Paper ID:
IJSDIP_2016_VOLUME 4_ISSUE 3_1-5
Author's Name:
Gergely Deák
Volume:
4
Issue:
Issue 3
Year:
2016
Pages:
01-05
Abstract
Big Data processing frameworks have revolutionized the way large-scale datasets are analyzed and processed. Apache Hadoop and Apache Spark are two leading technologies for batch processing, but they differ significantly in architecture and performance. This paper presents a comprehensive performance comparison between Hadoop MapReduce and Apache Spark using large-scale data processing benchmarks. We set up a 20-node cluster and execute Terasort and PageRank tasks to evaluate metrics such as execution time, CPU and memory usage, disk I/O, and fault recovery. Spark demonstrates a consistent performance advantage, completing jobs 3.7 times faster than Hadoop in most scenarios, primarily due to its in-memory computation model. Hadoop, while more disk-intensive, shows resilience in handling node failures thanks to its mature HDFS and speculative execution. We also evaluate ease of deployment and tuning complexity. Spark’s DAG-based engine offers superior optimization for iterative workloads, making it more suitable for machine learning and graph processing tasks. Conversely, Hadoop remains relevant for sequential batch jobs and legacy integration. The analysis concludes that while both frameworks have distinct strengths, Spark is generally more efficient for modern batch processing needs. Our findings provide practical guidance for organizations deciding between these two platforms for large-scale data analysis.
References
1. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster computing with working sets. Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, 10.
2. White, T. (2015). Hadoop: The Definitive Guide (4th ed.). O'Reilly Media.
3. Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., ... & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65.
4. Borthakur, D. (2007). The Hadoop Distributed File System: Architecture and design. Hadoop Project Website.
5. Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop Distributed File System. 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 1–10.
6. Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.
7. Lan, Y., Li, Q., Wu, J., & Wang, F. (2015). Comparative analysis of Hadoop and Spark frameworks on large-scale data processing. International Journal of Computer Applications, 113(1), 9–14.
8. Talluri Durvasulu, M. B. (2014). Understanding VMAX and PowerMax: A storage expert’s guide. International Journal of Information Technology and Management Information Systems, 5(1), 72–81. https://doi.org/10.34218/50320140501007
9. Karau, H., Konwinski, A., Wendell, P., & Zaharia, M. (2015). Learning Spark: Lightning-Fast Big Data Analysis. O'Reilly Media.
10. Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., ... & Baldeschwieler, E. (2013). Apache Hadoop YARN: Yet another resource negotiator. Proceedings of the 4th Annual Symposium on Cloud Computing, 1–16.
11. Guller, M. (2015). Big Data Analytics with Spark: A Practitioner's Guide to Using Spark for Large Scale Data Analysis. Apress.
12. Apache Software Foundation. (2016). Apache Spark Documentation. Retrieved from https://spark.apache.org
13. Apache Software Foundation. (2016). Apache Hadoop Documentation. Retrieved from https://hadoop.apache.org
14. Rezaei, M., & Yang, K. (2016). Analyzing the performance of Hadoop and Spark on big data applications. Journal of Computer Science, 12(7), 344–352.
15. Roy, S., & Mathew, A. (2015). Performance comparison of Hadoop and Spark for big data applications. 2015 International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM), 61–66.
16. Taleb, T., Dutta, S., Ksentini, A., & Iqbal, M. U. (2016). Data and service management in cloud-enabled big data environment. IEEE Network, 30(4), 10–16.
