IJSDIP Logo

International Journal of System Design and Information Processing

psnajournals.org.in

Performance Optimization of Machine Learning Pipelines on Apache Spark for Real-Time Big Data Analytics

Paper ID:
IJSDIP_2013_VOLUME 1_ISSUE 3_1- 6
Author's Name:
Dileep Kumar Ghattamaneni and Narasimha Rao Boinapalli
Volume:
1
Issue:
Issue 3
Year:
2013
Pages:
1-6

Abstract

The development of big data has put pressure on traditional machine learning systems to process vast amounts of data in real-time. Apache Spark's ease of use, combined with its support for real-time analytics, made it popular after its launch in 2010, thanks to its in-memory processing and iterative algorithm capabilities. This paper outlines optimization methods for efficiently implementing scalable machine learning pipelines on Apache Spark. We investigate the role of RDDs, data partitioning, in-memory caching, and pipeline parallelism on performance. Benchmark results for k-means and logistic regression models demonstrated a 2.5 times speed increase compared to Hadoop MapReduce while performing similar tasks. Our research addresses incomplete real-time learning workflows and sets requirements for effective large-scale machine learning systems.

Keywords

Apache Spark, big data, machine learning, RDD, real-time analytics, pipeline optimization, in-memory computing.

References

[1] M. Zaharia et al., “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing.” [2] M. Zaharia, M. Chowdhury, M. J. Franklin, and S. Shenker, “Spark: Cluster Computing with Working Sets.” [3] M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica, “Discretized streams,” in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, New York, NY, USA: ACM, Nov. 2013, pp. 423–438. doi: 10.1145/2517349.2522737. [4] A. Agarwal, O. Chapelle, M. Dudík, and J. Langford, “A Reliable Effective Terascale Linear Learning System,” 2014. [Online]. Available: http://hadoop.apache.org/. [5] C.-T. Chu et al., “Map-Reduce for Machine Learning on Multicore.” [6] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The Google file system,” in Proceedings of the nineteenth ACM symposium on Operating systems principles, New York, NY, USA: ACM, Oct. 2003, pp. 29–43. doi: 10.1145/945445.945450. [7] J. Dean and S. Ghemawat, “MapReduce,” Commun ACM, vol. 51, no. 1, pp. 107–113, Jan. 2008, doi: 10.1145/1327452.1327492. [8] D. Borthakur, “The Hadoop Distributed File System: Architecture and Design,” 2007. [Online]. Available: http://hadoop.apache.org/core/. [9] Y. Bu et al., “Scaling Datalog for Machine Learning on Big Data,” Mar. 2012, [Online]. Available: http://arxiv.org/abs/1203.0160 [10] F. Niu, B. Recht, C. Ré, and S. J. Wright, “HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent.” [11] U. Kang, C. E. Tsourakakis, and C. Faloutsos, “PEGASUS: A Peta-Scale Graph Mining System Implementation and Observations,” in 2009 Ninth IEEE International Conference on Data Mining, IEEE, Dec. 2009, pp. 229–238. doi: 10.1109/ICDM.2009.14. [12] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad,” in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, New York, NY, USA: ACM, Mar. 2007, pp. 59–72. doi: 10.1145/1272996.1273005. [13] M. Sachan and A. Bhattacharya, “Mining statistically significant substrings using the chi-square statistic,” Proceedings of the VLDB Endowment, vol. 5, no. 10, pp. 1052–1063, Jun. 2012, doi: 10.14778/2336664.2336677. [14] N. Gal-Oz, T. Grinshpoun, and E. Gudes, “Privacy Issues with Sharing and Computing Reputation across Communities.” [15] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad,” in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, New York, NY, USA: ACM, Mar. 2007, pp. 59–72. doi: 10.1145/1272996.1273005. [16] Y. Zhou, D. Wilkinson, R. Schreiber, and R. Pan, “Large-Scale Parallel Collaborative Filtering for the Netflix Prize,” in Algorithmic Aspects in Information and Management, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 337–348. doi: 10.1007/978-3-540-68880-8_32. [17] P. Talebifard and V. C. M. Leung, “Context-Aware Mobility Management in Heterogeneous Network Environments.” [18] H. Hu, J. Xu, C. Ren, and B. Choi, “Processing private queries over untrusted data cloud through privacy homomorphism,” in 2011 IEEE 27th International Conference on Data Engineering, IEEE, Apr. 2011, pp. 601–612. doi: 10.1109/ICDE.2011.5767862. [19] R. Bekkerman, M. Bilenko, and J. Langford, “Scaling Up Machine Learning Parallel and Distributed Approaches.”