Automatically Configuring Parallelism for Hybrid Layouts
Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review
Contributors
Abstract
Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.
Details
Original language | English |
---|---|
Title of host publication | New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings |
Editors | Tatjana Welzer, Vili Podgorelec, Aida Kamišalic Latific, Johann Eder, Robert Wrembel, Mikolaj Morzy, Mirjana Ivanovic, Johann Gamper, Theodoros Tzouramanis, Jérôme Darmont |
Publisher | Springer Verlag |
Pages | 120-125 |
Number of pages | 6 |
ISBN (print) | 9783030302771 |
Publication status | Published - 2019 |
Peer-reviewed | Yes |
Publication series
Series | Communications in Computer and Information Science |
---|---|
Volume | 1064 |
ISSN | 1865-0929 |
Conference
Title | 23rd European Conference on Advances in Databases and Information Systems |
---|---|
Abbreviated title | ADBIS 2019 |
Conference number | 23 |
Duration | 8 - 11 September 2019 |
Location | Hotel Park |
City | Bled |
Country | Slovenia |
External IDs
Scopus | 85072990842 |
---|---|
ORCID | /0000-0001-8107-2775/work/142253463 |
Keywords
ASJC Scopus subject areas
Keywords
- Big data, Hybrid storage layouts, Parallelism, Parquet, Spark