Automatically Configuring Parallelism for Hybrid Layouts
Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/Gutachten › Beitrag in Konferenzband › Beigetragen › Begutachtung
Beitragende
Abstract
Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.
Details
Originalsprache | Englisch |
---|---|
Titel | New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings |
Redakteure/-innen | Tatjana Welzer, Vili Podgorelec, Aida Kamišalic Latific, Johann Eder, Robert Wrembel, Mikolaj Morzy, Mirjana Ivanovic, Johann Gamper, Theodoros Tzouramanis, Jérôme Darmont |
Herausgeber (Verlag) | Springer Verlag |
Seiten | 120-125 |
Seitenumfang | 6 |
ISBN (Print) | 9783030302771 |
Publikationsstatus | Veröffentlicht - 2019 |
Peer-Review-Status | Ja |
Publikationsreihe
Reihe | Communications in Computer and Information Science |
---|---|
Band | 1064 |
ISSN | 1865-0929 |
Konferenz
Titel | 23rd European Conference on Advances in Databases and Information Systems |
---|---|
Kurztitel | ADBIS 2019 |
Veranstaltungsnummer | 23 |
Dauer | 8 - 11 September 2019 |
Ort | Hotel Park |
Stadt | Bled |
Land | Slowenien |
Externe IDs
Scopus | 85072990842 |
---|---|
ORCID | /0000-0001-8107-2775/work/142253463 |
Schlagworte
ASJC Scopus Sachgebiete
Schlagwörter
- Big data, Hybrid storage layouts, Parallelism, Parquet, Spark