Automatically Configuring Parallelism for Hybrid Layouts

Research output: Contribution to book/conference proceedings/anthology/reportConference contributionContributedpeer-review

Contributors

  • Rana Faisal Munir - , Chair of Databases, UPC Polytechnic University of Catalonia (Barcelona Tech) (Author)
  • Alberto Abelló - , UPC Polytechnic University of Catalonia (Barcelona Tech) (Author)
  • Oscar Romero - , UPC Polytechnic University of Catalonia (Barcelona Tech) (Author)
  • Maik Thiele - , Chair of Databases (Author)
  • Wolfgang Lehner - , Chair of Databases (Author)

Abstract

Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.

Details

Original languageEnglish
Title of host publicationNew Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings
EditorsTatjana Welzer, Vili Podgorelec, Aida Kamišalic Latific, Johann Eder, Robert Wrembel, Mikolaj Morzy, Mirjana Ivanovic, Johann Gamper, Theodoros Tzouramanis, Jérôme Darmont
PublisherSpringer Verlag
Pages120-125
Number of pages6
ISBN (print)9783030302771
Publication statusPublished - 2019
Peer-reviewedYes

Publication series

SeriesCommunications in Computer and Information Science
Volume1064
ISSN1865-0929

Conference

Title23rd European Conference on Advances in Databases and Information Systems
Abbreviated titleADBIS 2019
Conference number23
Duration8 - 11 September 2019
LocationHotel Park
CityBled
CountrySlovenia

External IDs

Scopus 85072990842
ORCID /0000-0001-8107-2775/work/142253463

Keywords

Keywords

  • Big data, Hybrid storage layouts, Parallelism, Parquet, Spark