Resilient store: A heuristic-based data format selector for intermediate results

Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/GutachtenBeitrag in KonferenzbandBeigetragenBegutachtung

Beitragende

  • Rana Faisal Munir - , UPC Universitat Politècnica de Catalunya (Barcelona Tech) (Autor:in)
  • Oscar Romero - , UPC Universitat Politècnica de Catalunya (Barcelona Tech) (Autor:in)
  • Alberto Abelló - , UPC Universitat Politècnica de Catalunya (Barcelona Tech) (Autor:in)
  • Besim Bilalli - , UPC Universitat Politècnica de Catalunya (Barcelona Tech) (Autor:in)
  • Maik Thiele - , Technische Universität Dresden (Autor:in)
  • Wolfgang Lehner - , Technische Universität Dresden (Autor:in)

Abstract

Large-scale data analysis is an important activity in many organizations that typically requires the deployment of data-intensive workflows. As data is processed these workflows generate large intermediate results, which are typically pipelined from one operator to the following. However, if materialized, these results become reusable, hence, subsequent workflows need not recompute them. There are already many solutions that materialize intermediate results but all of them assume a fixed data format. A fixed format, however, may not be the optimal one for every situation. For example, it is well-known that different data fragmentation strategies (e.g., horizontal and vertical) behave better or worse according to the access patterns of the subsequent operations. In this paper, we present ResilientStore, which assists on selecting the most appropriate data format for materializing intermediate results. Given a workflow and a set of materialization points, it uses rule-based heuristics to choose the best storage data format based on subsequent access patterns.We have implemented ResilientStore for HDFS and three different data formats: SequenceFile, Parquet and Avro. Experimental results show that our solution gives 18% better performance than any solution based on a single fixed format.

Details

OriginalspracheEnglisch
Titel Model and Data Engineering
Redakteure/-innenÓscar Pastor, Jesús M. Almendros Jiménez, Yamine Aït-Ameur, Ladjel Bellatreche
Herausgeber (Verlag)Springer Verlag
Seiten42-56
Seitenumfang15
ISBN (Print)9783319455464
PublikationsstatusVeröffentlicht - 2016
Peer-Review-StatusJa
Extern publiziertJa

Publikationsreihe

ReiheLecture Notes in Computer Science, Volume 9893
ISSN0302-9743

Konferenz

Titel6th International Conference on Model and Data Engineering, MEDI 2016
Dauer21 - 23 September 2016
StadtAlmeria
LandSpanien

Externe IDs

ORCID /0000-0001-8107-2775/work/142253538

Schlagworte

Schlagwörter

  • Big data, Data format, Data-intensive workflows, HDFS, Intermediate results

Bibliotheksschlagworte