DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines
Research output: Contribution to conferences › Paper › Contributed › peer-review
Contributors
Abstract
Integrated data analysis (IDA) pipelines-that combine data management (DM) and query processing, high-performance computing (HPC), and machine learning (ML) training and scoring-become increasingly common in practice. Interestingly, systems of these areas share many compilation and runtime techniques, and the used-increasingly heterogeneous-hardware infrastructure converges as well. Yet, the programming paradigms, cluster resource management, data formats and representations, as well as execution strategies differ substantially. DAPHNE is an open and extensible system infrastructure for such IDA pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware (HW) accelerators, and computational storage for increasing productivity and eliminating unnecessary overheads. In this paper, we make a case for IDA pipelines, describe the overall DAPHNE system architecture, its key components, and the design of a vectorized execution engine for computational storage, HW accelerators, as well as local and distributed operations. Preliminary experiments that compare DAPHNE with MonetDB, Pandas, DuckDB, and TensorFlow show promising results.
Details
| Original language | English |
|---|---|
| Publication status | Published - Jan 2022 |
| Peer-reviewed | Yes |
Conference
| Title | 12th Annual Conference on Innovative Data Systems Research |
|---|---|
| Abbreviated title | CIDR 2022 |
| Conference number | 12 |
| Duration | 9 - 12 January 2022 |
| Website | |
| Location | Chaminade Resort and Spa & Online |
| City | Santa Cruz |
| Country | United States of America |
External IDs
| ORCID | /0000-0001-8107-2775/work/176861684 |
|---|