Pipeline Group Optimization on Disaggregated Systems

Research output: Contribution to conferencesPaperContributed

Contributors

Abstract

While hardware disaggregation is considered the "next big thing" providing unique opportunities for database systems, the pipeline-based execution model is state-of-the-art in modern query engines on monolithic systems. Within this paper, we propose a lightweight way of adapting this pipeline-based model to disaggregated memory systems to soften the inherent overhead induced by arbitrary memory accesses. Instead of executing pipelines in strict isolation including a pipeline-local data transfer, we group pipelines with similar data access characteristics of concurrently running queries into pipeline groups. Each such pipeline group is then executed separately, but shared data across pipelines within each group is only transferred once from memory resources to compute resources and potentially re-used multiple times. This method dramatically reduces redundant data transfers and - in combination with a suitable caching strategy as well as a fast communication layer - increases the performance significantly in comparison to traditional pipeline-based execution of multiple queries.

Details

Original languageEnglish
Publication statusPublished - 2023
Peer-reviewedNo

Conference

Title13th Annual Conference on Innovative Data Systems Research
Abbreviated titleCIDR 2023
Conference number13
Duration8 - 11 January 2023
Website
LocationMövenpick Hotel Amsterdam City Centre & Online
CityAmsterdam
CountryNetherlands

External IDs

Scopus 85166217989