SProBench: Stream Processing Benchmark for High Performance Computing Infrastructure

Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/GutachtenBeitrag in KonferenzbandBeigetragenBegutachtung

Abstract

Recent advancements in data stream processing frameworks have improved real-time data handling, however, scalability remains a significant challenge affecting throughput and latency. While studies have explored this issue on local machines and cloud clusters, research on modern high-performance computing (HPC) infrastructures is yet limited due to the lack of scalable measurement tools. This work presents SProBench, a novel benchmark suite designed to evaluate the performance of data stream processing frameworks in large-scale computing systems. Building on best practices, SProBench incorporates a modular architecture, offers native support for SLURM-based clusters, and seamlessly integrates with popular stream processing frameworks such as Apache Flink, Apache Spark Streaming, and Apache Kafka Streams. Experiments conducted on HPC clusters demonstrate its exceptional scalability, delivering throughput that surpasses existing benchmarks by more than tenfold. The distinctive features of SProBench, including complete customization options, built-in automated experiment management tools, seamless interoperability, and an open-source license, distinguish it as an innovative benchmark suite tailored to meet the needs of modern data stream processing frameworks.

Details

OriginalspracheEnglisch
TitelEuro-Par 2025: Parallel Processing
Redakteure/-innenWolfgang E. Nagel, Diana Goehringer, Pedro C. Diniz
Herausgeber (Verlag)Springer Science and Business Media B.V.
Seiten268-282
Seitenumfang15
ISBN (elektronisch)978-3-031-99872-0
ISBN (Print)978-3-031-99871-3
PublikationsstatusVeröffentlicht - 2026
Peer-Review-StatusJa

Publikationsreihe

ReiheLecture notes in computer science
Band15902 LNCS
ISSN0302-9743

Konferenz

Titel31st International Conference on Parallel and Distributed Computing
KurztitelEuro-Par 2025
Veranstaltungsnummer31
Dauer25 - 29 August 2025
Webseite
OrtTechnische Universität Dresden
StadtDresden
LandDeutschland

Schlagworte

Schlagwörter

  • Benchmark suite, HPC cluster, Slurm, Stream Processing