Latency-aware Elastic Scaling for Distributed Data Stream Processing Systems

Research output: Contribution to conferencesPaperContributedpeer-review



Elastic scaling allows a data stream processing system to react to a dynamically changing query or event workload by automatically scaling in or out. Thereby, both unpredictable load peaks as well as underload situations can be handled. However, each scaling decision comes with a latency penalty due to the required operator movements. Therefore, in practice an elastic system might be able to improve the system utilization, however it is not able to provide latency guarantees defined by a service level agreement (SLA).

In this paper we introduce an elastic scaling system, which optimizes the utilization under certain latency constraints defined by a SLA. Specifically, we present a model, which estimates the latency spike created by a set of operator movements. We use this model to built a latency-aware elastic operator placement algorithm, which minimizes the number of latency violations. We show that our solution is able to reduce the 90th percentile of the end to end latency by up to 30% and reduce the number of latency violations by 50%. The achieved system utilization for our approach is comparable to a scaling strategy, which does not use latency as optimization target.


Original languageEnglish
Number of pages10
Publication statusPublished - 2014


Title8th ACM International Conference on Distributed Event-Based Systems (DEBS '14), ACM, 2014
Abbreviated titleDEBS'14
Conference number
Duration26 - 29 May 2014
Degree of recognitionInternational event
CityNew York
CountryUnited States of America

External IDs

Scopus 84903161696


Research priority areas of TU Dresden

DFG Classification of Subject Areas according to Review Boards