StreamHub: A Massively Parallel Architecture for High-performance Content-based Publish/Subscribe
Research output: Contribution to conferences › Paper › Contributed › peer-review
Contributors
Abstract
By routing messages based on their content, publish/subscribe (pub/sub) systems remove the need to establish and maintain fixed communication channels. Pub/sub is a natural candidate for designing large-scale systems, composed of applications running in different domains and communicating via middleware solutions deployed on a public cloud. Such pub/sub systems must provide high throughput, filtering thousands of publications per second matched against hundreds of thousands of registered subscriptions with low and predictable delays, and must scale horizontally and vertically. As large-scale application composition may require complex publications and subscriptions representations, pub/sub system designs should not rely on the specific characteristics of a particular filtering scheme for implementing scalability. In this paper, we depart from the use of broker overlays, where each server must support the whole range of operations of a pub/sub service, as well as overlay management and routing functionality. We propose instead a novel and pragmatic tiered approach to obtain high-throughput and scalable pub/sub for clusters and cloud deployments. We separate the three operations involved in pub/sub and leverage their natural potential for parallelization. Our design, named StreamHub, is oblivious to the semantics of subscriptions and publications. It can support any type and number of filtering operations implemented by independent libraries. Experiments on a cluster with up to 384 cores indicate that StreamHub is able to register 150 K subscriptions per second and filter next to 2 K publications against 100 K stored subscriptions, resulting in nearly 400 K notifications sent per second. Comparisons against a broker overlay solution shows an improvement of two orders of magnitude in throughput when using the same number of cores.
Details
Original language | English |
---|---|
Pages | 63-74 |
Number of pages | 12 |
Publication status | Published - 2013 |
Peer-reviewed | Yes |
Conference
Title | 7th ACM International Conference on Distributed Event-based Systems (DEBS '13), ACM, 2013 |
---|---|
Abbreviated title | DEBS '13 |
Conference number | |
Duration | 29 June 2013 |
Degree of recognition | International event |
Location | |
City | Arlington |
Country | United States of America |
External IDs
Scopus | 84881148912 |
---|
Keywords
Research priority areas of TU Dresden
DFG Classification of Subject Areas according to Review Boards
Keywords
- publish/subscribe, Scalability, Performance