StreamHub: A Massively Parallel Architecture for High-performance Content-based Publish/Subscribe

Research output: Contribution to conferencesPaperContributedpeer-review

Contributors

Abstract

By routing messages based on their content, publish/subscribe (pub/sub) systems remove the need to establish and maintain fixed communication channels. Pub/sub is a natural candidate for designing large-scale systems, composed of applications running in different domains and communicating via middleware solutions deployed on a public cloud. Such pub/sub systems must provide high throughput, filtering thousands of publications per second matched against hundreds of thousands of registered subscriptions with low and predictable delays, and must scale horizontally and vertically. As large-scale application composition may require complex publications and subscriptions representations, pub/sub system designs should not rely on the specific characteristics of a particular filtering scheme for implementing scalability. In this paper, we depart from the use of broker overlays, where each server must support the whole range of operations of a pub/sub service, as well as overlay management and routing functionality. We propose instead a novel and pragmatic tiered approach to obtain high-throughput and scalable pub/sub for clusters and cloud deployments. We separate the three operations involved in pub/sub and leverage their natural potential for parallelization. Our design, named StreamHub, is oblivious to the semantics of subscriptions and publications. It can support any type and number of filtering operations implemented by independent libraries. Experiments on a cluster with up to 384 cores indicate that StreamHub is able to register 150 K subscriptions per second and filter next to 2 K publications against 100 K stored subscriptions, resulting in nearly 400 K notifications sent per second. Comparisons against a broker overlay solution shows an improvement of two orders of magnitude in throughput when using the same number of cores.

Details

Original languageEnglish
Pages63-74
Number of pages12
Publication statusPublished - 2013
Peer-reviewedYes

Conference

Title7th ACM International Conference on Distributed Event-based Systems (DEBS '13), ACM, 2013
Abbreviated titleDEBS '13
Conference number
Duration29 June 2013
Degree of recognitionInternational event
Location
CityArlington
CountryUnited States of America

External IDs

Scopus 84881148912

Keywords

Research priority areas of TU Dresden

DFG Classification of Subject Areas according to Review Boards

Keywords

  • publish/subscribe, Scalability, Performance