DEBS Grand Challenge: Real Time Data Analysis of Taxi Rides using StreamMine3G

Research output: Contribution to book/conference proceedings/anthology/reportConference contributionContributedpeer-review

Contributors

Abstract

In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G followed by a thorough description of our implementation for the two queries that provide continuously up-to-date information about (i) the top-k most frequently driven routes and (ii) most profitable areas. Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1ms.

Details

Original languageEnglish
Title of host publicationProceedings of the 9th ACM International Conference on Distributed Event-Based Systems
Place of PublicationNew York, NY, USA
PublisherACM Press
Pages269-276
Number of pages8
ISBN (print)978-1-4503-3286-6
Publication statusPublished - 2015
Peer-reviewedYes

Publication series

SeriesDEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

External IDs

Scopus 84960952490

Keywords

Research priority areas of TU Dresden

DFG Classification of Subject Areas according to Review Boards

Keywords

  • CEP, ESP, complex event processing, event stream processing, fault tolerance, migration, scalability, state management, Distributed Systems, Distributed applications, Algoriths, Design, Reliability, Complex Event Processing, CEP, Event Stream Processing, ESP, Scalability, Migration, State Management, Fault Tolerance