DEBS Grand Challenge: Real Time Data Analysis of Taxi Rides using StreamMine3G

Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/GutachtenBeitrag in KonferenzbandBeigetragenBegutachtung

Beitragende

Abstract

In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G followed by a thorough description of our implementation for the two queries that provide continuously up-to-date information about (i) the top-k most frequently driven routes and (ii) most profitable areas.

Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1 ms.

Details

OriginalspracheEnglisch
TitelProceedings of the 9th ACM International Conference on Distributed Event-Based Systems
ErscheinungsortNew York, NY, USA
Herausgeber (Verlag)ACM Press
Seiten269-276
Seitenumfang8
ISBN (Print)978-1-4503-3286-6
PublikationsstatusVeröffentlicht - 2015
Peer-Review-StatusJa

Publikationsreihe

ReiheDEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

Externe IDs

Scopus 84960952490

Schlagworte

Forschungsprofillinien der TU Dresden

DFG-Fachsystematik nach Fachkollegium

Schlagwörter

  • CEP, ESP, complex event processing, event stream processing, fault tolerance, migration, scalability, state management, Distributed Systems, Distributed applications, Algoriths, Design, Reliability, Complex Event Processing, CEP, Event Stream Processing, ESP, Scalability, Migration, State Management, Fault Tolerance