DEBS Grand Challenge: Real Time Data Analysis of Taxi Rides using StreamMine3G
Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/Gutachten › Beitrag in Konferenzband › Beigetragen › Begutachtung
Beitragende
Abstract
In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G followed by a thorough description of our implementation for the two queries that provide continuously up-to-date information about (i) the top-k most frequently driven routes and (ii) most profitable areas.
Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1 ms.
Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1 ms.
Details
Originalsprache | Englisch |
---|---|
Titel | Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems |
Erscheinungsort | New York, NY, USA |
Herausgeber (Verlag) | ACM Press |
Seiten | 269-276 |
Seitenumfang | 8 |
ISBN (Print) | 978-1-4503-3286-6 |
Publikationsstatus | Veröffentlicht - 2015 |
Peer-Review-Status | Ja |
Publikationsreihe
Reihe | DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems |
---|
Externe IDs
Scopus | 84960952490 |
---|
Schlagworte
Forschungsprofillinien der TU Dresden
DFG-Fachsystematik nach Fachkollegium
Schlagwörter
- CEP, ESP, complex event processing, event stream processing, fault tolerance, migration, scalability, state management, Distributed Systems, Distributed applications, Algoriths, Design, Reliability, Complex Event Processing, CEP, Event Stream Processing, ESP, Scalability, Migration, State Management, Fault Tolerance