DEBS Grand Challenge: Real Time Data Analysis of Taxi Rides using StreamMine3G
Research output: Contribution to book/conference proceedings/anthology/report › Conference contribution › Contributed › peer-review
Contributors
Abstract
In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G followed by a thorough description of our implementation for the two queries that provide continuously up-to-date information about (i) the top-k most frequently driven routes and (ii) most profitable areas. Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1ms.
Details
Original language | English |
---|---|
Title of host publication | Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems |
Place of Publication | New York, NY, USA |
Publisher | ACM Press |
Pages | 269-276 |
Number of pages | 8 |
ISBN (print) | 978-1-4503-3286-6 |
Publication status | Published - 2015 |
Peer-reviewed | Yes |
Publication series
Series | DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems |
---|
External IDs
Scopus | 84960952490 |
---|
Keywords
Research priority areas of TU Dresden
DFG Classification of Subject Areas according to Review Boards
Keywords
- CEP, ESP, complex event processing, event stream processing, fault tolerance, migration, scalability, state management, Distributed Systems, Distributed applications, Algoriths, Design, Reliability, Complex Event Processing, CEP, Event Stream Processing, ESP, Scalability, Migration, State Management, Fault Tolerance