MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data
Research output: Contribution to conferences › Paper › Contributed › peer-review
Contributors
Abstract
In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.
Details
Original language | English |
---|---|
Number of pages | 6 |
Publication status | Published - 2019 |
Peer-reviewed | Yes |
External IDs
ORCID | /0000-0002-8491-770X/work/141543283 |
---|---|
ORCID | /0000-0003-1066-361X/work/142248189 |
ORCID | /0009-0003-0666-4166/work/151475577 |
ORCID | /0000-0002-5437-3887/work/154740504 |
Keywords
Sustainable Development Goals
Keywords
- Time series