MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data

Research output: Contribution to conferencesPaperContributedpeer-review

Abstract

In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.

Details

Original languageEnglish
Number of pages6
Publication statusPublished - 2019
Peer-reviewedYes

External IDs

ORCID /0000-0002-8491-770X/work/141543283
ORCID /0000-0003-1066-361X/work/142248189
ORCID /0009-0003-0666-4166/work/151475577
ORCID /0000-0002-5437-3887/work/154740504

Keywords

Sustainable Development Goals

Keywords

  • Time series