MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data

Publikation: Beitrag zu KonferenzenPaperBeigetragenBegutachtung

Abstract

In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.

Details

OriginalspracheEnglisch
Seitenumfang6
PublikationsstatusVeröffentlicht - 2019
Peer-Review-StatusJa

Externe IDs

ORCID /0000-0002-8491-770X/work/141543283
ORCID /0000-0003-1066-361X/work/142248189
ORCID /0009-0003-0666-4166/work/151475577
ORCID /0000-0002-5437-3887/work/154740504

Schlagworte

Ziele für nachhaltige Entwicklung

Schlagwörter

  • Time series