Examining Computational Performance of Unsupervised Concept Drift Detection: A Survey and Beyond

Research output: Preprint/documentation/report › Preprint

Abstract

Concept drift detection is crucial for many AI systems to ensure the system's reliability. These systems often have to deal with large amounts of data or react in real time. Thus, drift detectors must meet computational requirements or constraints with a comprehensive performance evaluation. However, so far, the focus of developing drift detectors is on detection quality, e.g.~accuracy, but not on computational performance, such as running time. We show that the previous works consider computational performance only as a secondary objective and do not have a benchmark for such evaluation. Hence, we propose a set of metrics that considers both, computational performance and detection quality. Among others, our set of metrics includes the Relative Runtime Overhead RRO to evaluate a drift detector's computational impact on an AI system. This work focuses on unsupervised drift detectors, not being restricted to the availability of labeled data. We measure the computational performance based on the RRO and memory consumption of four available unsupervised drift detectors on five different data sets. The range of the RRO reaches from 1.01 to 20.15. Moreover, we measure state-of-the-art detection quality metrics to discuss our evaluation results and show the necessity of thorough computational performance considerations for drift detectors. Additionally, we highlight and explain requirements for a comprehensive benchmark of drift detectors. Our investigations can also be extended for supervised drift detection.

Details

Original languageEnglish
Publication statusPublished - 17 Apr 2023
No renderer: customAssociatesEventsRenderPortal,dk.atira.pure.api.shared.model.researchoutput.WorkingPaper

External IDs

ORCID /0000-0003-3137-0648/work/146643299
ORCID /0000-0003-2549-7626/work/146643869
ORCID /0000-0001-9756-6390/work/146644783
ORCID /0000-0001-6684-2890/work/146646274

Keywords

Keywords

  • cs.LG, cs.PF