DataStreamAdapt: Unified Detection Framework for Gradual and Abrupt Concept Drifts

Abstract

Concept drift, the phenomenon where data distributions change over time, poses a significant challenge to maintaining the accuracy and reliability of predictive models in data stream environments. Traditional drift detectors often struggle to simultaneously handle both abrupt and gradual drifts, leading to delayed adaptation or excessive false positives. This study proposes DataStream Adapt, a unified and adaptive framework designed to detect and respond to both abrupt and gradual concept drifts in real-time data streams. The framework integrates a hybrid drift detection engine combining error-rate monitoring and statistical divergence, an adaptive threshold controller that adjusts sensitivity based on stream volatility, and a drift-aware ensemble classifier capable of reweighting or replacing base learners dynamically. The system is benchmarked using the synthetic Hyperplane dataset, designed to simulate controlled drift scenarios with known ground truths. Experimental results demonstrate that DataStream Adapt outperforms state-of-the-art baselines, including DDM, ADWIN, and EDDM. Specifically, it achieves a detection delay of 31.2 instances for abrupt drift and 64.8 instances for gradual drift, compared to 82.4 and 254.6 for DDM, respectively. The framework maintains a false positive rate of 0.041, significantly lower than ADWIN’s 0.147, while also achieving an F1-score of 0.89 on post-drift classification, outperforming all baselines. In conclusion, DataStream Adapt offers a scalable, interpretable, and low-latency solution for adaptive learning in evolving data environments, making it suitable for real-world deployment in applications such as fraud detection, predictive maintenance, and IoT analytics