IDEAL: Integrating the Detection of changes, Exploration of instance space, and Assessment of performance into Active Learning

Abstract

With Active Learning becoming more used in the context of data streams, there is also an increasing need to have an algorithm appropriate for this setting. As such, the active learner must be able to handle the challenges that arise in data stream-based settings.

This research aims to determine if an optimal Active Learning algorithm can detect changes unsupervised, assess its performance, and be representative of the distribution of the data stream. When combined, a trade-off in the weighting of the different components also needs to be considered

This master’s thesis proposes an active learning algorithm named IDEAL. IDEAL aims to operate in an evolving data stream where changes happen continuously based on three components. IDEAL aims to do this by Integrating the Detection of changes, Exploration of instance space, and Assessment of performance into Active Learning.

One synthetic data set and three real-world benchmark datasets have been used to evaluate the performance of IDEAL. IDEAL performed quite well, but it was not the most accurate Active Learning algorithm.

For future research, IDEAL is a good baseline for making a better Active Learning algorithm. The components’ weightings are scalable, and the option to add more or fewer components makes it easy to adjust the algorithm. More research into either the weightings between the three components or the Change Detector and Explorer components of IDEAL would be good.