Anomaly Detection in Application Log Data

Abstract

Many applications within the Flexyz network generate a lot of log data. This data used to be difficult to reach and search. It was therefore not used unless a problem was reported by a user. One goal of this project was to make this data available in a single location so that it becomes easy to search and visualize it. Additionally, automatic analysis can be performed on the log data so that problems can be detected before users notice them. This analysis is the core of this project and is the topic of a case study in the domain of application log data analysis.

We compare four algorithms that take different approaches to this problem. We perform experiments with both artificial and real world data. It turns out that the relatively simple KNN algorithm gives the best performance, although it still produces a lot of false positives. However, there are several ways to improve these results in future research.