Intelligent unit testing: how reasoning can improve automated testing
Summary
The thesis provides an alternative approach to automated software testing. This novel ap- proach is based on an intelligent BDI agent that is later extended in a multi-agent environment consisting on custom and third-party implementations. The BDI agents use logic, reasoning and a rule-based algorithmic process to achieve in-depth coverage. To do so, we introduce a new code coverage and code analysis approach that is based on a control flow graph that keeps track of the visited nodes, edges and edge-pairs.
The purpose of this study is to find whether and in what extend can BDI agents that are based on reasoning and logic, improve the current approaches of automated testing. To determine that, the research aims to find firstly how effective is a single BDI agent based on such symbolic AI approaches. Secondly, the research proceeds to determine how effective is a combination of other approaches, specifically T3 and Evosuite, when a BDI agent is used on top of those approaches. Lastly, the research examines whether this approach has any room for improvement and under what conditions.
The BDI agent and its testing process consist of three main parts: (i) analyze the control flow graph and initialize a knowledge base, (ii) interact with the target class randomly and update the knowledge base, (iii) apply logic, reasoning and a set of rules to construct all possible paths in each method. This process has been implemented for a single agent and was also divided into multiple agents with each one of them being responsible for only one part of this testing process. For the second part of the testing process, the implementations of T3 and Evosuite had been used to determine whether and reasoning part at the end has any influence to the process.
In the experiments the intelligent agent outperforms T3 in almost all cases. When the agents were used on top of T3 and Evosuite, certain cases showed that the agent can improve the coverage achieved by T3 and Evosuite. In fact, the results showed that the opposite is also possible. The intelligent agent can achieve better results when its random testing process is replaced with Evosuite or T3.