The influence of active learning model and prior knowledge choice on how long it takes to find hard-to-find relevant papers: Examining the variability of the time to discovery and the stability of its rank-orders

Abstract

["",""]Discovering "hard-to-find" relevant papers efficiently poses a challenge when employing active learning (AL)-assisted screening tools within the domain of systematic reviewing. Hard-to-find relevant papers if missed can have a significant impact on the outcomes of systematic reviews. The time to discovery (TD) measures how many records are needed to be screened in order to find a relevant paper, making it a useful tool to examine such papers. The main aim of this project was to investigate how AL model and prior knowledge choice influences the TD values of the hard-to find relevant papers and their rank-orders. A simulation study was conducted that consisted of two sets of simulations of the AL-aided screening process on a single dataset. The first set of simulations varied AL model across simulation runs, while keeping prior knowledge constant. The second set of simulations varied prior knowledge, while keeping AL model constant. The results demonstrated that AL model choice significantly influenced the TD values and the rank-order of the TD values of the hard-to-find relevant papers. Notably, the feature extractor had a larger impact on the TD values, as compared to the classifier. While prior knowledge choice did not significantly influence the TD values and the rank-order of the TD values. These findings demonstrate that AL model choice influences the difficulty of finding hard-to-find relevant papers, and that this should be considered when screening papers using an AL-aided tool. Furthermore, this paper highlights the use of the TD metric to examine the variation in the difficulty of finding relevant papers across different simulation setups. Future research should examine the characteristics of hard-to-find relevant papers to discover why they might take a long time to be found, and the effect of AL model choice on the TD values of such papers across multiple datasets in order to determine the generalisability of these findings.