Agent-Based Pyramid: A Model for Augmenting ASReview's Capability in Outlier Detection

Abstract

The exponential growth of academic literature presents significant challenges for researchers conducting systematic reviews, particularly in fields requiring exhaustive coverage such as medicine. This paper presents an improved algorithm based on ASReview for identifying outlier papers that might be overlooked by traditional screening methods. While machine learning-based active learning systems excel at finding similar documents, they often miss relevant papers with diverse characteristics in sparse feature regions.

We propose an Agent-Based Pyramid (ABP) structure that dynamically complements ASReview with systematic exploration capabilities. ABP operates in two modes after the stopping rule for the active learning has been met: (1) a manual mode where researchers input keywords for targeted exploration based on domain expertise, and (2) an automated mode that extracts predicted words of importance from ASReview's Random Forest classifier. The system monitors performance and automatically switches to agent-based exploration when detecting potential blind spots using these predicted words of importance to separate them from the manual keywords and to avoid any confusion with the RF model also being used for the prediction of relevance of records. Documents are organized hierarchically based on outlier scores calculated from either user keywords or machine-learned features.

Main innovations include flexible dual-mode operation (manual/automated), adaptive ML-agent switching, and hierarchical document organization with election-based prioritization. Experiments on SYNERGY datasets show superior performance compared to pure ASReview, with improved recall curves for finding 100% relevant papers in simulations across most datasets. The manual mode provides essential control for domain-specific reviews, while the automated mode demonstrates the framework's effectiveness without human intervention. This versatility makes ABP a valuable tool for comprehensive systematic reviews in high-stakes domains where missing critical studies has serious consequences.