Detecting educational policy reforms at scale using Retrieval Augmented Generation

Abstract

International surveys and assessments can measure educational efficiency, but often fail to find policies or external factors that cause changes in efficiency. Large-scale data on policy reforms is necessary to understand the impact of reforms and identifying reforms that positively benefit education. The World Education Reform Database (WERD) is a comprehensive, manually coded database on educational policy reforms. However due to the large scale and manual nature in the construction of this database, WERD can introduce human errors and limits scalability and update-ability.

In this paper, we propose EduPoliRAG, a large language model-based Retrieval-Augmented Generation (RAG) system designed to semi-automate the large scale detection and extraction of educational policies. EduPoliRAG is capable of dealing with many large highly in-depth policy documents in multiple languages simultaneously. Built upon GPT-4o, EduPoliRAG integrates a corpus of international and national policy reports on the Dutch education system in order to generate tabular data on policy reforms. Evaluation is conducted through a manual comparison of EduPoliRAG's outputs against the WERD database. While EduPoliRAG successfully generates structured tabular data on policy reforms across various education sectors and addresses some of the problems of WERD, further refinement is needed to improve the completeness and semantic coverage.