Connecting Software Specifications and Payroll Rules: Evaluating Sparse and BERT-Based Retrieval

Abstract

Each organization in the Netherlands that withholds taxes is legally required to file tax returns with the Dutch Tax Agency, which in turn sends these data to the Employee Insurance Agency (UWV). A tax return must comply with a set of rules. The technical specifications, intended for payrollsoftware developers, are contained in one document. Examples and explanations, intended for payroll administrators, are contained in another. When analysts or legal experts need to trace an informative rule back to a technical rule, they must manually locate relevant paragraphs as there is no tool to assist their search. We present a framework to automatically retrieve relevant paragraphs. As a baseline, TF–IDF achieved lower Recall@ 10 than BM25 across all metrics. BM25 reached Recall@10 = 0.55 on our manually annotated test set. A BERT bi-encoder, when applied as a reranker over BM25 results, achieved R@10 = 0.42. After domain-adaptive and denoising pre-training, the same bi-encoder reached R@10 = 0.44, yet both scores still trail the BM25 baseline of 0.55.