| dc.description.abstract | The growing interest in large language models like ChatGPT has opened new
possibilities for tasks requiring nuanced language understanding. ChatGPT's adaptability
makes it especially suitable for niche applications, even in the absence of task-specific
training data. In this context, Dutch health insurer Menzis is exploring AI-driven solutions to
reduce administrative workload. Menzis receives hospital-submitted action plans to monitor
and address excessive waiting times. These documents are typically written in free-text form
with varying style and expression, while Menzis requires the information in a standardized
format. This makes automated processing challenging and limits the potential for efficient,
scalable analysis.
This thesis investigates whether a carefully designed prompt can enable ChatGPT to
extract structured, semantically accurate information from such plans. A single prompt was
developed based on prompt engineering literature, iteratively refined within the ChatGPT
interface, and tested on a dataset of eight synthetic action plans and one real-world plan,
each synthetic one was crafted to reflect specific challenges. Evaluation focused on semantic
correctness, structural accuracy, hallucination resistance and robustness to variation, using a
qualitative approach.
The final prompt performed reliably across all test cases, achieving consistent
structural formatting and accurate content extraction with no observed hallucinations.
Minor variation in phrasing and interpretation occurred but the core information remained
intact. These findings suggest that prompt-based extraction is a promising method for
automating healthcare-related document processing. While human oversight remains
necessary for ambiguous or rare cases, further tuning based on output preferences may
reduce the need for manual review, supporting more efficient workflows at Menzis and
beyond. | |
