Comparing Eye-gaze and Transformer Attention Mechanisms in Reading Tasks

Abstract

As transformers become increasingly prevalent in NLP research, evaluating their cognitive alignment with human language processing has become essential for validating them as models of human language. This study compares eye-gaze patterns in human reading with transformer attention mechanisms to examine whether they can plausibly represent human attention during reading tasks. Our analysis validates previous findings with encoder models while extending the analysis to decoder architectures. We employ both statistical correlation analysis and predictive modeling using PCA-reduced representations of eye-tracking features across two reading tasks. We also examine the effect of different attention explanation methods (raw attention, attention flow, and gradient-based saliency) on the results. The findings reveal lower correlations and predictive capacity for the decoder model compared to the encoder model, with implications on the gap between behavioral performance and cognitive plausibility of different transformer designs.