Giulimondi, 7165838. Talking machines and linguistic cognition: a critical review of the use of large language models in linguistic theorizing

Abstract

Increasingly, linguistic studies are employing LLMs to explain the underlying mechanisms of human linguistic cognition by applying experimental methods to LLMS that were previously adopted to test human participants (Hueb- ner et al., 2021; Beguš et al., 2023; Piantadosi, 2023; Goldstein et al., 2020). Computational cognitive scientists have argued that the assumptions under- lying these research choices are incorrect(Guest and Martin, 2023; van Rooij et al., 2023) and a growing body of linguists is taking a critical stance to- wards LLMs (Martínez et al., 2023; Kodner et al., 2023; Katzir, 2023; Bender and Koller, 2020; Bender et al., 2021). However, meta-theoretical linguistic research is still scarce, and, so far, no systematic analysis of language stud- ies using LLMs as experimental tools was conducted. This thesis aims to understand how the use of LLMs in research is affecting theory building in linguistics. More specifically, this analysis will focus on two research ques- tions: 1) What is the theoretical relation of LLMs to human cognition, when they are used for linguistic research? 2) How valid is the use of LLMs in linguistic theory? The thesis will review ten linguistic articles and argue that they share the assumption that LLMs represent an artificial replication of human lin- guistic cognition. Moreover, drawing from Guest and Martin (2023); Guest (2024) and Sullivan (2022) theoretical framework, it will be discussed how LLMs used to generate human-like linguistic behavior represent a theoretical misuse. It will be shown how this misuse of LLMs is motivated by an industry- driven research mindset (Ahmed et al., 2023), which may be at the root of the theoretical misconceptions hypothesized in the first inquiry of this study. This analysis is relevant for the understanding of language technology by language professionals and the possible systemic misinterpretations at play in research in a human-machine era.