Towards a High-Resolution Mapping of Articulatory Kinematics in the Sensorimotor Cortex: Integrating Electrocorticography and Real-Time MRI

Abstract

Locked-in syndrome (LIS) is a condition where individuals are fully conscious but unable to communicate verbally due to complete paralysis of voluntary muscles, except for those controlling the eyes. For individuals with LIS, communication technology is vital to improving quality of life. Speech brain-computer interface (BCI) technologies offer a promising pathway for restoring communication by decoding neural signals directly into speech. However, effective speech BCIs require a detailed understanding of how speech is represented in the brain, specifically through the articulatory movements involved in speech production.

This thesis aims to establish a foundational mapping between articulatory kinematics and neural activity within the sensorimotor cortex (SMC), using electrocorticography (ECoG) and real-time magnetic resonance imaging (rtMRI). Unlike prior approaches that relied on phoneme-based or electromagnetic articulography (EMA) methods—which often provide incomplete or abstract representations of speech—rtMRI enables a comprehensive, continuous view of articulatory dynamics.

We utilized an autoencoder model, previously developed by Stolwijk (2022) and Wiemer (2024), to generate lower-dimensional representations of rtMRI data and assessed their alignment with ECoG recordings from a single participant. Our experiments identified an autoencoder configuration that effectively aligned rtMRI-based articulatory representations with ECoG data, suggesting that detailed articulatory kinematics are more accurately represented in the SMC than phoneme-level units. Incorporating neural-specific loss functions significantly enhanced this alignment, providing valuable insights into the neural representation of speech production.

The findings of this study provide a crucial step toward developing more effective speech BCIs for individuals with severe motor impairments. By focusing on the direct mapping of articulatory kinematics rather than abstract phonemic categories, future BCIs have the potential to generate more natural, fluent, and effective speech, offering an enhanced communication channel for those with LIS and other similar conditions.