What does your brain do when you listen to an audio book?  How and where does comprehension take place?

Coupling fMRI, which can identify active brain areas, and formal grammars, tools that can define the ways a language uses words, researchers form neurocomputational models of sentence comprehension.  So far, these models are limited.  There is not one model that reveals the total story of what the brain is doing while, for example, you listen to an audio book.  However, a recent paper offers new clues to building better models.

Researchers, including a UGA linguistics professor and graduate student, have published a paper that compares two grammars that are used in AI systems– context-free grammar (CFG) and combinatory categorial grammar (CCG).  They found that CCG-derived measures offer a better match to functional magnetic resonance images that were acquired while human participants listened to an audio book.

John Hale, UGA Arch Professor in World Languages and Cultures, and graduate student Donald Dunagan, UGA doctoral candidate in linguistics, are co-authors of the paper.

The researchers used CCG in a special way to enable it to deal with sentences like “Mary reads papers daily” without backtracking.

“Words like ‘daily’ in that sentence are hard for AI systems to anticipate," Hale said. "They incrementally process the sentence word-by-word and have to backtrack when they are surprised by something at the end that changes the structure of the sentence — and the meaning.”

Dunagan underlines that AI systems have to be flexible enough to deal with optional words like “daily”.  And a special computational operation, REVEAL, introduced by co-author Miloš Stanojević does just that.

The group also analyzed word-by-word predictions from a Large Language Model (LLM) similar to the one used in ChatGPT.  Comparing the ability of CCG and the LLM to account for brain activity, the group finds that CCG is superior.

The group’s paper, Modeling Structure-Building in the Brain with CCG Parsing and Large Language Models, appears in the current issue of the journal Cognitive Science.  Besides Hale and Dunagan, the authors include, Miloš Stanojević, Google DeepMind; Jonathan Brennan, Univ. of Michigan; and Mark Steedman, University of Edinburgh.