Bernstein member involved: Patrick Krauß

The question of how we learn language is a controversial issue among scholars. “There are two main lines of thought that deliver completely contradictory answers,” explains Dr. Patrick Krauss, a researcher in cognition and neurosciences at the Chair of Pattern Recognition at FAU. Universal grammar, whose most famous supporter is the American linguist Noam Chomsky, assumes that grammatical principles are innate and that children develop their mother tongue on the basis of these principles. In contrast to this generic approach, cognitive language models propose that the structure of language only becomes apparent through using it.

Task: Which word comes next?

Together with Dr. Achim Schilling, visiting scientist at Uniklinikum Erlangen and group leader at Heidelberg University, Krauss investigated whether grammatical structures can be derived spontaneously through using language. However, the experiments were not conducted using humans, but rather artificial intelligence. They trained a recurrent neural network using the novel “Gut gegen Nordwind” by the Austrian author Daniel Glattauer. The task: After being given nine words, the AI was asked to predict the tenth word in a text.

“Predicting the next word, event or image is a basic principle of how the human brain works. It is therefore often referred to as a predictive machine,” Krauss explains. “Recurrent language models work in a similar way: They take previously entered information to improve their output.” The decisive feature of the experiment was that the neural network was not given any information concerning syntax, grammar or word classes.

Even the scientists were surprised by the results: “AI was correct in a remarkably high number of cases,” says Patrick Krauss. “It is impressive when you consider that we were looking for the exact word, not just one with a similar meaning.” Similar results were obtained by a second neural network trained with the English novel “A Hitchhiker’s Guide to the Galaxy” by Douglas Adams. In order to ensure that the AI is not just learning the sentences off by heart, for the test the models only received text passages that they had not read during training.

Spontaneous learning of word classes and syntax

In order to find out how the AI came to its results, the researchers analyzed the individual processing steps. “Recurrent neural networks are organized in bi-directional long-short-term memory layers,” Krauss explains. “These layers work like a memory and give us insights into the various interim stages in processing.” The results were surprising: The language models started to group the input sequences, in other words the nine consecutive words, into special word classes. At the end, they were able to predict with a high level of probability whether a tenth word would be a verb, a noun or an adjective although they did not possess any information about classifying language.

“Our results prove that abstract linguistic categories such as types of word and syntax or grammar rules can be created spontaneously during neural processing of linguistic input,” says Patrick Krauss. “These findings question the assumption that the ability to recognize and categorize words on the basis of their grammatical function is innate and hard-wired in the human brain.” Rather, language is a complex adaptive system that is influenced both by biological and environmental factors. The study, Krauss continues, not only delivers new findings about how information is processed in the human brain. It may also contribute to improving language models, machine translation and AI systems in general.