Publications

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza

Transactions of the Association for Computational Linguistics (TACL)

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction

Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze

Northern European Journal of Language Technology (NEJLT)

The caused-motion construction (CMC, "She sneezed the foam off her cappuccino") is one of the most well-studied constructions in Construction Grammar (CxG). It is a prime example for describing how constructions must carry meaning, as otherwise the fact that "sneeze" in this context takes two arguments and causes motion cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at a statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and LLMs can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate OpenAI, Gemma3, Llama3, OLMo2, Mistral and Aya models for their understanding of the CMC using the newly collected corpus. We find that most models struggle with understanding the motion component that the CMC adds to a sentence.

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David Mortensen, Lori Levin

LREC-COLING 2024

In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads.

Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model

Leonie Weissweiler*, Valentin Hofmann*, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schütze, Kemal Oflazer, David Mortensen

EMNLP 2023

Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. However, there have been relatively few systematic inquiries into the linguistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ignore the remarkable ability of humans to generalize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of human language, like morphology. Here, we close these gaps by conducting the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages (specifically, English, German, Tamil, and Turkish). We apply a version of Berko's (1958) wug test to ChatGPT, using novel, uncontaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English. Overall, our results — through the lens of morphology — cast a new light on the linguistic capabilities of ChatGPT, suggesting that claims of human-like language skills are premature and misleading.

Construction Grammar Provides Unique Insight into Neural Language Models

Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, Hinrich Schütze

First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)

Construction Grammar (CxG) has recently been used as the basis for probing studies that have investigated the performance of large pretrained language models (PLMs) with respect to the structure and meaning of constructions. In this position paper, we make suggestions for the continuation and augmentation of this line of research. We look at probing methodology that was not designed with CxG in mind, as well as probing methodology that was designed for specific constructions. We analyse selected previous work in detail, and provide our view of the most important challenges and research questions that this promising new field faces.

The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, Hinrich Schütze

EMNLP 2022

Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC). We conduct experiments examining the classification accuracy of a syntactic probe on the one hand and the models' behaviour in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning. While human-like performance of PLMs on many NLP tasks has been alleged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge.

Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

Wesley Scivetti, Ethan Gotlieb Wilcox, Nathan Schneider, Kanishka Misra, Leonie Weissweiler

30th Conference on Computational Natural Language Learning (CoNLL)

Proceedings of the Second International Workshop on Construction Grammars and NLP

Claire Bonial, Melissa Torgbi, Leonie Weissweiler, Austin Blodgett, Katrien Beuls, Paul Van Eecke, Harish Tayyar Madabushi

Proceedings of the Second International Workshop on Construction Grammars and NLP

Findings of the UniDive 2025 shared task on multilingual Morpho-Syntactic Parsing

Omer Goldman, Leonie Weissweiler, Kutay Acar, Diego Alves, Anna Bączkowska, Gülşen Eryiğit, Lenka Krippnerová, Adriana Pagano, Tanja Samardzic, Luigi Talamo, Alina Wróblewska, Daniel Zeman, Joakim Nivre, Reut Tsarfay

Proceedings of The UniDive 2025 Shared Task on Multilingual Morpho-Syntactic Parsing

BabyLM's First Constructions: Causal interventions provide a signal of learning

Joshua Rozner, Leonie Weissweiler, Cory Shain

Empirical Methods in Natural Language Processing (EMNLP)

MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs

Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza

Transactions of the Association for Computational Linguistics (TACL)

Both direct and indirect evidence contribute to dative alternation preferences in language models

Qing Yao, Kanishka Misra, Leonie Weissweiler, Kyle Mahowald

Conference on Language Modeling (COLM)

Constructions are revealed in word distributions

Joshua Rozner, Leonie Weissweiler, Kyle Mahowald, Cory Shain

Empirical Methods in Natural Language Processing (EMNLP)

Linguistic generalizations are not rules: Impacts on evaluation of LMs

Leonie Weissweiler, Kyle Mahowald, Adele Goldberg

Second International Workshop on Construction Grammars and NLP (CxG+NLP)

Derivational Morphology Reveals Analogical Generalization in Large Language Models

Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze, Janet Pierrehumbert

Proceedings of the National Academy of Sciences (PNAS)

SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation

Raoyuan Zhao, Abdullatif Köksal, Yihong Liu, Leonie Weissweiler, Anna Korhonen, Hinrich Schütze

Findings of Empirical Methods in Natural Language Processing (EMNLP)

Models Can and Should Embrace the Communicative Nature of Human-Generated Math

Sasha Boguraev, Ben Lipkin, Leonie Weissweiler, Kyle Mahowald

Workshop on Mathematical Reasoning and AI, NEURIPS

Proceedings of the Sixth Workshop on Teaching NLP

Sana Al-Azzawi, Laura Biester, György Kovács, Ana Marasović, Leena Mathur, Margot Mieskes, Leonie Weissweiler

Proceedings of the Sixth Workshop on Teaching NLP

Collaborative Development of Modular Open Source Educational Resources for Natural Language Processing

Matthias Aßenmacher, Andreas Stephan, Leonie Weissweiler, Erion Çano, Ingo Ziegler, Marwin Härttrich, Bernd Bischl, Benjamin Roth, Christian Heumann, Hinrich Schütze

Sixth Workshop on Teaching NLP

Computational Approaches to Construction Grammar and Morphology

Leonie Weissweiler

LMU Munich

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R Mortensen, Lori Levin

Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)

Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

David R Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler

Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)

UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependencies

Leonie Weissweiler, Nina Böbel, Kirian Guiller, Santiago Herrera, Wesley Scivetti, Arthur Lorenzi, Nurit Melnik, Archna Bhatia, Hinrich Schütze, Lori Levin, Amir Zeldes, Joakim Nivre, William Croft, Nathan Schneider

Joint International Conference on Computational Linguistics, Language Resources and Evaluation

Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction

Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze

Northern European Journal of Language Technology (NEJLT)

Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model

Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schütze, Kemal Oflazer, David R Mortensen

Empirical Methods in Natural Language Processing (EMNLP)

Explaining Pretrained Language Models' Understanding of Linguistic Structures using Construction Grammar

Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, Hinrich Schütze

Frontiers in Artificial Intelligence

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives

Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank

Association for Computational Linguistics (ACL)

Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs

Yihong Liu, Haotian Ye, Leonie Weissweiler, Hinrich Schütze

Findings of Empirical Methods in Natural Language Processing (EMNLP)

A Crosslingual Investigation of Conceptualization in 1335 Languages

Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, Hinrich Schütze

Association for Computational Linguistics (ACL)

4.4 WG4: Finding Idiosyncrasy in Corpora

Nurit Melnik, Francis Bond, Maria Koptjevskaja-Tamm, Peter Ljunglöf, Alexandre Rademaker, Agata Savary, Leonie Weissweiler

Report from Dagstuhl Seminar 23191: Universals of Linguistic Idiosyncrasy in Multilingual Computational Linguistics

Construction Grammar Provides Unique Insight into Neural Language Models

Leonie Weissweiler, Taiqi He, Naoki Otani, David R Mortensen, Lori Levin, Hinrich Schütze

CxGs + NLP Workshop, Georgetown University Round Table (GURT)

The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, Hinrich Schütze

Empirical Methods in Natural Language Processing (EMNLP)

CaMEL: Case Marker Extraction without Labels

Leonie Weissweiler, Valentin Hofmann, Masoud Jalili Sabet, Hinrich Schütze

Association for Computational Linguistics (ACL)

Developing a stemmer for German based on a comparative analysis of publicly available stemmers

Leonie Weissweiler, Alexander Fraser

German Society for Computational Linguistics (GSCL)