BLiMP-NL: A corpus of Dutch minimal pairs and grammaticality judgements for language model evaluation

We present a corpus of Dutch 8400 sentence pairs intended for the grammatical evaluation of language models. Each pair has a grammatical sentence and a minimally different ungrammatical sentence. The corpus covers 84 paradigms, classified into 22 syntactic phenomena. Nine sentences of each paradigm were rated for acceptability by at least 30 participants, while self-paced reading time on each word was also recorded. We report on the grammaticality effects on acceptability ratings and reading times, as well as the extent to which language models' predictions match both the ground-truth grammaticality and human ratings.