| Sumario: | We compare the performance of state-of-the-art Large Language Models on a recently released benchmarking set for automated question answering for Icelandic and compare it with performance on questions from an Icelandic trivia game. We find that the models perform worse for questions on Icelandic subjects, specifically Icelandic culture, but somewhat surprisingly do better on a trivia game for people than on the benchmark set meant for language models, built around data that the model has seen during training. We also call into question some aspects of the benchmarking set and discuss what playing trivia games can tell us - if anything - about the capabilities of these models.
|
|---|