From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models

While Large Language Models have improved rapidly, many fundamental questions remain about how to evaluate the knowledge and reasoning abilities they acquire, and how such evaluations relate to the learning signals used in pre-training. In this paper, we propose ModelLog, a declarative probabilistic