BLAST: Benchmarking LLMs with ASP-based Structured Testing
arXiv:2604.22306v1 [cs.LO] 24 Apr 2026
Manuel Alejandro Borroto Santana1∗ , Erica Coppolillo1,2∗ , Francesco Calimeri1,3,4 , Giuseppe Manco1 , Simona Perri3 , Francesco Ricca3 1 University of Calabria, Rende, Italy 2 ICAR-CNR, Rende, Italy 3 Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy 4 DLVSystem, Rende, Italy {name.surname}@unical.it, {name.surname}@icar.cnr.it Abstract Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs.
Introduction Large Language Models (LLMs) (Zhao, Zhou, and et al., 2025) have been showing remarkable performance across a broad spectrum of tasks, including natural language understanding (Qin et al., 2024), dialogue systems (Ou et al., 2024), and code generation (Jiang et al., 2024). More recently, some efforts have been made to encompass advanced reasoning, knowledge representation, and logical formalisms (Borroto, Kareem, and Ricca, 2024; Lippolis et al., 2025; Valmeekam et al., 2025; Smirnov et al., 2024; Coppolillo et al., 2024). In this context, while several benchmarks have been proposed to assess LLM performance on imperative and web-oriented programming languages (e.g., C++, Java, HTML) (Ernst and Bavota, 2022; Peng et al., 2023; Dakhel et al., 2023), far less attention has been given to declarative paradigms such as Answer Set Programming (ASP) (Brewka, Eiter, and Truszczynski, 2011; Gelfond and Lifschitz, 1991), despite some recent initial efforts (Borroto, Kareem, and Ricca, 2024; Ishay, Yang, and Lee, 2023a; Coppolillo et al., 2024; Schrader et al., 2025). Rooted in logic programming and nonmonotonic reasoning, ASP is well-suited for modeling and solving complex AI problems (Brewka, Eiter, and Truszczynski, 2011). Over the years, ASP became increasingly adopted for its expressive language and the availability of efficient and reliable solvers (Gebser, Maratea, and Ricca, 2020), notably ∗
Equal contribution.
Clingo (Gebser et al., 2016) and DLV (Alviano et al., 2017). ASP as fully declarative paradigm has been successfully employed in both academic research and industrial applications, particularly in complex domains (e.g., scheduling, configuration, robotics, workforce planning, decision support) (Erdem, Gelfond, and Leone, 2016; Falkner et al., 2018). In recent years substantial effort has been devoted to developing programming environments and tools that support the creation of ASP specifications, including advanced editors, debuggers, testing frameworks, and utilities (Alviano, Cirimele, and Reiners, 2023; Hahn et al., 2023; Febbraro, Reale, and Ricca, 2011; Busoniu et al., 2013; Calimeri et al., 2018). Motivation. Despite these advancements, writing ASP code remains often challenging, especially for newcomers without a solid background in logic and mathematics. In this context, LLMs represent a promising avenue for supporting, and even automating, ASP coding. Recent studies have highlighted significant limitations in the ability of generalpurpose LLMs to generate correct and semantically sound ASP programs (Coppolillo et al., 2024). At the same time, other lines of research, including approaches specifically tailored to ASP encoding demonstrate that the overall direction is viable and promising (Ishay, Yang, and Lee, 2023a; Borroto, Kareem, and Ricca, 2024; Coppolillo et al., 2024; Schrader et al., 2025). However, the field still lacks systematic evaluations, standardized benchmarks, and well-defined metrics for rigorously assessing the ASP-coding capabilities of LLMs and, more generally, of any automatic approach. Contributions. To fill this gap, we introduce BLAST, a benchmarking methodology specifically designed to assess the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework that measures program quality through two novel metrics specifically designed for ASP code generation. The methodology encompasses both evaluation breadth (covering a diverse set of problems), and depth (testing robustness across multiple natural-language paraphrases of each problem). Crucially, BLAST imposes no guidance or syntactic constraints on input/output formats for the evaluated models. Moreover, existing procedures mostly rely on manual evaluations of the automated encoding outcomes (Borroto, Kareem, and Ricca, 2024; Coppolillo et al., 2024), resulting in time-consuming and potentially inconsistent assessments. In contrast, we
propose a systematic approach that automates the evaluation process, reducing human intervention while improving scalability, reliability and reproducibility. We assess BLAST via an experimental study that tackles ten well-known graph problems from the ASP literature using a pool of eight state-of-the-art LLMs. The main contribution of the present work can be summarized as follows: • We release a benchmark dataset with ten graph-related problems to be encoded in ASP. • We propose two novel evaluation methods to automatically assess the quality of the generated programs. Both approaches operate without imposing any constraints on the LLM’s input or output format, making the system robust to variations and renaming of input and output predicate symbols. • We evaluate eight state-of-the-art LLMs, from five different families, by considering the original problem description along with paraphrased versions, to assess the model flexibility in interpreting human-like crafted prompts. For the sake of reproducibility, the complete experimental setup, including code and evaluation data, is made publicly available.1
Related Work Automatic Code Synthesis. The advantages of automating code synthesis are well-recognized in the literature (Ernst and Bavota, 2022; Kalliamvakou, 2022; Peng et al., 2023; Dakhel et al., 2023), and automated program composition tools now support nearly all mainstream programming languages (Chen, Tworek, and et al., 2021). In this area, Large Language Models (LLMs) play a central role, and their performance has been extensively compared in the literature (Xu et al., 2022; Wang et al., 2023), while also the positive impact of fine-tuning models for code generation of imperative programming is established (Ma et al., 2024). In the context of declarative programming, a widely recognized research objective is to create tools that streamline and automate the development of ASP programs, thus bridging the gap between natural language specifications and ASP source code (Erdem and Yeniterzi, 2009; Fang and Tompits, 2017; Schwitter, 2018; Caruso et al., 2024). Early proposals focused on automating the resolution of logic puzzles presented in simplified English by translating their descriptions into ASP (Baral and Dzifcak, 2011), employing λcalculus and probabilistic combinatorial categorical grammars. Later, several efforts have been spent in the development of Controlled Natural Languages (CNLs) (Kuhn, 2014) for ASP programs; CNLs represent subsets of full natural languages, featuring restricted grammar and vocabulary. Among them, the BIOQUERYCNL (Erdem and Yeniterzi, 2009) defines the grammatical structure of a CNL and algorithm for transpiling queries into ASP; (Fang and Tompits, 2017) introduced a CNL approach for ASP that leverages LANA annotations, that was implemented in the SeaLion 1 https://anonymous.4open.science/r/ LLMs-ASP-Benchmark-DFC3/
IDE. We note that (Schwitter, 2018) developed a CNL called PENGASP for specifying and verbalizing ASP programs, while (Caruso et al., 2024) introduced CNL2ASP, an extensive publicly-available tool for converting controlled natural language into ASP programs.
ASP and LLMs. Powerful language tools such as LLMs have been exploited in conjunction with the ASP formalism in several ways. (Nye et al., 2021) introduced a dualsystem model based on GPT-3, which generates semantic parsers from natural language sentences and integrates them with reasoning modules. In a similar vein, (Yang, Ishay, and Lee, 2023) proposed that LLMs like GPT-3 can work as few-shot semantic parsers, transforming natural language into logical forms for ASP without necessitating distinct retraining for diverse question-answering tasks. (Ishay, Yang, and Lee, 2023b) utilized LLMs with prompt engineering to obtain ASP solutions for logic puzzles, leveraging the logic puzzle dataset from (Mitra and Baral, 2016). (Kareem et al., 2024) introduced LLM2LAS, which combines LLMs with ILASP (Law, Russo, and Broda, 2020) to learn commonsense knowledge from story-based Q&A, enabling the system to generalize with minimal examples and effectively answer unseen questions. Further, the effective use of LLMs in combination with ASP to perform a number of natural language understanding tasks has been explored and implemented in the STAR framework (Rajasekharan et al., 2023). Bauer et al. (2024) proposed NSGRAPH, an approach that integrates vision modules, LLM-based semantic parsing, and ASP reasoning for Visual Graph Question Answering, where LLMs extract ASP predicates from natural language questions that are then processed by an ASP solver. Beyond ASP, Voboril, Ramaswamy, and Szeider (2025) proposed StreamLLM, an approach that leverages LLMs to generate and validate streamlining constraints in MiniZinc models, achieving substantial speedups across multiple constraint satisfaction benchmarks. (Ren et al., 2025) introduced a benchmark focused on evaluating the ASP solving abilities of LLMs (i.e., entailment, verification and computation of answer sets). In contrast, BLAST focuses on the task of generating ASP code. (Borroto, Kareem, and Ricca, 2024) presented the NL2ASP tool, that constructs ASP programs from natural language specifications through a two-step architecture. NL2ASP works by translating the NL specifications into CNL statements, which are then converted into ASP code using the CNL2ASP tool. NL2ASP was implemented using well-known models for neural machine translation (NMT) and demonstrated promising performance. More recently, (Coppolillo et al., 2024) provides a first evaluation of LLMs performance in encoding simple, core ASP programs, further showing that LLMs are not good at ASP coding, but fine-tuning can be beneficial. However, no benchmark or methodology for systematic assessment of LLM in the general ASP coding task is provided (Coppolillo et al., 2024). Building on the potential of fine-tuning, (Schrader et al., 2025) introduced an ASP-solver-in-the-loop framework that improves LLM-based ASP code generation for combinatorial problems by using direct solver feedback
to curate data for supervised fine-tuning and to further improve robustness through a solver-guided search that includes best-of-N sampling. Evaluation Benchmarks. A substantial body of literature provides benchmarking datasets and test suites tailored to evaluating LLMs on imperative (e.g., Python, C++, Java) and web-oriented programming languages (e.g., HTML, CSS, JavaScript) such as (Xu et al., 2025; Shah, Genc, and Araci, 2025; Du et al., 2024; Chen, Tworek, and et al., 2021; Ding et al., 2023; Liu, Xu, and McAuley, 2023). Comparatively, we argue that much less attention has been devoted to declarative programming paradigms. A first benchmark for automatic translation from natural language to ASP was introduced by Borroto, Kareem, and Ricca (2024). However, it primarily focuses on controlled natural language (Caruso et al., 2024) and features input sentences with limited fluency. Recently, Ren et al. (2025) proposed a methodology to assess whether state-of-the-art LLMs can solve programs coded in ASP, comparing their capabilities to standard solvers. However, their goal (computing answer sets with LLMs) is different from ours (testing LLMs coding ability). To the best of our knowledge, no existing benchmark enables systematic automatic evaluation of LLMs ability to generate ASP programs. This work addresses this gap.
Preliminaries Large Language Model. A Large Language Model (LLM) is a function f that stochastically maps an input token sequence x = [x1 , . . . , xn ] to an output sequence y = [y1 , . . . , ym ], with x ∈ V ∗ and y ∈ W ∗ , where V and W are token vocabularies. Formally, f defines the conditional probability P (y|x), from which y is sampled, capturing the structure and semantics of natural language. LLMs typically adopt the Transformer encoder-decoder architecture (Vaswani et al., 2017). The input tokens are first embedded into dense vectors encoding contextual and positional information. These embeddings pass through the Transformer: the encoder applies self-attention and feedforward layers to model dependencies within the input, while the decoder combines masked self-attention, encoder-decoder attention, and feedforward transformations to generate outputs token by token. The decoder outputs a probability distribution over the output vocabulary via a linear layer followed by softmax: P (yi |x, y:i−1 ), where y:i−1 is the prefix up to token i−1. Generation proceeds stepwise from this distribution. Architectural variants exist (Naveed et al., 2024; Raiaan et al., 2024), but their internal design is orthogonal to our focus. LLMs are generally invoked via prompting: a textual query (or prompt) is provided as input to the model, which triggers content generation. Answer Set Programming. Answer Set Programming (ASP) is a declarative formalism for Knowledge Representation and Reasoning (Brewka, Eiter, and Truszczynski, 2011; Gelfond and Lifschitz, 1991). In ASP, a problem is encoded as a finite set of logical rules, and solutions correspond to
Input ASP
Target LLM
ASP Generation
Predicate Matcher ASP
ASP
Paraphraser
ASP
{ ... }
{ ... }
ASP Testing
{ ... }
Figure 1: Scheme of the overall proposed framework. Input consists of the textual specification of the problem, the target LLM to be evaluated, and the correct (gold) ASP program. The ASP Generation module comprises: the paraphraser, an LLM which paraphrases the original problem description in more human-styled texts; and the predicate matcher, an LLM which maps the predicates of the generated programs to the ones of the gold program. The predicate mappings and the gold encoding are finally provided to the ASP Testing module, which performs the evaluation.
answer sets. Rather than presenting a formal introduction to the language, we provide a representative example to highlight its expressive power: the classical 3-colorability problem, which requires assigning one of three colors to each node of an undirected graph such that adjacent nodes receive different colors. The following ASP encoding captures this specification. The input graph (nodes and edges) and the available colors are represented via facts. A choice rule assigns exactly one color to each node, and an integrity constraint eliminates colorings in which adjacent nodes share the same color: col(red). col(green). col(black). node(1..4). edge(1,2). edge(4,2). edge(1,3). 1 { colored(X,C) : col(C) } 1 :- node(X). :- edge(X,Y), colored(X,C), colored(Y,C).
ASP also provides constructs for optimization via weak constraints, which allow specifying preferences among answer sets. For instance, to minimize the number of nodes assigned the color red: :∼ colored(X,red). [1@1,X]
This example illustrates the declarative nature and expressiveness of ASP in modeling complex combinatorial problems. For a rigorous introduction to the ASP language and its semantics, please refer to the aforesaid literature.
BLAST Methodology The core of the approach consists in testing the generation capabilities of a target LLM. For this, we devise a methodology consisting of two main components: (i) an ASP Generation framework that, starting from a specification in natural language, provides a set of encodings of the problem in ASP; (ii) an ASP Testing infrastructure aimed at evaluating the performance of the target LLM. The overall scheme of the approach is detailed in Figure 1.
ASP Generation
1 {assign(N, C) : colour(C)} 1 :- node(N).
The framework for generating ASP initiates with a natural language description of the problem intended for ASP encoding, alongside a gold program that exemplifies the ASPbased solution. The generation process is driven by the following elements: (i) the target LLM, serving as the generator under evaluation, in charge of encoding the problem into ASP; (ii) an LLM instructed to paraphrase the initial problem description to make it more human-readable; (iii) an LLM that semantically matches the predicates of the resulting program with those in the gold program.
:- edge(X, Y), assign(X, C), assign(Y, C).
If there is no semantic match between some predicates, output ‘No semantic match’ only. Otherwise, produce the matches over all the predicates as a python dictionary. Do not generate any additional text.
The generated matching dictionary is the following: {‘node’:‘node’, ‘edge’:‘edge’, ‘color’:‘colour’, ‘chosen’:‘assign’}
Box 1: Target LLM Prompt Example [PROBLEM DESCRIPTION] Write an ASP program which models this problem, along with facts that make it instantiable. Generate the program only, without any additional text.
Predicate Matcher. It is important to remark that the target model is not given any instructions or constraints regarding the syntax, predicate names, or problem modeling to adopt. To normalize its output and enable comparison with the gold standard solution, we employ a further LLM, referred to as the Predicate Matcher. This model takes as input both the gold program and the encoding generated by the target model, and is prompted to perform a semantic match of the predicates used in the two programs. The result is returned as a Python dictionary mapping corresponding predicates. If no semantic similarity can be established between the programs, the model is instructed to return “No semantic match”. An example prompt of the predicate matcher is provided in Box 3, considering the Colorability problem. Box 3: Predicate Matcher Prompt Example Perform predicate matching on the following ASP programs based on the semantic similarity of the predicates: ASP Instance 1: node(1).
node(2).
node(3).
edge(1,2).
edge(1,3).
edge(2,1).
edge(2,3).
color(red).
color(green).
color(black). 1 {chosen(N,C) : color(C)} 1 :- node(N). :- edge(U,V), U < V, chosen(U,C), chosen(V,C).
ASP Instance 2: node(a).
node(b).
node(c).
edge(a, b).
edge(b, c).
colour(1).
colour(2).
edge(c, d). colour(3).
Gold Generated
0.5
0.0 Colorability Dominating set Hamiltonian cycle Hamiltonian path Hierarchical clustering Maximal clique Partition Slitherlink Stable marriage Traveling saleperson
Target LLM. As an initial step in our framework, the LLM under evaluation is prompted with the problem description and asked to provide an encoding in ASP language. Box 1 provides an example of the provided prompt. (Note that, this component can be replaced by any method generating ASP programs from natural language specifications.)
Semantics
1.0
Figure 2: Semantic comparison on the considered problems computed by using the “gold” matching dictionary (manually crafted by authors) and the generated one. DeepSeek-R1 has been used as the target LLM.
We emphasize three crucial aspects of the predicate matcher component. First, the predicate matching phase is independent of encoding correctness: the matcher LLM only semantically associates predicates between programs, without assessing their validity. Second, the matcher does not semantically alter a correct program. In other words, the quality of the target LLM performance (that is, its ability to generate a correct ASP program from a natural language problem description) remains consistent regardless of whether a “gold” predicate matching dictionary (manually crafted by the authors) or a generated one is used (see Figure 2, where we use DeepSeek-R1 as the representative target LLM). The only exception is the Traveling Salesperson problem, where performance is slightly better with the gold matching. By looking deeper into this problem, we realized that despite the better matching, the generated programs were still incorrect. Third, we remark that this result enables our methodology to achieve a high degree of generality, since it allows evaluating generated programs without imposing any prompting constraints on the LLMs. Indeed, we deliberately avoid penalizing encodings that differ only at the syntactic level, through the use of alternative predicate names. For instance, adopting reach(X) instead of reached(X) to represent the same concept should not negatively affect the evaluation, provided that the program is semantically correct. By abstracting away from superficial naming differences, our approach focuses on assessing genuine modeling and coding ability rather than sensitivity to arbitrary lexical choices.
Paraphraser. In addition to the technically detailed original problem statement, we further evaluate the adaptability of the target model to understand prompts that are formatted to be more relatable to humans. The goal is to evaluate the models not only in terms of breadth, by addressing a diverse set of problems, but also in terms of depth, by assessing their ability to adapt to less technical, more accessible textual descriptions intended for an “ASP layman”. To achieve this, we use another LLM as a paraphraser to generate two versions of the initial description. An example prompt is reported in Box 2 in the Supplementary Material. The paraphrasing process is conducted in two phases. Initially, a first version is derived from the original problem statement. This version is then employed as a basis for producing a second one. Both versions are subsequently used to prompt the target LLM with alternative specifications. Note that this phase is optional in the proposed framework. While we study the impact of paraphrased textual descriptions on model performance as an additional analysis, the target LLM can still be evaluated using only the original encoding.
ASP Testing To evaluate the ASP programs generated by the LLMs, we propose two methods: (i) a test-suite-based approach and (ii) a model-based approach. Both methods are associated with proper metrics that allow us to measure a form of semantic distance from the gold program. Both approaches assume that each problem has a fixed set of input and output predicates, corresponding to the ones of the gold program. This assumption is pragmatic, yet reasonably not too restrictive. Indeed, LLMs are expected to produce meaningful programs as written by a human; thus, syntactic variations of predicate names are very likely resolved by the predicate matcher. If not, we assume the program is not correct. Test-Suite-based Evaluation. This evaluation method relies on the unit testing mechanism implemented in the ASP-WIDE tool (Amendola et al., 2024), which uses an annotation-based test specification language inspired by JUnit. The objective is to create a test suite that thoroughly checks the program functionality. A test suite is a collection of test cases, each comprising assertions that verify a number of conditions on the expected program output. Based on the test suite results, we compute the test case accuracy as the ratio of passed test cases over the total number. To ensure this evaluation method is effective, it is crucial to design a robust test suite over most of the program’s expected behavior. To this end, we developed a guideline for constructing meaningful test cases. The first step involves creating test cases to check for unsatisfiability. Specifically, given a set of input instances that violate the problem constraints, the program is expected to return no answer set for each instance. Next, we define test cases that verify the satisfiability of the problem constraints by checking if valid instances comply with them. If needed, we further refine the test suite by adding test cases that validate the correctness of the program’s output atoms. For example, in the Colorability problem, it should never occur that a node remains uncolored. This is done by writing an ASP-WIDE test case with the following assertion:
@constraintForAll(constraint=":- node(U), not chosenColor(U,_).")
which ensures that every node U is assigned at least one color. Finally, we add test cases to check whether the program produces the expected number of answer sets. Model-based Testing. This method evaluates an ASP program Pt based on its ability to reproduce the correct (gold) (stable) models (GM) for a given set of instances provided via facts. The core idea relies on the assumption that, given a gold program Pg and a set of instances, the target program Pt is considered correct if and only if it produces the same (stable) models as Pg on those instances. To formalize this, we define the set of target program models (TPM) as the (stable) models of Pt , and the covered gold models (CGM) as the intersection of TPM and GM, i.e., CGM = GM ∩ TPM. To quantify the accuracy of Pt , we use the standard (micro) F1-score (Christen, Hand, and Kirielle, 2023), which combines precision and recall, defined as: Recall =
|CGM| |GM|
and
Precision =
|CGM| , |TPM|
The (micro) F1 score is then computed as the harmonic mean of precision and recall, providing a balanced measure of the program’s ability to reproduce the expected models. Notably, we retrieve the CGM set with high efficiency by mean of the Algorithm 1, which leverages an ASP solver. Rather than relying on full answer set enumeration of the program under test, that might quickly become infeasible, we modify the ASP program to directly target specific solutions. In particular, we constrain the search space to gold models only, thereby avoiding the overhead of enumerating irrelevant answer sets. Clearly, we carefully crafted out instance set to be sure that the enumeration of gold program answers is instead feasible. The algorithm takes as input the program to test Pt , a gold program Pg , and a set of instances Ins. For each instance I ∈ Ins, we compute the set of gold models ASg by computing all answer sets of Pg ∪ I using an ASP solver (line 5). Then, for each gold model Mg ∈ ASg , we create a set of constraints aimed at forcing Mg to be an answer set of Pt (line 8). Specifically, for each atom a over the output predicates, if a ∈ Mg then we add the constraint :- not a; otherwise, we add the constraint :- a. For the sake of clarity, let us consider the Colorability problem with 3 nodes and 3 colors and a gold model Mg where node 1 is colored as purple, node 2 as red, and node 3 as brown. In this case, create model constraints(M g) returns the following constraints: :- not chosenColor(3,brown). :- not chosenColor(2,red). :- not chosenColor(1,purple). :- chosenColor(1,brown). :- chosenColor(1,red). :- chosenColor(2,brown).
:- chosenColor(2,purple). :- chosenColor(3,red). :- chosenColor(3,purple).
Intuitively, in the answer set Mg , node 1 is colored as purple (i.e., chosenColor(1, purple) ∈ Mg ) and so :- not chosenColor(1, purple) is obtained. Conversely, node 1 is colored neither red nor brown, and so the constraints :- chosenColor(1, red) and :- chosenColor(1, brown) are obtained. Analogously, the remaining constraints are obtained for nodes 2 and 3. Note that, if the gold program contains weak constraints then create model constraints(M g) does not return strong constraints as in the previous example but a set of weak constraints with cost 1 and a priority higher than the maximum priority used in Pt . This ensures that the solver prioritizes the inclusion of these gold atoms in the resulting answer sets. After obtaining the set of constraints Cg , the program Pt ∪ Cg ∪ I is solved using the solver configured to return exactly one answer set M . This operation is performed by the function solve once in line 9 of the algorithm. Then, the gold model Mg is considered covered if Mg exactly matches M . M and Mg exactly match if, for each atom a over output predicates, it holds that a ∈ M if and only if a ∈ Mg . The next step is to identify the set of models generated by Pt that differ from the gold models (i.e., wrong models). Note that this last step is necessary in order to properly calculate T P M . To this end, we first augment Pt with the set of facts encoding the gold models in ASg . To do this, we used the predicate trueInGold/2, where the first term is an identifier (i.e., a positive integer) for a gold model Mg ∈ ASg , and the second term is a true atom in Mg . Note that, such facts are obtained considering only atoms over the output predicates. Then, the program Pt is further augmented with additional rules that discard all those models of Pt that exactly match one gold model. This augmented version of the program Pt is returned by the function aug program and is denoted as Pta . For example, let us consider again the Colorability problem and the gold model in which node 1 is colored as purple, node 2 as red, and node 3 as brown. In this case the program Pta is obtained by adding to the program Pt the rules reported in Listing 1. Listing 1: Rules and facts used to augment a program Pt targeting the Colorability problem. 1 2 3 4 5 6 7 8 9
trueInGold(1,chosenColor(2,red)). trueInGold(1,chosenColor(1,purple)). trueInGold(1,chosenColor(3,brown)). ... mod(X) :- trueInGold(X,_). trueInTested(chosenColor(X,Y)) :- chosenColor(X,Y). smallerMG(M) :- trueInGold(M, X), not trueInTested(X). smallerMt(M) :- mod(M), trueInTested(X), not trueInGold(M, X). :- mod(M), not smallerMG(M), not smallerMt(M).
Rules in lines 1-3 in Listing 1 represent the facts encoding the considered gold model (here we reported only one for the sake of readability, but all gold models are considered). The rule at line 6 is used to compute the true atoms in
Algorithm 1 Finding Pt covered gold models and overall models. Input: A program Pt , a program Pg , a set of instances Ins. Output: the sets of models GM , CGM , and WM. 1: GM = ∅ 2: CGM = ∅ 3: W M = ∅ 4: for I ∈ Ins do 5: ASg = solve all(Pg ∪ I). 6: GM = GM ∪ ASg . 7: for Mg ∈ ASg do 8: Cg = create model constraints(Mg ). 9: M = solve once(Pt ∪ Cg ∪ I). 10: if M = Mg then 11: CGM = CGM ∪ {Mg } 12: end if 13: end for 14: Pta = aug program(Pt , ASg ). 15: ASw = solve all(Pta ∪ I). 16: W M = W M ∪ ASw 17: end for 18: return GM, CGM, W M
a model of Pt . Note that this rule is domain-specific, as the output predicate names vary across different domains. Then, rules in lines 7-8 compute for each gold model, the differences with a candidate model of Pt . Finally, the constraint at line 9 forces the program to discard a model if it matches a gold model. Once the program Pta is obtained, the algorithm calls the solver to compute all models of Pta ∪ I (i.e., solve all(Pta ∪ I)). Obtained models are considered wrong. Finally, the algorithm returns the set of gold models over all instances GM, the set of covered gold models CGM ⊆ GM, and the set of wrong models WM, which is disjoint from GM. The value |TPM| can then be calculated as the sum of the number of elements of CGM and WM. Discussion on Metrics. The introduced metrics are intended to assess how well an ASP program semantically models a given problem. Among the two evaluation methods, the test-suite-based approach offers a more accurate measure of program correctness, as it involves white-box testing. This type of testing enables detailed inspection of a program internal behavior, rather than relying solely on its output models. However, the test-suite-based approach has some limitations; in fact, it requires a human expert to design the test cases, and the reliability of the final assessment depends heavily on the quality of the test suite. In contrast, the model-based approach offers a higher level of automation, as human intervention is limited to providing input instances and the gold program. This method, however, makes it more difficult to know the reasons why a program may be incorrect, which limits potential efforts to improve it. Indeed, we might inspect the results and discover some answer set is missing or wrong, but there is no clue on why it happens. On the other hand, specific test cases are devised
with the aim at testing a specific ability, and thus are more connected to a reason for the failure.
Dom
RQ3: How do the proposed semantic metrics compare, in terms of accuracy and efficiency?
Ham
RQ2: How does paraphrasing impact on performance?
Set ilton ian Cyc Ham le ilton ian Hier Path arch ical C Max lusterin g ima l Cli que Part ition Slith erlin k Stab le M a rriag Trav e eling Sale pers on
ty Colo rabi li
0.0
ing
RQ1: Are state-of-the-art LLMs effective in ASP coding, under both syntactic and semantic perspectives?
0.5
inat
We present our experimental evaluation, which tackles the following research questions:
Accuracy
Experimental Evaluation
1.0
Setup and Validation Protocol Dataset To evaluate the proposed approach, we created a dataset comprising the following ten graph-related problems: Colorability, (Connected) Dominating Set, Hamiltonian Cycle, Hamiltonian Path, Hierarchical Clustering, Maximal Clique, (Graph) Partitioning, Slitherlink, (Strong) Stable Marriage, and Traveling Salesman. We selected this set of tasks since they are standard problems used in many ASP competitions (Gebser, Maratea, and Ricca, 2017; Calimeri, Ianni, and Ricca, 2014; Alviano et al., 2013). Each problem includes a formal natural language description and the corresponding gold program. For each problem, we provide the related test suite and the input instances required by the evaluation methods outlined earlier. In the Supplementary Material, we provide the problem descriptions and details regarding the test suite design. Test Suite Validation. To ensure the quality of our test suite, we draw inspiration from Oetsch et al. (2012), who empirically demonstrated that the small-scope hypothesis of traditional testing is also applicable to ASP programs, i.e., many errors can be detected by testing a program against inputs involving only a small number of objects. Following Amendola et al. (Amendola et al., 2024), we employ a mutation-based testing strategy specifically designed for ASP (Oetsch et al., 2012). Given an ASP program, the tool randomly applies a set of modifications to its rules, generating mutants, altered versions of the original program that introduce subtle errors. For each gold program, we generate 15 mutants and execute the corresponding test suite on them using our selected input instances. After each execution, we iteratively refine the test suite and the input instances until none of the mutants achieves a perfect test score (i.e., 1.0). This iterative process ensures that the test suite is effective in detecting deviations from the expected program behavior. Tested Models. For the experimental evaluation, we adopt eight state-of-the-art LLMs from five distinct families: o4mini (OpenAI, 2025b), Claude-3.5-Sonnet (Anthropic, 2024), DeepSeek-R1 (DeepSeek, 2025), Gemini-1.5-Pro (Team and et al., 2024), Gemini-2.5-Flash (Comanici, Bieber, and et al., 2025), GPT-4o (OpenAI, 2024), GPT5 (OpenAI, 2025a), and LLaMa3.3-70B (Grattafiori and et al., 2024).
Figure 3: Performance of GPT-4o as predicate matcher. Error bars represent confidence intervals on ten different runs.
Settings. All tested models have been invoked via the dedicated APIs of OpenAI, Anthropic, Google, and DeepSeek, with a temperature of 0.7 (when allowed). In the case of LLaMa3.3-70B, we used a self-hosted instance using the Ollama tool. The LLM-related and ASP-testing code has been implemented in Python. The interaction with the ASPWIDE tool (developed in Spring Boot) was done through a RESTful API endpoint. The experiments have been conducted on a server equipped with an Intel(R) Xeon(R) CPU E7-8880 v4 @ 2.20GHz, 512GB of RAM, and running Debian 13 (6.1.0-32-amd64). Additional information and experimental results are provided as Supplementary Material. The experimental code is made publicly available.2
Results Predicate Matcher Accuracy. We first assess the predicate matcher accuracy in mapping predicates from the two input programs semantically. To achieve this, we manually verify the matches, assigning each generated dictionary a score ranging from [0, 1]. Specifically, we count the correct associations and average them over the total number of predicates. For each problem, we create ten separate matches to ensure statistical significance and report the average as the ultimate performance score. Figure 3 presents the results using GPT-4o as the predicate matcher. The accuracy of the model on this task comes as no surprise, given its well-documented abilities to perform text summarization and rephrasing (Adams et al., 2023; Zhou, Ringeval, and Portet, 2025); thus, we opt for GPT-4o also as a paraphraser. LLMs Performance (RQ1). Further, we assess the effectiveness of the evaluated models concerning both syntactic syntax and semantic accuracy. Figure 4 shows the average results for all examined problems, whereas Figure 5 displays scores for each individual problem. At this stage, we concentrate solely on the initial problem statements, with more 2 https://anonymous.4open.science/r/ LLMs-ASP-Benchmark-DFC3/
7 For Maximal Clique, we use only the model-based metric due to ASPWIDE run-time issues.
Stable marriage
Traveling saleperson
Partition
Slitherlink
Maximal clique
Hamiltonian path
Hierarchical clustering
Hamiltonian cycle
LLM Error Analysis. For completeness, we summarize the most common error patterns observed in syntactically valid but semantically incorrect programs. A frequent issue is the omission of constraints. For example, in the Hamiltonian Path problem, LLMs often fail to enforce that exactly one edge enters and one edge leaves each node, which is required to guarantee a proper path. In Stable Marriage, models sometimes omit bijectivity constraints ensuring that each man is matched with exactly one woman and vice versa. In optimization tasks involving weak constraints, mistakes in the specification of weights or priorities are also common. Another recurring problem concerns incorrect predicate signatures, such as wrong arity or mismatched arguments in input atoms. In some cases, additional constraints are introduced without a clear correspondence to the problem description; while not always harmful, they reveal a tendency to produce plausible but unjustified rules. Overall, LLMs tend to generate encodings that look reasonable at a superfi-
Paraphrase 1
Paraphrase 2
1.0
Trave li
ng sa lepers
on
ge marria Stable
rlink Slithe
Partit ion
path onian Hamil t
le n cyc tonia Hamil
ating Domin
Color
0.0
set
0.5
ability
Semantic Accuracy
Original Problem
Paraphrasing Impact (RQ2). Further, we want to assess whether a more natural, human-like style of the textual description might help in increasing performance. To this aim, we select DeepSeek-R1 as the representative LLM candidate, since it achieves one of the best syntactic-semantic trade-off while being the cheapest to probe. We compare its semantic accuracy when problem paraphrasings are applied. Figure 6 depicts the test-suite-based evaluation score7 , averaged over five runs, along with the confidence intervals. We see that, in most cases, performance is leveraged when the original problem description is used, despite some exceptions where the first paraphrasing slightly improves the semantic accuracy of the model (notably, with Dominating set, Partition, Stable marriage).
Colorability
Figure 5: Syntactic and semantic performance obtained by the evaluated LLMs across the considered problems. The test suite-based metric is used as the semantic reference7 . Each value represents the average of five different runs.
ue
in-depth analyses discussed subsequently. Here, semantic accuracy is determined using test suite-based evaluation. In comparison, syntactic correctness is evaluated by checking whether the input program results in any syntax errors during processing by the solver. To ensure results are statistically robust, each experiment is conducted by calling the target LLM five times, with outcomes then averaged. Two considerations can be made. First, as depicted in Figure 4, while most LLMs show an impressive accuracy under the syntactic perspective (with the notable exception of LLaMa3.3), they perform poorly in terms of semantics. Secondly, Figure 5 shows that, while simple problems such as Colorability or Dominating Set are easier to address, almost all LLMs fall short in encoding more complex tasks (e.g., Partition or Slitherlink). This suggests that state-ofthe-art LLMs still struggle to effectively encode declarative paradigms, such as Answer Set Programming.
1.0
(b) Semantic
al cliq
Figure 4: Trade-off of the evaluated LLMs between syntactic (Xaxis) and semantic accuracy (Y-axis). The semantic score is computed according to the test suite-based method7 . Results are averaged across all considered problems.
Stable marriage
(a) Syntactic
1.0
Maxim
0.8
ring
0.6
cluste
0.4
Syntactic
hical
0.2
Hiera rc
0.0 0.0
0.8
1 0.4 0.4 0.2 0.1 0.8 0 0.1 0.4 0.1 1 0.4 0.1 0.2 0 0.2 0.1 0.1 0 0.1 1 0.5 0.7 0.6 0.2 0.4 0 0 0.3 0.4 1 0.2 0.1 0.5 0 0 0 0 0.1 0.1 0.8 0.1 0.3 0.8 0 0.4 0 0 0.1 0 1 0.2 0.1 0 0 0 0 0 0.2 0 1 0.7 0.9 0.7 0.1 0.4 0 0.1 0.4 0.4 1 0.1 0.1 0 0 0 0 0.1 0 0
Traveling saleperson
0.2
Partition
0.4
0.6
Slitherlink
Colorability
0.6
0.4
Maximal clique
0.8
0.2
1 1 1 1 0.8 1 0.8 1 0.8 0.8 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0.8 0.6 1 1 1 1 0.8 1 0.8 1 1 0.4 0.6 1 1 1 1 1 0.6 1 0.4 0.4 0.4 1 1 0.8 0.8 0.8 1 1 0.8 0 0.8 0.6 1 1 1 1 0.8 1 0.8 1 0.8 1 1 0.4 0.6 0.4 0.8 0.4 0.4 0.2 0.2 0.6 Hamiltonian path
1.0
Semantic
0.0 o4-mini Claude3.5 DeepSeek-R1 Gemini1.5 Gemini2.5 GPT-4o GPT-5 LLaMa3.3
Dominating set
GPT-5 LLaMa3.3
Hierarchical clustering
Gemini2.5 GPT-4o
Hamiltonian cycle
DeepSeek-R1 Gemini1.5
Dominating set
o4-mini Claude3.5
Figure 6: Comparison of the semantic accuracy computed via the test-suite when problem paraphrasings are applied7 . DeepSeek-R1 is used as the reference LLM. Bars depict the average across five different runs, while error bars represent confidence intervals.
GPT-5 LLaMa3.3
Figure 7: Comparison between the two proposed semantic metrics8 . DeepSeek-R1 is the reference LLM. Values represent the difference between test-suite- and model-based scores, averaged across five runs, while lines indicate confidence intervals.
cial level, yet the formal rigor of ASP exposes subtle inaccuracies. Although such drafts may assist human programmers, our results indicate that fully automatic generation of correct ASP encodings remains challenging for current state-of-the-art models. Test-suite- vs Model-based (RQ3). Next, we compare the two proposed semantic metrics, both in terms of accuracy (Figure 7) and efficiency (Figure 8). Specifically, Figure 7 shows the difference in performance computed via the test-suite- and the model-based metric Therefore, a positive (resp., negative) score means that the performance is higher if the test-suite-based metric (resp., the model-based metric) is considered. Again, results are obtained by averaging five distinct runs, while the bars represent confidence intervals. We see that, across most problems and evaluated LLMs, the two metrics align. This alignment is also confirmed by the Pearson correlation coefficient ρ = 0.865. We finally compare the two metrics in terms of efficiency. Figure 8 depicts the testing times (in seconds) across the considered problems, using DeepSeek-R1 as the reference LLM. In all problems (except for Hierarchical clustering), the test-suite-based metric significantly reduces the computation time, resulting in the most efficient metric. (More details in the Supplementary Material.)
Conclusions and Future Work Although LLMs are known to excel at language understanding and code synthesis, their ability to generate correct ASP programs has remained comparatively underexplored. Research on the automatic composition of ASP code is growing; however, evaluations often rely on manual inspection or impose constraints on the LLM’s input/output format, making performance sensitive to superficial variations such as predicate renaming. As a consequence, the field still lacks systematic standardized benchmarks and rigorous met-
lepers on ng sa
rlink
marria ge
Trave li
Stable
Slithe
tering Partit ion
l clus
n pat h
Hiera rchica
le n cyc
tonia Hamil
set
tonia
ating
0
Hamil
lepers on ng sa Trave li
Stable
marria ge
link Slithe r
Partit ion
path
tering l clus Hiera rchica
le n cyc
tonia n Hamil
tonia Hamil
Domin
ating
set
−0.5
500
ability
0.0
Model-Based
1000
Color
0.5
−1.0
Test Suite-Based
Domin
Gemini2.5 GPT-4o
Testing time (s)
DeepSeek-R1 Gemini1.5
1.0
Color ability
Test suite- vs Model-based
o4-mini Claude3.5
Figure 8: Comparison between the two semantic metrics in terms of testing times (expressed in seconds). DeepSeek-R1 is used as the reference LLM. Bars represent the average across five runs, error bars indicate confidence intervals.
rics for assessing the ASP coding capabilities of LLMs and, more broadly, of automatic program coding approaches. In this paper, we addressed this gap by introducing BLAST, the first dedicated benchmarking methodology and accompanying dataset specifically designed to rigorously evaluate the accuracy of LLMs in generating ASP programs. Along with BLAST, we implement and release an ad-hoc benchmark, two testing methodologies and related correctness metrics. The proposed metrics measure the semantic distance between the generated encoding and the reference specification, without imposing constraints on predicate names or LLMs prompt structure. We show that the two measures consistently align in their assessment of correctness, while remaining complementary: one is more transparent and interpretable, at the cost of requiring human intervention, whereas the other is fully automatic but less directly explainable. Building on BLAST, we conduct an extensive empirical study of eight state-of-the-art LLMs from five different model families, evaluating their ability to encode ten well-known graph problems in ASP starting from textual descriptions. Experiments reveal that: (i) although LLMs often achieve high syntactic accuracy, they struggle with semantic correctness; (ii) their performance degrades significantly as task complexity increases; and (iii) rephrasing the problem specification does not lead to consistent improvements. As future work, we plan to address current limitations and explore several directions for extension. Although our evaluation adopts zero-shot prompting to ensure a uniform and controlled comparison, more advanced strategies, such as few-shot prompting, prompt tuning, or retrieval-augmented generation (RAG), deserve systematic investigation, and could lead to measure better performance of LLMs in generating ASP code. Moreover, our benchmark focuses on graph problems that are standard ASP system testing. Extending the benchmark to additional domains would further strengthen its scope and representativeness. Finally, although BLAST here evaluates general-purpose LLMs, our methodology is model-agnostic and could also be employed
to evaluate specialized techniques for ASP encoding.
GenAI Statement This paper has been fully conceptualized, designed, and written by the human authors. LLMs were used to proofread the paper and identify some typos and style issues, the authors claim the full ownership of the text. Importantly, LLM are used as integral components of the proposed framework, as detailed in the manuscript, and are subject solely to experimental evaluation.
Acknowledgments This work has been partially funded by MUR on D.M. 351/2022, PNRR Ricerca, CUP H23C22000440007; and partially supported by Moneying-Plus project (CUP J29I24001790005) selected within the framework of the PR FESR – FSE Calabria 2021/2027 and implemented with the support of the Italian State and the Calabria Region. The work has also been partially supported by PN RIC project ASVIN “Assistente Virtuale Intelligente di Negozio” (CUP B29J24000200005) and the PNRR project FAIR - Future AI Research (PE00000013), Spoke 9 – Green-aware AI, under the NRRP MUR program funded by the “NextGenerationEU”.
References Adams, G.; Fabbri, A.; Ladhak, F.; Lehman, E.; and Elhadad, N. 2023. From sparse to dense: GPT-4 summarization with chain of density prompting. In Proceedings of the 4th New Frontiers in Summarization Workshop, 68– 74. Association for Computational Linguistics. Alviano, M.; Calimeri, F.; Charwat, G.; Dao-Tran, M.; Dodaro, C.; Ianni, G.; Krennwallner, T.; Kronegger, M.; Oetsch, J.; Pfandler, A.; Pührer, J.; Redl, C.; Ricca, F.; Schneider, P.; Schwengerer, M.; Spendier, L. K.; Wallner, J. P.; and Xiao, G. 2013. The fourth answer set programming competition: Preliminary report. In LPNMR, volume 8148 of Lecture Notes in Computer Science, 42–53. Springer. Alviano, M.; Calimeri, F.; Dodaro, C.; Fuscà, D.; Leone, N.; Perri, S.; Ricca, F.; Veltri, P.; and Zangari, J. 2017. The ASP system DLV2. In LPNMR, volume 10377 of LNCS, 215–221. Springer. Alviano, M.; Cirimele, D.; and Reiners, L. A. R. 2023. Introducing ASP recipes and ASP chef. In ICLP Workshops, volume 3437 of CEUR WP. CEUR-WS.org. Amendola, G.; Mazzotta, G.; Ricca, F.; and Berei, T. 2024. Unit testing in ASP revisited: Language and test-driven development environment. Anthropic. 2024. Claude 3.5 sonnet model card addendum. Baral, C., and Dzifcak, J. 2011. Solving puzzles described in english by automated translation to answer set programming and learning how to do that translation. In AAAI Fall Symposium: Advances in Cognitive Systems, volume FS-11-01 of AAAI Technical Report. AAAI. Bauer, J. J.; Eiter, T.; Ruiz, N. H.; and Oetsch, J. 2024. Visual graph question answering with ASP and llms for language parsing. In ICLP, volume 416 of EPTCS, 15– 28. Borroto, M.; Kareem, I.; and Ricca, F. 2024. Towards automatic composition of ASP programs from natural language specifications. In IJCAI, volume abs/2403.04541, to appear. ijcai.org. Brewka, G.; Eiter, T.; and Truszczynski, M. 2011. Answer set programming at a glance. Busoniu, P.; Oetsch, J.; Pührer, J.; Skocovsky, P.; and Tompits, H. 2013. Sealion: An eclipse-based IDE for answerset programming with advanced debugging support. Calimeri, F.; Germano, S.; Palermiti, E.; Reale, K.; and Ricca, F. 2018. Developing ASP programs with ASPIDE and loide. Künstliche Intell. 32(2-3):185–186. Calimeri, F.; Ianni, G.; and Ricca, F. 2014. The third open answer set programming competition. Caruso, S.; Dodaro, C.; Maratea, M.; Mochi, M.; and Riccio, F. 2024. CNL2ASP: converting controlled natural language sentences into ASP. Chen, M.; Tworek, J.; and et al., H. J. 2021. Evaluating large language models trained on code.
Christen, P.; Hand, D. J.; and Kirielle, N. 2023. A review of the f-measure: Its history, properties, criticism, and alternatives. Comanici, G.; Bieber, E.; and et al., M. S. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Coppolillo, E.; Calimeri, F.; Manco, G.; Perri, S.; and Ricca, F. 2024. Llasp: fine-tuning large language models for answer set programming. In Proceedings of the 21st International Conference on Principles of Knowledge Representation and Reasoning, KR ’24. Dakhel, A. M.; Majdinasab, V.; Nikanjam, A.; Khomh, F.; Desmarais, M. C.; and Jiang, Z. M. J. 2023. Github copilot AI pair programmer: Asset or liability? DeepSeek. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Ding, Y.; Wang, Z.; Ahmad, W. U.; Ding, H.; Tan, M.; Jain, N.; Ramanathan, M. K.; Nallapati, R.; Bhatia, P.; Roth, D.; and Xiang, B. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Du, X.; Liu, M.; Wang, K.; Wang, H.; Liu, J.; Chen, Y.; Feng, J.; Sha, C.; Peng, X.; and Lou, Y. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24. Erdem, E., and Yeniterzi, R. 2009. Transforming controlled natural language biomedical queries into answer set programs. In BioNLP@HLT-NAACL, 117–124. ACL. Erdem, E.; Gelfond, M.; and Leone, N. 2016. Applications of answer set programming. Ernst, N. A., and Bavota, G. 2022. Ai-driven development is here: Should you worry? Falkner, A. A.; Friedrich, G.; Schekotihin, K.; Taupe, R.; and Teppan, E. C. 2018. Industrial applications of answer set programming. Fang, M., and Tompits, H. 2017. An approach for representing answer sets in natural language. In DECLARE, volume 10997 of LNCS, 115–131. Springer. Febbraro, O.; Reale, K.; and Ricca, F. 2011. ASPIDE: integrated development environment for answer set programming. In LPNMR, volume 6645 of LNCS, 317–330. Springer. Gebser, M.; Kaminski, R.; Kaufmann, B.; Ostrowski, M.; Schaub, T.; and Wanko, P. 2016. Theory solving made easy with clingo 5. In ICLP (Technical Communications), volume 52 of OASIcs, 2:1–2:15. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. Gebser, M.; Maratea, M.; and Ricca, F. 2017. The sixth answer set programming competition. Gebser, M.; Maratea, M.; and Ricca, F. 2020. The seventh answer set programming competition: Design and results. Gelfond, M., and Lifschitz, V. 1991. Classical negation in logic programs and disjunctive databases.
Grattafiori, A., and et al., A. D. 2024. The llama 3 herd of models. Hahn, S.; Sabuncu, O.; Schaub, T.; and Stolzmann, T. 2023. Clingraph: A system for asp-based visualization. Ishay, A.; Yang, Z.; and Lee, J. 2023a. Leveraging large language models to generate answer set programs. In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, KR ’23. Ishay, A.; Yang, Z.; and Lee, J. 2023b. Leveraging large language models to generate answer set programs. In KR, 374–383. Jiang, J.; Wang, F.; Shen, J.; Kim, S.; and Kim, S. 2024. A survey on large language models for code generation. Kalliamvakou, E. 2022. Research: quantifying github copilot’s impact on developer productivity and happiness. Kareem, I.; Gallagher, K.; Borroto, M. A.; Ricca, F.; and Russo, A. 2024. Using learning from answer sets for robust question answering with LLM. In LPNMR, volume 15245 of Lecture Notes in Computer Science, 112–125. Springer. Kuhn, T. 2014. A survey and classification of controlled natural languages. Law, M.; Russo, A.; and Broda, K. 2020. The ILASP system for inductive learning of answer set programs. Lippolis, A. S.; Saeedizade, M. J.; Keskisärkkä, R.; Zuppiroli, S.; Ceriani, M.; Gangemi, A.; Blomqvist, E.; and Nuzzolese, A. G. 2025. Ontology generation using large language models. In ESWC (1), volume 15718 of Lecture Notes in Computer Science, 321–341. Springer. Liu, T.; Xu, C.; and McAuley, J. 2023. Repobench: Benchmarking repository-level code auto-completion systems. Ma, Z.; Guo, H.; Chen, J.; Peng, G.; Cao, Z.; Ma, Y.; and Gong, Y.-J. 2024. Llamoco: Instruction tuning of large language models for optimization code generation. Mitra, A., and Baral, C. 2016. Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. In AAAI, 2779– 2785. AAAI Press. Naveed, H.; Khan, A. U.; Qiu, S.; Saqib, M.; Anwar, S.; Usman, M.; Akhtar, N.; Barnes, N.; and Mian, A. 2024. A comprehensive overview of large language models. Nye, M. I.; Tessler, M. H.; Tenenbaum, J. B.; and Lake, B. M. 2021. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In NeurIPS, 25192–25204. Oetsch, J.; Prischink, M.; Pührer, J.; Schwengerer, M.; and Tompits, H. 2012. On the small-scope hypothesis for testing answer-set programs. In KR. AAAI Press. OpenAI. 2024. Gpt-4o system card. OpenAI. 2025a. Introducing gpt-5. OpenAI. 2025b. Introducing openai o3 and o4-mini.
Ou, J.; Lu, J.; Liu, C.; Tang, Y.; Zhang, F.; Zhang, D.; and Gai, K. 2024. DialogBench: Evaluating LLMs as humanlike dialogue systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics. Peng, S.; Kalliamvakou, E.; Cihon, P.; and Demirer, M. 2023. The impact of AI on developer productivity: Evidence from github copilot. Qin, L.; Chen, Q.; Feng, X.; Wu, Y.; Zhang, Y.; Li, Y.; Li, M.; Che, W.; and Yu, P. S. 2024. Large language models meet nlp: A survey. Raiaan, M. A. K.; Mukta, M. S. H.; Fatema, K.; Fahad, N. M.; Sakib, S.; Mim, M. M. J.; Ahmad, J.; Ali, M. E.; and Azam, S. 2024. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. Rajasekharan, A.; Zeng, Y.; Padalkar, P.; and Gupta, G. 2023. Reliable natural language understanding with large language models and answer set programming. Ren, L.; Xiao, G.; Qi, G.; Geng, Y.; and Xue, H. 2025. Can llms solve asp problems? insights from a benchmarking study. In Proceedings of the 22nd International Conference on Principles of Knowledge Representation and Reasoning, KR ’25. Schrader, T. P.; Lange, L.; Kaminski, T.; Razniewski, S.; and Friedrich, A. 2025. A solver-in-the-loop framework for improving llms on answer set programming for logic puzzle solving. CoRR abs/2512.17093. Schwitter, R. 2018. Specifying and verbalising answer set programs in controlled natural language. Shah, N.; Genc, Z.; and Araci, D. 2025. Stackeval: benchmarking llms in coding assistance. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24. Smirnov, P.; Joublin, F.; Ceravola, A.; and Gienger, M. 2024. Generating consistent PDDL domains with large language models. Team, G., and et al., P. G. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Valmeekam, K.; Stechly, K.; Gundawar, A.; and Kambhampati, S. 2025. A systematic evaluation of the planning and scheduling abilities of the reasoning model o1. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017. Attention is all you need. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. Voboril, F.; Ramaswamy, V. P.; and Szeider, S. 2025. Generating streamlining constraints with large language models. J. Artif. Intell. Res. 84.
Wang, Y.; Le, H.; Gotmare, A. D.; Bui, N. D. Q.; Li, J.; and Hoi, S. C. H. 2023. Codet5+: Open code large language models for code understanding and generation. Xu, F. F.; Alon, U.; Neubig, G.; and Hellendoorn, V. J. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, 1–10. New York, NY, USA: Association for Computing Machinery. Xu, K.; Mao, Y.; Guan, X.; and Feng, Z. 2025. Web-bench: A llm code benchmark based on web standards and frameworks. Yang, Z.; Ishay, A.; and Lee, J. 2023. Coupling large language models with logic programming for robust and general reasoning from text. In ACL (Findings), 5186–5219. ACL. Zhao, W. X.; Zhou, K.; and et al., J. L. 2025. A survey of large language models. Zhou, Y.; Ringeval, F.; and Portet, F. 2025. Can gpt models follow human summarization guidelines? a study for targeted communication goals.