arXiv:2604.11699v1 [cs.CL] 13 Apr 2026
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning Jieying Xue∗
Phuong Minh Nguyen∗
Ha Thanh Nguyen
Center for Juris-Informatics, ROIS-DS, Tokyo, Japan [email protected]
Japan Advanced Institute of Science and Technology, Ishikawa, Japan [email protected]
Center for Juris-Informatics, ROIS-DS Tokyo, Japan [email protected]
May Myo Zin
Ken Satoh
Center for Juris-Informatics, ROIS-DS Tokyo, Japan [email protected]
Center for Juris-Informatics, ROIS-DS, Tokyo, Japan [email protected]
Abstract
CCS Concepts
This work aims to improve the generalization of logic-based legal reasoning systems by integrating recent advances in Natural Language Processing (NLP) with legal-domain adaptive few-shot learning techniques using Large Language Models (LLMs). Existing logic-based legal reasoning pipelines typically rely on fine-tuned models to map natural-language legal cases into logical formulas before forwarding them to a symbolic reasoner. However, such approaches are heavily constrained by the scarcity of high-quality annotated training data. To address this limitation, we propose a novel LLM-based legal reasoning framework that enables effective in-context learning through retrieval-augmented generation (RAG). Specifically, we introduce Legal2LogicICL, a few-shot retrieval framework that balances diversity and similarity of exemplars at both the latent semantic representation level and the legal text structure level. In addition, our method explicitly accounts for legal structure by mitigating entity-induced retrieval bias in legal texts, where lengthy and highly specific entity mentions often dominate semantic representations and obscure legally meaningful reasoning patterns. Our Legal2LogicICL constructs informative and robust few-shot demonstrations, leading to accurate and stable logical rule generation without requiring additional training. In addition, we construct a new dataset, named Legal2Proleg, which is annotated with alignments between legal cases and PROLEG logical formulas to support the evaluation of legal semantic parsing. Experimental results on both open-source and proprietary LLMs demonstrate that our approach significantly improves accuracy, stability, and generalization in transforming natural-language legal case descriptions into logical representations, highlighting its effectiveness for interpretable and reliable legal reasoning1 .
• Software and its engineering → Semantics; • Applied computing → Law.
∗ Both authors contributed equally to this paper. 1 Our code is available at https://github.com/yingjie7/Legal2LogicICL.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ICAIL’26, Singapore © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Keywords Diverse Demonstrations, Large Language Models, In-Context Learning, Legal Semantic Parsing, Legal Reasoning ACM Reference Format: Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh. 2026. Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning. In Proceedings of 21th International Conference on Artificial Intelligence and Law (ICAIL’26). ACM, New York, NY, USA, 10 pages. https: //doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Legal reasoning is an essential task of the legal domain, in which legal judgments are derived by interpreting statutory provisions together with case facts. However, the scale and complexity of legal systems, along with the ambiguity of natural language case descriptions, make the formalization of legal reasoning highly challenging. In particular, accurately transforming unstructured legal texts into structured, machine-interpretable logical representations remains a critical problem in legal AI, as it directly affects the reliability of automated legal reasoning systems. To support interpretable legal reasoning, the PROLEG framework was proposed [21, 22], providing an expressive formalism that enables legal professionals to understand reasoning processes and outcomes. However, PROLEGbased systems require inputs in the form of formal logical formulas, which creates a usability barrier for practitioners without expertise in logical modeling. To mitigate this issue, prior work has introduced multi-stage legal reasoning pipelines that translate natural language case descriptions into PROLEG fact formulas before inference [16]. In such pipelines, the semantic parsing stage is crucial, as errors in this step directly propagate to downstream reasoning. Existing approaches for this task can be broadly categorized into pattern-based methods [15], neural machine translation models [16, 17], and NER-based systems [27]. While these methods have achieved partial success, they suffer from limited generalization, high annotation costs, or brittleness to surface-level variations, particularly when handling complex legal structures and multi-entity relations.
ICAIL’26, June 8–16, 2026, Singapore
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh
Recent advances in LLMs offer a promising alternative, as LLMs exhibit strong reasoning and generation capabilities without taskspecific training. In this work, we propose Legal2LogicICL, a novel LLM-based legal semantic parsing framework that translates natural language legal case descriptions into PROLEG logical fact formulas through in-context learning, without requiring additional supervised fine-tuning. Our framework leverages the strong reasoning and generation capabilities of LLMs, while introducing structured guidance through carefully constructed in-context demonstrations. Specifically, we adopt a RAG paradigm [10] to retrieve relevant legal cases as in-context exemplars. In the In-Context Learning (ICL) setting, where model parameters remain fixed, the quality of retrieved exemplars plays a critical role in determining reasoning performance [11, 14]. Prior studies have shown that effective few-shot sets should be both relevant to the target instance and sufficiently diverse to avoid overfitting to superficial patterns [9, 13, 20, 23, 26]. Overly homogeneous exemplars may cause the model to overfit to surface-level similarities, thereby limiting robustness and generalization. Motivated by these observations, we propose a legal-oriented diversity-enhanced few-shot selection strategy, which explicitly controls exemplar diversity through two complementary mechanisms. First, we introduce a diversity control parameter 𝜆 to regulate the trade-off between semantic similarity and exemplar diversity during retrieval. Second, we address a fundamental limitation of conventional semantic similarity-based retrieval methods in the legal domain. Legal texts frequently contain long and highly specific entity mentions (e.g., party names, assets, contract identifiers, and descriptions of legal actions), which dominate semantic representations and bias retrieval toward surface-level entity overlap rather than meaningful legal structures. To mitigate this issue, we introduce an entity-agnostic, template-level similarity strategy. Instead of computing similarity over raw legal cases, we abstract legal texts into templates by removing concrete entity instantiations while preserving their underlying legal and logical structure. This enables the retrieval of structurally aligned exemplars even when surface entities differ. By combining semantically similar case-level exemplars with structurally similar template-level exemplars, our Legal2LogicICL constructs a diversity-aware hybrid few-shot set that balances contextual relevance and structural diversity. This prompting strategy provides flexible structural constraints while preserving the expressive power of LLMs in language understanding and generation, enabling robust domain adaptation without model fine-tuning. Experimental results demonstrate that Legal2LogicICL effectively mitigates entity-induced retrieval bias, improves generalization across diverse legal cases, and yields more accurate and stable generation of PROLEG-style logical rules. Overall, this work offers a principled and practical solution for converting natural language legal facts into structured logical representations and lays a solid foundation for future research on logic-based legal reasoning and decision-making.
primarily rely on template-based methods that extract entities or facts using predefined rules [2, 5, 8, 12]. While effective under constrained settings, such methods suffer from limited scalability and poor generalization to complex legal scenarios. To improve applicability, Nguyen et al. [16] introduced a multistage translation framework that combines end-to-end translation models with an additional error-correction module. Although pretrained models enhance flexibility, they remain vulnerable to overfitting and often fail to produce precise fine-grained details (e.g., temporal expressions), which are critical for logical reasoning. Zin et al. [27] further introduce a NER-based pipeline that enforces logical well-formedness via rule composition, but its reliance on surface forms makes it brittle under paraphrasing.
2 Related Work 2.1 Transforming Legal Text into Logical Forms Transforming natural language legal text into formal logical representations is essential for legal reasoning systems. Early approaches
2.2
Diversity in Few-shot In-Context Learning
Few-shot in-context learning (ICL) has shown strong performance in legal reasoning due to its scalability and independence from task-specific training. Prior work highlights the importance of demonstration selection, particularly semantic similarity between examples and queries [11, 18, 20]. Beyond similarity, recent studies emphasize the role of diversity in improving generalization [4, 7, 9, 14]. Diverse demonstrations help cover a broader structural space, which is especially important in compositional tasks where models must generate structured outputs (e.g., logical forms). In such settings, limited structural coverage can hinder generalization due to missing symbolic patterns. Taken together, prior studies indicate that diverse few-shot demonstrations provide more effective guidance for in-context learning by expanding the hypothesis space exposed to the model, thereby enabling stronger and more robust generalization. Motivated by these findings, we propose a diversity-aware few-shot selection framework tailored to the legal domain. While existing approaches emphasize semantic similarity or structural diversity in general NLP settings, legal reasoning poses additional challenges due to its reliance on specialized terminology, symbolic constraints, and heterogeneous reasoning patterns. Our method explores domainspecific strategies for increasing demonstration diversity, with the goal of exposing in-context learners to a broader and more representative set of legal reasoning structures, thereby enhancing generalization in few-shot settings.
3
Task Definition and Notations
We study the task of legal semantic parsing, which aims to transform natural-language legal case descriptions into structured logical representations (e.g., a set of PROLEG fact formulas) that are executable within a legal reasoning system. Legal Cases and Entities. Let 𝑙 ∈ L denote a natural-language legal case (example in Table 1). A legal case typically involves a set of concrete entities, such as legal parties (e.g., borrower, lender), objects (e.g., assets), agreements, temporal expressions, and legally relevant events. We denote the entity set associated with a legal case 𝑙 as E (𝑙) = {𝑒 1, 𝑒 2, . . . , 𝑒 | E (𝑙 ) | },
(1)
where each entity 𝑒𝑖 corresponds to a specific real-world instantiation appearing in the case description.
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning
Legal Templates. In the legal reasoning domain, different legal cases may share similar underlying legal structures while differing substantially in their concrete entities. To capture such structural regularities, we define a legal template as an entity-agnostic abstraction of a legal case. Formally, given a legal case 𝑙, a template function maps 𝑙 to a template 𝑡 = template(𝑙) by replacing each concrete entity mention in 𝑙 with its corresponding typed placeholder (or entity type) (e.g., {Borrower}, {Lender}, {Agreement}). template(·) : L → T
(2)
The resulting template preserves the narrative structure and legally meaningful relations of the case while suppressing entity-specific surface information. Fact Formula Generation. Given a legal template and a concrete entity set, a specific legal case can be constructed by filling the template placeholders with entities from E (𝑙). The goal of legal semantic parsing in this work is, given a legal case 𝑙, to identify the appropriate instantiations of PROLEG fact formulas that satisfy the predefined legal rules. We denote the target output as 𝑓 = {𝑓1, 𝑓2, . . . , 𝑓 | 𝑓 | } ∈ F ,
(3)
where each 𝑓𝑖 is an executable PROLEG fact corresponding to an entity-grounded legal predicate, as shown in the Facts field of Table 1. Task Formulation. Formally, the task can be defined as learning a transforming method (M) which maps a natural-language legal case 𝑙 to its corresponding set of fact formulas 𝑓 . M : L → F,
(4)
In this work, the mapping M is induced through few-shot incontext learning using LLMs, without any task-specific parameter fine-tuning. Instead, the model is guided by a small set of retrieved legal demonstrations that provide both semantically relevant and structurally diverse reasoning patterns.
4
Dataset Construction
Motivation. To evaluate the generalization ability of methods for transforming natural language legal cases into PROLEG format logical formulas within the Proleg system, we construct a new dataset named Legal2Proleg. This dataset is built by adopting a data-augmentation pipeline for legal cases introduced in Phuong et al. [19], which contains multiple contract types, including loan, lease, purchase, and copyright contracts. Table 2 compares the characteristics of our dataset with those of previous datasets in the legal domain. Overall, the prior dataset LegalCaseNER [27] is designed to evaluate the ability to detect object (entity) names in contracts. Although it contains the largest number of samples, it exhibits the lowest diversity (i.e., the smallest template2 vocabulary size) and is limited to purchase contracts only. In contrast, Legal2ProlegV0 3 [19] and our extended version, Legal2Proleg, demonstrate greater diversity in terms of template vocabulary size, the number of legal issues, and the variety of unique facts. Notably, neither LegalCaseNER nor Legal2ProlegV0 control for template overlap between the training 2 The template of each sample was constructed by substituting entity values with
corresponding entity names. 3 The original authors did not assign a name to their experimental dataset; we denote it as Legal2ProlegV0 to properly credit their contribution.
ICAIL’26, June 8–16, 2026, Singapore
Table 1: An Example Legal Case and Its Structured Representation in the Dataset Legal2Proleg Content Example - (denotaion) Legal Issue The borrower’s use of the assets in a manner that harms the lender’s interests or reputation has damaged the lender’s rights and reputation. Legal Case - (𝑙 𝑞𝑢𝑒𝑟 𝑦 ) Medical supplies were given to the hospital by the health organization as part of a supply agreement, intended for patient treatment. Instead, the hospital redistributed the supplies to external clinics without consent. This resulted in a shortage of supplies during a critical time, affecting patient care. These actions were discovered on 2023/08/20. Does the health organization have grounds for legal action to protect their reputation? Entities Set (or Slot Holders) - (E (𝑙 𝑞𝑢𝑒𝑟 𝑦 )) { "Borrower": "The hospital", "Object": "medical supplies", "Lender": "the health organization", "Agreement": "a supply agreement", "Harm": "a shortage of supplies during a critical time, affecting patient care", "T_discovery": "2023/08/20" } Template - (template(𝑙 𝑞𝑢𝑒𝑟 𝑦 )) {Object} was given to {Borrower} by {Lender} as part of {Agreement}, meant for patient treatment. Instead, {Borrower} redistributed supplies to external clinics without consent. This resulted in {Harm}. These actions were discovered on {T_discovery}. Does {Lender} have grounds for legal action to protect their reputation? Rules - (R) right_to_legal_action(_Lender, _Borrower, _Object) <= harm_to_lender_rights(_Borrower, _Lender, _Object), discovery_of_harm(_Lender, _Object, _T_discovery). harm_to_lender_rights(_Borrower, _Lender, _Object) <= borrower(_Borrower), lender(_Lender), owned_by(_Object, _Lender), borrowing_agreement(_Agreement), use_under_agreement(_Borrower, _Object, _Agreement), unlawful_use(_Borrower, _Object), reputational_damage(_Lender, _Harm). unlawful_use(_Borrower, _Object) <= use_under_agreement(_Borrower, _Object, _Agreement), violation_of_agreement(_Borrower, _Agreement). reputational_damage(_Lender, _Harm) <= harm_fact(_Harm, _Lender). discovery_of_harm(_Lender, _Object, _T_discovery) <= discovery_fact(_Lender, _Object, _T_discovery). Facts in Legal Case - (𝑓 𝑞𝑢𝑒𝑟 𝑦 ) borrower("The hospital"). lender("the health organization"). owned_by("medical supplies", "the health organization"). borrowing_agreement("a supply agreement"). agreement_fact("The hospital", "medical supplies", "a supply agreement"). violation_of_agreement("The hospital", "a supply agreement"). harm_fact("a shortage of supplies during a critical time", "the health organization") discovery_fact("the health organization", "medical supplies", "2023/08/20") % Legal Query Formula right_to_legal_action("the health organization", "The hospital", "medical supplies").
and testing sets, which introduces a risk of overfitting when applying supervised fine-tuning (SFT) methods. For example, BERT-based models, which strongly encode bidirectional contextual information, may memorize template-specific contexts during training and thus perform poorly on previously unseen templates. This issue is
ICAIL’26, June 8–16, 2026, Singapore
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh
Table 2: Statistics of the Legal2Proleg dataset and others. Dataset
LegalCaseNER
Legal2ProlegV0
Legal2Proleg
Source #Samples (×1000) #Templates #Template Vocab. #Entity Types #Legal Issues #Unique Facts #Contract Types
Zin et al. [27] ≈6.3 947 264 13 4 6 Purchase
Disclosed templ.
Yes
Phuong et al. [19] ≈5 240 907 27 9 31 Purchase, Lease, Loan, Copyright Yes
(this work) ≈1 1057 2897 45 24 114 Purchase, Lease, Loan, Copyright No
further examined in our experimental section. The construction of Legal2Proleg dataset proceeds through the following stages. 4.0.1 Annotation of PROLEG Rules. The annotator is first provided with a legal issue and an example case description (Table 1). At this stage, the annotator and analyzes the constraints specified in the contract (or legal document), based on this analysis, the annotator manually construct the logical causal relationships between contractual constraints and the relevant case facts using the PROLEG logic language. These relationships are represented as legal rules (R) or PROLEG logical trees, as illustrated in Table 1. All annotations are constructed under expert supervision and undergo careful verification and correction to ensure both logical correctness and legal consistency. 4.0.2 Annotation of Facts in Legal Cases. Given the legal rules and legal case obtained in the previous step, this stage focuses on annotating legally relevant facts in each legal case using PROLEG formulas (𝑓 ). An example illustrating the alignment between PROLEG formulas and the corresponding legal case information is shown in Table 1. In addition, the important entity types (or slot holders) are identified and collected for subsequent template-based legal case data augmentation [19]. 4.0.3 Data Augmentation of Legal Template. Given the provided legal issue, a case description, and predefined entity types, annotators construct realistic contractual scenarios covering diverse types of breaches, and then generate multiple corresponding logical templates. Following the augmentation pipeline of Phuong et al. [19], the content of templates and entity pairs can be generated with assistance from LLMs (e.g., gpt-5), but certainly need to be confirmed or rejected by annotators familiar with the legal issue. This design ensures structural consistency at the template level while maintaining substantive diversity at the case level, closely mirroring real-world legal reasoning settings. Finally, for each legal issue, numerous templates and entity pairs are synthesized and aligned with the corresponding legal rules and fact formulas, which serve as gold-standard data for machine learning systems.
5
Proposed Method
As illustrated in Figure 1, we propose Legal2LogicICL, a diversityoriented hybrid few-shot learning framework for legal translation from natural language into structured logical representations. Following the RAG [10] and few-shot prompting paradigm [3], the
framework constructs domain-aware demonstrations while incorporating diversity to guide LLMs toward structurally consistent and legally grounded reasoning, without task-specific fine-tuning. We introduce a two-level diversity-similarity balancing mechanism. At the latent semantic level, we propose DiverseSim, a retrieval algorithm that selects demonstrations by regulating the trade-off between similarity and diversity via a tunable hyper-parameter over encoded vector representations. At the structural text level, we design a legal-domain-aware hybrid retrieval strategy. While conventional case-based retrieval is often dominated by surfacelevel entity overlap, we introduce an entity-agnostic template-based mechanism that emphasizes underlying legal structures. By combining semantically relevant exemplars with structurally aligned templates, our method enhances both diversity and robustness in incontext demonstrations. We further decompose Legal2LogicICL into two stages: (1) Demonstration Selection, and (2) Prompting Construction and Inference, which are detailed in the following subsections.
5.1
Demonstration Selection
Motivated by prior work on diversity-aware retrieval for in-context learning [9, 14], we observe that two factors are particularly critical during demonstration retrieval: (1) semantic relevance to the query instance and (2) diversity among the selected demonstrations. Given a legal case query, the objective of this stage is to select demonstrations from the retrieval pool that are not only highly relevant to the query, but also mutually distinctive, so as to collectively cover a broad range of legal scenarios and reasoning patterns. This balance is especially important in the legal domain, where excessive similarity among demonstrations may bias the model toward narrow, entity-specific patterns and hinder generalization. Motivated by these considerations, we propose DiverseSim (Algorithm 1), a diversity-aware demonstration ranking algorithm tailored for the legal domain. DiverseSim selects a set of few-shot examples that jointly maximize relevance to the query while promoting structural and contextual diversity, which are subsequently used to construct the in-context prompt for logical inference. Algorithm 1 DiverseSim Input: a query legal case 𝑙 𝑞𝑢𝑒𝑟 𝑦 , a storage examples D𝑑𝑜𝑚𝑎𝑖𝑛 = { ⟨𝑙𝑖 , 𝑓𝑖 ⟩ }0≤𝑖<|D𝑑𝑜𝑚𝑎𝑖𝑛 | , number of candidates 𝑘, balancing weight of similarity and diversity scores 𝜆. Output: a set of selected demonstrations S 1: S ← { } ⊲ init the set of selected demonstrations 2: D ← D𝑑𝑜𝑚𝑎𝑖𝑛 3: e𝑞 = TextEmb(𝑙 𝑞𝑢𝑒𝑟 𝑦 ) 4: e𝑑 𝑖 = TextEmb(𝑙𝑖 ) foreach 𝑙𝑖 ∈ D 5: B = arg top-10𝑖<|D| sim(𝑒𝑖𝑑 , 𝑒 𝑞 ) ⊲ most similar to query as a boundary 6: while |𝑆 | < 𝑘 do ⊲ loop until enough demonstrations 7: e𝑠𝑗 = TextEmb(𝑙 𝑗 ) foreach 𝑙 𝑗 ∈ S 8: 𝑟𝑎𝑛𝑘𝑖 = 𝜆 × sim(𝑒𝑖𝑑 , 𝑒 𝑞 ) − (1 − 𝜆) × max( {sim(𝑒𝑖𝑑 , 𝑒 𝑠𝑗 ) } 𝑗 <|𝑆 | ) 9: R = {𝑟𝑎𝑛𝑘𝑖 }𝑖<|B| ⊲ ranking with redundancy penalty 10: S ← S ∪ argmax ( R ) ⊲ pick best element based on ranking scores (𝑙𝑖 ,𝑓𝑖 )
11: B←B\S 12: end while
⊲ avoid duplicated selections in next step
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning
Legal Query (
)
Template -
Sam rented office from Eva under the agreement contract to make a meeting office ... Eva discovered on 2021/04/07 that Sam used office to do illegal activity instead ...
{Borrower} rented {Asset} from {Lender} under the {Agreement} to make a ...
ICAIL’26, June 8–16, 2026, Singapore
Step 1: Demonstration Selection
Step 2: Prompting Construction and Inference Prompting
case-based examplars
Candidate pool (seen data) ...
template-based examplars
Legal case ( ) in semantic space
LLM
DiverseSim x
Figure 1: Overview of the proposed diversity-aware hybrid few-shot in-context learning framework. 5.1.1 Few-shot Selection via Query Content. Given a query legal case, we first retrieve the top-𝑛 semantically similar legal cases from a domain-specific corpus to construct high-quality contextual demonstrations. We hypothesize that legal cases exhibiting stronger semantic and structural similarity to the query are more likely to provide informative and reliable references for downstream logical reasoning. To operationalize this process, we construct a domain𝑁 , where each 𝑙 denotes specific corpus, denoted as D domain = {𝑙𝑖 }𝑖=1 𝑖 a legal case in the training corpus. Formally, given a query legal case 𝑙 query , we retrieve the top-𝑛 most semantically similar legal cases from D domain . During retrieval, semantic similarity is computed using cosine similarity between vector representations of legal cases. All candidate query-example pairs are ranked according to DiverseSim algorithm, and the top-𝑛 ranked cases are selected as few-shot exemplars. The parameter 𝜆 serves as an explicit diversity control factor, regulating the trade-off between semantic relevance to the query and diversity among the selected demonstrations. A larger 𝜆 emphasizes query relevance, while a smaller 𝜆 increases the diversity characteristic, encouraging the selection of mutually distinctive cases. Scase = DiverseSim(𝑙 𝑞𝑢𝑒𝑟 𝑦 , 𝐷 𝑑𝑜𝑚𝑎𝑖𝑛 , 𝑛, 𝜆)
(5)
Through this process, we obtain the top-𝑛 semantically similar legal cases, forming the case-based exemplars for subsequent hybrid fewshot in-context learning. 5.1.2 Entity-agnostic Few-shot Selection. While semantic case-based retrieval provides relevant contextual demonstrations, legal texts are sometimes dominated by lengthy and highly specific entity mentions, such as detailed descriptions of illegal activities, harms, assets, and contract identifiers. These entity-heavy expressions may distort similarity measurements and bias retrieval toward surface-level entity overlap rather than legally meaningful reasoning patterns. To mitigate this issue, we propose an entity-agnostic few-shot selection strategy based on template content. Specifically, given a legal case 𝑙𝑖 and a query case 𝑙 query , we first apply a template function 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 (·) to transform both cases into entity-agnostic templates. This function substitutes concrete entity instantiations with their entity type, preserving the underlying legal relations and structure of the case. The top-𝑚 most relevant exemplars are retrieved by
the DiverseSim algorithm. This template-based retrieval mechanism enables the selection of structurally aligned demonstrations even when the original cases differ substantially in their surface entities. The template-based similar exemplars are retrieved as: T = {(template(𝑙𝑖 ), 𝑓𝑖 )}𝑙𝑖 ∈ D𝑑𝑜𝑚𝑎𝑖𝑛 Stemplate = DiverseSim(𝑙 𝑞𝑢𝑒𝑟 𝑦 , T , 𝑚, 𝜆)
(6) (7)
where 𝑙𝑖 and 𝑓𝑖 represent the legal case and the corresponding fact formulas, respectively. As in the case-based retrieval, the parameter 𝜆 serves as a diversity score that balances relevance and diversity during exemplar selection. This process yields the top𝑚 template-based exemplars, serving as entity-agnostic few-shot demonstrations. 5.1.3 Diversity-aware Few-shot Combination. We integrate the retrieved template-based exemplars (the top-𝑚 templates) with the 𝑛 case-based exemplars into a unified few-shot prompt, forming a diversity-aware hybrid in-context learning environment. This complementary design enables the model to simultaneously leverage realistic legal contexts and structurally rich reasoning patterns, thereby improving robustness and generalization in legal logical inference. Sfew = Scase ∪ Stemplate
(8)
where the hyperparameters 𝑛 and 𝑚 control the balance between semantic and structural relevance in the in-context demonstrations, respectively.
5.2
Prompting Construction and Inference
This subsection describes the construction of the input prompt using the selected exemplars from the previous step. As illustrated in Table 3, the prompt is composed of two complementary types of demonstrations introduced in Section 5.1: case-based exemplars and template-based exemplars. These demonstrations are concatenated sequentially to form a unified in-context learning prompt, which is then provided to the large language model for legal semantic parsing. 𝑓 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 = LLM-decode(prompting(𝑙 𝑞𝑢𝑒𝑟 𝑦 , S𝑓 𝑒𝑤 ))
(9)
ICAIL’26, June 8–16, 2026, Singapore
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh
We defer a detailed analysis of the characteristics and effects of different few-shot retrieval strategies to the experimental section, where we systematically examine how various combinations of casebased and template-based exemplars influence model performance.
where correctness is defined by the accurate identification of all target entities. In contrast, our method directly translates a legal case description into a complete PROLEG fact formula, which necessitates a stricter, sentence-level exact match evaluation to faithfully reflect the precision requirements of automated legal reasoning. In addition, to provide a more comprehensive evaluation beyond exact matching, we introduce a semantic-aware metric, SoftMatch Accuracy, that preserves structural correctness while relaxing surface-form constraints on entities. Specifically, we require an exact match on the logical structure of the predicted formula, 𝑝𝑟𝑒𝑑 and measure semantic similarity between predicted 𝑒𝑘 and gold
Table 3: Prompting template combining case-based and template-based demonstrations. The blue text indicates content that is replaced with data from the selected few-shot set, Sfew , while the red text denotes the content to be generated by the LLM. ### You are an expert in the Semantic parsing task, which maps from legal cases to logical formulas (Note: following the exact function name defined in the fewshot samples).
𝑔𝑜𝑙𝑑
entities 𝑒𝑘
using cosine similarity: 𝑁
𝑆𝑜 𝑓 𝑡-𝑀𝑎𝑡𝑐ℎ 𝐴𝑐𝑐. = ### Input: {{ Selected legal case, 𝑙𝑖 ∈ S𝑓 𝑒𝑤 }} ### Logical Formulas Template: {{ template( 𝑓𝑖 ) where 𝑓𝑖 ∈ S𝑓 𝑒𝑤 }} ### Output: {{ Logical facts, 𝑓𝑖 ∈ S𝑓 𝑒𝑤 }} ... ### Input: {{ Query legal case, 𝑙 𝑞𝑢𝑒𝑟 𝑦 }} ### Logical Formulas Template: {{ template( 𝑓 𝑞𝑢𝑒𝑟 𝑦 ) }} ### Output: {{Logical facts, 𝑓 𝑞𝑢𝑒𝑟 𝑦 }}
6 Experiments 6.1 Datasets and Experimental Settings To evaluate the generalization ability of the proposed framework, Legal2LogicICL, we primarily conduct experiment on Legal2Proleg dataset, which is an extended version of the Legal2ProlegV0 dataset. In addition, we also conduct extensive experiments on the LegalCaseNER dataset to highlight the differing characteristics between the two datasets. The detailed comparison among three datasets is presented in Table 2. We evaluate our approach on a diverse set of representative large language models, including the open-source models Qwen3-8B, Qwen3-14B [25], Llama-3.1-8B-Instruct [6], and Phi-4 [1], as well as the proprietary model gpt-5.2 accessed via the OpenAI API. In our implementation of DiverSim, Qwen/Qwen3-Embedding-8B model was used to encode the query and compute the similarity. All experiments are conducted with five different random seeds, and results are reported as the average performance across runs to reduce variance due to stochastic sampling.
6.2
Evaluation Metric
For evaluation, we adopt exact match accuracy as the primary evaluation metric following previous works [19, 27] and Semanticaware Evaluation. This choice is motivated by the strict syntactic and compositional constraints of PROLEG fact formulas, where even minor deviations render a formula invalid for legal reasoning. Accordingly, a prediction is counted as correct only if the entire generated formula–including all PROLEG functions and entities– exactly matches the reference: 𝑁 1 ∑︁ pred gold 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = I 𝑓𝑖 = 𝑓𝑖 (10) 𝑁 𝑖=1 where 𝑁 is the number of evaluation samples. It is worth noting that LegalCaseNER [27] formulates the problem as an NER task,
1 ∑︁ pred gold I struct(𝑓𝑖 ) = struct(𝑓𝑖 ) 𝑁 𝑖=1
\
𝐾
×
1 ∑︁ pred gold sim Emb(𝑒𝑖,𝑘 ), Emb(𝑒𝑖,𝑘 ) (11) 𝐾 𝑘=1
where 𝐾 denotes entity index in legal fact formulas, struct(·) denotes the structural form of logical expressions with entity information removed, and Emb(·) represents the embedding function using Qwen/Qwen3-Embedding-8B model. This metric enables a more fine-grained evaluation by accounting for semantic equivalence between entities, while strictly enforcing structural correctness.
6.3
Experimental Results
6.3.1 Performance Comparison under Varying Training Data Proportions. We conduct a comparative evaluation between NER-based supervised models and our Legal2LogicICL framework on the Legal2Proleg dataset under varying training data splits, with the proportion of seen data ranging from 0.2 to 0.8. To construct these splits (seen data), we apply a repeated hold-out strategy. In the dataset construction stage, each template is used to generate one unique sample, ensuring that different samples correspond to different templates. During data splitting, the dataset is randomly partitioned into training and test sets while ensuring that templates appearing in the test set do not occur in the training set, thereby preventing template overlap across splits. As illustrated in Figure 2, when the seen data ratio is relatively high (0.8), the NER-based model LegalCaseNER achieves a competitive average performance of 88.51%, compared to 95% obtained by our method (phi-4). However, as the amount of labeled training data decreases, the performance gap between the two approaches becomes increasingly pronounced. In the low-resource setting with a seen data ratio of 0.2, the accuracy of the NER-based model drops sharply to 16.61%, whereas our Legal2LogicICL maintains a substantially higher performance of 83%. These results demonstrate that our Legal2LogicICL framework exhibits strong robustness across different data availability settings and is significantly less sensitive to the proportion of labeled training data. This property is particularly critical for real-world deployment, where user inputs are inherently open-ended and cannot be exhaustively covered by pre-annotated training cases. In contrast, the LegalCaseNER approach degrades severely in the presence of a large amount of unseen data, indicating its limited generalization capability under low-resource conditions. This limitation poses a significant risk in practical applications, as it
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning
1
Acc. 0.83
0.92 0.93 0.94
0.95
(ICL) Phi-4 14B (ours) (ICL) Qwen3-14B (ours) (ICL) gpt-5.2 (ours) (ICL) Qwen3-8B (ours) (SFT) LegalCaseNER [27]
0.5
0.2
0.4 0.5 0.6
0.8
#seen data
Figure 2: Performance comparison between NER-based method LegalCaseNER and few-shot learning methods Legal2LogicICL on the Legal2Proleg dataset with respect to the portion of seen data. Results are averaged over five random seeds, corresponding to the shaded lines. is unrealistic to guarantee sufficient labeled training data for every possible legal scenario encountered in free-form user inputs. For the ChatGPT experiment (gpt-5.2), due to the high cost of OpenAI services, we conducted evaluations only under the seendata ratio of 0.6 (dashed line). The results demonstrate that our Legal2LogicICL framework has the potential to work effectively across different LLMs. Further analysis of the gpt-5.2 results is provided in Section 6.4.4, with representative error examples presented in Table 8. 6.3.2 Effect of Diversity Control Parameter (𝜆). As shown in Figure 3, a moderate setting of the diversity control parameter (𝜆 = 0.6) consistently achieves the highest diversity across all evaluated model backbones. This observation highlights the importance of maintaining an appropriate level of diversity in latent semantic vector representations (deep-level). The results indicate that all LLMs exhibit the same performance trend. When the 𝜆 value is decreased (e.g., 𝜆 = 0.2), diversity is emphasized, yielding exemplars that remain relevant to the query while being more diverse with respect to one another. Conversely, when the 𝜆 value is increased (e.g., 𝜆 = 0.8), similarity is prioritized, resulting in exemplars that are more similar to the query and to each other. 0.94 0.92 0.9
Acc. Phi-4 14B Qwen3-14B 𝜆 0.2
0.4
0.6
0.8
0.84 Acc.
Qwen3-8B Llama-3.1-8B
0.82
𝜆 0.2
0.4
0.6
0.8
Figure 3: Performance comparison on the effect of the diversity control parameter (𝜆) across diverse LLMs under identical ICL experimental settings (seen data rate = 0.6). Results are averaged over five random seeds for each setting.
ICAIL’26, June 8–16, 2026, Singapore
6.3.3 Effect of hybrid few-shot strategy. As shown in Table 4, under identical data rate and diversity control settings, the choice of 𝑘shot strategy has a substantial impact on performance. Specifically, 3c and 5c denote three and five case-based few-shot examples, respectively, while 3c+3t (𝑛 = 3, 𝑚 = 3) represents our proposed hybrid strategy combining three case-based and two template-based examples. Here, in the setting of 5c, comparison of our DiverseSim (𝜆 = 0.6) with a cosine similarity–based method (𝜆 = 1, where diversity is disabled and only similarity-based examples are selected) shows that performance consistently drops across LLMs. This confirms that diversity-aware retrieval leads to better performance than purely similarity-based selection, demonstrating the effectiveness of the diversity characteristic. Across all evaluated configurations, the proposed 3c+3t strategy consistently achieves the best performance across four backbone models. These results indicate that increasing the number of semantically retrieved examples improves performance, but the model remains sensitive to entity-induced retrieval bias. The hybrid fewshot strategy incorporating both semantic cases and template-based entity-agnostic exemplars consistently yields further gains, even under the same total number of few-shot examples (k=5). This confirms that the proposed hybrid retrieval strategy effectively balances semantic relevance and structural diversity. Table 4: Ablation study on the impact of different 𝑘-shot strategies under identical seen data rate (= 0.6). 𝑘-shot
𝜆
Qwen3-8B
Llama-3.1-8B
Qwen3-14B
Phi-4
3c 5c 5c 3c+3t
0.6 1.0 0.6 0.6
0.7075 0.7899 0.8005 0.8472
0.7824 0.8171 0.8246 0.8327
0.8437 0.8930 0.8955 0.9216
0.8894 0.9141 0.9216 0.9412
6.3.4 Effect of Entity Name Bias. As shown in Table 5, we evaluate our proposed Legal2LogicICL framework on the LegalCaseNER dataset [27] and re-implement their proposed approach. The SFTbased NER model achieves near-perfect performance on this dataset. This outcome can be attributed to two primary factors: (1) the dataset contains a small number of patterns, with the smallest template vocabulary but the largest number of samples among the evaluated datasets (Table 2); and (2) the training and test sets share the same templates, which makes the model prone to overfitting and limits its robustness in real-world scenarios. In contrast, the main source of errors in our ICL-based approach Legal2LogicICL stems from minor variations in entity surface forms, such as missing articles (e.g., “a” or “the”). These errors are closely related to annotation consistency in the dataset and do not affect the logical soundness of the resulting legal reasoning representations.
6.4
Result Analysis
6.4.1 Diversity Analysis in Latent Semantic Vector Space. Here, we analyze the effect of DiverseSim in comparison with conventional cosine similarity for the few-shot exemplar selection process. Figure 4 visualizes the semantic representations of all legal cases in the
ICAIL’26, June 8–16, 2026, Singapore
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh
Table 5: Impact of entity name bias on model performance.
matching effectively increases structural diversity within the fewshot context, complementing semantic relevance and contributing to more robust in-context legal reasoning.
Seen data 40% 60% 80%
Legal2LogicICL (Qwen3-8B)
Legal2LogicICL (Qwen3-14B)
(SFT) LegalCaseNER (Roberta-large)
0.7973 0.8353 0.8466
0.9000 0.8975 0.8981
0.9957 0.9988 0.9953
training set (seen data) using t-SNE dimensionality reduction [24]. Different clusters correspond to distinct legal issues or contract types. This visualization demonstrates that the exemplars selected by DiverseSim exhibit are more widely distributed within each semantic cluster, demonstrating greater semantic diversity, whereas cosine similarity-based retrieval tends to concentrate exemplars within a narrower region around the query representation.
t-SNE visualization 40
Dim 2
20
Training samples DiverseSim. (ours) CosineSim. Query
0 20 40 60
40
20 0 20 40 Dim 1 Figure 4: Visualization of our ranking method (DiverseSim) with 𝜆 = 0.6. Each data point represents a sample in the Legal2Proleg dataset. The highlighted points indicate the few-shot exemplars selected by DiverseSim and by cosine similarity with respect to the query, respectively.
6.4.2 Diversity Analysis of Retrieved Few-shot Exemplars. To better understand the effectiveness of the proposed hybrid few-shot strategy, we analyze exemplar diversity under two retrieval settings: (i) semantic retrieval based on cosine similarity, and (ii) templateaware, entity-agnostic selection based on template-level matching. Table 6 presents representative examples from both methods. As illustrated in the table, semantic retrieval tends to select exemplars that are highly similar to the query at the surface level, including identical entities (e.g., the same names such as Alex and Jordan) and evaluative expressions (e.g., “What an inventive version!”), as well as nearly identical narrative structures. Beyond entity overlap, most retrieved examples exhibit minimal structural variation, indicating that the retrieval process is dominated by entity-level similarity and provides limited diversity in legal reasoning patterns. In contrast, the proposed template-aware entity-agnostic retrieval strategy abstracts away from concrete entities and emphasizes structural variations across cases. While the retrieved exemplars share the same high-level legal structure as the query case, they exhibit greater diversity in reasoning paths and predicate instantiations. This qualitative difference indicates that template-level
6.4.3 Error Analysis of LegalCaseNER. To better understand the limitations of NER-based legal semantic parsing methods, we conduct a qualitative error analysis of LegalCaseNER. Although NERbased approaches are capable of identifying surface-level entities in legal texts, we observe that they often fail to capture legally meaningful entities and relations, particularly when training data is limited. Table 7 presents a representative example illustrating several typical failure modes. For clarity, we highlight incorrect entity predictions in red and their corresponding gold-standard entities in blue, while leaving correctly predicted entities unmarked. Shallow and Incorrect Entity Boundary Detection. As shown in the example, the gold entity (the conference room)[Object] is incorrectly predicted as (the)[Asset] conference room. Here, the model assigns the entity label Asset to a function word (the), while excluding the actual legal object (conference room) from the entity span. This behavior indicates that the NER model relies heavily on local lexical cues rather than semantically coherent spans, leading to fundamentally incorrect entity grounding. Such boundary errors directly propagate to downstream logical parsing, resulting in invalid or incomplete fact formulas. Failure under Overlapping or Semantically Composite Entities. Legal texts frequently express harms, obligations, or violations as semantically rich clauses rather than isolated noun phrases. In the gold annotation, the harm is represented as a single composite entity: (Lucas’s reputation was compromised as clients began avoiding the venue due to the unexpected activities)[Harm]. However, LegalCaseNER fragments this span into multiple unrelated entities, such as (Lucas’s)[Lender] and (the venue)[Object]. This demonstrates that NER-based models struggle when a legally salient concept subsumes multiple surface-level entities, making it difficult to recover higher-order legal facts such as reputation damage. Confusion between Semantically Similar Entity Types. We further observe frequent confusion between temporally related entity types. In the example, the discovery time (2023/06/30)[T_discovery] is incorrectly labeled as (2023/06/30)[T_due]. This error reflects an inherent limitation of flat NER labeling schemes in distinguishing legally distinct but lexically similar temporal concepts. Such confusion can critically affect legal reasoning, as different temporal roles often trigger different legal consequences. Discussion. These reveal fundamental limitations of NER-based approaches. By design, LegalCaseNER focuses on identifying surfacelevel spans and assigning fixed entity labels, without explicitly modeling underlying legal structures or reasoning patterns. As a result, it is particularly fragile when faced with complex legal narratives, overlapping entity semantics, and limited training data. In contrast, our proposed framework avoids explicit entity labeling during inference and instead leverages structurally grounded, template-level reasoning through in-context learning, making it more robust to such entity-induced noise. 6.4.4 Error Analysis of Legal2LogicICL. In this section, we conduct an error analysis of the outputs generated by ChatGPT-5 (gpt-5.2).
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning
ICAIL’26, June 8–16, 2026, Singapore
Table 6: Qualitative comparison of few-shot exemplars retrieved by semantic-only and template-aware strategies. Case
Content
Query Case
Artist Jordan admires musician Alex and habitually listened to Alex’s composition. Jordan released a new composition by adjusting the vocals while maintaining the hook, sharing it as his own on SoundCloud. Eventually, Alex acknowledged, “What an inventive version!”, causing the composition to exceed one million plays.
Exemplar
Semantic-based Retrieved Exemplars
Template-based Retrieved Exemplars
Example_1
Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes. Musician Casey has a fan named Chris who played regularly Casey’s tune. Chris created a new track by changing the melody slightly but keeping the hook and uploaded it as Chris’s own on Instagram. Subsequently, Casey responded, “Amazing interpretation!”, leading to the track amassing a million streams. Artist Jamie has a follower in Sam who practiced daily Jamie’s single. Sam composed a new melody by modifying the chords but keeping the pre-chorus and published it on TikTok as Sam’s piece. Finally, Jamie expressed, “Fantastic transformation!”, resulting in the track garnering over three million plays.
Musician V is admired by person U and often vibed to V’s rhythm. U crafted a fresh tune by preserving the tempo but altering the arrangement, subsequently sharing it on Instagram as U’s composition. Eventually, V said, “This is captivating,”, leading the track to amass over two million streams. T is a fan of musician U and heavily sampled U’s piece. T crafted a new tune while keeping the vocal style retained but changed the tempo. T shared this on his SoundCloud as original.
Example_2
Example_3
Table 7: Representative Errors of LegalCaseNER (incorrect predictions in red, corresponding gold entities in blue) (Prediction) (Lucas)[Lender] provided (the)[Asset] conference room to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize ..., (Lucas’s)[Lender] reputation was compromised as clients began avoiding (the venue)[Object] due to the unexpected activities. The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_due]. (Gold Ouput) (Lucas)[Lender] provided (the conference room)[Object] to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize ..., (Lucas’s reputation was compromised as clients began avoiding the venue due to the unexpected activities)[Harm]. The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_discovery].
Table 8: Representative Errors of Legal2LogicICL (incorrect entities in red, corresponding gold entities in blue) (Prediction) borrower(’Mason’). lender(’Emma’). owned_by(’the laptop’, ’Emma’). borrowing_agreement(’lease78’). damage_fact(’Mason’, ’the laptop’). repair_payment_request_fact(’Mason’,’Emma’, ’the laptop’). repair_request_fact(’Mason’,’Emma’, ’the laptop’, ’2024/01/10’). demob :- block(right_to_dispute_repair_demand(’Emma’,’Mason’, ’the laptop’, ’lease78’)). (Gold) borrower(’Mason’). lender(’Emma’). owned_by(’a laptop’, ’Emma’). borrowing_agreement(’lease78’). damage_fact(’Mason’, ’a laptop’). repair_payment_request_fact(’Mason’,’Emma’, ’a laptop’). repair_request_fact(’Mason’,’Emma’, ’a laptop’, ’2024/01/10’). demob :- block(right_to_dispute_repair_demand(’Emma’,’Mason’, ’a laptop’, ’lease78’)).
Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes.
analysis reveals that a frequent class of errors corresponds to minor surface-form inconsistencies, as illustrated in Table 8. These errors mainly involve confusion between indefinite and definite articles (e.g., “a” and “the”), missing or redundant articles, as well as occasional omissions of punctuation symbols such as periods. Such errors account for 17 out of 38 incorrect cases, corresponding to approximately 5% of the total predictions. Although these cases are counted as errors under our strict evaluation protocol, they are largely attributable to annotation inconsistencies in the dataset, which was labeled by multiple annotators. Importantly, these surface-level variations may not affect the underlying legal semantics or the correctness of logical reasoning, and would not lead to erroneous conclusions in practical legal reasoning systems. We therefore report these results to provide a transparent and comprehensive assessment of model behavior. Effect of Miss Matching Entity Name. In order to evaluate the affect of miss matching entity name in the semantic parsing process of our Legal2LogicICL framework, we report the different between two evaluation metrics Exact-Match and Soft-Match (described in sec 6.2) in Table 9. Similar to the observation in gpt-5.2, we found that, our framework archived high performance in parsing the structure of logical expressions. Although the entity name some how hard to exactly match with the gold data but it can keep the major semantic meaning, which may not affect to the legal reasoning process, which this system can be contributed to. Table 9: Comparison on two evaluation metrics, Exact-Match Acc. and Soft-Match Acc., under settings of 3𝑐 + 3𝑡 and 𝜆 = 0.6.
7 We manually inspect the predictions produced under the setting of 𝜆 = 0.6 with a seen-data rate of 60%, using one representative random seed (overall result shown in Figure 2, dashed line). Our
Eval.Metric
Qwen3-8B
Llama-3.1-8B
Qwen3-14B
Phi-4
Exact-Match Acc. Soft-Match Acc.
84.72 92.18 (+7.46)
83.27 97.68 (+14.44)
92.16 97.52 (+4.91)
94.12 98.00 (+3.88)
Conclusion
This paper presents Legal2LogicICL, a novel in-context learning method for transforming natural language legal case into formal PROLEG logical facts. A key contribution of this work lies in the
ICAIL’26, June 8–16, 2026, Singapore
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, and Ken Satoh
design of a diversity-aware hybrid few-shot retrieval strategy, implemented via the DiverseSim ranking algorithm, which jointly considers semantic case-level similarity and entity-agnostic templatelevel matching. By balancing contextual relevance with structural diversity, Legal2LogicICL constructs more informative and robust in-context demonstrations, leading to more accurate and stable logical rule generation. In addition, we introduce Legal2Proleg, a new benchmark dataset featuring annotated legal rules and corresponding PROLEG logical formulas, to support the systematic evaluation of legal semantic parsing. Experimental results across both openand closed-source LLMs demonstrate that Legal2LogicICL consistently improves accuracy, stability, and generalization in parsing natural language legal cases into formal logical representations. Overall, this work provides a practical and effective few-shot pa radigm for explainable and reliable legal reasoning. Beyond the specifics of semantic parsing, the proposed retrieval strategy and in-context learning framework are generalizable and can be seamlessly integrated into legal reasoning systems, offering a scalable and interpretable foundation for future research in legal AI.
[11] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Eneko Agirre, Marianna Apidianaki, and Ivan Vulić (Eds.). Association for Computational Linguistics, Dublin, Ireland and Online, 100–114. doi:10.18653/v1/2022.deelio-1.10 [12] L Thorne McCarty. 2007. Deep semantic interpretations of legal texts. In Proceedings of the 11th international conference on Artificial intelligence and law. 217–224. [13] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 11048–11064. doi:10.18653/v1/2022.emnlpmain.759 [14] Phuong Minh Nguyen, Truong Dinh Do, and Minh Le Nguyen. 2025. Improving hierarchical semantic parsing with LLMs: Demonstration selection and chain-ofthought prompting via semantic fragment decoding. Knowledge-Based Systems 328 (2025), 114256. doi:10.1016/j.knosys.2025.114256 [15] María Navas-Loro, Ken Satoh, and Víctor Rodríguez-Doncel. 2018. ContractFrames: bridging the gap between natural language and logics in contract law. In JSAI International Symposium on Artificial Intelligence. Springer, 101–114. [16] Ha-Thanh Nguyen, Fungwacharakorn Wachara, Fumihito Nishino, and Ken Satoh. 2022. A multi-step approach in translating natural language into logical formula. In Legal Knowledge and Information Systems. IOS Press, 103–112. [17] Minh-Phuong Nguyen, Thi-Thu-Trang Nguyen, Vu Tran, Ha-Thanh Nguyen, Le-Minh Nguyen, and Ken Satoh. 2022. Learning to Map the GDPR to Logic Representation on DAPRECO-KB. In Intelligent Information and Database Systems, Ngoc Thanh Nguyen, Tien Khoa Tran, Ualsher Tukayev, Tzung-Pei Hong, Bogdan Trawiński, and Edward Szczerbicki (Eds.). Springer International Publishing, Cham, 442–454. [18] Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable Semantic Parsing via Retrieval Augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7683–7698. doi:10.18653/v1/2021.emnlp-main.607 [19] Nguyen Phuong, Nguyen Thanh, May Zin, and Ken Satoh. 2026. Data Augmented Pipeline for Legal Information Extraction and Reasoning. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law (ICAIL ’25). Association for Computing Machinery, New York, NY, USA, 481–482. doi:10. 1145/3769126.3769200 [20] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning To Retrieve Prompts for In-Context Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 2655–2671. doi:10.18653/v1/2022.naacl-main.191 [21] Ken Satoh. 2023. PROLEG: Practical Legal Reasoning System. Springer Nature Switzerland, Cham, 277–283. https://doi.org/10.1007/978-3-031-35254-6_23 [22] Ken Satoh, Masahiro Kubota, Yoshiaki Nishigai, and Chiaki Takano. 2009. Translating the Japanese presupposed ultimate fact theory into logic programming. In Legal Knowledge and Information Systems. IOS Press, 162–171. [23] Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained Language Models Yield Few-Shot Semantic Parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wentau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7699–7715. doi:10.18653/v1/2021.emnlp-main.608 [24] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605. http: //jmlr.org/papers/v9/vandermaaten08a.html [25] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [26] Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. arXiv preprint arXiv:2211.04486 (2022). [27] May Myo Zin, Ha Thanh Nguyen, Ken Satoh, Saku Sugawara, and Fumihito Nishino. 2023. Improving translation of case descriptions into logical fact formulas using legalcasener. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. 462–466.
Acknowledgments This work was supported by the “R&D Hub Aimed at Ensuring Transparency and Reliability of Generative AI Models” project of the MEXT, by JSPS KAKENHI Grant Numbers, 25H00522 and 25H01112, and by JST as part of Adopting Sustainable Partnerships for Innovative Research Ecosystem (ASPIRE), Grant Number JPMJAP25B2.
References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). [2] Imran Sarwar Bajwa, Mark G Lee, and Behzad Bordbar. 2011. SBVR Business Rules Generation from Natural Language Specification.. In AAAI spring symposium: AI for business agility. 2–8. [3] Tom Brown, Benjamin Mann, Nick Ryder, and et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1877–1901. [4] Amir DN Cohen, Shauli Ravfogel, Shaltiel Shmidman, and Yoav Goldberg. 2024. Diversity Over Quantity: A Lesson From Few Shot Relation Classification. arXiv preprint arXiv:2412.05434 (2024). [5] Shruti Gaur, Nguyen H Vo, Kazuaki Kashihara, and Chitta Baral. 2014. Translating simple legal text to formal representations. In JSAI International Symposium on Artificial Intelligence. Springer, 259–273. [6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [7] Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 4966–4979. doi:10.18653/v1/2022.findingsemnlp.365 [8] Nikolaos Lagos, Frederique Segond, Stefania Castellani, and Jacki O’Neill. 2010. Event extraction for legal case building and reasoning. In International Conference on Intelligent Information Processing. Springer, 92–101. [9] Itay Levy, Ben Bogin, and Jonathan Berant. 2023. Diverse Demonstrations Improve In-context Compositional Generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 1401–1422. https://aclanthology.org/2023.acl-long.78 [10] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459–9474. https://proceedings.neurips.cc/ paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf