arXiv:2604.20523v1 [cs.SE] 22 Apr 2026
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis Viet-Man Le
Thi Ngoc Trang Tran
Sebastian Lubos
[email protected] Graz University of Technology Graz, Austria
[email protected] Graz University of Technology Graz, Austria
[email protected] Graz University of Technology Graz, Austria
Alexander Felfernig
Damian Garber
[email protected] Graz University of Technology Graz, Austria
[email protected] Graz University of Technology Graz, Austria
Abstract We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 standard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88–89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy–cost trade-offs that inform model selection. These findings position LLMs as lightweight assistants for early variability validation.
CCS Concepts • Software and its engineering → Abstraction, modeling and modularity; Model-driven software engineering; Automated static analysis; • Computing methodologies → Natural language processing.
Keywords Feature Model Analysis Operations, Software Product Line Engineering, Software Product Line Scoping, Large Language Models ACM Reference Format: Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos, Alexander Felfernig, and Damian Garber. 2026. Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis. In The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3748522.3779903
1
Introduction
Software Product Line Engineering (SPLE) systematically manages variability and enables large-scale reuse in software-intensive systems [2, 9]. A critical early stage is Software Product Line (SPL) scoping, where engineers and domain experts define the boundaries
This work is licensed under a Creative Commons Attribution 4.0 International License. SAC ’26, Thessaloniki, Greece © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2294-3/2026/03 https://doi.org/10.1145/3748522.3779903
of the product line, identify candidate features, and make assumptions about variability [25]. Scoping outcomes have a profound influence on all subsequent phases, since they determine which requirements, architectures, and reusable assets will be engineered. Empirical studies confirm that incorrect or incomplete scoping decisions can cascade into costly rework and misalignment with business goals. Industrial practitioners continue to view scoping as one of the most persistent challenges in SPLE adoption [3]. To ensure correct variability decisions, researchers have developed a broad catalogue of automated analysis operations (AOs) for feature models (FMs) [5, 13]. These operations extract information from FMs and verify properties that are difficult or infeasible to assess manually, especially as models scale to hundreds or thousands of features [5]. Typical examples include detecting dead or false optional features, validating satisfiability, computing valid configurations, and estimating configuration-space size. Such analyses are vital for maintaining FM correctness and scalability [13]. They are typically applied only after an FM has been constructed during domain requirements engineering, delaying feedback until late in the process when corrections are costly and disruptive [14, 20]. This paper investigates the potential of applying automated analysis already during the scoping phase, where assumptions about variability are first articulated. We introduce an early validation workflow in which these assumptions are expressed in semi-formal textual blueprints [26] and analyzed directly with Large Language Models (LLMs). The workflow provides immediate feedback on feasibility, consistency, and potential defects before formal feature models are constructed, thereby bridging informal scoping practices with automated validation and enabling domain experts to iteratively refine scope decisions. This early validation step avoids the need for a solver or formal knowledge base, reducing modeling effort and enabling earlier detection and discussion of design issues. Recent advances in LLMs suggest that this approach is feasible. Studies show that, when carefully prompted, LLMs can perform deductive and logical inference beyond surface-level text understanding [17, 24, 30, 31, 37], and can map natural language into formal specifications such as answer set programs [18], constraint satisfaction problems (CSPs) [16, 26], or Universal Variability Language (UVL) feature models [11]. These findings indicate that LLMs may function as lightweight inference engines over semi-formal representations like blueprints. However, their out-of-the-box ability to perform feature model AOs on such inputs has not been
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
systematically studied. This paper addresses this gap through a large-scale empirical evaluation of 12 state-of-the-art LLMs on 16 AOs, comparing their accuracy, runtime, and failure modes against a solver-based oracle. The contributions of this paper are threefold. Firstly, we formalize and motivate a workflow for early product-line validation, which combines semi-formal blueprints with lightweight LLM-based AOs to provide feedback already during SPL scoping. Secondly, we conduct a large-scale systematic evaluation of off-the-shelf LLMs on feature model analysis tasks using blueprint inputs. Our study covers 12 LLMs (both general-purpose and reasoning-optimized) and 16 AOs across solver-free and solver-based categories. Finally, we systematically evaluate LLM accuracy, cost, and failure modes against the solver-based oracle FLAMA [12], providing insights into their suitability as lightweight assistants for SPL scoping.
2 Background and Related Work 2.1 Software Product Line Engineering The primary objective of SPLE is to reduce development cost and time-to-market while improving quality by deriving families of related products from shared assets rather than engineering each product independently [2, 9]. Its lifecycle is commonly organized into two complementary processes: domain engineering, which establishes reusable core assets, and application engineering, which derives concrete products from them [32]. Domain engineering is typically structured into four phases: (i) scoping, where productline boundaries, candidate features, and variability assumptions are defined; (ii) domain requirements engineering, which captures common and variable requirements; (iii) domain design and implementation, which produce reusable architectures and components; and (iv) domain testing, which validates the shared assets before reuse [2]. Application engineering then configures and assembles products from these assets, while evolution and maintenance activities ensure long-term adaptation of the product line [7].
2.2
Feature Models and Analysis Operations
A central artifact of domain requirements engineering is the feature model (FM), which represents a product line’s commonalities and variabilities through a hierarchical feature tree and cross-tree constraints such as requires and excludes [2, 19]. To validate FMs, the community has established automated analysis operations (AOs) [5, 13]. Solver-free AOs compute structural metrics (e.g., feature counts, tree depth), while solver-based AOs employ SAT/CSP/BDD engines to verify semantic properties such as satisfiability, dead features, and valid configurations [4, 13]. These operations transform FMs from descriptive artifacts into analyzable models, enabling systematic detection of defects difficult to assess manually at scale [5]. The Universal Variability Language (UVL) has emerged as a unified textual DSL for feature modeling [6]. In this study, we use FLAMA [12] to execute solver-based AOs on UVL inputs as our ground-truth oracle. We focus on the Boolean level of UVL, capturing feature hierarchies and cross-tree constraints.
2.3
SPL Scoping
Scoping defines the boundaries of a product line, identifies candidate features, and establishes variability assumptions [25, 32].
V.M. Le et al.
These decisions shape all subsequent engineering activities and are widely recognized as among the most critical yet challenging in SPLE [3]. Current validation practices rely heavily on informal stakeholder workshops, product roadmaps, and expert judgment, leaving consistency and feasibility unchecked until formal feature models exist [14, 20]. This delay causes late-stage rework when variability conflicts or infeasible assumptions surface during requirements engineering or implementation. Integrating automated analysis operations (AOs) directly into scoping can close this gap by providing early feedback on variability properties before formalization, enabling domain experts to iteratively refine scope decisions with confidence [5].
2.4
Large Language Models for Reasoning
Large Language Models (LLMs) such as GPT, Claude, and Gemini are transformer-based architectures with up to hundreds of billions of parameters, pre-trained on large text corpora [8]. Beyond traditional NLP tasks, they exhibit emerging reasoning capabilities when properly prompted, including deductive inference, logical implication, and structured problem solving [17, 30, 37]. Benchmarks such as LogicBench [31] and ZebraLogic [24] confirm this potential but also reveal limitations, e.g., accuracy drops under multi-step inference or complex constraints. Studies further note that many models behave as “greedy reasoners,” favoring short reasoning paths and failing under negation or deep logic. Prompting techniques, such as few-shot, chain-of-thought [8, 21, 35], and rationale decomposition [17], can elicit more structured reasoning without fine-tuning. These approaches have enabled LLMs to generate formal artifacts such as answer-set programs, CSPs, and UVL feature models directly from natural language [11, 16, 18, 26]. For product line engineering, this suggests that well-prompted LLMs may serve as lightweight inference engines for analyzing semi-formal blueprints, providing early validation without relying exclusively on solver technology.
3
Early Validation Workflow for SPL Scoping
We propose an early validation workflow that integrates automated analysis directly into the scoping phase. This is achieved by introducing a lightweight, semi-formal blueprint [26] and leveraging LLMs to analyze it. The workflow proceeds in three steps: (1) Step 1 - Blueprint creation: Domain experts consolidate scope boundaries, candidate features, and variability assumptions into a semi-formal blueprint. A blueprint is a set of textual semiformal constraints that collectively specify the feature hierarchy and cross-tree relations. Examples include “Feature A requires Feature B”, “Feature C can be Feature D or Feature E”, “Feature B excludes Feature D”, or “Feature E can be Feature G, Feature H, or both”. This representation is easy to author yet structured enough for automated checks, bridging free-text scoping notes and formal UVL models. Figure 1 (top-left panel) illustrates a concrete blueprint for a smartwatch product line. (2) Step 2 - LLM-based analysis. The blueprint is fed to an LLM analysis engine that executes AOs. Each AO is guided by a tailored prompt that encodes its reasoning task, ensuring the LLM applies an appropriate inference strategy. The set of AOs
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
is extensible, allowing any operation expressible in natural language to be incorporated into the workflow. (3) Step 3 - Feedback and refinement: The engine returns actionable findings such as inconsistencies, dead features, or product counts. Based on this feedback, the domain expert revises the blueprint and, where necessary, updates scoping outputs. All these three steps are repeated until stabilization. The iteration yields a validated blueprint and updated scoping dossier. These can (i) be compiled to a formal feature model for solver-based verification and (ii) seed draft UVL generation to reduce modeling effort [11]. In short, blueprints capture early variability decisions in a form LLMs can analyze, providing timely feedback before full formalization. The workflow provides the context for our empirical evaluation. Our study does not assess the workflow itself but investigates a critical question it raises: Are off-the-shelf LLMs capable of performing feature model analysis operations directly on semi-formal blueprints? The next section presents our methodology.
4 Methodology 4.1 Research Questions We evaluate whether off-the-shelf LLMs can execute AOs on semiformal blueprints and how they compare to a solver oracle (FLAMA [12]). The evaluation is structured around three research questions: RQ1 Accuracy. How accurately can different LLMs perform solverfree and solver-based AOs on blueprint inputs? RQ2 Cost. How do end-to-end costs (runtime and token usage) relate to achieved accuracy across models? RQ3 Failure modes. What types of errors do LLMs exhibit when performing AOs? How do these vary across model families?
4.2
LLMs Under Study
We evaluate 12 publicly available LLMs, spanning both generalpurpose (e.g., GPT-4.1, Claude Sonnet 4, DeepSeek Chat) and reasoning-optimized variants (e.g., Grok 4 Fast Reasoning, GPT5 mini, Gemini 2.5 Pro). All models are accessed via their official APIs without any fine-tuning, enabling a direct comparison between mainstream and reasoning-focused architectures. To ensure comparability, every model is queried under identical conditions: temperature=0, unrestricted context and output lengths within provider limits, and a uniform prompting and evaluation pipeline. This setup yields deterministic completions that expose each model’s inherent reasoning behavior. Table 1 lists the evaluated models along with their type, context-window size, and maximum output length.
4.3
Blueprints and Dataset
To evaluate LLM-based analysis in a controlled and reproducible setting, we derive blueprints from existing UVL feature models rather than from raw scoping inputs. While scoping artifacts represent the natural source for early validation, they are rarely standardized and lack solver-based ground truth. In contrast, UVL models are publicly available, semantically precise, and compatible with solver analysis, enabling reproducible comparison against an oracle.
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
We collected models from two established repositories, UVLHub [33] and the Feature-Model-Benchmark v1.0 [34], and selected ten representative cases covering both toy (e.g., SW, SMW) and large, real-world product lines (e.g., BDB, CNNl/f). Each model contains at least one cross-tree constraint and varies widely in feature count, tree depth, and constraint density. For each UVL model, we constructed a corresponding blueprint by restating its hierarchy and constraints in constrained natural language, as described in Section 3. For generalization-related AOs, each blueprint was paired with its variant obtained by swapping selected relationships (mandatory ↔ optional, or ↔ alternative). This setting yields controlled pairs that allow testing if one blueprint’s variability space includes the other. All blueprints were manually verified for semantic equivalence to their UVL sources. Table 2 summarizes structural metrics and blueprint sizes, ranging from fewer than 100 to over 70,000 tokens, thus spanning both compact and complex models.
4.4
Analysis Operations
We consider 16 commonly used AOs from feature model research and practice [4, 13], covering both structural metrics (e.g., feature counts, tree depth) and solver-based reasoning tasks (e.g., satisfiability, core features, configuration counting). Table 3 lists all evaluated AOs. Each AO is implemented through a dedicated prompt template that defines its reasoning steps and enforces a standardized output format for comparison with the solver-based oracle. AO12 (#valid configurations) is limited to the first eight FMs due to scalability constraints on the largest cases (CNNl, CNNf). For AO16 (generalization), each test involves a pair of blueprints: the original and its modified variant (see Section 4.3).
4.5
Evaluation Protocol
Prompt Design. For each AO, we use a three-part pipeline illustrated in Figure 1 using dead feature detection as an example. The figure shows how the system prompt, user prompt, and output contract work together to guide the LLM through the analysis of a smartwatch blueprint. We now describe each component: • System prompt. The system prompt defines the model’s role as a domain-specific assistant for feature model analysis. It introduces the blueprint format, explains the elements of feature models (features, hierarchy, and cross-tree constraints), and specifies the semantics and expected outcome of the target AO using precise definitions and result formats. • User prompt. Combines task instructions, illustrative examples, and the target input. It begins with a Learn from Examples section containing 2–4 exemplars that pair blueprints with their corresponding AO results. A subsequent step-bystep procedure describes the reasoning strategy (e.g., feature enumeration, constraint propagation, contradiction checking), followed by the blueprint to be analyzed. This design grounds the task in concrete examples and guides the model through a consistent, reproducible reasoning process. • Output contract. Each AO uses an XML-based output schema to ensure machine-checkable results. The contract enforces standardized tags (e.g., <dead_features>, <core_features>, <feature_model_analysis>) with
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
V.M. Le et al.
Table 1: Evaluated LLMs ordered by variant type, context-window capacity, and max output length. Model
Model ID
Type
Grok 4 Fast Non Reasoning [36] GPT-4.1 [27] Llama 4 Scout [28] Claude Sonnet 4 [1] DeepSeek V3.1 Chat [10] Grok 4 Fast Reasoning [36] Gemini 2.5 Flash [15] Gemini 2.5 Pro [15] Llama 4 Maverick [29] GPT-5 mini [27] Claude Sonnet 4 Thinking [1] DeepSeek V3.1 Reasoner [10]
grok-4-non-reasoning gpt-4.1 llama-4-scout claude-sonnet-4 deepseek-chat grok-4-reasoning gemini-2.5-flash gemini-2.5-pro llama-4-maverick gpt-5-mini claude-sonnet-4-think deepseek-reasoner
Context Window (tokens)
Max Output (tokens)
2M 1M 328K 200K 128K 2M 1M 1M 1M 400K 200K 128K
N/A 32K 16K 64K 8K N/A 65K 65K 16K 128K 64K 64K
General-purpose General-purpose General-purpose General-purpose General-purpose Reasoning-optimized Reasoning-optimized Reasoning-optimized Reasoning-optimized Reasoning-optimized Reasoning-optimized Reasoning-optimized
Notes: “K” = thousand tokens, “M” = million tokens, “N/A” = not specified.
Table 2: Feature models used in the experiments with structural metrics and blueprint sizes. Feature Model
SW
SMW
IDE
SMG
COM
SEA
CVE
BDB
CNNl
CNNf
#Features #Relationships #Cross-tree constraints Tree Depth
6 4 1 2
13 8 2 2
14 11 2 2
33 19 4 3
48 25 21 3
145 73 13 10
169 15 153 4
117 54 282 5
3,296 1,561 76 10
6,867 3,516 9 11
#Blueprint tokens
68
143
178
360
740
1,456
2,367
3,780
33,823
71,812
Abbreviations: SW=Sandwich, SMW=Smartwatch, IDE=IDE product line, SMG=Strategy Mobile Game, COM=Computer, SEA=Subsea Control System, CVE=Cybersecurity Vulnerability, BDB=Berkeley DB, CNNl=light CNN architectures, CNNf=full CNN architectures. Metrics extracted with FLAMA 2.0.1 and Glucose3; blueprint token counts measured in tokens.
Table 3: AOs considered in the experiments. ID
Solver-free AOs
ID
Solver-based AOs
AO1 AO2 AO3 AO4 AO5 AO6 AO7 AO8 AO9
#Features #Leaf Features Tree Depth #Mandatory Features #Optional Features #Or Groups #Alternative Groups #Requires #Excludes
AO10 AO11 AO12 AO13 AO14 AO15 AO16
Satisfiable/Void Configuration Satisfiable #Valid Configurations Core Features Dead Features False Optional Features Generalization
canonical content: integers for counts, true/false for booleans, and newline-separated or tag-enclosed lists for feature sets. Any deviation (e.g., unparseable text, missing tags, malformed lists) is automatically considered incorrect. For set-valued AOs, exact set equality with the solver-based oracle is required. Models are also instructed to include concise natural language justifications, such as explaining why a feature is classified as dead. This uniform pipeline ensures consistent results across models and AOs by prioritizing fair capability comparison over tailored
optimization. Full templates for all 16 AOs are provided in the replication package1 . Inference. Each triple (including model, blueprint, and AO) is executed once with deterministic decoding (temperature=0, greedy) via the provider’s public API only, i.e., no tool use, no solver calls, no post-processing. We record request/response payloads, the raw XML, the rationale, and wall-clock runtime. Besides, timeouts and truncations are kept (and evaluated) as produced. Metrics and error taxonomy. We report the results along the following two metrics: • Accuracy. Exact-match agreement with the FLAMA oracle, broken down by AO, blueprint, and model family (generalpurpose vs. reasoning-optimized). • Cost. End-to-end runtime (in seconds) and token usage (prompt+completion). Errors are categorized into four mutually exclusive failure modes aligned with our analysis: (i) Unparseable (violates XML contract), (ii) Format-correct but wrong (includes semantic slips such as misreading alternative vs. mandatory), (iii) Partial/Truncated (incomplete outputs due to context/output limits), and (iv) Hallucinated elements (items not in the blueprint). 1 Replication package: https://github.com/AIG-ist-tugraz/llm-blueprint-analysis
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
Feature Model Blueprint
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
System Prompt
- A [smartwatch] must have at least one type of [connector] and a [screen].
User Prompt
- The [connector] can be [gps], [cellular], or both.
You are a domain-specific AI assistant specializing in analyzing feature models described in “Blueprint Format”.
- The [screen] can be [analog] or [hires] (high resolution).
Your purpose is to:
- The [smartwatch] may include a [camera] and a [compass].
- Interpret the structural and semantic elements of a feature model…
- [compass] requires a [gps].
Follow these steps: Step 1: Learn from Examples: Use the annotated examples below to understand how dead features are inferred… Step 2: Reasoning:
- Detect dead features using logical reasoning and configuration validation.
- [camera] requires a [hires]. - [cellular] and [analog] exclude each other. - If the [screen] is selected, a [camera] must also be included. Analysis Results
For each feature:
- Provide explanations grounded in feature relationships and constraint interactions.
- Attempt to construct a valid configuration where:
Use the following foundational knowledge to support your analysis:
- Satisfies all relationships and constraints
- That feature is selected. - Avoids triggering any exclusion or logical conflict
Feature Model
<feature_model_analysis>
- If no valid configuration exists due to unsatisfiable relationship/constraint interactions, classify the feature as dead.
A feature model is a structured representation…
<dead_features>[[analog]]</dead_features> <explanation> [analog] is a dead feature because [screen] is mandatory, and [screen] must be either [analog] or [hires]. However, [screen] also requires [camera], and [camera] in turn requires [hires]. This creates a conflict if [analog] is selected, as it cannot satisfy the transitive requirement for [hires]. Therefore, [hires] is always included, and [analog] can never be selected.
Now, analyze the following feature model.
- Explain your reasoning using structural and logical justifications.
Formal Representation A feature configuration task is defined as a triple (F, D, C) where:…
LLMs
</explanation> </feature_model_analysis>
Wrap your detailed reasoning inside:…
Additional Concepts
*Output Format* At the end, wrap your final reasoning in:
- Dead features: Features that *cannot* be included in *any* valid configuration.
… Step 3: Analysis this feature model: Now, analyze the following model to identify any dead features.
Blueprint Format
<feature_model_blueprint_to_analyze>
The Blueprint format is a bullet-point, human-readable way for describing feature models… Each bullet clarifies the relationships between features,…
{{feature_model_blueprint}} </feature_model_blueprint_to_analyze>
Figure 1: Prompting pipeline for dead feature detection, showing how the system and user prompts guide reasoning and how the LLM outputs XML results for solver comparison. 80 40 70 100 70 80 90 100 90 50 90 80 70 90 50 30 50 100 70 70 80 100 100 50 80 70 60 100 30 20 50 90 80 60 80 90 80 50 50 70 60 100 60 10 40 100 70 60 80 80 80 25 60 60 50 90
AO16
AO15
AO14
AO13
AO12
AO11
AO9
AO10
AO8
AO7
AO6
AO5
AO4
llama-4-scout 30 20 30 20 30 40 50 40 70 90 90 13 40 40 20 40 AO3
We use FLAMA 2.0.1 [12] as the solver-based oracle, executing AOs on UVL inputs through the Glucose3 SAT solver and the DD library to obtain exact and reproducible ground-truth results for all comparisons. The LLM-based analysis engine and the evaluation framework are implemented in Python 3.10 within a unified evaluation harness. To ensure modularity and reproducibility, we employ LangChain [22] for standardized API integration, and LangGraph [23] for orchestrating prompting pipelines. All prompts, blueprints, model settings, evaluation scripts, raw outputs, and per-run runtimes are publicly released in the replication package1 .
claude100 50 sonnet-4 gpt-4.1 50 40 grok-4-non40 40 reasoning deepseek-chat 30 30
AO2
Baseline Oracle and Implementation
AO1
4.6
Figure 2: Accuracy (%) of general-purpose LLMs across AOs. grok-4100 80 100 80 90 100 80 90 100 100 100 100 60 90 70 100 reasoning gpt-5-mini 100 80 70 80 80 100 80 100 100 100 100 75 90 90 80 100
Accuracy Across AOs. Solver-based AOs yield higher accuracies than solver-free ones for both model families: 68.8% vs. 56.0% for general-purpose and 83.5% vs. 79.2% for reasoning-optimized 2 Supplementary appendix: https://doi.org/10.5281/zenodo.17913681
AO16
AO15
AO14
AO13
AO12
AO11
AO10
AO9
AO8
AO7
AO6
AO5
Figure 2 and Figure 3 summarize accuracy values across all 16 AOs. Overall, reasoning-optimized LLMs clearly outperform generalpurpose ones, achieving an average accuracy of 81.1% compared to 61.0%. This gap reflects stronger multi-step reasoning and constraint handling. Accuracy patterns, however, vary markedly across operation types and blueprint complexity.
gemini-2.560 60 80 50 80 80 80 70 80 100 90 63 70 80 60 80 flash llama-460 30 30 30 60 80 70 60 80 70 100 13 60 60 40 90 maverick AO4
RQ1: Accuracy of LLM-based AOs
AO3
5.1
claude100 70 80 60 80 100 70 80 90 100 100 75 90 90 80 80 sonnet-4-think deepseek90 80 100 30 50 100 80 90 100 100 90 75 90 80 70 90 reasoner
AO2
Results
Full tables and results are shown in Appendix2 .
AO1
5
gemini-2.5-pro 100 70 90 80 80 100 80 90 100 100 90 100 90 90 80 100
Figure 3: Accuracy (%) of reasoning-opt. LLMs across AOs.
LLMs. Rather than requiring exhaustive enumeration, these solverbased tasks are operationalized as verification. For instance, in AO10, the model is asked to search for contradictions in the blueprint and concludes “satisfiable” if none are found. AO11 analogously checks whether a given configuration violates any constraint before deciding its satisfiability. AO16 reduces to judging whether
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
V.M. Le et al.
claude100 100 sonnet-4
94
88
94
62
88
56
38
63
grok-4100 100 100 100 reasoning
94
88
88
60
73
gpt-4.1 100 100
81
81
56
62
50
69
31
50
gpt-5-mini 100 100 100 100 100 100
94
75
53
67
94
81
81
56
50
50
44
31
25 56
llama-4-scout
75
88
50
50
31
25
44
38
13
13
SMW
IDE
SMG
COM
SEA
CVE
BDB
CNNl
CNNf
Accuracy Across Blueprints. Figures 4 & 5 show accuracies aggregated by blueprint. Across both model families, accuracy decreases steadily with increasing blueprint complexity. All models achieve near-perfect performance on small and shallow blueprints (SW, SMW), moderate accuracy on medium-sized ones (IDE, SMG, COM), and substantial drops on complex models such as SEA, BDB, CNNl, and CNNf. These challenging cases combine large feature counts (up to 7,000), deep hierarchies (depth ≥10), and numerous cross-tree constraints (>300), which amplify semantic misunderstandings and reasoning limitations. Importantly, no single model achieves 100% accuracy across all blueprints, highlighting the difficulty of handling large, constraint-dense feature models. Model Ranking. Among reasoning-optimized LLMs, three models form the top-performing group: Grok 4 Fast Reasoning (89.7%), GPT-5 mini (88.9%), and Gemini 2.5 Pro (88.2%) (see Figure 5). They achieve perfect scores on several simple AOs and maintain stable performance on complex ones, suggesting potential for ensemblebased aggregation. The second-tier reasoning models, Claude Sonnet 4 Thinking (84.5%) and DeepSeek Reasoner (82.8%), show moderate drops on AO12 (both 75%) and AO15 (80% and 70%). They also exhibit structural counting weaknesses: Claude Sonnet 4 Thinking performs lower on AO4 (60%) and AO2/AO7 (70%), while DeepSeek Reasoner falls sharply on AO4 (30%) and AO5 (50%). Gemini 2.5 Flash
94
100
69
94
81
40
67
81
88
94
88
81
75
60
67
gemini-2.5100 100 100 flash
88
94
62
62
75
27
27
81
81
50
56
44
38
13
27
deepseek100 reasoner
Figure 4: Accuracy (%) of general-purpose LLMs on 16 AOs across 10 blueprints. one model subsumes another (i.e., no counterexample is identified). This verification-style framing lowers the need for complete search and tends to be more robust to local parsing noise. By contrast, solver-free AOs (e.g., AO4–AO9) require the model to parse and count structural relationships such as mandatory, optional, or alternative features; small semantic misunderstandings, e.g., interpreting “A must have B or C” as two mandatory children instead of an alternative group, lead to systematic counting errors and lower accuracies. Difficulty levels vary markedly across AOs and model families. For general-purpose LLMs, the hardest AOs are AO2 (leaf count), AO4 (mandatory count), and AO12 (#valid configurations), all averaging below 50% accuracy, reflecting persistent semantic errors and limited reasoning capacity. In contrast, reasoning-optimized LLMs achieve high performance (≥85%) on most tasks but still struggle with AO4 (58.6%) due to semantic misunderstandings, and moderately on AO12 and AO15, which demand constraint propagation or enumeration. Across both families, the easiest AOs are AO6, AO9, AO10, AO11, and AO16 (all above 85%), which involve simpler verification steps rather than complex structural reasoning.
claude100 100 100 sonnet-4-think
CNNf
19
CNNl
44
94
llama-4100 maverick
94
BDB
69
CVE
50
67
SEA
44
53
COM
62
81
SMG
69
100
IDE
81
81
SMW
94
gemini-2.5-pro 100 100 100 100 100
SW
deepseek-chat
SW
grok-4-non100 reasoning
94
Figure 5: Accuracy (%) of reasoning-optimized LLMs on 16 AOs across 10 blueprints. Table 4: Average runtime (in second), token usage, and accuracy across all AOs and blueprints for representative models. Model ID
Runtime (s)
Tokens
Accuracy (%)
grok-4-reasoning gemini-2.5-pro gpt-5-mini deepseek-reasoner claude-sonnet-4 llama-4-scout
3,046.7 2,390.8 6,118.3 9,508.4 1,168.5 680.2
505,874 602,251 843,251 561,290 439,523 374,608
89.7 88.2 88.9 82.8 78.3 42.7
FLAMA (Solver)†
18.45
–
100.0
† Results on formal inputs; solver cannot process semi-formal blueprints.
and Llama 4 Maverick perform worse (60–75%), mainly due to context overflow and output truncation. Within the general-purpose family (see Figure 4), Claude Sonnet 4 leads with 78.3%, followed by GPT-4.1 and Grok 4 Fast Non Reasoning (60–68%), while Llama 4 Scout is the weakest (42.7%). Takeaway. Reasoning-optimized models consistently outperform general-purpose ones on blueprint AOs, but accuracy declines with blueprint size/depth and remains uneven for structural counts (e.g., mandatory/alternative counting).
5.2
RQ2: Efficiency of LLM-Based AOs
Since AOs demand high precision, computational cost is meaningful only in relation to achieved accuracy. A model that spends more time but delivers correct results is preferable over a faster yet unreliable one. Table 4 reports average runtime, token usage, and accuracy across all AOs and blueprints. Reasoning-optimized LLMs are notably more resource-intensive, requiring 2.4k–9.5k seconds and 500K–840K tokens, compared to 0.7k–1.9k seconds and 370K–440K tokens for general-purpose models. The symbolic solver baseline (FLAMA) remains by far the fastest (18.45 seconds) but cannot process semi-formal blueprints directly. Accuracy–Cost Balance. High computational cost does not necessarily guarantee better accuracy. For instance, DeepSeek Reasoner is the slowest model (9,508 seconds) yet achieves only 82.8% accuracy,
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
while Grok 4 Fast Reasoning reaches 89.7% with one-third of the runtime. In contrast, Gemini 2.5 Pro delivers similar accuracy (88.2%) with the lowest runtime among reasoning models (2,391 seconds), representing the most efficient high-accuracy trade-off. GPT-5 mini attains top accuracy (88.9%) but at substantially higher runtime and token cost, suggesting diminishing returns. Meanwhile, generalpurpose models such as Claude Sonnet 4 offer a moderate balance (78.3%, 1,169 seconds), and lightweight models like Llama 4 Scout are fast yet unreliable (42.7%).
5.3
RQ3: Error Analysis
We identify three dominant failure modes. First, semantic slips in solver-free AOs (AO4–AO9), most notably misreading “A must have B or C” as two mandatory children instead of an alternative group. Second, incomplete propagation or enumeration in solver-based AOs, especially AO12 (#valid configurations) and AO15 (#false optional), which require constraint reasoning beyond surface parsing. Third, context and output limits on very large blueprints (CNNl, CNNf), causing truncation or early stopping. These problems intensify in deep or constraint-dense models (SEA, BDB, CNNl/f) and are most pronounced in AO12, AO15, and structural AOs (AO4–AO9). Detailed counts per model and AO are in Appendix2 . Model behaviors reflect these patterns. Grok 4 Fast Reasoning, GPT-5 Mini, and Gemini 2.5 Pro make few but systematic propagation errors. Claude Sonnet 4 often overcounts mandatory features due to confusion between or/alternative and mandatory relationships. DeepSeek Reasoner shows partial outputs while Gemini 2.5 Flash and Llama 4 Maverick suffer truncation near context limits. Lightweight models (Llama 4 Scout, DeepSeek Chat) tend to produce format-correct but semantically wrong answers. These recurring patterns underscore the need for disambiguation rules, output chunking, and model ensembles.
5.4
Overall Synthesis
Across RQ1–RQ3, reasoning-optimized LLMs approach solver-level accuracy on many analyses but remain sensitive to structural ambiguity, scale, and context limits. Most errors stem from incomplete constraint reasoning rather than random noise. In summary, LLMs can already serve as reliable early validators of variability models when paired with simple safeguards and ensemble strategies.
6
Discussion
Where LLMs fit in early scoping. Our results support a pragmatic placement of LLMs at the earliest stage, when blueprints are semiformal and solver-ready model is not yet available. In this context, reasoning-oriented models provide verification-style feedback with near-solver accuracy (e.g., for AO10, AO11, AO16), quick structural sanity checks (e.g., for AO1, AO3, AO6, AO9), and concise explanations that help stakeholders refine scope decisions before formalization. Accuracy and explainability over speed. Because early validation is iterative and not latency-critical, we prioritize correctness and interpretability over raw speed. The observed runtime/token overheads are acceptable in practice given the benefit of solver-like guidance without requiring a formal model. Reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro, GPT-5 mini) therefore offer a practical operating point: near-solver accuracy
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
with stable formatting and useful rationales that fit within minutesscale review cycles typical of scoping workshops. Leveraging model complementarity. Top performers exhibit complementary strengths across blueprints and AOs. Grok 4 Fast Reasoning demonstrates exceptional robustness on large, constraintdense models such as BDB, CNNl, and CNNf, where other models suffer truncation or incomplete propagation. In contrast, GPT-5 mini achieves the highest consistency on mid-size blueprints (COM, SEA) and complex solver-based tasks (AO12, AO15), benefiting from deeper constraint reasoning. Gemini 2.5 Pro performs best on compact to medium blueprints (CVE, SMG) and excels in maintaining strict XML conformance and stable formatting even under long outputs. These differences suggest that the models compensate for each other’s weaknesses: Grok handles scale and constraint density, GPT-5 mini excels in fine-grained constraint reasoning, and Gemini maintains strict output conformance. Simple ensemble aggregation (majority vote or confidence-weighted fusion) mitigates idiosyncratic errors and raises reliability with minimal engineering, matching the “low-friction” goal of early validation. Actionable guidance. To harden early analyses, we recommend four concrete measures aligned with the observed failure modes: (i) add short disambiguation rules that test group semantics before any counting (distinguish or/alternative from mandatory) to reduce semantic slips in AO4–AO9, (2) plan outputs for long blueprints by chunking lists and capping free text to avoid truncation and partial results on large cases (e.g., CNNl/CNNf), (iii) use model ensembles on medium and large blueprints (majority vote or confidence-weighted fusion across top models) to smooth per-model weaknesses in propagation and enumeration (notably AO12 and AO15), (iv) maintain human oversight through stakeholder review and verification protocols to prevent automation bias, as approximately 1 in 10 analyses may contain errors. Threats to Validity. Our evaluation uses blueprints derived from UVL models to ensure reproducibility and oracle comparability. While this enables consistent benchmarking, it may underrepresent ambiguity and noise found in truly informal scoping artifacts. Moreover, the 16 evaluated AOs cover representative but not exhaustive FM tasks. We also evaluated a broad set of public LLM APIs, but not all commercially or academically available models. Results may vary for future versions or domain-specific finetuning. Finally, our strict exact-match scoring penalizes partially correct outputs, which may underestimate the practical reasoning capabilities of models in near-missing cases.
7
Conclusion and Future Work
We evaluated whether off-the-shelf LLMs can perform feature model AOs directly on semi-formal blueprints, where early feedback is most valuable. Reasoning-optimized models reach near-solver accuracy and offer actionable explanations, making them practical assistants for SPL scoping. Their minute-scale runtime is acceptable for accurate, automation-ready feedback before formalization. Remaining issues, such as semantic slips and incomplete propagation, can be mitigated through ensemble use and prompt safeguards. Overall, LLMs provide a feasible path to automated, explainable
SAC ’26, March 23–27, 2026, Thessaloniki, Greece
variability checks prior to solver-based verification, though human oversight remains necessary given their 88–89% accuracy. Future work will expand both the analytical scope and the workflow. We plan to include additional AO (e.g., redundancy, atomic sets, diagnosis/conflict sets) to better approximate the full AO catalogue, and to conduct workflow evaluations in practice with domain engineers to assess usability and feedback quality. We also aim to support blueprint-to-UVL translation with round-trip consistency checks and to develop learned routers that select the most reliable model. Finally, we envision extending the workflow toward scope completeness checks, where LLMs ensure traceability between goals, scenarios, and features—detecting orphaned or missing links to reduce underscoping and misalignment with business objectives.
Acknowledgments The work presented in this paper has been developed within the research project GenRE funded by the Austrian Research Promotion Agency under the project number 915086.
V.M. Le et al.
[16]
[17]
[18]
[19] [20] [21]
[22] [23]
References [1] [2] [3]
[4] [5] [6]
[7]
[8] [9] [10] [11]
[12]
[13]
[14]
[15]
2025. All models overview - Anthropic — docs.anthropic.com. Retrieved September 29, 2025 from https://docs.claude.com/en/docs/about-claude/models/o verview#model-comparison-table. S. Apel, D. Batory, C. Kästner, and G. Saake. 2013. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer. M. Becker, R. Rabiser, and G. Botterweck. 2024. Not quite there yet: remaining challenges in systems and software product line engineering as perceived by industry practitioners. In Proceedings of the 28th ACM International Systems and Software Product Line Conference (SPLC ’24). ACM, Dommeldange, Luxembourg, 179–190. doi:10.1145/3646548.3672587. D. Benavides, A. Felfernig, J. Galindo, and F. Reinfrank. 2013. Automated Analysis in Feature Modelling and Product Configuration. In ICSR’13 (LNCS) number 7925. Springer, Pisa, Italy, 160–175. D. Benavides, S. Segura, and A. Ruiz-Cortes. 2010. Automated analysis of feature models 20 years later: A literature review. Inf. Sys., 35, 615–636, 6. D. Benavides, C. Sundermann, K. Feichtinger, J.A. Galindo, R. Rabiser, and T. Thüm. 2025. UVL: feature modelling with the universal variability language. Journal of Systems and Software, 225, 112326. doi:https://doi.org/10.1016/j.jss.2 024.112326. T. Berger, J.-P. Steghöfer, T. Ziadi, J. Robin, and J. Martinez. 2020. The state of adoption and the challenges of systematic variability management in industry. Empirical Software Engineering, 25, 3, (May 2020), 1755–1797. doi:10.1007/s106 64-019-09787-6. T. Brown et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, (Eds.) Vol. 33. Curran Associates, Inc., 1877–1901. P.C. Clements and L. Northrop. 2002. Software product lines. Addison-wesley. 2025. Models & Pricing | DeepSeek API Docs — api-docs.deepseek.com. Retrieved September 29, 2025 from https://api-docs.deepseek.com/quick_start/pr icing/. J. A. Galindo, A. J. Dominguez, J. White, and D. Benavides. 2023. Large language models to generate meaningful feature model instances. In Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A (SPLC ’23). ACM, Tokyo, Japan, 15–26. doi:10.1145/3579027.3608973. J. A. Galindo, J.-M. Horcas, A. Felferning, D. Fernandez-Amoros, and D. Benavides. 2023. Flama: a collaborative effort to build a new framework for the automated analysis of feature models. In Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume B (SPLC ’23). ACM, Tokyo, Japan, 16–19. doi:10.1145/3579028.3609008. J.A. Galindo, D. Benavides, P. Trinidad, A. Gutiérrez-Fernández, and A. RuizCortés. 2019. Automated Analysis of Feature Models: Quo Vadis? In 23rd International Systems and Software Product Line Conference - Volume A (SPLC ’19). ACM, Paris, France, 302. doi:10.1145/3336294.3342373. S. Ghosh, D. Elenius, W. Li, P. Lincoln, N. Shankar, and W. Steiner. 2016. Arsenal: automatic requirements specification extraction from natural language. In NASA Formal Methods. S. Rayadurgam and O. Tkachuk, (Eds.) Springer International Publishing, Cham, 41–46. 2025. Gemini models | Gemini API | Google AI for Developers — ai.google.dev. Retrieved September 29, 2025 from https://ai.google.dev/gemini-api/docs/mod els.
[24] [25]
[26]
[27] [28] [29] [30] [31]
[32] [33]
[34]
[35]
[36] [37]
L. Hotz, C. Bähnisch, S. Lubos, A. Felfernig, A., and J. Twiefel. 2024. Exploiting large language models for the automated generation of constraint satisfaction problems. 26th International Workshop on Configuration, ConfWS 2024. CEUR Workshop Proceedings, 3812, 91–100. J. Huang and K. C.-C. Chang. 2023. Towards reasoning in large language models: a survey. In Findings of the Association for Computational Linguistics: ACL 2023. A. Rogers, J. Boyd-Graber, and N. Okazaki, (Eds.) Association for Computational Linguistics, Toronto, Canada, (July 2023), 1049–1065. doi:10.18 653/v1/2023.findings-acl.67. A. Ishay, Z. Yang, and J. Lee. 2023. Leveraging Large Language Models to Generate Answer Set Programs. In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning. (Aug. 2023), 374–383. doi:10.24963/kr.2023/37. K. Kang, S. Cohen, J. Hess, W. Novak, and S. Peterson. 1990. Feature-oriented Domain Analysis (FODA) – Feasibility Study. Tech.Rep. – SEI-90-TR-21. C. Khor and R. R. Lutz. 2024. Enhancing the requirements engineering of configurable systems by the ongoing use of variability models. Requirements Engineering, 29, 3, (Sept. 2024), 303–328. doi:10.1007/s00766-024-00421-6. T. Kojima, S. (S.) Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, (Eds.) Vol. 35. Curran Associates, Inc., 22199–22213. LangChain. 2025. GitHub - langchain-ai/langchain: Build context-aware reasoning applications — github.com. Retrieved April 20, 2025 from https://github .com/langchain-ai/langchain. LangChain. 2025. GitHub - langchain-ai/langgraph: Build resilient language agents as graphs. — github.com. Retrieved April 20, 2025 from https://github.c om/langchain-ai/langgraph. B. Y. Lin, R. Le Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi. 2025. Zebralogic: on the scaling limits of LLMs for logical reasoning. In Forty-second International Conference on Machine Learning. L. Marchezan, E. Rodrigues, W. K. G. Assunção, M. Bernardino, F. P. Basso, and J. Carbonell. 2022. Software product line scoping: a systematic literature review. In Proceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A (SPLC ’22). ACM, Graz, Austria, 256. doi:10.1145/354693 2.3547012. K. Michailidis, D. Tsouros, and T. Guns. 2024. Constraint Modelling with LLMs Using In-Context Learning. In 30th International Conference on Principles and Practice of Constraint Programming (CP 2024) (Leibniz International Proceedings in Informatics (LIPIcs)). Vol. 307. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 20:1–20:27. doi:10.4230/LIPIcs.CP.2024.20. 2025. OpenAI models. Retrieved September 29, 2025 from https://platform.ope nai.com/docs/models. 2025. Llama 4 Scout - API, Providers, Stats — openrouter.ai. Retrieved September 29, 2025 from https://openrouter.ai/meta-llama/llama-4-scout. 2025. Llama 4 Maverick - API, Providers, Stats — openrouter.ai. Retrieved September 29, 2025 from https://openrouter.ai/meta-llama/llama-4-maverick. L. Pan, V. Ganesh, J. Abernethy, C. Esposo, and W. Lee. 2025. Can transformers reason logically? a study in SAT solving. In Forty-second International Conference on Machine Learning. M. Parmar, N. Patel, N. Varshney, M. Nakamura, M. Luo, S. Mashetty, A. Mitra, and C. Baral. 2024. Logicbench: towards systematic evaluation of logical reasoning ability of large language models. In ACL (1), 13679–13707. https://doi.o rg/10.18653/v1/2024.acl-long.739. K. Pohl, G. Böckle, and F. J. van der Linden. 2010. Software Product Line Engineering: Foundations, Principles and Techniques. (1st ed.). Springer Publishing Company, Incorporated. D. Romero-Organvidez, J. A. Galindo, C. Sundermann, J.-M. Horcas, and D. Benavides. 2024. Uvlhub: a feature model data repository using uvl and open science principles. Journal of Systems and Software, 216, 112150. doi:https://doi .org/10.1016/j.jss.2024.112150. C. Sundermann, V. F. Brancaccio, E. Kuiter, S. Krieter, T. Heß, and T. Thüm. 2024. Collecting feature models from the literature: a comprehensive dataset for benchmarking. In Proceedings of the 28th ACM International Systems and Software Product Line Conference (SPLC ’24). ACM, Dommeldange, Luxembourg, 54–65. doi:10.1145/3646548.3672590. J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter b., F. Xia, E. Chi, Q. V. Le, and D. Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, (Eds.) Vol. 35. Curran Associates, Inc., 24824–24837. XAI. 2025. Models and Pricing | xAI Docs — docs.x.ai. Retrieved September 29, 2025 from https://docs.x.ai/docs/models. J. Yan, C. Wang, J. Huang, and W. Zhang. 2024. Do large language models understand logic or just mimick context? CoRR, abs/2402.12091. https://doi.org /10.48550/arXiv.2402.12091.