Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering

Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering Quim Motger # Universitat Politècnica de Catalunya, Barcelona, Spain

Alessio Miaschi ItaliaNLP Lab, Istituto di Linguistica Computazionale “A. Zampolli” (CNR-ILC), Pisa, Italy

arXiv:2608.27621v1 [cs.SE] 27 Aug 2026

Xavier Franch Universitat Politècnica de Catalunya, Barcelona, Spain

Mohammad Amin Zadenoori University of Padua, Padua, Italy

Alessio Ferrari University College Dublin, Dublin, Ireland

Abstract Background. LLM outputs are highly sensitive to prompt formulation. Small changes in wording or phrasing can lead to substantial differences in output quality. This sensitivity is particularly important in software engineering, where prompts guide tasks such as requirements analysis, code generation, and artifact synthesis in general. Ineffective prompt formulations can produce unreliable artefacts, with consequences for both task outcomes and the broader development process. However, practitioners lack principled ways to assess prompt formulation before inference (i.e., LLM execution), making prompt selection dependent on resource-intensive LLM calls and trial-and-error refinement. Aims. We investigate whether measurable linguistic properties of prompts can predict LLM performance before inference, thereby supporting low-cost prompt selection and refinement. We validate this hypothesis in the context of binary requirements classification by targeting performance metrics (F1, F2, precision, recall). Method. We generate 9,000 linguistically controlled prompt variants derived from 100 initial prompts by varying 30 linguistic metrics. Variants are evaluated using five open-source LLMs on 625 requirements annotated with ground-truth labels. Regression predictors are trained using stratified 10-fold cross-validation with permutation-based significance testing. Feature importance analysis identifies cross-LLM and model-specific linguistic predictors. Results. Linguistic features significantly predict prompt performance across all targets (R2 ∈ [0.38, 0.42], q < 0.05). Syntactic and morphosyntactic features drive most of the predictive signal. Cross-LLM consistent predictors include compound dependency distribution, conjunction density, and raw-text indicators such as word and sentence length, reflecting sensitivity to domain-specific vocabulary and complex structures. Conclusions. These results draw practical observations for prompt engineering, including the overlap between linguistic patterns that reduce LLM performance and those that increase human comprehension difficulty, as well as the irrelevance of lexical variety as a prompt-quality dimension. More broadly, they show that linguistic profiling combined with standard regression models provides an effective, interpretable signal for prompt characterisation, applicable as a low-cost prior before engaging costly optimisation pipelines. 2012 ACM Subject Classification General and reference → Empirical studies; Human-centered computing → Natural language interfaces; Software and its engineering → Requirements analysis Keywords and phrases prompt engineering, requirements engineering, computational linguistics, requirements classification, regression modelling

2

Linguistic Properties of Prompts as Predictors of LLM Performance

1

Introduction

The increasing adoption of large language models (LLMs) in software engineering (SE) has demonstrated great potential to automate a wide range of tasks [18], from code generation [46] to requirements analysis [37]. However, LLM outputs are highly sensitive to prompt formulation: small changes in wording, structure or phrasing can lead to substantial differences in output quality [12]. This effect has been documented across task types and model families, from accuracy swings [40] to inconsistent model rankings [31] under prompt formatting changes. The consequences are particularly acute in SE. While reproducibility, interpretability of prompt artefacts, and developer trust are central concerns, even binary judgements have been shown to flip under minor paraphrasing [16]. This sensitivity makes prompt design a critical — yet poorly understood — determinant of task performance, with no established principles to guide SE practitioners in formulating effective prompts [36]. Prompt engineering has emerged as the community’s response to this challenge, aiming to formulate prompts that elicit reliable, high-quality outputs from LLMs. In SE and requirements engineering (RE), this work has so far been predominantly structural or templatebased, with researchers proposing design frameworks [10], role and context patterns [21], and task-specific templates for code generation [41, 4], test generation [33], and requirements classification [2]. Overall, systematic reviews confirm the centrality of structural design choices to LLM performance in these settings [19]. A complementary line of work applies automatic prompt engineering (APE), which frames prompt design as an optimisation problem and searches for high-performing prompts through iterative rewriting, meta-prompting, or reinforcement learning [52, 22, 48, 24, 43, 50]. Yet both directions have limitations: structural and template-based approaches typically prescribe a fixed design and do not account for how small linguistic variations within that design affect output, while APE requires repeated LLM calls to evaluate candidate prompts, making it computationally expensive and energyintensive [38]. What is missing is a lightweight way to estimate prompt effectiveness from linguistic properties of the prompt itself, before inference. Such a signal would complement both directions by constraining the search space, guiding prompt generation, and supporting the selection of promising prompts without running the LLM at every step. We therefore hypothesise that measurable linguistic properties of prompts are systematic predictors of LLM performance. If such relationships can be modelled, prompt designers can assess and improve prompts analytically before engaging costly optimisation pipelines, reducing the number of LLM interactions required during prompt development. To validate this hypothesis, a controlled empirical study is required in a setting where (i) prompt variability can be systematically manipulated, (ii) ground-truth performance can be reliably measured, and (iii) a sufficient volume of prompt variants can be generated and evaluated. Requirements engineering (RE) offers a particularly suitable instantiation: RE tasks such as requirements classification are well-defined, supported by established benchmarks, and have been shown to benefit from LLM-based automation [49]. Accordingly, the goal of this study is to validate the above hypothesis in the context of binary requirements classification, using a controlled experimental setting in which linguistic properties of prompts are systematically manipulated and their effect on performance — focusing on functional correctness as defined by ISO/IEC 25010 [20] — is measured and modelled. Our study makes the following contributions: A prompt variability framework that combines controlled paraphrasing — guided by targeted manipulation of linguistic metrics — with systematic regression-based analysis

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

3

of the impact of linguistic properties on LLM performance. A benchmark dataset of 9,000 linguistically diverse prompt variants for binary requirements classification, evaluated against five open-source LLMs and annotated with performance metrics. A ranked assessment of linguistic predictors of prompt performance, distinguishing features that generalise across LLMs from those that are model-specific, providing actionable guidance for prompt engineers targeting heterogeneous model deployments. The remainder of this paper is structured as follows. Section 2 reviews relevant background on prompt engineering and linguistic profiling. Section 3 describes the research method. Section 4 presents the results. Section 5 discusses findings and implications. Section 6 addresses threats to validity. Section 7 surveys related work. Section 8 concludes the paper.

2

Background

2.1

Prompt Engineering

Prompt engineering is the process of designing and refining prompts — i.e., textual inputs or queries given to LLMs — to maximise the performance of the model on specific tasks [28, 50]. More formally, prompt engineering can be cast as an optimisation problem [48, 50], where the objective is to identify the optimal prompt p∗ that maximises a task-specific evaluation metric over a ground-truth dataset D = {(xi , yi )}ni=1 , using an LLM M: p∗ = arg max f ({M(xi , p)}ni=1 , {yi }ni=1 ) p

(1)

D = {(xi , yi )}ni=1 is the ground-truth dataset, where each xi is an input instance and each yi is its corresponding ground-truth output; M is the LLM, which takes a prompt p and an input xi to produce a predicted output ŷi = M(xi , p). The prediction ŷi is expected to match, or approximate, the corresponding ground-truth output yi ; f is the task-specific evaluation function to be maximised. It compares the predictions induced by prompt p, namely {ŷi }ni=1 , with the ground-truth outputs {yi }ni=1 , and returns a scalar score indicating the effectiveness of the prompt execution. In the context of requirements classification, each xi represents a textual requirement, and each yi denotes its manually annotated class. For example, a requirement may be classified as functional, as in “When the user submits the form, the system shall validate the input”, or as non-functional, when it concerns quality attributes such as security, performance, or others, as in “The system shall respond within two seconds under normal load”. In this setting, the evaluation function f can be instantiated using a classification metric, such as accuracy or F1, to measure how well the predicted requirement classes match the manually annotated ones. Prompt engineering can be performed either manually or automatically, through APE approaches [52]. In manual prompt engineering, human experts craft and refine prompts based on their domain knowledge and on iterative evaluation against a subset of the groundtruth data. In APE, instead, the search for effective prompts is partially or fully automated, with candidate prompts being generated and evaluated according to f .

4

Linguistic Properties of Prompts as Predictors of LLM Performance

2.2

Linguistic Profiling

Linguistic profiling is the methodology of characterising a text through a structured set of measurable linguistic features extracted from automatic annotation. Originally developed for stylometric and authorship attribution tasks [44], the approach has since been extended to a broad range of NLP applications, including readability assessment [11], native language identification [27], and, more recently, the evaluation and interpretation of neural language models [29, 30]. The central idea is that texts can be represented as vectors of quantitative descriptors computed over multiple levels of linguistic annotation — typically raw text (e.g., document length, average word and sentence length), lexical (e.g., lexical density, type/token ratio), morphosyntactic (e.g., distribution of parts of speech and inflectional categories), and syntactic (e.g., dependency relation distributions, parse tree depth). These features capture stable, interpretable properties of linguistic structure rather than task-specific surface cues, making them well-suited to comparative analyses across texts, authors, genres, or models. More formally, given a text t, linguistic profiling produces a feature vector ϕ(t) = ⟨f1 (t), f2 (t), . . . , fn (t)⟩ ∈ Rn

(2)

where each fi is a function that maps the text — typically after automatic linguistic annotation (tokenisation, POS tagging, dependency parsing) — to a real-valued descriptor. The annotation step is commonly based on standardised formalisms such as Universal Dependencies [9], which provide cross-linguistically consistent representations of morphosyntactic and syntactic structure. The resulting vector ϕ(t) can then be used as input to downstream analyses, including classification, regression, correlation studies, or distributional comparison between groups of texts. Two properties make linguistic profiling well-suited to the present study. First, extracted features are interpretable: each descriptor maps to a well-defined linguistic property, linking predictive analysis to specific textual characteristics. Second, features are computed deterministically from raw text — no learned parameters, no LLM inference — making them inexpensive to compute and reproducible. Hence, we adopt linguistic profiling to characterise prompts as structured linguistic artefacts. By representing each prompt variant as a feature vector ϕ(p) over a comprehensive set of linguistic descriptors, we enable the systematic regression-based analysis of how linguistic structure relates to LLM performance.

3

Research Method

3.1

Design

Following the ABC framework for SE research [42], this work is classified as a knowledgeseeking empirical study combining two research strategies: a sample study, in which a large population of prompt variants is analysed to uncover statistical relationships between linguistic features and performance metrics, and a laboratory experiment, in which prompt variants are evaluated in a controlled environment using multiple LLMs. Two research questions guide the study: RQ1: To what extent can the performance of LLM-based requirements binary classification be predicted from linguistic features of prompts? RQ2: Which linguistic features show the highest predictive relevance in LLM-based requirements binary classification?

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

Stage 1 — Experimental space construction 100 definition pairs (Zadenoori et al. 2025)

Controlled Prompt Variation Qwen3:32B

9,000 prompt variants

Linguistic Feature Extraction Profiling-UD

Variability Validation Boxplot analysis

Stage 2 — Performance evaluation and prediction 655 annotated requirements

LLM Requirements Classification

Correctness metrics

(Dalpiaz et al. 2019)

5 open-source LLMs

(F1, F2, P, R)

RQ1 — Predictive feasibility

Regression Modelling 5 models · hyper-tuning 10-fold CV · permutation test

Feature Analysis Permutation importance

RQ2 — Linguistic predictor importance

Figure 1 Overview of the research methodology.

RQ1 addresses predictive feasibility: whether linguistic features of prompts contain sufficient information to predict performance outcomes. It is answered through stratified 10-fold cross-validation across four performance targets (Precision, Recall, F1, F2) using five regression models, producing predictive performance scores that indicate whether linguistic features are a viable predictor of prompt performance. RQ2 addresses linguistic predictor relevance, and is answered through permutation-based feature importance analysis on the best-performing regression setup from RQ1, compared across LLMs and performance targets. This analysis identifies which linguistic properties contribute most strongly to prompt performance prediction, both at the level of individual features and feature groups. It also examines whether highly relevant features exhibit stable patterns across LLMs or whether their relevance varies depending on the target model. Both RQs are framed as joint predictive analyses rather than per-feature statistical tests. Linguistic profiling produces a high-dimensional, correlated feature space in which classical feature-wise testing is hard to interpret and exposed to multiple-comparison issues. Regression with cross-validation and permutation-based significance testing instead evaluates whether the feature set carries reliable joint predictive signal, while subsequent feature-importance analysis recovers the interpretable patterns within that signal. The study therefore combines predictive utility with explanatory insight. To support this analysis, the methodology is structured into two stages, illustrated in Figure 1. Stage 1 constructs and validates the experimental space: it generates linguistically controlled prompt variants through targeted paraphrasing guided by a set of linguistic metrics, and verifies that the manipulation effectively increases linguistic variability across the prompt population. Stage 2 evaluates each prompt variant against multiple open-source LLMs on the binary requirements classification task, collecting performance metrics, and uses the resulting prompt–performance pairs to train regression models and analyse feature importance to answer the research questions.

3.2 3.2.1

Stage 1: Experimental Space Construction Task and Dataset

The target task is binary requirements classification: given a software requirement expressed in natural language, determine whether it is Functional (F) or Non-functional (NF). This is one of the most studied and well-understood tasks in the field of Natural Language Processing (NLP) for RE [51], supported by established benchmarks [32], making it particularly suitable

5

6

Linguistic Properties of Prompts as Predictors of LLM Performance

for controlled experimentation where precise performance measurement is required. We evaluate the task on the PROMISE-NFR dataset [7], the most popular dataset on requirements classification [32]. The dataset provides 625 annotated software requirements derived from the PROMISE repository [7]. For the prompt definitions, we reuse the dataset introduced by Zadenoori et al. [50], which provides 100 few-shot prompt variants based on alternative natural language definition pairs for the classes F and NF. These definitions vary in wording and phrasing while referring to the same underlying concepts, making them a natural starting population for linguistic manipulation. We focus on few-shot prompting as it has been shown to reduce LLMs’ sensitivity to prompt formulation [53]. The prompt structure itself — system context, examples, and classification instruction — is kept constant across all variants (Listing 1). Linguistic manipulation is applied exclusively to the requirements class definitions. This design choice reflects a deliberate trade-off between breadth and depth. Rather than varying task, dataset, or prompting strategy (e.g., zero-shot, few-shot, Chain-of-Thought) simultaneously, we fix these dimensions and conduct an in-depth empirical analysis of linguistic variability within a single, well-controlled setting. This is consistent with established practice in empirical SE, where studies focusing on a single task and benchmark are a recognised and productive approach to validating novel hypotheses before broader generalisation is attempted [13, 8, 39, 35]. If measurable linguistic properties of prompts prove predictive under these controlled conditions — across a large number of prompt variants, multiple linguistic metrics, and five LLMs — the hypothesis gains the empirical support needed to motivate replication across other RE tasks, datasets, and prompt strategies in future work. Listing 1 Prompts used for requirements classification (adapted from Zadenoori et al. [50]) SYSTEM PROMPT ------------As an expert system for classifying software requirements , your job is to carefully review each requirement and place it into one of these two classes : Functional , Non - functional USER PROMPT -- -- - - - - - - Definitions : Functional : "{ functional_def }" Non - functional : "{ no n_f un ct ional_def }" Examples : Functional : - The system shall allow modification of the display . - The system shall offer a display of all the Events in the exercise . Non - functional : - 90% of untrained realtors shall be able to install the product without instructions . - The product is expected to run on Windows CE and Palm operating systems . Requirement : "{ text }" Using the Definitions above , classify the requirement and provide the final label in the format : Label : [ Your Class Label Here ]

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

Table 1 Linguistic feature groups extracted using the Profiling-UD tool [3]. Group

Description

#

Example feature

Raw Text

Surface-level statistics on document, sentence, and word length.

4

tokens_per_sent: avg. number of tokens per sentence

Lexical Variety

Type/Token Ratio computed over standardised windows of 100 and 200 tokens, for both lemma and word form.

6

ttr_lemma_chunks_100: lexical variety over first 100 tokens

Morphosyntactic Information

Percentage distribution of the 17 Universal POS categories and overall lexical density.

33

upos_dist_VERB: proportion of main verbs in the text

Inflectional Morphology

Distribution of verbal inflection features: tense, mood, form, gender, number, and person.

14

verbs_tense_dist_Pres: proportion of present-tense verbs

Verbal Predicate Structure

Frequency and arity of verbal heads, including distribution of verbs by number of instantiated dependency links.

9

avg_verb_edges: avg. number of dependents per verbal head

Global & Local Parse Tree Structures

Syntactic tree depth, clause length, dependency link length, and prepositional chain depth.

9

avg_max_depth: avg. maximum depth of dependency trees

Order of Elements

Distribution of pre- and post-verbal positions of subjects and objects.

4

subj_pre: proportion of subjects preceding the verb

Syntactic tions

Rela-

Percentage distribution of the 37 Universal Dependencies relation types.

37

dep_dist_nsubj: proportion of nominal subject dependencies

Use of Subordination

Distribution, position, and embedding depth of subordinate clauses.

8

sub_proposition_dist: proportion of subordinate clauses

3.2.2

Linguistic Feature Extraction

To characterise linguistic properties, we adopt the linguistic profiling framework proposed by Brunato et al. [3]. In particular, we extract features using the Profiling-UD tool [3], a tool that allows the extraction of 124 properties representative of the linguistic structure underlying a sentence and derived from raw, morphosyntactic and syntactic levels of annotation based on the UD formalism [9]. These features have been shown to play a highly predictive role when leveraged by traditional learning models on various classification problems and can also be effectively used to profile the knowledge encoded in the internal representations of Language Models and to evaluate their abilities in following specific linguistic constraints [29, 30]. Table 1 summarises the nine feature groups extracted by Profiling-UD, together with the number of features per group and a representative example. For controlled prompt generation, we select 30 linguistic metrics spanning four groups: raw text, parse tree structure, verbal predicate structure, and subordination. These groups were chosen because they operate at distinct levels of linguistic description — surface, syntactic, and clausal — and have been shown to contribute independently and non-redundantly to text characterisation tasks such as readability assessment and genre classification [3, 29]. Manipulating metrics across these four dimensions therefore ensures that the generated

7

8

Linguistic Properties of Prompts as Predictors of LLM Performance

prompt variants differ along genuinely distinct linguistic axes, rather than along correlated surface properties that would reduce the effective diversity of the experimental space. The complete set of Profiling-UD metrics — covering all nine groups in Table 1 — is computed for all prompts and used in the full predictive analysis of Stage 2.

3.2.3

Controlled Prompt Generation

Prompt variants are generated using Qwen3:32B with the paraphrasing template shown in Listing 2, which instructs the model to modify a specific linguistic property while preserving semantic meaning. The choice of Qwen3:32B was motivated by two factors. First, at the time of the study, Qwen3 ranked among the top open-source instruction-tuned models on established benchmarks [25], making it a strong candidate for following precise linguistic manipulation instructions. Second, the 32B parameter size was selected based on preliminary experimentation: smaller models failed to produce consistent variability across surface-level metrics such as token count and sentence count, while the 32B variant reliably responded to targeted paraphrasing instructions within the constraints of our experimental infrastructure1 . For each of the 30 metrics, three variants are generated per definition pair corresponding to the targets minimise, equal, and maximise, yielding 100 × 30 × 3 = 9,000 definition pairs. To validate the experimental space, we compare the linguistic distributions of the 100 original definition pairs against the 9,000 generated variants. For each metric, we compute descriptive statistics — mean, standard deviation, interquartile range — and visualise distributional differences using boxplots. Increased dispersion in the extended dataset indicates that the targeted linguistic properties were successfully manipulated. Listing 2 Prompt used for controlled linguistic paraphrasing Paraphrase the text below . Return only the paraphrased text - no explanations , tags , or comments . Aim for a paraphrase that reflects the following linguistic feature : Metric : { metric_name } Description : { m et ri c_ de sc ri ption } Target : { target_range } Text : "{ functional_def } || { non_functional_def }"

3.3 3.3.1

Stage 2: Performance Evaluation and Prediction LLM Evaluation

The 9,000 prompt variants are evaluated using five open-source instruction-tuned LLMs: Qwen2-7B-Instruct, Falcon3-7B-Instruct, Granite-3.2-8B-Instruct, Ministral-8B-Instruct2410, and Meta-Llama-3-8B-Instruct. These models were empirically validated by Zadenoori et al. [50], demonstrating their suitability for LLM-based binary requirements classification and establishing them as a representative benchmark suite for this task. To measure performance, we focus on F1, F2, precision (P) and recall (R) metrics. These are averaged over the 625 requirements within the dataset [7], tested over each prompt variant.

1

All experiments were conducted on a machine equipped with an AMD Ryzen 9 7950X 16-core processor, 64 GB of RAM, and an NVIDIA GeForce RTX 4090 GPU (24 GB VRAM)

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

3.3.2

Regression Modelling and Feature Analysis

To answer the research questions, we frame prompt performance prediction as a regression problem: each prompt is represented as a feature vector x ∈ R124 of Profiling-UD linguistic metrics, and the target y is one of four performance indicators y ∈ {F 1, F 2, P, R} aggregated over the 625 requirements per prompt variant. We evaluate five regression models of increasing complexity: Decision Tree (DT), Linear Support Vector Machine (LSVM), Random Forest (RF), Gradient Boosting (GB), and Multilayer Perceptron (MLP). This selection covers a spectrum from interpretable linear and rule-based models to non-linear ensembles, following established practice in empirical SE studies that compare multiple model families to avoid bias towards any single algorithm [17]. The MLP is wrapped with target standardisation for numerical stability in regression. For each model, hyperparameters are tuned via Randomised Search (20 iterations, optimising R2 ) prior to evaluation. Model performance is assessed using 10-fold cross-validation stratified by experimental condition (metric_name × target_range), ensuring that prompt variants generated from the same metric and target are distributed across folds proportionally. We report three complementary evaluation metrics based on standard regression-based empirical SE studies [17, 34]: Coefficient of determination (R2 ): proportion of variance in the target variable explained by the model; ranges from −∞ to 1, where 1 indicates a perfect fit and values below 0 indicate the model performs worse than a constant mean predictor. Mean Absolute Error (MAE): average absolute difference between predicted and actual values, expressed in the target’s original unit. Root Mean Squared Error (RMSE): square root of the average squared prediction error; penalises large errors more strongly than MAE, making it sensitive to outliers. To assess statistical significance, we apply a permutation test on R2 with 500 permutations and Benjamini-Hochberg FDR correction across all target-model combinations. To address RQ1, we evaluate the contribution of each linguistic feature group independently by running the best-performing model using one feature family at a time (as defined in Table 1), comparing each subset’s R2 against the full-feature baseline. This analysis identifies which levels of linguistic description carry the most predictive signal for prompt performance, and whether any single group alone is sufficient to explain a meaningful proportion of variance. To address RQ2, permutation importance is computed on the best-performing model per target, providing a ranking of the linguistic features that most strongly contribute to each performance indicator. Rankings are computed separately per LLM and then compared across models and targets to identify features with high predictive relevance and to assess the stability of their importance. This analysis allows us to distinguish broadly relevant linguistic predictors from features whose importance varies across models or performance metrics.

4

Results

4.1

Linguistic Variability

To verify that controlled paraphrasing effectively expanded the experimental space, we compared the baseline prompt definitions (B) against the generated variants under the three

9

10

Linguistic Properties of Prompts as Predictors of LLM Performance

Figure 2 Linguistic variability analysis.

control targets minimise (Min), equal (Eq), and maximise (Max) across the 30 targeted metrics identified in Section 3.2.2. Results are reported in Figure 2. The distributions show systematic target-dependent shifts for a substantial subset of metrics. Quantitatively, 46.2% of targeted metrics moved in the expected direction under minimise and 61.5% under maximise. Considering stricter bidirectional controllability (i.e., correct movement in both minimise and maximise), 30.8% of metrics satisfied this criterion. In addition, the equal condition was closer to baseline than both extremes for 61.5% of metrics, indicating partial preservation of intermediate linguistic profiles. Partial controllability is explained by structural constraints inherent to the input and the metrics themselves. Some metrics are bounded by the nature of the seed definitions. For example, since each definition pair consists of one sentence per class, metrics such as n_sentences have a structural minimum of 1, leaving no meaningful room for minimise to operate. Other metrics — such as avg_prepositional_chain_len, prep_dist_2, and verb_edges_dist_0 — already present near-zero baseline values, making minimise ineffective not because the paraphrasing fails, but because the metric has no further room to decrease. In both cases, the absence of directional movement reflects a ceiling or floor imposed by the data rather than a failure of the manipulation procedure. Additionally, the generated variants increased dispersion relative to baseline: the median variability gain was approximately 2.12× when measured as interquartile range variability (i.e., IQRcontrolled /IQRbaseline ), and 88.9% of the metrics with non-zero baseline IQR showed an increase (IQRcontrolled /IQRbaseline > 1). This confirms that controlled paraphrasing

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

11

generally broadened linguistic coverage, providing a diverse experimental space.

4.2

Predictive Feasibility (RQ1)

Table 2 reports regression performance across the four performance targets. All model–target combinations are statistically significant (q < 0.05, permutation test with Benjamini-Hochberg correction). Random Forest (RF) consistently outperforms all other models, achieving an average of R2 = 0.405 across all target metrics. Ensemble methods (RF and GB) substantially outperform linear and shallow models (LSVM, DT), indicating that the relationship between linguistic features and prompt performance is non-linear and benefits from the interaction of multiple features. Table 2 Regression model performance across quality targets. Best values per target are bold. Avg_F1 Model

R2 MAE RMSE

Avg_F2 R2 MAE RMSE

Avg_Precision

Avg_Recall

R2 MAE RMSE

R2 MAE RMSE

DT 0.145 0.059 0.069 0.141 0.060 0.070 0.124 0.015 0.019 0.138 0.053 0.062 LSVM 0.164 0.059 0.069 0.158 0.060 0.069 0.144 0.016 0.019 0.155 0.053 0.061 MLP 0.290 0.051 0.063 0.287 0.052 0.064 0.269 0.014 0.017 0.284 0.046 0.056 GB 0.352 0.050 0.060 0.350 0.051 0.061 0.316 0.014 0.017 0.347 0.044 0.054 RF 0.418 0.045 0.057 0.413 0.046 0.058 0.377 0.013 0.016 0.410 0.041 0.051

Table 3 reports, for Avg_F1 as a representative performance target, the contribution of each linguistic feature group in isolation compared against the full-feature baseline (R2 = 0.418). Results reveal a two-tier structure. Syntactic Relations (R2 = 0.375) and Morphosyntactic Information (R2 = 0.331) each retain substantial predictive power alone, accounting for most of the signal recoverable from any single group. Parse Tree Structure (R2 = 0.210) contributes moderately. All remaining groups produce near-zero R2 . These results suggest that syntactic and morphosyntactic features drive predictability, while surface-level and clausal features might contribute only when combined with the richer feature space. Answer to RQ1 ¬ Linguistic features significantly predict prompt performance across all targets (q < 0.05,

R2 ∈ [0.38, 0.42] for RF), providing empirical support for the main hypothesis. ¬ Predictive power requires the full feature space. Syntactic Relations and Morphosyn-

tactic Information contribute most individually, suggesting that sentence structure and part-of-speech diversity are the primary linguistic drivers of prompt performance. ¬ An R2 ≈ 0.41 is particularly relevant given that many other factors influence LLM

output beyond linguistic structure. Linguistic profiling combined with standard regression models provides an interpretable, LLM-free signal for prompt characterisation, complementary to empirical LLM evaluation.

4.3

Linguistic Predictor Importance (RQ2)

Figure 3 reports group-level feature importance and direction profiles across the four performance targets. Raw Text features achieve the highest per-feature importance despite comprising only four features (mean importance ≈ 0.062), while Syntactic Dependencies and POS features contribute most in absolute terms. Vocabulary features (TTR variants) are

12

Linguistic Properties of Prompts as Predictors of LLM Performance

Table 3 Feature family contribution to R2 for Avg_F1 (Random Forest). ∆R2 is the difference from the full-feature baseline. All subsets are statistically significant (q < 0.05). # feat.

R2

∆R2

MAE

RMSE

All features

124

0.418

—

0.045

0.057

Syntactic Relations Morphosyntactic Information Global & Local Parse Tree Struct. Inflectional Morphology Order of Elements Raw Text Properties Lexical Variety Use of Subordination Verbal Predicate Structure

37 33 9 14 4 4 6 8 9

0.375 0.331 0.210 0.024 0.004 0.003 0.001 0.000 −0.005

−0.044 −0.087 −0.208 −0.394 −0.415 −0.415 −0.417 −0.418 −0.423

0.047 0.049 0.055 0.063 0.065 0.061 0.065 0.065 0.064

0.059 0.061 0.067 0.074 0.075 0.075 0.075 0.075 0.075

Feature subset

irrelevant across all targets (mean importance ≈ 0.000, mean rank > 95). The direction profile confirms that Raw Text is fully and consistently negative (weighted score = −1.0): longer definitions with denser words and more sentences systematically reduce performance. Tree Structure features are similarly negative (−0.91), while POS shows mixed directionality. Figure 4 shows the top-20 features stable across all four performance targets. These come exclusively from Syntactic Dependencies, POS, Tree Structure and Raw Text features. The top-ranked feature is compound dependency distribution (dep_dist_compound, mean rank 1.0, importance 0.21–0.33) — the proportion of dependencies forming noun compounds. Character-per-token ratio (char_per_tok, rank 2.0) and density of conjunction (dep_dist_conj, upos_dist_CCONJ, ranks 3.2–5.8) follow. Qualitative inspection reveals that these features trace domain-specific technical language: the most frequent compound pairs include quality → requirements, performance → efficiency, and system → operations, while the dominant long tokens are ISO/IEC 25010 quality attribute names (maintainability, compatibility, portability). The dominant coordinating conjunction is and, coordinating nominal chains of up to 23 items. These prompts are technically dense, noun-compound-heavy, and enumerative, which reflects the nature of RE documentation. Figures 5 and 6 report cross-LLM consistency. Pairwise Spearman ρ between LLMspecific feature rankings ranges from 0.86 to 0.95, and rank consistency across the four performance targets reaches ρ ≥ 0.97, confirming that the identified predictors are both metric-agnostic and largely model-agnostic. The most universally stable individual predictors are char_per_tok, dep_dist_conj, and avg_links_len, which appear in the top-10 for all five LLMs. Within this overall consensus, two patterns of model-specific divergence are worth noting. First, token count (n_tokens) ranks #1 for Meta-Llama-3-8B-Instruct but falls to ranks 12–51 for the remaining models, suggesting that LLaMA-3 is disproportionately sensitive to absolute prompt length — possibly reflecting differences in instruction-tuning data composition or in how few-shot context is handled during training. Second, pronoun and determiner distributions show rank standard deviations above 20 across LLMs, indicating that the way function-word categories are weighted varies substantially from model to model, even when the broader structural signal is stable. These cases indicate that, while the dominant linguistic predictors are model-agnostic, fine-grained predictor importance still carries useful information for model-specific prompt tuning.

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

(a) Importance and rank analysis.

(b) Direction analysis.

Figure 3 Linguistic feature group analysis.

Answer to RQ2 ¬ Feature importance rankings are highly consistent across all five LLMs (pairwise

Spearman ρ ∈ [0.86, 0.95]) and across all four performance targets (ρ ≥ 0.97). ¬ The most consistently harmful features are compound dependency distribution

(dep_dist_compound, rank #1), character-per-token ratio (char_per_tok, rank #2), and conjunction density (dep_dist_conj, upos_dist_CCONJ). ¬ Lexical diversity features (TTR variants) are universally irrelevant (mean rank > 95 for

all models and targets), and verbal inflection and subordination features are similarly uninformative. Vocabulary variability does not predict prompt performance. ¬ While cross-LLM consensus is high, some features show model-specific importance.

This confirms that LLM selection remains a relevant design variable.

5

Discussion

The results of RQ1 and RQ2 jointly show that prompt performance is not only affected by the semantic content of a prompt, but also by how that content is linguistically packaged. RQ1 shows that linguistic features provide a significant predictive signal for performance, while

13

14

Linguistic Properties of Prompts as Predictors of LLM Performance

Figure 4 Linguistic feature analysis on top-20 stable features.

Figure 5 Group-level, cross-LLM mean rank analysis.

RQ2 shows that this signal is concentrated in a specific structural profile — morphosyntactic patterns, long dependency arcs and dense coordination — rather than being uniformly distributed across linguistic dimensions. Notably, these are some of the same structural properties associated with increased processing cost in human language comprehension, from dependency-length effects [14] to the nominalisation patterns characteristic of expository and technical writing [15]. This convergence reinforces the interpretability dimension that motivated our approach: the features driving prompt performance are not opaque statistical artefacts of a particular model, but linguistic properties with independent theoretical grounding. Prompts should therefore be treated as SE artefacts that humans can analyse, reason about, and refine using well-established linguistic categories, a perspective that complements automatic optimisation. Moreover, these observations connect to a recent line of work on LLM linguistic fragility: Cao et al. [5] showed that surface-level stylistic variation (formality, readability, politeness) degrades RAG performance through

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

Figure 6 Ranked cross-LLM analysis of top linguistic features.

retrieval and generation errors. Our results extend this picture along a complementary axis — the fragility is not only stylistic but structural, and it can be characterised through measurable syntactic and morphosyntactic properties before any LLM is invoked. This, in turn, motivates a different role for linguistic profiling in prompt engineering: not as a post-hoc diagnostic, but as a pre-inference scoring signal. Raw Text features have high per-feature importance (Figure 3) yet weak standalone predictive power (Table 3). Length, sentence count, and word length therefore operate as proxies for deeper structural density rather than as explanatory factors in themselves, drawing their predictive weight from associated phenomena — compound noun phrases, long dependency arcs, coordinated nominal lists, and dense technical terminology. This pattern reflects the register of technical documentation: in our study, the patterns are tied to requirements and quality-attribute terminology, but the implication is broader. Prompts including content copied from standards, taxonomies, or expert documentation may be conceptually accurate while being linguistically difficult for LLMs to process, creating a tension between domain completeness and prompt usability. Making definitions more precise, exhaustive, and terminology-rich can simultaneously introduce the structural patterns that reduce prompt effectiveness. Expert knowledge should therefore be adapted rather than used verbatim: the prompt should preserve decision-relevant cues while reducing unnecessary syntactic and nominal density — but not unnecessary lexical variability. RQ2 shows that lexical-variety features (TTR variants) are universally irrelevant predictors, indicating that vocabulary diversity, often promoted as a writing virtue, carries no measurable benefit for prompt effectiveness. These findings yield actionable implications for prompt engineering in SE: 1. Prefer simple over exhaustive definitions. Prompt definitions should support the

15

16

Linguistic Properties of Prompts as Predictors of LLM Performance

target decision rather than reproduce the full conceptual richness of the domain. 2. Express one idea at a time. Avoid compressing several criteria, exceptions, or distinctions into a single long sentence. 3. Avoid long additive enumerations. Long lists joined by additive connectors such as and should be shortened, grouped, or split into separate statements. 4. Reduce compound-heavy noun phrases. Unpack compressed technical expressions when doing so improves readability, e.g., “requirements about system quality” instead of “system quality requirements”. 5. Use domain-specific terminology selectively. Keep technical terms that are necessary for the task, but avoid terminology that only increases domain density. 6. Do not optimise prompt length in isolation. Length is a warning signal, but the main risks are the structures that often accompany it: compounds, coordination chains, long dependency arcs, and dense nominal constructions. 7. Do not optimise for lexical variety. Replacing words merely to increase vocabulary richness is unlikely to help; consistency is preferable to lexical embellishment. 8. Penalise dense syntactic structures. Candidate prompts with high compound density, long dependency arcs, high coordination density, or dense nominal constructions can be deprioritised before LLM evaluation.

6

Threats to validity

We organise threats to validity following the taxonomy by Wohlin et al. [47]. Internal validity. Linguistic variability is generated exclusively using Qwen3:32B, without comparing alternative paraphrasing models. Prompt generation is instrumental to the study, not its object of investigation. We empirically validate that the generated variants produce significant expansion across multiple metrics, which is enough for our purposes. Future work may assess different generation models and how they affect linguistic distributions. Construct validity. We operationalise linguistic predictive importance through permutation importance. This measure is sensitive to feature correlation: when linguistic features are correlated, importance may be distributed arbitrarily between them, potentially underestimating the contribution of individual features. We partially mitigate this by reporting group-level contributions alongside individual rankings. Additionally, we operationalise prompt performance through four performance indicators derived from a binary classification task, which might not generalise to measure prompt quality in other contexts. A further construct threat concerns potential data leakage in the PROMISE-NFR dataset [6], since some requirements may have appeared in the pre-training data of the evaluated LLMs. To mitigate this, we conducted an additional leakage check by prompting each LLM with the first half of each requirement and comparing its generated continuation with the original second half using Jaccard similarity. Mean Jaccard scores remain low across models, ranging within the interval [0.105, 0.165], suggesting no strong evidence of systematic reconstruction of hidden requirement segments. Full results are available in the replication package. External validity. The study is scoped to a single task (binary requirements classification), dataset (PROMISE-NFR [6]), and prompt component (class definitions). This is a conscious design choice: rather than varying these dimensions simultaneously — which would reduce experimental control — we conduct an in-depth analysis across 9,000 prompt variants, 124 linguistic metrics, five LLMs and five regression models. The statistical significance and consistent findings across these dimensions validate our hypothesis in a well-controlled context,

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

establishing a replicable pipeline. Generalisation to other tasks, datasets and prompting strategies remains an open question that this work is explicitly designed to motivate. Conclusion validity. The five LLMs used in this study were selected based on their empirical validation for requirements classification [50]. While newer models may outperform absolute metric values, we observe consistent patterns across architecturally diverse models. This suggests that the identified linguistic predictors reflect properties of the task and prompt structure rather than idiosyncrasies of specific models.

7

Related work

Prompt engineering is now a practical requirement for applying LLMs to SE tasks. Recent studies examine prompting for code generation [21, 4], unit test generation [33], repository mining [10], and software startup support [1]. In RE, prompt engineering has been studied across tasks such as elicitation, validation, traceability, and classification [19]. Binkhonain and Alfayaz show that prompt-based LLMs, especially few-shot and persona-based variants, can perform competitively with fine-tuned models while reducing the need for task-specific annotated data [2]. These studies establish that prompt design affects SE outcomes and that prompt engineering is a viable alternative to retraining in data-scarce settings. However, existing SE research mostly treats prompts as task templates or strategy configurations. The usual unit of analysis is the prompting technique: zero-shot, few-shot, persona, chain-of-thought, task-specific prompting, or their combinations [21, 2]. Other studies focus on end-to-end workflows, such as prompt refinement for repository mining [10], prompt patterns for startup support [1], or benchmark comparisons for test and code generation [33, 4]. This literature shows that prompt structure influences task performance, but it does not explain which linguistic properties of prompts matter. In particular, SE studies rarely analyse prompts in terms of syntactic structure, morphosyntactic composition, dependency patterns, lexical density, or surface complexity. Work in NLP has started to address this linguistic dimension. Leidinger et al. analyse semantically equivalent prompts varying in mood, tense, aspect, modality, and synonym choice, showing that prompt performance varies across models and datasets and is not reliably explained by prompt length, perplexity, word frequency, or ambiguity [23]. Wahle et al. study controlled paraphrase types across morphology, syntax, lexicon, lexico-syntax, discourse, and other categories, showing that linguistic reformulations can change model behaviour across tasks and models [45]. Long et al. propose a broader property-centric view of prompt quality, including linguistic and human-oriented properties beyond output accuracy [26]. Cao et al. show that linguistic style variations such as formality, readability, politeness, and grammatical correctness can degrade RAG performance through retrieval and generation error propagation [5]. These studies show that linguistic form affects LLM behaviour, but they mostly analyse coarse linguistic categories or style dimensions. They also diagnose sensitivity after inference rather than using linguistic features to predict prompt effectiveness or guide APE search beforehand. This paper addresses these gaps by treating prompts as measurable linguistic artefacts. We link prompt linguistic features to LLM performance for SE tasks, identifying predictors useful for prompt selection, rewriting, and APE pre-filtering before costly inference.

17

18

Linguistic Properties of Prompts as Predictors of LLM Performance

8

Conclusions

Our findings suggest an alternative to computationally expensive, opaque APE approaches: linguistic profiling can serve as a fast, human-interpretable pre-filter that scores candidate prompts before any LLM is invoked. Stable linguistic predictors can be computed from raw text in milliseconds using off-the-shelf tools. An APE pipeline that incorporates this prior could deprioritise structurally dense candidates early, reducing the number of LLM evaluations required to reach a high-performing prompt. This is particularly relevant in resource-constrained or energy-sensitive deployment contexts. Several directions follow from this work. First, the experimental scope should be expanded along three axes: replication across other RE and SE tasks; extension of linguistic manipulation to the full prompt structure, covering system context, task instruction, and output specification; and evaluation on closed-source and larger-scale LLMs. Second, our framework manipulates prompts along independent linguistic dimensions; future work could explore joint manipulation of multiple features and study interaction effects, which our regression results suggest are non-trivial. Third, the catalogue of linguistic predictors could be integrated into APE toolchains as a scoring function, enabling hybrid approaches combining linguistic screening with semantic search over candidate prompts. Finally, the findings motivate the development of linguistically-aware prompt rewriting tools for SE practitioners.

Data Availability All data, prompts, model outputs, scripts, results and instructions needed to reproduce the study are available in an anonymised replication package: https://anonymous.4open. science/r/ESEM-D64C/. The package includes a README describing the processing pipeline and how to reproduce the analyses.

References 1

Thea Lovise Ahlgren, Helene Fønstelien Sunde, Kai-Kristian Kemell, and Anh Nguyen-Duc. Assisting early-stage software startups with LLMs: Effective prompt engineering and system instruction design. Information and Software Technology, page 107832, 2025.

2

Manal Binkhonain and Reem Alfayez. Are prompts all you need? Evaluating prompt-based Large Language Models (LLM) s for software requirements classification. Requirements Engineering, 30(4):423–443, 2025.

3

Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. Profiling-UD: A Tool for Linguistic Profiling of Texts. In Twelfth Language Resources and Evaluation Conference, pages 7145–7151, 2020.

4

Marc Bruni, Fabio Gabrielli, Mohammad Ghafari, and Martin Kropp. Benchmarking prompt engineering techniques for secure code generation with gpt models. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), pages 93–103, 2025.

5

Tianyu Cao, Neel Bhandari, Akhila Yerukola, Akari Asai, and Maarten Sap. Out of style: RAG’s fragility to linguistic variation. In 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.

6

Jane Cleland-Huang, Sepideh Mazrouee, Huang Liguo, and Dan Port. nfr, 2007. doi: 10.5281/zenodo.268542.

7

Jane Cleland-Huang, Raffaella Settimi, Xuchang Zou, and Peter Solc. Automated classification of non-functional requirements. Requirements Engineering, 12(2):103–120, 2007.

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

8

9 10

11

12

13

14 15 16

17 18

19

20 21

22

23 24

25

Xing Cui, Jingzheng Wu, Xiang Ling, and Tianyue Luo. We Know What You’re Looking For: Recommendation for Large-Scale Open Source Software . In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2025. Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. Universal dependencies. Computational linguistics, 47(2):255–308, 2021. Vincenzo De Martino, Joel Castano, Fabio Palomba, Xavier Franch, and Silverio MartínezFernández. A framework for using llms for repository mining studies in empirical software engineering. In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE), pages 6–11, 2025. Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. Read–it: Assessing readability of italian texts with a view to text simplification. In Proceedings of the second workshop on speech and language processing for assistive technologies, pages 73–83, 2011. Federico Errica, Davide Sanvito, Giuseppe Siracusano, and Roberto Bifulco. What did I do wrong? quantifying LLMs’ sensitivity and consistency to prompt engineering. In 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025. Katia R. Felizardo, Márcia S. Lima, Anderson Deizepe, Tayana U. Conte, and Igor Steinmacher. ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity. In 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, page 25–36, 2024. Richard Futrell, Kyle Mahowald, and Ed. Gibson. Large-scale evidence of dependency length minimization in 37 languages. National Academy of Sciences, 112(33):10336–10341, 2015. Michael Alexander Kirkwood Halliday and JR Martin. Writing Science: Literacy And Discursive Power. CRC Press, 1994. Shuo Han, Tao Tan, Yuantian Miao, Xiao Chen, and Nan Sun. Prompting instability: An empirical study of llm robustness in code vulnerability detection. In AI 2025: Advances in Artificial Intelligence, pages 233–245, 2026. Peng He, Bing Li, Xiao Liu, Jun Chen, and Yutao Ma. An empirical study on software defect prediction with a simplified metric set. Inf. and Soft. Tech., 59:170–190, 2015. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. ACM Trans. Softw. Eng. Methodol., 33(8), 2024. Kaicheng Huang, Fanyu Wang, Yutan Huang, and Chetan Arora. Prompt engineering for requirements engineering: A literature review and roadmap. In 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW), pages 548–557, 2025. ISO/IEC JTC 1/SC 7. Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model. Standard, International Organization for Standardization, 2023. Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, and Philipp Leitner. The impact of prompt programming on function-level code generation. IEEE Transactions on Software Engineering, 2025. Weize Kong, Spurthi Hombaiah, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. PRewrite: Prompt rewriting with reinforcement learning. In 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2024. Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. The language of prompting: What linguistic properties make a prompt successful?, 2023. arXiv:2311.01967. Dairui Liu, Boming Yang, Honghui Du, Derek Greene, Neil Hurley, Aonghus Lawlor, Ruihai Dong, and Irene Li. Recprompt: A self-tuning prompting framework for news recommendation using large language models. In 33rd ACM International Conference on Information and Knowledge Management, page 3902–3906, 2024. LLM Stats. Open LLM Leaderboard. https://llm-stats.com/leaderboards/ open-llm-leaderboard, 2025. Accessed: Jan 2025.

19

20

Linguistic Properties of Prompts as Predictors of LLM Performance

26

27 28

29

30 31

32 33

34 35

36

37 38

39

40

41

42 43

44

Do Xuan Long, Duy Dinh, Ngoc-Hai Nguyen, Kenji Kawaguchi, Nancy F. Chen, Shafiq Joty, and Min-Yen Kan. What makes a good natural language prompt? In 63rd Annual Meeting of the Association for Computational Linguistics, 2025. Shervin Malmasi and Mark Dras. Native language identification with classifier stacking and ensembles. Computational Linguistics, 44(3):403–446, 2018. Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. Prompt engineering in large language models. In International conference on data intelligence and cognitive informatics, pages 387–402, 2023. Alessio Miaschi, Dominique Brunato, Felice Dell’Orletta, and Giulia Venturi. Linguistic profiling of a neural language model. In 28th International Conference on Computational Linguistics, 2020. Alessio Miaschi, Felice Dell’Orletta, and Giulia Venturi. Evaluating large language models via linguistic profiling. In Empirical Methods in Natural Language Processing, 2024. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics, 12, 2024. Quim Motger, Carlota Catot, and Xavier Franch. Characterizing Datasets for LLM-based Requirements Engineering: A Systematic Mapping Study, 2026. arXiv:2510.18787. Wendkûuni Ouédraogo, Abdoul Kaboré, Yinghua Li, Haoye Tian, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. Prompt engineering in LLMs for automated unit test generation: A large-scale study. Empirical Software Engineering, 31(4):103, 2026. Luca Pascarella, Fabio Palomba, and Alberto Bacchelli. Fine-grained just-in-time defect prediction. Journal of Systems and Software, 150:22–36, 2019. Md Nafiu Rahman, Sadif Ahmed, Zahin Wahab, S M Sohan, and Rifat Shahriyar. Secret Breach Detection in Source Code with Large Language Models. In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 207–217, 2025. Krishna Ronanki, Simon Arvidsson, and Johan Axell. Prompt engineering guidelines for using large language models in requirements engineering. In Software Engineering and Advanced Applications: 51st Euromicro Conference, SEAA 2025, page 245–262, 2025. Krishna Ronanki, Beatriz Cabrero-Daniel, Jennifer Horkoff, and Christian Berger. Requirements Engineering Using Generative AI: Prompts and Prompting Patterns. 2024. Riccardo Rubei, Aicha Moussaid, Claudio Di Sipio, and Davide Di Ruscio. Prompt engineering and its implications on the energy consumption of large language models. In IEEE/ACM 9th International Workshop on Green and Sustainable Software, page 60–67, 2025. Iflaah Salman, Muhammad Waseem, Vladimir Mandic, and Rasanjana Dhanushkha De Alwis. A Vision for Debiasing Confirmation Bias in Software Testing via LLM . In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2025. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, 2024. arXiv:2310.11324. Jiho Shin, Clark Tang, Tahmineh Mohati, Maleknaz Nayebi, Song Wang, and Hadi Hemmati. Prompt engineering or fine-tuning: An empirical assessment of llms for code. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pages 490–502, 2025. Klaas-Jan Stol and Brian Fitzgerald. The ABC of software engineering research. ACM Transactions on Software Engineering and Methodology, 27(3):1–51, 2018. Hamed Taherkhani, Melika Sepidband, Hung Viet Pham, Song Wang, and Hadi Hemmati. Automated prompt engineering for cost-effective code generation using evolutionary algorithms. ACM Transactions on Software Engineering and Methodology, 2024. Hans Van Halteren. Linguistic profiling for authorship recognition and verification. In 42nd Annual Meeting of the Association for Computational Linguistics, pages 199–206, 2004.

Q. Motger, A. Miaschi, X. Franch, M. A. Zadenoori, and A. Ferrari

45 46

47 48 49

50

51

52

53

Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. Paraphrase types elicit prompt engineering capabilities. In Empirical Methods in Natural Language Processing, 2024. Guoqing Wang, Zeyu Sun, Sixiang Ye, Zhihao Gong, Yizhou Chen, Yifan Zhao, Qingyuan Liang, and Dan Hao. Do advanced language models eliminate the need for prompt engineering in software engineering? ACM Trans. Softw. Eng. Methodol., 2025. Claes Wohlin, Per Runeson, Martin Hst, Magnus C. Ohlsson, Bjrn Regnell, and Anders Wessln. Experimentation in Software Engineering. 2012. Qinyuan Ye, Mohamed Ahmed, Reid Pryzant, and Fereshte Khani. Prompt engineering a prompt engineer. In Findings of the ACL, pages 490–502, 2024. Mohammad Amin Zadenoori, Jacek Dąbrowski, Waad Alhoshan, Liping Zhao, and Alessio Ferrari. Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review, 2025. arXiv:2509.11446. Mohammad Amin Zadenoori, Liping Zhao, Waad Alhoshan, and Alessio Ferrari. Automatic prompt engineering: The case of requirements classification. In Requirements Engineering: Foundation for Software Quality, pages 217–225, 2025. Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso Letsholo, Muideen Ajagbe, ErolValeriu Chioasca, and Riza Batista-Navarro. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. ACM Comput. Surv., 54(3), 2021. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2023. arXiv:2211.01910. Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, 2024.

21

Related documents

• Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering#780924
• Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering#779549
• LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs#432595
• Read, Extract, Classify: A Tool for Smarter Requirements Engineering#178952
• LLM-Based Discovery of Latent Requirements from Stakeholder Conversations: Preliminary Results from Industry#307072