arXiv CSopen access

Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Preprint. Under review.

Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models Aleksandr Rubashevskii, Dzianis Piatrashyn, Preslav Nakov & Maxim Panov Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Abu Dhabi, UAE {Aleksandr.Rubashevskii, Maxim.Panov}@mbzuai.ac.ae

arXiv:2604.13991v1 [cs.CL] 15 Apr 2026

Abstract Large language models (LLMs) are prone to generating factually incorrect outputs. Recent work has applied conformal prediction to provide uncertainty estimates and statistical guarantees for the factuality of LLM generations. However, existing approaches are typically not prompt-adaptive, limiting their ability to capture input-dependent variability. As a result, they may filter out too few items (leading to over-coverage) or too many (under-coverage) for a given task or prompt. We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs, with applications to long-form generation and multiplechoice question answering. This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage. In addition, the approach naturally supports selective prediction, allowing unreliable claims or answer choices to be filtered out in downstream applications. We evaluate our approach on multiple white-box models across diverse domains and show that it significantly outperforms existing baselines in terms of conditional coverage.

Introduction

Large language models (LLMs) have demonstrated impressive performance across diverse applications (Zhao et al., 2023; Minaee et al., 2024). Despite this progress, they are still susceptible to hallucinations, producing fluent but factually incorrect outputs (Huang et al., 2025). This limitation is especially concerning in high-risk domains such as medicine, where even a few errors within extended generations can lead to significant consequences (Thirunavukarasu et al., 2023). To mitigate these risks, it is essential to develop methods with rigorous reliability guarantees. Conformal prediction offers a theoretically grounded approach to uncertainty quantification, providing distribution-free guarantees on error rates (Vovk et al., 2005; Angelopoulos & Bates, 2023). Conformal prediction has recently been applied to large language models in tasks such as long-form question answering (Mohri & Hashimoto, 2024) and multi-choice QA (Kumar et al., 2023). In these settings, conformal methods are typically used to construct prediction sets or filtering rules based on uncertainty scores, enabling selective prediction: the model either returns only high-confidence outputs or abstains from uncertain ones. For example, in long-form generation, individual claims or spans can be filtered based on their estimated reliability, while in multiple-choice settings, conformal prediction produces a subset of candidate answers guaranteed to contain the correct one with high probability. However, existing conformal procedures for LLMs lack adaptivity: a single calibrated quantile is applied uniformly across all test prompts, regardless of their difficulty, ambiguity, or rarity. While this guarantees marginal coverage on average, it can lead to substantial miscalibration at the prompt level (Cherian et al., 2024). For certain inputs, the method may exhibit over-coverage (an overly conservative threshold), whereas for others it may result in under-coverage (an insufficiently strict threshold). 1

Preprint. Under review.

1.0

0.8 0.7 0.6 0.5 0.4

Inventions Persons Inventions&Persons

0.9

Empirical coverage

0.9

Empirical coverage

1.0

Inventions Persons Inventions&Persons

0.8 0.7 0.6 0.5

0.5

0.6

0.7

0.8

Target coverage (a)

0.9

0.4

1.0

0.5

0.6

0.7

0.8

Target coverage (b)

0.9

1.0

Figure 1: Long-form QA target vs. empirical coverage for (a) Conformal Factuality and (b) Adaptive Conformal (ours). The conformal procedure is calibrated jointly on two categories (“Inventions” and “Persons”). While both methods achieve the target coverage marginally, only the adaptive approach closely approximates conditional coverage. We propose an adaptive conformal prediction approach for evaluating the factuality of large language models that accounts for the characteristics and difficulty of specific tasks. The proposed methodology is evaluated across multiple domains using various models and uncertainty quantification techniques. Figure 1 illustrates the main result of our work: standard conformal methods fail to achieve category-wise (conditional) coverage for heterogeneous prompts, whereas our adaptive approach improves conditional coverage while preserving marginal guarantees (see Section 3). Our contributions are as follows: 1. We propose a new conformal prediction approach for hallucination detection in LLMs that learns a prompt-adaptive correction to conformity scores via embeddingconditioned quantile regression. 2. We show that our method preserves the finite-sample marginal coverage guaranties of split conformal prediction while improving conditional coverage across heterogeneous prompts. 3. Experiments on long-form and multiple-choice question answering benchmarks across multiple LLMs show improved hallucination detection performance and more stable coverage compared to existing conformal methods.

Methodology

2.1

Background

Conformal prediction assumes exchangeable data { Xi , Yi }iN=+1 1 with input features Xi and output labels Yi , and a user-specified miscoverage level α. Using calibration data { Xi , Yi }iN=1 it constructs a prediction set Cα ( X ) such that for a new test point { X N +1 , YN +1 }: P YN +1 ∈ Cα ( X N +1 ) ≥ 1 − α. (1) This guarantee is marginal coverage, meaning the coverage holds on average over the distribution of X N +1 . Usually, it assumed that some predictive model fb( x ) was constructed that models the dependence between x and y. Let V ( x, y) be a nonconformity score function, where larger values indicate worse agreement between fb( x ) and y. Using a calibration set { Xi , Yi }iN=1 , define the calibration scores vi := V ( Xi , Yi ), i = 1, . . . , N. Then the conformal prediction set 2

Preprint. Under review.

(

Cα ( x ) =

y ∈ Y : V ( x, y) ≤ Q1−α

δv δ∞ ∑ N +i 1 + N + 1 i =1

!) ,

(2)

where Q1−α denotes the (1 − α)-quantile of a distribution, δv is a Dirac mass at v. In the next section, we show how conformal prediction can be adapted to ensure factuality of LLM generations. 2.2

Conformal Prediction for LLMs

LLMs typically generate free-form text rather than structured outputs. To enable finegrained factuality assessment, a popular approach is to decompose generated responses into atomic, verifiable claims. For example, the response “Paris is the capital of France and was founded in the 3rd century BC” can be split into claims such as (i) “Paris is the capital of France” and (ii) “Paris was founded in the 3rd century BC”. Let an LLM for a long-form QA task produce a finite set of candidate claims from input x: L( x ) = {c1 , . . . , cm } ⊂ C , where each ci is a verifiable atomic claim. Given a claim-level score s : C → R measuring uncertainty, define the filtered output at threshold t as Ft x, L( x ) := {c ∈ L( x ) : s(c) ≤ t}. (3) Intuitively, Ft retains only sufficiently low-uncertainty (i.e., confident) claims. Accordingly, the filtered set of claims produced by a large language model can be interpreted as a conformal prediction set, such as Cα in equation (2), as it restricts the output space to claims whose uncertainty scores do not exceed a calibrated threshold. Let w : C × Y → R be a claim-level factuality function (e.g., based on a pre-trained Natural Language Inference (NLI) model) that evaluates whether a claim is supported by the reference. We distinguish this from an uncertainty score s(c), which provides a model-based estimate of how likely a claim is to be incorrect and is used to rank and filter claims. In contrast, w(c, y) serves as an oracle that determines whether a claim is factually correct with respect to the ground-truth answer y. An illustrative example is provided in Appendix C.3. For the long-form QA setting, we define the score V ( x, y) as the largest uncertainty threshold such that all retained claims are factually correct: V ( x, y) = sup {t : ∀c ∈ Ft ( x, L( x )), w(c, y) ≥ β} ,

(4)

where β is a fixed task-dependent factuality threshold defining claim correctness. We compute the conformal threshold as the (1 − α)-quantile of the scores {vi = V ( xi , yi )}iN=1 on a calibration set. At test time, the final claim set is obtained by filtering according to the uncertainty scores: ( !) N δvi δ∞ L̄α ( x ) = c ∈ L( x ) : s(c) ≤ Q1−α ∑ + . (5) N+1 N+1 i =1 This procedure ensures the following marginal coverage guarantee: P c ∈ L̄α ( X N +1 ) : w(c, YN +1 ) ≥ β ≥ 1 − α.

(6)

The condition w(c, Y ) ≥ β plays a role analogous to the membership test Y ∈ Cα ( X ) in classical conformal prediction, defining whether a retained prediction is correct; see equation (1). Mohri & Hashimoto (2024) propose a related mechanism for long-form question answering using entailment-based sets. Multiple-Choice QA Setting. We further note that the reformulation of conformal prediction for long-form QA naturally extends to the multi-choice QA setting. In this case, the elements c in equation (3) correspond to candidate answer classes, and the filtration mechanism Ft produces a subset of predicted classes. The factuality function in equation (6) reduces to verifying whether the true class YN +1 is contained in the filtered set L̄α ( X N +1 ). 3

Preprint. Under review.

In this setting, the nonconformity score V ( x, y) can be defined using the least ambiguous classifier (LAC; Sadinle et al., 2019): V ( x, y) = 1 − [ p( x )]y ,

(7)

where [ p( x )]y denotes the predicted probability of the true class y. Under this formulation, the same marginal coverage guarantee is recovered: the true class belongs to the constructed prediction set with probability at least 1 − α. 2.3

Adaptive Conformal Prediction

Standard conformal prediction methods for LLMs rely on global thresholds and do not account for input-dependent variability, which can lead to substantial over- or under-coverage for specific inputs despite valid marginal guarantees. To address this limitation, we build on a class of methods that improve conditional coverage by transforming nonconformity scores using input-dependent normalization (see Section 4 for an overview of related works). In this framework, the transformed score is defined as Ṽ ( x, y) = f τ−(1x) V ( x, y) ,

(8)

where τ ( x ) is an estimate of the conditional (1 − α)-quantile of the original score. Such transformations aim to normalize the score so that its conditional quantiles are approximately invariant with respect to x, aligning the distributions across inputs. In this work, we consider a simple multiplicative normalization given by division by the estimated conditional quantile. This corresponds to the choice f t (v) = t · v, for which f t−1 (v) = v/t. This transformation can be interpreted as a local rescaling, reducing variability of the score across inputs and bringing conditional quantiles closer together. More generally, other transformations are possible within this framework. For example, additive normalization via shifting the score by its estimated conditional quantile can similarly reduce input dependence of the relevant quantile. Conformal prediction sets are then constructed using the transformed scores: ( !) N δ δ ṽ ∞ i C˜α ( x ) = y ∈ Y : Ṽ ( x, y) ≤ Q1−α ∑ + . N+1 N+1 i =1

(9)

This class of score-transformation methods has primarily been studied in regression settings and evaluated on relatively small-scale datasets. In contrast, we extend this framework to long-form LLM generation, where outputs consist of multiple sentences and atomic claims. In this setting, achieving approximate conditional validity is more challenging due to the need for large calibration data, informative input representations (e.g., prompt embeddings), and the complexity of long-form outputs. 2.4

Adaptive Conformal Factuality

Long-form QA. Dataset D consists of prompt–generation pairs x, L( x ) , where the model output L( x ) = {c1 , . . . , cm } is a set of extracted verifiable atomic claims. For each prompt Xi , i = 1, . . . , n, we additionally compute a sentence embedding e( Xi ). We split the dataset D into three disjoint subsets: Dcal1 , Dcal2 , and Dtest . We build on the filtration mechanism Ft and factuality function w introduced in Section 2.2. For the long-form QA setting, we define V ( x, y) as the maximal uncertainty threshold such that all retained claims are factually correct; see equation (4). In our setting, factuality is evaluated using binary labels, so w(c, y) ∈ {0, 1} and β = 1, meaning that all retained claims must be correct. (1) (1) ncal1 On Dcal1 , we compute scores V Xi , Yi and train a conditional quantile estimator i =1 (1) (1) (1) ncal1 τ̂ ( x ) (using the pinball loss) on the pairs e Xi , V Xi , Yi i =1 . We use τ̂ ( x ) as 4

Preprint. Under review.

Algorithm 1: Adaptive Conformal Factuality for Long-Form QA (1) (1) ncal1 Input: LLM L, miscoverage level α, calibration sets Xi , Yi i =1 and (2) (2) ncal2 Xi , Yi i =1 , pre-trained prompt embedding extractor e, function V ( x, y) from equation (4), test prompt x. 1 for i ← 1 to ncal do 1 (1) (1) 2 z i ← e Xi ; (1) (1) (1) 3 vi ← V Xi , Yi ; (1) (1) ncal1 4 Fit a conditional (1 − α )-quantile regressor τ̂ on zi , vi i =1 ; 5 for i ← 1 to ncal do 2 (2) (2) 6 z i ← e Xi ; (2) (2) 7 τ̂i ← τ̂ zi ; (2) (2) V Xi ,Yi (2) 8 vi ← ; (2) τ̂i (2) ncal vi i=12 ; 9 q̂1− α ← Q1− α s(c)

Output: L̄α ( x ) ← {c ∈ L( x ) : τ̂ ( x) ≤ q̂1−α }

shorthand for τ̂ (e( x )), where the conditional quantile is evaluated on the embedding e( x ). The details of this procedure are provided in Section 3.1. (2) (2) ncal2 On Dcal2 , we compute transformed scores Ṽ Xi , Yi i =1 : Ṽ

V (2) (2) Xi , Yi =

We then compute the conformal threshold Q1−α of these transformed scores.

(2)

Xi , Yi (2) τ̂ Xi (2)

(10)

(2) ncal2 i =1 as the (1 − α )-quantile

Ṽ Xi , Yi

At test time, we evaluate transformed scores of candidate claims and filter them using the calibrated threshold. Both the claim-level scores s(c) and the calibration thresholds V ( x, y) are normalized by τ̂ ( x ), so that they are expressed on the same scale and can be compared using a single global threshold. The resulting conformal prediction set is L̄α ( x ) =

s(c) (2) (2) ncal2 ≤ Q1−α {Ṽ ( Xi , Yi )}i=1 . c ∈ L( x ) : τ̂ ( x )

(11)

The resulting algorithm is summarized in Algorithm 1. The predictions satisfy marginal coverage guarantees as in equation (6). The corresponding theoretical result and its proof are provided in Appendix A. Multi-choice QA. The proposed method also applies to multiple-choice question answering. The same pipeline is used: training the conditional quantile estimator on Dcal1 , calibrating transformed scores on Dcal2 , and filtering on Dtest . The main differences are: (i) the prediction set L̄α ( x ) consists of classes rather than claims, (ii) the task-specific nonconformity score is given by the least ambiguous classifier (see equation (7)). The resulting conformal prediction set is L̄α ( x ) =

V ( x, y) (2) (2) ncal2 y ∈ Y: ≤ Q1−α Ṽ Xi , Yi . i =1 τ̂ ( x ) 5

(12)

1.0

0.9

Empirical coverage

Preprint. Under review.

0.8 0.7 0.6

Marketing Accounting Marketing&Accounting

0.5 0.4

0.5

0.6

0.7

0.8

Target coverage (a)

0.9

Marketing Accounting Marketing&Accounting

0.8 0.7 0.6 0.5 0.4

1.0

0.5

0.6

0.7

0.8

Target coverage (b)

0.9

1.0

Figure 2: Multi-choice QA experimental results: target vs. empirical coverage for (a) Conformal Factuality and (b) Adaptive Conformal method (ours). Results are shown for two prompt categories, with coverage reported for each category individually as well as jointly. The conformal threshold is calibrated jointly across both categories.

Experimental Study

3.1

Setup

Model generations are produced using Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), Llama-3.1-8B-Instruct (Grattafiori et al., 2024) and Gemma-3-12B-Instruct (Team et al., 2025). We extract prompt embeddings using the multilingual model multi-qa-mpnet-base-dot-v1 (Reimers & Gurevych, 2020) and reduce the resulting 768-dimensional embeddings to 32 dimensions via PCA. Further details on the dimensionality reduction procedure are provided in Appendix C.1. We split the data D into three disjoint subsets Dcal1 , Dcal2 and Dtest in proportions 0.3, 0.4 and 0.3, respectively. The conditional quantile is modeled with a two-layer MLP with ReLU. We repeat each experiment 10 times, randomly shuffling the data and performing a new split in each run. We report the mean and standard deviation across the runs. 3.2 3.2.1

Dataset Long-form QA

Following Shelmanov et al. (2025), we generate long-form samples for each of the 8 categories: Biographies, Cities, Movies, Inventions, Books, Artworks, Landmarks, and Events. All generations are decomposed and decontextualized into atomic claims, which are subsequently labeled using GPT-4o. Instead of generating the original 100 samples per category, we produce three times as many, resulting in 300 long-form LLM generations per category. The motivation for increasing the sample size is that we aim to provide per-prompt conformal guarantees on factuality. Moreover, the data is further divided into three disjoint subsets for conditional quantile training, calibration, and testing. Consequently, several hundred samples per category are required to obtain representative and robust estimates. As for the claim scoring function s(c), we consider several claim-level uncertainty measures for white-box models, including Maximum Probability, Maximum Token Entropy (Fomicheva et al., 2020), Perplexity (Fomicheva et al., 2020), Claim Condition Probability (Fadeeva et al., 2024), TokenSAR (Duan et al., 2024), Pointwise Mutual Information (Takayama & Arase, 2019). For data generation and claim-level uncertainty quantification, we use the LM-Polygraph library (Fadeeva et al., 2023). 6

Preprint. Under review.

Claim Scoring Method

Mistral 7B

Llama3 8B

Gemma3 12B

0.189 0.273 0.255 0.313 0.189 0.360 0.288

0.166 0.281 0.257 0.324 0.158 0.367 0.286

0.138 0.180 0.162 0.189 0.136 0.238 0.182

Random Baseline Maximum Probability Perplexity Max Token Entropy Pointwise Mutual Information Claim Conditioned Probability TokenSAR

Table 1: PR-AUC for long-form QA claim scoring functions on Mistral 7B, Llama3 8B and Gemma3 12B. Higher is better, best method per column is colored.

3.2.2

Multi-choice QA

Similar to Kumar et al. (2023) for multiple-choice question answering, we select 16 categories from the MMLU dataset (Hendrycks et al., 2021). Each data category has at least 100 questions, each question has 4 possible answers. Unlike the original paper, which applies conformal prediction independently within each category, we construct a single conformal predictor using data from all categories jointly and subsequently evaluate its performance separately for each category. Dataset statistics presented in Table 5. 3.3

Long-form QA Experimental Results

Claim Scoring Functions Comparison. First, we compare various claim-level uncertainty quantification methods for claim filtering. We evaluate performance using PR-AUC, which is more informative in imbalanced settings and directly captures the precision–recall trade-off when filtering incorrect claims. Table 1 shows that the Claim Conditioned Probability (CCP) method achieves the best performance across all evaluated generation models. By focusing on claim-specific uncertainty rather than non-task-relevant factors such as claim order or surface form variability, CCP consistently outperforms competing approaches. Based on these results, we use CCP as the claim scoring method in subsequent conformal prediction experiments for long-form QA. Calibration on Two Categories. We compare global quantile thresholding via Conformal Factuality (Mohri & Hashimoto, 2024) with our adaptive conformal approach based on transformed scores. Conformal Factuality applies a single quantile threshold computed jointly on Dcal1 and Dcal2 , which is then used at test time. In contrast, our method uses Dcal1 to train a conditional quantile estimator and Dcal2 to calibrate the transformed scores. In a long-form QA experiment, we select two categories with substantially different conformity score distributions: Biographies and Inventions. As shown in Figure 1, both methods satisfy the marginal conformal guarantee. However, global thresholding fails to achieve conditional coverage, resulting in over-coverage for Inventions (a more complex category) and under-coverage for Biographies (an easier category). In contrast, our adaptive conformal procedure preserves marginal coverage while achieving improved conditional coverage, yielding more consistent performance across categories as well as on the overall dataset. Calibration Using All Data. For this experiment, we calibrate the threshold jointly across all eight categories. Tables 2 and 3 report category-wise coverage and the fraction of removed claims at target coverage 0.80 for Mistral 7B and Gemma-3 12B, respectively, while results for LLaMA-3.1 8B are provided in Appendix B.1. Across models, adaptive conformal prediction improves coverage alignment while typically reducing the fraction of removed claims. For Mistral 7B, the largest gains occur in Landmarks, Inventions, and Artworks, with reduced removal in the first two. For Gemma-3 12B, similar improvements are observed in Persons, Artworks, and Events, along with reduced variability across categories. 7

Preprint. Under review.

Coverage

% Removed

Category

Original

Adaptive

Original

Adaptive

inventions persons artworks books cities movies landmarks events

85.12 ± 2.69 73.91 ± 2.32 68.00 ± 4.98 85.73 ± 3.75 86.35 ± 2.04 87.71 ± 3.12 80.39 ± 4.55 73.39 ± 6.58

80.33 ± 4.93 79.82 ± 3.62 76.90 ± 5.02 82.20 ± 3.60 77.21 ± 4.10 79.56 ± 3.69 79.89 ± 4.58 82.62 ± 2.55

88.18 ± 0.92 79.45 ± 0.73 80.99 ± 0.89 83.82 ± 1.30 86.54 ± 0.47 84.98 ± 0.62 80.67 ± 1.34 79.43 ± 1.66

81.52 ± 3.44 81.80 ± 4.02 81.04 ± 3.87 80.66 ± 3.50 80.79 ± 3.61 80.40 ± 2.76 80.53 ± 3.70 79.37 ± 3.04

Table 3: Gemma-3 12B results (mean ± std over seeds) at α = 0.20 with target coverage 0.80. problem. Given a set of problems P , the performance profile is defined as p ∈ P : r ps ≤ δ , ρs (δ) = |P |

(14)

representing the fraction of problems for which method s is within a factor δ of the best one. Figure 3 shows the resulting performance profiles. Across both models, the adaptive method consistently outperforms the original method, as indicated by its uniformly higher curve across nearly all values of δ. In particular, at δ = 1, it achieves the best calibration error on a larger fraction of problems, and remains closer to the best-performing method as δ increases. Overall, this demonstrates more robust and reliable calibration across heterogeneous categories.

Related Work

Recently, conformal prediction has been extended to large language models across several settings, including long-form generation (Mohri & Hashimoto, 2024), multiple-choice QA (Kumar et al., 2023), and response sampling (Quach et al., 2024). More broadly, improving conditional coverage has been studied via input-dependent normalization of conformity scores. Plassier et al. (2025) propose transforming scores to equalize conditional quantiles, closely related to normalized conformal prediction (Johansson et al., 2021; Lei et al., 2018). Other approaches use localization or reweighting, including kernelbased methods (Guan, 2023), quantile regression forests (Amoukou & Brunel, 2023), and learned score transformations (Xie et al., 2024). In the context of LLMs, Cherian et al. (2024) address conditional coverage via a boostingbased method that improves group-wise calibration. However, their approach relies on predefined groups and hand-crafted features, and requires solving a linear system for quantile estimation. In contrast, we achieve prompt-level adaptivity using learned representations and conditional quantile regression, without explicit grouping or feature engineering. A related direction considers domain-shift-aware conformal prediction, reweighting calibration samples based on similarity to test inputs (Lin et al., 2025).

Conclusion

We propose a new adaptive conformal prediction framework for large language models based on nonconformity score transformations via conditional quantile regression. The method preserves marginal guarantees while enabling prompt-dependent calibration and improving conditional coverage. Experiments across multiple models and domains show consistent gains over existing baselines, particularly for heterogeneous categories. Future work includes extending adaptive conformal methods to broader generation tasks, improving input representations, and strengthening theoretical guarantees. 9

Preprint. Under review.

References Salim I Amoukou and Nicolas JB Brunel. Adaptive conformal prediction by reweighting nonconformity score. arXiv preprint arXiv:2303.12695, 2023. URL https://arxiv.org/ abs/2303.12695. Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. URL https://www.emerald.com/ftmal/article-abstract/16/4/494/1332423/ Conformal-Prediction-A-Gentle-Introduction. John Cherian, Isaac Gibbs, and Emmanuel Candes. Large language model validity via enhanced conformal prediction methods. In Advances in Neural Information Processing Systems, volume 37, pp. 114812–114842, 2024. URL https://proceedings.neurips.cc/paper files/paper/2024/hash/d02ff1aeaa5c268dc34790dd1ad21526-Abstract-Conference. html. Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201–213, 2002. URL https://link.springer. com/article/10.1007/s101070100263. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5050–5063, 2024. URL https://aclanthology.org/2024.acl-long.276.pdf. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-polygraph: Uncertainty estimation for language models. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 446–461, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023.emnlp-demo.41. URL https://aclanthology.org/2023.emnlp-demo.41. Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. Fact-checking the output of large language models via token-level uncertainty quantification. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 9367–9385, 2024. URL https://aclanthology.org/2024.findings-acl.558.pdf. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555, 2020. doi: 10.1162/tacl a 00330. URL https://aclanthology.org/ 2020.tacl-1.35/. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/ abs/2407.21783. Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110(1):33–50, 2023. URL https://academic.oup.com/ biomet/article/110/1/33/6647831. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://openreview. net/pdf?id=d7KBjmI3GmQ. 10

Preprint. Under review.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55, 2025. URL https://dl.acm.org/doi/10. 1145/3703155. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825. Ulf Johansson, Henrik Boström, and Tuwe Löfström. Investigating normalized conformal regressors. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 01–08. IEEE, 2021. URL https://ieeexplore.ieee.org/abstract/document/9659853. Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404, 2023. URL https://arxiv.org/abs/ 2305.18404. Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018. URL https://www.tandfonline.com/doi/full/10. 1080/01621459.2017.1307116. Zhexiao Lin, Yuanyuan Li, Neeraj Sarna, Yuanyuan Gao, and Michael von Gablenz. Domain-shift-aware conformal prediction for large language models. arXiv preprint arXiv:2510.05566, 2025. URL https://arxiv.org/abs/2510.05566. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. URL https://arxiv.org/abs/2402.06196. Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. In International Conference on Machine Learning, pp. 36029–36047. PMLR, 2024. URL https://proceedings.mlr.press/v235/mohri24a.html. Vincent Plassier, Alexander Fishkov, Victor Dheur, Mohsen Guizani, Souhaib Ben Taieb, Maxim Panov, and Eric Moulines. Rectifying conformity scores for better conditional coverage. In Forty-second International Conference on Machine Learning, 2025. URL https: //openreview.net/forum?id=STEhUnCmdm. Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pzUhfQ74c5. Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020. URL https://arxiv.org/abs/2004.09813. Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234, 2019. URL https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1395341. Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, et al. A head to predict and a head to question: Pre-trained uncertainty quantification heads for hallucination detection in llm outputs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35700–35719, 2025. URL https: //aclanthology.org/2025.emnlp-main.1809/. 11

Preprint. Under review.

Junya Takayama and Yuki Arase. Relevant and informative response generation using pointwise mutual information. In Yun-Nung Chen, Tania Bedrax-Weiss, Dilek Hakkani-Tur, Anuj Kumar, Mike Lewis, Thang-Minh Luong, Pei-Hao Su, and Tsung-Hsien Wen (eds.), Proceedings of the First Workshop on NLP for Conversational AI, pp. 133–138, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4115. URL https://aclanthology.org/W19-4115/. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. URL https://www.nature.com/articles/ s41591-023-02448-8. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer, 2005. URL https://link.springer.com/book/10.1007/ 978-3-031-06649-8. Ran Xie, Rina F Barber, and Emmanuel J Candès. Boosted conformal prediction intervals. Advances in Neural Information Processing Systems, 37:71868– 71899, 2024. URL https://proceedings.neurips.cc/paper files/paper/2024/hash/ 842714f78c95096e20ac7d2591c5a24b-Abstract-Conference.html. 12

Preprint. Under review.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2):1–124, 2023. URL https://arxiv.org/abs/ 2303.18223.

Preprint. Under review.

Theoretical Result

Assume that τ ( x ) := Q1−α ( PV | X = x ) denotes the oracle conditional quantile of the nonconformity score given X = x. For every x, let f τ ( x) be a strictly increasing and continuous transformation, and define Vk := V ( Xk , Yk ), Ṽk := f τ−(1X ) (V ( Xk , Yk )), k = 1, . . . , N + 1. k

Define the prediction set ( L̄( x ) =

c ∈ L( x ) : f τ−(1x) (s(c)) ≤ Q(1−α)(1+ 1 ) N

1 N δṼi N i∑ =1

!) .

Let β ∈ R be a predefined factuality threshold. We say that L̄( x ) is factually correct for y if ∀c ∈ L̄( x ), w(c, y) ≥ β. We assume the following compatibility condition: for every ( x, y) and every threshold q ∈ R, n o n o ∀c ∈ L( x ) : f τ−(1x) (s(c)) ≤ q ⇒ w(c, y) ≥ β = f τ−(1x) (V ( x, y)) ≤ q . (15) Theorem. Let {( Xi , Yi )}iN=+1 1 be exchangeable, and assume that Ṽ1 , . . . , ṼN +1 are almost surely distinct. Then, for α ∈ N1+1 , 1 , 1 − α ≤ P(∀c ∈ L̄( X N +1 ), w(c, YN +1 ) ≥ β) < 1 − α +

1 . N+1

Proof. Since τ (·) is the oracle conditional quantile, the map

( x, y) 7→ f τ−(1x) (V ( x, y)) is fixed and applied independently to each pair ( Xk , Yk ). Therefore, by the exchangeability of {( Xk , Yk )}kN=+11 , the transformed scores Ṽ1 , . . . , ṼN +1 are also exchangeable. Let k α := ⌈( N + 1)(1 − α)⌉ . Since α ≥ N1+1 , we have k α ∈ {1, . . . , N }. By the definition of the empirical quantile, ! 1 N Q(1−α)(1+ 1 ) δṼi = Ṽ(kα ) , N N i∑ =1 where Ṽ(1) < · · · < Ṽ( N ) are the order statistics of Ṽ1 , . . . , ṼN . By the definition of L̄ and the compatibility condition (15), n o {∀c ∈ L̄( X N +1 ), w(c, YN +1 ) ≥ β} = ṼN +1 ≤ Ṽ(kα ) . Since Ṽ1 , . . . , ṼN +1 are exchangeable and almost surely distinct, the rank of ṼN +1 among Ṽ1 , . . . , ṼN +1 is uniformly distributed over {1, . . . , N + 1}. Therefore, kα P ṼN +1 ≤ Ṽ(kα ) = . N+1 Finally, by the definition of k α , ( N + 1)(1 − α) ≤ k α < ( N + 1)(1 − α) + 1. Dividing by N + 1 yields 1 kα 1−α ≤ < 1−α+ . N+1 N+1 Combining the above proves the result. 14

Preprint. Under review.

Additional Experimental Results

B.1

Additional Long-form QA Results

Coverage

% Removed