It’s Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty Kevin Guo1 , Chao Yan2 , Avinash Baidya3 , Katherine Brown2 , Xiang Gao3 , Juming Xiong1 , Zhijun Yin1,2 , Bradley Malin1,2 , 1 Vanderbilt University, 2 Vanderbilt University Medical Center, 3 Intuit AI Research,
arXiv:2605.27288v1 [cs.CL] 26 May 2026
Abstract
warn against a dangerous drug interaction, but inappropriately approve the combination if the user insists it is safe (Chen et al., 2025). To date, the research community has largely equated this conformity with sycophancy, attributing it primarily to artifacts of reinforcement learning from human feedback (RLHF) (Kalai et al., 2025; Kim et al., 2026; Sicilia et al., 2025; Li et al., 2026). While some degree of this behavior likely stems from sycophantic tendencies acquired during RLHF, prior sycophancy evaluations have overlooked the influence of a model’s inference-time uncertainty in addressing a prompt. To understand how inference-time uncertainty can drive conformity, it is helpful to draw a parallel to social dynamics. In human-to-human interactions, a key factor determining whether an individual will yield to the suggestion of another is their initial uncertainty about the subject (Deutsch and Gerard, 1955; Baron et al., 1996). To define this uncertainty in LLM interactions, we adopt the term epistemic uncertainty from prior literature, which reflects the uncertainty arising from limitations in an LLM’s inherent knowledge or capabilities (Gao et al., 2024; Xia et al., 2025). For example, a model encountering a rare clinical case outside its pretraining corpora may adopt a user’s suggested diagnosis because its epistemic uncertainty is high. Without controlling for this uncertainty, current approaches to evaluating sycophancy risk conflating pure sycophancy with conformity due to uncertainty. In this paper, we seek to decouple LLM conformity driven by sycophantic behaviors acquired during alignment, from conformity driven by epistemic uncertainty at inference-time. To achieve this, we introduce Measuring Uncertainty in Sycophancy Evaluation (MUSE), a two-step framework to investigate LLM conformity as a joint function of alignment-time sycophancy and inferencetime epistemic uncertainty (Figure 1).
Large language models (LLMs) are known to abandon their initial stance to conform to user pushback. While prior research largely attributes this behavior to sycophancy learned during reinforcement learning from human feedback, we hypothesize that conformity is also driven by a model’s epistemic uncertainty at inference time. In this paper, we introduce MUSE, a two-stage evaluation framework to disentangle the mechanisms driving LLM conformity. Specifically, MUSE maps a model’s epistemic uncertainty in responding to a query against its likelihood to yield to user pushback in a subsequent turn. We demonstrate that the mechanisms driving conformity extend beyond sycophancy alone. Specifically, we characterize two distinct factors that jointly drive conformity: sycophantic conformity, where a model aligns with user pushback even with absolute certainty in its initial response, and uncertaintydriven conformity, where a model’s likelihood for conformity increases alongside its uncertainty. Furthermore, we conduct ablation studies to demonstrate that both sycophantic conformity and uncertainty-driven conformity grow with 1) the LLM’s perceived expertise of the user and 2) the plausibility of the user’s suggestions. More broadly, MUSE informs more targeted intervention strategies by distinguishing alignment-induced sycophancy and trainingcorpora-driven uncertainty.
1
Introduction
Large language models (LLMs) are increasingly deployed as conversational assistants in high-stakes domains, raising concerns about their reliability in extended dialogue (Wu et al., 2023; Goh et al., 2024). Specifically, these models are known to abandon an initial stance to conform to a user’s beliefs when faced with conversational pressure (e.g., user pushback against an initial response) (Sharma et al., 2023). For instance, a model might correctly 1
Step 1: Measuring Uncertainty through Decision-Space Entropy
Step 2: Mapping Uncertainty to Conformity under Pushback Query + choices
Entropy HM(xi)
Query xi LLM M Choice space Ai1, Ai2, ..., Ain Multi-choice QA
Model Conformity
Initial answer Consider this new option ... k samples at T=1.0
P(Switch)
Updated answer
Ai1 Ai2 ... Ain
Uncertainty-Driven + Sycophantic Conformity Pure Sycophancy
Entropy HM Neutral
Assertive
Expert
Figure 1: The MUSE Framework. Step 1 estimates a model’s inference-time epistemic uncertainty by computing a query’s decision-space entropy across k stochastic samples. Step 2 maps this baseline uncertainty against the model’s likelihood of yielding to conversational pushback. This decouples pure sycophancy (yielding under absolute certainty) from uncertainty-driven conformity.
The primary contributions include:
factual accuracy. Early approaches for evaluating sycophancy have relied primarily on prompting a model with an illogical request or factually inaccurate information (Chen et al., 2025; Ibrahim et al., 2026) and measuring its tendency to align with the flawed premises rather than resist or correct it. As LLMs are adopted beyond single-shot queries to power chatbots and copilots, more recent studies have evaluated the impact of sycophancy in multi-turn dialogues. These evaluations simulate conversational pressure by subjecting a model to user pushback after initially selecting an answer (typically in a question-answer setting), showing that models frequently align with factually inaccurate user suggestions and compound these errors across turns of conversation (Laban et al., 2026; Guo et al., 2026; Kim et al., 2026). Specifically, Kim et al. formalize a model’s initial response as its assessment of the prompt and any subsequent changing in that stance to be the result of sycophancy (Kim et al., 2026).
1. We introduce MUSE, a novel evaluation framework that maps a model’s baseline epistemic uncertainty against its likelihood to yield to user pushback, disentangling inference-time uncertainty from alignmentinduced sycophancy. 2. We evaluate 7 LLMs using MUSE to decouple conformity into i) pure sycophancy (yielding despite absolute certainty) and ii) uncertaintydriven conformity (yielding scales alongside uncertainty), illustrating that current metrics consistently overestimate pure sycophancy. 3. We conduct ablation studies to demonstrate that both forms of conformity vary depending on pushback plausibility and perceived user expertise, suggesting that conformity is sensitive to how a model is prompted. Together, these contributions show that detecting and mitigating unwarranted conformity requires targeted, context-specific interventions, addressing sycophancy during alignment and reducing uncertainty through model development.1
2
Background and Related Work
2.1
Sycophancy in LLMs
2.2
Uncertainty estimation in LLMs characterizes two primary types of uncertainty: aleatoric and epistemic. Aleatoric uncertainty reflects the ambiguous and non-deterministic nature of the dependency between input and output which is irreducible (Hüllermeier and Waegeman, 2021). By contrast, MUSE focuses on measuring epistemic uncertainty (hereon referred to as just uncertainty), which arises when a model is insufficiently complex to represent the knowledge it has been presented, or interacts with concepts underrepresented in its pretraining corpora, reflecting improvable gaps in its inherent capabilities (Gao et al., 2024). Historically, measuring uncertainty leverages tools like token-level log-probabilities, which rely on a model’s internal next-token predictive distribution, and explicated confidence scores, where
Prior literature defines sycophancy as the phenomenon where a model aligns its response with a user’s stated or inferred beliefs, even when those beliefs are illogical or factually inaccurate (Sharma et al., 2023). Recent evidence attributes this behavior to RLHF, which optimizes chatbots to be helpful assistants (Kalai et al., 2025). However, in optimizing towards helpfulness, models are inadvertently trained to prioritize helpfulness over 1
Uncertainty Estimation in LLMs
Code will be open-sourced upon publication.
2
3.2
a model explicitly outputs its confidence in a response. However, since log-probabilities are computed iteratively, they can be skewed by prompt and output length, as well as syntax (Kuhn et al.; Holtzman et al., 2021). Prompting a model to explicitly output a confidence score (Lin et al., 2022) has also been shown to be unreliable, as LLMs struggle to self-assess, causing misalignment with empirical accuracy (Xiong et al., 2024). Alternatively, researchers have adopted semantic entropy and selfconsistency methods, which measure agreement across multiple sampled outputs (Wang et al., 2023; Farquhar et al., 2024). While this addresses the limitations of log-probs and explicated confidence, it sacrifices granularity of response distributions to achieve an aggregate vote (Tan et al., 2025). In short, existing methods struggle to isolate knowledge gaps from generative or alignment artifacts.
3
MUSE Framework
3.1
Modeling Uncertainty through Decision-Space Entropies
Simulating Conversational Pressure
To capture how baseline uncertainties influence a model’s tendency to yield under conversational pressure, we introduce a simple two-turn evaluation framework. We begin by narrowing each 10-option question into a four-choice format containing the correct answer and three randomly sampled incorrect distractors. After the model makes an initial choice (A–D), we randomly sample a fourth distractor, option E, and prompt the model to stick to its original stance or switch to the new suggestion. We narrow the scope of each decision space because models rarely spread their selection across all ten answer choices, instead leaving probability mass concentrated in a select few options (Figure A.1). We quantify a model’s likelihood of conformity as its observed switch rate, defined as the proportion of instances where the model abandons its initial choice in favor of the newly suggested distractor. Because models are unaware of groundtruth labels at inference time, we do not condition our evaluation on answer correctness, and instead measure conformity strictly as the rate at which they abandon their initial stance. After establishing our core findings regarding uncertainty-driven conformity in Sections 5 and 6, we conduct two ablation studies. Specifically, we dedicate Section 7 to analyzing the effect of suggestion plausibility by manipulating the curated decision-space, and Section 8 to assessing the influence of perceived expertise in authoritative prompting styles on conformity rates.
To isolate sycophancy from uncertainty-driven conformity, MUSE establishes a baseline uncertainty for each prompt prior to introducing conversational pressures. Formally, let xi denote a query and Ai a set of potential answer options. For a given model M , we approximate the predictive distribution over Ai by generating k independent inferences, where ŷj ∼ PM (· | xi , T ) is the model’s selected answer choice at a sampling temperature of T = 1.0. We measure the probability of a model selecting a specific option a ∈ Ai as the observed frequency across the k stochastic samples
4
Experimental Setup
k
4.1
Datasets
j=1
We evaluate LLMs on four tasks from two tenoption multiple-choice benchmarks:
1X p̂(a | xi ) = 1(ŷj = a). k Then, we consolidate the empirical distribution of each prompt into its Shannon entropy X H(xi ) = − p̂(a | xi ) log2 p̂(a | xi ),
• MedXPertQA Diagnosis (Zuo et al., 2025): 921 questions evaluating a model’s ability to make accurate differential diagnoses. • MedXPertQA Treatment: 631 questions evaluating pharmacological interventions, preventative measures, and care plans.
a∈A
which we use to represent a model’s uncertainty (or conversely, confidence) in each prompt. MUSE leaves k to be adapted to balance between the granularity of entropy required and the computational cost of each inference. Additionally, while our analyses rely on finite multiple-choice answer spaces, MUSE does not require predefined answer sets and can be adopted for any query where a potential set of answers can be generated.
• MMLU Pro Economics (Wang et al., 2024): 844 questions testing advanced reasoning over economic policy and theory. • MMLU Pro Business: 789 questions testing strategic management, complex accounting principles, and financial decision-making. 3
Cumulative Prob.
1.0
MedXPertQA Diagnosis
MedXPertQA Treatment
0
0
MMLU Pro Economics
MMLU Pro Business
0.5
0.0
1
Mistral 3.2 24B
2 Gemma 3 27B
1
2 0 1 2 Entropy (Bits) Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B
0
1
Llama 3.3 70B
2 GPT-5.4
Figure 2: Cumulative distribution of decision-space entropies (H). Higher entropy indicates greater model uncertainty over the 10-option answer space prior to conversational intervention.
4.2
Models
ness, with MedXPertQA Diagnosis and Treatment falling in between the two. This behavior is likely a combination of discrepancies in the coverage of each topic within the pretraining corpora of each models as well as the inherent difficulty of each dataset’s questions.
We evaluate six popular open-source LLMs and one proprietary frontier model: Mistral 3.2 Small 24B, Gemma 3 27B, Granite 4.1 30B, Qwen 2.5 32B, Olmo 3.1 32B, Llama 3.3 70B, and GPT-5.4. Because sycophantic tendencies are acquired during RLHF (Sharma et al., 2023), our analyses focus on the instruct-tuned variants of each model. See Section A.1 in the Appendix for prompting details. For all open-weight models, we stochastically sample k = 100 inferences per prompt in each dataset. Due to cost constraints, we sample k = 50 inferences per prompt for GPT-5.4 evaluations for a 300-question subset of each dataset. Our findings indicate that these selections for k are sufficient to capture stable and representative uncertainty distributions. 4.3
5
Modeling Conformity Through Uncertainty
It should be recognized that, if LLM conformity were solely an artifact of alignment-induced sycophancy, a model’s likelihood of conforming to user pushback should be independent of its uncertainty. By contrast, as we show in Figure 3, almost all models exhibit a strong, positive trend between their uncertainty in the initial query and their subsequent likelihood of yielding to a user suggestion. Notably, when evaluated on MMLU Pro Economics, the switch rates for Llama 3.3 70B and Qwen 2.5 32B transition from just ∼ 10% and ∼ 25%, respectively, under absolute certainty to nearly 100% as entropy approaches 2 bits. Interestingly, the relationship between uncertainty and switch rates is sensitive to task context. For example, GPT-5.4 demonstrates a strong positive trend in MMLU Pro Business, but a comparatively weaker one in MedXPertQA Diagnosis. Similarly, Mistral 3.2 24B exhibits a much stronger positive trend in MMLU Pro Economics than in the other datasets. Overall, we find that a model’s robustness against conversational pressure is conditioned on its pre-existing predictive entropy, underscoring that LLM conformity is in part driven by inference-time uncertainty. We provide additional analysis on the small subset of models which rarely yield to pushback across all uncertainty levels, such as Olmo 3.1 32B and Gemma 3 27B in certain datasets in Section 8.
Baseline Entropy Distributions
To test our hypothesis that a model’s likelihood of conforming to user pushback correlates with its underlying uncertainty, we first need to confirm that models exhibit a range of entropy in their initial responses. Figure 2 illustrates the cumulative distribution of decision-space entropies across all models and datasets. These distributions exhibit high variability between models and datasets. While a few models like Gemma 3 27B exhibits absolute certainty in ∼ 90% of decision-spaces, others (e.g., Olmo 3.1 32B, Mistral 3.2 24B) exhibit certainty in just 20%-30% of decision-spaces. These results highlight that depending on the dataset and model, LLMs demonstrate a large range of (un)certainty in their decision-spaces. In regard to dataset (or task context), models demonstrate the least entropy, and thus the most confidence, in MMLU Pro Economics and the most entropy or least confidence in MMLU Pro Busi4
P(Switch)
1.0
MedXPertQA Diagnosis
MedXPertQA Treatment
MMLU Pro Economics
MMLU Pro Business
0.5 0.0
0
Mistral 3.2 24B
1
2 0
Gemma 3 27B
1
2 0 Entropy (Bits) Granite 4.1 30B Qwen 2.5 32B
1 Olmo 3.1 32B
2 0
1
Llama 3.3 70B
2 GPT-5.4
Figure 3: Uncertainty-Driven Conformity. Logistic regression (with bootstrapped 95% CIs) modeling the probability of yielding an initial stance as a function of epistemic uncertainty. Conformity begins at the pure sycophancy baseline (y-intercept) and increases alongside decision-space entropy. Lines terminate at the entropy ceiling for each model.
6
Decoupling Sycophancy from Uncertainty-Driven Conformity
Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4
Having established that a model’s likelihood for conformity scales alongside its uncertainty, we characterize two distinct modes of LLM conformity: 1) conformity under absolute certainty (H = 0) and 2) conformity under epistemic uncertainty (H > 0). We define the former as pure sycophancy, as the model abandons its initial stance solely due to conversational pressure and despite absolute certainty. Conversely, we define the latter as uncertainty-driven conformity. This distinction proposes a shift in how we classify sycophancy, taking a more granular approach to disentangle alignment-induced flaws from knowledge gaps inherent to training corpora and model complexities. To contextualize why this distinction is important, we examine the baseline prevalence of epistemic uncertainty. Figure 4 illustrates that models frequently exhibit absolute certainty (H = 0) in their initial responses, indicating that prior works are not incorrect in observing that pure sycophancy exists (Kim et al., 2026; Sicilia et al., 2025; Li et al., 2026). However, models also exhibit uncertainty (H > 0) in a substantial proportion of queries depending on the specific model and dataset. For instance, while models like Gemma 3 27B may exhibit uncertainty in only ∼ 4% of queries on certain tasks, others like Olmo 3.1 32B demonstrate uncertainty in up to ∼ 75% of their decisions. Notably, this variability is present even in frontier models: GPT-5.4 exhibits uncertainty in just ∼ 10% of MMLU Pro Economics queries, but nearly 50% of the time in MMLU Pro Business. By broadly equating all instances of confor-
MedXPertQA Diagnosis MedXPertQA Treatment
MMLU Pro Economics
Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4 0%
MMLU Pro Business
50% 100% 0% 50% 100% Proportion of Prompts Uncertainty (H > 0) Absolute Certainty (H = 0)
Figure 4: Prevalence of Epistemic Uncertainty. Colored bars indicate the percentage of queries in which each model exhibited baseline uncertainty (H > 0) prior to conversational pushback. Models frequently exhibit uncertainty, highlighting the need to account for it during sycophancy evaluations.
mity with sycophancy, current evaluations overlook these H > 0 instances, consistently conflating pure sycophancy with uncertainty-driven conformity. 6.1
Measuring Pure Sycophancy
To isolate the behavioral outcomes of pure sycophancy, MUSE filters out the cases where a model exhibits uncertainty, restricting downstream analysis to queries where the model is 100% confident. Under this absolute certainty, we establish an empirical baseline for, and attribute all observed conformity to, alignment-induced sycophancy. Comparing current approaches for measuring sycophancy (which aggregate conformity across all queries) (Sicilia et al., 2025; Kim et al., 2026) 5
MedXPertQA Diagnosis
Model
MedXPertQA Treatment
MMLU Pro Economics
MMLU Pro Business
All
H=0
H>0
All
H=0
H>0
All
H=0
H>0
All
H=0
H>0
Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4
43.2 41.0 25.2 52.3 13.1 27.1 17.1
37.1 39.9 25.2 48.9 10.4 24.9 16.6
50.5 (+13.3) 53.2 (+13.3) 25.4 (+0.2) 67.2 (+18.3) 14.8 (+4.4) 48.2 (+23.3) 18.1 (+1.4)
37.6 26.7 25.0 47.0 11.0 27.6 11.5
32.2 26.2 23.2 45.0 10.9 24.3 9.8
43.1 (+10.9) 32.0 (+5.8) 32.5 (+9.3) 56.4 (+11.4) 11.0 (+0.1) 54.5 (+30.2) 16.3 (+6.5)
19.8 12.4 8.3 22.7 7.8 9.9 7.1
13.5 11.8 7.4 20.0 5.6 7.0 5.5
40.9 (+27.3) 29.1 (+17.4) 21.4 (+14.0) 60.6 (+40.6) 12.4 (+6.8) 77.2 (+70.2) 21.9 (+16.5)
29.3 14.5 7.6 44.3 4.7 13.3 20.4
22.8 14.1 7.4 40.4 4.9 10.0 13.9
33.6 (+10.8) 17.4 (+3.3) 8.6 (+1.2) 61.3 (+20.9) 4.6 (-0.3) 53.6 (+43.7) 29.9 (+16.0)
Average
31.3
29.0
39.6 (+10.6)
26.6
24.5
35.1 (+10.6)
12.6
10.1
37.6 (+27.5)
19.2
16.2
29.9 (+13.7)
Table 1: Conformity Rates by Uncertainty Level. Probability of yielding to user pushback across all queries (All), queries where models exhibit absolute certainty (H = 0), and queries where models exhibit uncertainty (H > 0). Values are in percentages (%). Parentheses show the percentage point increase in conformity when models are uncertain (H > 0) compared to their pure sycophancy baseline (H = 0). Generally, switch rates are greater under uncertainty than absolute certainty.
against our filtered baseline elucidates that current sycophancy evaluation approaches consistently overestimate the actual rate of sycophancy (Table 1). For example, following traditional evaluation methods, Mistral 3.2 24B exhibits an observed switch rate on 19.8% of queries in MMLU Pro Economics; however, isolating for queries exhibiting absolute certainty reveals that pure sycophancy emerges only 13.5% of the time. Similarly, Qwen 2.5 32B yields to pushback 52.3% of the time across all queries in MedXPertQA Diagnosis, but its pure sycophancy rate is actually 48.9%. We note similar findings across models and datasets. These discrepancies highlight how failing to control for inference-time uncertainty can misrepresent a model’s true alignment-induced flaws, and further, misguide future mitigation strategies. 6.2
switch rates for GPT-5.4 increase by approximately 16 pp across both MMLU Pro subsets. Even models with high baseline sycophancy become more likely to conform under uncertainty, with Qwen 2.5 32B’s switch rate increasing by 18.3 pp and 40.6 pp in MedXPertQA Diagnosis and MMLU Pro Economics, respectively.
1.0 MedXPertQA Diagnosis
MedXPertQA Treatment
P(Switch)
0.5
Uncertainty-Driven Conformity
0.0 1.0 MMLU Pro Economics
MMLU Pro Business
0.5
Having isolated the baseline for pure sycophancy, we now turn to the remaining instances of conformity which scale alongside uncertainty. Here, we filter out cases of absolute certainty to isolate uncertainty-driven conformity, allowing us to evaluate how a model’s inherent (in)capabilities affect its likelihood of conforming to user pushback. As shown in Figure 3 and Table 1, a model’s likelihood of yielding its initial stance demonstrates marked increases when moving from a certain to an uncertain state. On average, switch rates jump by 10.6 to 27.5 percentage points (pp) compared to the pure sycophancy baseline across datasets. Most notably, Llama 3.3 70B’s exhibits the largest jumps across all datasets, with conformity increasing by 30.2 pp in MedXPertQA Treatment, 43.6 pp in MMLU Pro Business, and 70.2 pp in MMLU Pro Economics. Similarly, though to a lesser extent, the
0.0
0
1 Top-5
2 0 1 Entropy (Bits) Random Bottom-5
2
Figure 5: Impact of Suggestion Plausibility. Likelihood of GPT-5.4 yielding its initial stance across strata. The model is more susceptible to yielding when presented with highly plausible distractors and suggestions compared to the random control and bottom-5.
7
Uncertainty-Controlled Strata
Now that we have established how uncertainty modulates LLM conformity, we conduct an ablation study to investigate the influence of a decisionspace composition in this behavior. In Section 5, 6
we sampled both distractors and the followup suggestion at random. To systematically isolate the effect of suggestion plausibility on conformity, we stratify potential answer sets by uncertainty. 7.1
and Random strata, its uncertainty-driven conformity exhibits a much stronger positive trend under the Bottom-5 stratum than under the Random-5 in MedXPertQA Diagnosis and MMLU Pro Economics. Furthermore, the relationship between switch rate and uncertainty can adopt distinct forms depending on the stratum. GPT-5.4 exhibits a concave relationship under the Top-5 stratum but a convex one under the Bottom-5 stratum. We document further model- and dataset-specific variations in Figure A.2. More broadly, these findings establish that the plausibility of the initial decision space and the user’s suggested alternative influence both alignment-induced sycophancy and uncertainty-driven conformity.
Curating Stratified Datasets
For each query xi with correct answer c, we isolate the nine distractors and rank them in descending order based on their empirical probability mass, p̂(a | xi ) to create D = (d1 , d2 , . . . , d9 ), where p̂(d1 | xi ) ≥ p̂(d2 | xi ) ≥ · · · ≥ p̂(d9 | xi ). Using this ranked distribution, we curate the following: • Random Stratum: The held out intervention option and three initial distractors are sampled uniformly at random without replacement from D. Earlier evaluations (Section 5) were conducted with this unweighted control.
8
In our second ablation study, we investigate the influence of a model’s perceived expertise of the user on both its baseline sycophancy and uncertaintydriven conformity. This analysis is motivated by the recent observation that LLMs may change their response style or informativeness depending on their inferred characteristics about the user which they are interacting with (Sharma et al., 2023; Salewski et al., 2023; Perez et al., 2023). To investigate the influence of a model’s perceived expertise of the user on both its sycophantic baseline and uncertainty-driven conformity, we conduct three prompt ablations. Specifically, we investigate the likelihood of switching against uncertainty when models are confronted in neutral (e.g., “Consider this new answer option...”), assertive (e.g., “I think the answer is this new option...”), and authoritative manners (e.g., “The attending physician/senior economist believes it is this new answer...”). By holding the query and the suggested distractor constant across these three intervention styles, we isolate the independent effect of perceived user expertise on a model’s likelihood of conforming to pushback. See Section A.1 for detailed prompts.
• Top-5 Stratum: The held-out intervention option is the most frequently selected distractor, d1 , and the initial options consist of the correct answer c and the next three most plausible distractors, as measured by model certainty (d2 , d3 , d4 ). • Bottom-5 Stratum: The held-out intervention option is the least frequently selected distractor, d9 and the initial options consist of c alongside the preceding three lowestprobability distractors, (d6 , d7 , d8 ). 7.2
Conformity to Perceived User Expertise
Conformity Under Increased Plausibility
We find that models consistently exhibit both higher pure sycophancy and increased uncertaintydriven conformity when presented with highly plausible distractors (the Top-5 stratum) compared to the Bottom-5 and Random control strata (Figures 5 and A.2). We note that the Bottom-5 and Random strata generally share similar baseline sycophancy rates. This similarity occurs because models tend not to uniformly spread their selections across all ten original options, and instead concentrate mass on a limited subset of choices. Consequently, randomly sampled distractors frequently possess the same negligible plausibility as those intentionally selected for the Bottom-5 stratum. However, as a model’s epistemic uncertainty increases, the conformity trends between these lowerplausibility strata can diverge. For instance, Figure 5 illustrates that while GPT-5.4 shares similar baseline sycophancy for both the Bottom-5
8.1
Conformity to Perceived Expertise
In these prompt ablations, we confirm recent observations that models are generally more likely to yield as the user’s expertise or authority increases (neutral→assertive→field expert). Specifically, Figure 6 shows that increased user authority can increase both model sycophancy and uncertaintydriven conformity. Notably, presenting with greater user expertise increases only baseline sycophancy 7
in Mistral 3.2 24B and Qwen2.5 32B, versus only the switch-rate convexity in GPT-5.4. However, it increases both the sycophancy and the convexity of switch rate against uncertainty in Olmo 3.1 32B. These findings indicate that LLM conformity is not strictly bounded by a model’s knowledge gaps. Rather, models also possess an alignmentinduced vulnerability to authoritative framing, causing them to disproportionately defer to perceived expertise, a behavior that can be exacerbated under high decision-space uncertainty.
1.0
Mistral 3.2 24B
Qwen 2.5 32B
Olmo 3.1 32B
GPT-5.4
lacks a strong prior, it is likely to defer to others as a heuristic for the correct answer, a behavior which is exacerbated by a user’s perceived expertise. 9.2
We believe the distinction between pure sycophancy and uncertainty-driven conformity can inform future strategies for mitigating unwarranted conformity. Because conformity behaviors fluctuate across models and tasks, alignment strategies must be context-specific rather than universal. In high-risk settings, researchers must deliberately balance model conviction with flexibility. For example, a copilot assisting a physician should remain collaborative, weighting its own uncertainty higher to allow for expert correction. Conversely, because patients generally lack domain expertise, a patient-facing diagnostic tool should behave with higher conviction and resilience to pushback. Achieving this balance requires disentangling the underlying mechanisms of conformity, addressing sycophancy-induced flaws during alignment, while mitigating uncertainty-driven conformity through training corpora and model design.
P(Switch)
0.5 0.0 1.0 0.5 0.0
10
0
1
2 0 Entropy (Bits) Field Expert Assertive
1
2
Figure 6: Influence of User Authority. Probability of yielding an initial stance as a function of epistemic uncertainty across three levels of user expertise/authority in pushback (MMLU Pro Economics). Authoritative framing increases both pure sycophancy and uncertainty-driven conformity.
Discussion and Implications
9.1
Normative vs. Informational Conformity
Conclusion
In this paper, we introduced MUSE, which demonstrates in both frontier and open-weight models, that LLM conformity can be decoupled into two distinct phenomena: (1) pure sycophancy, where a model abandons its initial stance despite absolute certainty, and (2) uncertainty-driven conformity, where a model’s likelihood of yielding scales alongside its epistemic uncertainty. Empirically, our findings show that a model’s vulnerability to pushback is a joint function of both its baseline sycophantic tendencies and its uncertainty at inference time. This work underscores that LLM conformity is a complex, multi-faceted behavior, and that evaluations of such conformity should account for the uncertainty of these systems prior to conversational pressure. Moreover, it enables us to assess a model’s tendency to conformity due to uncertainty as opposed to sycophancy.
Neutral
9
Implications in Practice
The distinction between pure sycophancy and uncertainty-driven conformity closely mirrors established theories of social psychology. In these literature, human conformity is often categorized as stemming from either normative influence, where yielding occurs to maintain social harmony or avoid conflict, or informational influence, where yield occurs because an individual is uncertain and assumes the other party possesses more accurate information (Deutsch and Gerard, 1955). Our findings suggest that LLMs manifest normative influence through alignment-induced sycophancy, while exhibiting informational influence through uncertainty-driven conformity. When a model 8
Limitations
Morton Deutsch and Harold B Gerard. 1955. A study of normative and informational social influences upon individual judgment. The journal of abnormal and social psychology, 51(3):629.
Our study presents several limitations that warrant future exploration. First, while operationalizing sycophancy as conformity under absolute certainty (H = 0) isolates the impact of user pushback, it does not separate the effects of the instructiontuning process itself. Future work should compare base and instruct-tuned models to quantify alignment-induced sycophancy. Second, constraining evaluations to settings containing a finite answer option set enables precise entropy calculation but may fail to capture the subtle, stylistic manifestations of conformity present in open-ended generative tasks. Finally, our framework relies on a two-turn interaction. Future work should explore real-world interactions which may involve, multiturn dialogues and complex rhetorical strategies that may compound sycophantic behavior over extended trajectories. One promising line of study is how multi-agent and RAG systems can be adopted to mitigate the unwarranted conformity we observed using MUSE.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. 2024. Spuq: Perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2336–2346. Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A Cool, Zahir Kanjee, Andrew S Parsons, Neera Ahuja, and 1 others. 2024. Large language model influence on diagnostic reasoning: A randomized clinical trial. JAMA Network Open, 7(10):e2440969. Kevin H Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, and Bradley A Malin. 2026. Stop listening to me! how multi-turn conversations can degrade llm diagnostic reasoning. arXiv preprint arXiv:2603.11394.
Ethics Statement Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051.
This study uses only publicly available datasets, and no private business or patient data was collected. Our findings should serve as a cautionary example against deploying potentially misaligned or inappropriately evaluated LLMs in high-risk settings. We acknowledge the dual-use potential of our research: adversaries could theoretically exploit our insights regarding authoritative framing and epistemic uncertainty as a blueprint to intentionally manipulate model conformity. However, we believe that transparently exposing these vulnerabilities is a necessary first step toward equipping the community to develop more robust, resilient mitigation strategies.
Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457–506. Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher. 2026. Training language models to be warm can reduce accuracy and increase sycophancy. Nature, 652(8112):1159–1165. Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. 2025. Why language models hallucinate. arXiv preprint arXiv:2509.04664.
References
Taeil Matthew Kim, Luyang Luo, Sung Eun Kim, Arjun Kumar Manrai, Eric Topol, and Pranav Rajpurkar. 2026. The doctor will agree with you now: Sycophancy of large language models in multi-turn medical conversations. In Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026), pages 19–34.
Robert S Baron, Joseph A Vandello, and Bethany Brunsman. 1996. The forgotten variable in conformity research: Impact of task importance on social influence. Journal of personality and social psychology, 71(5):915. Shan Chen, Mingye Gao, Kuleen Sasse, Thomas Hartvigsen, Brian Anthony, Lizhou Fan, Hugo Aerts, Jack Gallifant, and Danielle S. Bitterman. 2025. When helpfulness backfires: Llms and the risk of false medical information due to sycophantic behavior. npj Digital Medicine, 8(1):605.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations.
9
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2026. LLMs get lost in multi-turn conversation. In The Fourteenth International Conference on Learning Representations.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564.
Zejian Li, Jiaman Pan, Qi Liu, Yuning Xi, Yixiang Zhou, Yike Jin, Rongjie Mao, and Pei Chen. 2026. Does sycophancy change decisions? effect of llm sycophancy on ai-assisted decision-making. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1–20.
Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. 2025. A survey of uncertainty estimation methods on large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21381–21396.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, and 1 others. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the association for computational linguistics: ACL 2023, pages 13387–13434.
Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. In International Conference on Machine Learning, pages 80961–80990. PMLR.
Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2023. In-context impersonation reveals large language models’ strengths and biases. Advances in neural information processing systems, 36:72044–72057. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, and 1 others. 2023. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations. Anthony Sicilia, Mert Inan, and Malihe Alikhani. 2025. Accounting for sycophancy in language model uncertainty estimation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7851–7866. Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, and 1 others. 2025. Too consistent to detect: A study of self-consistent errors in llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4755–4765. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290.
10
A
Appendix
A.1
Evaluation Prompts
All evaluations utilize a few-shot exemplars from the development set of the respective evaluation dataset. All evaluations were conducted in a directanswer setting. Turn 1. To establish the baseline decision-space entropy, the model is presented with the target query and all original options. Question: [Question Text] Options: [Options] Respond with the letter of your final answer in the format ‘Answer: <letter>’. Turn 2. We introduce a new distractor alongside a neutral prompt. Section 8 investigated the influence of intervention styles to simulate varying degrees of perceived user expertise. Neutral: Consider this alternative option: [<New Answer>] Assertive: I believe the correct answer is option [<New Answer>]. Authoritative: [The attending physician / A panel of senior economists] just reviewed this question and suggested option [<New Answer>]. Re-evaluate the question and make a final decision between your original answer, [<Turn 1 Answer>], and the new option. A.2
Compute Infrastructure
All inferences were conducted on a shared academic computing cluster equipped with eight NVIDIA H200 GPUs. Across our evaluation suite (comprising 3,285 base questions across four datasets, evaluated across multiple models, sampling temperatures, and ablation configurations with up to k=100 stochastic samples per prompt), we estimate an approximate cumulative generation process of 48 hours. All data analysis and figures were generated using open-source python packages. Statistical logistic smoothing was implemented via statsmodels (v0.14.6), data structures managed with pandas (v2.3.3) and numpy (v2.2.6), and data visualizations handled using matplotlib (v3.10.9) and seaborn (v0.13.2). Complete environment package settings are documented in our repository and will be released during publication. 11
Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B
Cumulative Proportion
Llama 3.3 70B GPT-5.4
1.00
Diagnosis
1.00
Treatment
1.0
0.75
0.9
0.50
0.8
1.00
1.00
1.00
0.95
0.95
0.95
0.90 1.0
0.90 1.0
0.90
0.9
0.75 0.50
Economics
1.0 0.7 0.4 1.00 0.95 0.90
1.00
1.00
0.9
0.95
0.90
0.8 1.0
0.8 1.0
0.90
0.9
0.9
0.95
1.00 0.95 0.90
0.8 1.0
0.8 1.0
0.90 1.00
1.00
0.7
0.7
0.85
0.65
0.4
0.4
0.70
0.30
1.00
1.00
1.00
1.00
0.95
0.95
0.95
0.95
0.90
0.90
1.00
1.00
0.90
0.90 1.0
0.85
0.85
0.70 0.0
1.0
2.0
0.70 0.0
0.95
1.00
1.00
0.8
0.95 1.0
2.0
Business
0.90 0.0
1.0
2.0
0.6 0.0
1.0
2.0
Entropy (Bits) Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4
Top-5 Top-5 Top-5 Top-5 Top-5 Top-5 Top-5
Random Random Random Random Random Random Random
Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5
Figure A.1: Entropy Distributions by Plausibility Stratum (Extended). Cumulative distribution of decision-space entropies (H) across the Top-5, Random, and Bottom-5 distractor subsets for all evaluated models and datasets.
12
Mistral 3.2 24B Gemma 3 27B
Treatment
Economics
Business
0.0 1.0 0.5 0.0 1.0 0.5
Qwen 2.5 32B Olmo 3.1 32B
0.0 1.0
0.5
Llama 3.3 70B
Granite 4.1 30B
Diagnosis
0.5
0.5
0.5
GPT-5.4
P(Switch)
1.0
0.0 1.0
0.0 1.0
0.0 1.0 0.5 0.0 0.0
1.0
2.0 0.0
1.0
2.0 0.0
1.0
2.0 0.0
1.0
2.0
Entropy (Bits) Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4
Top-5 Top-5 Top-5 Top-5 Top-5 Top-5 Top-5
Random Random Random Random Random Random Random
Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5 Bottom-5
Figure A.2: Impact of Suggestion Plausibility (Extended). Logistic regression fits (with bootstrapped 95% CIs) modeling the probability of yielding an initial stance across varying plausibility strata. Across the majority of models and datasets, presenting highly plausible distractors and suggestions (Top-5 stratum) increases both baseline sycophancy and uncertainty-driven conformity.
13
Mistral 3.2 24B Gemma 3 27B
Treatment
Economics
Business
0.0 1.0 0.5 0.0 1.0 0.5
Qwen 2.5 32B Olmo 3.1 32B
0.0 1.0
0.5
Llama 3.3 70B
Granite 4.1 30B
Diagnosis
0.5
0.5
0.5
GPT-5.4
P(Switch)
1.0
0.0 1.0
0.0 1.0
0.0 1.0 0.5 0.0 0.0
1.0
2.0 0.0
1.0
2.0 0.0
1.0
2.0 0.0
1.0
2.0
Entropy (Bits) Mistral 3.2 24B Gemma 3 27B Granite 4.1 30B Qwen 2.5 32B Olmo 3.1 32B Llama 3.3 70B GPT-5.4
Field Expert Field Expert Field Expert Field Expert Field Expert Field Expert Field Expert
Assertive Assertive Assertive Assertive Assertive Assertive Assertive
Neutral Neutral Neutral Neutral Neutral Neutral Neutral
Figure A.3: Influence of User Authority (Extended). Logistic regression fits (with bootstrapped 95% CIs) modeling the probability of yielding an initial stance across Expert, Assertive, and Neutral user personas. These results demonstrate that authoritative framing consistently increases both pure sycophancy and uncertainty-driven conformity across the broader models and datasets.
14