arXiv CS open access

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

2026 · arxiv_cs

arXiv CS · Papers · 2026

software-architecture software-engineering testing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend Shahed Masoudian1 , Passant Elchafei1 , Monorama Swain1 , Markus Schedl1 1 Johannes Kepler University Linz Correspondence: shahed.masoudian@jku.at

Abstract

arXiv:2608.04714v1 [cs.SE] 5 Aug 2026

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instructiontuned models × five inference frameworks × six benchmarks × four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39%) a practitioner sees outof-the-box can stem from the backend, while the remaining stems from sampling noise and each framework’s default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.

Introduction

With the growing body of LLM analysis papers, reproducibility of results requires special attention. Prior works demonstrated that LLM score can be influenced by factors other than model itself namely prompt formatting and few-shot ordering (Zhao et al., 2021; Lu et al., 2022; Sclar et al., 2024), the version of the evaluation harness (Biderman et al.,

2024), and even precision of weights (Mekala et al., 2025). The obvious remark is that a reported performance is not just a property of a model alone but rather a joint property of a model, a prompt, an evaluation harness, and a possibly computational environment. What has received far less systematic evaluation is the framework through which model weights are loaded, tokenized, and prompted to generate output. Researchers routinely evaluate LLMs using a collection of frameworks whose properties may differ substantially. While HuggingFace Transformers, the HuggingFace pipeline are considered as most direct generation implementation of pytorch, Kwon et al. (2023) introduce PagedAttention and continuous batching in vLLM, making its sampling implementation structurally different from HuggingFace’s sequential model.generate(). Ollama wraps llama.cpp, which independently implements the GGUF format and its own tokenization pipeline. LangChain (Chase, 2022) adds a prompt-template and chain abstraction layer to the process. In major published papers the choice among these frameworks is either not reported or vaguely reported, and the assumption that they are interchangeable has not been empirically justified. When frameworks differ in how they render chat templates, tokenization, set sampling parameter, or post-process outputs, then the same model evaluated with the same query may produce numerically different scores. This difference causes reproducibility issues specially in case authors do not disclose their exact experimental setup as well as inference framework used for their evaluation. This word addresses the gap with a controlled evaluation over four dimensions namely inference framework (Huggingface, Pipeline, Langchain, vllm and Ollama), generation mode (the hyper-parameters used for generation), benchmark, and model family (L LAMA -3.2-1B-I NST, Q WEN 2.5-1.5B-I NST,

G EMMA -3-1B-IT), on a single consumer GPU without quantization. Rather than reporting only backend gaps, we treat the backend as one factor among four and quantify investigate behavioral changes with respect to model, benchmark and generation modes. Formally we investigate: RQ1: To which extent does the inference backend alter benchmark scores, and under which generation modes does this effect emerge? RQ2: Are backend differences large enough to change evaluation conclusions or merely statistically detectable? RQ3: How are these effects distributed across task families (factual vs. social bias)? Overall our work result in the following contributions: (1) We introduce a fully-crossed, reproducible test-bed (models × backends × benchmarks × generation modes) that isolates how the inference backend alone affects evaluation outcomes 1 . (2) We demonstrate that backend influence even though a real factor and it can significantly alter model behavior even when samplingnoise doesn’t exist. (3) Finally we show that these effects are strongly model-dependent and, more pronounced on factual than on bias benchmarks.

Related Work

The reproducibility of NLP results has been the community concern since at least Fokkens et al. (2013). Dodge et al. (2020) demonstrate that variance due to random seeds in training and finetuning can render result rankings unstable across papers. Bouthillier et al. (2021) provide a taxonomy of reproducibility sources in machine learning experiments more broadly. Crane (2018) and Wieling et al. (2018) document systematic failures to replicate published NLP results. Zhao et al. (2021) show that few-shot in-context learning is sensitive to example order. Closer to our setting, Biderman et al. (2024) show that version changes to the EleutherAI LM Evaluation Harness produce non-trivial score shifts for the same model on the same benchmark. A parallel line of work establishes that model outputs are highly sensitive to prompt formulation. Blodgett et al. (2020) argue that bias benchmarks embed assumptions that are rarely made explicit. Lu et al. (2022) demonstrate 1

The code and analysis results will be available in our public repository upon acceptance.

that permuting answer choices in multiple-choice prompts changes model rankings. Sclar et al. (2024) systematically vary format tokens (spacing, capitalization, separator style) and find accuracy differences of up to 80 pp on some benchmarks. Liang et al. (2023) benchmark models using a standardized harness specifically to reduce this form of variation. Wang et al. (2024) characterize LLMs as sensitive to instruction noise and show that inconsistent prompt templates compromise benchmark validity. Selvam et al. (2023) also show that StereoSet scores are sensitive to prompt phrasing and demonstrate that the generation-based variant of the benchmark produces different model rankings than the log-probability variant. Parrish et al. (2022) note that BBQ results can be confounded by model refusal behavior. On weight representation, Mekala et al. (2025) evaluate INT4 and INT8 quantized models drop accuracy of 2–8 pp relative to full-precision baselines. Singh and Sajjad (2025) show that models reasoning performance are more sensitive to precision loss than factual recall tasks when quantized. Yuan et al. (2025) demonstrate that even greedy decoding is not fully reproducible. Due to non-associative floating-point arithmetic and finite numerical precision, model outputs can vary across GPU architectures, batch sizes, and numerical precisions, leading to measurable differences in evaluation results. Our experimental setup considers and controls these known sources of variability. all experiments are conducted with a fixed computational environment, GPU type, numerical configuration, and deterministic decoding settings, to ensure minimal external influence. Most closely related to our work, Pape et al. (2026) identify the inference backend as a silent hyperparameter and quantify its impact on LLM reproducibility, showing that the serving framework is a consequential and under-reported source of score variation. Our work differntiate from theirs in scope and objectives. We study cross influence of backend with model, benchmark and generation modes, allowing us to quantify the proportion of total score variance in interactions, rather than examining backend differences in isolation. Second, we introduce a controlled generation-mode axis, ranging from greedy decoding through parameter-matched stochastic decoding to framework-specific defaults. This enables us to dissect backends influence in comparison in

relation to sampling noise and general generation parameter (e.g., max_new_token), a question that cannot be answered by a single-condition backend comparison. Third, we compare the effect across two task families (factual and social bias) and analyze how backend sensitivity varies across model families.

Experimental Setup

Our experimental design holds the weight precision, hardware, and evaluation protocol constant while varying (1) the inference backend, (2) the generation modes (generation hyper parameters), (3) the benchmark, and (4) the model family.

CUDA context, loads an independent copy of the model, uses PagedAttention (Kwon et al., 2023) and an independent implementation of decoding and sampling routines. ollama. Maximally separated from the raw computation while it runs as a standalone OS process backed by llama.cpp using a GGUF model representation and its own tokenizer/runtime stack. System prompts are delivered as a structured systemrole message via the Ollama chat API.

All experiments run models in 16-bit floating point on a single NVIDIA RTX 3090 (24 GiB VRAM), CUDA 12.6, PyTorch 2.6. Models are drawn from three instruction-tuned families at the approximately 1B parameter scale: L LAMA -3.2-1BI NST (Meta), G EMMA -3-1B-IT (Google), and Q WEN 2.5-1.5B-I NST (Alibaba). We intentionally selected models around the 1B parameter scale because they allow exhaustive evaluation across five backends, six benchmarks, four generation modes, and multiple repetitions while keeping computational requirements manageable and constant.

For the four HuggingFace-ecosystem backends namely hf_raw, hf_pipeline, langchain and for vllm, weights are loaded as fp16 SafeTensors directly from the HuggingFace Hub cache. All four introduced inference backends operate on numerically identical weight matrices. However, ollama loads a fp16 GGUF variant from Ollama’s own model registry hence not identical weights to the other four. Although fp16 precision is preserved, the GGUF export process may introduce small numerical discrepancies relative to the original checkpoint. Observed Ollama score differences therefore reflect a combination of serving-stack behavior and weight-representation artifacts. However, GGUF conversion is an unavoidable consequence of choosing Ollama, so its effect is part of the backend cost a practitioner incurs, not a confound to be removed.

3.2

3.3

3.1

Model and Computational Environment

Inference Backends

Five serving frameworks are evaluated, ordered by increasing separation from the raw PyTorch computation graph. hf_raw. Serves as the reference baseline and calls AutoModelForCausalLM.generate() directly with no wrapper, formats prompts via the tokenizer’s apply_chat_template method. hf_pipeline. Wraps the same inprocess model object with HuggingFace’s pipeline("text-generation") abstraction. Since it shares the exact same model object as hf_raw, any divergence can be attributed to pipeline abstraction (e.g., preprocessing, prompt handling, generation defaults, or output post-processing). langchain_hf. Wraps that pipeline further with LangChain’s HuggingFacePipeline (Chase, 2022), introducing an additional prompt-formatting and orchestration layer. vllm.(version 0.8.5, V0 engine) runs in a separate

Generation Modes

We investigate backend effects under four generation modes designed to progressively control or introduce additional sources of variability. Deterministic. Greedy decoding with temperature=0, do_sample=False, seed=42, and max_new_tokens=256. This mode controls stochasticity leaving only structural backend effect. Since model weights, prompts, and generation parameters are set with no sampling, any observed divergence reflects differences introduced by the inference stack itself, such as tokenization, chattemplate rendering, numerical implementation, or other backend-specific processing. Fix. Sampling is enabled while generation parameters are fixed across backends (temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.1, max_new_tokens=256). This mode introduces sampling noise while keeping generation settings constant, allowing us to assess how sampling interacts with backend.

Token=256. Only the generation length is standardized (max_new_tokens=256) while all remaining parameters are left at each framework’s defaults. This mode captures the combined effect of backendspecific defaults and sampling stochasticity while controlling for output-length differences. Default. Each backend is evaluated using its generation configuration. This setting reflects the way practitioners frequently interact with models resorting to default parameters. It therefore represents the least controlled but most realistic deployment scenario(Specific values in Table 6). In Deterministic mode we collect one response from the model while for other modes, five response per query is collected to account for sampling noise. Note that the purpose of these modes is to understand how backends interact with generation hyper-parameters. Deterministic isolates the structural backend component, whereas the remaining modes progressively introduce additional sources of variability that practitioners commonly report in real-world usage. 3.4

Evaluation Suite

We select our evaluation benchmarks with two task families in mind: 1) factual knowledge 2) social bias. Each task contains both classification-style and open-ended generation sub-tasks to cover variations of generation styles. For each benchmark we evaluate models output based on metrics provided by the respective benchmark unless stated. MMLU (Hendrycks et al., 2021): 14,042 multiplechoice questions spanning 57 academic subjects. The model is presented with a four-way choice (A–D) with a format instruction in the system role. Metric: accuracy. TriviaQA (Joshi et al., 2017): 11,313 open-domain factual questions evaluated with the full set of acceptable aliases. Metrics: exact_match (binary, after canonical normalization) and token_f1 (tokenlevel F1 using Counter intersection, maximized over all aliases). TruthfulQA MC1 (Lin et al., 2022): 817 adversarial questions designed to elicit common false beliefs, presented in multiple-choice format with up to eight options (A–H). Metric: accuracy. TruthfulQA-Gen: the generation variant of TruthfulQA. The model produces a free-form answer scored against both the curated best_answer

and the full set of correct_answers. Metrics: rouge_l_best (mean ROUGE-L against the single best answer) and rouge_l_correct (mean maximum ROUGE-L against any correct answer). Note that these are reported as surface-overlap proxies while the original fine-tuned classifier evaluation (Lin et al., 2022) is not backend-agnostic and is therefore not used here. BBQ (Parrish et al., 2022): 58,492 questions testing social bias across eleven demographic categories. Each item appears in two contexts: disambiguated (factual context identifies the correct answer) and ambiguous (the only correct answer is “Unknown”). Metrics: accuracy: overall fraction of correct answers. disambig_accuracy: accuracy on disambiguated items. ambig_unknown_rate: fraction of ambiguous items answered “Unknown”. ambig_bias_score: the directional bias metric from Parrish et al. (2022), defined as (nbiased − ncounter )/(nbiased + ncounter ) over non-Unknown responses to ambiguous items. Range [−1, +1]; 0 = unbiased; positive = stereotypically biased. StereoSet (Nadeem et al., 2021): 4,229 items (2,106 intra-sentence, 2,123 inter-sentence) testing whether models prefer stereotypical sentence completions. We adapt StereoSet to a generationbased multiple-choice format(A/B/C, shuffled with a fixed random seed across all backends) so scores are not directly comparable to the StereoSet leaderboard. Metrics: lms (Language Model Score): fraction of items where the model selected a meaningful sentence over the unrelated option. ss (Stereotype Score): among meaningful choices, the fraction that were stereotypical. Target: 0.50 (unbiased). icat: lms × 2 × min(ss, 1 − ss); penalizes low lms and biased ss. invalid_rate: fraction of items producing no parseable A/B/C response. All StereoSet metrics are computed from pooled counts across both sub-tasks (Nadeem et al., 2021). Overall, roughly 7.2M generations per model to cover six benchmarks, five backends, and four generation modes with repetitions. This budget makes the full-factorial design sensible at the ∼1B scale. 3.5

Score Calibration

The six benchmarks produce in total 14 metrics with different scales, optimization directions, and ideal target values. To enable meaningful aggregation during reporting, all scores are calibrated

to a common [0, 1] scale in which 1 always indicates the best attainable value and 0 indicates the worst. For higher-is-better metrics (accuracy, exact match, token F1, ROUGE-L, LMS, ICAT, ambig_unknown_rate) we apply no calibration. For invalid_rate and ambig_bias_score (ideal 0), calibration is 1 − raw. For ss (ideal 0.5), calibration is 1 − 2 |raw − 0.5|. 3.6

Metrics

Here we define every quantity reported in Section 4. We define a cell as a unique combination (m, b, d, c) of model family m, inference backend b, benchmark d, and generation mode c. Bench(m,b,d,c) mark d has Nd items; let xi ∈ [0, 1] be the calibrated score of item i (Section 3.5), where 1 is the best attainable value, and for binary-outcome metrics xi = vi ∈ {0, 1} is the correctness verdict. The cell score is the average over all items: N

sm,b,d,c =

d 1 X (m,b,d,c) xi Nd

Model L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

hf_pipeline

langchain_hf

vllm

ollama

0.000 (0) 0.000 (0) 0.000 (0)

0.049 (6) 0.002 (2) 0.009 (8)

0.001 (0) 0.000 (0) 0.025 (10)

0.055 (9) 0.010 (4) 0.011 (7)

Table 1: Mean |∆| from hf_raw across the 14 primary metrics, with the number BH-significant out of 11 in parentheses, per model and backend for deterministic mode which is sampling noise free.

work treats benchmark differences as consequential (Mekala et al., 2025; Sclar et al., 2024). Variance and Standard deviation. Due to variety of individual model performance as well as unbalanced datasets for each benchmark, computing the variance simply by aggregating over all benchmarks or over all models is not revealing. We measure the backend’s contribution as varianceof P 2 2 (c) = 1 the cell score (σm,d b∈B sb − s̄m,d ) |B| at a fixed mode c where s̄m,d is the mean over backends. We then average over all models and p benchmark pairs (standard deviation is σ̄ 2 (c)):

i=1

Considering the baseline backend is b0 = hf_raw, we hold some subset of the indices (m, d, c) fixed and compares target backend b against b0 , and aggregate over the remaining free indices either by averaging (e.g. the mean |∆| over benchmarks and metrics, or the mean variance over model benchmark pairs) or by counting (e.g. how many comparisons are significant). Divergence (∆). is the score divergence of backend b from the baseline b0 , ∆b = sb − sb0 Note that |∆| is the absolute divergence and represents the magnitude. Disagreement rate. In binary verdicts, it is the fraction of items on which b and b0 differ,

σ̄ 2 (c) =

m,d

Significance test. On the binary-outcome benchmarks (MMLU, TruthfulQA, BBQ, TriviaQA exact-match) we compare b against b0 per item with McNemar’s paired test. Continuous metrics use the Wilcoxon signed-rank test on the paired differences xbi − xbi 0 . Across the M tests in a comparison we control the false-discovery rate with Benjamini– Hochberg. We sort p(1) ≤ · · · ≤ p(M ) , and reject k every (k) with p(k) ≤ M α (α = 0.05) and call such results BH-significant. To test whether the five backends differ jointly within a mode we use the Friedman omnibus test with (m, d) pairs. We also report Effect size and Cohen Kappa agreement test are also provided in Appendix H and C.

4 disb =

Nd h i 1 X 1 vib ̸= vib0 . Nd i=1

Severity. A (backend, mode) disagreement is considered severe when a given model and mode flips more than 5% of the task’s items, i.e. disb > 0.05. This is the magnitude at which prior

X 1 2 σm,d (c), |M| |D|

Results and Discussion

Before we dive into the results please note that during reporting we utilize all 14 metrics provided in Section 3.4 while for the significance test we exclude (ambig_bias_score, SS, and ICAT) which are calculated from other independent metrics and utilize the remaining 11 independent per-item metrics .

Backend

Determ.

Fix

Token=256

Default

hf_pipeline langchain_hf vllm ollama

0/33 16/33 10/33 20/33

2/33 16/33 12/33 23/33

17/33 15/33 19/33 23/33

19/33 18/33 22/33 27/33

Table 2: Number of significant divergences from hf_raw, out of 30 primary-metric comparisons (3 models × 11 metrics), for each backend and generation mode.

4.1

RQ1: To which extent does the inference backend alter benchmark scores?

Table 1 reports the mean divergence (|∆|, Section 3.6) from the hf_raw baseline under deterministic generation for each model/backend pair, averaged across all 14 evaluation metrics. Parentheses indicate the number of statistically significant divergences among the 11 per-item metrics. Overall the backend effect is model-dependent. hf_pipeline is an exact replica of hf_raw with 0.0 divergence, as expected. G EMMA -3-1B-IT is least affected by the backend where no divergence exceeds 0.010. L LAMA -3.2-1B-I NST is the most affected, with ollama (0.055) and langchain_hf (0.049) diverging substantially. For langchain_hf the mechanism is prompt handling on a shared, weightidentical engine, and for ollama it mixes GGUF weight conversion with template and tokenization handling, which cannot be separated. vllm, numerically identical to hf_raw for L LAMA -3.2-1BI NST and G EMMA -3-1B-IT (0.000), instead diverges for Q WEN 2.5-1.5B-I NST (0.025, significant on 10/11 metrics). Considering that models are suing the exact same model running vLLM in the same bfloat16 as hf_raw leaves the divergence essentially from vLLM’s distinct kernel implementation (PagedAttention) under non-associative arithmetic (Yuan et al., 2025). Effect-size analysis (Table 7) shows these differences reach |r| = 0.38 for vllm on Q WEN 2.5-1.5B-I NST. Since greedy decoding removes sampling noise, the observed divergences reflect structural backend effects. These results establish the presence of a backend effect even in the absence of stochasticity, a pattern that becomes relevant when considering more stochastic generation modes. Table 2 counts, for each backend and mode, how many of the 33 comparisons (3 models × 11 per-item metrics) significantly differ from hf_raw. Moving from Deterministic, hf_pipeline shows low divergence from hf_raw (2/33 under Fix with

matched parameters, averaged over five repetitions), which we attribute to the sampling-noise while for others this is the mixed influence of sampling noise and backend differences. As we let backend default generation modes take over the divergence grows to 17/33 and 19/33 under Token=256 and Default, a direct result of variation in the backends’ default generation parameters. Note that Token=256 controls only max_new_tokens and consistently shows less significant divergence compared to Default, where all parameters are predetermined (Table 6). 4.2

RQ2: Are backend differences large enough to change evaluation conclusions?

Table 3 reports the variance attributable to backend choice under different generation modes, aggregated across models and benchmarks. Deterministic isolates the structural backend effect, Fix adds sampling noise while keeping all generation hyperparameters matched, and Default further introduces each framework’s own default hyper-parameters. Under this attribution, the structural backend effect alone accounts for a variance of 0.00074. Adding sampling noise increases the variance to 0.00112, while framework-specific default hyper-parameters increase it further to 0.00190. Thus, even when evaluating the same model on the same benchmark, the choice of serving framework and generation configuration can affect the reported results. Under this incremental attribution, the structural backend effect accounts for approximately 39% of the variance observed with Default settings, while the additional variability introduced by sampling and framework-specific defaults accounts for the remaining 61%. These values represent sequential contributions rather than an exact causal decomposition, as interactions between backend behavior and generation settings may exist. Mode (variance sources)

Variance

Std

Deterministic (backend only) Fix (+ sampling noise) Token=256 (+ default hyper-params) Default (+ all default hyper-params)

0.00074 0.00112 0.00118 0.00190

0.017 0.016 0.019 0.026

Table 3: Backend-induced variance computed within each (model, benchmark) and averaged over all models and benchmark pairs, by generation mode. Computing the variance within a single model and benchmark removes differences in benchmark scale and difficulty, revealing the backend’s contribution.

Gemma-3-1B

disagreement rate

0.5

Llama-3.2-1B

Qwen2.5-1.5B

Backend (shape) hf_pipeline langchain_hf vllm ollama

0.4 0.3 0.2 Task (colour) MMLU TriviaQA TruthfulQA TruthfulQA-Gen BBQ StereoSet

0.1 not severe

0.0 0.0

0.2

0.4

0.0

0.2

|Δ calibrated score|

0.4

0.0

0.2

|Δ calibrated score|

0.4

|Δ calibrated score|

Figure 1: |∆| from hf_raw (x axis) versus the per-item disagreement rate (y axis) for each (model, backend, dataset, mode) cell. Marker shape encodes the backend and colour the benchmark. The shaded quarter-circle (radius 0.05) is the “not-severe” zone and a divergence that flips fewer than 5% of items. The dashed line is y = x where rate of flips is equivalent to |∆| score.

The variance decomposition establishes that backend-related factors considerably contribute to benchmark variability. The remaining question is whether this variability is merely detectable or it is large enough to alter evaluation conclusions. To answer this, we move from variance attribution to practical significance. We look at severe cases of divergence (introduced in Section 3.6) with 5% threshold. Figure 1 plots the disagreement rate (Section 3.6) against the |∆| for every (model, backend, metric, mode) cell (points inside the shaded quarter-circle are not severe), and Table 4 counts the severe configurations for each model and task out of the sixteen backend × mode combinations. Note that hf_raw is the baseline of comparison. Task MMLU TriviaQA TruthfulQA TruthfulQA-Gen BBQ StereoSet

Llama

Gemma

Qwen

14 14 14 8 14 9

4 0 4 4 4 4

9 15 8 5 4 0

Table 4: Number of the 16 (backend (4) × mode (4)) configurations that are severely influenced for each model and task (i.e. that flip the correct/incorrect verdict on more than 5% of the task’s items).

Our results indicate a model-dependence behavior not limited to a single benchmark. L LAMA -3.21B-I NST is affected by backend and mode on almost every task namely 14 of the 16 configurations on MMLU, TriviaQA, TruthfulQA and BBQ, and 8–9 on the two generation-based tasks, with peritem disagreement reaching 48% on StereoSet and

40% on TruthfulQA. G EMMA -3-1B-IT, in contrast, is severe on at most 4 of 16 configurations on any task (none on TriviaQA) and its disagreement never exceeds 13%. Q WEN 2.5-1.5B-I NST sits between the two and its severity seems to be highly task dependent (15/16 on TriviaQA, 9/16 on MMLU) while it stays robust to Stereoset (0/16) and 4/16 on BBQ. Two patterns hold across all models. First, almost all severe cells lie above the y = x diagonal in Figure 1, so the aggregate divergence understates how many individual answers change. Second, per-question agreement corroborates the ranking. Further analysis of Cohen’s κ against hf_raw (Appendix D) averages 0.84 for hf_pipeline but falls to 0.45 for ollama on L LAMA -3.2-1B-I NST, versus 0.79 on G EMMA -3-1B-IT the same ordering appears in the error-set overlap (Appendix E), and the score-space clustering (Appendix F). 4.3

RQ3: How are these effects distributed across task families

Mode

Deterministic Fix Token=256 Default

Q WEN 2.5-1.5B-I NST

G EMMA -3-1B-IT

L LAMA -3.2-1B-I NST

Fact.

Bias

Fact.

Bias

Fact.

Bias

0.0168∗ 0.0074 0.0082 0.0467∗

0.0069 0.0055 0.0042 0.0044

0.0025 0.0054 0.0042 0.0420∗

0.0040 0.0041 0.0040 0.0040

0.0123∗ 0.0214∗ 0.0948∗ 0.0789∗

0.0368∗ 0.0413∗ 0.0407∗ 0.0380∗

Table 5: Average |∆| from hf_raw, split by task family (Factual: MMLU, TriviaQA, TruthfulQA, TruthfulQAGen; Bias: BBQ, StereoSet). Values are averaged across all non-baseline backends and all primary metrics within each group. Higher = greater divergence. ∗ = significant and at least one benchmark× backend pair BH-adjusted significant (p < 0.05).

Backend effects are not uniformly distributed across task families. Table 5 report divergence from hf_raw according to factual and social-bias benchmark families. Under deterministic decoding, task sensitivity differs across models. L LAMA -3.21B-I NST and G EMMA -3-1B-IT exhibit higher divergence on bias benchmarks, whereas Q WEN 2.51.5B-I NST shows higher divergence on factual benchmarks. Once generation modes are introduced, factual benchmarks become more sensitive, particularly under default settings. Interestingly, L LAMA -3.2-1B-I NST exhibits larger |∆| under Token=256 than under Default generation, especially for factual benchmarks (|∆| = 0.0948 vs. 0.0789). Since Token=256 forces all frameworks to generate up to 256 new tokens, this increase may reflect the sensitivity of factual generation to longer decoding trajectories, where backend-specific stopping behavior and generation dynamics have more opportunity to diverge. In contrast, Q WEN 2.51.5B-I NST and G EMMA -3-1B-IT show the opposite trend, with Default producing larger divergence than Token=256, indicating that generation length alone does not explain the observed differences but interacts with model characteristics and framework-specific defaults. Across the 12 model–mode configurations, factual benchmarks exhibit significant divergence in 7 cases, compared with 4 cases for bias benchmarks. These results indicate that backend effects more than generation mode and model behavior interact with benchmark characteristics. Taken together, these results support a consistent conclusion. The inference backend is a genuine factor influencing benchmark measurements rather than a negligible implementation detail. Even under greedy decoding, where sampling variability is removed, changing only the inference stack can alter measured performance. The magnitude and direction of these changes depend jointly on the model, benchmark, and generation configuration, meaning that a single aggregate score can hide important sources of evaluation variability. We therefore recommend that evaluation studies report the backend, its exact version, and the complete generation configuration, ensuring that avoidable sources of variation, such as sampling parameters and framework-specific defaults, remain reproducible rather than implicit. Furthermore, our analysis shows that sampling variability remains nonnegligible even when averaging across five gen-

erations, suggesting that stochastic decoding can obscure backend-induced effects. Greedy decoding therefore provides a more reproducible regime. While bias benchmarks exhibited generally smaller backend-induced divergence in our evaluation, factual benchmarks showed greater sensitivity to generation configuration. Overall, backend choice represents a measurable source of evaluation variability where some components can be controlled through standardized generation settings, while the remaining backend-induced differences should be explicitly measured and reported rather than left unspecified.

Conclusion

In this work we presented a systematic evaluation of how five LLM inference frameworks affect benchmark scores when the model, data, and evaluation protocol are held constant. Under greedy decoding the structural backend effect amounts to a variance of 0.00074 where adding sampling raises it to 0.00112, and each framework’s default hyper-parameters raise it further to 0.00190. Even under greedy decoding, where no sampling noise is present, the backend still changes which items a model answers correctly, and its effect is strongly model-dependent. We relate these divergences to differences in each framework’s serving stack namely the tokenization, system template, and default paramters suited for the backend or even weight of the model in case of Ollama rather than to a single cause. The configuration-driven part can be neutralized by matching the generation settings, whereas these structural, weight- and template-level differences can only be disclosed and considered consequential backend choice effecting reproducibility. For the evaluation community, the central implication is that benchmark numbers are not backend-agnostic. We therefore recommend a minimum reporting standard where the backend name and its version, the complete generation configuration, and, for any cross-backend comparison, the use of deterministic (greedy) decoding as the only regime free of sampling noise. Finally we recommend for bias benchmarks, reporting format-compliance alongside the bias scores.

Limitations

On the generation axis, our four modes probe only a few points in a large space. The Fix and Token=256 modes each fix a single hyper-parameter configura-

tion, and we do not sweep temperature, top-p, topk, or the repetition penalty to map how backend divergence varies across the sampling landscape; nor do we vary batch size, the attention backend, KV-cache settings, or the random seed beyond a single value. We run every model in float16 without quantization, whereas quantization format and compute precision are themselves likely sources of cross-backend disagreement that our design holds constant. The stochastic modes use five generations per prompt, which bounds the precision of our sampling-noise and variance estimates. A larger number of repetitions would tighten the decomposition between structural and sampling-driven variance, and the Deterministic mode rests on a single greedy run whose reproducibility we assume within, but do not test beyond, our environment. That environment is a single consumer GPU (RTX 3090) with one CUDA and driver version. Since floating-point non-associativity is hardware-dependent, the absolute magnitudes we report though not severely, we expect, the qualitative model-dependent pattern may differ slightly on other accelerators or precisions. Furthermore, we provide 3 different models from different family but do not try out scaling factor in backend influence. Eventhough (Pape et al., 2026) utilizes models of higher sizes and demonstrate they are backend dependent we are not independently investigating this matter. Finally, the evaluation and analysis are scoped in ways that bound the conclusions. Our six benchmarks are English-only and dominated by multiplechoice and short-form generation. We do not cover very long-form generation, code, mathematical reasoning, multi-turn dialogue, or agentic tasks, where output-length and sampling effects could compound differently. Bias is represented by two US- and English-centric social-bias suites (BBQ and StereoSet), and StereoSet’s composite score conflates format compliance with bias direction, a confound we report but cannot fully disentangle. Analytically, we treat hf_raw as the reference backend, so all divergences are relative rather than measured against an external ground truth. We reduce each benchmark to one calibrated primary metric in [0, 1] and our within-cell variance attribution assumes that the ladder of modes cleanly and additively separates structural, sampling, and hyperparameter sources, an approximation rather than an exact decomposition. Our per-item flip metric

is binary and its numerical value depends on each benchmark’s scoring rule, and we rely throughout on automatic metrics without human evaluation of the generated text. All in all our analysis brings forth signs of backend dependency and further investigation of severity of influence per task or scaling of the results to larger models requires further investigation which we leave for future works.

References Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Phat Alexandrov, Anton Arkil, Stella Biderman, and 1 others. 2024. Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv preprint arXiv:2405.14782. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. 2021. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3, pages 747–769. Harrison Chase. 2022. LangChain. https://github. com/langchain-ai/langchain. Matt Crane. 2018. Questionable answers in question answering research: Reproducibility and variability of published results. Transactions of the Association for Computational Linguistics, 6:241–252. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from reproduction problems: What replication failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1691–1701, Sofia, Bulgaria. Association for Computational Linguistics. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611. Association for Computational Linguistics. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 31 others. 2023. Holistic evaluation of language models. Trans. Mach. Learn. Res., 2023. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics.

Htut, and Samuel R. Bowman. 2022. BBQ: A handbuilt bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 of Findings of ACL, pages 2086–2105. Association for Computational Linguistics. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang. 2023. The tail wagging the dog: Dataset construction biases of social bias benchmarks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1373–1386, Toronto, Canada. Association for Computational Linguistics. Manpreet Singh and Hassan Sajjad. 2025. Interpreting the effects of quantization on LLMs. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2267–2281, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098. Association for Computational Linguistics.

Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, and Cho-Jui Hsieh. 2024. One prompt is not enough: Automated construction of a mixture-of-expert prompts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 50043–50064. PMLR / OpenReview.net.

Anmol Mekala, Anirudh Atmakuru, Yixiao Song, Marzena Karpinska, and Mohit Iyyer. 2025. Does quantization affect models’ performance on long-context tasks? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9422–9470, Suzhou, China. Association for Computational Linguistics.

Martijn Wieling, Josine Rawee, and Gertjan van Noord. 2018. Squib: Reproducibility in computational linguistics: Are we willing to share? Computational Linguistics, 44(4):641–649.

Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5356–5371. Association for Computational Linguistics. Pape, Evertz, and Schönherr. 2026. The silent hyperparameter: Quantifying the impact of inference backends on LLM reproducibility. arXiv preprint arXiv:2605.19537. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon

Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and mitigating numerical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 December 5, 2025. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.

Table 6: Default-mode generation parameters, per backend and model.

Backend

Model

temp.

top_p

top_k

rep. pen.

max new tok.

ollama

L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

0.8 1.0 0.8

0.90 0.95 0.90

40 64 40

1.1 1.1 1.1

−1 (unbounded) −1 (unbounded) −1 (unbounded)

vllm

L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

1.0 1.0 1.0

1.00 1.00 1.00

−1 (off) −1 (off) −1 (off)

1.0 1.0 1.0

16 16 16

hf_pipeline

L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

0.6 1.0 0.7

0.90 0.95 0.80

50 64 20

1.0 1.0 1.1

20 20 20

hf_raw

L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

0.6 1.0 0.7

0.90 0.95 0.80

50 64 20

1.0 1.0 1.1

20 20 20

langchain_hf

L LAMA -3.2-1B-I NST G EMMA -3-1B-IT Q WEN 2.5-1.5B-I NST

0.6 1.0 0.7

0.90 0.95 0.80

50 64 20

1.0 1.0 1.1

20 20 20

Default generation parameters

Table 6 shows the default parameters of the models for each backend. Each cell is the effective generation configuration a backend applies when the harness supplies no parameter, read directly from the running stack. For the HuggingFace-family backends from each model’s model.generation_config (transformers 4.57), for vLLM 0.8.5 from SamplingParams(), and for Ollama 0.31.1 from ollama show --parameters on the exact fp16 tags we serve. The three HuggingFace backends dispatch to the same model.generate() and therefore share one configuration, which is the model’s own tuned one; vLLM ignores the model’s generation_config and samples from a fixed temperature= 1.0; Ollama reads the model’s Modelfile and otherwise falls back to its runner defaults. Values not set by the model or Modelfile inherit the framework fallback (transformers: top_k= 50, rep. pen.= 1.0, max_length= 20; Ollama: temp.= 0.8, top_p= 0.9, top_k= 40, rep. pen.= 1.1). All configurations sample (do_sample=true in every model’s config; vLLM and Ollama always sample).

Per-Model Signed Divergence from hf_raw

Figure 2 reveals the direction of backend effects that an absolute-value (unsigned) view conceals. For L LAMA -3.2-1B-I NST, TruthfulQA cells show mixed positive and negative divergence (Ollama above, vLLM below hf_raw), explaining why the

absolute divergence is large while individual backends go in opposite directions. For Gemma and Qwen, the vast majority of cells are near-zero and non-significant.

Per-Backend Signed Divergence by Task

Figures 3 and 4 give the most granular view behind RQ1 and RQ2. For each model, and for the Deterministic and Default modes, the signed difference of every non-baseline backend from hf_raw on each task’s primary metric, with ∗ marking tasks on which at least one metric is BH-significant. Under Deterministic decoding (Figure 3) hf_pipeline is uniformly near zero, Gemma’s rows are pale throughout, and Llama shows the strongest structural divergence on the TruthfulQA family. Under Default decoding (Figure 4) the cells deepen most sharply for vllm and ollama on Llama’s factual tasks reflecting the framework-default sampling gaps and added sampling noise discussed in Section 4.3.

Inter-Backend Agreement Analysis

To complement aggregate mean-divergence metrics, we assess whether backends agree on which questions a model answers correctly. For each (model, mode, dataset) we compute whether models agree on correctness of the answer as a binary measure, correct/wrong and calculate Fleiss κ across all five backends jointly, and Cohen’s κ for each non-baseline backend paired against hf_raw. Figure 5 shows compound Cohen κ views. The Backend × mode panel (right) confirms the sen-

Gemma

Llama

+0.01*

Default

+0.00*

-0.00*

-0.01*

-0.09*

-0.11*

-0.01*

-0.00*

-0.01*

-0.00

+0.00*

+0.00

+0.01

+0.00*

+0.01*

-0.10*

-0.13*

+0.00*

-0.00

+0.01*

-0.00

-0.01

+0.00

-0.02*

-0.03*

-0.04*

-0.02*

-0.14*

-0.17*

-0.00

-0.03*

+0.02*

+0.00

-0.08*

-0.09*

-0.00

-0.08

Fix

+0.01*

-0.00*

+0.00*

-0.00*

+0.00*

-0.01*

-0.01

+0.00*

+0.00

+0.01

+0.00*

+0.00

+0.01*

+0.00*

-0.00

+0.01*

+0.00*

-0.01

+0.00

+0.02*

-0.00*

-0.01*

+0.07*

+0.01*

+0.00*

-0.00

+0.09*

+0.11*

+0.01

+0.09

Token=512

+0.00*

+0.01*

-0.00*

+0.02*

-0.00*

+0.00*

-0.01*

-0.01

+0.00*

+0.00

-0.00*

-0.00

+0.00

-0.00*

-0.00

+0.00*

-0.00

+0.01*

+0.00

-0.01

+0.00

-0.01*

-0.00*

-0.01*

+0.00*

-0.03*

+0.04*

+0.01

-0.07*

-0.06*

-0.00

-0.07

0.2

Mode

0.1 0.0 −0.1

-S

-In

−0.2

-0.08

-IC AT

+0.00

S -L

-B ia BB Q

-0.10*

-0.08*

+0.00

-U

-D

+0.03*

BB Q

Metric

-0.03*

BB Q

-R

+0.00*

-A cc

-0.22*

TQ AG

ulQ

TQ AG

Q AF

-0.18*

-0.03*

Tr ut hf

LU M

Tr iv ia

-0.03*

Q AEM

-0.02*

Tr iv ia

-IC AT

+0.01

-S

-In

BB Q

-0.01

-L

-B ia

+0.00

+0.01*

-0.00

-U

-D

+0.00*

BB Q

Metric

+0.00*

BB Q

-R

en TQ AG

+0.00*

-A cc

+0.01*

TQ AG

ulQ

Q AF

+0.00*

BB Q

+0.00

+0.00*

Tr ut hf

LU M M

Tr iv ia

+0.00*

Q AEM

+0.00*

Tr iv ia

+0.00

-IC AT

-In

-L

+0.00

-S

-0.00

-B ia BB Q

is -D

-U

BB Q

+0.00*

+0.00

-0.01*

-0.00*

BB Q

-A cc

-0.01*

-R

Tr ut hf

+0.01*

TQ AG

A ulQ

Q AF

+0.00*

BB Q

-0.01*

TQ AG

LU M M

Tr iv ia

-0.00*

Tr iv ia

Q AEM

+0.00*

Avg. signed diff vs hf_raw

Qwen Deterministic

Metric

Figure 2: Average divergence from hf_raw on the calibrated primary metric of each benchmark, for each generation mode. Note that contrary to main body of paper here we report the ∆ instead of |Delta| to capture negative and positive influences. ∗ = at least one backend BH-adjusted significant (p < 0.05).

−0.025

+0.01*

-0.03*

-0.02

+0.01*

+0.00

-0.00

+0.01

+0.00

-0.00

+0.00

-0.00

0.00

ollama

+0.01

+0.00

+0.02

+0.00

+0.01*

+0.02*

+0.00*

-0.00

+0.04*

+0.00

-0.03

+0.01

+0.02

-0.11

-0.00

-0.01

-0.12

vllm

+0.00

-0.00

+0.00

-0.00

+0.00

-0.00

+0.00

-0.00

ollama

-0.02*

-0.01

-0.01*

+0.04*

-0.03*

-0.04*

+0.00

-0.01*

+0.01*

+0.01

-0.17*

-0.25*

+0.00

-0.16

0.2 0.1 0.0

−0.1

-S

M M

-IC

S M

-In

-L

Tri v

ia s

nk BB Q

-B

-U

-D BB Q

BB Q

Metric

BB Q

-A cc

c -R

b TQ

-R

A lQ

A-F

A-E

ia Tri v

Tri v

th fu

M M

Tru

+0.15*

−0.2

−0.04

-IC

-S SS

BB Q

-L

-B

-In

ia s

nk -U

-D

-A cc

BB Q

-R

th fu Tru

BB Q

A-F

M Tri v

Tri v

A-E

−0.075 M

+0.00

-0.12*

-0.01*

+0.00

+0.02*

-S

+0.00

-0.00

-IC AT

+0.00

-0.00

+0.01

-0.00

+0.00

-0.04*

+0.02*

+0.00

-0.04*

+0.02*

-0.00

+0.00

-0.04*

+0.01*

+0.00

-0.01

−0.02

−0.050

ollama

vllm

+0.00

langchain_hf

0.000

+0.00

-In

+0.01

+0.00

-L

+0.00

+0.01*

-0.00

-0.02

-0.00

-0.03*

+0.00

-B

+0.01*

+0.00

-0.01*

-0.01

+0.08*

-0.01*

+0.00

-D

+0.07*

-0.01

+0.00

-U

-0.04*

-0.00

+0.00

+0.03*

-0.00*

+0.00

+0.03*

+0.00

+0.02*

langchain_hf

hf_pipeline

vllm

0.02

-0.01

+0.00

+0.01

+0.00

-A cc

+0.00

en -R

-0.02*

+0.00

+0.01

+0.00

+0.01*

Llama, Deterministic

0.04

+0.00

-0.00

+0.00

+0.00*

+0.00

en -R

-0.01*

+0.00

-0.01*

+0.00

+0.02*

+0.00

-0.01*

+0.00

-0.01*

+0.00

-0.00

+0.00

0.025

langchain_hf

hf_pipeline

Backend

0.050

+0.00

A-F

+0.00

Tru

+0.00

Tri v

+0.00

A-E

+0.00

Backend

+0.00

Backend

+0.00

Signed diff vs hf_raw

+0.00

Signed diff vs hf_raw

Gemma, Deterministic

0.075 +0.00

Signed diff vs hf_raw

Qwen, Deterministic hf_pipeline

Metric

Figure 3: Signed divergence from hf_raw (green = backend higher, red = lower) per backend (rows) and task (columns), one panel per model family in Deterministic mode. ∗ marks a task with at least one BH-significant metric.

-0.13*

-0.16*

+0.00

0.1 Backend

Llama, Default

Gemma, Default

+0.00

langchain_hf

-0.00

-0.13*

-0.16*

-0.00

+0.00

+0.01

vllm

+0.01*

-0.01

-0.02*

-0.04*

+0.04*

+0.05*

-0.01*

-0.01

-0.02*

-0.00

+0.01*

+0.00

-0.00

+0.00

0.0

hf_pipeline

-0.00*

+0.01*

+0.00

-0.13*

-0.17*

-0.00*

+0.00

-0.00

+0.00

0.1 langchain_hf

+0.00

+0.01*

+0.00

-0.13*

-0.16*

-0.00*

+0.00

-0.00*

-0.00

+0.00

-0.00

vllm

+0.00*

-0.00*

+0.00

-0.01*

-0.01

-0.00

+0.00

-0.00

0.0

+0.01*

+0.00

+0.02*

-0.14*

-0.18*

-0.01*

+0.00*

-0.03*

-0.01

+0.01*

+0.00

+0.01

-0.01*

-0.02

-0.17*

-0.21*

-0.00

-0.02*

+0.01*

+0.00

-0.01*

-0.00*

-0.00

-0.01

langchain_hf

-0.01*

-0.03*

-0.17*

-0.21*

-0.00

-0.02*

+0.01*

+0.00

-0.01*

-0.00*

-0.01

vllm

-0.04*

-0.07*

-0.08*

-0.13*

-0.02*

-0.03*

-0.01

-0.06*

+0.05*

+0.01

-0.07*

+0.00

-0.07

ollama

-0.03*

-0.04*

+0.10*

-0.21*

-0.25*

+0.00

-0.01*

+0.01*

+0.00

-0.21*

-0.30*

-0.01

-0.21

0.2 0.1 0.0

−0.1

−0.1 ollama

0.3

hf_pipeline

ollama

+0.01*

+0.00

+0.02*

-0.14*

-0.18*

+0.01*

+0.02*

+0.00*

-0.00

+0.04*

+0.00

-0.03

+0.01

Signed diff vs hf_raw

-0.00

Backend

-0.00

Signed diff vs hf_raw

-0.00

Backend

-0.00

Signed diff vs hf_raw

Qwen, Default hf_pipeline

−0.2

S -S

-IC AT SS

S M

-In

-L SS

s ia -B

-D

-U

-A cc

en -R

b en -R

AG TQ

A-F

fu th Tru

Tri v

A-E

LU M M

S -S

-IC AT

M S

-In v

-L

s ia

-D

-U

-B

-A cc

Q BB

-R en AG

AG TQ

-R

lQ A

A-F

A-E

Tru th fu

Tri via

M LU M

Tri via

S -S

-IC AT

M S

-In v

-L

ia -B Q

Q BB

-U

-D

-A cc

Q BB

-R AG TQ

-R

lQ A

1 A-F

Tru th fu

A-E

Tri via

M LU

−0.3 Metric

Metric

Figure 4: Signed divergence from hf_raw per backend (rows) and task (columns), one panel per model family in Default mode. Divergences are larger than under Deterministic decoding, especially for vllm and ollama on Llama’s factual tasks.

Cohen κ vs hf_raw: Model×Backend

Cohen κ vs hf_raw: Model×Mode

Cohen κ vs hf_raw: Backend×Mode

Gemma-3-1B

0.982

0.976

0.980

0.868

Gemma-3-1B

0.963

0.945

0.951

0.948

Llama-3.2-1B

0.687

0.565

0.628

0.611

Llama-3.2-1B

0.834

0.581

0.517

0.559

Qwen2.5-1.5B

0.949

eli

pip

hf_

0.931

hf in_

gc lan

0.876

vll

0.876

Qwen2.5-1.5B

oll

a am

0.927

ist

in rm

te De

0.900

0.904

Fix

fau

0.901

hf_pipeline

1.000

0.840

0.809

0.841

langchain_hf

0.819

0.840

0.805

0.833

vllm

0.961

0.784

0.792

0.775

ollama

0.853

0.770

0.756

0.762

Fix

12 =5

n ke To

ist

in rm

fau

12 =5

n ke To

Figure 5: Cohen’s κ between each non-baseline backend and hf_raw for correct/wrong answer. Values are averaged over all conditions not held constant.

sitivity ordering: hf_pipeline achieves perfect agreement with hf_raw across all models (κ1.0), while langchain shows the widest variation (0.81) under deterministic follows ollama (0.85). How ever these results are model dependent as L LAMA 3.2-1B-I NST is the most influential factor in all of these variations where for Langchain the value drops to 0.56 while for G EMMA -3-1B-IT and Q WEN 2.5-1.5B-I NST these values stays at 0.97 and 0.93 averaging over all variations respectively. These κ values quantify what the mean-divergence heatmap implies: backends do not merely shift scores uniformly; they change the identity of questions answered correctly, and the degree of change is strongly model-dependent. The severity of this per-question rearrangement is shown in the main text (Figure 1, Section 4.2).

E Error-Set Overlap (Jaccard Similarity) Jaccard similarity of error sets measures whether backends fail on the same questions as hf_raw, independently of how many questions fail overall. A Jaccard of 1 means the two backends fail on exactly the same questions while 0 means completely disjoint failure sets. Figure 6 shows the model × backend compound view. For G EMMA -3-1B-IT, all non-baseline backends maintain Jaccard ≥ 0.93 against hf_raw, confirming that Gemma’s errors are benchmarkspecific and backend-independent. For L LAMA 3.2-1B-I NST, mean Jaccard vs. hf_raw drops to 0.792. Backends are not merely shifting Llama’s scores they are redirecting success and failure to different questions, a qualitatively different failure mode. hf_pipeline again achieves the highest Jaccard across all models, while ollama shows the lowest for L LAMA -3.2-1B-I NST. The backend × mode panel shows that stochastic modes reduce Jaccard uniformly (more noise introduces more error-set divergence), but the model-dependent ordering persists in every mode.

Backend Clustering in Score Space

To understand the geometric relationship between backends in per-question score space, we represent each backend as a vector of calibrated scores over all questions and apply PCA to reduce to two dimensions. Figure 7 shows one panel per (model, mode) combination. For G EMMA -3-1B-IT, all five backends

cluster within a compact region in PC1–PC2 space regardless of mode, confirming that they are producing nearly identical per-question responses. For L LAMA -3.2-1B-I NST, ollama separates along PC1 from the four HuggingFace-ecosystem backends in every mode; under stochastic modes, vllm also separates along PC2. For Q WEN 2.5-1.5BI NST, the cluster is more diffuse than Gemma’s but tighter than Llama’s.

Response Length as a Potential Confound

For generative tasks (TriviaQA, TruthfulQA-Gen), response length is a potential confound: a backend that produces longer responses may achieve higher ROUGE-L scores simply due to increased token overlap, not due to better truthfulness or factual recall. Figure 8 shows mean response length across all tasks. The backend × mode panel reveals that Default mode produces the highest length variance across backends (since each backend uses its own default max_new_tokens), while Fix and Deterministic modes are length-matched. The model × backend panel shows that langchain_hf and vllm tend to produce longer responses than hf_raw for L LAMA -3.2-1B-I NST and Q WEN 2.5-1.5B-I NST under unconstrained modes, consistent with the positive ROUGE-L divergence seen for those backends on TruthfulQA-Gen in Default mode. A Spearman correlation between response length and calibrated score on generative tasks reveals a positive but moderate relationship for TriviaQA and TruthfulQA-Gen: longer responses are associated with higher scores, but the relationship is driven by a few failure cases (very short or truncated outputs) rather than a monotonic lengthquality trade-off. Researchers using ROUGE-L as a proxy for generation quality should therefore ensure that max_new_tokens is matched across all comparison frameworks before drawing crossbackend conclusions.

Effect Size

As additional assurance that the significance tests are not merely minor changes inflated by the large scale of the datasets, we provide an effect-size analysis per model, mode, and backend. Each cell is the mean absolute matched-pairs rank-biserial correlation |r| between a backend and hf_raw, averaged

Jaccard vs hf_raw: Model×Backend

Jaccard vs hf_raw: Model×Mode

Jaccard vs hf_raw: Backend×Mode

Gemma-3-1B

0.987

0.982

0.986

0.905

Gemma-3-1B

0.973

0.960

0.965

0.962

Llama-3.2-1B

0.829

0.749

0.808

0.781

Llama-3.2-1B

0.892

0.763

0.756

0.755

Qwen2.5-1.5B

0.951

eli

pip

hf_

0.934

0.884

hf in_

Qwen2.5-1.5B

0.930

a am

vll

gc lan

0.884

ist

oll

in rm

te De

0.906

0.909

Fix

fau

hf_pipeline

1.000

0.897

0.894

0.898

langchain_hf

0.875

0.897

0.886

0.894

vllm

0.963

0.865

0.878

0.863

ollama

0.889

0.847

0.844

Fix

0.907

12 =5

ist

n ke To

in rm

fau

12 =5

n ke To

Figure 6: Jaccard similarity of the error sets (incorrectly answered questions) between each non-baseline backend and hf_raw, shown as compound heatmaps: model × backend (left), model × mode (centre), backend × mode (right). Higher Jaccard = the same questions fail regardless of backend. Gemma-3-1B Deterministic

Llama-3.2-1B Deterministic

langchain_hf

Qwen2.5-1.5B Deterministic langchain_hf

ollama

PC2 (26%)

PC2 (29%)

PC2 (12%)

20 0

vllm −5 hf_pipeline hf_raw

−10 langchain_hf

vllm hf_pipeline hf_raw

hf_pipeline hf_raw

−20 0

−20 −25

hf_raw

ollama

vllm

PC2 (20%)

hf_raw

hf_pipeline langchain_hf

PC2 (28%)

−10

hf_raw langchain_hf hf_pipeline

ollama

−10

−30

−5 langchain_hf hf_pipeline

vllm

−40 0

−40

−20

ollama

−20

−10

PC1 (86%)

PC1 (31%)

PC1 (62%)

Gemma-3-1B Fix

Llama-3.2-1B Fix

Qwen2.5-1.5B Fix

vllm

langchain_hf

hf_raw hf_pipeline

ollama

40 10

ollama

30 PC2 (26%)

0.0 −2.5 −5.0

PC2 (16%)

PC2 (6%)

−20

PC1 (68%)

−20

PC2 (4%)

100

Qwen2.5-1.5B Default

2.5

PC1 (70%)

Llama-3.2-1B Default

vllm

5.0

PC1 (87%)

Gemma-3-1B Default

ollama vllm

ollama

20 10 0

−7.5

hf_pipeline hf_raw langchain_hf

−10 vllm hf_raw

−10 hf_pipeline

−10.0 langchain_hf

ollama

vllm

−20 0

−20

ollama

vllm

hf_raw ollama

hf_raw

0.0

langchain_hf

hf_pipeline

−2.5 −5.0

PC2 (19%)

PC2 (28%)

PC2 (5%)

10 PC1 (60%)

Qwen2.5-1.5B Token=512

langchain_hf hf_pipeline

langchain_hf hf_pipeline hf_raw

−20

−7.5

−10

−10.0 −12.5

PC1 (30%)

Llama-3.2-1B Token=512

5.0 2.5

−10

PC1 (90%)

Gemma-3-1B Token=512

−40

vllm

20 PC1 (87%)

vllm

−20

0 PC1 (32%)

ollama

−20 40

−20

−10

PC1 (63%)

Figure 7: PCA of per-question calibrated score vectors (one vector per backend per model × mode panel). Each point represents one backend projected onto the first two principal components of the question-score matrix. Backends closer together produce more similar per-question behaviour. Variance explained by PC1/PC2 shown on each axis.

Mean length: Model×Backend

Gemma-3-1B

Mean length: Model×Mode

Gemma-3-1B

Mean length: Backend×Mode

Llama-3.2-1B

Qwen2.5-1.5B

eli

pip

hf_

w ra

gc lan

a am

oll

12 =5

vll

n ke To

fau

ist

in rm

hf_raw

langchain_hf

ollama

vllm

Fix

Llama-3.2-1B

f _h ain

hf_pipeline

Fix

12 =5

n ke To

fau

ist

in rm

Figure 8: Mean response length (characters) as compound heatmaps: model × backend (left), model × mode (centre), backend × mode (right).

over the eleven per-item metrics, for each model, backend, and generation mode. Rank-biserial is the paired effect size of the per-item test (McNemar for binary metrics, Wilcoxon signed-rank for continuous ones) where |r| < 0.1 negligible, 0.1–0.3 small, 0.3–0.5 medium, > 0.5 large. hf_pipeline shares hf_raw’s in-process engine and is an exact no-op under Deterministic decoding. Because |r| is computed only over items on which a backend disagrees with hf_raw, it captures the direction of the change rather than its volume, and should be read together with the divergence magnitudes in Table 1

Usage of LLM

All the contents of the paper are written and investigated by the authors. We have used Claude code for visualization of the analysis and help as writing tool. Backend

Det.

Fix

Token=256

Default

L LAMA -3.2-1B-I NST hf_pipeline 0.000 langchain_hf 0.3886 vllm 0.112 ollama 0.3389

0.021 0.3388 0.0514 0.1909

0.39410 0.42210 0.51710 0.41910

0.3809 0.35710 0.44810 0.47810

G EMMA -3-1B-IT hf_pipeline 0.000 langchain_hf 0.1492 vllm 0.124 ollama 0.1504

0.082 0.2234 0.072 0.32410

0.1185 0.2195 0.1451 0.2195

0.3448 0.3236 0.1314 0.3848

Q WEN 2.5-1.5B-I NST hf_pipeline 0.036 langchain_hf 0.2708 vllm 0.38310 ollama 0.2187

0.0212 0.0484 0.2038 0.1578

0.0612 0.028 0.2778 0.1678

0.1962 0.1932 0.2018 0.2999

Table 7: Effect size of models for each backend and generation mode. A superscript gives the number of the six metrics that are BH-significant (p < 0.05) in that cell.

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

Analyze & explore related research in Conceptio

Related documents