How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

arXiv:2610.13764v1 [stat.ML] 9 Oct 2026

Manuel Hülskamp

Julius Kobialka Emanuel Sommer Thomas Nagler LMU Munich and Munich Center for Machine Learning

Abstract Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

1

INTRODUCTION

Prior-data fitted networks (PFNs) have emerged as powerful foundation models for tabular data (Müller et al., 2021; Hollmann et al., 2025). By learning to approximate Bayesian posterior predictive distributions (PPDs) via in-context learning (Brown et al., 2020) during pretraining, they perform inference on new downstream datasets in a single forward pass. Given their success, many works have proposed strategies to improve predictive performance, scale to larger datasets, or transfer the approach to other tasks (Qu et al., 2026; Zhang et al., 2025b; den Breejen et al., 2024; Thomas et al., 2024; Xu et al., 2025; Liu and Ye, 2025; Ma et al., 2026; Robertson et al., 2025; Balazadeh Meresht et al., 2025). The Bayesian interpretation is central to this success. A PFN is trained on datasets drawn from a

David Rügamer

Deployment error RΠ′ (q) − HΠ′ (§3.3)

Prior error

Coverage PΠ′ (Π)

Amortization error AΠ (q)

Context error CΠ

Optimization Estimation

Transport λ ΓΠ′ (q)

Approximation

Off-support (1−λ) OΠ′ (q)

Figure 1: Decomposition of the deployment error of a PFN (Proposition 2). This paper studies the amortization error, the only term controlled by pretraining, which splits into optimization, estimation, and approximation error (Eq. (8)). The context and prior error support this analysis: the context error is incurred even by the exact PPD, and the prior error arises when the deployment tasks differ from the prior (see Section 3.3).

synthetic prior, and its training objective is minimized by the PPD of that prior. In other words, a PFN is trained to minimize its risk under its own prior, and the PPD is the optimal predictor there. Surprisingly, the evidence that trained PFNs closely track the PPD they amortize is thin and mostly focused on whether transformers can do Bayesian inference at all via the PPD (Müller et al., 2021) and not how well they recover it. Moreover, pretraining is typically reduced to first-order optimization of a single network, on the grounds that effectively unlimited synthetic data should remove any uncertainty about the outcome. Whether this holds is a question about amortization uncertainty. Existing work on uncertainty in PFNs (Nagler and Rügamer, 2025; Fortini et al., 2026) studies the uncertainty that remains for a fixed network at inference time and is, by construction, blind to it. Furthermore, when PFNs are deployed on real data, the model likely faces a different distribution of tasks, and the optimal predictor changes with it. Hence, deployment is a distribution shift from the pretraining prior to the real-world task distribution, which further

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

complicates the error source analysis of deployed PFNs. Our Contribution In this work, we make the following contributions to study the sources of error and uncertainty in PFNs with a focus on amortization: 1. A decomposition of the error of any PFN under a deployment distribution, splitting into prior, context and amortization error. 2. A study of the amortization error of PFNs under its training prior, including a comparison to a realistic reference PPD (Section 2). 3. An extended investigation of the amortization error of PFNs under a deployment task distribution (Section 3). 4. An analysis of approximate Bayesian model averages over PFN weights (Section 4): Deep ensembles consistently improve predictions, whereas sampling and variational learning add little.

2

AMORTIZATION ERROR

2.1

Setup

Following Nagler (2023), a task is a law p over (X, Y ) ∈ X × Y with covariate law pX and conditional p(y | x), and a dataset Dn = {(Xi , Yi )}ni=1 consisting of n independent and identically distributed (iid) draws from it. A PFN assumes a prior Π over pairs (p, N ) of a task and a context size, whose marginal over N is the size distribution Ψ. Given (p, N =n), the context is Dn ∼ p⊗n , and the PFN targets the posterior predictive distribution (PPD) Z πΠ (y | x, Dn ) =

p(y | x) dΠ(p | x, Dn ).

(1)

At deployment, pairs follow a different distribution Π′ with size marginal Ψ′ . In the following, we study PFNs through the lens of risk minimization. Risks average over a task, a context Dn ∼ p⊗n and an independent query (X, Y ) ∼ p. We write Ep [·] := EDn ∼p⊗n E(X,Y )∼p [·] for a fixed pair (p, n) and EΠ [·] := E(p,N )∼Π Ep [·] when the pair is drawn as well, likewise for Π′ . Random variables are capitalized.

Proposition 1. Let RΠ′ (πΠ′ ) be finite. Then for every predictor q, RΠ′ (q) − RΠ′ (πΠ′ ) = 
 EΠ′ DKL πΠ′ (· | X, DN ) q(· | X, DN ) ≥ 0, (3) with equality if and only if q = πΠ′ almost surely under Π′ . In particular, πΠ′ is the unique minimizer of RΠ′ and RΠ′ (πΠ′ ), the Bayes risk of Π′ , is the lowest risk any predictor can attain on Π′ . The proof is given in Appendix B.1. A PFN qθ is pretrained on the population objective L(θ) := RΠ (qθ ).

With Π′ = Π, Proposition 1 thus gives the pretraining objective: a PFN is trained to minimize its risk under its own prior, the minimizer over all predictors is πΠ (Müller et al., 2021), and the excess risk 
 AΠ (qθ ) := RΠ (qθ )−RΠ (πΠ ) = EΠ DKL πΠ ∥ qθ ≥ 0 (5) is the amortization error. This error is the distance, in expected KL, of the network from the PPD it is supposed to amortize. In practice, Eq. (4) is approximated by empirical risk minimization over M datasets D(j) whose task and size (pj , nj ) are drawn from the prior, 1 Lemp (θ) := − M

Risk Under a Task Distribution

A PFN with parameters θ ∈ Θ is a measurable map qθ : (x, Dn ) 7→ qθ (· | x, Dn ) into the distributions on Y. Deployed on Π′ , its risk under the log loss is   RΠ′ (qθ ) := EΠ′ − log qθ (Y | X, DN ) .

(2)

The following proposition characterizes the optimal predictor under an arbitrary task distribution.

PM

j=1 log qθ

yj | xj , D(j) ,

(6)

optimized with first-order stochastic gradient methods such that, in general, we have no guarantee how close we get to the target πΠ . Figure 1 places the amortization error among the two other sources of error of a deployed PFN: the context error, incurred even by πΠ because a finite context leaves the task uncertain, and the prior error, incurred because real tasks follow Π′ = ̸ Π. We focus on the former and work towards its precise relation to the other two in Proposition 2. As a first step, we propose how to analyse the amortization error for a tractable but realistic reference prior in the following section. 2.3

2.2

(4)

Measuring Amortization Error

Measuring AΠ (q) = EΠ [DKL (πΠ ∥q)] directly requires knowledge of the true PPD, which is intractable for the priors used by current tabular foundation models. The evidence that trained PFNs closely track the PPD they amortize comes from Müller et al. (2021, 2024). The considered settings are low-dimensional compared to current structural causal model (SCM) prior (Qu et al., 2026; Hollmann et al., 2025), none use a current PFN architecture at scale, and only small context sizes

Manuel Hülskamp

Julius Kobialka

PFN large

Thomas Nagler

MILE (MCLMC), 24 chains

Oracle (true task)

Gaussian features, log-uniform Ψ

Gaussian features, uniform Ψ

0.4

0.12

0.12

0.11

0.11

0.3

0.10

n ≥ 1024 (magnified)

Prior-data NLL

Emanuel Sommer

SCM features, uniform Ψ

n ≥ 1024 (magnified)

0.10

1024

2048

4096

256

1024 4096 4

David Rügamer

n ≥ 1024 (magnified)

0.12 0.11 1024

2048

4096

256

1024 4096 4

1024

2048

4096

256

1024 4096

0.2 0.1 4

16

64

Number of revealed data points (n)

16

64

Number of revealed data points (n)

16

64

Number of revealed data points (n)

Figure 2: Comparison of PPD approximation quality to MCMC reference on a classification task across both Gaussian and TabICL prior feature distributions (SCM prior). The labeling mechanism is a Bayesian neural network with two hidden layers with 16 neurons each, and the large TabICL variant is trained on these priors from scratch. The PFN approximates the ground-truth PPD provided by the MCMC reference very well across different context sizes (Figure inspired by Müller et al., 2021). Each point is the risk on datasets of context size n; Ψ is the size distribution of pretraining.

are considered. Furthermore, the MCMC reference performed worse than the PFN under consideration. To fill this gap, we measure the amortization error not only in the setting introduced in Müller et al. (2021) with standard Gaussian features but also in a considerably more realistic setting by keeping the covariate generator of a state-of-the-art synthetic prior, so that the context tables have the marginal structure of current SCM priors (mixed feature types, correlations, non-Gaussian marginals). We replace the random mechanism mapping covariates to targets by a Bayesian neural network (BNN) fw defining the likelihood y | x, w ∼ p(y | fw (x)) with w drawn from a Gaussian prior, analogous to the BNN prior introduced in Hollmann et al. (2023). Because the covariate generator and w are drawn independently, this prior factorizes. Conditioning on x∗ leaves the posterior over w unchanged, and the PPD reduces to Z
πΠ (y | x∗ , Dn ) = p y | fw (x∗ ) dΠ(w | Dn ). (7) Making use of recent advances in both MCMC algorithm development (Robnik and Seljak, 2024) as well as sampling-based inference for BNNs (Sommer et al., 2025, 2026), we compare the prediction of PFNs against the PPD they are trained to amortize to obtain an empirical lower bound on the approximation error. MCMC yields a reference π̂Π against which DKL (π̂Π ∥q) can be computed for each pair (x∗ , Dn ). In Fig. 2, we plot the aggregate NLL of a PFN against a largebudget MCMC reference. In the inset plots, it is visible that the MCMC reference, where 24 MCMC chains are sampled per task, still dominates the PFN, however, the amortization gap appears small and mostly dependent on what datasets are probable under the prior

(deviations at small context settings for Ψ uniform vs. deviations at large context settings for Ψ loguniform). Details are given in Appendix C. While this approach is sufficient to study the general capabilities of transformers to approximate the PPD and measure quantitatively how good the approximation is in terms of the amortization error, two aspects complicate the analysis once we move to productiongrade synthetic priors and deployment on real-world datasets: We cannot compute the reference PPD, and deployment under a distribution Π′ different from the training distribution Π introduces a distribution shift (see Appendix C). The following studies these problems in more detail.

3

AMORTIZATION ERROR UNDER DEPLOYMENT

3.1

Amortization Error Without a Reference

Beyond tractable priors, we cannot compute a reference PPD, and the amortization error has to be studied without an (approximate) ground-truth. Viewing the PFN with parameters θ̂ via empirical risk minimization, the amortization error splits into three components AΠ (qθ̂ ) = RΠ (qθ̂ ) − RΠ (qθ̃M ) + RΠ (qθ̃M ) − RΠ (qθ∗ ) | {z } | {z } optimization

+ RΠ (q ) − RΠ (πΠ ), | {z } θ∗

estimation

(8)

approximation

where θ̃M minimizes the empirical risk for M training samples and θ ∗ = arg minθ∈Θ L(θ), i.e., minimizes the population risk (4). The estimation error is the part

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

that classical learning theory ties to the number of pretraining datasets M and expects to decrease at a 1/M rate (Nagler, 2023). This is where PFNs profit from effectively unlimited synthetic data. The approximation error is a property of the hypothesis class and is reduced only by enlarging it, which extensive work on architectures attempts (Qu et al., 2026; Grinsztajn et al., 2025; Kong et al., 2026). The optimization error is a consequence of the optimizer and for overparameterized networks trained on a non-convex risk, it is not guaranteed to decrease with model size or training duration. To confirm that the estimation error vanishes as expected, we can study K models that share all optimizer settings but differ in seed and training data, at several scales M1 < · · · < ML , and compute PK (k) 1 d EE(M )= K k=1 Ltest (θM ) − Ltest (θref ),

(9)

where θref is a reference checkpoint with Mref ≫ ML . Averaging over the K runs cancels the zero-mean optimization noise of individual runs. 3.2

Shared & Removable Amortization Error

Optimization and approximation error cannot be isolated in the same way, since neither θ̃M nor θ ∗ nor πΠ is tractable. What is observable is the functional spread across K independently trained ensemble members q1 , . . . , qK with identical M , model size, and optimizer settings. We measure this spread by the gend JSD between eralized Jensen–Shannon divergence AU the members’ predictive distributions and by the paird dis , both averaged wise argmax disagreement rate AU over test points (definitions in Appendix D). Neither requires labels. The following classical identity (Cover and Thomas, 2006) connects this spread to predictive performance. Lemma 1. For any distributionPp on Y and predictors K 1 q1 , . . . , qK with mixture q̄ = K k=1 qk , 1 K

PK

k=1 DKL (p ∥ qk ) = DKL (p ∥ q̄) + ∆p ,

(10)

  PK 1 where ∆p := Ey∼p log q̄(y) − K k=1 log qk (y) ≥ 0, with equality if and only if all qk agree p-almost everywhere. ∆p is the averaging gain, i.e., the amount by which the mixture improves on its average member in expected log loss under p. Applied with p = πΠ , Lemma 1 splits the amortization error itself: Corollary 1. For PFNs q1 , . . . , qK with mixture q̄,   P 1 . (11) k AΠ (qk ) = AΠ (q̄) + EΠ ∆πΠ K | {z } | {z } shared

removable ≥0

Thus, the average amortization error of K PFNs splits into two parts that differ in what can be observed without the PPD. The first, AΠ (q̄), is the error the members have in common as they jointly deviate from πΠ , which cannot be measured without a reference as in Section 2.3. The second term is the part of the amortization error that differs between the members and does not require Pa reference to compute. It equals 1 the log-loss gain K k RΠ (qk ) − RΠ (q̄) of the mixture over its average member, which can be estimated on datasets drawn from the prior, and it vanishes if and only if the members agree. The disagreement metrics d JSD and AU d dis are therefore label-free probes of AU this visible part, and blind to the shared one. Revisiting Eq. (8), if the estimation error vanishes at the expected rate while disagreement persists, the remaining amortization error is consistent with optimization error, possibly combined with residual approximation error: independent runs reach functionally distinct solutions that more data does not reconcile. To test whether the removable part is driven to zero d and both by the usual scaling levers, we study EE disagreement metrics along two axes, the number of pretraining datasets M (data axis) and the model size (capacity axis). Results are depicted in Fig. 3. Under Π′ ̸= Π, Proposition 1 says that the optimal predictor is πΠ′ , not the πΠ the PFN was trained towards. Deploying a PFN on real data is thus a distribution shift from Π to Π′ . The next result makes its cost explicit and reconciles in what sense model averaging can be beneficial. 3.3

Decomposing the Deployment Error

If the PFN training on Π does not perfectly align with the deployment distribution Π′ , this difference needs to be taken into account. To locate the deployment pairs (p, n) relative to the prior, the Lebesgue decomposition splits the deployment distribution uniquely into a part the prior can express Π̃, and a part Π′⊥ it cannot, Π′ = λ Π̃ + (1 − λ) Π′⊥ ,

Π̃ ≪ Π,

Π′⊥ ⊥ Π, (12)

with density r̃Π = dΠ̃/dΠ on the absolutely continuous part. The weight λ ∈ [0, 1] is the fraction of deployment pairs reachable by reweighting the prior. The singular part Π′⊥ lives on pairs to which Π assigns probability zero, and no reweighting reaches it. These include tasks the prior cannot generate as well as context sizes outside the pretraining range. As a baseline for the deployment risk, let HΠ′ := EΠ′ [− log p(Y | X)] be the expected conditional entropy of the deployment tasks, i.e., the risk of an oracle that knows each task p and predicts with p(y | x). The excess risk of a PFN over this oracle then splits as follows.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

TabICLv2

NanoTabPFN

JSD (nats)

10−2 10−3 103

104

0.1 0.0

105

Disagreement rate

0.2

10−1

Training scale M

103

David Rügamer

104

105

Training scale M

0.4

small

EE(̂ M)

100

Thomas Nagler

0.2

103

104

105

Training scale M

d d JSD , AU d dis ) Figure 3: Estimation error EE(M ) (log-log) on synthetic data and member disagreement on TabArena (AU versus the number of pretraining datasets M for the small models; the tiny models are shown in Fig. 10.

Proposition 2 (informal). Under mild regularity conditions, for every PFN q, RΠ′ (q) − HΠ′ = PΠ′ (Π) | {z } coverage h +λ AΠ (q) | {z }

amortization

+ CΠ + ΓΠ′ (q) |{z} | {z } context

+ (1 − λ) OΠ′ (q) . | {z }

i

transport

(13)

off-support

Coverage, amortization and context error are nonnegative; the transport and off-support terms are signindefinite. The formal statement with the definitions of all terms and its proof are given in Appendix B.2. The decomposition separates the deployment error of any PFN into three kinds of error with different levers (illustrated in Fig. 1), of which only one is controlled by pretraining. Context Error CΠ = RΠ (πΠ ) − HΠ is the excess risk of the exact PPD and does not involve the network: the price that even an exact Bayesian pays for a finite context. It is the object of Nagler and Rügamer (2025) and Ng et al. (2025), who separate it from aleatoric uncertainty via martingale posteriors (Fong et al., 2023), conditional on a fixed network. Prior Error Coverage, transport and off-support terms describe how the prior relates to the deployment distribution. Coverage is model-free and vanishes if and only if every deployment task can be expressed by the prior. The transport term is the covariance under Π between r̃Π and the PFN’s excess risk: it is positive when deployment overweights tasks on which the PFN is poor, so misplaced prior mass is not harmful per se. The off-support term is the PFN’s excess risk on Π′⊥ relative to the closest expressible task and can be negative when the network generalizes beyond its prior (Raven-

tós et al., 2023; Goddard et al., 2025). Quantifying these terms for PFNs remains open (Appendix A). Amortization Error The amortization error enters Eq. (13) unchanged: it is still evaluated under the training prior, weighted by the expressible share λ, and undefined on Π′⊥ , where there is no PPD to compare against. It remains the only term that pretraining controls. Importantly, under a deployment distribution Π′ ̸= Π, member disagreement no longer measures only the removable part EΠ [∆πΠ ] of the amortization error, since Corollary 1 identifies it only under the prior. On real data, Lemma 1 and the Lebesgue decomposition Eq. (12) combine into Corollary 2 (Appendix B.5): the log-loss gain of an ensemble over its average member splits into the removable amortization error of Corollary 1 and a sign-indefinite transport term, both weighted by λ, plus a non-negative off-support term. Disagreement on real data, and with it the gain of averaging, therefore mixes uncertainty left by pretraining with how the deployment distribution relates to the prior. Since pretraining does not constrain the members on tasks the prior cannot express, we expect disagreement to grow as deployment moves away from the prior’s support, e.g., for contexts beyond the pretraining range or datasets whose structure the prior does not generate.

4

BAYESIAN MODEL AVERAGES OVER PFN WEIGHTS

If part of the amortization error persists and is partially removable in the sense of Corollaries 1 and 2, i.e., if the outcome of pretraining is itself uncertain and deployment adds additional uncertainty, a Bayesian model average of the network weights is the natural remedy. A Tempered Posterior Over PFN Weights

Plac-

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? Table 1: Relative LPPD gain (%) of Bayesian model averages of TabICLv2 on 30 TabArena classification datasets at context sizes 1000 and 10000. DE : deep ensemble of 8 members (4 for the large model) over its average member. IVON : average over S posterior samples over the posterior mean of the same model. SG-MCMC : Bayesian deep ensemble over the DE it is warm-started from, sampled with pSMILE. ∗ : significant change, paired Wilcoxon over datasets, p < 0.05. context size 1000

context size 10000

Model

DE

IVON S=8

IVON S=64

SG-MCMC

DE

IVON S=8

IVON S=64

SG-MCMC

tiny small medium large

+3.10 +2.89 +2.67 +1.69

+2.38∗ +0.52∗ +0.42∗ +0.23∗

+2.48∗ +0.65∗ +0.44∗ +0.18∗

+0.27∗ +0.37∗ +0.07 +0.27∗

+2.65 +2.56 +2.82 +1.81

−0.13 −0.14 −0.31 −0.13

−0.24 −0.04 −0.14 −0.06

+0.35∗ +0.42∗ +0.08 +0.32∗

ing a Gaussian prior p(θ) = N (0, σ 2 I) on all weights and combining it with the PFN likelihood gives the posterior p(θ | TM ) ∝ exp(−M Lemp (θ)) p(θ), where TM denotes the collection of all pretraining datasets. Since M can be made arbitrarily large and has no direct meaning as a training dataset size, this posterior collapses onto minimizers of Eq. (4) as M → ∞. To retain meaningful uncertainty, we instead target the tempered posterior (Wenzel et al., 2020)
pβ (θ | TM ) ∝ exp − β Lemp (θ) p(θ), (14) in which β > 0 plays the role of an effective sample size and decouples posterior concentration from M . Equivalently, pβ is the unique minimizer over all distributions Q on Θ of the Gibbs variational objective EQ [Lemp (θ)] + β −1 DKL (Q ∥ p) (Knoblauch et al., 2022). In practice, its predictions are an average over S samples from (an approximation of) pβ , q̄(y | x, Dn ) = S1

PS

s=1 qθ (s) (y | x, Dn ).

(15)

We compare three approximations of pβ in Section 5.3. What Averaging Guarantees Applied with the true conditional p of a deployment task, Lemma 1 guarantees that the expected log loss of q̄ is never worse than that of the average member, and strictly better whenever the members disagree on that task, regardless of whether the task lies inside the support of the prior. On the expressible part it additionally reduces the amortization error by the removable part of Corollary 1. Moreover, for finite Y the disagreement metrics are computable from the members alone, and d JSD > 0 on a dataset certifies ∆p > 0 for every p AU with full support, without labels. It hence can serve as a label-free certificate of amortization uncertainty d JSD > 0 only certifies a strict gain of q̄ over (on Π′⊥ AU its average member as no πΠ exists).

5

EXPERIMENTAL STUDY

We now test the claims of Sections 3 and 4: Whether the removable part of the amortization error persists

under scaling (Section 5.2), and whether approximate Bayesian methods can improve PFNs under real-world task distributions (Section 5.3). 5.1

Setup

We focus our experiments on TabICLv2 at up to four different scales: (∼1.5M/2M/7.5M/9M). For some experiments we additionally consider nanoTabPFN (Pfefferle et al., 2025) at two sizes (∼1M/2M parameters). Both are pretrained on synthetic tasks from the TabICL prior (Qu et al., 2026) with up to 1024 rows, 100 features and 10 classes. Later, in Section 5.4, we extend the analysis to full-sized TabICLv2 models as well as regression. We mostly study TabICLv2 because of its greatly improved training efficiency compared to nanoTabPFN and because of its much more refined architecture. Every ensemble member, and every sampling chain, is trained on disjoint pretraining data. Depending on the experiment, we evaluate accuracy (ACC), ROC AUC and log pointwise predictive density (LPPD; Gelman et al., 2014) on a large synthetic test set drawn from the TabICL prior or real-world classification datasets from TabArena (Erickson et al., 2025) as well as TALENT (Liu et al., 2025), potentially filtered to the scope of the models. Details on implementation, architectures, and optimization are given in Appendix E. 5.2

Disagreement Along Scaling Axes

d Data Axis At fixed model size, we compute EE(M ) in Eq. (9) and both disagreement metrics across K=4 members for the two smallest model sizes, keeping the number of training steps fixed. The two behave d differently (Fig. 3): EE(M ) decreases steadily over the whole range for the small models of both architectures. d A log-log fit over all M gives EE(M ) ∝ M −α with α ∈ [0.86, 1.14] across architectures and sizes, consistent with the expected 1/M rate. Disagreement falls quickly for small M but flattens beyond M ≈ 104 . For larger training scale, the disagreement rate decreases only from 9.1% to 7.6% for TabICLv2 and from 6.7% to d JSD also reduces by at most 3.7% for nanoTabPFN. AU a factor of three. The tiny models and the synthetic test

Small

Medium

↑

0.835 0.830

AUROC

Accuracy

↑

0.840

0.825

−0.37

0.81

−0.38

0.80 0.79

0.77 5000

Context size (rows)

10000

David Rügamer

Single chain (mean)

0.82

0.815 2000

Deep ensemble −0.36

0.78 1000

Large

Thomas Nagler

0.83

0.820

500

Emanuel Sommer

↑

Tiny

Julius Kobialka

LPPD

Manuel Hülskamp

−0.39 −0.40 −0.41

500

1000

2000

5000

Context size (rows)

10000

−0.42

500

1000

2000

5000

Context size (rows)

10000

Figure 4: Classification test performance measured by accuracy, ROC AUC and LPPD on the TabArena dataset suite across a sweep of the maximal number of context rows for deep ensembles of TabICLv2 across a range of model sizes. The corresponding results for nanoTabPFN are shown in Fig. 13.

set show the same pattern (Figs. 10 and 11). Within the scanned range, additional pretraining data therefore mainly reduces the estimation error, while a small but persistent share of the disagreement remains. Capacity Axis As shown in Table 6 in the Appendix, disagreement also decreases with capacity, but d only slowly compared to the 30–50-fold drop of EE along the data axis. Increasing the model parameters d JSD on TabArena by 41% from 1.5M to 9M reduces AU d and AU dis from 7.5% to 5.9%. Contexts beyond the pretraining range do not reduce the disagreement rate either, and the full-size ensembles of Section 5.4 still gain over their members. Capacity thus likely mainly lowers the error shared by all members, consistent with a smaller approximation error, whereas the removable part on prior data changes less. Reconciling Results Section 2.3 shows that a PFN closely tracks the PPD it is supposed to amortize. Section 5.2, on the other hand, finds disagreement among K independently trained PFNs on a (small) fraction of predictions. At first glance, this might seem contradictory. In both cases, however, the differences are caused by a small fraction of test rows: the difference to the MCMC reference concentrates on a small share of inputs (Appendix C), and so does the averaging gain between independently trained members, where little gain falls on confidently predicted test points (Fig. 12). The test points with the largest averaging gains are plausibly those that are rare under the prior and hence contribute little to the pretraining objective, so that more pretraining or larger models barely reduces the uncertainty for these predictions. Averaging over independently pretrained models addresses exactly this residual. Under deployment, Corollary 2 adds a transport and a non-negative off-support term, leading to

a higher disagreement on datasets outside the prior’s support as empirically confirmed in Appendix F.9. 5.3

Bayesian Model Averages of PFNs

We approximate tempered posteriors of the form (14), with temperature and prior set per method, using three families of methods that differ in which part of the posterior they explore. All yield parameter samples that are averaged via Eq. (15); implementation details and hyperparameters are given in Appendix E.2. Deep Ensembles Deep ensembles (DEs; Lakshminarayanan et al., 2017) train K networks independently. These act as draws from distinct modes of pβ and thus capture uncertainty between basins (Fort et al., 2019). Deep ensembles improve on their average member in ACC, ROC AUC and LPPD at every model size and context size (Fig. 4), by 1.7–3.1% LPPD within and 1.8–2.7% beyond the pretraining range (Table 1). At 1000 rows, an ensemble of one size roughly matches a single member of the next larger size. Within-Basin Methods SG-MCMC and variational learning, instead, are more likely to capture uncertainty within a basin. We run the samplers adaSGHMC (Chen et al., 2014; Springenberg et al., 2016) and pSMILE (Sommer et al., 2026), starting chains from each DE member (Sommer et al., 2024), which forms a Bayesian deep ensemble (Sommer et al., 2025). We further train IVON (Shen et al., 2024), which fits a diagonal Gaussian to pβ . Both within-basin methods recover part of the removable error, but less compared to the DE. IVON improves on its posterior mean within the pretraining range at every size, by 0.2–2.5%, shrinking with capacity and saturating by S ≈ 8 (Appendix F.5). Beyond it, no number of samples helps (Table 7). SG-MCMC chains (pSMILE) warm-started

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? deep ensemble (8 members)

(a) TALENT classif.: LPPD ↑

average member

best member

(b) TALENT regression: nRMSE ↓

(c) TabArena classif.: Elo ↑ TabPFN-2.6

−0.360

TabICLv2 (official)

0.480

−0.362

DE-8 (ours)

−0.365

0.470

RealTabPFN-2.5

0.460

CatBoost (tuned+ens.)

TabDPT (tuned+ens.)

−0.367 −0.370

members (ours) 1 ≤ 1k

2 ≤ 10k

3 ≤ 60k

Training stage (max. rows)

1 ≤ 1k

2 ≤ 10k

3 ≤ 60k

Training stage (max. rows)

1400

1500

1600

Elo (95% CI)

Figure 5: Deep ensembles of full-size TabICLv2. (a, b) LPPD and nRMSE on TALENT datasets after each training stage: ensemble of 8 members, average member and best member (selected after the fact). The gap between ensemble and average member persists while all members improve. (c) TabArena classification Elo (95% CI) of the single members (dots), the 8-member ensemble, the official TabICLv2 checkpoint and further published methods. Note that the individual DE members are trained for only 27.5% of the steps that official TabICLv2 uses.

from the DE add up to 0.42% on top of it at both context sizes. adaSGHMC fails to provide reliable additional gains despite an extensive hyperparameter search (Appendix F.6). Within-basin methods therefore can capture residual amortization uncertainty, but a plain DE delivers most of the attainable gain at matched inference cost, needs no tempering or sampler hyperparameters, and its members train independently. 5.4

Deep Ensembles of Full-Size TabICLv2

Since DEs carry most of the gain, we test whether these findings persist at the scale of current state-of-the-art PFNs. We pretrain K=8 full-size TabICLv2 models (27.5M parameters) each for classification and regression, with the three-stage curriculum and preprocessing pipeline of Qu et al. (2026) but 27.5% of its training steps (details and extended results in Appendix F.7). After every training stage, the ensemble improves on its average member, and it does so on all 38 classification and 13 regression datasets of TabArena. For classification, it is also better than the best single member. The same also holds true for evaluation on TALENT (see Table 10). Giving each member different preprocessing configurations leaves the ensemble unchanged, so the gain stems from the independently pretrained weights. With a TabArena classification Elo of 1552, the ensemble lies within the confidence interval of the official TabICLv2 checkpoint (1575), while its members never communicate and can be trained in parallel in about 0.37× the official wall-clock time.

6

DISCUSSION

PFNs amortize the PPD of their prior remarkably well up to small deviations, which are confined to inputs that

are rare under the prior. For production priors without a reference, only the removable part of the amortization error is observable, and it shows the same picture: While the estimation error vanishes at the expected rate, a small but consistent disagreement between independently pretrained PFNs remains, concentrated on few inputs and barely reduced by more pretraining data or capacity. Model averaging addresses exactly this remaining uncertainty, and deep ensembles of PFNs deliver most of the attainable gain, while within-basin methods like SG-MCMC and variational inference provide little additional benefit. At full scale, an ensemble of independently pretrained TabICLv2 models matches the official checkpoint at a fraction of its wall-clock time, with the deep ensemble outperforming the best post-hoc selected single model on TabArena as well as TALENT. For practitioners, this suggests a simple recipe: pretraining several members in parallel instead of a single model yields better predictions and makes the amortization error partially measurable. Our analysis has limitations. The amortization gap is measured against finite-budget MCMC, not the exact PPD, and some studies use smaller models. Furthermore, while we expect our results to hold across the whole range of state-of-the-art PFN architectures, we empirically validate our claims for TabICLv2 and nanoTabPFN only. Next steps are studying adjustments to the pretraining pipelines to improve the amortization at the tails of the prior as well as investigating how to integrate the disagreement metrics as uncertainty signals at deployment.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

AI Use Statement In this work, we used generative AI tools to support the implementation of the presented methods in code and to draft and review mathematical proofs. We have not used generative AI tools to formulate research hypotheses and develop the theoretical and conceptual framework. Furthermore, other required disclosures such as data cleaning or data analysis are not applicable to this work. Additionally, we used generative AI tools to assist in the writing and presentation of this manuscript by improving readability and the presentation of the included figures. Furthermore, generative AI tools were used to explore and identify relevant literature in addition to our initial manual survey. We have reviewed all AI-assisted work. AI-generated code was reviewed manually as well as tested in detail. All modifications of the manuscript suggested by AI tools were carefully inspected. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Acknowledgements This work was supported by the Bavarian AI Foundation Model Initiative (AI-Bay), funded by the Bavarian State Ministry of Science and the Arts. The authors gratefully acknowledge the computational resources and support provided by the Erlangen National High Performance Computing Center (NHR@FAU) at FriedrichAlexander-Universität Erlangen-Nürnberg (FAU). Furthermore, the authors gratefully acknowledge the scientific support and resources of the AI service infrastructure LRZ AI Systems provided by the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities (BAdW), funded by the Bayerisches Staatsministerium für Wissenschaft und Kunst (StMWK), primarily through the BayernKI initiative. References Vahid Balazadeh Meresht, Hamidreza Kamkari, Valentin Thomas, Junwei Ma, Bingru Li, Jesse Cresswell, and Rahul Krishnan. CausalPFN: Amortized Causal Effect Estimation via In-Context Learning. Advances in Neural Information Processing Systems, 38, 2025. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. JAX: composable transformations of Python+ NumPy programs, 2018. URL https://github.com/jax-ml/ jax. Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-

Thomas Nagler

David Rügamer

Li Meng. Handbook of Markov Chain Monte Carlo. CRC Press, 2011. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerbertVoss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020. Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning, pages 1683–1691. PMLR, 2014. Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, Hoboken, NJ, second edition, 2006. ISBN 978-0471241959. Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The Frontier of Simulation-based Inference. Proceedings of the National Academy of Sciences, 117(48): 30055–30062, 2020. Francesco D’Angelo and Vincent Fortuin. Repulsive Deep Ensembles are Bayesian. Advances in Neural Information Processing Systems, 34:3451–3465, 2021. Alex O. Davies, Telmo de Menezes e Silva Filho, and Nirav Ajmeri. Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models. arXiv preprint arXiv:2605.06343, 2026. Felix den Breejen, Sangmin Bae, Stephen Cha, and Se-Young Yun. Fine-tuned in-context learning transformers are excellent tabular data classifiers. arXiv preprint arXiv:2405.13396, 2024. Moonjung Eo, Min-Kook Suh, Hye-Seung Cho, Jiwon Kim, Seoyoon Kim, Sangjun Nam, and Soonyoung Lee. EXAONE Tabular 1.0: Technical report. arXiv preprint arXiv:2608.25774, 2026. Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. arXiv preprint arXiv:2506.16791, 2025. Fabian Falck, Ziyu Wang, and Chris Holmes. Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, 2024.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale Posterior Distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5):1357–1391, 2023.

TabPFN-3.5: Technical report. arXiv:2609.17895, 2026.

arXiv preprint

Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A Loss Landscape Perspective. arXiv preprint arXiv:1912.02757, 2019.

Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github. io/posts/muon/, 2024.

Sandra Fortini, Kenyon Ng, Sonia Petrone, Judith Rousseau, and Susan Wei. Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inference. arXiv preprint arXiv:2602.04596, 2026.

Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on Bayes’ rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23 (132):1–109, 2022.

Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real-TabPFN: Improving tabular foundation models via continued pre-training with real-world data. arXiv preprint arXiv:2507.03971, 2025.

Julius Kobialka, Emanuel Sommer, Chris Kolb, Juntae Kwon, Daniel Dold, and David Rügamer. On the interplay of priors and overparametrization in Bayesian neural network posteriors. In Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research. PMLR, 2026.

Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6): 997–1016, 2014. Chase Goddard, Lindsay M. Smith, Vudtiwat Ngampruetikorn, and David J. Schwab. When Can InContext Learning Generalize out of Task Distribution? In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 19585– 19599. PMLR, 2025. Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, et al. TabPFN-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667, 2025. Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023. Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. Benjamin Jäger, Nick Erickson, Léo Grinsztajn, Felix Birkel, Klemens Flöge, Oscar Key, Kürşat Kaya, Jonas Kübler, Adèle Frankel, Tobias Schröder, et al.

Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, and Abhimanyu Das. TabFM: A zero-shot foundation model for tabular data. arXiv preprint arXiv:2609.37959, 2026. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017. J. Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151, 1991. Si-Yang Liu and Han-Jia Ye. TabPFN unleashed: a scalable and effective solution to tabular classification problems. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25, 2025. Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, Huai-Hong Yin, Tao Zhou, Jun-Peng Jiang, and Han-Jia Ye. TALENT: A tabular analytics and learning toolbox. Journal of Machine Learning Research, 26(226):1–16, 2025. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Jesse Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L Caterini, and Maks Volkovs. TabDPT: Scaling tabular foundation models on real data. Advances in Neural Information Processing Systems, 38, 2025. Yuchen Ma, Dennis Frauen, Emil Javurek, and Stefan Feuerriegel. Foundation models for causal inference

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

via prior-data fitted networks. In International Conference on Learning Representations, volume 2026, pages 79065–79098, 2026. Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. arXiv preprint arXiv:2112.10510, 2021. Samuel Müller, Noah Hollmann, and Frank Hutter. Bayes’ power for explaining in-context learning generalizations. arXiv preprint arXiv:2410.01565, 2024. Thomas Nagler. Statistical foundations of prior-data fitted networks. In International Conference on Machine Learning, pages 25660–25676. PMLR, 2023. Thomas Nagler and David Rügamer. Uncertainty quantification for prior-data fitted networks using martingale posteriors. arXiv preprint arXiv:2505.11325, 2025. Kenyon Ng, Edwin Fong, David T Frazier, Jeremias Knoblauch, and Susan Wei. TabMGP: Martingale posterior with TabPFN. arXiv preprint arXiv:2510.25154, 2025. Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, José Miguel Hernández-Lobato, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position: Bayesian deep learning is needed in the age of largescale AI. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39556– 39586. PMLR, 21–27 Jul 2024. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019. Alexander Pfefferle, Johannes Hog, Lennart Purucker, and Frank Hutter. nanoTabPFN: A lightweight and educational reimplementation of TabPFN. arXiv preprint arXiv:2511.03634, 2025. Pascal Pfeiffer, Dmitry Gordeev, Mathias Müller, Laura Fink, Joan Salvà Soler, Mark Landry, Branden Murray, Marcos V. Conde, and Sri Satish Ambati. TabH2O: A unified foundation model for tabular prediction. arXiv preprint arXiv:2605.18383, 2026.

Thomas Nagler

David Rügamer

Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, and Frank Hutter. Beyond IID: How general are tabular foundation models, really? arXiv preprint arXiv:2606.30410, 2026. Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: a tabular foundation model for in-context learning on large data. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25, 2025. Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In Fortythird International Conference on Machine Learning, 2026. Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36:14228–14246, 2023. Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Schölkopf. DoPFN: In-context learning for causal effect estimation. Advances in Neural Information Processing Systems, 38:174811–174848, 2025. Jakob Robnik and Uros Seljak. Fluctuation without dissipation: Microcanonical Langevin Monte Carlo. In Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, volume 253 of Proceedings of Machine Learning Research, pages 111–126. PMLR, 21 Jul 2024. Marvin Schmitt, Paul-Christian Bürkner, Ullrich Köthe, and Stefan T Radev. Detecting model misspecification in amortized Bayesian inference with neural networks. In DAGM German Conference on Pattern Recognition, pages 541–557. Springer, 2023. Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024. Emanuel Sommer, Lisa Wimmer, Theodore Papamarkou, Ludwig Bothmann, Bernd Bischl, and David Rügamer. Connecting the dots: Is modeconnectedness the key to feasible sample-based inference in Bayesian neural networks? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 45988–46018. PMLR, 21–27 Jul 2024.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Emanuel Sommer, Jakob Robnik, Giorgi Nozadze, Uros Seljak, and David Rügamer. Microcanonical Langevin ensembles: Advancing the sampling of Bayesian neural networks. In International Conference on Learning Representations, volume 2025, pages 15101–15122, 2025. Emanuel Sommer, Kangning Diao, Jakob Robnik, Uros Seljak, and David Rügamer. Can microcanonical Langevin dynamics leverage mini-batch gradient noise? In Forty-third International Conference on Machine Learning, 2026. Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust Bayesian neural networks. Advances in Neural Information Processing Systems, 29, 2016. Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony Caterini. Retrieval & fine-tuning for incontext tabular models. Advances in Neural Information Processing Systems, 37:108439–108467, 2024. Zeynep Türkmen, Kürşat Kaya, Alexander Pfefferle, and Frank Hutter. Towards evaluating data priors for tabular foundation models. arXiv preprint arXiv:2606.29241, 2026. Penghui Wang, Wei Liu, Hong Wang, Chengyue Huang, Yuxi Sun, Zirui Wang, Hongming Huang, Quan Wang, Chunxiao Liu, et al. Xiaomi-TabLDM: A tabular foundation model technical report. arXiv preprint arXiv:2609.03880, 2026. Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 681–688, 2011. Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In Proceedings of the 37th International Conference on Machine Learning, ICML’20, 2020. Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer. Towards efficient MCMC sampling in Bayesian neural networks by exploiting symmetry. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 459–474. Springer, 2023. Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. Mixture of in-context prompters for tabular PFNs. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 24630–24665, 2025.

Xingxuan Zhang, Gang Ren, Han Yu, Hao Yuan, Hui Wang, Jiansheng Li, Jiayun Wu, Lang Mo, Li Mao, Mingchao Hao, et al. LimiX: Unleashing structureddata modeling capability for generalist intelligence. arXiv preprint arXiv:2509.03505, 2025a. Xiyuan Zhang, Danielle Maddix Robinson, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael Mahoney, et al. Mitra: Mixed synthetic priors for enhancing tabular foundation models. Advances in Neural Information Processing Systems, 38: 15795–15840, 2025b. He Zhao, Ryan Thompson, Daniel M Steinberg, Ashfaqur Rahman, Edwin V Bonilla, and Cheng Soon Ong. From synthetic priors to model behavior: Structural coverage in tabular foundation models. arXiv preprint arXiv:2609.06912, 2026.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

CHECKLIST 1. For all models and algorithms presented, check if you include: (a) A clear description of the mathematical setting, assumptions, algorithm, and/or model. [Yes, Sections 2 to 4 as well as Appendices B and E] (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. [Not Applicable] (c) (Optional) Anonymized source code, with specification of all dependencies, including external libraries. [Yes, Appendix E.4] 2. For any theoretical claim, check if you include: (a) Statements of the full set of assumptions of all theoretical results. [Yes, Appendix B] (b) Complete proofs of all theoretical results. [Yes, Appendix B] (c) Clear explanations of any assumptions. [Yes, Appendix B] 3. For all figures and tables that present empirical results, check if you include: (a) The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL). [Yes, Appendix E.4] (b) All the training details (e.g., data splits, hyperparameters, how they were chosen). [Yes, Appendix E] (c) A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times). [Yes, Appendices E and F and Section 5] (d) A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider). [Yes, Appendices E and F.7] 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator If your work uses existing assets. [Yes, Appendix E] (b) The license information of the assets, if applicable. [Not Applicable, all code used is open-source under common licenses that allow usage and extension.] (c) New assets either in the supplemental material or as a URL, if applicable. [Not Applicable] (d) Information about consent from data providers/curators. [Not Applicable]

Thomas Nagler

David Rügamer

(e) Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. [Not Applicable] 5. If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to participants and screenshots. [Not Applicable] (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. [Not Applicable] (c) The estimated hourly wage paid to participants and the total amount spent on participant compensation. [Not Applicable]

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? Supplementary Materials

A

RELATED WORK

Prior-Data Fitted Networks Müller et al. (2021) introduced PFNs as transformers trained on synthetic data to approximate Bayesian posterior predictive distributions via in-context learning, with Nagler (2023) providing statistical foundations by showing that PFNs admit a frequentist ERM interpretation. This line of work has since catalysed a rapidly growing family of tabular foundation models (Hollmann et al., 2025; Grinsztajn et al., 2025; Garg et al., 2025; Kong et al., 2026; Pfeiffer et al., 2026; Jäger et al., 2026; Wang et al., 2026; Eo et al., 2026; Ma et al., 2025; Zhang et al., 2025a,b). Importantly, TabICL (Qu et al., 2025, 2026) provides a fully open-source alternative with strong scaling to large datasets. Taken together, these works establish in-context learning over synthetic priors as the dominant paradigm for tabular foundation models. Uncertainty in PFNs Recent works address uncertainty in PFN predictions at inference time. Nagler and Rügamer (2025) construct Bayesian posteriors via martingale posterior sampling; Fortini et al. (2026); Ng et al. (2025) prove an aleatoric/epistemic decomposition via a predictive CLT. Fortini et al. (2026) additionally argue that weight-space approaches to PFN uncertainty constitute a “category error,” since they concern the transformer’s meta-parameters rather than the task at hand. We believe this characterization conflates two orthogonal levels of uncertainty: Context uncertainty is conditional on a fixed predictor, while the amortization error, and in particular its removable part (Corollary 1), concerns the predictor itself, which stochastic pretraining makes random. Context-level methods cannot register cross-warmstart disagreement as uncertainty, yet our results will show this signal is measurable, persistent, and task-relevant. Closely related to the PFN literature is the stream of work focusing on simulation-based inference (SBI; Cranmer et al., 2020), which has also focused on the uncertainty in its approaches. Notably, the simulation gap problem, that is, prior misspecification degrading amortized approximations (Schmitt et al., 2023), corresponds to the prior error in our decomposition and is discussed in Section 3.3. Bayesian Deep Learning Bayesian deep learning maintains distributions over network weights for uncertainty quantification (Papamarkou et al., 2024). We build on SG-MCMC methods such as SGHMC (Chen et al., 2014; Springenberg et al., 2016) and the recent MILE/SMILE samplers (Sommer et al., 2025, 2026), as well as deep ensembles (Lakshminarayanan et al., 2017), which Fort et al. (2019) attribute to exploration of distinct loss basins and D’Angelo and Fortuin (2021) connect to a Wasserstein gradient-flow posterior. Variational learning with IVON (Shen et al., 2024) scales natural-gradient variational inference to large networks. All three are instantiated here over PFN weights, targeting a tempered posterior (Wenzel et al., 2020) to prevent collapse at large meta-training scale M . Are PFNs and In-Context Learners Bayesian? Whether trained in-context learners implement the Bayes rule of their pretraining distribution has been studied mostly outside tabular data. Raventós et al. (2023); Goddard et al. (2025) show for linear regression that transformers pretrained on few tasks match the Bayes rule of that finite task set, and above a task-diversity threshold instead match the Bayes rule of the wider task family, i.e., they depart from the pretraining prior. Falck et al. (2024) test the martingale property that every Bayesian predictive satisfies and find violations in large language models. For PFNs, Müller et al. (2021) compare against exact or sampling-based references for Gaussian-process and small Bayesian-neural-network priors. The concrete evidence that PFNs track the posterior predictive they are supposed to amortize comes mainly from Müller et al. (2021, 2024). Unfortunately, the released code of Müller et al. (2021) covers only part of their experiments and, to our understanding, contains inconsistencies that narrow what some of the experiments show, which is one of the reasons why we revisit and extend their experiments in this work.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

Evaluating Synthetic Priors Several recent works try to score synthetic priors without pretraining, by relating distances between synthetic and real datasets in (summary) dataset space to downstream performance (Türkmen et al., 2026; Davies et al., 2026; Zhao et al., 2026), with limited success. In the terms of Proposition 2, they target the prior error, where the decomposition makes explicit that it consists of a coverage term, a transport term, and an off-support term, which respond to different interventions.

B

PROOFS

Throughout, Π and Π′ are laws over pairs (p, n) of a task and a context size, and Dn denotes a context of realized size n. Quantities indexed by a task, such as Ep , Ep⋆ and ap below, are evaluated at the pair drawn with p. We collect the standing assumptions. Assumption 1 (Finiteness). All cross-entropies appearing below are finite. Assumption 2 (Projection). For every task p, the I-projection p⋆ ∈ arg minp̃∈S Ep [DKL (p(· | X) ∥ p̃(· | X))] exists and depends measurably on p, with p⋆ = p for p ∈ S. Assumption 3 (Support). Π(S × N) = 1 for S the support of the task marginal of Π. For a prior Π′ over pairs, the joint law of (p, N, Dn , X, Y ) is dΠ′ (p, n) p⊗n (dDn ) pX (dX) p(Y | X) dY ; we write m̄Π′ for the induced law of the observables (X, Dn ). We use three elementary facts. (F1) Label laws. Given (p, X, Dn ), the label has density p(· | X). Given (X, DRnR), it has density πΠ′ (· | X, Dn ): ′ by R the tower property over p and Tonelli’s theorem, P(Y ∈ C | X, Dn ) = C p(y | X) dy dΠ (p | X, Dn ) = π ′ (y | X, Dn ) dy for measurable C ⊆ Y. C Π (F2) Log-loss gaps are expected divergences. If Y has density u(· | Z) given Z and log u(Y | Z) is integrable, then for every predictive density w, conditioning on Z gives   
 E log u(Y | Z) − log w(Y | Z) = E DKL u(· | Z) ∥ w(· | Z) ∈ [0, ∞], which vanishes if and only if w(· | Z) = u(· | Z) almost surely. (F3) Change of measure. If ν ≪ µ are probability measures with density r, and f and rf are µ-integrable, then Eν [f ] = Eµ [rf ] = Eµ [f ] + Covµ (r, f ), since Eµ [r] = 1. B.1

Proof of Proposition 1

Proof. By (F1), Y has density πΠ′ (· | X, DN ) given (X, DN ), and log πΠ′ (Y | X, DN ) is integrable because RΠ′ (πΠ′ ) is finite. Applying (F2) with Z = (X, Dn ), u = πΠ′ and w = q gives     RΠ′ (q) − RΠ′ (πΠ′ ) = EΠ′ log πΠ′ (Y | X, DN ) − log q(Y | X, DN ) = EΠ′ DKL (πΠ′ ∥ q) ≥ 0, with equality if and only if q = πΠ′ for m̄Π′ -almost every (X, Dn ). Hence RΠ′ (q) ≥ RΠ′ (πΠ′ ) for every predictor q, and any q attaining this bound equals πΠ′ m̄Π′ -almost surely, i.e., πΠ′ is the minimizer of RΠ′ , unique up to m̄Π′ -null sets. B.2

Formal Statement and Proof of Proposition 2

For a ground-truth task p, let Hp := Ep [− log p(Y | X)] be its conditional entropy and 
 Ep (qθ ) := Ep DKL p(· | X) qθ (· | X, Dn )

(16)

its excess risk, so that RΠ′ (qθ ) − HΠ′ = EΠ′ [Ep (qθ )] with HΠ′ := EΠ′ [Hp ] by (F1) and (F2). Let S be the support of the task marginal of Π (Assumption 3) and let p⋆ be the I-projection of p onto S (Assumption 2), the task in the prior’s support closest to p in expected conditional KL. It defines the coverage error and the signed excess risk relative to p⋆ , 
 PΠ′ (Π) := EΠ′ DKL p(· | X) ∥ p⋆ (· | X) ≥ 0, (17)   ⋆ ⋆ Ep (qθ ) := Ep log p (Y | X) − log qθ (Y | X, Dn ) . (18)

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? (a) Error of a single PFN Coverage PΠ′ (Π) Transport λ ΓΠ′ (qθ )

Prior error

Off-support (1 − λ) OΠ′ (qθ ) Optimization RΠ (qθ̂ ) − RΠ (qθ̃M ) Deployment error RΠ′ (qθ ) − HΠ′

×λ

Amortization error AΠ (qθ )

Estimation RΠ (qθ̃M ) − RΠ (qθ∗ )

vs. reference PPD (Sec. 2.3)

d EE(M ) (Sec. 5.2)

Approximation RΠ (qθ∗ ) − RΠ (πΠ ) × λ Context error CΠ (b) Averaging K PFNs q1 , . . . , qK into their mixture q̄ Ensemble error RΠ′ (q̄) − HΠ′

Average member error P 1 ′ ′ k RΠ (qk ) − HΠ K

Shared amortization AΠ (q̄), decomposed as in (a) not removed by averaging

Removable amortization   λ E Π ∆ πΠ ≥ 0 Ensemble gain EΠ′ [ap ] LPPD gain; label-free: d JSD AU

Transport λ CovΠ (r̃Π , ap ) Off-support (1 − λ) EΠ′⊥ [ap ] ≥ 0

Figure 6: Error decompositions used in this paper. (a) The deployment error of a PFN splits into prior, amortization and context error (Proposition 2); the latter two enter with the expressible share λ, and the prior error comprises the coverage, transport and off-support terms. For the trained network qθ̂ , the amortization error splits further into optimization, estimation and approximation error (Eq. (8)). (b) For K PFNs, the error of the average member splits into the error of their mixture q̄, whose amortization error is the part shared by all members (Corollary 1), and the gain of averaging, which splits into the removable amortization error and the transport and off-support terms of Corollary 2.

Proposition 3 (Deployment decomposition, formal version of Proposition 2). Under Assumptions 1 to 3, and if r̃Π Ep (q) is Π-integrable, Eq. (13) holds for every PFN q, where AΠ (q) is the amortization error Eq. (5), PΠ′ (Π) the coverage error Eq. (17),
ΓΠ′ (q) := CovΠ r̃Π , Ep (q) ,

  OΠ′ (q) := EΠ′⊥ Ep⋆ (q)

(19)

the transport and off-support terms, and 
 CΠ := RΠ (πΠ ) − HΠ = EΠ DKL p(· | X) ∥ πΠ (· | X, DN ) ≥0 the context error.

(20)

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

Before moving to the proof, the following paragraph highlights the intuition and relevance of each of the five terms in Proposition 2: • Coverage PΠ′ (Π) depends on Π and Π′ only: it is the misspecification of the prior, measured by the KL divergence to the best expressible task. Importantly, it only compares the conditionals of Y given X, not the covariate law alone. Hence, a task whose covariate law the prior never generates, but whose conditional it does, incurs no coverage error. • Amortization AΠ (q) is the only term the pretraining objective targets: it vanishes if and only if q = πΠ (Proposition 1). ΓΠ′ and OΠ′ also depend on q, but they are not controlled by the objective as they also depend on the deployment distribution, which is unknown at training time. It is evaluated under Π, not Π̃, the reweighting towards deployment is accounted for in ΓΠ′ . • Context CΠ : by (F1), RΠ (πΠ ) and HΠ are the conditional entropies of Y given (X, DN ) and given (p, X, DN ), so CΠ = I(Y ; p | X, DN ), the information the unknown task still carries about the label. It is a property of Π alone and does not depend on the PFN q. Its decay in the context size n measures how quickly the prior’s tasks are identified from data, i.e., how hard the prior is independently of the architecture. • Transport ΓΠ′ (q) measures whether deployment favours tasks or context sizes on which q does well or poorly. If q is equally good on all pairs, it vanishes, however strongly Π′ reweights the prior. It is generally non-zero even for the exact PPD q = πΠ , since some tasks are harder to identify from n points than others. • Off-support OΠ′ (q) compares q with the best expressible task p⋆ on pairs to which the prior assigns no mass, i.e., inexpressible tasks or context sizes outside the pretraining range. Where their observables (X, Dn ) fall outside the support of m̄Π , πΠ is not determined and the pretraining objective does not constrain q, so the term is governed by the inductive bias of the architecture. OΠ′ (q) < 0 means the network generalizes beyond its prior, as observed for in-context learners pretrained on sufficiently diverse tasks (Raventós et al., 2023; Goddard et al., 2025). Proof. Let cp := Ep [DKL (p(· | X) ∥ p⋆ (· | X))], so PΠ′ (Π) = EΠ′ [cp ]. Splitting log p − log q = (log p − log p⋆ ) + (log p⋆ − log q) in Eq. (16) gives, for every task p, Ep (q) = cp + Ep⋆ (q),

(21)

with all terms finite by Assumption 1. Since Π̃ ≪ Π and Π(S × N) = 1, Π̃-almost every pair (p, n) has p ∈ S and thus has p⋆ = p, cp = 0 and Ep⋆ = Ep (Assumption 2). Hence PΠ′ (Π) = (1 − λ) EΠ′⊥ [cp ], and integrating Eq. (21) against Eq. (12) gives RΠ′ (q) − HΠ′ = λ EΠ̃ [Ep (q)] + PΠ′ (Π) + (1 − λ) OΠ′ (q). By (F3), EΠ̃ [Ep (q)] = EΠ [Ep (q)] + ΓΠ′ (q), and     EΠ [Ep (q)] = RΠ (q) − HΠ = RΠ (q) − RΠ (πΠ ) + RΠ (πΠ ) − HΠ = AΠ (q) + CΠ , which proves Eq. (13). Finally, AΠ (q) ≥ 0 by Proposition 1, and (F2) with Z = (p, X, Dn ), u = p(· | X) and w = πΠ gives CΠ = EΠ [DKL (p(· | X) ∥ πΠ (· | X, Dn ))] ≥ 0. B.3

Proof of Lemma 1

Proof. For every y with p(y) > 0,    P  P 1 1 k log p(y) − log qk (y) − log p(y) − log q̄(y) = log q̄(y) − K k log qk (y). K Taking Ey∼p gives Eq. (10), all terms being finite by Assumption 1. By strict concavity of log, the right-hand side is non-negative, with equality if and only if q1 (y) = · · · = qK (y). Hence ∆p ≥ 0, with equality if and only if the qk agree p-almost everywhere. B.4

Proof of Corollary 1

Proof. By Proposition 1 with Π′ = Π, which holds for every predictor, AΠ (q) = EΠ [DKL (πΠ ∥ q)] for each qk and for q̄. Apply Lemma 1 at each (X, Dn ) with p = πΠ (· | X, Dn ) and take EΠ . All terms are non-negative, so the expectation splits.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

B.5

Ensemble gain under the deployment distribution

  Corollary 2. Let q1 , . . . , qK be PFNs with mixture q̄ and, for a pair (p, n), let ap := Ep ∆p(·|X) , where ∆ from Lemma 1 is evaluated at the members’ predictions qk (· | X, Dn ). If all risks are finite and r̃Π ap is Π-integrable, then 1 K

P

k RΠ (qk ) − RΠ (q̄) = EΠ [ap ] ′

′

′

i h   = λ EΠ ∆πΠ + CovΠ (r̃Π , ap ) {z } | {z } | removable

transport

+ (1 − λ) EΠ′⊥ [ap ], | {z }

(22)

off-support

where the removable term is that of Corollary 1 and λ, r̃Π , Π′⊥ are as in Eq. (12). P P 1 1 ′ ′ Proof. Let g := log q̄(Y | X, Dn ) − K k log qk (Y | X, Dn ). Since all risks are finite, K k RΠ (qk ) − RΠ (q̄) = EΠ′ [g]. By (F1), E[g | p, X, Dn ] = ∆p(·|X) , so EΠ′ [g] = EΠ′ [ap ], which is the first equality. Equation (12) and (F3) give
EΠ′ [ap ] = λ EΠ [ap ] + CovΠ (r̃Π , ap ) + (1 − λ) EΠ′⊥ [ap ]. Finally, under Π, conditioning g on (X, Dn ) instead and using (F1) gives E[g | X, Dn ] = ∆πΠ (·|X,Dn ) , so EΠ [ap ] = EΠ [g] = EΠ [∆πΠ ].

C

A TRACTABLE SYNTHETIC PRIOR

The priors used in state-of-the-art PFNs are based on structural causal models, where random nodes and edges (together with functions they compute) are drawn according to some predefined scheme that together define a data-generating process with a certain underlying causal (latent) structure. To generate the feature and target observations, noise is then passed through this graph and the resulting feature and target nodes are appended to form a dataset from the prior (Qu et al., 2025, 2026; Grinsztajn et al., 2025). Let ξ collect the SCM parameters, i.e. the graph, the node functions, the response node and the assignment of nodes to features. Each ξ defines a task pξ , and the posterior Π(· | Dn ) is over ξ. MCMC methods evaluate the likelihood p(Dn | ξ) and the prior density of ξ pointwise, up to a common constant (Brooks et al., 2011). An SCM prior defines pξ only implicitly, as the pushforward of the exogenous noise through the structural equations. The following four properties of current SCM priors prevent the direct evaluation using MCMC methods: • Latent nodes. Only a subset of the nodes enters the dataset as features and label. With latent nodes U and rows conditionally independent given ξ, the likelihood that marginalizes over these latent nodes has no closed form for nonlinear node functions. • Degenerate and discontinuous likelihood. Discretizing a node into a categorical feature or a class label turns its likelihood into an indicator or an interval probability and tree-based node functions (Qu et al., 2025) are piecewise constant. The likelihood is then discontinuous in ξ, can vanish on most of the parameter space, and has zero or undefined gradients, which poses a severe problem for gradient-guided MCMC samplers that are needed in this high-dimensional setting (e.g., Robnik and Seljak, 2024). • Dataset-level operations. TabPFN draws its class boundaries from the realized continuous targets Qn (Hollmann et al., 2025). The rows are then not conditionally independent given ξ, and p(Dn | ξ) ̸= i=1 pξ (xi , yi ). TabICLv2 rejects datasets that fail its predictability filters (Qu et al., 2026), which only adds to the fact that the posterior is intractable, as one would have to account for this rejection step in the likelihood. • Variable dimension and multimodality. The parameter ξ contains the DAG, the number of nodes (2 to 32 in TabICLv2), the response node, the assignment of nodes to features and the weights of the node functions. The number of DAGs grows super-exponentially in the number of nodes. Additionally, the dimension of ξ changes with the graph.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

In order to still get a high-quality reference PPD via MCMC sampling, we turn to the BNN prior introduced in Hollmann et al. (2023) that we construct in the following way: A ReLU network with two hidden layers of width 8 or 16 maps d = 4 features to a logit fw (x). All weights and biases w are iid N
(0, 1), the standard prior choice in BNNs (Kobialka et al., 2026), and labels follow y | x, w ∼ Bernoulli σ(fw (x)) . The covariate law pX is one of two: • Gaussian features: X ∼ N (0, Id ). • SCM features: the TabICL SCM generator draws one MLP-based or tree-based SCM per table (probabilities 0.7 and 0.3). It converts columns to categorical codes with probability 0.2, clips at four standard deviations, and standardizes over the table. We discard the SCM’s own label. For each covariate law we pretrain TabICLv2 PFNs of three sizes (small, medium and large according to Table 5) for 50,000 steps of 64 tables. The number of rows per table is log-uniform or uniform on [16, 4608], and a uniform fraction in [0.1, 0.9] of the rows forms the context. This induces the size distribution Ψ. The number of features d is uniform on {1, . . . , 8}, zero-padded to 8. Every context size n ≤ ⌊0.9 · 4608⌋ = 4147 therefore lies in supp Ψ. With SCM features we pretrain only the smallest and largest PFN, both with uniform row counts. We evaluate the PFNs on 512 test tasks per hidden width at context sizes n ∈ {4, 8, . . . , 4096}. The test tasks are drawn independently of pretraining. • Gaussian features, nested design. Each task has a pool of 4096 rows. The context Dn is the first n rows of the pool, and all n share the same 128 queries, so differences across n reflect only the added rows. • SCM features, independent design. The first n rows of a processed table are not distributed like a pretraining table, because standardization and categorical conversion use the whole table. We therefore draw one independent table per task and n, with w shared across n. Each table has n context rows and Qn = min(9n, max(128, ⌈n/9⌉)) queries, so every evaluated table is an exact pretraining draw. As a reference π̂Π we approximate the PPD πΠ (y | x, Dn ) of Eq. (7) with Microcanonical Langevin Ensembles (MILE; Sommer et al., 2025). As MCMC samplers come with no finite-time convergence guarantees (Welling and Teh, 2011), and in the BNN literature it is additionally well known that a complete characterization of the whole posterior distribution is computationally intractable due to a combinatorially exploding number of posterior modes (Wiese et al., 2023), we employ a very large sampling budget and run, per task, 24 chains from a deep-ensemble warm start (Sommer et al., 2024), 50,000 warm-up and 10,000 sampling steps, where we keep every tenth draw. This way, we make sure that we capture a major part of the functional variability that the weight space posterior expresses. We then pool the chains with equal weight in a Bayesian model average. Correspondingly, in Fig. 7 it is clearly visible that adding further chains to the Bayesian model average saturates after roughly 12 chains. Our metric is the prior-data NLL: the mean of − log qθ (y | x, Dn ) over a task’s queries with their realized labels, averaged over tasks, i.e., an estimate of the risk Eq. (2) at context size n. We also report the oracle predictive p(y | x) = σ(fw (x)), which has access to the true w. Its risk at context size n lower-bounds that of πΠ by the context error at that size. Where the amortization error concentrates To characterize the amortization error, we evaluate Eq. (5), DKL (π̂Π ∥ qθ ), at each input (x, Dn ). The inputs are 1024 additional rows per task for Gaussian features and the queries for SCM features (we cannot use additional rows for the SCM features due to the preprocessing issue explained above). We use the large PFN with hidden width 16 and uniform row counts (values for log-uniform row counts are given in parentheses). The amortization error stems from few inputs, not from few tasks. The 10% of inputs with the largest KL carry 60–66% of the total at n = 64 and over 80% at n ≥ 1024, whereas the 10% of tasks with the largest KL carry at most a third. These inputs lie near the decision boundary: at n = 4096, the 20% of inputs with the smallest predicted class-1 probability carry 80% of the KL. Relative to the between-chain variability of the reference, the amortization error grows with n. The PFN’s KL to π̂Π exceeds that of a single chain by a factor of 1.7 (3.0) at n = 64 and 5.5 (11) at n = 4096. At n = 4096, the PFN predictive lies outside the range of all 24 chain predictives for over a third (two thirds) of inputs. The larger

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Gaussian Features

NLL of k-chain average NLL of k-chain average

n=4

5.00

5.00

n = 64

×10−4 + 0.4502

4.00 3.00 2.00 1.00

n = 4096

×10−4 + 0.1198

×10−4 + 0.1062

5.00

5.00

5.00

4.00

4.00

4.00

3.00

3.00

3.00

2.00

2.00

2.00

1.00

1.00

1.00

×10−4 + 0.4598

SCM Features

n = 1024

×10−4 + 0.2068

×10−4 + 0.2110

×10−4 + 0.1235

×10−4 + 0.1103

5.00

4.00 3.00 2.00 1.00 1

8

16

Chains (k)

24

5.00

5.00

4.00

4.00

3.00

3.00

2.00

2.00

1.00

1.00 1

8

16

Chains (k)

24

4.00 3.00 2.00 1.00 1

8

16

Chains (k)

24

1

8

16

24

Chains (k)

Figure 7: Holdout NLL of Bayesian model average of k pooled MCMC chains for classification. The NLL saturates as more chains are added, which means a small number of chains seems sufficient to capture the functional diversity of the posterior distribution of the weights.

gap under log-uniform row counts is consistent with that law placing less pretraining mass on large contexts under Ψ. At large n, the PFN also shrinks confident logits, i.e., it becomes underconfident. Aggregate miscalibration does not explain the amortization error: Against the oracle predictive p(y | x), both qθ and π̂Π have calibration slopes within 0.03 of 1 for n ≥ 64. A slope of 1 means that, averaged over inputs, the predictor is neither overconfident (slope < 1) nor underconfident (slope > 1). It is expected that the MCMC reference is well calibrated but it is interesting that the PFN is as well, as a predictor can be calibrated but still far from the PPD. In summary, the amortization error concentrates at the decision boundary and is most severe at large context sizes that pretraining covers sparsely. The chains also disagree most at the decision boundary, so there the error of the PFN cannot be separated pointwise from the remaining error of the reference. In aggregate, however, the PFN’s error exceeds the reference’s approximation error due to finite-budget MCMC several-fold. Testing the PFN under deployment distribution shifts Pretraining minimizes the empirical risk only on average over the prior’s input distribution. To test the PFN at inputs that are unlikely under the prior, we use the Gaussian feature prior and scale the features to x = sz with s ∈ {1.5, 2, 3} and draw labels from the prior’s network. Because w is independent of x, the posterior predictive under the training prior remains the target, and MCLMC estimates it with the same protocol. In Fig. 8, the PFN-MCLMC gap in NLL grows with s at every n ≥ 64. At n = 4096, the difference is larger than the excess of MCLMC over the oracle. Under distribution shift, pretraining choices that do not matter much at s = 1 start to matter more, such as the more pronounced differences between the log-uniform Ψ and the uniform Ψ. How closely the PFN matches the posterior predictive therefore depends on its pretraining input distribution. Synthetic prior amortization gap for regression For regression, we draw tasks y = fw (x) + σε with log σ ∼ N (0, 1) and Gaussian features, train TabICL on raw labels, and compare it with MILE (MCLMC, 24 chains) on the same model for n ≤ 4096. TabICL’s regression head predicts 999 quantiles q1 ≤ · · · ≤ q999 at levels αi = i/1000, not a density. Its density is the derivative of the piecewise-linear quantile function: q(y) = 10−3 /(qi+1 − qi ) for y ∈ [qi , qi+1 ), with exponential tails. The log density thus depends on the difference of two neighbouring outputs, and small errors in the predicted quantiles make it non-smooth. This inflates the NLL but barely moves the CRPS, which is linear in the quantile errors. Again, the PFN comes very close to the

Manuel Hülskamp

Julius Kobialka

Large PFN uniform Ψ

log-uniform Ψ PFN − MCLMC NLL (10−3 nats)

Emanuel Sommer

n = 64

15

Thomas Nagler

David Rügamer

uniform Ψ, standardized context

n = 256

n = 4096

10 MCLMC − oracle

5 0

1

1.5

2

3 1 1.5 2 3 1 Feature scale s (s = 1: pretraining input distribution)

1.5

2

3

Figure 8: PFN to MCLMC gap under covariate shift. Test features are x = sz with z ∼ N (0, I4 ) and labels come from the BNN prior (2 × 8 hidden neurons). The PFN was pretrained at s = 1. Each point is the mean difference over 256 test tasks. Bands are 95% task-bootstrap intervals of each mean. Curves show the large PFN pretrained with uniform or log-uniform table size Ψ, and the uniform-Ψ PFN with context-standardized features. Dotted grey (n = 4096 only): excess NLL of the MCLMC reference over the oracle.

reference MCMC PPD across the whole range of pretraining context sizes. The gap at intermediate context sizes comes almost entirely from tasks with low noise. In these tasks, the posterior concentrates quickly as data arrive, and the PFN lags behind: its predictive median lies further from the true function than MILE’s, and it compensates with a wider predictive. Tasks with high noise show no gap at any context size, and the gap disappears once the data determine the function. The same low-noise tasks expose a weakness of the reference at the largest context sizes. There, MILE’s equal-weight mixture over chains becomes over-dispersed, and the PFN predicts better than the reference. Part of the gap’s decline at large n therefore reflects the reference degrading, not only the PFN improving.

PFN small

PFN large

MILE (MCLMC), 24 chains

Oracle (true task)

Negative log-likelihood

1.6

2.5

1.4

1024

2048

4096

2.0 1.5

Prior-data CRPS

3.0 Prior-data NLL

CRPS

n ≥ 1024 (magnified)

3.0

1.025

2.5

1.000 0.975

2.0

n ≥ 1024 (magnified)

1024

2048

4096

256

1024

4096

1.5 1.0

4

16

64

256

1024

Number of revealed data points (n)

4096

4

16

64

Number of revealed data points (n)

Figure 9: BNN regression with hidden width 8 × 8, d = 4, 512 test tasks: prior-data NLL (left) and Continuous Ranked Probability Score (CRPS; right) against the number of context points n for the small and large PFN, the MILE (MCLMC) reference (24 chains), and the oracle. The PFN’s NLL gap to MILE stems mostly from the density implied by its 999 predicted quantiles and on CRPS, which needs no density, the PFN nearly matches MILE.

D

AMORTIZATION DISAGREEMENT METRICS

We define the disagreement metrics across K independently trained members {θ (k) }K k=1 , evaluated on N test points. All members share optimizer settings, M and model size and only random seeds and pretraining data differ.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Jensen–Shannon divergence The generalized JSD (Lin, 1991) splits the predictive entropy of the mixture P 1 q̄ = K q , (k) k θ



P P 1 1 H(q̄) = JSD qθ(1) , . . . , qθ(K) + K JSD qθ(1) , . . . , qθ(K) = K (23) k H qθ (k) , k DKL qθ (k) ∥ q̄ . With equal weights, it equals the mutual information between the prediction and the member index. We report its average over test points,
d JSD = 1 PN JSD qθ(1) (· | xi , Dn ), . . . , qθ(K) (· | xi , Dn ) . (24) AU i=1 N d JSD /H(q̄) expresses the disagreement as a fraction of the total predictive entropy. The ratio AU Pairwise disagreement rate

With ŷ (k) (x) = arg maxy qθ(k) (y | x, Dn ),

d dis = AU

 PN  (k) K −1 P 1 (xi ) ̸= ŷ (l) (xi ) . k<l N i=1 1 ŷ 2

(25)

Compared to the JSD, it registers disagreement only when the argmax flips, and hence underestimates functional disagreement; unlike the JSD, it measures disagreement at the level of decisions. d Estimation error EE(M ) is defined in Eq. (9). It is a biased estimator of the estimation error, since the population minimizer θ ∗ is replaced by a reference checkpoint θref trained on Mref ≫ ML datasets; a ground-truth minimizer of the population objective is not tractable.

E

EXPERIMENTAL SETUP

E.1

Implementation and data

Implementation The SG-MCMC experiments and the deep ensembles of Section 5 use our JAX (Bradbury et al., 2018) reimplementations of nanoTabPFN and TabICLv2 based on the official repository of Sommer et al. (2025). IVON and the full-size TabICLv2 deep ensembles of Section 5.4 are implemented in PyTorch (Paszke et al., 2019), building on the official TabICL codebase (Qu et al., 2026). Table 5 lists all architectures. Evaluation data We hold out the 45 datasets of the TALENT-tiny core set (Liu et al., 2025) as validation data, which we use only to monitor training progress and not for checkpoint selection, and report all TALENT results on the remaining 255 datasets (170 classification, 85 regression), which we refer to simply as TALENT. From TabArena (Erickson et al., 2025), the experiments of Section 5 use the 30 classification datasets within the scope of the small models (at most 10 classes and 100 features, a minority class of at least 1% and at most 20% missing values). Each dataset is evaluated with stratified 5-fold cross-validation, averaged over five random subsamples. The full-size models of Section 5.4 are evaluated on all 38 classification and 13 regression datasets of TabArena. Finally, BeyondArena (Purucker et al., 2026) collects datasets with grouped, clustered, and temporal structure that departs from the i.i.d. assumption of the synthetic prior; we use it to study member disagreement beyond the prior (Appendix F.9). E.2

Bayesian model averages

Deep ensembles (Section 5) All members are trained with AdamW (Loshchilov and Hutter, 2019) (β1 =0.9, β2 =0.99, weight decay 10−4 ) on batches of 64 synthetic datasets drawn from the TabICLv2 graph-SCM prior, each with 1024 rows, 1–100 features, up to 10 classes and a train fraction drawn from [0.3, 0.9]. The learning rate follows a cosine decay from its peak to 0.3× the peak over 60% of the maximum number of steps and stays constant afterwards; for nanoTabPFN, this decay is preceded by a linear warmup from 10% of the peak over 600 steps. The optimizer settings were chosen via an extensive grid search on the small architecture size and an additional smaller grid search of the learning rate settings per model size. Training stops early after 10 epochs without improvement of the cross-entropy on 100 held-out synthetic validation batches, and each member keeps its weights with the lowest validation loss. Members are trained in parallel and never share a pretraining dataset; the size-dependent settings are given in Table 2. The M -scaling runs (Section 5.2) use the tiny and small recipes with K=4 members per value of M , where M caps the number of unique datasets each member sees within the fixed step budget; the reference runs use M =106 (tiny) and M =1.5 × 106 (small).

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

Table 2: Size-dependent training settings of the deep ensembles in Section 5. Maximum steps are reached only if early stopping does not trigger.

Maximum epochs Maximum steps Peak learning rate, nanoTabPFN Peak learning rate, TabICLv2 Ensemble members K

tiny

small

medium

large

30 18,000 4 · 10−4 4 · 10−4 8

40 24,000 4 · 10−4 4 · 10−4 8

55 33,000 – 4 · 10−4 8

70 42,000 – 4 · 10−4 4

SG-MCMC Both samplers target pβ directly. Each chain is warm-started from one trained DE member (Sommer et al., 2024) and continues on pretraining data disjoint from that of all members. The samples of all chains are averaged with equal weight via Eq. (15), forming a Bayesian deep ensemble (Sommer et al., 2025). The DE explores uncertainty between basins, whereas each chain gathers information within its warm-start basin. Table 3 lists the settings used for all reported runs. The hyperparameters for both samplers were chosen via a two-stage grid search. In a first coarser step, all listed parameters were searched. Afterwards, in a second step, the step sizes and temperatures were tuned per model size. Table 3: SG-MCMC settings. Sizes vary as tiny / small / medium / large. Both samplers use the same number of gradient evaluations per chain; pSMILE takes two per step. adaSGHMC Step size Warm-up steps Sampling steps Thinning Gradient evaluations per chain Chains Temperature T Prior p(θ) Batch size

pSMILE

−4

2 · 10 10−4 500 250 2500 / 3000 / 2500 / 3500 1250 / 1500 / 1250 / 1750 every 10% of the sampling steps (10 samples per chain) 3000 / 3500 / 3000 / 4000 8/8/8/4 0.01 N (0, I) 64 datasets

IVON IVON minimizes the Gibbs objective EQ [Lemp (θ)] + β −1 DKL (Q ∥ p) over diagonal Gaussians Q = N (m, diag(s2 )), and its effective-sample-size hyperparameter is exactly β in Eq. (14). Predictions average over S draws from Q and are compared with the posterior mean m (S=0 in Table 7). Table 4 lists the settings used for all reported runs. Similar to the hyperparameter search for the SG-MCMC samplers, the parameters for IVON were also selected in a two-step procedure: First, all listed parameters were coarsely searched, and in a second step the effective sample size and learning rate were jointly optimized per model size. Table 4: IVON settings. All sizes additionally share gradient clipping at 10, a cosine schedule with 1% linear warm-up decaying to 10−6 , batches of 64 datasets and bfloat16 precision. tiny Effective sample size β Learning rate Training steps β1 β2 Weight decay Hessian initialization Clip radius

E.3

small 6

1.92 · 10 1.0 300,000

medium 6

3.84 · 10 1.0 200,000

6

6.4 · 10 0.3 200,000

large 1.92 · 107 0.3 200,000

0.9 0.99999 10−4 0.5 10−3

Full-size TabICLv2

Full-size TabICLv2 deep ensembles (Appendix F.7) We train K=8 members each for classification and regression, differing in the seeds of the weight initialization and of the synthetic data stream. Regression members

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

use bias-free layer normalization and a quantile head with 999 quantiles. All members are trained with Muon (Jordan et al., 2024) applied to all parameters, weight decay 0.01, batches of 64 datasets, bfloat16 precision and gradient clipping at 10, on four NVIDIA H200 GPUs per member. Each stage starts from the previous stage’s weights. Within a stage, the learning rate warms up linearly over the first 1% of steps and then follows a cosine decay to 10−7 . Following the curriculum of Qu et al. (2026), training proceeds in three stages: (1) 130,000 steps at learning rate 8 · 10−4 on datasets with up to 1024 rows and a train fraction drawn from [0.3, 0.9]; (2) 10,400 steps at 10−4 on 400–10,240 rows (log-uniform); and (3) 11,000 steps at 2 · 10−5 with gradient clipping at 1 on 400–60,000 rows, both with a train fraction of about 0.8. Datasets have 1–100 features and up to 10 classes throughout. Compared with the official recipe, stages 1 and 2 are shorter (500,000 and 40,000 steps in Qu et al. (2026)), stage 3 is slightly longer (10,000 steps), and training uses bfloat16 instead of float32. The total number of training steps was chosen such that the total training over all three stages per ensemble member took about 48 hours across a node with NVIDIA H200 GPUs. Attention modifications for bfloat16 training To train stably in bfloat16, we make two changes to the attention of TabICLv2. First, queries and keys are RMS-normalized per head before the dot product in the column, row and ICL transformers (QK-normalization; Henry et al., 2020). Second, following Pfeiffer et al. (2026),
we bound the length-dependent base scale of QASSMax as cmax tanh g(log n)/cmax with cmax =10, where g is the original base MLP and n the sequence length, and compute it in float32. The unbounded MLP extrapolates linearly in log n, so at sequence lengths beyond those seen in training it can saturate the attention and overflow; with the query modulation 1 + tanh(·) ∈ [0, 2], the total scale now lies in [−2cmax , 2cmax ]. At initialization, cmax tanh(x/cmax ) ≈ x, so the bounded variant starts out equal to the original. Both changes add fewer than 2,000 parameters. Table 5: Architecture hyperparameters. NanoTabPFN large is omitted due to its training cost; TabICLv2 full, used in Appendix F.7, follows the architecture of Qu et al. (2026) with added QK-normalization and bounded QASSMax scaling. In TabICLv2, the ICL stage operates on the concatenated CLS column tokens (ICL width = embedding dimension × CLS column tokens), and every block’s MLP hidden dimension is twice its width. All models predict up to 10 classes.

NanoTabPFN Parameters (rounded) Transformer layers Embedding dimension Attention heads MLP hidden dimension TabICLv2 Parameters (rounded) Blocks (column / row / ICL) Heads (column / row / ICL) Embedding dimension ICL width CLS column tokens Column inducing vectors Feature group size E.4

tiny

small

medium

large

full

1M 4 128 4 256

2M 4 192 6 384

4M 5 256 8 512

– – – – –

– – – – –

1.5M 2/2/4 4/4/6 64 192 3 64 3

2M 3/3/5 4/4/6 64 192 3 64 3

7.5M 3/3/5 8/8/8 128 384 3 128 3

9M 3/3/6 8/8/8 128 384 3 128 3

27.5M 3 / 3 / 12 8/8/8 128 512 4 128 3

Code

We will make our code available in the near future.

F

FURTHER RESULTS

F.1

Pretraining data scaling

We present the extended data-scaling analysis of Section 5.2 for the tiny models and the synthetic test set, showing how estimation error and member disagreement change with the number of pretraining datasets M .

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

TabICLv2

(a) Estimation error, synthetic

Thomas Nagler

NanoTabPFN

(b) JSD, TabArena

(c) Disagreement, TabArena

104

103

105

Disagreement rate

JSD (nats)

10−4

0.10 0.05 0.00

Training scale M

104

103

105

0.3

tiny

EE(̂ M)

100 10−2

David Rügamer

0.2 0.1 103

Training scale M

104

105

Training scale M

d Figure 10: As Fig. 3, for the tiny models: estimation error EE(M ) (log-log) on synthetic data and member disagreement d on TabArena versus the number of pretraining datasets M . Bands show ±1 standard error over the K=4 members (EE) or over datasets (disagreement).

TabICLv2

(a) JSD, synthetic Disagreement rate

(b) Disagreement, synthetic

0.2 0.1 103

104

Training scale M

Disagreement rate

JSD (nats)

103

0.1

Training scale M

105

104

105

104

105

Training scale M

0.4

small

0.2

104

0.2

105

0.3

103

0.4

tiny

JSD (nats)

0.3

0.0

NanoTabPFN

0.2 103

Training scale M

d JSD , AU d dis ) on the synthetic test set versus the number of pretraining datasets M , Figure 11: Member disagreement (AU complementing the TabArena results of Fig. 3. Top: tiny, bottom: small models.

F.2

Model capacity

Experimental details on the capacity axis scaling testing whether larger models disagree less by comparing member disagreement across four TabICLv2 sizes on the synthetic test set drawn from the prior and on the TabArena classification datasets at two context sizes.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? Table 6: Member disagreement (K=4) across TabICLv2 sizes on the synthetic test set and on the 30 TabArena classification d JSD in 10−3 nats, AU d dis in %. datasets (mean ± standard error over datasets). AU synthetic

F.3

TabArena, context 1000 TabArena, context 10000

Model

d JSD AU d dis AU

d JSD AU

d dis AU

d JSD AU

d dis AU

tiny (1.5M) small (2M) medium (7.5M) large (9M)

16.1 15.5 – 13.6

12.5 ± 2.0 11.4 ± 1.6 9.7 ± 1.5 7.4 ± 1.4

7.5 ± 1.4 7.5 ± 1.5 6.7 ± 1.4 5.9 ± 1.2

11.8 ± 1.6 11.0 ± 1.4 10.5 ± 1.3 8.5 ± 1.4

8.0 ± 1.5 7.8 ± 1.5 7.0 ± 1.4 6.5 ± 1.2

9.4 9.2 – 8.2

Disagreement concentration

To see where the averaging gain between independently pretrained TabICLv2 members comes from, we evaluate it per test point. For a test point (xi , yi ) with context Dn , the pointwise gain

1 gi = log q̄(yi | xi , Dn ) − K

PK

k=1 log qk (yi | xi , Dn )

≥0

cumulative share of averaging gain

is the log-probability that the mixture q̄ assigns to the observed label minus the members’ average log-probability of it. It is zero exactly when all members assign the same probability to yi , and its mean over the test points is the log-loss improvement of q̄ over its averagePmember, i.e., the empirical counterpart of ∆p in Lemma 1. We sort the test points by gi and plot the share of i gi carried by the fraction of points with the largest gain on the synthetic test set and on TabArena.

1.0

tiny (1.5M)

large (9M)

0.8 0.6 0.4

uniform synthetic (prior) TabArena, context 1000 TabArena, context 10000

0.2 0.0

0.0

0.1 0.2 0.3 0.4 fraction of test points (largest gain first)

0.5 0.0

0.1 0.2 0.3 0.4 fraction of test points (largest gain first)

0.5

Figure 12: Share of the total pointwise gain carried by the test points with the largest gain, for tiny (left) and large (right) TabICLv2 with K = 4 members. Dashed: gain spread evenly over all points.

F.4

Deep ensembles

We report the deep-ensemble scaling results for nanoTabPFN, complementing the TabICLv2 results of the main text.

Manuel Hülskamp

Julius Kobialka

Small

Deep ensemble

Thomas Nagler

−0.335

0.8525

0.840

−0.340

0.8500

0.835

0.8450

↑

0.830 0.825 0.820

0.8425

1000

2000

5000

Context size (rows)

10000

−0.355 −0.360 −0.370

0.810 500

−0.350

−0.365

0.815

0.8400 0.8375

−0.345 LPPD

↑

0.845

0.8475

David Rügamer

Single chain (mean)

0.8550

AUROC

Accuracy

↑

Tiny

Emanuel Sommer

−0.375 500

1000

2000

5000

Context size (rows)

10000

500

1000

2000

5000

Context size (rows)

10000

Figure 13: As Fig. 4, for nanoTabPFN: accuracy, ROC AUC and LPPD on the 30 TabArena classification datasets across the maximal number of context rows, for deep ensembles and their average single member at the tiny and small model sizes.

F.5

Variational sampling via IVON

Additional evidence on TALENT classification to test whether sampling from IVON’s variational posterior improves on its posterior mean, averaging S posterior samples for three TabICLv2 sizes. Table 7: IVON variational sampling on the 170 TALENT classification datasets. S = 0 is the posterior mean; S ≥ 1 averages S posterior samples without the mean. LPPD is the mean log predictive density per test point. + /− : significantly better/worse than the same model at S = 0 (paired Wilcoxon over datasets, p < 0.05).

F.6

Model

S

Accuracy ↑

ROC AUC ↑

LPPD ↑

tiny

0 1 8 16 32 64

0.8073 0.8035− 0.8068 0.8063 0.8077 0.8072

0.8737 0.8714− 0.8738 0.8742 0.8752 0.8749

−0.4464 −0.4558− −0.4493− −0.4488− −0.4484 −0.4489

small

0 1 8 16 32 64

0.8129 0.8125 0.8122− 0.8130 0.8134 0.8134

0.8812 0.8811− 0.8814 0.8816 0.8816 0.8817

−0.4463 −0.4488− −0.4470− −0.4465− −0.4460+ −0.4459

large

0 1 8 16 32 64

0.8169 0.8153− 0.8173 0.8171 0.8171 0.8173

0.8824 0.8812− 0.8824 0.8828+ 0.8827 0.8827

−0.4338 −0.4360− −0.4332 −0.4330 −0.4330 −0.4329

SG-MCMC sampling

Extended results comparing the two SG-MCMC samplers, adaSGHMC and pSMILE, each warm-started from the deep-ensemble members, to the deep ensemble itself on the 30 TabArena classification datasets. Despite extensive hyperparameter searches, no parameters for adaSGHMC could be found that result in the sampler moving without at the same time decreasing performance compared to the DE.

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? Table 8: Relative LPPD gain (%) of the Bayesian deep ensemble over the deep ensemble it is warm-started from, for both samplers, on the 30 TabArena classification datasets at context sizes 1000 and 10000. In parentheses: number of datasets on which the Bayesian deep ensemble improves on the deep ensemble. ∗ : significant change, paired Wilcoxon over datasets, p < 0.05. context size 1000 Model tiny small medium large

F.7

adaSGHMC −0.05 (11/30) −0.64∗ (4/30) – +0.12 (18/30)

pSMILE ∗

+0.27 (26/30) +0.37∗ (22/30) +0.07 (12/30) +0.27∗ (26/30)

context size 10000 adaSGHMC

pSMILE

−0.05 (11/30) −1.12∗ (5/30) – −0.52∗ (9/30)

+0.35∗ (26/30) +0.42∗ (25/30) +0.08 (13/30) +0.32∗ (26/30)

Full-size TabICLv2 deep ensembles

This section details the results summarized in Section 5.4 where we ask whether the removable amortization error persists at the scale of current state-of-the-art PFNs, as well as with the full preprocessing pipeline. Since within-basin methods added little in Section 5.3 and to reduce the computational costs, we restrict this section to deep ensembles. We also extend the analysis to regression, for which Proposition 2 and Lemma 1 hold unchanged with densities and differential entropies. Setup We pretrain K=8 models that largely follow the architecture and training setup of TabICLv2 (Qu et al., 2026) including the three-stage training curriculum, with details in Appendix E. Each member is run with the same inference-time ensemble of 8 preprocessing and permutation configurations. Results The gain persists at full scale. After each of the three training stages, the ensemble improves on its average member in LPPD, Brier score and ROC AUC (Fig. 5a and Table 10). On the 170 TALENT classification datasets, the LPPD gain is 0.8% at every stage. The ensemble is also significantly better than the best single member, selected after the fact, in all three metrics (p ≤ 2 · 10−4 at stage 3). On TabArena, the ensemble improves on its average member on all 38 classification datasets and on the best member on 27 (p = 0.01). More training does not remove it. All three stages improve every member, and most strongly on the large contexts that stages 2 and 3 train on: on contexts beyond 10,240 rows, the LPPD of the average member improves by 6.0% from stage 1 to stage 3. Yet the ensemble gain stays at 0.7–0.9% in every context-size bin and at every stage (Table 11). This is the full-scale counterpart of the data axis in Section 5.2: further pretraining improves the members without reconciling them. The gain also does not come from the inference-time preprocessing. Giving each member 8 of 64 distinct preprocessing configurations, instead of sharing the same eight, leaves the ensemble unchanged (TabArena Elo 1555 vs. 1552). The gain thus stems from the independently pretrained weights, i.e., from amortization uncertainty, rather than from the context-level randomness that the preprocessing ensemble already averages over. Ensembling as parallel pretraining. A single member reaches a TabArena classification Elo of 1460–1514. The eight-member ensemble reaches 1552, within the confidence interval of the official TabICLv2 checkpoint (1575; Fig. 5c). Since the members never communicate, they can be trained concurrently on separate nodes. The ensemble then requires about 3× the GPU hours of the official recipe at our throughput, but only about 0.37× its wall-clock time, at 8× the inference cost (Table 9). Ensembling (potentially shorter) independent pretraining runs is thus a simple way to parallelize PFN pretraining beyond a single node.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

Table 9: Training budget of one full-size member against the official TabICLv2 recipe (Qu et al., 2026). Throughput was measured for our setup (bfloat16, four H200 GPUs, batch size 64); the official durations apply our throughput to its step counts and are therefore an estimate. GPU hours count four GPUs. steps

wall-clock (h)

Stage (max. rows)

official

ours

steps/h

official

ours

1 (1024) 2 (10,240) 3 (60,000)

500,000 40,000 10,000

130,000 10,400 11,000

6,033 2,150 650

82.9 18.6 15.4

21.5 4.8 16.9

Total, one model GPU hours, one model GPU hours, 8-member ensemble

550,000

151,400

116.9 467

43.3 173 1,386

Table 10: Full-size TabICLv2 on TALENT after each training stage. Average member averages each dataset’s metric over the 8 members; best member is the member with the best mean at that stage, selected after the fact. ∗ : deep ensemble significantly better than the best member (paired Wilcoxon over datasets, p < 0.05). Classification (170 datasets) Stage LPPD ↑

Brier ↓

ROC AUC ↑

1

average member best member deep ensemble

−0.3713 −0.3693 −0.3684∗

0.2132 0.2122 0.2116∗

0.9021 0.9024 0.9032∗

2

average member best member deep ensemble

−0.3634 −0.3619 −0.3606∗

0.2089 0.2082 0.2073∗

0.9039 0.9045 0.9049∗

3

average member best member deep ensemble

−0.3621 −0.3605 −0.3592∗

0.2081 0.2073 0.2066∗

0.9042 0.9048 0.9051∗

Regression (85 datasets) Stage

nRMSE ↓

RMSE ↓

R2 ↑

1

average member best member deep ensemble

0.4874 0.4748 0.4774

18,394 17,495 18,211

0.6728 0.6887 0.6846

2

average member best member deep ensemble

0.4605 0.4532 0.4561

17,202 16,925 17,073

0.7030 0.7085 0.7070

3

average member best member deep ensemble

0.4589 0.4536 0.4548

17,218 17,061 17,106

0.7043 0.7087 0.7081

Table 11: Relative improvement (%) of the deep ensemble over its average member per context-size bin (training rows available as context) on TALENT. Classification uses LPPD, regression nRMSE; positive values are improvements. ≤1024

1024–10240

>10240

Classification Stage 1 Stage 2 Stage 3

n = 33 +0.85 +0.85 +0.85

n = 100 +0.70 +0.77 +0.78

n = 37 +0.86 +0.72 +0.75

Regression Stage 1 Stage 2 Stage 3

n = 21 +0.69 +0.73 +0.68

n = 43 +2.16 +1.09 +1.04

n = 21 +3.24 +0.94 +0.90

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

F.8

Ensemble size analysis

To test how many members a full-size TabICLv2 deep ensemble needs, we evaluated every possible sub-ensemble of the eight members, from K=1 to 8, and recorded the mean and standard deviation of LPPD (classification) and nRMSE (regression) across all sub-ensembles of each size on TALENT. Both metrics improve with every added member but quickly saturate (Fig. 14): four members already achieve about 85% of the full 8-member gain. Additional members mainly reduce the variance across sub-ensembles, so the result depends less on which members are included.

0.4625

nRMSE ↓

LPPD ↑

−0.3600

−0.3615

−0.3630

0.4600

0.4575

0.4550 1

2

3

4

5

6

7

8

1

2

3

ensemble size K

4

5

6

7

8

ensemble size K

Figure 14: Full-size TabICLv2 deep ensembles on TALENT: LPPD on the 170 classification datasets (left) and nRMSE on
8 the 85 regression datasets (right) versus ensemble size K. Lines show the mean, bands ±1 standard deviation over all K sub-ensembles of the eight stage-3 members.

F.9

Disagreement beyond the prior

fraction of datasets

Results for testing the hypothesis that full-size TabICLv2 members disagree more outside the prior’s support, comparing TabArena datasets within the support to BeyondArena datasets with shifted splits or beyond it.

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0 10−4

̂ (nats) AUJSD

10−3

10−2

0.0 10−1

TabArena, within prior support (n = 30)

10−5

10−4

̂ AUdis

10−3

10−2

10−1

BeyondArena OOD (n = 38)

d JSD Figure 15: Empirical CDFs over datasets of the disagreement between full-size TabICLv2 classification members, AU d dis (right, log scale), on TabArena datasets within the prior’s support (iid (left) and the pairwise disagreement rate AU splits, at most 100 features, 10 classes and 48,000 training rows) and on BeyondArena datasets with a temporal or grouped split or outside that task space. Group summaries in Table 12.

Manuel Hülskamp

Julius Kobialka

Emanuel Sommer

Thomas Nagler

David Rügamer

Table 12: Member disagreement of full-size TabICLv2 classification members on TabArena datasets within the prior’s support and on BeyondArena datasets beyond it. Prior support: at most 100 features, 10 classes and 48,000 training rows; BeyondArena OOD: a temporal or grouped split, or outside that task space. Text datasets are excluded. Each dataset’s value is the mean over its splits; p: one-sided Mann–Whitney U test, datasets are listed in Tables 13 and 14.

Datasets Split Features, median (range) Training rows, median (range) Classes, max Beyond prior support d JSD (nats) Median AU d dis Median AU

TabArena, within prior support

BeyondArena OOD

30 iid 20 (4–94) 3,637 (499–47,679) 5 0 0.0011 0.0099

38 20 iid, 7 temporal, 11 grouped 152 (3–22,215) 30,625 (114–1,000,350) 20 33 (23 >100 features, 17 >48,000 rows, 3 >10 classes) 0.0043 (p = 1.4 · 10−4 ) 0.0218 (p = 0.06)

Table 13: TabArena classification datasets within the prior’s support (all iid splits): number of features, mean number of training rows per split and number of classes. The eight remaining TabArena classification datasets lie beyond the support (more than 100 features: APSFailure, Bioresponse, hiva_agnostic, kddcup09_appetency, MIC; more than 48,000 training rows: customer_satisfaction_in_airline, GiveMeSomeCredit, SDSS17) and are not used for the experiments resulting in Table 12. Dataset Amazon_employee_access anneal bank-marketing Bank_Customer_Churn blood-transfusion-service-center churn coil2000_insurance_policies credit-g credit_card_clients_default diabetes Diabetes130US E-CommereShippingData Fitness_Club hazelnut-spread-contaminant-detection heloc HR_Analytics_Job_Change_of_Data_Scientists in_vehicle_coupon_recommendation Is-this-a-good-customer jm1 Marketing_Campaign maternal_health_risk NATICUSdroid online_shoppers_intention polish_companies_bankruptcy qsar-biodeg seismic-bumps splice students_dropout_and_academic_success taiwanese_bankruptcy_prediction website_phishing

Features

Train rows

Classes

9 38 13 10 4 19 85 20 23 8 47 10 6 30 23 12 24 13 21 25 6 86 17 64 41 15 60 36 94 9

21,846 599 30,141 6,667 499 3,333 6,548 667 20,000 512 47,679 7,333 1,000 1,600 6,973 12,772 8,456 1,149 7,257 1,493 676 4,994 8,220 3,940 703 1,723 2,127 2,949 4,546 902

2 5 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 3 3 2 3

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference? Table 14: BeyondArena classification datasets outside the prior’s support or with a shifted (temporal or grouped) split, with the reason for inclusion. Training rows: mean per split. Datasets with more than 700,000 training rows are evaluated with a single preprocessing configuration instead of eight. Dataset

Split

Feat.

Train rows

Cls

Outside the prior

amex_non_iid_1m asp_potassco_classification cardiotocography covertype dementia_prediction mice_protein_trisomy_discriminant micro_mass musk pancreatic_cancer_mouse_detection parkinsons_biomedical_voice_measurements sepsis_prediction_1m

grouped grouped grouped grouped grouped grouped grouped grouped grouped grouped grouped

190 137 23 14 9 77 1,083 167 6,772 24 43

1,000,350 808 1,417 341,750 247 720 381 4,399 121 130 970,175

2 11 3 3 3 8 20 2 2 2 2

shift, >100 feat., >48k rows shift, >100 feat., >10 cls shift shift, >48k rows shift shift shift, >100 feat., >10 cls shift, >100 feat. shift, >100 feat. shift shift, >48k rows

acquire_valued_shoppers_challenge anes_voting_2026 ghanas_indigenous_intel home_credit_default_stability_1m hotel_booking_demand ieee_fraud_detection kick

temporal temporal temporal temporal temporal temporal temporal

111 318 10 711 31 435 32

109,766 27,917 7,442 1,000,000 52,183 321,131 51,118

2 2 4 2 2 2 2

shift, >100 feat., >48k rows shift, >100 feat. shift shift, >100 feat., >48k rows shift, >48k rows shift, >100 feat., >48k rows shift, >48k rows

aps_failure biogeographical_ancestry_prediction bioresponse clock_protein_toxicity customer_satisfaction_in_airline give_me_some_credit hiva_agnostic home_credit_default_risk homesite_quote_conversion kdd_cup_09_appetency ljubljana_primary_tumor lung_cancer lung_cancer_epithelial_genexp mic porto_seguro prostate_cancer_detection santander_customer_satisfaction santander_customer_transaction_prediction sdss_17 sepsis_survival_minimal_clinical_records

iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid iid

170 104 1,776 1,117 21 10 1,518 504 295 212 17 12,600 22,215 111 37 15,154 307 600 11 3

50,667 423 2,501 114 86,587 100,000 2,563 205,005 173,835 33,333 201 131 125 1,133 396,804 215 47,387 133,333 52,035 73,469

2 10 2 2 2 2 2 2 2 2 11 4 2 8 2 2 2 2 3 2

>100 feat., >48k rows >100 feat. >100 feat. >100 feat. >48k rows >48k rows >100 feat. >100 feat., >48k rows >100 feat., >48k rows >100 feat. >10 cls >100 feat. >100 feat. >100 feat. >48k rows >100 feat. >100 feat. >100 feat., >48k rows >48k rows >48k rows

Related documents

• How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?#1298260
• A Bayesian approach to out-of-sample network reconstruction#1233786
• Objective Model Prior Probabilities in Variable Selection#1035431
• Efficient model exploration with the integrated nested Laplace approximation#975334
• Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models#794148