Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Shelly Francis-Meretzki * 1 Mirco Mutti 1 Yaniv Romano 1 Aviv Tamar 1
arXiv:2604.20472v1 [cs.RO] 22 Apr 2026
https://shellytechnion.github.io/TDQC.github.io
Abstract
that a model’s confidence aligns with the empirical outcome frequencies (Guo et al., 2017; Brier, 1950; Xiong et al., 2023; Jiang et al., 2021). When a model is well calibrated, high confidence predictions are correct most of the time, and low confidence predictions correspond to a higher chance of error. This is a desirable property as it enables downstream safety mechanisms that depend on the model’s confidence.
Recent advances in vision-language-action (VLA) models for robotics have highlighted the importance of reliable uncertainty quantification in sequential tasks. However, assessing and improving calibration in such settings remains mostly unexplored, especially when only partial trajectories are observed. In this work, we formulate sequential calibration for episodic tasks, where tasksuccess confidence is produced along an episode, while success is determined at the end of it. We introduce a sequential extension of the Brier score and show that, for binary outcomes, its risk minimizer coincides with the VLA policy’s value function. This connection bridges uncertainty calibration and reinforcement learning, enabling the use of temporal-difference (TD) value estimation as a principled calibration mechanism over time. We empirically show that TD calibration improves performance relative to the state-of-the-art on simulated and real-robot data. Interestingly, we show that when calibrated using TD, the VLA’s singlestep action probabilities can yield competitive uncertainty estimates, in contrast to recent findings that employed different calibration techniques.
Although calibration has been broadly studied, most of the research focuses on single-step decision problems, such as classification and regression (Popordanoska et al., 2024; Dheur & Ben Taieb, 2023). In these problems, the evaluation of the model’s error is direct: Each input is paired with a ground truth label, so that calibration can compare the model’s confidence to the empirical frequency of successes. However, neural networks are commonly used in sequential decision problems as well, for example, in reinforcement and imitation learning (Black et al., 2024; Guo et al., 2025). In sequential tasks, it is not obvious how to define success for individual steps. Think of a robot performing a manipulation task, which requires a sequence of inter-dependent “correct” actions to be solved, and success is only assessable at the end of the sequence. This raises a question: How to reason about calibration in sequential tasks? Differently from previous settings, (i) the ground-truth label is typically delayed; and (ii) the confidence in the current decision may depend on the confidence of future and past decisions.
1. Introduction
In this paper, we provide a formal framework for sequential calibration that encompasses (i, ii). We cast the calibration problem as learning a predictor that takes as input the model’s confidence over time to minimize a sequential version of the Brier score (Brier, 1950). The Brier score is a popular loss that captures both the accuracy and calibration of a predictor by computing the mean squared error between the prediction and the outcome frequency.
Modern neural network based AI systems are typically “black box”, raising concern in applications that require safety and reliability – where a system must not only be accurate but also correctly assess when it is likely to fail. While neural networks often provide a confidence together with their prediction, there is no built-in machinery to enforce that the confidence actually matches the probability of being correct. In this context, calibration explicitly requires
Our main conceptual insight shows that, when formulating a sequential task as a partially-observable Markov decision process, we can connect the problem of minimizing the sequential version of Brier score to the problem of learning a
1
Technion - Israel Institute of Technology. Correspondence to: Shelly Francis-Meretzki <[email protected]>. Preprint. April 23, 2026.
1
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 1. Sequential Brier scores across benchmarks. Sequential Brier score (lower is better) on an unseen validation set averaged over 21 random seeds (train/validation task splits). To compare calibration across rollouts with different lengths, we report Brier score over time quantiles. Each subplot corresponds to a (VLA model, benchmark) pair. Success prediction methods are based on sequences of features or action probabilities. Across all settings, our TD-based methods consistently outperform conventional predictors trained with binary cross entropy (BCE). For π0 action probabilities are not directly interpretable, hence probability-based TDQC variants are not reported. The dotted horizontal line represents the Brier score of a constant predictor that consistently outputs the empirical mean success rate computed over the seen tasks (see Remark 4.2).
value function, which is a popular predictor of task success in Reinforcement Learning (RL, Mannor et al., 2026). This fundamental link allows us to bridge algorithmic insights from RL to sequential calibration, such as the ubiquitous Temporal-Difference (TD) loss (Sutton & Barto, 2018; Mnih et al., 2013; Haarnoja et al., 2018), which bootstraps future value predictions to previous steps, resulting in a superior bias-variance tradeoff (Kearns & Singh, 2000). This provides the backbone for our Temporal-Difference Q-based Calibration (TDQC) method.
prediction in RL, drawing inspiration from the latter to derive a novel TD method for calibration (TDQC); • We empirically show that TDQC, accessing only action probabilities, matches or improves the calibration performance of SAFE accessing the hidden state of the policy in challenging sequential tasks (see Fig. 1 top). This is essential to measure calibration of black-box models, for which the hidden state is often not accessible from APIs; • We show that the TDQC loss, either applied to SAFE or to action probabilities, is an essential ingredient to achieve state-of-the-art early detection results in LIBERO for OpenVLA (Kim et al., 2024), π0 (Black et al., 2024), π0 -FAST (Pertsch et al., 2025) and UniVLA (Wang et al., 2025) models and in a Franka real-robot dataset collected with π0 -FAST (see Table 2). • Finally, as a by-product of the TDQC method, we show that the resulting value predictor can be used to further improve the success rate of a baseline policy by using it to guide the action selection within a set of actions sampled from the policy. In our experiments, this simple recipe yields a 15% increase of the success rate for OpenVLA in the LIBERO benchmark (see Figure 4).
To showcase the potential of sequential calibration, we turn our attention to Vision-Language-Action (VLA) models. VLAs are powerful architectures that have risen to the stateof-the-art for learning robot policies (Black et al., 2024). VLAs encode decision policies that map visual observations and language instructions to probabilities over control actions. Recent works have targeted calibration of those action probabilities to the long-term task success probabilities (Zollo & Zemel, 2025). In particular, Gu et al. (2025) propose SAFE, a method to learn a failure detector that takes as input the hidden state of the VLA and minimizes the cross entropy between the prediction and long-term task success, achieving state-of-the-art failure detection for VLAs.
2. Background
Our paper further advances the research in crucial areas:
2.1. Calibration and Accuracy of Predictors
• We provide the first formulation for calibration in sequential tasks, unifying recent works on failure detection in VLAs (Zollo & Zemel, 2025; Gu et al., 2025) with the broad calibration literature into a coherent framework that singles out the challenges of the sequential setting; • We fundamentally link sequential calibration to value
The accuracy of a prediction model is its frequency of making correct predictions. Calibration measures how well a model’s predicted confidence aligns with empirical outcome frequencies. We formally discuss these concepts in the context of binary classification. 2
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
large when outcomes remain ambiguous (η(F ) ≈ 1/2). In practice, accuracy is typically measured by the ROC AUC.
Definition 1 (Binary Classification). Let (X, Y ) be a pair of random variables where X ∈ X denotes an input, Y ∈ {0, 1} is a binary label, and P(X, Y ) is their joint distribution. A probabilistic classifier is a function f : X → [0, 1]. For input x, the model prediction f (x) is interpreted as the probability that the label is 1.
Calibration of a classifier refers to the process of taking a trained classifier f and an i.i.d. calibration dataset Nc Dcal = {(xi , yi )}i=1 ∼ P Nc and computing a classifier ′ f that is better calibrated. For neural network classifiers, f (x) = σ(logit(x)), where logit(x) is the output of the neural network and σ is a sigmoid function. An effective calibration method is temperature scaling (Guo et al., 2017), yielding f ′ (x) = σ(a · logit(x) + b), where a, b are calculated by maximizing likelihood on Dcal .
Typically, f is calculated from an i.i.d. dataset D = N {(xi , yi )}N i=1 ∼ P , where (xi , yi ) are drawn from P(X, Y ). The Brier score (Brier, 1950) is a strictly proper scoring rule that measures the accuracy of a predictor. Definition 2 (Brier Score for Binary Classifier). The Brier score is the mean squared ℓ2 distance between the prediction and the ground-truth label: h i 2 BS(f ) = EX,Y (f (X) − Y ) .
2.2. POMDPs A Partially Observable Markov Decision Process (POMDP, Åström, 1965) is a popular framework for sequential decision making. It is defined by a tuple (S, X , A, P, O, R, T ) where S is a space of states, X is a space of observations, A is a space of actions, P : S × A → ∆S is a transition model, O : S → ∆X is an observation model, R : S × A → R+ is a reward model, and T ∈ N is a time horizon.
Given a dataset D, it can be estimated as N X 2 c )= 1 BS(f (f (xi ) − yi ) . N i=1
At each step t < T , the POMDP is in a state st ∈ S. The agent observes xt ∼ O(st ) and takes an action at ∈ A. The process transitions to st+1 ∼ P(st , at ) and the agent collects a reward rt ∼ R(st , at ). The process ends in state sT . We denote hT := (x1 , a1 , r1 , . . . , xT ) ∈ H the trajectory of the process and by ht its prefix of t < T steps.2
The Brier score relates to the calibration and accuracy of the classifier by its two-component decomposition. Let F = f (X) denote the random event that the model predicts a particular value, and let η(F ) = P(Y = 1 | F ), the success probability conditioned on the prediction. The Brier score can be decomposed1 as BS(f ) = E (F − Y )2 = E (F − η(F ))2 + E η(F )(1 − η(F )) . (1)
The agent selects actions according to a policy π : H → ∆A with the goal of maximizing the expected cumulative future rewards. The latter is defined through a value function " T −1 # X π π Vi (hi ) = EhT ∈H rt + rT hi ,
The first term relates to calibration: the discrepancy between the predicted success probabilities and the true conditional event frequencies. A classifier is perfectly calibrated if F = η(F ). In practice, calibration is estimated by binning predictions into intervals {Bk }K k=1 and comparing predicted confidence with empirical frequencies. Let f¯k be the mean prediction in bin k, ȳk the empirical positive rate, and nk the bin size. PK Then the empirical calibration error takes the form N1 k=1 nk (f¯k − ȳk )2 , which is a binned estimator of E[(F − η(F ))2 ]. The commonly used Expected Calibration Error (ECE) replaces P the squared deviation by absolute K deviation, ECE(f ) = k=1 nNk |f¯k − ȳk |. It is important to note that calibration does not relate to the accuracy of the predictor. For example, a model that outputs f = E[Y ] independently of X is perfectly calibrated.
t=i
where the expectation is over the action sampled from π, and states, rewards, and observations sampled from the POMDP. Thus, the agent’s objective can be written as maxπ Vtπ (ht ), ∀ht ∈ H. It is typically assumed that the cumulative reward is bounded Vtπ (ht ) ∈ [0, Rmax ]. Finally, we can define an action value function (a.k.a. Q function) as " T −1 # X Qπi (hi , ai ) = EπhT ∈H rt + rT hi , ai . t=i
3. Related Work
The second term in the Brier score decomposition (1) relates to accuracy: it measures how informative the score is about the label. This term is small when predictions induce groups with nearly deterministic outcomes (η(F ) ≈ 0 or 1), and
Calibration in Sequential Settings. In online forecasting, calibration is defined over sequential rounds and analyzed under minimal or adversarial assumptions on the data-generating process (Qiao & Zheng, 2024). In contrast,
1 This decomposition is based on Murphy (1973). A selfcontained proof is given in Apx. B.
2 We denote as H := {∅, H1 , . . . HT } the power set of the trajectories up to length T , where Ht := (X × A × R+ )t .
3
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
our setting considers predictions along a single episodic trajectory with delayed terminal feedback. In temporal calibration (Leathart & Polaczuk, 2020), predictions at different time steps are treated independently, applying post-hoc calibration procedures across temporal slices. In survival analysis (Haider et al., 2020), time-dependent Brier scores are used to evaluate the time until a failure event, different from our task of success evaluation. In large language models (LLMs), uncertainty estimation and calibration are inherently sequential, and important for detecting hallucinations (Huang et al., 2024; Shorinwa et al., 2025; Ren et al., 2023; Xiong et al., 2023; Bar-Shalom et al., 2025). In contrast, our work concerns decision making in dynamic environments.
we improve upon SAFE by rigorously formulating the sequential calibration problem, proposing a TD calibration method, and showing that our method outperforms SAFE using exactly their features. In addition, while Gu et al. (2025) concluded that VLA action probabilities are insufficient for success prediction, we show that when calibrated using our method, action probabilities yield competitive performance. Other existing approaches, which SAFE outperforms, primarily consist of unsupervised out-of-distribution (OOD) detection methods (Xu et al., 2025; Sinha et al., 2024; Wong et al., 2022; Majumdar et al., 2025), which can be overly restrictive for generalist policies, and supervised, task-specific failure detectors (Gokmen et al., 2023; Xie et al., 2022; Farid et al., 2022; Ablett et al., 2020).
In reinforcement learning (RL, Mannor et al. 2026), predicting terminal success corresponds to value estimation, and temporal difference (TD) methods are standard. To the best of our knowledge, the connection with sequential calibration, through the Brier score, is novel. Concurrently with our work, van der Laan & Kallus (2025) connected value learning to calibration, in the context of offline RL. Our work differs in that we consider calibrating a pretrained policy, show a connection between the Brier score and value learning, and demonstrate empirical results on VLAs.
4. Problem Formulation We are given a POMDP policy π. Our goal is to understand, during the run of the policy, whether it will succeed or fail in its task. While this problem has been studied before in various forms (see Section 3), in the following we formally define it in a POMDP setting. As we shall later see, this formulation will allow us to propose new algorithms. We begin by defining a successful outcome. We say that a trajectory PhTT is successful when the cumulative reward R(hT ) = t=1 rt surpasses a threshold c ∈ [0, Rmax ]:
Calibration of VLAs. VLAs provide generalist robot policies. Recent architectures including RT-2 (Brohan et al., 2022), Octo (Octo Model Team et al., 2024), OpenVLA (Kim et al., 2024), π0 (Black et al., 2024), and π0 -FAST (Pertsch et al., 2025), directly map visual observations and textual instructions to low level robot actions, promising zero shot generalization to unseen tasks. However, it is well known that current VLAs exhibit limited success on tasks unseen during training/fine tuning, with reported success rates ranging between 30% and 60% (Kim et al., 2024; O’Neill et al., 2024; Black et al., 2024), motivating study of failure detection.
Y (hT ) = I{R(hT ) ≥ c} ∈ {0, 1}.
(2)
Thus, success depend on the entire trajectory of decisions, which are dependent. Note that for sparse binary rewards Y (hT ) = R(hT ). We would like to evaluate success during the run of the policy, and we henceforth define a sequential counterpart to the conventional Brier score (Brier, 1950) defined in Section 2.1. Let f : Ht → [0, 1] denote a success prediction function given the history ht . Definition 3. The sequential Brier score of a function f is h h ii 2 BSseq (f, t) := Eπht ∈Ht EπhT ∈HT (f (ht ) − Y (hT )) ht (3) where the expectations are taken over trajectories hT and their prefix ht induced by π.
Zollo & Zemel (2025) first studied confidence calibration in VLAs. They reported correlation between task error rates and action calibration error and mainly evaluated confidence on the first action. In contrast, we argue that calibration of individual actions, or without success labels is not sufficient for predicting rollout success (cf. Example 4.1); instead, we use a calibration dataset labeled with episodelevel outcomes and show substantially better calibration and failure prediction performance.
The sequential Brier score can be empirically estimated from a dataset of trajectories and success pairs Dcal = i i i {hiT , y i }N i=1 , where y = Y (hT ) and trajectories hT are sampled with π, as follows.
The most closely related state-of-the-art method is SAFE (Gu et al., 2025), which trains a single cross-task failure detector on internal VLA features and applies conformal prediction (Diquigiovanni et al., 2021) for thresholded decisions. While SAFE does not explicitly target calibration, the method has strong similarities with ours and can be interpreted as an implicit calibration procedure. Nonetheless,
N X 2 c seq (f, t) = 1 f (hit ) − y i . BS N i=1
(4)
Similarly to the conventional Brier score, the sequential Brier score relates to the calibration and accuracy of the 4
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models π ∗ (s0 ) = a′
π ∗ (s1 ) : a′ (0.5) a, r=0
s0
s1 a′ , r=1
minimizes the conventional Brier score for individual action prediction. However, for a threshold c ≤ 1 we have that Y (hT ) = 1 for all trajectories, while π(s1 ) = 0.5.
a, r=0 ⊥ a′ , r=0
As Example 4.1 shows, in a sequential task some actions may be uncorrelated with task success, making action probabilities alone-even if coming from an optimal policyirrelevant as success predictors. To connect this example to a practical VLA setting, when the robot is moving without grasping anything, the gripper action is irrelevant for task success; if a policy is trained to imitate experts that randomly open and close the gripper, it may exhibit high uncertainty for irrelevant actions. Calibrating to task success data is therefore important.
Figure 2. Two-step MDP from Example 4.1.
success predictor by its two-component decomposition. Let Ft = f (ht ) denote the random event that the model predicts a particular value at time t, and let η(Ft ) = P(Y (hT ) = 1 | Ft ), the success probability conditioned on the prediction at time t. The following decomposition holds: (5) BSseq (f, t) = Eπ (Ft − Y (hT ))2 = Eπ (Ft − η(Ft ))2 +Eπ η(Ft )(1 − η(Ft )) ,
We next discuss the worst sequential Brier score that we can expect in practice. h Given some partial i trajectory ht , we 2 π have that minf E (f − Y (hT )) ht = Eπ [ Y (hT )| ht ].
where the first term is referred to as sequential calibration, and the second is accuracy. Thus, minimizing the sequential Brier score leads to desired success predictors that are both calibrated and accurate.
Remark 4.2. Let Eπ [Y (hT )] denote the average task success rate. Any information that exists in the trajectory may be used to obtain a lower Brier score.
Our focus in this work is on finding a success predictor that yields a low Brier score, by computing f from the calibration dataset Dcal . Relating to calibration of classifiers (Section 2), we are particularly interested in whether the policy’s action probabilities π(h0 ), . . . , π(ht ) could be calibrated to yield satisfactory performance in success prediction. We therefore introduce the following definition.
5. Method In this section, we devise a method for sequential calibration. The previous discussion clarifies the relation between sequential calibration and prediction of the task success, through the minimization of the sequential Brier score. The way task success is defined in our setting, i.e., whether the total reward collected during a trajectory surpasses a given threshold (see Eq. 2), essentially implies that we can predict task success by predicting future rewards. The problem of future reward prediction is a staple in the RL literature (Mannor et al., 2026). The following theorem formalizes the connection between the two problems. Theorem 5.1. The future rewards predictor fˆ(ht ) that minimizes the Sequential Brier Score h h ii Eπht ∈Ht EπhT ∈HT (fˆ(ht ) − R(hT ))2 | ht
Definition 4. A success predictor f that depends only on action probabilities, i.e., f (ht ) = f (π(h0 ), . . . , π(ht )), is termed a black box success predictor. Black box success predictors can be employed when only action probabilities are available – a common setting when interfacing commercial foundation models. In contrast, we refer to predictors that require internal access to the VLA model, such as feature activations, as white box success predictors. In the following, we motivate calibration using the sequential Brier score, and explore edge cases. We begin with understanding the importance of the task success labels y i in the calibration dataset. One may hope that a set of trajectories h∗T = (x0 , a∗0 , . . . , xT −1 , a∗T −1 ), which contain expert action labels to states visited by the policy π, are sufficient for sequential calibration. The following example shows that we should not expect this approach to work well in general.
is given by: fˆ(ht ) = EπhT ∈HT [R(hT ) | ht ] =
t−1 X
r(hi ) + Qπt (ht , at ).
i=1
Theorem 5.1 fundamentally links our original problem of calibrating a given policy π with estimating its value function Qπ . In other words, learning the value function is equivalent to learning a calibrated predictor of eventual success. Value estimation is a well-studied problem in RL (Mannor et al., 2026) and allows us to add the family of value estimation algorithms to the calibration toolkit.
Example 4.1. Consider an MDP described in Figure 2 with horizon T = 2 and two actions, a and a′ . The initial state is s0 , and R(s0 , a) = 0, R(s0 , a′ ) = 1. At time step t = 1 the state transitions deterministically to s1 , and R(s1 , a) = R(s1 , a′ ) = 0. The terminal reward is 0. Training data is generated by the following optimal policy π ∗ (s0 ) = a′ w.p. 1, π ∗ (s1 ) = a′ w.p. 0.5, and assume that the learned policy is π = π ∗ . For each time step, π
In this paper, we consider access to a policy π and a set of N labeled trajectories Dcal = {(hiT , y i )N i=1 } generated with the 5
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Algorithm 1 TDQC
tion over tokens at each step. This design is used by RT2 (Zitkovich et al., 2023), OpenVLA (Kim et al., 2024), NORA (Hung et al., 2025) and Uni-VLA (Wang et al., 2025). Since these models resemble LLM-style decoders, we can adopt the common heuristic of using token probabilities and use a black-box success predictor (Def. 4).
Policy π, calibration dataset Dcal = {(hiT , y i )N i=1 }
input 1: Initialize network weights fθ , fθ− 2: for until convergence do 3: Sample hiT uniformly from Dcal 4: Compute the TDQC loss using a target network TP −1 G(θ) = (fθ (hit )−fθ− (hit+1 ))2 +(fθ (hiT )−y i )2
More generally, we can define ht via the policy’s internal representations, such as hidden states from intermediate layers. Using internal features as inputs to an auxiliary predictor is standard in the LLM calibration and probing literature (Azaria & Mitchell, 2023; Zou et al., 2023; Kossen et al., 2024), but requires access to the model’s weights.
t=1
5: Update θ ← θ − α∇G(θ) 6: Periodically update target network θ− ← θ 7: end for
policy, where the label denotes whether the trajectory was successful. The value estimation algorithm fits a parametric function fθ : Hπ → [0, 1] to the data by minimizing a loss N
L(θ) :=
5.1. Early Stopping with Conformal Prediction When a policy π is deployed in safety-critical scenarios, we may want to exploit fθ to stop early trajectories that have low probability of success, avoiding unnecessary damage to the system and its surroundings in failed attempts.
T
1 XX ℓ(fθ (hit ), y i ). N T i=1 t=1
To this purpose, we use fθ as a confidence score for future success, which is equivalent to our original definition of predicting future rewards for binary tasks. Then, fθ can be used to trigger early stopping when the predicted probability of success is low. This can be made formal using conformal prediction (CP, Shafer & Vovk, 2008; Jazbec et al., 2024; Ringel et al., 2024). Following Gu et al. (2025), we treat 1 − fθ as a failure predictor and raise a failure flag whenever 1 − fθ > δt , where δt is a time-varying upper threshold. To calculate that threshold at each time step we first set a significance level α ∈ (0, 1), which determines how conservative we want to be with the early stopping, and we separate a set of successful trajectories from the data before training, to avoid biasing the procedure. On this validation set, we invoke a functional conformal prediction routine (see Algorithm 4 in Apx. C and Diquigiovanni et al. 2021 for details) to compute a time-varying confidence upper bound {δt }Tt=1 . Finally, while the policy is deployed, we process the current trajectory ht with fθ and compare the resulting score with δt . Whenever 1 − fθ (ht ) > δt we stop π. The described procedure is reported in Algorithm 2. Under the exchangeability assumption (Vovk et al., 2005), the CP band guarantees that, for a new successful rollout, 1 − fθ remain below the threshold for all time t w.p 1 − α.
In RL, the single-step loss ℓ(fθ (hit ), y i ) typically takes two forms. One is called Monte Carlo (MC), and prescribes to solve a classification or regression problem between hit and y i . The other is Temporal Difference (TD), which bootstraps the value between two consecutive steps as ( 2 fθ (hit ) − fθ (hit+1 ) t<T i i ℓ(fθ (ht ), y ) = . (6) i i 2 fθ (ht ) − y t=T To ensure stability in training, we used updates with a target network, formally introduced by Mnih et al. (2015). The target network parameters θ− are updated with the current parameters θ every C steps and are held fixed between individual updates. While previous work (Gu et al., 2025) employs an MC loss, TD has not been considered in the context of calibration. TD is the common practice in RL (e.g., Haarnoja et al., 2018) for its well-established practical and theoretical benefits (Kearns & Singh, 2000). In the experiments, we will show that similar benefits extend to the calibration setting. Before that, in Algorithm 1, we provide a brief summary of our method, which we call TemporalDifference Q-based Calibration, TDQC for short. Then, we comment on how to extract information from the history in practice, e.g., when running VLA policies, and how having a calibrated policy opens the door to failure detection and other applications.
5.2. Application to Test-Time Guided Action Search
Extracting the History. In principle, the history ht should contain all signals available at time t that are informative about future success, e.g., the current observation, past actions, and any internal state carried by the policy. In practice, we consider two lightweight ways to extract ht from a VLA.
As an example downstream application of our calibration method, we consider action search at test time. Standard VLA models typically execute the single most probable action predicted by their policy. However, this greedy approach can lead to sub-optimal decisions, especially when attempting to generalize to new environments.
For token-based VLA architectures, actions are discretized into tokens, and the policy outputs a categorical distribu-
Related LLM-based approaches combine search with pro6
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Algorithm 2 Early Stopping
Algorithm 3 Q-Value Guided Action Search
input Policy π, dataset Dcal , confidence α 1: fθ ← TDQC(π, Dcal ) 2: Sample Dval i.i.d. from Dcal 3: {δt }T {Algorithm 4} t=1 ← ConfPred(π, fθ , Dval ) 4: for t = 1, . . . , T do 5: Execute action a ∼ π(xt ) 6: Append (at , rt , xt+1 ) to ht−1 7: if 1 − fθ (ht ) > δt then break 8: end for
input observation xt , policy π, learned Q-function fθ , simulator W , sample size M , threshold T̄ 1: at = arg maxa′ π(a′ |xt ) {Greedy action} 2: if fθ (xt , at ) ≤ T̄ then {High success confidence} 3: Execute at 4: else 5: for i = 1, . . . , M do (i) 6: at ∼ π(xt ) {Sample candidate action} (i) (i) 7: xt+1 ∼ W (xt , at ) (i) (i) 8: at+1 = arg maxa′ π(a′ |xt+1 ) {Greedy action} (i) (i) 9: Q(i) ← fθ (xt+1 , at+1 ) 10: end for 11: i⋆ ← arg maxi Q(i) (i⋆ ) 12: Execute at 13: end if
cess reward models (PRMs), which calculate the quality or correctness of intermediate reasoning steps (Lightman et al., 2023; Zhang et al., 2025). Analogously, to exploit calibration to this end, we propose to use the learned Qfunction fθ to guide action search toward trajectories that are more likely to succeed, inspired by RL methods that employ search at test time (Silver et al., 2016). Furthermore, when fθ indicates a high confidence that the trajectory will succeed, we can save compute resources and skip the search (this is the converse of early stopping, cf. Section 5.1).
2. Can black-box based calibration yield competitive results? 3. Can a calibrated model be used to improve success rate using test-time guided action search?
The pseudocode of the procedure is reported in Algorithm 3. Note that we assume access to an accurate forward model of the system dynamics W = P (xt+1 |ht , at ), which is easily available in simulated domains. For real-world domains, a world model of the dynamics may be learned (Schrittwieser et al., 2020); we leave such investigation for future work.
6.1. Vision-Language-Action Models We evaluate four state-of-the-art VLA policies: OpenVLA (Kim et al., 2024), UniVLA (Wang et al., 2025), π0 (Black et al., 2024), and π0 -FAST (Pertsch et al., 2025). These models span different action parameterizations, which directly changes what confidence signals can be extracted from their outputs. OpenVLA discretizes continuous action into tokens and exposes the corresponding token probabilities, which we use as the action probabilities. π0 -FAST and UniVLA also employs a tokenized representation and apply Discrete Cosine Transform (DCT) to convert continuous action sequences into discrete action tokens. Consequently, token probabilities do not match with the per-dimension continuous action and cannot be directly interpreted as such. The π0 VLA represents continuous action with a flow matching model, rather than per-step token probabilities. All models are open-sourced, allowing access to internal features when needed.
At each time step, given a VLA policy π and a simulator W , we sample from the policy M possible action candidates. For each candidate action i, we simulate a onestep look-ahead to observe the resulting next observation (i) (i) xt+1 ∼ W (xt , at ) , and subsequently select the next action using the standard model’s greedy selection strategy maxa′ π(xt+1 , a′ ). We then evaluate each resulting observation using Q = fθ (xt+1 , at+1 ) and choose the action with highest score a∗ = maxa′ Q(xt+1 , a′ ). We emphasize that fθ parameters are frozen and used only for test-time guided search. To reduce test-time compute using the calibrated model, we apply the action search only when the predicted Q value exceeds a threshold T̄ . We set T such that trajectories selected under this rule have an expected failure rate of at most T %.
Following Definition 4, we refer to methods that predict success using only the model’s observable outputs as black-box. For OpenVLA, our black-box approach uses the discrete action probabilities directly. For π0 -FAST and UniVLA, we observe that the FAST action tokenizer decodes tokens into discrete actions, where each token has its corresponding logits that span across the decoder’s vocabulary size. Our black-box approach takes these logits as input; see Apx. D.3 for more details on the extraction of those probabilities. For all white-box predictors, we use as input the features suggested in SAFE (Gu et al., 2025).
6. Experiments In this section, we empirically validate the connection between calibrating a policy π and estimating its value Qπt . We test TDQC, our sequentially calibrated success predictor in sequential tasks. We aim to answer the following: 1. TD loss improves calibration and failure detection results? 7
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Static Predictors
Learned Predictors (white box)
Learned predictors (black box)
VLA Model Benchmark Eval Task Split Max prob. Avg prob. Running Avg prob. Avg entropy Running Avg entropy SAFE-RNN SAFE-RNN-TDQC (Ours) SAFE-MLP BCE SAFE-MLP-TDQC (Ours) RNN-BCE RNN-TDQC (Ours)
OpenVLA LIBERO Seen ↓ Unseen ↓ 0.395 0.390 0.348 0.364 0.338 0.356 0.306 0.313 0.265 0.273 0.204 0.255 0.197 0.218 0.192 0.231 0.195 0.229 0.199 0.206 0.191 0.197
OpenVLA WidowX Seen ↓ Unseen ↓ 0.572 0.579 0.275 0.282 0.255 0.257 0.414 0.426 0.435 0.432 0.169 0.213 0.096 0.153 0.127 0.164 0.130 0.169 0.301 0.344 0.156 0.192
UniVLA LIBERO Seen ↓ Unseen ↓ 0.909 0.899 0.529 0.532 0.543 0.544 0.406 0.391 0.343 0.330 0.124 0.162 0.064 0.100 0.091 0.158 0.066 0.131 0.138 0.152 0.100 0.107
π0 -FAST LIBERO Seen ↓ Unseen ↓ 0.320 0.318 0.212 0.218 0.244 0.238 0.209 0.222 0.279 0.264 0.106 0.148 0.103 0.163 0.103 0.162 0.109 0.150 0.122 0.158 0.105 0.141
π0 -FAST Franka Seen ↓ Unseen ↓ 0.331 0.323 0.290 0.294 0.359 0.361 0.281 0.281 0.341 0.339 0.220 0.288 0.150 0.215 0.206 0.248 0.210 0.229 0.237 0.243 0.204 0.228
π0 LIBERO Seen ↓ Unseen↓ − − − − − − − − − − 0.123 0.172 0.061 0.097 0.075 0.137 0.068 0.128 − − − −
Table 1. Brier Score results on simulation and real robot experiment. Results are averaged over 21 seeds which determine different train-test splits of the tasks. ”−” indicates that the Brier score can’t be calculated on the method. The best performing methods are highlighted in the table. Our method, TDQC achieves lowest sequential Brier scores in all benchmarks.
Static Predictors
Learned Predictors (white-box)
Learned predictors (black box)
VLA Model Benchmark Eval Task Split Max prob. Avg prob. Running Avg prob. Avg entropy Running Avg entropy SAFE-RNN SAFE-RNN-TDQC (Ours) SAFE-MLP SAFE-MLP-BCE SAFE-MLP-TDQC (Ours) RNN-BCE RNN-TDQC (Ours)
OpenVLA LIBERO Seen ↑ Unseen ↑ 54.64 55.78 47.08 48.09 49.19 47.72 46.81 46.75 50.48 48.09 72.30 69.04 71.67 65.12 73.56 69.53 72.66 64.99 71.22 60.09 72.70 72.28 74.20 72.72
OpenVLA WidowX Seen ↑ Unseen ↑ 53.25 53.20 47.47 48.30 49.20 46.37 50.19 49.36 46.16 43.99 75.95 70.00 84.01 70.76 88.23 83.18 85.38 71.43 82.33 70.64 69.69 67.09 78.90 72.97
UniVLA LIBERO Seen ↑ Unseen ↑ 50.00 50.00 42.96 40.25 43.05 40.29 41.34 47.36 51.63 56.71 74.48 68.13 73.83 64.25 74.39 63.80 78.83 69.74 76.55 66.71 63.44 54.37 72.30 70.17
π0 -FAST LIBERO Seen ↑ Unseen ↑ 61.75 63.49 47.36 48.09 53.95 55.68 45.42 46.30 53.78 55.44 91.26 85.88 92.03 85.49 79.00 68.36 91.82 85.50 90.25 84.44 87.67 82.53 87.51 83.39
π0 -FAST Franka Seen ↑ Unseen ↑ 48.61 46.64 49.45 48.03 52.95 49.98 49.28 49.07 51.12 48.78 70.85 56.69 79.89 68.43 79.15 63.94 73.82 58.83 61.95 51.22 59.32 53.26 64.19 57.37
π0 LIBERO Seen ↑ Unseen↑ − − − − − − − − − − 79.96 65.03 88.66 82.94 86.33 79.75 89.71 80.53 86.57 72.07 − − − −
Table 2. ROC-AUC results on simulation and real robot experiment. Results are averaged over 21 seeds which determine different train-test splits of the tasks. ”−” indicates that the ROC-AUC can’t be calculated on the method. The best performing methods are highlighted in the table. Our method, TDQC achieves SOTA results on unseen tasks in 4 benchmarks out of 6.
6.2. Benchmarks
6.3. Baselines
LIBERO (Liu et al., 2023) is a popular simulated benchmark for VLA evaluation. We evaluated OpenVLA, π0 FAST and π0 on LIBERO-10 suite, which contains 10 long horizon tasks with diverse objects, layouts, and instructions, and is considered the most challenging suite (Gu et al., 2025; Zollo & Zemel, 2025). An episode in LIBERO is stopped once the robot completes its task. If the task has not been completed before some timeout duration, the trajectory is labeled as a failure. We let Ti denote rollout i’s stopping time: either the time of task completion, or the timeout duration on failure (see Gu et al. (2025) and Apx. D.4 for more details).
Static Predictors. We compared Max Prob, Avg Prob and Running Avg Prob presented in Zollo & Zemel (2025), Avg Entropy and Running Avg Entropy from Gu et al. (2025). These methods use the raw action probabilities per time step or apply simple, hand-crafted transformations, such as maximum or average over both action dimension and time, on the raw probabilities and on the probabilities’ entropy. More details and formulation appear in the Apx. D.5. Trained Predictors. We used the baselines from Gu et al. (2025), which provide, to the best of our knowledge, the state-of-the-art failure detection baselines for VLA models. SAFE-MLP uses a small MLP network and takes as input the internal feature vectors (hidden state) e from the model, i.e., ht = et . SAFE-RNN applies a linear projection layer to the input and then an LSTM (Hochreiter & Schmidhuber, 1997). For the evaluation, we followed the same networks and parameters used in Gu et al. (2025) as the f in SAFE methods. Note that both SAFE baselines use hidden states as input, hence are white-box methods.
Real-World WidowX We consider the WidowX data and OpenVLA checkpoints published in Gu et al. (2025). The dataset contains 532 rollouts on 8 lifting pick-and-place tasks. All tasks have rollouts with exactly Ti = 50 steps; if the policy succeeds earlier, the robot keeps taking actions in the environment until reaching Ti (see Gu et al. 2025 and Apx. D.4). Real-World Franka We consider the Franka Emika Panda Robot in Gu et al. (2025) using π0 -FAST-DROID checkpoints (Pertsch et al., 2025). The dataset contains 13 pick and place tasks, each with 30 successful and 30 failed rollouts. Each task k has the same number of steps Tk , regardless if they failed earlier or not, thus Ti = Tk for each rollout i in task k (see Gu et al. 2025 and Apx. D.4).
Apx. D.6 and Gu et al. (2025) provide more details on how the hidden features are extracted and on the architectures of both models. Here we note that the SAFE-MLP over time P loss P aggregates the scores P LMLP = [y (t − f (e )) + (1 − y ) θ t i i i t t fθ (et )] which induces a monotone, time-indexed scoring scheme rather than a calibrated probability. Its uncertainty estimation at each time step is between f (ht−1 ) ≤ f (ht ) ≤ t and 8
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
cannot be treated as a probability. Post-hoc normalization would be arbitrary and distort comparisons. Therefore, we keep the same MLP architecture but replaced the loss with a standard binary cross-entropy objective, yielding SAFEMLP BCE. We compared our method TDQC in two settings: (1) We use black-box success predictor (Def. 4) for OpenVLA, UniVLA and π0 -FAST as described in Section 6.1 combined with TDQC loss (Alg. 1). (2) We incorporate our TDQC loss (Alg. 1) on SAFE baselines where ht is the policy’s internal features at time step t. Note that both settings are alternative ways for approximating Qπt . Figure 3. ROC-AUC vs Brier score over all learned baselines in all benchmarks at the minimum rollout length. Points are grouped by method and split, with a dashed linear fit; the Spearman correlation is ρ = −0.686 which indicates high negative correlation.
6.4. TD loss improves calibration and failure detection results? Calibration. To test the VLA policies’ calibration, we measure the sequential Brier score (Definition 3), which measures calibration and accuracy as shown in Eq. 5. Each rollout i has a horizon Ti , and produces an episode-level binary success label Y (hiT ). To compare calibration across rollouts with different lengths, we report sequential Brier score in different quantiles q ∈ [0, 1], where q = t/Ti denotes a normalized time within rollout i.
task, we compute the minimum rollout length and evaluate ROC-AUC using the maximum prediction up to this timestep. Table 2 shows that predictors with TDQC loss perform better or on par with the best baselines for both white-box and black-box predictors. Based on Table 1 and Table 2, we conclude that TDQC sets the state-of-the-art on unseen tasks on OpenVLA LIBERO, π0 LIBERO, UniVLA LIBERO and real robot π0 -FAST Franka.
Table 1 shows that across models and benchmarks, TDQC improves calibration relative to non-TDQC variants, where SAFE-RNN-TDQC performs the best or on par for all π0 model variants. Moreover, Figure 1 reports sequential Brier scores at different time quantiles within an episode. Early in the rollout, when limited task information is available, TD-based predictors perform comparably to binary crossentropy baselines. As time advances, sequential Brier scores decrease for all methods, reflecting that explicitly modeling the temporal structure yields more accurate success probability estimates than step-wise or purely supervised training.
Finally, since ROC-AUC and sequential Brier are related, we observe in Figure 3 a consistent empirical trend. Methods with lower sequential Brier score tend to achieve higher ROC-AUC, suggesting that better calibrated success probabilities are useful for failure detection. The Spearman correlation of −0.686 highlights the strong connection between the two performance measures. Apx. E.10 examines correlation results per VLA. All VLAs display high correlation, with some VLAs showing higher correlation than others, indicating that their internal representations are more informative about task success.
The dotted horizontal line in Figure 1 represents the Brier score of a constant predictor that, instead of using the model’s task-specific success estimates, consistently outputs the empirical mean success rate computed over the seen tasks, as described in Section 4.2. Consequently, it constitutes an uninformative baseline. We observe that several BCE-based approaches yield little to no improvement relative to this baseline, whereas most TDQC-based approaches attain progressively lower Brier scores as the rollout advances, indicating superior calibration and more informative predictive distributions.
6.5. Can black-box based calibration yield competitive results? To evaluate calibration of action or token probabilities we measured the Brier score for learned black-box methods and static methods. Table 1 shows that black-box predictors achieve best or competitive results on all LIBERO benchmarks, when trained using TDQC. Further, TDQC significantly improves performance of black-box predictors across all benchmarks. In comparison, static baselines, which also use action probabilities as input but are not calibrated by learning from trajectory success labels, display significantly worse performance. These results suggest that the right calibration technique (here, TDQC) can extract useful signal even in a black-box setting.
Failure Detection. We evaluate failure detection using ROC-AUC, which measures how well a score ranks failed rollouts above successful ones and is widely used for uncertainty quantification in LLMs (Huang et al., 2023; Farquhar et al., 2024; Xia et al., 2025). A failure is flagged once the conformal prediction threshold is exceeded. For each 9
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
6.6. Application to guided test-time action search To evaluate the guided test-time search application described in Section 5.2, we measured the success rates of OpenVLA on three unseen tasks from the LIBERO-10 benchmark. We compared the standard, unmodified OpenVLA policy (Baseline) against several value guided search configurations, using Algorithm 3. In Figure 4 we show the success rate relative to the increase in test time compute. RNN with TDQC or BCE action selection methods uses the output of the fθ network for the guidance at all time steps (that is, the threshold in Algorithm 3 is T̄ = −∞), and are trained using either TDQC or BCE loss. Method TDQC - Thresh saves compute by applying a confidence threshold, as described in Algorithm 3. In this experiment we used a threshold T̄ = 0.35, which achieved a good tradeoff between success rate and compute. Apx.E.3 gives more detailed explanation and analysis.
Figure 4. Averaged success rates over additional simulation steps for action selection configurations. All experiments evaluate 3 unseen tasks from LIBERO-10 taken with OpenVLA, consisting of 50 rollouts each, avereged over 3 seeds. RNN-TDQC achieves highest success rates, while RNN-TDQC with thresholds tradeoff success rate and compute.
To increase variance in sampled actions, we generated 10 samples per timestep in all value guided methods using a sampling temperature of 1.5. All experiments are performed on 3 unseen tasks, containing 50 rollouts each.
Limitations. We emphasize three main limitations. First, we observe that the failure predictor generalizes across unseen tasks within an environment but not across environments, embodiments, or action parameterizations. For instance, we report in Apx. E.2 that transferring the OpenVLA LIBERO-10 predictor directly to the WidowX environment yields only 53.69 ± 5.07 ROC-AUC and a sequential Brier score of 0.413 ± 0.03. We believe that such broad generalization may require a different approach. Second, our experimental results are performed with binary episodic success; Note however that our formulation in Section 4 accounts for general reward functions and TDQC can easily be extended to such. Third, our guided action-selection operates under a one-step look-ahead. While effective in simulation, this requires environment access at inference time and is therefore not directly deployable in real-world settings without a forward model. Recent advances in world models for robotics (Badithela et al., 2025; Gemini Robotics Team et al., 2026) offer a promising path toward addressing this.
Figure 4 shows that RNN methods with top-10 probabilities gain significant improvement and outperform the baseline VLA policy, validating the use of learned scoring function (fθ ) for action selection. Notably, the RNN method trained with TDQC achieves the highest overall performance, reaching 55% success rate on average, an improvement of 13% over the regular baseline. While the BCE loss variant also improves upon the baseline, it peaks lower at 50%. Moreover, applying a threshold to the TDQC method (TDQC, Thresh 0.35) results in a slight performance drop compared to the unconstrained TDQC approach, but saves almost half of the additional compute. Interestingly, we observe a nearlinear relation between additional compute and success rate (the gray line in Figure 4 is a fit on all RNN top-10 methods). Additional details are provided in Apx. E.3.
7. Conclusion and Future Works
Future Works. We conclude with several potential future extensions of our work. Recent advances on world models (Gemini Robotics Team et al., 2026; Badithela et al., 2025) could be exploited to improve uncertainty estimation of robot policies in the real world. Our approach could also be extended for LLM calibration (Orgad et al., 2025; Radharapu et al., 2025), interpreting text generation as a sequential task.
In this paper, we proposed a novel formulation for calibration in sequential tasks. We showed that predictors calibrated using our temporal-difference based approach achieve SOTA performance in several VLA benchmarks. Importantly, our approach can work with only action probabilities, which is essential for calibration of black-box proprietary models accessed via APIs. We see interesting potential in using calibrated success predictors for action selection at test time. In particular, a calibrated predictor may be used to allocate search effort only to “difficult” decisions – a promising future direction for resource management.
Acknowledgments We thank Qiao Gu for helpful conversations and for releasing his code and data for SAFE. This research was funded by the European Union (ERC, Bayes-RL, 101041250). Views and opinions expressed are however those of the author(s) 10
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. Y. R. was supported by the Israel Science Foundation (ISF grant 729/21). He also acknowledges the additional support from the Career Advancement Fellowship at the Technion.
Diquigiovanni, J., Fontana, M., and Vantini, S. The importance of being a band: Finite-sample exact distributionfree prediction sets for functional data. arXiv preprint arXiv:2102.06746, 2021. Farebrother, J., Orbay, J., Vuong, Q., Taiga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value functions via classification for scalable deep rl. In International Conference on Machine Learning, 2024.
Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Farid, A., Snyder, D., Ren, A. Z., and Majumdar, A. Failure prediction with statistical guarantees for vision-based robot control. In Robotics: Science and Systems, 2022.
References
Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024.
Ablett, T., Marić, F., and Kelly, J. Fighting failures with fire: Failure identification to reduce expert burden in intervention-based learning. arXiv preprint arXiv:2007.00245, 2020.
Gemini Robotics Team, Choromanski, K., Devin, C., Du, Y., Dwibedi, D., Gao, R., Jindal, A., Kipf, T., Kirmani, S., Leal, I., et al. Evaluating gemini robotics policies in a veo world simulator. arXiv preprint arXiv:2512.10675, 2026.
Åström, K. J. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174–205, 1965. Azaria, A. and Mitchell, T. The internal state of an LLM knows when it’s lying. In Conference on Empirical Methods in Natural Language Processing, 2023.
Gokmen, C., Ho, D., and Khansari, M. Asking for help: Failure prediction in behavioral cloning through value approximation. In International Conference on Robotics and Automation, 2023.
Badithela, A., Snyder, D., Zha, L., Mikhail, J., O’Kelly, M., Dixit, A., and Majumdar, A. Reliable and scalable robot policy evaluation with imperfect simulators. arXiv preprint arXiv:2510.04354, 2025.
Gu, Q., Ju, Y., Sun, S., Gilitschenski, I., Nishimura, H., Itkina, M., and Shkurti, F. Safe: Multitask failure detection for vision-language-action models. Advances in Neural Information Processing Systems, 2025.
Bar-Shalom, G., Frasca, F., Lim, D., Gelberg, Y., Ziser, Y., El-Yaniv, R., Chechik, G., and Maron, H. Learning on llm output signatures for gray-box behavior analysis. arXiv preprint arXiv:2503.14043, 2025.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. pi 0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950, 1950. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
Haider, H., Hoehn, B., Davis, S., and Greiner, R. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(85): 1–63, 2020.
Dheur, V. and Ben Taieb, S. A large-scale study of probabilistic calibration in neural network regression. In International Conference on Machine Learning, 2023.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 11
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Huang, X., Li, S., Yu, M., Sesia, M., Hassani, H., Lee, I., Bastani, O., and Dobriban, E. Uncertainty in language models: Assessment through rank-calibration. In Conference on Empirical Methods in Natural Language Processing, 2024.
Majumdar, A., Sharma, M., Kalashnikov, D., Singh, S., Sermanet, P., and Sindhwani, V. Predictive red teaming: Breaking policies without breaking robots. In Conference on Robot Learning, 2025. Mannor, S., Mansour, Y., and Tamar, A. Reinforcement Learning: Foundations. Cambridge University Press, 2026. URL https://sites.google.com/view/ rlfoundations/home.
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., JuefeiXu, F., and Ma, L. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236, 2023.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Hung, C.-Y., Sun, Q., Hong, P., Zadeh, A., Li, C., Tan, U., Majumder, N., Poria, S., et al. Nora: A small opensourced generalist vision language action model for embodied tasks. arXiv preprint arXiv:2504.19854, 2025.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529–533, 2015.
Jazbec, M., Timans, A., Veljković, T. H., Sakmann, K., Zhang, D., Naesseth, C. A., and Nalisnick, E. Fast yet safe: Early-exiting with risk control. Advances in Neural Information Processing Systems, 2024.
Murphy, A. H. A new vector partition of the probability score. Journal of Applied Meteorology and Climatology, 12(4):595–600, 1973.
Jiang, Z., Araki, J., Ding, H., and Neubig, G. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962– 977, 2021.
Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024.
Kearns, M. J. and Singh, S. Bias-variance error bounds for temporal difference updates. In Conference on Learning Theory, 2000.
Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., and Belinkov, Y. Llms know more than they show: On the intrinsic representation of llm hallucinations. In International Conference on Learning Representations, 2025.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. Openvla: An open-source vision-languageaction model. In Conference on Robot Learning, 2024.
O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In International Conference on Robotics and Automation, 2024.
Kossen, J., Han, J., Razzak, M., Schut, L., Malik, S., and Gal, Y. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927, 2024. Leathart, T. and Polaczuk, M. Temporal probability calibration. arXiv preprint arXiv:2002.02644, 2020.
Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations, 2023.
Popordanoska, T., Tiulpin, A., and Blaschko, M. B. Beyond classification: Definition and density-based estimation of calibration in object detection. In WACV, 2024.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 2023.
Qiao, M. and Zheng, L. On the distance from calibration in sequential prediction. In Conference on Learning Theory, 2024.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
Radharapu, B., Saxena, E., Li, K., Whitehouse, C., Williams, A., and Cancedda, N. Calibrating llm judges: Linear 12
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
probes for fast and reliable uncertainty estimation. arXiv preprint arXiv:2512.22245, 2025.
Xia, Z., Xu, J., Zhang, Y., and Liu, H. A survey of uncertainty estimation methods on large language models. In Findings of the Association for Computational Linguistics, 2025.
Ren, A. Z., Dixit, A., Bodrova, A., Singh, S., Tu, S., Brown, N., Xu, P., Takayama, L., Xia, F., Varley, J., et al. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning, 2023.
Xie, A., Tajwar, F., Sharma, A., and Finn, C. When to ask for help: Proactive interventions in autonomous reinforcement learning. Advances in Neural Information Processing Systems, 2022.
Ringel, L., Cohen, R., Freedman, D., Elad, M., and Romano, Y. Early time classification with accumulated accuracy gap control. In International Conference on Machine Learning. PMLR, 2024.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In Conference on Learning Representations, 2023.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839): 604–609, 2020.
Xu, C., Nguyen, T. K., Dixon, E., Rodriguez, C., Miller, P., Lee, R., Shah, P., Ambrus, R., Nishimura, H., and Itkina, M. Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies. In Robot Evaluation for the Real World, 2025.
Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics, 2025.
Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., and Majumdar, A. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 2025.
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, 2023.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
Zollo, T. P. and Zemel, R. Confidence calibration in vision-language-action models. arXiv preprint arXiv:2507.17383, 2025.
Sinha, R., Elhafsi, A., Agia, C., Foutter, M., Schmerling, E., and Pavone, M. Real-time anomaly detection and reactive planning with large language models. arXiv preprint arXiv:2407.08735, 2024.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018. van der Laan, L. and Kallus, N. Bellman calibration for vlearning in offline reinforcement learning. arXiv preprint arXiv:2512.23694, 2025. Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world. Springer, 2005. Wang, Y., Li, X., Wang, W., Zhang, J., Li, Y., Chen, Y., Wang, X., and Zhang, Z. Unified vision-language-action model. In International Conference on Learning Representations, 2025. Wong, J., Tung, A., Kurenkov, A., Mandlekar, A., Fei-Fei, L., Savarese, S., and Martı́n-Martı́n, R. Error-aware imitation learning from teleoperation data for mobile manipulation. In Conference on Robot Learning, 2022. 13
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
A. Proof for Theorem 5.1 Fix a policy π and consider an episodic process with horizon T . Let ht denote the history available up to time t, as defined in Section 2.2. We study the problem of predicting the future rewards from partial information: at time t we observe ht and seek a predictor fˆ(ht ) of the total future rewards. R := R(hT ) =
T X
r(hi ),
(7)
i=1
where r(hi ) denotes the (possibly stochastic) reward at step i given the history ht . Theorem 5.1 states that the predictor minimizing the conditional mean squared error h h ii Eπht ∈Ht EπhT ∈HT (R(hT ) − fˆ(ht ))2 | ht is given by: fˆ(ht ) = EπhT ∈HT [R(hT ) | ht ] =
t−1 X
r(hi ) + Qπt (ht , at )
i=1
Meaning that, in order to minimize the sequential Brier Score, one must approximate the Q-function Qπt (ht , at ). Proof. We begin by denote R(hT ) := R for ease of notation. We recall that the optimal estimator in terms of MSE takes the form of EhT ∈HT [R | ht ]. Suppose we stop at step t and can only access ht , meaning we don’t know hT and thus, don’t have access to {r(hi )}Ti=t+1 , PT so we’ll replace EπhT ∈HT [ j=t+1 r(hj ) | ht ] with fˆ(ht ). t T h i X X markov EhT ∈HT R ht = EπhT ∈HT r(hi ) + r(hj ) ht = i=1
EπhT ∈HT
" t X
j=t+1
#
r(hi ) ht + EπhT ∈HT
i=1
T X
r(hj ) ht =
j=t+1
t X
r(hi ) + EπhT ∈HT
i=1
T X
r(hj ) ht =
j=t+1
t X
r(hi ) + fˆ(ht )
i=1
Where the last equality holds since we know the rewards and histories until time step t. Note that:
EπhT ∈HT
T X
r(hj ) ht = Qπt (ht , at ) − r(ht )
j=t+1
The amended sequential Brier Score for this case is: Eπht ∈Ht EπhT ∈HT R −
t X
!2 r(hi ) − fˆ(ht )
ht
i=1
Because hwe have information until time step t, then fˆ(ht ) is constant and can be treated as a number. Also, i P t EπhT ∈HT i=1 r(hi ) is a constant and can be outside of the inner expectation. Let’s look at the inner expectation: !2 t t X X EπhT ∈HT R − r(hi ) − fˆ(ht ) ht = EπhT ∈HT R2 | ht − 2 · EπhT ∈HT [R | ht ] · r(hi ) i=1
− 2 · EπhT ∈HT [R | ht ] · fˆ(ht ) + 2 ·
i=1 t X
r(hi ) · fˆ(ht ) +
i=1
t X i=1
14
!2 r(hi )
+ fˆ(ht )2
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
2 We can add and subtract EπhT ∈HT [R | ht ] from the equation: t X EπhT ∈HT R2 | ht − 2 · EπhT ∈HT [R | ht ] · r(hi ) − 2 · EπhT ∈HT [R | ht ] · fˆ(ht ) i=1
+2·
t X
r(hi ) · fˆ(ht ) +
i=1
t X
!2 2 2 + fˆ(ht )2 + EπhT ∈HT [R | ht ] − EπhT ∈HT [R | ht ]
r(hi )
i=1
= Var (R | ht ) − 2 · EπhT ∈HT [R | ht ] ·
t X
r(hi ) − 2 · EπhT ∈HT [R | ht ] · fˆ(ht )
i=1
+2·
t X
r(hi ) · fˆ(ht ) +
i=1
t X
!2 2 + fˆ(ht )2 + EπhT ∈HT [R | ht ]
r(hi )
i=1
= Var (R | ht ) +
EπhT ∈HT [R | ht ] −
t X
!2 r(hi ) − fˆ(ht )
i=1
Lets break down the term EπhT ∈HT [R | ht ]: EπhT ∈HT [R | ht ] = EπhT ∈HT
t T X X r(hi ) + r(hj ) | ht = i=1
t X
" r(hi ) + EπhT ∈HT
i=1
T X
(8)
j=t+1
# r(hi ) | ht =
t X
" r(hi ) + EπhT ∈HT
i=1
i=t+1
T X
# r(hi ) | ht
i=t+1
We can use Equation (8) and the inner expectation looks like: EπhT ∈H
h
i (R(hT ) − fˆ(ht ))2 | ht = Var (R | ht ) +
" EπhT ∈HT
T X
# r(hi ) | ht
!2 ˆ − f (ht )
i=t+1
The fˆ(ht ) that will minimize this term is (inverse of law of total expectation): " T # X π ˆ f (ht ) = E r(hi ) | ht = Qπ (ht , at ) − r(ht ) hT
t
i=t+1
B. Brier Score Decomposition By adding and subtracting η(F ) and expanding the square, (F − Y )2 = (F − η(F ) + η(F ) − Y )2 = (F − η(F ))2 + (η(F ) − Y )2 + 2(F − η(F ))(η(F ) − Y ). Taking expectations, the cross term vanishes since E[(η(F ) − Y ) | F ] = 0, and therefore E[(F − η(F ))(η(F ) − Y )] = 0. Moreover, since Y | F ∼ Bernoulli(η(F )), E (η(F ) − Y )2 | F = η(F )(1 − η(F )). 15
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Algorithm 4 Split-Conformal Trajectory Upper Band with Modulation input Calibration rollouts Dcal , miscoverage level α ∈ (0, 1), scoring network S(·; θ), rollout horizon T j N2 1 1: Split Dcal into two disjoint sets: DcalA = {P k }N k=1 and DcalB = {P }j=1 . 2: Compute mean successful trajectory on DcalA : 3: for t = 1, . . P . , T do N1 S(Ptk ; θ) 4: µt ← N11 k=1 5: end for 6: Compute trimming threshold γ on DcalA : 7: for k = 1, . . . , N1 do 8: Mk ← maxt∈[T ] S(Ptk ; θ) − µt 9: end for 1 10: γ ← Quantile1−α {Mk }N k=1 11: Define index set H for modulation computation: ( [N1 ], if (N1 + 1)(1 − α) > N1 , H= k k ∈ [N1 ] : maxt∈[T ] S(Pt ; θ) − µt ≤ γ , otherwise, 12: Compute modulation function scalA (t): 13: for t = 1, . . . , T do 14:
scalA (t) ← max S(Ptk ; θ) − µt k∈H
(9)
15: end for 16: Compute normalized max-deviation scores on DcalB : 17: for j = 1, . . . , N2 do n o µ −S(P j ;θ)
t Dj ← maxt∈[T ] tscal (t) A 19: end for 20: D ← {Dj | j = 1, . . . , N2 } 21: Compute conformal band width and upper bound: 22: h ← Quantile1−α (D) 23: for t = 1, . . . , T do 24: δt ← µt + h · scalA (t) 25: end for
18:
C. Functional Conformal Prediction We follow (Gu et al., 2025; Xu et al., 2025; Diquigiovanni et al., 2021) for one-sided CP band calculations to determine the time varying threshold δt and use two calibration datasets DcalA , DcalB . Under the exchangeability assumption (Vovk et al., 2005) and given a user-specified significance level α, the CP band guarantees that, for a new successful rollout, its failure probability 1 − fθ will lie within band δt for all times t with probability 1 − α. Conversely, if the failure probability of a test rollout rises above that threshold, we can label it as a failure with nominal confidence of 1 − α. See Alg. 4 for the detailed calculations. Note that this assumption holds for every time step t since we consider the maximal failure detection score until time step t and not 1 − fθ (ht ). Intuitively, Eq. (9) in Algorithm 4 removes the top α most extreme rollouts from set DcalA before calculating the width of the prediction bands. This ensures the actual band width represents the bulk of the data, not the outliers. Additionally, the maximum over t is taken because the conformal prediction band is intended to reflect the entire trajectory. We define D = {Dj | j = 1, . . . , N2 } as the collection of such maximum deviations from the mean. The band width h is computed as the (1 − α)-quantile of S, and the upper bound is given by τt = uppert = µt + h · scalA (t) 16
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Note that Dj ’s were dividing by scalA (t) in Eq. (9), we then multiply by this term in line 24 to map the results back to the original scale.
D. Experiments Details D.1. Code You can find the Github repository here: https://github.com/shellytechnion/TDQC This repository contains the implementation for this paper. The codebase is built upon and extends the Gu et al. (2025) work on failure detection in Vision-Language-Action (VLA) models. D.2. VLA models We show experiments on 4 state-of-the-art open source VLA models: OpenVLA (Kim et al., 2024), UniVLA (Wang et al., 2025), π0 -FAST (Pertsch et al., 2025) and π0 (Black et al., 2024). OpenVLA use MIT license; π0 and π0 -FAST use Apache-2.0 license. At the time of writing, UniVLA does not specify an open-source license in its repository. D.3. Action and Token Probability Modeling OpenVLA (Kim et al., 2024) represents an action with D discrete tokens, where each token represents one dimension of the (d) robot’s action space (DoF). For each dimension d ∈ {1, . . . , D}, the policy outputs logits zt and probabilities (d) (d) (d) (d) pt = softmax zt , pt,k = πθ At = k | ot , for action tokens k ∈ {1, . . . , K} in an action vocabulary of size K. At time t, the policy selects the top tokens (d)
(d)
at
= arg max pt,k k
for each action dimension d, and decodes them into a continuous action for execution by the robot. Both the UniVLA and π0 -FAST models use the FAST tokenizer (Pertsch et al., 2025), which applies a Discrete Cosine Transform (DCT) to encode continuous action trajectories into discrete action tokens. Given a window of size H, we define the raw continuous action sequence as A1:H = {a1 , a2 , . . . , aH }, where each step at is a d-dimensional vector. During training, the FAST tokenizer encodes this continuous sequence into a discrete token sequence La = [T1 , . . . , Tn ], where each token Ti is drawn from a vocabulary V of size |V|. Analogous to natural language processing, sequence lengths can vary, resulting in a variable-length (n) discrete representation. During inference, these discrete tokens are decoded back into (T ) continuous robot actions A1:H . For each token Ti , the model outputs logits zt i over the vocabulary, which are converted into probabilities via a softmax operation: (T ) (T ) pt i = softmax zt i (10) We refer to these as token probabilities, which we utilize in our “black-box” configuration as ht . D.4. Benchmarks Statistics Benchmark
LIBERO Real Franka Real WidowX
Number of Tasks
Number of rollouts
Seen
Unseen
Total
Train
Eval Seen
Eval Unseen
Total
7 10 6
3 3 2
10 13 8
210 450 250
140 150 133
150 180 149
500 780 532
Table 3. Benchmark statistics: task split into seen/unseen subsets and corresponding numbers of training and evaluation rollouts.
Table 3 summarize each benchmark statistics on the number of tasks and rollouts. We note that while TDQC methods requires training data, they exhibit an improvement over the baselines in the real-robot experiments where they are trained on a much smaller dataset. Table 4 shows the number of timesteps per benchmark and VLA model pairs.
17
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models Benchmark
VLA Model
Number of Environment Steps
LIBERO-10
OpenVLA π0 π0 -FAST UniVLA
199, 279 151, 755 183, 488 134, 746
WidowX
OpenVLA
26, 600
Franka
π0 -FAST
36, 180
Table 4. Episode horizon by benchmark and model. We report the maximum rollout length (environment time limit) used in each evaluation setting.
Model
LIBERO-10
OpenVLA π0 -FAST π0 UniVLA
52% 60.2% 82.2% 90.6%
Table 5. LIBERO-10 Task success rates across models
LIBERO-10 We evaluate the VLA models on LIBERO-10, which consists of 10 long-horizon manipulation tasks and contains the most diverse objects, environments and instructions among the 4 LIBERO suits, thus considered the most challenging suite. We kept the same initial conditions of each enviorment as specified in the benchmark details.3 We constructed a task-level split into training, validation, and testing: we train and validate on 7 tasks and evaluate generalization on the remaining 3 unseen tasks. The specific tasks split is re-sampled over random seeds, i.e., each seed induces a different set of test tasks. Within the 7 training tasks, we further split rollouts into train/validation with a 60%/40% ratio. Each task includes 50 randomized initializations (e.g., variations in object placement), meaning a total of 10 × 50 = 500 episodes. Table 5 reports the rollout success rates of all evaluated VLA policies. In LIBERO, the simulator terminates an episode as soon as the task succeeds, or when a maximum horizon is reached (520 steps for LIBERO-10). As a result, successful rollouts are often shorter than failed ones, which can cause a success estimators to induce information based on the length of the rollout alone. To ensure a fair comparison of ROC-AUC, we follow a fixed-cutoff method that was introduced in Gu et al. (2025). For each task, we compute the minimum rollout length observed across its episodes and use this value as the effective horizon T for evaluation Tk = minr∈k Tr where k denotes the task and r is a rollout. Meaning, we only report ROC-AUC values up to this termination time. This ensures that all rollouts for a given task are evaluated on comparable trajectory prefixes. Real Robot WidowX We used published data4 that tested OpenVLA pretrained on ”Open-X Magic soup++” dataset on real WidowX robot arm which achieved success rate of 45.8%. In this experiment, they collected 532 rollouts on the 8 tasks, with 244 successes and 288 failures. Each task has roughly the same number of rollouts. We constructed a task-level split into training, validation, and testing: we train and validate on 6 tasks and evaluate generalization on the remaining 2 unseen tasks. The specific tasks split is re-sampled over random seeds, i.e., each seed induces a different set of test tasks. When we evaluated on multiple seeds, each seed determines a different configuration of train and test tasks split. Within the 6 training tasks, we further split rollouts into train/validation with a 66%/33% ratio. For each task, they set a fixed number of allowed time steps (50) and all rollouts ended in the same time step, even if success occurred before that time step. Real-world Franka We used published data5 that tested π0 -FAST on various real-world experiments. For each task, the maximum time horizon was predetermined and constant inside each task. We split to train, validation and test such that we train and validate on 10 tasks and test on 3 tasks. The authors of (Gu et al., 2025) recorded the rollouts such that the dataset will have 50% success on each task. 3
https://github.com/Lifelong-Robot-Learning/LIBERO/tree/master/libero/libero/init_files/ libero_10 4 https://github.com/vla-safe/SAFE 5 https://github.com/vla-safe/SAFE
18
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
D.5. Static baselines Many VLA models discretize continuous actions into descrete action tokens. At time step t, the policy produces a distribution over tokens conditioned on the trajectory history and the language instruction, pt (a) = πθ (a | ht , l),
at = arg max pt (a), a
(11)
where ht denotes the available history (e.g., past observations and actions) and l is the language instruction. For robots with |D| |D| degrees of freedom (DoF), we write the predicted token vector as At = (a1t , . . . , at ), with corresponding probabilities |D| Pt = (p1t , . . . , pt ), where pit ≜ pt (ait ) ∈ (0, 1] is the probability assigned to the selected token for DoF i. Let Hti denote the entropy of the token distribution for DoF i at time t. We follow the LLM hallucinations literature (Xiong et al., 2023; Huang et al., 2024; Shorinwa et al., 2025; Ren et al., 2023) and Zollo & Zemel (2025) and used the same methods to measure uncertainty in VLA model. We compute the following uncertainty scores at each timestep. For probability-based scores we use the negative log-probability, so that larger values indicate higher uncertainty: Max Prob(t) = max(−log(pit )) i
|D|
Avg Prob(t) = −
1 X log(pit ) |D| i=1
Running Avg Prob(t) = −
t |D| 1 1 XX log(pij ) t |D| j=1 i=1
|D|
Avg Entropy(t) =
1 X i H |D| i=1 t
Running Avg Entropy(t) =
t |D| 1 1 XX i H t |D| j=1 i=1 j
The first two families use the confidence of the chosen action’s DoF (via − log pit ), i.e. the token in the action that corresponds to the highest probability, while the entropy scores uses the full token distributions. Finally, Running Avg Prob aggregates uncertainty over time via a running average, which can be interpreted as a lightweight temporal smoothing baseline (without learning). D.6. Trained Baselines SAFE-RNN uses an LSTM model with 1 layer and a hidden dimension of 256, and an additional linear layer is used to project the hidden states of LSTM into a single scalar state. SAFE-MLP uses a multi-layer perceptron with 2 layers and a hidden dimension of 256. RNN-BCE and RNN-TDQC methods use the same LSTM architecture for all benchmarks, except OpenVLA, where the LSTM is replaced by a GRU. In OpenVLA with action probabilities, we trained a GRU network with A two-layer MLP with and a 1 layer GRU, the GRU’s output is passed through a two-layer MLP head that produces a single scalar output. Since successes and failures from the generated rollouts are imbalanced, the losses on positive (failed) and negative (successful) rollouts are weighted by their inverse class frequency. Given the internal feature vectors E ∈ Rn×d produced by a VLA model, where n corresponds to token positions, diffusion steps, etc. and d is the feature dimension. Gu et al. (2025) aggregate E into a single fixed-dimensional feature vector e ∈ Rd before inputting to SAFE models. They ablate on which feature vector to use in order to achieve highest ROC-AUC. In π0 Gu et al. (2025) takes the feature vectors before the projection into the velocity field. The aggregation methods in π0 are denoted as agghori , agghdiff Other implementation details appear both in SAFE paper (Gu et al., 2025) and in https://github.com/vla-safe/SAFE. 19
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
D.7. Training Details Following SAFE, for both metrics sequential Brier Score and ROC-AUC, we follow the evaluation in SAFE and interpreted the uncertainty estimations (probabilities) as the probability of failure. We use AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay of 0.01 , and learning rate (lr) that was determined by grid search and a step LR scheduler with gamma determined by a grid search. All models were trained for 1000 epochs with batch size 512. We also ablate on applying L2 regularization λreg loss on the model weights to reduce weights. Note that for the TDQC method that gets as an input the model’s probabilities we entered each data point sequentially. We concatenated all the rollount and samples ”starting index” from which we took a batch size next steps. If the rollout ended in the middle of the batch - we reset the network’s hidden state. In SAFE methods, each rollout is considered as one data point and thus batch size of 512 translates to training on (at most) 512 rollouts in each iteration. All training and evaluation are done on a single NVIDIA RTX 5090 32GB GPU. D.8. Hyperparameter Tuning To tune TDQC, we perform a grid search over hyperparameters and select the configuration that maximizes sequential Brier score on a held-out validation set of rollouts from the same tasks as in training. Unlike Gu et al. (2025), which evaluates each hyperparameter setting across all random seeds and then reports the best-performing one (in terms of ROC-AUC), we decouple tuning and evaluation: we run the hyperparameter sweep on two tuning seeds and then fix the selected configuration and evaluate it on an additional nineteen seeds (21 total). Using a large number of evaluation seeds reduces selection bias and yields statistically meaningful comparisons. In Tables 6 to 11 we report the hyper-parameters we have searched over and the values of the best performance. For the SAFE methods - we used their chosen parameters for the LSTM and MLP methods. For MLP-BCE method we searched the same grid to find the best parameters. For SAFE methods without TD we choose the parameters that maximized the ROC-AUC, and for the TDQC parameters, we choose the ones that minimize the sequential Brier Score. Unless stated otherwise, in action probability methods (OpenVLA model) we concatenated the top 10 probabilities for each degree of freedom, used learning rate step size of 200 and batch size of 512. For all methods, LRλ = 1, and the number of layers is 2 for the projection in MLP and for LSTM method we used 1 LSTM layer. Table 6. Hyperparameter search space for OpenVLA + LIBERO benchmark. Method SAFE-RNN
SAFE-RNN-TDQC
SAFE-MLP SAFE-MLP BCE
SAFE-MLP-TDQC
RNN-BCE
RNN-TDQC
HParams aggtoken lr λreg aggtoken lr lr γ λreg aggtoken lr λreg aggtoken lr λreg aggtoken lr λreg LR-γ GRU hidden head hidden lr lr γ λreg GRU hidden head hidden lr lr γ λreg
Table 7. Hyperparameter search space for OpenVLA + WidowX benchmark.
Values First Last Mean 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 1 First Last Mean 1e-5 5e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 First Last Mean 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 1 First Last Mean 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 1 First Last Mean 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 0.8 0.1 256 512 256 512 1e-5 5e-5 1e-4 1e-3 0.1 0.8 0 1e-3 1e-2 1e-1 256 512 256 512 1e-5 5e-5 1e-4 1e-3 0.1 0.8 0 1e-3 1e-2 1e-1
Method SAFE-RNN
SAFE-RNN-TDQC
SAFE-MLP SAFE-MLP BCE
SAFE-MLP-TDQC
RNN-BCE
RNN-TDQC
20
HParams aggtoken lr λreg aggtoken lr lr γ λreg aggtoken lr λreg aggtoken lr λreg aggtoken lr λreg LR-γ GRU hidden head hidden lr lr γ λreg GRU hidden head hidden lr lr γ λreg
Values First Last Mean 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 1 First Last Mean 1e-5 5e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 First Last Mean 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 1 First Last Mean 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 1 First Last Mean 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 0.1 0.8 256 512 1024 256 512 1e-5 5e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 256 512 1024 256 512 1e-5 5e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models Table 9. Hyperparameter search space for π0 -FAST + Franka benchmark. Table 8. Hyperparameter search space for π0 -FAST + LIBERO benchmark. Method SAFE-RNN
SAFE-RNN-TDQC
SAFE-MLP
SAFE-MLP BCE
SAFE-MLP-TDQC
RNN-BCE RNN-TDQC
HParams aggtoken Feat lr λreg aggtoken Feat lr lr γ λreg aggtoken Feat lr λreg aggtoken Feat lr γ lr λreg aggtoken Feat lr λreg lr γ lr λreg lr γ lr λreg lr γ
Method SAFE-RNN
Values First Last Mean Encoded Pre-logits 3e-5 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 First Last Mean Encoded Pre-logits 1e-5 5e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 First Last Mean Encoded Pre-logits 3e-5 1e-4 1e-4 1e-3 1e-3 1e-2 1e-1 First Last Mean Encoded Pre-logits 0.1 0.8 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 First Last Mean Encoded Pre-logits 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 0.1 0.8 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8
SAFE-RNN-TDQC
SAFE-MLP
SAFE-MLP BCE
SAFE-MLP-TDQC
RNN-BCE
RNN-TDQC
SAFE-RNN
SAFE-RNN-TDQC
SAFE-MLP
SAFE-MLP BCE
SAFE-MLP-TDQC
HParams agghorri aggdiff lr λreg agghorri aggdiff lr lr γ λreg agghorri aggdiff lr λreg agghorri aggdiff lr lr γ λreg agghorri aggdiff lr lr γ λreg
Values Mean Pre-logits 128 256 1e-4 3e-4 1e-3 3e-3 1e-3 1e-2 1e-1 Mean Pre-logits 128 256 1e-5 5e-5 1e-4 3e-4 1e-3 0.10.8 0 1e-3 1e-1 Mean Pre-logits 1e-4 3e-4 1e-3 3e-3 1e-3 1e-2 1e-1 First Last Mean Pre-logits 128 256 0.1 0.8 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 Mean Pre-logits 128 256 1e-5 5e-5 1e-4 3e-4 1e-3 0 1e-3 1e-1 0.1 0.8 128 256 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 128 256 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8
Table 11. Hyperparameter search space for UniVLA + LIBERO benchmark.
Table 10. Hyperparameter search space for π0 + LIBERO benchmark. Method
HParams aggtoken Feat hidden dim lr λreg aggtoken Feat hidden dim lr lr γ λreg aggtoken Feat lr λreg aggtoken Feat hidden dim lr γ lr λreg aggtoken Feat hidden dim lr λreg lr γ hidden dim lr λreg lr γ hidden dim lr λreg lr γ
Method
Values First Last First&Last First Last First&Last 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 First Last First&Last First Last First&Last 1e-5 3e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 First Last First&Last First Last First&Last 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-2 1e-1 First Last First&Last First Last First&Last 1e-5 3e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1 First Last First&Last First Last First&Last 1e-5 3e-5 1e-4 3e-4 1e-3 0.1 0.8 0 1e-3 1e-1
SAFE-RNN
SAFE-RNN-TDQC
SAFE-MLP
SAFE-MLP BCE
SAFE-MLP-TDQC
RNN-BCE (Top 10) RNN-TDQC
21
HParams aggtoken lr λreg lr γ aggtoken lr λreg lr γ aggtoken lr λreg lr γ aggtoken lr λreg lr γ aggtoken lr λreg lr γ lr λreg lr γ lr λreg lr γ
Values First Last Mean 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 First Last Mean 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 First Last Mean 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 First Last Mean 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 First Last Mean 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8 1e-5 3e-5 1e-4 3e-4 1e-3 1e-3 1e-1 0 0.1 0.8
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
E. Additional Results E.1. Results variance In Table 12 and Table 13 we report the standard deviation for all results in Tables 1 and 2. Note that the reported values were averaged over 21 seeds which set the environment seed and determines the train-test split of tasks. Since different tasks have different difficulties it is reasonable to see large standard deviations in the tables. Tables 12 and 13 shows that across models and benchmarks, TDQC learning improves calibration relative to non-TDQC variants, with relatively lower standard deviations compared to the baselines, where SAFE-RNN-TDQC performs the best or on par for all π0 model variants. VLA Model Benchmark Eval Task Split Max prob. Avg prob. Running Avg prob. Avg entropy Running Avg entropy SAFE-RNN SAFE-RNN-TDQC (Ours) SAFE-MLP BCE SAFE-MLP-TDQC (Ours) RNN-BCE RNN-TDQC (Ours)
OpenVLA LIBERO Seen ↓ Unseen ↓ 0.395 ± 0.037 0.390 ± 0.028 0.348 ± 0.036 0.364 ± 0.061 0.338 ± 0.034 0.356 ± 0.059 0.306 ± 0.020 0.313 ± 0.030 0.265 ± 0.014 0.273 ± 0.026 0.204 ± 0.038 0.255 ± 0.059 0.197 ± 0.024 0.218 ± 0.020 0.192 ± 0.020 0.231 ± 0.020 0.195 ± 0.022 0.229 ± 0.022 0.199 ± 0.016 0.206 ± 0.022 0.191 ± 0.015 0.197 ± 0.021
OpenVLA WidowX Seen ↓ Unseen ↓ 0.572 ± 0.060 0.579 ± 0.012 0.275 ± 0.012 0.282 ± 0.019 0.255 ± 0.003 0.257 ± 0.002 0.414 ± 0.044 0.426 ± 0.095 0.435 ± 0.028 0.432 ± 0.054 0.169 ± 0.061 0.213 ± 0.075 0.096 ± 0.021 0.153 ± 0.056 0.127 ± 0.019 0.164 ± 0.034 0.130 ± 0.022 0.169 ± 0.038 0.301 ± 0.062 0.344 ± 0.048 0.156 ± 0.023 0.192 ± 0.039
UniVLA LIBERO Seen ↓ Unseen ↓ 0.909 ± 0.024 0.899 ± 0.048 0.529 ± 0.017 0.532 ± 0.035 0.543 ± 0.016 0.544 ± 0.031 0.406 ± 0.027 0.391 ± 0.063 0.343 ± 0.016 0.330 ± 0.039 0.124 ± 0.020 0.162 ± 0.014 0.064 ± 0.016 0.100 ± 0.026 0.091 ± 0.022 0.158 ± 0.036 0.066 ± 0.015 0.131 ± 0.028 0.138 ± 0.047 0.152 ± 0.065 0.100 ± 0.025 0.107 ± 0.056
π0 -FAST LIBERO Seen ↓ Unseen ↓ 0.320 ± 0.021 0.318 ± 0.022 0.212 ± 0.019 0.218 ± 0.024 0.244 ± 0.024 0.238 ± 0.045 0.209 ± 0.022 0.222 ± 0.033 0.279 ± 0.021 0.264 ± 0.046 0.106 ± 0.018 0.148 ± 0.042 0.103 ± 0.028 0.163 ± 0.060 0.103 ± 0.019 0.162 ± 0.053 0.109 ± 0.022 0.150 ± 0.035 0.122 ± 0.022 0.158 ± 0.053 0.105 ± 0.020 0.141 ± 0.045
π0 -FAST Franka Seen ↓ Unseen ↓ 0.331 ± 0.018 0.323 ± 0.017 0.290 ± 0.016 0.294 ± 0.024 0.359 ± 0.029 0.361 ± 0.026 0.281 ± 0.017 0.281 ± 0.021 0.341 ± 0.025 0.339 ± 0.027 0.220 ± 0.047 0.288 ± 0.055 0.150 ± 0.026 0.215 ± 0.038 0.206 ± 0.016 0.248 ± 0.023 0.210 ± 0.008 0.229 ± 0.013 0.238 ± 0.006 0.243 ± 0.005 0.205 ± 0.015 0.228 ± 0.022
π0 LIBERO Seen ↓ Unseen↓ − − − − − − − − − − 0.123 ± 0.043 0.172 ± 0.095 0.061 ± 0.019 0.097 ± 0.033 0.075 ± 0.018 0.137 ± 0.058 0.068 ± 0.021 0.128 ± 0.060 − − − −
Table 12. Brier Score results on simulation and real robot experiment (lower is better). Results are averaged over 21 seeds that determined different train-test split of tasks. ”−” indicates that the Brier score can’t be calculated on the method. The first and second best performing methods are highlighted in the table.
VLA Model Benchmark Eval Task Split Max prob. Avg prob. Running Avg prob. Avg entropy Running Avg entropy SAFE-RNN SAFE-RNN-TDQC (Ours) SAFE-MLP SAFE-MLP-BCE SAFE-MLP-TDQC (Ours) RNN-BCE RNN-TDQC (Ours)
OpenVLA LIBERO Seen ↑ Unseen ↑ 54.64 ± 5.36 55.78 ± 4.33 47.08 ± 4.07 48.09 ± 5.14 49.19 ± 4.16 47.72 ± 4.38 46.81 ± 4.22 46.75 ± 4.77 50.48 ± 4.34 48.09 ± 4.76 72.30 ± 4.73 69.04 ± 7.01 71.67 ± 4.00 65.12 ± 4.94 73.56 ± 3.85 69.53 ± 5.02 72.66 ± 3.39 64.99 ± 5.26 71.22 ± 3.23 60.09 ± 7.09 72.70 ± 4.26 72.28 ± 4.26 74.20 ± 3.70 72.72 ± 4.29
OpenVLA WidowX Seen ↑ Unseen ↑ 53.25 ± 3.36 53.20 ± 2.70 47.47 ± 3.99 48.30 ± 4.75 49.20 ± 4.51 46.37 ± 6.82 50.19 ± 4.64 49.36 ± 8.16 46.16 ± 5.38 43.99 ± 8.08 75.95 ± 7.29 70.00 ± 5.95 84.01 ± 3.89 70.76 ± 7.72 88.23 ± 4.37 83.18 ± 6.36 85.38 ± 3.49 71.43 ± 8.49 82.33 ± 3.71 70.64 ± 9.42 69.69 ± 6.29 67.09 ± 5.48 78.90 ± 7.46 72.97 ± 6.69
UniVLA LIBERO Seen ↑ Unseen ↑ 50.00 ± 0.00 50.00 ± 0.00 42.96 ± 8.89 40.25 ± 7.71 43.05 ± 6.14 40.29 ± 7.54 41.34 ± 9.42 47.36 ± 11.8 51.63 ± 8.50 56.71 ± 7.78 74.48 ± 9.49 68.13 ± 11.4 73.83 ± 8.52 64.25 ± 12.6 74.39 ± 7.73 63.80 ± 13.32 78.83 ± 5.86 69.74 ± 13.9 76.55 ± 5.68 66.71 ± 12.6 63.44 ± 7.92 54.37 ± 12.5 72.30 ± 7.52 70.17 ± 11.5
π0 -FAST LIBERO Seen ↑ Unseen ↑ 61.75 ± 8.85 63.49 ± 13.6 47.36 ± 7.64 48.09 ± 7.52 53.95 ± 4.42 55.68 ± 7.40 45.42 ± 7.44 46.30 ± 7.22 53.78 ± 5.27 55.44 ± 6.86 91.26 ± 3.18 85.88 ± 7.63 92.03 ± 3.37 85.49 ± 6.96 79.00 ± 12.5 68.36 ± 24.3 91.82 ± 3.23 85.50 ± 6.58 90.26 ± 3.22 84.40 ± 7.62 87.67 ± 4.11 82.53 ± 10.4 87.51 ± 3.65 83.39 ± 9.67
π0 -FAST Franka Seen ↑ Unseen ↑ 48.61 ± 4.58 46.64 ± 6.95 49.45 ± 5.41 48.03 ± 8.08 52.95 ± 3.92 49.98 ± 4.19 49.28 ± 4.34 49.07 ± 7.42 51.12 ± 4.53 48.78 ± 5.02 70.85 ± 5.92 56.69 ± 7.33 79.89 ± 6.80 68.43 ± 7.18 79.01 ± 4.17 63.69 ± 7.08 73.82 ± 3.61 58.83 ± 7.31 61.95 ± 6.47 51.22 ± 6.53 59.32 ± 4.55 53.26 ± 7.34 64.10 ± 4.11 57.37 ± 9.30
π0 LIBERO Seen ↑ Unseen↑ − − − − − − − − − − 79.95 ± 9.22 65.03 ± 20.5 88.66 ± 4.54 82.94 ± 10.7 86.33 ± 4.94 79.75 ± 13.0 89.71 ± 3.36 80.53 ± 13.8 86.57 ± 4.99 72.07 ± 18.4 − − − −
Table 13. ROC-AUC results on simulation and real robot experiment (higher is better). Results are averaged over 21 seeds that determined different train-test split of tasks. ”−” indicates that the ROC-AUC can’t be calculated on the method. The first and second best performing methods are highlighted in the table.
E.2. Generalization across benchmarks To assess the generalization capability of the success predictors, we trained an RNN-TDQC (top-10) predictor on OpenVLA trajectories from the LIBERO benchmark and evaluated it on OpenVLA trajectories from the WidowX benchmark. On the unseen WidowX tasks, the predictor achieved a ROC-AUC of 53.69 ± 5.07 and a sequential Brier score of 0.413 ± 0.03, compared to 72.97 ± 6.69 and 0.192 ± 0.039, respectively, on the original LIBERO tasks. These results suggest that the success predictor has limited generalization capabilities. E.3. Extended Evaluation for Application to Test-Time Action Search In order to extend out evaluations for test-time action search, we added to Figure 4ablation configurations: Probs Entropy is a heuristic approach that does not rely on a learned Q-value, serving as a natural, zero-shot baseline. The inclusion of this method is motivated by the need to determine whether a separately trained value network fθ is necessary; it tests the hypothesis that the base VLA model’s internal predictive certainty is a sufficient indicator of action quality. It is the ’Avg Entropy’ method from Tables 1 and 2, which achieved good sequential brier and ROC-AUC compared the the static baselines. It selects the action with lowest mean entropy across all D action dimensions: ( at = arg min at
D 1 X H PDF(d) D d=1
More details are in Apx. D.5. 22
)
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
(a) Success rate comparison across specific tasks, including ablation configurations.
(b) Average episode length (steps) per task, including ablation configurations.
(c) Averaged number of steps used action search per task, including ablation configurations.
Figure 5. Extended Analysis of Guided Action Search and TDQC Efficiency. The results demonstrate that RNN-TDQC provides the highest success rates, while the Threshold 0.35 variant offers a significant reduction in computational overhead by selectively triggering action search only when safety is at risk while maintaining high success rates.
While results in the paper summarized the average success rate across all tasks, we provide a more granular breakdown in Figure 5a-c to analyze per-task performance and computational trade offs. Figure 5a details the success rates for 3 unseen tasks: placing a bowl in a drawer (Task 3), moving two mugs to plates (Task 4), and placing a mug in a microwave (Task 9). We observe that TDQC-based methods consistently outperform the baseline across all unseen tasks, with RNN-TDQC achieving the highest success rate across all configurations. As shown in Figure 5c, while the TDQC-Threshold 0.35 variant achieves success rates that exceed the baseline, its primary advantage lies in computational efficiency. By comparing the average episode length (Figure 5b) with the actual number of steps triggered the action search (Figure 5c), it is evident that this strategy significantly reduces total compute requirements while maintaining competitive performance. E.4. Computational Cost For real robot deployment, test-time compute for TDQC is not more costly than baselines such as SAFE: both require only a forward pass of the MLP/LSTM at every time step, typically much lighter than the VLA forward pass. Furthermore, the network for TDQC with action probabilities as input is lightweight, and smaller than SAFE with features as input. RL theory suggests that TD-based methods require longer training than Monte-Carlo (MC) methods, thus, we evaluated all methods on the same GPU (NVIDIA RTX 5090) with 5 seeds on various computational metrics. Results are reported in Tables 14 and 15. From our analysis, the overhead from the TD loss is small, 25% on OpenVLA and 15% on π0 -FAST. E.5. quantitative results of RNN-TDQC with top 10 probabilities We visualize rollout trajectories together with the failure scores predicted by TDQC on OpenVLA LIBERO-10 benchmark. Figure 6 illustrates both failure and success cases. The green-shaded areas show the functional CP band. Once failure scores exceed the band, a failure flag is raised. Figure 6a shows an example of the flag not being raised during a successful rollout, whereas Figure 6b shows that the failure flag is triggered once the policy becomes stuck. Our quantitative analysis further indicates All methods reliably detect failure modes in which the policy gets stuck or oscillates with slow back-and-forth motions. For RNN methods, once the robot successfully grasps an object, the failure score drops, suggesting that the model recognizes this as a ’good’ action. Figure 7 shows a successful rollout with an informative failure score. The failure score rises when the policy becomes 23
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models Table 14. Computational cost of all failure prediction methods trained on OpenVLA LIBERO-10 benchmark. We report reserved and peak GPU VRAM (MB), number of trainable parameters (M), total training wall-clock time (s), and per-epoch time (s), averaged over five seeds. TD-0 loss adds ∼25% overhead relative to BCE for a given architecture. run name
vram reserved mb
num params M
train wall clock sec
peak vram mb
epoch time sec
MLP (features, BCE Loss) MLP (features, TD-0 Loss) RNN (features, BCE Loss) RNN (features, TD-0 Loss) RNN (top-10, BCE Loss) RNN (top-10, TD-0 Loss)
12104.8 ± 1818.37 12074.4 ± 1819.26 14883.6 ± 1516.95 14900.4 ± 1517.40 13308.38 ± 887.27 13506.19 ± 887.69
1.05 1.05 4.46 4.46 0.48 0.48
30.88 ± 0.95 38.31 ± 0.65 108.11 ± 0.91 137.83 ± 0.37 72.67 ± 0.53 97.12 ± 0.52
9881.54 ± 784.52 9885.00 ± 784.35 12381.49 ± 783.54 12381.46 ± 783.42 10296.98 ± 382.87 10496.06 ± 382.87
0.02 0.02 0.09 0.12 0.06 0.08
Table 15. Computational cost of all failure prediction methods trained on π0 -FAST LIBERO-10 benchmark. Columns report reserved and peak GPU VRAM (MB), number of trainable parameters (M), total training wall-clock time (s), and per-epoch time (s), averaged over five seeds. Compared to OpenVLA (Table 14), all methods are substantially more efficient: peak VRAM stays below 3.2 GB and training completes in under 40 s. LSTM (TD-0 Loss) – same params as BCE uses the same architecture and hyperparameters as LSTM (BCE Loss), providing a controlled comparison of loss functions. TD-0 adds only ∼13% to training time relative to BCE. run name
vram reserved mb
num params M
train wall clock sec
peak vram mb
epoch time sec
MLP (BCE Loss) MLP (TD-0 Loss) LSTM (BCE Loss) LSTM (TD-0 Loss) – same params as BCE LSTM (TD-0 Loss) TDQC (top-10 BCE Loss) TDQC (top-10 TD-0 Loss)
2320.00 ± 411.44 2366.00 ± 411.44 4101.60 ± 533.08 4123.60 ± 533.08 2619.20 ± 305.89 2450.00 ± 411.44 2448.00 ± 411.44
0.52 0.52 2.36 2.36 2.36 1.25 1.25
17.99 ± 0.56 20.67 ± 0.64 22.31 ± 0.29 25.34 ± 0.48 39.79 ± 0.82 32.09 ± 0.41 37.90 ± 0.33
1834.79 ± 174.62 1837.03 ± 174.63 3106.82 ± 165.88 3128.65 ± 158.33 2095.10 ± 174.91 1868.36 ± 174.61 1906.37 ± 174.44
0.01 0.01 0.01 0.01 0.03 0.02 0.03
temporarily stuck while trying to drop the alphabet soup into the basket around step 140. Then decreases after recovery around step 275, and increases again when the robot attempts to pick up the tomato soup at step 336, since grasping can fail. Overall, these scores are intuitive. Figure 8 compared BCE and TDQC-based methods. TDQC methods are more sensitive to changes in policy behavior and produce sharper, more localized responses over time. In contrast, BCE-based methods tend to vary more smoothly and exhibit less sensitivity to short-term policy changes. For MLP models, however, TD loss does not appear to improve prediction quality.
(a) Example of successful episode detected in RNN-TDQC (top-10 probabilities)
(b) Example of a failed episode detected by RNN-TDQC (top-10 probabilities) in the middle of the rollout
Figure 6. Failures and successes detected by RNN-TDQC (top-10 probabilities) align with the actual robot failures, as shown in the observations from OpenVLA + LIBERO-10 simulation. The green-shaded areas show the functional CP band. Once failure scores exceed the band, a failure flag is raised.
E.6. WidowX analysis The high ROC-AUC results in Table 2 on WidowX are obtained with SAFE-MLP, a heuristic that uses a cumulative loss, see Sec. 6.3, which appears to work well in terms of ROC-AUC in this specific benchmark. However, its connection with calibration is unclear. Note that SAFE-MLP is not reported in Table 1 since it cannot be used as a calibration metric. Figure 1 24
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 7. Successful rollout with informative failure scores of TDQC top 10 probabilities on OpenVLA LIBERO-10 benckmark. task: “put both the alphabet soup and the tomato sauce in the basket”. The failure score rises when the policy becomes temporarily stuck while trying to drop the alphabet soup into the basket around step 140. Then, failure score decreases after recovery around step 275, and increases again when the robot is positioned himself and attempts to pick up the tomato soup at step 336.
shows that w.r.t. a standard MLP/LSTM, the action-based predictor is competitive on WidowX. Additional video examples are provided on our project webpage. E.7. Ablation Studies We ablate several ways of incorporating temporal-difference learning when training a success predictor on OpenVLA in the LIBERO-10 benchmark. Specifically, we compare: • TD-0: one-step bootstrapping with a standard MSE objective on the predicted success probability. • TD-λ: multi-step bootstrapping using eligibility traces, which interpolates between TD-0 and Monte-Carlo targets via the trace parameter λ. • Categorical TD-0: a distributional variant where the scalar success target is represented as a categorical vector and trained via Binary cross-entropy loss, following the setup of (Farebrother et al., 2024). For TD-λ, we evaluate two trace values, λ ∈ {0.5, 0.8}, to study the effect of shorter versus longer credit assignment on sequential calibration.
(a) Brier scores evaluated at the minimal task time for all ablation methods. We see here that TD-0 with top 10 probabilities achieves the lowest Brier score
(b) ROC-AUC scores evaluated at the minimal task time for all ablation methods. We see here that TD-0 with top 10 probabilities achieves the highest ROC-AUC with the lowest variance
Figure 9. Ablation results for TD methods Overall, we see that TD-0 with the top 10 probabilities achieve best performance
25
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
(a) Example of failure score of a successful episode detected in SAFE-RNN-BCE (features). This is the same rollout as in Figure 7.
(b) Example of failure score of a successful episode detected in SAFE-RNN-TDQC (features). This is the same rollout as in Figure 7.
(c) Top-10 probabilities over time for RNN-BCE on the same suc- (d) Top-10 probabilities over time for RNN-TDQC on the same successful rollout. cessful rollout. Figure 8. Comparison between BCE and TD-based methods for the same rollout as in Figure 7. The top row shows the failure scores on features, while the bottom row shows the failure score on top-10 probabilities over time. Table 16. Performance degregation as a function of failed trajectories, evaluated on π0 -FAST LIBERO-10 (unseen tasks). We vary the proportion of failed trajectories retained during training from 100% down to 10%. Performance degrades gracefully until 30% but drops more sharply at 10%, suggesting the method is robust to moderate class imbalance.
Failed traj. %
Brier score ↓
ROC-AUC ↑
100% 60% 30% 10%
0.097 ± 0.033 0.101 ± 0.040 0.100 ± 0.045 0.135 ± 0.065
82.94 ± 10.7 79.52 ± 12.8 76.99 ± 16.6 59.49 ± 16.7
E.8. Sensitivity of the Early Stopping Figure 10 reports an extended analysis of TPR and FPR across significance levels, averaged over 21 seeds. A steep ascent in the TPR curve indicates that even tight thresholds successfully catch most failures, reflecting confidently high failure scores. TD-based methods consistently outperform BCE-trained predictors in this regard. Since CP bands are calibrated on successful rollouts (negative examples only), the FPR is lower-bounded by the i.i.d. assumption, represented by Y = X line in the plot. Deviations from this line reflect distribution shift between calibration and test tasks: a small shift produces results close to Y = X, while a larger shift, as seen in LIBERO, moves the curve further away. Notably, even under larger distributional shift, probability-based methods generalize better than feature-based methods. E.9. Degradation with fewer failed trajectories We evaluated LSTM-TD0 on π0 -FAST LIBERO-10 with 100%, 60%, 30%, and 10% of failed trajectories retained during training. Results on unseen tasks are shown in Table 3 in the attached link. The analysis shows that performance degrades gracefully until 30% and more sharply at 10%, suggesting robustness to moderate class imbalance. E.10. ROC-AUC vs Brier Score extended evaluation We add experimental results across six benchmarks of the relationship between uncertainty calibration and ROC-AUC in several VLA models. Figures 11a through 11f illustrate a critical relationship in the evaluation of VLA models: the link between a model’s uncertainty calibration and its predictive performance. All six scatter plots demonstrate a consistent 26
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 10. Additional failure detection analysis using thresholds obtained by functional CP. These plots show TPR (True positive rate, left column), and FPR (False positive rate, right column), w.r.t. the significance level α, for each evaluation benchmark. These plots are averaged across 21 seeds.
27
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
and strong negative correlation, with Spearman’s ρ ranging from −0.648 to a peak of −0.881 for OpenVLA on WidowX. This indicates that as the Brier Score at Stop Time (T̂ ) increases (representing higher calibration error), the ROC-AUC significantly decreases. The Brier Score measures the accuracy of probabilistic predictions. These results confirm that well-calibrated models (lower Brier scores) are substantially more capable of distinguishing between successful and failing states, resulting in a higher ROC-AUC. The lower correlation in UniVLA suggests that its internal uncertainty estimates are less reliable predictors of task success compared to the π0 family and OpenVLA. In Figure 11c, UniVLA data points are more dispersed around the fit line, indicating that a low Brier score does not guarantee high ROC-AUC as consistently as it does for other models.
(a) OpenVLA on LIBERO (ρ = −0.746)
(b) OpenVLA on WidowX (ρ = −0.881)
(c) UniVLA on LIBERO (ρ = −0.648)
(d) π0 -FAST on Droid (ρ = −0.783)
(e) π0 -FAST on LIBERO (ρ = −0.816)
(f) π0 on LIBERO (ρ = −0.857)
Figure 11. Analysis of VLA Calibration and Success Rates. (a-f) Scatter plots showing the strong negative correlation between Brier Score at Stop Time (T̂ ) and ROC-AUC across different model-benchmark pairs.
E.11. Relation between sequential Brier and ECE Let us be reminded that the Brier score relates to the calibration and accuracy of the classifier by its two-component decomposition. Let F = f (X) denote the random event that the model predicts a particular value, and let η(F ) = P(Y = 1 | F ), the success probability conditioned on the prediction. The Brier score can be decomposed as BS(f ) = E (F − Y )2 = E (F − η(F ))2 + E η(F )(1 − η(F )) . The first term relates to calibration: the discrepancy between the predicted success probabilities and the true conditional event frequencies. The second term in the Brier score decomposition relates to accuracy, and measures how informative the score is about the label. In order to prove this decomposition, we compared between the sequential Brier scores and ECE scores of all models and benchmarks, across 5 time quantiles (0.0, 0.2, 0.4, 0.6, 0.8) in Figure 16. Figures 12 to 15 shows the Sequential Brier score and ECE for seen and unseen sets respectively for different quantile times. In Figure 16 we report Spearman ρ which measures monotonic relationship between variables, uses rank ordering rather than actual values and ranges between 28
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 12. Brier scores val-seen sequential Brier score (lower is better) on the seen validation set averaged over 21 random seeds and all environments. We report Brier score in different time quantiles, where each subplot corresponds to a (model-benchmark) pair. For π0 , action probabilities are not directly interpretable, hence probability-based TDQC variants are not reported. Across all settings, our TD-based methods consistently outperform conventional predictors trained with binary cross entropy (BCE). For π0 action probabilities are not directly interpretable, hence probability-based TDQC variants are not reported. The dotted horizontal line represents the Brier score of a constant predictor that consistently outputs the empirical mean success rate computed over the seen tasks.
ρ ∈ [−1, 1]. In OpenVLA and π0 we see a strong monotonic relation between ECE and Brier scores where the red dashed line is the linear fitting on the points. We also note that π0 -FAST model does not show a strong relationship between the two metrics, which indicates that this model is less interpretable, and that the sequential Brier score affected more on the ROC-AUC component rather then the calibration component.
29
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 13. Brier scores val-unseen sequential Brier score (lower is better) on the unseen validation set averaged over 21 random seeds. We report Brier score in different time quantiles, where each subplot corresponds to a (model-benchmark) pair. For π0 , action probabilities are not directly interpretable, hence probability-based TDQC variants are not reported. Across all settings, our TD-based methods consistently outperform conventional predictors trained with binary cross entropy (BCE). For π0 action probabilities are not directly interpretable, hence probability-based TDQC variants are not reported. The dotted horizontal line represents the Brier score of a constant predictor that consistently outputs the empirical mean success rate computed over the seen tasks.
Figure 14. ECE scores val-seen ECE scores (lower is better) on the seen validation set averaged over 21 random seeds and all environments. We report ECE scores in different time quantiles, where each subplot corresponds to a (model-benchmark) pair. We see the correlation between lower Brier score and lower ECE scores in almost all settings. This highlights the Brier score decomposition shown in Section 2.1
Figure 15. ECE scores val-unseen ECE scores (lower is better) on the unseen validation set averaged over 21 random seeds and all environments. We report ECE scores in different time quantiles, where each subplot corresponds to a (model-benchmark) pair. We see the correlation between lower Brier score and lower ECE scores in almost all settings. This highlights the Brier score decomposition shown in Section 2.1
30
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Figure 16. ECE vs Brier scores We compared ECE scores to sequential Brier scores in all models and benchmarks.
31