Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
I. Introduction Consider a robotic manipulator fitted with a multi-fingered hand and an externally fixed camera tasked with grasping an object at an unknown pose. The robot faces an epistemic problem: contact parameters that govern grasp stability, namely friction coefficients, surface compliance, and contact modes, are inexact and can only be inferred indirectly from noisy tactile and proprioceptive feedback. Additionally, the accuracy of the computed object pose may be corrupted by camera calibration drift, lighting, or partial camera observations. These sources of uncertainty do not merely constitute a nuisance that may simply be averaged out, since a grasp whose expected quality metric is high under a certain object pose and friction prior may still fail catastrophically if the true pose and friction lie in the tails of their respective probability distributions [1], [2]. Robust grasping therefore demands planning algorithms that explicitly reason about the 1 C. Enwerem, J. S. Baras, and C. Belta are with the Department of Electrical & Computer Engineering and Institute for Systems Research, University of Maryland, College Park, MD, USA. Emails: {enwerem, baras, calin}@umd.edu. 2 S. Kalyanaraman is with Maryland Applied Graduate Engineering, A. James Clark School of Engineering, University of Maryland, College Park, MD, USA. Email: [email protected].
Stable
1
VNB (Ours)
Abstract— Contact variability, sensing uncertainty, and external disturbances make grasp execution stochastic. Expectedquality objectives ignore tail outcomes and often select grasps that fail under adverse contact realizations. Risk-sensitive POMDPs address this failure mode, but many use particlefilter beliefs that scale poorly, obstruct gradient-based optimization, and estimate Conditional Value-at-Risk (CVaR) with highvariance approximations. We instead formulate grasp acquisition as variational inference over latent contact parameters and object pose, representing the belief with a differentiable Gaussian mixture. We use Gumbel-Softmax component selection and location-scale reparameterization to express samples as smooth functions of the belief parameters, enabling pathwise gradients through a differentiable CVaR surrogate for direct optimization of tail robustness. In simulation, our variational neural belief improves robust grasp success under contact-parameter uncertainty and exogenous force perturbations while reducing planning time by roughly an order of magnitude relative to particle-filter model-predictive control. On a serial-chain robot arm with a multifingered hand, we validate grasp-and-lift success under object-pose uncertainty against a Gaussian baseline. Both methods succeed on the tested perturbations, but our controller terminates in fewer steps and less wall-clock time while achieving a higher tactile grasp-quality proxy. Our learned belief also calibrates risk more accurately, keeping mean absolute calibration error below 0.14 across tested simulation regimes, compared with 0.58 for a Cross-Entropy Method planner.
0
-1
T = 0.352
-1
0
1
Apply 12 N shear
Object Slips
1
Baseline
arXiv:2604.25897v1 [cs.RO] 28 Apr 2026
Clinton Enwerem1 , Shreya Kalyanaraman2 , John S. Baras1 , and Calin Belta1
0
-1
T = 0.062
-1
0
1
Before perturbation
After perturbation
Fig. 1. This paper develops a variational neural belief (VNB) parameterization for robust grasping, casting grasping as a sequential risk-sensitive decision process over an online contact-geometry belief. The grasp wrench space (GWS), W, (leftmost column; axes omitted) is the convex hull of resistible hand wrenches. The dotted circle marks the terminal inscribed-ball radius, εT , reported in the bottom-right corner of each GWS figure. Larger hulls containing the origin indicate greater disturbance rejection. VNB (top row) iteratively refines contact, producing a force-closed GWS that persists after a 12 N lateral shear force is applied at the object’s center of mass. A baseline planner (bottom row) commits to a single offline force-closed grasp before perturbation, but its GWS collapses under the same force, causing object slip. See Section VI for quantitative results.
full posterior distribution over contact and pose uncertainty, and not only its mean. Mainstream analytical grasp synthesis pipelines largely ignore uncertainty. Optimization-based planners [3], [4], [5] compute penetration-free, force-closure grasps under assumed contact properties, yet the resulting grasps are brittle to friction mismatch and pose error [1]. Data-driven methods [6], [7], [8] learn grasp proposals from large datasets but optimize expected performance and provide no mechanism to quantify or hedge against contact or pose risk. Methods that explicitly model uncertainty, including Deep Variational Bayes Filters [9] and probabilistic grasp learning frameworks such as Dex-Net 2.0 [10], maintain a belief over uncertain parameters. A dominant nonparametric belief representation is the weighted empirical measure defined by a particle set, which enables approximate belief-space optimization [11], [12]. However, particle-based beliefs inherit several limitations from particle filtering that complicate tail risk estimation. First, standard risk measures such as the Conditional Value-at-Risk (CVaR) require accurate tail estimation, but finite particle sets introduce variance unless particle counts are increased at proportional computational cost [13]. Second, resampling introduces discrete stochasticity that prevents pathwise gradient propagation. Third, par-
RealSense D435i
Robot Arm (6 DoF)
Robot Hand (11 DoF) Piezoresistive Tactile Sensor
Object Set
Orbbec Astra Pro Plus
Fig. 2. Hardware Platform: Our platform comprises a FAIR Innovation FR3 cobot (6 DoF), RealHand L6 robotic hand (11 DoF), two calibrated RGB-D cameras (RealSense D435i and Orbbec Astra Pro Plus), and representative primitives and YCB objects. We compare VNB-MPC with a Gaussian baseline under object-pose uncertainty and report results in Section VI, with hardware grasps shown in Fig. 7.
ticle allocation limits posterior estimation fidelity, especially in multimodal settings [14]. We address these limitations using a variational neural belief represented as a continuous distribution over contact parameters with learned transition and observation updates. By sampling Gaussian mixture beliefs with GumbelSoftmax component selection [15] and location-scale reparameterization [16], we obtain pathwise-differentiable samples with respect to the belief parameters. We then evaluate a smooth CVaR surrogate on these samples, propagating gradients through the belief while avoiding discontinuous resampling and the high-variance tail estimates induced by particle-based beliefs [17]. Our contributions are as follows: i. Differentiable Gaussian Mixture Belief : A Gaussian Mixture Model (GMM) belief representation whose reparameterized samples admit pathwise gradients with respect to all distribution parameters (Section III). ii. Risk-Sensitive Grasp Optimization: A smooth CVaRbased grasp quality objective with a pathwise gradient estimator through the reparameterized belief (Section IV). iii. Neural Belief Dynamics: Learned transition and observation models for differentiable belief updates (Section III-B). iv. Robust Grasping as Information-Guided Risk-Sensitive MPC: Grasping as a belief-space model-predictive control (MPC) problem that selects finger motion primitives to incrementally increase grasp quality while minimizing failure risk, aiming to achieve force closure under the current belief. Our risk-aware belief optimization framework reshapes the grasp wrench space during execution (see Figure 1), expanding the set of disturbances the grasp can resist (Section IV-E). We evaluate dexterous grasping under friction and stiffness uncertainty in MuJoCo across multiple regimes and perturbations, and validate hardware performance under pose uncertainty (Section V). A. Related Work 1) Belief-Space Planning for Dexterous Manipulation: Belief-space planning represents uncertainty over latent vari-
ables such as object pose and contact parameters using probability distributions [18], [19]. Particle filters are widely used due to their flexibility and multimodal capacity [12], [20], [21], but provide discrete approximations that suffer from sample degeneracy and do not support pathwise gradient propagation. Risk-aware methods incorporate coherent measures such as CVaR to account for tail uncertainty [22], yet particle-based beliefs require sampling-based gradient estimates. Our approach replaces particles with a continuous differentiable belief representation, enabling gradient-based optimization of risk-sensitive grasp objectives. 2) Neural Uncertainty Representations: Variational inference provides differentiable uncertainty representations optimized via gradient-based learning [23]. Gaussian mixtures support multimodal modeling with reparameterized sampling [16], and neural probabilistic models have been used to represent uncertainty in dynamics for planning [24], [25]. Distributional reinforcement learning models return uncertainty [26], [27], but do not capture physical contact uncertainty. In contrast, our method represents uncertainty directly over contact and pose parameters, enabling multimodal belief updates and pathwise risk optimization beyond meanperformance Gaussian MPC [28]. II. Problem Formulation We consider dexterous grasping under partial observability, where a robotic manipulator equipped with a multifingered hand interacts with a stationary object (see Figure 2 for our hardware setup) whose pose and contact properties are uncertain. We assume an obstacle-free tabletop environment and a 6-DoF arm positioned at a fixed pre-grasp configuration; only the fingers move during grasp execution, and time is indexed discretely from t = 0. At each discrete time, t = 0, 1, . . ., the system applies an action at ∈ A (a 6-dimensional vector of finger joint velocity commands), receives an observation ot (comprising an RGB-D-derived pose estimate with covariance, per-finger contact forces from piezoresistive tactile sensors, and joint encoder readings), and maintains a belief over latent (non-observable) physical parameters governing grasp stability. Notation: We use θ for latent physical parameters, ϕ for Gaussian-mixture belief parameters, and w ∈ R6 for objectframe contact wrenches. The set of feasible object wrenches is denoted by W. Definition 1 (Latent State). The latent state at time t is θ t = ξ o,t , ψ t ∈ Θ, (1) nc
t where ξo,t ∈ SE(3) is the object pose, ψ t = {ψti }i=1 , with ψti = (µit , κit , dit , sit ), comprises per-contact parameters at the nct active contacts: friction coefficient µit , stiffness κit , contact damping dit , and slip velocity magnitude sit , and Θ is the set of possible values of latent states. The contact count satisfies 0 ≤ nct ≤ ncmax , where ncmax ∈ N is the maximum possible number of contacts. In our experiments, ncmax = 5, matching the five piezoresistive tactile sensors on the robotic hand used in our hardware experiments (see Section V-A).
\β ∇ϕ CVaR
Observations
ot+1 , at Simulation (MuJoCo)
Belief Dynamics h t+1 ftrans [ht ; at ] fobs [hpred ; o]
Decoder h 7→ ϕ
ϕt+1
Belief p(θ | ϕ)
θi
GMM / SIREN
H[bt+H ] Risk-Aware MPC mina J(at:t+H )
Reparam. Gumbel-Softmax + loc.-scale
Cctc (θ i ) \β CVaR
a∗t:t+H
execute Hardware Evaluation
Fig. 3. Overview of Our Proposed Variational Neural Belief Grasping Framework. At each decision step, the neural belief dynamics update a latent embedding ht via a prediction network ftrans conditioned on actions and a correction network fobs conditioned on observations (joint angles, pose estimate with covariance, and contact geometry). We then evaluate grasp robustness under multimodal uncertainty along separate but connected simulation and hardware axes, with simulation providing controlled evaluation under known friction regimes, and hardware trials validating real-world transfer under object pose uncertainty (see Section V).
Since the latent state is not directly observable, the robot must instead reason over a probability distribution conditioned on its interaction history. Definition 2 (Belief State). Let p(·) denote the underlying probability density over latent parameters. The belief at time t is the posterior distribution bt (θ) = p(θ t | o1:t , a0:t−1 ),
(2)
where o1:t and a0:t−1 denote the observation and action histories. Observations comprise object pose estimates and contactrelated signals. Specifically, contact-related signals include per-finger normal forces from piezoresistive tactile sensors, binary contact indicators, and joint position/velocity readings. Let ot = (ξ̂ t , Σpose , yttact , qt , q̇t , ôt , ĉt ) denote the obt servation at time t, where ξ̂ t ∈ SE(3) is the estimated object pose, Σpose ∈ R6×6 is the pose estimation covariance det rived from the ICP registration residual (see Section V-C), yttact collects tactile measurements, qt , q̇t are joint positions and velocities, and ôt , ĉt ∈ [0, 1] are occlusion and segmentation confidence scores. For compactness, we factor the observation likelihood into a pose term, a tactile/contact term, and a visual-reliability term, as in p(ot | θ t ) ∝ N ξ̂ t | ξo,t , Σpose t (3) × p(yttact , qt , q̇t | ψ t ) × p(ôt , ĉt | θ t ), where the Gaussian term models pose estimation noise and the final factor captures observation reliability due to occlusion and segmentation uncertainty. The occlusion score ôt is computed as the fraction of object point-cloud points occluded by the hand mesh, and the segmentation score ĉt is the Intersection-over-Union between the predicted and groundtruth object masks (see Section V-C). Definition 3 (Grasp Wrench Space Under Uncertainty). Let θ = (ξ o , ψ) collect the latent contact geometry and friction parameters. For hand configuration qt and latent state θ t , the grasp wrench space is W(qt , θ t ) = G(qt , ξo,t ) f | f ∈ F(qt , ψ t ) , (4)
where G(q, ξ o ) is the grasp map [29], the linear mapping from contact forces to object wrenches, and F(q, ψ) is the admissible contact force set defined by Coulomb friction cones and unilateral contact constraints [30]. Since θ t is uncertain under the belief bt , the wrench space is a random set. Definition 4 (Risk-Sensitive Grasp Cost). Let C(qt , θ t ) denote the differentiable grasp cost functional (detailed in Section IV), which combines contact stability, friction margins, and slip penalties. The risk-sensitive grasp cost is defined as J(qt ; bt ) = ρβ C(qt , θ) , θ ∼ bt , (5) where ρβ is a coherent risk measure. We use CVaR (Conditional Value-at-Risk), which averages the worst (1−β) fraction of outcomes [31] (see Section IV for a formal definition). Definition 4 captures worst-case grasp cost over belief mass, rather than expected performance alone. Problem 1 (Risk-Sensitive Belief-Space Grasping). Determine an action sequence a0:T −1 that minimizes the cumulative risk-sensitive grasp cost subject to belief dynamics bt+1 = B(bt , at , ot+1 ), where B is the belief update operator realized by Algorithm 2 (Section III-B): PT −1 (6) mina0:T −1 t=0 ρβ C(θ t , at ) We assume that the initial belief b0 and the bounded problem horizon T ≤ Tmax ∈ N are known. Problem 1 defines a risk-sensitive belief-space optimal control problem, where optimal actions must hedge against latent contact uncertainty. Solving this problem requires a belief representation that supports efficient sampling and optimization of risk-sensitive objectives. We address this challenge by introducing a differentiable variational belief parameterization next. III. Approach: Variational Neural Belief Networks Our solution (Figure 3) represents the belief as a Gaussian mixture whose samples are differentiable functions of the distribution parameters via Gumbel-Softmax and location-scale reparameterization. This reparameterization closes the gradient gap left open by particle filters, connecting the CVaR
objective directly to the belief’s sufficient statistics. We learn belief dynamics from data with neural networks, and optimize actions over a receding horizon with a gradient-based MPC planner. A. Gaussian Mixture Beliefs We parameterize the belief as a mixture of K diagonalcovariance Gaussians PK (7) p(θ | ϕ) = k=1 πk N θ µk , diag(σ 2k ) , where ϕ = {ℓ, {µk , log σ k }K k=1 } collects the mixture logits ℓ = [ℓ1 , ℓ2 , . . . , ℓK ]⊤ ∈ RK and per-component means and log-standard-deviations. In typical mixture sampling, one first draws a component index, k ∼ Categorical(π), and then samples θ ∼ N (µk , diag(σ 2k )). However, since this discrete component selection is non-differentiable, we restore gradient flow through two mechanisms applied jointly, namely the Gumbel-Softmax relaxation [15] of the categorical draw and the standard location-scale reparameterization. The former produces relaxed assignment weights, ζ that are differen⊤ tiable with respect to the logits, with ζ = [ζ1 , . . . , ζK ] ∈ K−1 K−1 ∆ , where ∆ is the (K−1)-dimensional probability P simplex in RK , i.e., the set {ζ ∈ RK : ζ = 1}. The k ≥0 k latter proceeds by setting θ k = µk + σ k ⊙ ϵ, ϵ ∼ N (0, I), making each component sample differentiable with respect to µk and log σ k (⊙ denotes the element-wise Hadamard product). Applying the relaxation and location-scale reformulation steps, we can then write a single reparameterized GMM sample as PK (8) θ = k=1 ζk µk + σ k ⊙ ϵk , ϵk ∼ N (0, I), where the weights ζk are obtained via the Gumbel-Softmax relaxation exp((ℓk + gk )/τ ) , gk ∼ Gumbel(0, 1), (9) ζk = PK j=1 exp((ℓj + gj )/τ ) with temperature τ > 0 controlling exploration–exploitation: low τ yields near-deterministic component selection, while high τ blends hypotheses. Every operation in (8) is differentiable with respect to ℓ, µk , and log σ k 1 . B. Neural Belief Dynamics Instead of specifying a hand-crafted process and observation model, we learn differentiable belief dynamics, with the belief defined by a latent embedding ht ∈ Rdh that is propagated through a neural transition model, and subsequently corrected using observations hpred t+1 = ftrans [ht ; at ] , (10) ht+1 = fobs [hpred t+1 ; ot+1 ] where ftrans and fobs are multi-layer networks. A decoder (encapsulated by the DecodeBelief subroutine; see Table V for the architecture) maps ht+1 to belief parameters ϕt+1 , yielding the updated belief bt+1 . 1 For highly multimodal posteriors that resist mixture approximation, our framework also supports an implicit neural belief using sinusoidal representation networks (SIREN) [32] with Langevin sampling [33]; see Appendix B.
IV. Pathwise Risk Gradients via Action Optimization In this section, we derive the gradient of the CVaR objective (6) with respect to the planned action, enabling direct gradient-based grasp optimization. Appendix A gives the corresponding reparameterized-belief proof. A. Differentiable CVaR via the Dual Representation Let C : Rd → R be a differentiable cost depending on latent contact parameters θ and action a. The CVaR at confidence level β ∈ (0, 1) admits the dual form [31]: 1 E max C(θ, a) − η, 0 , CVaRβ [C] = min η + η∈R 1−β (11) where the expectation in (11) is taken over realizations of its argument corresponding to random values of θ drawn from the belief distribution, b. We set η to the empirical β-quantile and approximate the hinge with a softplus over N stochastic contact parameter realizations under the current belief, with η̂ capturing the empirical VaR threshold that defines the tail region: PN 1 \ β = η̂ + CVaR i=1 SoftPlus(C(θ i , a) − η̂; κρ ), (1−β)N (12) where θ i = g(ϵi , ϕ) are reparameterized samples. The κρ x ), is SoftPlus(x; κρ ) function, defined as κ−1 ρ log (1 + e a smooth approximation of the hinge (x)+ , where κρ con\β trols the sharpness of the approximation. This makes CVaR differentiable with respect to both a and ϕ. \ β Computation). For reparameterizRemark 1 (∇a CVaR able belief b(ϕ) and differentiable cost C, automatic differentiation yields: PN 1 \β = ∇a CVaR i=1 σκ (Ci − η̂)∇a C(θ i , a), (1−β)N (13) where Ci = C(θ i , a) and σκ is the sigmoid with sharpness κρ . The softplus bias κ−1 ρ log 2 is negligible for κρ = 5. Unlike particle filters, this formulation enables pathwise gradient-based action optimization without resampling. B. Differentiable Multimodal Cost Function We define a differentiable cost over uncertain contact realizations. For θ = (ξ o , ψ) and action a, let q(a) denote the hand configuration reached by executing a from the current state, and let F (q(a), ψ) and W(q(a), θ) be as in Definition 3. We evaluate grasp stability through the decomposition C(θ, a) = Cstr (a) + Cρ (a, ψ) + Cctc (ψ),
(14)
into closure strength (Cstr ), friction-limited robustness (Cρ ), and contact stability (Cctc ), respectively given by Cstr (a) = −αs ā − αg Gclose (a), Cρ (a, ψ) = −αr ReLU αn ā − min µ−1 , i i
(15) nct h i 1 X −µi −κi e +e + SoftPlus(si ) , Cctc (ψ) = c nt i=1 P where ā = n1j j aj is the mean joint closing rate with nj = Pnj 6 actuated finger joints, Gclose (a) = − i=1 pi (q(a)) −
cobj is a differentiable closure score computed as the negative sum of fingertip-to-object-center distances from forward kinematics, and (µi , κi , si ) denote friction, stiffness, and slip at contact i. The positive weights (αs , αg , αr , and αn ) scale the respective cost terms (see Table V). C. Visual Perception Cost We define a perception cost Cvis (ot ) that penalizes visuallyuncertain states ωpose tr(Σpose ) + ωocc SoftPlus(ôt ) + ωseg (1 − ĉt ), (16) t where Σpose , ôt , and ĉt are as in (3). We set λv = 0 in t simulation and evaluate Cvis only in hardware trials. D. Failure Probability We define the belief-space failure probability as the mass assigned to latent parameters under which the grasp cost exceeds a stability threshold τf : (17) P̂fail (bt ) = Prθ∼bt C(θ, at ) > τf . In practice, we estimate (17) via Monte Carlo using N samples PN (18) P̂fail = N1 i=1 1[C(θ i , at ) > τf ] , where θ i ∼ bt . For the variational belief, we replace the hard indicator with a sigmoid of sharpness κf to preserve differentiability, following standard smooth surrogate constructions used in variational inference and risk-sensitive optimization [16], [31]: PN soft (19) P̂fail = N1 i=1 σ κf (C(θ i , at ) − τf ) . In our framework, failure probability, CVaR, and belief entropy measure complementary aspects of robustness: P̂fail quantifies the mass of unstable configurations, CVaRβ their severity, and H[bt ] belief spread or entropy. E. MPC for Grasping with Variational Beliefs At the core of our method is a belief-space modelpredictive control (MPC) grasping formulation. At each step, the controller selects an action sequence that minimizes a risk-aware objective combining contact risk, perception cost, and belief entropy J(at:t+H ) = λc CVaRβ [Ct ] + (1 − λc ) Eθ∼bt [Ct ] + λv Cvis + γ H[bt+H ],
N
1 − λc X ∇at Ci N i=1 N X λc + σκ (Ci − η̂)∇at Ci , (1 − β)N i=1
J (k) ← \ β [C(θ (k) , a(k) CVaR t:t+H−1 )] + γ H[bt+H ] // (20) (k) P̂fail ← failure prob. via (19) (k)
k ⋆ ← arg mink J (k) s.t. P̂fail ≤ δ (k⋆ ) Execute at ← at , observe ot+1 ϕt+1 ← NeuralBeliefUpdate(ϕt , at , ot+1 ) if simulation then ε ← min max d⊤ w // see [34] ∥d∥=1 w∈W
else ε ← ε̂ := Q(ot+1 ) // hw. proxy (App. D) t←t+1 return grasp with quality ε Algorithm 2: NeuralBeliefUpdate Input: Latent state ht , action at , observation ot+1 , EMA rate αema Output: Updated beliefbt+1 parameterized by ϕt+1 pred ht+1 ← ftrans ht , at // Prediction (10) ht+1 ← fobs hpred // Correction (10) t+1 , ot+1 ϕt+1 ← (1 − αema ) ϕt + αema DecodeBelief(ht+1 ) return bt+1 = p(θ t+1 | ϕt+1 )
(20)
where Ct = C(θ, at ), λc ∈ [0, 1] blends expected cost (λc = 0) with full CVaR (λc = 1), H[bt+H ] denotes the terminal belief entropy, encouraging information-gathering actions, Cvis is the visual perception cost (16), and the weights λv ≥ 0 and γ ≥ 0 balance perception and exploration (we set λv = 0 in simulation experiments). To compute gradients of the contact-risk term with respect to the action, we differentiate through the sample mean and the soft CVaR approximation (12) of the contact cost C(θ, at ) ∇at Jcontact =
Algorithm 1: VNB–MPC for Grasping Input: Belief params. ϕ0 , CVaR level β, failure bound δ, target ε (εdes ), horizon H, grad. steps M , step size αu , samples N , max steps Tmax t ← 0, ε ← 0 while ε < εdes and t < Tmax do Per-component action optimization for k = 1, . . . , K do (k) // reparam. (8) {θ i }N i=1 ← g(ϵi , ϕk ) (k) at:t+H−1 ← ainit for j = 1, . . . , M do (k) (k) at:t+H−1 ← at:t+H−1 − \ β [C(θ (k) , a(k) αu ∇a CVaR t:t+H−1 )] // (13)
(21)
where Ci = C(θ i , at ), and σκ denotes a sigmoid with sharpness κ. The remaining differentiable terms in (20) add their standard gradients. This formulation enables direct gradientbased action optimization, providing finer control updates than sampling-based methods such as the Cross-Entropy Method (CEM) [35]. To mitigate local minima we use multistart optimization with three initial action magnitudes and select the action with the lowest risk-aware objective. Given the MPC objective in (20), Algorithm 1 presents our VNB–MPC solution routine that proceeds thus: the controller maintains a K-component Gaussian mixture belief over object pose and contact parameters. Because multimodal beliefs represent distinct contact hypotheses, we perform percomponent action optimization: the controller optimizes actions independently for each mixture component and selects
the one with the lowest risk-aware cost. This avoids optimizing against an averaged belief and lets the controller hedge against pose ambiguity, where a single finger motion can both disambiguate hypotheses and avoid catastrophic contact errors. We write ε̂ = Q(o) for the Ferrari–Canny grasp quality [34] estimated from tactile observations (see Appendix D). Since the focus of this work is on belief representations, we have omitted detailed derivations of the underlying motion controller and assume a belief-space controller that produces kinematically feasible, self-collision-free finger motions given sampled pose and contact realizations from the belief. V. Experiments A. Simulation Sandbox and Hardware Setup Simulation Environment: We evaluate Algorithm 1 on a dexterous grasping task using an 11-DoF hand attached to a 6-DoF arm in MuJoCo. We sample contact stiffness from κ ∼ N (1000, 2502 ) N/m, to yield stable simulation across all tested objects, and perturb object mass by ±20% of the nominal value. MuJoCo computes the effective contact friction as the geometric mean of the friction coefficients of the finger and object colliding surfaces: √ µeff = µf · µo , (22) where µf is the friction coefficient of the robotic hand’s finger material (assumed to be fixed), and µo is a per-object friction coefficient sampled per friction regime. Hardware Platform: Our hardware platform (Figure 2) consists of a 6-DoF FAIR Innovation FR3 robot arm equipped with a RealHand Inc. L6 multi-fingered hand with piezoresistive tactile sensing. The hand provides joint position control and tactile feedback for contact detection and post-grasp evaluation. Our perception stack utilizes two RGB-D cameras with complementary viewpoints: an Orbbec Astra Pro Plus mounted laterally about 46 cm from the workspace and a RealSense D435i mounted on the opposite side. The Astra provides the primary object pose estimate, while the D435i supplies a complementary estimate from a less occluded viewpoint before grasp execution. We place objects on a planar table within the arm’s workspace. B. Baselines We compare our approach against four baselines spanning risk-neutral and risk-aware belief-space MPC. Gaussian MPC (Gauss) uses a single Gaussian belief (K=1) with an expected-cost objective (λc = 0 in (20)). GaussianCVaR MPC (Gauss-Cvar) extends Gauss with a CVaR objective (12) (λc = 0.5) while retaining a single Gaussian belief and location-scale reparameterization for pathwise gradients. Particle Filter (Pf) MPC uses a particle-filter belief (N =100) with sampling-based optimization and CVaR estimated via particle sorting and truncation. CEM MPC (Cem) follows [35], using a Gaussian belief, population size 64, elite fraction 0.2, and 3 CEM iterations. Our method, Variational Neural Belief MPC (Vnb), uses a reparameterizable GMM belief (K=8) and the full risk-aware objective (20) with λc = 1.
TABLE I Dry-contact friction coefficients (µf = 1.0).
Object
µo Range
Nom. µo
µeff Range
PLA primitive LDPE bottle Alum./steel can Tennis ball (felt)
0.30–0.50 0.25–0.45 0.20–0.35 0.50–0.80
0.40 0.35 0.25 0.65
0.32–0.42 0.30–0.40 0.26–0.35 0.42–0.53
C. Multimodal Perception and Object Pose Estimation Using the sensors in the hardware platform described in Section V-A, we estimate the pose of the object to be grasped using ICP registration from the Orbbec Astra Pro Plus. The resulting ICP registration fitness and residual yield the pose covariance, Σpose , which enters the MPC objective (20). t When both cameras observe the object, the two ICP hypotheses are fused using covariance intersection, −1
(Σpose ) t
−1 = ωCI Σ−1 1 +(1−ωCI )Σ2 ,
ωCI ∈ [0, 1], (23)
with a fixed ωCI = 0.5 in our implementation. During grasp execution and for the lift-and-hold evaluation phase, tactile observations replace vision for contact detection and tactile grasp-quality proxy evaluation (see Sections D and V-D). D. Grasp Stress Tests and Evaluation Metrics Friction Regimes: To expose robustness differences between our approach and the baselines in Section V-B, we evaluate across three friction regimes of increasing difficulty. In the Nominal regime, we sample object friction as µo ∼ U[0.4, 1.0], yielding µeff ∈ [0.37, 0.59]. The Wide regime spans the full uncertainty range with µo ∼ U[0.15, 1.2], yielding µeff ∈ [0.23, 0.65], exposing planners to both lowand high-friction contacts. The Bimodal regime models multimodal contact conditions using a Gaussian mixture, µo ∼ 0.5 N (0.18, 0.032 )+0.5 N (1.0, 0.052 ), producing a bimodal distribution over effective friction. We select friction coefficients from Schneider & Company’s reference chart [36] under dry-contact assumptions. The fingertip collision geometries2 use a fixed µf = 1.0. By (22), the effective friction therefore depends on only the object’s surface. Table I summarizes the assumed object friction ranges and nominal values. We report results on seven objects comprising geometric primitives and select YCB objects [37] under four risk levels (β ∈ {0.5, 0.9, 0.95, 0.99}) and three randomization settings per condition. For hardware experiments, we evaluate object pose perturbations following the protocol in Section V-D. Figure 2 shows the real robotic manipulator and sensing setup. Simulation Stress Tests: In simulation, we subject each grasp to a two-phase stress test that evaluates lift stability and disturbance rejection. The robot first lifts the object vertically by 5 cm over 1 s to verify load-bearing capability. It then applies four lateral shear pulses of increasing magnitude 2 We initially set µ = 0.35 to match the PEEK fingertips of the real f hand. In MuJoCo this setting produced unstable contacts for several objects, so we used the simulator default µf = 1.0 for controlled comparisons and varied object friction instead.
Success Rate
VNB(Ours)
Robust Success
Nominal
Wide
TABLE II Grasp Stress test protocol and evaluation metrics
Pert. Survival
Bimodal
Fig. 4. Per-Regime Performance Comparison. Grouped bar chart comparing Cem and Vnb across friction regimes. Vnb matches or exceeds Cem in all three regimes, with the largest gains under nominal and bimodal friction.
(3.0, 5.0, 8.0, and 12.0 N) to test resistance to disturbanceinduced slip. We consider a grasp nominally successful if the object remains within 1 cm of the lifted pose and survives all shear pulses. We then evaluate grasp stability, robustness, belief convergence, and empirical failure rate: nsurv emp , (24) P̂fail =1− ntest where nsurv is the number of grasps surviving all perturbations out of ntest total trials. We compare this with the bel belief-predicted failure probability (18), P̂fail ≈ P (C > τf ), using a Gaussian CDF with τf = 5.8 (dimensionless). Hardware Experiments: We evaluate grasp stability under object pose uncertainty (ICP pose estimates) and structured pose perturbations. The protocol uses a 24 × 19 pegboard (1-inch / 25.4 mm pitch) with 10 mm dots rigidly fixed to the workspace, with the hand positioned near the board center so all perturbations lie within a reachable grid. For each object we define an M × M sub-grid centered on the nominal pose, where M = 2Noj + 1 (odd Noj ) or M = 2Noj (even Noj ), with Noj = ⌈s/25.4⌉ and s the support-side length in mm. We evaluate four offsets O = {(0, 0), (+1, 0), (0, +1), (+1, +1)} × 5 mm,
(25)
corresponding to the center, half-dot translations in +x and +y, and a 45◦ diagonal (≈ 7.07 mm). At each offset the ICP pipeline records the object pose X o , from which we o −1 o compute the error ∆X = (X̂ref ) X . The robot then lifts the object by 4 cm over 3 s and holds for 2 s while the Intel D435i monitors displacement. A trial succeeds if the object remains within 2 cm of the lifted reference during the hold phase. With three objects (two YCB and one primitive), four offsets, and two methods, Vnb and a Gaussian baseline (Gauss), our hardware evaluation comprises 24 trials total (12 per method). Appendix C provides the ICP scoring and pegboard-placement details. VI. Results & Discussion A. Simulation Results Table III compares all five methods across friction regimes, averaged over objects, risk levels, and seeds (time reports wall-clock seconds per episode). While nominal success is high for several methods, the perturbation protocol separates policies by whether nominal grasps remain stable under uncertain friction and external loading. We therefore emphasize success rate (SR), robust success, perturbation survival, and calibration error rather than raw worst-tail quality:
Metric Category
Definition
Nominal Stress Test (Sim.)
Vertical lift of 5 cm over 1 s, followed by lateral shear pulses of 3, 5, 8, and 12 N. Success requires displacement < 1 cm from the lifted pose.
Uncertain Friction Stress Test (Sim.)
28 perturbations: lateral impulses (3, 5, 8, 12 N; 4 directions; 0.15 s), torque impulses (0.3–1.0 Nm; 3 axes; 0.2 s), and friction drops µ ∈ {0.05, 0.10, 0.15}.
Per-Episode Stability
Lift height hlift , time-to-slip tslip , peak slip dmax slip , and failure mode (grasp, lift, slip, perturbation, none).
Robust Success Criterion
Nominal success and perturbation survival rate > 50%, where survival rate is the fraction of perturbations completed without exceeding displacement or rotation thresholds.
Hardware Evaluation
Approximate ε quality at grasp acquisition (ε̂), mean slip proxy (s̄). VNB(Our Method)
ε-metric
Percentage (%)
100 75 50 25 0
CEM
CEM
0.01 0.00 0
5
10
15
20
MPC Step, t Hand Approach
Grasp Consolidation
Grasp Maintenance
Fig. 5. Grasp Quality under Force Perturbations. Aggregate Ferrari– Canny ε over MPC steps for Vnb and Cem. Shaded regions mark approach, consolidation, and grasp maintenance. Vnb maintains higher ε and remains robust under lift perturbations (step 11), whereas expectation-driven Cem degrades during grasp maintenance.
once a grasp fails, terminal quality collapses to zero and no longer distinguishes whether a method failed often or rarely. Figure 4 and Figure 5 show per-regime stress-test performance and grasp-quality evolution against Cem. Gauss and Gauss-Cvar produce identical aggregate results across all regimes in our benchmark: the single-component Gaussian belief (K = 1) cannot represent distinct contact hypotheses, and the CVaR weighting did not change the selected actions after multi-start optimization. B. Hardware Results Table IV summarizes our hardware evaluation on three YCB objects (box, mustard bottle, soup can; see Figures 2 and 7) under pose perturbations across 4 offsets per object (n = 12 trials total). Both methods achieve 100% success on this small hardware set, so the hardware trials should be interpreted as an execution and convergence check rather than a statistically powered success-rate comparison. Within this setting, Vnb uses fewer steps and less runtime, terminating through force-threshold or tactile-quality stopping conditions across all trials. Gauss has a heavier upper tail in step count,
TABLE III Aggregate performance across friction regimes (mean over objects, β values, and seeds) in simulation. Robust = nominally successful & perturbation survival ≥ 50%. PertSurv = perturbation survival rate. ε = mean Ferrari–Canny quality metric [34] at episode termination. bel = belief-predicted failure probability. Quality = mean fraction of episodes achieving positive force closure (ε > 0) at termination. P̂fail emp bel − P̂ emp |. P̂fail = empirical failure probability (24). Best per regime in bold; |∆P̂ | = calibration error |P̂fail fail
ε↑
emp bel Quality ↑ P̂fail ↓ P̂fail ↓ |∆P̂ | Time (s)
Method
Regime SR (%) Robust % PertSurv %
Pf
nominal wide bimodal
83 44 56
67 39 28
57 31 29
0.0058 0.0039 0.0069
0.86 0.85 0.87
– – –
0.43 0.69 0.71
– – –
49.6 32.6 41.7
Gauss
nominal wide bimodal
100 71 54
75 63 42
69 56 36
0.0104 0.0100 0.0111
0.90 0.90 0.90
0.22 0.31 0.57
0.31 0.44 0.64
0.09 0.13 0.07
3.2 3.4 4.3
nominal Gauss-Cvar wide bimodal
100 71 54
75 63 42
69 56 36
0.0104 0.0100 0.0111
0.90 0.90 0.90
0.22 0.31 0.57
0.31 0.44 0.64
0.09 0.13 0.07
3.1 3.4 4.3
nominal wide bimodal nominal Ours (Vnb) wide bimodal
100 79 42 100 79 54
63 67 33 79 67 38
55 59 30 73 58 35
0.0101 0.0109 0.0119 0.0105 0.0106 0.0106
0.90 0.90 0.91 0.90 0.90 0.89
1.00 1.00 1.00 0.18 0.28 0.54
0.45 0.42 0.70 0.27 0.42 0.65
0.55 0.58 0.29 0.09 0.14 0.11
5.5 5.4 5.1 8.5 7.1 6.8
Lift (Box)
Mustard Bottle
Box
Soup Can
ε̂T = 1.539
ε̂T = 1.637
ε̂T = 3.066
Cem
Grasp
Lift (Mustard)
ε<0
ε=0.001
ε=0.001 (failure) ε=0.0022 (failure)
Cem
Pre-Grasp
Vnb
Fig. 7. Hardware Grasp Results under Vision Uncertainty. Post-lift frames of representative Vnb grasps executed on our hardware platform after 6D pose estimation (RealSense D435i). Here, we quantify terminal grasp quality using an ε-metric [34] inspired proxy ε̂T (×10−3 ) estimated from piezoresistive tactile observations (see Appendix D for details). ε<0
ε=0.03
ε=0.15 (success) ε=0.0021 (success)
Fig. 6. Grasp Comparison on Representative Objects. Top: Under friction uncertainty, Cem fails to achieve force closure (ε≈0, 0.0022) and drops the object during lift. Bottom: Our risk-aware Vnb achieves robust grasps (ε=0.15, 0.0021) and sustains a 50 mm lift.
with an interquartile range (IQR) extending to 11 steps, indicating less consistent convergence. The median step count is 6 for Vnb and 7 for Gauss; the median runtime is 11.5 s for Vnb and 14.2 s for Gauss. Vnb also has lower relative dispersion: CoV is 0.32 vs. 0.49 for step count and 0.41 vs. 0.78 for runtime. Terminal grasp quality ε̂T is higher for Vnb (median 1.6 × 10−3 vs. 0.9 × 10−3 ), with lower variance (CoV 0.48 vs. 1.21). The wider IQR of Gauss (0.1–2.2) reflects less consistent terminal tactile quality and suggests occasional lower-quality terminations. Both methods achieve a median of three final contacts. On the mustard bottle, peak and mean slip are comparable across methods (63.2 vs. 64.1 and 3.2 vs. 3.3, respectively), while Gauss has tighter slip dispersion on this object (CoV 0.13 vs. 0.42). The small sample size (n = 12 per method) limits statistical power; we report medians and IQRs rather than means and standard deviations.
C. Summary Together, our simulation and hardware results demonstrate that continuous, reparameterizable beliefs close the gradient gap between the planning objective and the belief state that particle filters leave open. The resulting pathwise gradients through the smooth CVaR surrogate improve robust success, perturbation survival, and calibration while retaining substantially lower wall-clock cost than particle-filter MPC. Our proposed composite objective (20) integrates contact risk, visual perception cost, and information-seeking entropy, enabling the grasp planner to hedge against multimodal uncertainty while avoiding hand configurations that occlude the object. Furthermore, our perturbation protocol across simulation and hardware experiments provides an additional evaluation mechanism beyond aggregate success rates: by comparing the belief-predicted failure probability (18) against the empirical perturbation failure rate (24), we can assess belief calibration (see the ∆P̂ column of Table III). A emp bel well-calibrated belief should satisfy P̂fail ≈ P̂fail , whereas overconfident beliefs underestimate failure mass and yield fragile grasps despite nominal success. Our results therefore indicate our method’s advantage in producing better-
TABLE IV Hardware evaluation over 12 trials per method and three objects. Both methods achieve 100% success on this small set; Vnb converges faster and with lower dispersion. Continuous metrics are median [IQR]; CoV is σ/µ. Vnb Metric
Med. [IQR]
Steps 6 [6–7] Time (s) 11.5 [7.5–14.5] −3 ε̂T (×10 ) 1.6 [1.5–2.1] Contacts 3 Max Slip3 63.2 [51.9–64.9] 3 Mean Slip 3.2 [2.6–3.3] 3
Gauss CoV ↓
Med. [IQR]
CoV ↓
0.32 0.41 0.48
7 [6–11] 14.2 [10.3–15.8] 0.9 [0.1–2.2] 3 64.1 [60.1–64.3] 3.3 [3.1–3.3]
0.49 0.78 1.21
0.42 0.41
0.13 0.13
Mustard bottle only. Slip values were comparable across other objects.
calibrated uncertainty estimates: across all regimes, Vnb consistently achieves small calibration errors (|∆P̂ | ≤ 0.14) while maintaining high success and robustness. In contrast, the sampling-based planner baseline (Cem) miscalibrates bel risk, predicting near-certain failure (P̂fail ≈ 1) even when empirical failure rates are moderate, indicating beliefs that fail to reflect actual grasp stability. VII. Conclusion We presented variational neural beliefs for risk-aware dexterous grasping. Our framework represents uncertainty in contact parameters and object pose with a differentiable belief, enabling pathwise gradients of a smooth CVaR surrogate and direct gradient-based optimization of grasp strategies. We combine contact risk, perception uncertainty, failure probability, and belief entropy in a composite objective that improves robustness while reducing uncertainty, and we use neural belief dynamics to construct a fully differentiable belief-space planner. Across three friction regimes and a 28test perturbation protocol, our method achieves higher robust success than sampling-based baselines such as CEM while planning 5–7× faster than particle-filter MPC in Table III. Future work will replace Langevin sampling with normalizing flows for faster reparameterized sampling, extend the risk objective to spectral and entropic coherent risk measures, meta-learn belief dynamics (Section III-B) across object classes to reduce calibration cost, and improve tactile contact detection for compliant, low-friction objects by adapting pressure thresholds to object compliance and surface properties. References [1] J. Weisz and P. K. Allen, “Pose Error Robust Grasping From Contact Wrench Space Metrics,” in 2012 IEEE International Conference On Robotics and Automation, pp. 557–562, 2012. ISSN: 1050-4729. [2] A. H. Li, P. Culbertson, and A. D. Ames, “Toward An Analytic Theory of Intrinsic Robustness for Dexterous Grasping,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2992–2999, 2024. ISSN: 2153-0866. [3] A. T. Miller and P. K. Allen, “GraspIt! A Versatile Simulator for Robotic Grasping,” IEEE Robotics & Automation Magazine, vol. 11, no. 4, pp. 110–122, 2004. [4] H. Li, Q. Ye, Y. Huo, Q. Liu, S. Jiang, T. Zhou, X. Li, Y. Zhou, and J. Chen, “TPGP: Temporal-Parametric Optimization with Deep Grasp Prior for Dexterous Motion Planning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 18106–18112, 2024.
[5] T. Liu, Z. Liu, Z. Jiao, Y. Zhu, and S.-C. Zhu, “Synthesizing Diverse and Physically Stable Grasps With Arbitrary Hand Structures Using Differentiable Force Closure Estimator,” IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 470–477, 2022. [6] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, C. Eppner, J. Leitner, J. Bohg, A. Morales, T. Asfour, D. Kragic, D. Fox, and A. Cosgun, “Deep Learning Approaches to Grasp Synthesis: A Review,” IEEE Transactions on Robotics, vol. 39, no. 5, pp. 3994–4015, 2023. [7] N. Khargonkar, N. Song, Z. Xu, B. Prabhakaran, and Y. Xiang, “NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Hands,” 2022. [8] R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang, “DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation,” 2023. [9] M. Karl, M. Soelch, J. Bayer, and P. van der Smagt, “Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data,” in International Conference on Learning Representations (ICLR), 2017. [10] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics,” Robotics: Science and Systems (RSS), 2017. [11] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and Acting in Partially Observable Stochastic Domains,” Artificial Intelligence, vol. 101, no. 1, pp. 99–134, 1998. [12] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics. MIT Press, 2005. [13] Y. Cheng, W. Ren, C. Xiu, and Y. Li, “Improved Particle Filter Algorithm for Multi-Target Detection and Tracking,” Sensors (Basel, Switzerland), vol. 24, no. 14, p. 4708, 2024. [14] J. Vermaak, A. Doucet, and P. Pérez, “Maintaining Multimodality through Mixture Tracking,” in Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1110–1116 vol.2, 2003. [15] E. Jang, S. Gu, and B. Poole, “Categorical Reparameterization with Gumbel-Softmax,” 2017. [16] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” arXiv preprint arXiv:1312.6114, 2013. [17] L. J. Hong, Z. Hu, and G. Liu, “Monte Carlo Methods for Value-atRisk and Conditional Value-at-Risk: A Review,” ACM Transactions on Modeling and Computer Simulation, vol. 24, no. 4, pp. 22:1–22:37, 2014. [18] R. Platt Jr, L. P. Kaelbling, T. Lozano-Perez, and R. Tedrake, “BeliefSpace Planning Assuming Maximum Likelihood Observations,” in Robotics: Science and Systems, vol. 6, pp. 37–44, 2010. [19] A. Petrovskaya and O. Khatib, “Global Localization of Objects via Touch,” IEEE Transactions on Robotics, vol. 27, no. 3, pp. 569–585, 2011. [20] R. Dyro, J. Harrison, A. Sharma, and M. Pavone, “Particle MPC for Uncertain and Learning-Based Control,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7127– 7134, 2021. ISSN: 2153-0866. [21] J. van den Berg, P. Abbeel, and K. Goldberg, “LQG-MP: Optimized Path Planning for Robots with Motion Uncertainty and Imperfect State Information,” The International Journal of Robotics Research, vol. 30, no. 7, pp. 895–913, 2011. [22] A. Hakobyan, G. C. Kim, and I. Yang, “Risk-Aware Motion Planning and Control Using CVaR-Constrained Optimization,” IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 4538–4545, 2019. [23] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational Inference: A Review for Statisticians,” Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017. [24] S. Depeweg, J. M. Hernández-Lobato, F. Doshi-Velez, and S. Udluft, “Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks,” in International Conference on Learning Representations (ICLR), 2017. [25] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models,” in Advances in Neural Information Processing Systems (NeurIPS), pp. 4754–4765, 2018. [26] C. Enwerem, A. G. Puranic, J. S. Baras, and C. Belta, “Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression,” in 2025 IEEE 64th Conference on Decision and Control (CDC), pp. 4890–4895, IEEE, 2025. [27] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos, “Distributional Reinforcement Learning with Quantile Regression,” in Pro-
ceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018. [28] Y. Pan and E. A. Theodorou, “Probabilistic Differential Dynamic Programming,” in Advances in Neural Information Processing Systems, vol. 27, Curran Associates, Inc., 2014. [29] R. M. Murray, Z. Li, and S. S. Sastry, A Mathematical Introduction to Robotic Manipulation. CRC Press, 1 ed., 2017. [30] D. Prattichizzo and J. C. Trinkle, “Grasping,” in Springer Handbook of Robotics (B. Siciliano and O. Khatib, eds.), pp. 671–700, Springer Berlin Heidelberg, 2008. [31] R. T. Rockafellar and S. Uryasev, “Optimization of Conditional Valueat-Risk,” Journal of Risk, vol. 2, no. 3, pp. 21–42, 2000. [32] V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit Neural Representations with Periodic Activation Functions,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 7462–7473, 2020. [33] P. Xu, J. Chen, D. Zou, and Q. Gu, “Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization,” in Advances in Neural Information Processing Systems, vol. 31, Curran Associates, Inc., 2018. [34] C. Ferrari and J. Canny, “Planning Optimal Grasps,” IEEE International Conference on Robotics and Automation (ICRA), vol. 3, pp. 2290–2295, 1992. [35] C. Pinneri, S. Sawant, S. Blaes, J. Achterhold, J. Stueckler, M. Rolinek, and G. Martius, “Sample-Efficient Cross-Entropy Method for Real-time Planning,” in Proceedings of the 2020 Conference on Robot Learning, pp. 1049–1065, PMLR, 2021. [36] Schneider & Company, “Coefficient of Friction Reference Chart,” 2026. Accessed: Feb. 25, 2026. [37] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The YCB Object and Model Set: Towards Common Benchmarks for Manipulation Research,” in 2015 International Conference on Advanced Robotics (ICAR), pp. 510–517, 2015. [38] S. Ding, Z. Su, G. Zhu, and L. Wang, “Envelope Quantile Regression,” Statistica Sinica, 2019. [39] J. T. LaFrance and L. D. Barney, “The Envelope Theorem in Dynamic Optimization,” Journal of Economic Dynamics and Control, vol. 15, no. 2, pp. 355–385, 1991.
Appendix A. Pathwise Risk Gradients via Action Optimization Theorem 1 (Reparameterized CVaR Gradient). Let b(ϕ) be a reparameterizable belief distribution with samples θ i = g(ϵi , ϕ), ϵi ∼ p(ϵ), and let C : Rd → R be continuously differentiable with E[|∇ϕ C(g(ϵ, ϕ))|] < ∞. Then the gradient of the soft CVaR estimator in (12) satisfies
Remark 2. The bias of the softplus approximation relative to the exact hinge is bounded by κ−1 ρ log 2, which vanishes as κρ → ∞. We treat η̂ as fixed with respect to ϕ in the stochastic estimator; alternatively, optimizing jointly over η recovers the envelope-theorem gradient [31]. B. Implicit Belief Variant and Hyperparameters When the posterior is highly multimodal or shaped by complex contact geometry, a Gaussian mixture may require an impractically large K. As an alternative highcapacity representation, we parameterize the unnormalized log-density directly with a sinusoidal representation network (SIREN) [32]: exp fω (θ) , (28) p(θ | ω) = Z(ω) where fω uses sinusoidal activations σ(x) = sin(ω0 x), and Z(ω) is the partition function. Sampling is performed via unadjusted Langevin dynamics with step size α: √ α θ j+1 = θ j + ∇θ log p(θ j | ω) + α ϵj , ϵj ∼ N (0, I). 2 (29) Because the score ∇θ log p is computed by automatic differentiation through fω , the resulting samples are approximately differentiable with respect to ω. Under standard smoothness assumptions on fω , with Lipschitz gradient constant L, the total-variation distance (TV(·, ⋆)) between the Langevin chain distribution pJ and the target distribution p⋆ satisfies [33] r dαL ⋆ TV(pJ , p ) ≤ + exp(−αλJ/2) TV(p0 , p⋆ ), (30) 4 where λ is the log-Sobolev constant of the target. In our implementation we use α = 10−3 , J = 50, and warm-start the chain from the previous belief’s samples. The approximate reparameterization guarantee follows from treating the Langevin chain as a differentiable function of ω, with bias controlled by the discretization error in (30).
N C. ICP Scoring and Pose Perturbation Details X 1 σκρ (C(θ i ) − η̂) ∇ϕ C(g(ϵi , ϕ)). For hardware pose estimation, we compute bounding-box (1 − β)N i=1 extents for the observed point-cloud cluster Pobs returned by (26) the Orbbec Astra Pro Plus and the object’s CAD model. We Proof. By the reparameterization trick, each sample θ i = generate 14 candidate rotations from cardinal-axis permutag(ϵi , ϕ) is a deterministic differentiable function of ϕ for tions and π-radian rotations, run ICP from each candidate, fixed ϵi . The softplus SoftPlus(x; κρ ) = κ−1 and score each registration using ρ log(1 + exp(κρ x)) is smooth with derivative σκρ (x) = (1 + s = sfit − λs ermse , (31) exp(−κρ x))−1 . Since C and g are differentiable by assumpwhere s is the ICP fitness score, e is the position-only fit rmse tion, applying the chain rule gives root-mean-square error in meters, and λs = 0.1. We select # " N X 1 the registration with the highest score and compute the fiSoftPlus(C(g(ϵi , ϕ)) − η̂; κρ ) ∇ϕ η̂ + nal object rotation as Ro = R⊤ (1 − β)N i=1 icp Rinit , where Rinit is the CAD model’s default rotation. The final pose estimate stores N X 1 the world-frame (F ) position and orientation Ro , and its R ≡ σκ (C(θ i ) − η̂) ∇θ C(θ i )∇ϕ g(ϵi , ϕ). (1 − β)N i=1 ρ covariance enters the MPC objective (20) through Cvis . (27) The pegboard is oriented with its longer edge, 24 dots, The gradient contribution through η̂ vanishes at the op- aligned with the yR axis and its shorter edge, 19 dots, aligned timal quantile by the envelope theorem [38], [39], which with the xR axis. The pegboard center pB is axis-aligned yields (26). with the robot base frame. We assign each object an initial
\β = ∇ϕ CVaR
24 in
19 in box {FR } yR xR
1 in pitch
128
yB
{FB } xB
Raw Pressure
255
0
(a) Schematic
(b) Hardware Setup
Fig. 8. Pose Perturbation Protocol. A rigid 24×19-inch pegboard defines a discrete planar coordinate frame {FB }, with origin pB axis-aligned to the robot base frame {FR }. We place objects at fixed dot offsets from pB , and apply controlled pose perturbations using the offset set in (25).
(a)
(b)
(c)
Fig. 9. Hardware Grasp and Tactile Readings. Sample hardware grasp (a) and corresponding 12 × 6 taxel arrays of analog pressure values in [0, 255] for the active thumb (b), index (c), and middle finger (d).
pk,t
TABLE V Architecture details and hyperparameters.
pj,t
n̂k,t
Group
Details
Belief nets
dh = 64, da = 6, do = 41, d = 26; transition MLP [dh + da , 128, 128, dh ]; observation MLP [dh + do , 128, 128, dh ]; decoder MLP [dh , 64, K(2d + 1)]; K = 8 for Vnb, K = 1 for Gauss/Cem; SIREN 3 × 128, ω0 = 30.
Observation layout
(41) pose 6, tr(Σpose ) 1, tactile forces 5, contacts 5, joint positions 11, joint velocities 11, visual scores 2.
Latent layout
(26) pose 6 and 5× contact parameters ×4. −4
Training
Adam, η = 3 × 10 , batch size 64; Gumbel-Softmax τ : 1.0 → 0.1 cosine over 500 epochs; κρ = 5.0, κf = 100.0; β ∈ {0.5, 0.9, 0.95, 0.99}; belief samples N = 256 online, 512 final evaluation.
MPC/planning
Tmax = 80, PF particles 100; CEM population/elite/iterations 64/0.2/3; 20 action candidates, close ∈ {0.05, 0.10, 0.15, 0.20}; λc = 1.0 Vnb, 0.5 Gauss-Cvar, 0 Gauss; λv = 0.3.
Cost weights
αs = 1.0, αg = 0.5, αr = 2.0, αn = 0.1.
Visual cost
ωpose = ωocc = ωseg = 1.0; σbase = 0.005 m; occlusion radius 0.05 m.
Perturbations
lateral forces {3, 5, 8, 12} N, 4 directions; torques {0.3, 0.6, 1.0} Nm, 3 axes; µ → {0.05, 0.10, 0.15}; drop/displacement thresholds 0.015/0.04 m; rotation threshold 0.3 rad.
Compute
NVIDIA RTX 4070, 12 GB; Vnb episode time ≈ 4–6 s; Pf episode time ≈ 65–72 s.
lattice location relative, in lattice counts, to the lattice at pB : box and soup can (−2, +1), and mustard bottle (−1, +1). The four evaluated offsets are the set O in (25). For each trial, we compute the realized SE(3) perturbation ∆X from the ICP estimate, then use the RealSense stream to monitor slip during the 4 cm lift and 2 s hold. A trial is successful if object displacement remains below 2 cm. D. Hardware Grasp Quality Proxy The robotic hand used in our hardware experiments provides a 6 × 12 pressure array of taxels, tactile pixels, at each fingertip; see Figure 9. Each taxel reports pressure in [0, 255], and we sum over the tactile-pad area to obtain a normal-force proxy 6 12 1 XX ˆ pi,u,v , (32) fi = ki u=1 v=1 where pi,u,v ∈ [0, 255] is taxel (u, v) at finger i, and ki is a finger-specific pressure-to-force calibration constant
(d)
(1)
C t̂i,t Fi,t ⊙ (2)
t̂i,t
object
n̂j,t
c
n̂i,t pi,t
C = {n̂ , t̂(1) , t̂(2) } Cbt = {i, j, k} | Fi,t i,t i,t i,t
Fig. 10. Contact-Frame Assignment. For each active finger i ∈ Cbt , forward kinematics gives the fingertip position pi,t . Dashed lines connect the inferred object center ct to active fingertip positions. Blue arrows denote (1) inward contact normals n̂i,t , the red arrow denotes t̂i,t , and ⊙ denotes (2) t̂i,t , pointing out of the page.
(kthumb = 25.5, kj = 204.0 for j ̸= thumb). A finger is in contact when fˆi > fmin = 0.05 N. However, because the tactile array lacks the full 6D contact wrench, our hardware pipeline cannot directly certify force closure. We therefore use a force-closure-inspired tactile grasp-quality proxy computed from active contact geometry. Let Cbt denote the hardware-estimated active contact set at time t, obtained from tactile activation. This is the hardware analogue of the contact set used to construct the grasp wrench space in the model-based planner. For each active finger i ∈ Cbt , we estimate the fingertip world position pi,t via forward kinematics and construct the unit contact normal ct − pi,t n̂i,t = , (33) ∥ct − pi,t ∥ where ct ∈ R3 is the FK-inferred object center. We then com(1) (2) C plete the right-handed contact frame Fi,t = {n̂i,t , t̂i,t , t̂i,t } (1) by choosing t̂i,t = n̂i,t × e, with e a non-parallel reference (2) (1) C direction, and t̂i,t = n̂i,t × t̂i,t . With Fi,t , we then compute a force-closure-inspired tactile proxy as ε̂t = Q (pi,t , n̂i,t , µ) i∈Cbt , (34)
where Q(·) is the Ferrari–Canny quality functional [34] evaluated on the approximate grasp wrench space induced by inferred contacts, linearized friction cones, fixed µ = 0.5, and unit normal forces. Since the contacts, normals, and friction are estimated, ε̂t is a force-closure-inspired tactile proxy rather than a direct certificate. We set ε̂t = 0 when |Cbt | < 2, since the approximate grasp wrench space is then not spanned.