AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation
arXiv:2604.11674v1 [cs.RO] 13 Apr 2026
Mingyang Li∗ , Haofan Xu∗ , Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui† , Zeyang Liu† , Xingyu Chen† , Xuguang Lan† School of Artificial Intelligence, Xi’an Jiaotong University
Abstract Simulation-based data generation has become a dominant paradigm for training robotic manipulation policies, yet existing platforms do not incorporate object affordance information into trajectory generation. As a result, tasks requiring precise interaction with specific functional regions—grasping a mug by its handle, pouring from a cup’s rim, or hanging a mug on a hook—cannot be automatically generated with semantically correct trajectories. We introduce AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into the manipulation data generation pipeline. AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector that enhances MLLM output tokens with multi-scale geometric features, to predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions. Built on NVIDIA Isaac Sim with cross-embodiment support (Franka FR3, Panda, UR5e, Kinova), VLM-powered task generation, and novel domain randomization using DA3-based 3D Gaussian reconstruction from real photographs, AffordSim enables automated, scalable generation of affordance-aware manipulation data. We establish a benchmark of 50 tasks across 7 categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) and evaluate 4 imitation learning baselines (BC, Diffusion Policy, ACT, Pi 0.5). Our results reveal that while grasping is largely solved (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging for current imitation learning methods, highlighting the need for affordance-aware data generation. Zero-shot sim-to-real experiments on a real Franka FR3 validate the transferability of the generated data.
1
Introduction
Simulation-based data generation has become a dominant paradigm for scaling robotic manipulation training data [Mu et al., 2024, Nasiriany et al., 2024, Gu et al., 2023, RoboVerse Team, 2025]. By collecting demonstrations in simulation, researchers can generate thousands of diverse trajectories at low cost, train visuomotor policies via imitation learning, and deploy them on real robots through sim-to-real transfer. However, a fundamental limitation persists across existing simulation platforms: none of them incorporates object affordance information into the trajectory generation pipeline. Object affordances—the functional regions through which objects are meant to be manipulated—are critical for tasks requiring precise, semantically correct interactions. Pouring from a cup requires * Equal contribution. † Corresponding author.
Preprint.
Figure 1: Overview of AffordSim. Natural language task descriptions are processed by a VLM to generate simulation scenes in Isaac Sim. VoxAfford predicts 3D affordance maps on object point clouds, guiding grasp pose estimation and motion planning to produce semantically correct manipulation trajectories. Comprehensive domain randomization, including DA3-reconstructed real backgrounds, enables zero-shot sim-to-real transfer.
grasping its handle and tilting from the rim; hanging a mug demands locating the hook-compatible opening. Current simulation platforms handle grasp pose estimation through one of two approaches, both of which fail for affordance-demanding tasks. Manual grasp design requires human engineers to hand-specify grasp poses for each object and task, which is labor-intensive and does not scale. Generic grasp estimation methods such as AnyGrasp [Fang et al., 2023] predict stable grasps without considering task semantics, often selecting poses that are physically stable but functionally incorrect (e.g., grasping a mug by its body when the task requires pouring). In our experiments, affordancedemanding tasks such as mug hanging achieve at most 10% policy success when trajectories are generated with generic grasp methods, compared to meaningful improvements with affordance guidance. We propose AffordSim, the first simulation framework that integrates affordance prediction into the robotic manipulation data generation pipeline. AffordSim uses our VoxAfford model [Yang et al., 2024b], an open-vocabulary 3D affordance detection method that enhances MLLM output tokens with multi-scale 3D geometric features, to predict affordance maps indicating where and how a robot should interact with objects. These affordance predictions guide grasp pose estimation and motion planning, producing trajectories that are both physically plausible and semantically aligned with task objectives. Built on NVIDIA Isaac Sim [NVIDIA, 2023], AffordSim features: • Affordance-aware trajectory generation. VoxAfford predicts 3D affordance heatmaps on object point clouds, guiding grasp pose selection toward task-relevant functional regions. • Automated task-scene generation. A VLM-powered pipeline converts natural language task descriptions into simulation scenes, supporting 4 robot embodiments (Franka FR3, Panda, UR5e, Kinova) without task-specific engineering. • Photorealistic domain randomization. Five randomization axes—including DA3-based [Yang et al., 2024a] 3D Gaussian reconstruction from real photographs—reduce the visual domain gap for sim-to-real transfer. 2
• A 50-task benchmark across 7 manipulation categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) with evaluations of 4 imitation learning baselines (BC, Diffusion Policy [Chi et al., 2023], ACT [Zhao et al., 2023], Pi 0.5 [Black et al., 2024]). Our benchmark reveals a key finding: while grasping is largely solved by current imitation learning methods (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging. This performance gap highlights the need for affordance-aware data generation to address these challenging manipulation tasks. Zero-shot sim-to-real experiments on a real Franka FR3 validate that policies trained exclusively on AffordSim data transfer to the real world, with success rates ranging from 60% on grasping to 10% on mug hanging, mirroring the affordance-complexity gradient observed in simulation.
2
Related Work
2.1
Simulation Platforms for Robotic Manipulation
A growing number of simulation platforms provide environments for generating robotic manipulation data. RoboTwin [Mu et al., 2024] combines digital twin environments with LLM-driven task generation and multi-embodiment support. RoboCasa [Nasiriany et al., 2024] focuses on household environments with large-scale scene generation. ManiSkill2 [Gu et al., 2023] provides GPU-parallelized environments for contact-rich manipulation. Meta-World [Yu et al., 2020] and CALVIN [Mees et al., 2022] offer standardized multi-task benchmarks for policy learning. LIBERO [Liu et al., 2024] introduces lifelong learning benchmarks with procedural task generation. RoboVerse [RoboVerse Team, 2025] unifies multiple simulators under a common interface. GenManip [Geng et al., 2025] uses LLM-driven simulation for generalizable instruction-following manipulation, focusing on language grounding rather than affordance-aware trajectory generation. However, none of these platforms integrates object affordance information into the trajectory generation process. Grasp poses are either manually designed or estimated by generic methods, limiting the range of tasks that can be automatically generated with semantically correct interactions. 2.2
3D Affordance Prediction
Affordance prediction has been studied extensively as a standalone perception problem. Where2Act [Mo et al., 2021] predicts per-point actionability for articulated objects. 3D AffordanceNet [Deng et al., 2021] learns affordance representations from 3D point clouds. GAPartNet [Geng et al., 2023] detects generalizable and actionable parts across object categories. Our concurrent work, VoxAfford [Yang et al., 2024b], advances this line by injecting multi-scale geometric features from a frozen 3D VQVAE encoder into MLLM output tokens via cross-attention, achieving state-of-the-art open-vocabulary 3D affordance detection (∼8% mIoU improvement) with zero-shot transfer to real robots. Despite progress in affordance prediction, these models have been studied in isolation from simulation data generation pipelines. AffordSim bridges this gap by integrating VoxAfford directly into the trajectory generation loop, enabling affordance-aware data collection at scale. 2.3
Automated Data Generation and Domain Randomization
LLM- and VLM-driven approaches have enabled automated task and scene generation. GenSim2 [Katara et al., 2024] uses LLMs to generate task code and reward functions. RoboGen [Wang et al., 2023] automates the generation of diverse manipulation tasks through LLM-based decomposition. These approaches focus on task diversity but remain agnostic to object affordances during trajectory collection. Domain randomization [Tobin et al., 2017, Peng et al., 2018] is a widely adopted strategy for closing the sim-to-real gap by diversifying visual appearance during training. We extend standard domain randomization with DA3-based [Yang et al., 2024a] 3D Gaussian reconstruction from real scene photographs, providing photorealistic background rendering from arbitrary viewpoints—a complement to texture-based randomization that better preserves real-world visual statistics. 3
3
AffordSim Framework
AffordSim is a simulation framework for generating affordance-aware robotic manipulation data. Given a natural language task description T (e.g., “pick up the mug by its handle and pour into the pan”), AffordSim automatically generates a simulation scene, predicts affordance maps on target objects, plans grasp poses guided by affordance predictions, executes motion plans to collect trajectories, and applies domain randomization for sim-to-real transfer. Figure 1 provides an overview of the full pipeline. 3.1
System Overview
The AffordSim pipeline consists of five stages: (1) A VLM converts T into a scene configuration S, specifying objects, their poses, the robot embodiment, and goal conditions. (2) For each target object, a point cloud P ∈ RN ×3 is captured from the simulation scene. (3) VoxAfford predicts an affordance map A : P → [0, 1], where ai = A(pi ) indicates the suitability of point pi for the specified interaction. (4) Affordance-guided grasp pose estimation selects a grasp g∗ ∈ SE(3) from the affordance-weighted candidate set. (5) A motion planner generates and executes a trajectory τ , and domain randomization is applied to the rendered observations. The system is built on NVIDIA Isaac Sim and supports four robot embodiments: Franka FR3, Franka Panda, UR5e, and Kinova. 3.2
Affordance-Aware Trajectory Generation
The core contribution of AffordSim is the integration of affordance prediction into grasp planning. Given the task description T and an object point cloud P, our VoxAfford model [Yang et al., 2024b] predicts a per-point affordance score ai ∈ [0, 1] indicating how suitable each surface point is for the specified manipulation. VoxAfford enhances MLLM output tokens with multi-scale geometric features from a frozen 3D VQVAE encoder via cross-attention, producing spatially-aware affordance masks. It operates in an open-vocabulary setting: it accepts arbitrary natural language affordance queries (e.g., “graspable handle,” “pourable rim”) and generalizes across object categories without task-specific fine-tuning. Affordance-guided grasp selection. Grasp pose estimation proceeds in two stages. First, the predicted affordance map A identifies the high-affordance region on the object surface. We sample multiple approach directions around this region and generate a set of candidate grasps G = {g1 , . . . , gK }, each oriented along a different approach vector toward the affordance peak. Second, each candidate gk is scored by jointly considering (i) the affordance values of the contacted surface and (ii) the kinematic feasibility with respect to the robot’s current configuration q: 1 X sk = ai · f (gk , q), (1) | {z } |Ck | pi ∈Ck kin sk | {z } saff k
where Ck ⊂ P denotes the set of points in contact with grasp gk , saff k measures the mean affordance value of the contact region, and skin = f (g , q) ∈ [0, 1] evaluates the reachability and collision-free k k feasibility of the grasp given the robot arm’s actual position. The selected grasp is g∗ = gargmaxk sk . This two-stage design ensures that the final grasp is both semantically correct (targeting functional regions such as a mug handle or cup rim) and physically executable by the specific robot embodiment. Motion planning. Given the selected grasp pose g∗ , we use cuRobo [Sundaralingam et al., 2023], a GPU-accelerated motion planning library, to generate a collision-free trajectory from the robot’s home configuration to the pre-grasp pose, execute the grasp, and plan the subsequent manipulation motion (e.g., lifting, tilting for pouring, placing). The full trajectory τ = {(ot , at )}Tt=1 consists of observation-action pairs, where ot includes RGB-D images and proprioceptive state, and at is the end-effector pose command. 3.3
VLM-Powered Task and Scene Generation
AffordSim automates the task-to-scene pipeline using a VLM. Given a natural language task description T , the VLM generates: (1) a list of required objects with their USD asset paths from the object 4
Figure 2: Domain randomization in AffordSim. Each column shows the cumulative effect of adding one randomization axis: object pose, lighting, background texture, and image noise. The rightmost column shows example background textures used for randomization. Three representative tasks are shown across rows.
library (500+ objects, 50+ real-scanned), (2) initial poses for each object on the workspace, (3) the target robot embodiment, and (4) task-specific goal conditions (e.g., “object A is inside container B”). This eliminates manual scene engineering and enables rapid scaling to new tasks. The system supports four robot embodiments—Franka FR3, Franka Panda, UR5e, and Kinova—with automatic kinematic adaptation (Figure 4). 3.4
Domain Randomization for Sim-to-Real Transfer
To close the visual gap between simulation and the real world, AffordSim applies domain randomization along five axes (Figure 2): (1) Background texture: random textures from a diverse texture library are applied to the workspace and surrounding surfaces. (2) Lighting: randomized number, position, intensity, and color of light sources. (3) Object texture: PBR material properties (albedo, roughness, metallicity) are perturbed. (4) Object pose: small random perturbations to initial object positions and orientations. (5) DA3 Gaussian background: we reconstruct the real deployment scene using DA3-based 3D Gaussian Splatting [Yang et al., 2024a] from 10–20 photographs. The reconstructed Gaussian field renders photorealistic background images from arbitrary viewpoints, which are composited behind the simulation scene. This approach preserves the visual statistics of the real environment (spatial layout, textures, lighting gradients) rather than sampling from a generic texture distribution, significantly reducing the domain gap. 3.5
Benchmark Design
We design a benchmark of 50 tasks organized into 7 manipulation categories of increasing affordance complexity (Figure 3): 1. Grasping (10 tasks): Pick up objects with varied geometries. Requires basic affordance (graspable surfaces). 2. Placing (10 tasks): Pick and place objects onto target surfaces or into containers. Requires release-point affordance. 3. Stacking (5 tasks): Stack objects in specified orders. Requires stable-surface affordance and sequencing. 4. Pushing/Pulling (6 tasks): Push objects to targets or pull open drawers/doors. Requires contactsurface affordance. 5. Pouring (8 tasks): Pour contents between containers of varying geometry. Requires rim affordance and tilt control. 6. Mug Hanging (3 tasks): Hang mugs on racks or hooks. Requires handle-opening affordance and precise alignment. 5
Figure 3: Task gallery. Representative tasks from the AffordSim benchmark across seven manipulation categories: grasping, placing, stacking, pushing/pulling, pouring, mug hanging, and long-horizon composite tasks. 7. Long-Horizon Composite (8 tasks): Multi-step tasks that chain primitives from different categories, e.g., pick-pour-place or open-and-place. Requires sequential affordance reasoning across multiple objects and interactions. Each task is evaluated with 30 simulation rollouts. The primary metric is task success rate (SR), defined as the fraction of rollouts that satisfy all goal conditions. For multi-step tasks, we additionally report sub-task success rates to localize failure modes (e.g., pick succeeds at 90% but pour fails at 3%).
4
Experiments
We evaluate AffordSim along four axes: (1) benchmark performance of imitation learning baselines, (2) cross-embodiment generalization, (3) the contribution of affordance integration, and (4) zero-shot sim-to-real transfer. Baselines. We evaluate four imitation learning methods: BC (behavioral cloning with an MLP policy), Diffusion Policy (DP) [Chi et al., 2023], ACT [Zhao et al., 2023], and Pi 0.5 [Black et al., 2024]. All policies are trained on 300 demonstrations per task generated by AffordSim, using RGB-D observations and proprioceptive state as input. Training details are provided in Appendix A. 4.1
Benchmark Evaluation
Each task is trained on 300 demonstrations with object pose randomization applied during data collection; all evaluation rollouts also include object pose randomization to test generalization. Task-specific randomization ranges are detailed in Appendix A. Table 1 presents results on representative tasks from each category. Across all baselines, grasping tasks achieve the highest success rates (53–93%), confirming that basic pick-up manipulation is largely solved by current methods. Placing and pushing/pulling tasks show moderate-to-high success (3–93%), with failures primarily occurring during the precision-placement or target-reaching phase rather than the initial grasp. The most striking finding is the sharp performance drop on affordance-demanding tasks. Pouring into a wide basket achieves 20–99% success, but pouring into a narrow cup drops to 1–43%—a significant degradation for the same manipulation primitive (pour) when the target container demands higher positional precision. Mug hanging, which requires the policy to align the mug handle with a hook or rack peg, achieves at most 47% success even for the best baseline. These results highlight that affordance-demanding tasks—where success depends on the precision of functional-region 6
Table 1: Benchmark evaluation results (Success Rate %) on representative tasks across six categories. Each task is evaluated over 30 rollouts. Sub-task success rates are shown where applicable. Best results per task in bold. Category
Task
BC ACT DP Pi 0.5
Grasping
pick_banana pick_red_cup
53 17
63 77
87 63
93 80
Placing
pick_banana_place_plate pick_cup_place_shelf pick_kettle_place_coffee machine
37 5 27
60 7 43
84 3 49
93 47 65
Stacking
stack_two_blocks stack_three_blocks
30 13
43 27
47 28
77 53
Push/Pull
push_box_to_target pull_drawer_open
23 37
63 53
67 58
78 86
Pouring
pour_basket_into_bowl pour_pan_into_bowl pour_cup_into_bowl
20 7 1
86 33 24
94 87 36
99 92 43
Mug Hanging
hang_mug_on_rack hang_mug_on_hook
0 0
10 7
17 13
47 33
pick_cup_pour_place_coffee_machine Long Horizon pick_cup_pour_hang_on_rack open_microwave_place_can
0 0 0
3 1 0
8 3 2
16 21 13
Average (all 17 tasks)
16
35
44
61
Figure 4: Cross-embodiment support. Three representative tasks executed by four robot embodiments: Franka FR3, Franka Panda, UR5e, and Kinova. AffordSim generates affordance-guided trajectories for each embodiment without task-specific tuning. interaction—remain significantly more challenging for current methods, underscoring the importance of affordance-aware data generation. Pi 0.5 achieves the highest average performance (61%), followed by DP (44%) and ACT (35%). BC lags significantly behind (16%), suggesting that foundation-model-based, diffusion-based, and action-chunking architectures better capture the multimodal action distributions characteristic of affordance-guided trajectories. Sub-task analysis. The sub-task breakdown reveals that grasping (the first sub-task) succeeds at 63–100% across pouring tasks, while the downstream pouring action fails dramatically for small containers. This localizes the bottleneck: current policies can learn affordance-guided grasps from data but struggle to execute the fine-grained, affordance-dependent manipulation that follows. 4.2
Cross-Embodiment Evaluation
We evaluate trajectory generation success across four robot embodiments on 10 representative tasks. AffordSim successfully generates affordance-guided trajectories for all embodiments, with generation success rates of 94% (Franka FR3), 92% (Franka Panda), 83% (UR5e), and 95% (Kinova). The variation reflects kinematic differences rather than affordance prediction failures, as the VoxAfford 7
Table 2: Affordance integration ablation. Trajectory generation success rate (%) under four grasp strategies. Manual: human-specified grasp poses without affordance. AnyGrasp: generic grasp estimation. VoxAfford: predicted affordance guidance (ours). Human Aff.: human-annotated affordance ground truth (oracle upper bound). Evaluated with Pi 0.5. Task
Manual
AnyGrasp
VoxAfford
Human Aff.
pick_banana_place_plate pour_into_pan pour_into_cup hang_mug_on_rack hang_mug_on_hook pick_screwdriver
100 67 83 87 83 100
47 20 0 0 0 53
98 80 63 10 17 97
100 93 80 90 87 100
Average
87
20
61
92
module is embodiment-agnostic. Notably, the UR5e exhibits lower success on orientation-sensitive tasks such as pouring, because its 6-DOF kinematics (versus 7-DOF for the other arms) limits wrist dexterity and reduces the feasible set of end-effector orientations. This confirms that AffordSim’s affordance-aware pipeline generalizes across robot platforms without task-specific tuning. 4.3
Affordance Integration Ablation
Table 2 compares four grasp strategies on affordance-demanding tasks: Manual (human-specified grasp poses without affordance reasoning), AnyGrasp [Fang et al., 2023] (generic grasp estimation), VoxAfford (our predicted affordance guidance), and Human Aff. (human-annotated affordance ground truth, serving as an oracle upper bound). Manual grasp design achieves the highest baseline performance (87%), yet each new object or task instance requires re-designing grasp poses from scratch—a labor-intensive process that does not scale. AnyGrasp (20%) performs poorly on affordance-demanding tasks, as generic grasp estimation cannot reason about task-relevant contact regions. VoxAfford (61%) substantially outperforms AnyGrasp, demonstrating the value of learned affordance guidance. On pouring tasks, VoxAfford achieves 80% (pan) and 63% (cup) vs. AnyGrasp’s 20% and 0%, because affordance guidance ensures the robot grasps the cup at a functionally correct position for tilting. On mug hanging, however, VoxAfford achieves only 10–17%, because the VoxAfford model was not trained with hanging-specific affordances: hanging a mug on a rack requires grasping the cup body rather than the handle, which differs from the conventional grasp affordances in the training data. AnyGrasp completely fails (0%) on these tasks as well. The Human Aff. column (92%) establishes the oracle upper bound: when ground-truth affordance labels replace VoxAfford predictions, success rates improve by ∼31 percentage points. This gap is largely driven by the mug-hanging tasks where VoxAfford lacks training coverage, indicating that expanding the affordance training set to include non-standard grasp regions is the primary avenue for improvement. On tasks where VoxAfford has adequate training data (e.g., pick-and-place, pouring, screwdriver), it already recovers over 80% of the oracle performance. 4.4
Zero-Shot Sim-to-Real Transfer
We deploy Pi 0.5 trained exclusively on AffordSim data to a real Franka FR3, evaluating on one representative task from each category (Table 3). The sim-to-real transfer success rate mirrors the affordance-complexity gradient observed in simulation: grasping achieves 60%, placing 30%, push/pull 40%, and stacking 20%, while pouring (20%) and mug hanging (10%) remain challenging. These results confirm two findings: (1) AffordSim-generated data supports effective sim-to-real transfer, and (2) the affordance-complexity bottleneck is not an artifact of simulation but reflects genuine difficulty in learning fine-grained functional interactions. 4.5
Domain Randomization Robustness
To evaluate whether domain randomization equips policies with robustness to environmental variations, we train DP and Pi 0.5 on three representative tasks (pick_banana, pick_cup_place_shelf, 8
Table 3: Zero-shot sim-to-real transfer on a real Franka FR3. Policies trained exclusively on AffordSim data (Pi 0.5). Each task evaluated over 10 trials. Category
Task
SR (%)
Grasping Placing Stacking Push/Pull Pouring Mug Hanging
pick_banana pick_cup_place_shelf stack_two_blocks push_box_to_target pour_into_pan hang_mug_on_rack
Average
60 30 20 40 20 10 30
Table 4: Domain randomization robustness. Sim-to-real transfer success rate (%) on a real Franka FR3 under different training and evaluation conditions. Standard: original deployment scene. Novel scene: modified tablecloth and background objects. Each task evaluated over 10 trials. DP
Pi 0.5
Setting
pick place pour pick place pour
w/o DR (standard) w/ DR (standard) w/ DR (novel scene)
10 20 20
0 0 0
0 0 0
40 50 50
10 20 20
0 10 10
pour_pan_into_bowl), each with 300 demonstrations, under two settings—with and without domain randomization—and evaluate on the real Franka FR3 in both the standard deployment scene and a novel scene with modified tablecloth and background objects (Table 4). Specific randomization parameters for lighting, texture, and tablecloth are provided in Appendix A. Policies trained without domain randomization achieve very low success rates in the standard scene (DP: 3% avg, Pi 0.5: 17% avg). Adding domain randomization during training improves performance notably, with Pi 0.5 increasing from 17% to 27% average success. Crucially, DR-trained policies maintain their performance even when the real scene is visually altered: Pi 0.5 achieves identical success (50/20/10%) in both standard and novel scenes, and DP likewise shows consistent results across scene conditions. This demonstrates that AffordSim’s domain randomization pipeline effectively endows policies with robustness to environmental perturbations, enabling deployment across varying real-world conditions without retraining.
5
Discussion and Limitations
Affordance prediction accuracy. AffordSim’s trajectory quality depends on VoxAfford’s affordance predictions. While VoxAfford generalizes well to common object categories, prediction accuracy degrades for novel objects with unusual geometries or under heavy occlusion. Failure cases include thin tool handles, transparent objects, and objects with ambiguous functional regions (e.g., symmetric containers with no clearly defined pouring side). Scope of manipulation. The current framework is limited to rigid-body manipulation. Deformable objects (cloth, rope, dough) require different simulation backends and affordance representations, which we leave for future work. The benchmark also does not cover in-hand manipulation or bimanual tasks, both of which introduce additional affordance reasoning challenges (e.g., coordinated functional region contact across two grippers). Background reconstruction. The DA3 Gaussian reconstruction requires 10–20 photographs of the real deployment scene. While this is a one-time effort per workspace, it limits portability to new environments without re-collection. An alternative would be diffusion-based style transfer, though at the cost of geometric accuracy. Broader impact. AffordSim accelerates the development of robotic manipulation systems by reducing the cost of training data generation. While this benefits applications in manufacturing, 9
logistics, and assistive robotics, autonomous manipulation capabilities could be misused in contexts where physical interactions should require human oversight. We release our code and benchmark under a permissive license to support reproducible research.
6
Conclusion
We introduced AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into robotic manipulation data generation. By combining VoxAfford for affordanceguided grasp planning, VLM-powered task generation, and photorealistic domain randomization with DA3-reconstructed real backgrounds, AffordSim produces semantically correct manipulation trajectories at scale. Our 50-task benchmark across 6 categories reveals that affordance-demanding tasks—pouring and mug hanging—remain a critical open challenge for current imitation learning methods, exposing affordance complexity as a missing evaluation dimension. Zero-shot sim-to-real experiments validate the practical utility of the generated data. We hope that AffordSim and its benchmark will serve as a catalyst for developing manipulation policies that reason about object affordances.
References Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. Robotics: Science and Systems (RSS), 2023. Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3D AffordanceNet: A benchmark for visual object affordance understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1778–1787, 2021. Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. AnyGrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023. Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. GAPartNet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. Haoran Geng et al. GENMANIP: LLM-driven simulation for generalizable instruction-following manipulation. arXiv preprint, 2025. Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. ManiSkill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659, 2023. Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. GenSim2: Scaling robot data generation with multi-modal and reasoning LLMs. arXiv preprint arXiv:2410.03645, 2024. Bo Liu, Yifeng Zhu, Chongkai Gao, Yizhou Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 2024. Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2Act: From pixels to actions for articulated 3D objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021. 10
Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. RoboTwin: Dual-arm robot benchmark with generative digital twins. arXiv preprint arXiv:2409.02920, 2024. Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parber, Tsung-Wei Lo, Avanika Joshi, Huihan Welborn, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523, 2024. NVIDIA. NVIDIA Isaac Sim. https://developer.nvidia.com/isaac-sim, 2023. Version 5.1. Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. IEEE International Conference on Robotics and Automation (ICRA), pages 3803–3810, 2018. RoboVerse Team. RoboVerse: Towards a unified platform for scalable and generalizable robot learning. arXiv preprint arXiv:2504.09837, 2025. Balakumar Sundaralingam, Siva Kumar Ber, Tucker Hermans, and Byron Boots. cuRobo: Parallelized collision-free robot motion generation. In IEEE International Conference on Robotics and Automation (ICRA), 2023. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30, 2017. Yufei Wang, Zhou Fan, Zackory Jia, Siddhartha Srinivasa, and Danfei Xu. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455, 2023. Toan Yang et al. DA3: Dense affordance annotation from 3D gaussian splatting. arXiv preprint, 2024a. Toan Yang et al. VoxAfford: Open-vocabulary affordance detection in 3D point clouds. arXiv preprint, 2024b. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL), 2020. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems (RSS), 2023.
11
A
Training Details
All imitation learning policies are trained on 300 demonstrations per task. Observations consist of two RGB-D camera views (wrist-mounted and third-person, 256×256 resolution) and 7-DoF proprioceptive state (joint positions). Actions are 7-DoF end-effector pose commands (position + quaternion orientation + gripper). BC. MLP with 3 hidden layers (256 units each), ReLU activations. ResNet-18 visual encoder. Trained for 500 epochs with batch size 64, learning rate 10−4 (Adam). Diffusion Policy (DP). U-Net denoiser with 256-dim latent, 100 diffusion steps at training, 10 DDIM steps at inference. Observation horizon To = 2, action horizon Ta = 8, prediction horizon Tp = 16. Trained for 500 epochs, batch size 64, learning rate 10−4 . ACT. CVAE architecture with d = 512 latent dimension, k = 100 action chunks. Transformer encoder-decoder with 4 layers, 8 heads. Trained for 500 epochs, batch size 64, learning rate 10−5 . Pi 0.5. Pretrained Pi 0.5 backbone fine-tuned on AffordSim data. Full fine-tuning of action head, frozen vision-language backbone. Fine-tuned for 100 epochs, batch size 32, learning rate 10−5 . All experiments are conducted on a single NVIDIA A100 80GB GPU. Training time ranges from 2 hours (BC) to 12 hours (Pi 0.5) per task.
B
Full Benchmark Results
The full 50-task benchmark results are available in the supplementary material and project website. Table 1 in the main paper shows representative tasks from each category.
C
Sim-to-Real Setup
The real-world evaluation uses a Franka FR3 robot arm with a Franka Hand gripper, mounted on a tabletop workspace. Two Intel RealSense D435 cameras provide RGB-D observations (wrist-mounted and third-person view at 45◦ elevation). The DA3 Gaussian background is reconstructed from 15 photographs of the workspace taken from diverse viewpoints. All policies are evaluated zero-shot without any real-world fine-tuning, with 10 trials per task.
12