InstructMesh: Selective Refinement of Generative 3D Models for Fabrication

arXiv:2608.28534v1 [cs.AI] 28 Aug 2026

InstructMesh: Selective Refinement of Generative 3D Models for Fabrication Faraz Faruqi

Ahmed Katary

Demircan Tas

ffaruqi@mit.edu MIT CSAIL Cambridge, MA, USA

atkatary@google.com Google Sunnyvale, CA, USA

tasd@mit.edu MIT CSAIL Cambridge, MA, USA

Theresa Hradilak

Ning Zhang

Jiaji Li

hradilak@mit.edu MIT CSAIL Cambridge, MA, USA

ningrz@mit.edu MIT CSAIL Cambridge, MA, USA

jiaji@mit.edu MIT CSAIL Cambridge, MA, USA

Fabian Manhardt

Martin Nisser

Vrushank Phadnis

fabianmanhardt@google.com Google Munich, Germany

nisser@uw.edu University of Washington Seattle, WA, USA

vrushank@google.com Google Mountain View, CA, USA

Ruofei Du

Federico Tombari

Megan Hofmann

ruofei@google.com Google XR San Francisco, CA, USA

tombari@google.com Google Zurich, Switzerland

m.hofmann@northeastern.edu Khoury College of Computer Sciences, Northeastern University Boston, MA, USA

Stefanie Mueller stefmue@mit.edu MIT CSAIL Cambridge, MA, USA

Figure 1: InstructMesh enables fabrication-relevant refinement of generative 3D models through selective editing. a) A user provides an initial prompt. b) The system generates a 3D model, but may include fabrication-related flaws (e.g., a sealed lid). c) The user selects the problematic region and provides a descriptive edit prompt. d) InstructMesh applies an edit operation in the latent-space to remove the lid while preserving other regions and regenerates the model. e) The final model is fabricated as a functional mug.

ABSTRACT Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UIST ’26, November 02–05, 2026, Detroit, MI, USA

Recent advances in generative AI allow users to create 3D models from text or images. However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws © 2026 Association for Computing Machinery. ACM ISBN 979-8-4007-2856-3/2026/11. . . $15.00 https://doi.org/10.1145/3830398.3830647

UIST ’26, November 02–05, 2026, Detroit, MI, USA

that compromise their intended use post-fabrication. We present InstructMesh, an interactive post-generation refinement tool that enables selective repair of generative 3D models through region selection and targeted operations, such as opening or sealing voids, or adjusting local thickness. Users can invoke edit operations via natural language prompts or slider controls. By operating directly on the intermediate latent representation, InstructMesh allows users to apply robust geometric corrections without requiring expert modeling skills. To inform our design, we first analyze common fabrication-related failure modes in outputs from state-of-the-art generative tools. We then conduct two user studies, demonstrating that novices can identify and perform fabrication-relevant repairs on generative outputs using InstructMesh, and revealing user preference for hybrid interfaces that combine slider controls with natural language input.

CCS CONCEPTS • Human-centered computing → Human computer interaction (HCI).

KEYWORDS Personal Fabrication; Generative AI; 3D modeling. ACM Reference Format: Faraz Faruqi, Ahmed Katary, Demircan Tas, Theresa Hradilak, Ning Zhang, Jiaji Li, Fabian Manhardt, Martin Nisser, Vrushank Phadnis, Ruofei Du, Federico Tombari, Megan Hofmann, and Stefanie Mueller. 2026. InstructMesh: Selective Refinement of Generative 3D Models for Fabrication. In The 39th Annual ACM Symposium on User Interface Software and Technology (UIST ’26), November 02–05, 2026, Detroit, MI, USA. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3830398.3830647

1

INTRODUCTION

Generative AI has enabled new forms of creative expression across modalities like text [11], images [43], music [33], and 3D geometry [22]. In the 3D domain, these advances now allow users with minimal modeling experience to generate novel 3D models from text or image prompts, bypassing the steep learning curve of traditional 3D modeling tools. However, while generative models produce visually plausible outputs, they are typically trained with image-based supervision, prioritizing appearance over the geometric properties required for real-world fabrication [17]. As a result, generated models could exhibit flaws such as sealed openings, insufficient wall thickness, or missing structural connections, that only become apparent when a model is fabricated and used in a downstream physical task. Bridging the gap between a visually plausible model and a physically usable one requires fabrication-relevant repairs that are difficult for novices to achieve. Mesh-editing tools such as Blender [13] or MeshMixer [46] can address geometric flaws, but require significant domain expertise and effort that novices may not have. Recent text-driven 3D editing methods such as DreamEditor [71] and TIP-Editor [70] allow localized edits guided by text prompts, but operate on neural radiance fields to target visual appearance rather than the fabrication-specific geometric corrections needed for realworld use. Similarly, tools like Style2Fab [17] and TactStyle [19] support functionality-aware transformations of existing designs,

Faruqi et al.

but assume that the relevant functional geometry already exists in the base model, focusing on preservation rather than repair. Finally, re-prompting the generative model is often unreliable: prior work shows that non-experts struggle to craft effective prompts [52, 62], often producing vague or overly rigid instructions that fix one issue, only to alter other desired details, trapping them in a frustrating prompt-refinement loop [32]. To bridge this gap, we introduce InstructMesh, an interactive post-generation refinement tool that supports fabrication-relevant repairs of generative 3D models by mapping user-specified edits to targeted operations on the model’s intermediate latent representation. Rather than forcing novices to edit the mesh directly, InstructMesh modifies the coarse voxel grid produced by the generative model’s encoder, then passes the edited representation through the decoder to produce a refined mesh. This allows the generative model to fill in low-level details such as texture and surface topology, while satisfying the user’s intended geometric correction. A key challenge in designing such tools for generative 3D models is determining how novices should express geometric corrections: through natural language, which is expressive but ambiguous, or through structured controls, which are precise but require understanding the available operations? To investigate this, InstructMesh provides two complementary interaction modes: a natural language interface through an LLM, and a slider interface exposing a library of parameterized operations. In both modes, users first highlight a problematic region on the mesh, then specify the desired change. A second design challenge arises from the opacity of latent-space editing: because the intermediate representation is non-visual, users cannot anticipate how their edit will affect the final mesh. To make this process controllable, InstructMesh provides a preview visualization — showing additive edits in green and subtractive edits in red, making the latent-space edit tangible before regeneration is triggered. This introduces WYSIWYG1 -style interaction into generative 3D modeling, allowing users to inspect, revise, and confirm edits on the intermediate representation, thereby maintaining direct control over the generative process. Together, these design choices enable us to investigate two research questions about novice-driven repair of generative 3D models for fabrication. First, can novices reliably identify and correct visually identifiable fabrication-relevant flaws using latent-space editing, a task that has traditionally required sophisticated mesh-manipulation skills? Second, how do the choice of interaction mode and the availability of visual feedback on the intermediate representation shape their editing experience? In summary, this work contributes: (1) A formative analysis (N = 120 models) identifying common fabrication-relevant geometric flaws in outputs from stateof-the-art 3D generative models, (2) A set of canonical latent-space operations that repair these flaws without requiring mesh manipulation expertise, paired with an interaction design combining natural language input and parameterized slider controls, and a preview visualization for user understanding, (3) Two user studies (N=12) demonstrating that novices can successfully identify and repair visually identifiable flaws 1What You See Is What You Get

InstructMesh

using InstructMesh, and that hybrid workflows with preview visualizations improve usability and transparency.

2

RELATED WORK

We situate our work at the intersection of human-computer interaction, generative AI, and computational fabrication. Specifically, we draw on prior research in three key areas: (1) support tools in digital fabrication workflows, (2) 3D generative AI workflows and their integration into design processes, and (3) human-in-the-loop customization techniques in 2D and 3D generative systems.

2.1

Supporting Users in Fabrication Workflows

3D printing is emerging as makers’ digital fabrication tool of choice [30]. Online repositories such as Thingiverse2 provide a low barrier to entry for exploring 3D printing as users can quickly download pre-made 3D designs and explore design-specific questions [16]. However, numerous studies have explored the practices of these novice makers [3, 25, 38, 45] and tend to find that makers struggle with modifications to existing designs due to limitations of current computer-aided design (CAD) workflows. Oehlberg et al. [39] observed that, even when designs are customizable, they lack the full range of modifiable features makers seek. Researchers have proposed several support tools that enable editing of 3D models by users with the goal to fabricate an object. Meshmixer [46] follows the approach of ‘Design to Fabricate’ [45] and allows users to manipulate 3D models with the intent to 3D print them afterwards. Several parametric design tools [47, 50, 51, 54] allow interactive customization of parametric designs by a novice user, while maintaining validity and manufacturability of the design. Tools also support specific retargeting of mechanical components [29, 44, 65], allowing users to recombine mechanical components between 3D models. Several optimization algorithms target post-fabrication physical properties of 3D models, such as balance [41], moment-of-inertia for spinnable models [2], or constrained optimization between center-of-mass and balance [40, 67]. Recent work has also demonstrated augmentation of generative models with mechanical simulation to ensure structural viability [15]. However, these fabrication support tools typically edit existing meshes or models, limiting the ability of novice users to freely explore new designs. Recent developments in generative AI workflows offer an alternative approach, enabling users to construct bespoke 3D models directly from image and text prompts.

2.2

3D Generative AI Workflows

Reconstructing 3D objects from a single image or text description is a longstanding challenge in computer vision. Recent machine learning methods address this problem by learning priors over large collections of 3D shapes to infer the full geometry of an object from a single image. Several approaches have been proposed, including GANs [8, 22], flow networks [28, 61], VAEs [36, 57], and diffusion models [10, 26]. The recent Large-Reconstruction Model (LRM) architecture [23] and its variants [4, 53, 60] have demonstrated generalizable, high-quality 3D reconstruction. Among these, Trellis [59] introduces a structured latent representation that separates 2 Thingiverse: https://www.thingiverse.com

UIST ’26, November 02–05, 2026, Detroit, MI, USA

geometry encoding from mesh decoding, achieving state-of-the-art reconstruction fidelity. Commercial platforms such as Meshy [35], Hunyuan3D [68], and CLAY [64] have further broadened access, enabling users to produce textured 3D models from text or image inputs. These generative methods have enabled support tools for creative exploration and rapid prototyping [48]. In addition to generation, a parallel line of work has explored text-driven editing of 3D models (see Lu et al. [34] for a survey). For example, DreamEditor [71] enables text-guided editing of 3D scenes by optimizing neural radiance fields, while TIP-Editor [70] extends this to accept both text and image prompts for localized edits. However, while these generative methods produce visually plausible 3D models, they are typically trained using image-based supervision. As a result, they prioritize appearance over physical or functional correctness [21]. This leads to 3D models that may look realistic but lack critical geometric features essential for realworld functionality. Consequently, models generated through these pipelines may exhibit failure modes that only become apparent when used in downstream tasks like simulation or 3D printing. To characterize the scope and prevalence of these flaws, we conducted a formative study examining outputs from a state-of-the-art generative model, which we describe in Section 3.

2.3

Human-in-the-Loop Customization with Generative Models

The challenges in generating functional models with generative AI stem from a broader limitation shared across generative modalities. Generative systems must predict complex outputs from underspecified inputs, introducing ambiguity that often leads to outputs misaligned with user intent. Novices struggle particularly with this ambiguity, crafting vague or overly rigid prompts and failing to debug faulty outputs [52, 62]. This has driven increasing interest in human-in-the-loop approaches that allow users to guide or correct the generative process. In the image domain, tools such as ControlNet [63], PromptPaint [12], PromptCharm [55], and Promptify [5] support iterative refinement through visual conditionings, regional prompting, and attention visualization. Similar trends have emerged with LLMs: ABScribe [42], AI Chains [56], and EvalLM [27] enable structured revision, inspectable intermediate steps, and evaluator-guided prompt refinement. In the 3D domain, part-based composition interfaces similarly give users structural control over generative outputs [18, 20]. Together, these systems illustrate a shift toward mixedinitiative workflows where users and models share creative agency through localized control, visual feedback, and iteration. InstructMesh extends these principles to 3D generative modeling. Where image-domain tools give users region-level control over appearance, InstructMesh gives users region-level control over fabrication-relevant geometry, enabling novices to correct structural flaws directly on generated models through selection, preview, and iterative revision.

3

FORMATIVE STUDY

A key challenge in using generative AI for 3D modeling is ensuring that outputs contain the geometric details needed for physical functionality such as appropriate wall thickness, openings, and

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Missing Openings

Fused Joints

Broken or Missing Bridges

Extraneous Artifacts

Fused or Duplicated Features

Faruqi et al.

Hollowing Errors

Wall Thickness Issues

Topological Errors

Truncated Features

Generated Model

3.4

Flaw Highlighted

Figure 2: Descriptive examples of flaws found in our formative study. Models were sourced from Thingiverse, reconstructed with 3D generative model Trellis [59] using their image as input, and evaluated manually for geometric functionality errors by reviewers.

manifold integrity. We hypothesize that current generative models replicate aesthetic properties but fail to capture these local geometric details, rendering models non-functional after fabrication. To test this, we conducted a formative study assessing geometric errors in reconstructions of popular 3D models from Thingiverse3 .

3.1

Dataset Selection and Generative 3D Reconstruction

Thingiverse is a popular online resource for novice and expert makers to share designs or things for 3D printing. We selected the 100 most popular ‘things’ (collections of models) from Thingiverse, skipping multi-part components without assembled versions since current generative tools produce single meshes. After preprocessing, the dataset comprised of 120 distinct unique 3D models (100 unique ‘things’ from Thingiverse). We rendered one image of each model and reconstructed it using Trellis [59], chosen for its stateof-the-art geometric fidelity. While we scope our analysis on Trellis for consistent reconstruction and evaluation, nearly all current 3D generative methods employ a similar two-stage encoder-decoder pipeline with image-based training objectives.

3.2

Inductive Taxonomy Development

We used an iterative qualitative coding method to develop our taxonomy of geometric flaws in 3D models - geometric errors in reconstruction that impact the functionality of the 3D model. For each of the 120 3D models, two expert annotators in 3D modeling compared the generated model and the original model, and described the issues in natural language. Their task was to describe each individual issue independently, and describe the change required. Next, they discussed these annotations and came to an agreement on the issues found.

3.3

process concluded when no new categories emerged from additional data, at which point the taxonomy had stabilized at nine flaw categories. We report these categories in Table 1 and show descriptive examples in Figure 2.

Iterative Categorization of Issues

Next, the two annotators independently grouped similar descriptions of issues, then collaboratively discussed and refined these groupings. We continued this process over multiple rounds of iterative refinement, merging overlapping categories and sharpening definitions through discussion until consensus was reached. The 3 https://www.thingiverse.com

Deductive Classification

After developing our taxonomy of fabrication-relevant geometric flaws, we performed a deductive classification of the dataset using the previously collected annotations. The two expert annotators revisited their initial descriptions and mapped each issue to one or more of the nine categories found in the taxonomy. Across the 120 models analyzed, we found that fabricationrelevant flaws were widespread: 94 models (78.3%) exhibited more than one distinct type of issue, highlighting the compounding nature of geometric failures in generative reconstructions. On average, each model exhibited 2.4 issues (SD = 1.10), suggesting that these flaws commonly co-occur and cannot be addressed through isolated fixes. The most frequently encountered issues were Extraneous Artifacts (65.8%), Missing Openings (48.6%), and Hollowing Errors (36.9%), while more localized structural flaws like Broken / Missing Bridges (10.8%) or Truncated Features (9.9%) appeared less frequently. These findings suggest that while state-of-the-art generative models achieve high visual fidelity, they overlook fine-grained geometric properties critical for fabrication, which can be difficult to capture through image-based supervision. The prevalence and diversity of these flaws motivate the design of post-generation tools that support targeted, user-guided geometric correction. We describe our approach in the following section.

4

SYSTEM OVERVIEW

InstructMesh enables users to generate 3D models with text or image prompts and iteratively refine them to fix fabrication-relevant issues. Recent 3D generative methods, including TRELLIS [59] and SPAR3D [24], employ a two-stage pipeline: an encoder produces a coarse latent structure (e.g., point cloud or voxel grid) and contextual features about appearance, and a decoder refines this information into a high-fidelity mesh. One key idea is that this separation enables controlled customization by modifying this intermediate representation. InstructMesh operates directly on this intermediate latent voxel representation. We use TRELLIS [59] as the generative backbone, which represents each shape as a structured latent over a 643 sparse voxel grid; we persist the latent structure alongside the decoded mesh so that user edits can be applied to this latent structure and decoded to produce a new mesh. Figure 3 illustrates the system design, which comprises of: (1) a set of latent-space operations derived from common fabrication flaws, (2) dual interaction modes—a natural language interface powered by an LLM and a slider interface exposing parameterized presets, and (3) a preview visualization that highlights additive edits in green and subtractive edits in red before they are committed. Together, these components provide fine-grained customization with generative 3D models while maintaining user’s transparency and control. We focus on repairing static functionality in 3D models, as current generative methods do not yet support dynamic mechanisms

InstructMesh

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Table 1: Taxonomy of Fabrication-Relevant Geometric Flaws in 3D Model Reconstructions. Based on 120 most popular 3D models on 3D model website Thingiverse. Flaw Category

Description

Missing Openings

Absence of essential through-holes, ventilation gaps, volumes to contain liquids or support airflow. Components meant to rotate or articulate (e.g., hinges, pivots, ball joints, axles) are fused, preventing intended motion. Crucial geometric connections between parts are missing, leaving the structure fragmented or unsupported. Unintended bumps, extensions, or disconnected mesh elements. Features are erroneously duplicated, stacked, or fused together, often due to generative ambiguity. Solid elements are mistakenly hollowed or vice versa. Non-uniform or insufficient wall thicknesses. Gaps, tears, or ambiguous topologies in the mesh. Extrusions or geometric extensions are prematurely cut off, resulting in incomplete or malformed shapes.

Fused Joints Broken or Missing Bridges Extraneous Artifacts Fused / Duplicated Features Hollowing Errors Wall Thickness Issues Topological Errors Truncated Features

Latent Decoder

Latent Encoder

Original Model

No

d

Yes

c

b

“Remove this lid”

“A mug with a dragon wrapped around it as the handle” a

Flaws?

Models Affected

Modified Voxel Representation f

Input Prompt

LLM

Identify Operation

(In Context Learning)

Figure 3: InstructMesh System Overview. (a) The user provides a text or image prompt. (b) The model is encoded into a latent voxel representation and (c) decoded into a textured mesh. (d) The user inspects the mesh for flaws; if none, the model proceeds to (g) fabrication. Otherwise, the user highlights a region and specifies an edit. (e) The latent operation is applied and previewed. (f) Once confirmed, the edited latents are decoded into a refined model. User specifies the edit either in text prompt or using direct action

b

c

d

10.8% 65.8% 27% 36.9% 20.7% 9.9% 9.9%

Preview

Edit Highlight

Subtractive Operation Preview

User prompt “”

Additive Operation Preview

User prompt “ ”

Refined Model

e

Figure 5: Preview visualizations of canonical operations. Top: a subtractive edit (“make a hole”) removes the sealed lid. Bottom: an additive edit (“connect these parts”) generates a handle. Red regions indicate subtractive changes; green indicates additive. Previews let users verify edits on the latent representation before triggering regeneration.

User previews edit before generation

Refined model is rendered in the UI

a

13.5%

Fabricated g Output

e

User highlights problematic region

48.6%

f

Figure 4: The InstructMesh web interface. (a) Users enter a text or image prompt. (b) The generated mesh is inspected for flaws. (c) Users paint the problematic region. (d) The edit is specified via natural language or direct selection. (e) A preview is shown for confirmation. (f) The refined model is generated.

(e.g., hinges or interlocking parts) that require precise geometric and mechanical constraints. Our goal is to address the static fabricationrelated flaws identified in our formative study, with the exception of Fused Joints, which involves articulated or mechanical components.

4.1

User Interface and Workflow

Figure 4 shows the InstructMesh web interface. Users begin by providing a text or image prompt to generate a 3D model (Fig. 4a), which is rendered for inspection (Fig. 4b). If a flaw is identified, they activate Highlight Region and paint the problematic area (Fig. 4c). Users can then specify the desired edit either by entering a natural language instruction or by selecting an operation from the slider interface (Fig. 4d). Before the edit is applied, the system presents a preview overlay: additive changes are shown in green and subtractive changes in red (Fig. 5). This makes latent edits transparent and gives users the opportunity to revise or cancel. Once confirmed, the operation is executed on the corresponding subset of the latent representation and decoded into an updated mesh (Fig. 4f). The edited model appears next to the original for side-by-side comparison, and users can continue refining, undo an operation, or download the final repaired model.

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Faruqi et al.

This workflow enables localized, in-situ corrections, abstracting away the complexity of latent manipulation, while providing transparent feedback loops through preview visualizations. Our goal with this design is to enable novices to iteratively refine generative 3D models with both expressiveness and control.

4.2

Canonical Operations to Modify Latent Representation

In this section, we describe our set of latent-space operations that enable localized, precise edits of a generative 3D model. Editing the latent voxel representation, rather than directly modifying the mesh, offers key advantages. Latent grids are coarser and lack explicit connectivity, making them easier to manipulate without navigating complex mesh topology. Additionally, the edited voxels are processed by the decoder, which combines them with learned appearance features to produce a coherent mesh, synthesizing missing geometry without requiring the topology-aware, texture-preserving edits required in tools like Blender. This enables novices to make structural corrections that would otherwise require expert-level mesh manipulation. 4.2.1 Mapping Selections to the Latent Representation. When a user highlights a region on the decoded mesh, the system maps this selection back to the latent voxel grid as follows. First, it recovers the decoder’s coordinate transformation by aligning the mesh’s bounding box to the voxel grid via scaling, rotation, and translation. The selected faces are then extracted as sub-meshes in this voxelaligned frame, where they serve as geometric references for the canonical operations: each sub-mesh identifies the voxels to modify. For example, fill uses convex-hull containment, while trim uses signed-distance queries. The modified voxel structure is then passed through the TRELLIS [59] decoder, which regenerates a coherent mesh from the edited representation. This pipeline is tolerant to imprecise selection through three mechanisms. First, the selection is not treated as an exact voxel mask: connected components below a minimum size are discarded, removing stray fragments from accidental brushing, while all remaining sub-meshes are retained to support disjoint selections (e.g., the two endpoints of a broken bridge). Second, operations act on the neighborhood of these sub-meshes using a proximity radius. Expand, for instance, grows voxels outward from the selection, so that missing a few faces still produces a coherent edit. Third, the preview renders the modified voxels directly, without invoking the decoder, allowing users to verify the actual effect of an operation before committing (Fig. 5). We provide pseudocode in Appendix A. With selections mapped onto the latent grid, the system next needs a vocabulary of edits to apply there. We defined a small set of canonical operations analogous to geometric primitives such as extrude or subtract, that are compatible with latent voxel grids and composable: individual edits can be combined or sequenced to address complex repair scenarios (Fig. 6). We group our operations into two classes: 4.2.2 Additive Operations. These increase voxel occupancy by extending surfaces outward, thickening thin regions, or filling enclosed volumes. They are used to reinforce fragile areas or close gaps.

Figure 6: Canonical operations on voxel representations. User highlights (pink) are mapped to regions in the latent voxel grid. Additive operations (left) grow occupancy via extrusion, expansion, or filling. Subtractive operations (right) reduce occupancy via trimming, erosion, or flattening. The modified grid is decoded to produce the refined model.

• Extrude extends the selected surface region by adding voxels along its average outward normal, useful for lengthening handles or creating protrusions. • Expand thickens structure by adding a uniform shell of voxels around the selection, reinforcing thin walls and fragile geometry. • Fill closes surface gaps or hollow volumes by populating voxels inside the convex hull of the selection, commonly used to solidify hollow parts or create bases and lids. 4.2.3 Subtractive Operations. These reduce occupancy by cutting away selected regions, thinning shells, or smoothing surfaces. They are used to remove unwanted material, hollow interiors, or prepare flat contact areas. • Trim removes voxels near the selected surface based on a signed distance field threshold, useful for cleaning artifacts or cutting away extraneous material. • Erode thins a surface or creates voids by subtracting an interior shell of voxels, often used to hollow parts or carve channels. • Flatten smooths a region toward a best-fit plane, preparing contact surfaces or flat bases for fabrication. Together, these operations form a compact, extensible vocabulary for repairing generated 3D models. The latent voxel representation provides a meaningful locus of operations, and the flexibility of decoding process into the final mesh allows us to create these representation-agnostic operations. Thus, these representation switches are abstracted away from the user who can focus on the high level design intent. Each operation can be parameterized (e.g., extrusion depth, erosion thickness), enabling a range of functional modifications. Formal definitions and pseudocode for all operations are provided in Appendix A. 4.2.4 Operation Prediction Using LLMs with In-Context Learning. To map natural language edit descriptions to canonical operations, we employ in-context learning (ICL) with OpenAI’s GPT-4 [1]. The LLM does not interpret or edit 3D geometry directly: it maps the user’s instruction to an operation and parameters from our fixed vocabulary, while spatial grounding comes from the user’s region selection. Constraining the LLM to select from a pre-parameterized set of operations rather than synthesize direct edits, keeps each edit interpretable and bounded. This is consistent with prior work

InstructMesh

showing that scoping an LLM to a fixed set of operations improves reliability in human-AI interaction [56]. We provide a set of labeled description-operation pairs drawn from the 20% development split of our formative dataset (see Section 5) directly in the prompt. For example, given the description “the mug has a sealed lid that should be open,” the model predicts erode as the appropriate operation. By including examples spanning all operation types and flaw categories, the model learns to map new user instructions to the fixed operation vocabulary without fine-tuning. We chose ICL over fine-tuning for its simplicity and adaptability: new operations can be incorporated by extending the example set without retraining. While our implementation uses GPT-4, the approach is model-agnostic: any modern LLM model can serve as the backend with the proposed ICL approach. We evaluate prediction accuracy in Section 5.

5

EVALUATING AND CALIBRATING OPERATIONS

In the previous section, we introduced a set of canonical operations for making fine-grained edits to the latent voxel representation prior to mesh regeneration. Each operation has tunable parameters that must be calibrated to reliably rectify the flaws identified in our taxonomy. To this end, we conducted a technical evaluation. We first randomize our dataset, and split it using a 80%-20% split. We used 20% set as a ‘development set’ to calibrate each operation, and use the 80% to evaluate its accuracy in resolving an issue. Using the development set, we manually calibrated each operation (e.g., expand, extrude, erode, fill) by highlighting affected regions and testing parameter ranges for their ability to resolve the target flaw. We found that different flaw contexts required distinct parameterizations. Balancing flexibility against interface complexity and LLM prediction accuracy, we defined twelve calibrated presets across six operations. These presets function as operational modes that can be invoked via natural language or the slider interface; detailed descriptions are provided in Appendix B.

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Figure 7: Success rates of canonical operations in resolving fabrication-relevant geometric flaws. This evaluation was conducted on 80% of flaw instances from our annotated dataset. For each flaw, a predefined canonical operation was applied through region selection and LLM-guided latent editing. time was 31.7s (SD = 3.7s). Applying region-specific latent operations increased this to 43.7s (SD = 5.2s), reflecting the added computation for localized manipulation and regeneration. Despite this overhead, edits remained well under a minute, supporting iterative design workflows. 5.1.3 LLM Accuracy for Operation Selection: Using the ICL setup described in Section 4.2.4, GPT-4 achieved 92.1% accuracy in predicting the correct canonical operation on the 80% test split, demonstrating that in-context learning is sufficient to map natural language instructions to our operation vocabulary without fine-tuning. 5.1.4 Implementation Details: The generative pipeline is implemented in Python and deployed on a cloud-based server (Intel CPU, 32 GB RAM, NVIDIA L4 GPU with 24 GB memory). Canonical latentspace operations were implemented using PyTorch, Pymeshlab [37] and Trimesh [14].

6 5.1

Technical Evaluation

5.1.1 Flaw Correction Accuracy: We tested our calibrated operations on the 80% test split from our formative study. The operations were conducted by the authors with InstructMesh, and the resulting 3D models were stored. To evaluate the repairs, an independent reviewer assessed each flaw and judged whether the correction was successful or unsuccessful. This evaluation was conducted at the flaw level, meaning each flaw was independently repaired, stored, and assessed by the independent reviewer. As shown in Figure 7, InstructMesh successfully repaired a high proportion of flaws across all categories. The most reliably repaired issues included Missing Openings (96.3%) and Wall Thickness Issues (95.65%), while the lowest success rate was observed for Fused / Duplicated Features at 83.33%. These results demonstrate that the latent operations implemented in InstructMesh are broadly effective across a diverse set of fabrication-relevant flaw types. 5.1.2 Runtime Efficiency: We also evaluated runtime performance. Using Trellis for image-to-3D generation, the average generation

USER EVALUATION I: NOVICE FLAW IDENTIFICATION AND REPAIR

For InstructMesh to be effective, novices must be able to (1) identify functional flaws in generated models and (2) repair them independently and efficiently using our system. We designed our first study to evaluate both capabilities.

6.1

Study Design

We recruited 12 participants (7 male, 5 female; ages 20–39, 𝑀 = 27.3, 𝑆𝐷 = 5.6) from a university campus, including undergraduate and graduate students across non-technical disciplines. None had prior experience with 3D modeling, CAD tools, or digital fabrication. Participants worked with five flawed 3D models randomly drawn from our formative study (Fig. 8), each containing between one and four annotated fabrication flaws. For each model, participants first inspected the mesh and identified flaws while thinking aloud. They then used InstructMesh to attempt repairs, with up to 10 minutes per model. Participants could address flaws in any order. After completing their edits, participants judged for each flaw whether the repair was successful or unsuccessful. An independent researcher,

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Figure 8: Overview of the five models used in our user study (M1–M5). (a) Original model. (b) Flawed version from a generative model. (c) Fabrication-related flaws highlighted in pink.

Figure 9: Flaw Identification Performance. (a) Precision and recall of flaw detection across five generated 3D models (M1–M5). (b) Average identification accuracy (left axis) and number of flaw instances (right axis) for each flaw type.

Figure 10: Repair Time and Effort. (a) Mean repair time and number of flaws per model, indicating that models with more flaws required longer repair times. (b) Mean repair time by flaw type. Simpler flaws like topological errors and fused features were repaired faster, while truncated features and missing openings took longer.

post user-study sessions, evaluated the final meshes in the same way. We measured task completion time, the number of flaws identified and repaired, and repair success rates based on both participant and expert judgments.

6.2

Results

We report findings across flaw identification, repair time, and repair success, using repeated-measures ANOVAs to test for effects of model and flaw type.

Faruqi et al.

Figure 11: Repair Success by Flaw Type. Participants successfully repaired most flaw types, with success rates exceeding 80% for all categories. 6.2.1 Error Identification. Participants successfully identified 90.4% of fabrication-relevant flaws in AI-generated models (Fig. 9), showing that novices could reliably recognize most issues. However, performance varied by both model and flaw type. A repeatedmeasures ANOVA revealed a significant effect of model (𝐹 (4, 55) = 3.06, 𝑝 < 0.05); post-hoc tests showed that flaws in Model M1 were significantly easier to detect than those in Model M5 (𝑝 < 0.05). This suggests that flaws in more geometrically complex models can be harder to perceive. Flaw type also had a significant effect (𝐹 (5, 139) = 19.24, 𝑝 < 0.0001). Topological issues were the hardest to detect, performing significantly worse than truncated features, extraneous artifacts, missing openings, and fused features. In contrast, extraneous artifacts and missing openings were among the most readily identified. Together, these results demonstrate that while novices can generally identify fabrication flaws, both the geometry of the model and the type of flaw strongly affect detectability. 6.2.2 Repair Times. Repair efficiency varied significantly across both models and flaw types (Fig. 10). A repeated-measures ANOVA revealed a significant effect of model on repair time (𝐹 (4, 55) = 7.27, 𝑝 < 0.001), with simpler models (e.g., M1) repaired significantly faster than more complex ones (M3, M4, M5). Flaw type also had a significant effect on repair time (𝐹 (5, 121) = 5.43, 𝑝 < 0.001). Truncated features (3.00 ± 0.95 min) took significantly longer to repair than topological errors (1.50 ± 0.52 min, 𝑝 < 0.01) and fused features (1.84 ± 0.75 min, 𝑝 < 0.05); missing openings (2.33 ± 1.24 min) also took longer than topological errors (𝑝 < 0.05). These results suggest that while flaws such as topological errors were harder to identify, they could be resolved quickly once recognized, whereas flaws like truncated features required more time-consuming interventions. 6.2.3 Repair Correction Performance. An independent expert in 3D modeling and fabrication evaluated all user-corrected meshes (Fig. 11). Each flaw was judged as successfully repaired or not, based on whether functionality was restored. Overall, 89.7% of flaws were successfully repaired, with only 16 failures out of 156 annotated flaws. A repeated-measures ANOVA showed no significant effect of model on repair correctness (𝐹 (4, 55) = 2.25, 𝑝 > 0.05), but flaw type had a significant effect (𝐹 (5, 121) = 2.33, 𝑝 < 0.05). Posthoc tests, however, did not reveal significant differences between specific flaw categories. This suggests that while some flaw types

InstructMesh

may be more challenging, InstructMesh enabled novices to produce successful fabrication-relevant repairs across diverse models. We also compared participants’ self-assessments with the expert’s ratings. No significant difference was found (𝐹 (1, 587) = 0.40, 𝑝 > 0.05), indicating close alignment between novices’ perception of successful repair and expert evaluation. Only 11 cases (7.1%) showed disagreement, where participants believed a flaw was fixed but the reviewer did not. Overall, these results suggest that novice users were able to identify and correct functional errors in AI-generated 3D models with a high degree of accuracy.

6.3

Participant Feedback

Participants responded positively to InstructMesh, highlighting its ease of use and low barrier to entry. P2 remarked, “I like that you can undo your changes, and I just have to describe my changes in text. It’s very easy to get used to the tool.” P6 emphasized that “I don’t feel the need to learn CAD software,” while another noted “First time 3D modeling - Didn’t know it could be easy!” Participants envisioned use cases spanning rapid prototyping, creative hobbyist printing, and educational contexts.

7

USER EVALUATION II: INTERACTION MODES AND PREVIEW VISUALIZATION

Our first user study demonstrated that novices can identify and repair fabrication-relevant flaws using InstructMesh. The second study examines how the choice of interaction mode shapes this experience, comparing two conditions: (1) an LLM-based interface where participants describe edits in natural language, and (2) a slider-based interface where participants select operations directly. We also investigate the role of preview visualization in supporting user understanding and clarity during latent-space editing. We address the following research questions: • RQ1 (Usability & Control): How do the LLM and slider interfaces differ in terms of ease of use, effort, clarity of intent expression, perceived accuracy, and sense of control? • RQ2 (Previews): How do latent-space previews influence participants’ understanding of edits, their confidence, and their decision to accept or revise? • RQ3 (Trade-offs): Do participants prefer natural language versus direct operation selection, or a combination?

7.1

Tasks and Procedure

We recruited 12 new participants (7 female, 5 male; compensated 20 USD), none with prior 3D modeling or CAD experience. Each completed 12 editing tasks from the same flawed models used in Study 1, split into two counterbalanced blocks of six: one with the LLM interface and one with sliders. For each task, participants were shown a flawed model and its target corrected version, and asked to rectify the error using the assigned interface. In the LLM condition, participants described desired edits in natural language, while in the slider condition, they selected operations and adjusted parameters directly. In both conditions, a preview visualization was shown before committing the edit.

UIST ’26, November 02–05, 2026, Detroit, MI, USA Strongly Disagree

Disagree

Somewhat Disagree

Agree

Somewhat Agree

Strongly Agree

disagree agree

LLM

2

Sliders

1 1 2

2

0

25%

25%

a

1

7

50%

25%

75%

3

Sliders

1 25%

5

2 0

4 25%

b

50%

“I could express my intent clearly”

LLM

8 3

Sliders 100%

e

75%

6 50%

25%

5 5

0

25%

2

3

3

50%

75%

100%

12

LLM

1

Sliders 25%

f

75%

“I felt in control while editing”

d

0

1

50%

3

1

“Using this interface required a lot of effort”

25%

1

25%

100%

c

2

0

1

5

“The edit was applied accurately”

Sliders

75%

2

LLM

3 4

3

5 1

Sliders

“It was easy to specify the edit”

LLM

2

LLM

2 6

3 0

9 25%

50%

75%

100%

“The preview helped me confirm or change the edit”

Figure 12: Likert results comparing LLM and slider interfaces. (a,c) LLM rated easier for specifying edits. (b,d) Sliders scored higher on accuracy and control. (e) LLMs required less effort. (f) Previews rated very highly across both conditions. After each block, participants completed six 7-point Likert items assessing ease of use, perceived accuracy, clarity of intent, sense of control, effort, and preview usefulness. This evaluation was informed by usability constructs from standardized questionnaires such as SUS [6], adapted to our study-specific tasks. A post-study questionnaire asked participants to compare interfaces and indicate preference for LLMs, sliders, or a hybrid workflow.

7.2

Results

We analyzed the questionnaire data using descriptive statistics and within-subject comparisons between the LLM and slider conditions. Figure 12 shows the results from the analysis. RQ1 (Usability & Control). Both interfaces were rated as usable, with subtle differences. The LLM interface was rated as less effortful (M = 2.67, SD = 1.23 vs. M = 3.25, SD = 1.36), while the slider interface was perceived as slightly more accurate (M = 5.17, SD = 1.19 vs. M = 5.00, SD = 1.21). Perceived control was comparable (both M = 5.58). This suggests a trade-off: LLMs lower the barrier to entry, while sliders provide a stronger sense of precision. RQ2 (Previews). Previews received consistently high ratings across both interfaces. Participants agreed the preview clearly showed what the system would do (LLM: M = 6.67, SD = 0.65; sliders: M = 6.75, SD = 0.45) and helped them decide whether to accept or revise (LLM: all participants rated 7.0; sliders: M = 6.75, SD = 0.45). One participant noted the preview “made invisible operations visible,” suggesting that previews are critical for building trust in non-WYSIWYG editing. RQ3 (Trade-offs). Participants were evenly split in overall preference (6 sliders, 6 LLMs), and all 12 favored a hybrid design combining both modes. Users preferring the slider mode emphasized precision: “I had more control of the edits rather than waiting to see if my intent was clearly understood by the LLM” (P1). Meanwhile, LLM

UIST ’26, November 02–05, 2026, Detroit, MI, USA

advocates highlighted expressiveness: “It was easier to specify the modification I wanted” (P6). Participants described the two modes as complementary, natural language for broad creative edits such as “generating initial sketches and making big fundamental changes” (P4), and sliders for precise local refinements such as “fill in a gap, make a hole” (P2). These findings suggest that novices value the expressiveness of LLMs and the precision of sliders, and see the hybrid combination with previews as the most effective design direction.

8

APPLICATIONS

We demonstrate InstructMesh across six application scenarios covering home decor, medical devices, personal accessories, and robotic enclosures. All examples were constructed with InstructMesh and printed on a Stratasys J55 multi-color 3D printer.

8.1

Home Decor and Functional Objects

Personal fabrication tools are increasingly used to design expressive yet functional home objects. We showcase a mug with a dragonshaped handle, where the generated model included a sealed lid. With InstructMesh, the lid was removed, producing a printable, functional version (Fig. 13a). We also demonstrate an octopusinspired liquid dispenser whose head and legs were sealed. We created a top opening, thickened the surrounding area, and added tunnels through each leg to enable fluid flow (Fig. 13f).

8.2

Medical and Assistive Devices

Personalized fabrication is increasingly important in “Medical Making” [31] and DIY Assistive Technology [7], where combining function and aesthetics improves adoption [49]. We generated “a knee brace in the style of blue denim jeans”. The model achieved the desired aesthetic but lacked functional openings for straps. Using localized edits, we created strap holes to produce a wearable, functional design, ready for printing (Fig. 13c).

8.3

Personal Accessories

InstructMesh also supports the refinement of personalized accessories. We generated “butterfly-shaped eyeglass frames” with ornate wing patterns. The initial model had filled lens regions, a thin bridge, and disconnected temples. Localized edits corrected these flaws into a functional design (Fig. 13d). We also generated “an ornate whistle in the shape of a seashell” that lacked acoustic chambers. Using InstructMesh, we added tunnels and refined the resonant geometry; the fabricated whistle produces a clear tone (Fig. 13b).

8.4

Robotic Enclosures

Finally, we demonstrate InstructMesh’s utility in robotics with a shrimp-shaped enclosure for a bristle bot [69]. We generated a “colorful shrimp with several legs” as an enclosure for a bristle bot housing a DC motor. The model’s front and back legs would cancel forward motion, and it lacked a cavity for electronics. We shortened obstructing legs, created an underbelly cavity, and added a motor tunnel, producing a printable enclosure that enabled directed walking (Fig. 13e).

Faruqi et al.

9

DISCUSSION AND FUTURE WORK

InstructMesh demonstrates how fabrication-aware editing can be integrated into generative 3D workflows through selective highlights and natural language interactions. In this section, we discuss the broader implications of our system design and study results, the current limitations, and future directions.

9.1

Insights and Design Guidelines

In Study 1, novices identified 90.4% of fabrication-relevant flaws and successfully repaired 89.7%, suggesting that post-generation repair can be made accessible through latent-space operations paired with region-based selection. This aligns with prior observations that novice makers can reason about physical properties of 3D models but lack the expert skills with 3D modeling tools to perform the relevant corrective operations [25, 39]. Across flaw types, detectability and repair difficulty were not always correlated (Figures 9, 10), suggesting that future tools could benefit from both automated flaw detection and operation recommendations. In Study 2, the even split between LLM and slider preference (6/6), combined with unanimous preference for a hybrid workflow (12/12), provides empirical support for dual-mode interaction design. From these findings, we distill four design guidelines: • Offer complementary input modes. Natural language lowers the barrier to entry; structured controls provide precision. Supporting both lets users match the mode to the task. • Provide previews for non-WYSIWYG edits. Latent-space operations are inherently opaque. Visual previews make edits tangible, improving transparency and trust. • Bridge intent and operation with parameterization. Calibrated presets make operations reliable while leaving room for adjustment, reducing cognitive load without oversimplifying. • Prioritize near real-time feedback. Edits and previews that complete within seconds are sufficient to maintain iterative workflows and user engagement.

9.2

Heuristics vs. Learned Approaches for Latent-Space Editing

We opted for heuristic-based operational design for several reasons. First, they are interpretable: each operation has a clear geometric meaning (e.g., “remove voxels within a distance threshold”) that can be previewed, parameterized, and explained to users. This interpretability is central to our WYSIWYG design goal. Second, they are composable: simple operations can be sequenced to address complex repairs, as demonstrated by combinations like fill + erode for creating bridges. Third, they require no training data, which is important given the absence of large-scale datasets of fabricationrelevant 3D edits. A learning-based approach, such as training a model to predict voxel modifications from natural language descriptions, could potentially handle a broader range of edits and generalize to novel repair scenarios. However, such an approach could limit the transparency and controllability that our user studies identified as critical for novices. We see the two approaches as complementary: heuristic operations provide a reliable, inspectable foundation, while learned

InstructMesh

UIST ’26, November 02–05, 2026, Detroit, MI, USA

Figure 13: Applications of InstructMesh across domains. (a) Dragon mug with lid removed. (b) A functional whistle with acoustic chambers added. (c) Denim-aesthetic knee brace with strap openings added. (d) Butterfly eyeglasses with hollowed lenses and connected temples. (e) Bristle bot enclosure with electronics cavity. (f) Octopus-inspired liquids dispenser with added fluid channels. components (such as our ICL-based operation prediction) can route user intent to the appropriate operation. Future work could explore hybrid architectures where learned models propose edits that are executed through interpretable geometric primitives. Although our implementation builds on TRELLIS, the approach is generalizable across similar two-stage generative backbones. The canonical operations act on a structured intermediate voxel representation, including recent models such as TRELLIS 2 [58] and SAM3D [9]. Similarly, the ICL-based operation mapping is modelindependent: we validated it with GPT-4, but the same in-context approach can route instructions to operations with any modern LLM. Our formative dataset, drawn from the most popular Thingiverse designs, reflects the contexts in which novices most commonly fabricate objects, and broader fabrication domains are important areas of future work.

9.3

Limitations

We identify several limitations and future directions for this work. We focused our evaluation on a single generative model, Trellis, due to its state-of-the-art reconstruction fidelity and to maintain a standardized evaluation pipeline. We will open-source our implementation to support future analysis across additional generative models as they continue to evolve. The 643 voxel grid bounds the granularity of repairs: a single voxel spans roughly 1.5% of a shape’s extent, which sets the smallest feature an edit can target. The flaws identified in our formative study are meso-scale and span many voxels, making them addressable, as our evaluation demonstrates. However, sub-voxel defects cannot be repaired in the current implementation. This is a limitation of grid resolution rather than of the operations themselves and higher-fidelity backbones can improve the minimum addressable feature size. InstructMesh supports the repair of fabrication-relevant flaws but relies on the user to diagnose them: the system assumes users can visually identify and select a problematic region. Our studies show novices do this reliably for visually apparent flaws (90.4%,

Study 1), but defects such as internal wall-thickness violations or tolerance issues may only surface during slicing or after a failed print. Automatic flaw detection and operation recommendation are natural and exciting future work. Our user studies were designed to evaluate whether novices can identify and rectify visually identifiable fabrication-relevant flaws within a generative workflow. We did not compare InstructMesh with alternative workflows such as iterative re-prompting or manual mesh repair. Such analysis would complement findings from this paper and is an important direction for future work. Similarly, our fabricated examples demonstrate applications rather than systematic printability validation. As scoped in Section 4, InstructMesh targets static geometry and cannot address flaws involving dynamic mechanisms such as hinges or interlocking parts. As these generative systems continue to evolve, future systems can incorporate dynamic simulations to evaluate functionality. Finally, the preview visualization uses red and green overlays, which may pose challenges for colorblind users; future iterations should explore alternative visual encodings such as patterns or shape cues [66].

10

CONCLUSION

In this work, we present InstructMesh, an interactive postgeneration refinement tool that enables novices to perform fabrication-relevant repairs on generative 3D models by mapping region selections and natural language instructions to canonical latent-space operations. A formative study revealed geometric flaws in state-of-the-art generative outputs, motivating a set of composable editing operations that can be triggered via text or slider controls. A preview visualization makes this latent-editing process transparent, supporting user trust and decision-making. Our user studies show that novices can detect and repair visually identifiable flaws in under ten minutes per model, and that hybrid workflows combining natural language with structured controls are preferred. We demonstrate practical applications spanning home decor, assistive devices, personal accessories, and robotics.

UIST ’26, November 02–05, 2026, Detroit, MI, USA

ACKNOWLEDGMENTS We would like to extend our sincere gratitude to the MIT-Google Program for Computing Innovation, the MIT-HPI Collaborative Research Program, and the MIT Felicis Program for their generous support, which made this research possible.

REFERENCES [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [2] Moritz Bächer, Emily Whiting, Bernd Bickel, and Olga Sorkine-Hornung. 2014. Spin-It: Optimizing Moment of Inertia for Spinnable Objects. ACM Trans. Graph. 33, 4, Article 96 (jul 2014), 10 pages. doi:10.1145/2601097.2601157 [3] Alexander Berman, Ketan Thakare, Joshua Howell, Francis Quek, and Jeeeun Kim. 2021. HowDIY: Towards Meta-Design Tools to Support Anyone to 3D Print Anywhere. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 491–503. doi:10.1145/3397481.3450638 [4] Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. 2024. Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement. arXiv preprint arXiv:2408.00653 (2024). [5] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–14. [6] John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7. [7] Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J. Chang, Megan Kelly Hofmann, Amy Hurst, and Shaun K. Kane. 2015. Sharing is Caring: Assistive Technology Designs on Thingiverse. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 525–534. doi:10.1145/2702123.2702525 [8] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. 2020. Learning gradient fields for shape generation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 364–381. [9] Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, et al. 2026. Sam 3d: 3dfy anything in images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7220–7232. [10] Gene Chou, Yuval Bahat, and Felix Heide. 2023. Diffusion-sdf: Conditional generative modeling of signed distance functions. In Proceedings of the IEEE/CVF international conference on computer vision. 2262–2272. [11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113. [12] John Joon Young Chung and Eytan Adar. 2023. Promptpaint: Steering text-toimage generation through paint medium-like interactions. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–17. [13] Blender Online Community. 2018. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam. http://www. blender.org [14] Michael Dawson-Haggerty. 2019. Trimesh. https://github.com/mikedh/trimesh. Computer software. [15] Faraz Faruqi, Amira Abdel-Rahman, Leandra Tejedor, Martin Nisser, Jiaji Li, Vrushank Phadnis, Varun Jampani, Neil Gershenfeld, Megan Hofmann, and Stefanie Mueller. 2025. MechStyle: Augmenting Generative AI with Mechanical Simulation to Create Stylized and Structurally Viable 3D Models. In Proceedings of the ACM Symposium on Computational Fabrication. 1–15. [16] Faraz Faruqi, Kenneth Friedman, Leon Cheng, Michael Wessely, Sriram Subramanian, and Stefanie Mueller. 2021. SliceHub: Augmenting Shared 3D Model Repositories with Slicing Results for 3D Printing. arXiv preprint arXiv:2109.14722 (2021). [17] Faraz Faruqi, Ahmed Katary, Tarik Hasic, Amira Abdel-Rahman, Nayeemur Rahman, Leandra Tejedor, Mackenzie Leake, Megan Hofmann, and Stefanie Mueller. 2023. Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AI. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–13. [18] Faraz Faruqi, Sean J Liu, George Fitzmaurice, and Justin Matejka. 2026. Compos3D: Interactive Part-Based Composition for Creative Control in Generative 3D Models. arXiv preprint arXiv:2607.12193 (2026).

Faruqi et al.

[19] Faraz Faruqi, Maxine Perroni-Scharf, Jaskaran Singh Walia, Yunyi Zhu, Shuyue Feng, Donald Degraen, and Stefanie Mueller. 2025. TactStyle: Generating Tactile Textures with Generative AI for Digital Fabrication. arXiv preprint arXiv:2503.02007 (2025). [20] Faraz Faruqi, Demircan Tas, Arthur Caetano, Niccolò Meniconi, Oğuz Arslan, Misha Sra, Ruofei Du, Stefanie Mueller, and Mustafa Doga Dogan. 2026. MiXR: Harvesting and Recomposing Geometry from Real-World Objects for In-Situ 3D Design. arXiv preprint arXiv:2605.09620 (2026). [21] Faraz Faruqi, Yingtao Tian, Vrushank Phadnis, Varun Jampani, and Stefanie Mueller. 2024. Shaping realities: Enhancing 3D generative AI with fabrication constraints. arXiv preprint arXiv:2404.10142 (2024). [22] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. 2022. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. In Advances In Neural Information Processing Systems. [23] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. 2023. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023). [24] Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M. Rehg, and Varun Jampani. 2025. SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images. arXiv:2501.04689 [cs.CV] https://arxiv.org/abs/2501.04689 [25] Nathaniel Hudson, Celena Alcock, and Parmit K. Chilana. 2016. Understanding Newcomers to 3D Printing: Motivations, Workflows, and Barriers of Casual Makers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 384–396. doi:10.1145/2858036.2858266 [26] Heewoo Jun and Alex Nichol. 2023. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 (2023). [27] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–21. [28] Roman Klokov, Edmond Boyer, and Jakob Verbeek. 2020. Discrete point flow networks for efficient point cloud generation. In European Conference on Computer Vision. Springer, 694–710. [29] Yuki Koyama, Shinjiro Sueda, Emma Steinhardt, Takeo Igarashi, Ariel Shamir, and Wojciech Matusik. 2015. AutoConnect: Computational Design of 3D-Printable Connectors. ACM Trans. Graph. 34, 6, Article 231 (nov 2015), 11 pages. doi:10. 1145/2816795.2818060 [30] Stacey Kuznetsov and Eric Paulos. 2010. Rise of the Expert Amateur: DIY Projects, Communities, and Cultures. In Proceedings of the 6th Nordic Conference on HumanComputer Interaction: Extending Boundaries (NordiCHI ’10). ACM, 295–304. doi:10. 1145/1868914.1868950 [31] Udaya Lakshmi, Megan Hofmann, Stephanie Valencia, Lauren Wilcox, Jennifer Mankoff, and Rosa I. Arriaga. 2019. "Point-of-Care Manufacturing": Maker Perspectives on Digital Fabrication in Medical Practice. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 91 (nov 2019), 23 pages. doi:10.1145/3359193 [32] Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023. 3DALLE: Integrating text-to-image AI in 3D design workflows. In Proceedings of the 2023 ACM designing interactive systems conference. 1955–1977. [33] Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13. [34] Lihua Lu, Ruyang Li, Xiaohui Zhang, Hui Wei, Guoguang Du, and Binqiang Wang. 2024. Advances in text-guided 3D editing: a survey. Artificial Intelligence Review 57, 12 (2024), 321. [35] Meshy AI. 2026. Meshy: 3D AI Toolbox for Generative Mesh and Texture Creation. https://www.meshy.ai/ Version 6.0, Accessed: 2026-03-29. [36] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. 2022. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 306–315. [37] Alessandro Muntoni and Paolo Cignoni. 2021. PyMeshLab. doi:10.5281/zenodo. 4438750 [38] Behnaz Norouzi, Marianne Kinnula, and Netta Iivari. 2021. Making Sense of 3D Modelling and 3D Printing Activities of Young People: A Nexus Analytic Inquiry. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 481, 16 pages. doi:10.1145/3411764.3445139 [39] Lora Oehlberg, Wesley Willett, and Wendy E. Mackay. 2015. Patterns of Physical Design Remixing in Online Maker Communities. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 639–648. doi:10.1145/2702123.2702175 [40] Romain Prévost, Moritz Bächer, Wojciech Jarosz, and Olga Sorkine-Hornung. 2016. Balancing 3D Models with Movable Masses. In Proceedings of the Conference on Vision, Modeling and Visualization (Bayreuth, Germany) (VMV ’16). Eurographics

InstructMesh

Association, Goslar, DEU, 9–16. [41] Romain Prévost, Emily Whiting, Sylvain Lefebvre, and Olga Sorkine-Hornung. 2013. Make It Stand: Balancing Shapes for 3D Fabrication. ACM Trans. Graph. 32, 4, Article 81 (jul 2013), 10 pages. doi:10.1145/2461912.2461957 [42] Mohi Reza, Nathan M Laundry, Ilya Musabirov, Peter Dushniku, Zhi Yuan “Michael” Yu, Kashish Mittal, Tovi Grossman, Michael Liut, Anastasia Kuzminykh, and Joseph Jay Williams. 2024. ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18. [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695. [44] Thijs Jan Roumen, Willi Müller, and Patrick Baudisch. 2018. Grafter: Remixing 3DPrinted Machines. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3173574.3173637 [45] Ryan Schmidt and Matt Ratto. 2013. Design-to-Fabricate: Maker Hardware Requires Maker Software. IEEE Computer Graphics and Applications 33, 6 (2013), 26–34. doi:10.1109/MCG.2013.90 [46] Ryan Schmidt and Karan Singh. 2010. Meshmixer: An Interface for Rapid Mesh Composition. In ACM SIGGRAPH 2010 Talks. Association for Computing Machinery, New York, NY, USA. doi:10.1145/1837026.1837034 [47] Adriana Schulz, Ariel Shamir, David I. W. Levin, Pitchaya Sitthi-amorn, and Wojciech Matusik. 2014. Design and Fabrication by Example. ACM Trans. Graph. 33, 4, Article 62 (jul 2014), 11 pages. doi:10.1145/2601097.2601127 [48] Yulin Shen, Yifei Shen, Jiawen Cheng, Chutian Jiang, Mingming Fan, and Zeyu Wang. 2024. Neural canvas: Supporting scenic design prototyping by integrating 3d sketching and generative AI. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18. [49] Kristen Shinohara, Cynthia L. Bennett, Wanda Pratt, and Jacob O. Wobbrock. 2018. Tenets for Social Accessibility: Towards Humanizing Disabled People in Design. ACM Trans. Access. Comput. 11, 1, Article 6 (mar 2018), 31 pages. doi:10.1145/3178855 [50] Maria Shugrina, Ariel Shamir, and Wojciech Matusik. 2015. Fab Forms: Customizable Objects for Fabrication with Validity and Geometry Caching. ACM Trans. Graph. 34, 4, Article 100 (jul 2015), 12 pages. doi:10.1145/2766994 [51] Evgeny Stemasov, Simon Demharter, Max Rädler, Jan Gugenheimer, and Enrico Rukzio. 2024. PARam: leveraging parametric design in extended reality to support the personalization of artifacts for personal fabrication. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–22. [52] Hariharan Subramonyam, Roy Pea, Christopher Lawrence Pondoc, Maneesh Agrawala, and Colleen Seifert. 2023. Bridging the Gulf of envisioning: Cognitive design challenges in LLM interfaces. arXiv preprint arXiv:2309.14459 (2023). [53] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. 2024. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision. Springer, 1–18. [54] Tom Veuskens, Florian Heller, and Raf Ramakers. 2021. CODA: A Design Assistant to Facilitate Specifying Constraints and Parametric Behavior in CAD Models. In Graphics Interface 2021. https://openreview.net/forum?id=1dLDPJeafRZ [55] Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2024. Promptcharm: Text-to-image generation through multi-modal prompting and refinement. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–21. [56] Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–22. [57] Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2019. Sagnet: Structure-aware generative network for 3d-shape modeling. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14. [58] Jianfeng Xiang, Xiaoxue Chen, Sicheng Xu, Ruicheng Wang, Zelong Lv, Yu Deng, Hongyuan Zhu, Yue Dong, Hao Zhao, Nicholas Jing Yuan, et al. 2026. Native and compact structured latents for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14419–14429. [59] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. 2024. Structured 3D Latents for Scalable and Versatile 3D Generation. arXiv:2412.01506 [cs.CV] https://arxiv.org/abs/ 2412.01506 [60] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. 2024. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191 (2024). [61] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision. 4541–4550.

UIST ’26, November 02–05, 2026, Detroit, MI, USA

[62] J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems. 1–21. [63] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision. 3836–3847. [64] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. 2024. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG) 43, 4 (2024), 1–20. [65] Ran Zhang, Thomas Auzinger, Duygu Ceylan, Wilmot Li, and Bernd Bickel. 2017. Functionality-Aware Retargeting of Mechanisms to 3D Shapes. ACM Trans. Graph. 36, 4, Article 81 (jul 2017), 13 pages. doi:10.1145/3072959.3073710 [66] Zhuohao Zhang, Haichang Li, Chun Meng Yu, Faraz Faruqi, Junan Xie, Gene SH Kim, Mingming Fan, Angus Forbes, Jacob O Wobbrock, Anhong Guo, et al. 2025. A11yShape: AI-Assisted 3-D Modeling for Blind and Low-Vision Programmers. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility. 1–20. [67] Haiming Zhao, Chengkuan Hong, Juncong Lin, Xiaogang Jin, and Weiwei Xu. 2016. Make it swing: Fabricating personalized roly-poly toys. Computer Aided Geometric Design 43 (2016), 226–236. doi:10.1016/j.cagd.2016.02.001 Geometric Modeling and Processing 2016. [68] Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. 2025. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501.12202 (2025). [69] Junyi Zhu, Lotta-Gili Blumberg, Yunyi Zhu, Martin Nisser, Ethan Levi Carlson, Xin Wen, Kevin Shum, Jessica Ayeley Quaye, and Stefanie Mueller. 2020. CurveBoards: Integrating breadboards into physical objects to prototype function in the context of form. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13. [70] Jingyu Zhuang, Di Kang, Yan-Pei Cao, Guanbin Li, Liang Lin, and Ying Shan. 2024. Tip-editor: An accurate 3d editor following both text-prompts and image-prompts. ACM Transactions on Graphics (ToG) 43, 4 (2024), 1–12. [71] Jingyu Zhuang, Chen Wang, Liang Lin, Lingjie Liu, and Guanbin Li. 2023. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 conference papers. 1–10.

Related documents

• InstructMesh: Selective Refinement of Generative 3D Models for Fabrication#792534
• InstructMesh: Selective Refinement of Generative 3D Models for Fabrication#788534
• ProxyBuild: Text-Guided Structured 3D Building Generation with Mesh-Anchored Procedural Proxies#1049128
• Building Expressive and Tractable Probabilistic Generative Models: A Review#819204
• Building Expressive and Tractable Probabilistic Generative Models: A Review#809212