arXiv CSopen access

Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Appearance Pointers Multimodal Region Control of Diffusion Transformers Rahul Sajnani1,2 Yulia Gryaditskaya2 Radomir Mech2 Srinath Sridhar1 Matheus Gadelha2 1 2 Brown University Adobe Research ivl.cs.brown.edu/research/appearance_pointers 1

4 1

arXiv:2607.19344v1 [cs.CV] 21 Jul 2026

4 5

2

2

5

3 BG Prompt: Cinematic Lighting illuminates Ship Deck…..

3

(b) Insertion / Editing

(a) Generation (with Fine & Sparse Control) 1 A small, plump chicken

5

1

in shades of brown and white….

2 3

2 A retro-style arcade machine…..

A sleek … modern car with a smooth, aerodynamic body

3

BG prompt: In classroom…

5

4

4

A boxy van … painted in a deep, matte black.

3

3

4

1

2 2

4

1

(c) Pose Control

(d) Multi-Modal Region Control 1

1

1

2 4 1

ı

insert an engineer with the material properties…..

3

insert a sunglass with the material properties…. “A tall, cylindrical soap dispenser made of …

1

1

2

1 3

4

“… walking in a park…”

“… on a red sofa … In a living room”

2

Fig. 1: Appearance Pointers enable precise, multimodal, region-aware image generation in a single denoising pass. Given spatial masks and heterogeneous conditioning signals — reference images or text descriptions — our region correspondence network produces compact pointer tokens that route a Diffusion Transformer toward the correct appearance cues at the appropriate spatial locations. Our work supports: (a) generation from fine or sparse regional layouts; (b) object insertion with material and style fidelity; (c) pose-conditioned generation; and (d) simultaneous multi-modal region control, combining image- and text-based descriptions in a single scene.

2

R. Sajnani et al. Abstract. Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone. Diffusion Transformers (DiTs) can natively ingest heterogeneous tokens stemming from texts and images, but they lack mechanisms for determining where and how these tokens should influence the output. We introduce appearance pointers, compact tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing token load. Our approach introduces the first modality-agnostic interface for localized multimodal control in a DiT without retraining the base model from scratch. Across a range of metrics, our single model reaches or surpasses the performance of modalityspecific state of the art methods, offering a simple and extensible path toward precise, region-aware, multimodal guidance in generative image synthesis.

1

Introduction

Recent advances in generative modeling have dramatically expanded the creative possibilities of machine-generated imagery. Artists, designers, and filmmakers can now synthesize realistic and diverse scenes by simply providing a text description. Yet, despite their expressive potential, current systems remain difficult to control in practice. Creative professionals often operate with specific visual intentions — precise materials, object layouts, and stylistic details that define the desired outcome. In contrast, text prompting provides only indirect and often unpredictable control, requiring extensive trial and error to achieve the intended composition. This gap between human intention and model controllability limits the integration of generative models into real-world creative pipelines. Diffusion Transformers (DiTs) — now core to many state-of-the-art image generators — offer an appealing pathway toward richer controllability because they can natively ingest heterogeneous token streams, including text embeddings and image tokens. However, they lack a mechanism to determine where and how these modality-specific tokens should influence the generated image. Simply providing more text or image tokens does not convey the spatial intent of the user, nor does it specify which appearance cues should be used in which regions. As a result, structured and localized user intent, expressed through text or reference images, remains difficult to translate into region-aware control. Existing controllable generation approaches attempt to address this through attention manipulation [62, 68, 69], specialized adapters [14, 23, 29, 30, 33, 63, 65], noise initialization strategies [26, 27, 41], or gradient-based guidance during inference [5, 9, 23, 34, 41, 53, 54, 64]. The majority of existing work is developed for older U-Net-based diffusion models and adopts inference-time approaches that

Appearance Pointers

3

are often slow and non-robust. Moreover, most are constrained to a single conditioning modality (e.g., text-only or image-only) and do not seamlessly combine heterogeneous cues [37, 62]. Even with the adoption of recent DiT backbones, these limitations restrict the practical use of diffusion models in tasks requiring multimodal and spatially grounded control.

Vanilla

Appearance Pointers

Generated Image Reference I Reference II Consider the image to the right where reference images were passed to a DiT to generate the image in the left column. Yellow/green overlays on the reference images correspond to tokens being attended by probe tokens in the generated image – represented by yellow/green stars. Top row showcases our method while bottom row shows a vanilla baseline that simply passes reference image tokens to the DiT during generation. Our method attends to the corresponding regions thanks to a new type of token – AppearancePointers. They are compact tokens designed to be ingested directly by a DiT to route the model toward the correct appearance cues at the correct spatial locations provided by the user . Appearance pointers are produced by a small region correspondence network that fuses text or reference images with their associated spatial masks (see small color-coded regions in the top left corner of the figure above). Rather than storing appearance directly, appearance pointers tell the DiT where to use the vanilla image and text tokens provided by the user. This mechanism preserves the architectural flexibility of DiTs while adding a lightweight and modular interface for localized multimodal guidance. We also carefully designed a spatial aggregation mechanism that fuses appearance information from multiple regions, allowing the DiT to handle several regional descriptions – from images or text – together, within a single denoising process. Despite its simplicity, the modular design of AppearancePointers allows it to be applied effectively in a variety of workflows – see Fig. 1 for illustrations and Table 1 for a comparison of the model capabilities with state-of-the-art approaches. More crucially, it extends capabilities of the model to condition a specific region with both image and text simultaneously enabling new applications such as material and text conditioned generation in Fig. 1 (d).

To train and evaluate this model, we created a synthetic dataset with image and text description for multiple elements within the same image. When regions are described only through text, our approach is the best or second best in all the six metrics. When images are used for regional description, our approach surpasses MSDiffusion [49] & DreamRenderer [68] in region adherence and identity preservation. Beyond these quantitative gains, appearance pointers offer a simple and extensible path toward precise, multimodal, region-aware control in generative modeling, enabling users to specify not only what should appear in

4

R. Sajnani et al.

Method InsertAnything [40] MS-Diffusion [49] Sigma-Gen [37] DreamRenderer [68] InstanceDiffusion [50] Seg2Any [62] Ours

Image Text ✓ ✓ ✓ ✓ × × ✓

× × × ✓ ✓ ✓ ✓

Region Control Fine

Sparse

✓ × ✓ × ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ × ✓

Insertion Generation MultiModal ✓ × × × × × ✓

× ✓ ✓ ✓ ✓ ✓ ✓

× × × × × × ✓

Table 1: Comparison of region-conditioned generation methods across key capabilities. Image and Text indicate the supported modalities for regional appearance description. Insertion denotes the ability to place an object into an existing scene, Generation denotes full image synthesis from regional descriptions, and MultiModal indicates whether a method can use both image and text description at the same time for the same region within a single generation pass. This enables novel use cases like the material assignment example in Fig. 2. Unlike all prior works, AppearancePointers supports every capability simultaneously, enabling flexible and spatially grounded multimodal control in a unified framework.

an image, but where and how they should be realized. To sum up, our main contributions are: – We introduce AppearancePointers, a compact representation that routes diffusion transformers to their appropriate regional signals in the form of images, text, and masks. – Our modular and flexible approach extends to multiple edit capabilities including: (a) region controlled generation, (b) insertion, (c) pose control, and (d) multi-modal region-controlled generation for multiple regions in a single denoising process. – We also propose a dataset AppearancePointers-37K containing region text descriptions, appearance images from novel views, and an automatic generation scheme to generate more data.

2

Related Works

For detailed discussions on multimodal controllable diffusion models, we refer the reader to recent surveys [4, 15, 39, 46, 58, 60]. Earlier methods operated on segmentation maps, employing specialized architectures [12, 31, 48] or general conditioning frameworks [23, 29, 30, 33, 63, 65], with some extending to bounding box layouts [20, 57]. There are several relevant capabilities on localized text/image conditioned image generation. One of our main goals while developing AppearancePointers is to enable a variety of workflows with the same framework without resorting to a big and complicated architecture. A summary of the features supported by AppearancePointers and a comparison with state-of-theart methods can be found in Table 1. Below we discuss previous literature on region-based controls for image synthesis.

Appearance Pointers

5

Bounding boxes. Early works explored bounding-box-guided text-to-image generation using object names [25, 54] or descriptive prompts [14, 38], often with a global scene-level prompt [14, 19, 38, 53, 54, 64], though these lack precise shape controllability. Training-free methods incorporate spatial or semantic guidance at inference time via noise initialization [26,27,41], cross-attention manipulation [1, 5, 25, 42, 68], and sometimes self-attention [22, 68, 69]. Taghipour et al. [42] use KL divergence for stronger attribute–region association. An alternative strategy estimates object-specific noises independently and fuses them via cross-attention [38]. Several methods apply gradient-based guidance w.r.t. crossattention alignment [5,41,54], cross- and self-attention consistency [64], or alignment augmented with a boundary-aware loss [53]. Training-based approaches introduce lightweight modules to fine-tune the backbone. GLIGEN [19] injects gated self-attention layers conditioned on grounding tokens. SSMG [14] uses a ControlNet-like branch encoding spatial and semantic features, with attention modeling inter-object relations. LayoutDiffusion [66] proposes a layout encoding and fusion module with cross-attention alignment in an end-to-end diffusion framework. Several works also target auto-regressive generation [10, 67]. While our method supports both coarse regions (like bounding boxes) and also has the capability of following precise regions. Additionally, it can be employed in both image generation and editing workflows. Semantic layouts. Many training-free methods manipulate cross-attention to strengthen concept influence within regions [13, 16], optionally restricting selfattention across segments [16]. Gradient-based guidance updates latents via text–latent cross-attention alignment [9,23,34] and sometimes self-attention [34], image-space losses projected to latent space [2], or a lightweight aligner module [21]. One approach [35] generates objects independently in early denoising steps before merging, while MultiDiffusion [3] optimizes the image at each noise step for global consistency. Training-based methods integrate attention manipulation to fine-tune the base model [51,55,61], or inject text- and mask-conditioned features via convolutional layers [59]. Most closely related, Seg2Any [62] builds on Flux with attention restriction during training and region contour maps for boundary precision, but does not support image references. In contrast, AppearancePointers unifies text and image regional conditioning within the same framework, allowing heterogeneous appearance cues to be combined across regions in a single denoising pass. Reference image guidance. For a comprehensive overview of appearance transfer, we refer to [52]. DreamRenderer [68] focuses on text-based region control and encodes reference images into text embeddings via Redux [17]. AnyDoor [7] uses DINOv2 [32] features injected via cross-attention alongside ControlNet [63]-style conditioning for masks and structure. MimicBrush [6] employs dual U-Nets, injecting reference attention keys and values into an imitative UNet. Insert Anything [40] builds on Flux with reference image, target region, and text conditioning; Ace++ [24] uses channel-wise concatenation but is outperformed by [40]. Unlike our work, these approaches support only a single region at a time, whereas AppearancePointers generates multiple regions (de-

6

R. Sajnani et al. Image Prompts (IPi)

Text Prompts (TPi)

Region Prompts (Ri)

Generated Image

∅

∅

I

I

Snow covered ground…

P1̂

(Section 3.1)

I

R

T

P2

P2

🔥

P2̂ Region Contour

R

T

P3

P3

P3̂

xt xt

Global Prompt (GP)

R

P1

Appearance Pointers

Place a desk with material properties as shown in the image

Generation Tokens (xt)

T

P1

A snowy village …. snow covered hills and trees.

G

P

G

P

I

P3

T

P3

I

P2

T

P2

I

P1

T

P1

E

I

AP

(Section 3.2)

T

AP

FLUX Diffusion Transformer

🔥

LoRA

Fig. 2: Overview of AppearancePointers. Our framework converts heterogeneous inputs (text, images, and spatial masks) into appearance pointers (yellow). After encoding via FLUX VAE and T5 text encoder, multimodal signals from image (I Pi ) and text (T Pi ), are fused with their spatial masks (Ri ) to form Appearance Pointers (I AP, T AP), dictating exactly which features map to specific regions. During sampling, the diffusion transformer receives noisy tokens, text/image tokens, and the appearance pointers as joint inputs to then synthesize a coherent and spatially aligned output that respects all region-specific instructions across modalities. See Sec. 3 for more details.

scribed by images or text) in a single diffusion pass. Furthermore, while these methods entangle conditioning signals globally or process regions sequentially, AppearancePointers routes each appearance cue to its correct spatial location via appearance pointer tokens, enabling precise and modular multimodal control without architectural redundancy.

3

Method

Preliminaries. Recent state-of-the-art image generative models [11, 17, 18] are predominantly based on transformer architectures that jointly process textual and visual information through full self-attention. These models operate on two input modalities: a text stream, which encodes the prompt description, and a visual stream, which encodes conditioning images and generative tokens. A common architectural design involves the use of a dual-transformer scheme, in which text and image tokens are first processed by modality-specific projection layers before being fused through full self-attention. The dual-transformer blocks promote alignment between textual and visual representations, after which a series of single-transformer blocks perform unified multimodal processing. Overview. Our method generates an image I ∈ RH×W ×3 that satisfies both global and regional conditions. The global condition is provided as a text prompt, while the regional conditions are defined as a set of n pairs R = {(Ri , Pi )}ni=1 . Here, each Ri ∈ {0, 1}H×W denotes a region mask, and Pi represents the corresponding local prompt, which may consist of tokens coming from text descriptions, reference images, or both. The model is tasked to generate an image

Appearance Pointers

7

where each region corresponding to Ri follows the content Pi , while the rest of the image aligns with the global text prompt (see Figure 2). While vanilla DiTs possess the ability of ingesting image or text tokens, simply passing additional tokenized information is not enough to inform the models about the localization and composition of the elements in the final image. To this end, we introduce a new type of input token responsible for informing the DiT where and how localized text and image information should be used during generation. We call these appearance pointers. Their role is to ensure correct linking between regions (Ri ) and prompts (Pi ). In practice, as shown in Fig. 2, the appearance pointers serve as a conditional input to the DiT model, alongside global and region-specific information — local image prompts and local text prompts complement our appearance pointers to provide fine-grained appearance details, while region contour maps promote fine-grained alignment with region boundaries. 3.1

Appearance Pointers

We first describe how we construct appearance pointers. To align with our objectives, a good regional representation R should enable multi-modal control while explicitly linking the local prompts Pi to the intended region Ri . To achieve this, we design a Region Correspondence Transformer that links and processes all modalities jointly, before passing them to the DiT generative model. We first describe how each region Ri and its corresponding prompt Pi are encoded, and then explain how these representations are aligned to form the appearance pointers. Region and Condition Encoding. Each regional information (Ri , Pi ) consists of a binary mask Ri and a local prompt Pi , which may be text, an image, or both. Image prompts are encoded with the VAE image encoder providing image tokens I Pi , while text prompts are encoded with the Flux Kontext T5 encoder [18] providing text tokens T Pi . To encode the mask itself, we treat it as an image input to the same VAE encoder. Before encoding, we augment the binary mask with coordinate information to provide explicit spatial grounding. Please see supplement for more details. The coordinate-enhanced mask Mi is then encoded and patchified in the same manner as an image prompt, yielding mask tokens with explicit spatial context. Region-Prompt Linking. After encoding masks, images, and text to the DiT latent space, our next goal is to link masks to their respective condition tokens. This linking enables DiTs to point to the desired condition signal and helps generate region conditioned images (see Sec. 1 for pointer attention visualization). To integrate spatial constraints with semantic content, we introduce a Region Correspondence Transformer, ΦRC to process tokens from the mask R Pi , image I Pi , and text T Pi prompts to produce two semantic feature maps, I Mi and T Mi for each region i: {}^{I}M_i, \; {}^{T}M_i := \mathbf {\Phi }_{RC}([\, \hat {^{R}\mathcal {P}_i}, \; ^{I}\mathcal {P}_i, \; ^{T}\mathcal {P}_i \,]). \label {eq:correspondence_tokens}

(1)

8

R. Sajnani et al.

I

Pi

Region Correspondence (ΦK)

VAE Encoder

I

Mi

🔥

Pî

R

Mask Transformer Image

🔥

VAE Encoder

Correspondence Transformer Mask 🔥 Transformer Text

A vintageinspired air Pi conditioner with a brushed copper finish,

T

Section 3.1: Appearance Pointer (Region-Prompt Linking)

T5 Encoder

T

Mi

Fig. 3: Region-Prompt Linking. To enable the full utilization of the multi-modal conditioning mechanism of the DiT generative backbone, we generate a set of feature maps targeting the image and text streams of the backbone DiT model. Note that either the image or the text prompts can be omitted, but we will always generate feature maps targeting the two conditioning streams of the DiT model. Refer to Sec. 3.1 for details.

These maps serve as modular semantic region representations that guide regionconditioned image generation. Specifically, I Mi and T Mi respectively are targeting the image and text streams of the backbone DiT model, enabling full utilization of its multi-modal conditioning mechanism. Within ΦRC , the region tokens undergo initial alignment via two lightweight mask transformer (2 layer self-attention) blocks aligning mask with the corresponding image and text condition. This mask transformer also undergoes token downsampling analogous to U-DiT [44] for efficiency (see Fig. 3). The processed mask tokens, image, and text tokens are concatenated along the sequence dimension and provided to a multi-modal Correspondence Transformer with self-attention blocks (consistent with the Flux MM-DiT attention). These attention blocks use separate query, key, and value projections for each tokens type. In practice, we also pass learnable tokens to account for missing modalities allowing the module to operate flexibly on image+mask, text+mask, or image+text+mask inputs. Crucially, ΦRC operates on each region independently and is diffusion-step independent. This enables us to run region correspondence only once per generation, significantly reducing overhead. Additionally, this design allows the number of regions to vary dynamically during inference. K = I for Image Pointer Region Aggregation. Learnable Token K = T for Text Pointer M With the regions linked to their corresponding conditions we can condition Region M Aggregation (Φ ) the diffusion transformer to generate region conAP Section 3.1: Appearance Pointer trolled images. However, M (Region Aggregation) naively injecting all region tokens along with their conditional tokens Fig. 4: Appearance pointers generation. The Reincreases the token count gion Aggregation Transformer creates a concise repreK

1

K

2

K

K

K

3

sentation of regional conditions, Appearance Pointers.

Appearance Pointers

9

drastically and is unfeasible with O(N 2 ) complexity for self-attention. Hence, we create Appearance Pointers by consolidating information across regions using a Region Aggregation Transformer into a single canvas producing Appearance Pointers (see Fig. 4). To preserve spatial structure, we perform this aggregation locally at each patch location. We first collect each semantic region embedding T Mi , I Mi ∈ RN ×C from Eq. (1) and stack them along the region dimension to obtain I M, T M ∈ RN ×R×C . Here, N is the number of spatial tokens, R is the number of regions and C is the channel dimension. Next, the region aggregation transformer performs region-wise self-attention for each patch independently, treating R as the sequence length. To consolidate the information, we prepend a learnable [CLS] token (A) to the region sequence at each patch. This "depth-wise" processing collapses the multi-region stack into a single, unified semantic canvas that matches the spatial dimensions of the backbone DiT. In practice, we employ two independent aggregation blocks, ΦTA and ΦIA , to process the text and image streams separately. We extract the updated [CLS] tokens to serve as the final appearance pointers: {}^{T}\mathcal {AP} &:= \Phi ^T_{A}([ {}^{T}A, {}^{T}\mathcal {M}]), \\ {}^{I}\mathcal {AP} &:= \Phi ^{I}_{A}([{}^{I}A, {}^{I}\mathcal {M}]). \label {eq:AP-main} (3) The appearance pointer aggregation module improves identity and coherence resulting better identity preservation and image coherence (Ablation Tab. 4). 3.2

Base DiT Conditioning

Dual Stream Conditioning. We pass the text (T AP) and image (I AP) appearance pointers to the image and text streams of the Flux Kontext model, respectively, to guide the generation process. Additionally, we include the individual local image I Pi and text T Pi prompt tokens. Concatenating all tokens yields the conditions for the image and text streams: X_t &:= [x_t; \, {}^{I}\mathcal {AP}; \, {}^{I}\mathcal {P}_1; \dots ; {}^{I}\mathcal {P}_n] \quad \text {(Image Stream)}\\ c &:= [{}^{G}\mathcal {P}; \, {}^{T}\mathcal {AP}; \, {}^{T}\mathcal {P}_1; \dots ; {}^{T}\mathcal {P}_n] \quad \text {(Text Stream)} (5) where xt are the noisy tokens. The denoising step of the model is then written as: x_{t-1} := \text {FLUX}(X_t, c, t).

(6)

Note that some of the local prompts I Pi and T Pi may be empty. Region Contour Guidance. The appearance pointers effectively link image regions with image and text prompts. However, its token-reduction design of the mask transformer and region aggregation occasionally smooth over finegrained details. To address this limitation, inspired by DreamRenderer [68] and

10

R. Sajnani et al.

Seg2Any [62], we aggregate all edges into a single boundary map, which is encoded using the FLUX VAE and provided to the DiT for improved precision (Ablation Tab. 4). Implementation Details. We train AppearancePointers using a flow-matching objective with log-normal time sampling on 8×A100 GPUs for three days. Following OminiControl [43], we apply LoRA with rank 128 to the newly introduced conditional Appearance Pointer tokens and edge tokens. All learnable parameters are trained using Prodigy [28] with a learning rate of 1.0. Additional training details and pseudo-code are provided in the supplement. Complexity. In terms of memory, our Region Aggregation and Region Correspondence modules consist of ∼400M parameters – a 3.33% increase over the number of parameters of the base model. In terms of time, it is important to highlight that Appearance Pointers are computed once for the whole inference procedure rather than per denoising timestep. Moreover, the design we propose with spatial downsampling and independent region processing reduces the attention computation complexity from O(T · (RNreg )2 ) to O(R(Nreg /k)2 ) , where k is the downsampling factor, R is the number of regions, Nreg is the number of tokens per region, ant T is the number or regions. This ensures negligible overhead relative to the base generation process maintaining high inference efficiency.

4

AppearancePointers-37K Data Generation

Existing personalization and (I) Scene Captioning LLM object-insertion datasets [37, Bowl, Man, Object, material, and Bottle, ... scene descriptions 43,62] lack fine-grained region Subject List Structured Representation captions. We instead gener- (II) Generation and Grounding ate a fine-grained, region-level Flux[dev] Grounding a suburban kitchen, ..., editing dataset enabling con- “Ina round ceramic bowl, ...” trolled variations in object orientation, texture, and ma- (III) Editing and VLM Checking LLM terial, with coverage across Flux Kontext small, medium, and large ob[dev] jects. We generate the dataset in three stages, described be- Data Examples low and visualized in Fig. 5. Scene Captioning. We begin by categorizing everyFig. 5: Overview of the dataset creation day objects into three scale pipeline. (I) We generate diverse scenes by samgroups: small (graspable ob- pling objects of varying scales and prompting an jects), medium (indoor items LLM to produce scene- and object-level descripsuch as beds, tables, and tions. (II) Images are synthesized using Flux.1 Dev chairs), and large (vehicles and grounded with Grounded SAM. (III) For each and infrastructure such as object, we produce pose and material edits and filter cars, buses, cranes, and build- them using a VLM. ings). For each scene, we sample a variable number of objects from these cat{

}

“Extract materials, put in neutral pose, orientation...”

“Subject1”: { Desc: “a white bowl”; Material: “matte white ceramic”; Grounding: “a white floral bowl”; }, “Subject2”: { ... }, ... “Scene”: “In a... kitchen...”;

Consistent change? VLM scoring

Appearance Pointers

11

egories and provide a coarse scene context description (e.g., “kitchen”, “living room”) to an LLM. Using Qwen 3 [56], we generate: (i) a global scene description, and (ii) object-level descriptions that include visual attributes, a material description, and a dedicated description for accurate object grounding. Generation and Grounding. The structured scene prompt is provided to Flux.1 Dev [17] to generate an image. We then segment each described object using Grounded SAM [36], guided by dedicated descriptors. This yields region masks linking each generated region to its text reference. Editing and VLM Checking. Following SIGMA-Gen [37], we extract each grounded object and generate two types of controlled edits: (i) pose perturbations, and (ii) material and texture variations, produced using Flux Kontext. Each edited crop is evaluated by an InternVL [8] vision–language model, verifying whether the edits remain consistent with the original descriptions through VLM-based scoring. Full pipeline details and prompting templates are provided in the supplementary material. Prompts

Input Regions

Ours

Seg2Any

DreamRenderer*

InstanceDiffusion

In a bright, well-lit hospital room, the atmosphere is calm and organized, with soft, natural light streaming through large windows that overlook a peaceful garden. The walls are painted in a soothing pastel green A compact, gasoline-powered lawn mower with a sleek, metallic frame A professional-looking nurse in her mid-30s, wearing a crisp, white uniform with a subtle blue trim The attic is bathed in soft, golden light filtering through dusty windows, casting long shadows across An older man in his late 50s stands near the motorcycle, wearing a blue denim jacket with a faded A vintage 1970s motorcycle sits on a wooden stand in the corner of the attic, its chrome-plated frame gleaming under the soft light The wood grain is visible through a thin layer of grime, and the top drawer is slightly ajar A large, antique filing cabinet stands against one wall, its surface covered in dust and age spots a bright, well-lit warehouse A vintage-style mechanical watch rests on a wooden surface, its face gleaming with a polished brass finish A sleek, modern shampoo bottle stands upright, its surface gleaming under soft ambient lighting. The bottle is cylindrical in shape, with a narrow neck that tapers into a wide base. Its body is made of clear, high-quality plastic that allows the vibrant blue

A compact, rectangular tissue box sits neatly on a shelf, its surface painted in a soft pastel pink hue A large, flat-screen television rests on a solid, dark wood stand The vintage film grain photo captures the heart of a grand, dimly lit hotel lobby A middle-aged man with a weathered face and a confident posture stands near the entrance of the hotel A large, brown raccoon with sleek fur and bright, curious eyes tie is a deep burgundy red with a subtle geometric pattern, and it is neatly knotted small wooden bench coffee cup In a cozy, well-lit coffee shop nestled in a quiet corner of a city street The stand is crafted from a dark, rich mahogany wood, with a smooth, polished finish that gives it a A modern, sleek treadmill sits on a sturdy wooden platform The pole is painted in a dull, muted green that contrasts slightly with the surrounding environment A large, flat-screen television rests on a solid, dark wood stand A classic three-color traffic light stands upright on a small metal pole

Fig. 6: Qualitative comparison on region-conditioned generation from textual descriptions. For each example, the left column shows the user-provided regional text prompts (color-coded for clarity), followed by the corresponding input region masks. We compare against Seg2Any, DreamRenderer* , and InstanceDiffusion, all of which support region-level textual guidance. Across diverse scenes— including indoor environments, object–person interactions, retail shelves, and multiobject arrangements—our results show stronger regional fidelity, more accurate object appearance, and better alignment with both global and local textual cues. DreamRenderer* additionally uses depth information, giving it access to geometric cues not available to other baselines. Despite this, our model produces images with higher semantic correctness and more consistent placement across regions.

12

R. Sajnani et al.

5

Experiments & Results

5.1

Quantitative Results Image Prompts

Input Regions

Ours

MSDiffusion

DreamRenderer* (with Depth)

Dataset. We benchmark our multimodal region correspondence model on our Appearance Pointers-37K (Sec. 4) dataset for region, text, and image conditioned generation. Baselines. As prior works do not support conditioning regions on both text and images, we benchmark AppearancePointers against unimodal regional models: InstanceDiffusion [50], DreamRenderer [68], and Seg2Any Fig. 7: Qualitative comparison on region[62], — for text+region-to- conditioned generation from image prompts. image generation (Tab. 2 Each example provides object reference images and Fig. 6). To evaluate (left) and user-defined region masks (second colimage+region-to-image gen- umn), specifying where each object should appear in the final scene. We compare against MSDiffusion, eration, we also compare Apa recent multi-subject generation method & DreampearancePointers against the Renderer* that has depth supervision. Across varprior state-of-the-art method, ied scenarios, our results show more accurate object MS-Diffusion [49] and Dream- identity and more faithful spatial placement, while Renderer* [68] (Tab. 3 and MSDiffusion & DreamRenderer often alter appearFig. 7). ance or places objects inconsistently. In addition to the described generation setting here, we also evaluate an editing variant of AppearancePointers that preserves background regions of an input image while performing targeted modifications. This allows the model to maintain existing content, providing finer control for region-specific edits. For further implementation details, additional examples and comparison against InsertAnything [40], we refer the reader to the supplementary material. Metrics. Our evaluation benchmark consists of generating 500 images under two conditions: text-only and image-only, both guided by global prompts with an average of 5 regions per image. For the image-only condition, the benchmark additionally includes novel viewpoints of the objects to be inserted into the specified regions. First, we evaluate our results using two global metrics: (i) CLIP-IQA [47], which assesses visual quality by comparing the CLIP embedding of the generated image with a ‘quality’ prompt; (ii) CLIP-T (Global), which measures alignment between the generated image and the global text prompt via the cosine similarity of their CLIP embeddings. We further evaluate region precision

Appearance Pointers

13

Table 2: Quantitative comparison on AppearancePointers-37K (textual region descriptions). Best result in gold, second best in silver. Higher is better for all metrics. Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑ InstanceDiffusion [50] DreamRenderer [68] Seg2Any [62]

86.39 87.75 89.11

27.54 28.03 27.67

35.02 44.20 50.05

41.04 33.84 36.37

26.70 28.43 29.30

93.37 93.24 94.59

Ours

90.40

27.24

56.09

40.35

28.93

95.02

Table 3: Quantitative comparison on AppearancePointers-37K (image region descriptions). Best results in gold. Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑ MSDiffusion [49] DreamRenderer* [68] Ours

89.66 92.08 93.29

24.89 27.37 27.25

45.61 64.20 69.31

28.86 40.11 40.97

31.08 30.02 29.62

74.85 89.56 95.57

using four metrics: (i) DINO-I, which measures fidelity by computing the cosine similarity between the CLS tokens of DINO features from the generated and ground truth image patches; (ii) CLIP-I, analogous to DINO-I but using CLIP image embeddings; (iii) CLIP-T, which assesses alignment between the generated image region and its corresponding region prompt via cosine similarity of CLIP embeddings; (iv) Class-agnostic MIoU [62], computed by prompting SAM2 on the ground truth regions within the generated image to evaluate shape consistency. Analysis. Table 2 and Table 3 compare our method against prior works for generation with text-conditioned regions and image-conditioned regions, respectively. Qualitative comparisons are shown in Fig. 6 and Fig. 7. As shown in Table 2, on text-conditioned region generation, our method achieves the highest scores in global image quality (CLIP-IQA: 95.02), region fidelity (CLIP-I: 90.40), and semantic alignment (DINO-I: 56.09) for text-regionto-image synthesis. We are also competitive in MIoU (40.35), indicating strong adherence to region masks. In contrast, Seg2Any and DreamRenderer often fail to strictly adhere to the specified region masks, which leads to lower MIoU scores despite competitive CLIP-T performance. For the image-region control benchmark (Table 3), our method outperforms both MS-Diffusion and DreamRenderer*, achieving a CLIP-I score of 93.29, MIoU of 40.97, and DINO-I score of 69.31. Our approach demonstrates both higher semantic alignment and improved adherence to specified region boundaries. Please see the supplement for more details and qualitative and quantitative comparison for subject insertion. Please see Sec. E for more experiments, baseline settings, and real-world comparisons on SACap dataset. Ablations: Table 4 presents ablations of our AppearancePointers method for the image-conditioned prompt setting. We use the image-conditioned prompt setting to evaluate the effectiveness of our method in preserving identity. We

14

R. Sajnani et al.

Table 4: Ablation study on image generation using image-conditioned prompts. We evaluate the contribution of each component in the Appearance Pointer model using the image-based regional description setting. Removing the appearance pointer aggregation significantly harms identity preservation (lower CLIP-I and DINOI), while slightly improving MIoU due to the absence of region fusion. Omitting the appearance pointer mask reduces spatial disentanglement, yielding weaker region alignment. Disabling position ID resampling degrades both identity and region consistency, especially in multi-region scenes. Finally, removing region contour guidance leads to substantial drops in MIoU and region fidelity. Together, these results highlight the importance of each component and confirm that the full appearance pointer formulation achieves the strongest balance of regional accuracy, spatial adherence, and identity preservation. Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑ w/o Region Aggregation Sec. 3.1 w/o Region Contour Guidance Sec. 3.2 w/o Position ID Resampling Multi-Region Mimic Sec. A.2 w/o Appearance Pointer Mask Sec. A.3 AppearancePointers (Ours)

89.93 85.57 93.01 93.23 93.29

27.35 23.89 27.2 27.25 27.25

54.47 41.62 67.46 68.93 69.31

41.49 35.81 40.75 41.09 40.97

28.45 28.89 29.47 29.60 29.62

93.48 94.47 95.34 95.83 95.57

first remove the region aggregation (Sec. 3.1), and instead of Appearance Pointers T AP and I AP, directly use T M and I M to condition the DiT. Region Aggregation improves subject identity (from DINO-I 54.47 to 69.31) and image quality (93.48 to 95.57). The Region Contour improves the MIoU from 35.81 to 40.97 improving region precision. See supplement for more detailed ablation and analysis of our work.

6

Conclusion

We introduced appearance pointers, a lightweight and modality-agnostic mechanism that enables Diffusion Transformers to interpret and apply region-specific user intent from text or image inputs. By routing the model toward the correct appearance cues at the correct spatial locations, appearance pointers complement the native multimodal flexibility of DiTs. Our region correspondence network and spatial aggregation module allow multiple regional descriptions to be combined within a single denoising process, supporting scalable and robust spatial control. In order to train and evaluate our model, we created a synthetic dataset with text and image descriptions of multiple regions in generated images. When compared to modality-specific approaches, our model reaches or surpasses the state-of-the-art in most of the metrics, despite being trained for multiple modalities. These findings demonstrate that appearance pointers provide a simple, extensible, and effective interface for precise multimodal guidance, enabling generative models to better realize not only what users intend to create, but where and how the content should appear. Limitations. We observe that our work occasionally overlooks finer regions and does not preserve finer-grained details, like identity of human faces. Moreover, we noticed degradation in the ability to follow regional information when a bigger number of regions is being prescribed; e.g. 10 or more regions. Please see supplement for visual failure cases and its discussion.

Appendix

A Implementation Details & Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Training Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Conditions’ RoPE Positional IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Appearance Pointer Mask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B Multi-Subject Insertion (Editing) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C Ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.1 Appearance Pointer Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E Baseline Comparison Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F Limitation & Discussion with Future Works . . . . . . . . . . . . . . . . . . . . . . . G Appearance Pointer Attention Map Analysis . . . . . . . . . . . . . . . . . . . . . . H Additional Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.1 Dataset VLM Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.2 Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A

Implementation Details & Pseudocode

A.1

Training Setting

15 15 16 16 17 17 18 20 20 28 28 29 30 30 31

We train with up to 7 regions (including background) at each training iteration, where one of the regions can be a background region. During training, we randomly replace the global prompt with a background only global prompt that does not describe the foreground objects 50% of the time. Using background prompt as the global prompt forces region modules to attend more to the region prompts and allows the model to work robustly in scenarios where no global prompt is provided. As described in section 4 in the main manuscript, we perform VLM checking for generated (i) pose variation and (ii) material and texture variation. We use the generated result for training only when InternVL [8] consistency scores exceed 0.84 for pose variation and 0.79 for texture/material variation. The equation below provides the sampling probability p of a region prompt being a subject image from the same viewpoint as in the groundtruth target, a subject image from a novel viewpoint, a material reference image, a text, or their combination:

^{I}\mathcal {P}, ^{T}\mathcal {P} = \begin {cases} p\in [0,0.4], \begin {cases} p\in [0.0,0.2] \text { \& (if consistent novel view)} \\ \;\; \text {novel view image \textbf {(I + T)}} \\ p\in (0.2, 0.4] \text { \& (if consistent material)}\\ \;\; \text {material image \textbf {(I + T)}} \\ \text {else, image crop (no text prompt) \textbf {(I)}} \end {cases} \\ p\in (0.4,0.6], \text {image crop (no text prompt) \textbf {(I)}} \\ p \in (0.6,1.0], \text {text only prompt \textbf {(T)}}, \end {cases} \label {eq:sampling_prob}

(7)

16

R. Sajnani et al.

where (I) denotes image prompt and (T) denotes text prompt with cases of (I+T) denotes both image and text prompts. During training, we center each reference subject in the prompt image and apply a black background mask before passing it to our model. Additionally, we sample coarse masks 30% of the time where the coarse mask can be a bounding box or an ellipse. Additionally, we provide the category name for the material prompt as the material alone does not denote the type of subject to be inserted. A.2

Conditions’ RoPE Positional IDs

Following [37, 43], we assign distinct, non-overlapping positional embeddings to each conditioning image and to the text-prompt tokens. For an image prompt, the positional IDs are defined as [1,\, R(i) \cdot H + h,\, w], and for text tokens as [0,\, R(i) \cdot H + \text {mean}({}^{T}h),\, 0], where R(i) is the integer region identifier and H is the height dimension of the positional-embedding grid for the generated image, and h, w are the row and column coordinates of the token within the region. Here, mean(T h) is the mean height of the region prompt linking tokens (T Mi ) corresponding to the regional text prompt T Pi . The appearance pointer tokens use positional IDs: [1, h, w] for the image appearance pointer, and [0, h, w] for the text appearance pointer. ID Resampling. To mimic a larger number of regions in a small-region training setting, we multiply the region position ID by a random integer multiplier. This allows the model to experience a wider range of positional IDs, which is necessary for scaling up at inference time. With this approach, the image prompt token positional ID becomes [1,\, R(i) \cdot m \cdot H + h,\, w], and the regional text prompt token positional ID becomes [0,\, R(i) \cdot m \cdot H,\, 0], where m is a randomly chosen integer in the range m \in [1,6]. A.3

Appearance Pointer Mask

In the presence of multiple regions, the appearance region pointer aggregation may find it difficult to select the most relevant features. We assist the appearance pointer aggregation by masking regions of interest using the input region mask. Before aggregation all region conditions to obtain the image (I AP) and text (T AP) appearance pointers (defined in the main document), we re-purpose

Appearance Pointers

17

the input regional masks (R P) to ensure non-zero features only in the prompts designated spatial regions. This is implemented as follows: {}^{T}\mathcal {AP} &:= \Phi ^{T}_{A}([{}^{T}A, {}^{T}\mathcal {M} \odot {}^{R}\mathcal {P}]), \\ {}^{I}\mathcal {AP} &:= \Phi ^{I}_{A}([{}^{I}A, {}^{I}\mathcal {M} \odot {}^{R}\mathcal {P}]). (9) The region-linked features T M ∈ RR×C×Ht ×Wt and I M ∈ RR×C×Hi ×Wi , the Region Aggregation Transformer networks ΦIA and ΦTA , and the learnable aggregation tokens I A and T A are defined in the main document. We omit the downsample function used to resize regional masks R P for clarity. ⊙ represents the Hadamard Product.

A.4

Architecture

The region correspondence block is a six layer multimodal transformer. Each transformer block within region correspondence has separate query, key, and value projections for text, image, and mask tokens. We also have a modulation block that takes the region’s clip feature vector to modulate the region similar to attention in FLUX Kontext dev. The tokens are then passed through an MLP with skip connection. Our hidden dimension is 768 and we perform a bottleneck operation that converts from the high dimensional FLUX embedding (D=3072) to 768. This makes our approach lightweight and fast without any perceivable change in performance. The region aggregation block is constructed in a similar way with the only difference that we don’t have modulation and instead skip connections with layer scale [45]. The attention of the region aggregation block is unimodal and much more lighter compared to correspondence block.

B

Multi-Subject Insertion (Editing)

A benefit of our modular region approach is that we can treat the background image and global text prompt as a region condition and seamlessly extend our approach to an editing task that allows multi-subject insertion (see Algorithm 1). We therefore benchmark our method against InsertAnything [40], which supports object insertion within a scene image. Since InsertAnything can only insert one object at a time, we construct a baseline by iteratively inserting objects. This procedure gives InsertAnything a conceptual advantage compared to our method, which inserts all objects simultaneously. Table 5 and Figure 8 show quantitative and qualitative results comparing our method against InsertAnything-based baseline. AppearancePointers outperforms iterative InsertAnything on CLIP-I, CLIP-T, MIoU, and CLIP-IQA, while remaining competitive on DINO-I, despite handling all regions at once, rather than resorting to an iterative approach. Moreover, the quality of the InsertAnything’s generated images deteriorates as the number of insertion operations increases.

18

R. Sajnani et al. Subjects

Edit Insertion

Iterative InsertAnything

Ours

Fig. 8: Multi-Subject Insertion. we perform multi subject insertion with AppearancePointers and benchmark against iterative InsertAnything. InsertAnything tends to degrade image results after repeated subject insertion and deforms objects such as the white bottle in row 1, the table and leaves in row 2.

C

Ablations

Table 6 presents ablations of our AppearancePointers method for the imageconditioned prompt setting. We use the image-conditioned prompt setting to evaluate the effectiveness of our method in preserving identity. w/o Appearance Pointer Aggregation:. We remove the aggregation step, and instead of Appearance Pointers T AP and I AP, directly use T M and I M. For these tokens, we then use the same LoRA fine-tuning approach as for the Appearance Pointers tokens. Without the aggregation step to Appearance Pointers tokens, the identity of the objects is not well preserved, as indicated by the lower DINO-I (54.47) and CLIP-I (89.93) scores, but not performing Appearance Pointer aggregation enables finer region control with higher MIoU (41.49) as the model does not need to learn how to aggregate regions.

Appearance Pointers

19

Table 5: Quantitative comparison on AppearancePointers-27K (Image Region Description with Background). Best result in gold. Higher is better for all metrics. AppearancePointers outperforms Iterative InsertAnything in CLIP-I, CLIP-T, MIoU & CLIP-IQA and performs competitively on DINO-I even with a setting that favors InsertAnything. Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑ Iterative InsertAnything [40] AppearancePointers (Ours)

94.52 94.52

27.46 27.69

76.32 75.11

40.63 45.44

29.74 30.03

85.82 95.47

Table 6: Ablation study on image generation using image-conditioned prompts. We evaluate the contribution of each component in the Appearance Pointer model using the image-based regional description setting. Removing the appearance pointer aggregation significantly harms identity preservation (lower CLIP-I and DINOI), while slightly improving MIoU due to the absence of region fusion. Omitting the appearance pointer mask reduces spatial disentanglement, yielding weaker region alignment. Disabling position ID resampling degrades both identity and region consistency, especially in multi-region scenes. Finally, removing region contour guidance leads to drop in MIoU. Together, these results highlight the importance of each component and confirm that the full appearance pointer formulation achieves the strongest balance of regional accuracy, spatial adherence, and identity preservation. Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑ w/o Region Aggregation Sec. 3.1 w/o Appearance Pointer Mask Sec. A.3 w/o Region Contour Guidance Sec. 3.2 w/o Position ID Resampling Multi-Region Mimic Sec. A.2 AppearancePointers (Ours)

89.93 93.23 85.57 93.01 93.29

27.35 27.25 23.89 27.2 27.25

54.47 68.93 41.62 67.46 69.31

41.49 41.09 35.81 40.75 40.97

28.45 29.60 28.89 29.47 29.62

93.48 95.83 94.47 95.34 95.57

w/o Appearance Pointer Mask:. The masking strategy, described in Sec. A.3, enhances subject identity preservation by ensuring that condition region tokens attend only to tokens within the target regions. This results in a higher MIoU of 69.31, compared to the baseline without the strategy, denoted in the table as w/o Appearance Pointer Mask. w/o Position ID Resampling Multi-Region Mimic:. The position ID sampling described in Sec. A.2, improves identity for larger number of regions and region adherence without which we obtain a lower DINO-I score of 67.46 and a lower MIoU of 40.75 w/o Region Contour Guidance:. Next, we perform an ablation study by removing the Region Contour Guidance. Table 4 shows that Region Contour Guidance improves all region metrics, increasing MIoU from 35.81 to 40.97. Performance vs. Number of Regions: Tab. 7 evaluates performance as the number of regions increases from 1 to 9 for image+region to image task. While identity preservation (CLIP-I) remains remarkably stable, regional DINOI and mIoU show a gradual decline. This is a consequence of increased spatial occlusions and overlapping boundaries in crowded scenes. Robustness to Mask Precision: Tab. 8 analyzes model performance across varying mask granularities (Fine vs. BBox vs. Ellipse). Our results demonstrate high stability in identity preservation (CLIP-I) and aesthetic quality (CLIP-

20

R. Sajnani et al. Table 7: Precision vs. Number of Regions. Metrics / Num Regions 1

CLIP-T CLIP-I ↑ Region DINO-I ↑ MIoU ↑ Global

CLIP-T ↑ CLIP-IQA ↑

2

3

4

5

6

7

8

9

30.50 29.81 29.5 28.96 28.39 28.19 26.99 26.41 25.13 95.07 95.41 95.41 95.51 94.87 95.13 94.84 95.64 94.94 85.22 85.49 85.56 84.03 82.03 80.45 78.58 76.96 74.80 51.19 51.95 51.88 49.13 48.35 48.05 46.33 44.13 43.63 34.87 34.96 34.90 34.98 36.27 36.20 35.19 35.87 35.87 94.62 92.96 93.65 93.97 94.38 95.02 93.07 93.65 93.18

IQA), even when using sparse inputs. The decrease in DINO-I and mIoU for coarse masks reflects the model’s flexibility in synthesizing novel subject orientations that respect the coarse spatial regions (ellipse & bbox). Table 8: Coarse Region Control Ablation.

Ablations

Region

Global

Coarse/Fine Regions CLIP-T CLIP-I DINO-I MIoU CLIP-T CLIP-IQA Blobs (Ellipse) 28.60 Bounding Box (Bbox) 28.79 Fine 28.93

C.1

93.72 94.31 95.13

76.53 80.32 84.03

44.94 34.93 47.86 34.91 53.87 35.00

93.91 93.02 93.51

Appearance Pointer Pseudocode

Please see Algorithm 1 pseudocode for a detailed pseudocode for implementing attention pointer.

D

Qualitative Results

Fig. 9, Fig. 10, Fig. 11, Fig. 12, and Fig. 13 show visuals of using our method with all conditions together, including material, image, and text conditions. We also perform in-the-wild image editing (Fig. 14) and layout-guided generation (Fig. 15) on Google images.

Appearance Pointers

21

Algorithm 1 Appearance Pointer Conditioned Image Generation Require: R P(Region Masks), I P(Region Images), T P(Region Text Prompts) Require: I Pbg (Background Image), timesteps (Generation Timesteps) Require: R(Number of Regions), G P(Global Prompt) Ensure: I := AppearancePointerConditionedGeneration(R P, I P, T P) 1: I Ms := List(), T Ms := List() 2: R P .append(R Pbg ), I P.append(I Pbg ⊙ R Pbg ), T P.append(G P) \triangleright Add Masked Global Image Prompt and Text Prompt for Editing Application 3: for i = 1, 2, . . . R + 1 do I \triangleright Region-Prompt Linking where 4: Mi , T Mi := ΦRC ([ R Pi , I Pi , T Pi ]) Learnable Embedding is Used if Image or Text Prompt is Missing I 5: Ms .append(I Mi ), T Ms .append(T Mi ) 6: end for 7: T M := Concat(T Ms ), I M := Concat(I Ms ) \triangleright If Appearance-Pointer Mask 8: if AppearancePointerMask then T 9: AP := ΦTA ([T A, T M ⊙ R P]) I 10: AP := ΦIA ([I A, I M ⊙ R P]) 11: else T 12: AP := ΦTA ([T A, T M]) I 13: AP := ΦIA ([I A, I M]) 14: end if 15: E := VAE(R P) \triangleright Get Contour Latent 16: xtimesteps := N (0, I) 17: for t = timesteps, timesteps − 1, . . . 0 do 18: Xt := [xt ; I AP; I P1 ; . . . ; I Pn ] (Image Stream) 19: c := [G P; T AP; T P1 ; . . . ; T Pn ] (Text Stream) 20: xt−1 := FLUX(Xt , c, t, E) (Add Contours E to Generative Model) 21: end for 22: I := Decode(x0 ) \triangleright Decode Generated Image 23: return I

22

R. Sajnani et al.

Prompt Condition 1

“insert a wallet with the material properties as described in the following image”

5

“A vintage ceramic bowl sits on a wooden table, its surface etched with intricate floral patterns that have faded slightly over time….”

Image Condition 1

Condition Types

2 3

4

2 3

4

BG Prompt: “In a dimly lit basement, the atmosphere is quiet and subdued, illuminated by a single overhead bulb casting soft, even light across the space….”

2

“insert a stool with the material properties as described in the following image.”

5

1

1

2 4

2 5

“A tall, layered cake with three distinct tiers, each covered in smooth vanilla frosting….”

Generated Image

3

5

4

BG Prompt: “In this cartoonstyle shopping mall scene, the atmosphere is bright and cheerful, with vibrant colors and playful elements that bring the environment to life.…”

1

3

1

2 2

BG Prompt: “In the heart of a quiet hospital room, the Van Gogh Starry Night style paints the scene with swirling, dreamlike brushstrokes of deep blues, vibrant yellows……“

3 3 1

2

“insert a child with the material properties as described in the following image”

3

“The vase is adorned with delicate, hand-painted floral patterns in muted pastel tones”

4

“A small, intricate keychain made of polished brass”

1

2 2

1 4

BG Prompt: “In the heart of a bustling train station, the scene is vibrant yet orderly, with a warm, golden light filtering through large windows that frame the view of a quiet platform outside.…”

3

“insert a desk top with the material properties as described in the following image.”

4

“A sleek, cylindrical water bottle made of high-quality plastic…”

5

“An ancient bronze coin, tarnished with age and time…..” BG Prompt: “The interior of a grand, ancient temple bathes in soft, natural light streaming through tall, arched windows. The walls are lined with intricately carved stone, adorned with faded murals depicting historical events and religious symbols….”

3

1

2

2 3 4

1 5

3

Fig. 9: MultiModal Region Conditioned Generation. AppearancePointers can generate images from heterogeneous condition signals that include materials, images and text prompts. Each region is numbered with the same number as the conditioning signal. The background is generated by the background description. Material conditioned regions are highlighted with bright purple color, image conditioned regions are highlighted with cyan color, and text conditioned regions are highlighted with red color in column 3.

Appearance Pointers

Prompt Condition 1

“insert a engineer with the material properties…..”

2

“insert a sunglass with the material properties as described in the following image”

4

“A tall, cylindrical soap dispenser made of frosted white glass…..”

Image Condition 1

Condition Types

23

Generated Image

2

1

3

3 4

BG Prompt: “In a bright, well-lit office space with large windows that allow natural daylight….” 2

1 3

“A small, dark-colored pigeon perched on a ledge, its feathers glossy and slightly iridescent under the light.…..”

4

“A slender wooden pencil, slightly curved at the tip, with a smooth, polished surface that reflects ambient light…..”

2

3

1 2

BG Prompt: “The scene is set inside a classic school building, rendered in the style of an oil on canvas painting…..”

2

3

“insert a fire engine with the material properties as described in the following image.…. “ “A sleek, modern filing cabinet sits to the left of the fire engine, its surface made of matte black…. “

4

1

2

3

2

BG Prompt: “In this museum exhibition hall, the atmosphere is calm and refined….” 1

Fig. 10: MultiModal Region Conditioned Insertion. AppearancePointers can insert subjects from heterogeneous condition signals that include materials, images and text prompts. Each region is numbered with the same number as the conditioning signal. The background is unchanged for the editing scenario. Material conditioned regions are highlighted with bright purple color, image conditioned regions are highlighted with cyan color, and text conditioned regions are highlighted with red color in column 3.

24

R. Sajnani et al. Conditions Handcrafted ring with a central diamond. Platinum band with brushed finish… 1

3

4

Generated Image 2 4

BG Prompt: Cozy rustic restaurant with warm lighting and elegant decor. A wooden table with a farmer in a denim shirt and leather apron… 2

3 1

A vintage alarm clock on a table. The clock has a glass dial and metal face 1

3

2

BG Prompt: A train station with bright, even lighting and clean, polished floors. Posters and advertisements cover walls in bold, colorful designs. 2

a tourist with the material properties 1

2

3 1

1

3

6

metallic keys with engravings on a wooden… 2

1

A fork with material properties as shown. 4 Tall glass bottle filled with thick red sauce… 5

3 5

4

4

BG Prompt: A basement with neutral beige walls and a large window… 6 A washing machine with digital interface….. 1 Wooden bench with weathered edges…

3

4

6 4

3

2

5

2

6

1

5

2

insert a pan with the material properties.. 1

1

5

4 2

Man in black leather jacket. Sharp jawline. Furrowed brow…. 2 Navy-blue backpack 3 made of nylon BG Prompt: Dimly lit movie theater lobby with faded film posters on walls. Ornate chandelier casts….. 4

Modern camera with polished metal body…. 1

5 6

3 1

6

3

4

6 3

White crib against wall with firm mattress… 2

5 4 5

6

2 1

Fig. 11: Image Generation and Editing Gallery AppearancePointers can perform regional edits and generation using heterogeneous conditions in the wild. Each region is accompanied by a number which relates the condition to the region.

Appearance Pointers Conditions A sleek, modern navyblue sofa with a minimalist design…. 1

3

25

Generated Image 3

A classic, tufted sofa with a rich, deep burgundy velvet upholstery. 2

1

BG Prompt: The skatepark is spacious and well-lit, with smooth concrete surfaces and various ramps and rails for skating…. 1

2

3

2

1

2 4 3

4

A grand, circular fountain with intricate… 1

4

5

A large, cylindrical wooden barrel with a wide opening at the top and a handle on one side. 2 A sleek, white sailboat with a triangular sail billowing in the wind. 3

A simple, dark green sturdy wheelbarrow… 1

4 3

BG Prompt: A serene way stretches across a vast, open landscape…. 5

1

2

3

2

4

5 BG Prompt: The playground is spacious and well-lit, with a variety of outdoor equipment scattered across the ground. 2

5

A stethoscope around the neck 1

4

A tall, middle-aged doctor with a confident posture, wearing a crisp, navy-blue buttondown shirt with subtle vertical stripes…. 2

4

3 1

5

4

1 6

2

5

BG Prompt: A modern factory with concrete floor…. 3

6 An acoustic guitar is being played by a man 1

3

4

5 4 3

Tall glass bottle on wooden table with red liquid inside. 2

5

1

2

Fig. 12: Image Generation and Editing Gallery AppearancePointers can perform regional edits and generation using heterogeneous conditions in the wild. Each region is accompanied by a number which relates the condition to the region.

26

R. Sajnani et al. Conditions A large, wheeled shopping cart with a bright red frame and a flat, steel-gray basket. … 1

3

4

Generated Image 5 2 4

A tall, middle-aged man wearing a navy-blue blazer….. 2

1

3

BG prompt:A modern airport terminal with sleek, white-tiled floors, large glass windows.. 5

2

1

3 1

Insert the given sculpture stone 1

2

BG prompt: A garden scene on a bright sunny day. 3

Cardboard tissue box with curled corners. Faded label…. 1

4

5

Elderly woman with gray hair in a bun. Floralpatterned dress. Calm expression... 2 Steel filing cabinet with dust and scratches….. 3 Sailboat with weathered wooden hull. Tattered sails show wear... 4

3

6

3

1

BG prompt: The boat deck is open and spacious, with a wooden floor and a clear view of the surrounding water. Two air conditioners are placed on either side of the deck, one modern and one vintage, both operating quietly and efficiently. 1

2

A slightly older office building with a brick exterior that has a warm, earthy tone. 1

2

5

3

2

3

BG prompt: Two office buildings in a snowy area …… 3

BG prompt: In a warehouse with boxes in the back…… 1

2

1

BG Prompt: A hospital room with a sailboat on a wooden dock…. 6

1

2

2

3

1

4

4

5 2

5

3

Fig. 13: Image Generation and Editing Gallery AppearancePointers can perform regional edits and generation using heterogeneous conditions in the wild. Each region is accompanied by a number which relates the condition to the region.

Appearance Pointers

Input Image

Condition Segments

Condition Edits 1

27

Generated Image

1

2

“A bright yellow pillow”

3

“A bright red pillow”

1

1

2

3

2

“Add tapestry on the floor with this material ” 2

1

Fig. 14: Image Editing with Images and Region Prompts. AppearancePointers can perform one-shot regional edits on images from the internet.

Input Image

Condition Edits

Condition Segments

1

Generated Image

2 1

2

“A bird house”

3

“A green parrot with red beak”

4

“A bright orange plate”

5

BG Prompt: A table with plants in the background and brown wall

5

3

4

Fig. 15: Layout Guided Generation. AppearancePointers can perform layout guided image generation. In this example, an image segmentation model is used to identify region of an input image. Then, we select a subset of these regions and generate a new image following the same layout. Region 1 is described by a new subject image, while the other regions have text descriptions.

28

E

R. Sajnani et al.

Baseline Comparison Setting

Following Seg2Any, MS-Diffusion, and InstanceDiffusion, we evaluate baselines using their original optimized weights. Retraining these models on our Appearance Pointers-37K dataset is avoided for the following reasons: (1) Metric Neutrality: Our benchmark utilizes distribution-neutral metrics (mIoU, DINO-I, CLIP-I, CLIP-T, CLIP-IQA) to measure spatial+text adherence, identity preservation and aesthetics. We explicitly exclude distributionsensitive metrics (FID/KID) to minimize inherent evaluation advantage. We do recognize some advantage distribution shift advantage might still and ran experiments on real data below. (2) Large-Scale Training: Prior works such as Seg2Any and MS-Diffusion train at large scale with millions of data samples. Retraining them risks creating weak baselines and is avoided by all prior works as well. Notably, our method utilizes the same model for both image- and text-based regional descriptions — a significantly more difficult setting that is not supported by prior works. However, we also show real-world results below and benchmark against Seg2Any & MS-Diffusion on Real-World generation. Table 9: Quantitative comparison on SACap-eval dataset (real-world images). Our work is competitive even when trained entirely on synthetic data with better aesthetics and image-text adherance (Zero-Shot setting). Minimal fine-tuning with only 2.3% of real-data improves region text adherence significantly. AppearancePointer is the only model that supports both image and text as regional guidance with the same weights (highlighted in red). Best results in gold, second best in silver. Training Data

Region Global CLIP-I↑ CLIP-T↑ DINO-I↑ MIoU↑ CLIP-T↑ CLIP-IQA↑

Task: Region+Text to Image Seg2Any Ours (Zero-Shot) Ours (2.3% Fine-Tune)

SACap-1M AP-37K AP-37K + SACap-23K

× × ×

26.24 25.57 25.78

× × ×

64.24 60.14 61.72

32.30 32.17 32.99

86.07 89.27 87.92

Private-3M AP-37K AP-37K

87.81 93.92 94.09

× × ×

46.86 72.16 72.63

53.53 59.64 62.46

31.95 32.78 33.02

74.91 86.23 86.27

Task: Region+Image to Image MS-Diffusion (Bbox) Ours (Zero-Shot, Bbox) Ours (Zero-Shot, Fine-Grained)

Real Data Quantitative Results: We evaluate our method on the real-world SACap-eval dataset in both zero-shot and fine-tuned settings (Tab. 9). To ensure a fair comparison with Seg2Any, which applies face blurring, we exclude human subjects from the evaluation and randomly sample 200 images for benchmarking. Our results demonstrate strong distribution transfer and effective performance with marginal fine-tuning data. See Fig. 16 (c).

F

Limitation & Discussion with Future Works

First, our method inherits the FLUX Kontext model’s difficulty in preserving intricate human facial identities, a limitation that likely reflects the constraints of the FLUX training distribution. Empirically, we observe that the model preserves identity more reliably when faces occupy larger spatial regions rather

Appearance Pointers

29

Fig. 16: (a) Precise Easy Segmentation in UI (clicks visualized in red). (b) Obtaining precise regions from other images using a single click and using them as conditions for generating/editing images. (c) Real-world regions+text to image generation zero-shot with AppearancePointers on SACap-eval. (d) Sparse Bbox subject conditioned image generation from MS-Bench.

than smaller ones. Second, because our dataset verification relies on large-scale VLM-based filtering, residual errors in the verification process may persist and potentially degrade appearance-transfer performance. Finally, conditioning on a large set of regions (about 10) can lead to leaking artefacts or degrade regions, as shown in Fig. 17. Our work AppearancePointers introduces a method that allows Pointing text/image/edit conditions to the desired region and shows competitive or better performance compared to unimodal prior works. A future work for our generic design could be to extend our work to other forms of conditioning modalities, for example, audio, video, 3D, and 4D. Another possible direction is improving the efficiency of the model to make it real-time for editing and scene control applications.

G

Appearance Pointer Attention Map Analysis Generated Image

We provide the attention map visualization in Figure 18 to understand what the pointer tokens are attending to during generation. The yellow queried points on the generated image and segments are queried to obtain the attention map reflecting the regions responsible for conditioning the segment. In each case, our appearance pointer highlights the region of correct region of interest. For example, the pattern on the chair, the right foot of the bench, the lamp, and the crane body each point to the corresponding conditioning signal.

Region Masks

Appearance Pointers Attention Visualization

Fig. 18: Appearance Pointer Points to Correct Regions of Interest displaying its efficacy in multimodal region-controlled generation.

30

R. Sajnani et al.

Region Segments 2

1

5 3

Region Conditions

9

4

1

A cute white bird is ying in the sky

2

A bright yellow airplane ying over the sky

3

A bright green bike waiting near the red car

4

A bright vibrant red car resting near the river

5

A tall man standing near the river

6

An orange toy speed boat

7

A dark brown table

8

A pink vase

9

A dark purple violet sofa

8

7 6

Generations

10

BG Prompt: … Near river with a city background… fl

10

BG Prompt: … Near river with a cartoonish city background…

10

BG Prompt: … Near river in an outdoor shopping center…

fl

Fig. 17: Failure Cases. We test scaling the number of regions using text prompt for AppearancePointers. Increasing number of regions upto 10 in this case can degrade region adherence and quality. Here, the sofa is not captured accurately.

H

Additional Dataset Details

H.1

Dataset VLM Check

Edits by FLUX Kontext may not necessarily be accurate and can have inconsistencies that can result in an erroneous signal for our model. Hence, we additionally use InternVL 3 VLM to rate the quality of the edits and prompt it to provide a summary of the reason for the rating (see examples in Listing 1.1 & Listing 1.2). Novel View Verification Listing 1.4 displays an example of prompting the InternVL [8] model to score the quality of the novel view generation of the subject of interest. Listing 1.1 displays the result obtained from the model to rate the overall consistency of the generated novel view.

Listing 1.1: Example Novel Pose Verification JSON {

Appearance Pointers

31

"overall_consistency_score": 0.8, "is_geometrically_consistent": true, "sharpness_comparison_score": 0, "is_novel_pose": false, "overall_consistency_reason": "Good Match (Clear resemblance, but minor, subtle inconsistencies in material or lighting are noticeable).", "geometric_consistency_reason": "N/A", "sharpness_comparison_reason": "N/A", "pose_novelty_reason": "N/A" }

Material/Texture Verification Listing 1.5 displays an example of prompting the InternVL [8] to score the quality of the material/texture sphere generated by FLUX-Kontext for the subject of interest. Listing 1.2 displays the result obtained from the InternVL model to rate the quality of the generated material with its reasoning. Listing 1.2: Example Material Verification JSON { "perceptual_similarity_score": 0.7, "albedo_color_fidelity_score": 0.8, "roughness_gloss_fidelity_score": 0.6, "texture_detail_fidelity_score": 0.8, "albedo_color_reason": "Sphere’s color is slightly lighter than the object.", "roughness_gloss_reason": "Sphere appears smoother than the object’s material.", "texture_detail_reason": "Sphere’s pattern is similar but lacks ....", "perceptual_consistency_reason": "Good match but some differences in texture depth and glossiness." }

H.2

Prompting

Here, we detail the additional details that we used to prompt LLM. Listing 1.3 displays the JSON prompt provided to Qwen 3 [56] to generate object descriptions, material descriptions, Grounding DINO captions, and more. In our work, we found succinct Grounding DINO captions necessary to improve ground score and succinct material descriptions to prompt FLUX Kontext [18] to improve material/texture sphere generation. Listing 1.3: Example for Generating Prompts and Object Descriptions for our Dataset { "task": "expert scene captioning and material extraction for 3D generation", "style": "{style}", "environment": "{environments}", "required_object": "{prompt_obj_string}", "people_details": "{people_detail_description}", "constraints": { "output_format": "JSON object", "total_word_count_min": 1000, "total_word_count_max": 3000, "scene_caption_word_count": 300, "image_quality": "avoid dark, low-resolution, or overly complex scenes", "visual_distinctiveness_rule": "ALL objects generated in the scene must be visually differentiable. Ensure colors, primary textures, and unique features DO NOT overlap between objects. For example, if Object A is ’smooth, dark mahogany,’ Object B must not be ’smooth, dark walnut.’",

32

R. Sajnani et al. "material_specificity_rule": "Material descriptions must be vivid, focusing on sensory qualities: the degree of shine, the feel of the texture, and how light interacts with the surface. Make sure you mention description of the dominant color and any texture elements. Avoid technical 3D terms." }, "instructions": [{ "step": 1, "name": "detailed_object_captions", "goal": "Generate a detailed, purely visual description (max 100 words per object) for each object listed in ’required_objects’. These descriptions must be entirely isolated: contain NO environmental context, NO relation to neighboring objects, and NO scene placement. Focus only on intrinsic visual attributes.", "subtasks": ["For the ’caption’ field, detail color, shape, and intricate surface patterns.", "For the ’dino_caption’ field, generate a succinct, precise phrase (under 10 words) focusing on the object’s function or unique geometry to maximize detection accuracy, avoiding reliance on simple color.", "**If the object is a person, the ’material_caption’ must describe the material and texture of their most prominent article of clothing (e.g ., shirt, jacket, or pants), detailing its weave, reflectivity, and drape.**. For all other objects, provide a rich, sensory and perceptual description of the object’s material properties, describing texture, reflectivity, and light interaction in the material caption with the **single** most prominent article of the material along with its color (under 10 words). DO NOT describe multiple materials or colors." ]}, { "step": 2, "name": "scene_caption", "goal": "Generate a final, vivid scene caption by seamlessly combining all objects from Step 1. Focus on **scene-level details** (placement, lighting, composition, object interaction) within the specified **environment** and **style**. Ensure all objects are in sharp focus and not part of a blurred background. The caption must be upto **300 words** long." } ], "output_structure": { "detailed_object_captions": { "object_1": { "name": "<object_1_name>", "caption": "<detailed object 1 visual caption>", "dino_caption": "<succinct dino caption for object 1 for better detection>", "material_caption": "<detailed visual and textural description of the object’s material>" }, "object_2": { "name": "<object_2_name>", "caption": "<detailed object 2 visual caption>", "dino_caption": "<succinct dino caption for object 2 for better detection>", "material_caption": "<detailed visual and textural description of the object’s material>" }, }, "scene_caption": "<scene caption, upto 300 words long>", "scene_background_caption": "<scene background caption, upto 300 words long>" }

}

Listing 1.4: Example for Rating the Quality of Novel View Pose { "task": "Expert simplified 3D view consistency and critical artifact analysis",

Appearance Pointers

33

"goal": "Given two images of the same object from different viewpoints, provide a single overall consistency score and three critical binary/categorical checks (Geometric, Sharpness, Pose). The ONLY output must be a single, valid JSON object that adheres strictly to the defined output_structure, with absolutely no preamble, explanation, or concluding text.", "prompt": "Analyze the two object views. Assume both images are renders of the *same 3D model* and *same material*. **CRITICAL RULE: IGNORE ALL PIXELATION AND BLOCKING ARTIFACTS** when determining scores. On a scale from 0.0 (inconsistent) to 1.0 ( perfectly consistent), score the **overall visual consistency**, encompassing perceptual match, material realism, and texture detail. **GEOMETRIC CHECK:** Set ** is_geometrically_consistent** to **true** if the 3D shape/scale is plausible and consistent, and **false** if there are significant errors. Provide an **integer score** for the sharpness comparison based on the rule: **0** if SAME SHARPNESS; **-1** if View 1 is MORE BLURRY; and **+1** if View 2 is MORE BLURRY. Additionally, set **is_novel_pose** to **true** if View 1 and View 2 have a DIFFERENT POSE or ORIENTATION, and **false** otherwise. Provide a brief, one-sentence justification for **every output**. **MAKE SURE YOU STICK TO THE RUBRIK AND FOLLOW IT EXACTLY**. **IGNORE THE WHITE BACKGROUND IN BOTH IMAGES. JUST FOCUS ON THE OBJECT ITSELF**.", "scoring_rubric": { "overall_consistency_score": { "0.9-1.0": "Excellent Match (Nearly indistinguishable; excellent consistency across all lighting, texture, and perceptual aspects, ignoring pixelation).", "0.7-0.89": "Good Match (Clear resemblance, but minor, subtle inconsistencies in material or lighting are noticeable).", "0.4-0.69": "Fair Match (Obvious inconsistencies in one or more areas, such as color shift or smeared textures).", "0.0-0.39": "Poor Match (Gross inconsistencies; views look like different objects or materials)." }, "is_geometrically_consistent": { "true": "3D shape, scale, and placement are plausible and consistent (minor artifacts acceptable).", "false": "Significant geometric errors are present (distortion, object shifting, hole-filling errors, or wrong scale)." }, "sharpness_comparison_score": { "+1": "View 2 is noticeably softer, has less fine detail, or appears more out-offocus than View 1 (View 2 is the blurrier image).", "0": "The perceptual sharpness and fine detail levels are visually identical between View 1 and View 2.", "-1": "View 1 is noticeably softer, has less fine detail, or appears more out-offocus than View 2 (View 1 is the blurrier image)." }, "is_novel_pose": { "true": "**NOVEL POSE**: View 2 is rotated/transformed and is different from view 1 (difference is easily visible).", "false": "**NO NOVELTY IN POSE**: View 2 is the same orientation as view 1 ( difference is minimal or invisible)." } }, "output_structure": { "overall_consistency_score": "float (0.0 to 1.0)", "is_geometrically_consistent": "bool (one of: true, or false)", "sharpness_comparison_score": "integer (one of: -1, 0, or 1)", "is_novel_pose": "bool (one of: true, or false)", "overall_consistency_reason": "string (e.g., ’Low score due to clear color shift and texture smearing, NOT pixelation’ or ’N/A’)", "geometric_consistency_reason": "string (e.g., ’Significant geometric distortion is visible in View 2’ or ’N/A’)", "sharpness_comparison_reason": "string (e.g., ’Score is +1 because fine edges are lost in View 2’ or ’N/A’)", "pose_novelty_reason": "string (e.g., ’Pose is novel because it is rotated 30 degrees around up-down axis’ or ’N/A’)" } }

34

R. Sajnani et al.

Listing 1.5: Example for Rating the Material/Texture Quality of the Generated Edit { "task": "expert graphics 3D shape and material understanding", "goal": "numerically score the fidelity of three distinct PBR attributes between the object and sphere images.", "prompt": "Analyze the two images (segmented object and sphere). On a scale from 0.0 to 1.0, score the degree to which the material properties and texture on the sphere accurately represent those of the object. Score the **Albedo/Color**, **Roughness/ Gloss**, **Perceptual Match** and **Texture/Pattern Detail** independently. Output the three scores as floating-point numbers. **MAKE SURE YOU STICK TO THE RUBRIK AND FOLLOW IT EXACTLY**. **IGNORE THE WHITE BACKGROUND IN BOTH IMAGES. JUST FOCUS ON THE OBJECT AND SPHERE ITSELF**.", "scoring_rubric": { "perceptual_similarity": { "0.9-1.0": "Excellent Match (Visually indistinguishable; the sphere perfectly captures the ’feel’ and realism of the object’s material).", "0.7-0.89": "Good Match (The sphere clearly represents the intended material, but a layman could spot minor flaws like poor lighting interaction or blurring)." , "0.4-0.69": "Fair Match (The material looks synthetic, plastic, or simply ’off’ compared to the real object, severely impacting believability).", "0.0-0.39": "Poor Match (The material is unrecognizable or looks like a gross simplification; fails the basic realism test)." }, "albedo_color": { "0.9-1.0": "Excellent Match (Nearly identical color, saturation, and lightness).", "0.7-0.89": "Good Match (Correct hue, but lightness or saturation is clearly wrong /off).", "0.4-0.69": "Fair Match (Significant color shift; related but incorrect hue).", "0.0-0.39": "Poor Match (Fundamentally different color)." }, "roughness_gloss": { "0.9-1.0": "Excellent Match (Highlights and surface diffusion perfectly match observed shininess/dullness).", "0.7-0.89": "Good Match (Generally correct, but highlights are slightly too sharp or too broad).", "0.4-0.69": "Fair Match (Significant misrepresentation of surface type, e.g., shiny plastic rendered as dull cloth).", "0.0-0.39": "Poor Match (Completely inverted property)." }, "pattern_detail": { "0.9-1.0": "Excellent Match (Texture pattern, scale, and orientation are identical ).", "0.7-0.89": "Good Match (Correct pattern used, but slightly tiled, scaled, or rotated incorrectly).", "0.4-0.69": "Fair Match (Right *type* of texture, but the *specific pattern* is different or map resolution is poor).", "0.0-0.39": "Poor Match (Completely different pattern applied)." } }, "output_structure": { "perceptual_similarity_score": "float (0.0 to 1.0)", "albedo_color_fidelity_score": "float (0.0 to 1.0)", "roughness_gloss_fidelity_score": "float (0.0 to 1.0)", "texture_detail_fidelity_score": "float (0.0 to 1.0)", "albedo_color_reason": "string (e.g., ’Sphere’s color is too light’ or ’N/A’)", "roughness_gloss_reason": "string (e.g., ’Sphere is too shiny’ or ’N/A’)", "texture_detail_reason": "string (e.g., ’Sphere’s pattern is different than object as it contains....’ or ’N/A’)", "perceptual_consistency_reason": "string (e.g., ’Low perceptual similarity because ..... ’ or ’N/A’)" } }

Appearance Pointers

35

References 1. Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., et al.: ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022) 2. Bansal, A., Chu, H.M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., Goldstein, T.: Universal guidance for diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 843–852 (2023) 3. Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: Multidiffusion: Fusing diffusion paths for controlled image generation. Proceedings of Machine Learning Research (2023) 4. Cao, P., Zhou, F., Song, Q., Yang, L.: Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279 (2024) 5. Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance. In: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE (2024) 6. Chen, X., Feng, Y., Chen, M., Wang, Y., Zhang, S., Liu, Y., Shen, Y., Zhao, H.: Zero-shot image editing with reference imitation. Advances in Neural Information Processing Systems 37 (2024) 7. Chen, X., Huang, L., Liu, Y., Shen, Y., Zhao, D., Zhao, H.: Anydoor: Zero-shot object-level image customization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6593–6602 (2024) 8. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198 (2024) 9. Couairon, G., Careil, M., Cord, M., Lathuiliere, S., Verbeek, J.: Zero-shot spatial layout conditioning for text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023) 10. Cui, X., Sun, Q., Wang, M., Li, L., Zhou, W., Li, H.: Layoutenc: Leveraging enhanced layout representations for transformer-based complex scene synthesis. ACM Transactions on Multimedia Computing, Communications and Applications 21(4), 1–21 (2025) 11. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Proceedings of the 41st International Conference on Machine Learning (2024), https://proceedings.mlr.press/v235/esser24a.html 12. Ham, C., Hays, J., Lu, J., Singh, K.K., Zhang, Z., Hinz, T.: Modulating pretrained diffusion models for multimodal image synthesis. In: ACM SIGGRAPH 2023 Conference Proceedings (2023) 13. He, Y., Salakhutdinov, R., Kolter, J.Z.: Localized text-to-image generation for free via cross attention control. arXiv preprint arXiv:2306.14636 (2023) 14. Jia, C., Luo, M., Dang, Z., Dai, G., Chang, X., Wang, M., Wang, J.: Ssmg: Spatialsemantic map guided diffusion model for free-form layout-to-image generation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38 (2024) 15. Jiang, R., Zheng, G.C., Li, T., Yang, T.R., Wang, J.D., Li, X.: A survey of multimodal controllable diffusion models. Journal of Computer Science and Technology 39(3), 509–541 (2024) 16. Kim, Y., Lee, J., Kim, J.H., Ha, J.W., Zhu, J.Y.: Dense text-to-image generation with attention modulation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7701–7711 (2023)

36

R. Sajnani et al.

17. Labs, B.F.: Flux.1: Official inference repository for flux open-weight models. https: //github.com/black-forest-labs/flux (2024) 18. Labs, B.F., Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., Kulal, S., Lacey, K., Levi, Y., Li, C., Lorenz, D., Muller, J., Podell, D., Rombach, R., Saini, H., Sauer, A., Smith, L.: Flux.1 kontext: Flow matching for in-context image generation and editing in latent space. ArXiv abs/2506.15742 (2025), https://api.semanticscholar.org/CorpusID: 279464475 19. Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 20. Li, Z., Wu, J., Koh, I., Tang, Y., Sun, L.: Image synthesis from layout with localityaware mask adaption. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021) 21. Liu, C., Liu, D.: Late-constraint diffusion guidance for controllable image synthesis. arXiv preprint arXiv:2305.11520 (2023) 22. Liu, J., Huang, T., Xu, C.: Training-free composite scene generation for layout-toimage synthesis. In: European Conference on Computer Vision. pp. 37–53. Springer (2024) 23. Lukovnikov, D., Fischer, A.: Enabling controlnet to follow localized descriptions using cross-attention control. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 310–327. Springer (2025) 24. Mao, C., Zhang, J., Pan, Y., Jiang, Z., Han, Z., Liu, Y., Zhou, J.: Ace++: Instruction-based image creation and editing via context-aware content filling. arXiv preprint arXiv:2501.02487 (2025) 25. Mao, J., Wang, X.: Training-free location-aware text-to-image synthesis. In: 2023 IEEE International Conference on Image Processing (ICIP). IEEE (2023) 26. Mao, J., Wang, X., Aizawa, K.: Guided image synthesis via initial image editing in diffusion model. In: Proceedings of the 31st ACM International Conference on Multimedia (2023) 27. Mao, J., Wang, X., Aizawa, K.: Semantic-driven initial image construction for guided image synthesis in diffusion model. arXiv preprint arXiv:2312.08872 (2023) 28. Mishchenko, K., Defazio, A.: Prodigy: An expeditiously adaptive parameter-free learner. ArXiv abs/2306.06101 (2023), https://api.semanticscholar.org/ CorpusID:259129271 29. Mo, S., Mu, F., Lin, K.H., Liu, Y., Guan, B., Li, Y., Zhou, B.: Freecontrol: Trainingfree spatial control of any text-to-image diffusion model with any condition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 30. Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y., Qie, X.: T2iadapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 (2023) 31. Niu, Y., Wu, L., Zhang, Y., Zhu, Y., Zhu, G., Wang, J.: Multi-model style-aware diffusion learning for semantic image synthesis. ACM Transactions on Multimedia Computing, Communications and Applications 20(11) (2024) 32. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 33. Peng, B., Wang, J., Zhang, Y., Li, W., Yang, M.C., Jia, J.: Controlnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06070 (Nov 2024)

Appearance Pointers

37

34. Phung, Q., Ge, S., Huang, J.B.: Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05427 (2023) 35. Qi, Z., Huang, G., Huang, Z., Guo, Q., Chen, J., Han, J., Wang, J., Zhang, G., Liu, L., Ding, E., et al.: Layered rendering diffusion model for zero-shot guided image synthesis. arXiv preprint arXiv:2311.18435 (2023) 36. Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024) 37. Saha, O., Krs, V., Mech, R., Maji, S., Blackburn-Matzen, K., Gadelha, M.: Sigmagen: Structure and identity guided multi-subject assembly for image generation. arXiv preprint arXiv:2510.06469 (2025) 38. Shirakawa, T., Uchida, S.: Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging. arXiv preprint arXiv:2403.03485 (2024) 39. Shuai, X., Ding, H., Ma, X., Tu, R., Jiang, Y.G., Tao, D.: A survey of multimodal-guided image editing with text-to-image diffusion models. arXiv preprint arXiv:2406.14555 (2024) 40. Song, W., Jiang, H., Yang, Z., Quan, R., Yang, Y.: Insert anything: Image insertion via in-context editing in dit. arXiv preprint arXiv:2504.15009 (2025) 41. Sun, W., Li, T., Lin, Z., Zhang, J.: Spatial-aware latent initialization for controllable image generation. arXiv preprint arXiv:2401.16157 (2024) 42. Taghipour, A., Ghahremani, M., Bennamoun, M., Rekavandi, A.M., Laga, H., Boussaid, F.: Box it to bind it: Unified layout control and attribute binding in text-to-image diffusion models. IEEE transactions on multimedia (2025) 43. Tan, Z., Liu, S., Yang, X., Xue, Q., Wang, X.: Ominicontrol: Minimal and universal control for diffusion transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14940–14950 (2025) 44. Tian, Y., Tu, Z., Chen, H., Hu, J., Xu, C., Wang, Y.: U-dits: Downsample tokens in u-shaped diffusion transformers. In: Advances in Neural Information Processing Systems (NeurIPS) 37 (2024) 45. Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., J’egou, H.: Going deeper with image transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 32–42 (2021), https://api.semanticscholar.org/CorpusID: 232428161 46. Wang, J., Hu, J., Ma, X., Ma, H., Wei, X., Wu, E.: Image editing with diffusion models: A survey. arXiv preprint arXiv:2504.13226 (2025) 47. Wang, J., Chan, K.C., Loy, C.C.: Exploring clip for assessing the look and feel of images. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 2555–2563 (2023) 48. Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., Li, H.: Semantic image synthesis via diffusion models. arXiv preprint arXiv:2207.00050 (2022) 49. Wang, X., Fu, S., Huang, Q., He, W., Jiang, H.: MS-diffusion: Multi-subject zeroshot image personalization with layout guidance. In: The Thirteenth International Conference on Learning Representations (2025), https://openreview.net/forum? id=PJqP0wyQek 50. Wang, X., Darrell, T., Rambhatla, S.S., Girdhar, R., Misra, I.: Instancediffusion: Instance-level control for image generation (2024) 51. Wang, Y., Zhang, W., Zheng, J., Jin, C.: Enhancing object coherence in layout-toimage synthesis. arXiv preprint arXiv:2311.10522 (2023)

38

R. Sajnani et al.

52. Wei, Y., Zheng, Y., Zhang, Y., Liu, M., Ji, Z., Zhang, L., Zuo, W.: Personalized image generation with deep generative models: A decade survey. arXiv preprint arXiv:2502.13081 (2025) 53. Xiao, J., Li, L., Lv, H., Wang, S., Huang, Q.: R&b: Region and boundary aware zero-shot grounded text-to-image generation. arXiv preprint arXiv:2310.08872 (2023) 54. Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., Shou, M.Z.: Boxdiff: Textto-image synthesis with training-free box-constrained diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023) 55. Xue, H., Huang, Z., Sun, Q., Song, L., Zhang, W.: Freestyle layout-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 56. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 57. Yang, B., Luo, Y., Chen, Z., Wang, G., Liang, X., Lin, L.: Law-diffusion: Complex scene generation by diffusion with layouts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023) 58. Yang, P., Cheung, N.M., Ma, X.: Text to image generation and editing: A survey. arXiv preprint arXiv:2505.02527 (2025) 59. Zeng, Y., Lin, Z., Zhang, J., Liu, Q., Collomosse, J., Kuen, J., Patel, V.M.: Scenecomposer: Any-level semantic image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 60. Zhan, Z., Chen, D., Mei, J.P., Zhao, Z., Chen, J., Chen, C., Lyu, S., Wang, C.: Conditional image synthesis with diffusion models: A survey. arXiv preprint arXiv:2409.19365 (2024) 61. Zhang, H., Duan, Z., Wang, X., Chen, Y., Zhang, Y.: Eligen: Entity-level controlled image generation with regional attention. arXiv preprint arXiv:2501.01097 (2025) 62. Zhang, H., Wang, S., Li, J., Wu, Z., et al.: Seg2any: Open-set segmentationmask-to-image generation with precise shape and semantic control. arXiv preprint arXiv:2506.00596 (2025) 63. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023) 64. Zhao, P., Li, H., Jin, R., Zhou, S.K.: Loco: Locally constrained training-free layoutto-image synthesis. arXiv preprint arXiv:2311.12342 (2023) 65. Zhao, S., Chen, D., Chen, Y.C., Bao, J., Hao, S., Yuan, L., Wong, K.Y.K.: Unicontrolnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems 36 (2024) 66. Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y., Li, X.: Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 67. Zheng, Z., Isobe, T., Shen, T., Jia, X., Zhao, J., Li, X., Ge, M., Li, B., Wang, Q., Li, D., et al.: Layout-conditioned autoregressive text-to-image generation via structured masking. arXiv preprint arXiv:2509.12046 (2025) 68. Zhou, D., Li, M., Yang, Z., Yang, Y.: Dreamrenderer: Taming multiinstance attribute control in large-scale text-to-image models. arXiv preprint arXiv:2503.12885 (2025) 69. Zhou, D., Xie, J., Yang, Z., Yang, Y.: 3dis-flux: simple and efficient multi-instance generation with dit rendering. arXiv preprint arXiv:2501.05131 (2025)