Leveraging Foundation Models for Causal Generative Modeling Aneesh Komanduri
Xintao Wu
University of Arkansas Fayetteville, Arkansas, USA [email protected]
University of Arkansas Fayetteville, Arkansas, USA [email protected]
arXiv:2605.23861v1 [cs.LG] 22 May 2026
Abstract Causal generative modeling is essential for developing reliable and transparent AI systems capable of counterfactual reasoning. While existing approaches focus on integrating causal constraints during the training of generative models, they often lack a unified framework to leverage the zero-shot reasoning capabilities of pretrained foundation models. We introduce FM-CGM, a modular framework for end-to-end visual causal reasoning using pretrained foundation models. FM-CGM formalizes the causal pipeline through three core components: a concept extractor, a concept manipulator, and a counterfactual generator. By leveraging a large reasoning model for causal inference and a text-to-image diffusion model for generation, our approach enables zero-shot causal discovery, intervention, and counterfactual generation. We then develop Causal Semantic Guidance (CSG), a cross-attention-based mechanism that ensures semantic interventions propagate to descendant concepts while preserving invariant regions. We empirically show that our approach can identify plausible causal structures and is suitable for faithful counterfactual image generation.
CCS Concepts • Computing methodologies → Causal reasoning and diagnostics.
Keywords causality, generative models, large vision-language models, diffusion models, counterfactual generation ACM Reference Format: Aneesh Komanduri and Xintao Wu. 2018. Leveraging Foundation Models for Causal Generative Modeling. In . ACM, New York, NY, USA, 8 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Causal generative modeling [7] has been studied both in the context of discovering causal concepts and their relationships from highdimensional data [22] and reasoning about counterfactual scenarios [19]. This direction has contributed to improving the reliability and transparency of AI systems. Existing work in causal generative modeling primarily focuses on integrating causality directly into the training process of generative models such as GANs [5], VAEs Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
[8, 14], and Diffusion models [9] by modeling relationships among generative causal factors. Several works have also studied counterfactual image generation using state-of-the-art diffusion models [21, 24]. Recently, due to the inference capabilities of large-scale generative models such as large language model (LLMs) and large reasoning models, there has been ongoing research utilizing the power of foundation models as domain experts to improve causal discovery and inference [25], and causal agent frameworks to automate tasks in scientific discovery [26]. However, there is no unified framework that leverages the capabilities of pretrained foundation models to discover generative factors, reason about interventions, and generate counterfactual scenarios, the complete pipeline of causal generative modeling. In this work, we propose a formalism for visual causal reasoning from the perspective of pretrained foundation models. We develop Foundation Model Powered Causal Generative Model (FM-CGM), a high-level abstraction for causal generative modeling using pretrained foundation model components. We formalize the key components of a visual causal generative model in this setting as (1) concept extractor, which infers a set of concepts and their causal relationships from a given image, (2) concept manipulator, a mechanism to perform interventions on concept variables and propagate causal effects, and (3) a counterfactual generator, a generation mechanism to map the concepts under intervention to a counterfactual image. We represent each component using a powerful large-scale foundation model. Specifically, we implement a practical method consisting of a reasoning VLM (Qwen3-VL) as the concept extractor and manipulator and text-to-image generative model (Stable Diffusion XL) as the counterfactual generator. To translate semantic interventions inferred by the concept manipulator into pixelspace counterfactual edits, we propose causal semantic guidance (CSG) a procedure that operates through the text-to-image diffusion model’s text-conditioning mechanism. Concretely, we leverage cross-attention attribution maps to localize the regions associated with each concept and its descendants and construct an inferencetime guidance mechanism that (a) amplifies edits for intervened concepts, (b) encourages consistent changes in descendant concepts, and (c) suppresses unintended changes to non-descendant (invariant) concepts. This yields counterfactual images that are both visually plausible and aligned with the intended causal semantics. Contributions. We summarize our contributions as follows (1) We propose a general and modular framework for end-to-end causal generative modeling with pretrained foundation model components including a concept extractor, concept manipulator, and counterfactual generator, each represented as a foundation model. (2) We develop causal semantic guidance (CSG), a cross-attentionbased guidance method that compositionally enforces the concept under intervention and its descendants, inferred by a large reasoning model, while preserving non-descendant concepts, enabling
Conference’17, July 2017, Washington, DC, USA
Komanduri et al.
faithful counterfactual generation. (3) We demonstrate empirically that our method can perform minimal and faithful counterfactual edits.
Î where ∀𝑖, 𝑓𝑖 : 𝑈𝑖 × 𝑗 ∈pa𝑖 𝐶 𝑗 → 𝐶𝑖 are causal mechanisms that determine each causal variable as a function of the parents and noise, 𝐶 pa𝑖 are the parents of causal variable 𝐶𝑖 .
2
Diffusion Probabilistic Models. Diffusion models are a class of likelihood-based generative models that consist of two main steps: forward and reverse diffusion. Given an input image 𝑋 0 sampled from some data distribution, the forward diffusion process defines a Markov chain of diffusion steps to slowly destroy the structure of data through a series of Gaussian noise perturbations defined as follows: √ √ 𝑋𝑡 = 𝛼¯𝑡 𝑋 0 + ( 1 − 𝛼¯𝑡 )𝜖 (2) Î𝑡 𝛼𝑖 . The reverse where 𝜖 ∼ N (0, I), 𝑡 ∼ Unif(0,𝑇 ) and 𝛼¯𝑡 = 𝑖=1 diffusion restores the structure of data by parameterizing a UNet model 𝜖𝜃 to predict the noise perturbation at each time step 𝑡 given input image 𝑋𝑡 and minimizing the mean squared error between the predicted and actual noise at each time step 𝑡. During inference, random noise 𝑋𝑇 is sampled from a standard Gaussian distribution and 𝜖𝜃 is applied iteratively to denoise 𝑋𝑇 to a clean image 𝑋 0 . The overall objective of the diffusion model is as follows
Related Work
Diffusion Models for Controllable Generation. Diffusion models [3, 13, 23] have quickly become one of the most effective tools for controllable image generation with models such as DALL-E [18], StableDiffusion [20], T2I-Adapter [12], ControlNet [31], etc. Several different techniques have been proposed to achieve controllable generation, such as fine-tuning, adaptation, and post-hoc optimization. Recent work in diffusion probabilistic models has focused on interpretable properties to enable generative control. [28] show that pretrained text-to-image diffusion models, such as Stable Diffusion, already have somewhat of a content-style disentanglement capability, and propose a lightweight optimization to learn a soft mixing of content and style-infused prompts for better disentanglement. [29] study disentanglement properties of diffusion probabilistic models by learning separate gradient fields. [2] propose LEDITS++, an image editing technique using cross-attention attribution maps capable of composing multiple concept edits. Causality-based Diffusion Models. [21] provide a causal interpretation of classifier guidance in diffusion models and propose a method to guide the generation of image counterfactuals using interventions on labeled data. [9] explore counterfactual generation in diffusion models through guidance from disentangled causal representations. [19] study counterfactual identifiability in highdimensions leveraging flow-matching. [24] develop a framework for counterfactual video editing in pretrained text-to-video generative models by utilizing VLMs to refine counterfactual prompts given a fixed causal graph. [27] formalize the notion of causal separability in text-to-image generative models through a counterfactual lens and show that concepts that are independently manipulable lie in orthogonal subspaces. Based on this intuition, they propose a simple projection mechanism to manipulate style components of generated images while preserving content. Large Vision-Language Models. Extending LLMs to incorporate visual inputs, large vision-language models (LVLMs) have shown impressive performance in tasks such as recognition and visual question answering (VQA). Previous work has shown that LVLMs can sometimes struggle to perform formal causal reasoning [6]. Recently, there has been extensive development in improving the reasoning abilities of LLMs and LVLMs through reinforcement learning techniques with verifiable rewards. Recently, there has been a large improvement in visual reasoning capabilities in vision language models, including QwenVL [1], OpenAI o1, and GeminiThinking. These models have shown remarkable performance on math, coding, and visual reasoning tasks.
3
Preliminaries
Structural Causal Model. [15]. A structural causal model (SCM) is defined as C = ⟨𝐶, 𝑈 , 𝐹 ⟩, where 𝐶 is the set of semantic concepts, 𝑈 is the set of exogenous noise factors, and 𝐹 is a set of independent causal mechanisms of the form 𝐶𝑖 = 𝑓𝑖 (𝑈𝑖 , 𝐶 pa𝑖 )
(1)
Lsimple = E𝑋𝑡 ,𝑡,𝜖 (∥𝜖 − 𝜖𝜃 (𝑋𝑡 , 𝑡)∥ 22 )
(3)
Latent diffusion models (LDMs) apply the same procedure in the latent space of a well-trained variational autoencoder that compresses the image data 𝑋 0 to a perceptually equivalent low-dimensional representation 𝑧 0 . Operating in the latent space has significant advantages such as faster training and sampling. The noise parameterization UNet 𝜖𝜃 is modified to perform denoising of the latent variables. Stable Diffusion (SD), a specific implementation of the latent diffusion model, is a text-to-image generative model that conditions the reverse diffusion process on a text prompt 𝑌 from a CLIP [17] text encoder. The prompt embedding influences the feature maps of the UNet through cross-attention mechanisms. The objective of Stable Diffusion is as follows LStableDiffusion = E𝑧𝑡 ,𝑡,𝜖 (∥𝜖 − 𝜖𝜃 (𝑧𝑡 , 𝑡, 𝑌 )∥ 22 )
4
(4)
Foundation Model Powered Causal Generative Model
In order to define a causal generative model, we require three components: (1) a mechanism to infer the semantic concepts and their causal relationships from an image (concept extractor), (2) an intervention mechanism to manipulate concepts and their descendants (concept manipulator), and (3) a generation mechanism to map intervened concepts to a counterfactual image (counterfactual generator). We describe each component in more detail and the desiderata for pretrained foundation models to simulate each component in the vision-language setting as follows: Concept Extractor. A concept extractor foundation model E maps an image 𝑋 and a text description 𝑌 to a discrete set of textual concepts 𝐶 and a directed acyclic graph G describing the relationships among concepts as follows: (𝐶, G) = E(𝑋, 𝑌 )
(5)
The concepts are assumed to be described by a semantic Structural Causal Model (SCM) C, where (conceptually) the joint distribution
Leveraging Foundation Models for Causal Generative Modeling
Conference’17, July 2017, Washington, DC, USA
Abduction Image-to-Noise Inversion
Text-to-Image Diffusion Model Large Vision-Language Model
noise seed
Prediction
gender “A photo of a woman”
𝑋
Concept Extractor (E)
Counterfactual Generator (G)
𝑌
facial hair
𝑋!"
𝒀𝒊𝒏𝒕 = “A photo of a man”
What is the result of changing gender from male to female?
𝒀𝒅𝒆𝒔𝒄 = “mustache”
gender Concept Manipulator (M) facial hair
Action
Figure 1: An overview of Foundation Model Powered Causal Generative Model (FM-CGM) consisting of a concept extractor, concept manipulator, and counterfactual generator enabled by foundation models for the causal concepts follows a Markov factorization: 𝑝 (𝐶 1, ..., 𝐶𝑛 ) =
𝑛 Ö
𝑝 (𝐶𝑖 |𝐶 pa𝑖 )
(6)
Foundation Model Powered Causal Generative Model. Formally, we define a class of visual causal generative models with pretrained foundation model components as follows:
𝑖=1
Concept Manipulator. Given an inferred set of concepts and their causal relationships, a concept manipulator foundation model M performs an intervention on a concept and propagates effects to descendant concepts consistent with the causal graph G 𝐶 ′ = M(𝑋, 𝑌, (𝐶, G), do(𝐶𝑖 = 𝑐𝑖′ ))
Definition 4.1. An Foundation Model Powered Causal Generative Model (FM-CGM) is a quadruplet F = ⟨C, E, M, G⟩ where: • C = (𝐶, G) is a semantic structural causal model describing a set of high-level concepts 𝐶 and their causal relationships G corresponding to the image 1 • E is a concept extractor represented by a large reasoning foundation model to infer a set of concepts 𝐶 and their causal relationships G, as described in Eq. 5 • M is a concept manipulator represented by a large reasoning foundation model to reason about interventions on causal concepts (e.g., do(𝐶𝑖 = male)), as described in Eq. 7 • G is a counterfactual generator represented by a large text-toimage generative foundation model to map interventions on high-level concepts to a counterfactual image, as described in Eq. 8
(7)
where 𝐶 ′ is the set of concepts after intervention on concept 𝐶𝑖 and 𝑐𝑖′ is the specific value of the intervention. Specifically, this component performs a do intervention on a concept variable (e.g., do(𝐶𝑖 = male) if 𝐶𝑖 describes the gender concept). Counterfactual Generator. A counterfactual generator foundation model G takes as input a set of concepts 𝐶 ′ after an intervention, and a representation of the factual image 𝑋 (for structural preservation), and generates a counterfactual image consistent with the intervention 𝑋𝐶𝐹 = G(𝐶 ′, 𝑋 )
(8)
The generation mechanism maps concepts to measurable variables in the high-dimensional image (e.g., pixel groups representing a concept). That is, 𝑋 = 𝑔(𝑋𝐶1 , . . . , 𝑋𝐶𝑛 , 𝜉) where 𝑔 is one-to-one mapping describing a set of 𝑋 -measurable concepts composing the generated image. Intuitively, the image 𝑋 is generated by abstract causal concept variables with causal mechanisms that are independent and modular with respect to the image. For example, in a text-to-image generative model, we can consider concepts to manifest in the image as high values in a region of the image’s attention maps corresponding to the semantic concept.
5
Methodology
In this section, we propose a practical foundation model-powered architecture to implement the FM-CGM, consisting of a Vision Language Model (VLM) to represent the concept extractor and concept manipulator and a text-to-image diffusion model (i.e., Stable Diffusion) to represent the counterfactual generator. The overall FM-CGM framework is shown in Figure 1.
1 A semantic structural causal model is one that only includes high-level concepts and
their relationships and not any causal mechanisms to comply with our setting
Conference’17, July 2017, Washington, DC, USA
Komanduri et al.
Concept Extractor Prompt You are an expert in causal reasoning, computer vision, and semantic scene analysis. Your task is to analyse images and identify the underlying causal structure of the scene. You always respond with valid, well-formed JSON and nothing else — no prose, no markdown, no explanation outside the JSON object. Carefully examine the provided image together with the following context description: <context> {prompt} </context> Your task has two parts: PART 1 — CONCEPT EXTRACTION First, identify 5 high-level semantic concepts visible in or strongly implied by the image. A concept should be a high-level attribute about the subject (e.g., age, gender, etc.). Then, for each, ask: what would change if I were to change this concept to a different value? Generate 2-3 additional concepts that would directly change as a result. Concepts should NOT be related to lighting or posture. For each concept record: • id — short unique identifier (c1, c2, . . . ) • name — concise label (2-5 words) • current_value — specific observable state (1–3 words) • description — one sentence explanation PART 2 — CAUSAL RELATIONSHIP DISCOVERY Reason about which concepts causally influence others (A → B). Only include relationships that are causally plausible. Produce a directed acyclic graph (DAG). For each edge record: • id — unique identifier (r1, r2, . . . ) • cause_id — id of the causing concept • effect_id — id of the effected concept • description — one sentence: how/why the cause produces the effect Also provide a one-sentence scene_summary. Keep your answer short. Do NOT explain. Return ONLY the following JSON object: { "concepts": [...], "relationships": [...], "scene_summary": "..." }
Figure 2: Concept extractor prompt
5.1
VLM-based Concept Extractor
In this section, we provide the details of the Concept Extractor. We represent the concept extractor as a pretrained reasoning visionlanguage model (VLM) that maps an image to a textual description of a concept. A concept variable is a categorical variable that takes exactly one value at a time and describes the visual scene. The resulting (text) concept serves as the semantic state for downstream counterfactual generation. We first set up our motivation behind using an VLM to extract causally related concepts. [30] conjecture that large-scale foundation models learn causal facts, but not physical mechanisms. We rely on this assumption to obtain a plausible causal structure over identified concepts (a meta-SCM) from a reasoning model, which is sufficient for constraining downstream counterfactual edits. The diffusion decoder (via guidance) then provides the image-level mechanism that realizes these semantic changes. We first detail a multi-stage approach to identifying useful causally related concepts by leveraging the reasoning abilities of large reasoning vision language models. We construct a robust prompt to identify the most important elements in a visual scene and their relationships as follows (1) Visual Description: We prompt the VLM to describe the visual scene for context to obtain a base prompt 𝑌 . (2) Concept Identification: We prompt the VLM to return a list of the most important and interpretable concepts (nodes) that describe the visual scene. We define this output as a concept set 𝐶 = {𝐶 1, . . . , 𝐶𝑛 } for the given image where a concept variable 𝐶𝑖 takes a specific valuation specified by 𝑐𝑖 . (3) Causal Graph: Given the identified set of concepts 𝐶, we prompt the VLM to identify edges between pairs of concepts (𝐶𝑖 , 𝐶 𝑗 ) if 𝐶𝑖 is a direct cause of 𝐶 𝑗 to generate a causal graph G.
We obtain a set of fully specified concepts 𝐶 along with a causal graph G with which we can reason about causal queries. The complete prompt template for VLM concept and graph identification is specified in Figure 2. For visual scenes, we find it reasonable to leverage strong reasoning-capable VLMs to output consistent concept sets and plausible causal graphs.
5.2
VLM-based Concept Manipulator
Define 𝐼 = {1, . . . , 𝑛} as the indices for concepts. For an intervention target 𝑖 ∈ 𝐼 with intervention do(𝐶𝑖 = 𝑐𝑖′ ) on a causal concept, we prompt the VLM to obtain a counterfactual concept set 𝐶 ′ such that (1) the intervention do(𝐶𝑖 = 𝑐𝑖′ ) is performed, (2) De(𝐶𝑖 ) (descendants) are updated to values consistent with the intervention and the causal graph, and (3) ND(𝐶𝑖 ) (non-descendants) are preserved. This semantic counterfactual state 𝐶 ′ is then translated into diffusion guidance as defined below. The prompt for the concept manipulator is specified in Figure 3.
5.3
Diffusion-based Counterfactual Generator
In this section, we provide the details of the Counterfactual Generator. Since we cannot mechanistically change the Stable Diffusion model to enforce structural causal mechanisms in the latent space, we primarily rely on guidance satisfying counterfactual minimality. In this section, we take inspiration from the LEDITS++ framework [2] and develop a VLM-informed counterfactual generation method Causal Semantic Guidance (CSG), a classifier-free guidance approach coupled with a VLM as the Concept Manipulator to achieve faithful counterfactual image generation. Following the Pearlian paradigm [15], we implement the counterfactual edit through a structured Abduction-Action-Prediction style procedure. Abduction. We first capture the noise 𝑧𝑇 specific to the factual image 𝑋 0 by performing an inversion of the sampling process. We
Leveraging Foundation Models for Causal Generative Modeling
Conference’17, July 2017, Washington, DC, USA
Concept Manipulator Prompt You are an expert at understanding visual scenes and reasoning about how changing one thing in a scene affects other things. You always respond with valid, well-formed JSON and nothing else — no prose, no markdown, no explanation outside the JSON object. You are given an image, a description of the scene it shows, a list of concepts present in it, and a list of relationships that describe how those concepts influence one another. <scene_summary> {scene_summary} </scene_summary> <concepts> {concepts_json} </concepts> <relationships> {relationships_json} </relationships> Your task: 1. PROPOSE 3 DIVERSE MANIPULATIONS • Each manipulation changes a DIFFERENT concept to a new value. • Manipulations should produce visually distinct scenes. • Pick one concept that affects many others, one that affects only a few, and one that affects none. • Keep each manipulation realistic and physically plausible. 2. WORK OUT WHAT ELSE CHANGES For each manipulation, think through every concept that would be different as a result of the changed concept. For each concept that changes, write its new value in 1–3 descriptive, unambiguous words that combine the concept variable and value (e.g. "wet road"). DO NOT INCLUDE CONCEPTS THAT STAY THE SAME. 3. LIST FINAL STATE OF ALL CONCEPTS THAT CHANGED Record the final value of every concept that changed. Each value must combine the concept variable and its new value into 1–3 words (e.g., "black hair"). 4. WRITE AN IMAGE GENERATION PROMPT Write a generation_prompt that adds to or modifies the following reference prompt: <reference_prompt> {base_prompt} </reference_prompt> Return ONLY the following JSON: { "interventions": [{"id": "intervention_1", "target_concept_id": "...", "target_concept_name": "...", "intervention_description": "...", "original_value": "...", "new_value": "...", "propagated_changes": [{"concept_id": "...", "concept_name": "...", "original_value": "...", "new_value": "...", "reason": "..."}], "final_concept_states": { "c1": "value", "c2": "value" }, "generation_prompt": "..." } ] }
Figure 3: Concept manipulator prompt utilize the fast second-order DPM-Solver++ inversion [11] to obtain the latent representation 𝑧𝑇 via forward diffusion. Action. Given the counterfactual states 𝐶 ′ obtained from the VLM Concept Manipulator, we can now utilize classifier-free guidance and disentangled image editing techniques to realize counterfactual image edits. Now, for a given intervened concept 𝐶𝑖 , we construct an interventional prompt 𝑌𝐶𝑖 following the style of the base prompt 𝑌 and a set of prompts 𝑌De(𝐶𝑖 ) for descendants. For the collection of descendant concepts, we define the guidance term as the following sum ∑︁ 𝛾 (𝑋𝑡 , 𝑌De(𝐶𝑖 ) ) = 𝜙 (𝜓 ; 𝑠𝑐 , 𝜆)𝜓 (𝑋𝑡 , 𝑌𝑐 ) (9) 𝑐 ∈De(𝐶𝑖 )
where 𝜙 applies an elementwise edit guidance scale 𝑠𝑐 , and 𝜓 is defined as 𝜓 (𝑋𝑡 , 𝑌𝑐 ) = 𝜖𝜃 (𝑋𝑡 , 𝑌𝑐 ) − 𝜖𝜃 (𝑋𝑡 )
(10)
The 𝜙 term scales elements of the image and its score estimate that are relevant to the prompt 𝑌 . A larger scaling factors 𝑠𝑐 naturally increases the effect of the intervention, and 𝜆 selects the relevant pixels. The term 𝜙 is an intersection of a binary masks 𝑀𝑐1 generated from UNet cross-attention layers for a concept 𝑐 and 𝑀 2 generated from the unconditioned noise estimate (thresholded by the value 𝜆), scaled by a factor 𝑠𝑐 . 𝜙 (𝜓 ; 𝑠𝑐 , 𝜆) = 𝑠𝑐 𝑀𝑐1 𝑀 2
(11)
The intersection of the masks focuses both on relevant image regions and fine-grained semantic information from the descendants. Intuitively, the 𝑀 2 preserves other features, including the nondescendant concepts ND(𝐶𝑖 ). Prediction. We define the score update as follows 𝜖𝜃 (𝑋𝑡 , 𝑌𝐶 ′ ) = 𝜖𝜃 (𝑋𝑡 , 𝑌𝐶𝑖 ) + 𝛾𝑡 (𝑋𝑡 , 𝑌De(𝐶𝑖 ) )
(12)
where the guidance term 𝛾 pushes the interventional score estimate 𝜖 (𝑋𝑡 , 𝑌𝐶𝑖 ) toward the descendant concepts estimate to ultimately realize a minimal counterfactual edit satisfying the causal graph.
6
Experiments
In this section, we empirically show as a demonstration of concept that we can produce accurate image counterfactuals using our proposed framework. Specifically, we show the result of VLM interventional reasoning coupled with the compositional editing technique compared to baseline editing methods. Datasets. We qualitatively evaluate our method on a small set of images generated directly from Stable Diffusion XL (SDXL), as shown in Figures 4 and 5. We also perform a larger scale experiment on two benchmark datasets: MS-COCO [10] and CelebA-HQ [4]. MS-COCO consists of images with natural language captions describing diverse everyday scenes such as objects, animals, outdoor environments, and complex multi-entity compositions. CelebA-HQ is a high-quality human face dataset of 30,000 celebrity portrait
Conference’17, July 2017, Washington, DC, USA
Original Image
DDPM Inversion
Komanduri et al.
DDIM Inversion
Original Image
CSG (Ours)
CSG (Ours)
facial hair
facial hair
gender
young
smooth skin no wrinkles
facial hair old
old
white hair
wrinkles
wrinkles
smiling
smiling
eyebrows raised
DDIM Inversion
gender
gender
white hair
DDPM Inversion
eyebrows raised
eyes narrow
(a)
eyes narrow
(b)
Figure 4: Image Counterfactual Generation on human facial characteristics for (a) close-up profile and (b) garden scene Original Image
DDPM Inversion
DDIM Inversion
CSG (Ours)
dry weather
dry hair fluffy hair sunny weather
no frost
warm glow
weather sunny
no umbrella
Figure 5: Counterfactual Generation on weather scenes images, each annotated with a descriptive text caption covering facial attributes such as age, gender, hair color, and expression. For each dataset, we randomly sample 75 images per dataset using a fixed seed to ensure reproducibility, for a total of 𝑁 = 150 examples. Sampling is performed without replacement over the full available split. Each sampled image and its associated caption are passed through the complete pipeline: concept extraction, manipulation, generation across all editing methods. Implementation. Our pipeline implements automated causal concept discovery and counterfactual generation across four distinct stages. First, a Qwen3-VL-30B-A3B-Instruct [1] vision-language model (VLM) analyzes an input image and a textual description to extract a set of high-level semantic concepts, such as hair color or age, and their downstream effects. This stage constructs a directed acyclic graph (DAG) of causal relationships, with inference performed at a temperature of 0.1 and a budget of 2,048 tokens. Second,
the VLM, conditioned on both the image and the discovered graph, proposes three diverse causal interventions at a temperature of 0.2 with a limit of 3,072 output tokens. Each intervention targets a unique concept, propagates its effects through the DAG, records all modified concept states as descriptive phrase-value pairs, and produces a text-to-image generation prompt. Third, counterfactual images are generated with Stable Diffusion XL [16] via DDIMInversion, DDPMInversion as baselines, and Causal Semantic Guidance. For DDIMInversion and DDPMInversion, we incorporate classifier-free guidance with the fully specified prompt with all concepts included. For CSG, we use the LEDITS++ pipeline and a DPM-Solver++ scheduler to invert the original image into noise over 100 steps with a source guidance scale of 3.5 and an inversion skip ratio of 0.15. The system then denoises the image under the CSG edit using the propagated concept values as per-concept editing prompts. This involves an edit guidance scale of 8.0, an edit threshold of 0.7, 10 warmup steps, and intersect masking, while falling back to the direct manipulation value if no downstream concepts are affected. Qualitative Evaluation. We evaluate our method based on several image samples generated from Stable Diffusion XL. We find that our approach consistently outperforms baselines in generating minimal counterfactually consistent edits based on the causal graph specified by the VLM. For example, in Figure 4a, intervening on the gender attribute should cause facial hair to change while preserving every other attribute in the original image. DDIM and DDPM Inversion introduce unnecessary variations, while CSG generates a minimal and consistent edit with downstream effects. We also find that our method works well when manipulating more complex concepts such as weather. In Figure 5, the first row shows an intervention on the weather from rainy to dry weather. The edit generated by our method shows a clear background and dry hair, which are inferred according to the VLM (e.g., sunny weather
Leveraging Foundation Models for Causal Generative Modeling
Conference’17, July 2017, Washington, DC, USA
Table 1: Evaluation Results CelebA-HQ
Method
VLM-Eff. ↑ DDIM Inversion DDPM Inversion CSG (Ours)
0.792 0.756 0.854
MS-COCO
Average
LPIPS ↓ VLM-Eff. ↑
LPIPS ↓ VLM-Eff. ↑
LPIPS ↓
0.4836 0.2244 0.1980
0.4759 0.2035 0.1750
0.4798 0.2139 0.1865
VLM Effectiveness Evaluation Prompt You look at images and answer simple questions about what is shown in them. You always respond with valid, well-formed JSON and nothing else — no prose, no markdown, no explanation outside the JSON object. Look at the given image. It was generated with the following prompt: <prompt> {generation_prompt} </prompt> The one thing that was changed to produce this image was: {target_concept_name}: {original_value} → {new_value} Below is a checklist of concepts and the value each one should have in the image. <checklist> {checklist_text} </checklist> Your task: 1. CHECKLIST EVALUATION For each item in the checklist, answer "yes" if that value is clearly present in the image, or "no" if it is not. 2. VERDICT ASSIGNMENT • "success" — all checklist items are "yes" • "partial" — most checklist items are "yes" but some are wrong • "failure" — the main changed concept is not visible, OR most checklist items are "no" 3. REASONING Write one or two sentences explaining your verdict. Return ONLY the following JSON: { "intervention_id": "...", "concept_checks": [ { "concept_name": "...", "expected_value": "...", "present": "yes|no" } ], "verdict": "success|partial|failure", "reasoning": "..." }
Figure 6: VLM effectiveness evaluation prompt
causes dry hair and fluffy hair) whereas baselines either add high variations or fail to incorporate causal changes. Quantitative Evaluation. We also provide a quantitative evaluation on a subset of the CelebA-HQ and MS-COCO datasets. Given a counterfactual image, we evaluate effectiveness of the intervention by using VLM-Eff, a VLM-based approach to identify whether the generated image contains each counterfactual concept value, similar to [24]. We include the effectiveness prompt in the Figure 6 for reference. To evaluate minimality, or how much of a minimal edit the generated image is, we utilize perceptual similarity to the original image, quantified using LPIPS with a VGG backbone. We
0.741 0.655 0.762
0.767 0.705 0.808
report the average VLM-Eff. and LPIPS score for all methods in Table 1. Our results demonstrate that CSG is capable of more accurate counterfactual edits, while satisfying minimality compared to standard editing techniques.
7
Conclusion
In this work, we propose Foundation Model Powered Causal Generative Model, an abstraction for causal generative modeling consisting of pretrained foundation model components. We then develop a framework using large vision-language models for concept extraction and manipulation and Stable Diffusion XL for counterfactual generation. We propose Causal Semantic Guidance, an inferencetime editing approach extending on LEDITS++ to ensure interventions and propagation to descendant concepts are sufficiently represented in the counterfactually generated image. Qualitative and quantitative empirical evaluation shows that our framework is promising for modularizing and automating the process of causal generative modeling by utilizing the power of foundation models.
Acknowledgements This work is supported in part by National Science Foundation under awards 1910284, 1946391, 2147375, the National Institute of General Medical Sciences of National Institutes of Health under award P20GM139768, and the Arkansas Integrative Metabolic Research Center at University of Arkansas.
References [1] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025). [2] Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinario Passos. 2024. LEDITS++: Limitless Image Editing using Text-to-Image Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems. [4] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations. [5] Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. 2018. CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training. In International Conference on Learning Representations. [6] Aneesh Komanduri, Karuna Bhaila, and Xintao Wu. 2025. CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. [7] Aneesh Komanduri, Xintao Wu, Yongkai Wu, and Feng Chen. 2024. From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling. Transactions on Machine Learning Research (2024). [8] Aneesh Komanduri, Yongkai Wu, Feng Chen, and Xintao Wu. 2024. Learning Causally Disentangled Representations via the Principle of Independent Causal Mechanisms. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence.
Conference’17, July 2017, Washington, DC, USA
[9] Aneesh Komanduri, Chen Zhao, Feng Chen, and Xintao Wu. 2024. Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models. In Proceedings of the 27th European Conference on Artificial Intelligence. [10] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision. [11] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2025. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research 22, 4 (2025), 730–751. [12] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence. [13] Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved Denoising Diffusion Probabilistic Models. In Proceedings of the 38th International Conference on Machine Learning. [14] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. 2020. Deep Structural Causal Models for Tractable Counterfactual Inference. In Advances in Neural Information Processing Systems. [15] Judea Pearl. 2009. Causality (2 ed.). Cambridge University Press, Cambridge, UK. [16] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In The Twelfth International Conference on Learning Representations. [17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning. [18] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning. 8821–8831. [19] Fabio De Sousa Ribeiro, Ainkaran Santhirasekaram, and Ben Glocker. 2025. Counterfactual Identifiability via Dynamic Optimal Transport. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [20] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Komanduri et al.
[21] Pedro Sanchez and Sotirios A Tsaftaris. 2022. Diffusion causal models for counterfactual estimation. Conference on Causal Learning and Reasoning (CLeaR). [22] Bernhard Scholkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021. Toward Causal Representation Learning. Proc. IEEE 109 (May 2021), 612–634. [23] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising Diffusion Implicit Models. In International Conference on Learning Representations. [24] Nikos Spyrou, Athanasios Vlontzos, Paraskevas Pegios, Thomas Melistas, Nefeli Gkouti, Yannis Panagakis, Giorgos Papanastasiou, and Sotirios A Tsaftaris. 2025. Causally steered diffusion for automated video counterfactual generation. arXiv preprint arXiv:2506.14404 (2025). [25] Aniket Vashishtha, Abbavaram Gowtham Reddy, Abhinav Kumar, Saketh Bachu, Vineeth N. Balasubramanian, and Amit Sharma. 2023. Causal Inference using LLM-Guided Discovery. In AAAI 2024 Workshop on ”Are Large Language Models Simply Causal Parrots?”. [26] Vishal Verma, Sawal Acharya, Devansh Bhardwaj, Samuel Simko, Yongjin Yang, Anahita Haghighat, Dominik Janzing, Mrinmaya Sachan, Bernhard Schölkopf, and Zhijing Jin. 2025. Causal AI Scientist: Facilitating Causal Data Science with Large Language Models. In NeurIPS 2025 Workshop on CauScien: Uncovering Causality in Science. [27] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. 2023. Concept Algebra for (Score-Based) Text-Controlled Generative Models. In Advances in Neural Information Processing Systems. [28] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. 2023. Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1900–1910. doi:10.1109/CVPR52729.2023.00189 [29] Tao Yang, Yuwang Wang, Yan Lu, and Nanning Zheng. 2023. DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems. [30] Matej Zečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. 2023. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal. Transactions on Machine Learning Research (2023). [31] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009