Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation Nikolai Röhrich1,2⋆† , Isabell Hans1,2† , Felix Krause3,4† Björn Ommer3,4 LMU Munich, Germany Konrad Zuse School of Excellence in Reliable AI (relAI), Germany 3 CompVis @ LMU Munich, Germany 4 Munich Center for Machine Learning (MCML), Germany 1

arXiv:2608.14172v1 [cs.CV] 14 Aug 2026

2

Abstract. Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer’s concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-α, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance. Keywords: Text-to-Image Generation · Diffusion Models · Guidance

1

Introduction

Text-to-image (T2I) diffusion models have achieved remarkable quality in generating images from natural language descriptions [9, 12, 34, 37]. By sampling a random latent and iteratively refining it, these models gradually transform the latent into a coherent image that aligns with a given text prompt. Two fundamental challenges persist, particularly in one-shot generation. First, T2I models lack reliability in tasks requiring precise local coherence. This is perhaps most apparent in outputs involving text, where models frequently produce misspellings or hallucinated characters [8], and in images involving human hands, which often exhibit incorrect finger counts, distortions, or implausible geometry [32]. ⋆ †

Corresponding author: n.roehrich@campus.lmu.de Equal contribution.

2

N. Röhrich et al.

CoG for better text rendering in FLUX.1-dev

CoG for enhanced aesthetics in PixArt-α

CoG for improved human hands in Stable Diffusion 3

Fig. 1. Concept Guidance (CoG) improves arbitrary quantifiable concepts across a variety of T2I models. CoG enables models to recover from classical failure modes while also delivering better generative quality for abstract concepts.

Second, Classifier-Free Guidance (CFG) [16] – the de facto standard guidance mechanism in T2I diffusion – provides no fine-grained control over the generation process. In CFG, generation is guided by performing a conditional and an unconditional forward pass and by extrapolating beyond the conditional noise prediction. This technique effectively controls global prompt alignment, but it entirely lacks the ability to control fine-grained semantic details and tends to underperform for complex prompts due to its global nature [26, 49]. Prominent attempts to address these limitations fall into three broad categories, each with significant drawbacks (complementary directions, such as refined guidance rules, are discussed in Section 2): (1) Fine-tuning methods like ControlNet [53] or T2I-Adapters [31] can impose spatial control but require extensive per-concept training and large datasets. (2) Gradient-based methods [3, 11] can guide generation towards a target but are computationally expensive. (3) Inference-time intervention, e.g. by manipulating cross-attention maps [14, 23], offers more flexibility, but often requires careful model-specific tuning. Our method, Concept Guidance (CoG), in contrast, is an out-of-the-box, concept-specific guidance mechanism for T2I models that requires no training, external models, gradients, or reverse-engineering of model internals. By introducing a novel notion of per-layer, per-concept mutual information, we demonstrate the varying degrees of influence that layers in T2I models have on the generation of specific semantic concepts (see Figure 3), and we exploit this information by reinforcing the influence of concept-relevant layers (see Figure 2).

Concept Guidance

3

We first propose a framework to measure the layer-wise target performance by using target-specific metrics. Then, we generate predictions with relevant layers skipped and guide by extrapolating the standard prediction away from those predictions, using performance-based weighting. This selectively amplifies the most concept-relevant layers, allowing for precise guidance. Additionally, CoG seamlessly integrates with CFG, allowing users to effortlessly steer both global prompt alignment and targeted semantics. We make the following contributions: 1. We propose a novel method of measuring the mutual information between diffusion network layers and target concepts, and thereby show that the properties of specific layers can be exploited for precise, per-concept guidance. We demonstrate that layers that work well for skip-based guidance are precisely the layers responsible for generating relevant concepts. 2. In light of our layer analysis, we present Concept Guidance, a concept-specific guidance method based on layer skipping. CoG is a simple yet effective method that enables precise, out-of-the-box guidance for T2I models and increases T2I generation performance for arbitrary (measurable) concepts. 3. Through extensive experiments, we demonstrate that Concept Guidance consistently outperforms Classifier-Free Guidance and alternative guidance mechanisms based on layer-skipping. Across all tested models, we achieve an average single-target performance increase of 8.1% compared to CFG.

2

Related Work

Training-Free Guidance via Model Perturbations Diffusion sampling is commonly steered with classifier guidance [11] which leverages gradients from an external classifier to steer generation, or classifier-free guidance (CFG) [16], which increases condition adherence by contrasting conditional and unconditional predictions. Recent work revisits guidance rules to reduce CFG artifacts, e.g., via adaptive projection (APG) [38] or manifold-constrained guidance (CFG++) [10], and to obtain CFG-like behavior without special unconditional training [39]. Autoguidance [22] instead contrasts the model with a weakened version of itself, e.g., an earlier training checkpoint, thus requiring access to training artifacts. A complementary training-free line constructs a “weak” model at inference time by perturbing the generator itself, including attention-maps [1,17,18], processed tokens [36], or self-guidance derived from the model’s own dynamics [25]. Layer skipping is another perturbation method widely exposed in modern diffusion pipelines, e.g., Stable Diffusion 3 [42], yet it remains underexplored as a controllable handle. Spatiotemporal Skip-Guidance (STG) uses a fixed single-layer skip to improve video quality [21]. Our work is closest in spirit to perturbationbased training-free guidance, but differs by making skip perturbations conceptdependent and by using weighted combinations of multiple layers rather than a fixed skip.

4

N. Röhrich et al.

Concept-Specific Control Beyond Prompting Beyond prompt engineering, conceptspecific control is often achieved by adding learnable components: low-rank adapters (LoRA) [19] enable parameter-efficient updates and support per-concept modules such as concept sliders [13], while other approaches train auxiliary controllers (e.g., adapters) for new conditioning modalities [31]. Relatedly, some methods train lightweight predictors or readout heads on frozen diffusion features and backpropagate through them during sampling to enforce targets [28]. A different family uses external guidance objectives at inference time—either via gradients from arbitrary guidance functions [3] or via energy-based losses built from off-the-shelf predictors [51], with newer formulations addressing proxy unreliability through trust-region style sampling [20]. In contrast, our goal is a practical mechanism for concept-specific improvement within an off-the-shelf generator: we require no training and no external predictors during denoising. Concept Localization and Interpretability A growing body of work probes where and how diffusion models represent text-conditioned concepts. Attention-centric analyses and interventions treat cross-attention as the primary locus of wordregion binding [7, 14], and attribution methods such as DAAM derive token-topixel maps from cross-attention aggregation [46]. Separately, feature-level probing reveals that semantic correspondences emerge in intermediate layers and vary strongly with depth [29, 45]. Information-theoretic perspectives quantify prompt–image dependence more robustly than raw attention, using MI-style decompositions [24, 52] and applying information-theoretic objectives to improve alignment [48]. Mechanistic and causal approaches further localize attributerelevant components for model editing [4, 5] or component attribution [33], and recent work identifies “vital” layers in transformer backbones for training-free editing [2]. Our work connects these threads by turning per-concept layer specialization into an actionable control signal: we localize concept-relevant layers through targeted interventions and use the resulting layers to guide generation.

3

Method

3.1

Preliminaries

T2I Diffusion Text-to-Image diffusion models [12,34,37] generate images aligned with a text prompt by iteratively denoising a randomly sampled latent into a noise-free image [15,41]. This reverse process is trained by adding noise to image samples from the target distribution during the forward process. Formally, given an image x0 , noisy latents xt at timestep t are given by q(xt | x0 ) = N (xt ;

√

ᾱt x0 , (1 − ᾱt )I),

(1)

where ᾱt controls the noise level. Parameters θ are updated based on the distance between predicted noise ϵθ (xt , t, c) and actual noise ϵ, thus solving min θ

Ex0 ,ϵ,t,c ∥ϵ − ϵθ (xt , t, c)∥2 .

(2)

Concept Guidance

ϵ[θ \ α]

ϵCoG ϵθ

ϵ[θ \ β] ϵ[θ \ γ]

–

Target Performance

+

5

Fig. 2. Concept Guidance precisely approximates the target direction by combining perlayer skip predictions using conceptrelevant layers (α, β, γ). It computes individual skip-layer noise predictions (ϵ[θ\α] , ϵ[θ\β] , ϵ[θ\γ] ) and extrapolates over them to precisely estimate the target direction.

Classifier Guidance Classifier Guidance [11] was introduced to steer unconditional diffusion models towards a distribution aligned with a desired mode y. Given a classifier pϕ (y | xt ), guidance is achieved by modifying the process to favor samples the classifier considers more likely. The distribution is adjusted as pθ (xt−1 | xt , y) ∝ pθ (xt−1 | xt ) pϕ (y | xt ),

(3)

which corresponds to adding a correction to the model’s score estimate based on the gradients of the external classifier. In practice, this results in ∇ log pθ (xt | y) = ∇ log pθ (xt ) + λ ∇ log pϕ (y | xt ),

(4)

where λ is a scalar guidance strength controlling the influence of the classifier on the generation process. The classifier gradients ∇ log pϕ (y | xt ) can be interpreted as the direction of strongest label-alignment in the latent space, y. Classifier-Free Guidance Classifier-Free Guidance [16] steers diffusion generation towards a target y specified by a text prompt c, without requiring a classifier. Instead, the model is trained to perform both unconditional and text-conditioned generation. Then, guidance is achieved by extrapolating from the unconditional prediction ϵθ (xt , t, ∅) beyond the conditional prediction ϵθ (xt , t, c): ϵ̃θ (xt , t, c) = (1 − λ)ϵθ (xt , t, ∅) + λϵθ (xt , t, c),

(5)

where λ ≥ 1 controls the strength of conditioning. Similar to Classifier Guidance, this process can be interpreted as determining and reinforcing the direction of strongest condition alignment in the latent space y. 3.2

Concept Guidance

Classifier Guidance is precise in guiding towards specific targets, while ClassifierFree Guidance is simple and effective for increasing text-to-image alignment. We introduce Concept Guidance to combine the best of both worlds: highly usable, target-specific guidance in the latent space of diffusion models. We exploit the fact that given a target y, different layers of T2I models are particularly impactful w.r.t. y. To approximate the noise-space direction of highest target performance y, we find such layers and precisely adjust their influence (see Figure 4).

6

N. Röhrich et al. 0.15

0.20 0.15

0.10

0.10 0.05 0.05

Text

Hands

L3 L4 L5 L6 L7 L8 L9 L1 0 L1 1 L1 2 L1 3 L1 4 L1 5 L1 6 L1 7 L1 8 L1 9 L2 0 L2 1 L2 2 L2 3 L2 4

L9 L1 0 L1 1 L1 2 L1 3 L1 4 L1 5 L1 6 L1 7 L1 8

L8

L7

L6

L5

L4

0.00

L3

0.00

Aesthetics

(a) FLUX.1-dev

Hands

Aesthetics

(b) PixArt-α

Fig. 3. Different concepts concentrate in different layers, enabling conceptspecific layer selection for guidance. Bar plots report per-layer mutual information in (a) FLUX.1-dev and (b) PixArt-α, revealing strong, concept-dependent variation.

Mutual Information Analysis We hypothesize to find consistent, concept-dependent patterns of distributed layer responsibilities in T2I models that could be exploited for concept-specific guidance. To validate our intuition and to provide insight into why guiding with skipped layers works [21, 42], we analyze the mutual information (MI) between each layer and a given concept. We adapt the work of Wang et al. [48], where MI is used to locate layers that are responsible for general text-to-image alignment. Wang et al. [48] formulate their notion as the expected difference between conditional and unconditional predictions: 2

I(x, c) = Et,ϵ κt ∥ϵθ (xt , t, c) − ϵθ (xt , t, ∅)∥ ,

(6)

where κt scales the contribution of each timestep, reflecting information flow at that denoising stage. To extend this framework for our method, we introduce per-layer MI by computing Equation (6) with a single skipped layer: I(x, c, i) = Et,ϵ κt ϵ[θ\i] (xt , t, c) − ϵ[θ\i] (xt , t, ∅)

2

,

(7)

where we denote the conditional noise prediction with layer i skipped as ϵ[θ\i] (xt , t, c), or simply ϵ[θ\i] . We then define per-concept, per-layer MI as the difference in MI given a positive and a negative text prompt c and c \ y that differ only in the presence of the target y. That is, the mutual information between the target concept y and images generated while skipping layer i is given by I(x, y, i) = I(x, c, i) − I(x, c \ y, i).

(8)

Layer-Skipping We skip concept-relevant layers by redefining the residual mapping as an identity function for each skipped layer, similar to STG [21]: Res(zl ) = fl (zl ) + zl ,

Res(zl ) = id(zl ) = zl ,

(9)

where zl denotes the feature representation at the l-th layer, and fl represents the nonlinear transformation in layer l. In Res, the block output is set equal to its input, thus bypassing fl . This preserves information flow while preventing additional perturbations, allowing for controlled modulation.

Concept Guidance Offline Layer Profiling (once per model / concept)

Inference Time Concept Guidance (for each denoising step)

Choose Model & Target Concept

Prompt 𝑐 + noisy latent 𝑥(

e.g., FLUX.1-dev & Aesthetics

For each Layer i: Generate samples with layer i skipped

Score generations Using target metric or VLM judge

Select top-k target layers

7

Skipped Prediction 𝜖! Skipped Prediction 𝜖" Skipped Prediction 𝜖#

…

Based on layer performance 𝜔! = 𝑝! − 𝑝∅

Weighted Negative ∑$∈& 𝜔$ 𝜖$ 𝜖neg = ∑$∈& 𝜔$

Standard Prediction 𝜖'

CoG Extrapolation 𝜖CoG = 1 − 𝜆 𝜖neg + 𝜆𝜖'

Fig. 4. Concept Guidance Workflow. Left: In a one-time profiling stage, each layer is skipped individually, generations are scored, and top-k layers are selected. Right: At inference, per-layer skipped predictions are aggregated into a performance-weighted negative prediction; CoG then extrapolates from beyond the standard prediction.

Locating Layer-Directions in Latent Space The fundamental intuition behind our method is to decompose layer-wise predictions into a target direction y and a residual r. Let di be the noise prediction with layer i amplified, then di = αi y + r i ,

(10)

where αi ≥ 0, and ri satisfies ⟨ri , y⟩ = 0. Then, the idea is to extract information about the magnitude of αi by profiling the effectiveness of single-layer skip-guidance. For each layer i, we generate a set of images by using the skipped prediction ϵ[θ\i] as a negative direction to guide away from, similar to the unconditional noise prediction in Equation (5). We measure the target performance pi of these resulting images. Given this performance observation, the per-layer performance can be represented as a sum of αi plus a noise component ϵi : pi = αi + ϵi ,

(11)

where we assume that {ϵi }i≥1 are i.i.d. with E[ϵi ] = 0, and that ϵi is independent of (αi , r i ). Observing performances then yields a per-layer impact distribution for a given concept. The computational cost of this procedure scales linearly with the number of layers and the number of samples, i.e., O(L · N ) for L layers and N samples per layer. Notably, profiling is performed only once per model and concept. We make our layer analysis and code available at https://github.com/ CompVis/concept_guidance. Pseudocode is provided in Algorithms 1 and 2. Performance-Weighted Multi-Layer Guidance To approximate y, we compute predictions for the top-k best performing layers and weigh their impact on the overall prediction using their observed performance pi . Let K be the set of bestperforming layers, then we compute an individual negative noise prediction with layer i skipped for all i ∈ K. Each prediction is weighed by the performance term pi relative to the target performance without layer-skipping p∅ . Specifically, the weight ωi for the prediction with layer i skipped is given by:

8

N. Röhrich et al. Optimizing for aesthetics with Concept Guidance...

0

a) ... generates faces with sharper features, ...

c) ... yields improved lighting, symmetry, and composition, ...

b) ... outputs cleaner architecture with more refined geometry, ...

d) ... and produces more diverse and detailed natural scenes.

Fig. 5. Optimizing aesthetics with CoG improves perceptual quality. Sideby-side examples compare CFG vs. CoG.

ωi = pi − p∅ .

(12)

Thus, each layer contributes to the guidance process only insofar as it improves target performance compared to standard generation. The final negative noise prediction is then given by a weighted mean over all k noise predictions: P ωi · ϵ[θ\i] P ϵneg = i∈K . (13) i∈K ωi Finally, we extrapolate ϵneg beyond the standard noise prediction ϵθ : ϵCoG = (1 − λ)ϵneg + λϵθ ,

(14)

where ϵθ is given by CFG according to Equation (5), and λ ≥ 1 controls the guidance strength. Thus, CoG integrates seamlessly with CFG and allows easily combining general prompt adherence with target-specific guidance.

4

Experiments

4.1

Setup

We evaluate our method on multiple models with varying architecture, size, and output quality, such as PixArt-α [9], Stable Diffusion 3 [12,42], Stable Diffusion 3.5 [43] and FLUX.1-dev [6]. As targets we choose text and hand generation since text-to-image diffusion models notoriously struggle with those concepts. To provide evidence that CoG also extends to more abstract concepts we choose general aesthetics. We evaluate with EasyOCR [47], a pretrained model from MediaPipe [27] and a pretrained model for evaluating aesthetics [40] based

Concept Guidance Optimizing for visible text with Concept Guidance...

9

Optimizing for human hands with Concept Guidance...

a) ... avoids minor typos, ...

c) ... generates realistic hands with the correct finger count, ...

b) ... or correctly displays full text instances where CFG fails.

d) ... especially in scenarios where hands interact with objects.

Fig. 6. CoG improves fine-grained local coherence for hard failure cases like visible text and hands. Side-by-side examples compare CFG vs. CoG.

on CLIP embeddings [35]. Furthermore, we investigate if improving a target concept with CoG creates a trade-off for overall image quality using HPSv3 [30], which aligns strongly with human preferences. Throughout, all guidance methods are evaluated on identical prompts, seeds, and scheduler settings, so reported improvements correspond to paired comparisons. Beyond these metrics, we show that CoG also generalizes to concepts without a hand-crafted metric by using a vision-language model as the scoring judge (Section C), and we release our per-model, per-concept layer configurations (Table 6). 4.2

Layer Analysis

We derive insights on layer-concept interactions from computing per-layer, perconcept MI according to Equation (8). Evaluating MI across all tasks for FLUX.1dev and PixArt-α (see Figure 3 (a) and (b)) reveals that (i) there exist large concept-dependent differences, which supports our claim that networks distribute concept-specific knowledge non-uniformly across layers and (ii) we find that layers with high per-concept MI are often located in the middle of the network. To validate our findings, we evaluate correlations between per-layer MI and the measured layer performance on the target metric, and find very high correlations in the range of [0.692, 0.956]. This strongly reinforces our intuition that performance-weighted, multi-skip Concept Guidance works by extrapolating the influence of the layers that are most responsible for the generation of target concepts. 4.3

Qualitative Results

CoG consistently improves visual quality across all concepts. We present comparisons of CFG and CoG in Figure 5 and Figure 6. We further provide extensive uncurated comparisons of CoG and several baselines in Figures 14 and 15.

10

N. Röhrich et al.

Table 1. Concept Guidance achieves Table 2. Concept Guidance can better performance across measured be applied to multiple concepts at tasks. † does not generate visible text. once. † does not generate visible text. Model

Method

PixArt-α†

CFG CoGsingle CoGmulti

Text (↑) Hands (↑) Aesthetics (↑) 0.357 0.496 0.509 ▲42%

6.667 6.709 6.722 ▲0.8%

CFG 0.476 0.651 CoGsingle 0.483 0.674 CoGmulti 0.490 ▲2.9% 0.684 ▲5.0%

6.197 6.333 6.348 ▲2.4%

CFG 0.580 0.646 CoGsingle 0.598 0.662 CoGmulti 0.610 ▲5.2% 0.679 ▲5.1%

6.276 6.408 ▲2.1% 6.368

CFG 0.481 0.672 FLUX.1-dev CoGsingle 0.499 0.655 CoGmulti 0.510 ▲6.1% 0.675 ▲0.5%

6.448 6.631 6.657 ▲3.2%

SD3

SD3.5

-

Text (↑) + Hands (↑) Model

Method

PixArt-α†

CFG CoGsingle CoGmulti

SD3

SD3.5

Text (↑) + Hands (↑) + Aesthetics (↑) Aesthetics (↑)

Text Hands Text

Aes.

Hands

Aes.

-

-

-

0.330 0.252 0.334

6.217 6.231 6.260

CFG 0.353 CoGsingle 0.354 CoGmulti 0.356

0.423 0.502 0.503

0.416 0.396 0.424

6.213 6.222 6.282

0.631 0.720 0.745

5.922 5.915 5.938

CFG 0.498 CoGsingle 0.493 CoGmulti 0.508

0.620 0.655 0.693

0.637 0.634 0.642

5.974 5.861 5.893

0.330 0.252 0.334

6.217 6.260 6.231

CFG 0.346 FLUX.1-dev CoGsingle 0.354 CoGmulti 0.377

0.554 0.494 0.538

0.377 0.418 0.431

6.606 6.628 6.657

0.668 0.677 0.710

6.205 6.215 6.301

-

Complex Compositional Tasks. Concept Guidance improves tasks that require fine-grained structural coherence and correct object interactions. For human hands, CoG visibly reduces common artifacts and produces more plausible results, particularly in complex scenarios where hands interact with other objects (Figure 6 d ). When generating visible text, CoG consistently generates text more faithful to the prompt. In some cases, CoG even successfully renders complete and correct text where CFG fails to produce any readable output (Figure 6 b). General Concepts. We find that Concept Guidance also improves concept-specific alignment beyond notorious failure cases of T2I models. When optimizing for the more general concept of aesthetics, CoG yields both general and prompt-specific enhancements. Overall, we find that CoG produces images with improved lighting, contrast, and compositional symmetry (Figure 5 c). Moreover, CoG adapts stylistic elements to the prompt. Human faces exhibit sharper features, and healthier skin tones (Figure 5 a). Architectural scenes appear more modern, clean, and luxurious (Figure 5 b). Lastly, natural landscapes display greater visual diversity, for instance, through a richer variety of vegetation (Figure 5 d ). 4.4

Quantitative Results

We evaluate CoG with multiple performance-weighted layer skips (CoGmulti ) using four models and three different target concepts, and compare our method against Classifier-Free Guidance [16] and Concept Guidance with only a single, target-optimized skipped layer (CoGsingle ). Across all settings and models, CoGmulti consistently outperforms both CFG and CoGsingle (see Table 1): Concept Guidance achieves an average performance increase of 8.1% compared to CFG, and at its best, a 42% increase for hand generation with PixArt-α. We also find consistent improvements across models. CoG achieves an average target performance increase of 3.3% for FLUX.1-dev [6], 3.5% for SD3 [42], 3.9% for SD3.5 [43], and 22% for PixArt-α [9]. Regarding different tasks, CoG yields stable but moderate increases for aesthetics, and higher increases for tasks

Concept Guidance Classifier-Free Guidance

11

Concept Guidance

a) Jointly Optimizing Hands & Aesthetics

b) Jointly Optimizing Hands & Text

c) Jointly Optimizing Aesthetics & Text

Fig. 7. Multi-target Concept Guidance simultaneously improves multiple concepts on FLUX.1-dev compared to CFG.

that require high local coherence. Specifically, CoG increases target performance by 2.0% for aesthetics, by 4.7% for text, and by 13.4% for hands. We suspect localized tasks are especially sensitive to guidance accuracy. Our intuition is that CoG’s improvements are caused by a better-aligned update direction. Combined Targets. We then evaluate CoG on multiple targets at once. Here, we only use layers for skipping that increase performance for all targets, and choose those that yield the highest combined gain. We find that while synergies between targets vary, CoG again consistently outperforms CFG (see Table 2). Using only a single layer for guidance is often worse than CFG, indicating that weighted skipping of multiple layers is essential for optimizing several targets. We show qualitative results in Figure 7, where CoG jointly resolves common failure cases. Table 3. Evaluation Metrics. Comparison against APG [38] and PAG [1]. (a) SD3 & SD3.5 Model

Method

SD3

CFG 6.358 CFG+APG [38] 6.524 PAG [1] 6.354 CoG CoG+APG

SD3.5

Aesth. Hands Text

6.494 0.707 0.491 6.600 0.746 0.410

CFG 6.293 CFG+APG [38] 6.462 PAG [1] 6.200 CoG CoG+APG

0.664 0.477 0.590 0.396 0.687 0.481

0.475 0.504 0.535 0.463 0.352 0.491

6.344 0.618 0.514 6.499 0.733 0.476

(b) PixArt & FLUX Model

Method

PixArt-α

CFG 6.622 CFG+APG [38] 7.096 PAG [1] 6.606 CoG CoG+APG

Aesth. Hands Text 0.388 0.399 0.401

– – –

6.658 0.437 7.143 0.499

– –

CFG 6.585 CFG+APG [38] 6.650 FLUX.1-dev PAG [1] 6.652 CoG CoG+APG

0.727 0.608 0.747 0.472 0.626 0.629

6.659 0.754 0.650 6.573 0.738 0.586

12

N. Röhrich et al. Table 4. Auxiliary Metrics.

Table 5. Human Preference Scores favor CoG over CFG on SD3.5 Model Method KDD↓ CLIP↑ LPIPS↑ [43]. Especially when optimizing for aesthetics, CoG yields preferred imCFG 1.589 31.05 0.649 ages. CoGtext – – – PixArt-α CoG 30.81 0.584 hands 2.127 CoGaesth. 1.679 30.98 0.655 Text (↑) Hands (↑) Aesthetics (↑) Mean (↑) SD3

CFG 0.869 31.97 CoGtext 0.961 31.77 CoGhands 0.967 31.77 CoGaesth. 0.955 31.69

0.635 0.650 0.645 0.653

CFG [16] CoGmulti

8.991 8.945

6.298 6.552

7.832 8.277

7.707 7.925

Win Rate

48.4%

62.6%

74.6%

61.87%

Stronger Baselines. We further compare CoG against two state-of-the-art trainingfree guidance methods: Adaptive Projected Guidance (APG) [38] and PerturbedAttention Guidance (PAG) [1]. As APG and PAG improve general guidance behavior while CoG contributes a concept-aware guidance direction, the two are complementary and can be combined (CoG+APG). As shown in Table 3 (a, b), CoG improves over CFG in every setting, and either CoG or CoG+APG is best overall in every cell. Complementarity is clearest for aesthetics and hands, where CoG+APG wins, while CoG alone is better for text. Quality and Diversity. We emphasize that CoG does not aim to improve general generation quality, but rather provides systematic, concept-aware inference-time steering; we therefore report auxiliary metrics to characterize the trade-off between general and concept-specific quality. We use Kernel DINO Distance (KDD) instead of FID, as FID is poorly aligned with perceptual quality for state-of-theart models [44, 50], as well as CLIP-Score and LPIPS (Table 4). CLIP-Score remains close to CFG, LPIPS shows no systematic diversity collapse, and KDD reflects the expected fidelity/control trade-off. Consistently, the HPSv3 scores reported below measure overall generation quality while steering toward a concept, rather than concept quality itself. Human Preference Comparison. We use HPSv3 [30] as a preference-based proxy to probe perceptual trade-offs under target optimization and present absolute values and win-rates vs. CFG in Table 5. We explicitly do not optimize Concept Guidance for HPSv3: we keep the standard, per-target settings and evaluate 5,000 samples per target on SD3.5 [43]. CoG is preferred for Hands (62.6%) and Aesthetics (74.6%), indicating that CoG is preferred by humans, especially when optimizing aesthetics. Text is near parity, indicating a minor trade-off. 4.5

Ablation Studies

Here we summarize three ablations; the detailed discussion, figures (Figures 9 and 10) and tables are provided in Section D. (i) Per-concept layers: selecting skipped layers per concept, rather than using a single fixed layer as in STG [21], improves target performance by up to 24.5% and on average 5.6% (Table 7). (ii) Number of skipped layers: performance improves with k up to a sweet spot around k = 2–3, with most of the gain already obtained from

Concept Guidance

13

a single relevant layer, while larger k eventually introduces interference between layer directions (Figure 10, Table 9); the additional inference cost is therefore opt-in (Section E.3). (iii) Aggregation strategy: performance-weighted aggregation of separate per-layer predictions outperforms both Naive single-pass multi-skipping (as in the HuggingFace SD3 pipeline [42]) and Uniform weighting (Figure 9, Table 8), suggesting that popular diffusion pipelines could be improved by incorporating our method. Finally, CoG is robust to the choice of guidance scale λ (Table 11). Limitations. Concept Guidance applies to concepts with a meaningful scoring signal: either an automatic metric or a VLM-based judge (Section C). Layer profiling is performed once per model/concept pair and should not be assumed to transfer across backbones, so rankings must be recomputed for new models (we release our configurations in Table 6 to avoid this cost for the models we study). At inference, CoG adds one noise prediction per skipped layer, increasing latency with k (Section E.3). In practice, a single layer already yields most of the benefit. Finally, CoG steers a targeted concept rather than improving general generation quality resulting in a potential trade-off.

5

Conclusion

We introduced Concept Guidance, a simple, general, and effective mechanism for precise latent control in text-to-image diffusion models. By identifying and amplifying the influence of concept-specific layers, CoG solves persistent, wellknown failure cases of standard guidance–like misspelled text and malformed hands – but can also optimize for more general concepts like overall aesthetics. Concept Guidance’s key strength is its usability: it is a plug-and-play component that integrates seamlessly with CFG, requires no training, gradients, or external models, and can be added to any existing pipeline with minimal modification. Acknowledgments. This work has been supported by the Horizon Europe project ELLIOT (GA No. 101214398), the German Federal Ministry for Economic Affairs and Energy within the project “NXT GEN AI METHODS – Generative Methoden für Perzeption, Prädiktion und Planung”, the project “GeniusRobot” (01IS24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), and the BMWE ZIM-project (No. KK5785001LO4) “conIDitional LoRA”. The authors gratefully acknowledge the Gauss Center for Supercomputing for providing compute through the NIC on JUWELS/JUPITER at JSC and the HPC resources supplied by the NHR@FAU Erlangen. Furthermore, this work was partially supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space.

14

N. Röhrich et al.

References 1. Ahn, D., Cho, H., Min, J., Jang, W., Kim, J., Kim, S., Park, H.H., Jin, K.H., Kim, S.: Self-rectifying diffusion sampling with perturbed-attention guidance. In: European Conference on Computer Vision. pp. 1–17. Springer (2024) 2. Avrahami, O., Patashnik, O., Fried, O., Nemchinov, E., Aberman, K., Lischinski, D., Cohen-Or, D.: Stable flow: Vital layers for training-free image editing. arXiv preprint arXiv:2411.14430 (2024), https://arxiv.org/abs/2411.14430 3. Bansal, A., Chu, H.M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., Goldstein, T.: Universal guidance for diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 843–852 (2023) 4. Basu, S., Rezaei, K., Kattakinda, P., Rossi, R., Zhao, C., Morariu, V., Manjunatha, V., Feizi, S.: On mechanistic knowledge localization in text-to-image generative models. arXiv preprint arXiv:2405.01008 (2024), https://arxiv.org/abs/2405.01008 5. Basu, S., Zhao, N., Morariu, V., Feizi, S., Manjunatha, V.: Localizing and editing knowledge in text-to-image generative models. arXiv preprint arXiv:2310.13730 (2023), https://arxiv.org/abs/2310.13730 6. Black Forest Labs,: Flux.1-dev. https://huggingface.co/black-forest-labs/FLUX. 1-dev (2024) 7. Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826 (2023), https://arxiv.org/abs/2301.13826 8. Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., Wei, F.: Textdiffuser: Diffusion models as text painters. Advances in Neural Information Processing Systems 36, 9353–9387 (2023) 9. Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al.: Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 (2023) 10. Chung, H., Kim, J., Park, G.Y., Nam, H., Ye, J.C.: Cfg++: Manifold-constrained classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08070 (2024) 11. Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780–8794 (2021) 12. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first international conference on machine learning (2024) 13. Gandikota, R., Materzyńska, J., Zhou, T., Torralba, A., Bau, D.: Concept sliders: Lora adaptors for precise control in diffusion models. In: European Conference on Computer Vision. pp. 172–188. Springer (2024) 14. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022) 15. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020) 16. Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022) 17. Hong, S.: Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. Advances in Neural Information Processing Systems 37, 66743–66772 (2024)

Concept Guidance

15

18. Hong, S., Lee, G., Jang, W., Kim, S.: Improving sample quality of diffusion models using self-attention guidance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7462–7471 (2023) 19. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR 1(2), 3 (2022) 20. Huang, W., Jiang, Y., Van Wouwe, T., Liu, C.K.: Constrained diffusion with trust sampling. arXiv preprint arXiv:2411.10932 (2024), https://arxiv.org/abs/ 2411.10932 21. Hyung, J., Kim, K., Hong, S., Kim, M.J., Choo, J.: Spatiotemporal skip guidance for enhanced video diffusion sampling. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 11006–11015 (2025) 22. Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., Laine, S.: Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems 37, 52996–53021 (2024) 23. Kim, J., Esmaeili, E., Qiu, Q.: Text embedding is not all you need: Attention control for text-to-image semantic alignment with text self-attention maps. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8031– 8040 (2025) 24. Kong, X., Liu, O., Li, H., Yogatama, D., Steeg, G.V.: Interpretable diffusion via information decomposition. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=X6tNkN6ate 25. Li, T., Luo, W., Chen, Z., Ma, L., Qi, G.J.: Self-guidance: Boosting flow and diffusion generation on their own. arXiv preprint arXiv:2412.05827 (2024), https: //arxiv.org/abs/2412.05827 26. Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models. In: European conference on computer vision. pp. 423–439. Springer (2022) 27. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., Lee, J., et al.: Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019) 28. Luo, G., Darrell, T., Wang, O., Goldman, D.B., Holynski, A.: Readout guidance: Learning control from diffusion features. arXiv preprint arXiv:2312.02150 (2023), https://arxiv.org/abs/2312.02150 29. Luo, G., Dunlap, L., Park, D.H., Holynski, A., Darrell, T.: Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems 36, 47500–47510 (2023) 30. Ma, Y., Wu, X., Sun, K., Li, H.: Hpsv3: Towards wide-spectrum human preference score. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15086–15095 (2025) 31. Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 4296–4304 (2024) 32. Narasimhaswamy, S., Bhattacharya, U., Chen, X., Dasgupta, I., Mitra, S., Hoai, M.: Handiffuser: Text-to-image generation with realistic hand appearances. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2468–2479 (2024) 33. Nguyen, Q.H., Phan, H., Doan, K.D.: Unveiling concept attribution in diffusion models. arXiv preprint arXiv:2412.02542 (2024), https://arxiv.org/abs/2412.02542

16

N. Röhrich et al.

34. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021) 35. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 36. Rajabi, J., Mehraban, S., Sadat, S., Taati, B.: Token perturbation guidance for diffusion models. arXiv preprint arXiv:2506.10036 (2025), https://arxiv.org/abs/ 2506.10036 37. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022) 38. Sadat, S., Hilliges, O., Weber, R.M.: Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In: The Thirteenth International Conference on Learning Representations (2024) 39. Sadat, S., Kansy, M., Hilliges, O., Weber, R.M.: No training, no problem: Rethinking classifier-free guidance for diffusion models. arXiv preprint arXiv:2407.02687 (2024), https://arxiv.org/abs/2407.02687 40. Schuhmann, C.: Laion-aesthetics. LAION Blog (August 2022), https://laion.ai/ blog/laion-aesthetics/ 41. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) 42. Stability AI,: Stable diffusion 3 medium. https://huggingface.co/stabilityai/ stable-diffusion-3-medium (2024) 43. Stability AI,: Stable diffusion 3.5 large. https://huggingface.co/stabilityai/ stable-diffusion-3.5-large (2024) 44. Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z., Caterini, A.L., Taylor, E., Loaiza-Ganem, G.: Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems 36, 3732–3784 (2023) 45. Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. Advances in neural information processing systems 36, 1363– 1389 (2023) 46. Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Kumar, K., Stenetorp, P., Lin, J., Ture, F.: What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.04885 (2022), https://arxiv.org/abs/2210.04885 47. Vedhaviyassh, D., Sudhan, R., Saranya, G., Safa, M., Arun, D.: Comparative analysis of easyocr and tesseractocr for automatic license plate recognition using deep learning algorithm. In: 2022 6th International Conference on Electronics, Communication and Aerospace Technology. pp. 966–971. IEEE (2022) 48. WANG, C., Franzese, G., Finamore, A., Gallo, M., Michiardi, P.: Information theoretic text-to-image alignment. In: The Thirteenth International Conference on Learning Representations (2025), https://openreview.net/forum?id= Ugs2W5XFFo 49. Wu, Q., Liu, Y., Zhao, H., Bui, T., Lin, Z., Zhang, Y., Chang, S.: Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7766–7776 (2023) 50. Yang, J., Geng, Z., Ju, X., Tian, Y., Wang, Y.: Representation fr\’echet loss for visual generation. arXiv preprint arXiv:2604.28190 (2026)

Concept Guidance

17

51. Yu, J., Wang, Y., Zhao, C., Ghanem, B., Zhang, J.: Freedom: Training-free energyguided conditional diffusion model. arXiv preprint arXiv:2303.09833 (2023), https: //arxiv.org/abs/2303.09833 52. Zawar, R., Dewan, S., Saxena, P., Chang, Y., Luo, A., Bisk, Y.: Diffusionpid: Interpreting diffusion via partial information decomposition. arXiv preprint arXiv:2406.05191 (2024), https://arxiv.org/abs/2406.05191 53. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3836–3847 (2023)

18

N. Röhrich et al.

Supplementary Material A

Layer Indices and Weights

To facilitate reproducibility and allow future work to apply Concept Guidance without the computational cost of layer profiling, we provide the exact configurations used in our experiments. For each model and target concept, we list the set of top-k layer indices K identified as most impactful, along with their corresponding importance weights ωi . As defined in Section 3.2, the weights represent the performance gain of the skipped layer relative to the baseline. Table 6. Layer Configurations. Top-k skipped layers (K) and their corresponding weights (Ω) for all evaluated models and target concepts. Model PixArt-α

SD3

SD3.5

Task

k Skipped Layers (K)

Human Hands [4]

Weights (Ω) [1.00, 0.88, 0.73, 0.63]

Aesthetics

[3]

[9, 20, 5]

[1.00, 0.28, 0.12]

Visible Text

[3]

[7, 11, 4]

[1.00, 0.55, 0.45]

Human Hands [4]

[9, 6, 5, 10]

[1.00, 0.48, 0.45, 0.21]

Aesthetics

[3]

[6, 7, 8]

[1.00, 0.71, 0.70]

Visible Text

[2]

[15, 10]

[1.00, 0.49]

Human Hands [2]

[7, 18]

[1.00, 0.98]

Aesthetics

[1]

[4]

[1.00]

Visible Text

[3]

[11, 3, 9]

[1.00, 0.85, 0.74]

FLUX.1-dev Human Hands [3] Aesthetics

B

[3, 23, 8, 18]

[3]

[9, 12, 15]

[1.00, 0.81, 0.78]

[15, 14, 13]

[1.00, 0.18, 0.09]

Convergence Analysis

Empirically, as demonstrated in Section 4.5, we find that CoG approximates the latent space direction of highest target performance y more precisely for a larger k, at least until a certain threshold. Here, we formalize our intuition and prove that under idealized assumptions, CoG predictions converge to y as the number of skipped layers goes to infinity. In practice, the number of skippable layers is of course bounded by the model architecture. Lemma 1. Let di = αi y + ri be the noise prediction of layer i, where y is the unit ground-truth direction, αi represents signal strength, and ri is a zero-mean residual vector. Let K be the set of k layers selected by CoG. The normalized CoG estimator P di ybk = Pi∈K i∈K di converges to the true direction y as the number of selected layers k → ∞, provided that the expected signal strength of selected layers is positive, i.e. E[αi | i ∈ K] = µ > 0.

Concept Guidance

19

P Proof. Let sk = k1 i∈K di be the mean vector of the selected layers. Substituting the decomposition di = αi y + ri into the sum, we have: ! ! 1X 1X sk = αi y + ri . k k i∈K

i∈K

We analyze the asymptotic behavior of the two terms on the right-hand side as k → ∞ by applying the Law of Large Numbers (LLN): 1. Signal Term: The coefficient of y is the sample mean of the signal strengths αi . By the LLN, this converges to the conditional expectation of α: 1X k→∞ αi −−−−→ µ. k i∈K

2. Residual Term: The second term is the sample mean of the independent residual vectors ri . Since E[ri ] = 0, the LLN dictates: 1 X k→∞ ri −−−−→ 0. k i∈K

Combining these results, the unnormalized estimator converges to the scaled ground truth: k→∞ sk −−−−→ µy. Since µ > 0, the magnitude ∥sk ∥ converges to µ∥y∥ = µ. Finally, the normalized estimator ybk satisfies: sk k→∞ µy ybk = −−−−→ = y. ∥sk ∥ µ

C

Optimizing Arbitrary Concepts with VLMs.

While we are bound to concepts for which valid metrics exist when reporting results in the experiment section, we find that using vision-language models (VLMs) as judges for layer-finding makes Concept Guidance available for arbitrary concepts, even when no valid metrics exist. We demonstrate the broad applicability of Concept Guidance by optimizing for the following additional targets (Figure 8): – Symmetry and Geometric Regularity: We optimize for images whose main subject displays strong structural symmetry and an overall regular geometric arrangement. This property is relevant across portraits, architecture, and design, yet it is difficult to measure. – Background Separation: We target images where the subject is clearly separated and visually dominant. Subject-background separation is a key component in product photography, close-up shots, and portraits.

20

N. Röhrich et al.

– Ukiyo-e Style: We optimize for images that match the visual characteristics of traditional Japanese ukiyo-e woodblock prints. This shows that Concept Guidance can target highly specific artistic styles.

Optimizing for symmetry and geometric regularity with Concept Guidance...

Optimizing for background separation with Concept Guidance...

Optimizing for Ukiyo-e style with Concept Guidance...

Fig. 8. Comparison of Classifier-Free Guidance [16] and Concept Guidance for additional targets where layers are profiled using a VLM (InternVL3-14B).

Concept Guidance

21

Table 7. Leveraging per-concept layer information yields significant gains over guiding with a fixed layer as is the case in Spatio-Temporal Skip-Guidance.

D

Model

Method

Text (↑)

PixArt-α

STG [21] CoGsingle

-

SD3

STG [21] CoGsingle

SD3.5 FLUX.1-dev

Hands (↑)

Aesthetics (↑)

0.384 0.478 ▲24.48%

6.502 6.606 ▲1.6%

0.441 0.462 ▲4.8%

0.662 0.664 ▲0.3%

6.359 6.359 ▲0.0%

STG [21] CoGsingle

0.531 0.564 ▲6.2%

0.540 0.603 ▲11.7%

6.297 6.344 ▲0.7%

STG [21] CoGsingle

0.490 0.500 ▲2.0%

0.595 0.643 ▲8.1%

6.724 6.865 ▲2.1%

Additional Ablation Results

This section provides the detailed ablation studies summarized in Section 4.5. Leveraging Concept-Wise Layers To measure the effect of leveraging different layers for guidance per concept, we compare CoG against STG [21], where a fixed single layer is skipped based on overall generation quality, regardless of the given concept. While STG is a method for video generation guidance, we port the approach to text-to-image generation by choosing the best overall layer based on FID scores. To demonstrate the effect achieved by per-concept information alone, we do not leverage performance-weighted multi-skipping, i.e. we use CoGsingle only. We find that incorporating per-layer, per-concept knowledge when choosing the skipped layer yields significant performance increases of up to 24.5%, and an average increase of 5.6% (Table 7). Number of Skipped Layers The idea behind CoG is to approximate the direction of highest target performance y more accurately for a larger k, which we also demonstrate theoretically in our convergence analysis. However, we also expect that in practice, there is a tradeoff between increased accuracy and negative synergies between multiple noise predictions for a larger k. In particular, we find that while multiple layers can each be beneficial, their resulting predictions may diverge. Thus, skipping too many layers with CoG can result in guiding with possibly conflicting noise predictions. To find the optimal tradeoff, we evaluate models with up to 5 skipped layers, and measure their impact on task-specific metrics. As shown in Figure 10 and Table 9, performance indeed generally improves with more skipped layers up to a certain point, with the optimal tradeoff being usually achieved around 2–3 skipped layers. Notably, significant gains are already achieved by skipping a single, concept-relevant layer. Multiple Layers and Contribution Weighting To demonstrate the benefit of computing weighted averages of per-layer noise predictions, we compare our method

22

N. Röhrich et al.

Table 8. Performance-weighted aggregation is key for Concept Guidance. We compare skipping all layers in the same pass (Naive) equal weighting of multiple predictions (Uniform), and performance-based weighting (CoG). Model

Guidance Method

Metric

Table 9. Increasing k trades off gains vs. interference, with a clear sweet spot. We sweep k (top-k improving layers); settings with fewer than five improving layers omit larger-k results. Across architectures, the optimum is approximately k = 3. Model

Number of Layers

Metric 1

Naive Uniform CoG (ours)

2

3

4

5

Hands (↑) 0.473 Aesthetics (↑) 6.680

0.484 0.502 0.535 0.501 6.709 6.729 -

SD3

Text (↑) 0.428 Hands (↑) 0.632 Aesthetics (↑) 6.194

0.445 0.455 0.444 0.452 0.666 0.653 0.675 0.704 6.257 6.259 6.191 6.169

0.610 0.679 6.361

SD3.5

Text (↑) 0.536 0.582 0.554 Hands (↑) 0.627 0.692 0.684 Aesthetics (↑) 6.376 6.295 6.309

0.510 0.675 6.657

Text (↑) 0.481 0.657 FLUX.1-dev† Hands (↑) Aesthetics (↑) 6.652

PixArt-α

Hands (↑) Aesthetics (↑)

0.481 6.470

0.482 6.714

0.509 6.722

PixArt-α

SD3

Text (↑) Hands (↑) Aesthetics (↑)

0.453 0.571 6.009

0.477 0.669 6.338

0.490 0.684 6.348

SD3.5

Text (↑) Hands (↑) Aesthetics (↑)

0.575 0.664 6.321

0.591 0.677 6.368

Text (↑) FLUX.1-dev Hands (↑) Aesthetics (↑)

0.483 0.159 6.628

0.478 0.666 6.607

†

0.539 0.649 6.291

0.548 0.665 6.295

0.478 0.509 0.491 0.667 0.670 0.655 6.654 6.662 -

0.487 0.641 -

against two additional baselines. For Uniform skip-guidance, separate noise predictions are computed per layer but are not weighted based on performance. In Naive skip-guidance, multiple layers are skipped within the same forward pass. Notably, Naive corresponds to the current implementation in the HuggingFace SD3 pipeline [42]. We hypothesize that Naive leads to degraded images due to strong manifold distortions induced by skipping several layers simultaneously. Moreover, we expect that without weighting, models underperform due to the lack of directional control on the noise manifold. Our experiments provide quantitative (Table 8) and qualitative (Figure 9) evidence that separate noise predictions are necessary and that contribution based weighting improves performance. Notably, our findings indicate that popular diffusion pipelines could be significantly improved by incorporating our method of computing separate predictions and combining them through weighted aggregation.

Naive

Uniform

Number of Layers

CoG

CFG

CoGsingle

CoGmulti

Fig. 9. Guidance Method Ablation. Fig. 10. Number of Layers Ablation.

Concept Guidance

E

Implementation Details

E.1

Code Availability

23

Our implementation of Concept Guidance, together with the layer profiling code and the per-model, per-concept layer configurations reported in Table 6, is publicly available at https://github.com/CompVis/concept_guidance. E.2

Computational Resources

For all experiments, models, and tasks, we use nodes of four NVIDIA A100 GPUs with 80 GBs of VRAM. E.3

Inference and Profiling Cost

Concept Guidance introduces a one-time, offline profiling stage and a small online inference overhead. For profiling, we use N = 100 prompts per concept. For example, for SD3 with L = 24 layers this amounts to N × L = 2400 generations and completes in ∼ 4 hours on a single A100. Profiling is performed only once per model/concept pair (Algorithms 1 and 2), and we release the resulting layer configurations (Table 6) so that CoG can be applied directly, without any profiling, for the models and concepts we study. At inference, the cost scales linearly with the number of skipped layers k, since each skipped layer requires one additional noise prediction (Table 10). The additional cost per skipped layer ranges from 0.4 s for PixArt-α to 12.4 s for FLUX.1-dev, i.e. it grows with model size; relative to a single CFG pass this corresponds to roughly 10% for PixArt-α and up to nearly a full additional pass for FLUX.1-dev. As shown in Table 1, significant gains are already obtained with a single skipped layer (CoGsingle ), so the overhead is opt-in and can be tuned to the desired level of concept control. E.4

Guidance Scales

Classifier-Free Guidance For CFG [16], we use the standard guidance scale per model. For Flux.1-dev [6], the default guidance scale is 3.5, for PixArt-α [9], it is 4.5, and for both Stable Diffusion 3 and 3.5 [42, 43], the default scale is 7.0. Table 10. Inference time (s) on an A100 GPU. Model PixArt-α Stable Diffusion 3 Stable Diffusion 3.5 FLUX.1-dev

CFG [16]

CoGsingle (k =1)

CoGmulti (k =2)

+Sec / Layer

3.5 4.4 5.9 12.9

3.9 6.2 7.7 25.3

4.3 8 9.5 37.7

0.4 1.8 1.8 12.4

24

N. Röhrich et al.

Concept Guidance To measure individual layer performance with residual skipping, we use a guidance scale of 2.0, as used in single-skip residual STG [21]. For CoG, we sweep over guidance scales ranging from 1.25 to 3.00 in 0.25 steps, and find that a guidance scale between 2.0 and 2.5 generally works best to achieve the results reported in Table 1. Full results are reported in Table 11. Table 11. CoG Guidance Scale Sweep. For most models and tasks, the optimal guidance scale falls in the range between 2.0 and 2.5. † PixArt-α does not generate any visible text. Model

Metric

Guidance Scale (λ) 1.25 1.50 1.75 2.00 2.25 2.50 2.75 3.00

Text (↑) PixArt-α

†

Hands (↑)

-

-

-

-

-

-

-

-

0.396 0.373 0.418 0.504 0.503 0.496 0.492 0.468

Aesthetics (↑) 6.647 6.678 6.695 6.723 6.716 6.759 6.740 6.734 SD3

Text (↑)

0.482 0.472 0.486 0.489 0.506 0.481 0.480 0.474

Hands (↑)

0.602 0.631 0.660 0.670 0.640 0.662 0.605 0.629

Aesthetics (↑) 6.169 6.198 6.250 6.248 6.257 6.260 6.217 6.241 SD3.5

Text (↑)

0.561 0.569 0.581 0.600 0.595 0.606 0.602 0.598

Hands (↑)

0.603 0.628 0.636 0.634 0.653 0.633 0.645 0.628

Aesthetics (↑) 6.324 6.337 6.349 6.419 6.388 6.378 6.351 6.356 Text (↑) FLUX.1-dev Hands (↑)

0.502 0.509 0.521 0.515 0.524 0.521 0.515 0.503 0.642 0.666 0.695 0.617 0.617 0.620 0.635 0.638

Aesthetics (↑) 6.663 6.711 6.766 6.753 6.789 6.779 6.785 6.815

E.5

Conditional Prompts

To ensure optimal diversity of conditional prompts, we generate prompts with different large language models. Specifically, we encourage LLMs to cover a wide range of prompts with respect to length, complexity and general theme. For instance, in text generation, we make sure that prompts contain a varying count of text instances, and that text within these instances has varying complexity. For text specifically, we find that strong models like FLUX.1-dev seldom struggle with prompts containing very simple text, thus we slightly adjust the complexity to model strength for evaluation. We provide sample prompts from our datasets for text (Figure 11), hands (Figure 12) and aesthetics (Figure 13).

Concept Guidance

"A squirrel scientist in a cluttered forest lab. Beakers bubble beside a chalkboard that reads 'NUTRIUM FORMULA v3'. The squirrel wears goggles and a lab coat with a stitched name tag: 'Dr. Acornstein'.", "A construction worker operating a crane wearing a helmet labeled 'SAFETY FIRST'", "A steam-powered mech piloted by a mouse. On the mech's chestplate is etched 'The Brass Claw'. Gauges display labels like 'pressure max' and 'oil level low'. The cockpit has sticky notes with scribbled text.", "A sign that says 'OPEN' in a bright, glowing neon font.", "a notebook with 'My Ideas' on the cover", "A green highway sign that reads 'New York - 50 Miles'.", "A detective’s office in the 1940s. The frosted glass door says 'MARLOWE INVESTIGATIONS'. A rotary phone sits on a desk beside a file labeled 'Case #23 – Vanishing Heiress'.", "A sushi chef slicing fish behind a counter. The sign above reads 'SATO’S SUSHI BAR'. A menu board lists 'Omakase – ¥12,000'.", "A team jersey with 'Tigers' in a classic, athletic, block font with an outline.", "A haunted attic filled with glowing artifacts. A dusty mirror has scrawled words: 'DON'T LOOK TWICE'. A book floats mid-air, its spine labeled 'Rituals of the Veil'. Faint writing glows on the floorboards."

Fig. 11. Prompt Samples for Text Generation.

"Portrait of a young woman sitting in a cozy cafe, warm lighting, holding a coffee cup", "A woman waving, both hands visible", "A musician playing an acoustic guitar on stage, human hands visible strumming the guitar strings", "An elderly person holding a book, old, wrinkled hands visible", "A single, normal hand, anatomically correct", "A mechanic fixing an engine, greasy, calloused hands", "A person showing approval, giving a 'thumbs up'", "A person standing, human hands visible with fingers interlaced behind their head" "A hand with firm, well-defined fingers, anatomically correct", "A person typing on a laptop, both hands visible on the keyboard"

Fig. 12. Prompt Samples for Hand Generation.

"A timeless portrait of a farmer tending golden wheat fields, backlit by the warm glow of sunset with dust particles dancing in the air", "a magnificent castle", "an atmospheric rainy city scene with glistening wet streets reflecting colorful neon lights and a moody, cinematic ambiance", "A majestic mountain", "a photorealistic portrait of a woman, 8k, uhd, studio lighting, detailed skin texture", "A majestic clock tower dominating a historic town square, captured during blue hour with warm illuminated clock face and architectural details", "A lush flower garden bursting with rich, saturated colors and delicate, intricate petals under gentle sunlight", "a beautiful abstract depiction of volumetric light, god rays, ethereal", "a sleek futuristic robot in a high-tech laboratory, dramatically lit with cool blue tones and atmospheric volumetric lighting", "an elegant dress"

Fig. 13. Prompt Samples for Aesthetics.

25

26

N. Röhrich et al.

E.6

Algorithms

We implement CoG according to Algorithm 1 and layer profiling according to Algorithm 2. Notably, once layers have been found using layer profiling, CoG can be added into existing pipelines by exchanging the layer-wise forward with a conditional forward that returns the input if for the current layer i, i ∈ K, according to Equation (9), and implementing guidance with the weighted negative noise prediction according to Equations (13) and (14).

Algorithm 1 Concept Guidance (CoG) Step Require: xt , t, c, K, Ω, CoG Scale λ, CFG Scale γ 1: ϵunc , ϵcond ← ϵθ (xt , t, ∅), ϵθ (xt , t, c) 2: ϵθ ← ϵunc + γ(ϵcond − ϵunc ) 3: ϵsum ← 0, W ← 0 4: for i ∈ K do 5: ϵskip ← ϵ[θ\i] (xt , t, c) 6: ϵsum ← ϵsum + ωi · ϵskip 7: W ← W + ωi 8: end for 9: ϵneg ← ϵsum /W 10: return (1 − λ)ϵneg + λϵθ

▷ Standard CFG ▷ Aggregate Skipped Predictions

▷ Apply CoG

Algorithm 2 CoG Layer Profiling Require: Model θ, Prompts C, Metric Φ(·), Top-k count 1: for c ∈ C do ▷ Compute Baseline Performance 2: x0 ← Sample(θ, c) 3: p∅ ← Φ(x0 ) 4: end for 5: p∅ ← p∅ /|C| 6: L ← [ ] ▷ Initialize Impact List 7: for layer i ∈ {1, . . . , L} do ▷ Compute Layer Impact 8: for c ∈ C do 9: x0 ← Sample(θ\i , c) 10: pi ← Φ(x0 ) 11: end for 12: pi ← pi /|C| 13: ωi ← max(0, pi − p∅ ) 14: Append (i, ωi ) to L 15: end for 16: Sort L by ωi descending ▷ Select Top Layers 17: K ← {i | (i, ωi ) ∈ L[: k]} 18: Ω ← {ωi | i ∈ K} 19: return K, Ω

Concept Guidance

F

27

Uncurated Samples

Complementing the qualitative results in Figures 5 and 6, we show uncurated samples for the typical failure case of text generation and for the general task of generating more aesthetic images. For both tasks, we choose random seeds and generate images with Classifier-Free Guidance [16], Naive Skip-Guidance (as found in SD3 [42]), Spatio-Temporal Skip-Guidance [21], and Concept Guidance, using the same seed for each method. Uncurated samples generated with Flux.1dev are shown in Figure 14 for text, and in Figure 15 for aesthetics.

Classifier-Free Guidance

Naive (SD3 Skip-Guidance)

Spatio-Temporal Skip-Guidance

Concept Guidance

Fig. 14. Uncurated Comparison of Classifier-Free Guidance (CFG) [16], Naive SkipGuidance as found in Stable Diffusion 3 [42], Spatio-Temporal Skip Guidance [21] and Concept Guidance for Text Generation (FLUX.1-dev).

28

N. Röhrich et al.

Classifier-Free Guidance

Naive (SD3 Skip-Guidance)

Spatio-Temporal Skip-Guidance

Concept Guidance

Fig. 15. Uncurated Comparison of Classifier-Free Guidance (CFG) [16], Naive SkipGuidance as found in Stable Diffusion 3 [42], Spatio-Temporal Skip Guidance [21] and Concept Guidance for Aesthetics (FLUX.1-dev).

Related documents

• SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention#155595
• Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning#820369
• RefCon: Iterative Refinement and Contrastive Memory Extraction for Context-Evolving Agent#1144110
• SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations#1144602
• From Skill Acquisition to Language Control#487827