Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment arXiv:2606.16799v1 [cs.CV] 15 Jun 2026
Zijie Meng Peking University, [email protected]
Abstract—Existing vision-language model (VLM)-based AI-generated image quality assessment (AIGIQA) methods suffer from a fundamental semantic-distortion dimensional conflict: monolithic representations optimized for semantic discrimination inherently entangle compositional understanding with low-level perceptual sensitivity, rendering them blind to fine-grained quality degradations. We introduce MST-CLIPIQA, a multi-scale two-stream framework that achieves hierarchical vision-language alignment through explicit representational decoupling. Our architecture leverages dual CLIP encoders with complementary patch granularities—coarse-grained streams capture global semantic coherence while fine-grained streams preserve textural signatures and artifact patterns. An information bottleneck-inspired gated fusion mechanism performs adaptive cross-scale distillation, with optional cross-attention enabling prompt-anchored correspondence evaluation when generation prompts are available. Extensive experiments across five benchmarks establish new state-of-the-art results, achieving average improvements of 1.11% SRCC on quality and 2.35% SRCC on text-image correspondence prediction, while maintaining efficiency with only ∼0.8M trainable parameters. Our project is available at https://github.com/ YMlinfeng/MST-CLIPIQA. Index Terms—AI-generated image quality assessment, IQA, vision-language models, multi-scale feature extraction, gated feature fusion
I. Introduction The rapid proliferation of AI-generated images (AIGIs) has introduced a fundamentally distinct quality assessment paradigm where perceptual fidelity, semantic authenticity, and text-image correspondence constitute inseparable quality dimensions that transcend conventional distortion-centric evaluation [1]–[3]. VisionLanguage Models (VLMs) pre-trained on massive imagetext corpora, most notably CLIP [4], have demonstrated remarkable zero-shot transfer capabilities by leveraging rich semantic priors [5,6]. However, a fundamental semanticdistortion dimensional conflict persists: the representational geometry of these models is inherently optimized for high-level semantic discrimination rather than low-level perceptual sensitivity [7,8]. This renders them systemati-
cally blind to fine-grained texture degradations, localized artifacts, and subtle generative anomalies that critically influence human quality judgments. This dimensional conflict originates from a scalesemantic entanglement inherent in single-scale visual representations. Human quality perception operates hierarchically: coarse-grained gestalt processing governs global coherence evaluation while fine-grained scrutiny detects local artifacts [9]. Yet monolithic feature extraction conflates these perceptual hierarchies into an undifferentiated embedding that sacrifices sensitivity at both ends of the spatial spectrum. The natural remedy of multiscale feature extraction introduces a secondary challenge of cross-scale information redundancy, where substantial representational overlap between adjacent granularities dilutes quality-discriminative signals when processed through conventional fusion strategies such as concatenation or learned linear combination [10]. This demands principled mechanisms for selective information distillation that contemporary architectures conspicuously lack. Furthermore, AIGIs uniquely exhibit generative semantic misalignment, including anatomically implausible compositions, physically impossible configurations, and promptcontent discordance. Standard VLM embeddings trained on naturalistic image-text pairs fundamentally struggle to encode such degradations, necessitating explicit crossmodal reasoning that leverages generation prompts as privileged semantic anchors. To systematically address these intertwined challenges, we propose MST-CLIPIQA, a Multi-Scale Two-stream framework that achieves hierarchical vision-language geometric alignment through explicit decoupling and subsequent reconciliation of semantic understanding with perceptual sensitivity. Our Multi-Scale Two-Stream Feature Extraction (MSTFE) architecture constructs complementary processing pathways at distinct spatial granularities: coarse-grained streams capture global semantic coherence through efficient long-range contextual aggregation, while fine-grained streams preserve high-frequency textural sig-
TABLE I Performance comparison on authenticity score prediction.
AIGCIQA2023
PKU-AIGIQA-4K
SRCC
PLCC
SRCC
PLCC
LinearityIQA [11] MUSIQ [12] HyperIQA [13] StairIQA [14] MANIQA [15] LIQE [16] AMFF-Net♯ [17] CLIP-AGIQA♯ [18]
0.6710 0.7185 0.7060 0.7352 0.7829 0.8010 0.7749 0.7940
0.6640 0.7152 0.6971 0.7347 0.7704 0.7893 0.7643 0.7797
0.6427 0.6348 0.7093 0.6835 0.2559 0.7823 -
0.6305 0.5999 0.6964 0.6891 0.2553 0.7805 -
MST-CLIPIQA MST-CLIPIQA*
0.8149 0.8170
0.8026 0.8049
0.7919 0.7993
0.7905 0.8003
Method
natures and localized artifact patterns. This design exploits the native patch-based tokenization flexibility of modern vision encoders without incurring explicit image pyramid construction overhead. Notably, our architecture remains agnostic to specific encoder implementations, enabling seamless integration with any lightweight backbone. The central innovation lies in our Gated Feature Fusion (GFF) module, which implements information bottleneck-guided cross-scale selective gating. A learnable gating network dynamically computes perdimensional selection coefficients that adaptively interpolate between multi-scale contributions, effectively compressing the joint representation to maximally preserve quality-predictive mutual information while filtering scalespecific redundancies. This achieves a principled balance between representational completeness and discriminative compactness with linear computational complexity. When generation prompts are available, a lightweight crossattention mechanism further grounds quality predictions in text-image semantic correspondence. Extensive experiments across multiple AIGI quality benchmarks demonstrate that MST-CLIPIQA consistently achieves state-of-the-art performance, with particularly pronounced improvements on challenging cases involving subtle generative artifacts and complex prompt-image semantic relationships. Our contributions are fourfold: • We formalize the semantic-distortion dimensional conflict in VLM-based IQA and propose a multiscale two-stream architecture (MSTFE) that explicitly decouples global semantic understanding from local perceptual sensitivity through complementary spatial granularity processing. • We introduce Gated Feature Fusion (GFF), an information bottleneck-inspired selective cross-scale gating mechanism that achieves adaptive quality-aware feature distillation with minimal computational overhead, substantially outperforming conventional fusion strategies.
We present a prompt-anchored cross-modal alignment framework that elevates generation prompts to semantic quality references, enabling explicit textimage correspondence verification for AIGI-specific quality dimensions. • We establish new state-of-the-art results on AGIQA3K [1], AIGCIQA2023 [2], and AIGIQA-20K [3], demonstrating robust generalization across diverse generative paradigms from diffusion-based synthesis to GAN-generated imagery. •
II. Method A. Overview Given an input image I ∈ R 𝐻 ×𝑊 ×3 with spatial resolution 𝐻 × 𝑊 and an optional generation prompt T = 𝐿 {𝑡𝑖 }𝑖=1 of length 𝐿, MST-CLIPIQA predicts a perceptual quality score 𝑞ˆ ∈ R through three cascaded stages. As illustrated in Fig. 1, our framework first employs Multi-Scale Two-Stream Feature Extraction (MSTFE) to construct scale-decoupled representations that resolve the semantic-distortion dimensional conflict (see Appendix A for theoretical analysis). Let E 𝜃 denote a frozen vision encoder parameterized by 𝜃 and T (𝑠) a granularity-specific tokenization operator with patch size 𝑃𝑠 , the dual-stream features are obtained as: F (𝑐) = E 𝜃 T (𝑐) (I) ∈ R 𝑁𝑐 ×𝐷 (1) (𝑓) (𝑓) 𝑁 𝑓 ×𝐷 F = E 𝜃 T (I) ∈ R (2) where 𝑁 𝑠 = 𝐻𝑊/𝑃𝑠2 denotes the token count for stream 𝑠 ∈ {𝑐, 𝑓 } and 𝐷 is the embedding dimension. The coarse-grained stream (𝑃𝑐 > 𝑃 𝑓 ) captures global semantic coherence while the fine-grained stream preserves local textural details. Subsequently, Gated Feature Fusion (GFF) adaptively combines multi-scale representations through learned per-dimensional interpolation z = GFF(f (𝑐) , f ( 𝑓 ) ), filtering task-irrelevant redundancies. Finally, a lightweight regression head maps the fused representation to 𝑞. ˆ We denote our model as MST-CLIPIQA when operating without text prompts, where the fused visual features are compared against learnable quality-aware text templates. When generation prompts are available, we denote the variant as MST-CLIPIQA*, which additionally incorporates cross-modal attention to explicitly verify promptcontent alignment. B. Multi-Scale Two-Stream Feature Extraction The MSTFE module exploits the native patch-based tokenization flexibility of vision-language encoders to construct scale-decoupled representations without explicit image pyramid construction. For an input image I, the tokenization operator T (𝑠) with patch size 𝑃𝑠 partitions it into 𝑁 𝑠 non-overlapping patches, each flattened and projected 2 via E 𝑝 ∈ R𝐷× ( 𝑃𝑠 ·3) into 𝐷-dimensional embeddings.
Fig. 1. The pipeline of the proposed CLIP-based multi-scale two-stream framework, which consists of five key modules: text feature extraction, multi-scale two-stream image feature extraction, gated feature fusion, text-image cross-attention, and score regression.
(a) LinearityIQA
(b) MUSIQ
(c) HyperIQA
(d) StairIQA
(e) MANIQA
(f) LIQE
(g) MST-CLIPIQA
(h) MST-CLIPIQA*
Fig. 2. Scatter plots of different IQA methods tested on the AGIQA-1K database. The curves are obtained by a three-order polynomial nonlinear fitting.
(𝑠) Together with a prepended [CLS] token xcls ∈ R𝐷 and (𝑠) ( 𝑁 +1) ×𝐷 𝑠 learnable positional embeddings Epos ∈ R , the in(𝑠) (𝑠) put sequence X (𝑠) = [xcls ; E 𝑝 · Flatten(T (𝑠) (I)) + Epos ]∈ ( 𝑁 +1) ×𝐷 R 𝑠 is processed through 𝑀 stacked transformer blocks. The stream-specific global features are extracted
from the final [CLS] representations: f (𝑐) = E 𝜃 (X (𝑐) ) [0],
f ( 𝑓 ) = E 𝜃 (X ( 𝑓 ) ) [0] ∈ R𝐷
(3)
where [·] [0] denotes extraction of the first token ([CLS]) from the output sequence.
TABLE II Comparison of multi-scale feature fusion methods.
Method
AGIQA-1K
AGIQA-3K
Quality
Quality
AIGCIQA2023 Quality
Average
Authenticity
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
Linear Interpolation AdaptiveAvgPool AdaptiveMaxPool Cross Attention
0.9031 0.8984 0.8938 0.8944
0.9094 0.9090 0.9112 0.9024
0.9062 0.9072 0.9065 0.9053
0.9244 0.9265 0.9256 0.9249
0.8641 0.8606 0.8615 0.8561
0.8731 0.8780 0.8767 0.8759
0.8032 0.8103 0.8011 0.8033
0.7937 0.7996 0.7889 0.7925
0.8692 0.8691 0.8657 0.8648
0.8752 0.8783 0.8756 0.8739
GFF (Ours)
0.8990
0.9149
0.9091
0.9282
0.8701
0.8835
0.8149
0.8026
0.8733
0.8823
TABLE III Comparison of MSTFE architecture variants.
Architecture
MSTFE-1 MSTFE-2 (Ours)
AGIQA-1K
AGIQA-3K
Quality
Quality
AIGCIQA2023 Quality
Average
Authenticity
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
0.8977 0.8990
0.9071 0.9149
0.9106 0.9091
0.9289 0.9282
0.8637 0.8701
0.8780 0.8835
0.8107 0.8149
0.7960 0.8026
0.8707 0.8733
0.8775 0.8823
TABLE IV Performance comparison on quality score prediction. Best results are in bold.
Method
AGIQA-1K
AGIQA-3K
AIGCIQA2023
AIGIQA-20K
PKU-AIGIQA-4K
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
LinearityIQA [11] MUSIQ [12] HyperIQA [13] StairIQA [14] MANIQA [15] LIQE [16] MA-AGIQA♯ [19]
0.8200 0.8506 0.8534 0.8640 0.8804 0.8927 -
0.8578 0.8850 0.8912 0.8899 0.9084 0.9117 -
0.8189 0.8338 0.8495 0.8543 0.8916 0.9009 0.8939
0.8309 0.8698 0.8923 0.8943 0.9194 0.9220 0.9273
0.7947 0.8261 0.8159 0.8313 0.8412 0.8608 -
0.8057 0.8382 0.8212 0.8376 0.8540 0.8774 -
0.7419 0.8344 0.8174 0.7911 0.8553 0.8655 0.8644
0.6838 0.8678 0.8417 0.8435 0.8891 0.8934 0.9050
0.6493 0.6801 0.7144 0.7247 0.7800 0.8030 -
0.6056 0.6697 0.7180 0.7145 0.7800 0.8001 -
MST-CLIPIQA MST-CLIPIQA*
0.8990 0.9091
0.9149 0.9180
0.9091 0.9085
0.9282 0.9283
0.8701 0.8608
0.8835 0.8754
0.8803 0.8936
0.9078 0.9149
0.8289 0.8261
0.8176 0.8180
The dual-stream design explicitly instantiates the hierarchical nature of human quality perception. Varying the patch size 𝑃𝑠 induces fundamentally different receptive field characteristics: larger patches (𝑃𝑐 ) enforce spatial pooling that emphasizes compositional structure and semantic plausibility, directly addressing global coherence evaluation. Conversely, smaller patches (𝑃 𝑓 ) preserve finegrained spatial locality, encoding textural patterns, edge sharpness, and localized artifact signatures that conventional VLM features systematically neglect. This transforms the entangled monolithic representation into a disentangled dual-component structure: f (𝑐) captures “what is depicted” while f ( 𝑓 ) encodes “how well it is rendered.”
C. Gated Feature Fusion Given the dual-stream features f (𝑐) , f ( 𝑓 ) ∈ R𝐷 , the GFF module learns to selectively combine multi-scale information through adaptive per-dimensional gating. We first concatenate the features and compute gate coefficients via a two-layer gating network with hidden dimension 𝐷 ℎ : g = 𝜎 W𝑔 · ReLU Wℎ [f (𝑐) ; f ( 𝑓 ) ] + bℎ + b𝑔 ∈ [0, 1] 𝐷 (4) where Wℎ ∈ R𝐷ℎ ×2𝐷 and W𝑔 ∈ R𝐷×𝐷ℎ are learnable projection matrices, bℎ ∈ R𝐷ℎ and b𝑔 ∈ R𝐷 are bias terms, [·; ·] denotes concatenation, and 𝜎(·) denotes the sigmoid function. The gate vector g determines the contribution of each scale at every feature dimension. The fused representation is then computed through element-
wise interpolation: z = g ⊙ 𝜙 𝑐 (f (𝑐) ) + (1 − g) ⊙ 𝜙 𝑓 (f ( 𝑓 ) ) ∈ R𝐷
(5)
where ⊙ denotes the Hadamard (element-wise) product, 1 ∈ R𝐷 is an all-ones vector, and 𝜙 𝑐 (·) = W𝑐 (·) + b𝑐 , 𝜙 𝑓 (·) = W 𝑓 (·) + b 𝑓 with W𝑐 , W 𝑓 ∈ R𝐷×𝐷 are streamspecific affine projections that align feature distributions before fusion. When 𝑔𝑖 → 1, dimension 𝑖 predominantly reflects coarse-grained semantics; when 𝑔𝑖 → 0, finegrained textural information dominates. MST-CLIPIQA (without prompts). When generation prompts are unavailable, we compute quality scores by measuring the similarity between z and a set of learnable quality-level text embeddings {e 𝑘 } 𝐾 𝑘=1 derived from templates (e.g., “a photo of {quality} quality”), followed by softmax-weighted aggregation. MST-CLIPIQA* (with prompts). When prompts T are available, we augment z with text-image correspondence through cross-modal attention: z′ = z + 𝛼 · CrossAttn(z, Etext (T)), where Etext (·) is the frozen text encoder and 𝛼 is a learnable scalar initialized to zero for stable training. This enables explicit verification of prompt-content alignment. Detailed formulations of the cross-attention mechanism are provided in Appendix B. The final quality score is obtained through a regression ′ head with residual connection: 𝑞ˆ = w⊤ 𝑜 (GELU(W1 z ) + ′ 𝐷×𝐷 W2 z ) + 𝑏 𝑜 , where W1 , W2 ∈ R are projection matrices, w𝑜 ∈ R𝐷 and 𝑏 𝑜 ∈ R are output weights and bias. We optimize the model using a composite loss L = LMSE + 𝜆Lrank that jointly minimizes prediction error and enforces pairwise ranking consistency, where 𝜆 controls the relative weight of the ranking term. The vision encoder E 𝜃 remains frozen throughout training, with only the lightweight fusion and regression parameters (∼0.8M) being updated. Complete training details including hyperparameter settings are provided in Appendix C. III. Experiment A. Databases and Experiment Settings Databases. We evaluate on five AIGIQA benchmarks: AGIQA-1K [20] (1,080 images), AGIQA-3K [21] (2,982 images), AIGCIQA2023 [22] (2,400 images), AIGIQA20K [23] (14,000 training images used), and PKUAIGIQA-4K [24] (4,000 images). These databases provide MOS annotations for quality, authenticity, and text-image correspondence. Evaluation Criteria. We adopt Spearman rank correlation coefficient (SRCC) for monotonicity and Pearson linear correlation coefficient (PLCC) for accuracy. Implementation Details. Experiments are conducted on NVIDIA A40 with PyTorch 1.11.0. We use Adam optimizer [25] with learning rate 5 × 10−6 , weight decay 1×10−3 , and batch size 8. Datasets are split 4:1 for training and testing.
TABLE V Performance comparison on text-image correspondence score prediction. Method
AGIQA-3K
AIGCIQA2023
PKU-AIGIQA-4K
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
CLIPScore [26] PickScore [27] ImageReward [28] LIQE [16] AMFF-Net♯ [17]
0.5207 0.6710 0.7297 0.7638 0.7513
0.6409 0.7252 0.7847 0.8480 0.8476
0.2337 0.5159 0.5870 0.7529 0.7782
0.2483 0.5136 0.5874 0.7468 0.7638
0.1492 0.5056 0.4832 0.7796 -
0.1969 0.5682 0.5829 0.7946 -
MST-CLIPIQA MST-CLIPIQA*
0.7895 0.8124
0.8666 0.8817
0.7729 0.7762
0.7615 0.7696
0.8059 0.8009
0.8141 0.8174
B. Results and Analysis Comparison with State-of-the-Art Methods. We compare against representative IQA methods including CNNbased approaches (LinearityIQA [11], MUSIQ [12], HyperIQA [13]), transformer-based methods (StairIQA [14], MANIQA [15]), VLM-based approaches (LIQE [16], CLIP-AGIQA [18], MA-AGIQA [19]), and text-image matching methods (CLIPScore [26], PickScore [27], ImageReward [28]). Tables IV, I, and V present performance comparisons on quality, authenticity, and text-image correspondence prediction, respectively. Methods marked with ‘♯’ indicate results from original papers. As shown in Table IV, both MST-CLIPIQA variants achieve state-of-the-art performance across all five benchmarks. Compared to the previous best method LIQE, MST-CLIPIQA achieves average improvements of 0.89% SRCC and 0.93% PLCC on quality prediction, while MST-CLIPIQA* further improves to 1.11% SRCC and 0.98% PLCC. For authenticity prediction (Table I), MSTCLIPIQA* outperforms LIQE by 1.12% SRCC and 1.31% PLCC on average, indicating that text-image alignment provides auxiliary cues for realism assessment. For correspondence prediction (Table V), MST-CLIPIQA* demonstrates particularly strong gains of 2.35% SRCC and 1.99% PLCC, significantly outperforming zero-shot text-image matching methods and validating the effectiveness of TICAM for prompt-aware evaluation. To provide intuitive comparison, Figure 2 presents scatter plots on AGIQA1K, where MST-CLIPIQA* exhibits the tightest clustering around the diagonal, indicating superior prediction accuracy and consistency. Effectiveness of Multi-Scale Feature Extraction. To validate the core contribution of MSTFE, we compare our dual-encoder design against single-encoder baselines. As shown in Table VI, employing complementary ViT-B/32 and ViT-B/16 encoders consistently outperforms either single encoder, with average improvements of 0.48% SRCC and 0.72% PLCC over the best single-scale baseline. This confirms that coarse-grained semantic features and finegrained textural features provide complementary information for quality assessment.
TABLE VI Effects of image encoder configuration. AGIQA-1K
Encoder
AIGCIQA2023
Average
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
ViT-B/16 ViT-B/32
0.8937 0.8980
0.9021 0.9064
0.8653 0.8613
0.8789 0.8751
0.8695 0.8685
0.8760 0.8751
ViT-B/32 + ViT-B/16
0.8990
0.9149
0.8701
0.8835
0.8733
0.8823
TABLE VII Ablation study. MSTFE: multi-scale two-stream feature extraction; CA: cross-attention with text prompts. MSTFE
✓ ✓
CA
✓ ✓
AGIQA-1K
AGIQA-3K
Quality
Corresp.
AIGCIQA2023 Auth.
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
0.8980 0.8990 0.9010 0.9091
0.9064 0.9149 0.9091 0.9180
0.7793 0.7895 0.8057 0.8124
0.8619 0.8666 0.8761 0.8817
0.8070 0.8149 0.8109 0.8170
0.7944 0.8026 0.7994 0.8049
Effects of Multi-Scale Feature Fusion Methods. Table II compares GFF against alternative fusion strategies including linear interpolation, adaptive pooling, and crossattention. GFF achieves the best average performance (0.8733 SRCC, 0.8823 PLCC), as the learnable gating mechanism enables adaptive selection of task-relevant information from each scale. Effects of MSTFE Architecture Variants. We compare two MSTFE architectures: MSTFE-1 with two complete CLIP models versus MSTFE-2 with dual image encoders sharing a single text encoder. Table III shows that MSTFE2 achieves comparable or better performance with lower computational cost, validating that multi-scale visual features can be effectively aligned within a unified text embedding space. Ablation Study. Table VII presents ablation results validating each component. Starting from the ViT-B/32 baseline, adding MSTFE improves SRCC by 0.10%–1.02% across different metrics. Incorporating cross-attention (CA) with text prompts yields larger gains of 0.30%– 2.64%, particularly for correspondence prediction where prompt information is directly relevant. Combining both components achieves the best performance, demonstrating their complementary contributions. IV. Conclusion We present MST-CLIPIQA, a multi-scale framework that addresses the semantic-distortion conflict in VLMbased AIGI quality assessment. By extracting global semantic features and fine-grained textural representations through dual-stream encoders with complementary patch sizes, our approach captures the hierarchical nature of human quality perception. The gated feature fusion enables adaptive integration of multi-scale information, while optional cross-attention with generation prompts enhances
text-image correspondence evaluation. Experiments across five benchmarks demonstrate state-of-the-art performance on quality, authenticity, and correspondence prediction with only ∼0.8M trainable parameters.
Appendix A. Image Quality Assessment (IQA) In recent years, the Contrastive Language-Image Pretraining (CLIP) model has attracted considerable attention in the field of Image Quality Assessment (IQA) due to its powerful vision-language alignment capabilities and rich prior knowledge learned from large-scale image-text pairs [29]–[34]. Wang et al. [35] pioneered the exploration of CLIP for assessing both the quality perception (i.e., the “look”) and abstract perception (i.e., the “feel”) of images, proposing CLIP-IQA which leverages effective prompt engineering and an antonym prompt pairing strategy (e.g., “Good photo” vs. “Bad photo”) to harness CLIP’s prior knowledge in a zero-shot manner, demonstrating that CLIP captures meaningful priors that generalize well to different perceptual assessments. Subsequently, Zhang et al. [16] proposed LIQE (Learning Image Quality via Vision-Language Correspondence), a CLIP-based blind image quality assessment method that employs a multitask learning paradigm to jointly learn three tasks— quality prediction, scene classification, and distortion type identification—by computing the joint probability from cosine similarities between visual and textual embeddings, thereby leveraging the correspondence between vision and language to predict image quality while benefiting from auxiliary task knowledge. With the rapid development of text-to-image generation models, recent research has increasingly focused on utilizing CLIP-based methods to evaluate AI-Generated Images (AIGIs), which present unique challenges including visual quality, authenticity, and text-image correspondence assessment. Zhou et al. [17] introduced AMFFNet (Adaptive Mixed-Scale Feature Fusion Network), a novel blind IQA framework that evaluates AGI quality from three dimensions—visual quality, authenticity, and text-image consistency—by employing a multi-scale input strategy inspired by the human visual system, utilizing an Adaptive Feature Fusion (AFF) block to adaptively fuse multi-scale features with learnable weights, and comparing semantic features from text and image encoders to assess text-to-image alignment. Qu et al. [36] introduced IP-IQA, a CLIP-based dual-stream framework that simultaneously processes AI-generated images and their corresponding textual prompts, featuring an Image2Prompt incremental pretraining strategy to bridge AGI-style visual and textual modalities, and incorporating a crossattention-based image-prompt fusion module along with a specially designed [QA] token to guide the model on quality-relevant aspects and enable effective image-text interaction. Unlike most existing methods that simply calculate similarity scores, they designed specialized modules to explicitly learn the deeper relationships between text prompts and input images. Peng et al. [37] proposed IPCE,
a CLIP-based AIGC image quality assessment method that emphasizes the correspondence between images and prompts by designing textual templates with five qualityrelated adverbs (e.g., “badly”, “poorly”, “fairly”, “well”, “perfectly”) to represent different levels of image-prompt correlation, transforming the assessment into classification probabilities and subsequently into a precise regression task, achieving the first place in the image track of the NTIRE 2024 Quality Assessment for AI-Generated Content Challenge. Tang et al. [18] proposed CLIP-AGIQA, a CLIP-based regression model for quality assessment of generated images that implements multi-category learnable prompts to fully utilize the textual knowledge encapsulated in CLIP, thereby enhancing prediction precision by moving beyond the limited contrastive similarity approach. However, despite these significant advances, existing methods have not fully explored the potential of CLIP in assessing the quality of AIGIs. First, most approaches utilize only a single pre-trained CLIP model for evaluation, which may limit the diversity of extracted features. Second, they primarily rely on calculating the similarity between text prompts and input images, which may constrain the model’s ability to capture relationships beyond mere similarity measures. B. Theoretical Background The Information Bottleneck framework addresses the fundamental problem of extracting relevant information from a source variable X about a target variable 𝑌 while discarding irrelevant details. For our multi-scale feature fusion setting, we identify X ≡ F = [f (𝑐) ; f ( 𝑓 ) ] as the concatenated multi-scale features and 𝑌 ≡ 𝑞 as the groundtruth quality score. a) Mutual Information.: For continuous random variables, the mutual information I (X; 𝑌 ) quantifies the statistical dependence between X and 𝑌 : I (X; 𝑌 ) = H (X) − H (X|𝑌 ) = H (𝑌 ) − H (𝑌 |X)
(6)
where H (·) denotes differential entropy and H (·|·) denotes conditional entropy: H (X) = −E 𝑝 (x) log 𝑝(x) , H (X|𝑌 ) = −E 𝑝 (x,𝑦) log 𝑝(x|𝑦) (7) b) IB Objective.: The IB principle seeks a compressed representation Z of X that preserves maximal information about 𝑌 while minimizing information about X itself. This is formalized as a constrained optimization problem: min I (X; Z)
𝑝 (z|x)
s.t.
I (Z; 𝑌 ) ≥ I0
(8)
Introducing a Lagrange multiplier 𝛽 −1 and reformulating as an unconstrained problem yields: LIB = I (X; Z) − 𝛽 ·I (Z; 𝑌 ) = −𝛽 · I (Z; 𝑌 ) +I (Z; X) (9)
which is equivalent to maximizing the objective in the main text.
c) Sparsity-Inducing Regularization.: To encourage explicit compression behavior, we can optionally augment the training objective with a gate sparsity penalty:
C. Variational Relaxation
𝐷
Direct optimization of Eq. (9) is intractable due to the difficulty of estimating mutual information for highdimensional continuous variables. Following the Variational Information Bottleneck (VIB) framework, we derive tractable upper and lower bounds. a) Compression Term Upper Bound.: Using a variational marginal 𝑟 (z), we obtain: h i I (X; Z) = E 𝑝 (x) KL 𝑝(z|x) ∥ 𝑝(z) h i ≤ E 𝑝 (x) KL 𝑝(z|x) ∥𝑟 (z) (10) where KL(·∥·) denotes the Kullback-Leibler divergence. The inequality follows from the non-negativity of KL divergence. b) Relevance Term Lower Bound.: Using a variational decoder 𝑞(𝑦|z): I (Z; 𝑌 ) = H (𝑌 ) − H (𝑌 |Z) ≥ H (𝑌 ) + E 𝑝 (z,𝑦) log 𝑞(𝑦|z)
(11)
Lsparse =
1 ∑︁ min 𝑔𝑖 , 1 − 𝑔𝑖 𝐷 𝑖=1
(15)
This term encourages gate values toward the extremes (0 or 1), promoting hard selection that maximizes compression. In practice, we find this regularization unnecessary as the MSE and ranking losses provide sufficient gradient signal for meaningful gate specialization. This appendix details the optional cross-modal attention module that enables prompt-anchored quality assessment when generation prompts are available. E. Motivation AI-generated images uniquely exhibit quality dimensions that require semantic reasoning beyond pure visual analysis. Specifically, prompt-content misalignment occurs when the generated image fails to faithfully represent the textual generation prompt. Standard visual features cannot detect such misalignment without access to the prompt itself. Our cross-modal attention mechanism addresses this by computing explicit text-image correspondence signals.
F. Architecture Combining these bounds yields the VIB objective: a) Text Encoding.: The generation prompt T = h i LVIB = E 𝑝 (x) KL 𝑝(z|x) ∥𝑟 (z) −𝛽·E 𝑝 (x,𝑦) E 𝑝 (z|x) log 𝑞(𝑦|z){𝑡1 , . . . , 𝑡 𝐿 } is processed through the frozen CLIP text (12) encoder to obtain a sequence of contextualized token embeddings: D. Connection to Gated Feature Fusion E𝑡 = TextEncoder(T) = [e1 , . . . , e 𝐿 ] ∈ R 𝐿×𝐷 (16) Our GFF module implements a deterministic approximation to the VIB framework tailored for multi-scale feature fusion. a) Deterministic Encoder.: Rather than learning a stochastic encoder 𝑝(z|x), we employ a deterministic gating function: z = 𝑓GFF (F; Θ𝑔 ) = g(F) ⊙ 𝜙 𝑐 (f (𝑐) ) + (1 − g(F)) ⊙ 𝜙 𝑓 (f ( 𝑓 ) ) (13) This can be viewed as a delta distribution encoder: 𝑝(z|x) = 𝛿 z − 𝑓GFF (F) . b) Implicit Compression.: The gating mechanism achieves implicit compression through selective dimension-wise interpolation. Let 𝑔𝑖 ∈ [0, 1] denote the 𝑖-th gate value. The effective information content of dimension 𝑖 is bounded by: (𝑓) I (𝑧𝑖 ; 𝐹𝑖 ) ≤ 𝑔𝑖 · I 𝜙 𝑐 ( 𝑓𝑖(𝑐) ); 𝐹𝑖 + (1 − 𝑔𝑖 ) · I 𝜙 𝑓 ( 𝑓𝑖 ); 𝐹𝑖 (14) When the gate learns to select the more quality-relevant scale for each dimension, it effectively implements taskaware information routing that discards scale-specific redundancies.
The [EOS] token embedding serves as the global text representation: e𝑡 = e 𝐿 ∈ R𝐷 . b) Cross-Attention Formulation.: We employ scaled dot-product attention with the fused visual representation z as the query and text embeddings as keys and values. First, we compute query, key, and value projections: 𝐷𝑣 ×𝐿 V = W𝑉 E⊤ 𝑡 ∈ R (17) where W𝑄 ∈ R𝐷𝑘 ×𝐷 , W𝐾 ∈ R𝐷𝑘 ×𝐷 , W𝑉 ∈ R𝐷𝑣 ×𝐷 are learnable projection matrices. The attention weights and output are computed as: Q⊤ K ∈ R 𝐿 , o = VA ∈ R𝐷𝑣 (18) A = softmax √ 𝐷𝑘 c) Multi-Head Extension.: For enhanced representational capacity, we employ multi-head attention with 𝐻 parallel attention heads: MultiHead(z, E𝑡 ) = W𝑂 · Concat o1 , . . . , o 𝐻 (19)
Q = W𝑄 z ∈ R 𝐷 𝑘 ,
𝐷 𝑘 ×𝐿 , K = W𝐾 E⊤ 𝑡 ∈ R
where each head ℎ operates with independent projec(ℎ) (ℎ) tions W𝑄 , W𝐾 , W𝑉(ℎ) of reduced dimension 𝐷 𝑘 /𝐻 and 𝐷 𝑣 /𝐻, and W𝑂 ∈ R𝐷×𝐷𝑣 is the output projection.
d) Residual Integration.: The cross-modal enhanced representation incorporates the attention output through a learnable residual connection: z′ = z + 𝛼 · LayerNorm MultiHead(z, E𝑡 ) (20) where 𝛼 is a learnable scalar initialized to zero. This zero initialization ensures that the model initially behaves identically to the prompt-free variant, with cross-modal information gradually incorporated as training progresses. G. Semantic Correspondence Interpretation The attention weights A ∈ R 𝐿 provide interpretable signals about which prompt tokens the model considers most relevant for quality assessment. High attention on specific tokens (e.g., object names, attributes) indicates that the model is verifying whether these semantic elements are faithfully represented in the image. This enables post-hoc analysis of prompt-content alignment failures that degrade perceived quality. This appendix provides comprehensive implementation details to ensure reproducibility. H. Architecture Specifications a) Vision Encoder.: We employ the CLIP ViT-B/32 vision encoder as our default backbone, which consists of 𝑀 = 12 transformer blocks with embedding dimension 𝐷 = 512. The encoder processes 224 × 224 images with the default patch size of 32 × 32. All encoder parameters remain frozen throughout training. b) Multi-Scale Configuration.: For the dual-stream architecture, we configure: ( Coarse stream: 𝑃𝑐 = 32, 𝑁 𝑐 = 49 (21) Fine stream: 𝑃 𝑓 = 16, 𝑁 𝑓 = 196 The fine-grained stream requires interpolation of positional embeddings from the original 7 × 7 grid to a 14 × 14 grid, implemented via bicubic interpolation. c) Gated Feature Fusion.: The gating network employs: Input dimension: 2𝐷 = 1024 (22) Hidden dimension: 𝐷 ℎ = 256 Output dimension: 𝐷 = 512 Stream-specific projections 𝜙 𝑐 , 𝜙 𝑓 are implemented as single linear layers without bias terms. d) Regression Head.: The quality regression MLP uses: Hidden dimension: 𝐷 𝑟 = 128 Activation: GELU Dropout: 0.1 (training only)
(23)
e) Cross-Modal Attention.: When enabled: Number of heads: 𝐻=8 Key/Query dimension: 𝐷 𝑘 = 512 Value dimension: 𝐷 𝑣 = 512 I. Training Protocol
(24)
a) Optimization.: We employ the Adam optimizer with the following hyperparameters: Learning rate: 𝜂 = 1 × 10−4 Weight decay: 𝜆 𝑤 = 1 × 10−2 Betas: (𝛽1 , 𝛽2 ) = (0.9, 0.999) Epsilon: 𝜖 Adam = 1 × 10−8
(25)
b) Learning Rate Schedule.: We employ a cosine annealing schedule with linear warmup: 𝑡 if 𝑡 ≤ 𝑇warmup 𝜂 · 𝑇warmup 𝜂𝑡 = 𝑡 −𝑇warmup 1 otherwise 𝜂min + 2 (𝜂 − 𝜂min ) 1 + cos 𝑇 −𝑇warmup 𝜋 (26) where 𝑇warmup = 5 epochs, 𝑇 = 50 total epochs, and 𝜂min = 1 × 10−6 . c) Loss Hyperparameters.: The composite loss function uses: 𝜆 = 0.1, 𝜖 = 0.05 (27) where 𝜆 balances MSE and ranking losses, and 𝜖 is the ranking margin. d) Data Augmentation.: Training images undergo: 𝑝 = 0.5 scale ∈ [0.8, 1.0], ratio ∈ [0.9, 1.1] brightness = 0.1, contrast = 0.1 (28) Validation and test images use center crop only. e) Batch Configuration.: Training uses a batch size of |B| = 32 per GPU. For multi-GPU training, we employ synchronized batch normalization equivalents where applicable. Random horizontal flip: Random resized crop: Color jitter:
J. Inference Protocol During inference, the model operates in evaluation mode with all stochastic elements (dropout) disabled. Input images are resized to 224 × 224 using bicubic interpolation followed by center cropping. No test-time augmentation is employed. The predicted quality score 𝑞ˆ is output directly without post-processing. K. Computational Requirements a) Parameter Count.: The trainable parameters of MST-CLIPIQA total approximately 0.8M, distributed as: Gating network (𝜓 𝜙 ) : ∼ 0.4M Stream projections (𝜙 𝑐 , 𝜙 𝑓 ) : ∼ 0.3M Regression head: ∼ 0.1M
(29)
The frozen CLIP ViT-B/32 encoder contains 86M parameters. b) Inference Speed.: On a single NVIDIA RTX 3090 GPU: ( Throughput: ∼ 180 images/second (30) Latency: ∼ 5.6 ms/image c) Training Time.: Full training on AGIQA-3K requires approximately 2 hours on a single A40 GPU. References [1] Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin, “Agiqa-3k: An open database for ai-generated image quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 8, pp. 6833–6846, 2023. [2] Jiarui Wang, Huiyu Duan, Jing Liu, Shi Chen, Xiongkuo Min, and Guangtao Zhai, “Aigciqa2023: A large-scale image quality assessment database for ai generated images: from the perspectives of quality, authenticity and correspondence,” in CAAI International Conference on Artificial Intelligence. Springer, 2023, pp. 46–57. [3] Chunyi Li, Tengchuan Kou, Yixuan Gao, Yuqin Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang, Weixia Zhang, Haoning Wu, et al., “Aigiqa-20k: A large database for ai-generated image quality assessment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 6327–6336. [4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763. [5] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy, “Exploring clip for assessing the look and feel of images,” in Proceedings of the AAAI conference on artificial intelligence, 2023, vol. 37, pp. 2555–2563. [6] Lorenzo Agnolucci, Leonardo Galteri, and Marco Bertini, “Qualityaware image-text alignment for opinion-unaware image quality assessment,” arXiv preprint arXiv:2403.11176, 2024. [7] Daekyu Kwon, Dongyoung Kim, Sehwan Ki, Younghyun Jo, Hyong-Euk Lee, and Seon Joo Kim, “Attiqa: Generalizable image quality feature extractor using attribute-aware pretraining,” in Proceedings of the Asian Conference on Computer Vision, 2024, pp. 4526–4543. [8] Zhenchen Tang, Zichuan Wang, Bo Peng, and Jing Dong, “Clipagiqa: boosting the performance of ai-generated image quality assessment with clip,” in International Conference on Pattern Recognition. Springer, 2024, pp. 48–61. [9] Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin, “Topiq: A topdown approach from semantics to distortions for image quality assessment,” IEEE Transactions on Image Processing, vol. 33, pp. 2404–2418, 2024. [10] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang, “Musiq: Multi-scale image quality transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 5148–5157. [11] Dingquan Li, Tingting Jiang, and Ming Jiang, “Norm-in-norm loss with faster convergence and better performance for image quality assessment,” in Proceedings of the 28th ACM International conference on multimedia, 2020, pp. 789–797. [12] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang, “Musiq: Multi-scale image quality transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 5148–5157.
[13] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang, “Blindly assess image quality in the wild guided by a self-adaptive hyper network,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 3664–3673. [14] Wei Sun, Huiyu Duan, Xiongkuo Min, Li Chen, and Guangtao Zhai, “Blind quality assessment for in-the-wild images via hierarchical feature fusion strategy,” in 2022 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), 2022, pp. 01–06. [15] Sidi Yang, Tianhe Wu, Shu Shi, Shan Gong, Ming Cao, Jiahao Wang, and Yujiu Yang, “Maniqa: Multi-dimension attention network for no-reference image quality assessment,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1190–1199, 2022. [16] Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma, “Blind image quality assessment via vision-language correspondence: A multitask learning perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14071–14081. [17] Tianwei Zhou, Songbai Tan, Wei Zhou, Yu Luo, Yuan-Gen Wang, and Guanghui Yue, “Adaptive mixed-scale feature fusion network for blind ai-generated image quality assessment,” IEEE Transactions on Broadcasting, 2024. [18] Zhenchen Tang, Zichuan Wang, Bo Peng, and Jing Dong, “Clipagiqa: Boosting the performance of ai-generated image quality assessment with clip,” in International Conference on Pattern Recognition. Springer, 2025, pp. 48–61. [19] Puyi Wang, Wei Sun, Zicheng Zhang, Jun Jia, Yanwei Jiang, Zhichao Zhang, Xiongkuo Min, and Guangtao Zhai, “Large multimodality model assisted ai-generated image quality assessment,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 7803–7812. [20] Zicheng Zhang, Chunyi Li, Wei Sun, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai, “A perceptual quality assessment exploration for aigc images,” arXiv preprint arXiv:2303.12618, 2023. [21] Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin, “Agiqa-3k: An open database for ai-generated image quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2023. [22] Jiarui Wang, Huiyu Duan, Jing Liu, Shi Chen, Xiongkuo Min, and Guangtao Zhai, “Aigciqa2023: A large-scale image quality assessment database for ai generated images: from the perspectives of quality, authenticity and correspondence,” arXiv preprint arXiv:2307.00211, 2023. [23] Chunyi Li, Tengchuan Kou, Yixuan Gao, Yu Shan Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang, Weixia Zhang, Haoning Wu, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai, “Aigiqa-20k: A large database for ai-generated image quality assessment,” 2024. [24] Jiquan Yuan, Fanyi Yang, Jihe Li, Xinyan Cao, Jinming Che, Jinlong Lin, and Xixin Cao, “Pku-aigiqa-4k: A perceptual quality assessment database for both text-to-image and image-to-image aigenerated images,” ArXiv, vol. abs/2404.18409, 2024. [25] Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [26] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi, “Clipscore: A reference-free evaluation metric for image captioning,” arXiv preprint arXiv:2104.08718, 2021. [27] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy, “Pick-a-pic: An open dataset of user preferences for text-to-image generation,” 2023. [28] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong, “Imagereward: Learning and evaluating human preferences for text-to-image generation,” 2023. [29] Zijie Meng, Yuanze Zeng, Xiang Chang, Tianshuo Xu, Fei Chao, Xixin Cao, Changjing Shang, and Qiang Shen, “Orpaint: a zeroshot inpainting model for oracle bone inscription rubbings with visual mamba block,” Science China Information Sciences, vol. 68, no. 8, pp. 189102, 2025.
[30] Zijie Meng, Jinming Che, Bingcai Wei, and Xixin Cao, “Make a game: A novel paradigm for interactive game rendering,” in ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026, pp. 1026– 1030. [31] Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, et al., “Omnidirector: General multi-shot camera cloning without cross-paired data,” arXiv preprint arXiv:2606.13432, 2026. [32] Zijie Meng, Jiwen Liu, Yufei Liu, Chengzhuo Tong, Xiaoqiang Liu, Yuanxing Zhang, Yulong Xu, and Pengfei Wan, “Argus: Stacked multi-view identity mosaic injection for subject-preserving video generation,” arXiv preprint arXiv:2606.11670, 2026. [33] Yufei Liu, Haoke Xiao, Jiaxing Chai, Yongcun Zhang, Rong Wang, Zijie Meng, and Zhiming Luo, “Synpo: Boosting training-free few-shot medical segmentation via high-quality negative prompts,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2025, pp. 594–603. [34] Bingcai Wei, Hui Liu, Chuang Qian, Zijian Li, Wangyu Wu, and Zijie Meng, “Robust single image sand removal by leveraging uncertainty-aware sam priors and prompt learning with refined perceptual loss,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 4932–4941. [35] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy, “Exploring clip for assessing the look and feel of images,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, pp. 2555–2563. [36] Bowen Qu, Haohui Li, and Wei Gao, “Bringing textual prompt to ai-generated image quality assessment,” 2024. [37] Fei Peng, Huiyuan Fu, Anlong Ming, Chuanming Wang, Huadong Ma, Shuai He, Zifei Dou, and Shu Chen, “Aigc image quality assessment via image-prompt correspondence,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2024, pp. 6432–6441.