LINE: LLM-based Iterative Neuron Explanations for Vision Models Vladimir Zaigrajew1,3 , Michał Piechota1,∗, Gaspar Sekula1,∗ , Przemysław Biecek1,2,3
arXiv:2604.08039v1 [cs.CV] 9 Apr 2026
1
Warsaw University of Technology, 2 University of Warsaw, 3 Centre for Credible AI [email protected]
Abstract Interpreting the concepts encoded by individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly black-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. We demonstrate that LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.18 on ImageNet and 0.05 on Places365, while discovering, on average, 29% of new concepts missed by massive predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, which enables polysemanticity evaluation and produces supporting visual explanations that rival gradient-dependent activation maximization methods. The source code will be made available soon.
1
Introduction
Deep Neural Networks (DNNs) have been widely adopted across various domains due to their performance and capabilities. However, limited progress in comprehending their opaque decisionmaking processes remains a major obstacle to their use in safety-critical applications, such as healthcare or the justice system. To address this problem, numerous explainable AI (XAI) methods (Selvaraju et al., 2017, Tjoa and Guan, 2020, Biecek and Samek, 2025) have been developed over the years to uncover how these models “think” and decide. Further need for more precise control over model behavior led to the development of the field of mechanistic interpretability (Bereska and Gavves, 2024) aimed to surgically uncover the inner workings of DNNs, focusing on identifying localized decision-making circuits (Conmy et al., 2023) or specific neuron activation patterns (Feldhus and Kopf, 2025). A significant problem in this area is to automatically translate neural activity into human-understandable semantic concepts. While neuron labeling has seen success in both vision (Bau et al., 2017, Hernandez et al., 2022, Kalibhat et al., 2023, Oikarinen and Weng, 2023, Bykov et al., 2024, Bai et al., 2025) and text (Bills et al., 2023, Kopf et al., 2025) domains, crucial limitations remain. Foremost, most concept assignments rely on predefined vocabularies (Bau et al., 2017, Kalibhat et al., 2023, Oikarinen and Weng, 2023, Bykov et al., 2024) that often lack the precise ground truth concept. Alternatively, methods relying on text-generative models like MILAN (Hernandez et al., 2022) frequently struggle to capture broad concepts, producing overly crisp descriptions. ∗
Bachelor Work.
1
Figure 1: Qualitative comparison of neuron descriptions. We show the top four activating images from the ImageNet-1K for three randomly selected neurons from the ResNet50 avgpool layer (left) and the ViT-B/16 heads layer (right). The descriptions are provided from our method (LINE), CLIP-Dissect (Oikarinen and Weng, 2023), and INVERT (Bykov et al., 2024) alongside their CoSy benchmark (Kopf et al., 2024) AUC scores. The best method according to the AUC score is highlighted in bold. Extended results for different layers and models are reported in Appendix I.
To address above limitations we propose LLM-based Iterative Neuron Explanations (LINE), the first iterative, automatic neuron naming framework for providing textual explanations of vision models (Figure 2, with corresponding visual results in Figure 1). LINE operates in a strictly training-free, black-box setting, leveraging the generative capabilities of text-to-image (T2I) models and the reasoning power of Large Language Models (LLMs). At its core, LINE utilizes an LLM to propose new potential concept’s descriptions based on a neuron’s concept scoreboard, initialized from a predefined vocabulary. For instance, while an initial step might incorrectly assign to a neuron a wreck label (Table 3), LINE’s iterative refinement reveals that the neuron activates stronger on the concept gatherer. To connect LLMs with vision models and identify concept activations, we synthesize proposed concepts via a T2I model and leverage the resulting images to extract neuron activations. Finally, these activations are evaluated by a special scoring function to quantify the correctness of the proposed concept and update the scoreboard with the new concept. To overcome the limitation of overly coarse concepts, an additional summary step is added at the end of the cycle to suggest and score higher-order concept reasoned from the final history (e.g., highlighting a global concept like farming). Through its highly transparent framework, LINE provides not only the top-scoring concept but also the generation history, the reasoning behind each new concept, and the generated visual samples, which may be used to evaluate neuron polysemanticity. Furthermore, by optimizing textual descriptions, LINE provides a gradient-free alternative to activation max2
imization approaches commonly used for bias discovery (Pennisi et al., 2025, Carnemolla et al., 2025). Contribution. In this work, we present LINE, a full black-box, training-free iterative pipeline for providing textual and visual explanations for vision model neurons. Our core contributions include: State-of-the-art Neuron Labeling: LINE significantly outperforms prior methods on the CoSy neuron labeling benchmark (Kopf et al., 2024), achieving AUC improvements of 0.18 on ImageNet and 0.05 on Places365, with statistically significant margins. Beyond Predefined Vocabularies: We demonstrate that relying solely on predefined vocabularies leads to suboptimal label descriptions. Through the iterative refinment process, LINE discovers up to 39% new, highly human-interpretable concepts missed even by massive predefined vocabularies. Visual Explanations: Compared to activation maximization methods, LINE identifies analogous model biases without requiring access to the gradients. Furthermore, it produces significantly more interpretable and natural visuals than rivaled approaches. We evaluate LINE across three dimensions: (1) quantitative superiority on the CoSy benchmark across four distinct architectures, (2) qualitative and causal validation of assigned labels via image ablation, and (3) a comparison of visual explanations against attribution and activation maximization methods for bias discovery tasks. Demonstrating strong performance across these areas, LINE provides a transparent and scalable pipeline for auditing DNN models.
2
Related work
Neuron Labeling. Early methods for automatic neuron labeling, pioneered by Network Dissection (Bau et al., 2017), linked hidden neurons to predefined concepts by computing the Intersection over Union (IoU) between neuron activation maps and ground-truth segmentation masks. INVERT (Bykov et al., 2024) expanded this by incorporating compositional logic and optimizing for a higher Area Under the Curve (AUC). To decouple descriptions from rigid datasets, MILAN (Hernandez et al., 2022) trained an LSTM-based captioning model on the human-annotated custom dataset (MILANNOTATIONS) to generate free-form descriptions. To further overcome the need for labeled corpora or segmentation masks like Broden (Bau et al., 2017), methods such as CLIP-Dissect (Oikarinen and Weng, 2023) and FALCON (Kalibhat et al., 2023) leveraged multimodal models like CLIP (Radford et al., 2021) to match top activation images against large-scale, open-vocabulary captions. More recently, Describe-and-Dissect (DnD) (Bai et al., 2025) introduced a training-free framework utilizing image-to-text (I2T) models to describe highly activating image crops, an LLM to propose captions from I2T descriptions, and a T2I model to synthesize potential concepts. While LINE shares DnD’s components like LLMs and T2I models, it fundamentally departs from standard single forward-pass methods through an iterative refinement process. Furthermore, when evaluated on the recent unified, architecture-agnostic benchmark, Concept Synthesis (CoSy) (Kopf et al., 2024), LINE achieves superior performance on deeper layers. Finally, because neurons are often highly polysemantic (Bricken et al., 2023), LINE enables a rapid and comprehensive assessment of multi-faceted neuron behavior by maintaining a history of top-scoring concepts. Activation Maximization (AM). AM interprets neural networks by generating inputs that maximize targeted neuron activations (Erhan et al., 2009, Zhu and Cangelosi, 2025). While early methods relying on the direct optimization of input pixels often produced unrealistic, hard-to-interpret images (Simonyan et al., 2013, Olah et al., 2017), recent approaches like DiffExplainer (Pennisi et al., 2025) and later DEXTER (Carnemolla et al., 2025) have modernized AM by optimizing the text prompts for diffusion models that generate AM images. However, as these methods predominantly 3
relay on gradient-based optimization, it imposes significant memory overhead and requires the entire pipeline to be fully differentiable, often limiting their architectural choices to older T2I models like Stable Diffusion 1.5 (Rombach et al., 2022). Since LINE does not require differentiability, it can seamlessly integrate modern T2I models such as SDXL (Podell et al., 2024) and FLUX.1 [dev] (Black Forest Labs, 2024). This flexibility enables the discovery of model biases by producing visually cleaner, more natural, and less out-of-distribution (OOD) AM samples. Generative Models for Explainability. The integration of generative models has significantly advanced XAI by bridging visual and textual reasoning. The reasoning capabilities of LLMs have enabled their deployment as iterative optimizers in tasks that can be in described in natural language (Yang et al., 2023) or in multimodal feedback loops for refining image caption descriptions (Ashutosh et al., 2025). Concurrently, generative image models have facilitated the creation of visual counterfactuals (Sobieski and Biecek, 2024, Sobieski et al., 2024) and synthetic evaluation sets (Kopf et al., 2024), mitigating the reliance on extensive human-labeled corpora. Recent advancements in agentic frameworks have introduced the first agent-based neuron interpretability pipelines, such as MAIA (Shaham et al., 2024) and OpenMAIA (Camuñas et al., 2025), which equip multimodal LLMs with unconstrained tools for exemplar selection, synthetic generation, and image editing. While these systems create complex, less-interpretable, multi-billion parameter workflows, LINE offers a simpler, highly transparent, and controllable architecture. It demonstrates that the iterative reasoning of small-scale LLMs (e.g., Llama 3.1 8B (Grattafiori et al., 2024)), when paired only with a T2I generative model, achieves state-of-the-art performance on neuron labeling tasks.
3
Method
In this section, we describe LINE, an automatic, training-free, and black-box framework for labeling vision model neurons. An overview of the LINE algorithm is illustrated in Figure 2 and detailed in Section 3.2. We establish formal notation in Section 3.1 (with a complete notation summary in Appendix A) and provide a comprehensive characteristic comparison against baseline methods in Appendix B).
3.1
Preliminaries
Let f : X → A denote a target vision network, where X ⊂ RC×H×W represents the input image domain and A denotes the activation space of a specific model layer under inspection. For Convolu′ ′ ′ tional Neural Networks (CNNs), the activation space typically resides in RC ×H ×W , where C ′ is the number of channels and H ′ × W ′ is the spatial resolution. To obtain a vector representation, we apply a spatial global pooling function p(·), yielding pooled activations a = p(f (x)), such that a ∈ RD , where D denotes the number of neurons. For layers that are inherently one-dimensional, such as global average pooling (avgpool), the identity mapping is used. For Vision Transformers (ViTs), we focus on the linear representations from the encoder or the classifier heads, which are natively one-dimensional. The objective of automated neuron labeling is to define an explanation function E : [D] → T that assigns semantic textual descriptions from a set T to each neuron, providing a human-interpretable concept for each of the D dimensions in A.
3.2
LINE: LLM-based Iterative Neuron Explanations
LINE is a training-free iterative approach that leverages four core components in a loop (see Figure 2). Similar to prior methods, LINE analyzes one neuron n at a time, allowing for parallel and 4
t = red fruit
Concept Proposal
Step 1.
Step 2.
Initialization
Image Synthesis
concept score strawberry 0.86 cherry 0.77
Summary Concept
Scoreboard
ImageNet-1K
for N iterations
N+1 iteration
At = [2.13, ..., 4.18]
Concept Scoring
Activation Extraction
Step 3.
Step 4.
st = 0.93
red fruit
Final concept
Figure 2: Overview of the LINE iterative framework. Step 1: An LLM proposes a new concept t based on the descriptions from the scoreboard H (e.g., proposing strawberry and pomegranate into red fruit). Step 2: A Text-to-Image (T2I) model generates a batch of diverse synthetic images illustrating the concept t. Step 3: These images are processed by the target vision model extracting concept activations At . Step 4: A scoring function (Equation (2)) converts At into score st , and the the H is updated with the result. Upon reaching the final iteration N , an additional final summary iteration (N + 1) evaluates the higher-order global concept reasoned from top-scoreboard descriptions; here, Step 1 is replaced by a Summary Concept step, while the remaining steps are unchanged. The highest-scoring concept from H is then returned.
independent explanations of each neuron in the model. Following an initialization phase creating initial concept scoreboard H, each iteration proceeds through concept proposal, image synthesis, activation extraction, and concept scoring steps to evaluate new potential concept. This process repeats for N iterations, concluding with a final summary step and yields the highest-scoring concept from H. Algorithm in Appendix C summarizes the proposed method. All the prompts used in the pipeline are detailed in Appendix D. Initialization. To avoid the slow convergence associated with uniform or zero-score distributions, we initialize the concept scoreboard H using labels from the ImageNet-1K validation set (Russakovsky et al., 2015), which provides a diverse vocabulary closely resembling the target model’s training distribution. For each of the K = 1000 classes, we pass M = 50 images through the target model and extract the neuron’s activations to form a control activation matrix Ainit ∈ RK×M . These activations are evaluated by the scorer to produce an initial score vector sinit ∈ RK . Finally, to balance exploitation and exploration, the scoreboard H is populated with the top 5 highest-scoring class labels alongside 5 randomly selected ones.
5
Concept Proposal. We employ LLM as in MILS (Ashutosh et al., 2025) pipeline to reason and propose a new concept t based on the already evaluated concepts in H. After N iterations, a final summary step is executed. The LLM is provided with the top three concepts from H to generate, generalized explanation. This summary concept is evaluated via the standard pipeline appending the results to H. Image Synthesis. We utilize T2I model to generate a set P of synthetic images for the proposed concept. To ensure visual diversity we use the prompt template: "A realistic photo of a {concept}, {angle}, {lighting}", where sampling for “angle” and “lighting” values is described in Appendix D. Activation Extraction. The generated set of images P are processed by the target model f to obtain the activation vector set At for the specific neuron n: At = [an (x1 ), an (x2 ), . . . , an (x|P| )],
(1)
where an (xi ) is the activation of neuron n for the i-th synthetic image. Concept Scoring. While prior works often used Mean Activation Difference (MAD) (Kopf et al., 2024) or the Area Under the Curve (AUC) relative to a reference set, our preliminary analysis indicates that the mean over A provides a more granular and continuous scoring signal s. Unlike AUC, which frequently saturates near 1.0 across multiple concepts, and MAD, whose results are strictly dependent on a control set, the simple mean avoids both of these limitations. By maximizing the neuron’s response, this continuous objective steers the LLM toward increasingly relevant semantic labels. Formally, we define the scoring function as: 1 X ψavg (A) = a, (2) |A| a∈A
where for concept t score function produce scalar value st = ψavg (At ). Updating Scoreboard. After each iteration, the scoreboard is updated with the latest result. We maintain a fixed number of results by removing one entry per update to limit the history size. If the newly evaluated concept t outscores the current minimum in H, we replace the lowest-scoring entry. Otherwise, to encourage exploration, we stochastically sample an existing concept r ∈ H for removal with a probability proportional to (maxc∈H sc ) − sr . This ensures higher-scoring concepts have a greater probability of survival.
4
Experiments
In this section, we provide a comprehensive evaluation of LINE across diverse model architectures and tasks. Throughout these experiments, we focus specifically on the deeper layers of the networks—these layers capture the complex, high-level semantics critical for safety auditing, in contrast to the low-level features (e.g., color or texture) typically found in earlier layers (LeCun et al., 2015). We parameterize LINE using Llama 3.1 8B (Grattafiori et al., 2024) and Stable Diffusion XL (SDXL), with a scoreboard size of |H| = 10 and |P| = 5 images generated per concept. Section 4.1 presents a quantitative assessment on the CoSy benchmark across several different architectures trained on ImageNet-1K and Places365. Section 4.2 provides a qualitative comparison 6
Table 1: Quantitative comparison on the CoSy benchmark. We compare LINE against CLIPDissect, INVERT, and DnD using the CoSy benchmark. Evaluations are performed on the avgpool layer of ResNet50 (trained on ImageNet-1k and Places365), avgpool layer of ResNet18 (trained on ImageNet-1k) and the encoder layer of ViT-B/16 (trained on ImageNet-1k). We report CoSy metrics MAD and AUC; for both metrics, higher is better (↑). Scores are averaged over 30 neurons. The best and second-best results in each row are highlighted in bold and underlined, respectively. The CoSy benchmark and its evaluation metrics are detailed in Appendix F. Model
Dataset
ResNet50
ImageNet
ResNet50
Places365
ResNet18
ImageNet
ViT-B/16
ImageNet
Metric
LINE
Dissect
INVERT
DnD
AUC MAD AUC MAD AUC MAD AUC MAD
0.97±0.04 5.10±2.16 0.94±0.10 4.19±2.25 0.96±0.05 4.61±2.00 0.94±0.09 2.00±0.61
0.87±0.20 3.59±2.61 0.89±0.15 3.53±2.32 0.91±0.10 3.78±2.30 0.69±0.20 0.78±0.89
0.88±0.14 2.53±1.67 0.81±0.19 1.97±1.83 0.83±0.19 2.08±1.62 0.76±0.24 0.96±0.99
0.76±0.20 2.03±2.62 0.74±0.21 1.66±1.88 0.73±0.24 1.79±2.49 0.56±0.24 0.27±1.03
with the baselines and causal validation leveraging image-to-image generative models to ablate labeled concepts and measure the resulting drop in neuron activation. In Section 4.3, we evaluate our visual explanations against established attribution and activation maximization methods using Salient ImageNet (Singla and Feizi, 2022). Finally, Sections 4.4 and 4.5 examine the impact of iterative steps and T2I model choice on LINE performance.
4.1
Quantitative Evaluation
To quantitatively evaluate LINE, we assess its performance on the CoSy benchmark (described in Appendix F), with results presented in Table 1. Benchmarking across three architectures and two datasets showcase that LINE consistently achieves superior performance across all metrics. In Appendix G, we confirm that the performance gains over the baselines are statistically significant. The surprisingly low performance of DnD we attribute to its reliance on image cropping—a structural weakness similarly observed in the CoSy evaluations of the crop-based FALCON (Bykov et al., 2024) method. Because both methods rely heavily on crops during their search phase, they often produce misleading, uninterpretable samples for the Vision-Language Model (VLM) to describe, forcing it to generate overly broad descriptions (e.g., diverse subjects showcase). Ultimately, these findings lead us to question whether image cropping is a viable strategy for neuron analysis. Impact of Iterative Refinement. To evaluate the refinement process, Table 2 reports the percentage of final labels proposed during the iteration process versus those selected from the initial vocabulary. The LLM in LINE generates the winning label for up to 39% of neurons in certain layers, averaging 22% on ImageNet-1K and 38% on Places365. These results, alongside the results of comparing LINE to open-vocabulary baselines in Table 1, demonstrate that zero-shot methods constrained by predefined vocabularies frequently miss optimal labels that LINE iteratively discovers. 7
Table 2: Proportion of predefined vs. generated labels in LINE. For each model, dataset, and layer combination, we explain a randomly selected set of neurons using LINE. We report the total number of investigated neurons, alongside the breakdown of final top-scoring concepts that were either selected from the initial vocabulary (Predefined) or newly proposed during the iterative loop (Generated). The bottom rows aggregate these totals per dataset. Even with a strong initialization vocabulary, LINE’s iterative refinement discovers novel, higher-scoring concepts, highlighting the limitation of the methods relying solely on predefined vocabularies. Model
Dataset
Layer
ResNet18 ResNet50
ImageNet ImageNet
ResNet50
Places365
ViT-B/16
ImageNet
avgpool avgpool layer4 avgpool encoder heads
Total
ImageNet Places365
4.2
# Neurons
Predefined
Generated
94 100 100 100 50 50
81 (86.2%) 81 (81.0%) 63 (63.0%) 61 (61.0%) 34 (68.0%) 32 (64.0%)
13 (13.8%) 19 (19.0%) 37 (37.0%) 39 (39.0%) 16 (32.0%) 18 (36.0%)
294 200
228 (77.6%) 124 (62.0%)
66 (22.4%) 76 (38.0%)
Qualitative evaluation
In Figure 1, we qualitatively compare the concept descriptions for randomly selected neurons from the models and layers evaluated in Section 4.1. The results confirm that the concepts proposed by LINE are highly consistent with the visual features in the top activating images. While CLIP-Dissect and INVERT occasionally provide similar labels, LINE’s descriptions are notably more accurate and human-interpretable (e.g., eel-like versus snake, or spiny-finned fish). Furthermore, despite superficial similarities in some baseline descriptions, scores obtained on the benchmark quantitatively confirm that LINE captures the underlying neuron semantics significantly more accurately. Causal Analysis. To verify the fidelity of assigned neuron labels, we perform an ablation study on ImageNet-1K images. Using an image-to-image (I2I) generative model, we remove only the object corresponding to the assigned neuron label while preserving the rest of the scene (details in Appendix H). As illustrated in Figure 3, removing the carousel or necklace objects results in
Table 3: Concept scoreboard for neuron 252 (gatherer). The table presents the final scoreboard for the neuron 252 from ResNet50 layer4. The Novel row indicates whether the proposed concept was newly generated by the LLM during the iterative process (✓) or selected from the initialization vocabulary (✗). Step “S” denotes the final summary step. Detailed reasoning and visual explanations are provided in Appendix E. Label
thresher
hartebeest
wreck
harvest machine
gatherer
threshing
debris removal
farming
Novel Step Score
✗ 0 0.90
✗ 0 0.69
✗ 0 0.73
✓ 1 1.09
✓ 6 1.33
✓ 8 0.98
✓ 9 0.71
✓ S 0.52
8
Figure 3: Causal impact of concept ablation on neuron activation. Using image-to-image generative models, we remove visual concepts associated with the neuron description from ResNet50 avgpool layer identified LINE. Original (left) and ablated (right) images are shown side-by-side, with normalized neuron activations displayed below each pair. Original images are outlined in blue, ablated versions in orange, and the resulting relative drop in activation is marked in green. Extended results featuring failure cases are provided in the Appendix H. significant drops in the activation magnitudes of neuron 403 and neuron 19, respectively, with values often reaching near zero. Similarly, for the neuron labeled lab coat, removing or changing the coat style produces a substantial decline in activation. While Figure 3 confirms that LINE labels accurately capture underlying causal patterns in neuron activation via I2I interventions, our expanded evaluation in the Appendix H highlights certain limitations of this evaluation protocol. Specifically, the generative process occasionally fails to ablate the target feature and introduces image reconstruction artifacts, raising the question of whether failures stem from incorrect neuron labeling or are simply artifacts of the I2I model. Scoreboard Analysis. In Table 3, we present the final scoreboard for the neuron 252 labeled as gatherer. The initial predefined labels (e.g., thresher, hartebeest) correctly anchor the LLM’s reasoning toward agricultural contexts. However, as these initial concepts were suboptimal, LINE iteratively discovered novel concepts that outperformed the original vocabulary, demonstrating that single-pass reasoning is often insufficient for finding optimal labels. Finally, based on the top three concepts, the LLM proposed farming as a global summary description (step “S”). Although this summary concept scored lower than gatherer, we hypothesize that because the neuron is highly specific to harvesting, a broader concept like farming introduces excessive visual noise in the T2I synthesis. 9
DiffExplainer Salient ImageNet
Feature 1067
Label 609 (Jeep)
Feature 1100
Feature 1208
Feature 1515
Feature 691
Activation: 11.52 Activation: 7.64 Activation: 0.10
Activation: 1.94 Activation: 5.75
Activation: 3.20
Activation: 0.88
Activation: 0.51
Activation: 6.60
Activation: 1.91 Activation: 4.09 Activation: 2.41 Activation: 3.18
Activation: 2.59
LINE
DEXTER
Activation: 2.47
tractors
jeep
tractors
minivan
minivan
Figure 4: Visual explanation comparison on Salient ImageNet (Singla and Feizi, 2022). We present saliency maps alongside visual explanations generated by LINE, DiffExplainer, and DEXTER for the top-5 core features of the “Jeep” class in RobustResNet50. Neuron activation values are displayed above each visual explanation, with the corresponding LINE neuron description provided at the bottom. Compared to AM methods, LINE produces explanations without visual artifacts common in DiffExplainer while achieving higher activations than DEXTER. Extended results for bias and core features on additional Salient ImageNet classes are provided in the Appendix I.
4.3
Visual Explanations
By coupling iterative concept search with image synthesis, LINE generates highly activating images that, alongside textual descriptions, naturally function as visual explanations. This enables a direct comparison with state-of-the-art Activation Maximization (AM) methods such as DiffExplainer and DEXTER. Following the evaluation protocol in Salient ImageNet (Singla and Feizi, 2022), Figure 4 presents visual explanations alongside Grad-CAM-like maps (Selvaraju et al., 2017) for top features that show meaningful correlations with the core class in a RobustResNet50, while additional results for both biased and core features are provided in Appendix I. While LINE consistently outperforms DEXTER, it often yields lower neuron activations than DiffExplainer. This is largely 10
0.8 0.6 1
3
5
7
ResNet-18 ResNet-50 9 11 13 15 17 19 S Iteration
10 8 6 4 2 0 1
Ratio of Discovered Top Concepts (%)
Relative Score (%)
1.0
3
5
7
9 11 13 15 17 19 S Iteration
Figure 5: Performance analysis across iteration steps. We extend the maximum iterations from 10 to 20 for 100 randomly selected neurons in the avgpool of ResNet50 (Places365) and ResNet18. Over successive iterations, we report the relative average best activation score (left panel) and the discovery rate of the optimal description (right panel) at each step. The “S” label denotes the final summary iteration. On average, the iterative loop consistently refines the proposed concepts, yielding monotonically increasing activation scores even until the final step. because standard AM methods often exploit non-descriptive artifacts through gradient optimization, creating a critical bottleneck by producing uninterpretable visual noise. While DEXTER mitigates this through a text-model proxy, LINE avoids such noise entirely as it does not require gradient optimization. Compared to DEXTER, LINE generates easy-to-interpret explanations that achieve higher activations (see Appendix I) while remaining fully compatible with black-box models. We attribute LINE’s lower activations on feature 755 of the “Bathing Cap” class, as well as on multiple features in the “Patio” class (see Appendix I), to its initialization phase. This phase seeds the scoreboard with ImageNet labels that are uncorrelated with the target feature, ultimately preventing LINE from successfully discovering it.
4.4
Ablation: Performance over Optimization Steps
To evaluate the impact of iterative loop, we extend the maximum number of iterations from 10 (plus a final Summary step) to 20. Figure 5 analyzes the step at which the optimal neuron label emerges for ResNet18 and ResNet50. Both architectures indicate that even after 10 iterations, the model still discovers novel concepts. While the most substantial progress is observed around 10 iterations, we continue to see marginal gains even at step 20. When comparing the frequency of the top-selected descriptions against the step at which they were discovered, we observe distinct spikes: the first iteration often provides a strong initial baseline, but major breakthroughs frequently occur around steps 8 and 15, with intermediate steps rarely yielding the winning concept. Importantly, the final summary step is strictly necessary, as the concept from this step most frequently achieves the highest overall activation score.
4.5
Ablation: Impact of Different Synthesize Models
The text-to-image (T2I) model is a core component of LINE. To evaluate its impact, we compare the older SD1.5, our SDXL baseline, and FLUX.1[dev] in Figure 6 on ResNet50. While concepts produced by FLUX.1 [dev] are most frequently selected as winners (43% vs. 27% for SDXL) and yield slightly higher average activations, the difference is not statistically significant (Appendix G). Interestingly, substituting the T2I model alters the top-selected concept in 80% of cases, yet these variants remain semantically and visually consistent (e.g., bookshop vs. bookcase). Qualitatively, we observe that each T2I model introduces distinct generative patterns to the images, altering visual aesthetics without compromising semantic intent. 11
concept: boat slip (score: 0.59) concept: waterfront structure (score: 0.87) concept: waterfront structure (score: 0.78)
Figure 6: Impact of text-to-image (T2I) models on LINE performance. We evaluate 30 random neurons from the avgpool layer of ResNet50 trained on Places365. The box plots (left) comparing the highest concept activation scores indicate that the FLUX model produces slightly higher activations. Corresponding descriptions and visual explanations for neuron 166 (right) illustrate the distinct generative priors of each T2I model: SD1.5 tends toward photorealistic outputs, whereas SDXL and FLUX lean into stylized aesthetics, with FLUX producing the most pronounced cinematic results.
5
Conclusion
As deep learning models become more prevalent in high-stakes applications, it becomes pivotal to ensure that the decision-making components of these models are understandable. To this end, we present LINE, the first training-free, black-box iterative pipeline for automated neuron labeling in vision models. LINE achieves state-of-the-art performance on neuron labeling benchmarks, driven by a highly transparent, iterative discovery process that yields both textual and visual explanations. Furthermore, LINE’s modular structure presents a highly scalable solution, allowing to seamlessly integrate future advancements in Language Models and text-to-image generators to push interpretability limits even higher. Limitations and Future Work. We identify three main limitations of LINE that provide natural opportunities for future research. First, the initialization step may be constrained by an imprecise initial vocabulary, especially if the investigated neurons are not closely related to the model’s predefined labels, as seen in Section 4.3. Expanding the initial vocabulary heavily increases computational overhead with diminishing returns. Conversely, while dynamically generating concepts from topactivating domain samples (Dunlap et al., 2024, Kim et al., 2024) yields more tailored concepts, it introduces a strong dependency on the specific subset of activating images. Second, because neurons are often polysemantic, LINE may converge on just one of the activating concepts while ignoring 12
others. A potential solution is to iteratively restart the pipeline while explicitly prompting the LLM and T2I models to ignore previously discovered concepts, forcing the exploration of secondary semantic triggers. Finally, a more fundamental challenge is that the LLM and T2I models must be capable of comprehending and generating the specific concepts learned by the target model. In specialized fields like medical or satellite imaging, LINE’s reliance on synthetic images becomes a bottleneck, as high-fidelity generative models for these domains are currently either non-existent or lack sufficient performance.
13
Acknowledgements Work on this project was financially supported by the Foundation for Polish Science (FNP) grant ‘Centre for Credible AI’ No. FENG.02.01-IP.05-0058/24, and carried out with the support of the Laboratory of Bioinformatics and Computational Genomics and the High Performance Computing Center of the Faculty of Mathematics and Information Science, Warsaw University of Technology.
References Kumar Ashutosh, Yossi Gandelsman, Xinlei Chen, Ishan Misra, and Rohit Girdhar. Llms can see and hear without any training. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=cJeFULIiot. Nicholas Bai, Rahul Ajay Iyer, Tuomas Oikarinen, Akshay R. Kulkarni, and Tsui-Wei Weng. Interpreting neurons in deep vision networks with language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=x1dXvvElVd. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum? id=ePUVetPKu6. Przemyslaw Biecek and Wojciech Samek. Model Science: Getting Serious About Verification, Explanation and Control of AI Systems. In Frontiers in Artificial Intelligence and Applications. IOS Press, 2025. doi: 10.3233/FAIA250784. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/ paper/index.html, 2023. Black Forest Labs. FLUX.1 [dev], 2024. URL https://github.com/black-forest-labs/ flux. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning, October 2023. Transformer Circuits Thread. Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina Höhne. Labeling neural representations with inverse recognition. Advances in Neural Information Processing Systems, 36, 2024. Josep Lopez Camuñas, Christy Li, Tamar Rott Shaham, Antonio Torralba, and Agata Lapedriza. OpenMAIA: a multimodal automated interpretability agent based on open-source models. In Mechanistic Interpretability Workshop at NeurIPS 2025, 2025. URL https://openreview.net/ forum?id=KitDRi76It. 14
Simone Carnemolla, Matteo Pennisi, Sarinda Samarasinghe, Giovanni Bellitto, Simone Palazzo, Daniela Giordano, Mubarak Shah, and Concetto Spampinato. DEXTER: Diffusion-guided EXplanations with TExtual reasoning for vision models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=baBhSzaSHI. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià GarrigaAlonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24199– 24208, 2024. Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009. Nils Feldhus and Laura Kopf. Interpreting language models through concept descriptions: A survey. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 149–162, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep features. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=NudBMY-tzDr. Neha Kalibhat, Shweta Bhardwaj, C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. In International Conference on Machine Learning, pages 15623–15638. PMLR, 2023. Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11082–11092, 2024. Laura Kopf, Philine L Bommer, Anna Hedström, Sebastian Lapuschkin, Marina Höhne, and Kirill Bykov. Cosy: Evaluating textual explanations of neurons. Advances in Neural Information Processing Systems, 37:34656–34685, 2024. Laura Kopf, Nils Feldhus, Kirill Bykov, Philine Lou Bommer, Anna Hedström, Marina MC Höhne, and Oliver Eberle. Capturing polysemanticity with PRISM: A multi-concept feature description framework. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=btJUnAPQ7j. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. International Conference on Learning Representations, 2023. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23915/distill.00007. 15
Matteo Pennisi, Giovanni Bellitto, Simone Palazzo, Isaak Kavasidis, Mubarak Shah, and Concetto Spampinato. Diffexplainer: Towards cross-modal global explanations with diffusion models. Computer Vision and Image Understanding, page 104559, 2025. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=di52zR8xgf. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3): 211–252, 2015. doi: 10.1007/s11263-015-0816-y. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum? id=mDw42ZanmE. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022. URL https://openreview.net/ forum?id=XVPqLyNxSyh. Bartlomiej Sobieski and Przemyslaw Biecek. Global counterfactual directions. In European conference on computer vision, pages 72–90. Springer, 2024. Bartlomiej Sobieski, Jakub Grzywaczewski, Bartłomiej Sadlej, Matthew Tivnan, and Przemyslaw Biecek. Rethinking visual counterfactual explanations through region constraint. In The Thirteenth International Conference on Learning Representations, 2024. Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (XAI): Toward medical xai. IEEE transactions on neural networks and learning systems, 32(11):4793–4813, 2020. 16
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2023. Hongbo Zhu and Angelo Cangelosi. Representation understanding via activation maximization. arXiv preprint arXiv:2508.07281, 2025.
17
Appendix for “LINE: LLM-based Iterative Neuron Explanations for Vision Models” A Notation
19
B Neuron Labeling Methods Comparison
20
C LINE Algorithm
22
D Prompts D.1 T2I Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 LLM Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23 23 23
E Detailed Iteration Process
29
F CoSy Benchmark Methodology
32
G Statistical Validation
34
H Causal Evaluations Pipeline
35
I
37
Extended Qualitative Comparison
18
A
Notation Table 4: Main notations used in this work.
Notation
Meaning Models & representations
f X A p D E T
Target vision model Input image domain Activation space of a specific model layer Spatial global pooling function Dimensionality of the pooled activation vector Explanation function assigning semantic descriptions Set of semantic textual descriptions Neuron labeling & evaluation
t n H At ψ ψavg st
Concept description Specific neuron identifier being analyzed LINE scoreboard Activation vector for a proposed concept t Scoring function Scoring function based on mean activation over A Scalar score produced by the scoring function for concept t Initialization phase
K M Ainit sinit
Number of classes used for initialization (1000) Number of images sampled per class (50) Initial activation matrix for analyzed neuron, Ainit ∈ RK×M Initial class-wise score vector, sinit ∈ RK LINE framework & iterations
N P r S
Number of refinement iterations (excl. summary iteration) Set of synthetic images generated by the T2I model Concept candidate sampled for removal/replacement Final summary iteration index (N + 1)
19
B
Neuron Labeling Methods Comparison
A comprehensive comparison between prior neuron labeling approaches and our proposed method (LINE) is provided below. To properly contextualize these advancements, we categorize the methods across several key characteristics, as summarized in Table 5. Table 5: Comparison of neuron labeling methods. We contrast prior approaches with LINE across several key characteristics: explanation output format, targeted neuron representation, optimization metric, and external method dependencies. We also denote whether each method is training-free and architecture-agnostic. Notably, LINE leverages LLMs and T2I models to generate rich, free-form explanations without imposing architectural constraints or requiring additional training pipelines. ∗ indicates the dependency is only required during specific phases of the method. Method
Explanation Format
Neuron Type
Target
Method Dependency
Training Free
Model Agnostic
Network Dissection (Bau et al., 2017)
fixed-label
conv.
IoU
seg. masks
✓
✓
MILAN (Hernandez et al., 2022)
free-form
conv.
WPMI
labeled corpus*, trained LSTM
✗
✓
openvocabulary
scalar
SoftWPMI
CLIP
✓
✓
openvocabulary
pre-det.
avg. CLIP score
CLIP
✓
✗
compositional
scalar
AUC
multi-labeled data
✓
✓
Describe-and-Dissect (DnD) (Bai et al., 2025)
free-form
scalar
avg. rank
I2T, LLM, T2I
✓
✓
LINE (Ours)
free-form
scalar
avg. activation
LLM, T2I
✓
✓
CLIP-Dissect (Oikarinen and Weng, 2023) FALCON (Kalibhat et al., 2023) INVERT (Bykov et al., 2024)
Explanation Format: The output format determines the expressive capacity of the explanation. Early approaches rely on fixed-label sets, which restrict the vocabulary to predefined concepts. While openvocabulary methods (e.g., CLIP-Dissect) expand this range using vision-language models (VLMs), they still require the user to provide a candidate concept vocabulary. Compositional approaches further increase complexity by building logical formulas from base concepts. Finally, free-form methods, including LINE, leverage the generative power of LLMs to produce unconstrained textual descriptions capable of capturing highly nuanced or abstract neuron behaviors. Neuron Type: The unit of analysis is often dictated by the model architecture. Early methods (e.g., NetDissect) primarily analyze spatial convolutional feature maps. To maintain relevance for modern architectures like Vision Transformers (ViTs), recent methods target specific scalar activations within a layer. Notably, while most methods are architecture-agnostic, some (e.g., FALCON) remain restricted to predetermined structures required by the tools like Grad-CAM (Selvaraju et al., 2017). Optimization Target: To map neurons to semantic concepts, methods optimize specific alignment metrics. Spatial grounding methods typically use Intersection over Union (IoU) to measure 20
overlap with segmentation masks. Other statistical approaches utilize continuous metrics such as Weighted Pointwise Mutual Information (WPMI) or Area Under the Curve (AUC). Generative and language-guided methods, such as LINE and DnD, utilize average activation or average rank to measure the correlation between the neuron’s peak firing patterns and the synthesized concept. Method Dependencies and Constraints: Requirement profiles vary significantly across methods. Fixed-label approaches historically necessitated expensive, dense segmentation masks or multi-labeled corpora, while MILAN requires a labeled corpus for training its LSTM-based captioner. In contrast, our framework leverages off-the-shelf large language models (LLMs) and text-to-image (T2I) models, ensuring it is strictly training-free and architecture-agnostic. This modularity allows for future scaling, as each component (LLM or T2I) can be upgraded in parallel with state-of-the-art advancements.
21
C
LINE Algorithm
In addition to the detailed description in Section 3.2 and the overview schema in Figure 2, we include the complete pseudocode for our iterative concept proposal and scoring algorithm in Algorithm 1. This outlines the exact step-by-step process LINE uses to initialize the search space, generate and evaluate potential concepts, and extract the best concept description. Algorithm 1: LINE Pipeline Input: Number of iterations N , target neuron n Output: Best textual concept tbest H ← initialize_scoreboard() // Initialize scoreboard for i ∈ {1, . . . , N } do t ← propose_concept(H) // LLM proposes a new concept t P ← generate_images(t) // T2I model generates image set P At ← extract_activations(n, P, t) // Collect neuron n activation st ← score_concept(At ) // Calculate concept score st H ← update_scoreboard(t, st ) // Update scoreboard with new score end tsummary , ssummary ← run_summary_step(H) // Final iteration H ← update_scoreboard(tsummary , ssummary ) // Add the final iteration score tbest ← get_best_concept(H) // Retrieve highest-scoring concept return tbest
22
D
Prompts
The complete set of prompt templates used for both the T2I and LLM components of the LINE pipeline is detailed below.
D.1
T2I Prompt
To generate diverse images for a given concept, we use the following template: Concept Image Synthesis A realistic photo of a {concept}, {angle}, {lighting} The environmental modifiers are uniformly sampled at random from the following sets of values: • angle: extreme close-up, wide angle shot, aerial view, low angle; • lighting: cinematic lighting, natural sunlight, studio lighting.
D.2
LLM Prompts
The LLM in LINE is guided by two distinct prompt templates: one utilized during the main refinement loop across all N iterations, and another exclusively for the final summary step (iteration S). To maximize the quality of the generated concepts, we employ standard reasoning techniques such as Chain-of-Thought (CoT) and few-shot prompting. This compels the model to articulate its rationale before predicting the final concept string, guided by in-context examples of the desired reasoning format. Main Loop Prompt: In the main loop, the current scoreboard H is provided to the LLM as concept_list, while all the concepts proposed by the LLM in the previous iterations are supplied in generation_history. This explicit constraint prevents the LLM model from falling into a loop of continuously proposing suboptimal solutions. Main Loop: Iterative Concept Generation You are an assistant that analyzes lists of concepts with scores and produces one new concept based on them. You MUST follow this output format: <thinking> (Short reasoning. List 3 potential concepts, check them against the forbidden list, and select the best one. Max 3 sentences.) </thinking> <answer> FINAL_CONCEPT </answer> Rules: • The <thinking> section must be extremely concise (max 8 sentences).
23
• The <answer> section contains only the final generated concept and nothing else. • You are not allowed to generate concepts present in the forbidden list. • You must generate exactly one new concept. • Keep the concept short (1–3 words). • It must relate to the highest-scored concepts. • It must generalize, unify, or creatively combine relevant concepts. • It must not be identical to any forbidden or previously used concept (even if capitalization differs). • You can be guided by the shape of objects. • Do not comment outside the required tags. • The <thinking> section must be short. Examples: List of concepts and their scores: • soccer ball: 0.93, • ladybug: 0.90, • dice: 0.87, • dalmatian: 0.84, • mushroom cap: 0.79, • tree leaf: 0.41, • hammer: 0.22, • envelope: 0.18 You are not allowed to generate these concepts: animal, insect, toy, tool <thinking> The highest-scoring concepts share a common visual pattern: they all feature black spots on a contrasting background. </thinking> <answer> polka dots </answer> List of concepts and their scores: • rain: 0.93, • wet: 0.89, • storm: 0.86, • blowing wind: 0.83, • handle: 0.77, • sandwich: 0.25,
24
• rocky mountain: 0.22, • television: 0.14 You are not allowed to generate these concepts: coat, water drops, storm, blowing wind <thinking> The high scores for rain, wet, storm, and handle suggest an object used for protection against weather. "Umbrella" fits perfectly as it has a handle and protects from rain. It is not on the forbidden list. </thinking> <answer> umbrella </answer> List of concepts and their scores: • wooden door: 0.95, • computer screen: 0.92, • old book: 0.89, • windows: 0.87, • dark chocolate: 0.83, • travel suitcase: 0.46, • rug: 0.29, • yellow flower: 0.17 You are not allowed to generate these concepts: door, book, window, screen <thinking> The highest-scoring objects all share a clear rectangular outline, defined by straight edges and right angles. </thinking> <answer> rectangle </answer> List of concepts and their scores: • pomegranate: 0.96, • strawberry: 0.96, • cherry: 0.84, • fruit: 0.77, • seeds: 0.19, • buckeye: 0.46, • aril: 0.55, • jewel: 0.38, • Bursting capsule: 0.392
25
You are not allowed to generate these concepts: fruit, seeds, apple, buckeye <thinking> Pomegranate, strawberry, and cherry are all red fruits. Since "fruit" is forbidden, I need a more specific descriptor. "Red fruit" generalizes these top items accurately and is allowed. </thinking> <answer> red fruit </answer> Task: Analyze the new list and produce a concept. List of concepts and their scores: {concept_list} You are not allowed to generate these concepts: {generation_history} Produce your answer in the required format. Keep thinking very short.
Summary Prompt: In the final iteration (S), the three highest-scoring concepts from the final scoreboard H are passed to the LLM as concept_list. The final prompt is designed to generalize and unify these top-tier concepts into a single high-level concept. Summary Iteration: Concept Summarization You are an assistant that analyzes lists of concepts with scores and generates a single summary concept. The higher the score, the more important the concept. You MUST follow this output format: <thinking> (Extremely concise reasoning. Identify the common thread among high-scoring items in 3-5 sentences max.) </thinking> <answer> SUMMARY_CONCEPT </answer> Rules: • The <thinking> section must be extremely concise (max 5 sentences). • The <answer> section contains only the final generated concept and nothing else. • Be specific and try to find the most generalizing concept.
26
• Combine or merge ideas from multiple provided concepts. • Only semantic meaning matters. • Keep the summary concept very short (ideally a single word or up to 3 words). • Do not score the new concept. • Generate exactly one concept. Examples: List of concepts and their scores: • lava: 0.94, • eruption: 0.91, • Ash: 0.78 <thinking> Lava, eruption, and ash are all direct components or results of a volcano. "Volcano" summarizes these distinct elements into a single source concept. </thinking> <answer> volcano </answer> List of concepts and their scores: • grass: 0.93, • crocodile: 0.86, • bushes: 0.83, • beach: 0.22 <thinking> Grass, crocodiles, and bushes share the visual characteristic of being green. This unifies the high-scoring concepts better than a location-based summary. </thinking> <answer> green </answer> List of concepts and their scores: • banana: 0.96, • lemon: 0.90, • pineapple: 0.84 <thinking> Banana, lemon, and pineapple are all fruits that are yellow. "Yellow fruit" captures both the category and the shared color attribute of the highest-scored items. </thinking>
27
<answer> yellow fruit </answer> Task: Analyze the new list and produce a summary concept. List of concepts and their scores: {concept_list} Produce your answer in the required format. Keep thinking very short.
28
E
Detailed Iteration Process
In this section, we provide a complete, step-by-step trace of the LINE framework’s refinement process. This trace includes the initial scoreboard (Table 6), the concept proposed at each iteration, its corresponding score, the synthesized images (visual explanations), and the LLM’s reasoning. These results supplement the final scoreboard presented in Table 3 and trace the behavior of neuron 252 (ultimately labeled as gatherer) from the ResNet50 layer4. Table 6: Initial Scoreboard for neuron 252 from ResNet50 layer4. We provide the starting scoreboard H used at the beginning of the LINE refinement process. The scoreboard contains concepts closely related to the final concept (gatherer), alongside lower-scoring, diverse concepts that facilitate broader exploration. We bold the best score and underline the second best. Concept
Score
thresher harvester hartebeest trailer truck wreck capuchin ant hoopskirt oil filter hard disc
0.74 0.88 0.78 0.43 0.51 0.29 0.17 0.10 0.06 0.12
Following the initialization, Table 7 details the inner refinement loop conducted at each step. LINE immediately identifies farming as the primary semantic path, occasionally diverging to functional concepts like cutting tool to explore the mechanistic properties of harvesting machines. However, this trace highlights two limitations: (1) the LLM occasionally stalls refinement by proposing duplicate concepts (e.g., separation in steps 7 and 10), and (2) it fixates on the highest-scoring path (agriculture), neglecting secondary features that could also capture the neuron’s inherent polysemanticity. Finally, Table 8 presents the full set of synthesized images per step. The visual variance observed for vague concepts like separation arises from randomness in the prompt (Section D). We found that for these highly abstract concepts, modifying visual characteristics (e.g., lighting, camera angle) often overshadows the core prompt during T2I generation.
29
Table 7: Iteration process of LINE. We provide the outputs from the LLM, the T2I, and the scoring function for each iterative step, building from them the Table 6. The best and second-best scores are highlighted in bold and underlined, respectively. Step
Concept
Score
Reasoning
T2I Image
1
harvesting machine
1.09
The top-scoring concepts are related to farming and harvesting. They all feature a machine or vehicle used for collecting or gathering something.
2
reaper
0.37
The top-scoring concepts are all related to agricultural equipment. They have a common theme of collecting or gathering.
3
cutting tool
0.39
The top-scoring concepts are harvesting machine, thresher, and harvester, all related to agricultural tools. They share a common theme of cutting or gathering crops.
4
reap
0.43
The top-scoring concepts are all related to agricultural machinery. They have a common theme of harvesting or reaping.
5
harvest
0.33
The top-scoring concepts are all related to agricultural activities, with a focus on harvesting or reaping.
6
gatherer
1.33
The top-scoring concepts are all related to agricultural activities or machines used for them. They share a common theme of gathering or collecting.
7
separation
0.39
The top-scoring concepts are related to farming and agriculture. They all involve collecting or removing crops by separating them from a larger entity.
8
threshing
0.98
The top-scoring concepts are related to farming and gathering. The process involves combining or generalizing these concepts.
9
debris removal
0.71
The top-scoring concepts involve collecting or processing crops, sharing a common thread of separating or removing unwanted parts.
10
separation (duplicate)
0.32
The concepts relate to farming and agriculture, specifically the idea of separating or removing unwanted parts during processing.
S
farming
0.52
Gatherer, harvesting machine, and threshing all share a common theme of agricultural activities. The high scores suggest a strong connection to farming.
30
Table 8: Iteration process of LINE (synthesized images). Expanding on Table 7, we present the full set of images generated by the T2I model for each step. Step
Concept
1
harvesting machine
2
reaper
3
cutting tool
4
reap
5
harvest
6
gatherer
7
separation
8
threshing
9
debris removal
10
separation (duplicate)
S
farming
Synthesized Images
31
F
CoSy Benchmark Methodology
Figure 7: CoSy evaluation framework. A schematic illustration of the CoSy framework for Neuron 80 in ResNet18’s avgpool layer. The figure is sourced from the original paper (Kopf et al., 2024). The CoSy benchmark (Kopf et al., 2024) evaluates the quality of open-vocabulary textual explanations for vision model neurons. Given the difficulty of finding natural datasets that perfectly isolate arbitrary concepts, CoSy leverages T2I generative models to synthesize validation data, similarly to how LINE generates concept images. The assumptions of CoSy are that if a concept label t accurately explains a neuron n, the neuron should activate significantly more on generated images representing t than on a baseline control (ImageNet validation (Russakovsky et al., 2015)) set of random natural images. The evaluation process, which is illustrated in Figure 7, follows three steps: 1. Generate Synthetic Data. Given a proposed concept label t (e.g., polka dots), a generative T2I model is used to synthesize a collection of images, denoted as in our work P. 2. Collect Neuron Activations. Both the synthetic images P and a control dataset of natural images Xcontrol ⊂ X are passed through the target vision network f and the activations are extracted from the specific neuron n. This yields two sets of scalar activations: • At : Activations from the synthetic concept images in P. • Acontrol : Activations from the natural control images in Xcontrol . 3. Score Explanations. A scoring function ψ(Acontrol , At ) is used to quantify the difference between the two activation distributions. A higher score indicates that the concept t is a better match for the neuron n. The benchmark evaluates these explanations using two complementary scoring functions that capture different aspects of neuron behavior: Area Under the ROC Curve (AUC): A non-parametric metric evaluating the neuron’s ability to act as a binary classifier. It represents the probability that a synthetic concept image activates the neuron 32
more strongly than a random control image: ψAUC (Acontrol , At ) =
1
X
|Acontrol | · |At |
X
1[a < b]
(3)
a∈Acontrol b∈At
where 1[·] is the indicator function. AUC is robust to outliers. Mean Activation Difference (MAD): A parametric metric measuring the magnitude of the activation shift. It calculates the difference in mean activations between the synthetic and control images, normalized by the control’s standard deviation: ψMAD (Acontrol , At ) =
µ(At ) − µ(Acontrol ) σ(Acontrol )
(4)
where µ and σ denote the mean and standard deviation. MAD quantifies the absolute extent to which concept t maximizes the neuron’s output.
33
G
Statistical Validation
To substantiate the hypothesized superiority of LINE on the CoSy benchmark, we perform a rigorous statistical significance analysis. This evaluation further ensures performance consistency across different T2I models. All statistical tests were conducted at a significance level of α = 0.05. First, we applied the Shapiro-Wilk test to determine whether the score differences follow a normal distribution. If the null hypothesis of normality could not be rejected (p > 0.05), we proceeded with a paired Student’s t-test. Otherwise, we used the nonparametric Wilcoxon signed-rank test, as the t-test assumes normality. Table 9: Statistical significance test results comparing LINE and prior methods. Expanding upon the CoSy benchmark results from Table 1, we calculate the statistical significance of the differences between LINE and the baselines (CLIP-Dissect, INVERT, and DnD). The p-values confirm that LINE achieves statistically significant improvements in almost all configurations, with the sole exception being the comparison against CLIP-Dissect on the Places365 dataset. Underlined values indicate a lack of statistical significance (p > 0.05). Model
Dataset
ResNet50
ImageNet
ResNet50
Places365
ViT-B/16
ImageNet
Method of reference
AUC
MAD
CLIP-Dissect INVERT Describe-and-Dissect (DnD) CLIP-Dissect INVERT CLIP-Dissect INVERT
0.0148 < 10−4 < 10−4 0.0926 < 10−4 < 10−4 < 10−4
0.0051 < 10−4 < 10−4 0.2371 < 10−4 < 10−4 < 10−4
Table 10: Statistical significance tests across T2I models. Building on the comparisons in Figure 6, we calculate the statistical significance of the activation differences for 30 randomly selected neurons from the ResNet50 avgpool layer when substituting the default SDXL model used in LINE with alternative models (FLUX.1[dev] and SD1.5). The reported p-values confirm that there is no statistically significant difference in performance between the models (p > 0.05). Model of reference
p-value
FLUX SD1.5
0.3054 0.4476
34
H
Causal Evaluations Pipeline
In this section, we detail the pipeline for generating ablated samples by removing the visual concepts identified by LINE (as illustrated in Figure 3). To remove the visual concept t (associated with neuron n) from an original image x, we utilized an image-editing generative model, specifically Qwen-Image-Edit-2511. Given input image x and a modification prompt, the model synthesizes the edited image x̂. The prompt is constructed as follows: Concept Image Removing Remove the {concept} from the image where {concept} corresponds to t. Using this setup, we generated ablated versions of the highly activating images from ImageNet-1K for a random subset of neurons in the ResNet50 avgpool layer, based on the descriptions provided by LINE. The resulting visual ablations, along with the corresponding changes in neuron activation, were initially presented in Figure 3. Figure 8 provides a broader set of edited images that highlight the current limitations of this automated ablation pipeline. Specifically, this trace reveals two primary issues: (1) the generative model occasionally fails to completely remove the targeted concept or erroneously alters unrelated features, requiring manual inspection; and (2) the resolution of the edited image x̂ sometimes differs from the original image x, which may introduce unexpected artifacts into the model’s behavior and artificially decrease activations (a phenomenon to which we attribute some of the activation increases observed in neuron 19).
35
Figure 8: Ablation of visual concepts derived from LINE. We show side-by-side comparisons demonstrating the effect of removing the LINE-defined concept t from highly activating images for the ResNet50 avgpool layer using an image-editing generative model. Concept removal via generative models is not always perfect and still requires manual inspection; for instance, removal failures can be observed in the 5th row for neuron 403 and the 1st row for neuron 49, whereas neuron 19 demonstrates largely successful ablations, but causing an increase in activation for two of five samples. Original images are outlined in blue, ablated versions in orange, and the resulting relative change in activation is marked green if there has been a decrease, or red if there has been an increase.
36
I
Extended Qualitative Comparison
To supplement the visualizations in Section 4.2, we provide additional qualitative results across several model architectures. Specifically, Figure 9 presents 6 classes from a RobustResNet50 model trained on ImageNet-1K. Building on the evaluation protocol from the DEXTER (Carnemolla et al., 2025) and extending the results from Figure 4, we selected 3 classes containing heavily spurious features and 3 classes relaying mainly on core features from Salient ImageNet (Singla and Feizi, 2022). For each evaluated class, we provided visual explanations of the top-5 features, categorized as either “core” (for the core classes) or “spurious” (for the spurious classes). In Table 11 we summarize all evaluated features from Salient ImageNet, alongside the corresponding feature labels generated by LINE and DEXTER. Furthermore, we present extended comparisons of the neuron descriptions generated by LINE, CLIP-Dissect, and INVERT for randomly selected neurons across different network architectures and layers in (Figure 10 and Figure 11)
Figure 9: Extended visual explanations on Salient ImageNet. Extending the Figure 4, we present results for 3 core classes (left) and 3 biased classes (right). Each class subfigure displays the top-5 features identified in the dataset, along with their attribution maps and visual from DiffExplainer, DEXTER, and LINE. Activation magnitudes appear above each image; LINE feature descriptions are provided below. (Continued on next page)
37
Figure 9: Continued extended visual explanations on Salient ImageNet. Extending the Figure 4, we present results for 3 core classes (left) and 3 biased classes (right). Each class subfigure displays the top-5 features identified in the dataset, along with their attribution maps and visual from DiffExplainer, DEXTER, and LINE. Activation magnitudes appear above each image; LINE feature descriptions are provided below.
38
Table 11: Detailed feature description and label comparison on Salient ImageNet. Expanding upon the visualizations presented in Figure 9, we summarize the class and top feature descriptions while indicating whether the evaluated classes exhibit bias (✗) or not (✓). Alongside each evaluated feature, we report the labels produced by DEXTER (using the prompt "a picture of a {label}") and LINE. The generated labels confirm the visual explanations provided in Figure 9, and notably, descriptions from both methods exhibit strong alignment for the non-biased classes. Class (idx)
Non-Bias
Jeep (609)
✓
Daisy (985)
✓
Rifle (764)
✓
Patio (706)
✗
Bathing Cap (433)
✗
Seat Belt (785)
✗
# Feature
DEXTER
LINE
1067 1100 1208 1515 691 1105 120 298 595 859 1259 1928 400 515 522 1016 1633 194 451 654 121 1340 1591 1609 755 1010 108 116 1493 50
tractor truck tractor truck ford flower daisy daisy daisy daisy pistol gun rifle gun pistol porch chair window residence room child shoulder look baby population leg face field railway park
tractors jeep tractors minivan minivan sulphur-crested cockatoo ground beetle daisy daisy centered creature heavy hauler damselfly musical revolver revolver assault rifle china cabinet animal cart microwave golfcart barber chair monkey cap swimwear colobus shower cap leather motor scooter eye cover car mirror car mirror steam locomotive
39
ResNet50 (Layer4)
ResNet50 (Avgpool)
Figure 10: Qualitative comparison of neuron descriptions in ResNet50. We extend the qualitative analysis from Figure 1 on the ResNet50 avgpool and layer4 layers. We show the top four activating images from ImageNet-1K alongside descriptions from LINE, CLIP-Dissect, and INVERT, and their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold. 40
ResNet18 (Avgpool)
ViT-B/16 (Heads)
Figure 11: Qualitative comparison of neuron descriptions in ResNet18 and ViT-B/16. We extend the qualitative analysis from Figure 1, focusing on the ResNet18 avgpool layer and ViT-B/16 heads. We present the top four activating images from ImageNet-1K, along with descriptions from LINE, CLIP-Dissect, and INVERT, alongside their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold. 41