Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

arXiv:2611.03597v1 [cs.CV] 2 Nov 2026

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks Yusong Zhao

Edgar Y. Walker

yusongz@illinois.edu University of Illinois Urbana-Champaign Urbana, Illinois, United States

eywalker@uw.edu University of Washington Seattle, Washington, United States

Hao Wang

Mi Lu

hwangml@illinois.edu University of Illinois Urbana-Champaign Urbana, Illinois, United States

lmi7@gatech.edu Georgia Institute of Technology Atlanta, Georgia, United States

Abstract

can demonstrate mixed selectivity [65], where they can respond to images with small overlap in shared features. As presented in Figure 1, the top images that maximally activate the brain region are distributed across the embedding space of the CLIP encoder [64]. Additionally, brain activity measured is often noisy, low-resolution, and partially observed, which introduces difficulties for analysis and modeling. Unlike previous works that aim to reconstruct entire images from brain activities [25, 48, 53, 71], our work focuses on understanding the feature representation of brain regions by generating new preferred images that maximize or minimize the activity of those regions. These synthetic visual stimuli can be deployed in follow-up neuroscience experiments to test new hypotheses, refine ROI functional maps, and accelerate discovery by guiding stimulus design. To achieve this goal, we propose Vis-Lens, an effective approach to interpret the feature represented by brain regions. Given the noisy nature of brain recordings, instead of using brain activities directly as input to the decoding model, our method refines the landscape of visual representation by employing brain activation to regularize the prior, serving as a soft constraint. We further integrate the structured landscape with a variational auto-encoder (VAE) and diffusion generative models to generate realistic preferred images that can effectively modulate target brain activations. Our method can facilitate the exploration of visual representations of brain regions while maintaining interpretability. It is validated with fMRI data from the Natural Scenes Dataset (NSD) [2], analyzing responses in the human visual cortex to complex natural stimuli. Meanwhile, due to the scarcity of image response data from human brains, our proposed framework is also extended to artificial neural networks (ANNs), allowing controlled testing of visual feature representations and interpretability in artificial systems. By integrating insights from both biological and artificial neural networks, we aim to bridge the understanding of visual representation strategies across natural and engineered systems. Our contributions can be outlined as follows:

Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of blackbox artificial neural networks and for understanding the visual representation in the human brain. We propose a structured visual representations landscape and design an activation-based prior that allows us to effectively regularize the landscape with activations from brain regions or neural network units. Our model Vis-Lens integrates a variational autoencoder and a diffusion model into an image generation framework. It enables the generation of natural, realistic preferred images by directly modifying the activationregularized latent codes without per-image iterative optimization. We demonstrate the effectiveness of our framework on both artificial neural networks and human visual cortex using functional magnetic resonance imaging (fMRI) data. Our framework improves activation modulation and image realism compared with a recent state-of-the-art method for generating preferred visual representations of these neural targets.

Keywords visual representation, brain decoding, variational autoencoder, preferred images, neural networks, feature visualization

1

Introduction

The exploration of visual representations in both biological and artificial neural networks has advanced our understanding of how complex visual stimuli are processed and represented in neural networks [4, 27, 41]. Previous studies have demonstrated the capacity of neurons in the medial temporal lobe (MTL) of the human brain to form invariant representations of complex stimuli [63], such as faces, landmarks, and objects, regardless of visual variations. These findings illustrate how single neurons can encode high-level, abstract percepts with remarkable specificity, leading to the hypothesis of sparse and invariant coding mechanisms in the brain [56]. While methods exploring visual representation in some brain regions work fine by learning directly from embeddings of popular image encoders like CLIP [17, 37], understanding the feature representations for many brain regions that are not well-studied remains challenging. For example, some higher-order brain regions

• We introduce an activation-regularized prior for VAE [30] that organizes its latent space into two distinct clusters, corresponding to codes that increase or decrease a target brain region’s activation. This design enables easy plugand-play integration with conventional generative models. • Our method synthesizes visual representations from specified brain regions, showing better performance to regularize 1

Zhao et al.

Mean Activations: 0.103

Brain Activations

CLIP Embedding

Top 5 Images Mean Activations: 0.233

CLIP Embedding Dim 1

Generated Top Images

Dim 2

Increase Activation

Original CLIP Space Sa Brain Activations mp lin g

Mean Activations: -0.029

Max

g lin mp Sa

Decrease Activation

Min

Structured Latent Dim 1

2 Structured Latent Dim

Bottom 5 Images Mean Activations: -0.179

Structured Latent Space

Generated Bottom Images

Figure 1: Original CLIP Space (Top): Original CLIP embedding space without any activation constraint: image embeddings that elicit high and low neural responses are dispersed and not clustered throughout the space without shared visual structure. Structured Landscape (Bottom): By encoding CLIP embeddings into the regularized latent space constrained with neuronal activations (after z-score normalization) as the prior, our method Vis-Lens could provide a more structured and representative landscape to better identify the representations and preferences of brain regions than sorting top-k images in a dataset. The results are based on brain activations from the NSD dataset [2]. In the landscape plots, the color bar and z axis indicate the activation of the target brain region given different images as inputs, while the x and y axes represent two latent dimensions, respectively. brain activations than a recent state-of-the-art baseline, especially for decreasing activations. It could also generate preferred visual inputs with better perceptual realism. Moreover, it provides a generalized framework which shows robustness on cross-subject biological data, and can also be transferred to visualizing features in artificial neural networks. • By generating controllable, high-quality visual stimuli that can modulate specific regions, our framework demonstrates new potential in vision-based brain computer interface applications and can be used to generate new hypotheses for future neuroscience studies.

2

enhanced synthesis quality. More superior generation models like Diffusion models [23, 67] advanced controllable, high-fidelity synthesis, integrating structured perturbations via SDEdit [42] and text-to-image alignment through CLIP [64]. IP-Adapter [83] refined latent spaces for semantic consistency, while GLIDE [52] and Stable Diffusion [67] further demonstrated photorealistic, text-guided image generation. Our work is developed based on these conventional generative model frameworks, it adopts VAEs to build up our proposed structured landscape and a Stable Diffusion model generation pipeline to synthesize preferred images conditioned on CLIP embeddings derived from the structured landscape. Feature visualization. Visualization techniques provide deep insights into neural network interpretability and functionality, spanning both biological and artificial systems. Early work by Erhan et al. [15] focused on minimal regularization approaches, while adversarial examples exposed vulnerabilities in neural networks [74]. Mahendran and Vedaldi [39] and Nguyen et al. [50] further expanded on interpretability with total variation regularization, counterexamples, and image blurring. Meanwhile, techniques such as DeepDream by Mordvintsev et al. [45, 46], employing jitter, multiscale visualization, and gradient normalization [54, 76], enhanced

Related Works

Image generative models. Advances in generative models have significantly transformed image synthesis. Foundational approaches such as GANs [18] led to extensions like Wasserstein GANs [3] and Conditional GANs [43], with fidelity improvements using spectral normalization [44], while VAEs [30] introduced probabilistic encoding, later combined with latent space regularization [24, 77]. Further, Cascaded generation [24] and autoregressive modeling [9] 2

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

3 Methodology 3.1 Problem Setting and Notation

clarity in feature visualization. For prior related work, Nguyen et al. [51] demonstrated generative adversarial synthesis of preferred inputs, emphasizing controlled image generation through learned priors. Our work extends this line by generating images with preferred features based on pre-defined priors in landscape shaped by activations. Brain decoding and most exciting images. Generative models leveraging brain activity as direct input have substantially advanced neural-to-visual synthesis. Direct neural-to-image synthesis methods convert fMRI activity into high-fidelity pictures with diffusion or GAN backbones, like BrainDiffusion and NeuroDM [37, 61], Brain2GAN [11], latent-diffusion variants [16, 58], and energy-guided approaches [60], while activation-optimized methods like Inception Loops [80] and NeuroGen [19] reveal neuronal tuning by iterating stimuli based on neural responses. Recent brain decoding studies have continued to improve visual reconstruction quality through stronger priors, better alignment spaces, and cross-subject or data-efficient formulations, including generative decoding with latent diffusion [58, 75], diffusion-prior-based reconstruction [69, 70], probabilistic visual-to-fMRI synthesis [40], and cross-subject decoding frameworks [81]. Recent multi-modal approaches [6, 14, 26, 28, 62, 79] integrate generative models for improved synthesis precision, while Controllable Mind Visual Diffusion [84], Seeing Beyond the Brain [10], and Reconstructing the Mind’s Eye [69] leverage structured diffusion priors for enhanced visual reconstructions. Beyond reconstruction, recent methods have also explored richer alignment spaces between brain signals and foundation-model representations, such as CLIP-based decoding [35] and self-supervised visual representations including DINO-style encoders [7]. Though dominated by fMRI data, [1, 12, 32, 33, 38, 73] explored on EEG data, obtained promising results. To further modulate activations, computational models [47, 59] and frameworks targeting single neuron contributions [5, 55] provide interpretable mappings of cortical activations to specific visual features. Latent space optimization strategies [66, 82] extend foundational decoding frameworks [25, 48, 53, 71] to achieve targeted neural activation modulation. BrainDiVE and BrainSCUBA[36, 37] uses a CLIP-based encoder and diffusion models to improve stimulus quality and semantic specificity. BrainACTIV [17], as the recent state-of-the-art and the first work on using diffusion models to generate images to regulate brain activations, extends these paradigms by conditioning synthesis with linearly fitted CLIP embeddings and brain activation patterns. Related efforts in controllable activation synthesis have also shown that generative models can be used not only to maximize neural responses but also to suppress or differentially modulate them, both in biological systems and in artificial networks [4, 49, 72]. Although modeling activations via CLIP embeddings does work in previous work, our method tries to take the noisy and mix-selective nature of brain activity data into consideration, building a novel structured landscape in VAE through CLIP embeddings, permitting more controllable and effective activation manipulation, and avoiding falling into local minimum in the optimization process.

Let I be the space of natural images, X be the space of CLIP embeddings, and Z be the latent space of a variational auto-encoder (VAE). For an input image 𝑖 ∈ I, the pretrained CLIP encoder E maps 𝑖 to a CLIP embedding 𝑥 = E (𝑖) ∈ X. Subsequently, the VAE encoder 𝑞𝜙 (𝑧 | 𝑥) produces a latent 𝑧 ∈ Z with mean 𝜇 and diagonal covariance Σ = diag(𝜎 2 ); the VAE decoder 𝑝𝜃 (𝑥 | 𝑧) reconstructs an embedding 𝑥 rec . Throughout, 𝑟 denotes a scalar activation value of a chosen target, i.e., the mean response of a selected ROI in human brain data after z-score normalization or the response of a selected unit in an artificial neural network. 𝜆, 𝜔 rec, 𝜔 KLD are scalar hyper-parameters.

3.2

Constructing a Structured VAE Landscape with Activation-Constrained Priors

To generate explainable and controllable preferred visual representations for a specific brain region or neural unit, we incorporate the target activation 𝑟 into the VAE prior and use it to organize the latent space. The key intuition is that neural signals are noisy and could have an asymmetric distribution. To better represent those properties, we design a prior for the neural responses that they have a bimodal distribution, with significant responses (most positive or negative) are expected to be centralized in each distribution. The greater the absolute value of activation is, the closer it will be to each distribution center, and responses close to zero are relatively loosely constrained. This design yields a structured latent landscape that supports direct latent manipulation at test time. The KL divergence (KLD) term in the VAE measures the divergence between the encoder’s approximate posterior, 𝑞𝜙 (z|x), and the prior, 𝑝𝜃 (z). We begin with its standard definition: ∫ 𝑞𝜙 (z|x) 𝐷 KL (𝑞𝜙 (z|x) ∥ 𝑝𝜃 (z)) = 𝑞𝜙 (z|x) log 𝑑z 𝑝𝜃 (z) (1)   = Ez∼𝑞𝜙 (z|x) log 𝑞𝜙 (z|x) − log 𝑝𝜃 (z) . Both posterior and prior are defined as multivariate Gaussian distributions with diagonal covariance matrices. In our framework, the approximate posterior is 𝑞𝜙 (z | x) = N (z; 𝝁, diag(𝝈 2 )). For the prior, we define an activation-constrained prior: 𝑝𝜃 (z) = N (z; 𝝁 prior, diag((𝝈 prior ) 2 )), which depends on the target activation value 𝑟 : ( 𝜇 pos, if 𝑟 > 0, 𝜆 prior prior , 𝜇𝑖 = 𝜎𝑖 = 𝑟𝑖 + 𝜖 𝜇 neg, otherwise.

(2)

Here, 𝜆 is a scaling hyperparameter, 𝜖 is a small constant to prevent division by zero, and 𝜇 pos , 𝜇 neg are predefined means for positive and negative activations, respectively. The role of 𝜇 pos and 𝜇neg is to provide two separated latent centers in VAE latent space for maximize activation increase and decrease. Meanwhile, variance prior term 𝜎𝑖 , which defines different priors for each data point 𝑖, decrease as |𝑟𝑖 | increases. By imposing such continuous loose to tight constraints for each data point, our method can map images to a structured space. Theoretically, the inputs will be centralized when 𝑟𝑖 approaches infinity, while in practice, we can sample input data 3

Zhao et al.

Phase I: Build Activation Constrained Landscape

VAE Encoder

Training Pairs

Training Images CLIP Encoder 𝜺

𝝁𝒛|𝒙 = 𝝁𝒑𝒓𝒊𝒐𝒓

VAE Decoder

𝒒𝝓 (𝒛|𝒙)

Reconstructed CLIP Embeddings

𝒑𝜽 (𝒙|𝒛) 𝚺𝒛|𝒙 =

CLIP embeddings (𝒙) +

Input images (𝒊𝒕𝒓𝒂𝒊𝒏 )

Structured Latent Space

𝟏 |𝒓|

𝒙𝒓𝒆𝒄𝒐𝒏

Activation-regularized prior Activations (r) Reconstruction Loss + KL Divergence

Phase II: Generate Preferred Images with Structured Latent Space Encode Testing Images into CLIP Embeddings

CLIP Encoder 𝜺

Input images (𝒊𝒕𝒆𝒔𝒕)

Trained VAE Encoder

𝒒𝝓 (𝒛|𝒙)

Interpolate in Structured Latent Space

z

Trained VAE Decoder

max

Preferred Images

𝒑𝜽 (𝒙|𝒛′)

min

CLIP embeddings (𝒙)

Guide Diffusion Model With CLIP Embeddings

Generated CLIP embeddings with preferred semantics (𝒙𝒈𝒆𝒏)

z → z’

Generated preferred images (𝒊𝒈𝒆𝒏)

Figure 2: Overview of Vis-Lens. The illustration depicts a two-stage approach integrating a structured landscape and variational autoencoder (VAE) generation with CLIP encoder and diffusion based generative model. Phase I: The input image, 𝑖 train , is encoded by a pretrained CLIP encoder E, CLIP embeddings 𝑥 are encoded into a latent representation 𝑧 by the VAE encoder. A regularization step using a activation-based prior N (𝜇prior, |𝑟1 | ) modifies the latent space for better alignment with activations, producing a regularized latent 𝑧. The VAE is also trained to reconstruct the input CLIP embeddings 𝑥 with 𝑥𝑟𝑒𝑐𝑜𝑛 . Phase II: The test images 𝑖 test , is encoded by same CLIP encoder E. Then, with fully trained VAE and the structured visual latent representation space, we can directly modify latents constrained by a specific feature represented by the selected regions, obtaining a new latent 𝑧 ′ . This transformed latent is decoded back into an intermediate generated CLIP embeddings 𝑥𝑔𝑒𝑛 , which is used to guild the image generation through a diffusion pipeline, yielding the final output 𝑖 gen as preferred images to modulate the activation 𝑟. points with maximize and minimize values at two pre-defined centers, and they are expected to increase or decrease the activations of targeted neurons. Given the definition of the approximate posterior and activationconstrained prior, the expectation in Eq. 1 can be computed analytically, yielding the closed-form KLD between two Gaussian distributions. Assuming that the VAE latent space has 𝑑𝑧 dimensions, the final activation-aware KLD is " # prior prior 𝑑𝑧 ∑︁ 𝜎𝑖 1 𝜎𝑖2 + (𝜇𝑖 − 𝜇𝑖 ) 2 KLDactivation = log − + . (3) prior 2 𝜎𝑖 2 2(𝜎 ) 𝑖=1

the prior. This modified KLD does more than regularize the latent space toward a standard Gaussian: it explicitly organizes latent codes according to activations. As a result, the learned latent space becomes more suitable for controllable activation modulation than a single linear direction in the original embedding space. With this activation-constrained prior, we can build a structured VAE latent space, where we can easily perform in-distribution sampling from a spherical space in a structured landscape.

Here, 𝜇𝑖 and 𝜎𝑖 are the mean and standard deviation of the apprior proximate posterior for the 𝑖-th latent dimension, respectively; 𝜇𝑖 prior and 𝜎𝑖 are the corresponding mean and standard deviation of

To produce preferred images, our method consists of two phases: a training phase that constructs the structured latent space, and a generation phase that synthesizes images from it. The VAE is

3.3

𝑖

4

Generate Preferred Visual Representations with Modified VAE

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

(A)

(B) Predicted Activations (r) 0.6

Ours

BrainACTIV

Ours

0.0

BrainACTIV -0.8

IPS4

0.6

Ours 0.0

BrainACTIV -0.8

SPL1

0.4

Ours 0.0

BrainACTIV -0.4

31a

𝑧

𝜇!"# -1.0

-0.8

-0.6

-0.4

-0.2

0.0

𝜇$%& 0.2

0.4

0.6

0.8

Interpolation (𝛼)

1.0

Interpolation (𝜶)

Figure 3: Activation change results of our method and BrainACTIV [17] (A) Generate preferred representations of multiple ROIs with example images from COCO dataset by our method and baselines. The alpha value in interpolation indicates the strength and direction of preferred semantic in generation. (B) Comparison between predicted activations of ROIs made by DINO-ViT encoder. Means and ranges of activations are presented for different alpha settings. Our method outperforms the baseline in terms of maximizing or minimizing the ROI activations. the only trainable component in the pipeline, while the pretrained CLIP encoder (E) and the image-to-image diffusion model remain frozen. Training. During training, only the VAE encoder 𝑞𝜙 (𝑧 | 𝑥) and decoder 𝑝𝜃 (𝑥 | 𝑧), which map the CLIP embedding to a latent representation 𝑧 and reconstruct it back to 𝑥 recon , are optimized by LVAE = 𝜔 recon · Lrecon + 𝜔 KLD · KLDactivation,

passed to the frozen diffusion model to synthesize the final preferred image 𝑖 ′ .

4 Experiments 4.1 Datasets

(4)

Natural Scenes Dataset (NSD). The Natural Scenes Dataset (NSD), introduced by Allen et al. [2], provides high-resolution fMRI recordings from eight subjects viewing approximately 10,000 MS COCO images [34], each repeated three times to enhance data consistency. Following standard fMRI encoding practices [48, 78], beta values are z-scored within sessions and averaged across repetitions. ROI masks are utilized to extract activation patterns from specific brain regions, serving as essential constraints in our framework to align the VAE latent space with observed brain responses in 4.3.1. ImageNet-mini. ImageNet-mini is a subset of the full ImageNet dataset [13], offering a diverse collection of images across various categories. This dataset serves as a benchmark for evaluating the performance of modulating artificial units from SIMCLR [8] in 4.3.2 and validating optimization performance under constrained image dimensions in 4.3.3.

where Lrecon = ∥𝑥 − 𝑥 recon ∥ 22 is the reconstruction loss between the original and decoded CLIP embeddings. The terms 𝜔 recon and 𝜔 KLD balance reconstruction fidelity against the activation-based regularization imposed by Eq. 3. Therefore, the VAE is trained not only to reconstruct CLIP embeddings, but also to reshape the latent space such that neural activation information becomes geometrically accessible for later manipulation. Generation. To synthesize new images, we first encode an input image 𝑖 with the frozen CLIP encoder to obtain its CLIP embedding 𝑥 = E (𝑖). We then feed 𝑥 into the trained VAE encoder to obtain a latent code 𝑧 = 𝑞𝜙 (𝑥) in the structured latent landscape. Next, 𝑧 is moved toward the predefined positive or negative prior center, producing a modified latent code 𝑧 ′ . The modified latent code is decoded into a new CLIP embedding 𝑥 ′ = 𝑝𝜃 (𝑥 | 𝑧 ′ ), which is then 5

Zhao et al.

Table 1: Comparison of activation change and realism between our method and the baseline on NSD datasets. Better values are in bold face. For activation changes, ↑ indicates that higher values are better, and ↓ indicates that lower values are better. Our method produces activation changes that are more consistently aligned with the intended direction. (a) NSD Results

ROIs

TE2p

IP0

IP1

OFA

FFA

EBA

OPA

Activation Increase (↑)

Our Method BrainACTIV

0.242 0.207

0.498 0.463

0.153 0.044

1.052 1.155

0.891 0.976

0.963 1.160

0.699 0.699

Activation Decrease (↓)

Our Method BrainACTIV

-0.136 0.149

-0.699 0.186

-0.073 0.092

-0.660 0.213

-0.580 -0.484

-0.810 -0.669

-0.824 -0.172

FID @ 2k (↓)

Our Method BrainACTIV

172.71 225.77

173.34 213.17

178.01 200.14

173.66 216.15

153.70 256.44

137.33 188.59

166.01 225.37

Table 2: Cross-subject generalization results on the NSD dataset. The table compares predicted activation changes obtained from the original subject-specific predictor and a held-out-subject predictor across several visual ROIs.

4.2

ROIs

OFA

FFA

EBA

VWFA

OPA

PPA

RSC

Activation Increase (original subj.) Activation Increase (held-out subj.)

0.869 0.861

0.851 0.975

0.860 0.790

0.867 0.648

0.731 0.629

0.966 1.061

0.942 0.940

Activation Decrease (original subj.) Activation Decrease (held-out subj.)

-0.526 -0.436

-0.382 -0.302

-0.564 -0.391

-0.279 -0.089

-0.557 -0.485

-0.610 -0.813

-0.775 -0.823

Models

4.3

Implementation of our framework. Our VAE uses a linear encoder to map CLIP embeddings into an activation-constrained latent space and a linear decoder to reconstruct CLIP embeddings for controlled image synthesis. We use a pretrained CLIP-ViT-H-14 encoder [64] to extract 1024-dimensional embeddings, and Stable Diffusion v1.5 with IP-Adapter [83] as the image generator. Baseline method. For every ROI, we re-implement BrainACTIV [17], a ridge-regularized linear model on ℓ2 -normalized CLIP embeddings, [ ∥𝑥𝑥 ∥ ·w + 𝑏 ] =⇒ 𝑟 , yielding a single modulation vector w ∈ R1024 . This vector is interpreted as the direction of maximal and minimal activation in CLIP space: zmax = w/∥w∥ and zmin = −w/∥w∥. At inference time, a test CLIP embedding is shifted only along zmax or zmin before being injected, via an unchanged adapter, into a frozen image-to-image diffusion model. This baseline therefore assumes that activation increase and activation decrease can be represented by a single symmetric direction in CLIP space, whereas our method models them independently. In-silico brain simulator. In our experiments, brain activations are evaluated with a DINO-ViT encoder [57]. DINO acts as a frozen feature extractor whose 12 layer outputs are fed to a single-layer ViT ensemble trained with ridge loss to predict neuronal responses. In line with the findings of [17], this encoder explains a large proportion of NSD voxel variance (typically 𝑅 2 > 0.6 in high-level ROIs), validating its use as a reliable neural readout. Because the encoder is not CLIP-based, it provides an independent estimate of neural activity.

Results

4.3.1 Generate Preferred Representations in Human Brain. With the structured landscape, we can obtain a better representation of a target brain region by sampling around the corresponding prior in latent space and generating images from those latent values, as shown in Figure 1. In our latent space, we interpolate each latent code 𝑧 toward a positive or negative prior, 𝑧 ′ = 𝛼 · 𝜇prior + (1 − 𝛼) · 𝑧, and pass it to a frozen SDEdit diffusion model with 𝛾 = 1.0; thus, the output image depends solely on the edited embedding, allowing a clean comparison with the baseline. Figure 3 and Table 1 show results on the NSD dataset. Our method produces activation changes in the intended direction across all reported ROIs, whereas the BrainACTIV [17] fails to do so in several cases, especially for activation decrease. In particular, our method yields a larger decrease in activation in all reported regions, supporting the use of an independent negative prior rather than simply reversing the maximal direction used in the BrainACTIV. For activation increase, the advantage of our method is most evident in less-studied or mixed-selective regions such as TE2p, IP0, and IP1, while performance is more comparable in strongly categoryselective regions such as OFA, FFA, EBA, and OPA. Our method also consistently produces more realistic visual representations, as indicated by lower FID scores [22]. Those results demonstrate that, compared with linear interpolation method in BrainACTIV, which the negative direction is simply the reverse of the positive one, the key advantage of our method is constructing a structured space to make in-distribution sampling easier, mitigating out-of-distribution 6

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Top 10 Images Unit Activations (r)

Original

(A)

12.5 10.0 7.5

BrainACTIV

5.0 2.5 0 -2.5

Our Method

-5.0

Original

BrainACTIV Our Method

(B)

Top 10 Images Unit Activations (r)

Original

1.00 0.75 0.50

BrainACTIV

0.25 0.00 -0.25 -0.50

Our Method

-0.75

Original

BrainACTIV Our Method

Figure 4: Generate preferred images with increasing unit activations in ANNs. (A) The top 10 images demonstrate a notable pattern of “green plant & insect” represented by a unit in the middle layer of the SimCLR model. Our method generates images that better match this pattern and increase activation more than the BrainACTIV [17]. (B) For a mid-layer unit in a more complex Vision Transformer (ViT), our method synthesizes images that can better activate the unit compared with the baseline, although the underlying concept appears more ambiguous. problem in linear interpolation method, which could produce consistently correct activation change, and generating more realistic images because sample from such latent space will avoid over-shift from original image representations. This is particularly useful when the target representation is noisy or mixed-selective. To examine the robustness of the predictor, we further compare activation modulation results obtained with the original subjectspecific predictor and with a held-out-subject predictor on the NSD dataset. The results are shown in Table 2. The comparable trends across the two predictors suggest that our conclusions are not driven solely by a single subject-specific model.

shows natural images that produce the largest responses from the chosen units in these ANNs. With our model trained on unit activations, editing the latent code toward the positive prior can generate representations that better activate the selected ANN units than the BrainACTIV [17] in these examples. 4.3.3 Optimize Unit Activations while Preserving Unrelated Features. To demonstrate the strength of our structured VAE latent space, we trained our model with ImageNet dataset. Standard VAE with a standard Gaussian prior. Then we constrain latent of VAE with multiple units activations, and only optimize the latent dimensions controlled by a certain unit. Compared to vanilla VAEs, our method achieves more significant changes in unit responses while preserving unrelated essential image characteristics. Besides modifying latent to prior values, which can manipulate activations and preserve features effectively, we further optimize the VAE’s latent variables 𝑧 using gradient-based methods, applying the Adam optimizer to adjust 𝑧 such that the activation 𝑟 of targeted network

4.3.2 Generate Preferred Representations in Artificial Neural Networks. Besides biological targets, we also test our approach on purely artificial targets: a mid-level unit inside a SimCLR encoder [8], a contrastive-learning framework with a ResNet-50 backbone, as well as a unit from a more complex Vision Transformer (ViT). Figure 4 7

Zhao et al.

B

Latent Modification (Ours)

Activation Change (Δ𝑟) (Higher is Better)

Original

Latent Optimization (Positive Prior) (Ours)

Latent Optimization (Ours)

Standard VAE Optimization (Positive)

Original

Default VAE Optimization Activation Change (Δ𝑟) (Lower is Better)

Latent to Positive (Ours)

A

3 2 1

𝚫𝒓 = 𝟏. 𝟏𝟎𝟑

𝜟𝒓 = 𝟏. 𝟐𝟔𝟕 Δ𝑟 = 0.635

0 -1 -2 0 0.2 0.4 0.6 0.8 1.0 Cosine Similarity (𝑠)

3 2 1 0 -1 -2 -3

Latent to Negative (Ours)

Latent Optimization (Negative Prior) (Ours)

Latent Modification (Ours)

Latent Optimization (Ours)

𝜟𝒓 = −𝟎. 𝟕𝟓𝟔

𝜟𝒓 = −𝟎. 𝟗𝟔𝟓

Standard VAE Optimization (Negative)

Default VAE Optimization

Δ𝑟 = −0.181

0 0.2 0.4 0.6 0.8 1.0 Cosine Similarity (𝑠)

Figure 5: a) Generating preferred images with different methods on ImageNet Dataset. Left 4 columns show optimizing images to increase their activations. Column 1 are four example input images. Column 2-3 are results of our method while column 4 represent optimizing the latent space with a standard VAE. Right 4 columns are results of optimization in the same methods presenting in the same order, with target of decreasing activations. b) shows the shift in the activation value between and the cosine similarity between randomly chosen 100 original images and their optimized results over different methods. Compared with baseline VAE, our method can modulate activation better as well as maintain similarity with original images, meaning unrelated features are preserved. units is either maximized or minimized. The optimization objective combines the desired activation change with a KL divergence (KLD) regularization term, ensuring that the latent representation remains aligned with the prior distribution. The results of latent modification and optimization on both VAEs are shown in Figure 5. To further validate the effectiveness of our optimization approach. Cosine similarity, computed using features extracted from a pre-trained ResNet18 [21] encoder, quantifies the preservation of image features. Results in Figure 5 indicate that our structured landscape approach not only achieves more robust activation changes but also maintains similarity to the original images compared to standard VAE methods, underscoring the efficacy of incorporating structured priors for latent optimization.

5

example, prior work has shown that attenuation of amygdala hyperreactivity is linked to improvement in anxiety and trauma-related symptoms [29], while reduction of subgenual cingulate hyperactivity has been associated with antidepressant treatment effects [20]. Thus, although our study is entirely in-silico and does not constitute a medical intervention, the ability to selectively suppress target-unit responses may be relevant for modeling clinically meaningful forms of neural modulation. Our framework can also extend to artificial neural networks as a controlled setting for studying feature visualization. Compared with the linear method, Vis-Lens does not assume that activation increase and decrease follow a single linear direction in CLIP space. Instead, it learns activation-specific regions in a structured latent space, and enables more controllable sampling, which is especially useful when neural responses are noisy and mixed-selective. There are also limitations to our work. The image generator and the CLIP encoder are frozen, so biases inherited from their pretraining data may propagate into the generated preferred images. In the future, we may scale the framework to additional modalities such as EEG, MEG or larger-scale brain datasets, and move beyond in-silico evaluation by testing the generated preferred images in human experiments; such closed-loop validation would further confirm the effectiveness of our framework and broaden its potential use in brain-computer-interface applications.

Discussion and Limitations

We developed Vis-Lens, a framework that imposes an activationregularized prior on a VAE to yield a structured latent landscape and then performs direct latent editing to steer activations in biological and artificial targets. This structured landscape can also modulate the activation of target units while preserving unrelated features. Vis-Lens shows clear advantages in changing activations in correct direction, keeping image realism and decreasing activations under our evaluation protocol. Inhibit activation is meaningful in clinical applications, reducing abnormally elevated activity in specific circuits has been associated with therapeutic benefit. For 8

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

References

[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips. cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf [23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems (NeurIPS). [24] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. 2022. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research 23, 47 (2022), 1–33. [25] Tomoyasu Horikawa and Yukiyasu Kamitani. 2017. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 8 (2017), 15037. [26] Shuo Huang, Wei Shao, Mei-Ling Wang, and Dao-Qiang Zhang. 2021. fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review. International Journal of Automation and Computing 18 (01 2021). doi:10.1007/ s11633-020-1263-y [27] David H Hubel and Torsten N Wiesel. 1962. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology 160, 1 (1962), 106. [28] Jingyang Huo, Yikai Wang, Yanwei Fu, and Jianfeng Feng. 2026. The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling. arXiv:2601.15071 [cs.CV] https://arxiv.org/abs/2601.15071 [29] Maria Ironside, Michael Browning, Tahereh Ansari, Christopher Harvey, Mama Sekyi-Djan, Sonia Bishop, Catherine Harmer, and Jacinta O’Shea. 2019. Effect of Prefrontal Cortex Stimulation on Regulation of Amygdala Response to Threat in Individuals With Trait Anxiety: A Randomized Clinical Trial. JAMA Psychiatry 76 (01 2019), 71–78. doi:10.1001/jamapsychiatry.2018.2172 [30] Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013). [31] Paulo H. Leocadio. 2025. Hugging Face Diffusers - Chapter 07. Hugging Face (Jan. 2025). doi:10.22541/au.173627634.49343603/v1 [32] Dongyang Li, Haoyang Qin, Mingyang Wu, Chen Wei, and Quanying Liu. 2025. BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings. arXiv:2507.09747 [cs.NE] https://arxiv.org/abs/2507.09747 [33] Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, Haoyang Qin, and Quanying Liu. 2024. Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion. arXiv:2403.07721 [cs.HC] https://arxiv.org/abs/2403.07721 [34] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312 [cs.CV] https://arxiv.org/abs/1405.0312 [35] Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. 2023. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding. arXiv preprint arXiv:2302.12971 (2023). [36] Andrew F. Luo, Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe. 2024. BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity. arXiv:2310.04420 [cs.LG] https://arxiv.org/abs/2310.04420 [37] Andrew F. Luo, Margaret M. Henderson, Leila Wehbe, and Michael J. Tarr. 2023. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. arXiv:2306.03089 [cs.CV] https://arxiv.org/abs/2306.03089 [38] Jun Ma and Tuukka Ruotsalo. 2024. Cognition-Supervised Saliency Detection: Contrasting EEG Signals and Visual Stimuli. In Proceedings of the 32nd ACM International Conference on Multimedia (Melbourne VIC, Australia) (MM ’24). Association for Computing Machinery, New York, NY, USA, 7744–7753. doi:10. 1145/3664647.3681037 [39] Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 5188–5196. [40] Weijian Mai, Jiamin Wu, Yu Zhu, Zhouheng Yao, Dongzhan Zhou, Andrew F. Luo, Qihao Zheng, Wanli Ouyang, and Chunfeng Song. 2025. SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning. arXiv:2508.10298 [cs.LG] https://arxiv.org/abs/2508.10298 [41] David Marr. 2010. Vision: A computational investigation into the human representation and processing of visual information. MIT press. [42] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. 2022. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. arXiv:2108.01073 [cs.CV] https://arxiv.org/ abs/2108.01073 [43] Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. In arXiv preprint arXiv:1411.1784. [44] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1802.05957 [45] Alexander Mordvintsev, Chris Olah, and Mike Tyka. 2015. Inceptionism: Going deeper into neural networks. Google Research Blog 5 (2015).

[1] Igor Abramov. 2025. EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Models. In Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25). ACM, 13522–13524. doi:10.1145/3746027.3754476 [2] Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Logan T. Dowdle, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. 2021. A massive 7T fMRI dataset to bridge cognitive and computational neuroscience. bioRxiv (2021). arXiv:https://www.biorxiv.org/content/early/2021/02/22/2021.02.22.432340.full.pdf doi:10.1101/2021.02.22.432340 [3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017). [4] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. 2019. Neural population control via deep image synthesis. Science 364, 6439 (2019), eaav9436. [5] David Bau et al. 2017. Network Dissection: Quantifying Interpretability of Deep Visual Representations. In Conference on Computer Vision and Pattern Recognition (CVPR). [6] Vinamra Benara, Chandan Singh, John X. Morris, Richard Antonello, Ion Stoica, Alexander G. Huth, and Jianfeng Gao. 2024. Crafting Interpretable Embeddings by Asking LLMs Questions. arXiv:2405.16714 [cs.CL] https://arxiv.org/abs/2405. 16714 [7] Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. 2024. Brain decoding: toward real-time reconstruction of visual perception. In International Conference on Learning Representations (ICLR). [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning (ICML). [9] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. 2018. PixelSNAIL: An Improved Autoregressive Generative Model. In Proceedings of the International Conference on Machine Learning (ICML). [10] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. 2023. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding. arXiv:2211.06956 [cs.CV] https://arxiv.org/abs/ 2211.06956 [11] T Dado, P Papale, et al. 2024. Brain2GAN: Feature-Disentangled Neural Encoding and Decoding of Visual Perception in the Primate Brain. PLoS Computational Biology (2024). https://journals.plos.org/ploscompbiol/article?id=10.1371/journal. pcbi.1012058 [12] Carlos de la Torre-Ortiz and Tuukka Ruotsalo. 2024. Perceptual Visual Similarity from EEG: Prediction and Image Generation. 11146-11155 pages. doi:10.1145/ 3664647.3685508 [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255. doi:10.1109/CVPR.2009.5206848 [14] Zhiwei Ding, Dat Tran, Kayla Ponder, Erick Cobos, Zhuokun Ding, Paul Fahey, Eric Wang, Taliah Muhammad, Jiakun Fu, Santiago Cadena, Stelios Papadopoulos, Saumil Patel, Katrin Franke, Jacob Reimer, Fabian Sinz, Alexander Ecker, Xaq Pitkow, and Andreas Tolias. 2023. Bipartite invariance in mouse primary visual cortex. doi:10.1101/2023.03.15.532836 [15] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal (2009). [16] Matteo Ferrante, Tommaso Boccato, Luca Passamonti, and Nicola Toschi. 2024. Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model. Journal of Neural Engineering 21, 4 (2024), 046001. [17] Diego Garcia Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Pascal Mettes, and Iris Groen. 2025. BrainACTIV: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=CGON8Btleu [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems (NeurIPS) (2014). [19] Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily J. Allen, Yihan Wu, Ghislain St-Yves, Thomas Naselaris, Kendrick Kay, Mert R. Sabuncu, and Amy Kuceyeski. 2022. NeuroGen: Activation optimized image synthesis for discovery neuroscience. NeuroImage 247 (2022), 118812. doi:10.1016/j.neuroimage.2021. 118812 [20] Itay Hadas, Yinming Sun, Pantelis Lioumis, Reza Zomorrodi, Brett Jones, Daphne Voineskos, Jonathan Downar, Paul Fitzgerald, Daniel Blumberger, and Zafiris Daskalakis. 2019. Association of Repetitive Transcranial Magnetic Stimulation Treatment With Subgenual Cingulate Hyperactivity in Patients With Major Depressive Disorder: A Secondary Analysis of a Randomized Clinical Trial. JAMA Network Open 2 (06 2019), e195578. doi:10.1001/jamanetworkopen.2019.5578 [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778. 9

Zhao et al.

[46] Alexander Mordvintsev, Chris Olah, and Mike Tyka. 2016. DeepDream—A code example for visualizing Neural Networks. Google Research Blog (2016). [47] N. Murty, Pouya Bashivan, Alex Abate, James Dicarlo, and Nancy Kanwisher. 2021. Computational models of category-selective brain regions enable highthroughput tests of selectivity. Nature Communications 12 (09 2021), 5540. doi:10. 1038/s41467-021-25409-6 [48] Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. 2011. Encoding and decoding in fMRI. Neuroimage 56, 2 (2011), 400–410. [49] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. 2017. Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4467–4477. [50] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 427–436. [51] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2016. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems (NeurIPS). 3387–3395. [52] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv:2112.10741 [cs.CV] https://arxiv.org/abs/2112.10741 [53] Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. 2011. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology 21, 19 (2011), 1641–1646. [54] Tormod Ø ygard. 2015. Gradient blurring for improved feature visualization. Personal communication (2015). [55] Chris Olah et al. 2017. Feature Visualization. Distill (2017). [56] Bruno A Olshausen and David J Field. 2004. Sparse coding of sensory inputs. Current opinion in neurobiology 14, 4 (2004), 481–487. [57] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. DINOv2: Learning Robust Visual Features without Supervision. arXiv:2304.07193 [cs.CV] https://arxiv.org/abs/2304.07193 [58] Furkan Ozcelik and Rufin VanRullen. 2023. Natural scene reconstruction from fMRI signals using generative latent diffusion. arXiv:2303.05334 [cs.CV] https: //arxiv.org/abs/2303.05334 [59] Paolo Papale, Daniela De Luca, and Pieter R. Roelfsema. 2024. Deep generative networks reveal the tuning of neurons in IT and predict their influence on visual perception. bioRxiv (2024). arXiv:https://www.biorxiv.org/content/early/2024/10/12/2024.10.09.617382.full.pdf doi:10.1101/2024.10.09.617382 [60] Paweł A. Pierzchlewicz, Konstantin Willeke, Arne Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas Tolias, and Fabian Sinz. 2023. Energy Guided Diffusion for Generating Neurally Exciting Images. doi:10.1101/2023.05.18.541176 [61] D Qian, H Zeng, et al. 2024. NeuroDM: Decoding and Visualizing Human Brain Activity with EEG-Guided Diffusion Model. Computer Methods and Programs in Biomedicine (2024). https://www.sciencedirect.com/science/article/pii/ S0169260724002098 [62] Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, and Rex Ying. 2025. MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding. arXiv:2502.15786 [q-bio.NC] https://arxiv.org/abs/2502.15786 [63] Rodrigo Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. 2005. Invariant visual representation by single neurons in the human brain. Nature 435, 7045 (2005), 1102–1107. [64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference on Machine Learning (ICML). https://arxiv.org/abs/2103.00020 [65] Mattia Rigotti, O. Barak, M. Warden, Xiao-Jing Wang, N. Daw, E. Miller, and Stefano Fusi. 2013. The importance of mixed selectivity in complex cognitive tasks. Nature 497 (2013), 585–590. doi:10.1038/nature12160 [66] Amanda Robinson, Genevieve Quek, and Thomas Carlson. 2023. Visual Representations: Insights from Neural Decoding. Annual Review of Vision Science 9 (03 2023). doi:10.1146/annurev-vision-100120-025301 [67] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV] https://arxiv.org/abs/2112.10752 [68] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. LAION-5B: An

open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=M3Y74vmsMcY [69] Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. 2023. Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. arXiv:2305.18274 [cs.CV] https://arxiv.org/abs/2305.18274 [70] Paul S. Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A. Norman, and Tanishq Mathew Abraham. 2024. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. arXiv:2403.11207 [cs.CV] https://arxiv.org/abs/2403.11207 [71] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. 2019. Deep image reconstruction from human brain activity. PLoS Computational Biology 15, 1 (2019), e1006633. [72] Matthew Shinkle and Mark D. Lescroart. 2025. Visualizing and Controlling Cortical Responses Using Voxel-Weighted Activation Maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 4864–4868. [73] Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang, and Xiaorong Gao. 2024. Decoding Natural Images from EEG for Object Recognition. arXiv:2308.13234 [cs.HC] https://arxiv.org/abs/2308.13234 [74] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. In arXiv preprint arXiv:1312.6199. [75] Yu Takagi and Shinji Nishimoto. 2023. High-Resolution Image Reconstruction With Latent Diffusion Models From Human Brain Activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14453–14463. [76] Mike Tyka. 2016. Regularizing feature visualization with bilateral filters. Personal communication (2016). [77] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. 2016. Conditional Image Generation with PixelCNN Decoders. In Advances in Neural Information Processing Systems (NeurIPS). [78] Marcel A.J. van Gerven. 2017. A primer on encoding models in sensory neuroscience. Journal of Mathematical Psychology 76 (2017), 172–183. doi:10.1016/j. jmp.2016.06.009 Model-based Cognitive Neuroscience. [79] Marcel A J van Gerven. 2021. Neural decoding with generative models. Trends in Cognitive Sciences 25, 7 (2021), 593–607. [80] Edgar Walker, Fabian Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul Fahey, Alexander Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. 2019. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience 22 (12 2019), 1–6. doi:10.1038/s41593019-0517-x [81] Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. 2024. MindBridge: A Cross-Subject Brain Decoding Framework. arXiv:2404.07850 [cs.CV] https://arxiv.org/abs/2404.07850 [82] Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. 2024. DREAM: Visual Decoding from Reversing Human Visual System. arXiv:2310.02265 [cs.CV] https://arxiv.org/abs/2310.02265 [83] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. arXiv:2308.06721 [cs.CV] https://arxiv.org/abs/2308.06721 [84] Bohan Zeng, Shanglin Li, Xuhui Liu, Sicheng Gao, Xiaolong Jiang, Xu Tang, Yao Hu, Jianzhuang Liu, and Baochang Zhang. 2023. Controllable Mind Visual Diffusion Model. arXiv:2305.10135 [cs.CV] https://arxiv.org/abs/2305.10135

10

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Supplementary Material Appendix Overview This appendix provides comprehensive details and additional results to support the findings presented in the main paper and ensure that all of the results can be reproduced. The appendix content is organized into the following sections: • Additional Results: Additional visualizations and analyses that further demonstrate the effectiveness of our proposed methods. More examples of enhanced sampling, local change results are presented, as shown in section A. • Experimental Settings: Comprehensive information about the datasets settings, like input size and how are the datasets divided and processed, model architectures and implementation details, then training configurations and computational resources. All of them can provide a clear understanding of our experimental setup, which are helpful to reproduce the results, as shown in section B. • Implementation Details: Detailed information about the algorithms we used to implement our method, including pseudocode, and corresponding explanations. All of them can provide a clear understanding of how the training and optimization are carried out, as shown in section C. • Predictive Power of In-silico brain simulator: Information about how much our trained In-silico brain simulator can predict the brain activity data, as shown in section D.

A Additional Results A.1 More Results for the NSD Dataset Here is a summarized table of more experimental results on NSD dataset: Table S1: Activation change and realism on additional ROIs. Better values are in bold.

ROIs

V1

V2

V3

V4

V8

LO1

LO2

V4t

Activation Increase (Δ𝑟 )

Ours BrainACTIV

0.257 0.157

0.080 0.116

0.247 0.226

0.486 0.488

0.696 0.721

0.583 0.582

0.498 0.517

0.757 0.767

Activation Decrease (Δ𝑟 )

Ours BrainACTIV

-0.291 0.102

-0.272 0.224

-0.573 0.274

-0.690 0.501

-0.406 0.608

-0.485 0.100

-0.490 -0.057

-0.601 -0.511

FID@2k

Ours BrainACTIV

164.87 224.29

157.84 201.56

167.44 240.64

173.24 231.60

160.07 216.80

136.05 208.23

186.67 255.06

178.62 197.61

ROIs

VWFA-1

PPA

RSC

OWFA

IP2

TPOJ1

TPOJ2

TPOJ3

Activation Increase (Δ𝑟 )

Ours BrainACTIV

1.108 1.019

0.952 0.926

0.923 0.954

0.718 1.088

0.091 0.084

0.456 0.509

0.661 0.751

1.014 1.091

Activation Decrease (Δ𝑟 )

Ours BrainACTIV

-0.391 0.350

-0.809 -0.579

-0.913 -0.804

-0.780 0.347

-0.213 0.061

-0.136 -0.134

-0.384 -0.234

-0.591 -0.484

FID@2k

Ours BrainACTIV

169.14 229.65

153.67 234.79

172.92 216.65

163.88 207.15

168.56 171.94

192.04 244.44

131.83 197.94

137.57 197.24

The full test-set results for the additional ROIs are provided in Figs. S1–S16.

11

Zhao et al.

A.1.1

ROI: IPS4. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S1: Full result of activation change on region IPS4, positive direction.

12

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S2: Full result of activation change on region IPS4, negative direction.

13

Zhao et al.

A.1.2

ROI: SPL1. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S3: Full result of activation change on region SPL1, positive direction.

14

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S4: Full result of activation change on region SPL1, negative direction.

15

Zhao et al.

A.1.3

ROI: 31a. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S5: Full result of activation change on region 31a, positive direction.

16

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S6: Full result of activation change on region 31a, negative direction.

17

Zhao et al.

A.1.4

ROI: 31pd. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S7: Full result of activation change on region 31pd, positive direction.

18

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S8: Full result of activation change on region 31pd, negative direction.

19

Zhao et al.

A.1.5

ROI: V1. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S9: Full result of activation change on region V1, positive direction.

20

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S10: Full result of activation change on region V1, negative direction.

21

Zhao et al.

A.1.6

ROI: V2. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S11: Full result of activation change on region V2, positive direction.

22

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S12: Full result of activation change on region V2, negative direction.

23

Zhao et al.

A.1.7

ROI: V3. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S13: Full result of activation change on region V3, positive direction.

24

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S14: Full result of activation change on region V3, negative direction.

25

Zhao et al.

A.1.8

ROI: V4. Original

Original

Vis-Lens

BrainACTIV

𝜶 = 𝟎. 𝟐

𝜶 = 𝟎. 𝟒

𝜶 = 𝟎. 𝟔

𝜶 = 𝟎. 𝟖

𝜶 = 𝟏. 𝟎

Figure S15: Full result of activation change on region V4, positive direction.

26

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Original

Original

Vis-Lens

BrainACTIV

𝜶 = −𝟎. 𝟐

𝜶 = −𝟎. 𝟒

𝜶 = −𝟎. 𝟔

𝜶 = −𝟎. 𝟖

𝜶 = −𝟏. 𝟎

Figure S16: Full result of activation change on region V4, negative direction.

27

Zhao et al.

A.2

More ANN Results

We present more examples on manipulating activations in SimCLR[8], shown in Figs. S17–S22, which further demonstrate the effectiveness of our method.

Top 10 Images

Unit Activations (r) 25

Original

20 15 10

BrainACTIV

5 0 -5 -10

Our Method

Original

BrainACTIV

Our Method

Figure S17: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.

Top 10 Images

Unit Activations (r) 1.0

Original

0.8 0.6 0.4 0.2

BrainACTIV

0.0 -0.2 -0.4

Our Method

Original

BrainACTIV Our Method

Figure S18: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV. 28

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Top 10 Images

Unit Activations (r) 4

Original

2 0

BrainACTIV -2 -4

Our Method

Original

BrainACTIV Our Method

Figure S19: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.

Top 10 Images

Unit Activations (r)

Original

4 3 2

BrainACTIV

1 0 -1

Our Method

Original

BrainACTIV Our Method

Figure S20: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. The top 10 images demonstrate a notable pattern represented by the unit in the deep layer of SimCLR model, that round objects appear in top images. Our method can also generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV. 29

Zhao et al.

Top 10 Images

Unit Activations (r) 15

Original

10 5 0

BrainACTIV

-5 -10

Our Method

Original

BrainACTIV

Our Method

Figure S21: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. The top 10 images demonstrate a notable pattern represented by the unit in the deep layer of SimCLR model, that grid objects appear in top images. Our method can also generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.

Top 10 Images

Unit Activations (r)

Original

6 4 2

BrainACTIV 0 -2

Our Method

Original

BrainACTIV Our Method

Figure S22: One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.

B Experimental Settings B.1 Dataset Details We will show how the dataset was processed in this part. 30

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Natural Scenes Dataset (NSD): [2]: we selected 1000 images presented to Subject 1 from the dataset as our test set, while using CLIP embeddings of 9000 images shown to same subject for training. We used same test images selected by the BrainACTIV. And for the measurement of activation change and FID score, we use all 1000 images presented to same subject as test set in COCO dataset [34] and their CLIP embeddings. 2k images for FID[22] calculation are the combination of 1k preferred images to maximize/minimize the activations, respectively. The CLIP embeddings are retrieved from the first output of CLIP encoder [64], the dimension is 1024 for every image. Imagenet-mini [13]: The train and test split follow the default settings in the orginal dataset, which divided the whole dataset into train/val sets, we use train as our train set and val as our testset. Train set consists of 34,745 images, and test set consists of 3,923 images. For the training process of SimCLR, we used following transform techniques in torchvision.transforms to edit the orginal Imagenet-mini images: t r a n s f o r m s . RandomResizedCrop ( 1 2 8 , s c a l e = ( 0 . 0 8 , 1 . 0 ) , i n t e r p o l a t i o n = Image . BICUBIC ) transforms . RandomHorizontalFlip ( ) t r a n s f o r m s . ToTensor ( )

B.2

Model Implementation Details

B.2.1

VAE Implementation and Training.

VAE:. For NSD and ImageNet dataset to learn CLIP embeddings, using VAE architecture [30] consists of an encoder and a decoder with the following layers: • Encoder (CLIP → latent) – FCenc : 1024 −→ 4096 (first 2048 dims are 𝜇, last 2048 dims are log 𝜎 2 ) • Latent layer – 𝜇 ∈ R2048 , log 𝜎 2 ∈ R2048 – Reparameterisation: 𝑧 = 𝜇 + 𝜖 ⊙ exp(0.5 log 𝜎 2 ) • Decoder (latent → CLIP) – FCdec : 2048 −→ 1024 (reconstructed CLIP embedding) This CLIP-to-CLIP VAE is distinct from the image-space VAE used in the latent optimization experiments, whose decoder produces images directly. The VAE training configurations are shown in the Table S2. B.2.2 Implementation of Diffusion Pipeline. All images are first converted to a 1024-dimensional representation with the frozen CLIP-ViTH/14 encoder released by LAION [68]. For every split of the Natural Scenes Dataset we iterate once over the raw-pixel loader, compute the embeddings and cache the resulting CLIP features. Subsequent experiments therefore operate purely in embedding space and never revisit the expensive vision backbone. Image synthesis uses the “img-to-img” variant of Stable Diffusion v1.5 from the Diffusers library [31]. We swap the original PNDM scheduler for DDIM (better speed/quality trade-off) and disable the safety checker to avoid unintended filtering. Every network in the pipeline—U-Net, text encoder, VAE, and the IP-Adapter [83] that injects the CLIP embedding into the cross-attention blocks—remains entirely frozen. We use an IP-Adapter checkpoint compatible with the ViT-H/14 image embeddings employed for feature extraction. During inference each RGB frame (512 × 512 px) is supplied together with a target CLIP embedding—either the baseline embedding obtained by ridge regression or the one produced by our activation-aware VAE. Unless noted otherwise we set the SDEdit strength to 𝛾 = 1.0; We run 50 DDIM steps per image, draw a single sample, and fix the random seed to 42 for strict comparability across all methods. B.2.3

Implementation of Other Models.

SimCLR. We train a SimCLR encoder with a ResNet-50 backbone on ImageNet-mini, as described in Section 4.3.2. Training uses the 128 × 128 image crops and augmentations specified above, with the model adjusted to accommodate this input resolution.

B.3

Metric Details

Quantifying Similarity. To further evaluate the structured landscape, we used similarity and realism metrics. For similarity between original images and images with activation optimized but unrelated features preserved, we use cosine similarity as follows. Given a set of images {𝐼𝑖 } with feature embeddings {f𝑖 }, the similarity score is expressed as: Cosine Similarity(f𝑖 , f 𝑗 ) =

f𝑖 · f 𝑗 ∥f𝑖 ∥∥f 𝑗 ∥

Implementation of Similarity Calculation: Following Section 4.3.3, feature preservation is quantified using a pretrained ResNet18 encoder [21]. We compute cosine similarity between the feature embedding of each original image and that of its corresponding optimized image. Higher similarity indicates greater preservation of image features. 31

Zhao et al.

Table S2: VAE Hyperparameters and Training Settings for NSD and ImageNet-mini datasets. Hyperparameter

NSD

ImageNet

Data Hyperparameters input_size n_channels

1024 dim embeddings 1

1024 dim embeddings 1

Model Hyperparameters latent_dim

2048

2048

0 128 150 Adam 1 × 10 −4 StepLR (step_size=10, 𝛾 = 0.9) 1.0

0 128 150 Adam 1 × 10 −3 StepLR (step_size=10, 𝛾 = 0.9) 1.0

Training Settings seed batch_size epochs optimizer lr scheduler grad_clip

Quantifying Realism: Frechét Inception Distance (FID):. To quantify overall realism we report the Frechét Inception Distance, computed on the 2 048-D pool-3 activations of a frozen Inception-v3 network. Let 𝜇𝑟 , Σ𝑟 and 𝜇𝑔 , Σ𝑔 be the empirical means and covariances of the real and generated image features, respectively. The FID is
FID = ∥𝜇𝑟 − 𝜇𝑔 ∥ 22 + Tr Σ𝑟 + Σ𝑔 − 2 (Σ𝑟 Σ𝑔 ) 1/2 . Lower values indicate that the generated distribution is closer to the real one. For the NSD evaluation, each method generates 1,000 images in the positive direction and 1,000 in the negative direction. We pool these into a single set of 2,000 generated images per method and compute one FID against the 1,000 original test images repeated twice. FID is computed with the torchmetrics implementation using feature=2048.

B.4

Computational Resources

All experiments were carried out on a single workstation equipped with 8 × NVIDIA RTX A5000 GPUs (each with 24 GB on-board memory, CUDA 12.2, driver 535.183), an AMD EPYC 7543 CPU (32 cores, 2.8 GHz) and 512 GB of system RAM. One GPU was used for model training or inference at a time. Training a CLIP-to-CLIP VAE on the NSD train split (≈ 9 k images) with a batch-size of 128 took ≈ 6.5 hours on a single A5000; Optimizing on Imagenet-mini completed in under 40 minutes. For diffusion-based generation (IP-Adapter + Stable-Diffusion v1.5, 50 DDIM steps) ,producing one full 2 k-image evaluation set (positives + negatives) took about 70 minutes.

32

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

C Implementation Details C.1 Training Details The following pseudocode summarizes the training objective and generation procedure described in Section 3.3.

Algorithm 1 VAE Training with Activation Regularization Require: 1: vae: Variational Autoencoder model 2: optimizer: Optimizer (Adam) 3: scheduler: Learning rate scheduler (StepLR) 4: num_epochs: Number of training epochs Ensure: 5: VAE with structured latent space and high-quality reconstruction 6: procedure TrainVAE(vae, optimizer, . . . )

for epoch = 1 to num_epochs do for each batch do 9: optimizer.zero_grad() 10: Forward Pass: 11: (𝜇, log 𝜎 2 ) ← vae.encode(data) 12: 𝜂 ∼ N (0, 𝐼 ); 𝑧 ← 𝜇 + exp( 21 log 𝜎 2 ) ⊙ 𝜂 13: 𝑥ˆ ← vae.decode(𝑧) 14: Compute Loss: 15: Reconstruction Loss ← ∥data − 𝑥ˆ ∥ 22 16: Compute Prior Parameters: 𝜆 17: 𝜎 prior ← activation_values +𝜖 ( 𝜇pos, if activation_values𝑖 > 0, prior 18: 𝜇𝑖 ← 𝜇neg, otherwise. 19: Compute KLD Loss from 𝜇, 𝜎, 𝜇 prior, 𝜎 prior using Eq. 3 20: Total Loss ← 𝜔 recon × Reconstruction Loss + 𝜔 KLD × KLD Loss 21: Backpropagation: 22: Total Loss.backward() 23: optimizer.step() 24: end for 25: scheduler.step() 26: end for 27: end procedure 7: 8:

Latent Modification and Image Generation. To generate high-quality preferred images that enhance activations based on the latent modification, the algorithm is shown as following block 2: 33

Zhao et al.

Algorithm 2 Vis-Lens: Preferred–Image Generation Require: 𝑖 in . . . RGB input image clip_enc . . . frozen CLIP–ViT-H encoder 𝑥 . . . 1024-d CLIP embedding of 𝑖 in vae . . . activation–aware CLIP→CLIP VAE 𝑧, 𝜇 . . . latent code / mean; 𝜇pos, 𝜇neg . . . activation priors ip_adapter . . . frozen IP-Adapter sd15 . . . Stable-Diffusion v1.5 (DDIM) 𝛼 . . . linear interpolation weight, 𝛾 . . . SDEdit strength 𝑟 dir ∈ {pos, neg} . . . desired ROI shift Ensure: preferred image 𝑖 out 1: procedure GenPreferred(𝑖 in , 𝑟 dir ) 2: 𝑥 ← clip_enc(𝑖 in ) 3: (𝜇, log 𝜎 2 ) ← vae.encode(𝑥); 𝑧 ← 𝜇 4: if 𝑟 dir = pos then 5: 𝜇prior ← 𝜇pos 6: else 7: 𝜇prior ← 𝜇neg 8: end if 9: 𝑧 ′ ← 𝛼 𝜇 prior + (1 − 𝛼)𝑧 10: 𝑥 gen ← vae.decode(𝑧 ′ ) 11: 𝑖 out ← ip_adapter.generate(clip = 𝑥 gen, image = 𝑖 in, 𝛾, 50 steps) 12: return 𝑖 out 13: end procedure

Here are more detailed explanations of our generation process:

• The input image 𝑖 in is projected to a CLIP embedding x with the frozen encoder. • The activation-aware VAE encodes x to latent mean 𝜇; we take 𝑧 = 𝜇 for a deterministic edit. • Following Section 4.3.1, the latent edit is 𝑧 ′ = 𝛼𝜇 prior + (1 − 𝛼)𝑧, where 𝜇 prior is the positive prior 𝜇 pos to raise activity or the negative prior 𝜇neg to suppress it. • Decoding z’ gives an intermediate CLIP embedding 𝑥 gen . • 𝑥 gen is injected—via a frozen IP-Adapter—into Stable-Diffusion v1.5. With SDEdit strength 𝛾 = 1.0 the reverse DDIM process relies entirely on 𝑥 gen ; smaller 𝛾 blends in low-level structure from the reference image. • The sampler returns the final preferred image 𝑖 out , which is used in all downstream analyses.

C.2

Details about Optimizing Unit Activations with Latent

The algorithm for optimizing neural network unit responses using latent variables from a VAE is listed as following algorithm block 3: 34

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Algorithm 3 Optimize Activation with Latent Variables Require: 1: net: Pre-trained neural network model 2: vae: Pre-trained Variational Autoencoder 3: layer_name: Target layer in net 4: unit_idx: Index of target unit 5: num_iterations: Number of optimization steps 6: learning_rate: Learning rate for optimizer 7: KLD_weight: Weight for Kullback-Leibler Divergence loss 8: target: Direction “max" or “min") 9: init_data: Initial data sample 10: J: Latent dimensions associated with the target unit Ensure: 11: Optimized responses scores, Controlled KLD loss 12: procedure OptimizeActivation(net, vae, . . .) 13: Initialize Latent Variable: 14: (𝜇, logvar) ← vae.encode(init_data) 15: 𝑧 ← 𝜇; 𝑧 0 ← 𝜇 16: Set Up Optimizer: 17: optimizer ← Adam(𝑧, lr=learning_rate) 18: Register Activation Hook on layer_name 19: for iteration = 1 to num_iterations do 20: Decode Latent to Image: 21: 𝑥 ← vae.decode(𝑧 ) 22: Forward Pass: 23: net(x) 24: Compute Loss: 25: if target = “max" then 26: loss ← −activation + 𝜔 KLD × KLD(𝑧 ) 27: else 28: loss ← activation + 𝜔 KLD × KLD(𝑧 ) 29: end if 30: Backpropagation: 31: optimizer.zero_grad() 32: loss.backward() 33: optimizer.step() 34: 𝑧 𝑗 ← (𝑧 0 ) 𝑗 for all 𝑗 ∉ J 35: end for 36: Remove Activation Hook 37: Return 𝑥, activation scores 38: end procedure

More Explanation of the Algorithm. This algorithm describes the image-space VAE optimization experiment in Section 4.3.3. To preserve unrelated features, only the latent dimensions J associated with the selected unit are updated; the remaining coordinates stay at their initial values. The VAE and target network remain fixed while the latent code is optimized to increase or decrease the target response. Initialization: We initialize the latent variable 𝑧 by encoding the input image using the VAE encoder: (𝜇, logvar) ← vae.encode(init_data) 𝑧←𝜇 Optimization Loop: For each iteration, we perform the following steps: 1. Decode Latent to Image: The current latent variable 𝑧 is decoded to generate an image 𝑥: 𝑥 ← vae.decode(𝑧) 2. Forward Pass and Activation Capture: The generated image 𝑥 is passed through the neural network net. We utilize a registered hook function on the target layer to capture the activation of the target unit during this forward pass: net(𝑥) 35

Zhao et al.

The hook function retrieves the target activation activation, which will be used in the loss computation. This approach allows us to monitor and manipulate the activation of specific units without altering the forward pass logic of the network. 3. Compute Loss: We compute the loss function, which consists of two components: the activation term and the Kullback-Leibler Divergence (KLD) regularization term. - Activation Term: Depending on the optimization direction (maximize or minimize), we use the negative or positive of the target unit’s activation: ( loss =

−activation, activation,

if target = “max" if target = “min"

- KLD Regularization Term: The KLD term ensures that the latent variable 𝑧 remains within a plausible region of the latent space. We consider two method for the KLD calculation. One is our method, and the other one is using the latent space of Standard VAE to compute KLD loss, which is our baseline. • Our Method: We calculate the KLD using a modified prior that incorporates the desired change in activation. Specifically, the prior mean 𝜇 prior is adjusted based on the target direction: ( 𝜇

prior

=

+1, −1,

if target = “max" if target = “min"

The KLD is then computed as:

KLDour (𝑧) =

! prior 𝑑𝑧  𝜎 1 ∑︁ 2 log 𝑖 −1 2 𝑖=1 𝜎𝑖 prior  𝜎 2 + (𝜇𝑖 − 𝜇𝑖 ) 2 + 𝑖 prior (𝜎𝑖 ) 2

prior

where 𝜎𝑖 is set to a small constant (e.g., 0.1). • Standard VAE: We calculate the KLD using the standard Gaussian prior with zero mean and unit variance: 𝑑

KLDstandard (𝑧) =

𝑧
1 ∑︁ 𝜇 2 + 𝜎𝑖2 − log(𝜎𝑖2 ) − 1 2 𝑖=1 𝑖

- Total Objective Function: The total loss combines the activation loss and the KLD regularization: loss = activation_loss + 𝜔 KLD × KLD(𝑧) where KLD(𝑧) is either KLDour (𝑧) or KLDstandard (𝑧) depending on the mode. 4. Backpropagation: We perform backpropagation to compute the gradients of the loss with respect to 𝑧 and update 𝑧 using the optimizer. optimizer.zero_grad() loss.backward() optimizer.step() Configurations: During the implementation, learning rate was set to 0.1, num_iterations to 10000 and 𝜔 KLD to 0.001.

D

Predictive Power of In-silico brain simulator

The following plots compare observed and predicted responses for the in-silico brain simulator described in Section 4.2. The simulator uses a frozen DINO-ViT feature extractor and a single-layer ViT ensemble over its 12 layer outputs. Separate simulators are trained for different ROI groups. In these fMRI plots, the displayed correlation is the arithmetic mean of the Pearson correlations computed separately for the ROIs in each group. 36

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Figure S23: Observed versus predicted responses. ROIs: OFA, FFA-1, EBA, VWFA-1, OPA, PPA, RSC, OWFA.

37

Zhao et al.

Figure S24: Observed versus predicted responses. ROIs: IPS4, SPL1, TO1, PHC2, VO2, PGi, VVC, TE2a.

38

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Figure S25: Observed versus predicted responses. ROIs: STSdp, 31a, 31pd, DVT, FST, Gp, PGs, PH.

39

Zhao et al.

Figure S26: Observed versus predicted responses. ROIs: TE2p, PHT, TPOJ1, TPOJ2, TPOJ3, IP0, IP1, IP2.

40

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Figure S27: Observed versus predicted responses. ROIs: V1, V2, V3, V4, V8, LO1, LO2, V4t.

41

Zhao et al.

Figure S28: Observed versus predicted responses. Occipital Channels: O1, Oz, O2, PO7, PO3, POz, PO4, PO8.

42

Related documents

• Recurrent Dynamics Give Rise to Individualized Category Representations#1078985
• Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video#1143420
• Recurrent network dynamics explain the time course of perceptual grouping in natural scenes#1201432
• StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching#1326591
• Classification and Reconstruction Processes in Deep Predictive Coding Networks: Antagonists or Allies?#1141382