Modality-agnostic decoding of vision and language from fMRI.

Modality-agnostic decoding of vision and language from fMRI - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice eLife . 2026 Apr 17;14:RP107933. doi: 10.7554/eLife.107933 Search in PMC Search in PubMed View in NLM Catalog Add to search Modality-agnostic decoding of vision and language from fMRI Mitja Nikolaus Mitja Nikolaus 1 Université de Toulouse, CNRS, CerCo, Toulouse, France Find articles by Mitja Nikolaus 1, †, ✉ , Milad Mozafari Milad Mozafari 2 Torus AI, Toulouse, France Find articles by Milad Mozafari 2, † , Isabelle Berry Isabelle Berry 1 Université de Toulouse, CNRS, CerCo, Toulouse, France Find articles by Isabelle Berry 1 , Nicholas Asher Nicholas Asher 3 Université de Toulouse, IRIT, Toulouse, France Find articles by Nicholas Asher 3 , Leila Reddy Leila Reddy 1 Université de Toulouse, CNRS, CerCo, Toulouse, France Find articles by Leila Reddy 1 , Rufin VanRullen Rufin VanRullen 1 Université de Toulouse, CNRS, CerCo, Toulouse, France Find articles by Rufin VanRullen 1 Editors: Yanchao Bi 4 , Yanchao Bi 5 Author information Article notes Copyright and License information 1 Université de Toulouse, CNRS, CerCo, Toulouse, France 2 Torus AI, Toulouse, France 3 Université de Toulouse, IRIT, Toulouse, France 4 Peking University, China 5 Peking University, China † These authors contributed equally to this work. ✉ Corresponding author. Roles Yanchao Bi : Reviewing Editor Yanchao Bi : Senior Editor Collection date 2026. © 2025, Nikolaus, Mozafari et al This article is distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use and redistribution provided that the original author and source are credited. PMC Copyright notice PMCID: PMC13090028  PMID: 41995708 Previous version available: This article is based on a previously available preprint with doi: https://doi.org/10.1101/2025.06.08.658221 . Previous version available: This article is based on a previously available preprint with doi: https://doi.org/10.7554/eLife.107933.1 . Previous version available: This article is based on a previously available preprint with doi: https://doi.org/10.7554/eLife.107933.2 . Abstract Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises six subjects watching both images and short text descriptions of such images, as well as the conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders and even outperform them when decoding captions and mental imagery. Furthermore, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition. Research organism: Human Introduction Several regions in the human brain have developed a high degree of specialization for particular lower-level perceptive as well as higher-level cognitive functions ( Kanwisher, 2010 ). For many higher-level functions, it is crucial to be able to manipulate inputs regardless of the modality in which a stimulus was perceived by the brain. Such manipulations can be performed thanks to representations that are abstracted away from particularities of specific modalities and are, therefore, modality-invariant . A range of theories have been developed to explain how and where in the human brain such invariant representations are created ( Damasio, 1989 ; Binder et al., 2009 ; Martin, 2016 ; Barsalou, 2016 ; Ralph et al., 2017 ). In this paper, we aim to develop modality-agnostic decoders that are specifically trained to leverage such modality-invariant representation patterns in the brain. In contrast to modality-specific decoders that are trained to be applied only in the modality that they were trained on, modality-agnostic decoders can be applied to decode brain patterns from multiple modalities, even without knowing a priori the modality the stimulus was presented in. Regarding the terminology that will be used in the following, we use the terms modality-agnostic and modality-specific to refer to the way decoders are trained. The term modality-dependent describes patterns in brain activity that are completely unrelated to stimuli presented in varying modalities. In contrast, modality-invariant patterns do contain a significant degree of shared representational structure between modalities ( Man et al., 2012 ). Evidence suggests that many regions in the brain contain mixtures of both types of information: A single region can show activation patterns with contributions of multiple modality-dependent and modality-invariant features ( Liuzzi et al., 2020 ; Dirani and Pylkkänen, 2024 ). Modality-agnostic decoders are explicitly trained to leverage the shared information in modality-invariant patterns of the brain activity. In order to build modality-agnostic decoders, large multimodal neuroimaging datasets with well-controlled stimuli across modalities are required. While large-scale datasets exist for vision ( Huth et al., 2012 ; Chang et al., 2019 ; Allen et al., 2022 ), language ( Brennan and Hale, 2019 ; Nastase et al., 2021 ; Schoffelen et al., 2019 ; Tang et al., 2023a ), and video (naturalistic movies) ( Aliko et al., 2020 ; Visconti di Oleggio Castello et al., 2020 ; Boyle et al., 2020 ), none of these contain a controlled set of equivalent stimuli that are presented separately in both modalities. For instance, the different modalities in movies (vision and language) are complementary but do not always carry the same semantics. Furthermore, they are not presented separately but simultaneously, impeding a study of the respective activity pattern caused by each modality in isolation. The analyses of this paper are based on SemReps-8K , a new large-scale multimodal fMRI dataset of six subjects each viewing more than 8000 stimuli which are presented separately in one of two modalities, as images of visual scenes or as descriptive captions of such images. In addition, the dataset also contains three imagery conditions for each subject, where they had to imagine a visual scene based on a caption description they had received before the start of the fMRI experiment. We exploit this new data to develop modality-agnostic decoders that are trained on brain imaging data from multiple modalities, which we demonstrate here for the case of vision and language. We find that modality-agnostic decoders trained on this dataset perform on par with their respective modality-specific counterparts for decoding images, despite the additional challenge of uncertainty about the stimulus modality. For decoding captions, the modality-agnostic decoders even outperform the respective modality-specific decoders (because the former, but not the latter, can leverage the additional training data from the other image modality). Additionally, we use this novel kind of decoders for a searchlight analysis to localize regions with modality-invariant representations in the brain. Previous studies that aimed to localize modality-invariant patterns were based on limited and rather simple stimulus sets and did not always agree on the exact location and extent of such regions (e.g. Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Devereux et al., 2013 ; Fairhall and Caramazza, 2013 ; Jung et al., 2018 ). We design a searchlight analysis based on a combination of modality-agnostic decoders and cross-decoding. The results reveal that modality-invariant patterns can be found in a widespread left-lateralized network across the brain, encompassing virtually all regions that have been proposed previously. Finally, we find that modality-agnostic decoders trained only on data with perceptual input also generalize to conditions during which the subjects were performing mental imagery and outperform both kinds of modality-specific decoders on this task. Related work Modality-invariant representations Decades of neuro-anatomical research (e.g. based on clinical lesions) and electrophysiology in non-human and human primates, as well as modern experiments leveraging recent brain imaging techniques have provided evidence that the activation patterns in certain brain areas are modality-dependent; for example, the occipital cortex responds predominantly to visual stimulation ( Felleman and Van Essen, 1991 ; Sereno et al., 1995 ; Grill-Spector and Malach, 2004 ), and a commonly left-lateralized network responds to language processing tasks ( Zola-Morgan, 1995 ; Fedorenko et al., 2010 ; Fedorenko et al., 2011 ; Friederici, 2017 ; Brennan, 2022 ). More recent research has started to focus on higher-level regions that respond with modality-invariant patterns, i.e., patterns that are abstracted away from any modality-dependent information. A modality-invariant region responds with similar patterns to input stimuli of the same meaning, even if they are presented in different modalities (e.g. the word ‘cat’ and picture of a cat). Several theories and frameworks on how the brain transforms inputs from different modalities into modality-invariant representations have been proposed. (Here, we do not attempt to disambiguate whether information is modality-invariant or completely abstracted away from input modalities, described in the literature as modality-independent Dirani and Pylkkänen, 2024 , supramodal Sanchez et al., 2020 , transmodal Ralph et al., 2017 , amodal Fairhall and Caramazza, 2013 , or abstract/conceptual Binder, 2016 (see also distinctions made in Barsalou, 2016 ). For our goal of building modality-agnostic decoders, it is only important that information be represented in a way invariant to the stimulus modality. We will come back to this point in the Discussion section.) The convergence zones view proposes that modality-dependent information coming from sensory cortices is integrated in multiple convergence zones that are distributed across the cortex, predominantly in temporal and parietal lobes ( Damasio, 1989 ; Tranel et al., 1997 ; Meyer and Damasio, 2009 ). These convergence zones are organized hierarchically, learned associations are used to create abstractions from lower-level to higher-level feature representations ( Simmons and Barsalou, 2003 ; Meyer and Damasio, 2009 ). A perceived stimulus first causes modality-dependent activity in the related low-level region (e.g. visual cortex), subsequently, higher-level convergence zones serve as relays that cause associated activity in other regions of the brain (e.g. the language network) ( Meyer and Damasio, 2009 ; Kiefer and Pulvermüller, 2012 ). According to Binder, 2016 , the most high-level convergence zones can become so abstract that they are representing amodal symbols. The GRAPES framework (Grounding Representations in Action, Perception, and Emotion Systems) also suggests that representations are distributed across temporal and parietal areas of the cortex. More specifically, they are hypothesized to be situated in areas connected to the perception and manipulation of the environment, as well as in the language system ( Martin, 2009 ; Martin, 2016 ). According to this theory, conceptual knowledge is organized in domains: For example, semantic information related to object form and object motion is represented within specific visual processing systems, regardless of the stimulus modality, and both for perception as well as imagination. The hub-and-spoke theory states that cross-modal interactions are mediated by a single modality-invariant hub, located in the anterior temporal lobes ( Rogers et al., 2004 ; Lambon Ralph et al., 2006 ; Patterson and Lambon Ralph, 2016 ; Ralph et al., 2017 ). The hub contains a ‘continuous distributed representation space that expresses conceptual similarities among items even though its dimensions are not independently interpretable’ ( Frisby et al., 2023 , p. 262). The spokes form the links between the hubs and the modality-dependent association cortices. Most importantly, semantic representations are not solely based in the hub; for a given concept, all spokes that are linked to modalities in which the concept can be experienced do contribute to the semantic representation. This explains why selective damage to spokes can cause category-specific deficits ( Pobric et al., 2010 ). This is conceptually similar to some aspects of the Global Workspace Theory , which assumes both a multimodal convergence of inputs towards a specific (network of) region(s), and the possibility of flexibly recruiting unimodal regions into this Global Workspace ( Baars, 1993 ; Baars, 2005 ). While these and other theories partly disagree on how modality-invariant information is represented in the brain, they agree that such information is distributed across the cortex, and possibly overlapping with the semantic network ( Binder et al., 2009 ; Huth et al., 2012 ; Andrews et al., 2014 ; Zwaan, 2016 ). Decoding of vision and language from fMRI Early approaches to brain decoding focused on identifying and reconstructing limited sets of simple visual stimuli ( Haxby et al., 2001 ; Cox and Savoy, 2003 ; Kay et al., 2008 ; Naselaris et al., 2009 ; Nishimoto et al., 2011 ). Soon after, attempts to decode linguistic stimuli could identify single words and short paragraphs with the help of models trained to predict features extracted from word embeddings ( Pereira et al., 2018 ). More recently, large-scale open-source fMRI datasets for both vision and language have become available ( Chang et al., 2019 ; Allen et al., 2022 ; Schoffelen et al., 2019 ; Tang et al., 2023b ) and allowed for the training of decoding models for a larger range and more complex naturalistic stimuli with the help of features extracted from deep learning models. For example, modality-specific decoders for vision can be trained by mapping the brain activity of subjects viewing naturalistic images to feature representation spaces of computational models of the same modality (i.e. vision models) ( Shen et al., 2019 ; Beliy et al., 2019 ; Lin et al., 2022 ; Takagi and Nishimoto, 2023 ; Ozcelik and VanRullen, 2023 ). Moreover, a range of studies provided evidence that certain representations can transfer between vision and language by evaluating decoders in a modality that they were not trained on Shinkareva et al., 2011 ; Man et al., 2012 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 ; Jung et al., 2018 . The performance in such cross-modal decoding evaluations always lags behind when compared to within-modality decoding. One explanation is that modality-specific decoders are not explicitly encouraged to pick up on modality-invariant features during training, and modality-dependent features do not transfer to other modalities (an exception is the case of modality-specific decoders that are trained using features from the other modality; such decoders might also rely on modality-invariant features, however these might be less easily decodable than the modality-dependent features). To address this limitation, we here propose to directly train modality-agnostic decoders , i.e., models that are exposed to multiple stimulus modalities during training in order to make it more likely that they are leveraging representations that are modality-invariant. Training this kind of decoder is enabled by the multimodal nature of our fMRI dataset: The stimuli are taken from COCO, a multimodal dataset of images with associated descriptive captions ( Lin et al., 2014 ). During the experiment, the subjects are exposed to stimuli in both modalities (images and captions) in separate trials. The modality-agnostic decoders learn to map brain activity associated with a specific stimulus independent of its presentation modality (image or caption) to high-level features extracted from a deep learning model. After training, a single modality-agnostic decoder can be used to decode stimuli from multiple modalities, leveraging representations that are invariant to the stimulus modality. Decoding of mental imagery Apart from decoding perceived stimuli, it is also possible to decode representations when subjects were performing mental imagery, without being exposed to any perceptual input. Different theories on mental imagery processes emphasize either the role of the early visual areas ( Kosslyn et al., 1999 ; Pearson, 2019 ) or the role of the high-level visual areas in the ventral temporal cortex and frontoparietal networks ( Spagna et al., 2021 ; Hajhajate et al., 2022 ; Liu et al., 2025 ). There is evidence for both kinds of theories in the form of neuroimaging studies that used decoding to identify stimuli during mental imagery. Some of these found relevant patterns in the early visual cortex ( Albers et al., 2013 ; Naselaris et al., 2015 ), others highlighted the role of higher-level areas in the ventral visual processing stream ( Stokes et al., 2009 ; Reddy et al., 2010 ; Lee et al., 2012 ; VanRullen and Reddy, 2019 ; Boccia et al., 2019 ) as well as the precuneus and the intraparietal sulcus ( Johnson and Johnson, 2014 ). These discrepancies can possibly be explained by differences in experimental design: For example, the early visual cortex might only become involved if the task requires the imagination of high-resolution details, which are represented in lower levels of the visual processing hierarchy ( Kosslyn and Thompson, 2003 ). Crucially, it has been shown that decoders trained exclusively on trials with perceptual input can generalize to imagery trials ( Stokes et al., 2009 ; Reddy et al., 2010 ; Lee et al., 2012 ; Johnson and Johnson, 2014 ; Naselaris et al., 2015 ), providing evidence that representations formed during perception overlap to some degree with representations formed during mental imagery ( Dijkstra et al., 2019 ). In our study, we explored to what extent these findings hold true for more varied and complex stimuli. Following previous approaches, we use decoders trained exclusively on trials where subjects were viewing images and captions and evaluated them on their ability to decode the imagery trials. We additionally hypothesized that mental imagery should be at least as conceptual as it is sensory and, therefore, primarily recruits modality-invariant representations. Consequently, modality-agnostic decoders should be ideally suited to decode mental imagery and outperform modality-specific decoders on that task. Localizing modality-invariant regions The first evidence for the existence of modality-invariant regions came from observations of patients with lesions in particular cortical regions which led to deficits in the retrieval and use of knowledge across modalities ( Warrington and Shallice, 1984 ; Warrington and Mccarthy, 1987 ; Gainotti, 2000 ; Damasio et al., 2004 ). Semantic impairments across modalities have also been observed in patients with the neurodegenerative disorder semantic dementia ( Warrington, 1975 ; Snowden et al., 1989 ; Jefferies et al., 2009 ). In early work exploring the possible locations of modality-invariant regions in healthy subjects, brain activity was recorded using imaging techniques while they were presented with a range of concepts in two modalities (e.g. words and pictures). Regions that were active during semantic processing of stimuli in the first modality were compared to regions that were active during semantic processing of stimuli in the second modality. The conjunction of these regions was proposed to be modality-invariant ( Vandenberghe et al., 1996 ; Moore and Price, 1999 ; Bright et al., 2004 ). While this methodology allows for the identification of candidate regions in which semantic processing of multiple modalities occurs, it cannot be used to probe the information represented in these regions. In order to compare the information content (i.e. multivariate patterns) of brain regions, researchers have developed Representational Similarity Analysis (RSA, Kriegeskorte et al., 2008 ) as well as encoding and decoding analyses ( Naselaris et al., 2011 ). More specifically, RSA has been used to find modality-invariant regions by comparing activation patterns of a candidate region when subjects are viewing stimuli from different modalities ( Devereux et al., 2013 ; Handjaras et al., 2016 ; Liuzzi et al., 2017 ). This comparison is performed in an indirect way, by measuring the correlation of dissimilarity matrices of activation patterns. In turn, cross-decoding analysis can be leveraged to identify modality-invariant regions by training a classifier to predict the category of a stimulus in a given modality, and then evaluating its performance to predict the category of stimuli that were presented in another modality ( Shinkareva et al., 2011 ; Man et al., 2012 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 ; Jung et al., 2018 ). However, all these studies relied on a predefined set of stimulus categories and can, therefore, not be easily extended to more realistic and complex stimuli, as we perceive them in our everyday life. We summarize candidates for modality-invariant regions that have been identified by previous studies in Appendix 4. This overview reveals substantial disagreement regarding the possible locations of modality-invariant patterns in the brain. For example, Fairhall and Caramazza, 2013 found modality-invariant representations in the left ventral temporal cortex (fusiform, parahippocampal, and perirhinal cortex), middle and inferior temporal gyrus, angular gyrus, parts of the prefrontal cortex as well as the precuneus. Shinkareva et al., 2011 found a larger network of left-lateralized regions, including additionally the left superior temporal, inferior parietal, supramarginal, inferior and inferior occipital, precentral and postcentral gyrus, supplementary motor area, intraparietal sulcus, cuneus, posterior cingulum as well as the right fusiform gyrus and the superior parietal gyrus, paracentral lobule on both hemispheres. In contrast, Jung et al., 2018 found modality-invariant representations only in the right prefrontal cortex. These diverging results can probably be explained by the limited number of stimuli as well as the use of artificially constructed stimuli in certain studies. Recent advances in machine learning have enabled another generation of fMRI analyses based on large-scale naturalistic datasets. Here, we present a new multimodal dataset of subjects viewing both images and text. Most importantly, the dataset contains a large number of naturalistic stimuli in the form of complex visual scenes and full sentence descriptions of the same type of complex scenes, instead of pictures of single objects and words as commonly used in previous studies. This data enables the development of modality-agnostic decoders that are explicitly trained to leverage features that are shared across modalities. Furthermore, we use this data to localize modality-invariant regions in the brain by applying decoders in a multimodal searchlight analysis. Methods fMRI experiment Six subjects (three female, age between 22 and 47 years, all right-handed and fluent English speakers) participated in the experiment after providing informed consent. The study was performed in accordance with French national ethical regulations (Comité de Protection des Personnes, ID 2019-A01920-57). We collected functional MRI data using a 3T Philips ACHIEVA scanner (gradient echo pulse sequence, TR = 2 s, TE = 30 ms, 46 slices with a 32-channel head coil, slice thickness = 3 mm with 0.2 mm gap, in-plane voxel dimensions 3×3 mm). At the start of each session, we further acquired high-resolution anatomical images for each subject (voxel size = 1 mm 3 , TR = 8.13 ms, TE = 3.74 ms, 170 sagittal slices). Scanning was spanned over 10 sessions (except for sub-01: 11 sessions), each consisting of 13–16 runs during which the subjects were presented with 86 stimuli. Each run started and ended with an 8 s fixation period. The stimulus type varied randomly inside each run between images and captions. Each stimulus was presented for 2.5 s at the center of the screen visual angle: 14.6 degrees; captions were displayed in white on a dark gray background (font: ‘Consolas’), the inter-stimulus interval was 1 s. Every 10 stimuli, there was a fixation trial that lasted for 2.5 s. Every 5 min there was a longer fixation trial for 16 s. Subjects performed a one-back matching task: They were instructed to press a button whenever the stimulus matched the immediately preceding one ( Figure 1 Panel A). In case the previous stimulus was of the same modality (e.g. two captions in a row), the subjects were instructed to press a button if the stimuli matched exactly. In the cross-modal case (e.g. an image followed by a caption), the button had to be pressed if the caption was a valid description of the image, and vice versa. Positive one-back trials occurred on average every 10 stimuli. Figure 1. Setup of the fMRI experiment. Open in a new tab ( A ) Setup of the main fMRI experiment. Subjects were seeing images and captions in random alternation. Whenever the current stimulus matched the previous stimulus, the subjects were instructed to press a button (one-back matching task). Images and captions for illustration only; actual size and stimuli as described in the text. ( B ) Number of distinct training stimuli (excluding trials that were one-back targets or during which the subject pressed the response button). The number of overlapping stimuli indicates how many stimuli were presented both as caption and as image. There was an additional set of 140 stimuli (70 images and 70 captions) used for testing. ( C ) Setup of the fMRI experiment for the imagery trials. Subjects were instructed to remember three image descriptions with corresponding indices (numbers 1–3). One of these indices was displayed during the instruction phase, followed by a fixation phase, and then the subjects were imagining the visual scene for 10 s. Images and captions were taken from the training and validation sets of the COCO dataset ( Lin et al., 2014 ). This dataset contains five matching captions for each image, of which we only considered the shortest one in order to fit on the screen and to ensure a comparable length for all captions. Spelling errors were corrected manually. As our training set, a random subset of images and another random subset of captions were selected for each subject. All these stimuli were presented only a single time (except for the case of one-back target trials, for which the second presentation was, however, excluded during the preprocessing of the data). Information on the number of training stimuli for each subject is shown in Figure 1 Panel B. Additionally, a shared subset of 140 stimuli (70 images and 70 captions) was presented repeatedly to each subject in order to reduce noise, serving as our test set (on average: 26 times, min: 22, max: 31) (contrary to the training stimuli which were randomly selected from the COCO dataset, the 70 test stimuli were chosen by hand to avoid, including multiple scenes that could match the same semantic description. The 70 chosen images, as well as their 70 corresponding captions, constituted the test set). These stimuli were inserted randomly between the training stimuli. Note that for each stimulus presented to the subject (e.g. an image), we also have access to the corresponding stimulus in the other modality (the corresponding caption from the COCO dataset), allowing us to estimate model features based on both modalities (vision model features extracted from the image and language model features extracted from the corresponding caption) as well as multimodal features extracted from both the image and the caption. In addition to these perceptual trials, there were three imagery trials for each subject (see also Figure 1 Panel C). Prior to the first fMRI scanning session, each subject was presented with a set of 20 captions (manually selected to be diverse and easy to visualize) that were not part of the perceptual trials, and they selected three captions for which they felt comfortable imagining a corresponding image. Then, they learned a mapping of each caption to a number (1, 2, and 3) so that they could be instructed to perform mental imagery of a specific stimulus, without having to present them with the caption again. The imagery trials occurred every second run, either at the beginning or the end of the run, so that each of the three imagery conditions were repeated on average 26 times (min: 23, max: 29). At the start of the imagery trial, the imagery instruction number was presented for 2 s, then there was a 1 s fixation period followed by the actual imagery period during which a light gray box was depicted for 10 s on a dark gray background (the same background that was also used for trials with perceptual input). The light gray box was meant to represent the area in which the mental image should be ‘projected.’ At the end of the experiment, the subjects drew sketches of the images they had been imagining during the imagery trials. We report dataset quality metrics on head motion and intersession alignment in Appendix 1. fMRI preprocessing Preprocessing of the fMRI data was performed using SPM12 ( Ashburner et al., 2014 ) via nipype ( Gorgolewski et al., 2011 ). We applied slice time correction and realignment for each subject. Each session was coregistered with an anatomical scan of the respective subject’s first session (downsampled to 2 mm 3 ). We created and applied explicit gray matter masks for each subject based on their anatomical scans using a maximally lenient threshold (probability >0). In order to obtain beta values for each stimulus, for each subject, we fit a GLM (using SPM12) on data from all sessions. We included regressors for train images, train captions, test images, test captions, imagery trials, fixations, blank screens, button presses, and one-back target trials. One-back target trials, as well as trials in which the participant pressed the button were excluded in the calculation of all training and test stimulus betas. As output of these GLMs, we obtained beta values for each training and test caption and image, as well as the imagery trials. Finally, we transformed the volume-space data to surface space Freesurfer ( Fischl, 2012 ). We used trilinear interpolation and the fsaverage template in the highest possible resolution (163,842 vertices on each hemisphere) as target. Modality-agnostic decoders The multimodal nature of our dataset allowed for the training of modality-agnostic decoders. We trained decoders by fitting ridge regression models that take fMRI beta values as input and predict latent representations extracted from a pretrained deep learning model. Further details on decoder training can be found in Figure 2 as well as Appendix 2. Figure 2. Training of modality-specific and modality-agnostic decoders. Open in a new tab ( A ) Modality-specific decoders are trained on fMRI data of one modality (e.g. subjects viewing images) by mapping it to features extracted from the same stimuli. ( B ) Modality-agnostic decoders are trained jointly on fMRI data of both modalities (subjects viewing images and captions). ( C ) To train decoders, features can be either extracted unimodally from the corresponding images or captions, or by creating multimodal features based on both modalities. For example, to train a modality-agnostic decoder based on features from a unimodal language model, we map the fMRI data of subjects viewing captions to features extracted from the respective captions using this language model, as well as the fMRI data of subjects viewing images to features extracted by the language model from the corresponding captions. We can also train modality-specific decoders on features from another modality, for example, by mapping fMRI data of subjects viewing images to features extracted from the corresponding captions using a language model. While modality-specific decoders are trained only on brain imaging data of a single modality, modality-agnostic decoders are trained on brain imaging data from multiple modalities and, therefore, allow for decoding of stimuli irrespective of their modality. More specifically, in our case, the modality-specific decoders are trained on fMRI beta values from one stimulus modality, e.g., when subjects were watching images ( Figure 2 panel A). Conversely, modality-agnostic decoders are trained jointly using fMRI data from both stimulus modalities (images and captions; Figure 2 panel B). For all decoders, the features that serve as regression targets can either be unimodal (e.g. extracted from images using a vision model) or multimodal (e.g. extracted from both stimulus modalities using a multimodal model, Figure 2 panel C). We considered features extracted from a range of vision, language, and multimodal models: For vision features, we considered ResNet ( He et al., 2016 ), ViT ( Dosovitskiy et al., 2020 ), and DINOv2 Oquab et al., 2023 ; for language features BERT ( Devlin et al., 2019 ), GPT2 ( Radford et al., 2019 ), Llama2 ( Touvron et al., 2023 ), mistral and mixtral ( Jiang et al., 2023 ). Regarding multimodal features, we extracted features from VisualBERT ( Li et al., 2019 ), BridgeTower ( Xu et al., 2023 ), ViLT ( Kim et al., 2021 ), CLIP ( Radford et al., 2021 ), ImageBind ( Girdhar et al., 2023 ), Flava ( Singh et al., 2022 ), Blip2 ( Li et al., 2023 ), SigLip ( Zhai et al., 2023 ), and Paligemma2 ( Steiner et al., 2024 ). In order to estimate the effect of model training, we further extracted features from a randomly initialized ImageBind model as a baseline. Further details on feature extraction can be found in Appendix 2. All decoders were evaluated on the held-out test data (140 stimuli, 70 captions, and 70 images) using pairwise accuracy calculated using cosine distance. Prior to calculating the pairwise accuracy, the model predictions for all stimuli were standardized to have a mean of 0 and a standard deviation of 1 (In the case of imagery decoding, the model predictions were standardized separately). In the case of cross-modal decoding (e.g. mapping an image stimulus into the latent space of a language model), a trial was counted as correct if the caption corresponding to the image (according to the ground truth in COCO) was closest. Figure 3 provides an overview of the evaluation metrics. A modality-specific decoder for images can be evaluated on its ability to decode images (Panel A, top) and in a cross-decoding setup for captions (Panel B, bottom). In the same way, we can compute the respective evaluation metrics for modality-specific decoders trained on captions. For the case of modality-agnostic decoders, we evaluate performance for decoding both images and captions using the same single decoder that is trained on both modalities (Panel C). Figure 3. Evaluation of modality-specific and modality-agnostic decoders. Open in a new tab The matrices display cosine similarity scores between features extracted from the candidate stimuli and features predicted by the decoder. The evaluation metric is pairwise accuracy, which is calculated row-wise: For a given matrix row, we compare the similarity score of the target stimulus on the diagonal (in green) with the similarity scores of all other candidate stimuli (in red). ( A ) Within-modality decoding metrics of modality-specific decoders. To compute within-modality accuracy for image decoding, a modality-specific decoder trained on images is evaluated on all stimuli that were presented as images. To compute within-modality accuracy for caption decoding, a modality-specific decoder trained on captions is evaluated on all caption stimuli. ( B ) Cross-modality decoding metrics of modality-specific decoders. To compute cross-modality accuracy for image decoding, a modality-specific decoder trained on captions is evaluated on all stimuli that were presented as images. To compute cross-modality accuracy for caption decoding, a modality-specific decoder trained on images is evaluated on all caption stimuli. ( C ) Metrics for modality-agnostic decoders. To compute modality-agnostic accuracy for image decoding, a modality-agnostic decoder is evaluated on all stimuli that were presented as images. The same decoder is evaluated on caption stimuli to compute modality-agnostic accuracy for caption decoding. Here, we show feature extraction based on unimodal features for modality-specific decoders and based on multimodal features for the modality-agnostic decoder. In practice, the feature extraction can be unimodal or multimodal for any decoder type (see also Figure 2 ). Results Modality-agnostic decoders We first compared the performance of modality-specific and modality-agnostic decoders that are trained on the whole brain fMRI data based on different unimodal and multimodal features. The average pairwise accuracy scores are presented in Figure 4 . Figure 5 presents pairwise accuracy scores separately for decoding images and for decoding captions. Results for individual subjects can be found in Appendix 6. Figure 4. Average decoding scores for modality-agnostic decoders (green), compared to modality-specific decoders trained on data from subjects viewing images (orange) or on data from subjects viewing captions (purple). Open in a new tab The metric is pairwise accuracy (see also Figure 3 ). Error bars indicate 95% confidence intervals calculated using bootstrapping. Chance performance is at 0.5. Figure 5. Decoding accuracy for decoding images (top) and for decoding captions (bottom). Open in a new tab The orange bars in the top row indicate within-modality decoding scores for images, the purple bars in the bottom row indicate within-modality decoding scores for captions. The purple bars in the top row indicate cross-decoding scores for images, the orange bars in the bottom row indicate cross-decoding scores for captions (see also Figure 3 ). Error bars indicate 95% confidence intervals calculated using bootstrapping. Chance performance is at 0.5. When analyzing the average decoding accuracy ( Figure 4 ), we find that modality-agnostic decoders perform better than modality-specific decoders. To support this observation, we performed two repeated measures ANOVAs (grouping the data by subject), once comparing the average accuracy values of modality-agnostic decoders (mean value: 81.86%) with those of modality-specific decoders trained on images (mean value: 78.15%), and once comparing modality-agnostic decoders to modality-specific decoders trained on captions (mean value: 72.52%). In both cases, the accuracy values for the two decoder types were significantly different. When comparing modality-agnostic decoders to modality-specific decoders trained on images: decoder\_type : β = 0.037 , S E = 0.004 , p < 1 ⋅ 10 − 18 ; and when comparing to modality-specific decoders trained on captions: decoder\_type : β = 0.093 , S E = 0.005 , p < 1 ⋅ 10 − 93 . This high performance (which can be attributed to the large training dataset used by modality-agnostic decoders) is achieved despite the additional challenge of not knowing the modality of the stimulus the subject was seeing. Furthermore, we observed that modality-agnostic decoders based on the best multimodal features (imagebind: 85.71 % ± 2.58 % ) do not perform substantially better than decoders based on the best language features (GPT2-large: 85.31 % ± 2.83 % ) and only slightly better than decoders trained on the best vision features (Dino-giant: 82.02 % ± 2.43 % ). This result suggests that high-performing modality-agnostic decoders do not necessarily need to rely on multimodal features; features extracted from language models can lead to equally high performance. When comparing the different architecture types of models for multimodal feature extraction (dual stream vs. single stream with early fusion vs. single stream with late fusion; cf. Panel C in Figure 2 ), we only observed a slight performance disadvantage for single-stream models with early fusion (Mean accuracy values for dual stream models: 85.04%; for single stream models with early fusion: 81.01%; for single stream models with late fusion: 83.63%). We confirmed this observation with a repeated measures ANOVA (grouping the data by subject), comparing the decoding accuracy values of modality-agnostic decoders based on different families of multimodal features. The only significant effect was: model\_family\_single\_stream\_early\_fusion : β = − 0.04 , S E = 0.011 , p < 1 e − 3 . When analyzing the performance specifically for decoding images ( Figure 5 , top), we find that modality-agnostic decoders perform as well as the modality-specific decoders trained on images (green bars in the top row are at the same level as the orange bars in Figure 5 ). We found no statistically significant difference in their performances. A repeated measures ANOVA (grouping the data by subject), comparing the image decoding accuracy values of modality-agnostic decoders with those of modality-specific decoders trained on images resulted in a p-value of p = 0.73 for the effect of the decoder_type. Furthermore, modality-agnostic decoders even outperform modality-specific decoders trained on captions for decoding captions (purple bars in the bottom row are lower than green bars in Figure 5 ). This was supported by a repeated measures ANOVA (grouping the data by subject), comparing the caption decoding accuracy values of modality-agnostic decoders with those of modality-specific decoders trained on captions. The result was: decoder\_type : β = 0.036 , S E = 0.006 , p < 1 ⋅ 10 − 8 . In other words, even if we know that a brain pattern was recorded in response to the subject reading a caption, we are more likely to decode it accurately if we choose to apply a decoder trained using both modalities than if we apply the appropriate decoder, trained only on captions. Finally, we found that the cross-modal decoding performance for decoding visual stimuli (images) using decoders trained on captions (mean value: 73.31%) is higher than the cross-modal decoding performance in the other direction (mean value: 68.03%), corroborating similar results from Tang et al., 2023a on movies and audio books (purple bars in top row are higher than orange bars in bottom row in Figure 5 ). We performed a repeated measures ANOVA (grouping the data by subject) to compare the pairwise accuracy values of both cross-decoding directions and found that the difference was significant: decoder\_type : β = 0.053 , S E = 0.004 , p < 1 ⋅ 10 − 31 (the decoder_type variable is indicating whether the decoder is trained on images or captions). Qualitative decoding results To obtain a better understanding of the decoding performance of the modality-agnostic decoders, we inspected the decoding results for five randomly selected test stimuli. We created a large candidate set of 41,118 stimuli by combining the test stimuli and the training stimuli from all subjects. For each stimulus, we ranked these candidate set stimuli based on their similarity to the predicted feature vector. As the test stimuli were shared among all subjects, we could average the prediction feature vectors across subjects to obtain the best decoding results. Figure 6 presents the results for decoding images using a modality-agnostic decoder trained on ImageBind features. We display the target stimulus along with the top-5 ranked candidate stimuli. We can observe some clear success cases (the train in the first row, a person eating pizza in the last row) but also failure cases (the teddy bear decoded as pizza). For the other stimuli, some aspects, such as the high-level semantic class (e.g. vehicle, animal, sports) are correctly decoded: For the cars on the highway (fifth row), the top-ranked images depict trains, which are also vehicles. For the dog (second row), the top images contain cats of similar colors. The footballer in the sixth row is decoded as a tennis player, and the horses of the seventh row are decoded as zebras. Regarding the giraffe (third row), the model appears to have picked up on the fact that there was a body of water depicted in the image. Figure 6. Decoding examples for image decoding using a modality-agnostic decoder. Open in a new tab The first column shows the image the subject was seeing and the five following columns show the candidate stimuli with highest similarity to the predicted features, in descending order. We display both the image and the caption of the candidate stimuli because the decoder is based on multimodal features that are extracted from both modalities. All images were taken from the CoCo dataset ( Lin et al., 2014 ). Note that these qualitative results are not directly comparable with previous work on retrieval or reconstruction using the NSD dataset ( Allen et al., 2022 ; Lin et al., 2022 ; Takagi and Nishimoto, 2023 ; Ozcelik and VanRullen, 2023 ), as our data was collected on a 3T MRI scanner with lower signal-to-noise ratio than NSD’s 7T MRI scanner. The ranking results for decoding captions are depicted in Figure 7 . The results are somewhat similar to the image decoding results. Stimuli that were decoded successfully when presented as image, such as the train are also decoded successfully when presented as caption; cases that were failures in the case of image decoding (e.g. the teddy bear) also fail here. However, the top-ranked stimuli for the dog (second row) do not always contain animals (the decoder seems to have picked up on the presence of a vehicle in the caption), and the horses of the seventh row do not get decoded correctly. The cars on the highway (fifth row) get decoded more accurately than during image decoding. Figure 7. Decoding examples for caption decoding using a modality-agnostic decoder. Open in a new tab For details, see caption of Figure 6 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). We additionally provide qualitative results for modality-specific decoders in Appendix 3. These results generally reflect the observations from the quantitative results: Modality-agnostic decoders perform similarly to modality-specific decoders evaluated in a within-modality decoding setup, but substantially better than modality-specific decoders when evaluated in cross-decoding setups. Modality-invariant regions Modality-agnostic decoders perform best when they can leverage modality-invariant representations. To provide insight into the spatial organization of such modality-invariant representations in the brain, we performed a surface-based searchlight analysis. Modality-invariant regions should contain patterns that generalize between stimulus modalities. Therefore, such regions should allow for decoding of stimuli in both modalities using a decoder that is trained to pick up on modality-invariant features, i.e., the decoding performance for images and captions of a modality-agnostic decoder should both be above chance. However, as a modality-agnostic decoder is trained on stimuli from both modalities, it could have learned to leverage certain features to project stimuli from one modality and different features to project stimuli from the other modality. We added two conditions to control that the representations directly transfer between the modalities by additionally training two modality-specific decoders and evaluating them according to their cross-decoding performance, i.e., we require that their decoding performance in the modality they were not trained on is above chance. These four conditions are summarized at the top of Figure 8 . Figure 8. Searchlight method to identify modality-invariant ROIs. Open in a new tab The top plots show performance (pairwise accuracy averaged over subjects) of modality-agnostic decoders for decoding images (top left) and decoding captions (top right). In the second row, we display cross-decoding performances: On the left, modality-specific decoders trained on captions are evaluated on images. On the right, modality-specific decoders trained on images are evaluated on captions. We identified modality-invariant ROIs as clusters in which all 4 decoding accuracies are above chance by taking the minimum of the respective t-values at each location, then performed threshold-free cluster enhancement (TFCE) to calculate cluster values. The plot only shows left medial views of the brain to illustrate the method; different views of all resulting clusters are shown in Figure 9 . We used ImageBind features for these searchlight analyses as they led to the highest decoding performance when using the whole brain data. The decoders were trained based on the surface projection of the fMRI beta values. For each vertex, we defined a searchlight with a fixed size by selecting the 750 closest vertices, corresponding to an average radius of ∼9.4 mm (details on how this size was selected are outlined in Appendix 5). We trained and evaluated a modality-agnostic decoder and modality-specific decoders for both modalities on the beta values for each searchlight location and each subject, providing us with decoding accuracy scores for each location on the cortex. Then we performed one-tailed t-tests to identify locations in which the decoding performance is above chance ( a c c > 0.5 ). We aggregated all four comparisons by taking the minimum of the four t-values at each spatial location. Finally, we performed threshold-free cluster enhancement ([, TFCE,)]smith_threshold-free_2009 to identify modality-invariant ROIs ( Figure 8 , bottom) (we used the default hyperparameters of h =2 and e=1 for surface-based TFCE Jenkinson et al., 2012 ). To estimate the statistical significance of the resulting clusters, we performed a permutation test, combined with a bootstrapping procedure to estimate a group-level null distribution (see also Stelzer et al., 2013 ). For each subject, we evaluated the decoders 100 times with shuffled labels to create per-subject chance-level results. Then, we randomly selected one of the 100 chance-level results for each of the six subjects and calculated group-level statistics (TFCE values) the exact same way as described in the preceding paragraph. We repeated this procedure 10,000 times resulting in 10,000 permuted group-level results. We ensured that every permutation was unique, i.e., no two permutations were based on the same combination of selected chance-level results. Based on this null distribution, we calculated p-values for each vertex by calculating the proportion of sampled permutations where the TFCE value was greater than the observed TFCE value. To control for multiple comparisons across space, we always considered the maximum TFCE score across vertices for each group-level permutation ( Smith and Nichols, 2009 ). The results of the surface-based searchlight analysis are presented in Figure 9 . The analysis revealed that modality-invariant patterns are actually widespread across the brain, especially on the left hemisphere. Peak cluster values were found in the left supramarginal gyrus, inferior parietal gyrus and posterior superior temporal sulcus. Regions belonging to the precuneus, isthmus of the cingulate gyrus, parahippocampus, middle temporal gyrus, inferior temporal gyrus, and fusiform gyrus also showed high cluster values. Figure 9. Searchlight results for modality-invariant regions. Open in a new tab Maps thresholded at threshold-free cluster enhancement (TFCE) value of 1508, which is the significance threshold value for which p < 10 − 4 based on the permutation testing. Regions with the highest cluster values are outlined and annotated based on the Desikan-Killiany atlas ( Desikan et al., 2006 ). Imagery decoding Finally, we evaluated the ability of decoders trained on the fMRI data with perceptual input to decode stimuli during the imagery conditions. For a modality-agnostic decoder trained on the whole-brain data, the imagery pairwise decoding accuracy reaches 84.48% (averaged across subjects and model features) when using the three imagery stimuli as the candidate set (note that we used the ground-truth caption and corresponding image from COCO in this candidate set, and not the sketches drawn by the subjects). When the whole test set is added to the candidate set (in total: 73 stimuli), the average pairwise accuracy drops to 72.47%. This substantial drop in performance is most likely explained by the fact that the predicted features for the imagery trials were standardized using only three stimuli, and this transformation emphasized differences that enabled distinguishing the three imagery trials but do not generalize to the larger test set (we also attempted decoding without standardization of the predicted feature vectors, but this led to much lower performance). As expected, we also found that modality-agnostic decoders are better suited for imagery decoding than modality-specific decoders. We compared the imagery decoding accuracy of both decoder types, taking into account the results for all features and all subjects. To this end, we performed two repeated measures ANOVAs (grouping the data by subject), once comparing the accuracy values of modality-agnostic decoders with those of modality-specific decoders trained on images, and once comparing modality-agnostic decoders to modality-specific decoders trained on captions. The average decoding accuracies were 69.42% for a modality-specific decoder trained on images and 70.02% for a modality-specific decoder trained on captions (vs. 72.47% for a modality-agnostic decoder, as mentioned above). In both comparisons, the accuracy values for the two decoder types were significantly different (when comparing modality-agnostic decoders to modality-specific decoders trained on images: decoder\_type : β = 0.03 , S E = 0.011 , p < 0.01 ; and when comparing to modality-specific decoders trained on captions: decoder\_type : β = 0.024 , S E = 0.012 , p < 0.04 ). Appendix 7 presents qualitative decoding results for the imagery trials for each subject as well as the sketches of the mental images drawn at the end of the experiment. As expected, the results are worse than those for perceived stimuli, but for several subjects, it was possible to decode some major semantic concepts. We further computed the imagery decoding accuracy during the searchlight analysis. Figure 10 shows the result clusters for decoding imagery (using the whole test set + the three imagery trials as potential candidates). To assess statistical significance, we employed the same permutation testing procedure with bootstrapping as used for identifying modality-invariant regions (10,000 permutations based on random combinations of 100 chance distributions for each subject). Figure 10. Searchlight results for imagery decoding. Open in a new tab Maps thresholded at threshold-free cluster enhancement (TFCE) values that surpass the significance threshold of p < 10 − 4 based on a permutation test. Maps thresholded at TFCE value of 3897, which is the significance threshold value for which p < 10 − 4 based on the permutation testing. We used the pairwise accuracy for imagery decoding using the large candidate set of 73 stimuli. We outlined the same regions as in Figure 9 to facilitate comparison. We observe that many regions that were found to contain modality-invariant patterns ( Figure 9 ) are also regions in which decoding of mental imagery is possible. One main difference is that the imagery decoding clusters appear to be less left-lateralized than the modality-invariant region clusters (peak cluster values can be found both on the right inferior parietal cortex and bilaterally in the precuneus). To estimate the overlap of the regions allowing for imagery decoding and modality-invariant regions, we calculated the correlation between the TFCE values that were used for identifying modality-invariant regions ( Figure 9 ) and the TFCE values for imagery decoding ( Figure 10 ). The Pearson correlation score for the left hemisphere is 0.41 ( p < 1 e − 8 ), and for the right hemisphere, 0.62 ( p < 1 e − 8 ). Importantly, these correlation scores are substantially higher when compared to the correlation with decoding accuracy of modality-specific decoders: The correlation between the TFCE values for imagery decoding and TFCE values for image decoding of a modality-specific decoder trained on images is 0.28 on the left hemisphere and 0.40 on the right hemisphere. When using TFCE values based on the caption decoding accuracy of a modality-specific decoder trained on captions, we obtain 0.19 on the left hemisphere and 0.45 on the right hemisphere. Discussion In this work, we introduced modality-agnostic decoders, which are trained to decode stimuli from different modalities using a single model. Such modality-agnostic decoders are specifically trained to pick up on modality-invariant representations, enabling a performance increase over modality-specific decoders when decoding linguistic stimuli in the form of captions as well as when decoding mental imagery. Previous work on decoding with large-scale brain imaging datasets has largely focused on modality-specific cortices, such as the visual cortex for decoding natural scenes ( Shen et al., 2019 ; Beliy et al., 2019 ; Lin et al., 2022 ; Takagi and Nishimoto, 2023 ; Ozcelik and VanRullen, 2023 ). Our work demonstrates the advantages of leveraging higher-level representations that are shared between modalities, and which we found to be present in widespread areas of the human cortex. In the following, we will further discuss the nature and location of the modality-invariant representations in light of previous experimental and theoretical work. The results of our searchlight analysis suggest that modality-invariant representations are found in temporal, parietal, and frontal regions. Peak cluster values are found on the border of the temporal and parietal cortices on the left hemisphere ( Figure 9 ). All areas with high cluster values confirm findings from previous studies: The left precuneus ( Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Popham et al., 2021 ; Handjaras et al., 2016 ), posterior cingulate/retrosplenial cortex ( Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 ), supramarginal gyrus ( Shinkareva et al., 2011 ), inferior parietal cortex ( Man et al., 2012 ; Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Devereux et al., 2013 ; Popham et al., 2021 ; Simanova et al., 2014 ; Handjaras et al., 2016 ), superior temporal sulcus ( Man et al., 2012 ), middle temporal gyrus ( Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Devereux et al., 2013 ; Handjaras et al., 2016 ), inferior temporal gyrus ( Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 ; Handjaras et al., 2016 ), fusiform gyrus ( Vandenberghe et al., 1996 ; Moore and Price, 1999 ; Bright et al., 2004 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 ), and parahippocampus ( Vandenberghe et al., 1996 ). However, previous studies have led to contradicting results regarding the locality of modality-invariant regions (they were identifying varying subsets of these regions; see also Appendix 4), probably due to the limited number and artificial nature of stimuli employed. Our method identified almost all of the previously proposed regions as regions with modality-invariant patterns, highlighting the advantage of our searchlight decoding approach and the large multimodal dataset in which subjects are viewing photographs of complex natural scenes and reading full English sentences. (The left superior occipital gyrus was not identified in our study, but in previous studies by Shinkareva et al., 2011 ; Vandenberghe et al., 1996 . However, we found that a major part of the left superior occipital sulcus represents modality-invariant information. Furthermore, Jung et al., 2018 found modality-invariant patterns in the right superior frontal gyrus. One major difference between their study and ours is that they used auditory input as a second modality instead of text. Further work is required to investigate to what extent the modality-invariant regions identified in our work generalize to all modalities.) According to a range of theories, modality-invariant representations are tightly linked to (lexical-) semantic representations ( Simmons and Barsalou, 2003 ; Binder et al., 2009 ; Meschke and Gallant, 2024 ). Most importantly, a range of studies that aimed to identify brain regions linked to semantic/conceptual representations by asking subjects to perform tasks that require semantic processing of words found evidence for such regions that overlap to a high degree with the regions identified in our study ( Fernandino et al., 2016 ; Martin et al., 2018 ; Carota et al., 2021 ; Fernandino et al., 2022 ; Tong et al., 2022 ). A strong link between these systems could also explain our result that modality-agnostic decoders based on unimodal representations from language models are performing as well as decoders based on multimodal representations ( Figure 4 ), as well as the partial left-lateralization of the identified modality-invariant regions. The fact that the presence of modality-invariant patterns is positively correlated with the imagery decoding performance in different locations provides further evidence that the identified patterns are truly modality-invariant (they are not only shared between vision and language, but also for mental imagery). We further found that decoders trained exclusively on data for which participants were exposed to perceptual input to generalize to imagery trials, confirming previous findings that were based on more limited stimulus sets ( Stokes et al., 2009 ; Reddy et al., 2010 ; Lee et al., 2012 ; Johnson and Johnson, 2014 ; Naselaris et al., 2015 ). Moreover, we found that modality-agnostic decoders outperform modality-specific decoders in terms of imagery decoding, demonstrating another advantage of this new type of decoder explicitly trained to pick up modality-invariant patterns. The findings of our searchlight analysis for imagery decoding suggest that mental imagery indeed involves a large network of regions across both hemispheres of the cerebral cortex. This includes high-level visual areas, parietal areas, such as the precuneus and inferior parietal cortex and several frontal regions, but also parts of the early visual cortex. Results are highly similar on both hemispheres, highlighting the involvement of large-scale bilateral brain networks during mental imagery of complex scenes. While there are lesion studies on hemispheric asymmetries that suggest that regions in the left hemisphere are crucial for mental imagery ( Farah, 1984 ; Bartolomeo, 2002 ), a more recent review that additionally considers evidence from neuroimaging and direct cortical stimulation studies suggests that frontoparietal networks in both hemispheres are involved in mental imagery, and that lateralization patterns can be found in the temporal lobes ( Liu et al., 2022 ). Such lateralization was found to depend on the nature of the imagined items, the imagination of objects and words involving the left inferior temporal cortex, while the imagination of faces and people was found to be more right-lateralized, and the imagination of complex scenes (as in our study) led to significant activity in both hemispheres ( O’Craven and Kanwisher, 2000 ; Steel et al., 2021 ; Spagna et al., 2021 ). Crucially, in more recent decoding studies, results were either observed bilaterally, or the analyses did not target hemispheric asymmetries ( Reddy et al., 2010 ; Lee et al., 2012 ) (but see Stokes et al., 2009 in which perception-to-imagery generalization of single letters was left-lateralized). Our work shows for the first time results of a searchlight analysis of imagery decoding of complex visual scenes. Above-chance decoding is possible on both hemispheres, with the highest decoding accuracies in the precuneus, the right inferior parietal cortex and the superior temporal sulcus. Future investigations with larger sets of imagined scenes could address the question of whether lateralization patterns depend on the nature of the imagined objects. According to the results of our searchlight analysis, the anterior temporal lobes are not among the regions with the highest probability of being modality-invariant, contradicting the hypothesis of the hub-and-spoke theory that these areas are the major semantic hub in the brain. However, MRI signals from these regions have a lower signal-to-noise ratio with standard fMRI pulse sequences ( Devlin et al., 2000 ; Embleton et al., 2010 ). A more targeted study with an adapted fMRI protocol would be required to shed light on the nature of patterns in these regions. More generally, the hub-and-spoke theory also puts emphasis on the role of spokes for the formation of conceptual representations ( Pobric et al., 2010 ; Ralph et al., 2017 ). Future work could be aimed at testing the hub-and-spoke theory proposal that features in hierarchically lower level representation spaces of the spokes are not directly relatable to features in the representation space of the hubs: Object representations in the spokes are based on interpretable features (e.g. shape, color, affordances of an object) and get translated into another representational format that is representing conceptual similarities (but its dimensions do not directly map to interpretable features) in the semantic hub ( Frisby et al., 2023 ). To test this hypothesis, modality-invariant representations in the anterior temporal lobes (measured with targeted fMRI pulse sequences) could be compared to representations in candidate regions for modality-specific spokes using RSA. The modality-invariant regions we found in the searchlight analysis can also be seen as candidates for convergence zones, in which increasingly abstract representations are formed ( Damasio, 1989 ; Tranel et al., 1997 ; Meyer and Damasio, 2009 ). To obtain further insight into the hierarchical organization of these zones, future work could take advantage of the improved temporal resolution of other brain imaging techniques, such as MEG to explore in which areas modality-invariant patterns are formed first, and how they are being transformed when spreading to higher-level areas of the brain ( Dirani and Pylkkänen, 2024 ; Benchetrit et al., 2024 ). In line with the GRAPES framework, ( Martin, 2009 ; Martin, 2016 ), we found that modality-invariant representations are distributed across temporal and parietal areas. To test the related hypothesis that conceptual information is organized in domains, we plan to use RSA to understand which kind of semantic information is represented in the different modality-invariant regions identified. Finally, our results can be interpreted with respect to the Global Workspace Theory. All modality-invariant regions are good candidate regions for a global workspace. They could, however, also be part of modality-specific modules that get activated in a modality-invariant fashion through a ‘broadcast’ operation as a stimulus is perceived consciously ( Baars, 1993 ; Baars, 2005 ). To distinguish these two cases, an experimental manipulation of attention could be used: according to Global Workspace Theory, attention is required for information to enter the workspace, but not for the workspace signals to reach other brain regions via broadcast. In the future, we plan to investigate how modality-invariant patterns are modulated by attention, by analyzing additional test sessions from the same subjects in which they were instructed in specific runs to pay attention to only one of the modalities (these sessions will be released as part of another future data set and publication) . To conclude, the results from our searchlight analysis so far are in line with all major theories on modality-invariant representations that were considered. As the dataset that this study was based on will be shared publicly, more targeted investigations can be performed by the research community in order to adjudicate between different theories. Limitations There are some important limitations to this study, which are mainly consequences of some design choices to enable large-scale data collection. First, the fMRI experiment relied on a one-back task, which required subjects to remember preceding stimuli, and therefore, brain activity in each trial could be influenced by the previous stimuli kept in working memory. Such unrelated activity could have impaired the performance of decoders, which were only trained to decode the presented stimulus. However, this limitation could also represent an opportunity for future work on modality-invariant representations in working memory, for example, by training decoders to decode the preceding stimuli from such working memory activity. Second, the inclusion of the cross-modal one-back task could have incentivized subjects to perform visual imagery when reading the captions, or to recall descriptive words (and/or have inner speech) when viewing images. Such behavior might have facilitated the spread of modality-invariant information in the cortex. It remains an open question to what degree such widespread modality-invariant information is also present during isolated reading and visual inspection of images, without any cross-modal task. Finally, it remains an open question whether the activation patterns in the modality-invariant regions identified in our study relate to abstract concepts or to lower-level features that are shared between the two modalities (see also Section Modality-invariant representations). Our current study did not aim to tackle this question, as it is not crucial for the goal of building modality-agnostic decoders. In the future, techniques, such as RSA and encoding models based on carefully designed feature spaces could be used to explore the nature of the representations in more detail. One major challenge to overcome will be the spatial overlap of modality-dependent, modality-invariant and abstract information in brain representations ( Fernandino et al., 2016 ; Liuzzi et al., 2020 ; Dirani and Pylkkänen, 2024 ). More generally, Binder, 2016 puts the dichotomy between modality-invariant and abstract representations into question, considering that ‘there is no absolute demarcation between embodied/perceptual and abstract/conceptual representation in the brain.’ (p. 1098). The author argues for a hierarchical system in which representational patterns become increasingly abstract, creating a continuum from actual experiential information up to higher-level conceptual information (see also Andrews et al., 2014 ). Acknowledgements This research was funded by grants from the French Agence Nationale de la Recherche (ANR: AI-REPS grant number ANR-18-CE37-0007-01 and ANITI grant number ANR-19-PI3A-0004) as well as the European Union (ERC Advanced grant GLoW, 101096017). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. We thank the Inserm/UPS UMR1214 Technical Platform for their help in setting up and for the acquisitions of the MRI sequences. Appendix 1 Dataset quality metrics Head motion Appendix 1—figure 1 and Appendix 1—figure 2 present information on head motion for each subject. All head motion estimates are based on the realignment parameters calculated using SPM12. We found that head motion estimates are largely stable and do not vary extensively between sessions and subjects. Appendix 1—figure 1. Head motion estimates for each subject. Open in a new tab The plots show the realignment parameters as computed by SPM12 (spm_realign) for estimating within-modality rigid body alignment. We multiplied the rotation parameters pitch, roll, and yaw (originally in radian units) by 50 in order to allow interpretation in terms of millimeters of displacement for a circle of diameter 10 cm (which is approximately the mean distance from the cerebral cortex to the center of the head) ( Power et al., 2012 ). The translucent error bands show 95% confidence intervals calculated using bootstrapping over all frames/runs from a session. Appendix 1—figure 2. Frame-wise displacement for each subject. Open in a new tab The measure indicates how much the head changed position from one frame to the next. We calculated framewise displacement as the sum of the absolute values of the derivatives of the six realignment parameters. The translucent error bands show 95% confidence intervals calculated using bootstrapping over all frames/runs from a session. Intersession alignment During preprocessing, all functional MRI data was coregistered to the anatomical scan of the first session. To assess to which degree the alignment of functional data of different sessions varies with respect to the alignment of the first session, we estimated mutual information between the coregistered scans and the reference anatomical scan. During coregistration, our pipeline based on SPM12 created a mean image of the functional data for each session to estimate transformation parameters specific to each scanning session. We calculated mutual information between the coregistered mean images and the anatomical scan of the first session. Then, we normalized all values with respect to the mutual information of the first session (i.e. the normalized mutual information value for session 1 for all subjects is 1). The results are shown in Appendix 1—figure 3 . The plot shows that the mutual information does not deviate strongly from the baseline (minimum value: 90%) and stays in an interval of ±5% for most subjects. Appendix 1—figure 3. Mutual information between the anatomical scan of the first session and functional data for each session as an indicator of intersession alignment. Open in a new tab All values were normalized based on the mutual information from the functional data from the first session. The raw mutual information scores for the alignment during the first session are: subject 1: 0.49; subject 2: 0.55; subject 3: 0.55; subject 4: 0.53; subject 5: 0.57; subject 6: 0.57. Appendix 2 Feature extraction details For each target stimulus (image or caption), our database also contained an equivalent stimulus in the other modality (caption or image). In this way, we could extract model features from the corresponding image for vision models, the corresponding caption for language models, and a multimodal representation of both image and caption for the multimodal models. We used publicly available pretrained models implemented in the Hugging Face Transformers library ( Wolf et al., 2020 ) or from their respective authors’ repositories. Model versions for unimodal models are as indicated in Figure 4 . For multimodal models, the exact version for CLIP was clip-vit-large-patch14 , for ViLT vilt-b32-mlm , for VisualBERT visualbert-nlvr2-coco-pre , for ImageBind imagebind_huge , for Bridgetower bridgetower-large-itm-mlm-itc , for Flava flava-full , for SigLip siglip-so400m-patch14-384 , and for Paligemma2 paligemma2-3b-pt-224 . We extracted language features from all models by averaging the outputs for each token, as this was established as common practice for the extraction of sentence embeddings from Transformer-based language models ( Krasnowska-Kieraś and Wróblewska, 2019 ; Reimers and Gurevych, 2019 ). For Transformer-based vision models, we compared representations extracted by averaging the outputs for each patch with representations extracted from [CLS] tokens in Appendix 2—table 1 . We found that for almost all models, the mean features allow for higher decoding accuracies. For all experiments reported in the main paper, we, therefore, only considered this method. Appendix 2—table 1. Feature comparison for vision models. Pairwise accuracy for modality-agnostic decoders based on vision features extracted by averaging the last hidden states of all patches (‘vision_features_mean’) compared to when using features extracted from [CLS] tokens (‘vision_features_cls’). The method leading to the best decoding performance for each model is highlighted in bold. Model Vision features Pairwise accuracy dino-base vision_features_cls 0.763 dino-base vision_features_mean 0.819 dino-giant vision_features_cls 0.750 dino-giant vision_features_mean 0.820 dino-large vision_features_cls 0.754 dino-large vision_features_mean 0.816 vit-b-16 vision_features_cls 0.788 vit-b-16 vision_features_mean 0.772 vit-h-14 vision_features_cls 0.785 vit-h-14 vision_features_mean 0.804 vit-l-16 vision_features_cls 0.788 vit-l-16 vision_features_mean 0.796 Open in a new tab For multimodal models, we also compared a range of techniques for feature extraction. The results are shown in Appendix 2—table 2 . Appendix 2—table 2. Feature comparison for multimodal models. The features are either based on the [CLS] tokens from the fused representations (‘fused_cls’), averaging over the fused tokens (‘fused_mean’), or averaging over tokens from intermediate vision and language stream outputs (‘avg’). For the last case, for some models, the vision and language features can also be either based on [CLS] or based on averaging over all tokens/patches. The method leading to the best decoding performance for each model is highlighted in bold. Model Features Vision features Language features Pairwise accuracy blip2 avg vision_features_cls lang_features_cls 0.851 blip2 fused_cls vision_features_cls lang_features_cls 0.710 blip2 fused_mean vision_features_cls lang_features_cls 0.740 bridgetower fused_cls 0.815 bridgetower fused_mean 0.788 clip avg vision_features_cls lang_features_cls 0.842 flava avg vision_features_cls lang_features_cls 0.842 flava fused_cls vision_features_cls lang_features_cls 0.752 flava fused_mean vision_features_cls lang_features_cls 0.772 imagebind avg vision_features_cls lang_features_cls 0.857 paligemma2 avg vision_features_cls lang_features_mean 0.829 paligemma2 avg vision_features_mean lang_features_mean 0.848 paligemma2 fused_mean vision_features_mean lang_features_mean 0.828 siglip avg vision_features_cls lang_features_cls 0.852 siglip avg vision_features_mean lang_features_cls 0.823 vilt fused_cls 0.759 vilt fused_mean 0.839 visualbert fused_cls 0.639 visualbert fused_mean 0.743 Open in a new tab For dual-stream multimodal models, we averaged the vision and language features to create the final multimodal feature representation. For single-stream multimodal features, we compared using representations extracted by averaging the outputs for each token with representations extracted from [CLS] token and found that the averaged output leads to better performance in almost all cases (see Appendix 2—table 2 ). Furthermore, some single-stream models (Flava, Paligemma2, and BLIP2) allow for feature extraction based on intermediate vision and language representations in addition to a direct extraction of multimodal features (based on fused representations from the multimodal stream). We found that averaging features from these intermediate vision and language representations leads to better performance (see Appendix 2—table 2 ). For all results reported in the main paper, we used the feature extraction method leading to best performance for each model. Decoder training details The decoders were linear ridge-regression models as implemented in the scikit-learn library ( Pedregosa et al., 2011 ). All training data was standardized to have a mean of 0 and a standard deviation of 1. The test data was standardized using the mean and standard deviation of the training data. The regularization hyperparameter α was optimized using fivefold cross-validation on the training set (values considered: α ∈ { 1 e 3 , 1 e 4 , 1 e 5 , 1 e 6 , 1 e 7 } ). Afterwards, a final model was trained using the best α on the whole training set. Appendix 3 Qualitative decoding results for modality-specific decoders In this section, we present qualitative decoding results for modality-specific decoders for the same examples as presented in Section Qualitative Decoding Results for modality-agnostic decoders. The results show that modality-agnostic decoders are as good as modality-specific decoders when evaluated in a within-modality decoding setup ( Appendix 3—figure 1 and Appendix 3—figure 3 ) but substantially better than modality-specific decoders when evaluated in a cross-decoding setup ( Appendix 3—figures 2 and 4 ). Appendix 3—figure 1. Decoding examples for image decoding using a modality-specific decoder trained on images (within-modality decoding). Open in a new tab The first column shows the image the subject was seeing and the five following columns show the candidate stimuli with highest similarity to the predicted features, in descending order. We display both the image and the caption of the candidate stimuli because the decoder is based on multimodal features that are extracted from both modalities. All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 3—figure 2. Decoding examples for caption decoding using a modality-specific decoder trained on images (cross-modality decoding). Open in a new tab For details, see caption of Figure 1 All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 3—figure 3. Decoding examples for caption decoding using a modality-specific decoder trained on captions (within-modality decoding). Open in a new tab For details, see caption of Appendix 3—figure 1 All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 3—figure 4. Decoding examples for image decoding using a modality-specific decoder trained on captions (cross-modality decoding). Open in a new tab For details, see caption of Appendix 3—figure 1 All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 4 Candidates for modality-invariant regions Appendix 4—table 1 shows candidates for modality-invariant regions that were identified by previous work. We only considered fMRI experiments involving multiple stimulus modalities. Many other studies relied on unimodal stimuli and semantic tasks to identify modality-invariant regions; these are, however, less directly comparable to our setup. Appendix 4—table 1. Candidates for modality-invariant regions, as identified by previous work. All these regions were also found in our analysis, except for the two regions marked with an asterisk (*). Hemi Region Studies that identified the region as modality-invariant Left hemisphere Superior occipital gyrus* Vandenberghe et al., 1996 ; Shinkareva et al., 2011 Middle occipital gyrus Shinkareva et al., 2011 Inferior occipital gyrus Shinkareva et al., 2011 ; Simanova et al., 2014 Superior temporal gyrus Shinkareva et al., 2011 Superior temporal sulcus Man et al., 2012 Middle temporal gyrus Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Devereux et al., 2013 ; Simanova et al., 2014 ; Handjaras et al., 2016 Inferior temporal gyrus Vandenberghe et al., 1996 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 ; Handjaras et al., 2016 Fusiform gyrus Vandenberghe et al., 1996 ; Moore and Price, 1999 ; Bright et al., 2004 ; Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Simanova et al., 2014 Hippocampus Vandenberghe et al., 1996 Parahippocampus Bright et al., 2004 ; Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 Perirhinal cortex Bright et al., 2004 ; Fairhall and Caramazza, 2013 Superior parietal cortex Shinkareva et al., 2011 Inferior parietal cortex Shinkareva et al., 2011 ; Handjaras et al., 2016 Cingulate gyrus Moore and Price, 1999 ; Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 Cuneus Shinkareva et al., 2011 Precuneus Shinkareva et al., 2011 ; Handjaras et al., 2016 ; Fairhall and Caramazza, 2013 ; Popham et al., 2021 Supramarginal gyrus Shinkareva et al., 2011 ; Handjaras et al., 2016 Angular gyrus Shinkareva et al., 2011 ; Fairhall and Caramazza, 2013 ; Devereux et al., 2013 ; Simanova et al., 2014 ; Handjaras et al., 2016 ; Popham et al., 2021 Intraparietal sulcus Shinkareva et al., 2011 ; Devereux et al., 2013 Temporoparietal junction Vandenberghe et al., 1996 ; Handjaras et al., 2016 Superior frontal gyrus Fairhall and Caramazza, 2013 Middle frontal gyrus Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 Inferior frontal gyrus Vandenberghe et al., 1996 ; Bright et al., 2004 ; Simanova et al., 2014 ; Liuzzi et al., 2017 ; Handjaras et al., 2016 Precentral gyrus Shinkareva et al., 2011 Postcentral gyrus Shinkareva et al., 2011 Paracentral lobule Shinkareva et al., 2011 Supplementary motor area Shinkareva et al., 2011 Right hemisphere Fusiform gyrus Shinkareva et al., 2011 ; Simanova et al., 2014 Superior temporal sulcus Man et al., 2012 Middle temporal gyrus Handjaras et al., 2016 Inferior temporal gyrus Handjaras et al., 2016 Angular gyrus Handjaras et al., 2016 ; Popham et al., 2021 Superior parietal cortex Shinkareva et al., 2011 Precuneus Shinkareva et al., 2011 ; Popham et al., 2021 ; Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 Cingulate gyrus Moore and Price, 1999 ; Fairhall and Caramazza, 2013 ; Handjaras et al., 2016 ; Jung et al., 2018 Parahippocampus Handjaras et al., 2016 Inferior parietal cortex Handjaras et al., 2016 Paracentral lobule Shinkareva et al., 2011 Middle frontal gyrus Simanova et al., 2014 ; Jung et al., 2018 Superior frontal gyrus* Jung et al., 2018 Inferior frontal gyrus Moore and Price, 1999 ; Simanova et al., 2014 Open in a new tab Appendix 5 Searchlight size In order to optimize the size of the searchlight for this analysis, we first ran a searchlight analysis with a fixed radius of 10 mm using a modality-agnostic decoder on the data from the first subject (sub-01). Due to the shape of the cortex, this leads of searchlights that contain varying numbers of vertices (on average: 897.4; max: 1580; min: 399). By observing the decoding scores as a function of the number of vertices, we find that performance peaks at 750 vertices ( Appendix 5—table 1 ). The final searchlight analysis was, therefore, performed with a searchlight of a fixed number of 750 vertices. The average radius with this number of vertices was 9.41 mm (max: 13.65 mm). Appendix 5—table 1. Average decoding scores (for images and captions) by number of vertices. Scores were calculated based on the results of a searchlight analysis with a radius of 10 mm. The accuracy values were grouped into bins based on the number of vertices that the searchlight was based on. # Vertices Pairwise accuracy (mean) 250 51.81% 500 52.35% 750 55.96% 1000 55.49% 1250 54.60% 1500 52.96% Open in a new tab Appendix 6 Per-subject results Results for individual subjects can be found in Appendix 6—figure 1 . Among all subjects, we found similar converging results for decoding accuracies when comparing models, feature modalities, and modality-agnostic with modality-specific decoders. Appendix 6—figure 1. Pairwise accuracy per subject. Open in a new tab For details, refer to Figure 5 Appendix 7 Qualitative imagery decoding results The following Appendix 7—figures 1 – 6 present qualitative decoding results for the imagery conditions using a modality-agnostic decoder. We present the results separately for each subject as each subject chose an individual set of three stimuli to perform mental imagery on. In each plot, the leftmost column shows the caption that was used as initial instruction as well as the subject’s sketch that they drew at the end of the experiment. We used the same large candidate set of 41 K stimuli as for the qualitative decoding results for the other conditions (see also Section Qualitative Decoding Results). Overall, we found that the decoding quality for imagery stimuli lags behind that for trials with perceived stimuli. This was expected and confirms the quantitative results reported in Section Imagery Decoding. Still, in several cases, some of the concepts are decoded correctly (e.g. a woman in the first row of Appendix 7—figure 1 ; winter sports in the second row of Appendix 7—figure 3 ; laptops/screens and multiple people in the third row of Appendix 7—figure 3 ). Appendix 7—figure 1. Imagery decoding for Subject 1 using a modality-agnostic decoder. Open in a new tab The first column shows the caption that was used to stimulate imagery and on top of it the sketch of their mental image that the subjects drew at the end of the experiment. The five following columns show the candidate stimuli with highest similarity to the predicted features, in descending order. We display both the image and the caption of the candidate stimuli because the decoder is based on multimodal features that are extracted from both modalities. All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 7—figure 2. Imagery decoding for Subject 2 using a modality-agnostic decoder. Open in a new tab For further details, refer to the caption of Appendix 7—figure 1 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 7—figure 3. Imagery decoding for Subject 3 using a modality-agnostic decoder. Open in a new tab For further details, refer to the caption of Appendix 7—figure 1 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 7—figure 4. Imagery decoding for Subject 4 using a modality-agnostic decoder. Open in a new tab For further details, refer to the caption of Appendix 7—figure 1 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 7—figure 5. Imagery decoding for Subject 5 using a modality-agnostic decoder. Open in a new tab For further details, refer to the caption of Appendix 7—figure 1 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). Appendix 7—figure 6. Imagery decoding for Subject 6 using a modality-agnostic decoder. Open in a new tab For further details, refer to the caption of Appendix 7—figure 1 . All images were taken from the CoCo dataset ( Lin et al., 2014 ). Funding Statement The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. Contributor Information Mitja Nikolaus, Email: mitja.nikolaus@posteo.de. Yanchao Bi, Peking University, China. Yanchao Bi, Peking University, China. Funding Information This paper was supported by the following grants: European Research Council

Advanced grant GLoW (101096017) to Rufin VanRullen. Agence Nationale de la Recherche

AI-REPS (ANR-18-CE37-0007-01) to Nicholas Asher, Leila Reddy, Rufin VanRullen. Agence Nationale de la Recherche

ANITI (ANR-19-PI3A-0004) to Nicholas Asher, Leila Reddy, Rufin VanRullen. Additional information Competing interests No competing interests declared. Author contributions Conceptualization, Resources, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing. Conceptualization, Resources, Data curation, Software, Formal analysis, Validation, Investigation, Methodology, Project administration. Resources, Data curation. Conceptualization, Funding acquisition, Methodology, Project administration. Conceptualization, Resources, Data curation, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Methodology, Project administration. Conceptualization, Data curation, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Methodology, Writing – original draft, Project administration, Writing – review and editing. Ethics Six subjects participated in the experiment after providing informed consent. The study was performed in accordance with French national ethical regulations (Comité de Protection des Personnes, ID 2019-A01920-57). Additional files MDAR checklist elife-107933-mdarchecklist1.pdf (8.4MB, pdf) Data availability All fMRI data is available at https://openneuro.org/datasets/ds007272 . Code for preprocessing and analyses is available at https://github.com/mitjanikolaus/multimodal_decoding (copy archived at Nikolaus, 2026 ). The following dataset was generated: Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. 2025. SemReps-8K. OpenNeuro. References Albers AM, Kok P, Toni I, Dijkerman HC, de Lange FP. Shared representations for working memory and mental imagery in early visual cortex. Current Biology. 2013;23:1427–1431. doi: 10.1016/j.cub.2013.05.065. [ DOI ] [ PubMed ] [ Google Scholar ] Aliko S, Huang J, Gheorghiu F, Meliss S, Skipper JI. A naturalistic neuroimaging database for understanding the brain using ecological stimuli. Scientific Data. 2020;7:347. doi: 10.1038/s41597-020-00680-2. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Allen EJ, St-Yves G, Wu Y, Breedlove JL, Prince JS, Dowdle LT, Nau M, Caron B, Pestilli F, Charest I, Hutchinson JB, Naselaris T, Kay K. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience. 2022;25:116–126. doi: 10.1038/s41593-021-00962-x. [ DOI ] [ PubMed ] [ Google Scholar ] Andrews M, Frank S, Vigliocco G. Reconciling embodied and distributional accounts of meaning in language. Topics in Cognitive Science. 2014;6:359–370. doi: 10.1111/tops.12096. [ DOI ] [ PubMed ] [ Google Scholar ] Ashburner J, Barnes G, Chen CC, Daunizeau J, Flandin G, Friston K, Kiebel S, Kilner J, Litvak V, Moran R. SPM12. Wellcome Trust Centre for Neuroimaging; 2014. [ Google Scholar ] Baars BJ. A Cognitive Theory of Consciousness. Cambridge University Press; 1993. [ Google Scholar ] Baars BJ. In: In Progress in Brain Research. Baars BJ, editor. Elsevier; 2005. Global workspace theory of consciousness: toward a cognitive neuroscience of human experience; pp. 45–53. [ DOI ] [ PubMed ] [ Google Scholar ] Barsalou LW. On staying grounded and avoiding quixotic dead ends. Psychonomic Bulletin & Review. 2016;23:1122–1142. doi: 10.3758/s13423-016-1028-3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Bartolomeo P. The relationship between visual perception and visual mental imagery: a reappraisal of the neuropsychological evidence. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2002;38:357–378. doi: 10.1016/s0010-9452(08)70665-8. [ DOI ] [ PubMed ] [ Google Scholar ] Beliy R, Gaziv G, Hoogi A, Strappini F, Golan T, Irani M. From voxels to pixels and back: self-supervision in natural-image reconstruction from fmri. Proceedings of the 33rd International Conference on Neural Information Processing Systems.2019. [ Google Scholar ] Benchetrit Y, Banville H, King JR. Brain decoding: toward real-time reconstruction of visual perception. In The Twelfth International Conference on Learning Representations.2024. [ Google Scholar ] Binder JR, Desai RH, Graves WW, Conant LL. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex. 2009;19:2767–2796. doi: 10.1093/cercor/bhp055. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Binder JR. In defense of abstract conceptual representations. Psychonomic Bulletin & Review. 2016;23:1096–1108. doi: 10.3758/s13423-015-0909-1. [ DOI ] [ PubMed ] [ Google Scholar ] Boccia M, Sulpizio V, Teghil A, Palermo L, Piccardi L, Galati G, Guariglia C. The dynamic contribution of the high-level visual cortex to imagery and perception. Human Brain Mapping. 2019;40:2449–2463. doi: 10.1002/hbm.24535. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Boyle JA, Pinsard B, Boukhdhir A, Belleville S, Brambatti S, Chen J, Cohen-Adad J, Cyr A, Fuente Rainville P, Bellec P. The Courtois project on neuronal modelling-first data release. In 26th annual meeting of the organization for human brain mapping.2020. [ Google Scholar ] Brennan JR, Hale JT. Hierarchical structure guides rapid linguistic predictions during naturalistic listening. PLOS ONE. 2019;14:e0207741. doi: 10.1371/journal.pone.0207741. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Brennan JR. Language and the Brain: A Slim Guide to Neurolinguistics. Oxford University Press; 2022. [ Google Scholar ] Bright P, Moss H, Tyler LK. Unitary vs multiple semantics: PET studies of word and picture processing. Brain and Language. 2004;89:417–432. doi: 10.1016/j.bandl.2004.01.010. [ DOI ] [ PubMed ] [ Google Scholar ] Carota F, Nili H, Pulvermüller F, Kriegeskorte N. Distinct fronto-temporal substrates of distributional and taxonomic similarity among words: evidence from RSA of BOLD signals. NeuroImage. 2021;224:117408. doi: 10.1016/j.neuroimage.2020.117408. [ DOI ] [ PubMed ] [ Google Scholar ] Chang N, Pyles JA, Marcus A, Gupta A, Tarr MJ, Aminoff EM. BOLD5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data. 2019;6:49. doi: 10.1038/s41597-019-0052-3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Cox DD, Savoy RL. Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage. 2003;19:261–270. doi: 10.1016/s1053-8119(03)00049-1. [ DOI ] [ PubMed ] [ Google Scholar ] Damasio AR. The brain binds entities and events by multiregional activation from convergence zones. Neural Computation. 1989;1:123–132. doi: 10.1162/neco.1989.1.1.123. [ DOI ] [ Google Scholar ] Damasio H, Tranel D, Grabowski T, Adolphs R, Damasio A. Neural systems behind word and concept retrieval. Cognition. 2004;92:179–229. doi: 10.1016/j.cognition.2002.07.001. [ DOI ] [ PubMed ] [ Google Scholar ] Desikan RS, Ségonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany RJ. An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. NeuroImage. 2006;31:968–980. doi: 10.1016/j.neuroimage.2006.01.021. [ DOI ] [ PubMed ] [ Google Scholar ] Devereux BJ, Clarke A, Marouchos A, Tyler LK. Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects. The Journal of Neuroscience. 2013;33:18906–18916. doi: 10.1523/JNEUROSCI.3809-13.2013. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Devlin JT, Russell RP, Davis MH, Price CJ, Wilson J, Moss HE, Matthews PM, Tyler LK. Susceptibility-induced loss of signal: comparing PET and fMRI on a semantic task. NeuroImage. 2000;11:589–600. doi: 10.1006/nimg.2000.0595. [ DOI ] [ PubMed ] [ Google Scholar ] Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2019. pp. 4171–4186. [ Google Scholar ] Dijkstra N, Bosch SE, van Gerven MAJ. Shared neural mechanisms of visual perception and imagery. Trends in Cognitive Sciences. 2019;23:423–434. doi: 10.1016/j.tics.2019.02.004. [ DOI ] [ PubMed ] [ Google Scholar ] Dirani J, Pylkkänen L. MEG Evidence that modality-independent conceptual representations contain semantic and visual features. The Journal of Neuroscience. 2024;44:e0326242024. doi: 10.1523/JNEUROSCI.0326-24.2024. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16x16 Words: transformers for image recognition at scale. In International Conference on Learning Representations.2020. [ Google Scholar ] Embleton KV, Haroon HA, Morris DM, Ralph MAL, Parker GJM. Distortion correction for diffusion-weighted MRI tractography and fMRI in the temporal lobes. Human Brain Mapping. 2010;31:1570–1587. doi: 10.1002/hbm.20959. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Fairhall SL, Caramazza A. Brain regions that represent amodal conceptual knowledge. The Journal of Neuroscience. 2013;33:10552–10558. doi: 10.1523/JNEUROSCI.0051-13.2013. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Farah MJ. The neurological basis of mental imagery: a componential analysis. Cognition. 1984;18:245–272. doi: 10.1016/0010-0277(84)90026-x. [ DOI ] [ PubMed ] [ Google Scholar ] Fedorenko E, Hsieh PJ, Nieto-Castañón A, Whitfield-Gabrieli S, Kanwisher N. New method for fMRI investigations of language: defining ROIs functionally in individual subjects. Journal of Neurophysiology. 2010;104:1177–1194. doi: 10.1152/jn.00032.2010. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Fedorenko E, Behr MK, Kanwisher N. Functional specificity for high-level linguistic processing in the human brain. PNAS. 2011;108:16428–16433. doi: 10.1073/pnas.1112937108. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Felleman DJ, Van Essen DC. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex. 1991;1:1–47. doi: 10.1093/cercor/1.1.1-a. [ DOI ] [ PubMed ] [ Google Scholar ] Fernandino L, Binder JR, Desai RH, Pendl SL, Humphries CJ, Gross WL, Conant LL, Seidenberg MS. Concept representation reflects multimodal abstraction: a framework for embodied semantics. Cerebral Cortex. 2016;26:2018–2034. doi: 10.1093/cercor/bhv020. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Fernandino L, Tong JQ, Conant LL, Humphries CJ, Binder JR. Decoding the information structure underlying the neural representation of concepts. PNAS. 2022;119:1119. doi: 10.1073/pnas.2108091119. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Fischl B. FreeSurfer. NeuroImage. 2012;62:774–781. doi: 10.1016/j.neuroimage.2012.01.021. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Friederici AD. Language in Our Brain: The Origins of a Uniquely Human Capacity. The MIT Press; 2017. [ DOI ] [ Google Scholar ] Frisby SL, Halai AD, Cox CR, Lambon Ralph MA, Rogers TT. Decoding semantic representations in mind and brain. Trends in Cognitive Sciences. 2023;27:258–281. doi: 10.1016/j.tics.2022.12.006. [ DOI ] [ PubMed ] [ Google Scholar ] Gainotti G. What the locus of brain lesion tells us about the nature of the cognitive defect underlying category-specific disorders: a review. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2000;36:539–559. doi: 10.1016/s0010-9452(08)70537-9. [ DOI ] [ PubMed ] [ Google Scholar ] Girdhar R, El-Nouby A, Liu Z, Singh M, Alwala KV, Joulin A, Misra I. ImageBind one embedding space to bind them all. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023. [ DOI ] [ Google Scholar ] Gorgolewski K, Burns CD, Madison C, Clark D, Halchenko YO, Waskom ML, Ghosh SS. Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Frontiers in Neuroinformatics. 2011;5:13. doi: 10.3389/fninf.2011.00013. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Grill-Spector K, Malach R. The human visual cortex. Annual Review of Neuroscience. 2004;27:649–677. doi: 10.1146/annurev.neuro.27.070203.144220. [ DOI ] [ PubMed ] [ Google Scholar ] Hajhajate D, Kaufmann BC, Liu J, Siuda-Krzywicka K, Bartolomeo P. The connectional anatomy of visual mental imagery: evidence from a patient with left occipito-temporal damage. Brain Structure & Function. 2022;227:3075–3083. doi: 10.1007/s00429-022-02505-x. [ DOI ] [ PubMed ] [ Google Scholar ] Handjaras G, Ricciardi E, Leo A, Lenci A, Cecchetti L, Cosottini M, Marotta G, Pietrini P. How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge. NeuroImage. 2016;135:232–242. doi: 10.1016/j.neuroimage.2016.04.063. [ DOI ] [ PubMed ] [ Google Scholar ] Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001;293:2425–2430. doi: 10.1126/science.1063736. [ DOI ] [ PubMed ] [ Google Scholar ] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. [ DOI ] [ Google Scholar ] Huth AG, Nishimoto S, Vu AT, Gallant JL. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron. 2012;76:1210–1224. doi: 10.1016/j.neuron.2012.10.014. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Jefferies E, Patterson K, Jones RW, Lambon Ralph MA. Comprehension of concrete and abstract words in semantic dementia. Neuropsychology. 2009;23:492–499. doi: 10.1037/a0015452. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Jenkinson M, Beckmann CF, Behrens TEJ, Woolrich MW, Smith SM. FSL. NeuroImage. 2012;62:782–790. doi: 10.1016/j.neuroimage.2011.09.015. [ DOI ] [ PubMed ] [ Google Scholar ] Jiang AQ, Sablayrolles A, Mensch A, Bamford C, Chaplot DS, Casas D, Bressand F, Lengyel G, Lample G, Saulnier L, Lavaud LR, Lachaux MA, Stock P, Scao TL, Lavril T, Wang T, Lacroix T, Sayed WE. Mistral7B. arXiv. 2023 doi: 10.48550/arXiv.2310.06825. [ DOI ] Johnson MR, Johnson MK. Decoding individual natural scene representations during perception and imagery. Frontiers in Human Neuroscience. 2014;8:59. doi: 10.3389/fnhum.2014.00059. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Jung Y, Larsen B, Walther DB. Modality-independent coding of scene categories in prefrontal cortex. The Journal of Neuroscience. 2018;38:5969–5981. doi: 10.1523/JNEUROSCI.0272-18.2018. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kanwisher N. Functional specificity in the human brain: A window into the functional architecture of the mind. PNAS. 2010;107:11163–11170. doi: 10.1073/pnas.1005062107. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kay KN, Naselaris T, Prenger RJ, Gallant JL. Identifying natural images from human brain activity. Nature. 2008;452:352–355. doi: 10.1038/nature06713. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kiefer M, Pulvermüller F. Conceptual representations in mind and brain: theoretical developments, current evidence and future directions. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2012;48:805–825. doi: 10.1016/j.cortex.2011.04.006. [ DOI ] [ PubMed ] [ Google Scholar ] Kim W, Son B, Kim I. ViLT: Vision-and-Language Transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning; 2021. pp. 5583–5594. [ Google Scholar ] Kosslyn SM, Pascual-Leone A, Felician O, Camposano S, Keenan JP, Thompson WL, Ganis G, Sukel KE, Alpert NM. The role of area 17 in visual imagery: convergent evidence from PET and rTMS. Science. 1999;284:167–170. doi: 10.1126/science.284.5411.167. [ DOI ] [ PubMed ] [ Google Scholar ] Kosslyn SM, Thompson WL. When is early visual cortex activated during visual mental imagery? Psychological Bulletin. 2003;129:723–746. doi: 10.1037/0033-2909.129.5.723. [ DOI ] [ PubMed ] [ Google Scholar ] Krasnowska-Kieraś K, Wróblewska A. Empirical linguistic study of sentence embeddings. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019. pp. 5729–5739. [ DOI ] [ Google Scholar ] Kriegeskorte N, Mur M, Bandettini PA. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience. 2008;2:4. doi: 10.3389/neuro.06.004.2008. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Lambon Ralph MA, Lowe C, Rogers TT. Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. Brain. 2006;130:1127–1137. doi: 10.1093/brain/awm025. [ DOI ] [ PubMed ] [ Google Scholar ] Lee SH, Kravitz DJ, Baker CI. Disentangling visual imagery and perception of real-world objects. NeuroImage. 2012;59:4064–4073. doi: 10.1016/j.neuroimage.2011.10.055. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Li LH, Yatskar M, Yin D, Hsieh CJ, Chang KW. VisualBERT: A simple and performant baseline for vision and language. arXiv. 2019 doi: 10.48550/arXiv.1908.03557. [ DOI ] Li J, Li D, Savarese S, Hoi S. BLIP-2: Bootstrapping Language-Image Pre-training with frozen image encoders and large language models. In International Conference on Machine Learning.2023. [ Google Scholar ] Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL. In: In Computer Vision – ECCV. Fleet D, Pajdla T, Schiele B, Tuytelaars T, editors. Springer; 2014. Microsoft COCO: common objects in context; pp. 740–755. [ DOI ] [ Google Scholar ] Lin S, Singh AK, Sprague T. Mind reader: reconstructing complex images from brain activities. Advances in Neural Information Processing Systems 35; 2022. pp. 29624–29636. [ DOI ] [ Google Scholar ] Liu J, Spagna A, Bartolomeo P. Hemispheric asymmetries in visual mental imagery. Brain Structure & Function. 2022;227:697–708. doi: 10.1007/s00429-021-02277-w. [ DOI ] [ PubMed ] [ Google Scholar ] Liu J, Zhan M, Hajhajate D, Spagna A, Dehaene S, Cohen L, Bartolomeo P. Visual mental imagery in typical imagers and in aphantasia: A millimeter-scale 7-T fMRI study. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2025;185:113–132. doi: 10.1016/j.cortex.2025.01.013. [ DOI ] [ PubMed ] [ Google Scholar ] Liuzzi AG, Bruffaerts R, Peeters R, Adamczuk K, Keuleers E, De Deyne S, Storms G, Dupont P, Vandenberghe R. Cross-modal representation of spoken and written word meaning in left pars triangularis. NeuroImage. 2017;150:292–307. doi: 10.1016/j.neuroimage.2017.02.032. [ DOI ] [ PubMed ] [ Google Scholar ] Liuzzi AG, Aglinskas A, Fairhall SL. General and feature-based semantic representations in the semantic network. Scientific Reports. 2020;10:8931. doi: 10.1038/s41598-020-65906-0. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Man K, Kaplan JT, Damasio A, Meyer K. Sight and sound converge to form modality-invariant representations in temporoparietal cortex. The Journal of Neuroscience. 2012;32:16629–16636. doi: 10.1523/JNEUROSCI.2342-12.2012. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Martin A. In: The Cognitive Neurosciences. Gazzaniga MS, editor. The MIT Press; 2009. Circuits in mind: the neural foundations for object concepts; pp. 1031–1045. [ DOI ] [ Google Scholar ] Martin A. GRAPES—Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain. Psychonomic Bulletin & Review. 2016;23:979–990. doi: 10.3758/s13423-015-0842-3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Martin CB, Douglas D, Newsome RN, Man LL, Barense MD. Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream. eLife. 2018;7:31873. doi: 10.7554/eLife.31873. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Meschke E, Gallant J. Mapping multimodal conceptual representations within the lexical-semantic brain system. In CCN.2024. [ Google Scholar ] Meyer K, Damasio A. Convergence and divergence in a neural architecture for recognition and memory. Trends in Neurosciences. 2009;32:376–382. doi: 10.1016/j.tins.2009.04.002. [ DOI ] [ PubMed ] [ Google Scholar ] Moore CJ, Price CJ. Three distinct ventral occipitotemporal regions for reading and object naming. NeuroImage. 1999;10:181–192. doi: 10.1006/nimg.1999.0450. [ DOI ] [ PubMed ] [ Google Scholar ] Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL. Bayesian reconstruction of natural images from human brain activity. Neuron. 2009;63:902–915. doi: 10.1016/j.neuron.2009.09.006. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Naselaris T, Kay KN, Nishimoto S, Gallant JL. Encoding and decoding in fMRI. NeuroImage. 2011;56:400–410. doi: 10.1016/j.neuroimage.2010.07.073. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Naselaris T, Olman CA, Stansbury DE, Ugurbil K, Gallant JL. A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes. NeuroImage. 2015;105:215–228. doi: 10.1016/j.neuroimage.2014.10.018. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Nastase SA, Liu YF, Hillman H, Zadbood A, Hasenfratz L, Keshavarzian N, Chen J, Honey CJ, Yeshurun Y, Regev M, Nguyen M, Chang CHC, Baldassano C, Lositsky O, Simony E, Chow MA, Leong YC, Brooks PP, Micciche E, Choe G, Goldstein A, Vanderwal T, Halchenko YO, Norman KA, Hasson U. The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Scientific Data. 2021;8:250. doi: 10.1038/s41597-021-01033-3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Nikolaus M. Multimodal_decoding. swh:1:rev:57f7a1c4a5da1c96887ed79f0507cbd286c727fdSoftware Heritage. 2026 https://archive.softwareheritage.org/swh:1:dir:358f81a6a99cdcdf3ba0129b4e5627b7170b3273;origin=https://github.com/mitjanikolaus/multimodal_decoding;visit=swh:1:snp:840f7a3b5e83b249f545ad61c124df2dab366bac;anchor=swh:1:rev:57f7a1c4a5da1c96887ed79f0507cbd286c727fd Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, Gallant JL. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology. 2011;21:1641–1646. doi: 10.1016/j.cub.2011.08.031. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] O’Craven KM, Kanwisher N. Mental imagery of faces and places activates corresponding stiimulus-specific brain regions. Journal of Cognitive Neuroscience. 2000;12:1013–1023. doi: 10.1162/08989290051137549. [ DOI ] [ PubMed ] [ Google Scholar ] Oquab M, Darcet T, Moutakanni T, Vo H, Szafraniec M, Khalidov V, Fernandez P, Haziza D, Massa F, El-Nouby A, Assran M, Ballas N, Galuba W, Howes R, Huang PY, Li SW, Misra I, Rabbat M, Sharma V, Synnaeve G. DINOv2: Learning robust visual features without supervision. arXiv. 2023 doi: 10.48550/arXiv.2304.07193. [ DOI ] Ozcelik F, VanRullen R. Natural scene reconstruction from fMRI signals using generative latent diffusion. Scientific Reports. 2023;13:15666. doi: 10.1038/s41598-023-42891-8. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Patterson K, Lambon Ralph MA. In: In Neurobiology of Language. Patterson K, editor. Elsevier; 2016. The hub-and-spoke hypothesis of semantic memory; pp. 765–775. [ DOI ] [ Google Scholar ] Pearson J. The human imagination: the cognitive neuroscience of visual mental imagery. Nature Reviews. Neuroscience. 2019;20:624–634. doi: 10.1038/s41583-019-0202-9. [ DOI ] [ PubMed ] [ Google Scholar ] Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V. Scikit-learn: machine learning in python. The Journal of Machine Learning Research. 2011;12:2825–2830. [ Google Scholar ] Pereira F, Lou B, Pritchett B, Ritter S, Gershman SJ, Kanwisher N, Botvinick M, Fedorenko E. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications. 2018;9:963. doi: 10.1038/s41467-018-03068-4. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Pobric G, Jefferies E, Lambon Ralph MA. Category-specific versus category-general semantic impairment induced by transcranial magnetic stimulation. Current Biology. 2010;20:964–968. doi: 10.1016/j.cub.2010.03.070. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Popham SF, Huth AG, Bilenko NY, Deniz F, Gao JS, Nunez-Elizalde AO, Gallant JL. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience. 2021;24:1628–1636. doi: 10.1038/s41593-021-00921-6. [ DOI ] [ PubMed ] [ Google Scholar ] Power JD, Barnes KA, Snyder AZ, Schlaggar BL, Petersen SE. Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion. NeuroImage. 2012;59:2142–2154. doi: 10.1016/j.neuroimage.2011.10.018. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog. 2019;1:1–24. [ Google Scholar ] Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I. Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning.2021. [ Google Scholar ] Ralph MAL, Jefferies E, Patterson K, Rogers TT. The neural and computational bases of semantic cognition. Nature Reviews. Neuroscience. 2017;18:42–55. doi: 10.1038/nrn.2016.150. [ DOI ] [ PubMed ] [ Google Scholar ] Reddy L, Tsuchiya N, Serre T. Reading the mind’s eye: decoding category information during mental imagery. NeuroImage. 2010;50:818–825. doi: 10.1016/j.neuroimage.2009.11.084. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using siamese BERT-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); 2019. [ DOI ] [ Google Scholar ] Rogers TT, Lambon Ralph MA, Garrard P, Bozeat S, McClelland JL, Hodges JR, Patterson K. Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychological Review. 2004;111:205–235. doi: 10.1037/0033-295X.111.1.205. [ DOI ] [ PubMed ] [ Google Scholar ] Sanchez G, Hartmann T, Fuscà M, Demarchi G, Weisz N. Decoding across sensory modalities reveals common supramodal signatures of conscious perception. PNAS. 2020;117:7437–7446. doi: 10.1073/pnas.1912584117. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Schoffelen JM, Oostenveld R, Lam NHL, Uddén J, Hultén A, Hagoort P. A 204-subject multimodal neuroimaging dataset to study language processing. Scientific Data. 2019;6:17. doi: 10.1038/s41597-019-0020-y. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Sereno MI, Dale AM, Reppas JB, Kwong KK, Belliveau JW, Brady TJ, Rosen BR, Tootell RBH. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science. 1995;268:889–893. doi: 10.1126/science.7754376. [ DOI ] [ PubMed ] [ Google Scholar ] Shen G, Horikawa T, Majima K, Kamitani Y. Deep image reconstruction from human brain activity. PLOS Computational Biology. 2019;15:e1006633. doi: 10.1371/journal.pcbi.1006633. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Shinkareva SV, Malave VL, Mason RA, Mitchell TM, Just MA. Commonality of neural representations of words and pictures. NeuroImage. 2011;54:2418–2425. doi: 10.1016/j.neuroimage.2010.10.042. [ DOI ] [ PubMed ] [ Google Scholar ] Simanova I, Hagoort P, Oostenveld R, van Gerven MAJ. Modality-independent decoding of semantic information from the human brain. Cerebral Cortex. 2014;24:426–434. doi: 10.1093/cercor/bhs324. [ DOI ] [ PubMed ] [ Google Scholar ] Simmons WK, Barsalou LW. The similarity-in-topography principle: reconciling theories of conceptual deficits. Cognitive Neuropsychology. 2003;20:451–486. doi: 10.1080/02643290342000032. [ DOI ] [ PubMed ] [ Google Scholar ] Singh A, Hu R, Goswami V, Couairon G, Galuba W, Rohrbach M, Kiela D. FLAVA: A Foundational Language And Vision Alignment Model. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022. [ DOI ] [ Google Scholar ] Smith SM, Nichols TE. Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference. NeuroImage. 2009;44:83–98. doi: 10.1016/j.neuroimage.2008.03.061. [ DOI ] [ PubMed ] [ Google Scholar ] Snowden J, Goulding PJ, Neary D. Semantic dementia: A form of circumscribed cerebral atrophy. Behavioural Neurology. 1989;2:167–182. doi: 10.1155/1989/124043. [ DOI ] [ Google Scholar ] Spagna A, Hajhajate D, Liu J, Bartolomeo P. Visual mental imagery engages the left fusiform gyrus, but not the early visual cortex: A meta-analysis of neuroimaging evidence. Neuroscience & Biobehavioral Reviews. 2021;122:201–217. doi: 10.1016/j.neubiorev.2020.12.029. [ DOI ] [ PubMed ] [ Google Scholar ] Steel A, Billings MM, Silson EH, Robertson CE. A network linking scene perception and spatial memory systems in posterior cerebral cortex. Nature Communications. 2021;12:2632. doi: 10.1038/s41467-021-22848-z. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Steiner A, Pinto AS, Tschannen M, Keysers D, Wang X, Bitton Y, Gritsenko A, Minderer M, Sherbondy A, Long S, Qin S, Ingle R, Bugliarello E, Kazemzadeh S, Mesnard T, Alabdulmohsin I, Beyer L, Zhai X. PaliGemma 2: A family of versatile VLMs for transfer. arXiv. 2024 doi: 10.48550/arXiv.2412.03555. [ DOI ] Stelzer J, Chen Y, Turner R. Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): random permutations and cluster size control. NeuroImage. 2013;65:69–82. doi: 10.1016/j.neuroimage.2012.09.063. [ DOI ] [ PubMed ] [ Google Scholar ] Stokes M, Thompson R, Cusack R, Duncan J. Top-down activation of shape-specific population codes in visual cortex during mental imagery. The Journal of Neuroscience. 2009;29:1565–1572. doi: 10.1523/JNEUROSCI.4657-08.2009. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023. [ DOI ] [ Google Scholar ] Tang J, Du M, Vo VA. Brain encoding models based on multimodal transformers can transfer across language and vision. In NeurIPS; 2023a. [ PMC free article ] [ PubMed ] [ Google Scholar ] Tang J, LeBel A, Jain S, Huth AG. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience. 2023b;26:858–866. doi: 10.1038/s41593-023-01304-9. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Tong J, Binder JR, Humphries C, Mazurchuk S, Conant LL, Fernandino L. A distributed network for multimodal experiential representation of concepts. The Journal of Neuroscience. 2022;42:7121–7130. doi: 10.1523/JNEUROSCI.1243-21.2022. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, Bikel D, Blecher L, Ferrer CC, Chen M, Cucurull G, Esiobu D, Fernandes J, Fu J, Fu W, Fuller B, Gao C, Goswami V, Goyal N, Hartshorn A, Hosseini S, Hou R, Inan H, Kardas M, Kerkez V, Khabsa M, Kloumann I, Korenev A, Koura PS, Lachaux MA, Lavril T, Lee J, Liskovich D, Lu Y, Mao Y, Martinet X, Mihaylov T, Mishra P, Molybog I, Nie Y, Poulton A, Reizenstein J, Rungta R, Saladi K. Llama 2: Open foundation and fine-tuned chat models. arXiv. 2023 doi: 10.48550/arXiv.2307.09288. [ DOI ] Tranel D, Damasio H, Damasio AR. A neural basis for the retrieval of conceptual knowledge. Neuropsychologia. 1997;35:1319–1327. doi: 10.1016/s0028-3932(97)00085-7. [ DOI ] [ PubMed ] [ Google Scholar ] Vandenberghe R, Price C, Wise R, Josephs O, Frackowiak RS. Functional anatomy of a common semantic system for words and pictures. Nature. 1996;383:254–256. doi: 10.1038/383254a0. [ DOI ] [ PubMed ] [ Google Scholar ] VanRullen R, Reddy L. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology. 2019;2:193. doi: 10.1038/s42003-019-0438-y. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Visconti di Oleggio Castello M, Chauhan V, Jiahui G, Gobbini MI. An fMRI dataset in response to “The Grand Budapest Hotel”, a socially-rich, naturalistic movie. Scientific Data. 2020;7:383. doi: 10.1038/s41597-020-00735-4. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Warrington EK. The selective impairment of semantic memory. The Quarterly Journal of Experimental Psychology. 1975;27:635–657. doi: 10.1080/14640747508400525. [ DOI ] [ PubMed ] [ Google Scholar ] Warrington EK, Shallice T. Category Specific Semantic Impairments. Brain: a Journal of Neurology; 1984. [ DOI ] [ PubMed ] [ Google Scholar ] Warrington EK, Mccarthy RA. Categories of Knowledge: Further Fractionations and an Attempted Integration. Brain: a Journal of Neurology; 1987. [ DOI ] [ PubMed ] [ Google Scholar ] Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing; 2020. [ DOI ] [ Google Scholar ] Xu X, Wu C, Rosenman S, Lal V, Che W, Duan N. Bridgetower: building bridges between encoders in vision-language representation learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37:10637–10647. doi: 10.1609/aaai.v37i9.26263. [ DOI ] [ Google Scholar ] Zhai X, Mustafa B, Kolesnikov A, Beyer L. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2023. [ DOI ] [ Google Scholar ] Zola-Morgan S. Localization of brain function: the legacy of Franz Joseph Gall (1758-1828) Annual Review of Neuroscience. 1995;18:359–383. doi: 10.1146/annurev.ne.18.030195.002043. [ DOI ] [ PubMed ] [ Google Scholar ] Zwaan RA. Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic Bulletin & Review. 2016;23:1028–1034. doi: 10.3758/s13423-015-0864-x. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] eLife. doi: 10.7554/eLife.107933.3.sa0 eLife Assessment Yanchao Bi Yanchao Bi 1 Peking University, China Reviewing Editor Find articles by Yanchao Bi 1 Author information Article notes Copyright and License information 1 Peking University, China Roles Yanchao Bi : Reviewing Editor Keywords: Convincing Keywords: Valuable PMC Copyright notice The study introduces a valuable dataset for investigating the relationship between vision and language in the brain. The authors provide convincing evidence that decoders trained on brain responses to both images and captions outperform those trained on responses to a single modality. The dataset and decoder results will be of interest to communities studying brain and machine decoding. eLife. doi: 10.7554/eLife.107933.3.sa1 Reviewer #2 (Public review): Anonymous Anonymous Reviewer Find articles by Anonymous Author information Copyright and License information Roles Anonymous : Reviewer PMC Copyright notice Summary: This work presents a modality-agnostic decoder trained on a large fMRI dataset (SemReps-8K), in which subjects viewed natural images and corresponding captions. The decoder predicts stimulus content from brain activity irrespective of the input modality and performs on par with-or even outperforms-modality-specific decoders. Its success depends more on the diversity of brain data (multimodal vs. unimodal) than on whether the feature-extraction models are visual, linguistic, or multimodal. Particularly, the decoder shows strong performance in decoding imagery content. These results suggest that the modality-agnostic decoder effectively leverages shared brain information across image and caption tasks. Strengths: (1) The modality-agnostic decoder compellingly leverages multimodal brain information, improving decoding accuracy-particularly for non-sensory input such as captions-showing high methodological and application value. (2) The dataset is a substantial and well-controlled contribution, with >8,000 image-caption trials per subject and careful matching of stimuli across modalities-an essential resource for testing theories about different representational modalities. Weakness: In the searchlight analysis aimed at identifying modality-invariant representations, although the combined use of four decoding conditions represents a relatively strict approach, the underlying logic remains unclear. The modality-agnostic decoder has demonstrated strong sensitivity in decoding brain activity, as shown earlier in the paper, whereas the cross-decoding with modality-specific decoders is inherently more conservative. If, as the authors note, the modality-agnostic decoder might have learned to leverage different features to project stimuli from different modalities, then taking the union of conditions would seem more appropriate. Conversely, if the goal is to obtain a more conservative result, why not focus solely on the cross-decoding conditions? The relationships among the four decoding conditions are not clearly delineated, and the contrasts between them might themselves yield valuable insights. As it stands, however, the logic of the current approach is not straightforward. eLife. doi: 10.7554/eLife.107933.3.sa2 Reviewer #3 (Public review): Anonymous Anonymous Reviewer Find articles by Anonymous Author information Copyright and License information Roles Anonymous : Reviewer PMC Copyright notice Summary: The authors recorded brain responses while participants viewed images and captions. The images and captions were taken from the COCO dataset, so each image has a corresponding caption and each caption has a corresponding image. This enabled the authors to extract features from either the presented stimulus or the corresponding stimulus in the other modality. The authors trained linear decoders to take brain responses and predict stimulus features. "Modality-specific" decoders were trained on brain responses to either images or captions while "modality-agnostic" decoders were trained on brain responses to both stimulus modalities. The decoders were evaluated on brain responses while the participants viewed and imagined new stimuli, and prediction performance was quantified using pairwise accuracy. The authors reported the following results: (1) Decoders trained on brain responses to both images and captions can predict new brain responses to either modality. (2) Decoders trained on brain responses to both images and captions outperform decoders trained on brain responses to a single modality. (3) Many cortical regions represent the same concepts in vision and language. (4) Decoders trained on brain responses to both images and captions can decode brain responses to imagined scenes. Strengths: This is an interesting study that addresses important questions about modality-agnostic representations. Previous work has shown that decoders trained on brain responses to one modality can be used to decode brain responses to another modality. The authors build on these findings by collecting a new multimodal dataset and training decoders on brain responses to both modalities. To my knowledge, SemReps-8K is the first dataset of brain responses to vision and language where each stimulus item has a corresponding stimulus item in the other modality. This means that brain responses to a stimulus item can be modeled using visual features of the image, linguistic features of the caption, or multimodal features derived from both the image and the caption. The authors also employed a multimodal one-back matching task which forces the participants to activate modality-agnostic representations. Overall, SemReps-8K is a valuable resource that will help researchers answer more questions about modality-agnostic representations. The analyses are also very comprehensive. The authors trained decoders on brain responses to images, captions, and both modalities, and they tested the decoders on brain responses to images, caption, and imagined scenes. They extracted stimulus features using a range of visual, linguistic, and multimodal models. The modeling framework appears rigorous and the results offer new insights into the relationship between vision, language, and imagery. In particular, the authors found that decoders trained on brain responses to both images and captions were more effective at decoding brain responses to imagined scenes than decoders trained on brain responses to either modality in isolation. The authors also found that imagined scenes can be decoded from a broad network of cortical regions. Weaknesses: The characterization of "modality-agnostic" and "modality-specific" decoders seems a bit contradictory. There are three major choices when fitting a decoder: the modality of the training stimuli, the modality of the testing stimuli, and the model used to extract stimulus features. However, the authors characterize their decoders based on only the first choice-"modality-specific" decoders were trained on brain responses to either images or captions while "modality-agnostic" decoders were trained on brain responses to both stimulus modalities. I think that this leads to some instances where the conclusions are inconsistent with the methods and results. First, the authors suggest that "modality-specific decoders are not explicitly encouraged to pick up on modality-agnostic features during training" (line 137) while "modality-agnostic decoders may be more likely to leverage representations that are modality-agnostic" (line 140). However, whether a decoder is required to learn modality-agnostic representations depends on both the training responses and the stimulus features. Consider the case where the stimuli are represented using linguistic features of the captions. When you train a "modality-specific" decoder on image responses, the decoder is forced to rely on modality-agnostic information that is shared between the image responses and the caption features. On the other hand, when you train a "modality-agnostic" decoder on both image responses and caption responses, the decoder has access to the modality-specific information that is shared by the caption responses and the caption features, so it is not explicitly required to learn modality-agnostic features. As a result, while the authors show that "modality-agnostic" decoders outperform "modality-specific" decoders in most conditions, I am not convinced that this is because they are forced to learn more modality-agnostic features. Second, the authors claim that "modality-specific decoders can be applied only in the modality that they were trained on" while "modality-agnostic decoders can be applied to decode stimuli from multiple modalities, even without knowing a priori the modality the stimulus was presented in" (line 47). While "modality-agnostic" decoders do outperform "modality-specific" decoders in the cross-modality conditions, it is important to note that "modality-specific" decoders still perform better than expected by chance (figure 5). It is also important to note that knowing about the input modality still improves decoding performance even for "modality-agnostic" decoders, since it determines the optimal feature space-it is better to decode brain responses to images using decoders trained on image features, and it is better to decode brain responses to captions using decoders trained on caption features. Comments on revised version: The revised version benefits from clearer claims and more precise terminology (i.e. classifying the decoders as "modality-agnostic" or "modality-specific" while classifying the representations as "modality-invariant" or "modality-dependent"). While the modality-agnostic decoders outperform the modality-specific decoders, I am still not convinced that this is because they are "explicitly trained to leverage the shared information in modality-invariant patterns of the brain activity". On one hand, the high-level feature spaces may each contain some amount of modality-invariant information, so even modality-specific decoders can capture some modality-invariant information. On the other hand, I do not see how training the modality-agnostic decoders on responses to both modalities necessitates that they learn modality-invariant representations beyond those that are learned by the modality-specific decoders. eLife. 2026 Apr 17;14:RP107933. doi: 10.7554/eLife.107933.3.sa3 Author response Mitja Nikolaus Mitja Nikolaus 1 Centre de recherche cerveau et cognition, Toulouse, France Author Find articles by Mitja Nikolaus 1 , Milad Mozafari Milad Mozafari 2 Torus AI, Toulouse, France Author Find articles by Milad Mozafari 2 , Isabelle Berry Isabelle Berry 3 Université de Toulouse, Toulouse, France Author Find articles by Isabelle Berry 3 , Nicholas Asher Nicholas Asher 4 Université de Toulouse, IRIT, Toulouse, France Author Find articles by Nicholas Asher 4 , Leila Reddy Leila Reddy 5 CerCo, CNRS, Toulouse, France Author Find articles by Leila Reddy 5 , Rufin VanRullen Rufin VanRullen 6 CNRS, UMR5549, Toulouse, France Author Find articles by Rufin VanRullen 6 Author information Article notes Copyright and License information 1 Centre de recherche cerveau et cognition, Toulouse, France 2 Torus AI, Toulouse, France 3 Université de Toulouse, Toulouse, France 4 Université de Toulouse, IRIT, Toulouse, France 5 CerCo, CNRS, Toulouse, France 6 CNRS, UMR5549, Toulouse, France Roles Mitja Nikolaus : Author Milad Mozafari : Author Isabelle Berry : Author Nicholas Asher : Author Leila Reddy : Author Rufin VanRullen : Author Collection date 2026. PMC Copyright notice The following is the authors’ response to the original reviews We would like to thank all reviewers for their constructive and in-depth reviews. Thanks to your feedback, we realized that the main objective of the paper was not presented clearly enough, and that our use of the same “modality-agnostic” terminology for both decoders and representations caused confusion. We addressed these two major points as outlined in the following. In the revised manuscript, we highlight that the main contribution of this paper is to introduce modality-agnostic decoders. Apart from introducing this new decoder type, we put forward their advantages in comparison to modality-specific decoders in terms of decoding performance and analyze the modality-invariant representations (cf. updated terminology in the following paragraph) that these decoders rely on. The dataset that these analyses are based on is released as part of this paper, in the spirit of open science (but this dataset is only a secondary contribution for our paper). Regarding the terminology, we clearly define modality-agnostic decoders as decoders that are trained on brain imaging data from subjects exposed to stimuli in multiple modalities. The decoder is not given any information on which modality a stimulus was presented in, and is therefore trained to operate in a modality-agnostic way. In contrast, modality-specific decoders are trained only on data from a single stimulus modality. These terms are explained in Figure 2. While these terms describe different ways of how decoders can be trained, there are also different ways to evaluate them afterwards (see also Figure 3); but obviously, this test-time evaluation does not change the nature of the decoder, i.e., there is no contradiction in applying a modality-specific decoder to brain data from a different modality. Further, we identify representations that are relevant for modality-agnostic decoders using the searchlight analysis. We realized that our choice of using the same “modality-agnostic” term to describe these brain representations created unnecessary debate and confusion. In order to not conflate the terminology, in the updated manuscript we call these representations modality-invariant (and the opposite modality-dependent). Our methodology does not allow us to distinguish whether certain representations merely share representational structure to a certain degree, or are truly representations that abstract away from any modality-dependent information. However, in order to be useful for modality-agnostic decoding, a significant degree of shared representational structure is sufficient, and it is this property of brain representations that we now define as “modality-invariant”. We updated the manuscript in line with this new terminology and focus: in particular, the first Related Work section on Modality-invariant brain representations, as well as the Introduction and Discussion. Public Reviews: Reviewer #1 (Public review): Summary: The authors introduce a densely-sampled dataset where 6 participants viewed images and sentence descriptions derived from the MS Coco database over the course of 10 scanning sessions. The authors further showcase how image and sentence decoders can be used to predict which images or descriptions were seen, using pairwise decoding across a set of 120 test images. The authors find decodable information widely distributed across the brain, with a left-lateralized focus. The results further showed that modality-agnostic models generally outperformed modality-specific models, and that data based on captions was not explained better by caption-based models but by modality-agnostic models. Finally, the authors decoded imagined scenes. Strengths: (1) The dataset presents a potentially very valuable resource for investigating visual and semantic representations and their interplay. (2) The introduction and discussion are very well written in the context of trying to understand the nature of multimodal representations and present a comprehensive and very useful review of the current literature on the topic. Weaknesses: (1) The paper is framed as presenting a dataset, yet most of it revolves around the presentation of findings in relation to what the authors call modality-agnostic representations, and in part around mental imagery. This makes it very difficult to assess the manuscript, whether the authors have achieved their aims, and whether the results support the conclusions. Thanks for this insightful remark. The dataset release is only a secondary contribution of our study; this was not clear enough in the previous version. We updated the manuscript to make the main objective of the paper more clear, as outlined in our general response to the reviews (see above). (2) While the authors have presented a potential use case for such a dataset, there is currently far too little detail regarding data quality metrics expected from the introduction of similar datasets, including the absence of head-motion estimates, quality of intersession alignment, or noise ceilings of all individuals. As already mentioned in the general response, the main focus of the paper is to introduce modality-agnostic decoders. The dataset is released in addition, this is why we did not focus on reporting extensive quality metrics in the original manuscript. To respond to your request, we updated the appendix of the manuscript to include a range of data quality metrics. The updated appendix includes head motion estimates in the form of realignment parameters and framewise displacement, as well as a metric to assess the quality of intersession alignment. More detailed descriptions can be found in Appendix 1 of the updated manuscript. Estimating noise ceilings based on repeated presentations of stimuli (as for example done in Allen et al. (2022)) requires multiple betas for each stimulus. All training stimuli were only presented once, so this could only be done for the test stimuli which were presented repeatedly. However, during our preprocessing procedure we directly calculated stimulus-specific betas based on data from all sessions using one single GLM, which means that we did not obtain separate betas for repeated presentations of the same stimulus. We will however share the raw data publicly, so that such noise ceilings can be calculated using an adapted preprocessing procedure if required. Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., Hutchinson, J. B., Naselaris, T., & Kay, K. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1), 116–126. https://doi.org/10.1038/s41593-021-00962-x (3) The exact methods and statistical analyses used are still opaque, making it hard for a reader to understand how the authors achieved their results. More detail in the manuscript would be helpful, specifically regarding the exact statistical procedures, what tests were performed across, or how data were pooled across participants. In the updated manuscript, we improved the level of detail for the descriptions of statistical analyses wherever possible (see also our response to your “Recommendations for the authors”, Point 6). Regarding data pooling across participants: Figure 8 shows averaged results across all subjects (as indicated in the caption) Regarding data pooling for the estimation of the significance threshold of the searchlight analysis for modality-invariant regions: We updated the manuscript to clarify that we performed a permutation test, combined with a bootstrapping procedure to estimate a group-level null distribution: “For each subject, we evaluated the decoders 100 times with shuffled labels to create per-subject chance-level results. Then, we randomly selected one of the 100 chance-level results for each of the 6 subjects and calculated group-level statistics (TFCE values) the exact same way as described in the preceding paragraph. We repeated this procedure 10,000 times resulting in 10,000 permuted group-level results.” Additionally, we indicated that the same permutation testing methods were applied to assess the significance threshold for the imagery decoding searchlight maps (Figure 10). (4) Many findings (e.g., Figure 6) are still qualitative but could be supported by quantitative measures. The Figures 6 and 7 are intentionally qualitative results to support the quantitative decoding results presented in Figures 4 and 5. (see also Reviewer 2 Comment 2) Figures 4 and 5 show pairwise decoding accuracy as a quantitative measure for evaluation of the decoders. This metric is the main metric we used to compare different decoder types and features. Based on the finding that modality-agnostic decoders using imagebind features achieve the best score on this metric, we performed the additional qualitative analysis presented in Figures 6 and 7. (Note that we expanded the candidate set for the qualitative analysis in order to have a larger and more diverse set of images.) (5) Results are significant in regions that typically lack responses to visual stimuli, indicating potential bias in the classifier. This is relevant for the interpretation of the findings. A classification approach less sensitive to outliers (e.g., 70-way classification) could avoid this issue. Given the extreme collinearity of the experimental design, regressors in close temporal proximity will be highly similar, which could lead to leakage effects. It is true that our searchlight analysis revealed significant activity in regions outside of the visual cortex. However, it is assumed that the processing of visual information does not stop at the border of the visual cortex. The integration of information such as the semantics of the image is progressively processed in other higher-level regions of the brain. Recent studies have shown that activity in large areas of the cortex (including many outside of the visual cortex) can be related to visual stimulation (Solomon et al. 2024; Raugel et al. 2025). Our work confirms this finding and we therefore do not see reason to believe that this is due to a bias in our decoders. Further, you are suggesting that we could replace our regression approach with a 70-way classification. However, this is difficult using our fMRI data as we do not see a straightforward way to assign the training and testing stimuli with class labels (the two datasets consist of non-overlapping sets of naturalistic images). To address your concerns regarding the collinearity of the experimental design and possible leakage effects, we trained and evaluated a decoder for one subject after running a “null-hypothesis” adapted preprocessing. More specifically, for all sessions, we shifted the functional data of all runs by one run (moving the data of the last run to the very front), but leaving the design matrices in place. Thereby, we destroyed the relationship of stimuli and brain activity but kept the original data and design with its collinearity (and possible biases). We preprocessed this adapted data for subject 1, and ran a whole-brain decoding using Imagebind features and verified that the decoding performance was at chance level: Pairwise accuracy (captions): 0.43 | Pairwise accuracy (images): 0.47 | Pairwise accuracy (imagery): 0.50. This result provides evidence against the notion that potential collinearity or biases in our experimental design or evaluation procedure could have led to inflated results. Raugel, J., Szafraniec, M., Vo, H.V., Couprie, C., Labatut, P., Bojanowski, P., Wyart, V. and King, J.R. (2025). Disentangling the Factors of Convergence between Brains and Computer Vision Models. arXiv preprint arXiv:2508.18226. Solomon, S. H., Kay, K., & Schapiro, A. C. (2024). Semantic plasticity across timescales in the human brain. bioRxiv, 2024-02. (6) The manuscript currently lacks a limitations section, specifically regarding the design of the experiment. This involves the use of the overly homogenous dataset Coco, which invites overfitting, the mixing of sentence descriptions and visual images, which invites imagery of previously seen content, and the use of a 1-back task, which can lead to carry-over effects to the subsequent trial. Regarding the dataset CoCo: We agree that CoCo is somewhat homogenous, it is however much more diverse and naturalistic than the smaller datasets used in previous fMRI experiments with multimodal stimuli. Additionally, CoCo has been widely adopted as a benchmark dataset in the Machine Learning community, and features rich annotations for each image (e.g. object labels, segmentations, additional captions, people’s keypoints) facilitating many more future analyses based on our data. Regarding the mixing of sentence descriptions and images: Subjects were not asked to visualize sentences and different techniques for the one-back tasks might have been used. Generally, we do not see it as problematic if subjects are performing visual imagery to some degree while reading sentences, and this might even be the case during normal reading as well. A more targeted experiment comparing reading with and without interleaved visual stimulation in the form of images and a one-back task would be required to assess this, but this was not the focus of our study. For now, it is true that we can not be sure that our results generalize to cases in which subjects are just reading and are less incentivized to perform mental imagery. Regarding the use of a 1-back task: It was necessary to make some design choices in order to realize this large-scale data collection with approximately 10 hours of recording per subject. Specifically, the 1-back task was included in the experimental setup in order to assure continuous engagement of the participant during the rather long sessions of 1 hour. The subjects did indeed need to remember the previous stimulus to succeed at the 1-back task, which means that some brain activity during the presentation of a stimulus is likely to be related to the previous stimulus. We aimed to account for this confound during the preprocessing stage when fitting the GLM, which was fit to capture only the response to the presented image/caption, not the preceding one. Still, it might have picked up on some of the activity from preceding stimuli, causing some decrease of the final decoding performance. We added a limitations section to the updated manuscript to discuss these important issues. (7) I would urge the authors to clarify whether the primary aim is the introduction of a dataset and showing the use of it, or whether it is the set of results presented. This includes the title of this manuscript. While the decoding approach is very interesting and potentially very valuable, I believe that the results in the current form are rather descriptive, and I'm wondering what specifically they add beyond what is known from other related work. This includes imagery-related results. This is completely fine! It just highlights that a stronger framing as a dataset is probably advantageous for improving the significance of this work. Thanks a lot for pointing this out. Based on this comment and feedback from the other reviewers we restructured the abstract, introduction and discussion section of the paper to better reflect the primary aim. (cf. general response above). You further mention that it is not clear what our results add beyond what is known from related work. We list the main contributions here: A single modality-agnostic decoder can decode the semantics of visual and linguistic stimuli irrespective of the presentation modality with a performance that is not lagging behind modality-specific decoders. Modality-agnostic decoders outperform modality-specific decoders for decoding captions and mental imagery. Modality-invariant representations are widespread across the cortex a range of previous work has suggested they were much more localized (Bright et al. 2004; Jung et al. 2018; Man et al. 2012; Simanova et al. 2014). Regions that are useful for imagery are largely overlapping with modality-invariant regions Bright, P., Moss, H., & Tyler, L. K. (2004). Unitary vs multiple semantics: PET studies of word and picture processing. Brain and language, 89(3), 417-432. Jung, Y., Larsen, B., & Walther, D. B. (2018). Modality-Independent Coding of Scene Categories in Prefrontal Cortex. Journal of Neuroscience, 38(26), 5969–5981. Liuzzi, A. G., Bruffaerts, R., Peeters, R., Adamczuk, K., Keuleers, E., De Deyne, S., Storms, G., Dupont, P., & Vandenberghe, R. (2017). Cross-modal representation of spoken and written word meaning in left pars triangularis. NeuroImage, 150, 292–307. https://doi.org/10.1016/j.neuroimage.2017.02.032 Man, K., Kaplan, J. T., Damasio, A., & Meyer, K. (2012). Sight and Sound Converge to Form Modality-Invariant Representations in Temporoparietal Cortex. Journal of Neuroscience, 32(47), 16629–16636. Simanova, I., Hagoort, P., Oostenveld, R., & van Gerven, M. A. J. (2014). Modality-Independent Decoding of Semantic Information from the Human Brain. Cerebral Cortex, 24(2), 426–434. Reviewer #2 (Public review): Summary: This study introduces SemReps-8K, a large multimodal fMRI dataset collected while subjects viewed natural images and matched captions, and performed mental imagery based on textual cues. The authors aim to train modality-agnostic decoders--models that can predict neural representations independently of the input modality - and use these models to identify brain regions containing modality-agnostic information. They find that such decoders perform comparably or better than modality-specific decoders and generalize to imagery trials. Strengths: (1) The dataset is a substantial and well-controlled contribution, with >8,000 image-caption trials per subject and careful matching of stimuli across modalities - an essential resource for testing theories of abstract and amodal representation. (2) The authors systematically compare unimodal, multimodal, and cross-modal decoders using a wide range of deep learning models, demonstrating thoughtful experimental design and thorough benchmarking. (3) Their decoding pipeline is rigorous, with informative performance metrics and whole-brain searchlight analyses, offering valuable insights into the cortical distribution of shared representations. (4) Extension to mental imagery decoding is a strong addition, aligning with theoretical predictions about the overlap between perception and imagery. Weaknesses: While the decoding results are robust, several critical limitations prevent the current findings from conclusively demonstrating truly modality-agnostic representations: (1) Shared decoding ≠ abstraction: Successful decoding across modalities does not necessarily imply abstraction or modality-agnostic coding. Participants may engage in modality-specific processes (e.g., visual imagery when reading, inner speech when viewing images) that produce overlapping neural patterns. The analyses do not clearly disambiguate shared representational structure from genuinely modality-independent representations. Furthermore, in Figure 5, the modality-agnostic encoder did not perform better than the modality-specific decoder trained on images (in decoding images), but outperformed the modality-specific decoder trained on captions (in decoding captions). This asymmetry contradicts the premise of a truly "modality-agnostic" encoder. Additionally, given the similar performance between modality-agnostic decoders based on multimodal versus unimodal features, it remains unclear why neural representations did not preferentially align with multimodal features if they were truly modality-independent. We agree that successful modality-agnostic and cross-modal decoding does not necessarily imply that abstract patterns were decoded. In the updated manuscript, we therefore refer to these representations as modality-invariant (see also the updated terminology explained in the general response above). If participants are performing mental imagery when reading, and this is allowing us to perform cross-decoding, then this means that modality-invariant representations are formed during this mental imagery process, i.e. that the representations formed during this form of mental imagery are compatible with representations during visual perception (or, in your words, produce overlapping neural patterns). While we can not know to what extent people were performing mental imagery while reading (or having inner speech while viewing images), our results demonstrate that their brain activity allows for decoding across modalities, which implies that modality-invariant representations are present. It is true that our current analyses can not disambiguate modality-invariant representations (or, in your words, shared representational structure) from abstract representations (in your words, genuinely modality-independent representations). As the main goal of the paper was to build modality-agnostic decoders, and these only require what we call “modality-invariant” representations (see our updated terminology in the general reviewer response above), we leave this question open for future work. We do however discuss this important limitation in the Discussion section of the updated manuscript. Regarding the asymmetry of decoding results when comparing modality-agnostic decoders with the two respective modality-specific decoders for captions and images: We do not believe that this asymmetry contradicts the premise of a modality-agnostic decoder. Multiple explanations for this result are possible: (1) The modality-specific decoder for images might benefit from the more readily decodable lower-level modality-dependent neural activity patterns in response to images, which are less useful for the modality-agnostic decoder because they are not useful for decoding caption trials. The modality-specific decoders for captions might not be able to pick up on low-level modality-dependent neural activity patterns as these might be less easily decodable. The signal-to-noise ratio for caption trials might be lower than for image trials (cf. generally lower caption decoding performance), therefore the addition of training data (even if it is from another modality) improves the decoding performance for captions, but not for images (which might be at ceiling already). Regarding the similar performance between modality-agnostic decoders based on multimodal versus unimodal features: Unimodal features are based on rather high-level features of the respective modality (e.g. last-layer features of a model trained for semantic image classification), which can be already modality-invariant to some degree. Additionally, as already mentioned before, in the updated manuscript we only require representations to be modality-invariant and not necessarily abstract. (2) The current analysis cannot definitively conclude that the decoder itself is modality-agnostic, making "Qualitative Decoding Results" difficult to interpret in this context. This section currently provides illustrative examples, but lacks systematic quantitative analyses. The qualitative decoding results in Figures 6 and 7 present exemplary qualitative results for the quantitative results presented in Figures 4 and 5 (see also Reviewer 1 Comment 4). Figures 4 and 5 show pairwise decoding accuracy as a quantitative measure for evaluation of the decoders. This metric is the main metric we used to compare different decoder types and features. Based on the finding that modality-agnostic decoders using imagebind features achieve the best score on this metric, we performed the additional qualitative analysis presented in Figures 6 and 7. (Note that we expanded the candidate set for the qualitative analysis in order to have a larger and more diverse set of images.) (3) The use of mental imagery as evidence for modality-agnostic decoding is problematic. Imagery involves subjective, variable experiences and likely draws on semantic and perceptual networks in flexible ways. Strong decoding in imagery trials could reflect semantic overlap or task strategies rather than evidence of abstraction. It is true that mental imagery does not necessarily rely on modality-agnostic representations. In the updated manuscript we revised our terminology and refer to the analyzed representations as modality-invariant, which we define as “representations that significantly overlap between modalities”. The manuscript presents a methodologically sophisticated and timely investigation into shared neural representations across modalities. However, the current evidence does not clearly distinguish between shared semantics, overlapping unimodal processes, and true modality-independent representations. A more cautious interpretation is warranted. Nonetheless, the dataset and methodological framework represent a valuable resource for the field. We fully agree with these observations, and updated our terminology as outlined in the general response. Reviewer #3 (Public review): Summary: The authors recorded brain responses while participants viewed images and captions. The images and captions were taken from the COCO dataset, so each image has a corresponding caption, and each caption has a corresponding image. This enabled the authors to extract features from either the presented stimulus or the corresponding stimulus in the other modality. The authors trained linear decoders to take brain responses and predict stimulus features. "Modality-specific" decoders were trained on brain responses to either images or captions, while "modality-agnostic" decoders were trained on brain responses to both stimulus modalities. The decoders were evaluated on brain responses while the participants viewed and imagined new stimuli, and prediction performance was quantified using pairwise accuracy. The authors reported the following results: (1) Decoders trained on brain responses to both images and captions can predict new brain responses to either modality. (2) Decoders trained on brain responses to both images and captions outperform decoders trained on brain responses to a single modality. (3) Many cortical regions represent the same concepts in vision and language. (4) Decoders trained on brain responses to both images and captions can decode brain responses to imagined scenes. Strengths: This is an interesting study that addresses important questions about modality-agnostic representations. Previous work has shown that decoders trained on brain responses to one modality can be used to decode brain responses to another modality. The authors build on these findings by collecting a new multimodal dataset and training decoders on brain responses to both modalities. To my knowledge, SemReps-8K is the first dataset of brain responses to vision and language where each stimulus item has a corresponding stimulus item in the other modality. This means that brain responses to a stimulus item can be modeled using visual features of the image, linguistic features of the caption, or multimodal features derived from both the image and the caption. The authors also employed a multimodal one-back matching task, which forces the participants to activate modality-agnostic representations. Overall, SemReps-8K is a valuable resource that will help researchers answer more questions about modality-agnostic representations. The analyses are also very comprehensive. The authors trained decoders on brain responses to images, captions, and both modalities, and they tested the decoders on brain responses to images, captions, and imagined scenes. They extracted stimulus features using a range of visual, linguistic, and multimodal models. The modeling framework appears rigorous, and the results offer new insights into the relationship between vision, language, and imagery. In particular, the authors found that decoders trained on brain responses to both images and captions were more effective at decoding brain responses to imagined scenes than decoders trained on brain responses to either modality in isolation. The authors also found that imagined scenes can be decoded from a broad network of cortical regions. Weaknesses: The characterization of "modality-agnostic" and "modality-specific" decoders seems a bit contradictory. There are three major choices when fitting a decoder: the modality of the training stimuli, the modality of the testing stimuli, and the model used to extract stimulus features. However, the authors characterize their decoders based on only the first choice-"modality-specific" decoders were trained on brain responses to either images or captions, while "modality-agnostic" decoders were trained on brain responses to both stimulus modalities. I think that this leads to some instances where the conclusions are inconsistent with the methods and results. In our analysis setup, a decoder is entirely determined by two factors: (1) the modality of the stimuli that the subject was exposed to, and (2) the machine learning model used to extract stimulus features. The modality of the testing stimuli defines whether we are evaluating the decoder in a within-modality or cross-modality setting, but is not an inherent characteristic of a trained decoder First, the authors suggest that "modality-specific decoders are not explicitly encouraged to pick up on modality-agnostic features during training" (line 137) while "modality-agnostic decoders may be more likely to leverage representations that are modality-agnostic" (line 140). However, whether a decoder is required to learn modality-agnostic representations depends on both the training responses and the stimulus features. Consider the case where the stimuli are represented using linguistic features of the captions. When you train a "modality-specific" decoder on image responses, the decoder is forced to rely on modality-agnostic information that is shared between the image responses and the caption features. On the other hand, when you train a "modality-agnostic" decoder on both image responses and caption responses, the decoder has access to the modality-specific information that is shared by the caption responses and the caption features, so it is not explicitly required to learn modality-agnostic features. As a result, while the authors show that "modality-agnostic" decoders outperform "modality-specific" decoders in most conditions, I am not convinced that this is because they are forced to learn more modality-agnostic features. It is true that for example a modality-specific decoder trained on fmri data from images with stimulus features extracted from captions might also rely on modality-invariant features. We still call this decoder modality-specific, as it has been trained to decode brain activity recorded from a specific stimulus modality. In the updated manuscript we corrected the statement that “modality-specific decoders are not explicitly encouraged to pick up on modality-invariant features during training” to include the case of decoders trained on features from the other modality which might also rely on modality-invariant features. It is true that a modality-agnostic decoder can also have access to modality-dependent information for captions and images. However, as it is trained jointly with both modalities and the modality-dependent features are not compatible, it is encouraged to rely on modality-invariant features. The result that modality-agnostic decoders are outperforming modality-specific decoders trained on captions for decoding captions confirms this, because if the decoder was only relying on modality-dependent features the addition of additional training data from another stimulus modality could not increase the performance. (Also, the lack of a performance drop compared to modality-specific decoders trained on images is only possible thanks to the reliance on modality-invariant features. If the decoder only relied on modality-dependent features the addition of data from another modality would equal an addition of noise to the training data which must result in a performance drop at test time.). We can not exclude the possibility that modality-agnostic decoders are also relying on modality-dependent features, but our results suggest that they are relying at least to some degree on modality-invariant features. Second, the authors claim that "modality-specific decoders can be applied only in the modality that they were trained on, while "modality-agnostic decoders can be applied to decode stimuli from multiple modalities, even without knowing a priori the modality the stimulus was presented in" (line 47). While "modality-agnostic" decoders do outperform "modality-specific" decoders in the cross-modality conditions, it is important to note that "modality-specific" decoders still perform better than expected by chance (figure 5). It is also important to note that knowing about the input modality still improves decoding performance even for "modality-agnostic" decoders, since it determines the optimal feature space-it is better to decode brain responses to images using decoders trained on image features, and it is better to decode brain responses to captions using decoders trained on caption features. Thanks for this important remark. We corrected this statement and now say that “modality-specific decoders that are trained to be applied only in the modality that they were trained on”, highlighting that their training process optimizes them for decoding in a specific modality. They can indeed be applied to the other modality at test time, this however results in a substantial performance drop. It is true that knowing the input modality can improve performance even for modality-agnostic decoders. This can most likely be explained by the fact that in that case the decoder can leverage both, modality-invariant and modality-dependent features. We will not further focus on this result however as the main motivation to build modality-agnostic decoders is to be able to decode stimuli without knowing the stimulus modality a priori. Recommendations for the authors: Reviewer #1 (Recommendations for the authors): I will list additional recommendations below in no specific order: (1) I find the term "modality agnostic" quite unusual, and I believe I haven't seen it used outside of the ML community. I would urge the authors to change the terminology to be more common, or at least very early explain why the term is much better suited than the range of existing terms. A modality agnostic representation implies that it is not committed to a specific modality, but it seems that a representation cannot be committed to something. In the updated manuscript we now refer to the identified brain patterns as modality-invariant, which has previously been used in the literature (Man et al. 2012; Devereux et al. 2013; Patterson et al. 2016; Deniz et al. 2019, Nakai et al. 2021) (see also the general response on top and the Introduction and Related Work sections of the updated manuscript). We continue to refer to the decoders as modality-agnostic, as this is a new type of decoder, and describes the fact that they are trained in a way that abstracts away from the modality of the stimuli. We chose this term as we are not aware of any work in which brain decoders were trained jointly on multiple stimulus modalities and in order not to risk contradictions/confusions with other definitions. Deniz, F., Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019). The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality. Journal of Neuroscience, 39(39), 7722–7736. https://doi.org/10.1523/JNEUROSCI.0675-19.2019 Devereux, B. J., Clarke, A., Marouchos, A., & Tyler, L. K. (2013). Representational Similarity Analysis Reveals Commonalities and Differences in the Semantic Processing of Words and Objects. The Journal of Neuroscience, 33(48). Nakai, T., Yamaguchi, H. Q., & Nishimoto, S. (2021). Convergence of Modality Invariance and Attention Selectivity in the Cortical Semantic Circuit. Cerebral Cortex, 31(10), 4825–4839. https://doi.org/10.1093/cercor/bhab125 Man, K., Kaplan, J. T., Damasio, A., & Meyer, K. (2012). Sight and Sound Converge to Form Modality-Invariant Representations in Temporoparietal Cortex. Journal of Neuroscience, 32(47), 16629–16636. Patterson, K., & Lambon Ralph, M. A. (2016). The Hub-and-Spoke Hypothesis of Semantic Memory. In Neurobiology of Language (pp. 765–775). Elsevier. https://doi.org/10.1016/B978-0-12-407794-2.00061-4 (2) The table in Figure 1B would benefit from also highlighting the number of stimuli that have overlapping captions and images. The number of overlapping stimuli is rather small (153-211 stimuli depending on the subject). We added this information to Table 1B. (3) The authors wrote that training stimuli were presented only once, yet they used a one-back task. Did the authors also exclude the first presentation of these stimuli? Thanks for pointing this out. It is indeed true that some training stimuli were presented more than once, but only for the case of one-back target trials. In these cases the second presentation of the stimulus was excluded, but not the first. As the subject can not be aware of the fact that the upcoming presentation is going to be a one-back target, the first presentation can not be affected by the presence of the subsequent repeated presentation. We updated the manuscript to clarify this issue. (4) Coco has roughly 80-90 categories, so many image captions will be extremely similar (e.g., "a giraffe walking", "a surfer on a wave", etc.). How can people keep these apart? It is true that some captions and images are highly similar even though they are not matching in the dataset. This might result in several false button presses because the subjects identified an image-caption pair as matching when in fact it wasn't intended to. However, as there was no feedback given on the task performance, this issue should not have had a major influence on the brain activity of the participants. (5) Footnotes for statistics are quite unusual - could the authors integrate statistics into the text? Thanks for this remark, in the updated manuscript all statistics are part of the main text. (6) It may be difficult to achieve the assumptions of a permutation test - exchangeability, which may bias statistical results. It is not uncommon for densely sampled datasets to use bootstrap sampling on the predictions of the test data to identify if a given percentile of that distribution crosses 0. The lowest p-value is given by the number of bootstrap samples (e.g., if all 10,000 bootstrap samples are above chance, then p < 0.0001). This may turn out to be more effective. Thanks for this comment. Our statistical procedure was in fact involving a bootstrapping procedure to generate a null distribution on the group-level. We updated the manuscript to describe this method in more detail. Here is the updated paragraph: “To estimate the statistical significance of the resulting clusters we performed a permutation test, combined with a bootstrapping procedure to estimate a group-level null distribution see also Stelzer et al., 2013. For each subject, we evaluated the decoders 100 times with shuffled labels to create per-subject chance-level results. Then, we randomly selected one of the 100 chance-level results for each of the 6 subjects and calculated group-level statistics (TFCE values) the exact same way as described in the preceding paragraph. We repeated this procedure 10,000 times resulting in 10,000 permuted group-level results. We ensured that every permutation was unique, i.e. no two permutations were based on the same combination of selected chance-level results. Based on this null distribution, we calculated p-values for each vertex by calculating the proportion of sampled permutations where the TFCE value was greater than the observed TFCE value. To control for multiple comparisons across space, we always considered the maximum TFCE score across vertices for each group-level permutation (Smith and Nichols, 2009).” (7) The authors present no statistical evidence for some of their claims (e.g., lines 335-337). It would be good if they could complement this in their description. Further, the visualization in Figure 4 is rather opaque. It would help if the authors could add a separate bar for the average modality-specific and modality-agnostic decoders or present results in a scatter plot, showing modality-specific on the x-axis and modality-agnostic on the y-axis and color-code the modality (i.e., making it two scatter colors, one for images, one for captions). All points will end up above the diagonal. We updated the manuscript and added statistical evidence for the claims made: We now report results for the claim that when considering the average decoding performance for images and captions, modality-agnostic decoders perform better than modality-specific decoders, irrespective of the features that the decoders were trained on. Additionally, we report the average modality-agnostic and modality-specific decoding accuracies corresponding to Figure 4. For modality-agnostic decoders the average value is 81.86\%, for modality-specific decoders trained on images 78.15\%, and for modality-specific decoders trained on captions 72.52\%. We did not add a separate bar to Figure 4 as this would add additional information to a Figure which is already very dense in its information content (cf. Reviewers 2’s recommendations for the authors). We therefore believe it is more useful to report the average values in the text and provide results for a statistical test comparing the decoder types. A scatter plot would make it difficult to include detailed information on the features, which we believe is crucial. We further provide statistical evidence for the observation regarding the directionality of cross-modal decoding. Reviewer #2 (Recommendations for the authors): For achieving more evidence to support modality-agnostic representations in the brain, I suggest more thorough analyses, for example: (1) Traditional searchlight RSA using different deep learning models. Through this approach, it might identify different brain areas that are sensitive to different formats of information (visual, text, multimodal); subsequently, compare the decoding performance using these ROIs. (2) Build more dissociable decoders for information of different modality formats, if possible. While I do not have a concrete proposal, more targeted decoder designs might better dissociate representational formats (i.e., unimodal vs. modality-agnostic). (3) A more detailed exploration of the "qualitative decoding results"--for example, quantitatively examining error types produced by modality-agnostic versus modality-specific decoders--would be informative for clarifying what specific content the decoder captures, potentially providing stronger evidence for modality-agnostic representations. Thanks for these suggestions. As the main goal of the paper is to introduce modality-agnostic decoders (which should be more clear from the updated manuscript, see also the general response to reviews), we did not include alternative methods for identifying modality-invariant regions. Nonetheless, we agree that in order to obtain more in-depth insight into the nature of representations that were recorded, performing analyses with additional methods such as RSA, comparisons with more targeted decoder designs in terms of their target features will be indispensable, as well as more in-depth error type analyses. We leave these analyses as promising directions for future work. The writing could be further improved in the introduction and, accordingly, the discussion. The authors listed a series of theories about conceptual representations; however, they did not systematically explain the relationships and controversies between them, and it seems that they did not aim to address the issues raised by these theories anyway. Thus, the extraction of core ideas is suggested. The difference between "modality-agnostic" and terms like "modality-independent," "modality-invariant," "abstract," "amodal," or "supramodal," and the necessity for a novel term should be articulated. The updated manuscript includes an improved introduction and discussion section that highlight the main focus and contributions of the study. We believe that a systematic comparison of theories on conceptual representations involving their relationships and controversies would require a dedicated review paper. Here, we focused on the aspects that are relevant for the study at hand (modality-invariant representations), for which we find that none of the considered theories can be rejected based on our results. Regarding the terminology (modality-agnostic vs. modality-invariant, ..) please refer to the general response. The figures also have room to improve. For example, Figures 4 and 5 present dense bar plots comparing multiple decoding settings (e.g., modality-specific vs. modality-agnostic decoders, feature space, within-modal vs. cross-modal, etc.); while comprehensive, they would benefit from clearer labels or separated subplots to aid interpretation. All figures are recommended to be optimized for greater clarity and directness in future revisions. Thanks for this remark. We agree that the figures are quite dense in information. However, splitting them up into subplots (e.g. separate subplots for different decoder types) would make it much less straightforward to compare the accuracy scores between conditions. As the main goal of these figures is to compare features and decoder types, we believe that it is useful to keep all information in the same plot. You are also suggesting to improve the clarity of the labels. It is true that the top left legend of Figures 4 and 5 was mixing information about decoder type and broad classes of features (vision/language/multimodal). To improve clarity, we updated the figures and clearly separated information on decoder type (the hue of different bars) and features (x-axis labels). The broad classes of features (vision/language/multimodal) are distinguished by alternating light gray background colors and additional labels at the very bottom of the plots. The new plots allow for easy performance comparison of the different decoder types and additionally provide information on confidence intervals for the performance of modality-specific decoders, which was not available in the previous figures. Reviewer #3 (Recommendations for the authors): (1) As discussed in the Public Review, I think the paper would greatly benefit from clearer terminology. Instead of describing the decoders as "modality-agnostic" and "modality-specific", perhaps the authors could describe the decoding conditions based on the train and test modalities (e.g., "image-to-image", "caption-to-image", "multimodal-to-image") or using the terminology from Figure 3 (e.g., "within-modality", "cross-modality", "modality-agnostic"). We updated our terminology to be clearer and more accurate, as outlined in the general response. The terms modality-agnostic and modality-specific refer to the training conditions, and the test conditions are described in Figure 3 and are used throughout the paper. (2) Line 244: I think the multimodal one-back task is an important aspect of the dataset that is worth highlighting. It seems to be a relatively novel paradigm, and it might help ensure that the participants are activating modality-agnostic representations. It is true that the multimodal one-back task could play an important role for the activation of modality-invariant representations. Future work could investigate to what degree the presence of widespread modality-invariant representations is dependent on such a paradigm. (3) Line 253: Could the authors elaborate on why they chose a random set of training stimuli for each participant? Is it to make the searchlight analyses more robust? A random set of training stimuli was chosen in order to maximize the diversity of the training sets, i.e. to avoid bias based on a specific subsample of the CoCo dataset. Between-subject comparisons can still be made based on the test set which was shared for all subjects, with the limitation that performance differences due to individual differences or to the different training sets can not be disentangled. However, the main goal of the data collection was not to make between-subject comparisons based on common training sets, but rather to make group-level analyses based on a large and maximally diverse dataset. (4) Figure 4: Could the authors comment more on the patterns of decoding performance in Figure 5? For instance, it is interesting that ResNet is a better target than ViT, and BERT-base is a better target than BERT-large. A multitude of factors influence the decoding performance, such as features dimensionality, model architecture, training data, and training objective(s) (Conwell et al. 2023; Raugel et al. 2025). Bert-base might be better than bert-large because the extracted features are of lower dimension. Resnet might be better than ViT because of its architecture (CNN vs. Transformer). To dive deeper into these differences further controlled analysis would be necessary, but this is not the focus of this paper. The main objective of the feature comparison was to provide a broad overview over visual/linguistic/multimodal feature spaces and to identify the most suitable features for modality-agnostic decoding. Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2023). What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? (p. 2022.03.28.485868). bioRxiv. https://doi.org/10.1101/2022.03.28.485868 Raugel, J., Szafraniec, M., Vo, H.V., Couprie, C., Labatut, P., Bojanowski, P., Wyart, V. and King, J.R. (2025). Disentangling the Factors of Convergence between Brains and Computer Vision Models. arXiv preprint arXiv:2508.18226. (5) Figure 7: It is interesting that the modality-agnostic decoder predictions mostly appear traffic-related. Is there a possibility that the model always produces traffic-related predictions, making it trivially correct for the presented stimuli that are actually traffic-related? It could be helpful to include some examples where the decoder produces other types of predictions to dispel this concern. The presented qualitative examples were randomly selected. To make sure that the decoder is not always predicting traffic-related content, we included 5 additional randomly selected examples in Figures 6 and 7 of the updated manuscript. In only one of the 5 new examples the decoder was predicting traffic-related content, and in this case the stimulus had actually been traffic-related (a bus). Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Citations Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. 2025. SemReps-8K. OpenNeuro. [ DOI ] [ PMC free article ] [ PubMed ] Supplementary Materials MDAR checklist elife-107933-mdarchecklist1.pdf (8.4MB, pdf) Data Availability Statement All fMRI data is available at https://openneuro.org/datasets/ds007272 . Code for preprocessing and analyses is available at https://github.com/mitjanikolaus/multimodal_decoding (copy archived at Nikolaus, 2026 ). The following dataset was generated: Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. 2025. SemReps-8K. OpenNeuro. Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd ACTIONS View on publisher site PDF (25.0 MB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Modality-Independent Core Brain Network for Language as Proved by Sign Language#1159820
• Extending Decoupled Attention to Dense Prediction and Masked Training for Multi-Channel Images#1036025
• NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.#382361
• MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation#794109
• MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation#790090