Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization Benjamin Léger
IID / Mila, Université Laval Quebec City (Quebec), Canada [email protected]
Kazem Meidani Capital One McLean, USA
arXiv:2604.08324v1 [cs.NE] 9 Apr 2026
Abstract Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.
Keywords Symbolic Regression, Genetic Programming, Latent Space Optimization, Deep Learning, Multi-modal Learning ACM Reference Format: Benjamin Léger, Kazem Meidani, and Christian Gagné. 2026. Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization. In Proceedings of Preprint – not peer-reviewed (Preprint). ACM, New York, NY, USA, 9 pages.
1
Introduction
While traditional Genetic Programming typically explores vast symbolic solution spaces through combinatorial search, recent methods [5, 12, 18, 20] – referred to here as Latent Space Optimization Genetic Programming, or LSO-GP – transform this combinatorial problem into continuous optimization. Leveraging the success of deep representation learning methods, these approaches use neural encoders to map symbolic expressions into continuous and semantically dense latent spaces. Traditional black-box optimizers are Preprint, 2026.
Christian Gagné
Canada-CIFAR AI Chair IID / Mila, Université Laval Quebec City (Quebec), Canada [email protected]
subsequently used to conduct the search over this space. However, most existing LSO-GP methods focus exclusively on encoding the symbolic structure (i.e., genotype) of candidate solutions, without capturing information about their numeric behavior (i.e., phenotype or semantics).. Symbolic Regression (SR) is, however, a multi-modal task [16, 19], as it aims to find interpretable symbolic equations from numeric data. Understanding the underlying genotype-phenotype mapping, that is, data-fitting accuracy, i.e., how well the solution fits the data, and has long been recognized as key for the design of effective SR heuristics [31]. The field of Semantic Genetic Programming [27] has demonstrated how semantic information can guide symbolic search more efficiently. Moreover, SR inherently requires optimizing two distinct types of objectives: numerical accuracy, or how well the solution fits the data, and symbolic relevance, which includes interpretability and meaningful syntactic patterns [3, 33]. Rather than treating these modalities separately, exploiting their interactions and mutual information is crucial for designing effective search algorithms [16]. Recent advances in multi-modal models offer a potential solution to this challenge. Vision-Language Models (VLMs) like CLIP [23] can map complex data to information-rich continuous spaces while learning relationships between different modalities of the same concept. This capability suggests multi-modal models could provide the missing ingredient for data-driven Genetic Programming: algorithms that explore continuous spaces which are simultaneously informative about both the numeric and symbolic nature of candidate solutions. Building on this idea, SNIP [19], inspired by CLIP, uses a pretrained bi-modal model to encode both numeric and symbolic modalities of mathematical equations. The model is trained to align these representations by mapping both modalities of the same equation close to each other in a shared latent space. For symbolic regression, SNIP conducts LSO search in the numeric latent space, with the premise that the learned inter-modal alignment will implicitly guide exploration of the symbolic space. However, the effectiveness of this approach depends on two critical assumptions. First, the LSO algorithm must actively exploit the learned alignment during search. Second, the alignment itself must be fine-grained enough to distinguish between symbolically different expressions. Existing literature on contrastive bi-modal models like CLIP reveals systematic failures to capture fine-grained semantic distinctions [6, 14, 25, 30], raising the question of whether SNIP’s alignment is sufficiently fine-grained for effective symbolic optimization. In this work, we investigate these two assumptions to better understand the strengths and limitations of multi-modal LSO for
Preprint, 2026,
symbolic regression. Specifically, we make the following contributions: (1) We empirically demonstrate that SNIP’s current LSO formulation does not actively exploit the learned cross-modal alignment during optimization, even as fitness improves. (2) We evaluate the granularity of SNIP’s cross-modal alignment using retrieval tasks adapted from the contrastive learning literature, revealing that the alignment is too coarse to reliably distinguish between structurally similar symbolic expressions. (3) We discuss implications for future work, identifying finegrained alignment as a critical direction for improving multimodal LSO methods and outlining potential paths forward for symbolic regression.
2 Preliminaries and background 2.1 Latent Space Optimization for SR Latent Space Optimization (LSO) transforms the combinatorial search in Genetic Programming (GP) algorithms into continuous optimization in learned representation spaces. The typical approach involves training an autoencoder to map symbolic expressions to continuous embeddings and reconstruct them via a decoder, minimizing syntactic reconstruction objectives such as cross-entropy loss. Once this space is constructed, gradient-free or gradient-based optimization algorithms can be utilized to search and sample candidates from this space, which are then decoded into symbolic expressions and evaluated for their fitness on the target data (as depicted in Fig 1-a). Prior LSO methods for symbolic regression [5, 12, 20] primarily encode symbolic structure without explicitly representing numeric semantics.
2.2
SNIP : Multi-modal Pretraining for Mathematical Equations
SNIP [19] is a multi-modal pre-training framework that jointly learns symbolic and numeric representations of mathematical expressions. Inspired by CLIP [23], SNIP uses contrastive learning to align both modalities in a shared latent space, aiming to capture the genotype-phenotype mapping between symbolic structure and numeric behavior. We provide an overview of SNIP’s architecture, training procedure, and its application to symbolic regression through latent space optimization. 2.2.1 Architecture and training. SNIP’s architecture consists of two transformer encoders: one processing symbolic expressions (as prefix-order token sequences), another processing numeric observations (input-output pairs (𝑥, 𝑦)), resulting in latent vector representations for each modality, respectively denoted as 𝑍𝑆 (for symbolic embeddings) and 𝑍𝑉 (for numeric embeddings) in the rest of the paper. Both encoders are trained with the same contrastive objective used in CLIP, the InfoNCE loss [22], learning to align embeddings of matched symbolic-numeric pairs while separating unrelated pairs (as illustrated in Fig 1-b):
LÉGER et al.
LInfoNCE = −
∑︁ (𝑆,𝑉 ) ∈ B
exp(𝑍𝑆 · 𝑍𝑉+ /𝜏) 𝑍 ∈ {𝑍 + ,𝑍 − } exp(𝑍 𝑆 · 𝑍 /𝜏)
log Í
𝑉
𝑉
exp(𝑍𝑉 · 𝑍𝑆+ /𝜏) + log Í 𝑍 ∈ {𝑍 + ,𝑍 − } exp(𝑍𝑉 · 𝑍 /𝜏) 𝑆
,
(1)
𝑆
where B is a minibatch of randomly generated mathematical equations, 𝜏 a learnable temperature coefficient, and 𝑍𝑆+ , 𝑍𝑉+ , 𝑍𝑆− and 𝑍𝑉− denote positive (matched) and negative (unmatched) pairs for each modality. Concretely, for a given expression 𝑓 with symbolic form 𝑆 and numeric observations 𝑉 , the positive pair is (𝑍𝑆 , 𝑍𝑉 ) from the same expression, while negative pairs are formed by all other expressions in the batch. This formulation encourages a form of geometric coherence between the two modalities in a “joint” learned space, with the main objective of training models that capture the underlying relationship between them. 2.2.2 Latent Space Optimization for Symbolic Regression. SNIP applies LSO to symbolic regression in a framework similar to the general approach described in Section 2.1, with a key difference: the search is conducted in the continuous latent space produced by the numeric encoder, rather than a discrete symbolic space. A decoder network, inherited and fine-tuned from prior neuro-generative SR work [11], maps latent vectors to symbolic expressions. The optimization procedure consists of the following steps: (1) The model encodes the target numerical dataset, resulting target in the latent vector 𝑍𝑉 ; (2) An initial population of latent vectors is initialized around target 𝑍𝑉 by randomly perturbing the data and latent vectors; (3) A gradient-free (black-box) optimizer, such as Grey Wolf Optimizer (GWO) [21], is employed to explore the continuous latent space, sampling new candidate vectors at each iteration; (4) Candidate vectors are decoded into symbolic expressions, whose constants are refined using BFGS algorithm [8]; (5) Solutions are evaluated for fitness (𝑅 2 ), which guides the optimizer’s search in subsequent iterations.
3 Alignment in Multi-Modal LSO 3.1 SNIP’s Performance: Promise and Gaps SNIP demonstrates promising capabilities for symbolic regression. The approach shows strong data-fitting accuracy and generates solutions with reasonable complexity on the SRBench benchmark [13]. Evidence suggests the model learns meaningful cross-modal relationships: LSO in the learned space yields better results than single-shot predictions decoded directly from the target’s numeric embedding, the model can quickly classify mathematical properties of expressions directly from their symbolic form, and visualizations reveal meaningful clusters of basic symbolic properties (e.g., number of variables, operator types) in the numeric space and vice versa (e.g., convexity, monotonicity in the symbolic space). However, the key hypothesis underlying SNIP’s multi-modal approach, that progressive moves in the continuous latent space translate to meaningful symbolic modifications via learned alignment, remains unvalidated. Recent work by Yu et al. [33] demonstrates that despite SNIP’s data-fitting performance, it struggles to
Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
a)
Preprint, 2026,
Symbolic expressions
b)
E2
c)
E3
Symbolic Encoder
Numeric Encoder
Numeric Encoder
Encode
Numeric data
S4
NB . SB
XYB
Symbolic Encoder
Evaluate
E* = sin(x1+1)
Similarity scores
Update
Decode
Sample solutions
Base expression : f*(x) = sin(x1+1)
Latent vectors
Latent Continuous
Optimizer
E1 = cos(x1+1) E2 = sin(x1*1) E3 = sin(x1+3) E4 = sin(x2+1)
Figure 1: Overview of Latent Space Optimization and multi-modal alignment. a) Traditional LSO-GP framework : a continuous optimizer searches the latent space of a symbolic autoencoder. b) SNIP’s [19] contrastive pre-training: symbolic and numerical encoders are trained to align embeddings of matched expression pairs with bat. c) Illustration of alignment granularity limitations : given a base expression 𝑓 ★ = sin(𝑥 + 1) and its numeric embedding 𝑍𝑉★ (labeled 𝑁 ★ in the figure) encoded from observations (𝑋, 𝑦★), the model should assign highest similarity between 𝑍𝑉★ and the symbolic embedding 𝑍𝑆★ (labeled 𝑆 ★) over embeddings of perturbed variants (𝑆 1, 𝑆 2, 𝑆 3 ). In practice, the learned alignment often fails to make this distinction. effectively retrieve relevant symbolic forms, showing significantly inferior symbolic retrieval rates compared to GP-based heuristics. Notably, SNIP fails to show substantial improvement over previous transformer-based generative models that include no explicit symbolic constraints in their search. This gap between data-fitting performance and symbolic retrieval raises fundamental questions about whether the learned alignment effectively guides symbolic search during optimization.
3.2
Limitations of Contrastive alignment and CLIP-like multi-modal models
SNIP’s training objective mirrors CLIP’s [23], which learns joint vision-language representations through contrastive learning. The extensive literature analyzing CLIP reveals systematic limitations that may be relevant for understanding SNIP’s performance. While CLIP achieves impressive zero-shot performance on many tasks, its alignment captures high-level semantic similarity rather than fine-grained structure. Studies show CLIP behaves as a “bag of concepts”, failing to bind attributes to objects or distinguish compositional relationships [14]. Similarity scores do not vary faithfully with semantic changes [30], and the model struggles with quantifiers, negations, and spatial relations [6, 25]. For SNIP, analogous limitations could mean the model captures coarse numerical behavior but fails to distinguish symbolically different expressions with similar outputs. These findings are particularly concerning for symbolic regression. Effective LSO in the symbolic space requires the ability to model how small symbolic modifications translate into behavioral changes. If CLIP’s contrastive alignment fails on fine-grained visuallinguistic distinctions despite its scale and success, SNIP’s alignment may similarly struggle to capture the precise symbolic-numeric relationships needed for principled symbolic search.
3.3
Measuring Cross-Modal Alignment
We refer to inter-modal alignment as how well the model associates the symbolic form of an expression with its numeric behavior (the two modalities) in the learned latent space. We quantify alignment using the cosine similarity between embeddings, which is the similarity function directly optimized by the InfoNCE training objective (Eq. 1): alignment = 𝑎(𝑍𝑆 , 𝑍𝑉 ) = cos(𝑍𝑆 , 𝑍𝑉 ) =
𝑍𝑆 · 𝑍𝑉 . ∥𝑍𝑆 ∥∥𝑍𝑉 ∥
(2)
The InfoNCE loss (Eq. 1) explicitly maximizes this similarity for matched symbolic-numeric pairs of the same expression during training while minimizing it for unmatched pairs. In principle, this learned alignment could guide the optimization, that is if a candidate solution’s symbolic embedding 𝑍𝑆 has high alignment with the target’s numeric embedding 𝑍𝑉∗ , this could indicate symbolic relevance to the problem. However, whether the alignment is actually exploited during LSO and whether it is fine-grained enough to support symbolic search remains an open question.
3.4
Research Hypotheses
Given the gap between SNIP’s data-fitting performance and its symbolic retrieval performance, combined with known limitations of contrastive alignment in similar models, we investigate two hypotheses for why the learned alignment may not effectively guide symbolic search: H1 – Algorithm exploitation: The LSO procedure does not actively exploit the learned cross-modal alignment during optimization, even if the alignment quality is sufficient. H2 – Alignment granularity: The learned alignment is too coarse-grained to distinguish between symbolically different expressions, preventing effective symbolic guidance even if the algorithm attempts to exploit it. These hypotheses are not mutually exclusive. Both could contribute to the observed gap in symbolic performance. In the following sections, we design experiments to test each hypothesis empirically.
Preprint, 2026,
4
Experimental Analysis
To address the hypotheses outlined above, we pose two research questions. • Does SNIP’s LSO actively exploit the learned alignment, or does optimization occur independently of cross-modal relationships? (Verifying H1) • Is the learned alignment fine-grained enough to support symbolic search, or does it only capture coarse behavioral similarities? (Verifying H2) We answer these questions through two complementary studies.
4.1
Study 1: Cross-Modal Alignment During Optimization
To verify H1, we investigate whether SNIP actively exploits crossmodal alignment during optimization. We run SNIP’s original LSO algorithm and track the following quantities for the best individual at each iteration 𝑡: • 𝑅𝑡2 fitness, indicating whether evolution improves data-fitting accuracy; • Inter-modal alignment between its symbolic embedding and the target’s numeric embedding (Eq. 2). While LSO gradient-free optimizer’s elitist nature ensures monotonically increasing 𝑅𝑡2 , the evolution of 𝑎𝑡 reveals whether the model’s search is guided by symbolic relevance, that is, how symbolically appropriate the current candidate solution is relative to the symbolic regression target. If the algorithm actively exploits the learned alignment to incorporate symbolic awareness, this quantity should progressively increase over the course of optimization. 4.1.1 Experimental details. We replicate SNIP’s experimental setup, hyperparameters, and model parameters (for full details, see [19]), running optimization for 80 iterations without early stopping to enable consistent comparison of metrics across the entire optimization trajectory. Dataset. We use the Feynman and Strogatz benchmark suites included in the popular symbolic regression benchmark SRBench [13], keeping all equations with input dimensions 𝐷 ≤ 10. Encoders. Symbolic and numeric embeddings used to compute alignment were obtained using SNIP’s original pre-trained encoders (not fine-tuned for LSO), as provided by the authors. Initialization and first measurement. Following SNIP’s LSO procedure, the population is initialized via dataset augmentation: subsampling (𝑝 1 = 15 individuals), target perturbation (𝑝 2 = 10 individuals), and latent perturbation (𝑝 3 = 25 individuals), totaling 𝑃 = 50 individuals. The first measurement point (𝑡 = 0) corresponds to the model’s one-shot prediction from the target data encoding. Measurement 𝑡 = 1 corresponds to the best individual after initialization and a BFGS constant refinement step. Measurement 𝑡 = 2 onwards correspond to the best individual after each GWO iteration. 4.1.2 Results. Evolution of both metrics is reported in Fig. 2. 𝑅 2 shows the expected increasing trend, improving from 0.73 at 𝑡 = 0 (one-shot prediction) to 0.96 at 𝑡 = 1 (after initialization and BFGS) and reaching 0.99 at 𝑡 = 80 (final iteration) on Feynman. The substantial 𝑅 2 improvement between 𝑡 = 0 and 𝑡 = 1 reveals the
LÉGER et al.
strong influence of initialization and constant refinement on datafitting accuracy. In contrast to numeric accuracy, alignment remains essentially flat throughout optimization, starting at 0.038 and decreasing slightly to 0.001 by the final iteration on the Feynman set. These results demonstrate that the search algorithm does not actively exploit the learned alignment during optimization. Even as numerical fitness steadily improves, alignment quality neither increases nor guides the search toward symbolically relevant solutions. This indicates that the LSO procedure operates independently of the cross-modal relationships learned during pretraining. Similar conclusions can be drawn with Strogatz equations. This confirms H1: SNIP’s LSO procedure does not actively exploit the learned cross-modal alignment during optimization.
4.2
Study 2: Alignment Granularity
We now address H2, by verifying whether the learned alignment is fine-grained enough to support symbolic search. While SNIP shows evidence of some cross-modal understanding (e.g., property prediction, operator clustering), effective symbolic optimization may require much finer granularity to distinguish between structurally similar expressions. We evaluate whether SNIP’s learned alignment can discriminate between structurally similar symbolic expressions through a retrieval task. Given a base expression 𝑓 and its corresponding numerical observations (𝑋, 𝑦), we generate 𝐾 −1 perturbed variants {𝑓1′, 𝑓2′, . . . , 𝑓𝐾′ −1 } that differ minimally from 𝑓 in symbolic structure. The task is to identify the correct expression 𝑓 among the 𝐾 candidates using only the alignment (cosine similarity, Eq. 2) between the symbolic embedding 𝑍𝑆 and the numerical embedding 𝑍𝑉∗ encoded from (𝑋, 𝑦). 4.2.1
Experimental details.
Dataset Construction. We evaluate on two datasets. First, the Feynman physics equations from SRBench with 2 ≤ 𝑑 ≤ 10 variables. The lower bound ensures applicability of variable substitution perturbations (which require at least two distinct variables), while the upper bound matches SNIP’s maximum capacity. After filtering equations without known ground-truth expressions and excluding 12 equations due to numerical issues or parsing failures, this yields 82 test cases. Second, 100 synthetic expressions generated using SNIP’s training data distribution. As these match the pretraining data, they might be seen as an upper bound on expected performance. Perturbation Strategies. For each base equation, we apply up to four symbolic perturbation types designed to create structurally similar but semantically different expressions, summarized in Table 1. For unary operators, we group operators into semantic families and allow swaps only within families: trigonometric (sin, cos, √ tan), exponential/logarithmic (exp, log), and power functions ( ·, (·) 2 , (·) 3 ). For binary operators, we allow swaps between any pair of the four arithmetic operators, covering both within-family swaps (e.g. + ↔ −, × ↔ ÷) and cross-family swaps (e.g. + ↔ ×, − ↔ ÷). When multiple instances of a swappable operator exist in an expression, one is selected uniformly at random. As not all perturbations are applicable to every equation (e.g., unary operator swap requires
Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
(a)
Table 2: Overall retrieval performance on both datasets. The model performs worse than random chance on Feynman, not better than random chance on synthetic data.
1.0 0.8
Fit quality (R²)
Preprint, 2026,
0.6 0.4 0.2
Metric
Feynman
Synthetic
Test cases Retrieval accuracy Random baseline Accuracy / Baseline Mean rank
82 18.3% 27.3% 0.67× 2.29 ± 0.92
100 23.0% 22.9% 1.00× 2.13 ± 0.85
Feynman Strogatz
0.0 0
10
20
30
40
50
60
70
80
Optimization iteration
(1) Load data (𝑋, 𝑦) with 𝑁 = 200 standardized input points; (2) Encode the numeric target: 𝑍𝑉∗ = encode𝑉 (𝑋, 𝑦) using SNIP’s original pre-trained encoder; (3) Encode all 𝐾 candidate expressions: 𝑍𝑆(𝑘 ) = encode𝑆 (𝑓𝑘 ) using the LSO-finetuned encoder; (4) Rank candidates by alignment 𝑎(𝑓𝑘 ) = cos(𝑍𝑆(𝑘 ) , 𝑍𝑉∗ ); (5) Record the rank of the true expression 𝑓 ∗ .
(b) 0.3
Feynman Strogatz
Alignment score
0.2
Evaluation Protocol. For each test case, we:
0.1 0.0
Metrics. We report retrieval accuracy: the fraction of test cases where the base expression ranks first. We compare against a random baseline corresponding to random selection over the 𝐾 valid candidates for each given evaluation case.
−0.1 −0.2 −0.3 0
10
20
30
40
50
60
70
80
Optimization iteration
Figure 2: Evolution of (a) 𝑅 2 fitness (top) and (b) cross-modal alignment (bottom) during LSO across all Feynman and Strogatz equations, averaged over all equations. While fitness improves consistently, alignment remains flat or decreases, indicating that optimization does not exploit cross-modal alignment to produce better symbolic solutions. Iteration 0 corresponds to the model’s one-shot prediction, iteration 1 to the best individual after initialization and BFGS constant refinement. Table 1: Perturbation types used to generate candidate expressions. Each perturbation modifies a single element of the expression tree while preserving overall structure. Type
Description
Examples
Unary operator swap
Replace unary operator with semantically related alternative Replace binary operator with any of the other three Modify a numeric constant Replace one variable with another
sin ↔ cos ↔ tan √ exp ↔ log; · ↔ (·) 2 ↔ (·) 3 + ↔ −; × ↔ ÷ + ↔ ×; − ↔ ÷ 2 → 3; 𝜋 → 𝑒; 𝑐 → 1.2𝑐 𝑥0 → 𝑥1
Binary operator swap Constant change Variable substitution
the presence of a swappable unary operator), each test case includes in average 𝐾¯ = 3.67 candidates (1 base + 2.67 perturbations) for the Feynman set and 𝐾¯ = 4.42 for the synthetic set.
4.2.2 Results. Table 2 presents retrieval performance on both datasets. On Feynman, the model achieves 18.3% accuracy, below the random baseline of 27.3 (0.67× baseline). This indicates that the alignment signal not only fails to discriminate between similar expressions, but actively misleads the ranking: perturbed expressions often have higher similarity to the target than the correct expression. On Synthetic data, accuracy matches the random baseline exactly (23.0%vs. 22.9%), meaning that the model performs no better than chance even on in-distribution expressions. Figure 3 shows the rank distribution for Feynman: the correct expression ranks first in only 15 cases (18.3%), while it most frequently ranks second (47.6%). Per-Perturbation Analysis. Table 3 shows which perturbation types most frequently “fool” the model on Feynman. Binary operator swaps are most problematic, fooling the model 65.8% of the time and accounting for 49% of all ranking failures. This suggests the alignment captures general functional form but cannot distinguish expressions that differ only in arithmetic operations. Variable substitutions are easiest to detect (only 24.4% fooling rate), indicating the model does encode some variable-specific information. This last observation is intuitively unsurprising: during pretraining, the numeric encoder receives observations ordered by variable index (i.e. as a stacked tensor [𝑋 0, 𝑋 1, . . . , 𝑋𝐷 ]) while the symbolic encoder processes variable through similarily annoted tokens (𝑥 0, 𝑥 1, . . . , 𝑥𝑑 ), making variable identity a straightforward correspondence to learn. Per-perturbation patterns are consistent on synthetic data: binary operator swaps remain most challenging (50.0% fooling rate) and variable substitutions easiest to detect (12.0% fooling rate).
Preprint, 2026,
LÉGER et al. 39
Table 4: Retrieval performance comparison: Feynman benchmark vs. synthetic data from training distribution.
100 80
30
60 20
17
15
11
10
40 20
0
Cumulative % of cases
Number of cases
40
Metric
Feynman
Synthetic
Test cases Retrieval accuracy Random baseline Accuracy / Baseline Mean rank
82 18.3% 27.3% 0.67× 2.29 ± 0.92
100 23.0% 22.9% 1.00× 2.13 ± 0.85
0 1
2
3
4
Rank of correct expression Count
Cumulative %
Figure 3: Distribution of ranks for the correct expression across 82 test cases of Feynman set. The correct expression most frequently ranks second (39 cases), with only 15 cases achieving rank 1. Table 3: Per-perturbation confusion analysis. “Fooling rate” indicates how often each perturbation type achieves higher similarity than the correct expression, “Wins” how often each perturbation is the ranks higher than all others. Perturbation Type Unary operator swap Binary operator swap Constant change Variable substitution
Occurrences
Fooling Rate
Wins (rank 1)
18 79 49 82
55.6% 65.8% 49.0% 24.4%
5 33 13 16
Impact of constant swaps. Since the handling and representations of constants have been shown to be challenging for transformerbased equation encoders [11, 15], and since SNIP performs a constant optimization step after each iteration, we repeat the experiment excluding constant perturbations entirely. Fig. 4 shows the impact of removing the “constant change” category on the overall results on the Feynman set. While excluding it improves accuracy from 18.3% to 23.2%, confirming that constant perturbations are indeed challenging, accuracy still falls short of the random baseline (23.2% vs 32.1%), demonstrating that the alignment’s limited discriminative power is not solely attributable to constant handling. The model also struggles with operator swaps, confirming that the coarse alignment granularity is a fundamental characteristic rather than a constant-specific artifact. Similar conclusions are noted with the Synthetic set.
5 Discussion 5.1 Summary of Findings Our experimental investigation addressed two research questions about multi-modal latent space optimization for symbolic regression. Study 1 (Sec. 4.1) demonstrated that SNIP’s LSO procedure does not actively exploit the learned cross-modal alignment during optimization. Despite steady improvements in numerical fitness (𝑅 2 ), alignment between candidate solutions and the target remains
flat or decreases throughout the search process. This reveals that the algorithm operates independently of the cross-modal relationships learned during pretraining. Study 2 (Sec. 4.2) evaluated whether the learned alignment is sufficiently fine-grained to support symbolic search. Through retrieval tasks with structurally similar expressions, we found that SNIP’s alignment is too coarse-grained to reliably distinguish between expressions that differ in operators or constants. The model achieves only 18.3% retrieval accuracy, below the 27.3% random baseline, indicating that perturbed expressions often align more strongly with targets than correct expressions. The model’s inability to differentiate solutions with potentially significant behavioral differences when their symbolic forms are close rules out the possibility of using alignment to guide symbolic space optimization. The combination of Studies 1 and 2 reveals a fundamental gap: alignment neither increases during optimization (Study 1), nor would such increases guide symbolic search effectively if they occurred (Study 2). This explains the symbolic retrieval gap observed by Yu et al. [33]. Furthermore, these results show that using alignment as an explicit objective would be bottlenecked by the model’s limited discriminative capacity: even if alignment increased during evolution, this would likely not result in meaningful symbolic improvements. Despite these limitations, our analysis is constructive: by isolating two precise bottlenecks—algorithmic exploitation and alignment granularity—it provides a concrete roadmap for improving multi-modal LSO methods. The potential of such methods remains significant: learning the phenotype-genotype mapping in a continuous, optimizable space could shift SR from hand-crafted heuristics to data-driven search paradigms. Realizing this potential requires addressing the specific challenges identified here.
5.2
Implications for Multi-Modal LSO
These results have broader implications beyond SNIP. They reveal a fundamental challenge for multi-modal latent space optimization methods in symbolic regression: learning cross-modal alignment that is simultaneously robust enough to emerge from contrastive pretraining and fine-grained enough to support symbolic search. The success of CLIP in vision-language tasks might suggest that similar architectures and objectives would naturally transfer to symbolic-numeric domains. Our findings demonstrate otherwise. The distinction between coarse-grained semantic alignment and fine-grained structural alignment is critical. In vision-language tasks, CLIP’s ability to capture high-level semantic similarity (e.g., “a dog playing in a park”) suffices for many applications. In symbolic
Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
(a)
(b) 40
Model Random
40
30
27% 23%
20
18%
10
Number of cases
32%
Accuracy (%)
Preprint, 2026,
All pert. Excl. const.
30 20 10
0
0 All perturbations
Excl. constant changes
1
2
3
4
Rank of correct expression
Figure 4: Sensitivity analysis excluding constant change perturbations on Feynman. (a) Accuracy comparison showing that even without constant changes, model accuracy (23.2%) remains below random baseline (32.1%). (b) Rank distribution comparison showing improved but still suboptimal performance when constant changes are excluded. regression, however, effective search requires understanding how minimal symbolic modifications (e.g., sin(𝑥) → cos(𝑥) or 𝑥 + 𝑦 → 𝑥 ×𝑦) translate to behavioral changes. This demands a qualitatively different level of alignment granularity. For the broader LSO-GP community, this suggests that incorporating numeric information into latent spaces requires more than simply adding a numeric encoder and applying contrastive learning. The alignment must be explicitly designed and evaluated for the fine-grained discriminative capacity that symbolic search demands. This represents both a challenge and an opportunity: methods that successfully achieve fine-grained alignment could substantially advance the field.
5.3
Future Directions
5.3.1 Algorithm-Level Improvements. A natural approach to leverage alignment would be to explicitly incorporate it into the optimization objective. For instance, the LSO algorithm could optimize both 𝑅 2 fitness and alignment quality (Equation (2)), guiding the search toward regions where candidate solutions are both numerically accurate and symbolically relevant to the target. However, our findings show this strategy is bottlenecked by alignment quality. Preliminary experiments confirm that while explicitly optimizing alignment increases 𝑎𝑡 during evolution, it does not improve symbolic retrieval performance when the underlying alignment is too coarse. This suggests that model-level improvements to alignment granularity are a prerequisite for effective algorithm-level exploitation. 5.3.2 Model-Level Improvements. Addressing coarse alignment in contrastive models is an active research area. Several techniques have been proposed to improve fine-grained alignment in CLIP and similar vision-language models [14, 30]. Adapting these techniques to mathematical expressions represents a promising direction. For symbolic regression, this might involve generating training data with carefully designed hard negatives (expressions that differ minimally but behave differently), incorporating structured contrastive objectives that explicitly reward fine-grained discrimination, or using auxiliary tasks that require detailed symbolic understanding.
Additionally, refined fine-tuning strategies after pretraining could improve alignment for specific SR applications. Such model-level improvements could then be combined with algorithm-level guidance mechanisms, enabling LSO procedures that truly exploit multimodal information for symbolic search.
5.3.3 Broader Applications. Beyond latent space optimization, improved multi-modal alignment could benefit symbolic regression in other ways. If alignment quality reaches sufficient granularity, it could serve as a learned symbolic similarity metric, offering advantages over hand-crafted measures like edit distance or tree distance. Such metrics could improve benchmarking methodologies and enable more principled evaluation of symbolic retrieval. Additionally, multi-modal representations could guide traditional GP operators, informing initialization, selection, or variation to maintain both numerical accuracy and symbolic coherence, similar to recent neural-guided GP methods [1, 10].
5.4
Limitations
Our analysis focuses on SNIP as a representative example of multimodal LSO for symbolic regression. While SNIP uses widely adopted design choices (Transformer encoders, InfoNCE loss, contrastive learning), investigating alignment quality in methods with different architectures or training objectives would provide additional insights into the generality of our findings. Our retrieval evaluation in Study 2 employed specific perturbation strategies designed to test fine-grained discrimination; alternative evaluation protocols could reveal complementary aspects of alignment quality. Finally, while our findings identify alignment granularity as a critical bottleneck, we have not proposed or evaluated concrete solutions to improve it. However, the precise identification of this bottleneck—and the retrieval-based evaluation framework we introduce—provide both the target and the tools for future work on this problem.
Preprint, 2026,
6 Related works 6.1 Neural Genetic Programming Several approaches have explored integrating neural methods with Genetic Programming, either by learning continuous representations for optimization or by using neural models to guide traditional GP operators. 6.1.1 Latent Space Optimization for GP. Several LSO-GP methods encode symbolic expressions into continuous latent spaces for optimization [5, 7, 12, 20]. These approaches focus on symbolic structure without explicitly representing numerical behavior. NEO [18] uses contrastive learning to enforce semantic locality in Boolean program synthesis, but exploits the finite output space and does not extend to symbolic regression. Our work investigates whether multi-modal approaches that jointly encode symbolic structure and numeric behavior can overcome these limitations. 6.1.2 Neural Guidance in Traditional GP. Rather than conducting search in learned latent spaces, other methods use neural models to guide traditional GP operators such as initialization, selection, and variation [1, 10, 17, 32]. These approaches maintain the discrete, tree-based representation of GP while leveraging neural networks to inform search decisions. If multi-modal alignment can be improved to achieve fine-grained discrimination, similar guidance mechanisms could be developed that account for both symbolic structure and numeric behavior.
6.2
Deep Learning for Symbolic Regression
Building on the success of Transformer-based generative models, several works have approached symbolic regression through one-shot inductive learning [4, 11, 15, 28]. These methods pretrain models on large-scale synthetic datasets to learn direct mappings from numerical observations to autoregressively decoded symbolic expressions, enabling instantaneous prediction without iterative search. While offering fast inference, these methods struggle with identifying underlying symbolic patterns, especially for out-ofdistribution test cases [33], motivating the inclusion of multi-modal information as in SNIP. Recent research has addressed fundamental limitations in neurogenerative SR methods. Yu et al. [34] focus on improving training data quality through structural plausibility filtering based on "Effective Information," ensuring training sets better represent real physical formulas. Related issues of memorization bias [24] and generalization [29] are actively being investigated. Additionally, several works explore using diffusion models instead of autoregressive generation [2, 9, 26], offering alternative approaches to symbolic expression generation. Since SNIP inherits components from prior neuro-generative work [11], including its numeric encoder, symbolic decoder, and training data generation, these improvements represent orthogonal directions that could be combined with better alignment strategies. Our findings on alignment quality are complementary to these decoder-focused improvements: even with perfect decoders and training data, coarse alignment would still limit the effectiveness of multi-modal LSO.
LÉGER et al.
6.3
Multi-Modal Approaches to Symbolic Regression
Several recent works share the perspective that symbolic regression should be approached as a multi-modal task requiring joint optimization or understanding of symbolic and numeric modalities. Li et al. [16] argue that SR cannot be properly modeled as sequenceto-sequence translation with word-to-word correspondence, but rather as a multi-modal task akin to image captioning. Their MMSR model optimizes symbolic, data-fitting, and alignment objectives simultaneously to predict expressions in a one-shot manner without iterative search. Bertschinger et al. [3] similarly emphasize simultaneous optimization of symbolic and numerical objectives, framing SR as dual optimization of "form and function." They use neuroevolution to evolve neural network weights that bridge these often orthogonal metrics. MDLformer [33] provides symbolic guidance by using Minimum Description Length (MDL) as a search objective, guiding Monte Carlo Tree Search toward symbolically principled solutions rather than purely minimizing prediction error. These works share our view that effective symbolic regression requires understanding and exploiting relationships between symbolic structure and numeric behavior. Our contribution is complementary: we investigate the fundamental requirement of finegrained cross-modal alignment that underlies all such approaches. Our findings suggest that achieving sufficiently fine-grained alignment is more challenging than prior work has acknowledged, and that alignment quality should be explicitly evaluated rather than assumed. This has implications for any multi-modal SR method, whether based on LSO, one-shot prediction, or evolutionary search.
7
Conclusion
We investigated the promise of multi-modal learning for Genetic Programming, focusing on SNIP’s application to symbolic regression. Through systematic experiments, we demonstrated two key limitations: (1) SNIP’s latent space optimization does not actively exploit the learned cross-modal alignment during search, and (2) the alignment learned through contrastive pre-training is too coarsegrained to distinguish between structurally similar expressions. These findings explain the gap between SNIP’s numerical success and its symbolic retrieval performance observed in prior work. While current bi-modal models do not yet achieve the fine-grained understanding required for principled symbolic search, our analysis clarifies the path forward: improving alignment granularity through refined training objectives, and designing optimization algorithms that explicitly leverage cross-modal relationships. We believe these directions can unlock the significant potential of multimodal approaches for symbolic regression.
References [1] Philipp Anthes, Dominik Sobania, and Franz Rothlauf. 2025. Transformer semantic genetic programming for symbolic regression. In Proceedings of the Genetic and Evolutionary Computation Conference. 952–960. [2] Tommaso Bendinelli, Luca Biggio, and Pierre-Alexandre Kamienny. 2023. Controllable neural symbolic regression. In International Conference on Machine Learning. PMLR, 2063–2077. [3] Amanda Bertschinger, James Bagrow, and Joshua Bongard. 2024. Evolving Form and Function: Dual-Objective Optimization in Neural Symbolic Regression Networks. In Proceedings of the Genetic and Evolutionary Computation Conference.
Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
277–285. [4] Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. 2021. Neural symbolic regression that scales. In International Conference on Machine Learning. PMLR, 936–945. [5] Victor Caetano, Matheus Cândido Teixeira, and Gisele Lobo Pappa. 2023. Symbolic Regression Trees as Embedded Representations. In Proceedings of the Genetic and Evolutionary Computation Conference. [6] Ziliang Chen, Tianang Xiao, Jusheng Zhang, Yongsen Zheng, and Xipeng Chen. 2025. Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens. arXiv preprint (2025). [7] Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. 2018. SyntaxDirected Variational Autoencoder for Structured Data. In International Conference on Learning Representations. [8] Roger Fletcher. 2013. Practical methods of optimization. John Wiley & Sons. [9] Xiaoxu Han, Chengzhen Ning, Jinghui Zhong, Fubiao Yang, Yu Wang, and Xin Mu. 2025. Discovering Mathematical Equations with Diffusion Language Model. arXiv preprint arXiv:2509.13136 (2025). [10] Xiaoxu Han, Jinghui Zhong, Zhitong Ma, Xin Mu, and Nikola Gligorovski. 2025. Transformer-Assisted Genetic Programming for Symbolic Regression. IEEE Computational Intelligence Magazine (2025). [11] Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and François Charton. 2022. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, Vol. 35. 10269–10281. [12] Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. 2017. Grammar variational autoencoder. In International conference on machine learning. PMLR, 1945–1954. [13] William La Cava, Bogdan Burlacu, Marco Virgolin, Michael Kommenda, Patryk Orzechowski, Fabrício Olivetti de França, Ying Jin, and Jason H Moore. 2021. Contemporary symbolic regression methods and their relative performance. Advances in neural information processing systems 2021, DB1 (2021), 1. [14] Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. 2022. Does CLIP Bind Concepts? Probing Compositionality in Large Image Models. arXiv preprint (2022). [15] Wenqiang Li, Weijun Li, Linjun Sun, Min Wu, Lina Yu, Jingyi Liu, Yanjie Li, and Songsong Tian. 2022. Transformer-based model for symbolic regression via joint supervised learning. In The Eleventh International Conference on Learning Representations. [16] Yanjie Li, Jingyi Liu, Min Wu, Lina Yu, Weijun Li, Xin Ning, Wenqiang Li, Meilan Hao, Yusong Deng, and Shu Wei. 2025. MMSR: symbolic regression is a multimodal information fusion task. Information Fusion 114 (2025), 102681. [17] Paweł Liskowski, Iwo Błądek, and Krzysztof Krawiec. 2018. Neuro-guided genetic programming: prioritizing evolutionary search with neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference. 1143–1150. [18] Paweł Liskowski, Krzysztof Krawiec, Nihat Engin Toklu, and Jerry Swan. 2020. Program Synthesis as Latent Continuous Optimization: Evolutionary Search in Neural Embeddings. In Proceedings of the Genetic and Evolutionary Computation Conference. 400–408. [19] Kazem Meidani, Parshin Shojaee, Chandan K Reddy, and Amir Barati Farimani. 2024. SNIP: Bridging mathematical symbolic and numeric realms with unified pre-training. In International Conference on Learning Representations. [20] Sebastian Mežnar, Sašo Džeroski, and Ljupčo Todorovski. 2023. Efficient generator of mathematical expressions for symbolic regression. Machine Learning 112, 11 (2023), 4563–4596. [21] Seyedali Mirjalili, Seyed Mohammad Mirjalili, and Andrew Lewis. 2014. Grey wolf optimizer. Advances in engineering software 69 (2014), 46–61. [22] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). [23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763. [24] Shun Sato and Issei Sato. 2025. Can Test-time Computation Mitigate Memorization Bias in Neural Symbolic Regression? arXiv preprint arXiv:2505.22081 (2025). [25] Shengbang Tong, Erik Jones, and Jacob Steinhardt. 2023. Mass-Producing Failures of Multimodal Systems with Language Models. Advances in Neural Information Processing Systems (2023). [26] Ryan T Tymkow, Benjamin D Schnapp, Mojtaba Valipour, and Ali Ghodshi. 2025. Symbolic-Diffusion: Deep Learning Based Symbolic Regression with D3PM Discrete Token Diffusion. arXiv preprint arXiv:2510.07570 (2025). [27] Leonardo Vanneschi, Mauro Castelli, and Sara Silva. 2014. A survey of semantic methods in genetic programming. Genetic Programming and Evolvable Machines 15, 2 (2014), 195–214. [28] Martin Vastl, Jonáš Kulhánek, Jiří Kubalík, Erik Derner, and Robert Babuška. 2024. Symformer: End-to-end symbolic regression using transformer-based architecture. IEEE Access 12 (2024), 37840–37849. [29] Henrik Voigt, Paul Kahlmeyer, Kai Lawonn, Michael Habeck, and Joachim Giesen. 2025. Analyzing Generalization in Pre-Trained Symbolic Regression. arXiv
Preprint, 2026,
preprint arXiv:2509.19849 (2025). [30] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. 2023. Equivariant similarity for visionlanguage foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11998–12008. [31] Stephan M Winkler, Michael Affenzeller, Bogdan Burlacu, Gabriel Kronberger, Michael Kommenda, and Philipp Fleck. 2018. Similarity-based analysis of population dynamics in genetic programming performing symbolic regression. In Genetic Programming Theory and Practice XIV. Springer, 1–17. [32] Piotr Wyrwiński and Krzysztof Krawiec. 2025. Learning Semantics-aware Search Operators for Genetic Programming. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 659–662. [33] Zihan Yu, Jingtao Ding, Yong Li, and Depeng Jin. 2025. Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length. In The Thirteenth International Conference on Learning Representations. [34] Zihan Yu, Guanren Wang, Jingtao Ding, Huandong Wang, and Yong Li. 2025. Beyond Formula Complexity: Effective Information Criterion Improves Performance and Interpretability for Symbolic Regression. arXiv preprint arXiv:2509.21780 (2025).