SLIM: Sparse Latent Steering for Interpretable and Property-Directed LLM-Based Molecular Editing Mingxu Zhang1 , Yuhan Li1 , Lujundong Li1 , Dazhong Shen2 * , Hui Xiong1† , Ying Sun3 * 1 The Hong Kong University of Science and Technology (Guangzhou) 2 Nanjing University of Aeronautics and Astronautics 3 The 63rd Research Institute, National University of Defense Technology, Nanjing [email protected], [email protected], [email protected]
arXiv:2605.10831v1 [cs.LG] 11 May 2026
Abstract
diffusion models (Lee et al., 2023; Igashov et al., 2024). Recently, large language models (LLMs) have demonstrated strong potential for molecular editing (Ye et al., 2025; Dey et al., 2025; Fang et al., 2024; Pei et al., 2023). LLMs acquire rich chemical priors, including structure–activity relationships, functional group chemistry, and pharmacophore patterns, through pretraining on massive corpora of chemical literature (Zhang et al., 2025b,a). Post training methods then bridge this chemical knowledge with SMILES-level structural transformations, positioning LLM-based editors to produce chemically meaningful modifications grounded in domain knowledge. Despite these advances, existing LLM-based editors provide no explicit handle for property control. Property-relevant information is implicitly entangled across the model’s dense hidden states, making it impossible to selectively amplify a target property without retraining. As a result, supervised fine-tuning offers only coarse control: a substantial fraction of generated edits fail to improve, or even degrade, the target property. Improving a specific property currently requires collecting new paired data and retraining the model, a costly process that scales poorly across properties and models. To address this, we propose SLIM, a plug-andplay framework that decomposes the editor’s hidden states into sparse, property-aligned features via a task-oriented Sparse Autoencoder (SAE) (Cunningham et al., 2024; Bricken et al., 2023). Unlike standard SAEs trained for reconstruction alone, our SAE is jointly trained with contrastive, predictive, and gradient-alignment objectives that shape the sparse basis to be property-aware. Per-property Importance Gates then learn to select, from this shared basis, the subset of features most relevant to each target property. To steer the model, we project a gradient-derived causal direction through the top-k selected features of the SAE, producing a sparse steering vector that precisely activates property-
Large language models possess strong chemical reasoning capabilities, making them effective molecular editors. However, propertyrelevant information is implicitly entangled across their dense hidden states, providing no explicit handle for property control: a substantial fraction of edits fail to improve or even degrade target properties. To address these issues, we propose SLIM (Sparse Latent Interpretable Molecular editing), a plug-and-play framework that decomposes the editor’s hidden states into sparse, property-aligned features via a Sparse Autoencoder with learnable importance gates. Steering in this sparse feature space precisely activates property-relevant dimensions, improving editing success rate without modifying model parameters. The same sparse basis further supports interpretable analysis of editing behavior. Experiments on the MolEditRL benchmark across four model architectures and eight molecular properties show consistent gains over baselines, with improvements of up to 42.4 points.
1
Introduction
Molecular editing, the task of modifying a given molecule to optimize specific properties while preserving its core structure, is a fundamental step in drug discovery and materials science (Hughes et al., 2011; Jiménez-Luna et al., 2020). In realworld pharmaceutical development, a promising lead compound often requires iterative structural refinements to improve druglikeness, synthetic accessibility, binding affinity, or other pharmacological properties. The ability to perform such targeted modifications efficiently and reliably holds great potential for accelerating the drug design pipeline. Early computational approaches to molecular optimization have relied on reinforcement learning (Loeffler et al., 2024; Zhuang et al., 2025) and * Corresponding authors. †
Corresponding authors.
1
relevant dimensions at inference time without modifying model parameters. The sparse basis further supports interpretable analysis, as individual SAE features encode recognizable chemical semantics. In summary, our contributions are as follows:
tokenization that reveals interpretable structure– property associations within LLM hidden states. More broadly, the linear representation hypothesis (Park et al., 2024) suggests that high-level concepts are encoded as directions in activation space, motivating methods that extract and manipulate these directions for controllable generation.
• We propose SLIM, a plug-and-play framework that decomposes dense LLM hidden states into sparse, property-aligned features for property-directed steering, requiring no retraining of the base molecular editor.
Sparse Autoencoders and Activation Steering. Neural networks represent more features than they have dimensions through superposition (Elhage et al., 2022). Sparse autoencoders (SAEs) recover these latent features by learning an overcomplete basis (Sharkey et al., 2022; Cunningham et al., 2024; Bricken et al., 2023). Templeton et al. (2024) show that SAE features scale to large models and capture semantically meaningful concepts. Gated SAEs (Rajamanoharan et al., 2024) improve feature quality by separating gating from magnitude estimation, and a recent survey (Gao et al., 2025) covers SAE architectures comprehensively. On the steering side, activation addition (Turner et al., 2023) modifies model behavior by adding direction vectors to hidden states during inference. Contrastive activation addition (Rimsky et al., 2024) computes directions as mean differences between positive and negative activations, and representation engineering (Zou et al., 2025) extends this with a top-down approach. However, existing SAEs are trained with reconstruction objectives alone and used only for post-hoc analysis, while steering methods rely on mean-difference directions that we show fail for molecular property control (§4.3). SLIM bridges these two lines by integrating taskoriented SAE training with gradient-based steering, producing sparse property-aligned features that enable precise, training-free property control.
• We introduce task-oriented SAE with contrastive, predictive, and gradient-alignment objectives and per-property Importance Gates, ensuring that the learned features are causally aligned with target properties. • Experimental results on the MolEditRL benchmark across four model architectures and eight molecular properties show consistent improvements of up to 42.4%. Case studies further show that individual SAE features encode interpretable chemical semantics.
2
Related Work
Molecular Editing. Molecular editing aims to modify a given molecule to optimize target properties while preserving structural similarity. Early approaches use reinforcement learning with scoring functions (Loeffler et al., 2024) or combine discrete diffusion with RL (Zhuang et al., 2025). More recently, LLM-based methods have shown strong results. DrugAssist (Ye et al., 2025) finetunes LLaMA-2 (Touvron et al., 2023) on paired molecular data, framing editing as instruction following. GeLLM3O (Dey et al., 2025) extends this to LLaMA-3 (Llama Team, AI @ Meta, 2024) and Mistral (Jiang et al., 2023), demonstrating crossmodel transferability. MolGen (Fang et al., 2024) adopts a BART-based (Lewis et al., 2020) encoderdecoder trained on SELFIES (Krenn et al., 2020) with chemical feedback. These methods optimize molecular properties through training but offer no mechanism to interpret or steer the learned representations at inference time.
3
Method
SLIM operates as a plug-and-play module on top of any molecular editing LLM. The pipeline consists of three stages: (1) a layer scan to identify the optimal intervention point, (2) task-oriented SAE training with gradient alignment, and (3) inferencetime activation steering via interpretable sparse directions. Figure 1 illustrates the overall framework.
Molecular Representation Learning. Understanding how models represent molecular structure and properties is central to interpretable editing. ChemATP (Zhang et al., 2025b) introduces a training-free chemical reasoning framework that leverages LLM internal representations. AtomDisc (Zhang et al., 2025a) proposes atom-level
3.1
Layer Scan
As shown in Figure 1 (Stage 1), different layers encode different levels of chemical abstraction. We select the layer l∗ whose hidden states are most informative about molecular properties by fitting 2
1
2 TASK-ORIENTED SAE
BEST LAYER SELECTION
A.Sparse Loss
Modify this molecule to increase its logP (lipophilicity). SMILES:CCCCCCC(C)NC(=O)c1ccc(C)cc1O
Decoder
🔥
🔥
Hidden State
Feature Probe
0.61
Layer 1
✕
★
0.89
✓
⋮ 0.27
Layer L
Sourced: 3.79 Model
DrugAssist GeLLM3O-L3
Layer 0
w_logP
MLP
w_QED
MLP
w_HBA
MLP
True Property vs Predicted Property
MSE
C.Contrastive Loss on hidden-state-derived sparse codes Similar value of property → Pull together
✕
0
0
1
1
Different value of property → Push apart 1
0
1 0
1 0
zj
zi positive pair hj
0
zi negative pair hj
hi
zj
Encoder
Avg R²
3
0.783
8
0.760
0.92 0.81 0.67 0.21 0.03 0
Layer 1
Layer 2
⋮
Base Edited Activation Activation
Layer l* (EDIT)
+
xᵢ (lower p)
Layer l*+1
⋮ Layer L
origin grad
gᵢ = ∇ log p_M( xʹᵢ | xᵢ )
Layer
Importanc Gate
1 0
D. Gradient Alignment Loss
SFT: 0.89 ✕
Sparse steering direction
Current SAE Activation z
1
hi
CCCCCCC(NC(=O)c1ccc (C)cc1O)C(N)=[NH2+]
Layer scan R²
Importance Gate
Sparse code
⋮ Layer l*
MSE reconstruction with Sparse loss
B.Supervised Property Loss
✓
⋯
Layer 2
Modify this molecule to increase its logP (lipophilicity). SMILES:CCCCCCC(C)NC(=O)c1ccc(C)cc1O
Embedding
Reconstruction
Property R² 0.12
Edit Residual Stream to Control Molecular Properties
At Chosen Layer l*
Encoder Embedding
Layer 0
3 INFERENCE-TIME STEERING
Obtain Properties Related Sparse Codes via SAE
Scan Layers to Find the Most Interpretable & Controllable
1
Decoder
x′ᵢ (higher p) origin grad
θ
recon grad
CCCCCCC(C)NC(=O)c1ccc(Oc2 ccc(C)cc2Br)cc1O
recon grad
Sourced: 3.79
Steered: 6.34 ✓
Figure 1: Overview of the SLIM framework. Stage 1: Ridge probes scan all layers to identify the optimal intervention point l∗ . Stage 2: A task-oriented SAE is trained at layer l∗ with four objectives: (A) sparse reconstruction, (B) supervised property prediction via per-property Importance Gates, (C) contrastive alignment of importance-gated sparse codes, and (D) gradient alignment to ensure the SAE basis faithfully represents causal steering directions. Stage 3: At inference time, a sparse steering vector is added to the residual stream at layer l∗ , directing the model toward improved molecular properties without modifying model parameters.
Importance Gates. As shown in Figure 1 (Stage 2B), each property p has a learnable Importance Gate wp ∈ RD that identifies which SAE features are relevant to p. The gated sparse code for property p is zp = wp ⊙ z, which feeds into per-property contrastive and predictive heads. This enables the SAE to learn property-aligned feature decompositions while sharing the same underlying sparse basis across all properties.
ridge regression probes (Park et al., 2024) on activations extracted from 5,000 ZINC molecules: 1 X 2 l∗ = arg max R (l, p), (1) l |P | p∈P
where P is the set of target properties and R2 (l, p) is the coefficient of determination for property p at layer l. The layer with the highest average R2 concentrates the most property-relevant information in its hidden states, making it the most effective point for steering: interventions at this layer can maximally influence generation because the representation is richest in the signals we aim to amplify. 3.2
Training Objective. Our task-oriented SAE is trained with a multi-objective loss: L = Lrecon + λc Lcontrast + λs Lsup + λsp Lsparse + λg L
grad , (3)
where Lrecon = ∥h − ĥ∥2 ensures the SAE faithfully reconstructs activations, preserving propertyrelated information needed for downstream generation. Lsparse = ∥z∥1 encourages activation sparsity, ensuring each molecule is represented by a small number of active features.
Task-Oriented SAE
As illustrated in Figure 1 (Stage 2A), we train a Gated SAE (Rajamanoharan et al., 2024) on hidden states h ∈ Rd extracted at layer l∗ from 50,000 molecules. The SAE uses D = 8d latent dimensions. Encoding produces a sparse code z ∈ RD :
Supervised Loss. As shown in Figure 1 (Stage 2B), Lsup trains a per-property MLP head fp to predict the property value from the gated code: P P (p) Lsup = |P1 | p∈P n1 ni=1 (fp (wp ⊙ zi ) − yi )2 ,
z = σ(Wg (h−bd ))⊙ReLU(Wm (h−bd )), (2) where Wg , Wm ∈ RD×d are gate and magnitude matrices, σ is the sigmoid function, and ⊙ denotes element-wise multiplication. Reconstruction is ĥ = Wd z + bd .
(p)
where yi is the oracle-computed property value. While Lcontrast only requires the gate to distinguish 3
high from low property values, Lsup demands that the selected features contain sufficient information to regress the precise numerical value, providing a stronger training signal for the Importance Gate.
where topk (·, k) retains the k largest-magnitude features. This decomposes the dense direction into a sparse, interpretable set of SAE features, where each feature has a known chemical semantics. At inference, the steering vector is added to the residual stream at all token positions in layer l∗ :
Contrastive Loss. As shown in Figure 1 (Stage 2C), Lcontrast is a group contrastive loss that operates on importance-gated sparse codes. For each property p, molecules in a batch are ranked by oracle-computed property values; the top 25% form the positive group P and the bottom 25% form the negative group N . The gated codes z̃i = norm(wp ⊙ zi ) are L2-normalized, and the loss is: P ⊤ (p) j∈P,j̸=i exp(z̃i z̃j /τc ) 1 P Lcontrast = − |P| i∈P log P exp(z̃⊤ z̃ /τ ) . k̸=i
i
k
(p)
h′l∗ ,t = hl∗ ,t + α · dsteer
c
4
N
1 X ∇hl∗ log pM (x′i | xi ) N ∥∇hl∗ log pM (x′i | xi )∥
(4)
i=1
where the gradient is averaged across all token positions and x′i has improved property value. This direction captures the perturbation to hl∗ that maximally increases the model’s probability of generating the property-improved target. The gradientalignment loss then encourages the SAE’s encodethen-decode pipeline to preserve this direction: L
grad =
X p∈P
1 − cos
(p) (p) Wd · topk enc(d ) , d
grad
grad
4.1
(5)
Properties and Metrics. We evaluate on eight molecular properties from the MolEditRL benchmark (Zhuang et al., 2025): QED (Bickerton et al., 2012), DRD2, logP, MW, RotBond, SA (Ertl and Schuffenhauer, 2009), HBA, and HBD (full descriptions in Appendix A). We report Acc@τ : the percentage of test molecules for which at least one generated candidate improves the target property and has Tanimoto similarity ≥ τ to the input, using Morgan fingerprints (Rogers and Hahn, 2010). We use τ = 0.15 (permissive) and τ = 0.65 (strict). The test set contains 500 molecules, with n = 5 candidates per molecule.
Inference-Time Activation Steering
As shown in Figure 1 (Stage 3), after training we extract a steering direction for each property p by (p) projecting dgrad through the trained SAE: (p)
(p) dsteer =
Wd · topk (enc(dgrad ), k) (p)
∥Wd · topk (enc(dgrad ), k)∥
Experimental Setup
Models. We evaluate SLIM on four molecular editing models spanning diverse architectures (Table 9): DrugAssist (Ye et al., 2025) (LLaMA-2-7B, l∗ = 3), GeLLM3O-LLaMA3 (Dey et al., 2025) (LLaMA-3.1-8B, l∗ = 8), GeLLM3O-Mistral (Mistral-7B, l∗ = 14), and MolGen (Fang et al., 2024) (BART-Large, l∗ = 5). This covers three decoder-only causal LMs and one encoder-decoder.
Joint Objective. These five losses serve complementary roles. Lrecon and Lsparse learn a faithful sparse decomposition of model hidden states. Lcontrast and Lsup shape the Importance Gates to select property-relevant features. Lgrad ensures the resulting sparse basis can faithfully reconstruct causal steering directions for downstream tasks. 3.3
Experiments
In this section, we empirically validate SLIM’s effectiveness and design choices. We aim to answer four research questions: RQ1: Does SLIM improve molecular editing across diverse model architectures and properties? (§4.2) RQ2: Are gradientbased steering directions essential, or do simpler alternatives like mean-difference (CAA) suffice? (§4.3) RQ3: Does the task-oriented SAE training provide gains beyond a vanilla reconstruction-only SAE? (§4.4) RQ4: Can SLIM’s steering directions compose for multi-property optimization? (§4.5)
Gradient-Alignment Loss. As shown in Figure 1 (Stage 2D), the key innovation is Lgrad , which ensures the SAE basis faithfully represents propertyrelevant steering directions. For each property p, we pre-compute a gradient-based direction from the SFT training pairs {(xi , x′i )}N i=1 : (p)
(7)
where α > 0 controls steering strength. This requires no retraining: the base model weights remain frozen, and only the added direction changes. For multi-property steering, directions combine via P (p) vector addition: dmulti = p αp · dsteer .
All same-group pairs are treated as positives and cross-group pairs as negatives, encouraging the Importance Gate to select features that separate molecules by property value.
dgrad =
∀t
(6)
4
[email protected] (%) DA
L3
Mi
MG
DA
L3
Mi
MG
Property
SFT
+S
SFT
+S
SFT
+S
SFT
+S
SFT
+S
SFT
+S
SFT
+S
SFT
+S
QED ↑ DRD2 ↑ logP ↑ MW ↑ RotBond ↑ SA ↓ HBA ↑ HBD ↑
77.0 47.6 55.8 41.8 36.4 67.6 50.6 49.6
77.6 50.6 61.2 46.8 41.2 67.6 77.4 74.2
95.8 88.2 88.2 86.8 55.8 94.0 45.0 28.8
97.2 88.8 89.6 89.8 64.2 94.8 47.6 42.2
71.0 52.4 58.0 51.4 38.6 65.4 49.4 37.4
98.2 90.8 94.4 93.0 75.0 95.8 54.2 47.4
42.4 37.2 46.2 48.2 43.2 44.4 42.6 33.8
76.2 83.6 86.4 87.0 85.6 49.4 55.0 40.8
60.4 35.8 43.4 32.2 27.0 54.8 38.4 37.2
61.8 42.6 51.4 38.6 33.8 55.6 56.0 55.2
32.4 29.6 43.2 44.6 26.4 32.6 20.8 13.8
34.4 30.4 48.6 48.4 29.8 35.4 27.2 23.8
52.2 40.2 44.0 37.0 29.6 49.8 37.0 25.4
48.6 47.2 60.2 63.8 43.6 55.4 30.6 30.0
19.4 23.0 27.2 22.6 18.0 25.2 23.0 16.8
36.6 33.4 46.2 46.4 35.4 31.2 32.4 18.2
Avg ∆
53.3 62.1 +8.8
72.8 76.8 +3.9
52.9 81.1 +28.1
42.3 70.5 +28.2
41.4 49.4 +8.0
30.4 34.8 +4.3
39.4 47.4 +8.0
21.9 35.0 +13.1
Table 1: Forward-direction results on the MolEditRL benchmark (500 test molecules, n = 5 candidates). +S denotes SFT with SLIM steering. ∆ = average gain over SFT. Bold = better of SFT vs. +S per cell.
4.2
Main Results
To answer RQ1, we evaluate SLIM on all four models across eight properties at both similarity thresholds (Table 1). As shown in Table 1, SLIM improves 31/32 model-property pairs at τ = 0.15. The gains are largest on models with weak SFT baselines: Mistral QED jumps from 71.0 to 98.2 (+27.2), and MolGen RotBond from 43.2 to 85.6 (+42.4), the single largest improvement. DrugAssist shows consistent moderate gains, with counting properties HBA (+26.8) and HBD (+24.6) as standouts. Even LLaMA-3, the strongest SFT model (72.8 avg), improves by +3.9 on average. At the strict threshold τ = 0.65, all four models maintain positive average gains (DA +8.0, L3 +4.3, Mi +8.0, MG +13.1), indicating that SLIM improves target properties without sacrificing molecular similarity.
Property
REINV.
MolEd.
+SLIM
REINV.
MolEd.
+SLIM
logP ↑ MW ↑ RotBond ↑ SA ↓ HBA ↑ HBD ↑
36.0 34.0 38.4 2.4 40.0 45.8
91.0 85.6 76.4 82.8 82.6 84.2
94.4 93.0 85.6 95.8 77.4 74.2
11.4 6.8 11.2 1.0 19.0 26.8
57.8 40.4 39.2 62.8 48.4 58.2
60.2 63.8 43.6 55.6 56.0 55.2
Avg
32.8
83.8
86.7
12.7
51.1
55.7
Table 2: Comparison with external baselines on six overlapping properties. REINVENT4 and MolEditRL results are from published papers on the same 500molecule test set. +SLIM reports the best result across our four base models for each property. Bold = best per property.
on MW at τ = 0.65 (+23.4 points). MolEditRL retains an edge on counting properties HBA and HBD at τ = 0.15, where its end-to-end RL optimization may better capture discrete structural changes.
Comparison with external baselines. To position SLIM against methods from different paradigms, we compare with REINVENT4 (Loeffler et al., 2024) (RL-based) and MolEditRL (Zhuang et al., 2025) (discrete diffusion + RL) using their published results on the same 500molecule test set. Table 2 reports the best SLIM result across our four base models for each property, compared against these external baselines on the six properties reported in both works (QED and DRD2 are omitted as MolEditRL does not report single-property results for these). As shown in Table 2, SLIM outperforms MolEditRL on 4/6 properties at both thresholds and achieves higher averages (86.7 vs. 83.8 at τ = 0.15; 55.7 vs. 51.1 at τ = 0.65), despite being a trainingfree plug-and-play module. The gains are largest
4.3
Ablation: Steering Direction
To answer RQ2, we compare SLIM’s gradientbased directions against simpler alternatives: SFT (no steering), a unit-norm Random vector, and CAA (Rimsky et al., 2024) (mean-difference between high- and low-property activations). All steered methods use the same per-property strength α. Table 3 shows per-property results on DrugAssist and Mistral. As shown in Table 3, on DrugAssist, Random and CAA are indistinguishable from SFT (avg ∆ < 2 at both thresholds), confirming that mean-difference directions fail to capture propertyrelevant steering information. On Mistral, Random and CAA inflate [email protected] for continuous properties (logP +30.8, MW +35.4) but decrease it for 5
DrugAssist (Layer 3) [email protected]
GeLLM3O-Mistral (Layer 14)
SFT
Rand.
CAA
SLIM
SFT
Rand.
CAA
SLIM
SFT
Rand.
CAA
SLIM
SFT
Rand.
CAA
SLIM
QED DRD2 logP MW RotBond SA HBA HBD
77.0 47.6 55.8 41.8 36.4 67.6 50.6 49.6
76.8 48.0 55.4 43.6 36.8 67.8 61.0 63.8
76.8 47.0 56.4 43.8 37.2 69.2 59.6 63.0
77.6 50.6 61.2 46.8 41.2 67.6 77.4 74.2
60.4 35.8 43.4 32.2 27.0 54.8 38.4 37.2
61.0 36.4 43.8 31.6 27.8 55.6 43.8 47.2
61.6 37.4 43.0 30.2 26.0 57.2 37.6 48.4
61.8 42.6 51.4 38.6 33.8 55.6 56.0 55.2
71.0 52.4 58.0 51.4 38.6 65.4 49.4 37.4
96.4 87.2 88.8 86.8 50.2 97.4 33.8 28.4
96.6 88.0 86.4 85.8 50.4 97.0 39.4 19.8
98.2 90.8 94.4 93.0 75.0 95.8 54.2 47.4
52.2 40.2 44.0 37.0 29.6 49.8 37.0 25.4
47.6 40.6 56.6 57.8 28.2 53.0 18.8 16.6
47.2 39.2 49.6 55.0 24.8 51.0 21.4 13.2
48.6 47.2 60.2 63.8 43.6 55.4 30.6 30.0
Avg
53.3
54.2
54.5
62.1
41.4
43.4
42.7
49.4
52.9
71.1
70.4
81.1
39.4
39.9
37.7
47.4
Table 3: Steering direction ablation on DrugAssist and Mistral. Four steering directions compared: no steering (SFT), random unit vector (Rand.), contrastive activation addition (CAA), and SLIM. All steered methods use the same per-property strength α. Bold = best per row.
counting properties (HBA −15.6, HBD −17.6 with CAA), revealing that mean-difference directions trade property gain for structural disruption. SLIM is the only method that improves [email protected] on both models (DA +8.0, Mi +8.0 vs. CAA’s +1.3 and −1.7), demonstrating that gradient-based directions preserve molecular similarity while increasing target properties. 4.4
(Lrecon , Lcontrastive , Lsupervised ). Table 5 shows results on Mistral and MolGen. The gradient alignment loss is essential for robust steering. On MolGen, a clean ordering SFT < w/o Lgrad < SLIM holds across all 8 properties at [email protected], with Lgrad contributing +4.4 average improvement. On Mistral, the pattern reveals a striking failure mode: without Lgrad , counting properties (HBA, HBD) degrade below SFT (−11.0 and −9.2 at [email protected]), even though continuous properties improve substantially. This indicates that Lgrad specifically aligns the SAE basis with the gradient structure needed for discrete property control. At [email protected], the gap widens further: SLIM gains +6.5 (Mi) and +3.5 (MG) over the ablated variant, confirming that gradient alignment preserves molecular similarity during steering.
Ablation: SAE Training Objective
To answer RQ3, we compare Vanilla SAE (Lrecon only) and our full Task-Oriented SAE (Eq. 3), both using the same pre-computed gradient direction. The key question is whether the SAE basis must be aligned with property-relevant axes, or whether reconstruction alone suffices. Table 4 shows per-property results. As shown in Table 4, a consistent ordering emerges: SFT < Vanilla SAE < SLIM. Vanilla SAE already provides partial gains (DA: +2.4, Mi: +25.2 at [email protected]), suggesting that even a reconstruction-only sparse bottleneck filters noise from the gradient direction. The task-oriented training adds further improvement (DA: +6.4, Mi: +3.0 over Vanilla at [email protected]), with the largest gains on counting properties: HBA improves by +9.2 over Vanilla on DA, and RotBond by +11.2 on Mi. The cosine similarity between Vanilla and taskoriented directions is near zero (< 0.05 for DA, < 0.12 for Mi), confirming that our losses fundamentally reorganize the latent basis rather than merely refining it.
4.5
Multi-Property Steering
To answer RQ4, we test whether SLIM’s sparse steering directions can be composed for multiproperty optimization. Directions combine via P (p) vector addition: dmulti = p αp · dsteer . Table 6 evaluates six property pairs on DrugAssist and Mistral, where a candidate is “successful” only if it improves all target properties simultaneously with Tanimoto ≥ τ . As shown in Table 6, SLIM improves joint success on 5/6 pairs at both thresholds. The largest gain is on Mi HBA+HBD (+13.6 at [email protected], +6.0 at [email protected]), counting properties that are individually difficult but benefit from complementary feature directions. DA logP+MW also shows strong improvement (+9.4/+4.0), as both properties share lipophilicity-related features. The composability of steering directions is a direct consequence of the sparse decomposition: because each prop-
Gradient alignment loss. We further isolate the contribution of the gradient alignment loss Lgrad by training a variant with λgrad = 0 (denoted w/o Lgrad ), retaining all other task-oriented losses 6
DrugAssist (Layer 3) [email protected]
GeLLM3O-Mistral (Layer 14)
SFT
Van.
SLIM
SFT
Van.
SLIM
SFT
Van.
SLIM
SFT
Van.
SLIM
QED DRD2 logP MW RotBond SA HBA HBD
77.0 47.6 55.8 41.8 36.4 67.6 50.6 49.6
77.4 47.8 55.4 44.0 38.8 67.2 68.2 66.8
77.6 50.6 61.2 46.8 41.2 67.6 77.4 74.2
60.4 35.8 43.4 32.2 27.0 54.8 38.4 37.2
62.0 38.4 45.8 35.8 30.4 56.6 48.2 50.4
61.8 42.6 51.4 38.6 33.8 55.6 56.0 55.2
71.0 52.4 58.0 51.4 38.6 65.4 49.4 37.4
97.2 90.6 91.4 90.8 63.8 96.2 55.4 39.4
98.2 90.8 94.4 93.0 75.0 95.8 54.2 47.4
52.2 40.2 44.0 37.0 29.6 49.8 37.0 25.4
50.6 45.0 61.2 60.2 36.6 54.4 31.0 26.4
48.6 47.2 60.2 63.8 43.6 55.4 30.6 30.0
Avg
53.3
55.7
62.1
41.4
45.9
49.4
52.9
78.1
81.1
39.4
45.7
47.4
Table 4: SAE training objective ablation on DrugAssist and Mistral. Three settings compared: no steering (SFT), steering with vanilla SAE (Lrecon only), and steering with task-oriented SAE (Eq. 3). All use the same gradient direction. Bold = best per row. GeLLM3O-Mistral (Layer 14) [email protected]
MolGen-Large (Layer 5)
SFT
w/o Lgrad
SLIM
SFT
w/o Lgrad
SLIM
SFT
w/o Lgrad
SLIM
SFT
w/o Lgrad
SLIM
QED DRD2 logP MW RotBond SA HBA HBD
71.0 52.4 58.0 51.4 38.6 65.4 49.4 37.4
96.4 88.8 89.6 87.4 58.0 95.2 38.4 28.2
98.2 90.8 94.4 93.0 75.0 95.8 54.2 47.4
52.2 40.2 44.0 37.0 29.6 49.8 37.0 25.4
47.6 44.6 55.8 58.2 30.4 53.8 20.4 16.6
48.6 47.2 60.2 63.8 43.6 55.4 30.6 30.0
42.4 37.2 46.2 48.2 43.2 44.4 42.6 33.8
74.8 79.0 78.0 83.6 75.2 46.8 52.0 39.6
76.2 83.6 86.4 87.0 85.6 49.4 55.0 40.8
19.4 23.0 27.2 22.6 18.0 25.2 23.0 16.8
34.0 31.2 37.8 43.8 25.4 30.4 30.2 18.8
36.6 33.4 46.2 46.4 35.4 31.2 32.4 18.2
Avg
52.9
72.8
81.1
39.4
40.9
47.4
42.3
66.1
70.5
21.9
31.5
35.0
Table 5: Gradient alignment loss ablation on Mistral and MolGen. w/o Lgrad removes gradient alignment while retaining Lrecon , Lcontrast , Lsup . Bold = best per row.
erty’s direction activates a distinct subset of SAE features (Table 7), adding directions produces minimal interference.
5
[email protected] Model Properties DA DA DA Mi Mi Mi
Case Study: Feature Interpretability
A key advantage of SLIM over black-box steering is that each SAE feature encodes a specific molecular property, not just a structural motif. Table 7 quantifies this: for each property, the top Importance Gate feature strongly discriminates molecules by property value. For six of eight properties, the best Importance Gate feature achieves |ρ| ≥ 0.52 (Table 7). Feature F31553 (ρ = +0.93) encodes molecular weight with a 251 Da gap between high- and low-activation quartiles. Feature F5418 (ρ = −0.85) captures molecular complexity, cleanly separating drug-like (QED=0.77) from non-drug-
SFT +SLIM SFT +SLIM
logP+MW 42.2 QED+SA 63.8 DRD2+SA 16.6 HBA+HBD 16.4 QED+SA 91.4 DRD2+SA 73.4
51.6 64.4 17.4 30.0 93.4 73.6
38.8 54.2 13.4 11.4 75.8 51.0
42.8 54.2 15.0 17.4 77.4 51.4
Table 6: Multi-property steering via direction addition on DrugAssist and Mistral. Joint success = candidate improves all target properties simultaneously with Tanimoto ≥ τ .
like (QED=0.27) molecules. For counting properties (HBD) and binary properties (DRD2), singlefeature correlations are weaker (ρ ≈ 0.32), indicating multi-feature encoding, consistent with the composite steering directions that successfully im7
Top-25% Bot-25%
∆
ρ
F31553 MW F5418 QED F17369 RotBond F31602 logP F18632 HBA F17369 SA
574.0 0.27 10.60 6.25 6.66 3.97
323.1 +250.9 +0.93 0.77 −0.50 −0.85 4.93 +5.67 +0.68 2.63 +3.62 +0.65 3.45 +3.21 +0.63 3.01 +0.96 +0.52
F4291 F7906
2.07 0.13
0.98 0.08
HBD DRD2
Source
SFT
+SLIM
4.51
1.82 (−2.69)
7.82 (+3.31)
5
5 (±0)
12 (+7)
4
4 (±0)
13 (+9)
logP↑
Prop.
+1.10 +0.32 +0.06 +0.32
HBA↑
Feat.
Rot.↑
Table 7: SAE feature interpretability (DrugAssist, Layer 3, 50K ZINC molecules). For each property, we report the best feature among the top-50 Importance Gate features ranked by |ρ|. Top/Bot-25% = mean property value of molecules with highest/lowest 25% feature activation. ρ = Spearman rank correlation between feature activation and property value.
Figure 2: Molecular editing examples on DrugAssist. Each row shows source, SFT output, and SLIM output for one property. Numbers below molecules indicate property values and changes (∆). Tanimoto similarities (SFT / SLIM): logP 0.65 / 0.79, HBA 0.69 / 0.75, RotBond 0.67 / 0.72.
prove these properties (Table 1). To confirm that features are not merely correlational, we steer using individual SAE decoder columns (α = 2.0, 500 test molecules). Single-feature steering produces targeted molecular changes: F3714 (logP-relevant) adds aromatic groups (+3.3 avg logP), F28089 (HBD-relevant) introduces NH groups (+1–2 donors), with 13–96 contrast pairs per feature where SFT fails but singlefeature steering succeeds. Full molecule visualizations are in Appendix B. Figure 2 shows representative cases where SFT fails but SLIM succeeds, each reflecting a distinct failure mode of the SFT editor. Property-reversing edit (logP). The source molecule (logP=4.51) contains an isopentyl chain contributing to hydrophobicity. SFT removes this chain and replaces it with a hydroxyl group, decreasing logP to 1.82, a structurally valid but property-reversing edit. SLIM instead replaces the methoxypyridine ring with a dibromophenoxy moiety, adding halogenated aromatic bulk that raises logP to 7.82 (+3.31) with higher structural similarity (Tanimoto 0.79 vs. 0.65). Ineffective edit (HBA). SFT removes a side chain but leaves acceptor count unchanged (5→5). SLIM introduces a guanidinium-like group with multiple nitrogen and oxygen atoms, adding 7 new acceptor sites (5→12) by activating SAE features associated with nitrogen-rich heterocycles. Overly conservative edit (RotBond). SFT modifies a substituent on the triazole ring but does not introduce any additional rotatable bonds (4→4). SLIM instead extends the molecule with a flexible alkenyl-alkyl chain, adding 9 rotatable bonds
(4→13, Tanimoto 0.72), overcoming SFT’s tendency toward rigid-scaffold-preserving edits. These cases share a common pattern: SFT produces valid edits but lacks a mechanism to ensure they align with the target property. SLIM provides this directional signal through property-relevant SAE features.
6
Conclusion
We presented SLIM, a plug-and-play framework that decomposes the dense hidden states of molecular editing LLMs into sparse, property-aligned features, enabling training-free property control at inference time without modifying model parameters. Our task-oriented SAE, trained with contrastive, predictive, and gradient-alignment objectives alongside per-property Importance Gates, produces a sparse basis that is causally aligned with target properties rather than merely correlated. Experiments on the MolEditRL benchmark across four architectures and eight properties show that SLIM consistently improves property-directed editing accuracy, with gains of up to 42.4 percentage points. Case studies further reveal that the learned SAE features encode interpretable chemical semantics and that steering corrects specific failure modes of SFT editors, including property-reversing edits, ineffective modifications, and overly conservative structural changes. 8
Limitations
large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025. ArXiv:2503.05613.
SLIM has several limitations. The steering strength and intervention layer require per-property and permodel tuning, adding setup cost before deployment. The SAE training pipeline introduces additional computational overhead beyond the base SFT model, though inference-time cost is negligible. Finally, our evaluation relies on computable property oracles, and the framework has not yet been validated on experimental assay endpoints.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. James P. Hughes, Stephen Rees, S. Barrett Kalindjian, and Karen L. Philpott. 2011. Principles of early drug discovery. British Journal of Pharmacology, 162(6):1239–1249. Ilia Igashov, Hannes Stärk, Clément Vignac, Victor Garcia Satorras, Pascal Frossard, Max Welling, Michael Bronstein, and Bruno Correia. 2024. Equivariant 3d-conditional diffusion model for molecular linker design. In Nature Machine Intelligence, volume 6, pages 417–427.
References G. Richard Bickerton, Gaia V. Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins. 2012. Quantifying the chemical beauty of drugs. Nature Chemistry, 4(2):90–98.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, and 5 others. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread.
José Jiménez-Luna, Francesca Grisoni, and Gisbert Schneider. 2020. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573–584.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations. ArXiv:2309.08600.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru-Guzik. 2020. Selfreferencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024.
Vishal Dey, Xiao Hu, and Xia Ning. 2025. GeLLMO: Generalizing large language models for multiproperty molecule optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ArXiv:2502.13398.
Greg Landrum and 1 others. RDKit: Open-source cheminformatics. https://www.rdkit.org. Seul Lee, Jaehyeong Jo, and Sung Ju Hwang. 2023. Exploring chemical space with score-based out-ofdistribution generation. In Proceedings of the 40th International Conference on Machine Learning.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer Circuits Thread. ArXiv:2209.10652.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.
Peter Ertl and Ansgar Schuffenhauer. 2009. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of Cheminformatics, 1(1):8. Yin Fang, Ningyu Zhang, Zhuo Chen, Lingbing Guo, Xiaohui Fan, and Huajun Chen. 2024. Domainagnostic molecular generation with chemical feedback. In International Conference on Learning Representations. ArXiv:2301.11259.
Llama Team, AI @ Meta. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Hannes H. Loeffler, Jiazhen He, Alessandro Tibo, Jon Paul Janet, Alexey Voronov, Lewis H. Mervin, and Ola Engkvist. 2024. REINVENT 4: Modern AI-driven generative molecule design. Journal of Cheminformatics, 16:20.
Dong Gao and 1 others. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of
9
Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of PMLR, pages 39643–39666.
Zeng. 2025. DrugAssist: A large language model for molecule optimization. Briefings in Bioinformatics, 26(1):bbae693. ArXiv:2401.10334. Mingxu Zhang, Dazhong Shen, and Ying Sun. 2025a. AtomDisc: An atom-level tokenizer that boosts molecular LLMs and reveals structure–property associations. arXiv preprint arXiv:2512.03080.
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. 2023. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1102–1123.
Mingxu Zhang, Dazhong Shen, Qi Zhang, and Ying Sun. 2025b. ChemATP: A training-free chemical reasoning framework for large language models. arXiv preprint arXiv:2512.19240.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024. Improving dictionary learning with gated sparse autoencoders. In Advances in Neural Information Processing Systems, volume 37. ArXiv:2404.16014.
Yuanxin Zhuang, Dazhong Shen, and Ying Sun. 2025. MolEditRL: Structure-preserving molecular editing via discrete diffusion and reinforcement learning. arXiv preprint arXiv:2505.20131.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering Llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. ArXiv:2312.06681.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representation engineering: A top-down approach to AI transparency. In International Conference on Learning Representations. ArXiv:2310.01405.
David Rogers and Mathew Hahn. 2010. Extendedconnectivity fingerprints. Journal of Chemical Information and Modeling, 50(5):742–754.
A
Lee Sharkey, Dan Braun, and Beren Millidge. 2022. Taking features out of superposition with sparse autoencoders. Alignment Forum.
Experimental Settings
Evaluation Metrics. We follow the MolEditRL benchmark protocol (Zhuang et al., 2025). For each test molecule x and property p, we generate n = 5 candidate molecules {x′1 , . . . , x′n } via nucleus sampling. A candidate x′i is considered successful if it satisfies three conditions: (1) it is a chemically valid SMILES string (verified by RDKit (Landrum et al.) parsing), (2) it improves the target property relative to the input (oracle(x′i ) > oracle(x) for increase tasks, or < for decrease tasks), and (3) its Tanimoto similarity to the input molecule meets or exceeds the threshold: Tanimoto(x, x′i ) ≥ τ . Tanimoto similarity is computed on Morgan fingerprints (Rogers and Hahn, 2010) with radius 2 and 2048 bits. We report Acc@τ : the percentage of test molecules for which at least one candidate is successful. We evaluate at two thresholds: τ = 0.15 (permissive, allowing larger structural changes) and τ = 0.65 (strict, requiring high structural preservation).
Teague Sterling and John J. Irwin. 2015. ZINC 15 – ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324–2337. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, and 7 others. 2024. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248.
Test Set. We use the 500-molecule test set from the DrugAssist evaluation suite, which is a standardized subset also adopted by MolEditRL. These molecules span diverse drug-like chemical space from the DrugAssist training distribution.
Geyan Ye, Xibao Cai, Houtim Lai, Xing Wang, Junhong Huang, Longyue Wang, Wei Liu, and Xiangxiang
10
Property
Oracle
Dir.
QED
RDKit QED (Bickerton et al., 2012) — quantitative estimate of druglikeness Random Forest classifier on Morgan FP (radius 2, 2048 bits) RDKit Crippen MolLogP — octanol-water partition coefficient RDKit MolWt — molecular weight in Daltons RDKit NumRotatableBonds SA scorer (Ertl and Schuffenhauer, 2009) — synthetic accessibility (lower = easier) RDKit NumHAcceptors — hydrogen bond acceptor count RDKit NumHDonors — hydrogen bond donor count
↑
DRD2 logP MW RotBond SA
HBA HBD
from larger strengths (α ≥ 1.0). MolGen (BART, d = 1024) requires ∼10× larger α than the 4096-dimensional causal LMs. Table 11 lists the selected values.
↑
Contrastive Loss Details. The InfoNCE contrastive loss Lcontrast operates per property. For each property p, molecules in a training batch are ranked by their oracle-computed property value. The top 25% molecules form the positive set and the bottom 25% form the negative set. For each positive anchor, its importance-gated code zp = wp ⊙z is projected through a learned 2-layer MLP to a 128-dimensional embedding space. The InfoNCE loss with temperature τc = 0.07 is: P X j∈pos,j̸=i exp(simij /τc ) (p) Lcontrast = − log P k̸=i exp(simik /τc )
↑ ↑ ↑ ↓
↑ ↑
i∈pos
(8) where simij is the cosine similarity between the projected embeddings of molecules i and j. The total contrastive loss averages over all properties: P (p) Lcontrast = |P1 | p Lcontrast .
Table 8: Molecular properties and oracles. Dir. indicates optimization direction. All oracles are deterministic and computed from molecular structure. Model
Architecture
Layer l∗
d
DrugAssist (DA) GeLLM3O-L3 (L3) GeLLM3O-Mi (Mi) MolGen (MG)
LLaMA-2-7B LLaMA-3.1-8B Mistral-7B BART-Large
3 8 14 5
4096 4096 4096 1024
Table 9: Models evaluated. Layer l∗ is selected by ridge probing (§3.1). d = hidden dimension. SAE expansion = 8×.
Supervised Loss Details. The supervised loss Lsup uses a per-property 2-layer MLP (hidden dim 256, ReLU activation) that takes the importancegated code zp as input and predicts the normalized property value. The loss P is MSE averaged over properties: Lsup = |P1 | p MSE(ŷp , yp ). Instruction Templates. For causal LMs (DrugAssist, LLaMA-3, Mistral), we use the LLaMA-2 chat format:
Properties and Oracles. Table 8 describes all eight molecular properties and their computational oracles.
<s>[INST] Modify this molecule to {increase/decrease} its {property description}: {SMILES} [/INST]
Models. Table 9 summarizes the four base models evaluated.
Property descriptions use full natural language names to match SFT training prompts: For MolGen (BART encoder-decoder), the input is the source SMILES directly, and the model generates the target SMILES via beam search (beam width 5).
Hyperparameters. Table 10 lists all key hyperparameters for each component.
Steering Strength Selection. The steering strength α is tuned per property by Computational Infrastructure. All experiments evaluating on a held-out 100-molecule were conducted on NVIDIA A100 80GB GPUs. validation split. We search over α ∈ SFT training uses 2 GPUs with DeepSpeed ZeRO{0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0, 5.0, 10.0} 2. SAE training and evaluation use a single GPU. and select the α maximizing [email protected] on the Gradient direction computation requires fp32 prevalidation set. Optimal values vary by model cision and takes approximately 2 hours per model and property type: composite properties (QED, per property on a single A100. SAE training conSA) prefer smaller strengths (α ≤ 0.5), while verges in approximately 30 minutes. Inferencecounting properties (HBA, HBD, RotBond) benefit time steering adds negligible overhead (<1% la11
SFT (LoRA (Hu et al., 2022))
Task-Oriented SAE
LoRA rank r LoRA alpha αlora Target modules Learning rate Batch size Epochs
16 32 q_proj, v_proj 2e-4 (cosine) 128 3
Architecture Expansion factor Learning rate Batch size Epochs Molecules
Gated SAE 8× (d → 8d) 1e-3 (Adam) 256 50 50K (ZINC)
Precision Pairs per property Token aggregation Per-example norm
Optimizer Precision
AdamW bf16
Decoder norm λc (contrastive) λs (supervised) λsp (sparsity) λg (gradient) Top-k for steering
Unit-norm cols 0.1 0.1 1e-3 0.5 200
Candidates/molecule Temperature Top-p (nucleus) Max new tokens Thresholds τ
Gradient Direction fp32 5,000 Mean all positions L2 before avg
Inference 5 0.8 0.95 256 0.15, 0.65
Table 10: Full hyperparameter settings. Steering strength α is tuned per property on a held-out validation set; optimal values are listed in Table 11.
{SMILES} is replaced with the source molecule at inference time.
Property DA L3 Mi MG QED 0.07 1.5 0.5 5.0 DRD2 0.7 0.7 0.3 10.0 logP 0.7 0.7 0.5 10.0 MW 0.5 1.5 0.3 10.0 RotBond 1.0 1.0 1.0 10.0 SA 0.2 0.1 0.1 5.0 HBA 2.0 1.0 1.0 10.0 HBD 2.0 2.0 1.0 10.0
B.2
Figures 5–10 show molecular editing examples using the full SLIM steering pipeline on DrugAssist. Red-highlighted regions indicate structural modifications relative to the source molecule. Each figure shows cases where SFT fails to improve the target property but SLIM succeeds. For each case, we report the property value, change (∆), and Tanimoto similarity (Sim) to the source molecule.
Table 11: Per-property steering strength α selected on validation set. Composite properties (QED, SA) use small α; counting properties (HBA, HBD) use large α. MolGen (d = 1024) requires larger α than 4096-dim models. Property
Prompt Description
QED DRD2 logP MW RotBond SA HBA HBD
“drug-likeness (QED)” “DRD2 binding activity” “lipophilicity (logP)” “molecular weight” “number of rotatable bonds” “synthetic accessibility” “number of hydrogen bond acceptors” “number of hydrogen bond donors”
GeLLM3O-Mistral GeLLM3O-LLaMA3 MolGen-Large
Table 12: Property descriptions used in instruction prompts.
tency increase) as it only involves a single vector addition per forward pass.
B
Case Studies
B.1
Single-Feature Steering
SLIM Steering Examples
Tables 13–16 show molecular editing examples using single SAE feature steering, demonstrating that individual features capture interpretable chemical semantics. All cases use the DrugAssist model with Layer-3 SAE steering. The instruction prompt for each property group is shown above the table; 12
[INST] Modify this molecule to increase its lipophilicity (logP): {SMILES} [/INST] SAE Feature F3714, semantics: phenyl 94%, halogen 79%, logP↑165%
Source
SFT
+SLIM
Property / Similarity
SFT: 2.72 (∆-2.57, ✗) Sim 0.72 SLIM: 8.61 (∆+3.32, ✓) Sim 0.75
SFT: 3.34 (∆-2.88, ✗) Sim 0.68 SLIM: 9.02 (∆+2.81, ✓) Sim 0.81
SFT: 0.59 (∆-2.68, ✗) Sim 0.65 SLIM: 6.24 (∆+2.97, ✓) Sim 0.62
SFT: -0.44 (∆-2.84, ✗) Sim 0.68 SLIM: 4.97 (∆+2.57, ✓) Sim 0.64
SFT: -6.05 (∆-3.57, ✗) Sim 0.69 SLIM: 0.92 (∆+3.39, ✓) Sim 0.67
Table 13: Case study: steering logP with Feature F3714. Source logP values: 5.29, 6.21, 3.27, 2.40, -2.47. SFT decreases logP in every case; SLIM steering increases it by +2.6 to +3.4.
Source
SFT
+SLIM
Source
SFT
+SLIM
0.56
0.45 (-0.11) Sim 0.68 0.75 (+0.19) Sim 0.54
0.51
0.50 (-0.01) Sim 0.67 0.69 (+0.17) Sim 0.70
0.49
0.46 (-0.04) Sim 0.70 0.64 (+0.15) Sim 0.71
0.24
0.17 (-0.07) Sim 0.75 0.37 (+0.14) Sim 0.56
0.51
0.38 (-0.13) Sim 0.57 0.63 (+0.13) Sim 0.74
0.48
0.30 (-0.18) Sim 0.71 0.57 (+0.09) Sim 0.62
Figure 3: SLIM steering examples: QED↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
13
[INST] Modify this molecule to increase its drug-likeness (QED): {SMILES} [/INST] SAE Feature F4829, semantics: compact heterocyclic scaffolds, MW∼200, QED↑, MW↓95%
Source
SFT
+SLIM
Property / Similarity
SFT: 0.46 (∆-0.25, ✗) Sim 0.76 SLIM: 0.77 (∆+0.06, ✓) Sim 0.62
SFT: 0.38 (∆-0.16, ✗) Sim 0.75 SLIM: 0.56 (∆+0.02, ✓) Sim 0.64
SFT: 0.44 (∆-0.01, ✗) Sim 0.69 SLIM: 0.61 (∆+0.17, ✓) Sim 0.76
SFT: 0.57 (∆-0.12, ✗) Sim 0.71 SLIM: 0.80 (∆+0.10, ✓) Sim 0.65
SFT: 0.61 (∆-0.03, ✗) Sim 0.68 SLIM: 0.78 (∆+0.14, ✓) Sim 0.72
Table 14: Case study: steering QED with Feature F4829. Source QED values: 0.71, 0.54, 0.45, 0.69, 0.64. SFT decreases QED in every case; SLIM steering increases it by +0.02 to +0.17.
Source
SFT
+SLIM
Source
SFT
+SLIM
0.16
0.11 (-0.05) Sim 0.69 0.29 (+0.13) Sim 0.69
0.14
0.08 (-0.06) Sim 0.65 0.26 (+0.12) Sim 0.60
0.02
0.01 (-0.01) Sim 0.67 0.13 (+0.11) Sim 0.61
0.10
0.10 (+0.00) Sim 0.73 0.17 (+0.07) Sim 0.68
0.13
0.12 (-0.01) Sim 0.76 0.19 (+0.06) Sim 0.69
0.09
0.01 (-0.08) Sim 0.62 0.13 (+0.04) Sim 0.70
0.10
0.07 (-0.03) Sim 0.80 0.14 (+0.04) Sim 0.78
0.06
0.03 (-0.03) Sim 0.67 0.10 (+0.04) Sim 0.70
Figure 4: SLIM steering examples: DRD2↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
14
[INST] Modify this molecule to increase its number of hydrogen bond donors: {SMILES} [/INST] SAE Feature F28089, semantics: NH 72%, phenyl only 40%, low logP
Source
SFT
+SLIM
Property / Similarity
SFT: 2 (∆-1, ✗) Sim 0.65 SLIM: 4 (∆+1, ✓) Sim 0.64
SFT: 3 (∆0, ✗) Sim 0.71 SLIM: 4 (∆+1, ✓) Sim 0.69
SFT: 2 (∆0, ✗) Sim 0.71 SLIM: 3 (∆+1, ✓) Sim 0.71
SFT: 3 (∆0, ✗) Sim 0.66 SLIM: 4 (∆+1, ✓) Sim 0.77
SFT: 2 (∆0, ✗) Sim 0.68 SLIM: 4 (∆+2, ✓) Sim 0.67
Table 15: Case study: steering H-bond donors with Feature F28089. Source HBD values: 3, 3, 2, 3, 2. SFT fails to increase donors (unchanged or decreased); SLIM steering adds +1 to +2.
15
[INST] Modify this molecule to increase its number of hydrogen bond acceptors: {SMILES} [/INST] SAE Feature F18533, semantics: piperidine 33%, ester 26%, phenyl 84%
Source
SFT
+SLIM
Property / Similarity
SFT: 3 (∆0, ✗) Sim 0.67 SLIM: 4 (∆+1, ✓) Sim 0.63
SFT: 2 (∆-2, ✗) Sim 0.79 SLIM: 6 (∆+2, ✓) Sim 0.79
SFT: 6 (∆0, ✗) Sim 0.68 SLIM: 7 (∆+1, ✓) Sim 0.75
SFT: 1 (∆-2, ✗) Sim 0.72 SLIM: 5 (∆+2, ✓) Sim 0.63
SFT: 6 (∆0, ✗) Sim 0.56 SLIM: 7 (∆+1, ✓) Sim 0.54
Table 16: Case study: steering H-bond acceptors with Feature F18533. Source HBA values: 3, 4, 6, 3, 6. SFT fails to increase acceptors (unchanged or decreased); SLIM steering adds +1 to +2.
16
Source
SFT
+SLIM
Source
SFT
+SLIM
-2.47
-5.36 (-2.89) Sim 0.75 1.32 (+3.79) Sim 0.64
4.51
1.82 (-2.69) Sim 0.65 7.82 (+3.31) Sim 0.79
3.81
1.89 (-1.92) Sim 0.75 7.03 (+3.22) Sim 0.77
6.39
3.02 (-3.38) Sim 0.67 9.44 (+3.05) Sim 0.74
5.92
3.25 (-2.67) Sim 0.65 8.93 (+3.01) Sim 0.62
5.48
2.81 (-2.67) Sim 0.71 8.49 (+3.01) Sim 0.74
3.05
0.46 (-2.58) Sim 0.65 5.86 (+2.82) Sim 0.75
5.76
3.10 (-2.66) Sim 0.71 8.57 (+2.80) Sim 0.83
Figure 5: SLIM steering examples: logP↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
Source
SFT
+SLIM
Source
SFT
+SLIM
542
496 (-46) Sim 0.67 733 (+192) Sim 0.69
589
473 (-116) Sim 0.62 729 (+140) Sim 0.61
550
494 (-56) Sim 0.67 677 (+126) Sim 0.77
490
393 (-97) Sim 0.63 614 (+124) Sim 0.73
541
436 (-104) Sim 0.72 665 (+124) Sim 0.75
536
481 (-55) Sim 0.75 660 (+124) Sim 0.72
496
391 (-104) Sim 0.71 614 (+118) Sim 0.82
310
280 (-30) Sim 0.70 424 (+114) Sim 0.74
Figure 6: SLIM steering examples: MW↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
17
Source
SFT
+SLIM
Source
SFT
+SLIM
4
4 (+0) Sim 0.67
13 (+9) Sim 0.72
9
6 (-3) Sim 0.65
15 (+6) Sim 0.81
10
9 (-1) Sim 0.62
15 (+5) Sim 0.77
5
4 (-1) Sim 0.76
9 (+4) Sim 0.65
8
5 (-3) Sim 0.76
11 (+3) Sim 0.74
10
7 (-3) Sim 0.77
13 (+3) Sim 0.70
4
4 (+0) Sim 0.70
6 (+2) Sim 0.68
6
5 (-1) Sim 0.67
8 (+2) Sim 0.77
Figure 7: SLIM steering examples: RotBond↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
Source
SFT
+SLIM
Source
SFT
+SLIM
4.19
4.50 (-0.31) Sim 0.70 2.67 (+1.52) Sim 0.60
3.24
3.42 (-0.17) Sim 0.77 2.62 (+0.62) Sim 0.67
2.66
3.77 (-1.11) Sim 0.71 2.16 (+0.51) Sim 0.43
3.14
3.57 (-0.43) Sim 0.74 2.83 (+0.30) Sim 0.58
3.67
3.95 (-0.28) Sim 0.76 3.41 (+0.26) Sim 0.89
3.33
3.56 (-0.23) Sim 0.73 3.19 (+0.13) Sim 0.77
3.65
4.02 (-0.37) Sim 0.73 3.62 (+0.03) Sim 0.72
3.48
3.54 (-0.06) Sim 0.74 3.45 (+0.03) Sim 0.69
Figure 8: SLIM steering examples: SA↓ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
18
Source
SFT
+SLIM
Source
SFT
+SLIM
5
5 (+0) Sim 0.69
12 (+7) Sim 0.75
5
3 (-2) Sim 0.72
9 (+4) Sim 0.53
6
4 (-2) Sim 0.76
9 (+3) Sim 0.69
4
2 (-2) Sim 0.69
7 (+3) Sim 0.62
6
4 (-2) Sim 0.78
9 (+3) Sim 0.69
2
2 (+0) Sim 0.68
5 (+3) Sim 0.75
2
2 (+0) Sim 0.68
5 (+3) Sim 0.62
5
5 (+0) Sim 0.70
8 (+3) Sim 0.76
Figure 9: SLIM steering examples: HBA↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
Source
SFT
+SLIM
Source
SFT
+SLIM
4
2 (-2) Sim 0.76
7 (+3) Sim 0.69
5
4 (-1) Sim 0.81
8 (+3) Sim 0.76
2
2 (+0) Sim 0.65
5 (+3) Sim 0.73
2
2 (+0) Sim 0.63
5 (+3) Sim 0.67
4
2 (-2) Sim 0.72
7 (+3) Sim 0.59
2
2 (+0) Sim 0.71
5 (+3) Sim 0.77
2
2 (+0) Sim 0.69
5 (+3) Sim 0.78
2
2 (+0) Sim 0.71
5 (+3) Sim 0.76
Figure 10: SLIM steering examples: HBD↑ (DrugAssist). Red highlights indicate structural modifications. Each case shows SFT failing while SLIM succeeds.
19
Source
SFT
+SLIM
Source
SFT
+SLIM
0.41
0.25 (-0.15) Sim 0.71 0.72 (+0.32) Sim 0.53
0.47
0.37 (-0.10) Sim 0.70 0.69 (+0.22) Sim 0.81
0.52
0.18 (-0.34) Sim 0.57 0.69 (+0.17) Sim 0.75
0.44
0.39 (-0.05) Sim 0.76 0.56 (+0.13) Sim 0.66
0.53
0.45 (-0.08) Sim 0.69 0.59 (+0.06) Sim 0.67
0.84
0.66 (-0.18) Sim 0.61 0.87 (+0.03) Sim 0.57
0.84
0.81 (-0.03) Sim 0.80 0.87 (+0.03) Sim 0.76
0.86
0.76 (-0.10) Sim 0.71 0.88 (+0.02) Sim 0.74
Figure 11: SLIM steering examples: QED↑ (GeLLM3O-Mistral).
Source
SFT
+SLIM
Source
SFT
+SLIM
0.17
0.04 (-0.13) Sim 0.76 0.46 (+0.29) Sim 0.56
0.05
0.02 (-0.03) Sim 0.71 0.24 (+0.19) Sim 0.70
0.20
0.19 (-0.01) Sim 0.62 0.36 (+0.16) Sim 0.63
0.04
0.02 (-0.02) Sim 0.72 0.16 (+0.12) Sim 0.69
0.15
0.10 (-0.05) Sim 0.60 0.26 (+0.11) Sim 0.55
0.16
0.03 (-0.13) Sim 0.58 0.27 (+0.11) Sim 0.72
0.02
0.01 (-0.01) Sim 0.69 0.13 (+0.11) Sim 0.77
0.04
0.04 (+0.00) Sim 0.72 0.14 (+0.10) Sim 0.72
Figure 12: SLIM steering examples: DRD2↑ (GeLLM3O-Mistral).
20
Source
SFT
+SLIM
Source
SFT
+SLIM
6.75
6.75 (-0.00) Sim 0.89 9.50 (+2.75) Sim 0.81
4.25
3.75 (-0.50) Sim 0.88 6.96 (+2.71) Sim 0.78
5.73
5.49 (-0.25) Sim 0.83 8.17 (+2.44) Sim 0.78
4.05
2.25 (-1.81) Sim 0.77 6.49 (+2.44) Sim 0.85
8.69
7.73 (-0.97) Sim 0.71 11.09 (+2.39) Sim 0.69
5.84
3.18 (-2.65) Sim 0.66 8.14 (+2.30) Sim 0.73
8.12
7.48 (-0.64) Sim 0.85 10.29 (+2.16) Sim 0.82
3.61
2.63 (-0.99) Sim 0.68 5.75 (+2.14) Sim 0.65
Figure 13: SLIM steering examples: logP↑ (GeLLM3O-Mistral).
Source
SFT
+SLIM
Source
SFT
+SLIM
592
535 (-57) Sim 0.79 716 (+125) Sim 0.71
555
507 (-48) Sim 0.64 679 (+124) Sim 0.68
371
305 (-66) Sim 0.67 482 (+111) Sim 0.68
596
567 (-29) Sim 0.81 707 (+111) Sim 0.80
458
424 (-34) Sim 0.76
550 (+91) Sim 0.61
656
554 (-102) Sim 0.80 742 (+86) Sim 0.72
583
498 (-85) Sim 0.75
669 (+86) Sim 0.79
521
392 (-129) Sim 0.67 601 (+80) Sim 0.81
Figure 14: SLIM steering examples: MW↑ (GeLLM3O-Mistral).
21
Source
SFT
+SLIM
Source
SFT
+SLIM
8
8 (+0) Sim 0.75
24 (+16) Sim 0.43
9
8 (-1) Sim 0.76
18 (+9) Sim 0.82
7
5 (-2) Sim 0.83
13 (+6) Sim 0.69
6
5 (-1) Sim 0.77
12 (+6) Sim 0.66
9
8 (-1) Sim 0.71
14 (+5) Sim 0.88
8
5 (-3) Sim 0.67
13 (+5) Sim 0.63
4
2 (-2) Sim 0.79
9 (+5) Sim 0.72
12
11 (-1) Sim 0.77
16 (+4) Sim 0.78
Figure 15: SLIM steering examples: RotBond↑ (GeLLM3O-Mistral).
Source
3.80
SFT
+SLIM
Source
3.82 (-0.02) Sim 0.79 3.65 (+0.15) Sim 0.75
3.28
SFT
3.30 (-0.02) Sim 0.83 3.27 (+0.00) Sim 0.70
Figure 16: SLIM steering examples: SA↓ (GeLLM3O-Mistral).
22
+SLIM
Source
SFT
+SLIM
Source
SFT
+SLIM
11
11 (+0) Sim 0.79
14 (+3) Sim 0.80
4
4 (+0) Sim 0.76
7 (+3) Sim 0.71
5
4 (-1) Sim 0.64
8 (+3) Sim 0.44
5
5 (+0) Sim 0.79
7 (+2) Sim 0.78
4
4 (+0) Sim 0.74
6 (+2) Sim 0.71
6
5 (-1) Sim 0.83
8 (+2) Sim 0.46
4
4 (+0) Sim 0.78
6 (+2) Sim 0.82
5
5 (+0) Sim 0.72
7 (+2) Sim 0.60
Figure 17: SLIM steering examples: HBA↑ (GeLLM3O-Mistral).
Source
SFT
+SLIM
Source
SFT
+SLIM
2
2 (+0) Sim 0.82
7 (+5) Sim 0.65
3
3 (+0) Sim 0.62
7 (+4) Sim 0.45
2
2 (+0) Sim 0.68
5 (+3) Sim 0.57
2
2 (+0) Sim 0.70
5 (+3) Sim 0.62
2
2 (+0) Sim 0.76
4 (+2) Sim 0.57
2
2 (+0) Sim 0.69
4 (+2) Sim 0.75
3
3 (+0) Sim 0.75
5 (+2) Sim 0.88
2
2 (+0) Sim 0.79
4 (+2) Sim 0.73
Figure 18: SLIM steering examples: HBD↑ (GeLLM3O-Mistral).
23
Source
SFT
+SLIM
Source
SFT
+SLIM
0.22
0.20 (-0.02) Sim 0.80 0.52 (+0.30) Sim 0.69
0.81
0.76 (-0.05) Sim 0.65 0.84 (+0.03) Sim 0.69
0.87
0.78 (-0.09) Sim 0.62 0.89 (+0.02) Sim 0.73
0.34
0.29 (-0.05) Sim 0.87 0.36 (+0.02) Sim 0.63
0.84
0.72 (-0.12) Sim 0.66 0.84 (+0.00) Sim 0.46
0.81
0.78 (-0.03) Sim 0.80 0.81 (+0.00) Sim 0.84
Figure 19: SLIM steering examples: QED↑ (GeLLM3O-LLaMA3).
Source
SFT
+SLIM
Source
SFT
+SLIM
0.16
0.06 (-0.10) Sim 0.61 0.43 (+0.27) Sim 0.64
0.02
0.00 (-0.02) Sim 0.61 0.24 (+0.22) Sim 0.56
0.03
0.03 (+0.00) Sim 0.74 0.22 (+0.19) Sim 0.62
0.03
0.01 (-0.02) Sim 0.71 0.16 (+0.13) Sim 0.74
0.14
0.09 (-0.05) Sim 0.64 0.23 (+0.09) Sim 0.74
0.15
0.14 (-0.01) Sim 0.63 0.23 (+0.08) Sim 0.56
0.08
0.06 (-0.02) Sim 0.79 0.15 (+0.07) Sim 0.72
0.16
0.08 (-0.08) Sim 0.64 0.22 (+0.06) Sim 0.62
Figure 20: SLIM steering examples: DRD2↑ (GeLLM3O-LLaMA3).
24
Source
SFT
+SLIM
Source
SFT
+SLIM
0.46
-0.96 (-1.42) Sim 0.76 5.25 (+4.79) Sim 0.40
12.21
9.74 (-2.47) Sim 0.59 14.89 (+2.67) Sim 0.86
5.36
5.11 (-0.25) Sim 0.44 7.84 (+2.48) Sim 0.81
8.12
6.11 (-2.01) Sim 0.82 10.29 (+2.16) Sim 0.82
5.46
3.34 (-2.12) Sim 0.81 7.20 (+1.74) Sim 0.56
5.24
5.15 (-0.09) Sim 0.68 6.87 (+1.63) Sim 0.52
8.69
6.37 (-2.33) Sim 0.77 10.18 (+1.48) Sim 0.72
6.77
6.52 (-0.25) Sim 0.80 8.13 (+1.37) Sim 0.74
Figure 21: SLIM steering examples: logP↑ (GeLLM3O-LLaMA3).
Source
SFT
+SLIM
Source
SFT
+SLIM
366
270 (-96) Sim 0.73 626 (+259) Sim 0.54
500
466 (-34) Sim 0.71 688 (+188) Sim 0.69
472
347 (-125) Sim 0.62 648 (+176) Sim 0.53
480
379 (-100) Sim 0.48 647 (+167) Sim 0.52
516
297 (-218) Sim 0.53 665 (+149) Sim 0.86
467
453 (-14) Sim 0.80 608 (+141) Sim 0.74
534
534 (+0) Sim 0.79
672 (+138) Sim 0.83
759
701 (-58) Sim 0.79 895 (+136) Sim 0.80
Figure 22: SLIM steering examples: MW↑ (GeLLM3O-LLaMA3).
25
Source
SFT
+SLIM
Source
SFT
+SLIM
6
6 (+0) Sim 0.58
10 (+4) Sim 0.61
6
6 (+0) Sim 0.42
9 (+3) Sim 0.64
12
10 (-2) Sim 0.78
15 (+3) Sim 0.58
11
10 (-1) Sim 0.65
14 (+3) Sim 0.58
8
5 (-3) Sim 0.52
11 (+3) Sim 0.52
12
11 (-1) Sim 0.80
15 (+3) Sim 0.50
6
5 (-1) Sim 0.58
8 (+2) Sim 0.54
9
8 (-1) Sim 0.79
11 (+2) Sim 0.65
Figure 23: SLIM steering examples: RotBond↑ (GeLLM3O-LLaMA3).
Source
SFT
+SLIM
Source
SFT
+SLIM
4
3 (-1) Sim 0.49
8 (+4) Sim 0.65
4
4 (+0) Sim 0.82
7 (+3) Sim 0.73
7
6 (-1) Sim 0.78
9 (+2) Sim 0.78
8
7 (-1) Sim 0.87
10 (+2) Sim 0.79
7
5 (-2) Sim 0.64
9 (+2) Sim 0.50
7
5 (-2) Sim 0.72
9 (+2) Sim 0.70
5
5 (+0) Sim 0.76
7 (+2) Sim 0.55
6
4 (-2) Sim 0.67
8 (+2) Sim 0.69
Figure 24: SLIM steering examples: HBA↑ (GeLLM3O-LLaMA3).
26
Source
SFT
+SLIM
Source
SFT
+SLIM
3
2 (-1) Sim 0.81
4 (+1) Sim 0.68
3
2 (-1) Sim 0.58
4 (+1) Sim 0.50
2
2 (+0) Sim 0.65
3 (+1) Sim 0.65
2
2 (+0) Sim 0.76
3 (+1) Sim 0.60
2
1 (-1) Sim 0.40
3 (+1) Sim 0.42
2
2 (+0) Sim 0.79
3 (+1) Sim 0.66
3
1 (-2) Sim 0.69
4 (+1) Sim 0.77
2
1 (-1) Sim 0.62
3 (+1) Sim 0.61
Figure 25: SLIM steering examples: HBD↑ (GeLLM3O-LLaMA3).
Source
SFT
+SLIM
Source
SFT
+SLIM
0.21
0.05 (-0.16) Sim 0.46 0.74 (+0.53) Sim 0.37
0.33
0.25 (-0.08) Sim 0.33 0.79 (+0.46) Sim 0.15
0.17
0.11 (-0.07) Sim 0.51 0.61 (+0.44) Sim 0.21
0.43
0.04 (-0.39) Sim 0.28 0.62 (+0.19) Sim 0.16
0.34
0.19 (-0.15) Sim 0.33 0.44 (+0.10) Sim 0.37
0.32
0.30 (-0.02) Sim 0.22 0.41 (+0.10) Sim 0.22
0.56
0.19 (-0.37) Sim 0.17 0.61 (+0.04) Sim 0.17
0.18
0.12 (-0.06) Sim 0.51 0.22 (+0.04) Sim 0.20
Figure 26: SLIM steering examples: QED↑ (MolGen-Large).
27
Source
SFT
+SLIM
Source
SFT
+SLIM
0.08
0.08 (+0.00) Sim 0.28 0.23 (+0.15) Sim 0.32
0.25
0.13 (-0.12) Sim 0.25 0.39 (+0.14) Sim 0.21
0.15
0.06 (-0.09) Sim 0.35 0.27 (+0.12) Sim 0.17
0.11
0.01 (-0.10) Sim 0.24 0.20 (+0.09) Sim 0.18
0.23
0.06 (-0.17) Sim 0.23 0.30 (+0.07) Sim 0.22
0.08
0.05 (-0.03) Sim 0.51 0.14 (+0.06) Sim 0.27
0.14
0.07 (-0.07) Sim 0.51 0.19 (+0.05) Sim 0.42
0.14
0.11 (-0.03) Sim 0.20 0.18 (+0.04) Sim 0.17
Figure 27: SLIM steering examples: DRD2↑ (MolGen-Large).
Source
SFT
+SLIM
Source
SFT
+SLIM
7.36
6.40 (-0.96) Sim 0.51 12.03 (+4.67) Sim 0.45
0.66
-5.13 (-5.78) Sim 0.16 4.00 (+3.34) Sim 0.19
7.04
3.32 (-3.72) Sim 0.28 9.68 (+2.64) Sim 0.32
2.56
-2.28 (-4.85) Sim 0.21 4.96 (+2.40) Sim 0.20
1.99
-0.92 (-2.91) Sim 0.17 4.34 (+2.35) Sim 0.19
2.88
1.66 (-1.22) Sim 0.18 4.98 (+2.11) Sim 0.29
8.06
6.61 (-1.45) Sim 0.18 9.94 (+1.88) Sim 0.15
5.39
3.73 (-1.65) Sim 0.68 6.90 (+1.52) Sim 0.27
Figure 28: SLIM steering examples: logP↑ (MolGen-Large).
28
Source
SFT
+SLIM
Source
SFT
+SLIM
450
251 (-199) Sim 0.24 738 (+288) Sim 0.18
453
216 (-237) Sim 0.28 738 (+285) Sim 0.32
514
232 (-281) Sim 0.23 792 (+278) Sim 0.24
453
323 (-129) Sim 0.23 718 (+265) Sim 0.28
726
679 (-47) Sim 0.18 840 (+114) Sim 0.19
576
556 (-20) Sim 0.30
609
561 (-48) Sim 0.68
656 (+47) Sim 0.15
759
616 (-143) Sim 0.28 787 (+28) Sim 0.28
625 (+49) Sim 0.35
Figure 29: SLIM steering examples: MW↑ (MolGen-Large).
Source
SFT
+SLIM
Source
SFT
+SLIM
11
7 (-4) Sim 0.23
47 (+36) Sim 0.21
11
4 (-7) Sim 0.24
21 (+10) Sim 0.24
10
8 (-2) Sim 0.15
16 (+6) Sim 0.55
10
3 (-7) Sim 0.18
16 (+6) Sim 0.19
11
3 (-8) Sim 0.28
16 (+5) Sim 0.32
13
8 (-5) Sim 0.23
16 (+3) Sim 0.27
14
6 (-8) Sim 0.30
16 (+2) Sim 0.33
7
5 (-2) Sim 0.35
9 (+2) Sim 0.29
Figure 30: SLIM steering examples: RotBond↑ (MolGen-Large).
29
Source
2.65
SFT
+SLIM
Source
3.65 (-1.00) Sim 0.51 2.06 (+0.59) Sim 0.21
3.23
SFT
+SLIM
5.23 (-2.00) Sim 0.30 2.70 (+0.52) Sim 0.38
Figure 31: SLIM steering examples: SA↓ (MolGen-Large).
Source
SFT
+SLIM
Source
SFT
+SLIM
8
7 (-1) Sim 0.27
11 (+3) Sim 0.20
3
3 (+0) Sim 0.29
5 (+2) Sim 0.28
8
6 (-2) Sim 0.15
10 (+2) Sim 0.61
3
2 (-1) Sim 0.28
4 (+1) Sim 0.37
6
6 (+0) Sim 0.68
7 (+1) Sim 0.26
5
5 (+0) Sim 0.29
6 (+1) Sim 0.32
Figure 32: SLIM steering examples: HBA↑ (MolGen-Large).
Source
SFT
+SLIM
Source
SFT
+SLIM
4
3 (-1) Sim 0.23
7 (+3) Sim 0.27
2
1 (-1) Sim 0.28
4 (+2) Sim 0.38
3
3 (+0) Sim 0.32
5 (+2) Sim 0.20
2
2 (+0) Sim 0.24
4 (+2) Sim 0.22
2
1 (-1) Sim 0.15
4 (+2) Sim 0.51
3
3 (+0) Sim 0.19
5 (+2) Sim 0.22
2
2 (+0) Sim 0.22
3 (+1) Sim 0.18
3
2 (-1) Sim 0.51
4 (+1) Sim 0.43
Figure 33: SLIM steering examples: HBD↑ (MolGen-Large).
30