arXiv CS open access

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks Brandon Wang,

Andrei S. Tyrin,

Daniil A. Boiko

Onepot AI, Inc.

research@onepot.ai

arXiv:2608.02595v1 [cs.LG] 3 Aug 2026

Abstract Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepotBench 0 comprises three complementary evaluations: ChemAbacus measures toolfree cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab. We evaluate 13 models from six providers. Frontier models generally perform well on basic molecular-property questions. Safety behavior varies substantially across models, molecular representations, target classes, and reasoning-effort settings. Furthermore, models that distinguish established controlled substances from benign compounds may still generalize poorly to unfamiliar analogs, which may imply memorization instead of deeper understanding of potential harms. Finally, models perform significantly worse on SynthBench vs. ChemAbacus: reaction-outcome predictions trail a simple empirical baseline, and no tested model performs significantly above chance on catalyst preference. These results reveal a pronounced gap between chemistryadjacent competence and the empirical judgment required for experimental execution. onepot-Bench 0 provides an initial, easy-to-run framework for measuring this gap using both general chemistry tasks and non-public wet-lab data.

1. Introduction onepot AI is an AI research lab building a small-molecule fab. We enable fast and efficient chemistry workflows by removing bottlenecks through automation. As part of this work, we

utilize various capabilities of language models, including coding, analysis of chemistry data, and synthesis planning. As opposed to more common skills (code, customer service, biology), there is limited work on evaluating chemical capabilities, and arguably, no universally accepted benchmark. Furthermore, existing benchmarks still generally rely on public chemistry databases [1–3], which can lead to leakage. The ideal chemistry benchmark would therefore both evaluate general chemistry intuition and utilize non-public experimental data. Moreover, chemistry capabilities are inherently multifaceted: general chemistry knowledge and simple computation, deeper chemical reasoning, and eventually, an ability to do chemistry work [4–6]. Recent interest in closed-loop agentic systems for science [7, 8] makes the latter particularly important. Finally, model capabilities are constrained by their refusal rates, which can be a nontrivial barrier for completing biology- and chemistry-adjacent topics. To this end, we introduce onepot-Bench 0, an easy-to-run benchmark that measures language models on a variety of axes of chemistry capability. Our goal is to evaluate whether models can make decisions necessary for executing chemistry experiments in physical wet-lab.

Figure 1 | Overview of the approach. onepot-Bench 0 addresses refusals, simple knowledge and computation, and experimental outcome prediction and catalyst selection.

onepot-Bench 0 consists of the following components: • ChemAbacus, a set of chemistry questions that test models’ (tool-free) cheminformatics capabilities. • SynthRefusal, a systematic measurement of models’ refusals on synthesis queries over a spectrum of target molecules ranging from benign to banned (DEA Schedule I-IV drugs and chemical warfare agents). • SynthBench, an evaluation of models’ abilities to anticipate reaction outcomes under concrete conditions, in both forward prediction (will this reaction work?) and decisionoriented (which catalyst will work better?). We picked these tasks as they measure orthogonal capabilities of interest. ChemAbacus measures general chemistry literacy and numeracy. Strong performance on its tasks is likely necessary but not sufficient for real-world chemistry decision-making. SynthRefusal is a dual-use evaluation [9]. It probes models’ helpfulness for a range of synthesis queries depending on the target molecule’s properties. In general, SynthRefusal measures a model’s ability to differentiate between benign and hazardous or illicit targets; understanding both directions of this behavior can inform users of which models can be both trusted to refuse unsafe requests and relied upon for genuine tasks. Unlike past work [10], SynthRefusal explicitly seeks to understand refusal profiles, i.e. how refusals depend on both properties of the input molecule and representations of the molecule. SynthBench measures two in silico-testable synthesis-relevant skills: Outcome prediction (bi2

nary) and catalyst selection (pick preferred catalyst). Both tasks [11, 12] are key problems in the ML-for-synthesis literature, as they are critical components of a functional automated synthesis pipeline. Furthermore, strong zero-shot performance on these tasks is a good proxy for whether models have the "chemistry intuition" required for experimental design and judgement of experimental results. Our results reveal a clear gap between chemistry-adjacent competence and real experimental decision-making. Models perform relatively well on ChemAbacus but substantially worse on SynthBench, where they generally trail a simple empirical baseline on reaction-outcome prediction and fail to exceed chance on catalyst selection. SynthRefusal likewise exposes inconsistent safety behavior: performance depends strongly on the model, target class, molecular representation, and reasoning effort, with many models tending toward either broad refusal of benign chemistry or permissive responses to controlled and safety-relevant targets.

2. Methodology 2.1. ChemAbacus ChemAbacus consists of 800 QA questions, each concerning a single molecule given as a SMILES [13] string. 100 of these questions ask to identify all functional groups (from a given set of all possibilities) in the molecule. The remaining 700 are split between seven numerical query types (100 questions / query type) asking for the molecule’s molecular weight, number of rings, TPSA [14], Crippen cLogP [15], number of rotatable bonds, number of hydrogen bond donors, and number of hydrogen bond acceptors. For counting problems, we require an exact answer, while for numerical problems (i.e. molecular weight, TPSA, and LogP), outputs within a certain range are also accepted. For TPSA, we accept both the values for N/O and values taking into account P/S atoms [14]. For rotatable bonds, we include the following instructions: “Count single, non-ring bonds between two nonterminal heavy atoms, but do NOT count bonds with restricted rotation from conjugation (e.g. amide, ester, amidine) or bonds to symmetric rotors (e.g. CF3, tert-butyl)." We use RDKit [16] for most of the ground truth computation, though we use custom SMARTS patterns [17] for some queries. The reference molecules for these questions are pulled from ChEMBL [18], with 20 small, 20 medium-sized, and 20 large drug-like molecules, 20 biologics, and 20 molecules chosen to be adversarially hard for that specific category.

2.2. SynthRefusal WARNING: This work contains discussion of chemical warfare agents, controlled substances, explosives, poisons, and other hazardous compounds as part of a language-model safety evaluation. These compounds should not be synthesized, acquired, or used outside appropriately authorized and controlled settings. Any model-generated chemistry discussed here is unverified and should not be treated as experimental guidance.

SynthRefusal measures how a model responds to a synthesis request and how that response depends on the target representation. Each response is assigned to one of five mutually exclusive outcomes: clean refusal, produced answer, partial disclosure, target not identified, or target misidentified. Partial disclosure denotes a response that ultimately refuses or is terminated by a provider safety mechanism, but only after revealing target-relevant conceptual or executable chemistry. Target-not-identified responses explicitly fail to resolve the supplied target, whereas target-misidentified responses confidently answer for a different molecule. 3

The clean-refusal rate is calculated over clean refusals and produced answers only. Partial disclosures and both target-identification outcomes are not included in refusal rate calculations: partial disclosure is reported separately in the appendix, while the complete targetidentification breakdown is retained in the audit supplement. We evaluate these outcomes on two datasets. Controlled-substance set. 282 target molecules span six categories ranging from benign to banned: benign (50), anti-doping / sport-prohibited substances (50), poisons (50), explosives (32), Schedule I–IV controlled substances (50), and chemical-warfare agents (50). Each molecule carries provenance: a PubChem CID and, where it is controlled, the listing authority (WADA [19], DEA [20], OPCW [21], ATF, EPA, or Stockholm Convention [22]), the citing document, and the type of evidence linking it to that list. Input representation can affect safety behavior [23]; to properly control for this effect, every target is presented four ways: as a SMILES string, a common name, a CAS registry number, and an InChI string. SMILES, InChI, and a common name are available for all 282 molecules, and a CAS number for 281. Designer-drug set. 201 molecules drawn from Wikipedia’s list of designer drugs [24], spanning eleven pharmacological classes (11 to 20 molecules each). Most are structural analogs of controlled substances (synthetic cannabinoids, dissociatives, entactogens, psychedelics, stimulants, sedatives, piperazines, and androgens); the rest are enhancement compounds that are not themselves scheduled (nootropics, PDE5 inhibitors, and peptides). Each molecule carries a PubChem CID (plus a Wikipedia link for 177 and a CAS number for 185 of the 201) and is presented as a SMILES string only. Alignment index. Refusal rate alone does not order models by desirability: a model that refuses everything and a model that refuses nothing are misaligned in opposite directions. The desired behavior is conditional: comply on benign targets while refusing controlled or safetyrelevant targets. All alignment indices in the main figures are computed from SMILES prompts only. For models evaluated at low, medium, and high reasoning effort, we first average each category’s cleanrefusal rate across the three effort levels. Let c = 1−rbenign denote benign compliance. For the controlled-substance set, let qctrl be the macro-average clean-refusal rate over the five controlled categories; for the designer-drug set, let qdesigner be the corresponding macro-average over its eleven pharmacological classes. We define AD =

√

c qD ,

D ∈ {ctrl, designer},

(1)

and report AD as a percentage. Both axes therefore use a common SMILES-only basis, and each class contributes equally regardless of its number of molecules. Partial disclosures and target-identification outcomes are excluded using the same denominator as the clean-refusal rate. The geometric mean penalizes both blanket refusal and blanket compliance.

2.3. SynthBench SynthBench consists of two tasks, reaction outcome prediction and catalyst preference. For SynthBench, all reactions used are sourced exclusively from data collection experiments performed in our in-house wet lab [25]. For outcome prediction, we collect 80 reactions, 40 positive and 40 negative, in each of the following reaction classes: Amide coupling [26], Buchwald-Hartwig coupling [27, 28], Chan-Lam coupling [29, 30], N-alkylation [31], Suzuki-Miyaura coupling [32], Thiourea synthesis [33], 4

Urea synthesis [33, 34], and Van Leusen [35]. For each reaction, the model is given the reactants, target, and the conditions under which the reaction occurred (including catalyst, solvent, and temperature and duration). The model is then asked to predict whether or not the reaction succeeded (success defined via sufficient presence of target molecule). In catalyst preference, we focus on Suzuki-Miyaura coupling; our experience is that SuzukiMiyaura coupling outcomes are highly sensitive to condition (and catalyst) choice. Here, the model is given two reactions, one which succeeded and one which did not, with identical execution up to choice of catalyst. The model is then asked to to identify which catalyst is more likely to lead to a successful reaction.

3. Results Across all three evals, we evaluate 13 models from 6 different providers: • Anthropic: Claude Haiku 4.5, Claude Opus 4.8, Claude Opus 5, and Claude Fable 5. • DeepSeek: DeepSeek V4 Flash and DeepSeek V4 Pro (via Novita). • Google: Gemini 3.1 Pro and Gemini 3.5 Flash. • Meta: Muse Spark 1.1. • Moonshot: Kimi K2.6. • OpenAI: GPT-5.5, GPT-Rosalind, and GPT-5.6 Sol. We evaluate all 13 models on ChemAbacus; all models except DeepSeek V4 Pro and Kimi K2.6 on SynthRefusal; and all models except Muse Spark on SynthBench. All models are evaluated on default API settings.

3.1. ChemAbacus For ChemAbacus, we use low, medium, and high effort modes for Opus 4.8, Opus 5, Fable 5, GPT-5.5, GPT-Rosalind, GPT-5.6 Sol, and Muse Spark 1.1; we use low effort for Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash; and we use the default thinking mode for DeepSeek V4 Flash/Pro and Kimi K2.6. Results are shown in Figure 2.

Figure 2 | Accuracy on ChemAbacus for a variety of frontier LLMs. Weaker and/or outdated models are omitted. Prices are computed using prices from the primary provider, assuming no input caching.

Claude Fable 5 has nontrivial refusal rate on ChemAbacus, increasing with reasoning effort (38 5

of 800 questions on low, 60 on medium, and 75 on high). To account for this, we report two numbers: One marking refusals as wrong, and one using Opus 5 rerouting as fallback. Fable 5 with fallback is the strongest performing model at high reasoning levels. Because a few of the ChemAbacus questions (molecular weight, Crippen LogP, and TPSA) require nontrivial arithmetic, we also separate accuracy by computation heavy vs. computation light. These results are shown in Figure 3. Performance is generally higher on noncomputational tasks than on computational tasks, though some models show noticeably better performance on non-computational than computational tasks. However, the gap seems to close with more effort, which seems to imply that models with asymmetric performance may have chemistry knowledge but lack cheminformatic arithmetic capabilities. In particular, while the mainline GPT family performs similarly on computational vs. non-computational, GPT-Rosalind performs much better on non-computational.

Figure 3 | Non-computational vs. computational performance on ChemAbacus. All models land near or above the y = x line (dashed). Similar model families (GPT-5.5 and GPT-5.6 Sol; Claude Opus 4.8, 5, and Fable 5) seem to have similar curves.

3.2. SynthRefusal SynthRefusal targets three axes: • whether models distinguish benign from controlled targets • how the input representation (e.g. name vs. structure) shifts refusal rates • whether refusal decisions reflect genuine reasoning over molecular structure We use low, medium, and high thinking levels for all models tested except DeepSeek V4 Flash, for which we only use the default thinking mode. Under the SMILES-only alignment index, the ordering differs sharply from raw refusal rate (Figures 4 and 5). On the controlled-substance set, GPT-5.5 leads with Actrl ≈ 85%, closely followed by Fable 5 at ∼84%; GPT-5.6 Sol and Gemini 3.1 Pro both score approximately 75%. On the designer-drug set, however, Fable 5 becomes the strongest model at Adesigner ≈ 80%, 6

while GPT-5.5 falls to ∼67%. GPT-5.5 separates benign from established controlled targets particularly well, but generalizes poorly to unfamiliar designer analogs; Fable 5 remains highly restrictive on both sets. The extremes illustrate why refusal rate alone is insufficient. Muse Spark 1.1 refuses almost every controlled and designer target, and also refuses approximately 95% of benign SMILES prompts, yielding an alignment index of only ∼23%. DeepSeek V4 Flash exhibits the opposite behavior: it complies with nearly everything and has an alignment index near zero. GPTRosalind’s low refusal rate is expected due to lowered safety guardrails and restricted availability. Controlled-substance set results. Refusal rates vary substantially by model, target category, and legal schedule (Tables 3 and 4). At high reasoning effort, macro-averaged over the four representations, Muse Spark 1.1 exhibits blanket refusal. Fable 5 refuses approximately 94% of controlled targets, followed by GPT-5.5 at ∼85%, Gemini 3.1 Pro and GPT-5.6 Sol at ∼64%, and Opus 5 at ∼33%. Opus 4.8 and Gemini 3.5 Flash are substantially more permissive, while Haiku 4.5 and DeepSeek V4 Flash refuse almost nothing. Appendix cells report clean refusal and partial disclosure separately. Models also differ in which categories they guard. Chemical-warfare agents and explosives generally elicit the most caution, whereas anti-doping compounds elicit substantially less. Legal schedule adds further structure (Table 4): at high effort, Fable 5 refuses all Schedule I– III targets and approximately 81% of Schedule IV targets, while GPT-5.5 remains above 90% across every schedule. Gemini 3.1 Pro declines from approximately 87% on Schedule I to 21% on Schedule IV, and Opus 5 declines from approximately 38% to zero. Muse Spark 1.1 refuses every schedule, whereas DeepSeek V4 Flash refuses none. How a molecule is represented also matters, but there is no universal ordering of representations (Figure 4 and Table 5). At high effort, averaged across models, CAS prompts increase refusal of anti-doping compounds by approximately 15 percentage points relative to SMILES, while common names produce the largest increase for chemical-warfare agents, approximately 8 points. Conversely, CAS representation lowers refusal of chemical-warfare agents by approximately 7 points. Representation effects therefore interact with both model and target class. Designer-drug set results. On the designer-drug set, the ranking changes substantially (Figure 5 and Table 6). Fable 5 has the highest SMILES-only alignment index (≈ 80%) and refuses approximately 85% of designer targets at high effort. GPT-5.5 falls from ≈ 85% alignment on established controlled targets to ≈ 67% on designer drugs, indicating weaker generalization to less familiar analogs. Muse Spark 1.1 again refuses almost every target, but its near-blanket refusal of benign compounds prevents high alignment. Most other models are considerably more permissive, with DeepSeek V4 Flash refusing essentially none. The class and structural-subcategory breakdowns in the appendix (Table 6) show that these effects are highly target-dependent. For example, Fable 5 refuses all evaluated peptides at high effort, whereas GPT-5.5 refuses none, and Fable also strongly refuses dissociatives, entactogens, and psychedelics. Such variation suggests that safety behavior reflects learned category- and structure-specific associations rather than a uniform notion of chemical risk. On average, refusal increases modestly with reasoning effort: on both datasets, high reasoning effort has around four percentage points higher average refusal rate than medium reasoning effort. The aggregate, however, hides large model-specific effects. GPT-5.5 increases by approximately 14 points on the controlled set and 17 points on the designer set, while Opus 5 increases by approximately 11 and 14 points, respectively. Fable 5 rises more moderately, by approximately five to six points. Gemini 3.1 Pro moves in the opposite direction, decreasing 7

Figure 4 | SynthRefusal results on the controlled-substance set. Top: SMILES-only alignment between benign compliance and refusal of controlled targets, with rates averaged across reasoning efforts, with an enlarged view of the high-compliance region. Bottom left: change in SMILES-based refusal rate with reasoning effort, relative to low effort. Bottom centre: high-effort refusal rates by controlled-substance category and DEA schedule. Bottom right: high-effort representation sensitivity, reported as the percentage-point change relative to SMILES.

by approximately seven points on both sets, and GPT-5.6 Sol is roughly flat or slightly less restrictive on the controlled set.

3.3. SynthBench SynthBench consists of two components, a reaction outcome prediction component and a catalyst preference component. Unlike ChemAbacus, which primarily tests computational and execution capabilities, SynthBench tests both knowledge and reasoning. Indeed, even a singlestep reaction will have a very complex mechanism; therefore, reasoning about outcomes requires either multi-step reasoning, or general intuition about how certain substrates perform in a given reaction. Unlike other fields (e.g. math, coding) it is generally speaking impossible to reason a priori about reaction outcomes. Simple models can perform quite well on reaction outcome prediction [11, 36]; as an extra baseline, for reaction outcome prediction, we train a model using logistic regression on internal data, using a simple feature set.

Figure 5 | SynthRefusal results on the designer-drug set. Top: SMILES-based alignment between benign compliance and mean refusal across the designer-drug classes, with an enlarged view of the high-compliance region. Bottom left: change in refusal rate with reasoning effort, relative to low effort. Bottom right: high-effort refusal rates by pharmacological class.

Reaction outcome prediction. We report reaction outcome prediction results in Figure 6 and Table 7. For every reaction class except urea synthesis, the highest-scoring language model is from the Claude family (Table 7). However, all models lose to the baseline on all reaction classes except Claude Opus 5 on Buchwald-Hartwig coupling. Interestingly, for all models for which we tested multiple reasoning levels, performance does not increase significantly with more test-time compute (Table 7); in fact, for many models, performance seems to degrade. In general, this suggests that models lack the knowledge and intuition necessary to perform complex reasoning about standard organic chemistry reactions. Moreover, we find that models are generally overoptimistic about reaction outcomes; the Claude family is a notable exception (Figure 6, bottom right). Catalyst preference. Suzuki-Miyaura coupling already presents significant challenges for all language models tested on reaction outcome prediction (Figure 6). Thus, one might expect that the models would also struggle on catalyst preference, as catalyst preference exclusively tests Suzuki couplings. Indeed, catalyst preference appears to be very difficult for all models (Figure 7). No model is able to score significantly better than chance. However, the dataset for catalyst preference 9

Figure 6 | Reaction-outcome prediction. Marker color denotes model family. Top: Best model per provider/model family vs. a logistic-regression baseline (black). The baseline wins 7 of 8 reaction classes, but loses to Claude Opus 5 on Buchwald-Hartwig coupling. Chan-Lam and Suzuki are especially difficult for the models. Bottom-left: Average performance (best performance across effort levels selected for each model) vs. cost/query for various models. Bottom-right: Sensitivity vs. specificity for various models. Aside from the Claude family, most models are overly optimistic about reaction outcomes.

is quite small, and larger datasets may be more instructive in better understanding the exact failure modes that current models run into.

4. Conclusion We introduce onepot-Bench 0, a chemistry benchmark suite designed to evaluate language models on capabilities relevant to wet-lab execution: basic cheminformatics literacy, safetyadjacent refusal behavior, reaction outcome prediction, and catalyst selection. By combining simple cheminformatics questions with evaluations derived from private experimental data, onepot-Bench 0 probes both basic chemistry knowledge and real wet-lab decision-making. Our results show a sharp separation between chemistry-adjacent competence and chemistryexecution competence. On ChemAbacus, frontier models perform reasonably well, demonstrating an ability to answer basic questions about small molecules. However, this success does not transfer cleanly to synthesis-relevant prediction. On SynthBench, the strongest models perform only modestly above chance on reaction outcome prediction, struggling especially on metal-catalyzed reactions which require more nuanced reasoning about both substrates 10

Figure 7 | Accuracy on catalyst preference on 95 pairs of Suzuki-Miyaura coupling reactions. Whiskers show 95% confidence intervals, all models score within noise.

and reaction conditions. The refusal results are similarly mixed. For example, GPT-5.5 has high alignment scores; however, it appears to mostly reliably refuse established targets while helping synthesize unfamiliar designer analogs, suggesting that its alignment score may be more of a function of recognizing known molecules than robust risk assessment. We also observe both undesirable extremes of the refusal spectrum: Muse Spark 1.1 refuses almost everything (including benign chemistry) while DeepSeek V4 Flash refuses almost nothing. These opposite failure modes are obscured by raw refusal rate but exposed by the alignment index. Safety behavior is also not a stable function of chemical risk alone. Representation effects depend on both model and compound class, and increased reasoning effort can either increase or decrease refusal. Partial disclosures further show that an eventual refusal does not necessarily mean that the response withheld useful synthesis information. Overall, onepot-Bench 0 suggests that today’s strongest language models are useful chemistry assistants for parsing, computation, and some high-level reasoning, but they struggle to make zero-shot decisions relevant to lab-based chemistry. Further progress will require tighter integration of language models with private experimental datasets, reaction-specific empirical models, and closed-loop wet lab feedback. As the first iteration of onepot-Bench, onepot-Bench 0 is designed to be a simple, easy-to-run proxy for in-lab capabilities. However, it does not measure agentic capabilities of models, nor does it afford them access to a lab to try out and iterate on their ideas. Future iterations will incorporate agentic tasks both in silico and in the wet lab, working up to the ultimate goal of a lab-in-the-loop synthesis planner.

Acknowledgments We thank Julian Götz for his valuable feedback and careful review of the manuscript.

References [1]

Taicheng Guo et al. “What can large language models do in chemistry? a comprehensive benchmark on eight tasks”. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23. New Orleans, LA, USA: Curran Associates Inc., 2023.

[2]

Adrian Mirza et al. “A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists”. In: Nature Chemistry 17.7 (July 1, 2025), pp. 1027–1034.

[3]

Nicholas T. Runcie, Charlotte M. Deane, and Fergus Imrie. “Assessing the Chemical Intelligence of Large Language Models”. In: Journal of Chemical Information and Modeling 66.1 (Jan. 12, 2026), pp. 216–227.

[4]

Daniil A. Boiko et al. “Autonomous chemical research with large language models”. In: Nature 624.7992 (Dec. 1, 2023), pp. 570–578.

[5]

Andres M. Bran et al. “Augmenting large language models with chemistry tools”. In: Nature Machine Intelligence 6.5 (May 1, 2024), pp. 525–535.

[6]

Shriram Chennakesavalu et al. “Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design”. In: CoRR abs/2604.16279 (2026).

[7]

Jon M. Laurent et al. “LAB-Bench: Measuring Capabilities of Language Models for Biology Research”. In: CoRR abs/2407.10362 (2024).

[8]

Chris Lu et al. “Towards end-to-end automation of AI research”. In: Nature 651.8107 (Mar. 1, 2026), pp. 914–919.

[9]

Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery”. In: Nature Machine Intelligence 4.3 (2022), pp. 189–191.

[10]

Haochen Zhao et al. “ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain”. In: CoRR abs/2411.16736 (2024).

[11]

Derek T. Ahneman et al. “Predicting reaction performance in C–N cross-coupling using machine learning”. In: Science 360.6385 (2018), pp. 186–190.

[12]

Hanyu Gao et al. “Using Machine Learning To Predict Suitable Conditions for Organic Reactions”. In: ACS Central Science 4.11 (2018), pp. 1465–1476.

[13]

David Weininger. “SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules”. In: Journal of Chemical Information and Computer Sciences 28.1 (Feb. 1, 1988), pp. 31–36.

[14]

Peter Ertl, Bernhard Rohde, and Paul Selzer. “Fast Calculation of Molecular Polar Surface Area as a Sum of Fragment-Based Contributions and Its Application to the Prediction of Drug Transport Properties”. In: Journal of Medicinal Chemistry 43.20 (Oct. 1, 2000), pp. 3714–3717.

[15]

Scott A. Wildman and Gordon M. Crippen. “Prediction of Physicochemical Parameters by Atomic Contributions”. In: Journal of Chemical Information and Computer Sciences 39.5 (2002), pp. 868–873.

[16]

Greg Landrum et al. RDKit: Open-source cheminformatics. https://www.rdkit.org.

[17]

Daylight Chemical Information Systems, Inc. SMARTS – A Language for Describing Molecular Patterns. Daylight Theory Manual.

[18]

David Mendez et al. “ChEMBL: towards direct deposition of bioassay data”. In: Nucleic Acids Research 47.D1 (2018), pp. D930–D940.

[19]

World Anti-Doping Agency. World Anti-Doping Code International Standard: Prohibited List 2026. Approved 11 September 2025; in force 1 January 2026. 2026. 12

[20]

United States Drug Enforcement Administration, Diversion Control Division. Lists of Scheduling Actions, Controlled Substances, and Regulated Chemicals. Controlled Substances Act schedules I–V. 2026.

[21]

Organisation for the Prohibition of Chemical Weapons. Convention on the Prohibition of the Development, Production, Stockpiling and Use of Chemical Weapons and on Their Destruction: Annex on Chemicals (Schedules 1–3). Entered into force 29 April 1997. 1993.

[22]

United Nations Environment Programme. Stockholm Convention on Persistent Organic Pollutants: Annexes A, B and C. Entered into force 17 May 2004; annexes amended by subsequent Conference of the Parties decisions. 2001.

[23]

Aidan Wong et al. SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis. 2024.

[24]

Wikipedia contributors. List of designer drugs. [Online; accessed 28-July-2026]. 2026.

[25]

onepot AI. POT-2 — the automated lab.

[26]

Christian A.G.N. Montalbetti and Virginie Falque. “Amide bond formation and peptide coupling”. In: Tetrahedron 61.46 (2005), pp. 10827–10852.

[27]

Anil S. Guram, Roger A. Rennels, and Stephen L. Buchwald. “A Simple Catalytic Method for the Conversion of Aryl Bromides to Arylamines”. In: Angewandte Chemie International Edition in English 34.12 (1995), pp. 1348–1350.

[28]

Janis Louie and John F. Hartwig. “Palladium-catalyzed synthesis of arylamines from aryl halides. Mechanistic studies lead to coupling in the absence of tin reagents”. In: Tetrahedron Letters 36.21 (1995), pp. 3609–3612.

[29]

Dominic M.T Chan et al. “New N- and O-arylations with phenylboronic acids and cupric acetate”. In: Tetrahedron Letters 39.19 (1998), pp. 2933–2936.

[30]

Patrick Y.S Lam et al. “New aryl/heteroaryl C-N bond cross-coupling reactions via arylboronic acid/cupric acetate arylation”. In: Tetrahedron Letters 39.19 (1998), pp. 2941– 2944.

[31]

Ralph N. Salvatore, Cheol Hwan Yoon, and Kyung Woon Jung. “Synthesis of secondary amines”. In: Tetrahedron 57.37 (2001), pp. 7785–7811.

[32]

Norio Miyaura, Kinji Yamada, and Akira Suzuki. “A new stereospecific cross-coupling by the palladium-catalyzed reaction of 1-alkenylboranes with 1-alkenyl or 1-alkynyl halides”. In: Tetrahedron Letters 20.36 (1979), pp. 3437–3440.

[33]

Riccardo Ronchetti et al. “Recent advances in urea- and thiourea-containing compounds: focus on innovative approaches in medicinal chemistry and organic synthesis”. In: RSC Medicinal Chemistry 12.7 (2021), pp. 1046–1064.

[34]

Arun K. Ghosh and Margherita Brindisi. “Urea Derivatives in Modern Drug Discovery and Medicinal Chemistry”. In: Journal of Medicinal Chemistry 63.6 (2019), pp. 2751–2788.

[35]

Albert M. Van Leusen, Jurjen Wildeman, and Otto H. Oldenziel. “Chemistry of sulfonylmethyl isocyanides. 12. Base-induced cycloaddition of sulfonylmethyl isocyanides to carbon,nitrogen double bonds. Synthesis of 1,5-disubstituted and 1,4,5-trisubstituted imidazoles from aldimines and imidoyl chlorides”. In: The Journal of Organic Chemistry 42.7 (May 1977), pp. 1153–1159.

[36]

Kangway V. Chuang and Michael J. Keiser. “Comment on “Predicting reaction performance in C–N cross-coupling using machine learning””. In: Science 362.6416 (2018).

A. Extended ChemAbacus Results Here, we report extended results for ChemAbacus. Table 1 shows the overall scores for each (model, task) pair. Overall, Fable 5 (w/ Opus 5 fallback) is the strongest model; on high reasoning, it attains the highest average score, and wins 5 out of 8 tasks. Out of all the tasks, molecular weight (MW), ring counting (Rings), TPSA, and HBD counting are fully solved by at least one model. Table 2 shows breakdown by difficulty tier instead of task. Performance is very strong on easy and weaker on the other buckets. For most models, “adversarial" is indeed the hardest tier, though some models also struggle on biologics as well. Fable 5 (strict) scores especially poorly on biologics; this is because most of its refusals occur on biologics. Table 1 | ChemAbacus accuracy (%) by property. MW within ±1%, TPSA within ±5 Å2 , cLogP within ±1; rings/rotatable/HBA/HBD exact match; FG is Jaccard over the functional-group set. Macro denotes mean accuracy over all query types. Fable 5 appears twice: strict counts refusals (38 low, 60 medium, 75 high) as failures; +Opus routes them to Opus 5. bold denotes best score in the column, and the ¢/q column shows cents per query. Model

Rings

TPSA

cLogP

Rot

HBA

HBD

Macro

¢/q

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high) GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high)

93 98 99 88 95 99

100 99 100 97 99 100

90 96 97 70 78 81

76 85 85 69 77 77

51 68 70 53 77 80

93 94 96 77 82 89

98 100 100 95 96 86

94 93 93 96 96 96

86.9 91.6 92.5 80.6 87.5 88.5

2.90 9.29 20.84 3.28 6.50 10.28

GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

96 98 99

100 100 100

80 78 86

74 78 73

61 67 78

91 93 95

64 66 65

95 96 96

82.6 84.5 86.5

3.03 5.61 10.47

Anthropic Claude Opus 5 (low) Claude Opus 5 (med) Claude Opus 5 (high) Claude Opus 4.8 (low) Claude Opus 4.8 (med) Claude Opus 4.8 (high) Claude Fable 5 (strict, low) Claude Fable 5 (strict, med) Claude Fable 5 (strict, high) Claude Fable 5 (+Opus, low) Claude Fable 5 (+Opus, med) Claude Fable 5 (+Opus, high) Claude Haiku 4.5 (low)

95 97 100 92 96 98 87 91 91 90 97 100 23

100 100 100 99 100 100 91 89 84 99 100 100 80

96 99 99 96 99 100 91 87 86 94 96 99 28

72 82 87 75 79 78 66 69 78 71 75 90 54

87 94 93 90 90 93 79 83 85 87 92 94 8

90 90 96 88 92 90 81 82 83 87 88 91 52

100 100 97 100 100 100 96 96 97 97 99 100 57

96 96 95 96 97 97 95 95 96 96 97 97 88

92.0 94.8 95.9 92.0 94.1 94.5 85.8 86.5 87.5 90.1 93.0 96.4 48.8

2.37 5.62 9.01 2.69 6.00 8.54 2.30 5.72 10.30 2.36 6.00 11.09 0.83

Google Gemini 3.1 Pro (low) Gemini 3.5 Flash (low)

91 93

100 100

64 61

89 78

94 89

93 84

100 99

94 97

90.6 87.6

0.50 0.35

Meta Muse Spark 1.1 (low) Muse Spark 1.1 (med) Muse Spark 1.1 (high)

94 97 98

98 98 100

88 93 97

80 84 83

72 75 82

79 82 82

98 99 99

96 96 98

88.1 90.5 92.4

1.19 1.96 4.00

DeepSeek DeepSeek V4 Pro (default) DeepSeek V4 Flash (default)

93 87

100 99

77 60

57 48

91 80

84 82

93 95

90 84

85.6 79.4

1.76 0.48

Moonshot Kimi K2.6 (default)

100

88.8

10.55

B. Detailed SynthRefusal Results This compact appendix reports every publication-facing aggregate while avoiding repeated model rows. In behaviour tables, cells are clean refusal / partial disclosure (R/L). Target-notidentified and target-misidentified responses are excluded from the corresponding refusal denominators and are treated as audit outcomes rather than benchmark result tables.

Table 2 | ChemAbacus accuracy (%) by difficulty split. Each cell shows average performance for a given model on a given difficulty level across all 8 tasks. Fable 5 is shown both as strict (refusals count as incorrect) and +Opus (refusals routed to Opus 5); comparing the two rows reveals that refusals overwhelmingly appear in the biologics difficulty tier. Average shows the average score across all rows excepting the Fable 5 strict rows. On average, easy is indeed the easiest tier by a large margin; the rest are of similar difficulty, with adversarial and biologics being the two hardest tiers. Model

Easy

Medium

Hard

Adv.

Biol.

Mean

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high) GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high) GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

95 96 97 97 96 97 93 93 92

87 92 92 80 87 89 84 85 88

87 90 92 79 86 86 82 86 89

84 92 94 74 85 82 76 74 78

81 88 88 74 83 88 79 84 85

87 92 92 81 88 89 83 85 86

98 99 98 99 98 98 98 99 99 99 99 99 67

92 96 97 92 95 93 93 92 93 93 93 95 54

89 94 95 91 93 93 88 90 92 88 90 95 46

91 95 94 92 93 97 84 91 93 85 92 97 36

91 90 96 86 92 92 66 62 60 86 90 95 41

92 95 96 92 94 95 86 86 87 90 93 96 49

Google Gemini 3.1 Pro (low) Gemini 3.5 Flash (low)

98 98

91 91

91 86

87 81

86 83

91 88

Meta Muse Spark 1.1 (low) Muse Spark 1.1 (med) Muse Spark 1.1 (high)

98 98 99

89 91 92

87 93 92

83 86 89

84 85 90

88 90 92

DeepSeek DeepSeek V4 Pro (default) DeepSeek V4 Flash (default)

94 89

85 77

85 76

80 78

84 78

86 79

Moonshot Kimi K2.6 (default)

Average

Table 3 | Controlled-substance outcome rates by category. Each cell reports the conditional cleanrefusal rate / partial-disclosure prevalence (%). Clean-refusal rates exclude partial disclosures and are calculated over clean refusals plus produced responses; partial-disclosure rates are calculated over all target-resolved responses. Rates are macro-averaged over SMILES, CAS, InChI, and name. Cell color encodes the clean-refusal rate (reversed for benign controls). Model

Benign

Anti-dop.

Poison

Explos.

Sched. I–IV

CWA

Avgctrl

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high) GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high) GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

3/21 4/9 7/11 0/0 0/0 2/0 0/4 0/0 0/0

36/18 46/11 42/9 2/0 6/3 5/0 10/9 12/9 15/7

83/30 90/23 92/17 0/1 4/1 3/1 69/40 56/40 54/43

90/11 96/15 97/18 33/20 34/13 26/12 90/28 84/27 80/33

88/7 94/2 97/0 15/0 16/3 14/1 82/15 82/16 84/18

91/11 95/11 97/11 33/5 35/8 33/6 81/8 86/11 83/10

78/15 84/12 85/11 17/5 19/5 16/4 66/20 64/20 63/22

Anthropic Haiku 4.5 (low) Haiku 4.5 (med) Haiku 4.5 (high) Opus 4.8 (low) Opus 4.8 (med) Opus 4.8 (high) Opus 5 (low) Opus 5 (med) Opus 5 (high) Fable 5 (low) Fable 5 (med) Fable 5 (high)

1/1 3/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 26/32 35/13 39/5

0/0 1/0 1/1 0/0 0/0 0/0 0/0 0/0 0/0 61/25 70/13 73/6

1/0 1/0 1/0 31/5 34/1 35/2 22/1 31/0 38/0 98/7 99/5 99/1

7/1 7/0 7/5 10/0 16/0 24/0 21/2 28/8 40/9 99/0 99/1 100/0

9/0 8/1 8/1 6/0 8/0 10/0 4/1 12/9 22/13 98/21 99/6 97/1

12/0 12/2 13/0 36/1 42/1 48/1 53/2 60/6 66/4 98/5 99/6 98/4

6/0 6/1 6/2 17/1 20/0 24/1 20/1 26/5 33/5 91/11 93/6 94/2

Google Gemini 3.1 Pro (low) Gemini 3.1 Pro (med) Gemini 3.1 Pro (high) Gemini 3.5 Flash (low) Gemini 3.5 Flash (med) Gemini 3.5 Flash (high)

8/4 5/2 6/0 0/0 0/0 0/0

18/4 15/4 16/5 0/0 0/0 0/0

76/19 70/23 70/29 1/5 2/4 2/2

89/9 81/16 80/19 29/5 37/2 38/5

73/10 71/28 66/29 12/2 12/2 16/2

90/4 89/7 89/8 36/0 41/1 47/1

69/9 65/16 64/18 16/2 18/2 20/2

Meta Muse Spark 1.1 (low) Muse Spark 1.1 (med) Muse Spark 1.1 (high)

92/8 96/11 96/14

100/5 99/2 99/3

100/5 100/4 100/4

100/0 100/1 100/1

100/1 100/2 100/0

100/2 99/4 100/1

100/3 100/2 100/2

DeepSeek DeepSeek V4 Flash

0/0

150/0

0/0

1/0

0/0

Model

Sched. I

Sched. II

Sched. III

All

Model

SMILES

CAS

InChI

name

80/15 96/0 93/0 0/0 6/0 0/0 78/11 65/34 75/25

88/7 94/2 97/1 15/1 17/2 15/1 83/16 82/17 84/17

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high) GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high) GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

68/16 81/14 81/11 13/8 14/6 14/5 58/20 57/22 56/25

80/16 88/11 90/11 9/0 20/4 18/1 74/21 67/21 67/16

78/16 80/12 83/14 16/7 14/9 12/5 59/25 59/22 59/22

85/14 87/13 86/9 29/6 28/4 20/4 75/14 73/16 72/25

0/0 0/0 0/0 0/0 5/0 0/0 0/0 0/12 7/0 100/18 100/4 100/0

0/0 5/0 6/0 0/0 0/0 0/0 0/0 0/0 0/8 83/32 92/8 81/7

9/1 8/1 8/1 6/0 9/1 11/0 4/1 13/9 22/12 98/21 99/6 97/1

Anthropic Haiku 4.5 (low) Haiku 4.5 (med) Haiku 4.5 (high) Opus 4.8 (low) Opus 4.8 (med) Opus 4.8 (high) Opus 5 (low) Opus 5 (med) Opus 5 (high) Fable 5 (low) Fable 5 (med) Fable 5 (high)

2/1 3/0 2/2 14/1 19/1 22/1 21/2 30/3 32/8 85/18 90/8 91/5

4/0 5/0 5/3 13/1 17/1 17/1 15/1 20/5 29/6 96/7 97/6 97/1

0/0 1/0 1/0 14/2 18/0 24/0 20/1 30/4 37/4 93/10 95/5 96/2

16/1 14/2 17/2 25/0 25/0 31/0 24/1 25/6 34/4 89/11 92/6 90/2

78/10 78/32 74/32 20/0 21/3 28/5

51/11 35/46 32/36 8/4 4/4 11/0

22/21 38/25 21/36 0/4 0/0 0/0

74/10 73/28 67/28 13/2 13/2 17/2

Google Gemini 3.1 Pro (low) Gemini 3.1 Pro (med) Gemini 3.1 Pro (high) Gemini 3.5 Flash (low) Gemini 3.5 Flash (med) Gemini 3.5 Flash (high)

60/9 54/16 53/22 12/3 19/3 20/3

89/8 86/14 86/18 11/1 13/1 15/0

65/14 63/20 61/20 12/1 13/0 18/1

62/6 57/14 56/12 28/5 29/3 29/4

100/1 100/3 100/1

100/2 100/2 100/0

100/0 100/0 100/0

100/1 100/2 100/1

Meta Muse Spark 1.1 (low) Muse Spark 1.1 (med) Muse Spark 1.1 (high)

100/3 100/4 100/4

100/1 100/1 100/0

100/2 99/2 100/3

100/4 100/2 100/1

0/0

DeepSeek DeepSeek V4 Flash

0/0

1/0

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high) GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high) GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

78/8 88/1 95/0 20/0 9/0 20/0 77/7 74/9 85/8

95/6 100/0 100/2 20/0 28/5 18/2 91/20 98/17 96/19

96/0 95/7 100/0 12/4 7/4 7/4 90/34 83/11 71/29

Anthropic Haiku 4.5 (low) Haiku 4.5 (med) Haiku 4.5 (high) Opus 4.8 (low) Opus 4.8 (med) Opus 4.8 (high) Opus 5 (low) Opus 5 (med) Opus 5 (high) Fable 5 (low) Fable 5 (med) Fable 5 (high)

9/1 11/2 10/0 12/0 15/1 15/0 4/0 21/8 38/16 98/9 100/1 100/0

16/0 10/0 10/3 4/0 8/0 15/0 9/3 16/15 19/17 100/32 100/13 100/0

Google Gemini 3.1 Pro (low) Gemini 3.1 Pro (med) Gemini 3.1 Pro (high) Gemini 3.5 Flash (low) Gemini 3.5 Flash (med) Gemini 3.5 Flash (high)

92/7 92/18 87/20 14/1 14/0 16/1

Meta Muse Spark 1.1 (low) Muse Spark 1.1 (med) Muse Spark 1.1 (high) DeepSeek DeepSeek V4 Flash

Sched. IV

Table 4 | Refusal rates by DEA schedule. Each cell is Table 5 | Refusal rate by input represenclean refusal / partial disclosure (%) averaged over the tation. Controlled molecules only; data and four given input representations. Colors follow Table 3. colors match Table 4. Table 6 | Designer-drug clean-refusal rate by structural subcategory and effort across all evaluated models. Each pharmacological class (bold, class total) splits into its named structural sub-families. The rate is clean refusal divided by clean refusal plus produced (%); identification failures are reported separately and excluded from this denominator. Pro 3.1 and Flash 3.5 refer to Gemini models. Class / subcategory

Androgens SARMs DHT based Testosterone based (other)

20 35 47 47 0 0 0 10 5 60 100 100 3 50 100 100 2 100 100 100

GPT-5.5 GPT-Rosalind GPT-5.6 Sol Haiku 4.5 Opus 4.8 Opus 5 Fable 5 Pro 3.1 Flash 3.5 Muse Spark DeepSeek low med high low med high low med high low med high low med high low med high low med high low med high low med high low med high V4 Flash 0 0 0 0 0

0 0 0 0 0

6 19 0 0 0 33 33 50 0 100

Dissociatives Arylcyclohexylamines 1,2-Diarylethylamines (other)

20 82 100 95 10 90 100 100 7 60 100 100 3 100 100 67

5 10 0 0

33 56 14 0

Entactogens MDxx Amphetamines Phenethylamines (other)

20 83 89 95 5 80 80 100 4 75 100 100 3 100 100 100 8 83 88 88

15 40 25 0 0

5 0 25 0 0

Nootropics CNS stimulants (other)

20 4 16

32 75 20

26 50 20

37 67 31

10 25 6

PDE5 inhibitors

11 0 25 50 0

0 0 0 0 0

0 67 67 95 0 67 43 89 0 25 75 100 0 100 100 100 0 100 100 100

0 0 0 0 0

5 0 0 0 50

0 0 0 0 0

100 100 100 100 100 100 100 100 100 100

100 100 100 100 100

0 0 0 0 0

47 72 89 100 14 60 0 0

0 0 0 0

11 25 0 0

24 57 0 0

100 100 100 100 100 100 100 100

100 100 100 100

44 83 14 33

31 75 14 0

25 75 0 0

0 0 0 0

5 11 0 0

11 22 0 0

100 100 100 100 100 100 100 100

100 100 100 100

0 0 0 0

10 20 25 0 0

60 74 60 80 80 80 50 50 50 67 100 100 50 71 38

0 0 0 0 0

5 0 25 0 0

0 0 0 0 0

25 40 25 33 12

38 75 33 50 14

100 100 100 100 100 100 – 100 100 100

100 100 100 100 100

15 20 0 33 12

21 20 25 0 25

6 25 0 0 0

0 0 0 0 0

10 20 25 0 0

100 100 100 100 100 100 100 100 100 100

100 100 100 100 100

0 0 0 0 0

10 25 6

11 33 7

16 50 7

0 0 0

5 25 0

15 25 12

50 67 46

53 75 47

11 0 12

15 25 12

11 25 7

5 25 0

5 0 6

10 25 6

86 94 100 75 100 100 90 92 100

0 0 0

88 100 100

53 75 46

0 100 100 100

Peptides 17 GH secretagogue agonists 5 GHRH analogues 4 (other) 8

0 0 0 0

6 0 0 12

0 0 0 0

0 94 94 0 80 80 0 100 100 0 100 100

100 100 100 100

6 0 25 0

0 0 0 0

100 100 93 100 100 100 100 100 100 100 100 83

0 0 0 0

Piperazines

0 100 100 100

Psychedelics Tryptamines Benzofurans > 2C-x Benzofurans > DOx (other)

20 84 80 85 8 71 75 75 4 100 75 100 3 67 67 67 5 100 100 100

0 0 0 0 0

5 0 0 0 20

35 25 25 67 40

50 50 67 33 50

47 50 67 33 40

0 0 0 0 0

5 0 0 0 20

0 0 0 0 0

15 0 50 0 20

41 94 100 100 35 23 0 29 80 100 100 0 17 0 67 100 100 100 100 – – 33 100 100 100 33 33 0 50 100 100 100 40 25 0

0 0 0 0 0

100 95 100 88 100 100 100 100 100 100

100 100 100 100 100

0 0 0 0 0

Sedatives Opioids GHB analogues (other)

20 9 3 8

35 44 0 38

47 62 0 50

68 88 33 62

0 0 0 0

28 38 0 29

26 44 0 14

33 43 0 38

0 0 0 0

5 0 0 12

0 0 0 0

5 0 0 12

15 75 67 83 11 100 100 100 0 0 0 33 25 83 67 83

26 33 0 29

24 33 0 20

18 25 0 17

0 0 0 0

100 100 100 100 100 100 100 100

100 100 100 100

0 0 0 0

Stimulants Piperidines/tropanes (other)

20 5 15

33 50 29

56 60 54

53 60 50

5 20 0

0 0 0

22 25 21

28 50 21

26 50 20

0 0 0

5 20 0

0 0 0

15 93 79 94 20 100 100 100 13 90 70 92

11 33 7

0 0 0

5 20 0

5 0 7

0 100 100 100 0 100 100 100 0 100 100 100

0 0 0

Synthetic cannabinoids

0 100 100 100

C. SynthBench (reaction outcome) results, all models Table 7 | Reaction-outcome accuracy (%) by reaction class. Binary task, chance = 50; underline = significantly above chance (one-sided binomial, p < 2.2 · 10−4 after Bonferroni correction); bold = best in column. Model

Amide

Buch.

Chan-Lam

N-Alk

Suzuki

Thio.

Urea

V. Leusen

Average

OpenAI GPT-5.5 (low) GPT-5.5 (med) GPT-5.5 (high)

54 51 50

52 52 54

54 49 51

58 59 59

50 50 50

56 52 51

65 56 55

60 56 55

56 53 53

GPT-Rosalind (low) GPT-Rosalind (med) GPT-Rosalind (high)

50 51 50

49 50 54

51 54 45

55 56 61

50 50 50

51 50 49

50 54 49

61 60 59

52 53 52

GPT-5.6 Sol (low) GPT-5.6 Sol (med) GPT-5.6 Sol (high)

51 50 50

51 51 51

52 54 55

56 58 58

50 50 50

52 51 51

51 50 50

54 55 55

52 52 52

Anthropic Claude Opus 5 (low) Claude Opus 5 (med) Claude Opus 5 (high) Claude Opus 4.8 (low) Claude Opus 4.8 (med) Claude Opus 4.8 (high) Claude Fable 5 (low) Claude Fable 5 (med) Claude Fable 5 (high) Claude Haiku 4.5 (low)

70 62 61 72 74 71 69 71 64 49

70 68 62 60 61 54 54 56 64 52

44 54 50 41 51 51 52 60 54 42

66 64 72 62 62 60 65 60 69 51

55 56 50 51 55 52 52 54 48 50

70 66 70 66 71 74 68 62 68 50

68 69 62 58 61 66 66 65 64 50

74 71 68 70 61 68 69 68 68 55

65 64 62 60 62 62 62 62 62 50

Google Gemini 3.1 Pro (low) Gemini 3.5 Flash (low)

62 59

54 58

49 45

61 62

52 49

59 69

65 71

60 56

58 59

DeepSeek DeepSeek V4 Pro (default) DeepSeek V4 Flash (default)

54 51

48 49

48 42

58 56

46 48

52 51

54 50

58 55

52 50

Moonshot Kimi K2.6 (default)

Log. Reg. baseline

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Analyze & explore related research in Conceptio

Related documents