Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?

Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?

arXiv:2610.01616v1 [cs.CL] 1 Oct 2026

Laura van Weesep*,1,2 , Riccardo Tedoldi*,1,3 , Jens Sjölund2 , Hossein Azizpour3,4 , Susanne Winiwarter1,5 , Ola Engkvist1,6 , Jon Paul Janet1 , Samuel Genheden1 , Juan Viguera Diez1 1 Molecular AI, Discovery Sciences R&D, AstraZeneca Department of Information Technology, Uppsala University 3 Robotics, Perception & Learning, KTH Royal Institute of Technology 4 Science for Life Laboratory, Stockholm, Sweden 5 Drug Metabolism and Pharmacokinetics, Research and Early Development, Cardiovascular, Renal and Metabolism (CVRM), BioPharmaceuticals R&D, AstraZeneca 6 Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Sweden 2

*

equal contributors

Correspondence: lauradesire.vanweesep@astrazeneca.com, riccardo.tedoldi@astrazeneca.com

Abstract The emergence of foundation models for molecular property prediction requires a high degree of AI data readiness, including reliable metadata annotation. However, both public repositories and industrial screening databases suffer from missing, inconsistent, or conflated assay annotations. In this work, we quantify the extent of missing annotations in PubChem for the BioAssay Ontology (BAO) assay format and physical detection method fields and investigate whether open-source and proprietary large language models (LLMs) can reliably predict and audit metadata annotations directly from the assay text. In our assessment, we found that the annotation coverage across PubChem’s ∼2 million bioassays is critically sparse, 36% lacking an assay format, 89% a BioAssay type, and >99.9% any BAO-mapped assay format or detection technology term. This motivates the need for automated test-metadata curation. Using evaluation sets derived from PubChem and ChEMBL, we assess the agreement of seven open-source and proprietary LLMs with existing silver labels. Recall is at least 0.96 for biochemical and cellbased assay formats, with a similar pattern for detection technology, although disagreements increase on under-represented classes. Manual inspection shows that many of these disagreements trace back to inconsistencies between silver sources rather than to LLM error. Moreover, in a qualitative study with a senior industrial curator, LLM-generated evidence prompted the expert to revise some of their own labels, showing LLMs can flag potentially mislabeled assays. Across the study, performance differences between proprietary and open-source models were small. Together, these results suggest LLMs can support the large-scale annotation and auditing of assay metadata, though per-class reliability estimates and targeted human review remain necessary before such labels enter downstream ML pipelines.

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: AI Data Readiness for Scientific Discovery.

1

Introduction

The FAIR data principles [1] introduced standards to make data findable, accessible, interoperable, and reusable, yet AI readiness for many datasets remains limited. In particular, missing, inconsistent, and incorrect metadata remain widespread problems in scientific data repositories. Bioassay collections are a notable example and a fundamental source of data for machine learning models for drug discovery. A bioassay measures the effect of a substance on a biological system to obtain insight into the expected biological activity. PubChem [2] is the largest public chemical database, containing more than 100 million compounds and bioassay results from more than a million assays. Many entries lack structured metadata on the nature of the experiment, such as the assay format and the detection method. The lack of metadata in PubChem reflects issues such as records that predate relevant standards or inconsistent use of terminology among contributors. Industrial assay databases present related problems, including missing historical data and the use of single labels, such as the “assay category” to describe several distinct aspects of an assay at once. The BioAssay Ontology (BAO) [3, 4] addresses the ambiguity challenge by decomposing assays into orthogonal axes, including the biological format and the physical detection method, the endpoint measured and the target with each a formal class definition. This distinction allows for assays of the same format but with a different detection method to be distinguished. It aims to provide a framework for mapping unstructured descriptions to standardized entities. PubChem’s native “BioAssay Type” field mixes format, modality, and endpoint into composite labels such as In vivo|Toxicity or Cellbased|RNAi for instance. Disentangling such labels retroactively or filling in missing labels manually does not scale to the level of millions of PubChem records or decades of legacy industrial assays. Large language models offer a promising approach to close this gap. For instance, instruction-tuned LLMs have recently been shown to perform well on ontology alignment in biomedical fields [5, 6]. Nevertheless, their performance on ontologies with multiple dimensions, such as BAO, remains underexplored. We address this gap by evaluating LLMs on the assignment of BAO assay formats and assay detection methods. Our contributions can be summarized as: (i) a systematic analysis of missing and inconsistent assay format and detection technique metadata in PubChem BioAssay’s bulk-download distribution, (ii) an assessment of four open-source and three closed-source LLMs on text-conditioned BAO assay format classification, using ChEMBL cross-references as a reference label followed by further exploration for physical detection method and subsets of PubChem and (iii) manual review of LLM vs. current-label disagreement, further backed up by a curator, in which we found examples of a senior industrial curator agreeing with the LLM over the original dataset label. Together, these results show that LLMs can detect likely mislabeled assays and in the future support automated annotation1 .

2

Background and Related Work

2.1 Bioassay Ontology (BAO) The BioAssay Ontology (BAO) was introduced to give the output of high-throughput screening campaigns explicit description-logic semantics rather than free-text labels [3], and later extended and modernized to cover a wider range of assay types and to interoperate with adjacent ontologies (cell line, target, and unit vocabularies) [4]. The BAO model focuses on subdividing assays into orthogonal axes, including categories related to the bioassay component, biology, endpoint, format, and method component. However, achieving comprehensive coverage of all bioassays has proved challenging. For example, in 2014, BAO could not adequately represent all transporter assays in ChEMBL [7]. Since then, it has undergone numerous updates to expand its capacity to describe and categorize diverse assay types. The current version (uploaded on the 6th of May 2026 v2.8.19) contains 7918 separate classes2 and is a product of over a decade of iterative validation against real-world chemical biology datasets. Early efforts to apply BAO relied on manual annotation. Vempati et al. [8] manually annotated 944 PubChem assays using a drop-down menu. They showed that they could map 299 distinct PubChem names to 20 controlled BAO terms, demonstrating the potential of the ontology to reduce terminological ambiguity and reveal additional relationships between assays. Industry has also 1 Code and data will be publicly released upon acceptance of the full paper. 2 http://www.bioassayontology.org/

2

adopted the BAO framework. AstraZeneca utilized this framework to annotate 381 internal assays run from 2005–2013, along with 233 PubChem assays to get insight into the technology gaps between their internal screening and external screening, compare compound–activity overlap and flag technology-specific artifacts [9]. However, manual annotation of protocols remains laborious and does not scale to the millions of bioassays available in public data repositories. 2.2 Annotated public data repositories In this work, we use three sources of bioassay annotations: PubChem, ChEMBL and BARD. PubChem [2] is the most extensive collection of publicly available data from high-throughput screening and medicinal chemistry experiments, which contains descriptions of protocols, target and results that many screening laboratories, biotech, pharmaceutical companies, and academic institutions have deposited since 2004 [10], currently comprising ∼1.9M bioassays (AIDs) as of the 2026 bulk download. However, records vary greatly on how and if the experimental metadata was reported, and often the information is present in form of unstructured free-text. To address this, the BioAssay Research Database (BARD) [11] produced a curated set of BAO annotations on PubChem records from the NIH Molecular Libraries Program (MLP). Curators mapped free-text PubChem protocols onto BAO terms, and the annotations were reviewed by MLP scientists involved in the original experiments. Although the BARD portal has been retired, its 4 477 curated annotations for 1 000 assays remain available in PubChem [12] as a bulk download. Out of these assays 997 have an annotated detection method and 998 had an annotated assay format. The BARD annotations for these assays and some assays beyond this, appear under the “BioAssay Annotations”. ChEMBL, in turn, is a manually curated bioactivity database that collects bioactivity data extracted primarily from the scientific literature [13, 14] and based on voluntary data depositors comprising ∼2.0M assays. ChEMBL curators assign standardized assay and target metadata and link some records to PubChem assay ID (AID) identifiers, an independent source of reference annotations for a subset of PubChem assays. ChEMBL has also introduced direct-deposition workflows with structured submission formats, depositor-defined identifiers, and validation procedures, aimed at improving metadata consistency [14, 15]. Nevertheless, many of these annotations lack meaningful BAO terms; ∼10% of the assays have unspecified assay formats, detection-method labels are entirely absent, and some existing annotations were generated using rule-based methods that may be unreliable [16]. These three annotation sources differ not only in their coverage of assays and metadata coverage but also in the way their labels are produced. Native PubChem metadata are derived from voluntary, free-text depositor submissions and have been found to lack the consistent structure needed for crossassay comparison [11]. ChEMBL improves on this through expert bioactivity curation, although the metadata coverage is limited. BARD was purpose-built for accuracy, and its expert validation step is absent from both PubChem’s native fields and ChEMBL’s assay annotations to our knowledge. However it only covers a small subset of PubChem. For assays annotated in all three sources, we therefore treat BARD-derived labels as the most trustworthy reference, ChEMBL as a secondary also partial source, and native PubChem metadata as the least reliable baseline. Because of these reliability gaps, several groups have built curated subsets or rule-based cleanup pipelines rather than fixing annotations at the source. Curated benchmarks such as LIT-PCBA [17] and MF-PCBA [18] clean HTS data and discard unreliable records, while ScrubChem applies logic-based text-processing rules to curate PubChem [19]. More recently, CAPRICHO [20] addressed curation of ChEMBL bioactivity data for QSAR modeling by flagging quality issues rather than removing them, letting users assess how each curation decision impacts the results. Its case studies show that assay metadata strongly affects whether measurements can be combined: grouping by assay type and cell type improved cross-assay agreement. However, CAPRICHO relies on existing ChEMBL metadata fields and on keyword-based queries of free-text assay descriptions. This highlights the need for more complete and structured assay annotations, which we address here using LLMs. Overall, although public repositories contain millions of bioassays, their BAO annotations are often missing, incomplete or unreliable, motivating the development of scalable, automated annotation methods. 2.3 Bioassay curation with language models LLMs offer a promising approach to automated assay annotation. LLMs are increasingly used to extract metadata from biomedical records. For example, ChIP-GPT uses a LLaMA-based model to 3

extract chromatin immunoprecipitation metadata from free-text records [5]. Similarly, Ikeda et al. [6] used Llama 3 to extract cell line and gene perturbation terms from BioSample descriptions. In both studies, the LLM method outperformed the traditional mapping approach. For assay format annotation specifically, recent studies have explored AI-assisted pipelines. One approach uses a named entity recognition (NER) model to extract the assay type from a curated description, followed by text2term [21] to ground free-text metadata to biomedical ontology terms [16]. Other methods, including a retrieval-then-prompt framework [22, 23] and other similar pipelines such as RELATE [24], ground LLM outputs to formal ontology terms rather than free text. This helps address a common issue with LLMs: their tendency to generate non-standard labels. To our knowledge, whether the same strategies could reliably assign BAO labels (i.e., format, detection, target) from free text consistently is unexplored and we aim to address this gap.

3

Methods

Dataset Construction. To evaluate the performance of LLMs for predicting bioassay metadata classes, we constructed our dataset from the PubChem BioAssay bulk JSON data and combined this with the associated annotations of assays and comment files (detailed in Appendix A). This process yielded 1 770 568 AIDs (containing a name, comment, and/or description) out of 1 994 310 downloaded in total. Because some metadata fields were empty or missing in the bulk files despite being available through the PubChem web interface, we additionally queried PubChem’s PUG REST endpoint for the assays included in our analysis. We used the returned records to supplement the assay title, description, comments and protocol when available. We also queried the ChEMBL 37 database3 and retained records that could be matched to a PubChem assay. We extracted the corresponding BAO assay format annotations, excluded PubChem AIDs that mapped to multiple ChEMBL entries, and discarded entries whose BAO assay format label was assay format. We also removed records with duplicate descriptions to not feed the same text field to the LLM multiple times. We followed the same de-duplication procedure and label cleanup for the AIDs for which BAO assay annotations were available in PubChem but not in ChEMBL. We similarly extracted and filtered data from assays containing physical detection method metadata. We mapped assay format and physical detection method labels to depth-one categories in the BAO hierarchy (Appendix Fig. 1 and Appendix Fig. 2). PubChem and ChEMBL labels that could not be mapped to a depth-one BAO category were discarded. This procedure yielded three final evaluation sets: 1 097 PubChem assays with assay format annotations from ChEMBL, 226 PubChem-only annotated assays with assay format labels, and 308 PubChem assays with physical detection method labels. Because the available reference labels are inherited from existing database annotations that included inconsistencies and disagreements with current annotation standards, we refer to them as “silver labels” as they might not reflect the ground truth according to the current BAO system. In this study, silver labels are derived from existing PubChem, ChEMBL, and BioAssay Research Database annotations and are used to evaluate agreement with LLM-generated labels. Task definition and LLMs utilized. We prompt LLMs to predict the assay format or assay physical detection method based on the title, description, comments, and protocol. We used three proprietary models: GPT-4o, Gemini 3.6 Flash, and Claude Sonnet 4.6, all accessed through APIs. We also evaluated four open-weights models: Gemma 4 31B, GPT-OSS 20B, Qwen 3.5, and Llama 3.3 70B, run locally via Ollama v0.17.4. Further details on the parameters used to run the models can be found in Appendix F. For prompt-ablation experiments, we selected a stratified subset of 500 assays from the ChEMBL assay format set and evaluated the same assays under different prompt conditions, including alternative label orderings and prompts without the definition of the label. The complete prompts can be found in Appendix E. For models run through Ollama, we enforced the label vocabulary at generation time using Ollama’s JSON-Schema format= parameter.

4

Missingness and Inconsistency in PubChem Bioassay Metadata

We first investigated the quality and completeness of the metadata provided in the PubChem BioAssay bulk distribution. Out of the 1 994 310 assays in the bulk downloaded, only 89% had a name, 3 https://ftp.ebi.ac.uk/pub/databases/ChEMBL/ChEMBLdb/releases/ChEMBL_37/ChEMBL_37_sqlite.

tar.gz

4

description, and comment, with less than 10% unique descriptions. We followed up by analyzing the metadata (see Appendix Table 5), focusing specifically on four metadata categories: Assay Format, BAO Assay Format, BAO Detection Technology, and BioAssay Type. In Appendix A, we report statistics of the distributions across the four metadata dimensions in Tables 6, 7, 8 and 9. The labels show substantial heterogeneity, both in their level of detail and in the aspect of the assay they describe: some refer to the assay modality, others to the endpoint measured. We initially examined the BioAssay Type field to see whether it could inform BAO assay format assignment, but found it unsuitable as the vocabulary conflates format (e.g., Cell-based, Biochemical), methodology (RNAi), and endpoint (Toxicity), and a non-negligible fraction of entries combine multiple labels that are inconsistent with the BAO ontology (Table 9). A related source of variation is the PubChem Assay Format field itself, which offers only three categories (Cell-based, Organism-based, Biochemical), whereas the BAO assay format branch is far richer. This limited vocabulary likely contributes to annotation discrepancies. When curators are forced to pick from three options they may assign a different label than they would given the full BAO hierarchy. This motivates the use of BAO labels for assessing whether LLMs can assign assay formats, instead of using the PubChem Assay Format field itself. We also noticed inconsistencies from the available data in the PubChem web interface. To illustrate, the AID 1885, is reported as Cell-based in the web interface, but as Cell-based|Organism-based in the bulk download. Moreover, inconsistencies can occur within a single PubChem entry: the same assay may be associated with multiple conflicting BAO labels. From the 236 assays that had both the BARD and ChEMBL assay format labels on PubChem in the bulk BARD download, for 24 assays there was a disagreement (10.2%). Further details and case studies can be found in Appendix E.1. Additionally, the coverage of the metadata fields is limited. The data from the PubChem bulk download present severe amounts of missing data, with 36% out of the 1.9 million entries missing the assay format, and less than 0.1% of the assays having a BAO assay format and physical detection method annotated in the comments. All in all, PubChem contains substantial bioassay annotation issues, including missing and inconsistent information.

5

Results: An LLM-based Auditing Workflow

Given the missingness, heterogeneity, and inconsistencies identified in Section 4, we evaluated whether LLMs could provide a scalable approach to reviewing and assigning bioassay metadata labels. Model predictions were compared against existing annotations derived from PubChem, ChEMBL, and BARD, which we refer to as silver labels. Because these references may themselves be incomplete, inconsistent, or erroneous, the per-class classification metrics reported below should not be interpreted as definitive annotation accuracy. To mitigate these limitations, we manually inspect a subset, keeping in mind that annotations from different sources might differ in trustworthiness. 5.1

Reliability against silver sources

We first assess agreement among the LLMs using Cohen’s κ [25]. On the ChEMBL assay format evaluation set, all seven LLMs achieve pairwise Cohen’s κ > 0.9, with similar agreement on the BAO detection tasks. The agreement drops on the smaller PubChem-BAO subset, but is still high with κ > 0.8 for all models except Gemma 3 27B, where we see lower agreement with the other LLMs. Full agreement matrices for all three experiments along with a definition of the Cohen’s κ are provided in Appendix G. Next, we evaluate whether the consensus signal tracks silver-label sources, using ChEMBL assay format labels as a reference. Table 1 reports classification metrics for each assay format class. A majority voting consensus of LLMs agrees with the ChEMBL label in 96% of the cases. Cell-based and biochemical format are classified consistently well with a recall at least 96% for all models including the open-source ones. Performance on the organism-based format varies more across models: Gemma 3 27B achieves the highest precision, but makes the fewest predictions for this class. The majority vote achieves a recall of 87% on the organism-based format with a precision of 77%, noticeably lower than the performance on the cell-based and biochemical format. The cell-free and tissue-based formats show substantially lower precision and recall for all models, with at most one correctly predicted instance in either class. These results might suggest that LLMs are unreliable for rare assay format classes (only 6 and 2 entries in this dataset). However, manual inspection showed that for 7 out of 8 of these assays the BARD label was actually different from the ChEMBL label 5

Table 1: Per-class recall / precision on the ChEMBL baseline (assay format). Values shown as R/P; support (ntrue ) shown once per column in the header row. The two lowest-support classes (tissuebased and cell-free, each n<10) are too small for reliable aggregate metrics. Invalid predictions were excluded for these metrics. Precision is reported as “–” when no predictions were made. Model

Acc.

mF1

mF1top3

cell-based (n=784)

biochemical (n=252)

organism-based (n=53)

tissue-based (n=6)

cell-free (n=2)

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o

0.963 0.957 0.954

0.560 0.578 0.551

0.934 0.919 0.919

0.98 / 0.99 0.97 / 0.99 0.96 / 0.99

0.96 / 0.98 0.96 / 0.97 0.96 / 0.97

0.88 / 0.83 0.94 / 0.71 0.91 / 0.74

0.00 / – 0.00 / – 0.00 / –

0.00 / 0.00 0.50 / 0.08 0.00 / 0.00

GPT-OSS 20B Gemma 3 27B Gemma 4 31B Llama 3.3 70B

0.956 0.979 0.951 0.976

0.602 0.576 0.545 0.572

0.908 0.960 0.909 0.954

0.97 / 0.98 0.99 / 0.98 0.96 / 0.99 0.99 / 0.98

0.96 / 0.97 0.99 / 0.97 0.96 / 0.97 0.99 / 0.98

0.77 / 0.79 0.85 / 0.98 0.91 / 0.70 0.85 / 0.94

0.17 / 1.00 0.00 / – 0.00 / – 0.00 / –

0.00 / 0.00 0.00 / 0.00 0.00 / 0.00 0.00 / 0.00

Majority vote

0.961

0.553

0.921

0.97 / 0.98

0.97 / 0.97

0.87 / 0.77

0.00 / –

0.00 / 0.00

Table 2: Categorization of the 43 ChEMBL assays where the majority of LLMs disagreed with the ChEMBL BAO assay format label, and how many of each pattern were sent for senior industrial curator review (Table 13). Disagreement pattern

Count

%

Sent for review

LLMs support the BARD label against ChEMBL ChEMBL and BARD agree and LLMs disagree with both ChEMBL and BARD disagree and LLMs disagree with both (novel prediction) No BARD label available; ChEMBL-only ground truth

17 10

40% 23%

4 10

2

5%

2

14

33%

1

Total

43

100%

17

and LLMs agreed with the BARD label and for the last, the expert curator agreed with the LLM prediction. We will elaborate on this in the following paragraph. We further inspected the 43 assays in which the majority of models disagreed with the ChEMBL label in Appendix H. For 29 of these assays, PubChem BARD labels are also available. As we show in Table 2, out of these, in 17 cases (59 %) the LLM consensus agreed with the BARD PubChem label but not ChEMBL. This finding is particularly notable because BARD annotations are generated through a comparatively thorough process that includes additional verification not applied to the ChEMBL labels. For the cases in which ChEMBL and BARD agreed, but LLMs did not, further inspection showed that the models did not unanimously disagree on the ChEMBL label (for full details see Appendix Table 12). If the ChEMBL label is correct in these instances, a structured debate among models could potentially help resolve the disagreement. Expert Annotator Review For 17 assay AIDs, we conducted a further manual review by asking domain experts to classify each assay. We included all assays for which the ChEMBL and PubChem BARD labels agreed but the LLM consensus differed. In addition, we randomly selected four cases in which the LLM consensus agreed with the PubChem BARD label but disagreed with the ChEMBL label. We also included cases in which the LLM consensus disagreed with both the ChEMBL and BARD labels. We shared with the expert annotator the same information as with the LLMs: title, description, comment and protocol. Experts highlighted that classifying these examples was challenging. In 6 cases they agreed with the majority of LLMs immediately. In further six cases, they agreed after seeing the evidence provided by the LLMs. In four additional cases, the experts considered the LLM classification plausible—three of them after reviewing the model-provided evidence—but did not assign a definitive label. They noted that some assays could reasonably fit multiple classes or lacked sufficient information for an unambiguous classification. These observations suggest that uncertainty or multiple applicable labels may be worth representing explicitly in the database. We realized that the annotator initially placed substantial weight on the format type stated in the assay title, for which in many cases they changed their mind after seeing LLM evidence. For example, 6

for AID 2695, the title was “Fluorescence Cell-Based Dose Screen to Determine Inhibitors of C. albicans Viability”. The expert prediction was cell-based and the LLM predicted organism format. Upon closer inspection, and after reviewing the LLMs’ reasoning, the expert concluded that the LLM classification was correct. The expert had initially overlooked the fact that, according to the official BAO definition, “single-celled organisms” should be classified as an organism format. This could be a result of the discrepancy between the assay format and the BAO assay format on ChEMBL. Finally, for assay AID 588769, the LLMs identified a likely mislabeled datapoint in PubChem and agreed with the expert’s classification. For AID 1470, the expert considered the classification highly uncertain; nevertheless, the expert’s assessment was consistent with the LLM consensus and differed from both the ChEMBL and PubChem labels, providing further evidence of alignment between the LLMs and human judgement. Together, these cases suggest that, when grounded in BAO definitions, LLM predictions may help identify errors in both database annotations and initial expert assessments. However, this capability was not consistent across all cases. For AID 504631, the expert agreed with the ChEMBL label rather than with the label supported by the LLM consensus. The expert correctly noted that the assay used a living organism, whereas the LLMs frequently confused this category with a cell-line format (see Appendix H). Thus, although the LLMs identified the relevant evidence, they drew the wrong conclusion from it. To follow up on this, we conducted a second experiment where we used the third-party BAO annotations already available in PubChem. In Table 3 we observe again LLM agreement with BARD labels compared to third party PubChem annotations. In the majority of cases where the BARD label did not align with the third party BAO assay format, LLMs agree with BARD. This highlights that there is a lack of consensus on what the ground truth should be. Table 3: Per-class recall / precision on the PubChem subset. Rare classes in grey. Tissue-based format has n = 0 in the PubChem subset and is omitted. Acc.

mF1

mF1top3

biochemical

cell-based

organism-based

cell-free

(n=120)

(n=100)

(n=4)

(n=2)

vs. third party PubChem Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o

R/P R/P R/P

0.649 0.655 0.668

0.350 0.349 0.356

0.534 0.533 0.551

0.54 / 0.88 0.56 / 0.87 0.59 / 0.87

0.78 / 0.84 0.77 / 0.82 0.76 / 0.84

0.50 / 0.07 0.50 / 0.07 0.50 / 0.09

1.00 / 0.08 1.00 / 0.08 1.00 / 0.07

GPT-OSS 20B Gemma 3 27B Gemma 4 31B Llama 3.3 70B

R/P R/P R/P R/P

0.650 0.765 0.646 0.730

0.353 0.397 0.348 0.403

0.549 0.614 0.547 0.569

0.53 / 0.89 0.73 / 0.86 0.55 / 0.86 0.72 / 0.87

0.79 / 0.84 0.82 / 0.84 0.76 / 0.84 0.75 / 0.83

0.50 / 0.10 0.50 / 0.14 0.50 / 0.11 0.50 / 0.08

1.00 / 0.06 0.50 / 0.08 1.00 / 0.05 1.00 / 0.18

Majority vote

R/P

0.646

0.345

0.533

0.55 / 0.86

0.76 / 0.84

0.50 / 0.08

1.00 / 0.07

(n=50)

(n=108)

(n=11)

(n=22)

vs. BARD Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o

R/P R/P R/P

0.855 0.874 0.817

0.642 0.659 0.561

0.810 0.810 0.727

0.93 / 0.93 0.94 / 0.92 0.96 / 0.86

0.81 / 0.99 0.83 / 0.99 0.81 / 0.99

1.00 / 0.44 1.00 / 0.42 0.55 / 0.30

0.84 / 0.73 0.86 / 0.86 0.68 / 0.58

GPT-OSS 20B Gemma 3 27B Gemma 4 31B Llama 3.3 70B

R/P R/P R/P R/P

0.832 0.785 0.827 0.801

0.574 0.455 0.542 0.545

0.690 0.718 0.665 0.759

0.96 / 0.98 0.96 / 0.67 0.96 / 0.96 0.98 / 0.70

0.81 / 0.99 0.88 / 0.99 0.81 / 0.99 0.81 / 0.99

0.27 / 0.17 0.45 / 0.42 0.18 / 0.12 0.91 / 0.42

0.91 / 0.71 0.09 / 0.18 0.91 / 0.59 0.32 / 0.78

Majority vote

R/P

0.864

0.637

0.789

0.96 / 0.96

0.81 / 0.99

0.82 / 0.38

0.91 / 0.74

5.2 Extending to a Second BAO Axis: Detection Technology To test whether our auditing workflow generalizes beyond a single BAO axis, we repeated the analysis on the physical detection technology axis. BAO’s detection hierarchy contains 12 classes, compared to 6 in assay format, at the depth-one level. Moreover, ChEMBL provides fewer cross-referenced labels, so we treat these results as a complementary probe. Table 4 reports the precision and recall per class. LLMs achieve high recall on most categories, including fluorescence, luminescence and radiometry. The performance on spectrophotometry is more moderate, with the mass spectrometry results and label-free methods having too few labels to draw definitive conclusions. Again, we see no big differences between open-source and proprietary models. 7

Table 4: Per-class recall / precision on the BAO detection technology task. Rare classes (n < 10) in grey. Model

Acc.

mF1

fluorescence (n=141–159)

luminescence (n=67–80)

spectrophotometry (n=54–55)

radiometry (n=10–12)

mass spec. (n=1)

label free (n=1)

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o

0.905 0.899 0.893

0.831 0.819 0.770

0.91 / 0.97 0.92 / 0.95 0.89 / 0.99

0.99 / 0.94 0.96 / 0.93 0.96 / 0.93

0.80 / 0.91 0.75 / 0.93 0.80 / 0.96

0.90 / 0.90 0.83 / 0.91 0.83 / 0.91

1.00 / 0.20 1.00 / 0.20 1.00 / 0.20

1.00 / 1.00 1.00 / 1.00 1.00 / 0.50

GPT-OSS 20B Gemma 3 27B Gemma 4 31B Llama 3.3 70B

0.883 0.886 0.892 0.883

0.817 0.822 0.719 0.817

0.89 / 0.97 0.89 / 0.97 0.92 / 0.97 0.89 / 0.96

0.96 / 0.93 0.96 / 0.94 0.95 / 0.94 0.96 / 0.93

0.75 / 0.93 0.76 / 0.95 0.75 / 0.93 0.76 / 0.91

0.83 / 0.91 0.83 / 0.91 0.83 / 0.91 0.83 / 0.91

1.00 / 0.20 1.00 / 0.20 1.00 / 0.20 1.00 / 0.20

1.00 / 1.00 1.00 / 1.00 1.00 / 0.25 1.00 / 1.00

Majority vote

0.890

0.819

0.91 / 0.96

0.96 / 0.94

0.75 / 0.93

0.83 / 0.91

1.00 / 0.20

1.00 / 1.00

To understand if the performance against silver labels is representative, and whether other questionable entries in PubChem might benefit from further review, we manually inspected 34 assays where the majority of LLMs disagreed with the label. BARD disagreed with the third-party annotation in the bulk download in all 24 cases where a BARD annotation was available. In 15 out of these 24 cases, our LLMs agreed with BARD, highlighting that the predictions that disagreed with the "silver" truth might be predominantly attributable to inaccuracies in the silver labels rather than genuine model error. A recurring source of ambiguity was the occurrence of assay-supporting methods, such as "quantitative reverse transcription PCR" and "cell number determination," which do not map cleanly onto any BAO physical detection method subcategory. LLMs variably placed such assays under simple measurement, visual observation, or a physically-adjacent readout method (e.g., fluorescence, when the underlying chemistry, such as TaqMan, is fluorescent based), suggesting that forcing LLMs to output a single physical detection method is suboptimal for some assays. There was one AID (1259354) with the notation “AlphaLISA: fluorescence intensity” in the assay dataset. We were unsure of whether AlphaLISA should be in the fluorescence methods, the luminescence method or the proximity-based method. All our LLMs tested agreed that it should be the luminescence method, which given the alphascreen method being in that category, could be reasonable. To probe whether the remaining disagreements between BARD or ChEMBL and the LLMs could be explained, we asked the expert curator to review 11 records. In 6 cases the expert judged the assay genuinely undefinable and in every one of these, non-erroring models likewise self-reported low confidence, evidence that models can recognize when a physical detection method truly cannot be determined from the text, even as they still emit a best-guess label. In a further 3 cases (588778, 624286, 651578), the expert’s own annotation explicitly named more than one acceptable detection method rather than a single ground truth: for 588778, the expert confirmed the assay genuinely uses both spectrophotometry (growth) and fluorescence (motility discrimination), while for 624286 and 651578 the expert accepted visual observation as one of several defensible labels for manual plaque counting. Only 2 of the 11 assays remain cases of true single-label disagreement: for AID 623918, the expert’s chosen label (simple measurement) matched the model majority (4 of 7 models) against the label in BARD. For AID 623968, the expert’s label (spectrophotometry) was tied 3–3 among non-error models against fluorescence, reflecting a genuine two-step-protocol ambiguity (RNA quantification by absorbance vs. RT-PCR detection via TaqMan fluorescence) rather than an outright model error. For the full analysis we refer to Appendix I. 5.3 Calibrated confidence and Prompt Ablations Intrigued by the finding that all the LLMs labelled the assay which the expert labeled as “cannot define” with low confidence, we examined the relationship between LLM-reported confidence and correctness against the silver label (see Appendix J). For the 1097 ChEMBL datapoints, over all the seven LLMs tried, we saw that for the medium confidence labels given by Claude and GPT-OSS 20B, the disagreement rate at 33% and 100% was much higher than for the instances predicted high confidence and a disagreement rate of only up to 5%. For the detection task we saw more spread confidence. On the high confidence labels there is a disagreement rate of up to only 10%, with higher disagreement for the medium high labels and low confidence labels. It is also noteworthy that only 8

Claude Sonnet 4.6 and GPT-OSS 20B model use the medium confidence, though other models give low confidence to some of their predictions as well. Combined with reasoning traces, this enables a practical triage: high-confidence predictions that agree with BARD can be accepted; high-confidence disagreements are surfaced with their supporting evidence for human review; and low-confidence predictions are routed to curators along with the model’s reasoning. Nevertheless, further verification is necessary, especially since these are disagreement rates against silver labels. Next to having a calibrated confidence, it is important that the LLMs predict based on the information given, not merely the order of the labels presented. To investigate the sensitivity of our results to prompt design, we evaluated two perturbations of the baseline prompt on the ChEMBL assay format subset. First the order swap where we reversed the alphabetical order of the initial experiment. Second, the format without definitions, which removed the BAO class definitions and only keeps the assay format labels. Results show that when swapping the order, labels change in less than 1.2% of the cases for proprietary models and 2.6% for the open-source models, indicating low ordering bias (see Appendix K). Removing the definitions, however, changes 3.1%, 10.8%, and 3.8% of predictions for Claude Sonnet 4.6, Gemini 3.6 Flash, and GPT-4o respectively. Gemini in particular shows more changes away from the silver label, which implies that Gemini might rely more heavily on the in-prompt definitions. For the open-source models the change is 4.2% for Llama 3.3 70B, up to 8.8% for GPT-4o. Interestingly, without the definitions there are also a number of flips without the definitions towards the silver label. This could mimic the human pattern of guessing the definition and consequently assigning it to a certain (silver) label, without grounding it in the exact definition asked for.

6

Discussion and Conclusion

Our audit indicates that PubChem’s bulk distribution offers incomplete assay metadata: roughly 36% of records lack an assay format, and fewer than 0.1% carry a structured BAO annotation. We also observe non-trivial disagreement across resources, 10% among the 236 assays with both BARD and ChEMBL format labels available in the bulk download. These gaps likely reflect the optional nature of several metadata fields at deposition time, together with the limited vocabulary of PubChem’s native Assay Format field relative to the richer BAO hierarchy. To address this gap, we evaluated seven LLMs, both proprietary and open-source, and found that they agree strongly with each other and recover well-supported assay format classes with recall above 0.96. Manual inspection of the cases where LLMs disagreed with silver labels showed that these errors are frequently traceable to inconsistencies between sources, or to mislabeling, rather than to the models themselves: in over a third of cases where the majority LLM prediction disagreed with ChEMBL, the LLMs in fact aligned with the PubChem/BARD label, and our senior annotator revised their own labels in several cases after seeing the model’s cited evidence. Differences between proprietary and open-source models were small throughout, suggesting cheaper open-weight models are a viable option for annotation at scale, though at a cost of $3–9 to audit 1000 records even proprietary models are orders of magnitude cheaper than manual re-curation(Appendix L). Taken together, results support a cautious claim that LLMs may improve upon individual existing annotations in cases of source disagreement. Based on this, we foresee a specific role for LLMs in the annotation pipeline, both as annotators and as auditors. LLMs flag inconsistent records, surface the textual evidence behind a disagreement, and route ambiguous cases to human curators, with per-class reliability estimates attached before any label is allowed to enter a downstream ML pipeline. These findings and the proposed framing come with limitations that follow from our results, silver labels are only partially reliable, so recall computed against them should be read as agreement with existing curation, not as ground-truth accuracy; where BARD, ChEMBL, and PubChem disagree, recall figures for the “losing” source will look artificially low even when the model is correct. Moreover, our evaluation stays at BAO depth one, and the live PUG JSON endpoint, the only consistently up-to-date source, currently rate-limits exhaustive crawls, which constrained how deep and how broad our dataset could go when sticking to PubChem. Our qualitative validation also has its own limitations: expert review was conducted with a single senior curator, so our findings reflect one expert’s judgment rather than consensus across annotators. Finally, we identified one failure mode worth flagging explicitly: for AID 504631, several LLMs quoted the correct evidence (that Plasmodium falciparum is a single-celled organism) yet still defaulted to a cell-based label instead of organism-based. This shows that even high-confidence 9

predictions backed by correct evidence can fail at the final step of mapping that evidence onto the BAO class definitions, and argues for continued human spot-checking of whether the cited evidence actually supports the assigned label in these known tricky cases. Beyond auditing existing annotations, the process itself surfaced gaps in BAO: the ontology currently lacks a definition for the “biosensor” detection technology, and the definitions of “simple measurement” and “visual observation” methods proved ambiguous enough to divide both LLMs and our expert curator. This suggests that LLM-based auditing can feed back into ontology maintenance, not only into data curation. As next steps, we would like to experiment with deeper BAO granularity where support permits, expanding the dataset that we test on, utilize industrial datasets and LLM-mediated debate to resolve residual silver-label conflicts, ingredients we believe necessary to bring legacy bioassay repositories to the readiness that foundation models for molecular property prediction now demand. LLM disclosure. LLMs were extensively used throughout this work. Primarily, as part of the experimental setup. LLMs were used for brainstorming, input for paper writing, and supported coding, mainly for plotting and analysis of the results. All written content was authored and edited by the authors.

7

Acknowledgments

This research was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Computational resources were provided on the Berzelius system funded by the Knut and Alice Wallenberg foundation and operated by NAISS in combination with resources provided by AstraZeneca.

10

References [1] [2] [3]

[4]

[5]

[6]

[7]

[8]

[9]

[10] [11]

[12] [13] [14] [15]

[16]

[17]

[18]

[19]

[20]

M. D. Wilkinson et al. “The FAIR Guiding Principles for scientific data management and stewardship”. In: Scientific Data 3.1 (Mar. 2016). DOI: 10.1038/sdata.2016.18. S. Kim et al. “PubChem 2025 update”. In: Nucleic Acids Research 53.D1 (Nov. 2024), pp. D1516–D1525. DOI: 10.1093/nar/gkae1059. U. Visser et al. “BioAssay Ontology (BAO): a semantic description of bioassays and highthroughput screening results”. In: BMC Bioinformatics 12.1 (2011). DOI: 10.1186/14712105-12-257. S. Abeyruwan et al. “Evolving BioAssay Ontology (BAO): modularization, integration and applications”. In: Journal of Biomedical Semantics 5.S1 (2014). DOI: 10.1186/2041-14805-s1-s5. O. Cinquin. “ChIP-GPT: a managed large language model for robust data extraction from biomedical database records”. In: Briefings in Bioinformatics 25.2 (Jan. 2024). DOI: 10.1093/ bib/bbad535. S. Ikeda et al. “Extraction of biological terms using large language models enhances the usability of metadata in the BioSample database”. In: GigaScience 14 (2025). DOI: 10.1093/ gigascience/giaf070. B. Zdrazil et al. “Transporter assays and assay ontologies: useful tools for drug discovery”. In: Drug Discovery Today: Technologies 12 (2014), e47–e54. DOI: 10.1016/j.ddtec.2014. 03.005. U. D. Vempati et al. “Formalization, Annotation and Analysis of Diverse Drug and Probe Screening Assay Datasets Using the BioAssay Ontology (BAO)”. In: PLoS ONE 7.11 (Nov. 2012). Ed. by D. Cox, e49198. DOI: 10.1371/journal.pone.0049198. L. Zander Balderud et al. “Using the BioAssay Ontology for Analyzing High-Throughput Screening Data”. In: SLAS Discovery 20.3 (Mar. 2015), pp. 402–415. DOI: 10 . 1177 / 1087057114563493. Y. Wang et al. “PubChem’s BioAssay Database”. In: Nucleic Acids Research 40.D1 (Dec. 2011), pp. D400–D412. DOI: 10.1093/nar/gkr1132. E. Howe et al. “BioAssay Research Database (BARD): chemical biology and probedevelopment enabled by structured metadata and result types”. In: Nucleic Acids Research 43.D1 (Dec. 2014), pp. D1163–D1170. DOI: 10.1093/nar/gku1244. N. L. of Medicine. BioAssay Research Database (BARD). PubChem data source, Data Source ID 15370. Last updated July 25, 2019. Accessed August 27, 2026. 2019. A. Gaulton et al. “ChEMBL: a large-scale bioactivity database for drug discovery”. In: Nucleic Acids Research 40.D1 (2011), pp. D1100–D1107. DOI: 10.1093/nar/gkr777. D. Mendez et al. “ChEMBL: towards direct deposition of bioassay data”. In: Nucleic Acids Res. 47.D1 (Jan. 2019), pp. D930–D940. B. Zdrazil et al. “The ChEMBL Database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods”. In: Nucleic Acids Research 52.D1 (Nov. 2023), pp. D1180–D1192. DOI: 10.1093/nar/gkad1004. I. Smit et al. “Integrating artificial intelligence and manual curation to enhance bioassay annotations in ChEMBL”. In: Journal of Cheminformatics 18.1 (Feb. 2026). DOI: 10.1186/ s13321-026-01165-x. V.-K. Tran-Nguyen, C. Jacquemard, and D. Rognan. “LIT-PCBA: An Unbiased Data Set for Machine Learning and Virtual Screening”. In: Journal of Chemical Information and Modeling 60.9 (Apr. 2020), pp. 4263–4273. DOI: 10.1021/acs.jcim.0c00155. D. Buterez et al. “MF-PCBA: Multifidelity High-Throughput Screening Benchmarks for Drug Discovery and Machine Learning”. In: Journal of Chemical Information and Modeling 63.9 (Apr. 2023), pp. 2667–2678. DOI: 10.1021/acs.jcim.2c01569. J. B. Harris et al. “ScrubChem: Cleaning of PubChem Bioassay Data to Create Diverse and Massive Bioactivity Datasets for Use in Modeling Applications (SOT)”. In: (2017). DOI: 10.23645/EPACOMPTOX.5176861. D. A. Araripe et al. “CAPRICHO: Interpretable Quality Flagging and Flexible ChEMBL Bioactivity Curation for QSAR Modeling”. In: Journal of Chemical Information and Modeling (2026). DOI: 10.1021/acs.jcim.6c00802.

11

[21] R. S. Gonçalves et al. “The text2term tool to map free-text descriptions of biomedical terms to ontologies”. In: Database 2024 (2024). DOI: 10.1093/database/baae119. [22] M. Taboada et al. “Ontology matching with Large Language Models and prioritized depth-first search”. In: Information Fusion 123 (Nov. 2025), p. 103254. DOI: 10.1016/j.inffus.2025. 103254. [23] F. Liu et al. “Self-Alignment Pretraining for Biomedical Entity Representations”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021, pp. 4228–4238. DOI: 10.18653/v1/2021.naacl-main.334. [24] O. Olasunkanmi et al. “RELATE: Relation Extraction in Biomedical Abstracts with LLMs and Ontology Constraints”. In: Proceedings of the Fifth Machine Learning for Health Symposium. Ed. by P. Argaw et al. Vol. 297. Proceedings of Machine Learning Research. PMLR, Dec. 2026, pp. 1178–1193. DOI: 10.48550/arXiv.2509.19057. [25] J. Cohen. “A Coefficient of Agreement for Nominal Scales”. In: Educational and Psychological Measurement 20.1 (Apr. 1960), pp. 37–46. DOI: 10.1177/001316446002000104.

12

Appendix A

Dataset Creation Details

To evaluate the performance of LLMs for predicting bioassay metadata classes, we performed a bulk download from PubChem BioAssay of the following resources: the JSON zip files4 , bioassays.tsv.gz5 , Aid2CategorizedComment.gz6 , and Aid2Annotation.gz7 . Seven of the JSON zip files, containing categorized bioassay names, comments, and descriptions, were empty at the time of download: 1258001_1259000, 1343001_1344000, 1344001_1345000, 1641001_1642000, 1642001_1643000, 1643001_1644000, and 1644001_1645000. This is itself a small illustration of the data-quality issues affecting PubChem’s bulk distribution that motivate this work. In total, we successfully downloaded 1 994 310 BioAssay AIDs, of which 1 770 568 contained a name, a comment, and/or a description. To mitigate this missing data, for the AIDs used in our analysis we additionally queried PubChem’s PUG REST endpoint8 directly, which contained a description for all assays of interest.

B

Dataset information from bulk PubChem

Tables 5, 6, 7, 8, and 9 report the statistics of the data collected from the PubChem BioAssay database bulk download. Raw data can be downloaded from https://ftp.ncbi.nlm.nih.gov/pubchem/ Bioassay/. Table 5: Assay coverage and uniqueness for description, comment, and name fields (Total AIDs = 1 994 310) Field AID Comment Description Name Deposit Date Assay Format Comment.BAO: assay format Comment.BAO: detection technology

Non-empty

% of Total

Unique Values

% Unique (of non-empty)

1 994 310 1 769 245 1 770 567 1 770 568 1 910 340 1 278 238 280 348

100.0% 88.7% 88.8% 88.8% 95.8% 64.1% 0.0% 0.0%

1 994 310 647 580 173 040 1 559 216 1 979 3 8 16

100.00% 36.60% 9.77% 88.06% 0.10% 0.00% 2.50% 4.31%

B.1 Dataset tables investigated labels statistics The tables 6, 7, 8, and 9 investigate for each of the selected labels the coverage over which description, comment, or name are present. Table 6: Distribution of assay formats available after bulk-downloading and processing PubChem BioAssay, broken down by commented vs. total counts and comment coverage. With Comment counts assays that have a description, comment, or name; Total counts all assays in the category; Coverage is the annotation rate per category (With Comment / Total); Percentage is each category’s share of the commented subset. Assay Format

With Comment

Total

Coverage

Percentage

Organism-based Cell-based Biochemical

609 321 541 113 101

663 381 614 738 119

91.85% 88.02% 84.87%

52.96% 47.03% 0.01%

1 150 535

1 278 238

90.01%

100.00%

Total

4 https://ftp.ncbi.nlm.nih.gov/PubChem/Bioassay/JSON/ 5 https://ftp.ncbi.nlm.nih.gov/PubChem/Bioassay/Extras/bioassays.tsv.gz 6 https://ftp.ncbi.nlm.nih.gov/PubChem/Bioassay/Extras/Aid2CategorizedComment.gz 7 https://ftp.ncbi.nlm.nih.gov/PubChem/Bioassay/Extras/Aid2Annotation.gz 8 https://PubChem.ncbi.nlm.nih.gov/rest/pug_view/data/bioassay/\{aid\}/JSON

13

Table 7: Distribution of BAO Assay Formats available after bulk-downloading and processing PubChem BioAssay, broken down by commented vs. total counts and comment coverage. With Comment counts assays that have a description, comment, or name; Total counts all assays in the category; Coverage is the annotation rate per category (With Comment / Total); Percentage is each category’s share of the commented subset. BAO Assay Format

With Comment

Total

Coverage

Percentage

cell-based format biochemical format: protein format: single protein format biochemical format: protein format: protein complex format cell based format biochemical format: protein format: Single protein format organism-based format cell-free format: subcellular: cytosol format

112 91 27 9 5 4 2

119 103 34 12 6 4 2

94.12% 88.35% 79.41% 75.00% 83.33% 100.00% 100.00%

44.80% 36.40% 10.80% 3.60% 2.00% 1.60% 0.80%

Total

250

280

89.29%

100.00%

Table 8: Distribution of BAO Detection Technologies available after bulk-downloading and processing PubChem BioAssay, broken down by commented vs. total counts and comment coverage. With Comment counts assays that have a description, comment, or name; Total counts all assays in the category; Coverage is the annotation rate per category (With Comment / Total); Percentage is each category’s share of the commented subset. BAO Detection Technology

With Comment

Total

Coverage

Percentage

AlphaLISA: fluorescence intensity absorbance fluorescence: flow cytometry fluorescence: fluorescence intensity fluorescence: fluorescence polarization fluorescence: fret: htrf fluorescence: fret: tr-fret label free technology: isothermal titration calorimetry label free technology: mass spectrometry luminescence: bioluminescence luminescence: chemiluminescence radiometry: scintillation counting: filter assay radiometry: scintillation counting: scintillation proximity assay spectrophotometry: absorbance spectrophotometry: absorbance_ | | _fluorescence: fluorescence intensity

1 2 37 112 21 2 10 1 1 1 79 2 2 46 1

1 2 37 115 24 3 10 1 1 1 86 6 6 54 1

100.00% 100.00% 100.00% 97.39% 87.50% 66.67% 100.00% 100.00% 100.00% 100.00% 91.86% 33.33% 33.33% 85.19% 100.00%

0.31% 0.63% 11.64% 35.22% 6.60% 0.63% 3.14% 0.31% 0.31% 0.31% 24.84% 0.63% 0.63% 14.47% 0.31%

Total

318

348

91.38%

100.00%

14

Table 9: Distribution of BioAssay Types available after bulk-downloading and processing PubChem BioAssay, broken down by commented vs. total counts and comment coverage. With Comment counts assays that have a description, comment, or name; Total counts all assays in the category; Coverage is the annotation rate per category (With Comment / Total); Percentage is each category’s share of the commented subset. BioAssay Type With Comment Total Coverage Percentage Biochemical Biochemical | Cell-based Biochemical | Cell-based | In vivo Biochemical | Cell-based | Toxicity Biochemical | In vitro Biochemical | In vivo Biochemical | RNAi Biochemical | Toxicity Cell-based Cell-based | In vitro Cell-based | In vivo Cell-based | Organism-based Cell-based | Organism-based | Toxicity Cell-based | RNAi Cell-based | Toxicity In vitro In vitro | In vivo In vitro | Organism-based In vitro | Toxicity In vivo In vivo | Organism-based In vivo | Organism-based | Toxicity In vivo | Toxicity Organism-based Organism-based | Toxicity RNAi RNAi | Toxicity Toxicity

1 866 18 1 2 24 68 1 337 3 578 49 2 27 3 52 289 25 224 30 67 1 644 22 663 12 3 255 573 28 115 7 149 444

3 155 22 1 3 48 68 1 385 4 184 55 2 39 3 52 347 26 604 53 67 1 658 23 965 12 3 266 602 40 124 8 167 493

59.14% 81.82% 100.00% 66.67% 50.00% 100.00% 100.00% 87.53% 85.52% 89.09% 100.00% 69.23% 100.00% 100.00% 83.29% 94.81% 56.60% 100.00% 99.16% 94.57% 100.00% 100.00% 95.86% 95.18% 70.00% 92.74% 87.50% 89.22%

0.90% 0.01% 0.00% 0.00% 0.01% 0.03% 0.00% 0.16% 1.73% 0.02% 0.00% 0.01% 0.00% 0.03% 0.14% 12.22% 0.01% 0.03% 0.80% 10.98% 0.01% 0.00% 0.12% 0.28% 0.01% 0.06% 0.00% 72.41%

Total

206 382

229 260

90.02%

100.00%

15

C

BAO format tree small-molecule physicochemical format BAO_0000100

protein format BAO_0000224 biochemical format BAO_0000217

protein complex format BAO_0000223 single protein format BAO_0000357

nucleic acid format BAO_0000225 organism-based format BAO_0000218 cell-based format BAO_0000219

Transwell chamber format BAO_0020015

tissue-based format BAO_0000221

nucleosome format BAO_0000007 cell membrane format BAO_0000249

assay format component BAO_0003113

assay format BAO_0000019

cytosol format BAO_0000250 microsome format BAO_0000251 mitochondrion format BAO_0000252 subcellular format BAO_0000220

nuclear extract format BAO_0000253 lysate based BAO_0000408 liposome format BAO_0020002 S9 fraction format BAO_0020010 tissue membrane format BAO_0020012 synaptosomal fraction format BAO_0020013

cell-free format BAO_0000366

whole cell lysate format BAO_0000254

synaptosomal P2 fraction format BAO_0020014

rabbit reticulocyte lysate format BAO_0000255

plasma format BAO_0020003 serum format BAO_0020004 blood format BAO_0020011

Figure 1: Figure was made by downloading the BAO complete OWL and creating the subtree in PowerPoint.

16

D

BAO detection tree

Figure 2: Figure was made by downloading the BAO complete OWL and creating the subtree in PowerPoint.

17

E

Prompts used

Three prompt variants were evaluated for classifying bioassay records into one of five BAO assay format classes (cell-based, biochemical, organism-based, tissue-based, cell-free) using Claude via AWS Bedrock. We left out the small-molecule physicochemical format in the prompt as this was not part of our silver labels. All variants share the same input schema (Title / Description / Comment / Protocol) and the same JSON output contract; they differ only in how the label and definitions are presented. For the physical detection method we used a prompt similar to the baseline. • Experiment 1 — Baseline: full BAO definitions, of assay format, alphabetical class order. • Experiment 2 — Order swap: full BAO definitions, reversed class order (probes ordering bias). • Experiment 3 — Without definitions: class names only, no definitions (probes reliance on label semantics vs. prior knowledge). • Experiment 4 — Physical Detection Method: full BAO definitions of assay physical detection technology.

18

Experiment 1 — baseline (with definitions) Presents each of the 5 classes with its official BAO definition and subcategory list. Order: biochemical → cell-based → cell-free → organism-based → tissue-based. System prompt — Baseline Assay Format Prompt You are an expert curator working with the BioAssay Ontology (BAO), a structured vocabulary used by ChEMBL and PubChem to standardize how bioassay experiments are described. The category definitions below are taken directly from BAO’s official class definitions. Given the Description, Comment, and Protocol of a bioassay record (when available), classify its assay format into exactly one of the following categories, based on the BioAssay Ontology (BAO). Categories (with definitions and typical phrasing you may see): 1. biochemical format An in vitro format used to measure the activity of a biological macromolecule (purified protein or nucleic acid). Most often homogeneous, but can be heterogeneous when a solid phase (e.g. beads) immobilizes the macromolecule. Subcategories: protein format, nucleic acid format, protein complex format, single protein format. 2. cell-based format Uses living cells of eukaryotic origin, derived from a multicellular organism; heterogeneous assay type. Typically permanent (immortal) cell lines, primary cells, or stem cells. Distinct from single-celled organism-based formats (e.g. bacteria, yeast). Subcategory: Transwell chamber format. 3. cell-free format Originates from a cell but does not use intact (live) cells; excludes biochemical assays. Usually homogeneous; heterogeneous when components are bead-immobilized. Subcategories: subcellular, whole cell lysate, plasma, serum, blood, nucleosome, cell membrane, cytosol, microsome, mitochondrion, nuclear extract, lysate-based, liposome, S9 fraction, tissue membrane, synaptosomal fraction, rabbit reticulocyte lysate, synaptosomal P2 fraction. 4. organism-based format Uses a living organism, single or multi-celled; heterogeneous assay type. 5. tissue-based format Uses a tissue derived from a living organism; heterogeneous assay type. Rules: • Use the strongest evidence across all provided fields. • If multiple categories seem plausible, pick the one most directly supported by explicit terminology in the text, not inferred biology. • Respond with valid JSON only — no markdown, no extra text. Output format: {"bao_format": "<cell-based format|biochemical format| organism-based format|tissue-based format| cell-free format>", "confidence": "<high|medium|low>", "evidence": "<short quote or paraphrase from the text that supports your choice>"}

Experiment 2 - Order swap (with definitions, reversed order) Identical wording and definitions as Experiment 1, but the class list is reversed: tissue-based → organism-based → cell-free → cell-based → biochemical. Isolates the effect of presentation order on the model’s choice.

19

System prompt — order_swap You are an expert curator working with the BioAssay Ontology (BAO). . . (introductory paragraph identical to Experiment 1) Categories (with BAO definitions): 1. tissue-based format — Uses a tissue derived from a living organism; heterogeneous assay type. 2. organism-based format — Uses a living organism, single or multi-celled; heterogeneous assay type. 3. cell-free format — Originates from a cell but does not use intact (live) cells; excludes biochemical assays. Full subcategory list identical to Experiment 1. 4. cell-based format — Uses living cells of eukaryotic origin from a multicellular organism; heterogeneous. Immortal cell lines, primary cells, or stem cells. Distinct from single-celled organism-based formats. 5. biochemical format — In vitro format to measure activity of a purified biological macromolecule (protein or nucleic acid). Subcategories: protein, nucleic acid, protein complex, single protein. Rules and Output format are identical to Experiment 1.

Experiment 3 — Without definitions (labels only) Strips out all BAO definitions and subcategory lists. The model sees only the five class names and must rely on its pretraining prior about what each term means. Measures how much of the classifier’s performance depends on the definitions vs. label semantics. System prompt — results_without_definitions You are an expert curator working with the BioAssay Ontology (BAO). . . (introductory paragraph identical to Experiment 1) Given the Description, Comment, and Protocol of a bioassay record (when available), classify its assay format into exactly one of the following categories, based on the BioAssay Ontology (BAO). Categories: 1. biochemical format 2. cell-based format 3. cell-free format 4. organism-based format 5. tissue-based format Rules: • Use the strongest evidence across all provided fields. • If multiple categories seem plausible, pick the one most directly supported by explicit terminology in the text, not inferred biology. • Respond with valid JSON only — no markdown, no extra text. Output format: identical to Experiments 1 and 2.

20

Experiment 4 — baseline (with definitions), detection method Presents each of the 12 BAO physical detection method classes with its official definition and subcategory list. Order: biosensor → fluorescence → imaging → isometric tension recording → label free → luminescence → mass spectrometry → proximity-based → radiometry → simple measurement → spectrophotometry → visual observation. System prompt — Baseline Physical Detection Method Prompt You are an expert curator working with the BioAssay Ontology (BAO), a structured vocabulary used by ChEMBL and PubChem to standardize how bioassay experiments are described. The category definitions below are taken directly from BAO’s official class definitions. Given the Description, Comment, and Protocol of a bioassay record (when available), classify its physical detection method into exactly one of the following categories, based on the BioAssay Ontology (BAO). Categories (with definitions): 1. biosensor method This category has no subcategories. 2. fluorescence method Detection techniques that use the principles of fluorescence, whereby incident light excites a fluorophore which then emits light at lower energy (higher wavelength). The emitted light is typically from the visible portion of the UV–Visible spectrum. Subcategories: fluorescence resonance energy transfer, fluorescence polarization, flow cytometry, thermal shift, fluorescence intensity, homogeneous time resolved fluorescence, time resolved fluorescence resonance energy transfer. 3. imaging method The field of using microscopes to see objects that cannot be seen with the unaided eye. Subcategories: tomography, microscopy, electron microscopy, optical microscopy, scanning probe microscopy, scanning electron microscopy, transmission electron microscopy, fluorescence microscopy, confocal microscopy, brightfield microscopy, phase contrast microscopy, dark field microscopy, differential interference contrast microscopy, scanning near-field optical microscopy, vibrational spectroscopic imaging, atomic force microscopy, scanning tunneling microscopy, total internal reflection fluorescence microscopy, Raman imaging, infrared imaging, near-infrared imaging, mid-infrared imaging. 4. isometric tension recording method Isometric tension recording is a technique used to measure tension. This category has no subcategories. 5. label free method Allows detection of binding interactions and cell-based reactions without conventional labels (e.g. fluorescent probes). Advantages: (a) functional activity without modifying binding partners with labels, (b) binding interactions independent of functional activity, and (c) cell-based GPCR assays without engineering cell lines to over-express receptors. Subcategories: optical based, nuclear magnetic resonance, circular dichroism, electrical sensor, x-ray crystallography, isothermal titration calorimetry, quartz crystal microbalance, surface plasmon resonance, bio layer interferometry, fiber optic waveguide, optical waveguide grating, resonant waveguide grating, microelectrode measurement, carbon nanotube based sensor, current clamp, patch clamp, voltage clamp. 6. luminescence method Emission of light from a substance from an electronically excited state reached by a physical, mechanical, or chemical mechanism. Subcategories: bioluminescence, chemiluminescence, AlphaScreen, bioluminescence resonance energy transfer, chemiluminescence resonance energy transfer. 7. mass spectrometry Used to detect compounds bound to protein targets; unbound compounds are separated by affinitybased, size-exclusion, or ultrafiltration methods, then bound compounds are eluted from the protein and detected by MS. Throughput varies with the separation method (typically 200–2700 compounds per run). Also used to determine the chemical structure of molecules: samples are vaporized, ionized, and the mass-to-charge ratio of the particles is analyzed. This category has no subcategories. 8. proximity-based method A biophysical approach used to monitor the spatial closeness of proteins or molecules within live cells by exploiting measurable signals generated through interactions (e.g. energy transfer) between tagged molecular components. For example, Bioluminescence Resonance Energy Transfer (BRET) uses

21

energy transfer from a donor enzyme (e.g. luciferase) to an acceptor fluorophore, giving a quantifiable proximity signal. Widely applied to protein–protein interactions, protein–ligand interactions, receptor proximity, and intracellular signaling. Advances such as NanoBRET and NanoBiT have expanded scope further. This category has no subcategories. 9. radiometry method Detection technology using radioactive tracers (e.g. filter assay, SPA). Subcategories: scintillation counting, radiometric imaging method, Cherenkov counting method, gamma counting method, autoradiography, phosphorimaging. 10. simple measurement method Any method entailing basic measurements requiring no special instrumentation (e.g. measuring tumor size with calipers, or a subject’s weight with a scale). This category has no subcategories. 11. spectrophotometry method Uses a spectrophotometer to measure the amount of light a sample absorbs: a beam of light is passed through the sample and the intensity of light reaching a detector is measured. Subcategories: absorbance, transmittance, atomic absorption spectrophotometry. 12. visual observation method Any method entailing primarily routine examination, requiring no special instrumentation. This category has no subcategories. Rules: • Use the strongest evidence across all provided fields. • If multiple categories seem plausible, pick the one most directly supported by explicit terminology in the text, not inferred biology. • Respond with valid JSON only — no markdown, no extra text. Output format: {"bao_format": "<biosensor method|fluorescence method| imaging method|isometric tension recording method| label free method|luminescence method| mass spectrometry|proximity-based method| radiometry method|simple measurement method| spectrophotometry method|visual observation method>", "confidence": "<high|medium|low>", "evidence": "<short quote or paraphrase from the text that supports your choice>"}

Shared user-turn template All experiments use the same per-record user message; missing fields are replaced with the literal marker (not provided) so the model is never shown nan or a silently blank line. User prompt template (shared across all experiments) Title: $record_title Description: $description Comment: $comment Protocol: $protocol

22

E.1 Case studies of annotation inconsistencies in PubChem To illustrate the annotation inconsistencies summarized in Section 4, in this section we discuss two representative case studies provide further statistics on the BARD-ChEMBL cross-source disagreements. Case 1: Conflicting assay format labels (AID 1885). AID 1885 is reported as Cell-based in the PubChem web interface, but as Cell-based|Organism-based in the bulk download. Although it is difficult to determine exactly how BioAssay Types are annotated, the protocol and description could plausibly support both categories: the assay runs in mouse fibroblasts (suggesting a cellbased format), while the target is a living microorganism (which could suggest an organism-based format). According to the BAO, however, the organism-based format label refers only to intact living organisms9 , so it should not apply here since no intact organism was used. The PubChem web interface correctly displays only the cell-based annotation. The same ambiguity likely affects the other 42 entries carrying Cell-based|Organism-based annotations in PubChem, as well as many further combinations such as In Vitro|Organism-based. Case 2: Disagreement between third-party sources (AID 588585). Next to annotations from external sources, PubChem also displays assay annotations obtained from third-party sources, and these do not always agree with each other or with the BARD annotations. Furthermore, the BARDlinked bulk download10 does not cover all BARD-annotated assays shown on PubChem. AID 588585 presents an example for this. It is labelled “biochemical” under the categorised-comment BAO field but “subcellular” in the BARD-derived BioAssay Annotations tab, despite not appearing in the BARD bulk download at all. Across the JSON files corresponding to AIDs in the BARD bulk download, we found only one third-party annotated BAO label, on which BARD and the third party did agree. Agreements on BARD–ChEMBL labels In the bulk download of BARD we found 236 assays with both BARD and ChEMBL assay format labels on PubChem. Out of these, on 24 (10.2%) assays, the two sources disagreed on the label. The disagreements comprised: 12 cases of Cell-based vs. Organism-based; 6 cases of Biochemical vs. Cell-based; 3 cases of Tissue-based vs. Cell-based; remaining cases including small-molecule physicochemical (BARD) vs. Biochemical (ChEMBL). Together, these observations show that even highly manual, expert-driven curation does not guarantee consistency, either within a single record or across sources.

9 https://bioportal.bioontology.org/ontologies/BAO?p=classes&conceptid=http%3A%2F%2Fwww. bioassayontology.org%2Fbao%23BAO_0000218 10 See https://pubchem.ncbi.nlm.nih.gov/source/BioAssay%20Research%20Database%20(BARD).

23

F

Model Execution Parameters

This appendix covers the exact runtime configuration used for each of the seven models evaluated in this work. Any response whose predicted label fell outside the allowed BAO class set was flagged as invalid. F.1 Structured output To make outputs comparable across providers, JSON output was enforced at the API level wherever supported: • GPT-4o (Azure OpenAI) and Gemini 3.6 Flash (Vertex AI, OpenAI-compatible passthrough): response_format={"type": "json_object"}. • Claude Sonnet 4.6 (AWS Bedrock Converse API): no native JSON-mode flag is exposed by the Converse API, so JSON adherence was obtained purely via prompt instructions and parsed. • Open-weights models (Ollama v0.17.4): a JSON-Schema was passed via the format parameter in such a way that the model literally cannot emit a label outside the allowed vocabulary. F.2

Proprietary models (API-served) Parameter

GPT-4o

Gemini 3.6 Flash

Claude Sonnet 4.6

Provider / route

Azure OpenAI

Endpoint / API

chat.completions

AWS Bedrock (via AI Gateway) bedrock-runtime.converse

Model / deployment ID

gpt-4o

Temperature Max output tokens JSON-mode enforced?

0.0 provider default yes (json_object)

Vertex AI (via AI Gateway) OpenAI-compatible passthrough google/ gemini-3.6-flash 0.0 provider default yes (json_object)

us.anthropic. claude-sonnet-4-6 0.0 provider default no (prompt-only)

Table 10: Runtime configuration for the three proprietary API-served models. Temperature was fixed at 0.0 across all models to obtain deterministic (greedy) decoding for reproducibility. Maximum output length was left at the provider default: the response is a compact JSON object with three short fields, well below any provider cap. F.3 Open-weights models (Ollama v0.17.4) The open-weights models were served locally on the national cluster. Each SLURM job started a dedicated ollama serve process, ran the classifier via ollama.chat(), and logged the GPU power draw once per second via nvidia-smi for downstream energy/CO2 accounting (see Appendix L). All calls used a fixed seed=42 to make decoding reproducible. Parameter

Gemma 4 31B

GPT-OSS 20B

Gemma 3 27B

Llama 3.3 70B

Ollama model tag Context window (num_ctx) Max output (num_predict) Temperature Seed Structured output OLLAMA_NUM_PARALLEL OLLAMA_CONTEXT_LENGTH

gemma4:31b 9000 8192 0.0 42 JSON-Schema 4 8192

gpt-oss:20b 9000 8192 0.0 42 JSON-Schema 4 8192

gemma3:27b 9000 8192 0.0 42 JSON-Schema 4 8192

llama3.3:70b 9000 8192 0.0 42 JSON-Schema 4 8192

Table 11: Runtime configuration for the four open-weights models served via Ollama v0.17.4. All open-weights runs used JSON-Schema-constrained decoding (Ollama’s format= parameter), which enforces the enum of valid BAO labels at generation time.

24

G

Inter-model agreement (Cohen’s κ)

For each of the three classification tasks, the BAO assay format on the ChEMBL evaluation set, BAO assay format on the PubChem-BAO subset, and BAO detection technology, we computed the pairwise Cohen’s κ between all evaluated LLMs. Cohen’s κ measures agreement between two annotators beyond what would be expected by chance, with κ = 1 indicating perfect agreement and κ = 0 indicating chance-level agreement. Values above 0.8 are typically considered “high” agreement and values between 0.6 and 0.8 “substantial” agreement. It is defined as: κ = Pr(a)−Pr(e) 1−Pr(e) , where Pr(a) is the observed proportion of agreement between the two annotators and Pr(e) is the proportion of agreement expected under chance [25]. We utilized cohen_kappa_score from scikit-learn’s metrics to calculate the Cohen’s κ value.

Figure 4: Pairwise Cohen’s κ between the seven LLMs on the smaller PubChem-BAO subset (n = 226). Agreement is substantial (κ > 0.8 for most pairs) but lower than on the ChEMBL benchmark; disagreements concentrate on the organism-based and cell-free classes.

Figure 3: Pairwise Cohen’s κ between the seven evaluated LLMs on the ChEMBL assay format benchmark (n = 1097). All model pairs exceed κ > 0.9, indicating near-perfect agreement across proprietary and open-weight models.

Figure 5: Pairwise Cohen’s κ between the three proprietary LLMs (Claude Sonnet 4.6, Gemini 3.6 Flash, GPT-4o) and four open source LLMs (GPT-OSS 20B, Gemma 3 27B, Gemma 4 41B, Llama 70B) on the BAO detection technology task. Agreement patterns mirror the assay format results: near-perfect agreement.

25

H

Full manual-inspection table for ChEMBL assay format disagreements

For reproducibility, we provide the full row-level annotation of the 43 assays discussed in Section 5, including PubChem AIDs, both silver-standard labels, the top-predicted LLM label, whether a BARD annotation was available, and the vote count of the seven evaluated models. Table 12. Table 12: Manual inspection of the 43 assays where the majority of LLMs disagreed with the ChEMBL BAO assay format label. All labels are BAO assay formats; the “ format” suffix is omitted for space. ndis. counts models disagreeing with ChEMBL, nal. counts models agreeing with ChEMBL; seven models were attempted per row unless noted otherwise. Rows shaded in gray were sent for, and received, senior industrial curator review (Table 13). ndis. nal.

AID ChEMBL

BARD (if diff.)

Top pred.

Comment

1984 2121 2217 2613

cell-based cell-based biochemical whole-cell lysate (subclass of cell-free) cell-based

cell-based cell-based biochemical cell-free

LLMs support BARD label LLMs support BARD label LLMs support BARD label LLMs support BARD label

7 7 7 7

0 0 0 0

cell-based

LLMs support BARD label

7

0

single-protein (part of biochemical) cell-based

biochemical

LLMs support BARD label

7

0

cell-based

LLMs support BARD label

7

0

cell-based

cell-based

LLMs support BARD label

7

0

cell-based

cell-based

LLMs support BARD label

7

0

cell-free

LLMs support BARD label

7

0

cell-free cell-free

LLMs support BARD label LLMs support BARD label

6 6

1 1

cell-based

LLMs support BARD label

5

2

cell-based

LLMs support BARD label

5

2

biochemical cell-based

LLMs support BARD label LLMs support BARD label

5 5

2 2

organismbased organismbased biochemical

LLMs support BARD label

4

3

ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree

6

1

6

1

5

2

5

2

5

2

ChEMBL and BARD agree; LLMs disagree

5

2

biochemical biochemical cell-based tissuebased

588382 organismbased 588766 cell-free

588780 tissuebased 602210 tissuebased 602211 tissuebased 2592 tissuebased

whole-cell lysate (part of cell-free) 1470 biochemical cell-free 2588 tissuewhole-cell based lysate (part of cell-free) 488745 organismcell-based based 488752 organismcell-based based 504546 cell-based biochemical 504832 organismcell-based based 2467 cell-based organism-based 492958 cell-based

cell-based

588769 cell-free

cell-free

1913 biochemical protein (subclass cell-free of biochemical) 2693 cell-based cell-based organismbased 488788 biochemical single-protein cell-free (part of biochemical) 492957 cell-based cell-based organismbased

continued on next page

26

Table 12 – continued from previous page AID ChEMBL

BARD (if diff.)

Top pred.

Comment

2096 cell-based

cell-based

2695 cell-based

cell-based

492959 cell-based

cell-based

651893 cell-based

cell-based

organismbased organismbased organismbased organismbased cell-free

ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree ChEMBL and BARD agree; LLMs disagree LLMs disagree with both ChEMBL and BARD LLMs disagree with both ChEMBL and BARD – –

504494 biochemical cell-based 463208 biochemical cell-based

ndis. nal. 4

3

4

3

4

3

4

3

5

2

4

2

7 7

0 0

651720 cell-based 1159512 organismbased 449765 organismbased 651733 cell-based 652150 cell-based

– –

cell-free (6 att.) biochemical cell-based

–

cell-based

–

7

0

– –

– –

7 6

0 1

1295 504507 743295 1159566

– – – –

biochemical organismbased cell-free cell-free biochemical organismbased organismbased cell-based

– – – –

5 5 5 5

2 2 2 2

–

5

2

–

4

3

–

4

3

–

4

3

–

4

3

cell-based biochemical cell-based cell-based

1159571 cell-based

–

504631 organismbased 652151 cell-based

–

1159567 cell-based

–

1159570 cell-based

–

–

organismbased organismbased organismbased

27

Table 13: Assays sent for expert review, with ChEMBL label, top LLM prediction, and expert annotation. Top pred.

Short title

1470 biochemical

cell-based

Discovery of novel allosteric cell-free, but modulators of the M1 muscarinic could also be receptor: Agonist NMS binding at biochemical M1

492958 cell-based

organismbased

588769 cell-free

biochemical

Counterscreen for AddAB inhibitors: absorbance-based bacterial cell-based high throughput dose response assay for inhibitors of bacterial viability Late stage assay provider results from the probe development effort to identify inhibitors of plasma platelet activating factor acetylhydrolase (pPAFAH): fluorescence-based dose response biochemical gel-based competitive Activity-Based Protein Profiling (ABPP) assay for HTS compounds Luminescence-based dose response biochemical high throughput screening assay for inhibitors of the Heat Shock Protein 90 (HSP90) Fluorescence Cell-Based Dose Screen to Determine Inhibitors of S. cerevisiae Viability

28

AID ChEMBL

1913 biochemical

cell-free

2693 cell-based

organismbased

Expert format

organismbased

biochemical

Expert reason

Comment

In protocol: “Membranes were prepared from M1-expressing CHO cells”; format appears to be cell membranes but could be classified as biochemical as well see short title; you might even classify it as “organism based” since these are bacteria cells: later adapted when we discussed definition fluorescence-based dose response biochemical gel-based competitive Activity-Based Protein Profiling (ABPP) assay for HTS compounds

Expert finds multiple labels plausible

Expert agrees with LLM majority: ChEMBL/BARD label incorrect (expert initially leaned towards ChEMBL, but adapted after discussion) Expert agrees with LLM majority: ChEMBL/BARD label incorrect

biochemical

see short title (it is using a “reticulocyte lysate” so classifying it as “cell-free” makes some sense as well)

Expert agrees with ChEMBL: LLM majority plausible

organismbased

see short title (but could be Expert agrees with LLM majority: defined as organism based since ChEMBL/BARD label incorrect it is an organism, S. cerevisiae), (expert initially leaned towards but later adapted when we ChEMBL, but adapted after discussed definition discussion) continued on next page

Table 13 – continued from previous page AID ChEMBL

Top pred.

Short title

Expert format

Expert reason

Comment

488745 organismbased

cell-based

in protocol: “. . . Four microliters Expert agrees with LLM majority: of infected erythrocytes . . . ” ChEMBL label incorrect (LLMs agree with BARD)

488752 organismbased

cell-based

488788 biochemical

cell-free

Quantitative high throughput cell-based screen for delayed death inhibitors of the malarial parasite plastid, 96 hour incubation Quantitative high throughput cell-based screen for delayed death inhibitors of the malarial parasite plastid, 48 hour incubation Ras-converting Enzyme/Cell biochemical Proliferation Pathway Measured in Biochemical System Using Plate Reader

492957 cell-based

organismbased

Counterscreen for AddAB organisminhibitors: absorbance-based based bacterial cell-based high throughput dose response assay to identify inhibitors of RecBCD

see short title (again, could be defined as organism based . . . )

Expert agrees with LLM majority: ChEMBL label incorrect (LLMs agree with BARD)

a bit ambiguous in the title, but likely biochemical using proteases from either yeast or human

Expert agrees with ChEMBL: After seeing evidence from the LLMs: they pointed out: it does say that they do membrane preparations (I was mostly reading the sentence “This assay has been used for characterization of both yeast and human CaaX proteases and for characterization of inhibitors.”; I tried to look up the reference, but could only see the abstract), so also LLM might be correct, ambiguous case Expert agrees with LLM majority: ChEMBL/BARD label incorrect (expert initially leaned towards ChEMBL, but adapted after discussion)

29

as for assay 488745

continued on next page

Table 13 – continued from previous page Top pred.

Short title

Expert format

504494 biochemical

cell-free

Late stage assay provider results biochemical from the probe development effort to identify inhibitors of PAFAH2: Fluorescence-based biochemical gel-based Activity-Based Protein Profiling (ABPP) IC50

463208 biochemical

cell-free

cell-free Fluorescence Biochemical Dose Confirmation to Identify Inhibitors of Yeast HDACs

30

AID ChEMBL

Expert reason

Comment

see short title

Expert agrees with ChEMBL: After seeing evidence: “The protocol definitely says that it uses ‘soluble proteome’ of cells, which your last LLM explicitly defined as ‘cell lysate’, which then would fit in the ‘cell-free format’. If ‘biochemical format’ requires a purified single protein or other macromolecule, one can also argue that a proteome is not just a single protein, but a mixture and since it is derived from a cell (but not a cell anymore), cell-free is fine. ‘The assay uses a soluble proteome (cell lysate) of BW5147-derived murine T cells, which is explicitly a cell-free system.’ So the question really is whether the ‘proteome’ is purified enough to become ‘biochemical’ – or not . . . ”, so also LLM might be correct Yeast nuclear extracts (YNE) Expert agrees with LLM majority: were used; could be biochemical, ChEMBL/BARD label incorrect but does not seem to be an isolated biological macromolecule continued on next page

Table 13 – continued from previous page

31

AID ChEMBL

Top pred.

Short title

Expert format

Expert reason

Comment

2096 cell-based

organismbased

Luminescence Cell-Based Dose Confirmation HTS to Identify Inhibitors of Polyadenylation.

organismbased

Expert agrees with LLM majority: ChEMBL/BARD label incorrect

2467 cell-based

organismbased

Fluorescence Cell-Based Retest of organismC. albicans Growth in the based Presence of Fluconazole

2695 cell-based

organismbased

Fluorescence Cell-Based Dose Screen to Determine Inhibitors of C. albicans Viability

organismbased

Title/protocol (“cells are lysed. . . ”) suggest cell-based, but it is yeast, so probably organism-based “Whole cell phenotypic screening” would indicate cell-based, however C. albicans is an organism already in short title, but later adapted when we discussed definition

492959 cell-based

organismbased

organismbased

504631 organismbased

cell-based

651893 cell-based

organismbased

Absorbance-based bacterial cell-based high throughput dose response assay for inhibitors of AddAB recombination protein complex A cell-based HTS for delayed death inhibitors of the malarial parasite plastid Measured in Microorganism System Using Plate Reader Inhibition of Trypanosoma brucei rhodesiense STIB900 Measured in Cell-Based System Using Plate Reader

see short title, adapted when discussing definition

Expert agrees with LLM majority: ChEMBL/BARD label incorrect (expert initially leaned towards ChEMBL, but adapted after discussion)

organismbased

organismbased

Expert agrees with LLM majority: ChEMBL label incorrect (LLMs agree with BARD)

Expert agrees with LLM majority: ChEMBL/BARD label incorrect (expert initially leaned towards ChEMBL, but adapted after discussion) see short title (again, could be Expert agrees with LLM majority: defined as organism based . . . ), ChEMBL/BARD label incorrect but adapted when discussing (expert initially leaned towards ChEMBL, but adapted after discussion) Title would indicate cell-based, Expert agrees with ChEMBL: however Plasmodium falciparum LLM majority incorrect (see is an organism Appendix H.1)

H.1 Failure case: AID 504631 A notable disagreement concerned assay AID 504631 using the single-celled pathogens Plasmodium falciparum. Under the BAO ontology, cell-based format is explicitly restricted to eukaryotic cells derived from a multicellular organism (cell lines, primary cells, stem cells) and is explicitly distinguished from organism-based format, which covers any living organism, single- or multi-celled. Bacteria and yeast are given as the canonical single-celled examples. A single-celled eukaryotic parasite such as P. falciparum therefore falls under organism-based, not cell-based, even though it is eukaryotic and is cultured in wells. Table 14 shows model predictions for AID 504631, “A cell-based HTS for delayed death inhibitors of the malarial parasite”. Four of seven models (Gemma 3 27B, GPT-OSS 20B, GPT-4o, Llama 3.3 70B) predicted cell-based format with high confidence, while three (Claude, Gemini, Gemma 4 31B) correctly predicted organism-based format. Notably, the erroneous predictions are not due to missing information: the cited evidence in several cases (e.g., GPT-OSS 20B, Llama 3.3 70B) directly quotes text referring to “the malarial parasite” or a “microorganism system,” yet the models still defaulted to cell-based. This suggests the failure is not one of information retrieval but of ontological reasoning: models appear to pattern-match on surface cues such as “cells cultured in wells” or the word “parasite” co-occurring with “cell,” rather than applying the BAO-specific distinction. The assay’s own title (“cell-based HTS”) likely reinforces this bias, as several models’ cited evidence draws directly on title wording rather than the underlying biology. Table 14: Model predictions for AID 504631, illustrating the cell-based/organism-based confusion for single-celled eukaryotic organisms. True label: organism-based format. Model

Prediction

Correct?

Cited evidence

Claude

organism-based

✓

Gemini

organism-based

✓

Gemma 4 31B

organism-based

✓

Gemma 3 27B

cell-based

✗

GPT-OSS 20B

cell-based

✗

GPT-4o

cell-based

✗

Llama 3.3 70B

cell-based

✗

“...parasites cultured in the presence of test compounds...measuring susceptibility of the living malarial parasite organism.” “The susceptibility of the Dd2 Plasmodium falciparum line against novel small molecules will be determined using a SYBR green-based fluorescence assay.” “The susceptibility of the Dd2 Plasmodium falciparum line...Parasites are cultured in the presence of serial dilutions of test compounds.” “A cell-based HTS for delayed death inhibitors of the malarial parasite plastid...Parasites are cultured in the presence of serial dilutions of test compounds...” “Parasites are cultured in the presence of serial dilutions of test compounds...using a SYBR green-based fluorescence assay in 384-well plates” “...explicitly indicates the use of living cells (parasites) in the experimental setup.” “A cell-based HTS for delayed death inhibitors of the malarial parasite plastid Measured in Microorganism System Using Plate Reader”

32

I

Detection Disagreement Details

This appendix reports the full record-level breakdown of the 34 physical detection method assays where the majority of LLMs disagreed with the PubChem bulk-download label (Table 15), followed by the per-model evidence spans for which the expert did not agree with the majority of LLMs. (Table 17). Table 15: Manual inspection of the 34 physical detection method assays where the majority of models disagreed with the third-party BAO detection label in the PubChem bulk download. All labels are BAO physical detection methods; the “ method” suffix is omitted for space. nmaj. is the number of models supporting the majority prediction; nal. is the number of predictions that aligned with the third-party BAO label instead. Seven models were attempted per row unless noted otherwise. Rows shaded in gray were additionally reviewed by a senior industrial curator (see Table 16 for per-model evidence spans). AID BAO label (third-party)

Majority pred.

588725 spectrophotometry radiometry

Bioassay annotation (PubChem)

Comment

7/0

588782 fluorescence

scintillation counting (part of radiometry) spectrophotometry absorbance (part of spectrophotometry) luminescence luminescence

602192 fluorescence

luminescence

7/0

602194 fluorescence

luminescence

588768 fluorescence

LLMs agree with BARD LLMs agree with BARD LLMs agree with BARD chemiluminescence (part LLMs agree with of luminescence) BARD chemiluminescence (part LLMs agree with of luminescence) BARD label-free method (BARD) LLMs disagree with all sources (no other) –

nmaj. /nal.

602231 spectrophotometry isometric tension recording 602245 spectrophotometry isometric tension recording 602337 fluorescence spectrophotometry absorbance (part of spectrophotometry) 602339 fluorescence mass mass spectrometry spectrometry 602341 fluorescence mass mass spectrometry spectrometry 602351 fluorescence mass mass spectrometry spectrometry 602353 fluorescence mass mass spectrometry spectrometry fluorescence 624079 spectrophotometry fluorescence 1259354 fluorescence

luminescence

(no other)

623968 spectrophotometry fluorescence method

quantitative reverse transcription PCR (this is not a physical detection method but an assay supporting method) 588778 fluorescence spectrophotometry (no other) 602418 luminescence fluorescence (no other) 588816 fluorescence simple (no other) measurement method 602170 spectrophotometry visual observation (no other) method 624103 fluorescence simple (no other) measurement method

7/0 7/0

7/0 7/0 7/0

LLMs agree with BARD LLMs agree with BARD LLMs agree with BARD LLMs agree with BARD LLMs agree with BARD LLMs agree with BARD LLM prediction seems reasonable –

7/0

3/3 (6 att.)

– – –

5/2 3/0 3/1 (6 att.)

–

3/1

–

3/2 (6 att.)

7/0 7/0 7/0 7/0 7/0 6/0

continued on next page

33

Table 15 – continued from previous page AID BAO label (third-party)

Majority pred.

Bioassay annotation (PubChem)

624135 fluorescence

simple (no other) measurement method 624334 fluorescence visual observation (no other) method 623916 spectrophotometry simple cell number determination measurement (not a BAO format, but assay design method) 623917 radiometry simple cell number determination measurement (not a BAO format, but assay design method) 623918 spectrophotometry simple cell number determination measurement (not a BAO format, but method assay design method) 623919 spectrophotometry simple cell number determination measurement (not a BAO format, but assay design method) 623930 radiometry luminescence luminescence 623954 spectrophotometry simple measurement 623956 spectrophotometry simple measurement 624085 spectrophotometry fluorescence

Comment

nmaj. /nal.

–

3/2 (6 att.)

–

2/0 (6 att.)

–

5/0

–

5/0

–

4/0

–

5/0

LLMs agree with BARD cell number determination –

6/0 (6 att.) 4/0

cell number determination –

4/0

fluorescence

6/1

LLMs agree with BARD fluorescence LLMs agree with 624090 spectrophotometry fluorescence BARD 624092 spectrophotometry fluorescence fluorescence LLMs agree with BARD 624286 luminescence visual observation plaque counting (not BAO – label) 651578 luminescence visual observation plaque counting (not BAO – label)

6/1 6/1 6/0 (6 att.) 6/0 (6 att.)

Table 16: Detection-method assays sent for expert review, with third-party (PubChem) label, LLM majority prediction, and expert annotation. AID 3rd-party

Majority pred.

Short title

Expert format

Expert reason

623968 spectrophotometry fluorescence

588778 fluorescence

602418 luminescence

Inhibitors of the Hepatitis spectrophotometry Similar work to the CHRM C Virus non-structural probe-development assays protein 3 helicase (NS3) but “late stage,” so more information is given, e.g. “RNA concentration is determined by reading absorbance at 260 nm” spectrophotometry A High Throughput fluorescence Protocol: “. . . fluorescence Screening Assay for (spectroscopy intensity was determined by Inhibitors of Bacterial also fine after reading. . . ” Motility in Vibrio seeing LLM cholerae evidence) fluorescence Summary of probe cannot define A summary of various development efforts to efforts per the title; no identify inverse agonists information on what was of LRH-1 (NR5A2) actually done for each effort continued on next page

34

Table 16 – continued from previous page AID 3rd-party

Majority pred.

Short title

Expert format

588816 fluorescence

simple measurement

Identify agonists of the human cholinergic receptor, muscarinic 1 (CHRM1) Probe development efforts to identify activators of Methionine sulfoxide reductase A (MsrA) Identify agonists of the human cholinergic receptor, muscarinic 5 (CHRM5) Identify agonists, PAMs, and antagonists of the human cholinergic receptor, muscarinic 4 (CHRM4) Small molecule inhibitors of the oncogenic and cytokinetic protein MgcRacGAP Cell-based colony formation assay to identify inhibitors of the recombinationpromoting activity of RecBCD in V66 E. coli Virus Titer Reduction Secondary Screen for Compounds that Inhibit VEEV (TC-83 strain)

cannot be defined No information; this is pre-work to identify a probe, not really an assay in itself cannot be defined As above (probe-development pre-work)

602170 spectrophotometry visual observation

624103 fluorescence

simple measurement

624135 fluorescence

simple measurement

624334 fluorescence

visual observation

623918 spectrophotometry simple measurement

624286 luminescence

visual observation

651578 luminescence

visual observation

As above (VEEV TC-83 titer reduction)

Expert reason

cannot be defined As above

cannot be defined As above

cannot be defined No information

simple measurement

Not exactly the same assay as 623916/623917 but same principle

not easy – simple measurement, visual observation, or imaging

Crystal violet solution added; number of plaques per well determined by visual counting; expert reluctant to use “visual observation,” reserving that category for larger-organism behavioral readouts As above

as above (624286)

Table 17: Detection technology predictions for selected AIDs across all models. AID

3rd-Party Label

588816 fluorescence method

Expert Label

cannot be defined

Model

Prediction

Conf.

Evidence Span

Llama 3.3 70B

simple measurement ✗

low

GPT-4o

visual observation ✗

low

No specific detection method mentioned No specific detection method or instrumentation mentioned.

Claude Sonnet ERROR ✗ 4.6 Gemini 3.6 fluorescence ✗ Flash

low

Gemma 3 27B

simple measurement ✗

low

Gemma 4 31B

label free ✗

low

Title indicates probe development for human muscarinic receptor 1 (CHRM1), typically fluorescence-based. . . No information provided. Conservative assumption. . . probe development efforts to identify agonists of CHRM1 Continued on next page

35

Table 17 – continued AID

3rd-Party Label

spectro602170 photometry method

Expert Label

cannot be defined

Model

Prediction

Conf.

Evidence Span

GPT-OSS 20B

simple measurement ✗

low

Llama 3.3 70B

fluorescence ✗

low

GPT-4o

visual observation ✗

low

Claude Sonnet spectrophotometry ✗ 4.6

low

Gemini 3.6 fluorescence ✗ Flash Gemma 3 27B simple measurement ✗

low

No detection method described No explicit evidence, but MsrA assays often associated with fluorescence-based detection. . . No explicit detection method or instrumentation mentioned. MsrA activity assays commonly use spectrophotometric or fluorescence-based detection. . . Inferred from typical probe development assays for MsrA No information provided; defaulting to most basic method. . . Title mentions ’probe development efforts’, often visual/basic observation. . . No protocol provided

Gemma 4 31B

spectro623918 photometry method

simple measurement

visual observation ✗

low

GPT-OSS 20B

visual observation ✗

low

Llama 3.3 70B

simple measurement ✓

high

GPT-4o

visual observation ✗

high

Claude Sonnet simple measurement ✓ 4.6 Gemini Flash

624103 fluorescence method

624135 fluorescence method

cannot be defined

low

3.6 visual observation ✗

medium

high

Gemma 3 27B

simple measurement ✓

high

Gemma 4 31B

visual observation ✗

high

GPT-OSS 20B

simple measurement ✓

high

Llama 3.3 70B

simple measurement ✗

low

GPT-4o

visual observation ✗

low

colony forming units per ml of culture Differential plating to measure colony formation relies on visual observation. . . Frequency of recombinants determined by differential plating and counting. . . Frequency of recombinants determined by differential plating and calculating. . . Counting colonies (colony forming units) to determine viability. . . Frequency of recombinants determined by differential plating Plated to measure viability and determine recombinant frequency No specific detection method mentioned No explicit detection method, techniques, or instrumentation mentioned. . . Muscarinic receptor agonist assays commonly use fluorescence-based detection. . . No explicit physical detection method described. No detection method stated; lack of detail. . .

Claude Sonnet fluorescence ✗ 4.6

low

Gemini 3.6 fluorescence ✗ Flash Gemma 3 27B simple measurement ✗

low low

Gemma 4 31B GPT-OSS 20B

ERROR ✗ simple measurement ✗

low

No protocol or description provided

Llama 3.3 70B

simple measurement ✗

low

No specific detection method mentioned Continued on next page

cannot be defined

36

Table 17 – continued AID

3rd-Party Label

Expert Label

Model

Prediction

Conf.

Evidence Span

GPT-4o

visual observation ✗

low

No explicit detection method, techniques, or instrumentation mentioned. . . Muscarinic receptor assays commonly use fluorescencebased detection (e.g., HTRF). . . CHRM4 probe development typically utilizes fluorometric functional assays. . . Only indicates probe development efforts, no detection method. . .

Claude Sonnet fluorescence ✗ 4.6

low

3.6 fluorescence ✗

low

Gemini Flash

624334 fluorescence method

cannot be defined

Gemma 3 27B

simple measurement ✗

low

Gemma 4 31B GPT-OSS 20B

ERROR ✗ simple measurement ✗

low

No detection method described

Llama 3.3 70B

simple measurement ✗

low

GPT-4o

visual observation ✗

low

Claude Sonnet spectrophotometry ✗ 4.6

low

No specific detection method mentioned No explicit detection method or instrumentation mentioned. . . Classification based on common methods used in small molecule inhibitor screens. . . No explicit method mentioned; luminescence inferred as common. . . No information provided; defaulting to most basic. . .

Gemini Flash

spectro623968 photometry method

588778 fluorescence method

spectrophotometry method

fluorescence & spectrophotometry

3.6 luminescence ✗

low

Gemma 3 27B

simple measurement ✗

low

Gemma 4 31B GPT-OSS 20B

ERROR ✗ visual observation ✗

low

No detection method described

Llama 3.3 70B

spectrophotometry ✓

high

GPT-4o

spectrophotometry ✓

high

RNA concentration determined by reading absorbance at 260 nm RNA concentration determined by reading absorbance at 260 nm.

Claude Sonnet ERROR ✗ 4.6 Gemini 3.6 fluorescence ✗ Flash Gemma 3 27B spectrophotometry ✓

high high

Gemma 4 31B

fluorescence ✗

high

GPT-OSS 20B

fluorescence ✗

high

Llama 3.3 70B

spectrophotometry ✓

high

GPT-4o

fluorescence ✓

high

Claude Sonnet spectrophotometry ✓ 4.6

high

Real time RT-PCR performed with TaqMan chemistry RNA concentration determined by reading absorbance at 260 nm. RT-PCR using TaqMan chemistry used to quantify HCV RNA levels Real time RT-PCR performed with TaqMan chemistry Absorbance at 615nm on a Perkin Elmer Envision Fluorescence intensity determined using excitation/emission to differentiate motility. . . Primary read was absorbance at 615nm to monitor bacterial growth and motility. . . Continued on next page

37

Table 17 – continued AID

3rd-Party Label

Expert Label

Model

Conf.

Evidence Span

3.6 spectrophotometry ✓

high

Gemma 3 27B

fluorescence ✓

high

Gemma 4 31B

spectrophotometry ✓

high

GPT-OSS 20B

spectrophotometry ✓

high

Primary read was absorbance at 615 to monitor bacterial growth and motility. . . Fluorescence intensity determined by reading with excitation wavelength of 535. . . Read using absorbance at 615nm on a Perkin Elmer Envision. . . "read using absorbance at 615nm on a Perkin Elmer Envision"

Llama 3.3 70B

fluorescence ✗

low

GPT-4o

visual observation ✗

low

Gemini Flash

602418 luminescence method

cannot define

Claude Sonnet fluorescence ✗ 4.6

low

3.6 fluorescence ✗

low

Gemini Flash

624286 luminescence method

simple meas., visual obs., or imaging

Prediction

Gemma 3 27B

simple measurement ✗

low

Gemma 4 31B

label free ✗

low

GPT-OSS 20B

simple measurement ✗

low

Llama 3.3 70B

visual observation ✓

high

GPT-4o

visual observation ✓

high

Claude Sonnet ERROR ✗ 4.6 Gemini 3.6 visual observation ✓ Flash

651578 luminescence method

not easy; possibly visual

high

Gemma 3 27B

visual observation ✓

high

Gemma 4 31B

visual observation ✓

high

GPT-OSS 20B

visual observation ✓

high

Llama 3.3 70B

visual observation ✓

high

GPT-4o

visual observation ✓

high

No explicit evidence, but fluorescence commonly used in HTS for inverse agonists No explicit detection method or instrumentation mentioned. LRH-1 probe development assays commonly use fluorescence-based methods (e.g., TR-FRET). . . Probe development for LRH-1 typically utilizes TR-FRET/fluorescence assays. . . No information provided; assuming basic assay. . . Probe development to identify inverse agonists often uses label-free methods like SPR. . . No detection method described Number of plaques in each well determined by visual counting. Protocol states number of plaques determined by visual counting. . .

Number of plaques in each well determined by visual counting Relies on ’visual counting’ of plaques after crystal violet staining. . . Number of plaques in each well determined by visual counting. "number of plaques in each well determined by visual counting" Number of plaques in each well determined by visual counting. Protocol states number of plaques determined by visual counting. . .

Claude Sonnet ERROR ✗ 4.6 Continued on next page

38

Table 17 – continued AID

3rd-Party Label

Expert Label

Model

Conf.

Evidence Span

3.6 visual observation ✓

high

Gemma 3 27B

visual observation ✓

high

Gemma 4 31B

visual observation ✓

high

GPT-OSS 20B

visual observation ✓

high

Number of plaques in each well determined by visual counting Relies on ’visual counting’ of plaques after crystal violet staining. . . Plates incubated and stained again; washed after. . . "number of plaques in each well determined by visual counting"

Gemini Flash

39

Prediction

J

Confidence calibration on both tasks

(a) Error rate by predicted confidence level for silver-label classification in the ChEMBL baseline dataset.

(b) Error rate by predicted confidence level for silver-label detection results.

Figure 6: Error rates across predicted confidence levels for silver-label predictions. The top panel shows results for the ChEMBL baseline, while the bottom panel shows the corresponding results for the detection task. Each bar represents the proportion of incorrect predictions within a confidence category, with the number of samples shown in parentheses.

40

K

Prompt ablation details

Figure 7: Effect of two prompt perturbations on proprietary-LLM predictions for the ChEMBL assay format subset. Removing the BAO definitions (Without defs) changes far more predictions than reversing the class order (Order swap), and disproportionately so for Gemini 3.6 Flash (10.8%). Bar segments indicate whether each changed prediction moved from a different label to the silver label (green), from the silver label to another label (red), or between two labels that are different from the silver label (orange); percentages above each bar give the total fraction of predictions that changed.

41

L

Resource Usage: Cost and Energy

This appendix reports the compute cost, wall-clock throughput, and (proxy) energy consumption of the LLM auditing workflow. The open-weights models were served locally via Ollama v0.17.4 on a single node of the Berzelius cluster. Wall-clock time was measured per SLURM job. Note that this could be improved for open-source models. The energy per record is a proxy computed as GPU-hours × PGPU . We treat this as a proxy: it excludes non-GPU consumption, such as CPU and cooling overhead. For rerunning with the same settings we would like to declare that GPT-OSS 20B and Gemma 3 27B and Gemma 4 were hosted on a single NVIDIA A100-SXM4-40GB, LLaMA 3.3 70B on a single NVIDIA H200, each with 32 CPU cores allocated. Table 18 reports per-model resource usage across all five experimental conditions. Proprietary models (Claude Sonnet 4.6, Gemini 3.6 Flash, GPT-4o) were accessed via API and are billed per token; we report USD cost per completion. Open-weight models (Gemma 3 27B 27B, Gemma 4 31B, GPT-OSS 20B, Llama 3.3 70B) were self-hosted, incurring no API cost but measurable energy draw; we report energy consumption per completion in place of a dollar cost. No energy figures are available for the proprietary models, as token-level energy accounting is not exposed by the respective providers. Table 18: Per-model token usage, parse failures, and cost/energy accounting across all experimental conditions. Input and completion token counts are experiment-wide totals. Cost is reported for API-accessed (proprietary) models; energy for self-hosted (open-weight) models. “–” indicates not applicable. Experiment

Model

N Parse fails

Input Output Cost Energy tokens tokens ($) (kWh)

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o ChEMBL baseline Gemma 3 27B Gemma 4 31B GPT-OSS 20B Llama 3.3 70B

1097 1097 1097 1097 1097 1097 1097

29 0 0 0 0 0 0

2,302,201 2,040,815 1,998,562 2,081,404 2,086,917 2,236,521 2,008,532

90,590 54,349 67,554 64,558 63,083 51,689 52,330

9.09 3.47 5.67 – – – –

– – – 6.44 6.45 6.85 6.18

Order swap

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o Gemma 3 27B Gemma 4 31B GPT-OSS 20B Llama 3.3 70B

500 500 500 500 500 500 500

12 1,053,952 0 939,568 0 907,504 0 946,568 0 948,589 0 1,013,895 0 912,284

41,224 24,503 30,607 29,901 30,711 24,958 24,164

4.16 1.59 2.57 – – – –

– – – 2.93 2.94 3.12 2.81

PubChem+defs

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o Gemma 3 27B Gemma 4 31B GPT-OSS 20B Llama 3.3 70B

226 226 226 226 226 226 226

21 0 0 0 0 0 0

417,023 363,743 362,937 372,331 373,235 410,707 365,753

19,332 11,523 13,537 12,763 13,367 10,608 10,983

1.70 0.63 1.04 – – – –

– – – 1.16 1.16 1.26 1.13

Without defs

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o Gemma 3 27B Gemma 4 31B GPT-OSS 20B Llama 3.3 70B

500 500 500 500 500 500 500

13 0 0 0 0 1 0

840,452 757,068 722,504 764,068 766,089 834,022 726,784

39,411 22,889 29,221 29,382 27,634 31,697 21,688

3.42 1.31 2.10 – – – –

– – – 2.38 2.38 2.60 2.25

Claude Sonnet 4.6 Gemini 3.6 Flash GPT-4o Gemma 3 27B

308 308 308 308

34 0 0 0

786,449 680,024 682,481 684,336

21,815 15,790 17,906 18,903

2.69 1.03 1.89 –

– – – 2.11

Detection

continued on next page

42

Table 18 – continued from previous page Experiment

Model

N Parse fails

Gemma 4 31B GPT-OSS 20B Llama 3.3 70B

308 308 308

43

3 0 0

Input Output Cost Energy tokens tokens ($) (kWh) 685,861 737,417 689,615

41,668 13,925 15,818

– – –

2.18 2.25 2.12

Related documents

• Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?#1156887
• Semantic knowledge improves molecular machine learning for chemical toxicity prediction.#422908
• AI-readiness Criteria for Biomedical Data#240474
• MetaPerch: Learning from metadata for bioacoustics foundation models#370314
• Development of Metadata Schemas For Collaborative Research Centers#206006