BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
arXiv:2604.16241v1 [cs.CL] 17 Apr 2026
Jiacheng Shen2† , Masato Hagiwara1† , Milad Alizadeh1 , Ellen Gilsenan-McMahon1 , Marius Miron1 , David Robinson1 , Emmanuel Chemla1 , Sara Keen1 , Gagan Narula1 , Mathieu Laurière3‡ , Matthieu Geist1‡ , Olivier Pietquin1‡ 1
Earth Species Project
2
NYU Center for Data Science; NYU Shanghai
3
NYU Shanghai Center for Data Science; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai
Abstract Large language models (LLMs) have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language models handle specialized animal-related knowledge under a unified closed-book evaluation protocol. We introduce BAGEL, a Benchmark for evaluating Animal knowledGe Expertise in Language models. BAGEL is constructed from diverse scientific and reference sources, including bioRxiv, Global Biotic Interactions, Xeno-canto, and Wikipedia, using a combination of curated examples and automatically generated closed-book question-answer pairs. The benchmark covers multiple aspects of animal knowledge, including taxonomy, morphology, habitat, behavior, vocalization, geographic distribution, and species interactions. By focusing on closed-book evaluation, BAGEL measures animal-related knowledge of models without external retrieval at inference time. BAGEL further supports fine-grained analysis across source domains, taxonomic groups, and knowledge categories, enabling a more precise characterization of model strengths and systematic failure modes. Our benchmark provides a new testbed for studying domain-specific knowledge generalization in language models and for improving their reliability in biodiversity-related applications.
1
Introduction
Instruction-tuned LLMs and chat-style assistants have rapidly improved on a wide range of knowledge and reasoning tasks (Ouyang et al., 2022; OpenAI, 2023), leading to strong performance on broad-domain evaluations such as MMLU (Hendrycks et al., 2021) and scienceoriented benchmarks such as ScienceQA (Lu et al., 2022). These gains have fueled interest in using LLMs as general-purpose interfaces for scientific information access, synthesis, and question answering. However, strong aggregate performance on broad benchmarks does not by itself establish whether models reliably encode specialized long-tail knowledge about the natural world, especially when answering questions that require species-level facts, ecological relations, or natural-history reasoning. This gap is increasingly important because language and foundation models are already being explored for biodiversity and animal-related applications. In ecology, recent studies have used LLMs to extract structured ecological information from scientific literature, including host–pathogen records and large-scale species interactions (Gougherty and Clipp, 2024; Keck, Broadbent, and Altermatt, 2025), and have begun † These authors contributed equally to this work
†† Co-supervising authors
1
BAGEL BENCHMARK
Benchmark / line
Primary setting
Relation to animals / biodiversity
BAGEL
Text-only, closed- Directly centered on book 4-option MC animal and naturalfrom four curated history knowledge source tracks
MMLU (Hendrycks et al., 2021)
Broad text MC across 57 subjects
Key contrast with BAGEL Reference row: combines encyclopedic animal knowledge (Wikipedia), interaction reasoning among animal species (GloBI), animal-focused scientific-literature reasoning (bioRxiv), and bioacoustic text about animals (Xeno-canto) under one unified protocol.
Only incidental biol- BAGEL is animal-centered, sourceogy / life-science cov- heterogeneous, and reports source/doerage main breakdowns rather than a broad academic aggregate. ScienceQA (Lu et Multimodal school- Includes natural- BAGEL targets higher-level natural-history al., 2022) science MC science curriculum content and uses text-only closed-book content questions built from biodiversity sources. BEANS; BEANS- Bioacoustic au- Directly about animal BAGEL removes audio and tests whether Zero (Hagiwara, dio tasks; audio– sounds models can answer textual questions about Hoffman, et al., language modeling vocalization properties and species knowl2022; Robinson edge. et al., 2024) Ecological knowl- Textual evaluation of Shared theme: BAGEL is centered on animal natural hisedge eval (Dorm ecological LLM com- biodiversity-related tory and organism-level facts, using one et al., 2025) petence (mixed task scientific knowledge closed-book MC format across four curated types) in language models biodiversity text sources; Dorm et al. study ecological competence more broadly with a heterogeneous textual task suite. EnvironmentalBroad environment BAGEL is narrower but deeper on animal EnviroExam; ELLE (Y. Huang domain exams / QA and eco-environment and natural-history knowledge, with exet al., 2024; Guo, coverage plicit species, interaction, literature, and bioacoustic-source structure. N. Li, and M. Xu, 2025) PubMedQA; Biomedical QA / Biological, but BAGEL shifts the focus from biomedical BLURB (Jin et biomedical NLP centered on hu- language understanding to organismal dial., 2019; Gu et al., man health and versity, ecology, and natural-history rea2021) biomedicine soning.
Table 1. Representative neighboring benchmarks and where BAGEL differs most clearly. Rows are illustrative rather than exhaustive; descriptions were checked against primary papers / official benchmark descriptions.
to test ecological knowledge more directly, finding uneven performance across ecological tasks (Dorm et al., 2025). In animal communication and bioacoustics, prior work has introduced several important resources, including BEANS, a benchmark covering a broad range of animal sound tasks (Hagiwara, Hoffman, et al., 2022); ISPA, a text-like transcription scheme for animal sounds (Hagiwara, Miron, and Liu, 2024); and NatureLM-audio, an audio-language foundation model built upon text-only LLMs(Llama) for bioacoustics (Robinson et al., 2024). More broadly, biodiversity-focused foundation models have also emerged in other modalities, such as BioCLIP for fine-grained recognition across the tree of life (Stevens et al., 2024). Despite this momentum, most prior work emphasizes information extraction, audio understanding, or visual recognition rather than evaluating whether text-only LLMs can answer closed-book questions about animals. As a result, it remains unclear how well current language models generalize across the heterogeneous forms of knowledge that matter for animal expertise, such as taxonomy, morphology, behavior, habitat, vocalization, geographic distribution, and species interactions. To address this gap, we introduce BAGEL, a Benchmark for closed-book evaluation of Animal knowledGe Expertise in Language models.1 BAGEL aggregates 11,852 multiple-choice questions derived from four complementary sources: Wikipedia, Global Biotic Interactions (GloBI) (Poelen, Simons, and Mungall, 2014), bioRxiv, and Xeno-canto (Vellinga and Planqué, 2015). Together they target four animal-centered skills—encyclopedic knowledge about animals (Wikipedia), ecological interaction reasoning (GloBI), scientific-literature reasoning about animals (bioRxiv), and text-only bioacoustic-domain knowledge about animal vocalizations (Xeno-canto). By design, BAGEL tests not only overall accuracy but also robustness across source domains and source-specific dimensions, providing a more fine-grained view of what current models do and do not know about animals and natural history. 1 Dataset release: https://huggingface.co/datasets/EarthSpeciesProject/BAGEL.
2
BAGEL BENCHMARK
2 Related Work LLMs and foundation models for biodiversity and animal-related applications. The use of language and foundation models in biodiversity-relevant settings is growing rapidly, but the literature is still fragmented across application areas and modalities. In ecology, LLMs have been explored as tools for extracting structured knowledge from text, including ecological variables from disease reports (Gougherty and Clipp, 2024) and species interactions from large scientific corpora (Keck, Broadbent, and Altermatt, 2025). Recent evaluation work has also begun to probe whether general-purpose LLMs possess ecological knowledge directly, reporting a substantial gap between relatively strong factual or taxonomic recall and weaker ecological reasoning or conservation-oriented judgment (Dorm et al., 2025). In parallel, animal-centered foundation-model work has expanded in bioacoustics: BEANS established a public benchmark covering multiple animal-sound tasks (Hagiwara, Hoffman, et al., 2022); ISPA proposed a text-based representation for transcribing animal sounds and connecting them to languagemodel-style methods (Hagiwara, Miron, and Liu, 2024); and NatureLM-audio introduced an audio-language foundation model tailored to bioacoustics, with strong zero-shot generalization across taxa and tasks (Robinson et al., 2024). Outside text and audio, BioCLIP demonstrates that biodiversity-specific foundation models can substantially improve fine-grained recognition across a wide taxonomic range (Stevens et al., 2024). Adjacent Earth-science domains have likewise moved toward specialized language models, including K2 for geoscience and OceanGPT for ocean science (Deng et al., 2023; Bi et al., 2023). Together, these studies show clear momentum toward AI systems specialized for nature and environmental data, but they do not directly evaluate closed-book animal knowledge in text-only LLMs. General knowledge and science benchmarks. Large language models are commonly evaluated using broad knowledge benchmarks such as MMLU (Hendrycks et al., 2021), which measure multitask performance across many academic subjects. In science-focused evaluation, ScienceQA (Lu et al., 2022) provides a large benchmark of science questions with associated explanations and multimodal context. These benchmarks are valuable for measuring general scientific competence, but they do not specifically target fine-grained knowledge of animals, biodiversity, or natural history. Biomedical and scientific QA benchmarks. A separate line of work studies domain-specific evaluation in biomedicine and scientific question answering. BLURB (Gu et al., 2021) aggregates multiple biomedical NLP tasks into a unified benchmark, while PubMedQA (Jin et al., 2019) focuses on question answering over biomedical research abstracts. BioASQ (Nentidis et al., 2023) has also established a long-running shared task centered on large-scale biomedical semantic indexing and question answering. These resources demonstrate the value of domainspecific evaluation, but they primarily focus on biomedical or clinical knowledge rather than biodiversity and natural history. Environmental and ecological benchmarks. More recently, several benchmarks have moved closer to environmental and ecological applications. EnviroExam (Y. Huang et al., 2024) evaluates environmental science knowledge of large language models using curriculumbased questions, and ELLE (Guo, N. Li, and M. Xu, 2025) proposes a QA benchmark for ecoenvironment applications. Work on ecological knowledge evaluation is also beginning to emerge, with recent evidence that strong general-purpose LLMs retain only partial and taskdependent ecological competence (Dorm et al., 2025). These efforts are important adjacent steps, but they emphasize broad environmental science, ecology, or sustainability topics rather than animal-centered expertise. In contrast, BAGEL foregrounds animal-centered evaluation under one protocol: encyclopedic species facts (Wikipedia), ecological interactions among taxa (GloBI), animal-relevant scientific literature reasoning (bioRxiv), and bioacoustic-domain textual knowledge (Xeno-canto), with accuracy reported per source. Our Contributions. BAGEL complements prior work by focusing on a distinct but underexplored evaluation axis: closed-book question answering about animals and natural history
3
BAGEL BENCHMARK
Figure 1. Overview of the BAGEL benchmark curation pipeline: four source-specific preparation tracks feed domain prompts into a shared generator, followed by quality checks, four-option formatting, and option-order shuffling.
grounded in heterogeneous biodiversity-relevant sources. Rather than testing general academic knowledge or biomedical reasoning alone, BAGEL is designed to measure how well language models handle species-level knowledge, ecological relations, and animal-focused factual generalization across multiple source domains. Table 1 makes the contrasts concrete with representative neighbors.
3
Benchmark Construction
Figure 1 summarizes the end-to-end curation workflow; subsections below describe each source domain. The four corpora are not intended to exhaust real-world natural-history competence; they are public, machine-accessible anchors over complementary animal-centered skills: encyclopedic knowledge about animals, ecological interaction reasoning among taxa, scientific-literature reasoning on animal-focused preprints, and text-only bioacoustic knowledge derived from animal recordings. 3.1 Wikipedia The Wikipedia subset targets encyclopedic knowledge about animals: English Wikipedia species articles supply the evidence, and each item probes closed-book recall of taxon-specific facts a reader would normally take from such an article, without access to the article at test time.2 Article retrieval. For each candidate taxon, structured metadata (scientific name and higher-level classification) is used to resolve the English article title through Wikidata (linking the taxon name to an item and reading its English Wikipedia sitelink), after which the article plain-text extract is retrieved via the MediaWiki API. Pages with missing text or extracts shorter than 1,000 characters are excluded so that stubs and minimally informative articles do not dominate the benchmark. Text preparation. Extracts longer than 180,000 characters are truncated before generation, with preference for paragraph- or sentence-boundary cuts so that the retained prefix remains coherent. This cap keeps prompts within practical context-window limits of the generation model. No additional manual editing of article text is performed beyond this cap. Question synthesis. Questions are generated through the GPT-4o-mini API using the system–user template in Appendix A.1.1. The model may emit up to eight four-option, singleanswer items per species, each assigned to one of eight thematic dimensions: Taxonomy; Be2 Portal https://www.wikipedia.org/; taxa are linked through Wikidata and article plain-text extracts are retrieved
via the MediaWiki API.
4
BAGEL BENCHMARK
havior (including social behavior where applicable); Communication; Morphology; Habitat; Cognition; Geographic Distribution; and Diet. Every item must be justified solely by explicit statements in the supplied extract; dimensions not supported by the text are skipped. When the article mentions vocalization, geographic range, or feeding, the prompt encourages at least one item in Communication, Geographic Distribution, or Diet, respectively. Parsed outputs are kept only if they pass simple structural checks (valid dimension label, exactly four options, and a designated correct option that matches one option string exactly). At evaluation time, models see only the stem and answer choices; article text and construction metadata are withheld, consistent with the closed-book protocol in Section 4. 3.2
Global Biotic Interactions (GloBI)
The GloBI subset targets ecological interaction reasoning among species. Tabular records in the Global Biotic Interactions exchange format describe directed links between a source taxon and a target taxon together with an interaction-type label and optional locality, coordinates, observation time, life-stage or body-part fields, habitat, and bibliographic provenance.3 Preprocessing. We read at most 10,000 rows from the record table and harmonize column names across common GloBI export conventions. Rows without both endpoint taxa and an interaction-type label are excluded. Each retained row is converted into a short naturallanguage summary of the interaction together with optional locality, date, and coordinate metadata, plus dataset- and reference-level provenance. Balanced subsampling. From the annotated pool we select 3,500 interactions, stratifying by interaction type so that the empirical distribution over relation labels is more uniform than under uniform random sampling of rows. Within each type, rows with richer contextual metadata (locality, coordinates, dates) are preferred when ties arise, using a fixed random seed for reproducibility; the final subset is shuffled before question synthesis. Multiple-choice synthesis. For each selected interaction, we prompt GPT-4o-mini through the OpenAI API with the natural-language interaction summary as the only textual evidence. The instruction format is given in Appendix A.1.1. The model must output exactly one closed-book, four-option item labeled as either Masked participant identification or Masked interaction type inference, obeying constraints that discourage trivial verb cues and encourage ecologically plausible distractors. Malformed or unparseable outputs are discarded. 3.3 bioRxiv The bioRxiv subset targets scientific-literature reasoning about animals. We harvest animalrelated articles directly from the public bioRxiv website4 using its month-indexed advanced search interface, restricting to posts dated from 2023 onward and to four subject areas: Animal Behavior and Cognition, Ecology, Evolutionary Biology, and Zoology. Preprocessing. Title, abstract, and article body are concatenated into one working document. For body text, we normalize whitespace, remove exact duplicate paragraphs, and drop paragraphs that begin with ”Figure”. We then apply a light text-cleaning pass (newline/pagenumber cleanup and whitespace normalization). We do not apply an additional hard length cap for bioRxiv documents since the text is within the context window that GPT-4o-mini can handle. Multiple-choice synthesis. Each document is presented to GPT-4o-mini via the OpenAI API. The prompt specification appears in Appendix A.1.1. The model must return exactly one four-option item of type Result Interpretation: the stem must state enough of the experimental setup and findings that the item is answerable without the PDF, and all claims must be grounded in the supplied excerpt. Malformed outputs are discarded. The released benchmark contains 2,183 questions (Table 2). 3 GloBI portal and indexed interaction data: https://www.globalbioticinteractions.org/.
4 bioRxiv server: https://www.biorxiv.org/.
5
BAGEL BENCHMARK
3.4 Xeno-canto The Xeno-canto subset focuses on text-based bioacoustic knowledge concerning animals, including typical vocalization characteristics and coarse-grained acoustic structures at the species level. It is derived from the Xeno-canto community repository (Vellinga and Planqué, 2015)5 , which provides user-contributed recordings and associated species metadata. At test time, models receive only the question stem and four options; spectrograms and audio are withheld, so scores reflect vocabulary and reasoning over animal sounds in text, not perception of a played waveform.Appendix Figure 3 provides a demonstration of the Xeno-canto MCQs generation pipeline. Species coverage. Xeno-canto items are built only for species that already appear in the Wikipedia construction pipeline: we use the same scientific-name inventory as for the Wikipedia subset, so every taxon with a candidate recording is one for which we also derived species-level questions from English Wikipedia (subsection 3.1). Recording retrieval and filtering. Recordings are retrieved from Xeno-Canto through its public query interface. For each species we request clips between 5 and 600 seconds in length, exclude material released under no-derivatives licenses, and retain at most 100 accepted files per species before further processing. We also align the species list to an internal recording-level metadata inventory with richer taxonomic and recording attributes than a minimal species-level export, matching on unified scientific names. Audio is analyzed at 16 kHz; clips shorter than five seconds or with peak amplitude below 0.1 are dropped, and longer files are truncated to the first 30 seconds for feature extraction. Each retained waveform receives a scalar clarity score from a lightweight signal-processing heuristic that combines band-limited signal-to-noise (0.5–10 kHz), separation of harmonic versus percussive energy, mean spectral flatness, and a penalty for clipping. When many candidates exist per species, we randomize order and keep a small fixed number to diversify recordings while controlling cost. Question synthesis. For recordings with usable vocalization labels, free-text type strings are normalized: mixed or ambiguous labels (e.g., simultaneous ”song” and ”call”), empty tags, or ”uncertain” entries are skipped, and otherwise we retain a single canonical phrase. We render a log-frequency spectrogram of at most 30 seconds (audio resampled at 44.1 kHz for display) and provide it together with common and scientific names to GPT-4o-mini through the OpenAI API. The instruction schema (Appendix A.1.1) asks for exactly one four-option item in one of four randomly drawn topic families—dominant frequency range, call or syllable duration, modulation pattern, or harmonic structure versus broadband noise—with respective sampling weights of approximately 65%, 15%, 10%, and 10%. The model should use the image only as support for species-typical acoustic descriptions and must not mention spectrograms, files, or recordings in the question; the stem must explicitly name the vocalization type (e.g., song versus call). 3.5 Mitigating answer-position imbalance. After initial construction, we found that the index of the correct option among the four ordered choices was highly skewed across domains (Appendix F, Table 9). Such imbalance is problematic for evaluation: recent work shows that large language models are sensitive to option order in multiple-choice tests and can exhibit systematic selection bias toward particular answer labels independent of content (C. Zheng et al., 2024; Pezeshkpour and Hruschka, 2024). To reduce the risk that models exploit positional shortcuts, we apply a random permutation of the four options for each item with a fixed random seed, updating the stored correct label or index accordingly. Table 10 summarizes the resulting distribution over all 11,852 multiple-choice items in the released shuffled split: the empirical frequency of each gold position stays within about one percentage point of the nominal 25% under a balanced design. Figure 2 provides one representative example item from each source domain, illustrating heterogeneity across encyclopedic animal knowledge (Wikipedia), interaction reasoning among animal species (GloBI), scientific-literature reasoning about animals (bioRxiv), and bioacoustic text (Xeno-canto). Additional examples for every domain-specific dimension are provided in Appendix B. 5 Recordings and species metadata: https://www.xeno-canto.org/.
6
BAGEL BENCHMARK
bioRxiv
Wikipedia
Dimension: Result Interpretation Level: easy Question: In a study of the sea anemone Nematostella vectensis, researchers found that animals kept in environments with gravel substrate produced significantly more clonal progeny through transverse fission compared to those without substrate. Given that substrate enhances fission rates, what can be inferred about its role in asexual reproduction? Options: A. Substrate provides a mechanical advantage that facilitates the physical process of fissioning. B. Substrate increases the genetic diversity of the clones produced during fission. C. Substrate reduces the metabolic waste that inhibits fission in high-density populations. D. Substrate alters the hormonal balance in Nematostella, promoting faster growth. Gold: A. Substrate provides a mechanical advantage that facilitates the physical process of fissioning.
Dimension: Communication Level: medium Question: What is one of the distinct call types of the Andean flamingo? Options: A. Whistle B. Chirp C. Growl D. Peep Gold: D. Peep
GloBI Dimension: Masked participant identification Level: hard Question: Which taxon is likely to be the host in the interaction with Pallisentis nagpurensis? Options: A. Cichlidae B. Channa striata C. Heteropneustes fossilis D. Pallisentis nagpurensis Gold: B. Channa striata
Xeno-canto Dimension: Dominant frequency range Level: easy Question: What is the typical dominant frequency range of the call of the Red Warbler (Cardellina rubra)? Options: A. ∼3–5 kHz B. ∼1–2 kHz C. ∼10–12 kHz D. ∼7–9 kHz Gold: A. ∼3–5 kHz
Figure 2. Representative example items from each BAGEL domain. Level indicates the reference difficulty label from Table 3 (agreement between GPT-5.4 and Claude Opus 4.6 on the released level field). The four panels are chosen to include easy, medium, and hard strata across domains.
4
Experimental Setup
4.1 Models We evaluate two frontier closed-source models—GPT-5.4 and Claude Opus 4.6—and a ladder of open-weight models from SmolLM2-360M (Ben Allal et al., 2025) upward, including five Qwen3 family models (0.6B, 4B, 8B, 14B, and 32B with reasoning mode turned off) (A. Yang et al., 2025), Gemma 3 27B IT (Team et al., 2025) and Gemma-7B (Gemma Team, 2024), Llama 3.1-8B Instruct (Grattafiori et al., 2024), Mistral-7B-Instruct (Jiang et al., 2023), and Phi-4 (Abdin et al., 2024). 4.2 Evaluation Protocol All models are evaluated in a closed-book multiple-choice setting. At test time, the model is given a unified prompt containing only the task instruction, the question stem, and four enumerated answer options; the source passage used during benchmark construction is not provided at inference time. We use deterministic decoding with seed 0 and greedy generation for the reported results, so each model is represented by a single run rather than an average across seeds. We report accuracy on each source domain and an overall score across domains. The evaluation prompt is provided in Appendix A.1.2. 4.3
Dataset Statistics
Table 2 summarizes the composition of BAGEL: 11,852 four-option, single-answer multiplechoice questions across bioRxiv (2,183), GloBI (3,500), Wikipedia (1,927), and Xeno-canto (4,242), along with source-specific dimensions and each category’s share within its domain. Table 3 reports how many items fall into each reference difficulty level (easy / medium / hard) when defined by agreement between GPT-5.4 and Claude Opus 4.6; the same labels are stored
7
BAGEL BENCHMARK
Domain
Dimension
bioRxiv
Result Interpretation
GloBI GloBI Wikipedia Wikipedia Wikipedia Wikipedia Wikipedia Wikipedia Wikipedia Wikipedia Xeno-canto Xeno-canto Xeno-canto Xeno-canto
bioRxiv (# = 2,183)
# Questions
Share in Domain
2,183
100.0%
2,120 1,380
60.6% 39.4%
353 296 281 268 234 234 233 28
18.3% 15.4% 14.6% 13.9% 12.1% 12.1% 12.1% 1.5%
2,747 649 427 419
64.8% 15.3% 10.1% 9.9%
GloBI (# = 3,500) Masked interaction type inference Masked participant identification Wikipedia (# = 1,927) Behavior Diet Geographic Distribution Taxonomy Communication Habitat Morphology Cognition Xeno-canto (# = 4,242) Dominant frequency range Call / syllable duration Harmonic structure / tonality vs. broadband Modulation pattern
Table 2. Composition of BAGEL across source domains and source-specific dimensions. The full benchmark contains 11,852 questions, with average question length of 180.54 characters and average option length of 35.68 characters. Domain Wikipedia Xeno-canto GloBI bioRxiv All
# Items 1,927 4,242 3,500 2,183 11,852
Easy 1,681 (87.2%) 1,836 (43.3%) 2,549 (72.8%) 1,957 (89.6%) 8,023 (67.7%)
Medium 163 (8.5%) 1,218 (28.7%) 439 (12.5%) 135 (6.2%) 1,955 (16.5%)
Hard 83 (4.3%) 1,188 (28.0%) 512 (14.6%) 91 (4.2%) 1,874 (15.8%)
Table 3. Reference difficulty strata induced by GPT-5.4 and Claude Opus 4.6 under our evaluation protocol: easy if both models answer correctly, medium if exactly one is correct, and hard if both are wrong.
in the released data as level. We treat these as reference difficulty strata rather than absolute item difficulty, and a subset of “hard” items may also reflect option-level ambiguity. Reported lengths are measured in characters on the released benchmark files: the mean question stem length is 180.5 and the mean length of a single option string is 35.7 (averaged over all options).
5
Results and Discussion
5.1 Main leaderboard Table 4 reports the main results on BAGEL. Three findings stand out. First, performance varies substantially across source domains, even for the strongest models. For example, frontier closed-source models perform very strongly on Wikipedia and bioRxiv, yet remain notably weaker on Xeno-canto. This gap indicates that BAGEL is not merely measuring broad factual recall, but also source-sensitive expertise in animal and natural history knowledge. Second, open-weight models span a wide band below the proprietary frontier. Among the open models in Table 4, Gemma 3 27B IT reaches the strongest overall open score (0.6789) under our protocol, followed closely by Llama 3.1 Instruct-8B (0.6522) and Phi-4 (0.6511), while Qwen332B is highest on Wikipedia accuracy. None of the included open models surpass GPT-5.4’s overall score (0.7601), highlighting a remaining gap to the proprietary entry in this evaluation setup. Third, the smallest open models remain near the random baseline on aggregate (Table 4), while mid-sized models already reach non-trivial accuracy on text-heavy domains yet can fail on Xeno-canto, indicating uneven scaling of biodiversity-related competence. Then we evaluate five instruct checkpoints from the same Qwen3 line—0.6B, 4B, 8B, 14B, and 32B—under an identical prompt and greedy seed-0 protocol (Table 4). Overall accuracy rises sharply from near-random performance at 0.6B (0.2464) into the 0.57–0.65 band for 4B–32B, and GloBI, Wikipedia, and bioRxiv scores generally increase with size up to 32B. Xeno-canto
8
BAGEL BENCHMARK
Model
GloBI
Wiki
bioRxiv
Xeno-canto
Overall
Empirical Random Guess
0.2686
0.2247
0.2652
0.2520
0.2549
Closed-Source Frontier Models GPT-5.4
0.7720
0.8988
0.9441
0.5926
0.7601
Claude Opus 4.6
0.8100
0.9305
0.9107
0.5601
0.7587
Open-Weight Models smolLM2 - 360M
0.2477
0.2372
0.2510
0.2504
0.2476
Qwen3 - 0.6B
0.2523
0.2532
0.2405
0.2414
0.2464
Qwen3 - 4B
0.5866
0.6767
0.8901
0.3581
0.5753
Gemma-7B
0.4806
0.6487
0.8543
0.2791
0.5046
Mistral-7B-Instruct-v0.3
0.6060
0.6834
0.8557
0.2949
0.5532
Llama 3.1 Instruct - 8B
0.6386
0.7348
0.8882
0.5045
0.6522
Qwen3 - 8B
0.6586
0.7654
0.8992
0.3932
0.6253
Phi-4
0.7251
0.7971
0.9244
0.3831
0.6511
Qwen3-14B
0.6954
0.7940
0.9111
0.4125
0.6499
Gemma 3 27B IT
0.7471
0.8012
0.9098
0.4481
0.6789
Qwen3-32B
0.7074
0.8106
0.9185
0.3751
0.6441
Table 4. Accuracy by domain on BAGEL. Open-weight models are ordered by increasing approximate size. Within the open-weight block, the best value in each column is shown in bold (leaders can differ by column). GPT-5.4 and Claude Opus 4.6 are closed-source references; between those two, the stronger score in each column is underlined.
behaves differently: accuracy moves from 0.2414 (0.6B) through 0.3581 (4B), 0.3932 (8B), and 0.4125 (14B), then falls at 32B (0.3751). Thus the largest Qwen3 checkpoint is not the strongest on Xeno-canto even though it leads on the text-heavy sources. Because Xeno-canto contributes many items, Qwen3-14B slightly edges Qwen3-32B on Overall (0.6499 vs. 0.6441). Figure 4 in the Appendix plots the same five checkpoints with domain-wise curves. Overall, these results show that BAGEL separates models not only by aggregate capability, but also by robustness across heterogeneous sources. Appendix Tables 5–7 break down accuracy by source-specific question type (dimensions match Table 2). 5.2
Discussion
Construct validity and interpretation. BAGEL aggregates four complementary animalcentered tracks—encyclopedic facts (Wikipedia), pairwise species-interaction reasoning (GloBI), scientific-literature-style stems (bioRxiv), and text-only items grounded in bioacoustic metadata (Xeno-canto)—under a single closed-book, four-option MC protocol. Under this setup, reported scores reflect what models can produce from parameters and instructions alone, without access to source passages or external retrieval; they should not be equated with field identification skills, expert ornithological practice, or perception of waveforms when audio is withheld by design (Section 3). Because the four tracks emphasize different surface forms of knowledge, aggregate accuracy is best treated as a coarse summary: substantive conclusions require domain-level and, where available, dimension-level accuracy (Table 4; Appendix Tables 5–7). Following standard concerns about multiple-choice artifacts, we apply independent random permutations of the four answer options at the item level to mitigate systematic gold-position imbalance in the released split (Section 3; Appendix F). Even after this mitigation, some models exhibit uneven emission of option letters, so measured accuracy can still co-vary with format sensitivity in ways that are orthogonal to “content mastery” in the narrow sense.
9
BAGEL BENCHMARK
Heterogeneity and limited cross-domain transfer. The main leaderboard underscores a recurrent pattern: models that achieve strong accuracy on text-rich domains can remain comparatively weak on Xeno-canto (Table 4). This pattern is not an artifact of a small Xeno subset— the Xeno-canto track contributes a large share of items—yet it is frequently the lowest-accuracy column for frontier and open models alike. Such cross-track gaps motivate reading BAGEL as a portfolio evaluation in which overall ranking is secondary to diagnosing which competencies are present or absent (Hendrycks et al., 2021). In practical terms, high performance on encyclopedic or abstract-style items does not license the conclusion that a model will reliably handle acoustics-oriented wording at scale. The Xeno-canto gap: item types, stem statistics, and scaling. The relative weakness on Xeno-canto does not reflect a single uniform deficit, but rather substantial heterogeneity across its underlying subtasks. Appendix Table 7 decomposes accuracy by the four acoustic topic families used during construction (dominant frequency range, call or syllable duration, harmonic structure versus broadband noise, and modulation pattern). The spread across these dimensions is large: for GPT-5.4, accuracy reaches roughly 0.71 on dominant-frequency-range items but only about 0.30 on modulation-pattern items, with other models exhibiting similarly wide within-domain variation and differing strengths across dimensions. The aggregate Xeno-canto score therefore averages over subtasks of markedly different difficulty, which partially explains why it can lag Wikipedia even when both tracks are nominally “about” the same species list. To complement this item-type decomposition, Appendix Table 8 analyzes the linguistic properties of English question stems, using wordfreq unigram Zipf scores as a proxy for generalEnglish word frequency (Speer, 2018). Relative to Wikipedia stems, Xeno-canto stems exhibit (i) substantially higher density of tokens drawn from a fixed bioacoustic keyword list (mean hits per stem: 2.03 versus 0.05), (ii) a larger fraction of low-frequency tokens (Zipf < 3: 18.4% versus 11.8%), and (iii) lower mean unigram Zipf (5.04 versus 5.33). These statistics indicate that Xenocanto operates in a distinct lexical register, combining domain-specific acoustic terminology with less frequent general-English vocabulary. However, lexical rarity alone does not explain performance: mean stem Zipf shows negligible correlation with per-item correctness on Xenocanto for Gemma 3 27B IT (Spearman ρ ≈ −0.03), suggesting that difficulty arises from the interaction of domain-specific vocabulary, reasoning demands, and distractor design rather than from word frequency alone. Finally, scaling behavior further differentiates Xeno-canto from text-heavy domains. Within the Qwen3 family, Xeno-canto accuracy is non-monotonic with parameter count, with the 32B checkpoint underperforming the 14B checkpoint on this domain despite improving on Wikipedia and bioRxiv (Figure 4). This divergence indicates that gains in general language modeling or factual recall do not uniformly translate to improvements in bioacoustic text competence. These results are consistent with the possibility that text-based bioacoustic QA constitutes a partially distinct challenge, although the gap may also reflect differences in lexical register, item construction, and subtask composition. Under our protocol, Xeno-canto evaluates textual reasoning about sound and should be viewed as complementary to, rather than a substitute for, audio-centric animal-sound benchmarks (Hagiwara, Hoffman, et al., 2022; Robinson et al., 2024). Multiple-choice ambiguity in BAGEL. Our findings from the manual audit in Appendix G align with a broader literature showing that multiple-choice evaluation can be confounded by insufficiently discriminative answer options, including semantically overlapping distractors, multiple plausible answers, and the forced single-answer assumption (Palta et al., 2024; Balepur, Rudinger, and Boyd-Graber, 2025; W. Xu et al., 2025). Surveys and systems likewise emphasize distractor quality and option distinctness (Alhazmi et al., 2024; Bitew et al., 2023; Amanlou et al., 2026), while recent work argues for moving beyond strict MC scoring toward generative or matching-based evaluation where appropriate (Chandak et al., 2025). Superficial factors such as option ordering can also shift scores (Pezeshkpour and Hruschka, 2024). We do not claim these issues are unique to any one BAGEL domain: we highlight GloBI below because interaction-centric prompts surface them clearly in our manual review, but analogous risks can arise wherever items are generated under a single-answer protocol.
10
BAGEL BENCHMARK
6
Conclusion and limitations
We introduced BAGEL, an 11,852-item closed-book MC benchmark from bioRxiv, GloBI, Wikipedia, and Xeno-canto. Experiments show large domain spread, a gap to a proprietary reference, persistently weaker Xeno-canto scores for many models, and complementary strengths across open-weight families; Appendix F discusses MC positional effects. Limitations. Results are reported with one seed and greedy decoding, so stochastic decoding or ensembling could shift rankings. BAGEL is closed-book only in the sense that models are evaluated without source passages or external retrieval at inference time; it does not establish that benchmark content was absent from pretraining corpora. Because several source domains are public, scores may reflect a mixture of source familiarity, broader parametric recall, multiple-choice test-taking skill, and task-specific reasoning, which this setup does not cleanly disentangle. The benchmark is English-only and inherits taxonomic, geographic, and source-selection biases from its sources. MC accuracy does not measure calibration, abstention, or open-ended reasoning. Corpus choice is pragmatic and axis-covering rather than exhaustive of natural-history competence; field guides, occurrence corpora, and expert exams are out of scope but complementary. The Xeno-canto-derived domain is text-only and cannot disentangle acoustic jargon from deep understanding. A targeted audit of the “hard” subset indicates that a minority of items contain insufficiently discriminative answer options; the “hard” split should be interpreted as relative challenge under generative multiple-choice design rather than a clean estimate of biological reasoning difficulty in every item. Because questions are generated automatically from source materials, benchmark quality also depends on generation prompts, filtering rules, and distractor construction, which may introduce artifacts not reducible to the source content itself.
11
BAGEL BENCHMARK
References Abdin, Marah, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, et al. (2024). “Phi4 Technical Report.” In: arXiv preprint arXiv:2412.08905. Alhazmi, Elaf, Quan Z. Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi (2024). Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation. arXiv: 2402.01512 [cs.CL]. Amanlou, Mohammad et al. (2026). “KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration.” In: arXiv preprint arXiv:2602.20135. Balepur, Nishant, Rachel Rudinger, and Jordan Lee Boyd-Graber (2025). “Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above.” In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria, pp. 3394–3418. Ben Allal, Loubna, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. (2025). “SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model.” In: arXiv preprint arXiv:2502.02737. Bi, Zhen, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, and Huajun Chen (2023). “OceanGPT: A Large Language Model for Ocean Science Tasks.” In: arXiv preprint arXiv:2310.02031. Bitew, Semere Kiros, Johannes Deleu, Chris Develder, and Thomas Demeester (2023). “Distractor generation for multiple-choice questions with predictive prompting and large language models.” In: arXiv preprint arXiv:2307.16338. Chandak, Nikhil, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping (2025). Answer Matching Outperforms Multiple Choice for Language Model Evaluation. arXiv: 2507 . 02856 [cs.CL]. Deng, Cheng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, Zhouhan Lin, and Junxian He (2023). “K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization.” In: arXiv preprint arXiv:2306.05064. Dorm, Filip, Joseph Millard, Drew Purves, Michael Harfoot, and Oisin Mac Aodha (2025). “Large Language Models Possess Some Ecological Knowledge, but How Much?” In: bioRxiv. Gemma Team (2024). “Gemma: Open Models Based on Gemini Research and Technology.” In: arXiv preprint arXiv:2403.08295. Gougherty, Andrew V. and Hannah L. Clipp (2024). “Testing the Reliability of an AI-Based Large Language Model to Extract Ecological Information from the Scientific Literature.” In: npj Biodiversity 3, p. 13. Grattafiori, Aaron, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. (2024). “The Llama 3 Herd of Models.” In: arXiv preprint arXiv:2407.21783. Gu, Yu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon (2021). “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.” In: ACM Transactions on Computing for Healthcare 3.1, pp. 1–23.
12
BAGEL BENCHMARK
Guo, Jing, Nan Li, and Ming Xu (2025). “Environmental Large Language Model Evaluation (ELLE) Dataset: A Benchmark for Evaluating Generative AI Applications in EcoEnvironment Domain.” In: arXiv preprint arXiv:2501.06277. Hagiwara, Masato, Benjamin Hoffman, Jen-Yu Liu, Maddie Cusimano, Felix Effenberger, and Katie Zacarian (2022). “BEANS: The Benchmark of Animal Sounds.” In: arXiv preprint arXiv:2210.12300. Hagiwara, Masato, Marius Miron, and Jen-Yu Liu (2024). “ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds.” In: arXiv preprint arXiv:2402.03269. Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (2021). “Measuring Massive Multitask Language Understanding.” In: International Conference on Learning Representations (ICLR). Huang, Yu, Liang Guo, Wanqian Guo, Zhe Tao, Yang Lv, Zhihao Sun, and Dongfang Zhao (2024). “EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models.” In: arXiv preprint arXiv:2405.11265. Jiang, Albert Q., Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. (2023). “Mistral 7B.” In: arXiv preprint arXiv:2310.06825. Jin, Qiao, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu (2019). “PubMedQA: A Dataset for Biomedical Research Question Answering.” In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pp. 2567–2577. Keck, François, Henry Broadbent, and Florian Altermatt (2025). “Extracting Massive Ecological Data on State and Interactions of Species Using Large Language Models.” In: bioRxiv. Lu, Pan, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan (2022). “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.” In: Advances in Neural Information Processing Systems (NeurIPS). Nentidis, Anastasios, Georgios Katsimpras, Anastasia Krithara, Salvador Lima López, Eulàlia Farré-Maduell, Luis Gasco, Martin Krallinger, and Georgios Paliouras (2023). “Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering.” In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Lecture Notes in Computer Science. OpenAI (2023). “GPT-4 Technical Report.” In: arXiv preprint arXiv:2303.08774. Ouyang, Long, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe (2022). “Training Language Models to Follow Instructions with Human Feedback.” In: Advances in Neural Information Processing Systems (NeurIPS). Palta, Shramay, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger (2024). “Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning.” In: Findings of the Association for Computational Linguistics: EMNLP 2024. Miami, Florida, USA, pp. 3451–3473. Pezeshkpour, Pouya and Estevam Hruschka (2024). “Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions.” In: Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006–2017.
13
BAGEL BENCHMARK
Poelen, Jorrit H., James D. Simons, and Chris J. Mungall (2014). “Global Biotic Interactions: An Open Infrastructure to Share and Analyze Species-Interaction Datasets.” In: Ecological Informatics 24, pp. 148–159. Robinson, David, Marius Miron, Masato Hagiwara, Benno Weck, Sara Keen, Milad Alizadeh, Gagan Narula, Matthieu Geist, and Olivier Pietquin (2024). “NatureLM-audio: an AudioLanguage Foundation Model for Bioacoustics.” In: arXiv preprint arXiv:2411.07186. Speer, Robyn (2018). wordfreq: Access to word frequency data in many languages. https://github. com/rspeer/wordfreq. Stevens, Samuel, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, WeiLun Chao, and Yu Su (2024). “BioCLIP: A Vision Foundation Model for the Tree of Life.” In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Team, Gemma et al. (2025). Gemma 3 Technical Report. arXiv: 2503.19786 [cs.CL]. URL: https: //arxiv.org/abs/2503.19786. Vellinga, Willem-Pier and Robert Planqué (2015). “The Xeno-Canto Collection and Its Relation to Sound Recognition and Classification.” In: Working Notes of CLEF 2015 Conference. Vol. 1391. CEUR Workshop Proceedings. Xu, Weijie, Shixian Cui, Xi Fang, Chi Xue, Stephanie Eckman, and Chandan K. Reddy (2025). SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions. arXiv: 2506. 00643 [cs.CL]. Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. (2025). “Qwen3 Technical Report.” In: arXiv preprint arXiv:2505.09388. Zheng, Chujie, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang (2024). “Large Language Models Are Not Robust Multiple Choice Selectors.” In: arXiv preprint arXiv:2309.03882.
14
BAGEL BENCHMARK
A Appendix A.1 Prompts We used the following prompts in this project. A.1.1 Benchmark Generation Prompt
Wikipedia domain benchmark generation prompt System: You are a scientific question generator. Given a Wikipedia article about a single species, generate up to eight multiple-choice questions (no freeform). Only use information explicitly mentioned in the text. Each question must: - Fall under one of these dimensions: Taxonomy, Behavior, Communication, Morphology, Habitat, Cognition, Geographic Distribution, Diet. - For Behavior, consider social behavior (e.g., solitary vs. social, group size, social structure) when present. - Have exactly four options and exactly one correct answer. - Be factual, unambiguous, and grounded in the given Wikipedia text (no outside knowledge). - Skip dimensions not covered in the text. If the text mentions vocalization and communication, include at least one question about Communication. If the text mentions where the species is found (e.g., native range, regions, countries), include at least one question about Geographic Distribution. If the text mentions feeding or diet, include at least one question about Diet.\n\n User:Species metadata: ScientificName: {ScientificName} CommonNames: {CommonNames} Taxonomy: Kingdom={Kingdom}; Phylum={Phylum}; Class={Class}; Order={Order}; Family={ Family}; Genus={Genus} WikipediaTitle: {WikipediaTitle} WikipediaText: \”\”\”{WikipediaText}\”\”\” Instructions: Generate up to EIGHT multiple-choice questions across relevant dimensions. Return output as a JSON list under this schema: {{ ”questions”: [ {{ ”dimension”: ”<one of Taxonomy|Behavior|Communication|Morphology|Habitat|Cognition| Geographic Distribution|Diet>”, ”question”: ”<question text>”, ”options”: [”<opt1>”, ”<opt2>”, ”<opt3>”, ”<opt4>”], ”answer”: ”<correct option>” }} ] }}\n\n Assistant:\n\n
15
BAGEL BENCHMARK
GloBI domain benchmark generation prompt System: You are creating benchmark-quality self-contained multiple-choice questions. Return only one valid JSON object User: You are generating benchmark-quality multiple-choice questions for the ecological interaction domain of an animal expertise benchmark. You will be given a short source context derived from a GloBI ecological interaction record. Your task is to generate exactly ONE self-contained multiple-choice question. The benchmark is CLOSED-BOOK: - The final test-taker will NOT see the source context. - The question must stand on its own. - The question must NOT refer to ”the passage”, ”the text”, ”the record”, ”the source”, or any hidden context. - Use ONLY information supported by the provided source context. Do NOT use outside knowledge. Your goal is to produce a question that tests ecological reasoning, not keyword matching. Select exactly ONE dimension that is most suitable for the source context: - Masked interaction type inference - Masked participant identification Dimension guidance: - Masked participant identification: hide one participant and ask which taxon best fills the missing ecological slot. - Masked interaction type inference: hide the relation label and ask which interaction type best fits the interaction pattern. High-value question criteria: - The item must require at least one inference step. - The correct answer should NOT be recoverable from a single cue word alone. - The species names and ecological clues should matter to solving the question. - The question should test ecological understanding, not simple paraphrase or label recognition. Anti-leakage rules: - Do NOT use verbs or phrases that directly reveal the correct interaction type, such as: ”consumes”, ”eats”, ”predates on”, ”parasitizes”, ”pollinates”, ”infects”, ”hosts”, ”mutualist”, ” commensal”, ”parasite”, ”predator”, ”prey”. - Do NOT use wording that is an obvious synonym or near-restatement of the correct answer. - Do NOT write a question that can be answered correctly even if the taxa are replaced with ”species X ” and ”species Y”. - Do NOT make the correct answer obvious from one lexical clue in the stem. - If the source context only supports a trivial label-retrieval question, reject it. For masked interaction type inference, do not ask for the interaction label directly unless the stem requires integrating multiple clues. Prefer questions where the solver must infer the interaction from ecological roles, asymmetry, or biological consequences, not from a single action verb. Requirements for the question: - Include only the clues necessary to support a non-trivial inference. - Prefer ecological reasoning over direct restatement. - Do not simply restate the source context verbatim unless necessary. - Do not overgeneralize from a single documented interaction. - For masked-inference questions, the missing field must be reasonably inferable from the remaining clues. - Avoid questions where multiple options could plausibly be correct. - Use clear scientific language. - The question should read like an ecological reasoning problem, not like a hidden database entry. Requirements for options:
16
BAGEL BENCHMARK
- Exactly four options labeled A, B, C, and D. - Exactly one correct answer. - Distractors must be plausible. - Distractors should reflect realistic ecological confusions, such as: - reversed interaction direction - wrong ecological role - wrong interaction type - plausible but incorrect taxon - unsupported stronger claim Reject low-value questions such as: - direct label retrieval from a cue word - simply copying a species name with no reasoning value - masked questions where the answer is not actually inferable from the clues - items where the stem already contains the answer in nearly explicit form - items that test generic ecology vocabulary rather than animal expertise Before deciding the item is usable, internally check: 1. Can the answer be solved from a single cue word or phrase? 2. Would the item still work the same if the species names were replaced with generic placeholders? 3. Is the correct option mostly a paraphrase of the stem? 4. Does the item require ecological reasoning rather than lexical matching? If the answer to any of 1-3 is yes, strongly prefer rejection. Avoid wording such as: - ”documented” - ”recorded” - ”reported” - ”observed” - ”in the record” - ”according to the source” - ”the passage states” Return valid JSON in one of these two formats. USABLE: { ”usable”: ”true”, ”dimension”: ”<one of: Masked participant identification | Masked interaction type inference>”, ”question”: ”<self-contained question>”, ”options”: { ”A”: ”<option A>”, ”B”: ”<option B>”, ”C”: ”<option C>”, ”D”: ”<option D>” }, ”correct_answer”: ”<A|B|C|D>”, ”supporting_evidence”: [ ”<short evidence span 1>”, ”<short evidence span 2>” ], ”rationale”: ”<brief explanation grounded in the source context>” } REJECTION: { ”usable”: ”false”, ”rejection_reason”: ”<one of: too_sparse | answer_not_self_contained | low_reasoning_value | masked_field_not_inferable | lexical_leakage>”
17
BAGEL BENCHMARK
} SOURCE CONTEXT: {Interaction record}\n\n Assistant:\n\n
BioRxiv domain benchmark generation prompt System: You are creating benchmark-quality self-contained multiple-choice questions. Return only one valid JSON object. User: You are creating benchmark-quality multiple-choice questions from a bioRxiv paper. The final benchmark model will see ONLY the question and answer options. Therefore, the question itself must include all necessary context for answering. Read the paper and generate exactly ONE self-contained 4-choice multiple-choice question. Use ONLY information from the paper. Do NOT use outside knowledge. Only generate the following type of dimension questions: - Result Interpretation — interpreting experimental results
Requirements: - The question must be fully self-contained. - Include the minimum necessary scientific context inside the question stem so that the question is answerable without access to the paper. - The question stem may be longer than usual, but it must remain concise and focused. - The question stem must include setup and evidence when needed, but must NOT directly restate the correct answer or final conclusion in answer-like wording. - Do NOT create a question that can be answered only by having seen the paper. - The question must test scientific reasoning, not trivial word matching. - Distractors must be scientifically plausible and belong to the same conceptual category as the correct answer. - Provide exactly four options labeled A, B, C, and D. - Exactly one option must be correct. - Do not mention the dimension in the question text. - Do not include unsupported information. Question-writing guidance: - Prefer a stem that includes: 1. the study system or biological setting, 2. the relevant manipulation, comparison, or observation, 3. the key evidence or result pattern, 4. the reasoning task. - Avoid copying long sentences directly from the paper. - Lightly rewrite the source information into a natural, self-contained scientific scenario. - Do not make the stem so compressed that the correct answer becomes obvious from wording alone. Before finalizing, check: - Can a model answer the question from the stem alone? - Does the stem avoid directly revealing the correct answer? - Are the options plausible and same-category? - Is the question testing reasoning rather than recall of the paper? Return valid JSON in this schema: { ”dimension”: ”Result Interpretation ”, ”question”: ”<self-contained question stem>”, ”options”: { ”A”: ”<option A>”,
18
BAGEL BENCHMARK
”B”: ”<option B>”, ”C”: ”<option C>”, ”D”: ”<option D>” }, ”correct_answer”: ”<A|B|C|D>”, ”supporting_evidence”: [ ”<short copied or lightly trimmed evidence span 1>”, ”<short copied or lightly trimmed evidence span 2>” ], ”reason”: ”<brief explanation of why the correct answer is supported by the paper>” } PAPER: {paper}\n\n Assistant:\n\n
Xeno-canto domain benchmark generation prompt System: You are an expert in bioacoustics. Use the provided spectrogram only as background to infer general, species-level acoustic facts. Write ONE multiple-choice question about the specified TOPIC for this ’species vocalizations. Do NOT mention or refer to any image/recording/ spectrogram in the question. The question must read like a factual quiz item about typical acoustic properties, not about a specific recording. CRITICALLY: include the VOCALIZATION TYPE explicitly in the question text (e.g., ’song’, ’call’, ’flight call’). User: Animal: {common_name} ({ scientific_name}). Vocalization type: {Vocalization type}. Topic: {topic_label}. Guidance: {guidance} Use appropriate units ({units_hint}). Provide 4 options (–AD) with plausible distractors; exactly one correct. Include approximate numeric ranges where applicable. Do NOT reference any image, recording, or spectrogram. IMPORTANT: The question MUST explicitly mention the vocalization type ’{vt}’. Examples of reasonable value styles: {examples}.\n\n Assistant: \n\n
A.1.2
Evaluation Prompt
Evaluation prompt You are answering a multiple-choice closed-book benchmark question for testing animal expertise. Choose exactly one answer.\n Output exactly one capital letter: A, B, C, or D.\n Do not output any explanation, words, punctuation, or extra text.\n\n Question: {question}\n Options: A. {option_a} B. {option_b} C. {option_c} D. {option_d} Answer:
B
Example Benchmark Items
19
BAGEL BENCHMARK
B.1 Wikipedia Example item Dimension: Taxonomy Level: easy Question: What is the scientific name of the Andean flamingo? Options: A. Phoenicopterus andinus B. Phoenicoparrus andinus C. Phoenicopterus chilensis D. Phoenicoparrus chilensis Gold answer: Phoenicoparrus andinus Example item Dimension: Geographic Distribution Level: easy Question: Where is the Andean flamingo native to? Options: A. The Amazon rainforest B. The Andes mountains of South America C. The coastal regions of Chile D. The plains of Argentina Gold answer: The Andes mountains of South America Example item Dimension: Diet Level: easy Question: What is the primary feeding strategy of the Andean flamingo? Options: A. Carnivorous predation B. Filter feeding C. Scavenging D. Grazing on grass Gold answer: Filter feeding Example item Dimension: Behavior Level: easy Question: How do Andean flamingos adapt their foraging behavior when grouped with other flamingo species? Options: A. They forage alone regardless of group B. They adopt the foraging patterns of the species they are grouped with C. They only forage at night D. They stop foraging altogether Gold answer: They adopt the foraging patterns of the species they are grouped with
20
BAGEL BENCHMARK
Example item Dimension: Communication Level: medium Question: What is one of the distinct call types of the Andean flamingo? Options: A. Chirp B. Peep C. Whistle D. Growl Gold answer: Peep Example item Dimension: Morphology Level: easy Question: What distinguishes the Andean flamingo from other flamingo species? Options: A. Its bright red plumage B. Its yellow legs and three-toed feet C. Its long neck D. Its ability to fly at high altitudes Gold answer: Its yellow legs and three-toed feet Example item Dimension: Habitat Level: easy Question: In which type of environment do Andean flamingos primarily live during the summer? Options: A. Forests B. Salt lakes C. Grasslands D. Deserts Gold answer: Salt lakes Example item Dimension: Cognition Level: easy Question: What cognitive ability is noted in kea regarding problem-solving? Options: A. They can memorize songs B. They can solve logical puzzles C. They can mimic human speech D. They can navigate using stars Gold answer: They can solve logical puzzles
21
BAGEL BENCHMARK
B.2
GloBI
Example item Dimension: Masked participant identification Level: hard Question: Which taxon is likely to be the host in the interaction where Pearsonema plica is an endoparasite? Options: A. Procyon lotor B. Ursus americanus C. Canis lupus D. Lynx rufus Gold answer: Procyon lotor (Verbatim from the released BAGEL GloBI JSONL; underlying GloBI record: Pearsonema plica endoparasiteof Procyon lotor.) Example item Dimension: Masked interaction type inference Level: easy Question: In an ecological interaction where a tree species is negatively affected by a fungal organism, what type of interaction is most likely occurring? Options: A. The tree is providing nutrients to the fungus. B. The fungus is benefiting at the expense of the tree. C. The tree is benefiting the fungus without any harm. D. The tree and fungus are mutually benefiting each other. Gold answer: The fungus is benefiting at the expense of the tree. B.3 bioRxiv Example item Dimension: Result Interpretation Level: easy Question: In a study of the sea anemone Nematostella vectensis, researchers found that animals kept in environments with gravel substrate produced significantly more clonal progeny through transverse fission compared to those without substrate. Given that the presence of substrate is shown to enhance fission rates, what can be inferred about the role of substrate in the asexual reproduction of Nematostella vectensis? Options: A. Substrate provides a mechanical advantage that facilitates the physical process of fissioning. B. Substrate increases the genetic diversity of the clones produced during fission. C. Substrate reduces the metabolic waste that inhibits fission in high-density populations. D. Substrate alters the hormonal balance in Nematostella, promoting faster growth. Gold answer: Substrate provides a mechanical advantage that facilitates the physical process of fissioning.
22
BAGEL BENCHMARK
B.4 Xeno-canto Example item Dimension: modulation pattern Level: hard Question: What common frequency modulation (FM) pattern is typically observed in the song of the Red Warbler (Cardellina rubra)? Options: A. Rising glide B. Sinusoidal vibrato C. Falling glide D. Trill Gold answer: Rising glide Example item Dimension: dominant frequency range Level: medium Question: What is the typical dominant frequency range of the song of the Red Warbler (Cardellina rubra)? Options: A. ∼9–11 kHz B. ∼3–5 kHz C. ∼6–8 kHz D. ∼1–2 kHz Gold answer: ∼3–5 kHz Example item Dimension: call/ syllable duration Level: easy Question: What is the typical duration of a single call for the Red Warbler (Cardellina rubra)? Options: A. 80–150 ms B. 200–400 ms C. 0.8–1.2 s D. 1.5–2.0 s Gold answer: 80–150 ms Example item Dimension: harmonic structure / tonality vs. broadband Level: hard Question: Are the calls of the Red Warbler (Cardellina rubra) typically characterized as tonal with harmonics or broadband/noisy? Options: A. Strong harmonics B. Weak harmonics C. Broadband D. Tonal whistle Gold answer: Strong harmonics
C
Xeno-canto generation pipeline demonstration plot
Figure 3 shows the pipeline to create text-based bioacoustic knowledge MCQs.
23
BAGEL BENCHMARK
LLM (gpt-4o-mini)
Question: What is the typical dominant frequency range of the call of the Western Corella (Cacatua pastinator)? Options: A: 0.5–1 kHz B: 2–4 kHz C: 4–6 kHz D: 10–12 kHz Answer: B
Figure 3. Xeno-canto question generation pipeline. The process involves feeding a log-frequency spectrogram of a bioacoustic recording into GPT-4o-mini to generate structured, multiple-choice questions based on visual acoustic features.
D
Accuracy by question type within each source domain
Table 6 shows the breakdown accuracy of all the models across Behavior, Cognition, Communication, Diet, Geographic Distribution, Habitat, Morphology type questions on the Wikipedia domain. Table 7 shows the breakdown accuracy of all the models across Dominant frequency range and Call/syllable duration, Harmonic vs broadband, and Modulation type of text-based bioacoustic knowledge questions in the Xeno-canto domain. Table 5 shows the breakdown accuracy among Masked interaction type inference and masked participant identification type questions in the GloBI domain.
E Qwen3 family performance on BAGEL Figure 4 shows the Qwen3 family accuracy curve on BAGEL.
F
Answer-position bias in the initial release
After inspecting model outputs on the initial BAGEL release (before option shuffling), we found substantial answer-position skew in the benchmark itself. Table 9 shows that the correct option was often concentrated in a single position within a domain, despite the multiple-choice format. This skew appears to have created a shortcut that some small models could exploit under greedy decoding. Table 10 reports the distribution after the option-shuffling mitigation described at the end of Section 3. We then examined whether models showed corresponding output collapse or positional preference. Table 11 reports the distribution of first-option letters emitted by five representative open models under the same seed-0 greedy-decoding setting used in the main paper. Qwen30.6B collapses almost entirely to option A across all four domains, while smolLM2-360M collapses primarily to option B. Larger models do not collapse as completely, but several still exhibit strong positional skew, especially toward option B on Wikipedia and Xeno-canto. Their apparent accuracy in some domains therefore tracks the dominant gold-option position rather than necessarily reflecting genuine domain competence. This artifact is important for interpreting the initial leaderboard. In particular, strong scores from small models on some domains should not be read as direct evidence of robust animal-
24
BAGEL BENCHMARK
Model
Masked interaction type inference
Masked participant identification
smolLM2
0.258
0.233
Qwen3-0.6B
0.254
0.250
Qwen3-4B
0.801
0.257
Gemma-7B
0.591
0.312
Mistral-7B-Instruct-v0.3
0.757
0.374
Llama 3.1 Instruct-8B
0.808
0.378
Qwen3-8B
0.814
0.420
Phi-4
0.880
0.488
Qwen3-14B
0.850
0.459
Gemma 3 27B IT
0.888
0.530
Qwen3-32B
0.855
0.481
Claude Opus 4.6
0.921
0.640
GPT-5.4
0.909
0.561
Table 5. Accuracy by question type on the GloBI subset of BAGEL. Open-weight models are ordered by increasing approximate parameter count. Best value in each column is underlined. Model
Behavior
Cognition
Communication
Diet
Geographic Distribution
Habitat
Morphology
Taxonomy
smolLM2
0.227
0.250
0.218
0.236
0.242
0.222
0.279
0.239
Qwen3-0.6B
0.246
0.179
0.239
0.236
0.246
0.256
0.292
0.272
Qwen3-4B
0.635
0.607
0.650
0.720
0.754
0.812
0.605
0.578 0.679
Gemma-7B
0.524
0.571
0.538
0.693
0.751
0.846
0.545
Mistral-7B-Instruct-v0.3
0.606
0.643
0.679
0.736
0.786
0.838
0.541
0.616
Llama 3.1 Instruct-8B
0.677
0.643
0.692
0.777
0.790
0.859
0.665
0.705
Qwen3-8B
0.722
0.643
0.731
0.838
0.826
0.897
0.691
0.672
Phi-4
0.737
0.857
0.752
0.845
0.854
0.932
0.704
0.761
Qwen3-14B
0.722
0.821
0.718
0.845
0.875
0.923
0.734
0.750
Gemma 3 27B IT
0.700
0.750
0.765
0.895
0.872
0.923
0.708
0.769
Qwen3-32B
0.742
0.607
0.774
0.865
0.879
0.936
0.712
0.799
Claude Opus 4.6
0.870
0.857
0.897
0.939
0.989
0.983
0.893
0.963
GPT-5.4
0.819
0.786
0.850
0.912
0.979
0.970
0.863
0.929
Table 6. Accuracy by question type on the Wikipedia subset of BAGEL. Open-weight models are ordered by increasing approximate parameter count. Best value in each column is underlined; ties are jointly underlined.
knowledge competence when the dominant output letter aligns with the dominant gold-answer position. In follow-up experiments, we therefore shuffled answer options to remove this benchmark shortcut.
G
Illustrative manual review on GloBI.
We manually sampled GloBI items to characterize recurring construction issues; this is not a full relabeling or prevalence study, and we do not quantify rates here. We observed three illustrative buckets. (i) Multi-plausible answers and taxonomic overlap. For example, globi:2:0 asks for a likely host of an endoparasite without restating the host from the source interaction; broad host ranges can make more than one species-level option defensible. globi:205:0 and globi:525:0 can pair a concrete host species with a higher taxon, so multiple options are plausible depending on whether “taxon” is read at species or higher rank. globi:207:0 and globi:303:0 mix species and subspecies names for the same lineage, creating taxonomy-granularity collisions. (ii) Under-specified interaction prompts. Flower-visitation templates such as globi:21:0, globi:420:0, globi:736:0, globi:785:0, and globi:1436:0 can admit more than one interaction la-
25
BAGEL BENCHMARK
Model
Dominant freq. range
Call / syll. dur.
Harmonic vs. broadband
Modulation
smolLM2
0.248
0.257
0.269
0.236
Qwen3-0.6B
0.237
0.271
0.225
0.239
Qwen3-4B
0.375
0.342
0.403
0.227
Gemma-7B
0.270
0.314
0.321
0.241
Mistral-7B-Instruct-v0.3
0.271
0.327
0.377
0.317
Llama 3.1 Instruct-8B
0.584
0.325
0.464
0.305
Qwen3-8B
0.427
0.322
0.297
0.382
Phi-4
0.443
0.308
0.286
0.205
Qwen3-14B
0.442
0.421
0.323
0.301
Gemma 3 27B IT
0.521
0.260
0.389
0.325
Qwen3-32B
0.403
0.290
0.393
0.303
Claude Opus 4.6
0.671
0.435
0.286
0.310
GPT-5.4
0.711
0.439
0.351
0.301
Table 7. Accuracy by question type on the Xeno-canto subset of BAGEL. Columns follow the dimension labels in Table 2 (harmonic structure / tonality vs. broadband; modulation pattern). Open-weight models are ordered by increasing approximate parameter count. Best value in each column is underlined. Source domain
Items
Mean Zipf
Rare tok. (%)
Bio-term hits/stem
Wikipedia
1,927
5.33
11.8
0.05
Xeno-canto
4,242
5.04
18.4
2.03
Table 8. Lexical statistics of English question stems (options excluded), computed with wordfreq (Speer, 2018). Higher Zipf indicates more frequent unigrams in general English. “Rare tok.” is the mean fraction of stem tokens with Zipf < 3. “Bio-term hits” counts tokens from a fixed bioacoustic keyword list (e.g., harmonic, khz, modulation).
bel when “visits flowers” alone does not pin down pollination outcomes; globi:47:0 mixes relationship-type wording with mechanistic phrasing so two options can appear simultaneously acceptable for a parasite–host association. (iii) Record-specific keys. Some host items key to a particular interaction record even when alternative hosts are plausible without the source string, so performance partially reflects record recovery rather than species biology alone. Our two-model “hard” stratum should therefore be read as a relative difficulty signal under generative MCQ noise rather than a guarantee of a single objectively correct option in every case; future releases should include additional adjudication, taxonomic de-duplication, and tighter prompt constraints where appropriate.
26
BAGEL BENCHMARK
1.0
BAGEL accuracy versus Qwen3 model size (seed 0, greedy)
Accuracy
0.8 0.6 0.4 0.2 0.0
0.6B
GloBI Wikipedia bioRxiv
Xeno-canto Overall
4B 8B 14B Qwen3 instruct checkpoint (parameters)
32B
Figure 4. Qwen3 instruct family on BAGEL (seed 0, greedy): accuracy versus model size for each source domain and Overall. GloBI, Wikipedia, and bioRxiv generally improve up to 32B, while Xeno-canto peaks at 14B and drops at 32B under our protocol—the same non-monotonicity discussed in the text.
Domain
A
B
C
D
bioRxiv
659 (30.2%)
1,143 (52.4%)
325 (14.9%)
56 (2.6%)
GloBI
2,135 (61.0%)
658 (18.8%)
633 (18.1%)
74 (2.1%)
Wikipedia
199 (10.3%)
1,244 (64.6%)
445 (23.1%)
39 (2.0%)
Xeno-canto
618 (14.6%)
3,231 (76.2%)
333 (7.8%)
60 (1.4%)
Table 9. Correct-answer position distribution in the initial BAGEL release before option shuffling. Percentages are computed over the evaluated subset used by the benchmark loader.
Domain
A
B
C
D
bioRxiv
519 (23.8%)
556 (25.5%)
551 (25.2%)
557 (25.5%)
GloBI
883 (25.2%)
877 (25.1%)
869 (24.8%)
871 (24.9%)
Wikipedia
487 (25.3%)
479 (24.9%)
490 (25.4%)
471 (24.4%)
Xeno-canto
1,034 (24.4%)
1,062 (25.0%)
1,053 (24.8%)
1,093 (25.8%)
All domains
2,923 (24.7%)
2,974 (25.1%)
2,963 (25.0%)
2,992 (25.2%)
Table 10. Correct-answer position distribution after option shuffling, counting individual multiplechoice items in the released evaluation files (n=11,852).
27
BAGEL BENCHMARK
Model
Domain
A
B
C
D
Qwen3-0.6B
bioRxiv
2,178 (99.8%)
3 (0.1%)
1 (0.0%)
1 (0.0%)
Qwen3-0.6B
GloBI
3,500 (100.0%)
0
0
0
Qwen3-0.6B
Wikipedia
1,927 (100.0%)
0
0
0
Qwen3-0.6B
Xeno-canto
4,242 (100.0%)
0
0
0
smolLM2-360M
bioRxiv
1 (0.0%)
2,168 (99.3%)
14 (0.6%)
0
smolLM2-360M
GloBI
138 (3.9%)
3,276 (93.6%)
85 (2.4%)
1 (0.0%)
smolLM2-360M
Wikipedia
85 (4.4%)
1,517 (78.7%)
323 (16.8%)
2 (0.1%)
smolLM2-360M
Xeno-canto
1 (0.0%)
4,225 (99.6%)
16 (0.4%)
0
Qwen3-8B
bioRxiv
731 (33.5%)
1,056 (48.4%)
319 (14.6%)
77 (3.5%)
Qwen3-8B
GloBI
1,640 (46.9%)
681 (19.5%)
757 (21.6%)
422 (12.1%)
Qwen3-8B
Wikipedia
432 (22.4%)
989 (51.3%)
398 (20.7%)
108 (5.6%)
Qwen3-8B
Xeno-canto
1,367 (32.2%)
1,965 (46.3%)
741 (17.5%)
169 (4.0%)
Llama 3.1 Instruct-8B
bioRxiv
716 (32.8%)
1,065 (48.8%)
331 (15.2%)
71 (3.3%)
Llama 3.1 Instruct-8B
GloBI
1,655 (47.3%)
812 (23.2%)
829 (23.7%)
204 (5.8%)
Llama 3.1 Instruct-8B
Wikipedia
453 (23.5%)
1,016 (52.7%)
373 (19.4%)
85 (4.4%)
Llama 3.1 Instruct-8B
Xeno-canto
1,131 (26.7%)
2,818 (66.4%)
229 (5.4%)
64 (1.5%)
Table 11. Distribution of emitted option letters in the initial BAGEL release for representative open models under seed-0 greedy decoding. Smaller models exhibit near-complete collapse to a single option, while stronger models still show substantial positional skew in several domains, especially Wikipedia and Xenocanto.
28