FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth

FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth Josef Chen

Erim Hayretci

Independent Researcher

Imperial College London

August 2026

arXiv:2608.20574v1 [cs.AI] 20 Aug 2026

Abstract Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14418 model–task cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate at r = 0.89 (rank ρ = 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0–69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result.

1

Introduction

observations per model pair instead of inferring a ranking from sparse votes or comparing endpoints on different task subsets. The paper makes three contributions: 1. an executable culinary benchmark with 29904 modelindependent portfolio scores and a single interpretable 0–100 metric; 2. a 27-model, 534-task complete-core evaluation with simultaneous uncertainty and paired, multiplicitycontrolled tests; and 3. a content-addressed release in which every public number traces to frozen tasks, exact routes, raw responses, and a dependency-light offline scorer.

Benchmarks are strongest when the evaluator can be executed. Code can be tested, database states can be inspected, and games can be scored. Culinary reasoning is usually evaluated with a model judge, a handful of human preferences, or factual questions whose answers do not capture the quality of a complete decision. That leaves two problems: the judge is entangled with the systems being tested, and exact match throws away useful differences between plausible answers. FlavourBench turns a culinary runtime into a benchmark environment. Epicure represents 1,790 ingredients in a 300-dimensional space and exposes deterministic operations for substitution, pairing, dietary feasibility, and regional composition [12, 13]. A task compiler converts these operations into three-of-eight selection problems. It enumerates every candidate portfolio and freezes a continuous 0–100 score map before model execution. An answer is therefore scored by table lookup, without a model judge and without post-hoc interpretation. This setup separates the reference system from the evaluated systems. Epicure is not a leaderboard entry and is not “100% accurate.” It defines the task-specific reward surface. A model receives 100 on a task only if it selects that task’s highest-scoring portfolio; other valid portfolios receive partial credit. The benchmark measures agreement with a published executable culinary environment, not universal human taste. We use that environment to evaluate 27 current endpoints. Every model faces the same 534 tasks and 56 frozen answer scores per task. The complete common core supports direct paired comparisons on 534 shared

2

Executable truth

culinary

ground

2.1

A reference environment, not a model judge

The benchmark binds an exact Epicure release, evidence bundle, application build, ingredient count, and embedding dimensionality by hash. Changing any of these inputs creates a new benchmark task-set identity. This is analogous to fixing a simulator or a code-test suite: the reference is explicit and reproducible, but its external validity remains a scientific question rather than an assumption hidden inside a judge prompt. For each task t, let At contain all 56 three-item subsets of eight candidates. Epicure assigns a raw utility ut (a) to every valid a ∈ At . Invalid portfolios under a dietary or category constraint receive zero. Valid utilities are 1

Epicure runtime 1,790 ingredients versioned 300-D space

Task compiler 534 shared prompts 3 decision families

Exhaustive oracle
8 = 56 portfolios 3 dense 0–100 map

Frontier panel 27 exact endpoints no fallback

Statistical release score + simultaneous CI all 351 paired tests

Set parser three distinct labels exact common core

Fixed before score inspection: task eligibility, all score maps, parser, taskselection hash, inference seeds, and the exact route for each model block

Figure 1: FlavourBench compiles a versioned culinary runtime into dense selection tasks. The evaluated models never judge one another and do not access Epicure. All score maps and the per-panel contracts are fixed before collection. The complete-core task set is selected from response validity alone, before inspecting any model selection or score. make the culinary decision, not whether it can copy a runtime return value after being told which operation to call.

normalized within the task: st (a) = 100

ut (a) − minb∈At ut (b) . maxb∈At ut (b) − minb∈At ut (b)

(1)

Every task has a unique 100-point optimum. The exact chance P baseline is not a generic percentage; it is |At |−1 a st (a) for that frozen task.

2.2

Evaluation design

3.1

One primary score

For model m, panel p, family f , and task t, let Ymt be the frozen portfolio score and let Tpf be the score-blindly selected set of 89 tasks completed by every model. The FlavourBench Score is

Three complete-core decision families

The ranked core contains 534 tasks: 89 from each family in each of two independently compiled panels, spanning 534 unique anchor ingredients. Tasks sharing an anchor move together as one cluster in inference. Candidate sets are selected across validation strata, then ordered and labeled by a task-derived hash. No evaluated model writes an item, distractor, or score. Regional-composition tasks are released as a supplemental track but are not in the 27-model leaderboard because they do not support an equally complete common core. The score maps have substantial resolution rather than one correct key plus 55 equivalent mistakes. Table 2 reports the exact random-portfolio baseline, separation of the best and second-best portfolios, and the number of distinct score values. Constraint tasks are intentionally sparse because infeasible portfolios score zero; the other families provide much finer graded rewards.

2.3

3

Fm =

3 2 1 XX 1 X Ymt . 2 · 3 p=1 |Tpf | f =1

(2)

t∈Tpf

It ranges from 0 to 100 and equals the ordinary mean over all 534 cells because the six panel–family strata have equal size. Every ranked endpoint contributes exactly 534 valid responses. Failures are neither assigned zero nor silently dropped model by model: they determine whether a task can enter the shared core, using response status and parser validity only. The task-selection hash is fixed before loading any selected portfolio or score.

3.2

Frontier endpoint panel

The panel spans OpenAI, Anthropic, Google, xAI, Meta, Kimi, Qwen, Z.ai, DeepSeek, ByteDance, Thinking Machines, MiniMax, NVIDIA, Mistral, Tencent, and Cohere. It includes GPT-5.6 Sol, Terra, and Luna; Claude Opus, Sonnet, and Fable 5; Gemini 3.1 Pro and 3.6 Flash; Grok 4.6; Llama 4 Maverick; Muse Spark 1.2 and Muse Glimmer 30B; Kimi K3; Qwen 3.8 Max and Qwen3.8 2.4T A95B; GLM 5.2 and GLM 5.3; DeepSeek V4 Pro 0813 and V4 Flash; Seed 2.1 Turbo; Inkling; MiniMax M3; Nemotron 3.5 Lightning; Mistral Large 3; Tencent HY 3; Cohere Command A; and Cohere Command R+ (08-2024). Z.ai supplied written permission for one finite GLM 5.3 benchmark run, conditioned on the adapter not becoming a permanent running function. We bind that permission and its hash in the release, require the returned identity to equal glm-5.3, and expose no standing endpoint.1

Prompt and parser contract

Prompts show the decision context and eight labeled candidates. The final line must begin with the FINAL_SELECTION marker and contain three distinct A–H labels separated by commas. No prompt supplies a concrete valid triplet. The parser takes the final exact marker line, uppercases the labels, sorts them as an unordered set, and performs one lookup in the frozen score map. Explanatory text does not affect the result. Missing markers, duplicate labels, provider failures, and abnormal finishes remain in the raw response ledger but cannot enter the common core. The model-only condition is deliberate. Epicure supplies the ground truth but is not exposed as a tool during ranking. Thus the score measures whether a model can

1 https://docs.z.ai/guides/llm/glm-5.3

2

Family

Decision

Epicure utility

Substitution

choose a three-item replacement portfolio for an anchor ingredient choose three ingredients to accompany an anchor choose a feasible portfolio under diet and processing limits

0.8 anchor similarity +0.2 portfolio none coherence 0.65 anchor affinity +0.35 portfolio none coherence 0.7 anchor similarity +0.3 coherence diet and maximum NOVA level

Pairing Constraints

Hard constraint

Tasks 178 178 178

Table 1: The three complete-core task families share one response and scoring interface but probe different culinary decisions. Each task contains eight candidates and 56 frozen scores. Family Substitution Pairing Constraints

3.4

Exact chance Top gap Distinct scores 45.2 45.0 3.4

5.6 5.6 37.3

56 56 4

All inferential choices are bound in the content-addressed analysis plan. • Scores. 50000 bootstrap samples are drawn over ingredient anchors; tasks that share an anchor across panels always move together. Each resample is reduced to six panel–family means and then equally weighted. We report pointwise percentile intervals and a simultaneous max-t band across all 27 model scores. • Pairs. Every one of the 351 model pairs uses the same 534 tasks. Two-sided anchor-cluster sign-flip tests use 100000 Monte Carlo draws and Holm correction at familywise α = .05. We also report paired mean differences, bootstrap intervals, and Cohen’s dz . • Chance. Each model is compared with the exact taskwise mean over all 56 portfolios. The 27 paired sign-flip tests form a separate Holm family. • Ranks. Point ranks are accompanied by bootstrap rank intervals and contiguous statistical groups. Models that are not significantly separated are not presented as a resolved total order. The design has 534 independent anchor clusters and 534 paired cells per contrast. Statistical claims are read directly from simultaneous intervals and Holm-adjusted tests; point-rank differences without corrected support are shown as unresolved.

Table 2: Frozen task-map diagnostics for 178 scheduled tasks per family, computed before model execution. Chance is the uniformly random portfolio mean; top gap is the median best–second-best margin. Routes were chosen before each scored block and automatic fallback is disabled. Most endpoints use an exact OpenRouter route; Qwen 3.8 Max uses Alibaba, Fable 5 uses Google Vertex, GLM 5.3 uses the finite Z.ai route above, and selected DeepSeek blocks use an exact GMICloud or DeepInfra route. When transport validation required a replacement, the complete model block was rerun under one new route; successful cells from superseded blocks are not pooled. The manifest binds the requested model, returned identity contract, endpoint tag, provider, supported parameters, and execution contract for both panels. The exact per-model routes appear in Section A. For endpoints that advertise a reasoning-effort control, hidden reasoning is excluded and effort is fixed to the lowest transport-stable setting recorded in the exact analysis plan. Endpoints without that control use their providerfixed behavior. This prevents hidden deliberation from silently consuming the answer budget while leaving the scored response format identical across routes.

3.3

Uncertainty and multiplicity

Score-blind complete-core construction

4

Results

4.1

A statistically resolved FlavourBench leaderboard

All 27 endpoints receive a score on the same 534 tasks. Grok 4.6 has the largest point estimate, 65.1, with simultaneous 95% interval [61.0, 69.2]. Across all 351 prespecified paired contrasts, 101 remain significant after Holm correction. A point rank is therefore a compact summary, not a claim that every adjacent model is distinguishable. Figure 2 shows the score scale, simultaneous intervals, and rank groups; Table 3 gives exact values. The ranked matrix is rectangular: every row contains 534 valid, parser-confirmed responses, so no endpoint is advantaged by answering a different subset. Provider failures and filtered calls remain available in the full execution ledger, but none appears as a zero-valued culinary

The two task compilers each produce 160 candidates per family. After collection, the release re-scores response syntax with one versioned parser and constructs a validity matrix. Within every panel–family stratum it selects the first 89 eligible task IDs under a fixed SHA-256 ordering. A task is eligible only when all 27 models completed it and produced a parseable three-item set. The selection code does not load the chosen ingredients or their scores until the task IDs are fixed. This yields a balanced common core without favoring any model or post-selecting easy tasks by observed score.

3

5

decision and no incomplete row enters this leaderboard.

4.2

Does the ordering replicate on a second task panel?

The exhaustive task map makes FlavourBench useful beyond a static leaderboard. For a task state xt and selected portfolio a, Epicure supplies a deterministic reward r(xt , a) = st (a). This creates three immediate training interfaces: 1. supervised learning from the optimal portfolio and the full ordering of 56 candidates; 2. preference or contrastive learning from any pair (ai , aj ) with known reward difference; 3. offline or online policy optimization using dense bounded rewards and held-out anchors. The present experiment evaluates models rather than training them. The release supplies the environment, reward function, and anchor identifiers required for a clean next experiment: train on development reward maps and measure transfer on disjoint held-out anchors. Direct optimization on the public test map would measure memorization rather than culinary learning.

The two panel-specific scores use the same frozen metric and 89 tasks per family, but disjoint task IDs. Their model-level Pearson correlation is 0.89 and their rank correlation is 0.80. These are descriptive stability diagnostics, not a model-selection rule; the primary leaderboard and its uncertainty use both panels and cluster shared ingredient anchors. Figure 3 exposes every model rather than reducing replication to a single coefficient.

4.3

Which pairwise conclusions survive correction?

The score is continuous, and each model pair is compared on the same 534 tasks rather than on a small number of arena votes. Figure 4 visualizes the multiplicity-controlled results. Grey cells are scientifically useful: they mark pairs for which the shared task evidence does not support a directional claim at familywise α = .05. This is different from declaring the two models equal.

4.4

6

Rank uncertainty is visible, not hidden

Bootstrap rank intervals expose how often small score differences change order under plausible resampling of ingredient anchors. Figure 6 separates stable regions of the table from visually tempting but statistically unsupported micro-ranks. This is stricter than sorting point estimates and printing an ordinal list without uncertainty.

4.6

Reproducibility

The release is organized around content-addressed inputs and append-only outputs. It includes: • both exact content-addressed task sets (534 scheduled tasks and 534 unique anchors), with prompts, candidates, all 56 portfolio scores, oracle provenance, task strata, and task-set digests; • the 27-model manifest, exact provider route per model, fallback policy, supported parameters, endpoint contract, and route digest; • all response records used to construct the 14418-cell ranked matrix, including raw text, completion state, usage, latency, parser output, task score, and digest; • all content-addressed provider-attempt events referenced by those responses, including normalized and native finish reasons used to distinguish refusals from ordinary completions; • the frozen joint analysis plan, anchor-cluster contract, seeds, 50,000 bootstrap and 100,000 sign-flip counts, all 351 pairwise rows, cross-panel diagnostics, and the final statistical release; and • a verifier and asset builder that reconstruct the leaderboard CSV, pairwise CSV, LaTeX tables, and vector figures from local files without provider access. Responses are written once under a model slot and cell identifier. Existing valid cells are skipped on resume; conflicting artifacts fail closed. Provider fallback is disabled, so the route shown in the manifest is the route measured. The task maps and analysis plan were frozen before the clean primary lineage; transport-calibration responses are retained separately and never enter the leaderboard.

Aggregate scores conceal different culinary profiles

Family-level results in Figure 5 show that similar aggregate scores can arise from different strengths. Constraint tasks penalize infeasible portfolios directly, while pairing and substitution reward graded semantic structure. The three columns are related but not interchangeable.

4.5

From benchmark to training signal

Real prompts and model decisions

Table 4 reports one deterministically selected commoncore task from each family with responses from the highestand lowest-point-estimate endpoints. Selection follows a fixed hash rule rather than task score. The public case-study JSON contains the complete prompt, all eight choices, the optimal portfolio, both raw model responses, selected ingredient identities, and exact scores. The examples illustrate why continuous scoring is preferable to exact match: two non-optimal portfolios can differ substantially in culinary utility.

4

27 frontier models on the identical 534-task core Grok 4.6

65.1 G1

Gemini 3.1 Pro

65.0 G1 64.2 G1

GPT-5.6 Sol Pro Muse Spark 1.2

63.8 G1

GPT-5.6 Terra Pro

63.7 G1

Claude Fable 5

63.4 G1

GPT-5.6 Luna Pro

62.6 G1

Claude Opus 5

62.5 G1

Qwen3.8 A95B

62.1 G1

Kimi K3

62.1 G1

Gemini 3.6 Flash

62.0 G1

DeepSeek V4 Pro 0813

62.0 G1

Qwen 3.8 Max

61.5 G1

Tencent HY 3

61.5 G1

MiniMax M3

60.9 G1

GLM 5.3

60.6 G1 59.9 G2

Muse Glimmer 30B Seed 2.1 Turbo

59.7 G2

Inkling

59.6 G2

Claude Sonnet 5

59.5 G2 58.5 G2

GLM 5.2

57.4 G2

Nemotron 3.5 Lightning

56.7 G2

Command A DeepSeek V4 Flash

55.4 G2

Mistral Large 3

55.4 G2 53.7 G3

Llama 4 Maverick 47.9 G3

Command R+

30

35

40

45

50

55

Mean exact chance 60

65

70

FlavourBench Score (0–100)

Figure 2: FlavourBench Score with simultaneous 95% max-t intervals. The letter-free group numbers are inferential tiers: a point rank is shown for navigation, while models inside an unresolved group should not be read as a statistically established ordering.

7

Related work

rower and complementary: common-task frontier model measurement against a versioned culinary decision environment, with exhaustive partial-credit maps and paired inference.

General-purpose evaluation spans broad suites such as HELM [10], difficult knowledge tests such as MMLU-Pro and GPQA [16, 14], live contamination- resistant sets [17, 7], and human preference systems such as Chatbot Arena [2]. Model judges increase scale but introduce their own bias and calibration problems [20]. FlavourBench instead takes the executable- environment route used by software and agent benchmarks: SWE-bench runs repository tests [8], BFCL evaluates function calls [11], and τ -bench checks interaction outcomes against database state [19]. Culinary datasets and benchmarks cover ingredient networks, recipe understanding, planning, nutrition, and food knowledge [1, 5, 15, 3, 18, 6, 4, 9]. Our focus is nar-

8

Limitations

FlavourBench Score measures agreement with one published Epicure release, not universal human taste. Within-task min–max normalization makes the 0–100 scale comparable while discarding absolute raw-utility magnitudes. The tasks evaluate constrained ingredient selection rather than full recipe generation, sensory execution, safety advice, or long-horizon kitchen planning. The common-core requirement removes differential miss5

Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Model

FB Score

Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+

simultaneous 95% CI

rank 95% CI

group

[61.0, 69.2] [60.8, 69.1] [60.1, 68.4] [59.6, 67.9] [59.5, 67.8] [59.2, 67.5] [58.5, 66.8] [58.4, 66.6] [57.9, 66.2] [57.9, 66.2] [57.7, 66.2] [57.8, 66.1] [57.4, 65.6] [57.3, 65.7] [56.8, 65.1] [56.4, 64.8] [55.9, 63.9] [55.4, 64.0] [55.4, 63.8] [55.4, 63.7] [54.2, 62.7] [53.2, 61.6] [52.5, 60.9] [51.1, 59.7] [51.4, 59.5] [49.6, 57.7] [43.7, 52.0]

[1, 5] [1, 6] [1, 8] [1, 10] [1, 11] [2, 13] [4, 14] [3, 15] [4, 17] [5, 16] [4, 17] [5, 17] [7, 18] [6, 19] [8, 20] [8, 20] [12, 21] [12, 22] [12, 22] [13, 22] [16, 23] [19, 25] [20, 25] [22, 26] [22, 26] [24, 26] [27, 27]

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3

65.1 65.0 64.2 63.8 63.7 63.4 62.6 62.5 62.1 62.1 62.0 62.0 61.5 61.5 60.9 60.6 59.9 59.7 59.6 59.5 58.5 57.4 56.7 55.4 55.4 53.7 47.9

Table 3: The complete automated leaderboard. Every score and every pairwise comparison uses the identical 534-task core. Rank intervals come from the anchor-cluster bootstrap; groups summarize the Holm-controlled pairwise graph. Family

Prompt

Higher-ranked selection

Substitution Select three alternatives to ‘boursin cheese‘ caciocavallo, fromage blanc, grana padano that best preserve its dairy role, regional (100) context, and mutual portfolio coherence. Pairing Select the three-ingredient bundle with the parsnip, cipollini onion, porcini mushroom strongest learned pairing to ‘sweetbread‘ (92) and the best internal coherence. Constraints For a dish centered on ‘potato‘, select three parsley, black pepper, thyme (100) vegetarian complements with NOVA processing level at most 2; among valid portfolios, maximize learned pairing and coherence.

Lower-ranked selection fromage blanc, quail egg, goose egg (15)

parsnip, pickled onion, peppadew pepper (0) cheese, bay leaf, parsley (0)

Table 4: Three common-core response examples. Parentheses give the fixed task score. The complete prompts and unabridged responses are released in machine-readable form. ingness from ranking, but it also restricts the estimand to tasks completed by every endpoint; the released supplemental tracks retain the rest. Results bind particular model routes and collection dates. The dense reward surface supports learning experiments, but this paper does not test whether optimizing Epicure reward transfers to human cooking outcomes.

9

reconstructable from public artifacts.

References [1] Yong-Yeol Ahn, Sebastian E. Ahnert, James P. Bagrow, and Albert-László Barabási. Flavor Network and the Principles of Food Pairing. Scientific Reports, 1:196, 2011.

Conclusion

[2] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8359– 8388. PMLR, 2024.

FlavourBench makes culinary reasoning measurable without asking one language model to grade another. A versioned runtime enumerates every candidate decision, a 27-model complete panel produces one continuous score, and paired uncertainty distinguishes resolved differences from apparent rank order. The result is an automated leaderboard that can be rerun, audited, and used as a dense training environment. Under this executable culinary ground truth and the released 534-task, 534-cluster distribution, every score, interval, contrast, and figure is

[3] Donghee Choi, Mogan Gim, Donghyeon Park, Mujeen Sung, Hyunjae Kim, Jaewoo Kang, and Jihun 6

Teal: row higher Red: row lower Grey: unresolved at familywise α=.05

Independent task panels test ranking stability

Holm-controlled paired comparisons

Pearson r=0.89 Spearman ρ=0.80

Grok 4.6

Grok 4.6

Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2

Gemini 3.1 Pro

GPT-5.6 Terra Pro

Muse Spark 1.2 GPT-5.6 Terra Pro GPT-5.6 Sol Pro DeepSeek V4 Pro 0813

65

Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5

Qwen 3.8 GPT-5.6 Max Luna Pro Gemini GLM 5.3 Tencent HY3.6 3 Flash Muse Glimmer 30B Qwen3.8 A95B

Qwen3.8 A95B Kimi K3

Seed 2.1 Turbo

Gemini 3.6 Flash

Inkling

60

DeepSeek V4 Pro 0813

Nemotron 3.5 Lightning

Row model

Panel 2 FlavourBench Score

Opus 5 Fable 5 KimiClaude K3 MiniMax M3 Claude

GLM 5.2 Claude Sonnet 5 Command A Mistral Large 3 DeepSeek V4 Flash Llama 4 Maverick

Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo

55

Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash

50

Mistral Large 3 Llama 4 Maverick

Command R+

Command R+

Llama 4 Maverick

Command A

Mistral Large 3

DeepSeek V4 Flash

Nemotron 3.5 Lightning

Inkling

GLM 5.2

Seed 2.1 Turbo

Claude Sonnet 5

GLM 5.3

MiniMax M3

Muse Glimmer 30B

Tencent HY 3

Qwen 3.8 Max

Kimi K3

Gemini 3.6 Flash

DeepSeek V4 Pro 0813

Claude Opus 5

Qwen3.8 A95B

Claude Fable 5

GPT-5.6 Luna Pro

65

Muse Spark 1.2

60

GPT-5.6 Terra Pro

Grok 4.6

55

Panel 1 FlavourBench Score

Gemini 3.1 Pro

50

GPT-5.6 Sol Pro

Command R+

Column model

Figure 3: Panel-1 versus panel-2 FlavourBench Score for every endpoint. The diagonal marks exact agreement. Each point aggregates 89 tasks per family; the joint analysis uses 534 unique anchor clusters.

Figure 4: All paired model contrasts after Holm correction. Teal means the row model scores higher; red means lower; grey is unresolved. uation of Large Language Models for Code. arXiv preprint arXiv:2403.07974, 2024.

Choi. CookingSense: A Culinary Knowledgebase with Multidisciplinary Assertions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3983–3996, Torino, Italia, 2024. ELRA and ICCL.

[8] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In The Twelfth International Conference on Learning Representations, 2024.

[4] Tome Eftimov, Ana Gjorgjevikj, Matej Martinc, Gjorgjina Cenikj, Saso Dzeroski, and Barbara Koroušić Seljak. FoodBench-QA: Overview of the Shared Task on Grounded Food and Nutrition Question Answering. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) at LREC 2026, 2026.

[9] Song Jin, Juntian Zhang, Xun Zhang, Zeying Tian, Fei Jiang, Guojun Yin, Wei Lin, Yong Liu, and Rui Yan. DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35289–35310. Association for Computational Linguistics, 2026.

[5] Neelansh Garg, Apuroop Sethupathy, Rudraksh Tuwani, et al. FlavorDB: A Database of Flavor Molecules. Nucleic Acids Research, 46(D1):D1210– D1216, 2018.

[10] Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023.

[6] Andong Hua, Mehak Preet Dhaliwal, Laya Pullela, Ryan Burke, and Yao Qin. NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions. In The Thirteenth International Conference on Learning Representations, 2025.

[11] Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 48371–48392. PMLR, 2025.

[7] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Eval7

Rank uncertainty remains visible behind point estimates

Performance decomposes by culinary decision family Grok 4.6

66

71

58

Gemini 3.1 Pro

69

72

54

GPT-5.6 Sol Pro

66

72

54

Muse Spark 1.2

66

70

55

GPT-5.6 Terra Pro

66

72

53

Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro

100

Claude Fable 5

68

71

51

GPT-5.6 Luna Pro

65

70

54

Claude Opus 5

67

70

50

Qwen3.8 A95B

66

68

52

Kimi K3

65

74

47

Kimi K3

Gemini 3.6 Flash

68

66

52

Gemini 3.6 Flash

DeepSeek V4 Pro 0813

66

70

50

Qwen 3.8 Max

67

68

49

Tencent HY 3

66

68

51

MiniMax M3

65

70

48

64

70

48

Muse Glimmer 30B

65

67

48

Seed 2.1 Turbo

65

66

48

Inkling

64

66

50

Claude Sonnet 5

65

69

44

GLM 5.2

64

67

44

Nemotron 3.5 Lightning

65

64

43

Command A

65

64

42

DeepSeek V4 Flash

63

63

40

Claude Opus 5

80

60

40

Qwen3.8 A95B

Mean Epicure score

GLM 5.3

Claude Fable 5 GPT-5.6 Luna Pro

DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling

20

Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A

0

DeepSeek V4 Flash Mistral Large 3

Mistral Large 3

60

66

40

Llama 4 Maverick

Llama 4 Maverick

59

61

40

Command R+

Command R+

57

55

32

Substitution

Pairing

Constraints

27

25

23

21

19

17

15

13

11

9

7

5

3

1

Rank (lower is better)

Figure 5: Mean score by family on the common core. Each family contributes 178 tasks per model. Values share the 0–100 scale but retain family-specific decision semantics.

Figure 6: Point rank and 95% bootstrap rank interval for every endpoint. Narrow intervals imply a stable location; overlapping intervals mark unresolved local order.

[12] Jakub Radzikowski and Josef Chen. Epicure: Multidimensional Flavor Structure in Food Ingredient Embeddings. arXiv preprint arXiv:2604.22776, 2026.

Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A Challenging, ContaminationLimited LLM Benchmark. In The Thirteenth International Conference on Learning Representations, 2025.

[13] Jakub Radzikowski and Josef Chen. Epicure: Navigating the Emergent Geometry of Food Ingredient Embeddings. arXiv preprint arXiv:2605.22391, 2026. [14] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling, 2024.

[18] Zirui Wu, Xiao Liu, Jiayi Li, Lingpeng Kong, and Yansong Feng. Recipe2Plan: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 4279–4301, Suzhou, China, 2025. Association for Computational Linguistics.

[15] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3020–3028, 2017.

[19] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In The Thirteenth International Conference on Learning Representations, 2025.

[16] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In Advances in Neural Information Processing Systems, volume 37, 2024.

[20] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLMas-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36, 2023.

[17] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti

8

A

Exact execution routes Model

Backend

Panel 1 route

Panel 2 route

Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+

openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter zai_coding_direct openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter openrouter

xai/zdr google-ai-studio openai/flex meta openai/flex google-vertex/global openai/flex azure/us alibaba morph google-vertex/us gmicloud/fp8 alibaba gmicloud/bf16 venice/fp8 zai-coding-plan-direct fireworks seed/fp8 deepinfra/fp8 amazon-bedrock/claude-on-aws coreweave/fp4 venice/fp4 cohere deepinfra/fp4 mistral digitalocean cohere

xai/zdr google-ai-studio openai/flex meta openai/flex google-vertex/global openai azure/us alibaba morph google-vertex/us gmicloud/fp8 alibaba gmicloud/bf16 venice/fp8 zai-coding-plan-direct fireworks seed/fp8 deepinfra/fp8 amazon-bedrock/claude-on-aws coreweave/fp4 venice/fp4 cohere deepinfra/fp8 mistral digitalocean cohere

Table 5: Frozen execution routes by panel. Automatic fallback is disabled; route differences indicate complete, score-blind model-block replacements rather than cell pooling.

B

Per-family scores Model

Substitution

Pairing

Constraints

66.1 69.0 66.2 65.9 66.1 68.2 64.5 67.2 66.0 65.5 68.1 66.4 66.6 66.0 65.4 63.7 65.1 65.3 63.6 65.3 63.6 65.3 64.7 63.1 60.4 59.4 56.7

70.9 71.6 72.5 70.3 72.1 70.6 69.7 69.9 67.8 73.7 66.2 69.5 68.5 67.9 69.9 69.6 66.8 66.2 65.8 69.2 67.2 64.3 63.9 63.2 65.8 61.4 54.9

58.2 54.2 54.1 55.0 52.8 51.3 53.7 50.4 52.5 47.0 51.6 49.9 49.5 50.5 47.6 48.4 47.8 47.6 49.5 44.1 44.5 42.5 41.6 40.1 40.0 40.2 32.0

Grok 4.6 Gemini 3.1 Pro GPT-5.6 Sol Pro Muse Spark 1.2 GPT-5.6 Terra Pro Claude Fable 5 GPT-5.6 Luna Pro Claude Opus 5 Qwen3.8 A95B Kimi K3 Gemini 3.6 Flash DeepSeek V4 Pro 0813 Qwen 3.8 Max Tencent HY 3 MiniMax M3 GLM 5.3 Muse Glimmer 30B Seed 2.1 Turbo Inkling Claude Sonnet 5 GLM 5.2 Nemotron 3.5 Lightning Command A DeepSeek V4 Flash Mistral Large 3 Llama 4 Maverick Command R+

Table 6: Mean score in each 178-task family. Every entry uses the same tasks for every model; the equal-family mean is the FlavourBench Score.

9

C

Protocol constants Quantity Models Collection panels Primary tasks per model Unique anchor clusters Tasks per family Candidate ingredients Selected ingredients Scored portfolios per task Complete response cells Bootstrap replicates Sign-flip replicates Pairwise hypotheses Shared cells per pair Common-core rule

Frozen value 27 2 534 534 178 8 3 56 14418 50000 100000 351 534 Status/parser only; score-blind

Table 7: Core design and inference constants fixed before clean primary execution.

10

Related documents

• FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training#924642
• FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training#920408
• MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs#821455
• Neural Language Models’ sensitivity to long-distance dependencies and their island constraints is not universal: a comparison between English and Dutch humans and models#331597
• SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia#988004