MongoBench: Do LLMs Know What a MongoDB Query Will Do?

arXiv:2611.09065v1 [cs.SE] 6 Nov 2026

MongoBench: Do LLMs Know What a MongoDB Query Will Do? Qibai Chen

Haoting Qiu

Yezhou Cheng

Mengxiao Qian

Independent Researcher United States qibaic@alumni.cmu.edu

Independent Researcher United States qiu.haot@northeastern.edu

University of Wisconsin–Madison United States cheng222@wisc.edu

Independent Researcher United States qmxresearch@gmail.com

Abstract—LLM agents increasingly operate databases on their own: they generate queries and run them against live data. Two checks are commonly used to judge such an agent—whether it produces a valid query, and whether it reaches the right final answer—but neither tells us whether the agent knows what its query will actually do before it runs. We argue that this consequence prediction ability must be evaluated in its own right, and we build MongoBench to measure it: given a collection of documents and a MongoDB find() filter, a model must return the exact set of matching _ids, with ground truth from a real mongod server. The 300 items each compose three or four documented matching pitfalls and are proposed symmetrically by three vendors’ models, so results stratify by item source. Evaluating four models—frontier systems from three vendors and one open-weight reasoner—accuracy ranges from 70.1% to 92.1%, a 22-point spread. Each proposer mispredicts 10–31% of the items it wrote itself, so producing a query does not imply predicting its effect. A failure-mode analysis shows errors are better predicted by which semantic pitfall a query exercises— range comparisons and null—than by surface measures such as document count or the number of composed rules. Prepending a verified rule sheet helps unevenly; executing the original filter against the engine yields reliable match sets, but delegation must be checked: on 28% of items (averaged over three runs) one model’s executed filters never passed a canonical check against the intended query, and it answered about 22 of them wrong, while the other three passed on every item. We release the benchmark, the mongod oracle, and the evaluation code with all prompts at https://github.com/research-repo-bai/MongoBench. Index Terms—large language models, LLM agents, MongoDB, NoSQL, query semantics, benchmark, program comprehension

I. I NTRODUCTION LLM coding agents now resolve real software-engineering tasks with little human oversight [1], and in doing so increasingly read, write, and execute database queries on their own. To judge whether such an agent is trustworthy, we usually look at two things: can it produce a syntactically valid query, and does it reach a correct final answer (for instance, after running the query and reading the result). Both miss a capability that sits between them and is safety-critical in its own right: before an agent runs a query, does it know which objects the query will affect? An agent that emits a valid delete filter need not know which documents that filter removes; an agent that can call a database tool need not pass the tool the query it meant. Predicting the consequence of an action, and preserving that action’s meaning when handing it to an executor, is what lets an agent estimate a blast radius, explain an empty result, or

Accepted at CAIT 2026. Author’s accepted manuscript.

veto a wrong query before damage is done. This matters most when running the query first is not an option: a destructive or irreversible write, a code review with only the schema or a sample in context, or any step that must judge a query before it touches live data. We make this consequence prediction ability the object of study. For document stores it has a clean, machine-checkable form: given documents D and a find() filter Q, which _ids does Q match under the engine’s rules? Research on LLMs and document-store queries has instead focused on generation—translating natural language to a query string [2], [3]—which does not test whether the model understands the matching semantics of what it writes. Those semantics are full of documented pitfalls: a scalar condition also matches arrays containing the value; multiple conditions on an array may be met by different elements unless $elemMatch is used; null also matches missing fields; a range comparison against null matches nothing; and equality with an embedded document is field-order sensitive. A model that writes a valid filter while mispredicting which documents it affects can, if that filter drives a delete, cause unintended and silent changes to the data. We present MongoBench and study four questions: RQ1 (capability) how accurately do LLMs predict match sets? RQ2 (failure structure) what makes a query hard—its semantics, or its surface complexity? RQ3 (source bias) when models both propose and answer items, is there an own-item advantage? RQ4 (mitigation) how much do rule injection, per-document decomposition, and engine delegation help, and is delegation safe? Our contributions are: (C1) MongoBench, 300 items that frame consequence prediction as an exact, engine-checkable task over composed MongoDB footguns (Sec. III); (C2) an evaluation of four current models including an open-weight reasoning model, with repeated runs and paired tests, plus a failure-mode analysis showing errors are better predicted by semantic pitfall type than by document count or the number of composed rules (Sec. V); (C3) evidence that consequence prediction is distinct from both producing a query (models mispredict 10–31% of items they wrote) and obtaining a final answer (a model that alters query semantics while delegating) (Secs. V, VI); (C4) a mitigation study yielding concrete guidance for agent builders (Sec. VI).

II. R ELATED W ORK Query understanding. Rahaman et al. [4] probe LLM understanding of SQL via tasks such as explanation and equivalence; DBLifeBench [5] spans the relational database lifecycle. MongoBench targets the document model, whose matching rules differ sharply from SQL (implicit array traversal, null/missing conflation, order-sensitive embedded-document equality), and asks for an exact executable answer rather than a description. Query generation. Text-to-SQL benchmarks such as Spider [6] and BIRD [7] score generated queries by execution; SM3-Text-to-Query [2] and Text-to-NoSQL [3] extend generation to MongoDB. A model can pass these while misjudging what its query matches—precisely the gap MongoBench measures. Execution reasoning. CRUXEval [8] asks models to predict the output of short Python functions, and HumanEval [9] popularized execution-based checking of generated code. MongoBench applies output prediction to a declarative query language whose semantics are fixed by an engine rather than by a general-purpose interpreter, so there is a single authoritative answer and no hidden program state. This difference matters for agents: an imperative program’s effect is usually read off by running it, whereas a declarative filter is often reasoned about before execution—exactly the setting where a mispredicted match set can turn into unintended data loss. General constraint- and instruction-following benchmarks [10]– [12] evaluate whether a model obeys stated requirements, not whether it predicts the effect of an action it emits. To our knowledge MongoBench is the first benchmark of LLM comprehension of document-store query-matching semantics.

act field set in the exact order; (8) dot-notation into arrays— a.b traverses an array of sub-documents, and a.0 indexes positionally; (9) nested arrays—implicit matching descends only one array level; and (10) $size/$all—exact length and containment, including $all with $elemMatch. Fig. 2 shows two in isolation. Composition. Every item composes three or four of these families at once: a numbered specification names only the neutral topics involved (e.g. “negation when the field is an array”), and the proposer must design an item whose correct answer depends on all of them. Proposers never see rules, answers, or other items. Collections hold 7.9 documents on average. We exclude single-rule items to target composition, which is the open question. Proposers and symmetry. The 300 items are proposed in equal shares (100 each) by opus-4-8, gpt-5.5, and gemini-3.1-pro—the same three models evaluated—so item source is an explicit factor and any own-item advantage is measurable (RQ3). Ground truth and validity. Gold match sets are computed by executing each filter on a real mongod server (v8.2.9) [13], independent of the proposer, following execution-based evaluation practice [7], [9]. All gold sets were recomputed on mongod 7.0.34 and agree exactly, ruling out version-specific quirks. We use mongod directly rather than an in-process emulator. Items are kept only on model-independent validity criteria (parses as strict JSON, executes, in-range document count, not a duplicate). Because a composed filter containing, e.g., a $gt:null conjunct trivially matches nothing, empty gold sets are capped at 20% per proposer; the resulting alwaysempty baseline scores 14.0%.

III. T HE M ONGO B ENCH B ENCHMARK

IV. E XPERIMENTAL S ETUP

Fig. 1 gives an overview. Task and metrics. Each item is a collection D (JSON documents with integer _ids) and a find() filter Q; the model returns the _id set that Q matches. The primary metric is exact-set accuracy: correct only if the predicted set equals the gold set—a preview that misses one document is wrong. We also report per-document accuracy and Jaccard similarity to separate matching judgment from set construction. Footgun families. Items draw on ten documented behaviors, each a rule where MongoDB diverges from a naïve reading: (1) array-implicit matching—a scalar condition {f:v} also matches a document whose f is an array containing v; (2) implicit-AND vs. $elemMatch—two conditions on one array field may be satisfied by two different elements unless $elemMatch forces a single element; (3) null/missing— {f:null} matches documents where f is null, missing, or an array containing null; (4) negation over arrays—$ne/$nin exclude a document if any array element matches; (5) type bracketing—comparisons never cross BSON types, so {f:{$gt:5}} never matches a string; (6) comparison vs. null—$gt/$lt against null match nothing, while $gte/$lte match null and missing; (7) embeddeddocument equality—a whole-document value requires the ex-

Models. We evaluate frontier models from three vendors—claude-opus-4-8 [14], gpt-5.5 [15], and gemini-3.1-pro-preview [16] (hereafter gemini-3.1-pro)—and one open-weight reasoning model accessed through DeepSeek’s deepseek-reasoner API endpoint [17] (we name it by that API identifier). All were accessed via vendor APIs in late 2026. All three run at medium reasoning effort; deepseek-reasoner reasons by default (its effort is not configurable), which we note when comparing. We report model versions and settings rather than equalize them. Protocol. The baseline prompt (E0) shows documents and filter as JSON and asks for a final line ANSWER: [ids]; no model produced an unparseable answer in any run. The main experiment is repeated three times per model; we report mean ± std, and for paired model comparisons take each item’s majority outcome and apply McNemar’s exact test [18] with Holm correction [19]. Enhancement conditions (Sec. VI) are each run three times. V. R ESULTS RQ1: capability. Table I reports baseline accuracy. gpt-5.5 leads at 92.1%, opus-4-8 reaches 85.1%, and

1. Propose

2. Ground truth

3 frontier models: claude-opus-4-8 gpt-5.5 gemini-3.1-pro each item composes 3– 4 of 10 footgun families; neutral topics only

real mongod 8.2 runs each filter re-verified on 7.0 empty-gold capped 20%

propose

3. Evaluate (E0) gold

→ 300 items gold decoupled from proposer

predict the matching _id set (exact-set) 4 solvers, incl. deepseek-reasoner (open-weight) 3 runs each

E0

4. Enhance

5. Analyze

E1 verified rule sheet E2 per-document E3 engine tool mongo_find 3 runs each

accuracy & families source bias (3×4) failure modes error direction (over/under-match)

Fig. 1. The MongoBench pipeline. Three vendors’ frontier models propose items that compose three or four documented MongoDB matching pitfalls; a real mongod server computes ground truth, decoupled from the proposer; four models (including one open-weight) are evaluated; and three interventions are measured.

(b) embedded-document order. D: {_id:1,a:{b:1,c:2}}, {_id:2,a:{c:2,b:1}}, {_id:3,a:{b:1,c:2,d:0}} Q: {a:{b:1,c:2}} Gold: {1} Whole-document equality is field-order and field-set sensitive, so 2 and 3 do not match; treating it as a subset or order-insensitive predicts {1, 2, 3}. Fig. 2. Two footguns (verified on mongod). Both read naturally but match counter-intuitively; MongoBench items compose several such rules. TABLE I BASELINE ACCURACY (E0, 3- RUN MEAN ± STD , n=300). ACCURACY IS THE 3- RUN MEAN ; PER - DOCUMENT ACCURACY AND JACCARD ARE FROM ONE RUN . TABLES ABBREVIATE D E E P S E E K - R E A S O N E R AS D E E P S E E K .

Model gpt-5.5 opus-4-8 deepseek gemini-3.1-pro Always-empty

comparison_null (89)

82

96

43

44

dot_array (94)

82

84

72

68

type_bracketing (112)

86

96

53

79

null_missing (117)

85

95

74

75

embedded_doc_eq (122)

88

94

70

80

70

array_implicit (102)

90

91

73

78

60

size_all (85)

84

94

75

80

nested_array (114)

84

97

73

79

implicit_and_elemMatch (128)

90

95

81

82

neg_array (87)

94

98

83

89

100

90

80

accuracy (%)

(a) range vs. null. D: {_id:1,f:5}, {_id:2,f:null}, {_id:3} Q: {f:{$gt:null}} Gold: ∅ A model reading $gt:null as “f is not null” predicts {1}.

50

40

-4-8

pus e-o

ud cla

30

r -5.5 -pro asone gpt ni-3.1 i k-re gem epsee de

Accuracy (%)

per-doc / Jaccard

92.1±0.5 85.1±0.7 72.3±0.9 70.1±0.7

98.7 / 96.4 96.8 / 92.8 93.3 / 82.5 92.2 / 80.4

Fig. 3. Accuracy (%) by footgun family (majority of 3 runs); an item counts toward every family it composes. Families sorted by mean accuracy.

14.0

–

(gemini-3.1-pro scores 70% and 70%), consistent with the failure analysis below: difficulty comes from the kind of rule, not the number stacked. RQ2: what makes a query hard. Fig. 3 breaks accuracy down by family. Comparison against null is hardest— gemini-3.1-pro 43% and deepseek-reasoner 44% vs. gpt-5.5 96%—followed by type bracketing: the rules that contradict the “SQL-like” intuition that a comparison with null means “is not null” and that cross-type values are ordered. To ask what drives difficulty, we label each item with structural features and relate them to the number of models that fail it. The strongest signal is an empty gold set (93% of such items are missed by at least one model vs. 37%, Fisher 3.0×10−12 ); being a property of the answer, we treat it as descriptive and exclude it (and result-set size) from the predictor below. Among pre-execution features, a range comparison (50% vs. 19%) and a null literal are most associated with failure. The surface measures we tested are not: accuracy does not fall as the collection grows (Pearson -0.03; Table VI shows no monotonic trend over the 5–12 documents per item) and composing four families is no harder than three (standardized coefficient -0.12). A logistic model

gemini-3.1-pro and deepseek-reasoner trail at 70.1% and 72.3%—a 22-point spread among current models on a task with a 14.0% trivial baseline. Reasoning budgets are not matched across vendors (deepseek-reasoner’s effort is not configurable), so this spread reflects capability and effort together, not capability alone (Sec. VII); the peritem and safety findings below do not depend on the ranking. Run-to-run std is below one point. Per-document accuracy is far higher than exact-set accuracy for every model (e.g. gemini-3.1-pro 92.2% per-doc vs. 70.7% exact-set), so a weak model typically misjudges only one or two documents per item—in production that single misjudged document is the silent error. All pairwise accuracy differences are significant (McNemar’s exact test, Holm-corrected p < 0.05) except deepseek-reasoner vs. gemini-3.1-pro (p=0.06). Only 4 items are missed by all four models, so these are model-dependent blind spots rather than universally impossible items. Composition depth barely matters: items joining four families are no harder than three for any model

TABLE II S TRONGEST PREDICTORS THAT AN ITEM IS FAILED BY AT LEAST ONE MODEL ( STANDARDIZED LOGISTIC - REGRESSION COEFFICIENTS ; + = HARDER ).

Query feature

std. coef.

range comparison ($gt/$lt) negation ($ne/$nin) $elemMatch null literal dot-notation path $exists

+0.73 −0.53 −0.48 +0.46 +0.46 −0.18

effect

Proposer (n=100)

harder easier easier harder harder easier

opus-4-8 gpt-5.5 gemini-3.1-pro

TABLE III D IRECTION OF ERRORS (E0, AMONG WRONG ITEMS ). OVER - MATCH = PREDICTION IS A STRICT SUPERSET OF GOLD ; UNDER - MATCH = STRICT SUBSET.

Model opus-4-8 gpt-5.5 gemini-3.1-pro deepseek

TABLE IV ACCURACY (%) BY ITEM SOURCE ( ROWS ) AND SOLVER ( COLUMNS ), MAJORITY OF 3 RUNS . U NDERLINED : A MODEL ANSWERING ITS OWN ITEMS . O NLY THESE THREE MODELS PROPOSE ITEMS .

over-match

under-match

incomp.

33% 82% 81% 55%

63% 14% 17% 39%

5% 5% 2% 6%

opus-4-8 gpt-5.5 gemini-3.1-pro deepseek 89 91 78

96 90 96

72 69 69

83 66 78

TABLE V E NHANCEMENT ACCURACY (%), 3- RUN MEAN . E1: PREPEND A VERIFIED 10- RULE SHEET. E2: FORCE PER - DOCUMENT JUDGMENT. E3: LET THE MODEL RUN THE FILTER VIA A M O N G O _ F I N D TOOL .

Model

E0

E1 rules

E2 per-doc

E3 engine

gpt-5.5 opus-4-8 deepseek gemini-3.1-pro

92.1 85.1 72.3 70.1

91.2 93.2 93.3 95.3

92.0 89.3 74.1 72.2

100.0 100.0 99.7 92.1

though the sources differ in difficulty (see Sec. VII). VI. M ITIGATION AND THE D ELEGATION G AP

over the pre-execution features alone predicts failure at 62% held-out accuracy (leave-one-out cross-validation; 67% insample) against a 55% base rate; Table II lists its strongest coefficients. Within the features we measured, difficulty tracks which semantic pitfall a query exercises more than its document count or the number of composed rules—evidence of genuine misunderstanding, though these remain associations (items compose several families) rather than isolated causes. (Queries with $ne/$nin are, interestingly, easier: their larger, more intuitive result sets are harder to get wrong.) Error direction. When a model is wrong, does it overor under-predict the match set? Table III classifies each error as an over-match (predicted a strict superset of gold), under-match (strict subset), or incomparable set. Errors lean toward over-matching overall (63% vs. 33%), but the direction is model-specific: gpt-5.5 and gemini-3.1-pro overmatch on ∼82% of their errors, whereas opus-4-8 undermatches on 63%. This matters for safety: an over-matching preview overstates a delete’s blast radius (false alarm), while an under-matching one understates it—silently leaving matched documents unaccounted for. Because no model errs consistently in the “safe” direction, an agent cannot assume its mispredictions are conservative; this is a further argument for delegating to the engine (Sec. VI). RQ3: proposing an item is not solving it. Table IV crosses item source with solver. Each proposer fails on items it wrote itself: opus-4-8, gpt-5.5, and gemini-3.1-pro get 11%, 10%, and 31% of their own items wrong, so generating a query that exercises a footgun does not imply predicting its result—producing a query and predicting its consequence are distinct abilities. On every source, the best solver is a model other than the item’s author (gpt-5.5 on opus-4-8’s and gemini-3.1-pro’s items, opus-4-8 on gpt-5.5’s),

We compare three interventions against E0 (Table V), each run three times. Rule injection (E1) prepends a one-line rule per family (Fig. 5), each statement verified against mongod. It helps unevenly: gemini-3.1-pro rises to 95.3% and deepseek-reasoner to 93.3%, while gpt-5.5 is unchanged (91.2% vs. 92.1%; the small drop is not significant, paired p ≈ 0.24). Per family (Fig. 4), the lift is largest where models lacked the rule—comparison vs. null (+29), type bracketing—and smaller for dot-notation into arrays (+8 points). This pattern is consistent with a knowledge gap in the first case and difficulty applying a known rule in the second. Rule injection thus appears to patch missing knowledge more than faulty application, and does not help every model. Decomposition (E2), judging each document before assembling the set, has a small, inconsistent effect (it raises opus-4-8 by about four points, to 89.3%, but leaves the other three within two) and is not a reliable substitute. Engine delegation (E3) gives the model a tool that runs a filter on the collection and returns the matching _ids (up to ten calls). opus-4-8 and gpt-5.5 reach 100% and deepseek-reasoner 99.7%. But delegation is not automatically safe. We canonicalize each executed filter against the intended one, treating order-irrelevant rewrites (reordered top-level conditions or operators) as preserving and orderrelevant ones (an embedded document’s field order, or a changed key) as altering; this check is conservative, as it may flag rewrites that are in fact equivalent. Across all four models and all three runs, only gemini-3.1-pro produced items on which it called the tool yet no executed filter passed this check—28% of items on average (27–30% across runs); the other three passed on every item. On about 22 of those items per run (range 16–29), gemini-3.1-pro’s final answer was also wrong. Fig. 6 illustrates a concrete

E0 baseline

E1 lift +29

comparison_null type_bracketing

+19

array_implicit

+14

nested_array

+14

null_missing

+12

embedded_doc_eq

+11

size_all

+10 +10

implicit_and_elemMatch +8

dot_array

+5

neg_array

0

20

40

60

80

100

accuracy (%), pooled

Fig. 4. Per-family accuracy before (E0) and after (E1) rule injection, pooled over models. Large lift is consistent with a knowledge gap the rule closes; small lift, with difficulty applying a rule the model has now been given. Array-implicit: {f:v} matches f==v OR an array elem ==v Implicit-AND: {f:{$gt:a,$lt:b}} diff elems ok; $elemMatch=one null/missing: {f:null} matches null, missing, or [..null..] Compare vs null: $gt/$lt:null match NOTHING; $gte/$lte:null match null/missing Type bracket: comparisons stay within one BSON type $ne/$nin: exclude doc if ANY array elem matches Embedded-doc eq: requires exact field set AND order Dot notation: a.b into array elems; a.0 positional; does not reach arrays nested in arrays Nested arrays: implicit match descends one level only $size/$all: exact length; $all=contains all (impl.)

Fig. 5. The E1 rule sheet (one line per family, abbreviated; each rule was verified against mongod before use, full text released). This paragraph is prepended to the E0 prompt.

alteration that changes the matching result: asked to run {..., "v.0":{$exists:true}}, gemini-3.1-pro quoted the positional-index key as the literal string "’v.0’", so the executed filter tested a field literally named v.0 instead of the first array element; eight tool calls later it still returned the empty set instead of the correct {1,2}. The engine answered faithfully—but to a different query than intended. Adversaries can exploit this gap to make agents issue backend queries that slip past access controls [20]; here it opens benignly, with no adversary. Guidance for agent builders. (1) For previewing or validating a query, execute the original filter against the engine and return its result verbatim—do not let the model reconcile the engine’s answer with its own prediction. (2) If in-context reasoning is unavoidable, include a verified rule sheet for the declarative footguns, but expect residual errors on executionstyle rules such as dot-notation. (3) Do not rely on perdocument decomposition as a safeguard. VII. T HREATS TO VALIDITY Model coverage and budget. We evaluate four models at their default or medium reasoning settings; rea-

intended: {a:{b:1,c:2}, v:{$gt:3,$lt:7}, v.0:{$exists:true}} executed: {a:{b:1,c:2}, v:{$gt:3,$lt:7}, ’v.0’:{$exists:true}} gold {1,2} returned {} (8 tool calls, never corrected)

Fig. 6. Engine delegation can still fail: gemini-3.1-pro alters the query’s meaning (quoting a positional-index key) before executing it, so a faithful engine returns the wrong set. TABLE VI ACCURACY (%) BY COLLECTION SIZE ( MAJORITY OF 3 RUNS ). ACCURACY SHOWS NO MONOTONIC DECLINE AS THE COLLECTION GROWS . #docs (n) 5 (34) 6–7 (75) 8 (85) 9–12 (106)

opus-4-8 gpt-5.5 gemini-3.1-pro deepseek 74 81 91 90

91 96 92 95

74 69 73 67

76 80 71 76

soning budgets are not matched across vendors, and deepseek-reasoner’s effort is not configurable, so crossmodel gaps conflate capability and effort. LLM-proposed items and source difficulty. Items are written by the evaluated frontier models, with no human-written control; gold never depends on a proposer, and we report accuracy stratified by source (RQ3). The three sources differ in difficulty (emptygold items number 3, 20, and 19 across proposers), so ownitem effects cannot be fully separated from source difficulty and we do not claim a difficulty-matched null; a matched, human-controlled source × solver analysis is future work. Rule sheet as upper bound. The E1 rules name the same ten footgun families the items are built from, so E1 measures accuracy when the relevant rule is supplied verbatim—an upper bound on in-context help, not what an agent would have unprompted. Saturation. Only four items are missed by all four models, so the hardest stratum is near the ceiling of what these models cannot do; a harder item pool may be needed to separate future models. Scope. We cover find() filters, not projections, updates, or aggregation pipelines; extending the consequence-prediction framing to those is future work. VIII. C ONCLUSION Evaluating an agent that acts on a stateful system should not stop at “can it produce an action” or “can it get an answer”; it must also ask whether the agent knows what its action will do, and whether it preserves that action’s meaning at the point of execution. MongoBench measures the first for MongoDB find() filters and surfaces a concrete instance of the second. All four current models misjudge composed matching footguns—failures tracking semantic pitfall type more than document count or the number of composed rules—rule sheets help unevenly, and even engine delegation can be undone by a model that silently rewrites the query. The same question applies to any setting where what a model emits is itself an action whose effect a deterministic system can adjudicate. A code edit has a test suite; an access-control rule has a policy evaluator; a configuration change has the system it configures.

In each case an agent can generate the artifact and can run the checker, yet, as here, neither establishes that it predicted what the artifact would do, nor that the artifact handed to the checker still meant what the agent intended. Where such a ground-truth system is cheap to call, the engineering answer is to call it and use its verdict verbatim; the open evaluation question is how well agents know, in advance, what their own actions will do. We release MongoBench—the items, the mongod oracle, and the evaluation code with all prompts at https://github.com/research-repo-bai/MongoBench—as a template for building such consequence-prediction benchmarks in other executable domains. ACKNOWLEDGMENT As detailed in Secs. III and V, large language models are both the subjects of this evaluation and the proposers of the composed items, whose answers were computed by an independent mongod oracle and checked for source bias. AI tools were also used to assist with implementation and writing. All methodological choices, analyses, and conclusions are the authors’. R EFERENCES [1] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representations (ICLR), 2024. [2] S. Sivasubramaniam, C. Osei-Akoto, Y. Zhang, K. Stockinger, and J. Fürst, “SM3-Text-to-Query: Synthetic multi-model medical text-toquery benchmark,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. [3] J. Lu, Y. Song, Z. Qin, H. Zhang, C. Zhang, and R. C.-W. Wong, “Bridging the gap: Enabling natural language queries for NoSQL databases through text-to-NoSQL translation,” arXiv preprint arXiv:2502.11201, 2025. [4] A. Rahaman, A. Zheng, M. Milani, F. Chiang, and R. Pottinger, “Evaluating SQL understanding in large language models,” in Proceedings of the 28th International Conference on Extending Database Technology (EDBT), 2025. [5] S. Zheng, D. Shi, Y. Li, X. Yi, L. Wang, and G. de Melo, “Evaluating LLMs in database scenarios: A lifecycle benchmark for assessing their potential in core database tasks,” arXiv preprint arXiv:2608.03794, 2026. [6] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A largescale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. [7] J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. C. C. Chang, F. Huang, R. Cheng, and Y. Li, “Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. [8] A. Gu, B. Rozière, H. J. Leather, A. Solar-Lezama, G. Synnaeve, and S. Wang, “CRUXEval: A benchmark for code reasoning, understanding and execution,” in Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. [9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [10] J. Tso, P. Schmittou, Q. Huynh, and J. Hutchins, “ConstraintBench: Benchmarking LLM constraint reasoning on direct optimization,” arXiv preprint arXiv:2602.22465, 2026.

[11] T. Zhang, C. Zhu, Y. Shen, W. Luo, Y. Zhang, H. Liang, T. Zhang, F. Yang, M. Lin, Y. Qiao, W. Chen, B. Cui, W. Zhang, and Z. Zhou, “CFBench: A comprehensive constraints-following benchmark for LLMs,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. [12] L. Zhang, J. Wang, S. He, C. Zhang, Y. Kang, B. Li, J. Wen, C. Xie, M. Wang, Y. Huang, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, D. Zhang, and Q. Zhang, “DI-BENCH: Benchmarking large language models on dependency inference with testable repositories at scale,” in Findings of the Association for Computational Linguistics (ACL Findings), 2025. [13] MongoDB Manual, MongoDB Inc., 2024, https://www.mongodb.com/ docs/manual/. [14] Anthropic, “Claude Opus 4.8 system card,” https://www.anthropic.com/ claude-opus-4-8-system-card, 2026. [15] OpenAI, “GPT-5.5 system card,” https://openai.com/index/ gpt-5-5-system-card/, 2026. [16] Google DeepMind, “Gemini 3.1 Pro model card,” https://deepmind. google/models/model-cards/gemini-3-1-pro/, 2026. [17] DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [18] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [19] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [20] Z. Rao, W. Zhu, C. A. Lu, Z. Chen, W. Niu, L. Guan, B. Li, and Z. Xiang, “FragFuse: Bypassing access control of large language model agents via memory-based query fragmentation and fusion,” arXiv preprint arXiv:2606.15609, 2026.

Related documents

• An Exact Rounds-versus-Queries Trade-off for Pointer Chasing, Machine-Checked in Lean 4#1393789
• The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion#1013670
• The Window Is in the Reasoning: When LLM Agents Decide to Follow Prompt Injections#1388614
• Replies to the Critics of Knowing and Checking: an Epistemological Investigation#729814
• Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges#1050383