arXiv CSopen access

MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning

MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning Charbel Al Bateh

[email protected]

Department of Electrical and Computer Engineering Lebanese American University Byblos, Lebanon

Samer Saab Jr.

[email protected]

Department of Electrical and Computer Engineering Lebanese American University Byblos, Lebanon

arXiv:2607.07391v1 [cs.AI] 8 Jul 2026

Abstract Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and longhorizon dialogue. We introduce MIRA-Math, a benchmark for a narrower diagnostic capability: solving mathematical problems whose full latent state has a unique answer, but whose solver-facing view is missing exactly one necessary atomic fact. The solver must request the missing information in natural language under a strict budget and then integrate the returned fact into an exact final answer. A fixed constrained LLM responder sees only the dataset-provided atomic fact and must either offer the quoted fact when the request matches it, or decline otherwise. Thus, instance generation, typed hint specifications, validation, and final-answer verification are deterministic, while request metrics are measured under a fixed LLM-mediated responder channel. MIRA-Math contains 2,310 generated instances from 22 typed mathematical families spanning algebra, probability, linear systems, discrete structures, signal processing, Markov chains, circuits, interpolation, and numerical boundary-value problems. Experiments across frontier and small models show that request success and final-answer accuracy are separable: models may ask for the right fact yet fail the downstream computation, or fail before obtaining the canonical hint. We release generators, verifiers, prompts, run metadata, and dataset documentation to support reproducible evaluation of minimal information requesting in mathematical reasoning. Keywords: benchmark, dataset generation, mathematical reasoning, information acquisition, clarification, partial observability, large language models

1 Introduction Large language models are increasingly evaluated on mathematical reasoning, tool use, and interactive problem solving. Yet many high-profile math benchmarks assess only the final answer under full information (Cobbe et al., 2021; Hendrycks et al., 2021; Balunović et al., 2025), while broad interactive benchmarks mix multiple sources of difficulty, including tool selection, retrieval, web navigation, external APIs, and long-context management (Mialon et al., 2023; Liu et al., 2023; Qin et al., 2023). These settings are valuable, but they make it difficult to isolate a basic bottleneck that appears whenever a model does not initially possess all relevant information: can the model recognize the missing fact, ask for that fact precisely, and use the answer correctly? ©2026 Al Bateh and Saab.

Al Bateh and Saab

MIRA-Math is designed to isolate this capability. Each benchmark instance is generated from a complete mathematical state with a unique answer, but the solver model receives only a private view that is deliberately insufficient, missing exactly one necessary atomic fact. A fixed information-holder model receives only the missing fact and is constrained by a structured-output protocol: if the solver’s request semantically matches the information it holds, it returns that fact; otherwise, it returns a declination stating that it does not have the requested information. The information holder does not solve, explain, negotiate, or volunteer extra hints. The benchmark therefore measures minimal information requesting under a controlled responder channel, not open-ended multi-agent collaboration. This framing is intentionally data-centric. Following the emphasis on public benchmark infrastructure and responsible dataset design in recent data-centric machine-learning work (Oala et al., 2024; Orr and Crawford, 2024), MIRA-Math includes generator code, exact verifiers, canonical hint specifications, prompt templates, run metadata, and dataset documentation. The benchmark is closest to diagnostic benchmark datasets that expose hidden structure or evaluation gaps through carefully controlled data construction (Matsubara et al., 2024; Zhang et al., 2024a,b; Telyatnikov et al., 2025), but its target capability is different: information acquisition under partial mathematical observability. Contributions.

This paper makes three contributions:

• We define a reproducible protocol for minimal information requesting in mathematical reasoning, with typed atomic hints, a fixed constrained LLM responder using structured offer/decline outputs, deterministic dataset validation, and exact answer checking. • We report baseline evaluations across multiple solver models and prompt regimes, over a benchmark of 2,310 instances showing that first-request success, request hit rate, and final accuracy capture distinct failure modes. • We provide dataset documentation, intended-use guidance, and a maintenance plan.

2 Related Work MIRA-Math is positioned at the intersection of mathematical reasoning benchmarks, clarification and information-acquisition benchmarks, active reasoning under incomplete information, and data-centric benchmark design. Table 1 summarizes the closest literature streams and the distinction between MIRA-Math and prior work. 2.1 Final-answer mathematical reasoning is not enough Mathematical reasoning benchmarks such as GSM8K and MATH have been central for measuring whether language models can produce correct final answers from complete problem statements (Cobbe et al., 2021; Hendrycks et al., 2021). More recent evaluations increase difficulty, reduce contamination, or track competition-level performance (Balunović et al., 2025). These benchmarks are important, but they usually assume that the model receives all information required to solve the problem. As a result, they do not isolate the ability 2

MIRA-Math

Table 1: Positioning of MIRA-Math relative to nearby benchmark streams. Stream

Typical focus

Full-information math and diagnostic benchmarks (Cobbe et al., 2021; Hendrycks et al., 2021; Balunović et al., 2025; Matsubara et al., 2024) Clarification and incompleteinformation reasoning (Min et al., 2020; Gan et al., 2024; Li et al., 2025; Zhou et al., 2025; Huang et al., 2025) Agent and tool-use benchmarks (Mialon et al., 2023; Liu et al., 2023; Qin et al., 2023; Zhou et al., 2023) Multi-agent communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Das et al., 2019; Wang et al., 2020; Wu et al., 2023; Li et al., 2023; Chen et al., 2023) Data-centric benchmark design (Oala et al., 2024; Orr and Crawford, 2024; Gebru et al., 2021; Mitchell et al., 2019)

Solve fully specified problems, often The solver’s private view is intentionally with exact final-answer scoring or con- incomplete; success requires requesting trolled generators. and then using a missing atomic fact.

How MIRA-Math differs

Ask useful questions or detect missing MIRA-Math uses typed mathematical information in underspecified tasks. families, a fixed constrained information holder, and exact verification of final mathematical integration. Evaluate web navigation, tools, re- MIRA-Math removes tools and envitrieval, APIs, and long-horizon work- ronment management to isolate minimal flows. mathematical information requesting. Study coordination and message passing The responder is a fixed constrained inunder partial observability. formation holder, not a strategic teammate, so failures are more localized to request precision, responder matching, or mathematical integration.

Emphasize public artifacts, documen- MIRA-Math releases generators, veritation, intended use, and reproducible fiers, typed hint schemas, responder specevaluation. ifications, prompts, run metadata, and documentation.

to notice that a problem is underdetermined, identify the missing fact, ask for it, and then use it correctly. MIRA-Math targets this missing capability directly. It should therefore not be read as a replacement for full-information mathematical benchmarks. Instead, it complements them by turning information acquisition into an explicitly measured step. A model that is strong on ordinary math benchmarks may still fail in MIRA-Math if it cannot identify the missing atomic fact or if it asks for an irrelevant quantity. 2.2 Controlled diagnostic benchmarks A second relevant stream uses controlled generation and exact evaluation to reveal hidden model weaknesses. Symbolic and scientific benchmark work has shown the value of generator-controlled tasks, exact verifiers, and diagnostic decompositions (Matsubara et al., 2024; Somasekharan et al., 2026). Data-centric benchmark papers similarly emphasize that the dataset, generator, evaluation protocol, and documentation are part of the scientific contribution (Zhang et al., 2024a,b; Telyatnikov et al., 2025). MIRA-Math follows this diagnostic tradition but uses controlled generation for a different purpose. Each instance is generated from a latent complete mathematical state that fixes a unique answer, while the solver’s private view omits a necessary atomic fact. This global-well-posed/local-underdetermined construction lets us evaluate not only whether the model eventually gets the answer right, but also whether it requested the correct missing information along the way. 3

Al Bateh and Saab

2.3 Clarification and information acquisition Clarification benchmarks study whether models ask useful questions when instructions, dialogue states, or user requests are ambiguous or underspecified (Min et al., 2020; Gan et al., 2024; Zhang et al., 2025; Zhao et al., 2026). This line of work is closely related in motivation in that practical assistants should not always guess when required information is absent. However, many clarification settings involve subjective ambiguity, natural human preferences, or dialogue-level uncertainty. These properties are valuable for realism but make it difficult to assign exact credit for a particular request. MIRA-Math reduces this ambiguity by construction. The full latent state has a unique target answer, and the missing information is represented by typed atomic hints specified by the generator. The solver may still ask in free-form natural language, but the response channel is constrained. A fixed information-holder LLM sees only its private constraints and must choose between a structured offer and a structured decline. The accepted/declined events are therefore logged in an auditable schema, while the semantic matching decision itself remains LLM-mediated. The deterministic components are the generator invariants, canonical hint specification, instance validation, and final-answer verification. 2.4 Active and missing-information reasoning benchmarks The closest recent work studies reasoning under incomplete information. QuestBench formalizes underspecified reasoning as missing variable assignments and evaluates whether models can identify the minimal necessary question in logic, planning, and math-derived tasks (Li et al., 2025). AR-Bench broadens the setting to active reasoning, where models must ask questions to acquire missing evidence in interactive environments (Zhou et al., 2025). CRITIC-math focuses directly on incomplete mathematical problems and evaluates whether large reasoning models proactively ask for information rather than hallucinating an answer (Huang et al., 2025). MIRA-Math is closest to this family of benchmarks, but it makes a different design choice. QuestBench emphasizes selecting the right clarification from a finite set of options; MIRA-Math requires open-ended natural-language requests that must be precise enough to be accepted by a fixed constrained information-holder LLM. AR-Bench emphasizes broader active reasoning in interactive scenarios; MIRA-Math restricts the environment to a single controlled mathematical information channel. CRITIC-math studies whether models detect incompleteness in mathematical problems; MIRA-Math additionally evaluates whether the model identifies the exact missing atomic slot and integrates the returned value into an exact solution. These differences make MIRA-Math a narrower but more controlled diagnostic for minimal mathematical information requesting. 2.5 Agent, tool-use, and communication benchmarks Broad agent benchmarks evaluate capabilities such as tool selection, API use, retrieval, web navigation, and long-horizon planning (Mialon et al., 2023; Liu et al., 2023; Qin et al., 2023; Zhou et al., 2023). LLM multi-agent frameworks and multi-agent reinforcement-learning work further study how agents coordinate and communicate under partial observability (Foerster et al., 2016; Sukhbaatar et al., 2016; Das et al., 2019; Wang et al., 2020; Wu 4

MIRA-Math

et al., 2023; Li et al., 2023; Chen et al., 2023). These settings are useful, but they conflate information acquisition with many other sources of difficulty. MIRA-Math intentionally avoids this conflation. The second role is not an autonomous collaborator and does not solve, negotiate, explain, or strategically communicate. It is a fixed LLM-mediated information holder over dataset-provided atomic hints: the holder sees only its private constraints, must quote an explicit constraint when offering information, and otherwise returns a structured declination. This design makes MIRA-Math a benchmark of request precision and mathematical integration, not a benchmark of general multi-agent collaboration. 2.6 Data-centric benchmark design Finally, MIRA-Math is a data-centric benchmark contribution. Work on dataset documentation, model cards, and responsible benchmark design emphasizes the need to specify intended uses, limitations, quality-control procedures, and maintenance plans (Gebru et al., 2021; Mitchell et al., 2019; Orr and Crawford, 2024; Oala et al., 2024). Following this view, the MIRA-Math release is intended to include generated JSONL files, family generators, exact verifiers, canonical hint specifications, prompt templates, run metadata, raw logs, and dataset documentation. The benchmark’s contribution is therefore not only a collection of problems, but a reproducible evaluation protocol for a specific capability: asking for the minimal missing mathematical fact needed to solve an underdetermined problem.

3 Benchmark Design 3.1 Task Formalization A MIRA-Math instance is generated from a latent complete state z. The full state determines a unique target answer y = f (z). The solver receives a private view vA = ϕA (z) that omits one necessary atomic fact. The hint source stores that atomic hint h ∈ H(z), each with a typed identifier such as a missing coefficient, boundary value, congruence, transition probability, or interpolation point. The benchmark enforces two conditions: global well-posedness:

|{y : y is consistent with vA ∪ H(z)}| = 1,

(1)

local underdetermination:

|{y : y is consistent with vA }| > 1.

(2)

3.2 Interaction Protocol At evaluation time the solver is shown its private view, the target question, and the request budget B. At each turn the solver may either issue a request or provide a final answer. Requests are free-form natural language. A fixed information-holder model receives its own private view containing the missing atomic fact and must respond using a structured schema. If the request semantically matches the fact it holds, the responder returns that fact; otherwise, it returns a declination and the solver may try again until the request budget is exhausted. For example, in a recurrence instance the solver may see the recurrence rule and a(0) but not a(1). A request such as “What is the missing value of a(1)?” maps to the canonical initial-condition hint and receives the stored value. A vague request such as “Can you help 5

Al Bateh and Saab

Figure 1: Overview of the MIRA-Math protocol, illustrated with a successful circuit missing resistance trace. Agent A receives a private view with one atomic fact hidden, asks for the missing slot in natural language, and receives either a structured offer or decline from a fixed constrained information-holder that sees only its private constraint. After receiving the hint, Agent A must still compute the final answer and pass the family-specific exact verifier.

me solve the recurrence?” is declined because it does not identify an allowed atomic hint. In a variable-slot family, such as a linear system with one missing coefficient, the request must identify the missing coefficient slot rather than merely ask for “the missing information.” This constrained responder channel is a central design choice. In an unconstrained twomodel dialogue, the responder could solve part of the problem, volunteer extra information, negotiate, or provide explanations. MIRA-Math removes these behaviors by giving the information holder only its private constraints and requiring a structured offer-or-decline response. The resulting request metrics should be interpreted as measuring the solver’s information-requesting behavior under a controlled LLM responder, not as a measure of general multi-agent teamwork or a purely deterministic request-matching oracle. Responder implementation. In the released reference runner, the information holder is implemented as a fixed LLM responder with a flat structured-output schema. Its only valid outputs are offer, with has exact match=true, a quoted private constraint, and a hint string; or decline, with a fixed declination message. If the structured output cannot be parsed, the runner attempts conservative raw-output extraction and otherwise records a declination. This design prevents the responder from solving the task, volunteering extra information, or giving partial clues, but it does not make semantic matching itself deterministic. Consequently, request metrics should be interpreted as solver performance under this fixed constrained responder channel. Exact mathematical correctness is still evaluated by deterministic family-specific verifiers. 3.3 Family Types The 22 families are partitioned into two types. Type A families have a fixed hint slot: the missing information is structurally determined by the family. For example, a Bayes-rule instance always withholds the prior, and a recurrence instance always withholds one initial condition. Type B families have a variable hint slot: the missing position changes by instance, such as a missing transition probability, boundary value, coefficient, interpolation 6

MIRA-Math

point, resistor, or grid cell. Type B therefore requires the solver to locate which slot is missing before phrasing the request. This design supports diagnosis of two separate subskills. Type A primarily tests whether the model knows what kind of fact is needed and can compute after receiving it. Type B additionally tests whether the model can inspect its local view, identify the absent slot, and request that slot precisely. 3.4 Metrics We report the following metrics. • Accuracy: fraction of instances whose final answer normalizes to the ground truth under type-aware comparison. • Request hit rate: mean per-instance ratio of accepted requests to fixed-responder messages, offers/(offers + declines). • First-request success: fraction of instances in which the solver’s first request receives a structured offer from the fixed responder. • Average requests: mean number of requests the solver issues per instance. • Average declinations: mean number of declinations from the fixed responder per instance. • Average hints: mean number of accepted offers the solver receives per instance. • Average tokens: approximate transcript cost, computed as mean transcript word-count × 1.3 per instance and summed across the solver and responder messages. Trace-level decomposition. To separate information-acquisition failures from downstream mathematical failures, we also classify each transcript using three binary indicators. Let Oi indicate that instance i received at least one structured offer; let Ci indicate that an offered hint matches the instance’s canonical atomic-hint specification under an offline audit of the quoted constraint and machine-readable hint fields; and let Fi indicate that the final answer is accepted by the exact verifier. We report: 1 X 1 X 1 X pno-canonical = (1 − Ci ), pintegration-fail = Ci (1 − Fi ), psuccess = Fi . N N N i

i

i

The first quantity captures failures to acquire the canonical missing fact under the fixed responder channel. The second captures cases where the missing fact was acquired but the solver did not integrate it into a correct mathematical answer. The third is the usual final-answer accuracy. We separately log raw offers Oi so that any rare noncanonical offer can be distinguished from a genuine solver request success. Accuracy measures mathematical completion, while request metrics measure information acquisition under the fixed responder channel. Their separation is important: a model may fail to elicit the canonical missing fact, may elicit it but still fail the mathematics, or may solve correctly after receiving it. The trace-level decomposition above makes these cases explicit rather than relying only on aggregate accuracy and request hit rate. 7

Al Bateh and Saab

4 Dataset Construction 4.1 Current Release The current MIRA-Math release is the 20/50 typed dataset: 20 instances per difficulty level for each Type A family and 50 instances per difficulty level for each Type B family. This yields 660 Type A instances, 1,650 Type B instances, and 2,310 instances in total. Each instance stores the family name, type, difficulty, random seed, solver prompt, target question, typed atomic-hint specification, canonical hint values, exact answer, and verifier metadata. The 22 families (described in Appendix C) cover Bayesian inference, Chinese remainder reconstruction, coordinate geometry, triangle path-sum inversion, linear systems with a separator variable, rank-one matrix completion, moment recovery, phase retrieval, piecewise functions, rank-deficient linear systems with a shared variable, recurrences, birth-death chains, series-parallel resistor networks, deconvolution, discrete tomography, eigenvectorconstrained matrix questions, discrete Laplace equations, linear systems with a missing coefficient, Markov chains, polynomial interpolation, portfolio variance, and hidden Markov models. 4.2 Generator Invariants and Quality Control Every generator enforces family-specific nondegeneracy conditions. Examples include rejecting singular linear systems, parallel lines, nonunique phase-retrieval instances, ambiguous tomography targets without a sufficient hint cell, and rational answers whose denominators exceed family-specific thresholds. All numeric answers are represented exactly as integers, tuples, or reduced rational values. Verifiers compare normalized exact values rather than surface strings. For families where local underdetermination can be checked algebraically, the generator uses rank, uniqueness, or constraint-satisfaction tests. For finite discrete families, the generator can enumerate consistent completions. These checks are part of the dataset artifact, so users can regenerate the release, create larger splits, or audit any instance from its seed. 4.3 Data Format, Access, and Maintenance The public release includes generated JSONL files, family generator code, exact verifiers, prompt templates, responder specifications, model-run scripts, and raw logs. The repository and dataset pages are: Code and documentation: https://github.com/cedar-lau/mira-math Generated data: https://huggingface.co/datasets/samersaabjr/MIRA-MATH/ The generated benchmark data are released under CC BY 4.0, and the generator, verifier, runner, and evaluation code are released under the MIT License. The maintenance plan is to version releases by generator version, seed set, and prompt protocol. Future releases will preserve the current 20/50 typed split as a regression suite while adding larger held-out seed sets and optional multi-hint variants. Each release will include checksums, schema documentation, and a changelog documenting changes to generators, prompts, matching rules, and verifiers. 8

MIRA-Math

5 Experimental Protocol We evaluate solver models in two prompt regimes: zero-shot (ZS) and four-shot (4S). The four-shot prompt provides examples of valid minimal requests and final-answer formatting without reusing test instances. In all reported experiments, the information-holder model is fixed to gpt-4o-mini; the solver model is the variable under evaluation. The responder receives only its private constraints and is prompted to return a flat structured output with one of two types: offer, when it judges the request to semantically match one of its private constraints, or decline, otherwise. A valid offer must include has exact match=true, the exact quoted private constraint, and the extracted hint string. A declination uses a fixed message and must not reveal what other information the responder holds. Thus the responder is fixed and constrained, but it is not a deterministic matching oracle. The released runner stores the full transcript for each instance, including Agent A requests, Agent B offer/decline decisions, quoted constraints, extracted hints, parsing errors when present, final answers, and exact-verifier outcomes. This makes it possible to recompute aggregate metrics and the trace-level decomposition in Section 6 without rerunning model calls, provided the raw transcripts are retained. The main results report six solver models. Exact decoding settings, model identifiers, timestamps, prompts, and raw transcripts are included with the release artifacts so that future users can reproduce the runs or update the baselines as model APIs and open-weight checkpoints change.

6 Results Requesting and solving are distinct. Figure 2 summarizes final-answer accuracy across solver models and prompting regimes, while Appendix B.1 retains the full numeric request metrics. The aggregate results show that high request performance does not necessarily imply high final-answer accuracy. For example, in the zero-shot setting, gemini-2.5-flash has the highest request hit rate and first-request success among the completed models, but its final accuracy remains below gpt-5.1 and Gemma-4. Conversely, Gemma-4 obtains the highest zero-shot final accuracy while not achieving the highest request metrics. This validates the benchmark’s central diagnostic split: asking for the right fact and using it correctly are separable skills. Trace snapshots make the diagnostic split concrete. To make the aggregate metrics easier to interpret, Appendix D gives short, offline-audited transcript snapshots for three representative outcomes: no canonical hint acquired, canonical hint acquired but final answer wrong, and full request–resolve–solve success. These snapshots are not used to estimate prevalence and should not be read as representative samples. Instead, they show what the logged metrics mean at the transcript level. For example, a no-hint trace shows a solver cycling through plausible but wrong missing rates until the request budget is exhausted; a hint-but-wrong trace shows the solver receiving the prior needed for Bayes’ rule but still submitting an incorrect posterior; and a full-success trace shows the solver requesting the missing resistor value and then computing the equivalent resistance exactly. Table 2 summarizes the examples and their diagnostic role. 9

Al Bateh and Saab

Figure 2: Overall final-answer accuracy by solver model and prompting regime. Bars are grouped by solver model, with separate zero-shot (ZS) and four-shot (4S) results. The full numeric counterpart, including request hit rate, first-request success, average requests, declinations, and token estimates, is retained in Appendix B.1.

Request acquisition versus mathematical integration. The transcript decomposition confirms that final accuracy should not be read as a pure requesting score. In the reference protocol, an instance can fail because the solver never obtains the canonical atomic hint, or because it obtains the hint but performs the subsequent computation incorrectly. We therefore report the disjoint trace-level categories in Table 3: no canonical hint acquired, canonical hint acquired but final answer wrong, and final answer correct. This decomposition is especially important for families such as Chinese-remainder reconstruction, phase retrieval, polynomial interpolation, and portfolio variance, where models can often obtain the relevant hint but still fail exact arithmetic, symbolic reconstruction, or answer normalization. Variable-slot families are not uniformly harder. Figure 3 shows that Type B families, despite requiring the solver to identify which slot is absent, are not uniformly harder in final-answer accuracy. Strong models often score higher on Type B than on Type A. This is not a contradiction: Type A contains several families where the request is easy but the downstream computation is brittle, such as exact Chinese-remainder reconstruction, path-sum inversion, moment recovery, and phase retrieval. The Type A/Type B distinction should therefore be interpreted as a missing-slot diagnostic, not as a total difficulty ranking. Appendix B.1 gives the corresponding request-quality metrics, including hit rate, first-request success, and average number of requests by family type. 10

MIRA-Math

Table 2: Illustrative transcript snapshots used to interpret the aggregate metrics. These are hand-audited examples, not a frequency estimate. Full transcripts and raw logs are released with the benchmark artifacts. Tag Outcome class

Example family

T1

No canonical hint acquired

birth death missing rate

T2

Canonical hint acquired, final answer wrong

T3

Canonical hint acquired, final answer correct

bayes missing prior

circuit missing resistance

Instance

Diagnostic role

bir-000768 The solver recognizes that a rate is missing but repeatedly asks for the wrong rate, so the responder only declines and no final answer is submitted. bay-000044 The solver asks for and receives the prior P (H), but the downstream Bayes calculation is wrong. This is an integration failure rather than a request failure. cir-000924 The solver identifies the missing resistor, receives it, and computes the exact equivalent resistance. This illustrates the intended successful loop.

Difficulty interacts with family structure. Figure 4 shows that smaller models can degrade sharply with difficulty, while stronger models are less monotonic. This reflects the benchmark design: difficulty scaling may increase coefficient ranges, dimensions, denominators, or target-index depth depending on the family. Per-family and per-type reporting remain essential, because a single aggregate score can hide whether a model fails at requesting, exact arithmetic, linear algebra, combinatorial search, or answer normalization. The full difficulty-by-type breakdown is provided in Appendix B.1. Four-shot prompting is not uniformly beneficial. Figures 2–4 show that the tested four-shot prompt does not uniformly improve final-answer accuracy. The Type A/Type B split helps interpret this pattern. Demonstrations can help when the missing slot is structurally stable: in Type A families, the solver can often reuse a demonstrated pattern such as asking for a prior, initial condition, threshold, or missing equation. By contrast, Type B families require instance-level slot localization: the solver must inspect the current problem and identify which coefficient, transition, boundary value, grid cell, resistor, interpolation point, or emission entry is absent. A small four-shot prompt necessarily covers only a few request archetypes, and may therefore induce exemplar anchoring: the solver may imitate the demonstrated request forms instead of adapting to the specific missing slot in the current instance. We treat this as a prompt-sensitivity hypothesis rather than a causal claim. The main result is that demonstrations of the interaction protocol do not automatically improve minimal-information reasoning; in some families they improve first-request compliance while leaving mathematical integration unchanged or worse. Appendix D includes 11

Al Bateh and Saab

Table 3: Trace-level decomposition of failures. “No canonical hint” means that the transcript did not contain an accepted offer matching the instance’s canonical atomichint specification under the offline hint audit. “Hint but wrong” means that the canonical hint was acquired but the exact final-answer verifier rejected the solver’s answer. “Correct” is the standard final-answer accuracy. All values are computed from raw transcripts; no additional model calls are required to reproduce this decomposition from the released logs.

Format

No canonical hint

Hint but wrong

Correct

Acc. given canonical hint

gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

ZS ZS ZS ZS ZS ZS

0.213 0.200 0.126 0.187 0.231 0.393

0.540 0.111 0.385 0.075 0.266 0.574

0.248 0.689 0.490 0.739 0.503 0.033

0.314 0.861 0.560 0.908 0.654 0.054

gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

4S 4S 4S 4S 4S 4S

0.210 0.197 0.196 0.216 0.250 0.375

0.557 0.116 0.477 0.074 0.267 0.581

0.233 0.687 0.328 0.710 0.483 0.044

0.295 0.855 0.407 0.906 0.644 0.070

Agent A model

an illustrative prompt-anchoring snapshot (T4), and Appendix B.1 gives the full per-family four-shot deltas. Taken together, these results show that MIRA-Math should be read less as a single leaderboard and more as a diagnostic decomposition. The same aggregate accuracy can arise from different mechanisms: failure to localize the missing slot, successful acquisition followed by incorrect computation, or correct end-to-end request and solution. The tracelevel logs, per-family metrics, and Type A/Type B split are therefore intended to be used jointly when comparing models or prompt strategies.

7 Failure Modes and Diagnostic Use The benchmark exposes at least two recurring failure modes. Request-acquisition failures occur when the solver does not obtain the canonical missing fact under the fixed responder channel. These can happen because the solver asks for a wrong slot, asks too vaguely, or repeatedly probes quantities that are already present in its private view. Integration failures occur when the solver receives the canonical hint but still computes, formats, or normalizes the final answer incorrectly. 12

MIRA-Math

Figure 3: Type A versus Type B final-answer accuracy by solver model and prompting regime. Type A families have a fixed missing-information slot, whereas Type B families require instance-level localization of the missing slot. The full numeric tables for Type A/Type B accuracy and request quality are retained in Appendix B.1.

Several families make this distinction visible. In fixed-slot families such as bayes missing prior, the request can be easy while exact arithmetic remains brittle. In variable-slot families such as birth death missing rate, the solver must first localize the absent rate before it can apply a standard computation. Other families, such as circuit missing resistance, show the intended successful pattern: identify the missing slot, request it precisely, and integrate the returned value into the exact verifier-compatible answer. Appendix D gives compact transcript snapshots for these cases. The examples are illustrative rather than representative; aggregate claims are based on the tables above. Their purpose is to show how the same final incorrect score can correspond to different underlying failures: no canonical hint acquired, canonical hint acquired but wrong downstream computation, or successful request and solution. This is why the release includes both exact verifiers and raw interaction logs.

8 Intended Uses and Limitations Intended uses. MIRA-Math is intended for evaluating models, prompts, and solver loops that must operate under partial mathematical information. Appropriate uses include measuring request precision, comparing communication-efficiency strategies, auditing whether a model can identify missing facts, and studying the relationship between clarification success and final mathematical accuracy. 13

Al Bateh and Saab

Figure 4: Overall final-answer accuracy by difficulty level, solver model, and prompting regime. Bars are grouped by solver model and split by difficulty level D1–D3 for zero-shot and four-shot prompting. The full numeric difficulty tables, including average request counts and the difficulty-by-family-type accuracy split, are retained in Appendix B.1.

Out-of-scope uses. The benchmark should not be used as evidence that a model can collaborate with another autonomous agent. The second role is a fixed, constrained information holder, not a teammate. The benchmark should also not be treated as a general measure of tool use, web navigation, social reasoning, or real-world decision-making. Limitations. The tasks are synthetic and mathematically structured. This improves controllability and exact verification but limits realism. The current release uses one required atomic hint per instance; many real tasks require multiple facts, or uncertain evidence. The typed hint schema and constrained responder protocol is useful for reproducibility but may understate the difficulty of communicating with humans or open-ended systems. The responder channel is controlled but not deterministic: in the reported experiments, a fixed LLM information holder performs the semantic offer-or-decline decision under structuredoutput constraints. This means request metrics can still reflect occasional responder matching errors, although the final-answer verifier and dataset validity checks are deterministic. Finally, model results can become stale as APIs and open-weight checkpoints change; the release is designed to support re-running baselines rather than treating the reported ranking as permanent.

9 Conclusion MIRA-Math introduces a controlled benchmark for minimal information requesting in mathematical reasoning. Each instance is generated from a complete latent mathematical state with a unique answer, but the solver receives only a locally underdetermined view 14

MIRA-Math

and must request exactly the missing atomic fact needed to complete the solution. This construction turns information acquisition into an explicit, measurable step rather than leaving it implicit inside final-answer accuracy. The results show that this step is not redundant with ordinary mathematical solving. Across the tested models, request hit rate, first-request success, and final-answer accuracy diverge substantially: some models obtain the canonical hint but fail exact downstream computation, while others fail before acquiring the resolving fact. The Type A/Type B split and transcript-level decomposition further show that fixed missing slots, variable missing slots, arithmetic integration, and answer normalization can fail independently. The benchmark is intentionally narrow. It does not claim to measure open-ended collaboration, tool use, or human clarification behavior. Instead, it provides a reproducible diagnostic setting with typed atomic hints, a fixed constrained LLM responder, deterministic validation, exact family-specific verifiers, and released transcripts. We hope this makes MIRA-Math useful both as a baseline suite for future models and as an analysis tool for studying when language models ask for missing mathematical information rather than guessing.

Broader Impact Statement MIRA-Math has positive potential as a controlled diagnostic for safer and more transparent use of language models in settings where systems should ask for missing information instead of guessing. The main risks are overclaiming and benchmark overfitting. The benchmark does not measure open-ended collaboration, human communication, or real-world decision-making, and we state these limitations explicitly. Since the data are synthetically generated mathematical problems, privacy and consent risks are minimal. We mitigate misuse by releasing generator code, verifiers, documentation, and intended-use guidance, and by recommending held-out seed sets for future evaluations.

Acknowledgments and Disclosure of Funding The authors are responsible for the dataset design, generated instances, and evaluation protocol.

References Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions. arXiv preprint arXiv:2505.23281, 2025. doi: 10.48550/arXiv.2505.23281. Weize Chen, Yusheng Su, Cheng Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. arXiv preprint arXiv:2308.10848, 2023. doi: 10.48550/ arXiv.2308.10848. 15

Al Bateh and Saab

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. doi: 10.48550/arXiv.2110.14168. Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communication. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1538–1546. PMLR, 2019. URL https://proceedings.mlr. press/v97/das19a.html. Jakob N Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06676, 2016. doi: 10.48550/arXiv.1605.06676. Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. ClarQ-LLM: A benchmark for models clarifying and requesting information in taskoriented dialog. arXiv preprint arXiv:2409.06097, 2024. doi: 10.48550/arXiv.2409.06097. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. doi: 10.1145/3458723. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021. doi: 10.48550/arXiv.2103.03874. Youcheng Huang et al. Beyond solving math quiz: Evaluating the ability of large reasoning models to ask for information in incomplete mathematical problems. arXiv preprint arXiv:2508.11252, 2025. Belinda Z Li, Been Kim, and Zi Wang. Questbench: Can llms ask the right question to acquire information in reasoning tasks? arXiv preprint arXiv:2503.22674, 2025. doi: 10.48550/arXiv.2503.22674. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for “mind” exploration of large language model society. arXiv preprint arXiv:2303.17760, 2023. doi: 10.48550/arXiv.2303.17760. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. doi: 10.48550/arXiv.2308.03688. Yoshitomo Matsubara, Naoya Chiba, Ryo Igarashi, and Yoshitaka Ushiku. Rethinking symbolic regression datasets and benchmarks for scientific discovery. Journal of Datacentric Machine Learning Research, 2024. URL https://openreview.net/forum?id= qrUdrXsiXX. 16

MIRA-Math

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023. doi: 10.48550/arXiv.2311.12983. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5783–5797. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.466. URL https://aclanthology.org/2020.emnlp-main.466/. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229, 2019. doi: 10.1145/3287560.3287596. Luis Oala, Manil Maskey, Lilith Bat-Leah, Alicia Parrish, Nezihe Merve Gürel, Tzu-Sheng Kuo, Yang Liu, Rotem Dror, Danilo Brajovic, Xiaozhe Yao, Max Bartolo, William A Gaviria Rojas, Ryan Hileman, Rainier Aliment, Michael W Mahoney, Meg Risdal, Matthew Lease, Wojciech Samek, Debojyoti Dutta, Curtis G Northcutt, Cody Coleman, Braden Hancock, Bernard Koch, Girmaw Abebe Tadesse, Bojan Karlaš, Ahmed Alaa, Adji Bousso Dieng, Natasha Noy, Vijay Janapa Reddi, James Zou, Praveen Paritosh, Mihaela van der Schaar, Kurt Bollacker, Lora Aroyo, Ce Zhang, Joaquin Vanschoren, Isabelle Guyon, and Peter Mattson. DMLR: Data-centric machine learning research— past, present and future. Journal of Data-centric Machine Learning Research, 2024. URL https://openreview.net/forum?id=2kpu78QdeE. Will Orr and Kate Crawford. Building better datasets: Seven recommendations for responsible design from dataset creators. Journal of Data-centric Machine Learning Research, 2024. URL https://openreview.net/forum?id=6bd8BrRKTW. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. doi: 10.48550/arXiv.2307.16789. Nithin Somasekharan, Ling Yue, Yadi Cao, Weichao LI, Patrick Emami, Pochinapeddi Sai Bhargav, Anurag Acharya, Xingyu Xie, and Shaowu Pan. CFDLLMBench: A benchmark suite for evaluating large language models in computational fluid dynamics. Journal of Data-centric Machine Learning Research, 2026. URL https://data.mlr.press/ volumes/03.html. Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. arXiv preprint arXiv:1605.07736, 2016. doi: 10.48550/arXiv.1605. 07736. Lev Telyatnikov, Guillermo Bernardez, Marco Montagna, Mustafa Hajij, Martin Carrasco, Pavlo Vasylenko, Mathilde Papillon, Ghada Zamzmi, Michael T Schaub, Jonas Ver17

Al Bateh and Saab

hellen, Pavel Snopov, Bertran Miquel-Oliver, Manel Gil-Sorribes, Alexis Molina, VICTOR GUALLAR, Theodore Long, Julian Suk, Patryk Rygiel, Alexander V Nikitin, Giordan Escalona, Michael Banf, Dominik Filipiak, Liliya Imasheva, Max Schattauer, Alvaro L Martinez, Halley Fritze, Marissa Masden, Valentina Sánchez, Manuel Lecha, Andrea Cavallo, Claudio Battiloro, Matthew Piekenbrock, Mauricio Tec, George Dasoulas, Nina Miolane, Simone Scardapane, and Theodore Papamarkou. Topobench: A framework for benchmarking topological deep learning. Journal of Data-centric Machine Learning Research, 2025. URL https://openreview.net/forum?id=07sTzyEVtY. Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9908–9918. PMLR, 2020. URL https: //proceedings.mlr.press/v119/wang20i.html. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multiagent conversation. arXiv preprint arXiv:2308.08155, 2023. doi: 10.48550/arXiv.2308. 08155. Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, and Robert D Nowak. Labelbench: A comprehensive framework for benchmarking adaptive labelefficient learning. Journal of Data-centric Machine Learning Research, 2024a. URL https://openreview.net/forum?id=Y2QcZfwHE7. Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection. Journal of Data-centric Machine Learning Research, 2024b. URL https://openreview.net/forum?id=cnnTnJQigs. Dataset Certification. Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi. Modeling future conversation turns to teach llms to ask clarifying questions. Proceedings of the International Conference on Learning Representations, 2025. Jiale Zhao, Ke Fang, and Lu Cheng. When and what to ask: Askbench and rubric-guided rlvr for llm clarification. arXiv preprint arXiv:2602.11199, 2026. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information? In Proceedings of the International Conference on Machine Learning, 2025. 18

MIRA-Math

Dataset Certification MIRA-Math is a synthetically generated benchmark. Each instance is produced from a recorded generator family, difficulty level, random seed, typed atomic-hint specification, exact answer, and verifier metadata. The release includes generated data, generator code, exact verifiers, prompt templates, responder specifications, model-run scripts, raw interaction logs, and documentation. The dataset contains mathematical problem text and synthetic numeric values rather than personal data. Intended uses and out-of-scope uses are described in Section 8.

Appendix A. Implementation and Reproducibility Details Released artifacts. The release contains the family generators, instance validator, exact family-specific solvers, prompt templates, structured-output schemas, reference LangGraph runner, scoring code, generated JSONL files, run metadata, and raw transcripts. Each generated instance records the family name, difficulty, random seed, agent-private constraints, machine-readable canonical atomic-hint specification, global solution, answer type, and verifier metadata. Instance validation. The validator checks three properties for each instance: global uniqueness of the full latent problem, local underdetermination of each agent’s private view, and sufficiency of the canonical atomic hint when applied to the solver’s view. These checks are family-specific and deterministic. The final-answer verifier also uses normalized exact values rather than surface-form string matching. Responder protocol. The reference responder is a fixed information-holder LLM. It receives only its private constraints and Agent A’s current request. It must emit a flat structured object with either offer or decline. An offer must set has exact match=true, quote the exact private constraint, and provide the extracted hint. A decline must not reveal alternative information. If structured parsing fails, the runner attempts conservative raw-output extraction; if that also fails, the response is recorded as a declination. Therefore, semantic matching is controlled and auditable but LLM-mediated. Transcript scoring. For each transcript, the scorer records final-answer accuracy, request attempts, offer count, declination count, first-request success, requests before first offer, token estimate, and exact expected/actual normalized answers. We additionally compute the trace-level indicators used in Table 3: whether a canonical hint was acquired, whether the solver produced a correct final answer after acquiring it, and whether a noncanonical offer occurred. Versioning. To make future reruns comparable, each release records the generator version, seed set, prompt-protocol version, responder model identifier, solver model identifier, decoding settings, code commit, dataset checksum, and transcript checksum. Model rankings should be interpreted as a snapshot under these settings because API-backed model behavior can change over time. Exact run configuration. Tables 4 and 5 record the concrete settings used for all reported runs so that the main reproducibility facts are available without consulting the 19

Al Bateh and Saab

Table 4: Shared run configuration, identical across all reported solver models and both prompting regimes. The request budget scales with instance difficulty d as 3d. Datasets are released as JSONL with a fixed generation seed; each instance additionally stores its own generation seed. Setting

Value

Responder (Agent B) Decoding temperature Max new tokens Request budget (max attempts) Runs per instance Inference-time seed Per-instance timeout Zero-shot dataset Four-shot dataset Dataset generation seed Protocol / method

gpt-4o-mini (OpenAI) 0.0 (Agent A and Agent B) unset (provider/model default) 3/6/9 at difficulty 1/2/3 1 (single pass) 1234 3600 s (parallel runner) family types 20 50.jsonl (2,310 inst.) family types 20 50 few shot.jsonl (2,310 inst.) 1234 llm (single-shot request, then final answer)

Table 5: Per-solver exact model identifiers, serving backend, and run dates. The responder (Agent B) is gpt-4o-mini in every configuration; for the gpt-4o-mini solver, Agents A and B are the same model. Date ranges span the initial run and any subsequent resumption waves for instances that timed out or errored. Solver (Agent A)

Exact model ID

Backend

Run dates (2026)

gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

gpt-4o-mini gpt-5.1 gemini-2.5-flash google/gemma-4-31b-it x-ai/grok-4.20 Meta-Llama-3.1-8B-Instruct (quantized, 16k ctx)

OpenAI API OpenAI API Google GenAI / OpenRouter OpenRouter OpenRouter Ollama (local)

Mar 13 Mar 23 Apr 09–21 Apr 17–19 Apr 22–26 May 07–09

release artifacts. All runs use greedy decoding (temperature = 0) for both agents, a single pass per instance, and a fixed request seed of 1234; residual provider-side nondeterminism may remain where the backend does not honor the seed. Models are referenced by provider alias on the dates shown; the underlying API snapshots and open-weight checkpoints are not version-pinned and may change.

Appendix B. Additional Results and Full Numeric Tables B.1 Full numeric tables for main-text figures Tables 6–10 provide the full numeric values underlying Figures 2–4. The main text visualizes the central accuracy patterns for readability, while these appendix tables retain the complete request-quality and token-usage metrics. 20

MIRA-Math

Table 6: Full overall performance metrics on MIRA-Math. This table is the numeric counterpart of Figure 2 and includes request hit rate, first-request success, average requests, declinations, and token estimates. Agent A Model gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

Type

Acc

Hit Rate

1st-Req OK

Avg Req

Avg Decl

Avg Tokens

ZS

0.248 0.690 0.490 0.739 0.503 0.033

0.596 0.701 0.793 0.720 0.661 0.433

0.493 0.643 0.735 0.652 0.587 0.329

2.55 2.19 1.69 2.12 2.34 3.35

1.77 1.39 0.82 1.30 1.57 2.74

507 1065 533 1099 1536 165

4S

0.233 0.687 0.328 0.710 0.483 0.044

0.604 0.719 0.722 0.703 0.634 0.450

0.510 0.665 0.670 0.648 0.555 0.347

2.60 2.09 2.06 2.26 2.46 3.35

1.81 1.29 1.25 1.47 1.71 2.72

494 1013 672 1107 1513 173

Table 7: Full performance by family type. Type A denotes fixed missing-information slots and Type B denotes variable missing-information slots. This table is the numeric counterpart to Figure 3. Accuracy Agent A Model gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

Type

1st-Req OK

Avg Requests

Type A

Type B

Type A

Type B

Type A

Type B

ZS

0.359 0.608 0.348 0.627 0.364 0.077

0.203 0.723 0.546 0.783 0.558 0.015

0.542 0.615 0.658 0.598 0.591 0.364

0.473 0.655 0.765 0.673 0.586 0.315

2.36 2.52 2.23 2.45 2.30 3.02

2.63 2.06 1.48 1.98 2.36 3.48

4S

0.326 0.629 0.332 0.614 0.338 0.079

0.196 0.710 0.326 0.748 0.541 0.030

0.612 0.661 0.676 0.550 0.656 0.547

0.470 0.667 0.668 0.687 0.515 0.267

2.36 2.32 2.23 2.62 2.30 2.94

2.70 2.00 1.99 2.11 2.52 3.51

B.2 Per-family and prompt-effect results The following tables provide additional per-family result summaries and four-shot effect analyses for each reported solver model and prompt regime.

21

Al Bateh and Saab

Table 8: Full accuracy and average-request breakdown by difficulty level. This table is the numeric counterpart to Figure 4. Accuracy Agent A Model gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

Avg Requests

Format

D1

D2

D3

D1

D2

D3

ZS

0.312 0.671 0.470 0.723 0.448 0.044

0.220 0.720 0.506 0.766 0.546 0.026

0.191 0.684 0.499 0.731 0.530 0.026

1.90 1.70 1.56 1.62 1.76 2.16

2.56 2.24 1.70 2.16 2.34 3.47

3.39 2.78 1.86 2.72 3.10 4.76

4S

0.291 0.665 0.381 0.702 0.416 0.071

0.209 0.733 0.329 0.751 0.526 0.030

0.182 0.670 0.257 0.679 0.526 0.021

1.81 1.68 1.68 1.63 1.81 2.09

2.69 2.15 2.09 2.31 2.47 3.47

3.54 2.56 2.52 3.01 3.28 4.88

Table 9: Full accuracy split by difficulty level and family type. This table supports the discussion of how difficulty interacts with fixed-slot and variable-slot family structure. Type A Agent A Model gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

Type B

Format

D1

D2

D3

D1

D2

D3

ZS

0.338 0.512 0.277 0.519 0.308 0.069

0.380 0.650 0.390 0.690 0.400 0.080

0.365 0.690 0.400 0.705 0.400 0.085

0.302 0.735 0.548 0.805 0.505 0.034

0.156 0.748 0.552 0.796 0.604 0.004

0.122 0.678 0.538 0.742 0.582 0.002

4S

0.285 0.512 0.300 0.481 0.292 0.081

0.330 0.695 0.355 0.685 0.365 0.085

0.375 0.715 0.350 0.715 0.370 0.070

0.294 0.726 0.414 0.791 0.466 0.068

0.160 0.748 0.318 0.778 0.590 0.008

0.104 0.652 0.220 0.664 0.588 0.002

22

MIRA-Math

Table 10: Full request-quality breakdown by family type, including hit rate, first-request success, and average request count. This table supports the claim that request acquisition and final-answer accuracy capture different failure modes. Hit Rate Agent A Model gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst gpt-4o-mini gpt-5.1 gemini-2.5-flash Gemma-4 Grok-4.20 Llama-3.1-8B-Inst

Format

1st-Req OK

Avg Requests

Type A

Type B

Type A

Type B

Type A

Type B

ZS

0.653 0.659 0.704 0.659 0.681 0.498

0.573 0.715 0.829 0.745 0.653 0.407

0.542 0.615 0.658 0.598 0.591 0.364

0.473 0.655 0.765 0.673 0.586 0.315

2.36 2.52 2.23 2.45 2.30 3.02

2.63 2.06 1.48 1.98 2.36 3.48

4S

0.671 0.702 0.723 0.618 0.704 0.587

0.577 0.726 0.722 0.737 0.606 0.396

0.612 0.661 0.676 0.550 0.656 0.547

0.470 0.667 0.668 0.687 0.515 0.267

2.36 2.32 2.23 2.62 2.30 2.94

2.70 2.00 1.99 2.11 2.52 3.51

23

Al Bateh and Saab

Table 11: Per-family accuracy on the 20/50 typed dataset (gpt-4o-mini, 2310 instances). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.750 0.000 0.033 0.000 0.933 0.167 0.000 0.000 0.950 0.133 0.983

1.000 1.000 0.500 0.000 0.283 0.967 0.000 0.650 1.000 0.350 1.000

0.800 0.000 0.067 0.000 0.783 0.133 0.000 0.033 0.800 0.033 0.933

1.000 1.000 0.567 0.000 0.350 1.000 0.000 0.967 1.000 0.300 1.000

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.227 0.560 0.127 0.080 0.593 0.000 0.093 0.153 0.081 0.153 0.167

0.387 1.000 0.127 0.013 0.547 0.020 0.007 0.207 0.819 0.960 1.000

0.280 0.540 0.127 0.147 0.593 0.000 0.140 0.100 0.027 0.141 0.060

0.400 0.993 0.080 0.113 0.300 0.007 0.047 0.193 0.780 0.832 0.993

24

MIRA-Math

Table 12: Full per-family metrics (gpt-4o-mini, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.750 0.000 0.033 0.000 0.933 0.167 0.000 0.000 0.950 0.133 0.983

1.000 1.000 0.597 0.003 0.421 0.983 0.000 0.650 1.000 0.419 1.000

1.000 1.000 0.500 0.000 0.283 0.967 0.000 0.650 1.000 0.350 1.000

1.00 1.00 2.17 6.00 2.72 1.02 6.00 1.35 1.00 2.68 1.00

0.00 0.00 1.18 5.98 1.72 0.02 6.00 0.35 0.00 1.78 0.00

1.00 1.00 0.78 0.02 1.00 1.00 0.00 0.65 1.00 0.62 1.00

345 312 465 826 607 311 908 497 342 555 358

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 149 150 150

0.227 0.560 0.127 0.080 0.593 0.000 0.093 0.153 0.081 0.153 0.167

0.405 1.000 0.133 0.087 0.580 0.102 0.036 0.359 0.784 0.960 1.000

0.387 1.000 0.127 0.013 0.547 0.020 0.007 0.207 0.819 0.960 1.000

2.79 1.00 4.23 2.91 2.25 4.41 4.32 3.04 1.96 1.04 1.00

1.95 0.00 3.60 2.73 1.33 3.72 3.83 2.21 0.97 0.04 0.00

0.65 1.00 0.34 0.16 0.77 0.53 0.36 0.63 0.61 0.96 1.00

566 489 560 424 555 597 688 559 389 535 548

Table 13: Full per-family metrics (gpt-4o-mini, four-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.800 0.000 0.067 0.000 0.783 0.133 0.000 0.033 0.800 0.033 0.933

1.000 1.000 0.658 0.010 0.503 1.000 0.000 0.983 1.000 0.228 1.000

1.000 1.000 0.517 0.000 0.233 1.000 0.000 0.967 1.000 0.017 1.000

1.00 1.00 2.07 5.97 2.45 1.00 6.00 1.03 1.00 3.43 1.00

0.00 0.00 1.22 5.93 1.45 0.00 6.00 0.03 0.00 2.72 0.00

1.00 1.00 0.85 0.03 1.00 1.00 0.00 1.00 1.00 0.72 1.00

248 271 358 747 625 258 725 307 185 778 341

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.280 0.540 0.127 0.147 0.593 0.000 0.140 0.100 0.027 0.140 0.060

0.544 0.997 0.206 0.190 0.640 0.176 0.401 0.529 0.728 0.939 0.996

0.353 0.993 0.093 0.107 0.533 0.013 0.260 0.313 0.607 0.900 0.993

2.81 1.01 4.54 2.69 2.69 4.87 4.17 2.87 1.83 1.16 1.01

1.89 0.01 4.01 2.38 1.85 4.20 3.53 2.00 0.89 0.17 0.01

0.91 1.00 0.53 0.31 0.85 0.67 0.64 0.87 0.95 0.99 1.00

580 298 736 486 513 908 629 572 392 314 234

25

Al Bateh and Saab

Table 14: Four-shot effect: accuracy change per family (gpt-4o-mini, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Positive values indicate improvement. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.750 0.000 0.033 0.000 0.933 0.167 0.000 0.000 0.950 0.133 0.983

0.800 0.000 0.067 0.000 0.783 0.133 0.000 0.033 0.800 0.033 0.933

+0.050 0.000 +0.033 0.000 −0.150 −0.033 0.000 +0.033 −0.150 −0.100 −0.050

1.000 1.000 0.500 0.000 0.283 0.967 0.000 0.650 1.000 0.350 1.000

1.000 1.000 0.567 0.000 0.350 1.000 0.000 0.967 1.000 0.300 1.000

0.000 0.000 +0.067 0.000 +0.067 +0.033 0.000 +0.317 0.000 −0.050 0.000

Type A average

660

0.359

0.326

−0.033

0.542

0.612

+0.070

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.227 0.560 0.127 0.080 0.593 0.000 0.093 0.153 0.081 0.153 0.167

0.280 0.540 0.127 0.147 0.593 0.000 0.140 0.100 0.027 0.141 0.060

+0.053 −0.020 0.000 +0.067 0.000 0.000 +0.047 −0.053 −0.054 −0.013 −0.107

0.387 1.000 0.127 0.013 0.547 0.020 0.007 0.207 0.819 0.960 1.000

0.400 0.993 0.080 0.113 0.300 0.007 0.047 0.193 0.780 0.832 0.993

+0.013 −0.007 −0.047 +0.100 −0.247 −0.013 +0.040 −0.013 −0.039 −0.128 −0.007

Type B average

1650

0.203

0.196

−0.007

0.473

0.470

−0.003

Overall

2310

0.248

0.233

−0.015

0.493

0.510

+0.017

B

26

MIRA-Math

Table 15: Per-family accuracy on the 20/50 typed dataset (gpt-5.1). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.117 0.017 1.000 0.700 0.000 0.933 1.000 0.917 1.000

1.000 1.000 0.067 0.017 0.133 0.667 0.000 1.000 1.000 0.883 1.000

1.000 0.000 0.317 0.050 1.000 0.717 0.000 0.983 1.000 0.850 1.000

1.000 1.000 0.267 0.083 1.000 0.683 0.017 1.000 1.000 0.217 1.000

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.153 0.567 0.933 0.227 0.693 0.833 0.807 0.960 0.800

0.967 1.000 0.033 0.233 0.853 0.533 0.520 0.640 0.467 0.980 0.973

0.953 0.980 0.153 0.513 0.960 0.227 0.573 0.847 0.847 0.973 0.787

0.920 1.000 0.107 0.173 0.927 0.520 0.427 0.647 0.660 0.993 0.967

27

Al Bateh and Saab

Table 16: Full per-family metrics (gpt-5.1, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.117 0.017 1.000 0.700 0.000 0.933 1.000 0.917 1.000

1.000 1.000 0.166 0.031 0.425 0.718 0.002 1.000 1.000 0.903 1.000

1.000 1.000 0.067 0.017 0.133 0.667 0.000 1.000 1.000 0.883 1.000

1.00 1.00 4.72 5.75 2.72 2.22 5.97 1.00 1.00 1.37 1.00

0.00 0.00 4.37 5.70 1.72 1.37 5.95 0.00 0.00 0.43 0.00

1.00 1.00 0.35 0.05 1.00 0.85 0.02 1.00 1.00 0.93 1.00

392 791 1161 1211 1035 774 1154 957 451 998 619

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.153 0.567 0.933 0.227 0.693 0.833 0.807 0.960 0.800

0.974 1.000 0.066 0.366 0.875 0.607 0.622 0.810 0.604 0.986 0.982

0.967 1.000 0.033 0.233 0.853 0.533 0.520 0.640 0.467 0.980 0.973

1.09 1.00 5.59 2.41 1.52 2.62 2.35 1.47 2.47 1.05 1.07

0.10 0.00 5.40 1.84 0.59 1.85 1.55 0.48 1.61 0.06 0.07

0.99 1.00 0.19 0.57 0.93 0.77 0.79 0.99 0.85 0.99 1.00

804 561 1250 847 675 1470 1513 1752 1486 732 1501

Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.317 0.050 1.000 0.717 0.000 0.983 1.000 0.850 1.000

1.000 1.000 0.349 0.094 1.000 0.708 0.022 1.000 1.000 0.548 1.000

1.000 1.000 0.267 0.083 1.000 0.683 0.017 1.000 1.000 0.217 1.000

1.00 1.00 3.87 5.37 1.00 2.33 5.87 1.00 1.00 2.07 1.00

0.00 0.00 3.32 5.25 0.00 1.57 5.82 0.00 0.00 1.17 0.00

1.00 1.00 0.55 0.12 1.00 0.77 0.05 1.00 1.00 0.90 1.00

404 783 1084 1149 676 721 1165 1081 453 1149 529

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.953 0.980 0.153 0.513 0.960 0.227 0.573 0.847 0.847 0.973 0.787

0.939 1.000 0.131 0.316 0.938 0.594 0.531 0.818 0.740 0.996 0.981

0.920 1.000 0.107 0.173 0.927 0.513 0.427 0.647 0.660 0.993 0.967

1.23 1.00 5.30 2.48 1.31 2.59 2.72 1.42 1.87 1.01 1.05

0.27 0.00 5.07 1.97 0.35 1.83 2.01 0.43 0.98 0.01 0.05

0.96 1.00 0.23 0.51 0.96 0.76 0.71 0.99 0.89 1.00 1.00

816 554 1223 828 624 1421 1421 1622 1338 743 1337

Table 17: Full per-family metrics (gpt-5.1, four-shot, 20/50 typed dataset).

28

MIRA-Math

Table 18: Four-shot effect: accuracy change per family (gpt-5.1, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.117 0.017 1.000 0.700 0.000 0.933 1.000 0.917 1.000

1.000 0.000 0.317 0.050 1.000 0.717 0.000 0.983 1.000 0.850 1.000

0.000 0.000 +0.200 +0.033 0.000 +0.017 0.000 +0.050 0.000 −0.067 0.000

1.000 1.000 0.067 0.017 0.133 0.667 0.000 1.000 1.000 0.883 1.000

1.000 1.000 0.267 0.083 1.000 0.683 0.017 1.000 1.000 0.217 1.000

0.000 0.000 +0.200 +0.067 +0.867 +0.017 +0.017 0.000 0.000 −0.667 0.000

Type A average

660

0.608

0.629

+0.021

0.615

0.661

+0.046

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.153 0.567 0.933 0.227 0.693 0.833 0.807 0.960 0.800

0.953 0.980 0.153 0.513 0.960 0.227 0.573 0.847 0.847 0.973 0.787

−0.027 −0.020 0.000 −0.053 +0.027 0.000 −0.120 +0.013 +0.040 +0.013 −0.013

0.967 1.000 0.033 0.233 0.853 0.533 0.520 0.640 0.467 0.980 0.973

0.920 1.000 0.107 0.173 0.927 0.520 0.427 0.647 0.660 0.993 0.967

−0.047 0.000 +0.073 −0.060 +0.073 −0.013 −0.093 +0.007 +0.193 +0.013 −0.007

Type B average

1650

0.723

0.710

−0.013

0.655

0.667

+0.013

Overall

2310

0.690

0.687

−0.003

0.643

0.665

+0.022

B

29

Al Bateh and Saab

Table 19: Per-family accuracy on the 20/50 typed dataset (gemini-2.5-flash, 2310 instances). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.983 0.000 0.633 0.000 0.000 0.233 0.000 0.000 1.000 0.000 0.983

1.000 1.000 0.517 0.267 0.033 0.417 0.033 0.983 1.000 0.983 1.000

0.867 0.000 0.300 0.000 0.000 0.567 0.000 0.000 0.950 0.000 0.967

1.000 1.000 0.550 0.050 0.050 0.850 0.017 1.000 1.000 0.917 1.000

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.640 0.827 0.000 0.473 0.940 0.067 0.193 0.393 0.873 0.860 0.740

0.987 1.000 0.240 0.220 0.980 0.540 0.867 0.607 0.980 1.000 1.000

0.267 0.573 0.000 0.300 0.667 0.000 0.107 0.427 0.213 0.547 0.487

0.807 0.993 0.027 0.113 0.840 0.527 0.513 0.567 0.967 1.000 0.993

30

MIRA-Math

Table 20: Full per-family metrics (gemini-2.5-flash, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.983 0.000 0.633 0.000 0.000 0.233 0.000 0.000 1.000 0.000 0.983

1.000 1.000 0.617 0.267 0.356 0.467 0.054 0.992 1.000 0.988 1.000

1.000 1.000 0.517 0.267 0.033 0.417 0.033 0.983 1.000 0.983 1.000

1.00 1.00 2.37 4.52 2.93 2.87 5.78 1.02 1.00 1.05 1.00

0.00 0.00 1.60 4.25 1.93 2.25 5.68 0.02 0.00 0.05 0.00

1.00 1.00 0.77 0.27 1.00 0.62 0.10 1.00 1.00 1.00 1.00

285 335 424 641 516 445 671 762 216 497 273

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.640 0.827 0.000 0.473 0.940 0.067 0.193 0.393 0.873 0.860 0.740

0.990 1.000 0.426 0.381 0.988 0.617 0.928 0.800 0.986 1.000 1.000

0.987 1.000 0.240 0.220 0.980 0.540 0.867 0.607 0.980 1.000 1.000

1.02 1.00 2.91 2.34 1.03 2.39 1.17 1.38 1.03 1.00 1.00

0.03 0.00 2.24 1.75 0.03 1.67 0.17 0.39 0.04 0.00 0.00

0.99 1.00 0.67 0.59 1.00 0.72 1.00 0.99 0.99 1.00 1.00

451 311 487 540 245 1048 617 830 671 441 547

Table 21: Full per-family metrics (gemini-2.5-flash, four-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.867 0.000 0.300 0.000 0.000 0.567 0.000 0.000 0.950 0.000 0.967

1.000 1.000 0.699 0.050 0.354 0.879 0.017 1.000 1.000 0.953 1.000

1.000 1.000 0.550 0.050 0.050 0.850 0.017 1.000 1.000 0.917 1.000

1.00 1.00 2.22 5.75 2.92 1.55 5.97 1.00 1.00 1.15 1.00

0.00 0.00 1.33 5.70 1.95 0.57 5.95 0.00 0.00 0.15 0.00

1.00 1.00 0.88 0.05 0.97 0.98 0.02 1.00 1.00 1.00 1.00

261 501 406 658 544 268 676 894 223 567 270

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.267 0.573 0.000 0.300 0.667 0.000 0.107 0.427 0.213 0.547 0.487

0.884 0.997 0.071 0.210 0.893 0.578 0.563 0.773 0.973 1.000 0.997

0.807 0.993 0.027 0.113 0.840 0.527 0.513 0.567 0.967 1.000 0.993

1.27 1.01 5.43 2.63 1.46 2.78 2.73 1.48 1.09 1.00 1.01

0.29 0.01 5.15 2.30 0.51 2.12 2.08 0.49 0.11 0.00 0.01

0.98 1.00 0.28 0.33 0.95 0.66 0.65 0.99 0.98 1.00 1.00

476 289 724 422 332 1813 1149 1039 887 470 634

31

Al Bateh and Saab

Table 22: Four-shot effect: accuracy change per family (gemini-2.5-flash, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Positive values indicate improvement. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.983 0.000 0.633 0.000 0.000 0.233 0.000 0.000 1.000 0.000 0.983

0.867 0.000 0.300 0.000 0.000 0.567 0.000 0.000 0.950 0.000 0.967

−0.117 0.000 −0.333 0.000 0.000 +0.333 0.000 0.000 −0.050 0.000 −0.017

1.000 1.000 0.517 0.267 0.033 0.417 0.033 0.983 1.000 0.983 1.000

1.000 1.000 0.550 0.050 0.050 0.850 0.017 1.000 1.000 0.917 1.000

0.000 0.000 +0.033 −0.217 +0.017 +0.433 −0.017 +0.017 0.000 −0.067 0.000

Type A average

660

0.348

0.332

−0.017

0.658

0.676

+0.018

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.640 0.827 0.000 0.473 0.940 0.067 0.193 0.393 0.873 0.860 0.740

0.267 0.573 0.000 0.300 0.667 0.000 0.107 0.427 0.213 0.547 0.487

−0.373 −0.253 0.000 −0.173 −0.273 −0.067 −0.087 +0.033 −0.660 −0.313 −0.253

0.987 1.000 0.240 0.220 0.980 0.540 0.867 0.607 0.980 1.000 1.000

0.807 0.993 0.027 0.113 0.840 0.527 0.513 0.567 0.967 1.000 0.993

−0.180 −0.007 −0.213 −0.107 −0.140 −0.013 −0.353 −0.040 −0.013 0.000 −0.007

Type B average

1650

0.546

0.326

−0.220

0.765

0.668

−0.098

Overall

2310

0.490

0.328

−0.162

0.735

0.670

−0.065

B

32

MIRA-Math

Table 23: Per-family accuracy on the 20/50 typed dataset (Gemma-4, 2310 instances). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.200 0.000 1.000 0.950 0.017 0.850 1.000 0.883 1.000

1.000 1.000 0.083 0.000 0.067 0.883 0.200 1.000 1.000 0.350 1.000

0.983 0.000 0.133 0.000 0.950 0.867 0.033 0.983 1.000 0.817 0.983

1.000 1.000 0.117 0.000 0.367 0.583 0.000 0.983 1.000 0.000 1.000

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.413 0.947 0.887 0.147 0.773 0.893 0.700 0.960 0.913

0.947 1.000 0.233 0.620 0.853 0.320 0.647 0.607 0.200 1.000 0.973

0.920 0.987 0.320 0.927 0.847 0.107 0.667 0.873 0.773 0.980 0.833

0.907 1.000 0.060 0.580 0.800 0.247 0.553 0.740 0.727 1.000 0.940

33

Al Bateh and Saab

Table 24: Full per-family metrics (Gemma-4, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.200 0.000 1.000 0.950 0.017 0.850 1.000 0.883 1.000

1.000 1.000 0.105 0.000 0.408 0.907 0.214 1.000 1.000 0.617 1.000

1.000 1.000 0.083 0.000 0.067 0.883 0.200 1.000 1.000 0.350 1.000

1.00 1.00 5.18 6.00 2.68 1.30 4.72 1.00 1.00 2.07 1.00

0.00 0.00 4.98 6.00 1.68 0.33 4.47 0.00 0.00 1.18 0.00

1.00 1.00 0.20 0.00 1.00 0.97 0.25 1.00 1.00 0.88 1.00

314 460 893 2019 588 480 1771 712 288 740 323

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.413 0.947 0.887 0.147 0.773 0.893 0.700 0.960 0.913

0.963 1.000 0.299 0.764 0.874 0.336 0.740 0.802 0.424 1.000 0.987

0.947 1.000 0.233 0.620 0.853 0.320 0.647 0.607 0.200 1.000 0.973

1.09 1.00 3.85 1.55 1.59 4.39 1.94 1.40 2.97 1.00 1.03

0.11 0.00 3.42 0.60 0.69 4.03 1.11 0.40 2.21 0.00 0.03

0.98 1.00 0.43 0.95 0.91 0.36 0.83 1.00 0.75 1.00 1.00

561 404 2866 839 633 3704 1044 914 1082 605 844

Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.983 0.000 0.133 0.000 0.950 0.867 0.033 0.983 1.000 0.817 0.983

1.000 1.000 0.121 0.000 0.600 0.655 0.017 0.992 1.000 0.414 1.000

1.000 1.000 0.117 0.000 0.367 0.583 0.000 0.983 1.000 0.000 1.000

1.00 1.00 5.18 6.00 2.13 2.42 5.78 1.02 1.00 2.33 1.00

0.00 0.00 5.05 6.00 1.13 1.52 5.72 0.02 0.00 1.50 0.00

1.00 1.00 0.13 0.00 1.00 0.90 0.07 1.00 1.00 0.83 1.00

309 455 816 1953 534 746 2049 850 265 659 312

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.920 0.987 0.320 0.927 0.847 0.107 0.667 0.873 0.773 0.980 0.833

0.916 1.000 0.145 0.734 0.818 0.256 0.629 0.867 0.775 1.000 0.969

0.907 1.000 0.060 0.580 0.800 0.247 0.553 0.740 0.727 1.000 0.940

1.47 1.00 4.83 1.62 1.88 4.63 2.62 1.29 1.81 1.00 1.07

0.54 0.00 4.49 0.69 1.02 4.35 1.91 0.29 0.97 0.00 0.07

0.93 1.00 0.34 0.93 0.86 0.27 0.71 1.00 0.84 1.00 1.00

700 406 2986 802 728 3385 1327 930 783 571 848

Table 25: Full per-family metrics (Gemma-4, four-shot, 20/50 typed dataset).

34

MIRA-Math

Table 26: Four-shot effect: accuracy change per family (Gemma-4, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Positive values indicate improvement. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

1.000 0.000 0.200 0.000 1.000 0.950 0.017 0.850 1.000 0.883 1.000

0.983 0.000 0.133 0.000 0.950 0.867 0.033 0.983 1.000 0.817 0.983

−0.017 0.000 −0.067 0.000 −0.050 −0.083 +0.017 +0.133 0.000 −0.067 −0.017

1.000 1.000 0.083 0.000 0.067 0.883 0.200 1.000 1.000 0.350 1.000

1.000 1.000 0.117 0.000 0.367 0.583 0.000 0.983 1.000 0.000 1.000

0.000 0.000 +0.033 0.000 +0.300 −0.300 −0.200 −0.017 0.000 −0.350 0.000

Type A average

660

0.627

0.614

−0.014

0.598

0.550

−0.048

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.980 1.000 0.413 0.947 0.887 0.147 0.773 0.893 0.700 0.960 0.913

0.920 0.987 0.320 0.927 0.847 0.107 0.667 0.873 0.773 0.980 0.833

−0.060 −0.013 −0.093 −0.020 −0.040 −0.040 −0.107 −0.020 +0.073 +0.020 −0.080

0.947 1.000 0.233 0.620 0.853 0.320 0.647 0.607 0.200 1.000 0.973

0.907 1.000 0.060 0.580 0.800 0.247 0.553 0.740 0.727 1.000 0.940

−0.040 0.000 −0.173 −0.040 −0.053 −0.073 −0.093 +0.133 +0.527 0.000 −0.033

Type B average

1650

0.783

0.748

−0.035

0.673

0.687

+0.014

Overall

2310

0.739

0.710

−0.029

0.652

0.648

−0.004

B

35

Al Bateh and Saab

Table 27: Per-family accuracy on the 20/50 typed dataset (Grok-4.20, 2310 instances). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.900 0.000 0.217 0.000 0.000 0.283 0.000 0.567 1.000 0.233 0.800

1.000 1.000 0.133 0.117 0.317 0.933 0.000 1.000 1.000 0.000 1.000

0.967 0.017 0.150 0.017 0.050 0.317 0.000 0.467 0.717 0.167 0.850

1.000 1.000 0.217 0.083 0.983 0.917 0.017 1.000 1.000 0.000 1.000

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.893 0.993 0.047 0.107 0.527 0.080 0.653 0.807 0.420 0.920 0.693

0.660 1.000 0.093 0.167 0.787 0.213 0.293 0.527 0.713 0.993 1.000

0.833 0.960 0.020 0.073 0.640 0.120 0.633 0.707 0.327 0.920 0.713

0.373 1.000 0.093 0.020 0.640 0.267 0.233 0.573 0.500 0.987 0.980

36

MIRA-Math

Table 28: Full per-family metrics (Grok-4.20, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.900 0.000 0.217 0.000 0.000 0.283 0.000 0.567 1.000 0.233 0.800

1.000 1.000 0.467 0.119 0.553 0.938 0.000 1.000 1.000 0.419 1.000

1.000 1.000 0.133 0.117 0.317 0.933 0.000 1.000 1.000 0.000 1.000

1.00 1.00 2.67 5.18 2.32 1.25 6.00 1.00 1.00 2.90 1.00

0.00 0.00 1.87 5.05 1.32 0.30 6.00 0.00 0.00 2.00 0.00

1.00 1.00 0.80 0.13 1.00 0.95 0.00 1.00 1.00 0.90 1.00

267 401 5595 2802 636 330 1791 1563 288 961 325

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.893 0.993 0.047 0.107 0.527 0.080 0.653 0.807 0.420 0.920 0.693

0.765 1.000 0.104 0.220 0.810 0.325 0.429 0.727 0.807 0.995 1.000

0.660 1.000 0.093 0.167 0.787 0.213 0.293 0.527 0.713 0.993 1.000

1.74 1.00 5.35 2.61 2.04 4.20 3.56 1.82 1.60 1.03 1.00

0.79 0.00 5.23 2.31 1.18 3.71 2.88 0.87 0.64 0.03 0.00

0.95 1.00 0.13 0.30 0.86 0.49 0.68 0.95 0.96 1.00 1.00

1304 478 3273 697 1602 3024 2460 2088 892 594 1262

Table 29: Full per-family metrics (Grok-4.20, four-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.967 0.017 0.150 0.017 0.050 0.317 0.000 0.467 0.717 0.167 0.850

1.000 1.000 0.567 0.103 0.989 0.929 0.028 1.000 1.000 0.128 1.000

1.000 1.000 0.217 0.083 0.983 0.917 0.017 1.000 1.000 0.000 1.000

1.00 1.00 2.27 5.32 1.03 1.37 5.87 1.00 1.00 4.43 1.00

0.00 0.00 1.35 5.18 0.03 0.42 5.82 0.00 0.00 3.83 0.00

1.00 1.00 0.92 0.13 1.00 0.95 0.05 1.00 1.00 0.60 1.00

266 433 1563 3235 475 485 2008 1039 305 945 383

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.833 0.960 0.020 0.073 0.640 0.120 0.633 0.707 0.327 0.920 0.713

0.583 1.000 0.093 0.081 0.711 0.373 0.403 0.761 0.675 0.990 0.990

0.373 1.000 0.093 0.020 0.640 0.267 0.233 0.573 0.500 0.987 0.980

2.32 1.00 5.51 2.90 2.56 4.21 3.53 1.66 1.95 1.04 1.02

1.43 0.00 5.42 2.73 1.73 3.69 2.85 0.69 1.02 0.04 0.02

0.89 1.00 0.09 0.17 0.83 0.52 0.68 0.97 0.93 1.00 1.00

2332 481 1937 333 2335 3747 2720 1923 1027 594 1424

37

Al Bateh and Saab

Table 30: Four-shot effect: accuracy change per family (Grok-4.20, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Positive values indicate improvement. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.900 0.000 0.217 0.000 0.000 0.283 0.000 0.567 1.000 0.233 0.800

0.967 0.017 0.150 0.017 0.050 0.317 0.000 0.467 0.717 0.167 0.850

+0.067 +0.017 −0.067 +0.017 +0.050 +0.033 0.000 −0.100 −0.283 −0.067 +0.050

1.000 1.000 0.133 0.117 0.317 0.933 0.000 1.000 1.000 0.000 1.000

1.000 1.000 0.217 0.083 0.983 0.917 0.017 1.000 1.000 0.000 1.000

0.000 0.000 +0.083 −0.033 +0.667 −0.017 +0.017 0.000 0.000 0.000 0.000

Type A average

660

0.364

0.338

−0.026

0.591

0.656

+0.065

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.893 0.993 0.047 0.107 0.527 0.080 0.653 0.807 0.420 0.920 0.693

0.833 0.960 0.020 0.073 0.640 0.120 0.633 0.707 0.327 0.920 0.713

−0.060 −0.033 −0.027 −0.033 +0.113 +0.040 −0.020 −0.100 −0.093 0.000 +0.020

0.660 1.000 0.093 0.167 0.787 0.213 0.293 0.527 0.713 0.993 1.000

0.373 1.000 0.093 0.020 0.640 0.267 0.233 0.573 0.500 0.987 0.980

−0.287 0.000 0.000 −0.147 −0.147 +0.053 −0.060 +0.047 −0.213 −0.007 −0.020

Type B average

1650

0.558

0.541

−0.018

0.586

0.515

−0.071

Overall

2310

0.503

0.483

−0.020

0.587

0.555

−0.032

B

38

MIRA-Math

Table 31: Per-family accuracy on the 20/50 typed dataset (Llama-3.1-8B-Instruct, 2310 instances). Families sorted by Type then name. n = instances per family. zero-shot

four-shot

Type

Family

n

Acc

1st-Req

Acc

1st-Req

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.017 0.000 0.133 0.000 0.050 0.183 0.000 0.000 0.450 0.000 0.017

1.000 0.983 0.017 0.000 0.000 0.000 0.000 0.000 1.000 0.000 1.000

0.033 0.000 0.133 0.000 0.017 0.000 0.000 0.067 0.600 0.000 0.017

1.000 1.000 0.083 0.000 0.967 0.000 0.000 1.000 1.000 0.000 0.967

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.000 0.007 0.000 0.093 0.053 0.000 0.000 0.000 0.000 0.007 0.007

0.200 1.000 0.000 0.000 0.193 0.040 0.000 0.060 0.307 0.733 0.933

0.007 0.027 0.000 0.180 0.053 0.000 0.007 0.013 0.013 0.027 0.000

0.140 0.460 0.000 0.113 0.280 0.013 0.113 0.053 0.253 0.627 0.880

39

Al Bateh and Saab

Table 32: Full per-family metrics (Llama-3.1-8B-Instruct, zero-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.017 0.000 0.133 0.000 0.050 0.183 0.000 0.000 0.450 0.000 0.017

1.000 0.983 0.447 0.000 0.406 0.500 0.000 0.138 1.000 0.002 1.000

1.000 0.983 0.017 0.000 0.000 0.000 0.000 0.000 1.000 0.000 1.000

1.00 1.03 2.35 6.00 2.57 2.00 6.00 4.23 1.00 6.00 1.00

0.00 0.05 1.35 6.00 1.57 1.00 6.00 3.57 0.00 5.98 0.00

1.00 0.98 1.00 0.00 1.00 1.00 0.00 0.67 1.00 0.02 1.00

154 251 209 167 235 118 167 173 179 169 160

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.000 0.007 0.000 0.093 0.053 0.000 0.000 0.000 0.000 0.007 0.007

0.409 1.000 0.001 0.057 0.375 0.111 0.086 0.146 0.491 0.838 0.966

0.200 1.000 0.000 0.000 0.193 0.040 0.000 0.060 0.307 0.733 0.933

3.45 1.00 5.97 2.91 3.50 5.51 5.41 4.94 2.97 1.52 1.07

2.64 0.00 5.97 2.79 2.79 5.25 5.17 4.55 2.22 0.52 0.07

0.81 1.00 0.01 0.13 0.71 0.26 0.23 0.39 0.75 1.00 1.00

190 92 167 96 184 191 203 190 156 134 145

Table 33: Full per-family metrics (Llama-3.1-8B-Instruct, four-shot, 20/50 typed dataset). Type

Family

n

Acc

Hit Rate

1st-Req

Avg Req

Avg Decl

Avg Hints

Avg Tok

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.033 0.000 0.133 0.000 0.017 0.000 0.000 0.067 0.600 0.000 0.017

1.000 1.000 0.478 0.000 0.983 0.000 0.000 1.000 1.000 0.008 0.983

1.000 1.000 0.083 0.000 0.967 0.000 0.000 1.000 1.000 0.000 0.967

1.00 1.00 2.30 6.00 1.03 6.00 6.00 1.00 1.00 5.98 1.03

0.00 0.00 1.30 6.00 0.03 6.00 6.00 0.00 0.00 5.93 0.03

1.00 1.00 1.00 0.00 1.00 0.00 0.00 1.00 1.00 0.05 1.00

117 192 207 197 195 167 167 175 182 172 172

B

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.007 0.027 0.000 0.180 0.053 0.000 0.007 0.013 0.013 0.027 0.000

0.337 0.729 0.000 0.183 0.448 0.101 0.283 0.144 0.439 0.765 0.926

0.140 0.460 0.000 0.113 0.280 0.013 0.113 0.053 0.253 0.627 0.880

3.83 1.55 6.00 2.70 3.31 5.33 4.33 5.04 3.39 1.85 1.32

3.05 0.55 6.00 2.41 2.56 5.06 3.70 4.64 2.67 0.86 0.34

0.79 1.00 0.00 0.29 0.75 0.27 0.63 0.40 0.71 0.99 0.98

212 174 166 108 176 193 200 195 169 167 127

40

MIRA-Math

Table 34: Four-shot effect: accuracy change per family (Llama-3.1-8B-Instruct, 20/50 typed dataset). ∆ = four-shot accuracy − baseline accuracy. Positive values indicate improvement. Accuracy

1st-Request Success

Type

Family

n

Base

4S

∆

Base

4S

∆

A

bayes missing prior crt reconstruction geometry coordinates graph path sums linear system separator matrix completion moment problem phase retrieval piecewise missing thresh. rankdef linear shared recurrence missing init

60 60 60 60 60 60 60 60 60 60 60

0.017 0.000 0.133 0.000 0.050 0.183 0.000 0.000 0.450 0.000 0.017

0.033 0.000 0.133 0.000 0.017 0.000 0.000 0.067 0.600 0.000 0.017

+0.017 0.000 0.000 0.000 −0.033 −0.183 0.000 +0.067 +0.150 0.000 0.000

1.000 0.983 0.017 0.000 0.000 0.000 0.000 0.000 1.000 0.000 1.000

1.000 1.000 0.083 0.000 0.967 0.000 0.000 1.000 1.000 0.000 0.967

0.000 +0.017 +0.067 0.000 +0.967 0.000 0.000 +1.000 0.000 0.000 −0.033

Type A average

660

0.077

0.079

+0.002

0.364

0.547

+0.183

birth death missing rate circuit missing resistance deconvolution discrete tomography eigenvector missing entry laplace grid linear sys missing coeff markov missing transition poly interpolation portfolio var missing c. steady state missing em.

150 150 150 150 150 150 150 150 150 150 150

0.000 0.007 0.000 0.093 0.053 0.000 0.000 0.000 0.000 0.007 0.007

0.007 0.027 0.000 0.180 0.053 0.000 0.007 0.013 0.013 0.027 0.000

+0.007 +0.020 0.000 +0.087 0.000 0.000 +0.007 +0.013 +0.013 +0.020 −0.007

0.200 1.000 0.000 0.000 0.193 0.040 0.000 0.060 0.307 0.733 0.933

0.140 0.460 0.000 0.113 0.280 0.013 0.113 0.053 0.253 0.627 0.880

−0.060 −0.540 0.000 +0.113 +0.087 −0.027 +0.113 −0.007 −0.053 −0.107 −0.053

Type B average

1650

0.015

0.030

+0.015

0.315

0.267

−0.048

Overall

2310

0.033

0.044

+0.011

0.329

0.347

+0.018

B

41

Al Bateh and Saab

Appendix C. Problem Family Generators Each family below defines a parameterised class of two-agent reasoning problems. A generator samples an instance at random given a difficulty level d ∈ {1, 2, 3} and a random seed. For each instance, Agent A and Agent B each receive a disjoint subset of the constraints; neither subset alone determines the answer, but together they do, and exactly one atomic hint from B to A resolves A’s ambiguity. The 22 families are split into two types: • Type A — Fixed hint slot (11 families). The missing information is structurally determined by the family: it is always the same kind/position regardless of the specific instance. Agent A can predict what to request without scanning its constraints. • Type B — Variable hint slot (11 families). The missing slot varies per instance — it could be any one of multiple possible positions. Agent A must first identify which slot is absent before formulating its request.

Type A — Fixed Hint Slot

A1.

Bayesian Inference with Missing Prior (bayes missing prior)

Setup.

Three rational parameters are sampled: P (H) = ppnd ,

P (E | H) = ssnd ,

P (E | ¬H) = ffnd ,

with denominators drawn from Unif[2, 3+2d] and numerators chosen so all three lie strictly in (0, 1) and P (E | H) ̸= P (E | ¬H). The posterior is computed via Bayes’ rule: P (H | E) =

P (E | H) P (H) . P (E | H) P (H) + P (E | ¬H) (1 − P (H))

Instances where the posterior denominator exceeds 200 are rejected. Information split. • Agent A: the likelihood P (E | H) and the false-positive rate P (E | ¬H). • Agent B: the prior P (H). B’s atomic hint is always the prior P (H). Answer.

The posterior P (H | E) as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). Fraction denominators drawn from [2, 3 + 2d] (max 5, 7, 9), controlling numerical complexity. 42

MIRA-Math

A2.

Chinese Remainder Theorem Reconstruction (crt reconstruction)

Setup. Three distinct primes (m1 , m2 , m3 ) are drawn uniformly at random (without replacement) from the pool {3, 5, 7, 11, 13, 17, 19}. An integer x ∼ Unif[0, M ) is sampled, where M = m1 m2 m3 , and the three residues ri = x mod mi are computed. By the Chinese Remainder Theorem the solution in [0, M ) is unique. Information split. • Agent A: congruences x ≡ r1 (mod m1 ) and x ≡ r2 (mod m2 ). • Agent B: congruence x ≡ r3 (mod m3 ). B’s atomic hint is always the third congruence (m3 , r3 ). Answer.

The integer x.

Difficulty scaling. This family does not scale with difficulty: the prime pool and problem structure are identical for all d. Accordingly, in mix generation it is always run at d = 1 regardless of the requested difficulty. A3.

Coordinate Geometry with Missing Line (geometry coordinates)

Setup. An intersection point (x, y) is drawn uniformly from [−(4 + d), 4 + d]2 . Two non-parallel lines through (x, y) are generated with integer coefficients in [−(3 + d), 3 + d]: a1 x + b1 y = c1 ,

a2 x + b2 y = c2 ,

subject to a1 b2 − a2 b1 ̸= 0 (non-parallel). A reference point Q = (x + ∆x, y + ∆y) is placed at an integer Euclidean distance from (x, y) using p a Pythagorean pair (∆x, ∆y) ∈ {(3, 4), (4, 3), (5, 12), (12, 5)}. The target is d((x, y), Q) = ∆x2 + ∆y 2 , guaranteed to be an integer. Information split. • Agent A: the first line equation and the coordinates of Q. • Agent B: the second line equation and the coordinates of Q. B’s atomic hint is always the second line equation. Answer.

The integer distance d((x, y), Q).

Difficulty scaling (d ∈ {1, 2, 3}). • Intersection point drawn from [−(4 + d), 4 + d]2 (range ±5, ±6, ±7). • Line coefficients drawn from [−(3 + d), 3 + d] (range ±4, ±5, ±6). 43

Al Bateh and Saab

A4.

Triangle Path Sums (graph path sums)

Setup. A triangle graph has three edges with positive integer weights e01 , e12 , e20 ∼ Unif[1, 2 + d]. The three path sums (one per pair of edges sharing a vertex) are: s1 = e01 + e12 ,

s2 = e12 + e20 ,

s3 = e01 + e20 .

Any two path sums leave one degree of freedom, while all three path sums determine the three edge weights uniquely. Information split. • Agent A: path sums s1 and s2 . • Agent B: path sum s3 = e01 + e20 . B’s atomic hint is always the equation e01 + e20 = s3 . Answer.

The triple (e01 , e12 , e20 ).

Difficulty scaling (d ∈ {1, 2, 3}). A5.

Edge weights drawn from [1, 2 + d] (max 3, 4, 5).

Linear System with Separator Variable (linear system separator)

Setup. Unknowns (a, b, c) are drawn uniformly from [−(5 + d), 5 + d]3 . Two linearly independent equations in (a, b, c) are sampled with integer coefficients in [−(3 + d), 3 + d], subject to the constraint that, when c is substituted, the residual 2 × 2 system in (a, b) is uniquely solvable (i.e. the 2 × 2 submatrix on columns a, b has rank 2). Information split. • Agent A: the two linear equations. • Agent B: the exact value of c. B’s atomic hint is always c = value. Answer.

The triple (a, b, c).

Difficulty scaling (d ∈ {1, 2, 3}). • Solution drawn from [−(5 + d), 5 + d]3 (range ±6, ±7, ±8). • Coefficients drawn from [−(3 + d), 3 + d] (range ±4, ±5, ±6). A6.

Rank-1 Matrix Completion (matrix completion)

Setup. A 2 × 2 rank-1 matrix M = uv⊤ is constructed from vectors u = (u0 , u1 )⊤ and v = (v0 , v1 )⊤ with ui , vi ∼ Unif[1, 2 + d]:     u0 v0 u0 v1 a b M= = . u1 v0 u1 v1 c d By the rank-1 identity, m11 = m01 m10 /m00 (valid since m00 = u0 v0 ≥ 1). 44

MIRA-Math

Information split. • Agent A: entries m00 = a and m01 = b (first row). • Agent B: entry m10 = c (position [1, 0]). B’s atomic hint is always m10 = c. Answer.

The integer m11 = bc/a.

Difficulty scaling (d ∈ {1, 2, 3}). 5). A7.

Factor entries ui , vi drawn from [1, 2 + d] (max 3, 4,

Moment Problem on Three-Point Support (moment problem)

Setup. A discrete probability distribution on support {0, 1, 2} is specified by unnormalised weights (q0 , q1 , q2 ) summing to a common denominator D = 6+2d. Three linear constraints determine the distribution uniquely: (sum) q0 +q1 +q2 = D,

(1st moment) q1 +2q2 = m1 ,

(2nd moment) q1 +4q2 = m2 .

Values q0 , q1 ∼ Unif[1, D − 2] are drawn subject to q2 = D − q0 − q1 > 0, and m1 , m2 are derived accordingly. Information split. • Agent A: the sum constraint and the first-moment equation q1 + 2q2 = m1 . • Agent B: the sum constraint and the second-moment equation q1 + 4q2 = m2 . Each two-equation subsystem has rank 2 in three unknowns (ill-posed). B’s atomic hint is always the second-moment equation. Answer.

The triple (q0 , q1 , q2 ).

Difficulty scaling (d ∈ {1, 2, 3}). Common denominator D = 6 + 2d (values 8, 10, 12), controlling the magnitude of the weights and moments. A8.

Phase Retrieval via DFT Magnitudes (phase retrieval)

Setup. A signal x = (x0 , x1 , x2 , x3 ) ∈ Z4 is drawn with x0 ̸= 0 and |xi | ≤ 2 + min(d, 2). The magnitude-squared spectrum is computed using closed-form expressions for a length-4 real DFT: |x̂0 |2 = (x0 + x1 + x2 + x3 )2 , |x̂1 |2 = (x0 − x2 )2 + (x3 − x1 )2 , |x̂2 |2 = (x0 − x1 + x2 − x3 )2 , |x̂3 |2 = (x0 − x2 )2 + (x1 − x3 )2 . Note that |x̂1 |2 = |x̂3 |2 in this formulation. By rejection sampling, instances are kept only when (i) the magnitude vector alone admits multiple solutions (i.e. A is locally ill-posed), and (ii) the magnitude vector together with sign(x0 ) admits exactly one solution. 45

Al Bateh and Saab

Information split. • Agent A: the four magnitude-squared values (|x̂0 |2 , |x̂1 |2 , |x̂2 |2 , |x̂3 |2 ). • Agent B: the sign bit sign(x0 ) ∈ {−1, +1}. B’s atomic hint is always sign(x0 ). Answer.

The full signal (x0 , x1 , x2 , x3 ).

Difficulty scaling (d ∈ {1, 2, 3}). Signal value range |xi | ≤ 2 + min(d, 2) (bounds 3, 4, 4 — caps at d = 2, so d = 3 is identical to d = 2). A9. Piecewise Linear Function with Missing Threshold (piecewise missing threshold) Setup.

A two-branch piecewise linear function is defined: ( a0 x + b0 if x < t, f (x) = a1 x + b1 if x ≥ t,

with slopes ai ∼ Unif[−(2 + d), 2 + d], intercepts bi ∼ Unif[−(4 + 2d), 4 + 2d], query point x ∼ Unif[−(5 + 3d), 5 + 3d], and threshold t drawn uniformly from a symmetric interval, with x ̸= t enforced and the two branch values at x required to be distinct (so the active branch is not degenerate). Information split. • Agent A: the query point x and both branch formulas, but not t. • Agent B: the query point x and the threshold t. Without t, A cannot determine which branch is active. B’s atomic hint is always t. Answer.

The integer f (x).

Difficulty scaling (d ∈ {1, 2, 3}). • Slope range ±(2 + d) (max ±3, ±4, ±5). • Query and threshold range ±(5 + 3d) (max ±8, ±11, ±14). A10.

Rank-Deficient Linear System, Shared Variable (rankdef linear shared)

Setup. A solution (x, y, z) ∈ Z3 is drawn uniformly from [−(5 + d), 5 + d]3 . Four integercoefficient equations are generated: rows 1 and 2 form a rank-2 subsystem; rows 3 and 4 are each independently sufficient to complete the system to rank 3. Formally, all four rows have coefficients in [−(3 + d), 3 + d] \ {0} and are generated by rejection to satisfy: rank(rows 1, 2) = 2,

rank(rows 1, 2, 3) = 3, 46

rank(rows 1, 2, 4) = 3.

MIRA-Math

Information split. • Agent A: equations 1 and 2 (rank 2 in 3 unknowns, ill-posed). • Agent B: equations 3 and 4 (either one suffices for A). B’s atomic hint is always equation 3. Answer.

The triple (x, y, z).

Difficulty scaling (d ∈ {1, 2, 3}). • Solution drawn from [−(5 + d), 5 + d]3 (range ±6, ±7, ±8). • Coefficients drawn from [−(3 + d), 3 + d] \ {0} (range ±4, ±5, ±6). A11.

Recurrence Missing Initial Condition (recurrence missing init)

Setup.

A second-order linear recurrence over the integers: a(n + 2) = r1 a(n + 1) + r2 a(n),

n ≥ 0.

Parameters are sampled as r1 ∼ Unif[1, 3 + d], r2 ∼ Unif[−2, 2], and initial conditions a(0), a(1) ∼ Unif[−3 − d, 3 + d]. A target index n∗ ∼ Unif[4, 7 + d] is drawn, and the target value a(n∗ ) is computed by forward iteration. Information split. • Agent A: the recurrence coefficients r1 , r2 and the initial condition a(0). • Agent B: the recurrence coefficients r1 , r2 and the initial condition a(1). Neither agent can compute a(n∗ ) alone. B’s atomic hint is always a(1). Answer.

The integer a(n∗ ).

Difficulty scaling (d ∈ {1, 2, 3}). • Recurrence coefficient r1 drawn from [1, 3 + d] (max 4, 5, 6). • Initial conditions and coefficients drawn from [−3 − d, 3 + d] (range ±4, ±5, ±6). • Target index n∗ drawn from [4, 7 + d] (max 8, 9, 10).

Type B — Variable Hint Slot

47

Al Bateh and Saab

B1.

Birth-Death Chain with Missing Rate (birth death missing rate)

Setup. A birth-death Markov chain on states {0, 1, . . . , n − 1} with n = d + 2 is defined by birth rates λi (transitions i → i + 1 for i = 0, . . . , n − 2) and death rates µi (transitions i → i − 1 for i = 1, . . . , n − 1), all drawn from Unif[1, 3 + 2d]. The unique stationary distribution satisfies detailed balance: i−1 Y λk πi = π0 · , µk+1

i = 1, . . . , n − 1,

k=0

P normalised so i πi = 1. One rate—either some λi or some µi —is chosen uniformly at random to be hidden. A target state s∗ ∼ Unif[0, n − 1] is drawn. Instances where πs∗ has denominator exceeding 300 are rejected. Information split. • Agent A: all birth and death rates except the one missing rate, in natural-language form. • Agent B: the single missing rate (which rate it is varies per instance). Answer.

The stationary probability πs∗ as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). • Number of states n = d + 2 (3, 4, 5 states). • Rates drawn from [1, 3 + 2d] (max 5, 7, 9). B2.

Series-Parallel Resistor Network (circuit missing resistance)

Setup. A random binary tree with n = 2d + 1 leaves is built by recursive splitting. Each leaf is a named resistor R1 , . . . , Rn with a positive integer resistance drawn from Unif[1, 5 + 3(d − 1)]. Each internal node is independently labelled series or parallel with equal probability. The equivalent resistance is evaluated recursively: Rseries = RL + RR ,

Rparallel =

RL RR . RL + RR

One resistor Rk (chosen uniformly) is withheld. The circuit topology is described to both agents in natural language. Information split. • Agent A: the full circuit topology and all resistance values except Rk (which resistor varies per instance). • Agent B: the value of Rk . Answer.

The equivalent resistance Req as a reduced fraction. 48

MIRA-Math

Difficulty scaling (d ∈ {1, 2, 3}). • Number of resistors n = 2d + 1 (3, 5, 7). • Resistance values drawn from [1, 5 + 3(d − 1)] (max 5, 8, 11). B3.

Deconvolution with Missing Measurement (deconvolution)

Setup. A signal x ∈ Zn and a kernel h ∈ Zm are sampled with entries in [−(2 + d), 2 + d] and h0 ̸= 0. The full output vector y = x ∗ h ∈ Zn+m−1 is computed via linear convolution: yk =

m−1 X

hi xk−i ,

k = 0, . . . , n + m − 2.

i=0

The n + m − 1 equations in x form an overdetermined linear system. By rejection, an index k ∗ is found such that removing yk∗ makes the system rank-deficient, while the full system uniquely determines x. Information split. • Agent A: the kernel h and all output values yk except yk∗ (which index varies per instance). • Agent B: the missing output value yk∗ . Answer.

The signal x = (x0 , . . . , xn−1 ).

Difficulty scaling (d ∈ {1, 2, 3}). • Signal length n = 3 + min(d, 2) (4, 5, 5 — caps at d = 2). • Kernel length m = 2 for d ≤ 1, else m = 3. • Entry magnitudes bounded by 2 + d (max 3, 4, 5). B4.

Discrete Tomography with Ambiguous Cell (discrete tomography)

3×3 Setup. P A 3 × 3 binary grid G ∈ {0, 1} P is sampled uniformly at random. Its row sums ri = j Gij and column sums cj = i Gij are computed. The row and column sums alone are typically consistent with multiple binary grids. A target cell (it , jt ) is chosen from those that are ambiguous (i.e. their value differs across consistent completions). A hint cell (ih , jh ) ̸= (it , jt ) is found by search: revealing its value must reduce the consistent completions to exactly one.

Information split. • Agent A: all three row sums r0 , r1 , r2 and all three column sums c0 , c1 , c2 . • Agent B: the value of the hint cell Gih jh (which cell varies per instance). Answer.

The binary value Git jt ∈ {0, 1}. 49

Al Bateh and Saab

Difficulty scaling. This family does not scale with difficulty: the grid is always 3 × 3 and the generation logic is identical for all d. Accordingly, in mix generation it is always run at d = 1 regardless of the requested difficulty. B5.

Matrix Entry Hidden from Eigenvector (eigenvector missing entry)

Setup. An n × n integer matrix M (with n = d + 1) is constructed such that a known integer vector v is an eigenvector with eigenvalue λ: M v = λv. The key construction: one component vj ∗ = 0, so the eigenvector equation for row i∗ , n−1 X

Mi∗ c vc = λ vi∗ ,

c=0

contains no term involving Mi∗ j ∗ (that term vanishes). Therefore Mi∗ j ∗ is unconstrained by the eigenvector equations.PA target column t∗ is drawn ensuring Mj ∗ t∗ ̸= 0, so that A cannot compute (M 2 )i∗ t∗ = k Mi∗ k Mkt∗ without knowing Mi∗ j ∗ . Information split. • Agent A: all entries of M except Mi∗ j ∗ , and all components of v (which entry is missing varies per instance). • Agent B: the entry Mi∗ j ∗ . Answer.

The integer (M 2 )i∗ t∗ .

Difficulty scaling (d ∈ {1, 2, 3}). • Matrix dimension n = d + 1 (2 × 2, 3 × 3, 4 × 4). • Entry magnitudes bounded by 3 + 2(d − 1) (max 3, 5, 7). • Eigenvalue λ drawn from [1, 3 + 2(d − 1)] (max 3, 5, 7). B6.

Discrete Laplace Equation on a Grid (laplace grid)

Setup. An n × n grid has unknown values uij on every cell. Interior cells satisfy the discrete Laplace equation: 4 uij = ui−1,j + ui+1,j + ui,j−1 + ui,j+1 ,

(i, j) interior.

Boundary values are drawn as integer multiples of a scaling factor (24 for n = 4, 224 for n = 5) chosen to ensure integer interior solutions. One boundary cell (i∗ , j ∗ ) is chosen uniformly at random and withheld; it is verified that A’s system is rank-deficient without this value. Information split. • Agent A: all discrete Laplace equations and all boundary values except ui∗ j ∗ (which cell varies per instance). • Agent B: the missing boundary value ui∗ j ∗ . 50

MIRA-Math

Answer.

The integer value uit jt at a target interior cell.

Difficulty scaling (d ∈ {1, 2, 3}). • Grid size n = 4 for d ≤ 1, n = 5 for d ≥ 2 (4 × 4 grid with 4 interior cells; 5 × 5 with 9 interior cells). • Boundary value magnitudes scale linearly with d within each grid size. B7.

Linear System with Missing Coefficient (linear system missing coeff)

Setup. An n × n system Ax = b (with n = d + 1) is generated with a random nonsingular integer matrix A and right-hand side b (entries in [−8, 8]). The unique solution x is computed by Gaussian elimination. One coefficient Ai∗ j ∗ (chosen uniformly) is withheld. Instances where xk∗ has denominator exceeding 300 are rejected. Information split. • Agent A: all coefficients Aij except Ai∗ j ∗ , and all right-hand-side values bi , in natural language (which coefficient is missing varies per instance). • Agent B: the missing coefficient Ai∗ j ∗ . Answer.

The solution component xk∗ as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). • System size n = d + 1 (2 × 2, 3 × 3, 4 × 4). • Coefficient magnitudes bounded by 4 + 3(d − 1) (max 4, 7, 10). B8. Markov Chain with Missing Transition Probability (markov missing transition) Setup. An n-state ergodic Markov chain is generated with a random row-stochastic transition matrix P (each row sampled by drawing positive integer parts and normalising). One row index r∗ and a column partition are chosen uniformly at random: • A knows (n − 2) entries of row r∗ . • B knows exactly one entry of row r∗ at hint column c∗ . P • The remaining entry is derivable from j Pr∗ j = 1. All P other rows are fully known to A. The unique stationary distribution π satisfies πP = π, i πi = 1. Information split. • Agent A: all rows of P completely, except row r∗ where only n − 2 entries are known (which row and column vary per instance). • Agent B: entry Pr∗ c∗ . 51

Al Bateh and Saab

Answer.

The stationary probability πs∗ as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). • Number of states n = min(d + 1, 4) (2, 3, 4 — caps at d = 3). • Transition probability denominators up to n·(2+d), controlling fractional complexity. B9.

Polynomial Interpolation with Missing Point (poly interpolation)

Setup. A polynomial p of degree δ = 2 + min(d, 2) is generated by sampling δ + 1 integer coefficients from [−(3 + d), 3 + d]. A set of δ + 1 distinct integer evaluation points is drawn, and the values p(xi ) computed. By Lagrange interpolation, δ + 1 distinct points uniquely determine a degree-δ polynomial, so δ points leave it underdetermined. A fresh query point x∗ (outside the interpolation set) is sampled and p(x∗ ) computed as the target. Information split. • Agent A: the δ evaluation pairs (x0 , p(x0 )), . . . , (xδ−1 , p(xδ−1 )) (which point is held back varies per instance). • Agent B: the one remaining pair (xδ , p(xδ )). Answer.

The integer p(x∗ ).

Difficulty scaling (d ∈ {1, 2, 3}). • Degree δ = 2 + min(d, 2) (3, 4, 4 — caps at d = 2, so d = 3 is identical to d = 2 in structure but with wider coefficient range). • Coefficient magnitudes bounded by 3 + d (max 4, 5, 6). B10. Portfolio Variance with Missing Correlation (portfolio variance missing corr) Setup. A portfolio of n = d + 1 assets has weights wi , standard deviations σi ∼ Unif[1, 5], and pairwise correlations ρij ∈ (−1, 1) for i < j (drawn as rational multiples of 1/D for D ∈ {2, 3, 4}). Portfolio variance is: σp2 =

n−1 X n−1 X

wi wj σi σj ρij ,

ρii = 1.

i=0 j=0

One correlation pair (a, b) with a < b is chosen uniformly at random and withheld. Instances where σp2 has denominator exceeding 500 are rejected. Information split. • Agent A: all weights wi , all standard deviations σi , and all except ρab (which pair varies per instance). • Agent B: the correlation ρab . 52

n 2

pairwise correlations

MIRA-Math

Answer.

The portfolio variance σp2 as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). Number of assets n = d + 1 (2, 3, 4), giving correlation pairs (1, 3, 6 — of which one is always missing).

n 2

B11. Hidden Markov Model with Missing Emission (steady state missing emission) Setup. A Hidden Markov Model has n = d + 1 states and m = min(d + 1, 3) observation symbols. The transition matrix P is fully specified (each row a random distribution over states), and the emission matrix has entries E[s][o] = probability of observing symbol o from state s (each row a random distribution over symbols). The stationary distribution π of P is computed by solving πP = π. The marginal observation probability for a target symbol o∗ is: P (observe o∗ ) =

n−1 X

πs E[s][o∗ ].

s=0

One emission entry E[s∗ ][o∗ ] (chosen uniformly) is withheld. Information split. • Agent A: the full transition matrix P and all emission entries except E[s∗ ][o∗ ] (which state/symbol entry varies per instance). • Agent B: the missing emission E[s∗ ][o∗ ]. Answer.

The observation probability P (observe o∗ ) as a reduced fraction.

Difficulty scaling (d ∈ {1, 2, 3}). • Number of states n = d + 1 (2, 3, 4). • Number of symbols m = min(d + 1, 3) (2, 3, 3 — caps at d = 2).

Appendix D. Illustrative Transcript Snapshots This appendix gives compact snapshots of representative transcripts. The snapshots are included to make the qualitative failure modes in Section 7 concrete. They are not a sampling protocol and are not used to estimate prevalence. Aggregate conclusions should be drawn from the quantitative tables in the main text and Appendix A. The full transcripts and raw JSON logs are included with the released artifacts. For each snapshot, Agent A is the solver and Agent B is the fixed constrained information holder. We report only the fields needed to interpret the trace: the instance, the canonical missing fact held by Agent B, the salient request/offer/decline sequence, and the exactverifier outcome. We exclude noncanonical offers from the integration-failure examples. If the responder offers a constraint that does not match the canonical atomic hint, the trace is categorized separately as a responder-matching anomaly rather than as a solver integration failure. 53

Al Bateh and Saab

T1: No canonical hint acquired. This trace illustrates a request-acquisition failure. The solver identifies the general family and realizes that a birth–death rate is missing, but repeatedly asks for wrong rate slots. Agent B never issues an offer, so the solver exhausts its request budget without the resolving constraint. Model/mode: gpt-4o-mini, zero-shot Instance: bir-000768 Family: birth_death_missing_rate Target: pi_2 Canonical missing fact held by B: mu_4 = 2 A requests: "the birth rate lambda_2" B declines. A requests: "the death rate mu_0" B declines. A requests: "the death rate mu_2" B declines. A requests: "the birth rate lambda_2" B declines. A requests: "the birth rate lambda_1" B declines. A requests: "the birth rate lambda_3" B declines. A requests: "the birth rate lambda_0" B declines. A requests: "the death rate mu_3" B declines. A requests: "the birth rate lambda_4" B declines. Outcome: no answer submitted. Ground truth: 1/9. Failure type: request-acquisition failure. This example shows why average request count and declination count are informative: the solver is not merely failing the final arithmetic, but failing to localize the hidden slot under the request budget. T2: Canonical hint acquired, but final answer wrong. This trace illustrates an integration failure. The solver asks for the correct missing prior and receives it, but then submits an incorrect posterior. Model/mode: gpt-4o-mini, zero-shot Instance: bay-000044 Family: bayes_missing_prior Target: P_H_given_E 54

MIRA-Math

A’s private facts: P(E|H) = 1/9, P(E|not_H) = 1/2 Canonical missing fact held by B: P(H) = 5/8 A requests: "the prior P(H)" B offers: P(H) = 5/8 A final answer: 5/29 Ground truth: 10/37. Failure type: integration failure after successful hint acquisition. The request was successful, so this transcript should not be counted as a gap-identification failure. The incorrect score comes from the downstream computation or simplification step. T3: Canonical hint acquired and final answer correct. This trace illustrates the intended successful loop. The solver identifies the absent resistor, receives the missing value, and computes the exact equivalent resistance. Model/mode: gpt-4o-mini, zero-shot Instance: cir-000924 Family: circuit_missing_resistance Target: R_eq Canonical missing fact held by B: R3 = 7 ohms A requests: "the resistance of R3" B offers: R3 = 7 ohms A computes: G1 = R3 + R7 = 7 + 6 = 13. G2 = R2 + R5 = 11 + 4 = 15. G3 = R1 + G2 = 8 + 15 = 23. G4 = G3 + R6 = 23 + 9 = 32. G5 = G4 + R4 = 32 + 1 = 33. R_eq = (13 * 33) / (13 + 33) = 429/46. A final answer: 429/46 Ground truth: 429/46. Failure type: none; full request--resolve--solve success. T4: Optional prompt-anchoring snapshot for four-shot analysis. This trace is included only to illustrate a prompt-sensitivity hypothesis. It should not be interpreted as a prevalence estimate. In this moment-recovery instance, the solver repeatedly asks for generic named moments rather than asking for the concrete missing equation held by Agent B. Model/mode: gemini-2.5-flash, four-shot Instance: mom-000404 Family: moment_problem 55

Al Bateh and Saab

Target: q0, q1, q2 A’s private equations: q0 + q1 + q2 = 12 q1 + 2q2 = 3 Canonical missing fact held by B: q1 + 4q2 = 5 A requests: "the 2nd moment" B declines. A requests: "the 1st moment" B declines. A requests: "the 0th moment" B declines. A requests: "the value of moment_0" B declines. A requests: "the value of moment_1" B declines. A requests: "the value of moment_2" B declines. A requests: "the 0th moment" B declines. A requests: "the 2nd moment" B declines. A requests: "the 3rd moment" B declines. Outcome: no answer submitted. Ground truth: q0 = 10, q1 = 1, q2 = 1. Interpretation: possible exemplar anchoring or mismatch between generic moment labels and the concrete constraint held by the responder. This example supports the qualitative interpretation in Section 6: four-shot demonstrations may encourage protocol-shaped requests without guaranteeing instance-specific slot localization. It is not used as evidence that this behavior is frequent.

56

Supplementary Dataset Card for MIRA-Math Charbel Al Bateh

[email protected]

Department of Electrical and Computer Engineering Lebanese American University Byblos, Lebanon

Samer Saab Jr.

[email protected]

Department of Electrical and Computer Engineering Lebanese American University Byblos, Lebanon

1 Dataset Summary MIRA-Math is a synthetic benchmark for minimal information requesting and mathematical reasoning under partial information. Each instance is generated from a complete latent mathematical state with a unique answer. The solver receives a locally underdetermined private view that is missing exactly one necessary atomic fact. To solve the instance, the solver must request the missing fact in natural language under a fixed request budget and then integrate the returned fact into an exact final answer. The benchmark is designed to separate two capabilities that are conflated by standard full-information mathematical benchmarks: identifying and requesting the missing mathematical information, and performing the downstream computation after receiving it. Instance generation, typed atomic-hint specifications, validation, and final-answer verification are deterministic. Request acceptance in the reported reference protocol is measured under a fixed constrained LLM-mediated responder channel.

2 Access The release artifacts are available at the following locations: Code and documentation: https://github.com/cedar-lau/mira-math Generated benchmark data: https://huggingface.co/datasets/samersaabjr/MIRA-MATH/ The generated benchmark files are hosted as downloadable JSONL artifacts. The code repository contains the generators, verifiers, prompts, runner, scoring utilities, documentation, and reproduction scripts. The submitted release is identified by the repository commit hash, dataset version, and file checksums recorded with the artifact release. Subsequent updates will use version tags or archived releases so that the version corresponding to this paper remains recoverable.

3 Released Artifacts The release consists of the following artifacts: ©2026 Al Bateh and Saab.

Al Bateh and Saab

• Frozen generated benchmark instances in JSONL format. • Family-specific Python generators and seed lists for regenerating the benchmark. • Instance validators that check global uniqueness, local underdetermination, and sufficiency of the canonical atomic hint. • Exact answer verifiers and normalization utilities. • Machine-readable canonical atomic-hint specifications and typed hint identifiers. • Prompt templates for zero-shot and four-shot evaluation. • Structured-output schemas and prompts for the fixed constrained information-holder responder. • Reference runner and scoring scripts. • Raw interaction transcripts, run metadata, and aggregate result files for the reported baselines. • Documentation, changelog, license files, and checksums.

4 Dataset Composition The current release is the 20/50 typed split. It contains 2,310 instances across 22 typed mathematical families. The families are divided into two groups: • Type A: fixed missing-information slots. There are 11 Type A families, with 20 instances per difficulty level for each family, for 660 Type A instances in total. • Type B: variable missing-information slots. There are 11 Type B families, with 50 instances per difficulty level for each family, for 1,650 Type B instances in total. Each family is generated at three difficulty levels unless the family definition is intentionally non-scaling. Difficulty can affect coefficient ranges, dimensions, denominators, target-index depth, grid size, or other family-specific parameters. The 22 families cover Bayesian inference, Chinese remainder reconstruction, coordinate geometry, triangle path-sum inversion, linear systems with a separator variable, rank-one matrix completion, moment recovery, phase retrieval, piecewise functions, rank-deficient linear systems with a shared variable, recurrences, birth-death chains, series-parallel resistor networks, deconvolution, discrete tomography, eigenvector-constrained matrix questions, discrete Laplace equations, linear systems with a missing coefficient, Markov chains, polynomial interpolation, portfolio variance, and hidden Markov models.

5 Instance Fields and Data Format Each generated instance records the information needed for validation, evaluation, and regeneration. The exact field names are defined by the released schema, but conceptually each instance includes: 2

MIRA-Math Dataset Card

• family name, family type, difficulty level, and generation seed; • solver-facing private view and target question; • responder-facing private constraint containing the missing atomic fact; • machine-readable canonical atomic-hint identifier and value; • exact target answer and answer type; • verifier metadata and normalization information; • prompt-regime metadata where applicable. The frozen JSONL files are intended for direct evaluation. The generator code and fixed seed lists are provided so that users can regenerate the same release and audit individual instances from their seeds.

6 Generation Procedure Each instance is sampled from a family-specific generator using recorded random seeds. Generators use exact arithmetic and rejection sampling to enforce nondegeneracy, local underdetermination of the solver view, global uniqueness after adding the canonical atomic hint, and verifier-compatible answers. Family-specific checks include rank tests, uniqueness tests, exact rational normalization, finite completion enumeration, denominator thresholds, and structural constraints such as nonsingularity or nonparallelism. The generation process enforces the following high-level invariant: the solver’s private view alone is insufficient to determine the target answer, while the solver’s private view plus the dataset-provided atomic hint is sufficient for a unique answer. This construction makes the benchmark a diagnostic of missing-information requesting rather than a collection of ordinary fully specified math problems.

7 Labels, Hints, and Verifiers Each instance contains an exact answer represented as an integer, reduced rational, tuple, vector, or family-specific structured object. Final answers are evaluated by family-specific exact verifiers after normalization rather than by free-form string matching. The missing information is represented by a typed canonical atomic-hint specification. Examples include a missing prior, congruence, coefficient, boundary value, transition probability, resistor value, interpolation point, or emission entry. The solver may ask for this information in free-form natural language, but successful acquisition requires the request to identify the held atomic fact precisely enough to be accepted by the fixed constrained responder.

8 Interaction and Evaluation Protocol In the reference evaluation protocol, the solver is shown its private view, the target question, and a request budget. At each turn, the solver may request information or submit a final 3

Al Bateh and Saab

answer. The information holder is a fixed constrained LLM responder. It receives only its private constraint and the solver’s current request. It must return a structured offer when it judges the request to match the held fact, or a structured decline otherwise. A valid offer includes has exact match=true, the quoted private constraint, and the extracted hint string. A decline must not reveal alternative information. This responder channel is controlled but not deterministic: semantic matching is LLMmediated. The deterministic components are the family generators, canonical hint specifications, validators, stored answers, and exact final-answer verifiers. Request metrics should therefore be interpreted as solver performance under this fixed constrained responder channel, not as performance under a deterministic symbolic request-matching oracle.

9 Reported Metrics The benchmark supports both final-answer and request-level metrics: • Accuracy: fraction of instances whose final answer is accepted by the exact verifier. • Request hit rate: accepted requests divided by responder messages, averaged per instance. • First-request success: fraction of instances where the first request is accepted. • Average requests: mean number of solver requests per instance. • Average declinations: mean number of fixed-responder declinations per instance. • Average hints: mean number of accepted offers per instance. • Trace-level decomposition: no canonical hint acquired, canonical hint acquired but final answer wrong, and final answer correct. The trace-level decomposition is computed from released transcripts and offline hint audits, without additional model calls.

10 Quality Controls Quality controls include deterministic seeding, exact arithmetic, family-specific nondegeneracy filters, schema validation, instance validators, verifier tests, and release checksums. For finite discrete families, validators can enumerate consistent completions. For algebraic families, validators use rank, uniqueness, or constraint-satisfaction tests where appropriate. The release includes checksums for the frozen JSONL files and released transcript archives. Future releases will add continuous-integration tests that regenerate benchmark subsets, validate schemas, and run exact verifiers on sampled instances.

11 Intended Uses Appropriate uses include: • evaluating whether a model can recognize a missing mathematical fact; 4

MIRA-Math Dataset Card

• measuring how precisely a solver can request a necessary atomic hint; • testing whether a model can integrate a returned hint into an exact mathematical answer; • comparing prompts, solver loops, request-budget strategies, and communication-efficiency methods; • auditing the gap between request acquisition and downstream mathematical integration; • regression testing mathematical reasoning systems under controlled partial information.

12 Out-of-Scope Uses MIRA-Math should not be used as evidence that a model can collaborate with an autonomous teammate. The second role is a fixed constrained information holder, not a strategic collaborator. The benchmark should also not be treated as a general measure of tool use, web navigation, retrieval, social reasoning, human dialogue, or real-world decisionmaking. Success on MIRA-Math should not be used to claim broad agentic ability without additional evidence from settings designed for those capabilities.

13 Limitations The benchmark is synthetic and mathematically structured. This improves control, auditability, and exact verification, but limits realism. Each current instance requires exactly one atomic hint; many real tasks require multiple facts, uncertain evidence, or iterative human clarification. The typed hint schema and constrained responder protocol make the benchmark reproducible but may understate the difficulty of communicating with humans or open-ended systems. The reported request metrics depend on a fixed LLM-mediated responder channel. Although the responder is constrained and auditable, semantic offer-or-decline decisions can still reflect occasional responder matching errors. Model rankings may also become stale as API-backed models and open-weight checkpoints change. The release is therefore intended to support reproducible reruns and future baselines rather than a permanent leaderboard.

14 Licensing The generated benchmark data are released under the CC BY 4.0 license. The generator, verifier, runner, and evaluation code are released under the MIT License. Users should cite the MIRA-Math paper and dataset when using the benchmark.

15 Hosting and Maintenance Plan The maintainers will preserve the 20/50 typed split as a stable regression suite. Future releases may add larger public splits, private held-out seeds, multi-hint variants, alternative prompt protocols, or alternative responder implementations. Each release will include: • a versioned dataset identifier; 5

Al Bateh and Saab

• generator version and code commit; • seed lists and prompt-protocol version; • dataset and transcript checksums; • a changelog covering changes to generators, prompts, matching rules, verifiers, and scoring scripts; • documentation describing backward compatibility with the original 20/50 typed split. The Hugging Face dataset page should remain the primary location for downloadable generated JSONL files, while the code repository should remain the primary location for generation, validation, and evaluation code.

16 Ethical Considerations MIRA-Math uses synthetically generated mathematical problem text and synthetic numeric values. It does not contain human-subject data, private user data, or scraped personal information. Privacy and consent risks are therefore minimal. The main ethical risk is overclaiming: benchmark performance should not be presented as evidence of broad collaboration, human communication skill, open-ended tool use, or deployment readiness. The paper and release documentation mitigate this risk by specifying intended uses, out-of-scope uses, and limitations.

17 Author Responsibility Statement The authors are responsible for the benchmark design, generated instances, code, documentation, evaluation protocol, and released metadata. The authors have verified that the release does not violate third-party rights, that released model-output transcripts are shared in accordance with applicable provider terms, and that the selected data and code licenses are stated in the repository and dataset page.

6