NumBench: Diagnosing Counting Failures in Text-to-Image Models

NumBench: Diagnosing Counting Failures in Text-to-Image Models Sandeep Wadhwa1 , Mayank Vatsa1 , Richa Singh1 , Parrva Chirag Shah1 , Prakhar Galriya2

arXiv:2608.28206v1 [cs.CV] 28 Aug 2026

1

Indian Institute of Technology Jodhpur, India 2 Shiv Nadar University, India

Abstract Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce NumBench, a benchmark of 640,000 prompts spanning 1,600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (cw-NPS), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14,400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.

Introduction Counting remains a persistent failure mode of modern T2I systems. A prompt such as “generate 51 raccoons” often yields far fewer objects; prompts requesting two categories additionally produce category mixing, merged instances, and default scene sizes (Figure 1). These failures matter wherever generated images must be numerically correct, including educational content, synthetic training data, accessibility tools, and digital commerce. Even when deployment counts are smaller, extending the range to 100 exposes where models cease to preserve distinct instance evidence and separates low-count success from scalable numeric control. Existing evaluation does not isolate the causes. General compositional benchmarks include only small counting subsets (Saharia et al. 2022; Ghosh, Hajishirzi, and Schmidt 2023; Huang et al. 2023; Bakr et al. 2023; Hu et al. 2023); counting-specific sets remain limited in count range, category diversity, or controlled spatial variation (Kajić et al. 2024; Binyamin et al. 2025; Sun et al. 2026; Zafar et al. 2026). Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Generate image of 51 raccoons

Generate image of 9 Traffic cones and 15 little blue herons

Generate image of 74 cookies and 56 panna cotta

Midjourney

Grok

Gemini

SORA

ChatGPT

Figure 1: Counting failures across five commercial T2I systems. High-count prompts produce missing or merged objects, category confusion, and repeated default patterns. System labels identify the evaluated interfaces; exact collection dates are recorded in the release metadata. Moreover, detector-based metrics can confuse generator errors with detector misses, especially in crowded scenes. We address this gap with three contributions. First, NumBench provides 640,000 prompts over 1,600 categories and counts 1–100. A controlled design crosses single- versus double-category composition, free-form versus guided layout, and clean versus modified appearance instructions. Second, we introduce a process model for spatial collision. It predicts how limited spatial capacity creates rapidly growing undercount and how coordinated layouts attenuate this loss. Third, we propose cw-NPS, a calibrated detector-ensemble metric, and evaluate nine systems at substantially larger scale than prior counting studies. The evaluation highlights a consistent hierarchy. Requested count is the dominant source of difficulty, followed by layout, composition, and appearance conditions. Specialized methods improve low-count results but remain weak at high counts. Among guided layouts, grids perform best, consistent with the coordination prediction of the process model. Human agreement and detector–human correlation weaken above count 50, so we treat that region as a stress test rather than a fully validated count estimate, and we release prompts, detector outputs, and annotations to support evaluation under the same protocol.

Benchmark

Max Prompts Count Categories Count Layout Appear. Comp. Joint

DrawBench† (Saharia et al. 2022) PaintSkills† (Cho, Zala, and Bansal 2023) GenEval† (Ghosh, Hajishirzi, and Schmidt 2023) T2I-CompBench† (Huang et al. 2023) HRS-Bench† (Bakr et al. 2023) GeckoNum (Kajić et al. 2024) Zafar et al. (Zafar et al. 2026) CoCoCount (Binyamin et al. 2025) QUANT-Bench (Sun et al. 2026)

∼200 7,330 553 6,000 45,000 1,386 3,674 ∼200 475

∼10 ∼5 4 ∼5 ∼10 ∼10 25 ∼10 25

< 20 ∼80 80 ∼200 700+ ∼30 147 ∼80 19

× × ✓ × × ✓ ✓ ✓ ✓

× × × × × × × × ×

× × × × × × × × ×

× × × × × ✓ × ✓ ×

× × × × × × × × ×

NumBench

640,000

100

1,600

✓

✓

✓

✓

✓

Table 1: Comparison with counting benchmarks. † Counting is one component of a broader suite. Joint indicates that count, layout, appearance, and composition are controlled within one benchmark design.

Related Work Benchmarks. DrawBench (Saharia et al. 2022), PaintSkills (Cho, Zala, and Bansal 2023), GenEval (Ghosh, Hajishirzi, and Schmidt 2023), T2I-CompBench (Huang et al. 2023), HRS-Bench (Bakr et al. 2023), and TIFA (Hu et al. 2023) evaluate counting as one component of broader compositionality. Focused resources include GeckoNum (Kajić et al. 2024), CoCoCount (Binyamin et al. 2025), QUANT-Bench, introduced with QUOTA (Sun et al. 2026), and the FSC-147-based protocol of Zafar et al. (Zafar et al. 2026). None jointly spans counts 1–100, 1,600 categories, composition, layout, and appearance variation at NumBench scale. Evaluation and generation. GenEval uses detection-based evaluation (Ghosh, Hajishirzi, and Schmidt 2023), while TIFA uses visual question answering (Hu et al. 2023). Most counting work relies on a single detector with a fixed threshold (Binyamin et al. 2025; Zafar et al. 2026). General-purpose evaluators such as GenEval and TIFA assess broader prompt– image consistency; our direct baseline is b-NPS because it represents the fixed-threshold detector-counting protocol used in prior exact-count evaluations. Generation methods modify count conditioning (Binyamin et al. 2025; Kang et al. 2025), use detector feedback (Zafar et al. 2026), plan scenes with language models (Lian et al. 2024; Wu et al. 2024), or impose spatial control (Li et al. 2023; Zhang, Rao, and Agrawala 2023; Dahary et al. 2024). NumBench supplies a common diagnostic testbed for these approaches.

A Process Model for Spatial Collisions Consider a prompt requesting n instances of one category. At the rendered object scale, suppose the image offers M resolvable regions. Assigning two requested instances to the same region may merge them into one connected visible instance. Let R be the number of occupied regions after placement. Theorem 1 (Collision deficit). If n ≤ M instances are assigned independently and uniformly to M regions, then  n−1 Y i 1− ≤ e−n(n−1)/(2M ) , (1) P(R = n) = M i=0

and

n 2

 n3 . (2) +O E[n − R] = M M2 Thus, when n/M is small, the expected visible deficit grows approximately quadratically rather than linearly. The model describes one mechanism only: it cannot explain overcounting, category substitution, or a generator’s learned preference for common scene sizes. To represent spatial coordination, let a placement use an unused region with probability λ and otherwise choose uniformly, where q = 1 − λ. Proposition 2 (Coordination). For q > 0,  Mh q n i E[R] = 1− 1− , (3) q M
  n n3 . (4) E[n − R] = q 2 + O M M2 

For λ = 1, every instance occupies an unused region and R = n. Structured layout guidance should improve counting when it increases λ or effective capacity M . This yields a testable qualitative prediction: regular layouts should degrade more slowly than weakly coordinated layouts. Our experiments test the predicted layout ordering through numeric precision and treat agreement as diagnostic consistency rather than causal identification. Proof intuition. Let Iu indicate whether region u is occupied. A region remains empty with probability (1 − 1/M )n , so linearity of expectation gives E[R] = M [1−(1−1/M )n ].
n Expanding the power yields the leading deficit 2 /M , which counts pairwise competition for the same region. With coordination, the recurrence for expected occupancy replaces 1/M by q/M ; hence the leading deficit is multiplied by q = 1 − λ. Complete proofs are provided in the supplementary material. The model yields two additional implications. First, increasing canvas resolution helps only when it increases the number of usable regions at the rendered object scale; upsampling after generation should not recover merged instances.

Dimension

Levels Allocation

Composition 2 80K single + 80K double Count 100 Equal mass at each integer Category 1,600 Marginally balanced assignment Spatial layout 4 Free-form / grid / scene / random Appearance 2 Clean + modified Total

640,000 prompts

Table 2: NumBench construction. Every core prompt receives a free-form version, one guided counterpart, and clean and modified appearance forms. Second, a layout method may improve counting by increasing M , increasing λ, or both. These mechanisms are observationally distinct only with directional count errors and controlled object scale. NumBench supplies the factors needed for such future tests, while the present paper uses the model to motivate and interpret the layout axis.

NumBench NumBench begins with 160,000 core prompts: 80,000 single-category and 80,000 double-category prompts. The vocabulary contains 1,600 unique categories curated from ImageNet, COCO, Objects365, Caltech-101, Caltech-256, Food101, and CUB-200. Synonyms, plural variants, and formatting duplicates are removed. Each single-category prompt pairs one category with a count in [1, 100]. Every category appears 50 times and every integer appears 800 times. Each double-category prompt pairs two distinct categories with independently assigned counts; every category appears 100 times and every integer appears 1,600 times as a requested count. Category and count assignments are balanced marginally rather than forming a complete category–count crossing. The core set is expanded along two axes. Layout adds a free-form version and a guided version using grid, scene, or random; guided forms are balanced across the three descriptors. After appearance expansion, free-form contributes 320,000 prompts and each guided form contributes approximately 106,667. Every core prompt therefore has a free-form reference but only one guided counterpart; factor summaries compare the four layout levels separately rather than pooling guided prompts. Appearance adds a clean and a modified version. Modified prompts combine two or three terms from blurry, deformed, low quality, overlapping, merged, partial, cropped, and artifacts. The final total is 160K×2×2 = 640K prompts. The scale comes from balanced coverage of count, category, composition, layout, and appearance cells rather than repeated paraphrases of one condition. NumBench is a controlled diagnostic distribution rather than an estimate of everyday prompting; the Experiments section includes a natural-language transfer test. Vocabulary and appearance semantics. Category names are converted to lowercase singular forms; clear synonyms, plural variants, and formatting duplicates are merged, while ambiguous names are removed when one phrase denotes unrelated concepts. Source labels and dataset provenance re-

Group

Descriptors

Visual qual- blurry, deformed, ity low quality, artifacts Instance ev- overlapping, idence merged, partial, cropped

Interpretation Primarily changes fidelity and detector confidence. Can make the requested cardinality visually ambiguous.

Table 3: The modified-appearance condition combines quality degradation with instance-separation difficulty; analyses keep these roles conceptually distinct. main attached to every normalized category. The appearance axis contains two semantically different groups (Table 3). Separating them matters because one group mainly degrades image quality, whereas the other changes the visible evidence from which either a detector or a person must infer cardinality.

Confidence-Weighted Numeric Precision A fixed-threshold count from one detector is vulnerable to detector-specific bias, confidence drift, and recall loss in crowded scenes. We use OW-DETR (Gupta et al. 2022), Grounding DINO (Liu et al. 2024), and OWL-ViT (Minderer et al. 2022). Each detector is queried through its native category interface, and class-aware non-maximum suppression is applied independently for every requested category before aggregation. The release preserves category mappings, raw proposals, calibrated confidences, and retained boxes, allowing each reported score to be audited. Additional vocabularycoverage details are provided in the supplement. (0.5) For reference, b-NPS uses hard counts ndi from each detector at threshold .5: ! P (0.5) k 1 X |D−1 d ndi − ni | b-NPS = max 0, 1 − . k i=1 ni (5) It removes single-detector dependence but still treats a .51 detection as fully present and a .49 detection as absent. Detector d returns raw confidence ρraw dij for proposal j of requested category i. We first apply temperature scaling (Guo et al. 2017),
raw ρsc (6) dij = σ logit(ρdij )/τd , with temperatures fitted on a disjoint 5,000-image calibration set. The fitted values are 1.15, 1.42, and 0.93 for OWDETR, Grounding DINO, and OWL-ViT. We then use the pre-specified reference-conditioned threshold  n ≤ 3, .7, T (n) = .3, (7) n ≥ 10,  .7 − .4(n − 3)/7, otherwise, and Pform = P the scsoft ensemble count n̂i D−1 d j ρsc 1[ρ ≥ T (n )]. For k requested i dij dij categories, ! k 1 X |n̂i − ni | cw-NPS = max 0, 1 − . (8) k i=1 ni

The score lies in [0, 1]. Calibration and thresholding are fixed before evaluating any of the nine systems. We pre-specify one threshold schedule and do not claim that it is optimal; paired b-NPS results test whether the main conclusions depend on confidence weighting alone. Because T (n) uses the requested count, cw-NPS is an evaluation score, not an independent count estimator. The schedule begins changing above count 3 and reaches .3 at count 10. It favors proposal recall in crowded scenes and can make signed errors less negative; a fixed .5 threshold has the opposite risk when crowding lowers detector confidence. Accordingly, the paper uses cw-NPS for absolute numeric precision but does not infer a universal undercount direction from it.

Experiments Systems and protocol. We evaluate five commercial systems (OpenAI image generation, Gemini, Sora, Midjourney, and Grok), two open models (SDXL and FLUX.2 [dev]), and two specialized methods (CountGen (Binyamin et al. 2025) and Bounded Attention (Dahary et al. 2024)). Commercial outputs were collected between January 2025 and July 2026 using default settings; per-system collection windows are recorded in the release metadata. OpenAI outputs were generated in January–February 2025 via ChatGPT, which at that time used DALL-E 3. Open and specialized systems use all 640,000 prompts with four images per prompt; commercial systems use a balanced 3,200-prompt subset with four images per prompt. All outputs are 1024 × 1024. The four locally run systems produce 10.24 million images; commercial systems add 64,000. We cache detector proposals so b-NPS and cw-NPS use identical raw outputs. The commercial subset is stratified by count range, composition, layout, and appearance. Factor effects are reported as the average best–worst difference across factor levels, averaged over the nine systems, rather than pooling all layout prompts into one guided category.

Performance Collapses as Counts Rise Every method declines across count ranges (Table 4). On single-category prompts, CountGen falls from .692 at counts 1–25 to .129 at 76–100; Bounded Attention falls from .623 to .110. Both open models fall below .09 in the final range. Specialized methods therefore help at low and moderate counts but do not solve high-count generation. CountGen loses 81.4% of its single-category score between the first and last ranges; the corresponding loss for Bounded Attention is 82.3%. The broad ordering is preserved under b-NPS, indicating that the collapse is not created solely by confidence weighting. Across the table, b-NPS is generally higher because hard thresholding treats every retained proposal as a full instance. On single-category prompts, the average b–cw gap across the four open and specialized methods is .043 for counts 1–25 and .028 for counts 76–100. The same trend appears for commercial systems (Figure 2). The strongest displayed single-category score is below .75 in the 1–25 range, and all displayed systems are at or below .16 in the 76–100 range. Commercial results are snapshots because unversioned services can change without notice.

Figure 2: cw-NPS on the balanced 3,200-prompt subset for five commercial systems, with SDXL included as an openmodel reference. S and M denote single-category and twocategory prompts, respectively. Composition changes method rankings. CountGen leads every single-category range, whereas Bounded Attention leads every double-category range. At counts 1–25, Bounded Attention reaches .641 on double-category prompts versus .610 for CountGen. Subject-wise spatial control may therefore help category separation, although double-category prompts are not uniformly harder for every system and range.

Observed Sensitivity to Benchmark Factors We use the balanced design to measure descriptive score variation across factor levels. For factor f , within each cell defined by the remaining factors we compute the relative contrast between the best and worst levels of f , (best − worst)/best, and average these contrasts over cells and systems. The resulting values lie in [0, 1] and measure proportional sensitivity, not absolute cw-NPS differences. These values identify strong associations within NumBench; they are not causal coefficients. The resulting hierarchy is count range (.81), layout (.57), composition (.51), and appearance condition (.45). Figure 3 shows that specialized methods retain a low-count advantage but all systems collapse toward the origin above 50 objects. Among the three guided settings, grid has the highest average cw-NPS (.45), while scene (.31) and random (.32) are similar. This ordering is consistent with Proposition 2: regular placement can reduce competition for resolvable regions. The comparison remains observational and does not include the matched free-form aggregate needed for a complete fourlayout contrast. Under the cell-balanced aggregation defined above, appearance condition has a mean best–worst difference of .45. This aggregate is not a uniform .45 reduction within every count range. Blur, deformation, low quality, and artifacts primarily change visual quality; overlap, merging, cropping, and partial objects directly change the visible evidence for counting. The supplementary material separates these groups.

Metric Validation and Transfer Replacing the ensemble with any single detector shifts aggregate cw-NPS by at least .10, showing that detector choice materially affects measurement. The human study below evaluates the ensemble score. Three annotators independently counted 1,600 images per system, yielding 14,400 images. Each count range con-

Single Category 26–50 b cw

‡

51–75 b cw

Double Category ‡

76–100 b cw

1–25 b cw

26–50 b cw

51–75‡ b cw

76–100‡ b cw

Class

Method

1–25 b cw

Open Open

SDXL FLUX.2 [dev]

.602 .553 .402 .360 .250 .222 .099 .076 .483 .445 .301 .271 .152 .125 .051 .038 .608 .560 .409 .367 .259 .231 .105 .081 .489 .450 .309 .278 .157 .131 .056 .040

Specialized CountGen .722 .692 .520 .485 .345 .311 .160 .129 .642 .610 .443 .410 .255 .230 .111 .088 Specialized Bounded Attention .669 .623 .482 .442 .315 .284 .142 .110 .671 .641 .465 .435 .279 .251 .125 .100

Table 4: Basic (b) and confidence-weighted (cw) numeric precision on the full NumBench evaluation. Broad degradation and method ordering persist under both metrics. ‡ Values above count 50 are stress-test results because human agreement and detector–human correlation weaken in this range.

Figure 3: Factor analysis. Axis and legend labels use shorthand: “Prompt Noise” denotes the appearance condition, “Object Config.” denotes composition, and “Single/Double-Object” denotes single- and double-category prompts. (a) Count range has the largest effect (normalized values). (b) Specialized methods improve low-count performance but not the high-count regime. (c) Composition changes difficulty across count ranges.

Count range Fleiss’ κ Detector–human r 1–25 26–50 76–100‡ Overall

.82 .58 .33 —

> .80 > .80 ≈ .40 > .75

Table 5: Aggregate human validation for the reported strata. The source annotations do not provide separate κ or detector– human r estimates for 51–75. ‡ High-count stress test.

tributes 400 images per system, with coverage across layouts, appearance conditions, and both composition settings. Table 5 summarizes the reported validation strata. Agreement is strongest through count 50, making this the better-validated evaluation regime. Results above 50 remain informative as stress tests, but their absolute values should be interpreted cautiously. Annotators saw the image and target category, counted distinct visible instances, and could mark ambiguous cases. Merged objects were counted by visible bodies when separable; cropped objects were counted only when a distinct instance remained clear. Consensus counts replace detector counts in Equation 8 to form the human score. Finally, we construct CoCoCount+ by combining 200

prompts from CoCoCount (Binyamin et al. 2025), 19 counting prompts from DrawBench (Saharia et al. 2022), and 24 prompts used in prior comparative studies (Binyamin et al. 2025; Dahary et al. 2024). Its 243 natural-language prompts all request fewer than ten objects. Method rankings and performance gaps remain broadly consistent with NumBench, and no system exceeds .67 cw-NPS. Qualitative cases also separate total-count errors from category-binding errors: a model may generate five kittens while assigning the wrong color ratio. The examples show that visually plausible outputs can still contain cardinality or category-binding errors, supporting a dedicated count metric rather than visual quality alone. Interpreting the high-count regime. Three uncertainties accumulate as requested count rises. First, generators increasingly merge, crop, or occlude instances, so even the intended visual unit becomes ambiguous. Second, detector recall and confidence decline as objects become smaller and more crowded. Third, human annotators disagree more often in the same scenes. The paired b-NPS/cw-NPS table shows that the broad performance collapse survives two counting rules, but neither rule converts visually ambiguous scenes into exact ground truth. We therefore use counts 51–100 to test whether a method preserves usable numeric evidence under severe crowding, not to claim equally precise mea-

Grok

Gemini

SORA

DALL-E 3

SDXL

b-NPS(D): 0.4612

b-NPS(D): 0.7407

b-NPS(D): 0.6482

b-NPS(D): 0.6061

b-NPS(D): 0.6431

cw-NPS(D): 0.4762 cw-NPS(H): 0.4492

cw-NPS(D): 0.7196 cw-NPS(H): 0.7071

cw-NPS(D): 0.6568 cw-NPS(H): 0.6111

cw-NPS(D): 0.6095 cw-NPS(H): 0.6516

cw-NPS(D): 0.6526 cw-NPS(H): 0.6061

Figure 4: Human validation for “11 cats and 9 dogs.” Detector-based and human-count cw-NPS differ by at most .05 for each shown system. This example illustrates agreement in the validated count range; the aggregate correlation analysis, not a single image, is the main evidence. surement across all 100 counts. This distinction also explains why the process model is interpreted through layout ordering and absolute precision rather than through an unqualified signed-undercount claim.

What the Benchmark Diagnoses NumBench separates three failure families that are often collapsed into one score. Cardinality errors change the number of visible target instances. Binding errors preserve the total but assign instances to the wrong category or attribute, as in the color-ratio example in Figure 5. Visibility errors create partial, merged, or ambiguous instances that make both human and automated counting uncertain. Equation 8 evaluates category-wise cardinality and therefore penalizes many binding errors, but it cannot fully resolve ambiguous visibility without a stronger instance-level reference. Category-wise scoring is essential. Suppose a prompt requests three cats and two dogs, but the image contains five cats and no dogs. A total-only evaluator reports an exact count of five. In contrast, the category-wise score is 1 − 12 (2/3 + 1) = 1/6 before clipping, correctly exposing the allocation failure. The same principle applies to attributes such as color when they define separate requested groups. This distinction matters for model development. A countconditioning method should be judged on cardinality after controlling layout and appearance. A spatial-control method should additionally report category binding on doublecategory prompts. An evaluator should distinguish its bettervalidated regime from high-count stress tests rather than presenting all scores as equally certain. NumBench supports all three analyses through factor labels, category-wise detector outputs, and human annotations. Table 6 clarifies how the benchmark should guide interventions. Prompt or count-conditioning methods primarily target cardinality; subject-wise layout controls should be examined for both cardinality and binding; and evaluator improvements should focus on visibility ambiguity and detector robustness. This prevents an apparent gain in one failure family from

being presented as a complete solution to counting. Reproducibility and Release: The complete 640,000prompt dataset is uploaded as the supplementary submission. After acceptance, the full release will include every prompt and factor label, source codes, and meta-data associated with our experiments.

Limitations and Broader Impact Measurement validity. cw-NPS remains detector-mediated: its three detectors reduce single-model dependence but may share training-data biases, miss the same small objects, or confuse similar categories. The count-conditioned threshold favors recall in crowded scenes and can make signed errors less negative, whereas a fixed threshold can exaggerate undercount as confidence falls with object size; we therefore report absolute precision, pair it with b-NPS, and avoid claiming a universal error direction. The human study provides an independent reference, but human counting also destabilizes in dense scenes, so results above 50 should be read as stress tests of preserved numeric evidence rather than equally precise count estimates. Model and construct validity. The collision process is a deliberately simple null model: it assumes a fixed effective capacity M and explains only merging-induced deficits, not overcounting, category substitution, object-scale or scenestructure changes, or learned scene-size priors. Grid guidance is observationally consistent with stronger coordination, but the present comparison cannot separate larger effective capacity, less overlap, larger objects, simpler backgrounds, or improved detector confidence. The model is thus a testable mechanism proposal, not a complete account of counting failure. Dataset and temporal validity. Controlled templates support paired factor analysis but exclude paraphrases, number words, multilingual instructions, and long natural-language prompts; CoCoCount+ offers an external check but contains only 243 low-count prompts. The appearance axis intentionally mixes quality degradation with instance-evidence modification, so pooled effects should not

Grok

Sora

Midjourney

Dall-E 3

Gemini

BA

Flux

CountGen

SDXL

FID vs cw-NPS

Figure 5: Transfer to CoCoCount+. Left: generations for “3 ginger kittens and 2 grey kittens” expose exact-count, color-binding, missing-object, and overcount failures even in the low-count regime. Right: FID versus cw-NPS across the nine systems; perceptual quality does not predict numeric precision, and no evaluated system exceeds .67 cw-NPS on the 243 prompts. Failure

Observable symptom

Cardinality

Too few or too many vis- Count, layout ible target instances

Binding

Correct total but wrong Composition, category category or attribute allocation Merged, partial, cropped, Appearance, layout or ambiguous instances

Visibility

Most relevant axes

What NumBench mea- Recommended reportsures ing Category-wise absolute count error through b-NPS and cw-NPS Separate target counts for every requested category

Scores by count range and composition Per-category counts on double-category prompts

Detector confidence plus Human validation and human ambiguity and explicit stress-test labels agreement

Table 6: NumBench separates three failure families that require different evidence and reporting. A single total-count score cannot distinguish them. be read as one homogeneous corruption, and categories inherit the visual scope of their source datasets. Commercial systems are unversioned snapshots collected between January 2025 and July 2026, and later service updates may change their results. These boundaries motivate releasing prompts, timestamps, detector proposals, and open-model outputs so future systems can be evaluated under the same protocol.

Ethical Statement: NumBench is intended to improve evaluation for applications requiring visual correctness. The work introduces no new image-generation capability. Categories are drawn from established recognition datasets, with offensive labels and sensitive personal attributes excluded. Commercial outputs are redistributed only when platform terms permit. The complete prompt dataset is included with the supplementary submission. Metadata, open-model outputs, detector proposals, calibration code, and evaluation scripts will be released under a permissive license upon acceptance.

Conclusion

NumBench turns T2I counting into a controlled diagnostic problem spanning counts 1–100, 1,600 categories, and 640,000 prompts. A spatial-collision model explains why limited coordination can produce rapidly growing visible deficits, while cw-NPS supports evaluation at scale. Across nine systems, requested count dominates all measured factors; layout and composition also matter, and no evaluated method retains strong numeric precision at high counts. Beyond aggregate scores, the benchmark separates cardinality, binding, and visibility failures, so a method can be credited for the specific failure family it addresses rather than for counting as a whole. Human and external evaluations support scalable use in the low-to-mid count regime, while higher counts remain stress tests, and the released prompts, detector proposals, and annotations allow every reported score to be audited and future systems to be evaluated under the same protocol. Progress requires coordinating many distinct instances while preserving category binding and countable visual evidence.

References Bakr, E. M.; Sun, P.; Shen, X.; Khan, F. F.; Li, L. E.; and Elhoseiny, M. 2023. HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 20041–20053. Binyamin, L.; Tewel, Y.; Segev, H.; Hirsch, E.; Rassin, R.; and Chechik, G. 2025. Make It Count: Text-to-Image Generation with an Accurate Number of Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13242–13251. Cho, J.; Zala, A.; and Bansal, M. 2023. DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3043–3054. Dahary, O.; Patashnik, O.; Aberman, K.; and Cohen-Or, D. 2024. Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation. In Proceedings of the European Conference on Computer Vision, 432–448. Ghosh, D.; Hajishirzi, H.; and Schmidt, L. 2023. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. In Advances in Neural Information Processing Systems, volume 36, 52132–52152. Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, 1321–1330. PMLR. Gupta, A.; Narayan, S.; Joseph, K. J.; Khan, S.; Khan, F. S.; and Shah, M. 2022. OW-DETR: Open-World Detection Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9235–9244. Hu, Y.; Liu, B.; Kasai, J.; Wang, Y.; Ostendorf, M.; Krishna, R.; and Smith, N. A. 2023. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 20406–20417. Huang, K.; Sun, K.; Xie, E.; Li, Z.; and Liu, X. 2023. T2ICompBench: A Comprehensive Benchmark for Open-World Compositional Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, 78723– 78747. Kajić, I.; Wiles, O.; Albuquerque, I.; Bauer, M.; Wang, S.; Pont-Tuset, J.; and Nematzadeh, A. 2024. Evaluating Numerical Reasoning in Text-to-Image Models. In Advances in Neural Information Processing Systems, volume 37, 42211– 42224. Kang, W.; Galim, K.; Koo, H. I.; and Cho, N. I. 2025. Counting Guidance for High Fidelity Text-to-Image Synthesis. In Proceedings of the Winter Conference on Applications of Computer Vision, 899–908. Li, Y.; Liu, H.; Wu, Q.; Mu, F.; Yang, J.; Gao, J.; Li, C.; and Lee, Y. J. 2023. GLIGEN: Open-Set Grounded Text-toImage Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22511– 22521.

Lian, L.; Li, B.; Yala, A.; and Darrell, T. 2024. LLMGrounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. Transactions on Machine Learning Research. Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; Zhu, J.; and Zhang, L. 2024. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In Proceedings of the European Conference on Computer Vision, 38–55. Minderer, M.; Gritsenko, A.; Stone, A.; Neumann, M.; Weissenborn, D.; Dosovitskiy, A.; Mahendran, A.; Arnab, A.; Dehghani, M.; Shen, Z.; Wang, X.; Zhai, X.; Kipf, T.; and Houlsby, N. 2022. Simple Open-Vocabulary Object Detection with Vision Transformers. In Proceedings of the European Conference on Computer Vision, 728–755. Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E.; Ghasemipour, S. K. S.; Ayan, B. K.; Mahdavi, S. S.; Gontijo-Lopes, R.; Salimans, T.; Ho, J.; Fleet, D. J.; and Norouzi, M. 2022. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In Advances in Neural Information Processing Systems, volume 35, 36479– 36494. Sun, W.; Du, Y.; Liu, G.; Zheng, Y.; and Snoek, C. G. M. 2026. QUOTA: Quantifying Objects with Text-to-Image Models for Any Domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 6381–6390. Wu, T.-H.; Lian, L.; Gonzalez, J. E.; Li, B.; and Darrell, T. 2024. Self-Correcting LLM-Controlled Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6327–6336. Zafar, O.; Cohen, Y.; Wolf, L.; and Schwartz, I. 2026. Detection-Driven Object Count Optimization for Text-toImage Diffusion Models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1885–1894. Zhang, L.; Rao, A.; and Agrawala, M. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836–3847.

Related documents

• NumBench: Diagnosing Counting Failures in Text-to-Image Models#784959
• NumBench: Diagnosing Counting Failures in Text-to-Image Models#783369
• DualCount: Structurally Consistent Density and Point Modeling for Zero-Shot Object Counting#1011345
• FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoders#927574
• Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild#353004