React-ing to Grace Hopper 200 Five Open-Weights Coding Models, One React Native App, One GH200, One Weekend
arXiv:2604.17187v1 [cs.SE] 19 Apr 2026
or: why SWE-Bench rankings mispredicted which model actually shipped
Alex Potanin School of Computing Australian National University [email protected] April 2026
Abstract We evaluate five state-of-the-art open-weights coding language models — Kimi-K2.5 (at Q3 and Q4 quantizations), GLM-5.1, Qwen3-Coder-480B, and DeepSeek-V3.2 — on a single multi-file React Native application generation task on NVIDIA GH200 576 GB hardware. The task specifies authentication, per-user per-day counting, and web compatibility, and is evaluated on whether the generated project runs out-of-the-box and on feature-level correctness. We find that SWE-Bench rankings do not predict task performance: Kimi-K2.5 at aggressive 3-bit quantization (UD-Q3_K_XL, 480 GB) produces the most complete and specification-compliant output, outranking models with substantially higher SWE-Bench Pro scores. We document three novel deployment findings: (1) default temperature=0 in coding tools causes sampling hangs with reasoning-model architectures, (2) reasoning-model thinking traces can leak through integration tools’ file-path parsers, and (3) web-platform adaptation of native-mobile APIs is a universal training-data gap across every model tested. We also map the hardware-tier structure of April 2026 open-weights coding models, identifying two architectural schools and showing that the efficiency school (10–15 B active parameters) delivers equivalent SWE-Bench results at roughly 1/7th the hardware cost of the scale school (32–40 B active parameters).
1
Introduction
By April 2026, open-weights coding language models have closed most of the gap with proprietary frontier systems on standard benchmarks. On SWE-Bench Verified, MiniMax M2.5 (230B/10B active, permissive license) reports 80.2%, within 1 pp of the best proprietary result.1 GLM-5.1 (754B/40B active, MIT licensed) achieves state-of-the-art 58.4% on SWE-Bench Pro, surpassing GPT-5.4 and Claude Opus 4.6 on that specific benchmark. Kimi-K2.5 (1T/32B active), DeepSeekV3.2 (671B/37B active), and Qwen3-Coder-480B (480B/35B active) all cluster within a few benchmark points. For self-hosting practitioners, the open question is different: given a natural product task — “build a working application that does X” — which model produces the best artifact, and is benchmark ranking a useful predictor? We investigate this with a single prompt that requires multi-file scaffolding, authentication, per-user data isolation, per-day semantic bucketing, and web-compatibility adaptation. 1 Throughout this paper, we report SWE-Bench Verified and SWE-Bench Pro numbers from the publishing labs’ own materials. Our contribution is orthogonal to their correctness: we evaluate a specific application-generation task, not reproduction of SWE-Bench itself.
1
Our findings are counterintuitive to benchmark-based selection and provide three actionable deployment lessons for teams planning institutional self-hosting.
2
Experimental Setup
2.1
Hardware and inference backend
All inference ran on a single NVIDIA GH200 576 GB node (96 GB HBM3 plus 480 GB LPDDR5X unified memory via NVLink-C2C, 72 ARMv8.2 cores) provided by the Australian National University Computer Vision group for a single weekend. We served models through llama-server from llama.cpp build b8797, using Unsloth Dynamic 2.0 GGUF quantizations in all cases. Expert FFN layers were offloaded to system memory via the –cpu-moe flag (equivalent to -ot ".ffn_.*_exps.=CPU"); attention layers and KV cache remained in HBM.
2.2
Client and orchestration
The client was a MacBook Pro M1 Max 64 GB running aider 0.86.2 in whole edit-format mode, connecting via SSH tunnel to the GH200. Sampling parameters were configured per each model publisher’s official recommendation: temperature 1.0 / top_p 0.95 for Kimi-K2.5, MiniMax, and DeepSeek-V3.2; temperature 0.6 / top_p 0.95 for GLM-5.1; temperature 0.7 / top_p 0.8 / top_k 20 for Qwen3-Coder.
2.3
The task
A single prompt was issued to each model under identical conditions: create react-native app that allows user to create account and login and then count kangaroos seen per day and make sure it runs on the web aider was configured to auto-commit and apply whole-file edits. We recorded token counts, generation speed, elapsed time, and the resulting project directory. Each generated project was then separately evaluated for: out-of-the-box runnability (npm install && npx expo start –web completes and the application starts), credential validation, per-user data isolation, per-day semantic bucketing of counts, history retention, working logout, and web-platform safety (no blocking reliance on native-only APIs such as Alert.alert).
2.4
Models tested
Table 1 lists the exact quantizations used and their on-disk sizes. All GGUFs were downloaded from Unsloth’s Hugging Face repositories in Dynamic 2.0 format, which uses per-tensor importance-aware quantization rather than uniform bit-width reduction. Table 1: Models evaluated. All quantizations are Unsloth Dynamic 2.0 GGUF. Model
Quant
Kimi-K2.5 Kimi-K2.5 GLM-5.1 Qwen3-Coder-480B DeepSeek-V3.2
UD-Q3_K_XL UD-Q4_K_XL UD-IQ4_XS UD-Q4_K_XL UD-Q4_K_XL
2
Size
Total / Active
License
480 GB 622 GB 370 GB 276 GB 430 GB
1T / 32B 1T / 32B 754B / 40B 480B / 35B 671B / 37B
Modified MIT Modified MIT MIT Apache 2.0 MIT
3
Deployment Findings
Before presenting application-level results, we document three deployment-level findings that affected the experiment itself and appear to be underdocumented in the community.
3.1
Default temperature=0 causes sampling hangs on reasoning models
Aider, like most code-edit tools descended from OpenAI’s function-calling conventions, defaults to temperature=0 to maximize determinism of code edits. For Kimi-K2.5 — which Moonshot documents as requiring temperature=1.0, top_p=0.95 — this caused multi-minute inference stalls with no visible output. nvidia-smi showed the server process in the R (running) state with 0% memory-bandwidth utilization and ≈ 2% SM utilization, indicating the sampler loop was spinning on low-probability token decisions rather than thrashing memory. We initially misdiagnosed this as HBM memory thrashing for the larger Kimi-K2.5 Q4 quantization (622 GB), which runs close to the unified-memory capacity of the GH200. Forcing temperature=1.0 and top_p=0.95 client-side via extra_params in aider’s model-metadata JSON restored generation to a healthy 7.9 tok/s for Q4 and 17 tok/s for Q3. Teams deploying reasoning-mode open-weights coding models with aider or similar tools should expect to override sampling defaults; model publishers’ recommended parameters are not optional.
3.2
Reasoning tokens can leak into integration-tool file-path parsers
Aider’s whole edit format asks the model to emit, per file, a line with the bare filename followed by a fenced code block containing the file contents. DeepSeek-V3.2 returned a response whose first line was Let’s start with App.js:</think>App.js — prose leading directly into a stray closing </think> tag from the model’s reasoning stream, followed by the intended filename. Aider interpreted the entire line as a file path and committed the generated content to "Let’s start with App.js:</think>App.js" instead of the project-root App.js, placing package.json’s main import out of reach and making the project non-bundlable. Other files in the same response parsed correctly. We are not aware of this failure mode being previously documented. It is distinct from the well-known problem of reasoning blocks consuming the output-token budget: here the reasoningto-content boundary visually resembles the next file-path marker well enough to corrupt parsers that expect clean newline-delimited filenames. Any tool that parses model output for filesystem actions needs a defensive filter that strips spurious thinking-block markers and prose-before-path lines before path interpretation.
3.3
Web-platform adaptation of native-mobile APIs is a universal trainingdata gap
Every model we tested used react-native’s Alert.alert for user-facing confirmations and validations. Alert.alert is a no-op on react-native-web; it returns silently without displaying anything. The task prompt explicitly required web compatibility (“make sure it runs on the web”), yet no model substituted window.confirm or conditional-platform dispatch. Concrete user-visible consequences ranged from silent form-validation failures (Qwen3-Coder blocking signup silently) to destructive-action buttons that do nothing (DeepSeek-V3.2’s “Clear History” button passes its destructive callback to an Alert that never fires). This suggests that React Native’s platform-divergence documentation is underrepresented in the post-training corpora used by all five labs. Because the failure is silent and the underlying code 3
compiles cleanly, it will not surface in benchmarks that grade on compilation or SWE-Bench-style patch-application, yet it breaks the task at the deployed-artifact level.
4
Application-Level Results
Table 2 summarizes the ranking. “Runs OOB” means npm install && npx expo start –web produced a bundled application that loaded in the browser without manual intervention. Table 2: Per-feature outcomes across the five models on the React Native kangaroo-counter task. Identical prompt, sampling parameters per publisher, whole-edit format. Kimi Q3
Kimi Q4
GLM-5.1
Qwen3-C
DS-V3.2
(a)
Runs out-of-the-box Auth validates credentials Per-user data isolation Per-day counting History persisted Logout works Web-safe (no Alert reliance)
✓ ✓ ✓ ✓ ✓7d ✓ ✗
✓ ✓ ✓ today only ✓7d ✓ ✗
✗ ✓ ✓ today only ✗ ✓ ✗
✓ ✓ ✓ ✗ ✗ ✓ ✗blocks signup
✗ (b) ✗ (c) ✗ ✗ (d) ✓ ✗ (e) ✗blocks save
Generation speed (tok/s) Output tokens (approx)
17 4.8 k
7.9 4.8 k
≈15 5.5 k
≈20 4.2 k
≈14 5.1 k
(a) GLM-5.1 generated a Firebase-based auth flow and did not inline any fallback credentials, so the app cannot start without a user-supplied firebaseConfig.js. (b) DeepSeek-V3.2’s App.js was committed under a directory path derived from prose spillover (Section 3.2), so the Expo entry point cannot resolve. (c) signIn(token) accepts any token; signUp is literally await signIn(token) with a hardcoded ’fake-jwt-token’. Any email/password combination logs in. (d) “Save Today’s Count” resets the counter to zero and appends a new history row, so one date can have multiple rows with partial counts. (e) HomeScreen.handleLogout clears the token in context but, per a code comment in the generated file, the model explicitly declined to implement navigation on logout. User remains on Home until page reload.
4.1
Discussion by model
Kimi-K2.5 Q3 (480 GB) — the quiet winner. Produced the only fully spec-compliant app. Per-user data keyed on userId_date, seven-day rolling history, imperative navigation with proper navigation.replace calls, working logout. The single weakness is the universal Alert.alert issue. 17 tok/s sustained generation from a 3-bit quantized 1-trillion-parameter model on a single 96 GB GPU is itself a notable result. Kimi-K2.5 Q4 (622 GB) — quality-equivalent, speed-halved. Produced a near-identical architectural design as Q3: same Context-based auth, same per-user isolation. The only behavioral regression from Q3 is that the counter is “today only” with a seven-day history, rather than Q3’s continuous per-day buckets. At 7.9 tok/s (vs Q3’s 17 tok/s), the Q4 quantization pays a roughly 2× speed penalty for a 1-feature quality gain. For single-user interactive use on this hardware, Q3 is the better operating point. GLM-5.1 IQ4_XS (370 GB) — technically correct, operationally useless. GLM5.1 chose Firebase Authentication over local credentials. The code is clean, the auth flow is real, per-user isolation is genuine. But the app cannot start without the user providing firebaseConfig.js. This is the opposite failure mode from DeepSeek-V3.2: a model that 4
over-engineered to an enterprise-tier solution when the prompt asked for something minimal. It suggests GLM-5.1’s post-training has weighted “production-looking” output heavily enough that it no longer interprets “create an account and login” as “local auth is fine”. The SOTA-on-SWEBench-Pro model produced the worst product decision of the five. Qwen3-Coder-480B Q4 (276 GB) — clean code, misread spec. Auth works, per-user isolation works, but the model treated “count kangaroos seen per day” as a single running counter with no date semantics. There is no per-day bucket, no history. Code quality per-file is high and the project starts cleanly, but the feature set is fundamentally a single-counter app rather than a logged-per-day app. This is the only non-reasoning model in the comparison (Qwen3-Coder explicitly does not emit <think> blocks), and the resulting response is the most concise, but the spec reading suffered. DeepSeek-V3.2 Q4 (430 GB) — broken at multiple levels. Tooling failure (prose in path, Section 3.2) prevents the project from bundling at all. Past the one-line filesystem fix, the auth layer accepts any credentials, per-user isolation is absent (history is stored under a single global kangarooHistory key), per-day semantics are incorrect (same-date counts append rather than accumulate), and logout is acknowledged as unimplemented in a code comment the model itself left behind. The README is the most polished of the five.
5
Hardware Implications
The task-level ranking is orthogonal to model size: Kimi Q3 at 480 GB beats DeepSeek-V3.2 at 430 GB; Qwen3-Coder at 276 GB beats GLM-5.1 at 370 GB. But the results interact with hardware in a specific way worth making explicit. Table 3 organizes the relevant open-weights landscape by target hardware tier. Two architectural schools are clearly visible: a scale school of models with 32–40 B active parameters that target institutional-tier hardware (400 GB+ unified memory, such as the GH200 used here or two 256 GB Apple Mac Studios in MLX cluster configuration), and an efficiency school of models with 10–15 B active parameters that target consumer-tier hardware (128–256 GB Apple Studio class, DGX Spark class). Table 3: Open-weights coding model landscape, April 2026. SWE-Bench is Verified (nonthinking) unless noted. Model
Total
Active
Q4 size
SWE-Bench
Scale school (32–40 B active) Kimi-K2.5 1T
32B
622 GB
76.8%
GLM-5.1 DeepSeek-V3.2
754B 671B
40B 37B
430 GB 430 GB
≈80 / 58.4 Pro 67.8 / ≈76 think
Qwen3-Coder-480B
480B
35B
276 GB
72–75%
Efficiency school (10–15 B active) MiniMax M2.5 230B
10B
230 GB FP8
80.2%
MiniMax M2.7 MiMo-V2-Flash
10B 15B
108 GB IQ4 187 GB Q4
≈80 ≈77
230B 309B
5
Studio 256 GB viable? No (only TQ1, unusable for code) No (only 1-bit quants fit) Tight (IQ2_XXS 240 GB) Yes, at IQ3_XXS 210 GB (≈57% Aider) Yes (FP8 fits with headroom) Yes (Q8 at 220 GB) Yes (Q4 fits with headroom)
For Mac Studio M3 Ultra 256 GB deployment (≈A$11 k), the scale-school models are structurally out of reach: only 1-bit or aggressive 2-bit quantizations fit, at which point communitymeasured quality degradation is substantial. The efficiency-school models fit comfortably, at roughly identical SWE-Bench Verified scores. The gap in cost-per-capability is large: a 2×256 GB Studio cluster reaches 512 GB at A$22 k, just barely fitting Kimi-K2.5 at Q3 (under MLX pipeline parallelism with inter-box Thunderbolt 5 bandwidth as the bottleneck), whereas a single 256 GB Studio running MiniMax M2.5 FP8 reports the same SWE-Bench Verified for half the hardware outlay. GH200-class hardware (>A$75 k list, or ≈US$30/hour cloud) delivers Kimi-K2.5 Q3 at 17 tok/s comfortably but against an efficiency-school baseline that is within benchmark variance at ≈1/7th the cost. The task-level finding (Section 4) and the hardware finding (this section) together suggest that, for self-hosting in 2026, model architecture matters more than parameter count for single-user coding. We observed this qualitatively: MiniMax M2.7 UD-IQ4_XS (108 GB, 10 B active) ran at 30 tok/s on the same hardware where Kimi-K2.5 Q3 (480 GB, 32 B active) ran at 17 tok/s.
6
Limitations
We emphasize several limitations of this study. Single-prompt, single-seed evaluation. Each model generated one application from one prompt with its publisher-recommended sampling parameters. We did not run multiple seeds, did not re-prompt after failure, did not allow iterative debugging, and did not vary the prompt wording. A multi-seed study would quantify the variance of each observation. One task, one stack. The task is a React Native web-compatible application. Findings about Alert.alert-on-web do not generalize to other stacks or other native-to-web translation boundaries. Findings about per-day bucketing may not generalize to tasks without implicit temporal semantics. Tool-specific results. Aider with whole-edit-format was the orchestration layer. Both the temperature-zero hang (Section 3.1) and the filepath-parsing leak (Section 3.2) are aider-specific in their surface manifestation, though we expect their underlying causes (reasoning-model sampling sensitivity, thinking-block leakage into structured output) to recur in other tool integrations. Quantization mix. We used one publisher’s dynamic quantizations (Unsloth) rather than native-FP8 weights where available. MiniMax M2.5 FP8 and Qwen3-Coder-480B FP8 were not evaluated here because they would require a vLLM-based inference stack that we did not stabilize during the weekend of GH200 access. Quality conclusions for those models in Table 3 rest on publisher-reported benchmarks, not our own measurement. Hardware availability and weekend scope. All measurements were collected during a single weekend of borrowed GH200 access. We did not explore full-precision inference, did not run standard SWE-Bench reproduction to cross-check publisher numbers, and did not test every model family (notably, MiMo-V2-Flash and MiniMax M2.5/M2.7 were characterized for hardware fit but not evaluated on the React Native task).
7
Conclusion
SWE-Bench Verified and SWE-Bench Pro rankings did not predict per-task outcomes on a concrete React Native application-generation task. Kimi-K2.5 at 3-bit quantization (the smallest Kimi we tested) produced the most complete application of the five models, outranking GLM5.1 (SOTA on SWE-Bench Pro) and DeepSeek-V3.2 (highest SWE-Bench Verified among the scale-school models).
6
The divergence tracks a specific pattern: the models that failed the task failed at integration and specification-reading rather than at code generation. GLM-5.1’s auth code is correct; its product decision (mandatory Firebase) made the app unusable. Qwen3-Coder-480B’s code is clean; it missed the per-day semantics in the prompt. DeepSeek-V3.2’s README is the most polished of the five; a reasoning-token leak caused its App.js to land in the wrong directory. Kimi-K2.5 Q3’s advantage is not sharper code per-file but a more complete and more conservative interpretation of what was asked. For practitioners choosing a self-hosted coding model in 2026, we suggest that benchmark ranking be treated as necessary but far from sufficient. Direct task-level evaluation on artifacts representative of the intended workload should be run before hardware commitments are made.
Data and Code Availability Generated projects, server and client configurations, and sampling-parameter settings are available on request. Models and quantizations used are listed with their exact Unsloth revision identifiers in Table 1; download commands were standard hf download invocations against the unsloth/ Hugging Face namespace.
Acknowledgments The NVIDIA GH200 576 GB node used for all measurements was loaned by the Computer Vision group at the Australian National University School of Computing for a single weekend; we gratefully acknowledge their generosity. Anthropic’s Claude Opus 4.6 was used throughout the experiment as a pair-debugging and writing assistant: it helped draft the llama.cpp server configurations, the aider metadata and configuration files, the speed-test scripts, and the download orchestration; it assisted in diagnosing the temperature=0 sampling-hang (Section 3.1) and the App.js-in-prose-path parsing failure (Section 3.2) through iterative conversation; and it helped draft and edit this paper. All experimental decisions, all measurements, and all interpretations are the author’s. Any errors are also the author’s. This work was carried out independently of the author’s other roles. It is not funded by any specific grant and does not represent the position of any affiliated institution.
References [1] Moonshot AI. Kimi-K2.5 model card. https://huggingface.co/moonshotai/Kimi-K2.5, 2026. [2] Z.AI. GLM-5.1: From Vibe Coding to Agentic Engineering. https://huggingface.co/ zai-org/GLM-5.1, 2026. [3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. https://huggingface.co/deepseek-ai/DeepSeek-V3.2, 2025. [4] Qwen Team. Qwen3-Coder-480B-A35B-Instruct. Qwen3-Coder-480B-A35B-Instruct, 2025.
https://huggingface.co/Qwen/
[5] MiniMax AI. MiniMax-M2.5 model card. MiniMax-M2.5, 2026.
https://huggingface.co/MiniMaxAI/
[6] MiniMax AI. MiniMax-M2.7 model card. MiniMax-M2.7, 2026.
https://huggingface.co/MiniMaxAI/
7
[7] Xiaomi LLM-Core. MiMo-V2-Flash: Multi-Token Prediction with Hybrid Attention. https: //huggingface.co/XiaomiMiMo/MiMo-V2-Flash, 2025. [8] Unsloth AI. Dynamic 2.0 GGUF quantization methodology. https://unsloth.ai/blog/ dynamic-v2, 2025. [9] Paul Gauthier. Aider: AI pair programming in your terminal. https://aider.chat, 2023–2026. [10] Georgi Gerganov and contributors. llama.cpp: Inference of Meta’s LLaMA model in pure C/C++. https://github.com/ggml-org/llama.cpp, 2023–2026. [11] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024. https://arxiv.org/abs/2310.06770.
8