Benchmarking System Dynamics AI Assistants: Cloud Versus Local Large Language Models on CLD Extraction and Discussion Terry Leitch1
arXiv:2604.18566v1 [cs.AI] 20 Apr 2026
1
ruxton.ai, [email protected] April 2026
Abstract We present a systematic evaluation of large language model families—spanning both proprietary cloud APIs and locally-hosted open-source models—on two purpose-built benchmarks for System Dynamics AI assistance: the CLD Leaderboard (53 tests, structured causal loop diagram extraction) and the Discussion Leaderboard (interactive model discussion, feedback explanation, and model building coaching). On CLD extraction, cloud models achieve 77–89% overall pass rates; the best local model reaches 77% (Kimi K2.5 GGUF Q3, zero-shot engine), matching mid-tier cloud performance. On Discussion, the best local models achieve 50–100% on model building steps and 47–75% on feedback explanation, but only 0–50% on error fixing—a category dominated by long-context prompts that expose memory limits in local deployments. A central contribution of this paper is a systematic analysis of model type effects on performance: we compare reasoning vs. instruction-tuned architectures, GGUF (llama.cpp) vs. MLX (mlx lm) backends, and quantization levels (Q3 / Q4 K M / MLX-3bit / MLX-4bit / MLX-6bit) across the same underlying model families. We find that backend choice has larger practical impact than quantization level: mlx lm does not enforce JSON schema constraints, requiring explicit prompt-level JSON instructions, while llama.cpp grammar-constrained sampling handles JSON reliably but causes indefinite generation on long-context prompts for dense models. We document the full parameter sweep (t, p, k) for all local models, cleaned timing data (stuck requests excluded), and a practitioner guide for running 671B–123B parameter models on Apple Silicon.
Contents 1 Introduction
4
2 Benchmarks 2.1 CLD Leaderboard (53 tests) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Discussion Leaderboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Prompt Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 4 5 5 5
3 Models Evaluated 3.1 Cloud API Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Local Open-Source Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 6 6
1
4 Model Type Effects on Performance 4.1 Architecture Class: Reasoning vs. Instruction-Tuned . . . . . . . . . . . . . . . . . . 4.2 Inference Backend: llama.cpp vs. mlx lm . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Quantization Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 7 8 9
5 CLD Leaderboard Results 5.1 Overall Leaderboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Category Deep Dive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Conformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Iterative Model Building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.4 Causal Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 9 9 9 9 10 11
6 Discussion Leaderboard Results 11 6.1 Per-Category Results (Best Variation) . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.2 Context Window Limits and OOM Failures . . . . . . . . . . . . . . . . . . . . . . . 12 7 Parameter Sensitivity
12
8 Timing and Throughput
12
9 Failed Models: Infrastructure vs. Model Quality 13 9.1 Mistral Large 2411 (Q6 K, 123B Dense) . . . . . . . . . . . . . . . . . . . . . . . . . 13 9.2 DeepSeek R1-0528 (IQ4NL, 671B) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 10 Practitioner Guide 14 10.1 mlx lm: JSON Output Requires Explicit Prompt Instructions . . . . . . . . . . . . . 14 10.2 mlx lm: No Context-Size Flag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 10.3 mlx lm: Metal OOM on Long Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . 14 10.4 Kimi K2.5: Thinking Mode Flag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 10.5 llama.cpp: Grammar Sampling Hangs on Dense Long-Context Models . . . . . . . . 14 10.6 Jinja Template Variables for Thinking Mode (Cross-Model) . . . . . . . . . . . . . . 14 10.7 mlx lm Python Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10.8 mlx lm Default Token Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10.9 Retry Pattern for Partial Runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10.10Stuck Server: Kill and Restart After Client Disconnect . . . . . . . . . . . . . . . . . 15 11 Cloud vs. Local Head-to-Head: Architecture Hypothesis Test 15 11.1 Hypothesis Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 11.2 Mixed-Model Analysis: Can Local Match Cloud Overall? . . . . . . . . . . . . . . . 16 11.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 12 Discussion 17 12.1 Architecture Class Is the Dominant Performance Predictor . . . . . . . . . . . . . . 17 12.2 Backend Choice Has Larger Practical Impact Than Quantization . . . . . . . . . . . 17 12.3 The Iteration Gap and Its Implications . . . . . . . . . . . . . . . . . . . . . . . . . . 18 12.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 13 Conclusion
18
2
A Energy Scenario Analysis: Cloud vs. Local Inference 19 A.1 Energy per Query and per Correct Answer . . . . . . . . . . . . . . . . . . . . . . . 19 A.2 The Utilisation Paradox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.3 Cooling and Physical Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.4 Idle Power: The Hidden Cost of GPU Infrastructure . . . . . . . . . . . . . . . . . . 21 A.5 Implications for AI Data Centre Design . . . . . . . . . . . . . . . . . . . . . . . . . 21 A.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 B Edge AI Appliances: Implications for System Dynamics Deployments 22 B.1 Data Centre vs. Edge: The Deployment Decision . . . . . . . . . . . . . . . . . . . . 22 B.2 Minimum Viable Hardware Tiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 B.3 Task Routing Extended to Hardware Routing . . . . . . . . . . . . . . . . . . . . . . 23 B.4 Data Sovereignty and Regulated Industries . . . . . . . . . . . . . . . . . . . . . . . 24 B.5 The SD AI Appliance Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 B.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3
1
Introduction
Causal loop diagrams (CLDs) are a foundational tool of the System Dynamics methodology, encoding feedback structure, polarity, and causal pathways among model variables [1]. Automating CLD extraction from natural-language text—and enabling AI-assisted model discussion and critique— could substantially accelerate System Dynamics model construction and the teaching of the method. Large language models (LLMs) are natural candidates: they must parse causal language, identify polarities, track loop semantics, and produce structured graph output. Yet structured extraction under strict schema constraints is among the more demanding LLM tasks [2], and iterative model refinement adds further complexity. The Discussion task additionally requires models to reason about model dynamics, identify feedback-driven behaviour, and diagnose formulation errors—tasks that require genuine domain understanding rather than pattern-matching. This paper evaluates a broad set of LLMs across both task families and contributes: 1. Ranked leaderboards for CLD extraction and Discussion tasks, comparing cloud and local models. 2. A systematic analysis of model type effects: reasoning vs. instruction-tuned architectures, GGUF vs. MLX backends, and quantization levels. 3. A full parameter sweep (t, p, k) for all local models, with analysis of optimal configurations per model class. 4. Cleaned timing analysis (stuck requests excluded) by task category for the top local models. 5. A practitioner guide documenting deployment challenges—JSON enforcement, context limits, template bugs, and crash recovery— encountered when running frontier open-source models on Apple Silicon.
2
Benchmarks
2.1
CLD Leaderboard (53 tests)
Each test presents a text passage and asks the model to extract a structured CLD in JSON format, encoding variables, directed links, polarities, and optionally loop labels. Tests are evaluated by exact structured match against a ground-truth schema: a response is scored pass if and only if the extracted JSON matches the ground-truth on all required fields (variable names, edge directions, polarities, and any specified cardinality constraints), after normalisation of variable name casing and whitespace. Reasoning traces or chain-of-thought content prior to the final JSON are ignored in scoring; only the final structured output is evaluated. All benchmark code, ground-truth schemas, raw results JSON, and model profiles are available at https://github.com/sd-ai/sd-ai (path: evals/). Model profiles (evals/model-profiles/*.json) document the exact LM Studio and mlx lm settings used for each model; run configurations (evals/run-configs/*.json) reproduce the exact parameter sweeps. Conformance (18 tests). Schema compliance: cardinality constraints, required vs. optional fields, correct polarity encoding. Models must not hallucinate extra variables or omit required links. Qualitative Causal Reasoning (3 tests). Second-order reasoning: identify mediated causation, distinguish direct from feedback-driven effects, handle ambiguous causal language. 4
Iterative Model Building (8 tests). The model receives an existing CLD plus a passage describing additions, and must return the updated CLD with all pre-existing relationships preserved exactly. Qualitative Translation (24 tests). Extract a full CLD from natural-language system descriptions spanning ecology, economics, public health, and engineering domains.
2.2
Discussion Leaderboard
The Discussion benchmark evaluates models acting as a System Dynamics mentor/critic via the Seldon engine. Tests are drawn from three categories: Model Building Steps. Given a modelling question, provide structured coaching toward correct model construction steps. Tests range from simple (single-step guidance) to medium (multi-step modelling sequences). Feedback Explanation. Explain the feedback dynamics driving model behaviour. Simple tests involve single-loop explanations; medium tests require multi-loop causal tracing. Error Fixing Suggestions. Identify and explain formulation errors in a provided model. These tests carry the longest prompts (80–146k tokens including full model context) and are the most demanding in terms of context window requirements.
2.3
Prompt Engines
For CLD extraction, tests run under two prompt strategies, referred to consistently throughout this paper as: • few-shot (qualitative engine): chain-of-thought system prompt with structured extraction examples included in context. • zero-shot (qualitative-zero engine): task instructions only, no in-context examples. Both engine names appear in result table footnotes where space requires abbreviation (qual./qual-zero). For Discussion, all tests run under the Seldon mentor engine.
2.4
Experimental Setup
All runs use seed 4242. Local models are served via LM Studio (v1.x) or mlx lm.server on an Apple Mac Studio (M3 Ultra, 512 GB unified memory, 2025) and queried through an OpenAIcompatible REST API [9]. Cloud APIs are accessed via their respective SDKs. Concurrency is set to 1 and tests run sequentially to avoid memory contention. 1
5
Table 1: Cloud API models evaluated. Model
Family
Type
Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Pro Preview Gemini 3.1 Pro Preview Gemini 3 Flash Preview GPT-5.1 GPT-5.2 Claude Sonnet 4.5 Claude Opus 4.5 o4-mini
Google Gemini Google Gemini Google Gemini Google Gemini Google Gemini OpenAI OpenAI Anthropic Anthropic OpenAI
Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Instruction-tuned Reasoning
Table 2: Local models evaluated. Backend: L=llama.cpp, M=mlx lm. Type: R=Reasoning, I=Instruction-tuned. Model
Quant
Be.
Ty.
Params
Ctx
Storage
Kimi K2.5 Kimi K2.5 DeepSeek V3.2 DeepSeek V3.2 Qwen 3.5 397B Qwen 3.5 397B GLM-5 Llama 4 Maverick Mistral Large 2411 DeepSeek R1-0528
GGUF Q3 MLX-3bit Q4 K M MLX-4bit Q4 K M MLX-6bit MLX-4bit Q4 K S Q6 K IQ4NL
L M L M L M M L L L
R R I I I I R I I R
671B 671B ∼671B ∼671B 397B 397B ∼9B 17B×128E 123B ∼671B
16–197k 164k 64–164k 164k 64k 64k 64–197k 64k 64k 33k
∼250 GB ∼200 GB ∼400 GB ∼320 GB ∼230 GB 301 GB ∼6 GB ∼100 GB ∼92 GB ∼360 GB
3
Models Evaluated
3.1
Cloud API Models
3.2
Local Open-Source Models
Mistral Large and DeepSeek R1 were excluded from the main leaderboard comparison due to systematic deployment failures unrelated to model capability (see Section 9).
4
Model Type Effects on Performance
A key contribution of this evaluation is systematic evidence that how a model is deployed —architecture class, inference backend, and quantization format—has material impact on task performance, independent of the underlying model weights. 1 The 512 GB M3 Ultra configuration was available as a custom-order option and is no longer listed in Apple’s current Mac Studio lineup. Specifications are documented in [9].
6
4.1
Architecture Class: Reasoning vs. Instruction-Tuned
We classify models into two architecture classes based on whether they generate an explicit internal reasoning (“thinking”) chain before producing their final answer. This is an operational classification based on observable output behaviour and published model descriptions, not a claim about internal architecture. • Reasoning models: Kimi K2.5, GLM-5, DeepSeek R1, o4-mini. These models emit extended chain-of-thought tokens or a reasoning content field before the final answer. o4-mini is classified as reasoning based on OpenAI’s published description of its “o-series” chain-of-thought design [4]. • Instruction-tuned models: DeepSeek V3.2, Qwen 3.5, Llama 4 Maverick, Mistral Large, and the remaining cloud API models. These produce responses without an observable extended thinking phase. This binary classification is a simplification; some instruction-tuned models may use internal chainof-thought not exposed via the API. The observed behavioural differences (temperature sensitivity, zero-shot preference) are empirical patterns from this benchmark, not theoretical predictions from the classification. Temperature sensitivity diverges by class (observed tendency). In this benchmark, reasoning models exhibit stronger performance degradation at t>0 than instruction-tuned models, though the evaluated set is small and confounds model family, parameter count, and deployment tooling. Kimi K2.5 peaks at t=0 (greedy) and degrades measurably at t=0.1. GLM-5 drops from 60% to 34% at t=0.3—a 26pp drop within a single model configuration. Instruction-tuned models (Qwen 3.5, Llama 4 Maverick) show minimal sensitivity (≤4pp variation across t ∈ {0, 0.1}) on this benchmark. These are observed tendencies in this evaluation, not general laws; broader validation across more models and tasks would be required to draw stronger conclusions. Zero-shot engine preferred by reasoning models. Reasoning models show a consistent preference for zero-shot prompting over few-shot in this benchmark (most clearly Kimi K2.5: +13pp, and in the excluded-model analysis DeepSeek R1), whereas some instruction-tuned models benefit more from few-shot structural guidance. GLM-5 is an exception in our local set: despite being operationally classified here as a reasoning model, its best results occur under the few-shot engine. The mechanism: few-shot extraction examples anchor the reasoning chain of thinking models to the example’s structure, causing them to follow the example’s causal pattern even when it does not match the target passage. Instruction-tuned models lack this problem—they treat examples as format guidance without reasoning over their content. Top-k sampling: effects across model classes. Adding k=20 alongside p=0.9 yielded the best or near-best result for GLM-5 (60%) and also improved DeepSeek V3.2 under the tested configuration (66%), but this pattern should not be interpreted as limited to the operationally defined reasoning class. Llama 4 Maverick and Qwen 3.5 (instruction-tuned) showed no benefit from the same combination. The combination narrows the candidate distribution in a complementary way—top-p caps probability mass, top-k caps candidate count—and its interaction with architecture class warrants further investigation.
7
Iteration task: reasoning models do not automatically win. Despite their reasoning chains, most reasoning models fail the iterative CLD update task. Kimi K2.5 scores only 0–3/8 on iteration; DeepSeek R1 scores 0/8. GLM-5 (reasoning, 9B) achieves 6/8—better than all other local models regardless of architecture class. This suggests that iterative update capability is not a direct function of chain-of-thought depth but may relate to training data composition or architectural memory features.
4.2
Inference Backend: llama.cpp vs. mlx lm
We ran matched model families under both backends (DeepSeek V3.2, Kimi K2.5, Qwen 3.5) to isolate backend effects from weight effects. JSON schema enforcement.
The most significant backend difference is JSON output reliability:
• llama.cpp: Supports response format: {type:"json schema"} via grammar-constrained sampling. Enforces the schema at token level; output is always valid JSON matching the Zod schema. • mlx lm: Silently ignores response format entirely—both json schema and json object modes. Without explicit prompt-level JSON instructions, the model returns free-text narrative responses. This causes 100% “Bad JSON” failures if the harness assumes schema enforcement. To correct for this, we appended an explicit JSON-only instruction to the system prompt when structuredOutput: false is set (mlx lm models): CRITICAL: Your entire response MUST be a single valid JSON object with no text before or after it. No markdown, no explanation, no code fences. Only output this exact structure: {“response”: “...”, “feedbackInformationRequired”: true—false}
After this fix, mlx lm models produced valid JSON reliably. Backend performance on CLD extraction. Table 3 compares DeepSeek V3.2 and Kimi K2.5 across backends at matched parameter configurations. Table 3: Backend comparison: GGUF/llama.cpp vs. MLX/mlx lm on CLD extraction (best variation, 53 tests). Scores are best variation per engine type. Model
Backend
DeepSeek V3.2 DeepSeek V3.2
Q4 K M (llama.cpp) MLX-4bit (mlx lm)
Kimi K2.5 Kimi K2.5
GGUF Q3 (llama.cpp) MLX-3bit (mlx lm)
Qwen 3.5 397B Qwen 3.5 397B
Q4 K M (llama.cpp) MLX-6bit (mlx lm)
Overall
C
CR
I
T
Avg(s)
70% 70%
11 17
1 2
4 1
22 19
209s 249s
77% n/a (DISCUSS only)
16 –
2 –
1 –
23 –
150s –
64% 62%
11 14
0 0
0 0
23 19
110s 261s
For DeepSeek V3.2, the backends produce the same overall score (70%) but with strikingly different category profiles: MLX-4bit excels at conformance (17/18 vs. 11/18) while Q4 K M leads on iteration (4/8 vs. 1/8) and translation (22/24 vs. 19/24). This pattern suggests that the backends apply chat templates and attention differently, with effects that are category-specific. For Qwen 3.5, Q4 K M edges MLX-6 (+2pp overall), primarily from translation gains in zeroshot mode. 8
Backend latency. On CLD tests (after removing stuck requests >3600s), llama.cpp is generally faster than mlx lm at the same quantization tier (Q4 K M: 110–209s vs. MLX-4/6: 249–261s avg). However, mlx lm avoids the grammar-constrained sampling hang that affects llama.cpp on longcontext dense models (Section 9).
4.3
Quantization Level
Q4 K M vs. MLX-6 (Qwen 3.5, 397B). At matched configurations (t=0.1, p=0.95, qualitative engine), Q4 K M and MLX-6 produce virtually identical scores (62% both), with minor categorylevel differences within measurement noise. This is consistent with the general finding that 4-bit quantization does not meaningfully degrade structured extraction quality at 397B scale [3]. Q3 vs. Q4 (Kimi K2.5). The MLX-3bit Kimi variant was only evaluated on Discussion simple groups (medium+ groups OOM at 164k context, Section 6.2). On the overlapping tests, MLX3bit performance is competitive: 75% on feedback explanation (vs. 67% for GGUF Q3) and 100% on model building steps (matching GGUF Q3). However, the 3-bit quantization limits the effective context to ∼60k tokens before Metal GPU memory is exhausted, severely restricting the test coverage achievable on this hardware. MLX-4bit vs. higher (GLM-5). GLM-5 at MLX-4bit achieves competitive scores (60% CLD, 75% iteration) despite being a 9B parameter model—far smaller than the 397–671B models in this evaluation. This suggests that for structured extraction tasks, model size interacts with architecture in non-obvious ways: GLM-5’s strong iteration performance cannot be explained by parameter count alone.
5
CLD Leaderboard Results
5.1
Overall Leaderboard
5.2
Category Deep Dive
5.2.1
Conformance
Cloud models cluster at 11–16/18. Kimi K2.5 (zero-shot, t=0) and DeepSeek V3.2 MLX-4 both reach 16–17/18—matching or exceeding Claude Opus 4.5 (16/18). The hardest conformance tests involve maximum-cardinality constraints: local models hallucinate extra links 49% of the time on these tests, versus 33% for cloud models. Minimum-cardinality tests are passed at much higher rates by both (72% local, 83% cloud), consistent with a general LLM bias toward comprehensiveness over constraint satisfaction. 5.2.2
Translation
Translation is the most competitive category for local models. Kimi K2.5 (zero-shot, t=0) achieves 23/24, equal to Gemini 2.5 Flash. DeepSeek V3.2 Q4KM and Qwen 3.5 Q4KM (zero-shot) both achieve 22–23/24. Even Llama 4 Maverick reaches 11–12/24. The one universally difficult test involves multi-hop causation through an implicit intermediate variable—all models show elevated failure here.
9
Table 4: CLD Leaderboard — best variation per model (53 tests). C=Conformance (/18), CR=Causal Reasoning (/3), I=Iteration (/8), T=Translation (/24). Best local result per column in bold. Model / Config
Overall
C
CR
I
T
Engine
89% (47/53) 85% (45/53) 83% (44/53) 83% (44/53) 81% (43/53) 81% (43/53) 79% (42/53) 75% (40/53)
15 13 15 13 16 11 13 14
1 0 1 0 0 1 0 2
7 8 5 8 6 7 7 1
24 24 23 23 21 24 22 23
qualitative qualitative qualitative qualitative qualitative qualitative qualitative qual-zero
Local Open-Source Models Kimi K2.5 GGUF Q3 77% (41/53) DeepSeek V3.2 Q4 K M 70% (37/53) DeepSeek V3.2 MLX-4 70% (37/53) Qwen 3.5 Q4 K M 64% (34/53) Qwen 3.5 MLX-6 62% (33/53) GLM-5 MLX-4 60% (32/53) Llama 4 Maverick Q4KS 47% (25/53)
16 11 17 11 14 10 14
2 1 2 0 0 1 0
1 4 1 0 0 6 0
23 22 19 23 19 15 12
qual-zero, t=0 qual-zero, t=0, p=1 qual-zero, t=0.1, p=0.95 qual-zero, t=0, p=1 qualitative, t=0.1 qualitative, t=0, k=20 qualitative
Cloud API Models Gemini 2.5 Flash Gemini 3.1 Pro Preview Claude Sonnet 4.5 GPT-5.1 Claude Opus 4.5 Gemini 3 Pro Preview Gemini 2.5 Pro o4-mini
5.2.3
Iterative Model Building
Iteration is the sharpest cloud/local separator. Table 5: Iteration scores (out of 8) by model. Model Gemini 3.1 Pro / GPT-5.1 Gemini 2.5 Flash Claude Opus 4.5 GLM-5 MLX-4 (best) DeepSeek V3.2 Q4KM Kimi K2.5 GGUF Q3 Qwen 3.5 (all configs) Llama 4 Maverick
Score 8/8 7/8 6/8 6/8 4/8 1/8 0/8 0/8
Notes Cloud leaders
Best local; 9B model Zero-shot preferred Did not succeed on iteration under tested configurations
Failure analysis shows a consistent local model pattern: models drop pre-existing relationships from the prior CLD while hallucinating new feedback edges that “close loops” mentioned implicitly in the new text. This holds even when the system prompt explicitly instructs preservation. The failure rate increases with CLD length (tests 5–8 with 4–7 pre-existing relationships are passed by essentially no local model), consistent with a context-length-dependent degradation as the preservation instruction competes with the extraction goal. GLM-5’s 6/8 success—the only local model near cloud performance—remains an open question given its smaller scale (9B vs. 397–671B). Its architecture may include memory features that better separate prior from new context, or its training data may include more iterative-update examples.
10
5.2.4
Causal Reasoning
With only 3 tests, this category has very high variance; no statistically meaningful confidence intervals can be derived, and a single test represents 33pp. Results should be interpreted as indicative rather than definitive. Kimi K2.5 (few-shot engine, t=0) scores 2/3—the highest among all local models and equal to o4-mini. Most other models score 0–1/3. The tasks require explicit loopawareness (tracing second-order effects through balancing vs. reinforcing loops) that most current LLMs appear to lack regardless of scale.
6
Discussion Leaderboard Results
6.1
Per-Category Results (Best Variation)
Table 6 reports the best single variation per model per Discussion category. Stuck requests (>3600s) are excluded from timing; scores reflect valid responses only. Table 6: Discussion Leaderboard — best variation per model per category. Timing excludes stuck requests (>3600s). Model
Error Fixing Score Avg(s)
Feedback Expl. Score Avg(s)
Model Building Score Avg(s)
Kimi K2.5 GGUF Q3 DeepSeek V3.2 Q4KM Kimi K2.5 MLX-3bit GLM-5 MLX-4 DeepSeek V3.2 MLX-4 Llama 4 Maverick
50% 50% 0%* 0% 0% 0%
67% 67% 75% 17% 0% 33%
100% 100% 100% 50% 0% 0%
59s 975s 1s 1295s 41s 5s
1138s 1527s 381s 1235s 308s 57s
74s 42s 29s 18s 42s 18s
*MLX-3bit ran only simple groups (medium+ prompts OOM at 164k context)
Model Building Steps. Three models achieve a perfect 100%: Kimi K2.5 GGUF Q3, DeepSeek V3.2 Q4KM, and Kimi K2.5 MLX-3bit (on simple tests). DeepSeek V3.2 is notably faster on model building (42s vs. 74s for GGUF Q3), reflecting more direct responses without extended reasoning chains. GLM-5 achieves only 50%—a surprise given its CLD iteration strength, suggesting that model building coaching requires different capabilities than structured graph update. Feedback Explanation. This is the category where MLX-3bit Kimi outperforms GGUF Q3 (75% vs. 67%). The mlx lm backend, despite its lack of JSON schema enforcement, produces higher-quality explanatory content on simple feedback questions. The latency advantage is also substantial: 381s vs. 1138s, reflecting the smaller effective prompt size in the simple group. GLM5 scores only 17% despite scoring 60%+ on CLD extraction, confirming that feedback dynamics explanation requires deeper domain-grounded reasoning than schema compliance. Error Fixing. Only Kimi K2.5 GGUF Q3 and DeepSeek V3.2 Q4KM achieve non-zero scores (both at 50%). This category carries the longest prompts (80–146k tokens); mlx lm models cannot process them without OOM crashes at the tested context settings. The 50% ceiling for all models suggests that even the top local models struggle with the complex multi-error formulation analysis these tests require.
11
6.2
Context Window Limits and OOM Failures
The error fixing tests expose a hard constraint for mlx lm-based models: prompts of 80–146k tokens exhaust Metal GPU memory on the Mac Studio at 164k context, even with 512 GB unified RAM. The failure mode is an uncaught std::runtime error from Metal: “Command buffer execution failed: Insufficient Memory (kIOGPUCommandBufferCallbackErrorOutOfMemory)”. This occurs because the Metal command buffer for attention over 80k+ tokens exceeds the GPU’s addressable command memory, which is a separate constraint from system RAM. GGUF models served by llama.cpp do not exhibit this failure because llama.cpp manages KVcache as CPU-accessible unified memory (pageable), not as a fixed GPU command buffer. Implication: For long-context Discussion tasks, llama.cpp backends are required. mlx lm is competitive only for prompts up to ∼60k tokens on this hardware.
7
Parameter Sensitivity
Table 7: Impact of temperature on CLD overall pass rate (qualitative engine, 53 tests). Best temperature per model in bold. Model
t=0, p=1
t=0, p=0.95
t=0, p=0.9/k=20
t=0.1
t=0.3
64% 57%
– 55%
– 60%
57% 58%
– 34%
Instruction-tuned models DeepSeek V3.2 62% Qwen 3.5 Q4KM 62% Llama 4 Maverick 47%
57% 62% 47%
66% 58% 47%
62% 62% 47%
– – –
Reasoning models Kimi K2.5 (qual.) GLM-5 MLX-4
For Discussion tasks, Kimi K2.5 MLX-3bit peaks at t=0.1/p=0.9/k=20 (75% feedback explanation, 100% model building)—slightly different from the CLD optimum, suggesting that the conversational Discussion task benefits from marginal sampling diversity where the structured CLD task does not.
8
Timing and Throughput
Table 8 summarises per-category timing for the top 3 local models on CLD (stuck requests >3600s excluded). Stuck requests were uniformly from long-context tests where the server continued processing an orphaned HTTP request after client disconnect; excluding them yields clean per-response latency. Kimi K2.5 tail latency at t=0.1. At t=0 (greedy), Kimi K2.5 averages 150s/test. At t=0.1, individual translation tests required up to 250 minutes, with total run time exceeding 22 hours. The root cause is reasoning chain non-termination: non-zero temperature can trigger extended self-critique loops before the model converges. At t=0, the chain follows a deterministic path and terminates promptly. For very large reasoning models, t=0 wins on both accuracy and speed.
12
Table 8: CLD per-category avg latency, top 3 local models (stuck requests removed). Table uses the clean-timing subset only; therefore Kimi’s overall score is 76% here rather than the full-leaderboard 77%. Model Kimi K2.5 GGUF Q3 DeepSeek V3.2 Q4KM Qwen 3.5 Q4KM
Conform.
Causal
Iter.
Trans.
Overall
Score
227s 240s 148s
145s 311s 176s
78s 265s 0s*
114s 157s 123s
150s 209s 110s
76% 67% 62%
*0s = all timed out (0% score on iteration)
Cloud model reference. For comparison: Claude Sonnet 4.5 averages 17s/test (83% score); Gemini 2.5 Flash averages 17s/test (89% score). The best local model (Kimi K2.5, 150s avg) is 9× slower than the best cloud model at comparable accuracy.
9
Failed Models: Infrastructure vs. Model Quality
Note: the models below are excluded from the main leaderboard due to infrastructure failures, not model capability ceilings. Partial success scores are reported here for completeness.
9.1
Mistral Large 2411 (Q6 K, 123B Dense)
All 4 configurations scored 11% (6/53), with 0/24 translation and 0/8 iteration. Root cause: llama.cpp grammar-constrained sampling causes indefinite generation on large-context prompts for dense models. Iteration tests include 13k–40k token prompts; grammar enforcement enters loops that never terminate on these. The 6/53 passes came exclusively from short-context conformance tests. This is a known llama.cpp issue with grammar-constrained decoding under long KV-cache for dense (non-MoE) architectures. Partial result: 11% (6/53) on conformance-only tests under working configurations. This reflects a deployment failure, not a model capability ceiling; re-evaluation under plain JSON mode is recommended before drawing conclusions about Mistral Large’s capability on these tasks.
9.2
DeepSeek R1-0528 (IQ4NL, 671B)
R1 scored 19% (10/53) on the qualitative engine and 11% on zero-shot, with 0/24 translation. Root cause: at t=0, R1 generates the “no relationships” example JSON (empty structure) for any translation prompt with the full few-shot system prompt (∼13k chars). Direct API calls with short prompts work correctly, confirming this is a prompt-length anchoring interaction, not a capability failure. Additionally, a harness bug was discovered: systemModeUser was set to ’developer’ instead of ’system’, causing all system messages to be silently dropped in R1’s Jinja template. Partial result: 19% (10/53) on short-context conformance tests before the prompt-length failure was identified. R1’s CLD capability under corrected conditions remains an open question.
13
10
Practitioner Guide
10.1
mlx lm: JSON Output Requires Explicit Prompt Instructions
mlx lm.server ignores response format (both json schema and json object). Without intervention, the model returns free-text narrative responses, causing 100% JSON parse failures. Fix: append an explicit JSON-only instruction to the system prompt when using mlx lm backends. The instruction must specify the exact field names and types; a generic “respond in JSON” is insufficient for reliable compliance.
10.2
mlx lm: No Context-Size Flag
mlx lm.server does not expose a --context-size flag (as of the version tested). Context window is set by the model’s default or by LM Studio. The contextLoaded field in a model profile is metadata only—verify actual context via GET /v1/models.
10.3
mlx lm: Metal OOM on Long Prompts
Prompts >60–80k tokens can exhaust the Metal GPU command buffer memory regardless of system RAM. The error is a hard crash: [METAL] Command buffer execution failed: Insufficient Memory. Use llama.cpp for any task with prompts exceeding this length. There is no workaround short of reducing context.
10.4
Kimi K2.5: Thinking Mode Flag
Both Kimi K2.5 variants require explicit opt-out of reasoning mode: • GGUF via LM Studio: Set reasoningParsing:
false in the model profile’s lmStudioSettings.
• MLX via mlx lm.server: Pass --chat-template-args ’{"thinking": false}’ at server start. The correct variable name is thinking (not thinking mode); verify via the model’s chat template.jinja if uncertain. Without this, Kimi routes all output to reasoning content and returns an empty content field, causing parse failures.
10.5
llama.cpp: Grammar Sampling Hangs on Dense Long-Context Models
Grammar-constrained sampling (json schema mode) in llama.cpp can hang indefinitely on prompts >10k tokens for dense (non-MoE) models. This affects Mistral Large and potentially other dense architectures. MoE models (Kimi, DeepSeek, Qwen) do not exhibit the same behaviour. If a dense model is required: disable grammar sampling, use plain JSON mode or explicit prompt instructions, and set aggressive timeouts.
10.6
Jinja Template Variables for Thinking Mode (Cross-Model)
• Kimi K2.5: thinking:
false
• DeepSeek models: thinking mode:
"chat" or thinking:
• Qwen 3 models: enable thinking:
False
false (check model’s chat template.jinja)
LM Studio reads from tokenizer config.json (field: chat template), not a separate .jinja file. Both files must be patched in sync if overriding defaults. 14
10.7
mlx lm Python Version
Important: use Python 3.13 (python3.13 -m mlx lm server ...). The legacy Python 3.9 install silently falls back to incomplete model implementations for newer architectures (Kimi K2.5, DeepSeek V3.2), producing wrong results with no error message. This is one of the most common silent failure modes for MLX users.
10.8
mlx lm Default Token Limit
mlx lm.server defaults to max tokens=512. Set maxTokens in the model profile to at least 8,000 for any reasoning model whose thinking chain may exceed 512 tokens.
10.9
Retry Pattern for Partial Runs
After any crash or timeout, use --retry-from <results.json> --errors-only to re-run only failed tests. Passing results are read from the prior file, avoiding re-running potentially thousands of tests.
10.10
Stuck Server: Kill and Restart After Client Disconnect
When a benchmark client disconnects mid-request (timeout, crash), mlx lm continues processing the orphaned prompt internally. If the next request arrives while the prior prompt is still processing, the server queues it—causing multi-hour waits. After any abnormal client exit, kill the server process and restart before the next run.
11
Cloud vs. Local Head-to-Head: Architecture Hypothesis Test
To evaluate whether our architecture-class conclusions hold against the current state of the art, we compare the top 3 cloud models with the top 4 local models across all four CLD categories. We include 4 local models (rather than 3) because GLM-5, despite ranking 9th overall (60%), is the only local model with competitive iteration performance and is essential for the mixed-model analysis below. Cloud timing is not captured; local timing reflects best-variation averages with stuck requests (>3600s) excluded. Cells where a local model matches or exceeds the cloud leader are marked † and represent direct deployment substitution opportunities.
11.1
Hypothesis Evaluation
H1: Translation is competitive for local models — Supported in this benchmark. Kimi K2.5 achieves 96% on translation (vs. 100% cloud ceiling), a gap of just one test. DeepSeek V3.2 Q4KM achieves 92%. The remaining 4–8pp gap likely reflects a small number of multi-hop implicitcausation tests rather than a systematic capability difference. H2: Iteration is the sharpest cloud/local divide — Supported in this benchmark. Cloud leaders score 88–100% on iteration; the best local model overall (Kimi K2.5) scores only 13%. DeepSeek V3.2 reaches 50%—the strongest result among the overall top-3 local models, but still 38–50pp below the cloud ceiling. GLM-5 (60% overall) is the lone local outlier at 75%, confirming that iteration capability is not a function of overall model strength but appears architecture- or training-specific.
15
Table 9: Cloud vs. local head-to-head (CLD, best variation per model). C=Conformance (/18), CR=Causal Reasoning (/3), I=Iteration (/8), T=Translation (/24). † = local matches or exceeds best cloud score in that category. Model
Overall
C
CR
I
T
Avg/test
Cloud (best = Gemini 2.5 Flash) Gemini 2.5 Flash Gemini 3.1 Pro Preview GPT-5.1
89% 85% 83%
83% 72% 72%
33% 0% 0%
88% 100% 100%
100% 100% 96%
n/a n/a n/a
Local Kimi K2.5 GGUF Q3† (zero, t=0) DeepSeek V3.2 Q4KM (zero, t=0) DeepSeek V3.2 MLX-4 (zero, t=0.1) GLM-5 MLX-4 (qual., t=0, k=20)
77% 70% 70% 60%
89%† 61% 94%† 56%
33%† 0% 0% 33%†
13% 50% 13% 75%
96% 92% 79% 63%
116s 169s 206s 119s
Mixed local (best model per category)
91%
94%†
67%†
75%
96%†
—
H3: Reasoning models lead on causal reasoning — Consistent with the benchmark evidence, with substantial uncertainty due to n=3. The only local models to score on causal reasoning at all are Kimi K2.5 (reasoning, 33%†—matching the cloud leader) and GLM-5 (reasoning, 33%†—also matching). All instruction-tuned local models score 0%, consistent with the hypothesis. Notably, two of three top cloud models also score 0% on causal reasoning, suggesting this category is genuinely hard regardless of deployment tier—reasoning architecture is necessary but not sufficient. H4: Local reasoning models lead on conformance — Supported in this benchmark, and reversed. DeepSeek V3.2 MLX-4 (zero-shot) achieves 94% on conformance—11pp above the cloud leader (Gemini 2.5 Flash, 83%). Kimi K2.5 (zero-shot) achieves 89%, also above the cloud ceiling. These are the clearest deployment-substitution opportunities in the dataset: conformance checks can be run entirely locally with better accuracy than any current cloud model. H5: Architecture class predicts zero-shot preference — Broadly supported in the evaluated configurations. All four local models’ best configurations are either zero-shot (Kimi, DeepSeek) or use the qualitative (few-shot) engine only for the reasoning models that benefit from structural scaffolding (GLM-5). All three top cloud models use the qualitative few-shot engine. This aligns with the architecture hypothesis: reasoning models internalise structural guidance from thinking chains; instruction-tuned models benefit from in-context examples.
11.2
Mixed-Model Analysis: Can Local Match Cloud Overall?
If we route each query to the best-performing local model per category—rather than using a single model for everything—the combined result is: The mixed local stack achieves 91% overall—exceeding the best single cloud model (Gemini 2.5 Flash, 89%) by 2pp. Important caveat: this is a post hoc upper bound for a category-routed local ensemble, constructed by selecting the best local model per category after observing results. It is not a directly benchmarked end-to-end system. A deployed task router would require a reliable query classifier, and performance may differ from this theoretical ceiling. Nevertheless, the result is meaningful as a routing hypothesis: the required per-category performance already exists in the
16
Table 10: Mixed local model composition: best local model per category. † = local matches or exceeds best cloud score in that category. Combined 91% is a post hoc upper bound, not a benchmarked end-to-end system. Category
Best Local Model
Local Score
Cloud Best
Conformance Causal Reasoning Iteration Translation
DeepSeek V3.2 MLX-4 (zero) Kimi K2.5 GGUF Q3 (qual.) GLM-5 MLX-4 (qual.) Kimi K2.5 GGUF Q3 (zero)
94%† 67%† 75% 96%†
83% 33% 100% 100%
Combined
(task-routed)
91%
89%
local model pool, and no single category requires a model that cannot be run on the evaluated hardware. The only remaining local weakness is iteration (75% vs. 100% cloud), which is the one category where routing to a cloud model would still be warranted for maximum performance. For workflows that do not require iterative CLD updates—the majority of translation and conformance workloads—the benchmark evidence suggests that fully local deployment is feasible and can be competitive with, and in some categories exceed, current cloud API performance.
11.3
Summary
The head-to-head confirms that the cloud/local performance gap is not uniformly distributed. It is reversed on conformance (local leads by 11pp), essentially absent on translation (4pp gap), hard for everyone on causal reasoning (both tiers struggle), and real but not insurmountable on iteration. A task-routing architecture—DeepSeek V3.2 MLX-4 for conformance, Kimi K2.5 for translation and causal reasoning, GLM-5 for iteration— achieves 91% overall with no cloud dependency.
12
Discussion
12.1
Architecture Class Is the Dominant Performance Predictor
Across both benchmarks, model architecture class (reasoning vs. instruction-tuned) is a stronger predictor of task-specific performance than parameter count or quantization: • Reasoning models (Kimi K2.5, GLM-5) dominate on causal reasoning (2/3) and conformance (16–17/18), but require t=0 and zero-shot prompting. • Instruction-tuned models (DeepSeek V3.2, Qwen 3.5) are more robust to prompt style but have lower ceilings on causal tasks. • GLM-5 at 9B outperforms all other local models on iteration (6/8), demonstrating that architecturespecific factors—possibly training on iterative editing tasks—can dominate scale effects.
12.2
Backend Choice Has Larger Practical Impact Than Quantization
For practitioners, the choice of inference backend (llama.cpp vs. mlx lm) has more immediate impact on task success than quantization level:
17
• mlx lm’s lack of JSON enforcement requires explicit prompt engineering that cloud-oriented harnesses may not include by default. • llama.cpp’s grammar sampling is reliable for MoE models but dangerous for dense models on long-context prompts. • Quantization at Q4/MLX-4+ appears to have negligible accuracy impact at 397–671B scale for these structured extraction tasks.
12.3
The Iteration Gap and Its Implications
The near-complete failure of large local models on iterative CLD updates (with GLM-5 as the sole exception) suggests that iterative structured editing is a distinct capability not captured by overall language understanding. We hypothesise this requires either training data with document-editing examples (update the existing structure, preserve what was there) or architectural features that separate prior-context representation from in-context extraction. The practical implication: taskrouting systems that identify iterative update requests and route them to cloud models (or GLM-5) could achieve cloud-level performance without full cloud deployment costs.
12.4
Limitations
1. Single seed. All runs use seed 4242. Individual test pass/fail may vary; scores represent one deterministic evaluation. The causal reasoning category (n=3) is particularly susceptible to seed-level variance; results there should be treated as indicative. 2. Hardware specification. All benchmark runs reported here were conducted on an Apple Mac Studio (M3 Ultra, 512 GB unified memory, 2025). Appendix A uses published M-series Ultra power-draw figures as order-of-magnitude estimates rather than direct power-meter measurements. 3. Context window constraints. Kimi K2.5 GGUF ran at 16k context (LM Studio limitation at the time); its iteration scores might improve at 64k+. Kimi K2.5 MLX-3bit is limited to simple Discussion groups due to Metal OOM on longer prompts. 4. No prompt optimisation. No model-specific tuning was performed. Task-specific prompts would likely improve all results, especially for reasoning models on iterative tasks. 5. Infrastructure exclusions. Mistral Large and DeepSeek R1 results reflect tooling or promptlength failures, not model capability ceilings. 6. Architecture classification is behavioural. The “reasoning” vs. “instruction-tuned” classification used throughout is based on observable output behaviour (presence of extended chainof-thought tokens or a reasoning content field) and published model descriptions. For closedsource cloud models, the internal architecture may differ from the operationally defined class; the observed performance differences are empirical patterns from this benchmark, not predictions derived from confirmed architectural knowledge.
13
Conclusion
We evaluated local and cloud LLMs across CLD extraction and System Dynamics Discussion benchmarks, finding structured patterns of capability differences that are explained more by architecture class and deployment backend than by raw parameter count or quantization. 18
Key findings: • Best local model overall (CLD): Kimi K2.5 GGUF Q3 (zero-shot, t=0): 77%, matching mid-tier cloud. Use it for translation and conformance. • Best local for Discussion: Kimi K2.5 GGUF Q3 and DeepSeek V3.2 Q4KM share top honours (100% model building, 67% feedback explanation, 50% error fixing). • Best local for iteration (CLD): GLM-5 MLX-4 (6/8), despite 9B parameters. Route iterative update tasks here. • Backend matters more than quantization: mlx lm requires explicit JSON prompt engineering; llama.cpp requires caution on dense models at long context. Q4/Q3 quantization does not meaningfully degrade accuracy at 397B+ scale. • Reasoning models require t=0: Non-zero temperature causes both accuracy degradation and catastrophic tail latency in reasoning models. Zero-shot prompting is consistently preferred. • Task routing appears promising: The post hoc category-routed local upper bound reaches 91%, but a deployed router remains to be implemented and evaluated prospectively. All benchmark code, ground-truth schemas, raw results JSON, model profiles, and run configurations are available in the public repository at https://github.com/sd-ai/sd-ai (evals/).
A
Energy Scenario Analysis: Cloud vs. Local Inference
Note: This analysis combines measured wall-clock times from our benchmark with published energy estimates and manufacturer specifications. Cloud token costs and GPU server power draws carry uncertainty of ±50%; the directional conclusions are robust across this range. Output token counts were not directly measured; we use ∼800 tokens/query as a representative estimate for CLD responses. This section is explicitly framed as a scenario analysis, not a directly measured result. Figures should be treated as order-of-magnitude estimates. ∗ The 76% pass rate in Table 11 differs from the 77% leaderboard figure for Kimi K2.5 because the timing table uses only the clean-timing subset (stuck requests >3600s excluded, n=102 of 106 total); the 77% figure uses all responses.
A.1
Energy per Query and per Correct Answer
At face value, cloud appears 3.7–6.4× more energy-efficient per query. This reflects the enormous advantage of massive batching: a shared H100 cluster serves thousands of concurrent users across the same physical silicon, amortising hardware energy over many tokens. A dedicated Mac Studio, by contrast, runs one request at a time at full hardware power.
A.2
The Utilisation Paradox
The infrastructure comparisons below are modeled scenarios derived from benchmark latencies plus external power and utilisation assumptions; they are not direct metered measurements of deployed fleets. The per-token comparison above is misleading for organisations considering dedicated inference infrastructure, because it assumes the cloud hardware runs at near-100% utilisation. Real enterprise AI inference workloads are bursty—utilisation rates of 15–40% are used here as a plausible enterprise-utilisation scenario. 19
Table 11: Estimated energy consumption per benchmark query and per correct answer (best variation, clean timing data, stuck requests excluded). Mac Studio: 140 W inference draw, PUE 1.02 (convective cooling). Cloud: approximately 0.002 kWh/1,000 output tokens in the baseline scenario, including data-centre PUE 1.25 (literature estimate for batched H100 inference at high utilisation). Model
Wh/query
Wh/correct ans.
Pass rate
Avg latency
Cloud (shared, high-utilisation infrastructure) Gemini 2.5 Flash 1.6 Gemini 3.1 Pro Preview 1.6 GPT-5.1 1.6
1.8 1.9 1.9
89% 85% 83%
17s 51s 42s
Local (dedicated Mac Studio Ultra, 512 GB) Kimi K2.5 GGUF Q3∗ 5.9 DeepSeek V3.2 Q4KM 8.3 DeepSeek V3.2 MLX-4 10.3 GLM-5 MLX-4 9.1
7.8 12.5 16.9 17.3
76% 67% 61% 53%
150s 209s 260s 230s
For dedicated infrastructure serving 100,000 queries/day (approximately a mid-size enterprise deployment): Table 12: Infrastructure comparison: 100,000 queries/day, dedicated hardware. Mac Studio: 150 s/query, 174 units required (24/7). H100 estimate: 671B-class model on 2×H100 at ∼5 s/query with batching; 12 dual-H100 servers required. Platform
Units
Fleet power
Energy/day
Cooling
Mac Studio Ultra (512 GB) H100 dual-server nodes
174 12
24 kW 110 kW
583 kWh 2,640 kWh
Convective, no chiller Chilled water required
Ratio (Mac/H100)
×14
×0.22
×0.22
Dedicated Mac Studio infrastructure for the same throughput draws 4.5× less power and consumes 4.5× less energy than an equivalent H100 deployment, despite requiring 14× more physical units. The crossover point—where the H100 fleet’s batching efficiency reaches parity with Mac Studio—occurs near 85–90% sustained utilisation under the assumptions used here, a level that only the largest consumer AI services maintain.
A.3
Cooling and Physical Infrastructure
The energy comparison understates the total infrastructure difference because GPU servers require active cooling that Apple Silicon does not: • H100 data centre PUE: 1.2–1.5 typical (modern hyperscaler to legacy facility). Chilled water loops, CRAC units, raised floors, and power conditioning add 20–50% overhead on top of the compute draw. • Mac Studio PUE: ∼1.02. The M3 Ultra’s thermal design uses convective cooling within the chassis; no external chiller is required. A rack of Mac Studios can be cooled by standard office HVAC.
20
• Power density: H100 servers draw ∼10 kW/unit, requiring specialised high-density power distribution (30–60 A circuits per rack). Mac Studios draw 150 W/unit and run on standard 15 A office circuits. For organisations that do not already own GPU data-centre infrastructure, the build-out cost— chiller plants, power conditioning, raised flooring—often exceeds the hardware cost itself. A Mac Studio deployment requires no infrastructure beyond standard office power and networking.
A.4
Idle Power: The Hidden Cost of GPU Infrastructure
A critical factor absent from per-query analyses is idle power consumption. LLM inference workloads are inherently bursty: queries cluster around business hours, product events, and user sessions. A GPU server that is provisioned for peak load but sitting idle at 3 AM still draws 2–4 kW. • H100 server idle: ∼2–3 kW (GPU memory refresh, CPU, fans, networking). At 30% average utilisation, ∼70% of energy is consumed doing no useful work. • Mac Studio idle: ∼25 W. At 30% utilisation, the time-averaged power draw is 0.3 × 140 W + 0.7 × 25 W ≈ 60 W—versus ∼2,500 W for a comparable H100 server at 30% utilisation. At 30% utilisation, the effective energy per useful query is: Mac Studio :
140 W × 150 s/3600 ≈ 19 Wh/query 0.3
(0.3 × 5,600 + 0.7 × 2,500) W × 5 s/3600 ≈ 37 Wh/query 0.3 At 30% utilisation, Mac Studio is approximately 2× more energy-efficient per query than a dedicated H100 deployment, even before accounting for PUE differences. H100 server (2-GPU) :
A.5
Implications for AI Data Centre Design
The implications discussed here should be read as conditional on the scenario assumptions in A.1– A.4, especially utilisation, batching, and hardware-power estimates. These results suggest several non-obvious implications for organisations planning AI inference infrastructure: The efficiency crossover is utilisation-dependent. Shared cloud infrastructure (hyperscaler APIs) remains the most energy-efficient option per token for organisations whose workloads can coexist with other users. Dedicated GPU infrastructure only achieves cloud-level efficiency above ∼85% sustained utilisation—a threshold few enterprise deployments reach. Below this threshold, high-efficiency edge hardware (Apple Silicon or equivalent) is the more sustainable choice. Performance parity changes the calculus. The conventional assumption has been that local models sacrifice performance for privacy or cost. Our results demonstrate that for CLD extraction, the best local configuration (task-routed, 91%) exceeds the best single cloud model (89%). When performance parity is achievable, the energy argument for local deployment strengthens substantially.
21
Cooling infrastructure is a multiplier. For organisations in regions where electricity is expensive or carbon-intensive, the ∼1.3–1.5× PUE overhead of GPU data centres compounds the compute energy cost. A 200-node Mac Studio cluster deployable in standard office space, consuming ∼28 kW total, represents a qualitatively different infrastructure footprint than a 12-server H100 rack potentially requiring substantial facility upgrades, often on the order of hundreds of thousands of dollars. The data centre scaling question. The current wave of AI data centre construction—projected at hundreds of gigawatts globally by 2030 [10]—is predominantly designed around GPU servers. If inference workloads at enterprise and edge scale can be served by high-efficiency unified-memory hardware at 4–5× lower power density, the aggregate energy and water consumption of the AI industry could be substantially lower than current projections. The constraint is throughput per unit: Mac Studio handles one request at a time, limiting peak concurrency. Architectural advances that enable higher concurrency on unified-memory hardware (or equivalent Apple Silicon successors) could shift this calculus materially.
A.6
Limitations
1. Output token counts were not directly measured. The 800-token estimate affects per-query cloud energy; the relative comparison is stable across 500–1,500 tokens. 2. Cloud energy estimates (approximately 0.002 kWh/1,000 tokens in the baseline scenario) are derived from published literature and operator disclosures [11, 12]; actual figures vary by model, provider, and data-centre location. 3. Mac Studio power draw was not directly metered; 140 W is based on third-party measurements of M3 Ultra under sustained GPU load. 4. The H100 throughput estimate for 671B-class models assumes simple batching; in practice, speculative decoding and quantised server-side models may improve cloud throughput 2–4×.
B
Edge AI Appliances: Implications for System Dynamics Deployments
B.1
Data Centre vs. Edge: The Deployment Decision
The conventional framing of AI deployment presents two options: consume cloud APIs, or build a data centre. Our benchmarks suggest a third path—edge AI appliances—that is already viable for domain-specific AI assistance at the performance levels required for System Dynamics work. An edge AI appliance is a self-contained inference device that runs locally at the point of use: a research institution, a university classroom, a government agency, or an enterprise with data-residency requirements. The Mac Studio used throughout this benchmark is, functionally, such a device. It requires no specialised infrastructure, runs silently on office power, and can be administered by a non-specialist. The critical question is whether an edge appliance can deliver useful performance. This benchmark provides a direct answer for the SD AI use case.
22
B.2
Minimum Viable Hardware Tiers
Except where explicitly marked as measured on the benchmark platform, the tier estimates in this section are projections derived from model size, storage footprint, and observed 9B- and frontiermodel behaviour. Our results, combined with public hardware specifications, allow us to define three practical deployment tiers: Table 13: Edge AI hardware tiers for SD AI assistance. Benchmark scores are best-variation CLD pass rates. GLM-5 9B scores are from Discussion iteration category (best local). “Viable” means ≥60% on the primary task category. Tier
Representative device
RAM
Best CLD%
Notes
†
7B–9B models only Qwen3, GLM-5 class Kimi K2.5, DeepSeek class
Entry MacBook Air M3 (16 GB) 16 GB ∼40–50% Mid Mac Mini M4 Pro (64 GB) 64 GB ∼60–70%† Full Mac Studio M3/M4 Ultra (192–512 GB) 192–512 GB 77% (measured) † Projected from 9B-class model performance; not directly benchmarked.
The entry tier (16 GB) can host 7B–9B quantised models. Our GLM-5 MLX-4bit result is instructive here: at approximately 6 GB storage and 9B parameters, it achieves 75% on the iteration category—matching the cloud causal-decoder engine on that specific task. A GLM5-class model is deployable on a MacBook Air, the most widely available personal computer in academic and enterprise settings. The full tier (192–512 GB Mac Studio Ultra) supports frontier-scale quantised models including Kimi K2.5 GGUF Q3 (∼250 GB) and DeepSeek V3.2 Q4 K M (∼200 GB). At this tier, measured CLD performance (77%) equals mid-tier cloud models. With task routing across models, the post hoc upper bound reaches 91%—matching the best single cloud model overall.
B.3
Task Routing Extended to Hardware Routing
Section 11 established that different models excel on different task categories. The hardware-tier analysis adds a second dimension: different tasks may also be addressable at different hardware tiers. • Iteration and simple model building steps are well-served by 9B-class models on entrytier hardware. GLM-5 achieves 75% on iteration at ∼119 s/query on the Mac Studio; projected latency on a Mac Mini M4 Pro would be comparable or faster due to higher memory bandwidth per dollar in the M4 generation. • CLD extraction and conformance require 100B+ quantised models for competitive performance (≥70%). Full-tier hardware is required. • Error fixing is currently impractical locally due to long-context requirements (80–146k tokens) that exhaust Metal GPU command buffers on mlx lm. This category remains cloud-dependent until mlx lm context handling improves or GGUF-backed local inference matures for this task type. A hardware-aware routing layer could therefore direct queries to the minimum-sufficient hardware tier, reducing energy consumption and cost while preserving accuracy.
23
B.4
Data Sovereignty and Regulated Industries
Cloud AI APIs present data-governance challenges for several categories of System Dynamics use: • Healthcare and public health models: patient flow, epidemic dynamics, and resource allocation models frequently involve sensitive or embargoed data. Cloud transmission may create compliance obligations under HIPAA, GDPR, or national health-data regulations. • Defence and national security models: force deployment, logistics, and adversarial dynamics models are self-evidently unsuitable for cloud APIs operated by foreign or commercial entities. • Commercially sensitive strategy models: merger integration, competitive dynamics, and market entry models represent trade secrets. Many organisations prohibit transmission of strategic planning data to third-party AI services. • Offline and disconnected environments: field research, expeditionary settings, and airgapped networks require inference that does not depend on internet connectivity. For many of these use cases, edge AI appliances may be the most practical or policy-compliant deployment path. Our results demonstrate that the performance penalty for edge deployment is now modest: 77% vs. 89% for the best single cloud model on CLD extraction, closing to parity on task routing.
B.5
The SD AI Appliance Concept
The results support a concrete product concept: a System Dynamics AI appliance—a dedicated edge device pre-configured with SD-specific models, prompts, and evaluation infrastructure, deployed at the institution or research group level. • Hardware: Mac Studio M3/M4 Ultra (192 GB minimum recommended). No GPU data-centre infrastructure required. • Software: llama.cpp backend (GGUF format) for large models; mlx lm for small models on simple tasks. SD-AI open-source inference stack. • Task routing: Kimi K2.5 for conformance/relationship extraction; GLM-5 for iteration coaching; cloud fallback for long-context error fixing (optional). • Performance: 77% CLD extraction; 75% Discussion iteration; comparable to mid-tier cloud APIs. • Data governance: all inference on-premises; no model data leaves the device. Future Apple Silicon generations with higher memory bandwidth or larger unified-memory configurations would likely narrow the gap further between local and cloud inference for this workload. Combined with continued model quality improvements in the open-source ecosystem (the Kimi K2.5, DeepSeek V3, and Qwen3 families all showed strong CLD performance), the trajectory favours local deployment for domain-specific SD AI work.
24
B.6
Limitations
The hardware-tier projections for entry and mid tiers are extrapolated from 9B-class model behaviour and have not been directly benchmarked. The GLM-5 iteration result was obtained on the Mac Studio; inference speed on lower-memory devices may differ. The SD AI appliance concept requires further engineering work to implement the task-routing layer and automate model selection.
References [1] Sterman, J.D. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. McGraw-Hill. [2] Wei, J., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022. [3] Frantar, E., et al. (2022). GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv:2210.17323. [4] OpenAI (2024). OpenAI openai-o1-system-card/
o1
System
Card.
https://openai.com/index/
[5] Apple MLX Team (2024). mlx-lm: LLM inference and fine-tuning with MLX. https://github. com/ml-explore/mlx-lm [6] Gerganov, G., et al. (2023). llama.cpp: Inference of LLaMA model in pure C/C++. https: //github.com/ggerganov/llama.cpp [7] Moonshot AI (2025). Kimi K2.5 Technical Report. https://huggingface.co/moonshotai/ Kimi-K2.5 [8] DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437. [9] Apple Inc. (2025). Mac Studio (M3 Ultra, 512 GB)—Technical Specifications. https://www. apple.com/mac-studio/specs/ [10] International Energy Agency (2024). Electricity 2024: Analysis and Forecast to 2026. IEA, Paris. https://www.iea.org/reports/electricity-2024 [11] Luccioni, A.S., Viguier, S., & Ligozat, A.L. (2023). Estimating the carbon footprint of BLOOM, a 176B parameter language model. Journal of Machine Learning Research, 24(253), 1–15. [12] Patterson, D., et al. (2022). The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7), 18–28.
25