HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads Justice Owusu Agyemang∗1,2,3 , Jerry John Kponyo†3 , Obed Kwasi Somuah‡2 , Elliot Amponsah§3 , Godfred Manu Addo Boakye¶3 , and Kwame Opuni-Boachie Obour Agyekum‖2
arXiv:2604.17111v1 [cs.DC] 18 Apr 2026
1
Sperix Labs VIA Cybersecurity Lab, KNUST 3 Quantum and Assistive Technologies Lab, KNUST 2
April 2026
Abstract
compute. An ablation study reveals that transparent retry—not admission control—is the single most critical primitive, but the primitives are most effective in combination. Real-world validation against Ollama confirms that HiveMind adds under 3 ms of proxy overhead per request. The system is open-source under the MIT license.
When multiple LLM coding agents share a ratelimited API endpoint, they exhibit resource contention patterns analogous to unscheduled OS processes competing for CPU, memory, and I/O. In a motivating incident, 3 of 11 parallel agents died from connection resets and HTTP 502 errors—a 27% failure rate—despite the API having sufficient aggregate capacity to serve all 11 sequentially. We present HiveMind, a transparent HTTP proxy that applies five OSinspired scheduling primitives—admission control, rate-limit tracking, AIMD backpressure with circuit breaking, token budget management, and priority queuing—to eliminate the failure modes caused by uncoordinated parallel execution. The proxy requires zero modifications to existing agent code and supports Anthropic, OpenAI, and local model APIs via autodetected provider profiles. Our evaluation across seven scenarios (5–50 concurrent agents) shows that uncoordinated agents fail at 72–100% rates under contention, while HiveMind reduces failures to 0–18% and eliminates 48–100% of wasted
1
Introduction
The emergence of tool-augmented large language models has shifted software-engineering assistants from suggestion engines to autonomous agents that read, write, and execute code on a developer’s behalf [1, 2, 3, 4]. When users spawn multiple such agents in parallel—a natural pattern for tasks like generating test suites, writing proof-of-concept exploits, or refactoring across modules—the agents compete for shared resources: API rate limits (requests and tokens per minute), network connections (concurrent connection limits per endpoint), context windows (fixed per model), and API-key quotas (billing and access limits). This resource contention leads to agent failures. The pattern is structurally identical to the contention that motivated operatingsystem schedulers: multiple processes competing for CPU, memory, and I/O without co-
∗
[email protected], [email protected] [email protected] ‡ [email protected] § [email protected] ¶ [email protected] ‖ [email protected] †
1
Table 1: Results of 11 uncoordinated concurrent agents (April 15, 2026). Outcome Completed successfully Died (ECONNRESET) Died (HTTP 502) Tokens wasted (dead agents)
Count
%
8 2 1
73 18 9
control via condition variables, provideraware rate-limit tracking, AIMD backpressure with circuit breaking, per-agent token budgets, and priority queuing with dependency DAGs—requiring zero modifications to existing agent code. 3. An evaluation across seven scenarios showing 72–100% failure reduction, and an ablation study revealing that transparent retry is the single most critical primitive. 4. An open-source implementation supporting Anthropic, OpenAI, Azure OpenAI, Google AI, and local models (Ollama, MLX) via auto-detected provider profiles. The remainder of this paper is organised as follows. Section 2 reviews the relevant background. Section 3 describes the proxy architecture and five scheduling primitives. Section 4 covers key implementation decisions. Section 5 reports evaluation results, ablation, and realworld validation. Section 6 surveys related work. Section 7 discusses tradeoffs and limitations, and Section 8 concludes.
∼135 K
ordination leads to thrashing, starvation, and deadlock [5, 6]. Yet current agent orchestration frameworks—LangChain [7], CrewAI [8], AutoGen [9], Semantic Kernel [10]—treat the LLM API as an unlimited resource, providing composition mechanisms (chains, crews, multiagent conversations) but not resource management. They are, in OS terms, running a multiprocess system without a scheduler. Motivating observation. On April 15, 2026, we spawned 11 concurrent Claude Code agents to generate proof-of-concept scripts for security findings. All 11 shared one Anthropic API key through a single network proxy. Three agents died: two from ECONNRESET and one from HTTP 502. Each dead agent had consumed approximately 45 000 tokens before failing—a total waste of ∼135 000 tokens and ∼15 minutes of wall time. The eight surviving agents completed successfully because they happened to stagger their requests enough to avoid the bottleneck. Key insight: if the 11 agents had been staggered by just 5 seconds each, all 11 would have succeeded. The problem is not capacity—it is coordination.
2
Background
2.1
LLM Coding Agents
A growing class of developer tools embed an LLM in an edit–test–commit loop. Claude Code [1], Cursor [3], GitHub Copilot [4], OpenAI Codex CLI [2], and Devin [11] each grant the model access to the local filesystem, a shell, and often a language server. The SWE-bench benchmark [12] and the SWE-agent framework [13] have further demonstrated that agents can resolve real GitHub issues end-to-end, making reliable API access a critical capability. Each agent is a long-running, stateful process that makes repeated API calls over a multi-turn conversation. A single agent session may consume 50 000–500 000 tokens across dozens of API calls, with each call dependent on the previous response. When an API call fails mid-session, the agent typically cannot recover: it has consumed tokens, modified files, and accumulated context that is lost on restart.
Contribution. We present HiveMind, a scheduling system that applies OS scheduling principles to concurrent LLM agent workloads. The contributions are: 1. A formal analogy mapping OS resourcemanagement concepts (admission control, congestion control, budgeting, priority scheduling) to the LLM agent domain (Table 2). 2. A transparent HTTP proxy that implements five scheduling primitives—admission 2
2.2
OS Scheduling Principles
HiveMind
The resource contention patterns exhibited by concurrent LLM agents are structurally identical to those solved by operating system schedulers [5, 14]: • Admission control. Limiting the number of concurrent processes to prevent thrashing. Dijkstra’s semaphore [15] is the classical mechanism. • Congestion control. TCP’s Additive Increase / Multiplicative Decrease (AIMD) algorithm [16, 17] adjusts sending rate based on observed congestion signals (packet loss, increased RTT). • Circuit breaking. The circuit breaker pattern [18] stops sending requests to a failing service, allowing it to recover before resuming load. • Resource budgeting. Per-process memory limits, CPU quotas, and the OOM killer prevent any single process from monopolising shared resources. • Priority scheduling. Shortest-job-first and priority queues [5] ensure that highvalue or short tasks are serviced before long or low-priority ones.
Admission Gate
Agent 1 Rate Limiter
Agent 2
AIMD + Circuit
.. N Agent .
Token Budget
Upstream API
Retry
Figure 1: Architecture of HiveMind. Agents connect to the local proxy; requests pass through five scheduling layers before reaching the upstream API. The proxy is transparent: agents require zero code changes.
and the upstream LLM API provider (Figure 1). Agents make normal API calls to http://localhost:8765/v1/messages; HiveMind applies all scheduling logic before forwarding to the upstream provider. This design has four advantages: (1) zero agent modification—works with any framework, SDK, or language; (2) provider agnostic—same proxy for Anthropic, OpenAI, Ollama, or any OpenAI-compatible endpoint; (3) observable— all traffic flows through one measurement point; 2.3 The OS–LLM Agent Analogy (4) composable—can chain with other proxies We formalise the mapping between OS concepts (e.g., Burp for security testing). and LLM agent orchestration in Table 2. This analogy is not merely illustrative—it is struc3.1 Admission Control turally precise. Each OS mechanism addresses a specific resource contention failure mode that The admission controller limits the number of has a direct counterpart in the LLM agent do- concurrent in-flight API requests. We model it main. as a gated counter protected by a condition variable. Let Cmax be the maximum concurrency and 2.4 Why Existing Frameworks Fail A the count of active requests. A request is adTable 3 compares the scheduling capabilities of mitted when A < Cmax ; otherwise it waits on a existing agent orchestration frameworks. None condition variable: provides the full set of primitives needed to man( age concurrent API access. true, A < Cmax admit(r) = (1) wait, otherwise
3
Architecture
On release, A is decremented and one waitHiveMind is implemented as a transparent ing request is notified. The condition-variable HTTP reverse proxy that sits between agents design (rather than a semaphore) supports safe 3
Table 2: Structural mapping between OS resource management and LLM agent scheduling. Each row identifies an OS mechanism, its HiveMind counterpart, and the failure mode it addresses. OS Concept
HiveMind Equivalent
Resource
Failure Mode
Process
LLM agent
–
CPU time slice Memory I/O bandwidth Process scheduler Virtual memory OOM killer TCP congestion ctrl. Circuit breaker Fork bomb protection Nice levels
API request slot Context window Network connections Admission gate + queue Checkpointing Token budget enforcer AIMD backpressure Backpressure circuit Max agent limit Task priority
RPM/TPM Fixed per model Conn. limits Concurrency slots Disk Token pool Latency signal Error rate Key quota Sched. order
Stateful, long-running, resourceconsuming Starvation under contention Cannot be shared or paged ECONNRESET, HTTP 502 Thrashing, stampede Context loss on eviction Runaway agent monopolises API Throughput collapse Cascading failure Unbounded spawn Low-value work blocks high-value
Table 3: Scheduling capabilities of existing minute (TPM) sliding-window counter is preframeworks. ✓ = full, ∼ = partial. seeded from the detected provider profile (Section 4.2). This provides throttling before the first System Adm. Rate BP Bud. Pri. API response arrives and for providers that send Claude Code – – – – – no rate-limit headers (e.g., Ollama). Each call LangChain [7] – ∼ – – – to wait_if_throttled() records a timestamp; CrewAI [8] – – – – ∼ when the window count reaches the RPM limit, AutoGen [9] – – – – – subsequent requests block until the oldest entry Sem. Kernel [10] – ∼ – – – expires. HiveMind ✓ ✓ ✓ ✓ ✓ dynamic resizing of Cmax by the backpressure 3.3 AIMD Backpressure with Circuit Breaking controller: when Cmax increases, all waiters are notified; when it decreases, the new limit takes The backpressure controller adapts TCP congeseffect naturally as active requests complete. tion control principles [16, 17] for LLM API concurrency. Let ct denote the concurrency level 3.2 Rate-Limit Tracking at time t, ℓ̄ the average latency over a sliding window of W samples, and Ltarget the latency The rate limiter operates at two levels: target: Header-based (reactive). After each API response, the proxy parses provider-specific rate-limit headers (anthropic-ratelimit-requests-remaining, x-ratelimit-remaining-requests, retry-after) and proactively pauses all agents when remaining capacity falls below a configurable threshold (default: 10% of the limit with ≤ 2 requests remaining).
ct+1 =
min C , c + α , max t
max Cmin , ct · β ,
max Cmin , ct · β ,
if ℓ̄ ≤ Ltarget if ℓ̄ > Ltarget
on error (429, 502, reset) (2) where α is the additive increase step (default: 0.5) and β is the multiplicative decrease factor (default: 0.5). Concurrency adjustments are pushed directly to the admission controller via Sliding-window counters (proactive). A a held reference, eliminating the lag of a polling requests-per-minute (RPM) and tokens-per- loop. 4
Algorithm breaker.
1:
AIMD
with
circuit
Closed
e/n ≥ τ
t > topen +Tcool probe fails
probe succeeds
Input: latency sample ℓ or error event Result: Updated concurrency ct , circuit state
Open
Half-Open
if error event then 2 ct ← max(Cmin , ct · β); 3 e ← e + 1; n ← n + 1; 4 push ct to admission controller; 5 if n ≥ N and e/n ≥ τ then 6 circuit ← open; 7 topen ← now; 8 end 9 else if latency sample ℓ then 10 append ℓ to window; 11 n ← n + 1; 12 if update interval elapsed then 13 ℓ̄ ← mean(window); 14 if ℓ̄ ≤ Ltarget then 15 ct ← min(Cmax , ct + α); 16 else 17 ct ← max(Cmin , ct · β); 18 end 19 push ct to admission controller; 20 end 21 else if success and circuit = half-open then 22 circuit ← closed; 1
Figure 2: Circuit breaker state machine. The circuit opens on sustained errors, transitions to half-open after a cooldown, and closes on a successful probe request. tracted from API response bodies. At 85% utilisation, the agent receives a warning. At 100%, the agent is checkpointed (state saved to disk) and stopped, analogous to the OS OOM killer.
3.5
Priority Queue with Dependency DAG
Tasks are ordered by: (1) priority level (Critical > High > Normal > Low), (2) estimated token cost (shortest-job-first within the same priority), (3) creation time (FIFO tiebreaker). Dependencies between tasks are tracked as a directed acyclic graph with cycle deCircuit breaker. A circuit breaker [18] over- tection; a task is not eligible for scheduling until lays the AIMD controller. The breaker monitors all its predecessors have completed. error rate over a sliding window of N requests (default: N = 20). When the error rate exceeds 3.6 Transparent Retry a threshold τ (default: τ = 0.50), the circuit proxy intercepts retryable opens, causing the proxy to fast-fail all incom- The errors—HTTP 429, 502, 503, 529, ing requests with HTTP 503 and a Retry-After header. After a cooldown period Tcool (default: ECONNRESET, RemoteProtocolError (“server 10 s), the circuit transitions to half-open: a sin- disconnected”)—and retries transparently with gle probe request is allowed through. If the probe exponential backoff plus jitter. The retry delay succeeds, the circuit closes and normal operation for attempt k is: resumes; if it fails, the circuit re-opens. dk = min dmax , dbase · 2k + U (0, dbase )
state =
3.4
open,
half-open, closed,
(4)
if ne ≥ τ, n ≥ N
where dbase = 1 s, dmax = 30 s, and U (0, dbase ) if open and t > topen + Tcool is uniform jitter. If a Retry-After header is if half-open probe succeeds present, it overrides the computed delay. From (3) the agent’s perspective, the request simply takes longer—the error is never surfaced.
Token Budget Management 3.7
Streaming Support
Each agent is assigned a token ceiling from a global pool. The budget manager tracks cumu- HiveMind passes through Server-Sent Events lative input and output tokens per agent, ex- (SSE) streams without buffering, forwarding 5
chunks as they arrive from the upstream API. Table 4: Default provider profile parameters. Token counts are extracted from message_delta Values are overridden by explicit user configuand message_start events in the SSE stream. ration. The admission slot is held for the duration of RPM TPM Max C Ltarget the stream and released on completion or error. Provider
4
Anthropic OpenAI Azure Google AI Ollama Generic
Implementation
50 60 60 60 1000 60
80K 150K 120K 100K 10M 100K
5 10 10 8 2 5
3 000 ms 2 000 ms 3 000 ms 2 000 ms 10 000 ms 2 000 ms
HiveMind is implemented in Python 3.11 as an asyncio-based HTTP proxy using Uvicorn and Starlette, with httpx for upstream connections. The system registers as an MCP server exposing eight tools (hm.submit, hm.batch, hm.status, hm.priority, hm.budget, thropic, OpenAI, Azure OpenAI, Google AI, Olhm.metrics, hm.config, hm.setup) and si- lama, and a generic fallback), each specifying: multaneously serves as a standalone proxy via • Default RPM and TPM limits hivemind proxy. • Default max concurrent connections • Rate-limit header field names 4.1 Condition Variable vs. Semaphore • Retryable status codes • AIMD tuning parameters (α, β, Ltarget ) The admission controller initially used • Authentication header name asyncio.Semaphore. Dynamic resizing reProvider detection is automatic via regex quired mutating the semaphore’s internal matching on the upstream URL (e.g., _value attribute—undefined behaviour in api.anthropic.com → Anthropic). The CPython that silently broke under concurrent detected profile pre-seeds the rate limiter’s load when the backpressure controller reduced sliding-window counters and configures AIMD concurrency while requests were in flight. parameters, so the system is correctly tuned We replaced the semaphore with an explicit before the first API response arrives. counter A protected by an asyncio.Condition Table 4 shows the default parameters for each wrapping an asyncio.Lock. Acquiring a slot provider. waits on the condition until A < Cmax ; releasing decrements A and calls notify(1). When Cmax increases, notify_all() wakes all waiters so they can re-check the predicate. When Cmax 4.3 Direct Backpressure–Admission decreases, no action is needed: the new limit Wiring takes effect naturally as active requests complete and new ones find the predicate false. The backpressure controller holds a direct refThis design makes dynamic resizing a safe erence to the admission controller, set durO(1) operation rather than an undefined muta- ing proxy initialisation via set_admission(). tion of internal state. When the AIMD algorithm adjusts ct , the new value is pushed immediately to the admission controller via set_max_concurrency(), which 4.2 Provider Detection and Profiles atomically updates Cmax and notifies waiters Each LLM API provider has different rate-limit if concurrency increased. This eliminates the header formats, default concurrency limits, retry polling loop used in earlier designs, where a backsemantics, and endpoint patterns. HiveMind ground scheduler task periodically synced the maintains a registry of six provider profiles (An- two controllers. 6
Token Counting Streams
from
SSE
100
Failure Rate (%)
4.4
80
100% Direct HiveMind
100%
100%
100%
100%
73%
60 For streaming responses, token counts are embedded in the SSE event stream. The proxy 40 parses message_start events (which contain 18% 20 10% 10% 10% input token counts) and final message_delta 0 events (which contain output token counts) withro-5 icro-10 icro-20 icro-50 lay-11 stress y-spike mic m m m nc rep out buffering the stream. For non-streaming relate sponses, token counts are extracted directly from the JSON response body. When neither source Figure 3: Failure rates by scenario. Direct mode provides counts, a heuristic estimate of 1 token (red) fails catastrophically at 10+ agents; Hiveper 4 characters is used. Mind (green) reduces failures to 0–18%.
5
Evaluation
At 5 agents, both modes succeed—there is no contention. At 10+ agents, uncoordinated execution fails catastrophically (72–100% failure rate), while HiveMind reduces failures to 0– 18%. The residual failures in replay-11 and stress scenarios arise from error injection rates that exceed the retry budget.
We evaluate HiveMind along three axes: (1) failure-rate reduction across seven scenarios, (2) an ablation study isolating the contribution of each primitive, and (3) real-world validation against local model APIs.
5.1
Wall-time trade-off. HiveMind takes longer in absolute wall time because it serialises requests through the rate-limit window rather than letting agents stampede and die. Direct mode “finishes fast” only because most agents fail immediately. When measured against completed work, HiveMind’s throughput is strictly higher. Figure 3 visualises the failure rate reduction across all scenarios. Figure 4 shows the scaling behaviour: direct mode completes zero agents beyond 5 concurrent, while HiveMind scales linearly.
Methodology
We evaluate using a mock API server that simulates realistic LLM API behaviour in both Anthropic and OpenAI response formats. The mock supports configurable rate limits (requests per minute), error injection (random HTTP 502 and connection resets at specified rates), provider-specific rate-limit headers (anthropic-ratelimit-* and x-ratelimit-*), latency (base plus jitter plus configurable spikes), concurrency limits, and SSE streaming in both formats. Mock agents make N sequential API calls simulating multi-turn coding sessions. Each agent either completes all turns or “dies” on the first unrecoverable error, matching observed real-world behaviour where agents cannot recover mid-session.
5.3
Ablation Study
To measure the individual contribution of each scheduling primitive, we run the replay-11 scenario with various primitives disabled (Table 6).
Surprising finding. Our initial hypothesis was that admission control alone would suffice. 5.2 Scenarios and Results The ablation disproves this: admission-only still Table 5 describes the seven evaluation scenarios, produces 81.8% failure because it limits conand Table 5 compares direct (uncoordinated) ex- currency but does not handle rate-limit errors ecution against HiveMind-managed execution. or connection resets. Transparent retry is 7
Table 5: Evaluation scenarios and results. Error rates are p502 + preset . ∆f is the change in failure rate (percentage points); ∆w is the reduction in tokens consumed by dead agents.
50 40
Scenario
Agents
RPM
micro-5 micro-10 micro-20 micro-50 replay-11 stress lat.-spike
5 10 20 50 11 20 10
50 50 50 50 60 20 60
Direct HiveMind
30 20 10
Failure Rate
Rate
Direct
HiveMind
∆f
∆w
0% 0% 0% 0% 8%+5% 10%+5% 0%
0% 100% 100% 100% 73% 100% 100%
0% 10% 10% 0% 18% 10% 0%
0 −90 −90 −100 −55 −90 −100
– −100% −94% −100% −48% −100% −100%
Why not per-agent retry? Per-agent retry (e.g., via tenacity) is the natural first response, but it lacks centralised coordination. When 10 agents each independently retry after a 429 error, the retries arrive simultaneously—the “thundering herd” [19]—re-triggering the rate limit. HiveMind’s centralised retry serialises retries through the admission gate, preventing amplification.
Direct HiveMind
200
Tasks/min
Agents Completed
Error
150 100 50 0
0
20
40
20
Concurrent Agents
40
Concurrent Agents
Figure 4: Scaling behaviour. Left: agents that complete successfully. Right: effective throughput (tasks/min). Direct mode throughput drops 5.4 Real-World Validation to zero beyond 5 agents. We validated HiveMind against two local model servers (Table 7): Ollama [20] serving Qwen 3.5Table 6: Ablation study on the replay-11 sce- 4B (GGUF, Q4_K_M) and an MLX inference nario. Each row disables one primitive; “Full” server serving Qwen 3.5-4B-4bit. Each test enables all. used 10 agents making 3 turns each, with the –compare flag running direct mode first, then Configuration Alive Dead Fail% Finding HiveMind mode. Full HiveMind No admission No rate limit No backpressure No retry
11 11 11 10 4
0 0 0 1 7
0.0 0.0 0.0 9.1 63.6
Adm. only
2
9
81.8
Baseline Compensated Table 7: Real-world validation against local Compensated model servers (10 agents × 3 turns). Marginal Most Server Mode Alive Fail% Time critical Insufficient
the single most impactful primitive, reducing failures from 63.6% (without it) to near-zero (with it). However, the primitives are most effective in combination: retry handles transient errors, admission prevents connection exhaustion, rate limiting prevents errors from occurring in the first place, and backpressure provides finegrained stability.
Ollama Ollama
Direct HiveMind
10/10 10/10
0% 0%
30.5 s 28.5 s
MLX MLX
Direct HiveMind
10/10 10/10
0% 0%
3.9 s 3.6 s
Local models handle concurrency gracefully (they queue internally), so these tests do not trigger the stampede scenario. They do, however, confirm that HiveMind adds negligible overhead: <3 ms per proxied request, and in the 8
Wasted Tokens (K)
Full HiveMind 81.8%
Admission Only No Admission No Ratelimit 9.1%
No Backpressure
0
20
40
60
Failure Rate (%)
7.5 5.0 2.5 0.0
ro-5 icro-10 icro-20 icro-50 lay-11 m m m rep
mic
63.6%
No Retry
Direct HiveMind
10.0
80
100
ss ke stre cy-spi n e t la
Figure 6: Wasted tokens by scenario (thousands). Direct mode (red) wastes 1–12K tokens Figure 5: Ablation study results. Removper scenario; HiveMind (green) reduces waste ing retry causes the largest degradation (63.6% to near-zero. failure); admission-only is insufficient (81.8%). Other primitives are compensated by the remaining ones. by 96–97%. Ollama case, HiveMind was actually 7% faster than direct access because its admission gate (Cmax = 2) matched Ollama’s natural concurrency and reduced internal queuing contention. An earlier test run produced one MLX failure (10%) caused by a RemoteProtocolError (“server disconnected”). Adding this pattern to the retryable-error list (Section 3.6) resolved the issue; subsequent runs achieve 10/10 across both servers.
5.5
6
Related Work
Agent orchestration frameworks. LangChain [7], CrewAI [8], AutoGen [9], and Semantic Kernel [10] focus on agent composition— chains, crews, multi-agent conversations—but not on resource management. They assume the API is always available and delegate retry to per-request libraries. HiveMind is complementary: it sits below any of these frameworks, managing the shared API resource that they all depend on.
Cost of Wasted Compute
Token waste translates directly to monetary cost. Table 8 shows the cost of wasted tokens (tokens consumed by agents that ultimately failed) across our evaluation suite, extrapolated to a daily workload of 10 runs.
API rate-limiting libraries. Libraries like tenacity and backoff provide retry logic at the individual request level but lack system-wide coordination. Each agent retries independently, potentially amplifying load during rate-limit Table 8: Daily cost of wasted tokens at current windows—the “thundering herd” problem [19]. Anthropic pricing (per million input tokens), as- HiveMind centralises retry decisions across all agents sharing an API key. suming 10 evaluation runs per day. Model
Direct
HiveMind
Savings
Haiku ($0.80/M) Sonnet ($3/M) Opus ($15/M)
$0.35 $1.31 $6.55
$0.01 $0.05 $0.24
97% 96% 96%
TCP congestion control. Our AIMD backpressure controller directly adapts the Additive Increase / Multiplicative Decrease algorithm from TCP Tahoe/Reno [16, 17, 21]. The key insight is that API latency serves the same role as network round-trip time: it signals congestion before requests are dropped. Unlike TCP, we do not implement slow start (APIs have known baseline concurrency) or fast recovery (the concurrency space is too small for multiplicative
At Opus-tier pricing, uncoordinated agents waste $6.55/day from our seven-scenario suite alone. In production workloads with 20–50 agents running continuously, waste scales to hundreds of dollars per day. HiveMind reduces this 9
probing).
Circuit breaker pattern. Nygard [18] introduced the circuit breaker as a stability pattern for distributed systems. Our circuit breaker adapts this for the LLM API context: the errorrate threshold is tuned for API-level failures (429, 502), the half-open probe uses a real API request rather than a health check, and the state is co-located with the AIMD controller so that circuit events also trigger concurrency reduction.
Staged event-driven architecture. Welsh et al.’s SEDA [22] proposed decomposing Internet services into stages connected by queues, with each stage applying admission control independently. HiveMind’s pipeline (admission → rate limit → backpressure → forward → retry) follows the same staged pattern, though our stages are co-located in a single process rather than distributed across threads.
OS scheduling theory. The correspondence between LLM agent scheduling and process scheduling has not been previously formalised in the literature. Classical scheduling theory— shortest-job-first, priority scheduling, multilevel feedback queues [5, 14]—applies directly when the “CPU” is an API request slot and “processes” are stateful agents with unpredictable execution times. The concurrency primitives (semaphores [15], condition variables, monitors [23]) translate directly to the async-I/O domain.
7
Discussion
7.1
Design Tradeoffs
Proxy vs. SDK integration. We chose a transparent HTTP proxy over SDK-level integration (e.g., a custom httpx transport or a LangChain callback). This sacrifices per-request metadata (the proxy cannot read agent-internal state) but gains universality: the same proxy works for Python, TypeScript, Go, and shellbased agents without any code changes. The MCP server mode provides richer integration for agents that support tool use. Condition variable vs. semaphore. The condition-variable admission gate adds ∼50 µs of overhead per acquire/release compared to a raw semaphore. This is negligible relative to API latency (typically 500–5000 ms) and eliminates undefined behaviour during dynamic resizing. AIMD tuning. The default AIMD parameters (α = 0.5, β = 0.5, Ltarget = 2000 ms) are conservative. Provider profiles override these: Ollama uses β = 0.7 (gentler decrease, since local inference doesn’t benefit from aggressive backoff) and Ltarget = 10 000 ms (local models are inherently slower). These defaults can be further tuned via the hm.config tool at runtime. Circuit breaker placement. The circuit breaker is co-located with the AIMD controller rather than implemented as a separate middleware layer. This ensures that circuit-open events also reduce the AIMD concurrency level, preventing a burst of requests when the circuit closes.
SWE-bench and agent reliability. SWEbench [12] and SWE-agent [13] evaluate agent 7.2 Limitations success rates on real GitHub issues but do not • Single-machine scope. The current imisolate API-access failures as a distinct cause of plementation runs on one machine. Distask failure. Our work addresses a failure mode tributed scheduling across multiple mathat is orthogonal to agent capability: an agent chines sharing an API key is architecturally may be perfectly capable of solving a task but supported via Redis-backed state but not still fail because its API call was dropped by the yet evaluated at scale. provider. 10
• Token estimation. When provider tokenizers are unavailable, token counting uses a heuristic (4 chars/token). This underestimates for languages with long tokens (e.g., CJK) and overestimates for code with short identifiers. • Mock evaluation. Our primary evaluation uses a mock API server supporting both Anthropic and OpenAI response formats. While it simulates realistic behaviour (rate limits, errors, latency, streaming), the stampede failure mode requires a cloud API with hard rate limits to trigger reliably— local models queue gracefully. • Dynamic priority. Priority is set at submission time. Automatic priority adjustment based on observed progress (e.g., promoting agents near completion) is future work. • No cross-agent coordination. HiveMind manages API access but does not coordinate agents’ filesystem operations or tool calls. Two agents writing to the same file remain a user-level concern.
with dependency DAGs—in combination eliminate the failure modes that currently plague parallel agent execution. The transparent proxy architecture requires zero changes to existing agents, making HiveMind a drop-in improvement for any multi-agent workflow. Our evaluation across seven scenarios shows that uncoordinated agents fail at 72–100% rates under contention, while HiveMind reduces failures to 0–18%. An ablation study yields a key insight for the field: in the current API landscape, transparent centralised retry is more important than admission control for agent survival, but both are most effective in combination. This suggests that LLM agent orchestration systems should prioritise retry coordination over simple concurrency limiting. Real-world validation against local model servers confirms that HiveMind adds under 3 ms of proxy overhead per request. Autodetected provider profiles ensure that the system is correctly tuned for each API provider out of the box. A 174-test suite validates correctness across all scheduling primitives, and the system is 7.3 Future Directions open-source under the MIT license at https: Production-scale validation against cloud APIs //github.com/jayluxferro/hivemind. (Anthropic, OpenAI) with 20–50 concurrent agents would strengthen the empirical claims. Integrating provider-specific tokenizers would References improve budget accuracy. A multilevel feedback queue (promoting short-running agents, demot- [1] Anthropic, “Claude code: An agentic coding tool.” https://docs.anthropic.com/ ing long-running ones) could improve average en/docs/claude-code, 2025. Accessed: completion time. Finally, combining HiveMind 2026-04-15. with task-level resilience systems (checkpointing, decomposition) would provide end-to-end fault tolerance from the API layer to the agent layer. [2] OpenAI, “Codex CLI: Open-source coding agent.” https://github.com/openai/ codex, 2025. Accessed: 2026-04-15.
8
Conclusion
We have presented HiveMind, a scheduling system that applies OS scheduling principles to concurrent LLM agent workloads. The five scheduling primitives—admission control via condition variables, provider-aware rate-limit tracking, AIMD backpressure with circuit breaking, per-agent token budgets, and priority queuing 11
[3] Anysphere Inc., “Cursor: The AI code editor.” https://cursor.com, 2024. Accessed: 2026-04-15. [4] GitHub, “GitHub copilot.” https: //github.com/features/copilot, 2024. Accessed: 2026-04-15.
[5] A. Silberschatz, P. B. Galvin, and [14] A. S. Tanenbaum and H. Bos, Modern Operating Systems. Pearson, 4th ed., 2015. G. Gagne, Operating System Concepts. Wiley, 10th ed., 2018. [15] E. W. Dijkstra, “Cooperating sequential processes,” Technical Report EWD-123, [6] E. G. Coffman, M. J. Elphick, and Technological University Eindhoven, 1965. A. Shoshani, “System deadlocks,” ACM Computing Surveys, vol. 3, no. 2, pp. 67– [16] V. Jacobson, “Congestion avoidance and 78, 1971. control,” in Proceedings of ACM SIGCOMM, pp. 314–329, 1988. [7] H. Chase, “LangChain.” https://github. com/langchain-ai/langchain, 2022. Ac[17] D.-M. Chiu and R. Jain, “Analysis of the cessed: 2026-04-15. increase and decrease algorithms for congestion avoidance in computer networks,” [8] J. Moura, “CrewAI.” https://github. Computer Networks and ISDN Systems, com/joaomdmoura/crewAI, 2024. Accessed: vol. 17, no. 1, pp. 1–14, 1989. 2026-04-15. [9] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, [18] M. T. Nygard, Release It! Design and Deploy Production-Ready Software. Pragmatic E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, Bookshelf, 2nd ed., 2018. A. H. Awadallah, R. W. White, D. Burger, and C. Wang, “AutoGen: Enabling next[19] J. Dean and L. A. Barroso, “The tail at gen LLM applications via multi-agent conscale,” in Communications of the ACM, versation,” 2023. vol. 56, pp. 74–80, 2013. [10] Microsoft, “Semantic kernel.” https: [20] Ollama, “Ollama: Run large language mod//github.com/microsoft/semanticels locally.” https://ollama.com, 2024. Ackernel, 2023. Accessed: 2026-04-15. cessed: 2026-04-15. [11] Cognition Labs, “Devin: The first AI soft[21] M. Allman, V. Paxson, and E. Blanware engineer.” https://devin.ai, 2024. ton, “TCP congestion control,” RFC 5681, Accessed: 2026-04-15. IETF, 2009. [12] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, [22] M. Welsh, D. Culler, and E. Brewer, K. Pei, O. Press, and K. Narasimhan, “SEDA: An architecture for well“SWE-bench: Can language models resolve conditioned, scalable internet services,” in real-world GitHub issues?,” 2024. Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP), [13] J. Yang, C. E. Jimenez, A. Wettig, K. Liber, pp. 230–243, 2001. S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces [23] M. Herlihy and N. Shavit, The Art of Mulenable automated software engineering,” tiprocessor Programming. Morgan Kauf2024. mann, revised 1st ed., 2012.
12