arXiv:2605.25701v1 [cs.DC] 25 May 2026
Neural Router: Semantic Content Matching for Agentic AI LAURI LOVÉN, Future Computing Group, University of Oulu, Finland ABHISHEK KUMAR, University of Jyväskylä, Finland ALEXANDER ENGELHARDT, Department of Computer Science, University of Helsinki, Finland ALAA SALEH, Department of Computer Science, University of Helsinki, Finland ROBERTO MORABITO, Department of Communication Systems, EURECOM, France XIAOLI LIU, Department of Computer Science, University of Helsinki, Finland NASER HOSSEIN MOTLAGH, Department of Computer Science, University of Helsinki, Finland SASU TARKOMA, Future Computing Group, University of Oulu, Finland and Department of Computer Science, University of Helsinki, Finland Large language models (LLMs) can serve as the semantic-matching engine of a content-based publish/subscribe broker for agentic AI across the edge–cloud computing continuum, bridging the vocabulary and modality gaps that defeat keyword and embedding filters. Framed as offline multi-label retrieval over three public datasets spanning social-media, legal, and smart-home sensor domains (six LLMs, seven baselines), our central contribution is a two-crossover cost–accuracy characterisation: an analytical context-window crossover below which a CoverAndMerge compression pipeline reduces LLM invocations, and an empirical discriminationcapacity crossover above which matching accuracy collapses independently of context budget, by a modeldependent factor of parameter count and training generation. Two findings carry practical weight: above the discrimination crossover, compression cannot recover accuracy and only frontier-scale models clear large subscription sets; and there backend choice dominates configuration choice, so model selection, not pipeline tuning, is the primary operator lever. We accompany this with three composable algorithms and a per-cluster Quality-of-Experience framework for autonomic LLM-tier selection. CCS Concepts: • Computer systems organization → Self-organizing autonomic computing; Cloud computing; • Theory of computation → Distributed computing models; • Computing methodologies → Natural language processing; • Networks → Middleware for databases. Additional Key Words and Phrases: Computing continuum, autonomic systems, large language models, content-based publish/subscribe, semantic matching, multi-label retrieval, agentic AI, Quality-of-Experience, cost-accuracy trade-off
1
Introduction
As autonomous AI agents increasingly operate across the computing continuum, from cloud services through network-edge infrastructure to IoT devices, they must discover and consume information produced by other agents and services whose data formats, vocabularies, and modalities they do not control [Saleh et al. 2025a; Tarkoma et al. 2023]; [Lovén et al. 2026]. A monitoring agent may need regulatory alerts written in legal jargon; a smart-environment agent must interpret raw sensor activations as human activities; a social-analytics agent consumes user-generated text in constantly evolving slang. Content-based publish/subscribe (pub/sub) systems [Carzaniga and Wolf 2003; Eugster et al. 2003] provide a natural communication substrate for such loosely coupled, data-driven interactions, but traditional designs rely on keyword matching or attribute-based filters [Carzaniga et al. 2001; Tarkoma 2012] that are increasingly limited as the semantic gap between publishers and Authors’ Contact Information: Lauri Lovén, [email protected], Future Computing Group, University of Oulu, Oulu, Finland; Abhishek Kumar, University of Jyväskylä, Jyväskylä, Finland; Alexander Engelhardt, Department of Computer Science, University of Helsinki, Helsinki, Finland; Alaa Saleh, Department of Computer Science, University of Helsinki, Helsinki, Finland; Roberto Morabito, Department of Communication Systems, EURECOM, Biot, France; Xiaoli Liu, Department of Computer Science, University of Helsinki, Helsinki, Finland; Naser Hossein Motlagh, Department of Computer Science, University of Helsinki, Helsinki, Finland; Sasu Tarkoma, Future Computing Group, University of Oulu, Oulu, Finland and Department of Computer Science, University of Helsinki, Helsinki, Finland.
0:2
Lovén et al.
AGENT Brand monitoring
AGENT Sensor
NEURAL ROUTER
AGENT Legal library
AGENT Social
!
AGENT Compl.
AGENT Activity rec.
Fig. 1. Conceptual overview of the Neural Router. Two pure-publisher agents (Sensor, Legal library) and four prosumer agents (Social, Activity recognition, Compliance, Brand monitoring) communicate through a single broker that matches published content to each agent’s natural-language interest via LLM reasoning; prosumers both publish and subscribe (content/interest in, matched content out). The highlighted link (bold red) marks the widest vocabulary/modality gap (raw sensor activations vs. a plain-language interest) where keyword and embedding matching fail by construction. Illustrative, not a deployment; the content types correspond to the evaluated domains (Section 4).
subscribers grows; embedding-based semantic-matching pipelines (e.g., Sentence-BERT cosine over the same description vocabulary) provide a natural intermediate, but stop short of the abductive reasoning required when modalities or vocabularies do not align. This paper studies the matching engine for such a broker as an offline multi-label retrieval problem; deployment-side concerns (online operation, fault tolerance, multi-tenant load) are scoped explicitly in Section 6.3. Recent advances in Large Language Models (LLMs), built on deep neural architectures such as Transformers [Devlin et al. 2019; Vaswani et al. 2017], have demonstrated remarkable semantic understanding of text. A growing body of work uses LLMs for routing, but the term is overloaded: in the LLM literature, “routing” typically means selecting which model should handle a given query [Hu et al. 2024; Ong et al. 2025; Wang et al. 2025], or classifying user intent to trigger a predefined action [Briggs and Aurelio AI 2024]. These systems route queries to models. In contrast, content-based pub/sub requires matching published content against subscriber interests, a fundamentally different problem in which the LLM must evaluate whether a piece of content satisfies a natural-language interest description. To date, no system places an LLM inside the pub/sub matching loop. Surveys on message brokers for generative AI [Saleh et al. 2025a] and pub/sub for edge intelligence [Saleh et al. 2024] identify the need to adapt brokers for AI workloads but do not propose LLM-based matching. Recent poster work explores LLM-generated notifications [Isahagian et al. 2023], and the AI Interconnect concept envisions LLMs as orchestrators within pub/sub [Tarkoma et al. 2023], but neither replaces the matching function itself. In this paper, we propose the Neural Router (Fig. 1), a content-based pub/sub matching engine that makes LLMs a first-class component of the content matching loop. The design explicitly trades per-invocation latency for semantic accuracy: embedding-based clustering, subscription compression, batching, and parallel LLM invocation collectively keep end-to-end matching latency within the bounds acceptable for analytics, notification, and contentdissemination workloads. Our contributions are:
Neural Router: Semantic Content Matching for Agentic AI
0:3
(1) A cost-accuracy trade-off characterization for LLM-based content matching with two crossover points: an analytically derived context-window crossover predicting when subscription compression becomes cost-effective, and an empirically identified discrimination-capacity crossover above which LLM matching accuracy declines, with the magnitude of decline determined by the model’s effective capability rather than by parameter count alone (Section 3). (2) A semantic matching architecture with three composable algorithms (OptimizeSubscriptions, MatchEvents, CoverAndMerge) and a QoE-based backend assignment mechanism for heterogeneous LLM backends (Section 3). (3) An empirical validation across three content domains (CardiffNLP tweets, EUR-Lex legal documents, CASAS smart-home sensor logs), evaluating six LLM backends (open-weight Qwen 2.5 at 1.5B/7B/32B and Mistral 7B; closed-API Claude Haiku and Sonnet) and seven ablation configurations against seven baselines, including zero-shot NLI with DistilBARTMNLI. The results confirm the cost model’s polarity: raw matching (A0) dominates above the context-window crossover, while below it the compression pipeline’s empirical recall is conditional on the merged set’s discrimination structure (Section 5.4). The four-model open-weight sweep shows the discrimination-capacity decline at |S|=201 is monotonic in parameter count within the Qwen family and is lifted, at fixed scale, by newer training generations (Mistral 7B > Qwen 2.5 7B at matched scale; broader cross-family validation deferred) (Sections 4 and 5). (4) A QoE-based heterogeneous backend assignment framework, with an empirical analysis identifying the calibration-sample-to-discrimination-gap ratio as the binding parameter. The framework routes subscription clusters to backends by per-cluster min-max-normalised weighted scoring of accuracy, cost, and latency, with three operator presets (accuracy_first, balanced, cost_first) yielding three distinct assignments. On CardiffNLP at a Qwen-2.5 (7B, 32B) gradient in the offline-batch single-tenant regime, round-robin is a strong calibration-free baseline — closing 83% of the F1 gap at 0.7× the larger-tier latency — and greedy QoE separates from it only at cost_first; a calibration-fraction sweep ({0.05, . . . , 1.00}) never crosses 𝛼=0.05 (Friedman 𝑝=0.290), so the ratio holds only in its weak, cost-side form on this gradient. Value under load skew, throughput heterogeneity, backend failure, and SLA constraints is left to deployment studies (Sections 5 and 6.3). The Neural Router specifies a single-node content matching engine. The question of how multiple matching engine instances can be federated across a distributed computing continuum (e.g., across 6G network slices) is an important extension that we discuss as future work in Section 6. The remainder of this article is organised as follows. Section 2 surveys related work and positions our contribution. Section 3 presents the Neural Router’s design, including the problem statement, architecture, algorithms, and cost model. Section 4 describes the experimental setup. Section 5 presents the results. Section 6 discusses limitations, implications, and future directions. Section 7 concludes. 2
Related Work
We review five bodies of work that surround the Neural Router: traditional content-based pub/sub (Section 2.1), distributed stream processing platforms (Section 2.2), LLM routing and semantic routing in the machine-learning community (Section 2.3), recent efforts that bring LLMs closer to publish/subscribe (Section 2.4), and autonomic / adaptive systems for the computing continuum (Section 2.5). Section 2.6 positions our contribution.
0:4
2.1
Lovén et al.
Content-based publish/subscribe
Content-based pub/sub matches messages to subscribers by evaluating predicates over message content, rather than relying on pre-assigned topic labels [Carzaniga et al. 2001; Eugster et al. 2003; Jacobsen 2009]. The central challenge is efficient matching: as the number of subscriptions and the dimensionality of the content grow, the broker must evaluate an increasing number of filter predicates per event. Traditional systems express subscriptions as conjunctions of attribute-value constraints [Carzaniga et al. 2001; Tarkoma 2012]. The algorithmic lineage of efficient subscription matching includes the Yan-Garcia-Molina counting algorithm, SIENA’s covering and merging relations [Carzaniga and Wolf 2003], BE-Tree, JEDI, Hermes, and PADRES [Li and Jacobsen 2005]; CoverAndMerge (Algorithm 4) is structurally analogous to SIENA’s covering / merging operations, but uses an LLM in place of the syntactic-subsumption test — a deliberate design choice motivated by the natural-language subscription descriptions targeted by this work. Scalability beyond syntactic-filter regimes has been pursued through space-partitioning techniques that cluster subscriptions hierarchically [Wang and Ma 2014] and Kafka extensions with specialised low-latency topic types [Qian et al. 2021]. Network-level optimisations embed matching into SDN data planes via P4 [Wernecke et al. 2022] or OpenFlow [Bhowmik et al. 2018]. Privacy-preserving variants apply order-preserving encryption [Li et al. 2020] or data-splitting techniques [Denis et al. 2020] to enable encrypted matching. Publisher-side filtering has been proposed for edge deployments where bandwidth is constrained [Wang et al. 2024]. A common limitation of these systems is their reliance on structured, attribute-based filter languages. Subscriptions must be expressed as Boolean predicates over named fields, which limits expressiveness when content is unstructured text, sensor data, or other high-dimensional modalities. Semantic pub/sub systems such as OpenPubSub [Zaarour et al. 2022] address part of this gap by using embedding-based similarity for routing in peer-to-peer networks, but the matching function remains a fixed cosine threshold rather than a learned or reasoned evaluation. 2.2
Distributed stream processing
Modern data infrastructure relies on distributed stream processing platforms that share pub/sub’s concern with high-throughput event dissemination but differ in their matching semantics. Apache Kafka [Kreps et al. 2011] provides durable, partitioned log-based messaging with topic-based routing and configurable delivery guarantees. Apache Flink [Carbone et al. 2015] extends the stream processing model with stateful operators, event-time semantics, and exactly-once processing, enabling complex event processing (CEP) over continuous data flows. Ray [Moritz et al. 2018] provides a general-purpose distributed computing framework that has become a standard substrate for distributed LLM serving and training. Efficient LLM serving systems such as vLLM [Kwon et al. 2023] and SGLang [Zheng et al. 2024] build on these platforms to optimise memory management and structured generation for high-throughput inference. These platforms are complementary to the Neural Router: they provide the transport and execution substrate on which a semantic matching broker can be deployed, but they do not themselves perform content-based semantic matching. Kafka routes by topic partition, Flink by user-defined operator logic, and Ray by task scheduling. None evaluates whether an event’s meaning satisfies a subscriber’s interest. 2.3
LLM routing and semantic routing
In the machine-learning community, “routing” refers to selecting which model should handle a given query, optimising cost–quality trade-offs across a pool of LLMs. RouteLLM [Ong et al.
Neural Router: Semantic Content Matching for Agentic AI
0:5
2025] trains a router on preference data to decide, per query, whether to invoke a strong or weak model, achieving substantial cost savings on standard benchmarks with minimal quality loss. RouterBench [Hu et al. 2024] provides a systematic benchmark and evaluation framework for such multi-LLM routing systems. The vLLM Semantic Router [Wang et al. 2025] extends this idea to serving infrastructure, using semantic embeddings to direct queries to specialised model replicas within the vLLM serving engine. A related line of work uses semantic similarity for intent classification: the Semantic Router framework [Briggs and Aurelio AI 2024] classifies user utterances into predefined intent categories using embedding distance, triggering deterministic actions without LLM invocation. This is effectively a dispatch mechanism that routes queries to handlers. All of these systems route queries to models or actions. The Neural Router addresses a fundamentally different problem: it uses an LLM to match published content against natural-language subscriber interests within a pub/sub broker. The LLM is not the destination of a route but the engine that evaluates the matching function. 2.4
LLMs in publish/subscribe
Several recent works bring LLMs and pub/sub closer together, though none places the LLM inside the matching loop. Saleh et al. [Saleh et al. 2025a] survey how existing message broker architectures (Kafka, RabbitMQ, MQTT, NATS, Redis) can be adapted to serve generative AI workloads, such as prompt routing, model output distribution, and agentic communication. They identify architectural requirements for AI-native brokers but do not propose LLM-based matching. A companion survey [Saleh et al. 2024] reviews pub/sub for edge intelligence, cataloguing how pub/sub has been used in IoT and edge computing and identifying open challenges for AI-driven enhancements. Isahagian et al. [Isahagian et al. 2023] demonstrate LLM-enhanced pub/sub in which subscribers express interests in natural language and the system uses an LLM to generate customised notifications from matched publications. Their focus is on notification generation (post-match enrichment), not on the matching function itself: the underlying matching still relies on conventional mechanisms. Tarkoma et al. [Tarkoma et al. 2023] propose the AI Interconnect, a multi-layer semantic pub/sub architecture for 6G systems that envisions LLMs as orchestrators for prompt routing, inference result dissemination, and model-update distribution. The LLM orchestrates the pub/sub fabric but does not serve as the content-matching engine. The Neural Pub/Sub paradigm [Lovén et al. 2023] introduces the concept of using neural models for content-based routing in the computing continuum, proposing a distributed architecture in which neural matching replaces traditional predicate evaluation. The present paper instantiates this concept for a single broker, providing the formal problem statement, algorithms, cost model, and experimental evaluation. 2.5
Autonomic and adaptive systems for the computing continuum
In TAAS terms the Neural Router is an adaptive system: a closed calibration-and-assignment loop (Algorithm 1) selects an LLM backend per cluster under operator-preference weights, and the costmodel crossover predicates partition the deployment regime. We position the contribution against the autonomic-computing tradition originating with Kephart and Chess [Kephart and Chess 2003] and the self-adaptive-software taxonomy of Salehie and Tahvildari [Salehie and Tahvildari 2009], which together developed control-theoretic, MAPE-K, requirements-aware, and most recently AI-augmented architectural styles for self-management [Nascimento et al. 2023]. Closely related at TAAS, the MemIndex framework [Saleh et al. 2025b] proposes intent-indexed adaptive memory
0:6
Lovén et al.
management for LM-based multi-agent pub/sub — a complementary layer to the broker-level matching function 𝜇 addressed here. The Neural Router is in the AI-augmented direction (LLM as decision component, per-cluster QoE calibration as adaptation trigger), with the QoE scalarisation echoing QoS-aware service composition [Ardagna and Pernici 2007; Zeng et al. 2004] applied to LLM-tier selection. We do not adopt MAPE-K vocabulary specifically because the calibration loop is one-shot at design time rather than a continuously running cycle; a runtime extension that lifts 𝑊cross and |S| cross into online predicates is left to follow-on work. The Edge-HPC/Cloud computing continuum [Beckman et al. 2020; Donta et al. 2023; Kokkonen et al. 2023; Parashar 2025] extends these concerns across heterogeneous tiers and argues that autonomic management techniques must evolve to handle the resulting distribution and dynamic behaviour. We engage this framing analytically — the cost-model crossover and the heterogeneous-backend QoE assignment apply at any tier — but defer cross-tier deployment to follow-on work; the experimental backends in this paper span HPC (CSC Mahti / Puhti) and cloud (Anthropic API), with the edge-tier 𝑊 =4K regime addressed analytically only. 2.6
Positioning
The five bodies of work above are complementary rather than competing. Traditional content-based pub/sub uses structured filters and provides no semantic understanding; embedding-based variants such as OpenPubSub use a fixed cosine threshold and no reasoning. Stream-processing platforms (Kafka, Flink, Ray) and modern LLM-serving systems (vLLM, SGLang) are transport and execution substrates that do not themselves perform content matching. LLM routing systems (RouteLLM, RouterBench, the vLLM Semantic Router, Aurelio’s Semantic Router) select which model handles a query, the inverse of our problem. Recent LLM-pub/sub work uses LLMs for orchestration [Tarkoma et al. 2023], post-match notification generation [Isahagian et al. 2023], or overall architectural framing [Lovén et al. 2023; Saleh et al. 2025a], never as the matching engine itself. The Neural Router occupies this gap: it defines a formal matching function evaluated by an LLM, supported by embedding-based clustering, cover/merge compression, and a context-window-aware cost model for batch processing. 3
Neural Router Design
3.1 Problem Statement We consider a content-based publish/subscribe (pub/sub) system in which subscribers express their interests as free-text natural-language descriptions. Let S = {𝑠 1, . . . , 𝑠𝑛 } denote the active subscriptions and E = {𝑒 1, . . . , 𝑒𝑚 } the published events. A matching function 𝜇 (𝑒 𝑗 , 𝑠𝑖 ) ∈ {0, 1} returns 1 when event 𝑒 𝑗 satisfies subscription 𝑠𝑖 ; the routing problem is to compute, for every 𝑒 𝑗 , the set 𝑅(𝑒 𝑗 ) = {𝑠𝑖 | 𝜇 (𝑒 𝑗 , 𝑠𝑖 ) = 1} and deliver accordingly. The design objective is to maximise F1 subject to latency and cost constraints, encoding an explicit accuracy–latency trade-off : LLM-based matching offers richer semantic understanding than syntactic filters, but each invocation carries nontrivial latency (tens to hundreds of milliseconds depending on model and deployment). Traditional pub/sub routers evaluate 𝜇 with Boolean filters or keyword matching, which fail under paraphrase, synonymy, and contextual meaning; the Neural Router addresses this gap by implementing 𝜇 through an LLM, supported by embedding-based clustering to partition the matching workload into tractable units. The targeted regime is one where per-event latency on the order of seconds is acceptable (analytics dashboards, notification feeds, content recommendation, regulatory-alert dissemination); real-time control loops (sub-millisecond matching) are outside scope.
Neural Router: Semantic Content Matching for Agentic AI
0:7
OptimizeSubscriptions
offline
Subscriptions S
Embed 𝜃 (𝑠)
𝑘-Means Cluster
Cover/ Merge
Assign LLM LLM Semantic Matching 𝜇 (𝑒, 𝑠)
centroids v̄𝑐
MatchEvents
online
Events E
Embed 𝜃 (𝑒)
Cosine Filter 𝜏
Append 𝑒 to 𝑐.queue
Match Results 𝑀 ⊆ E ×S
Batch/ Pack
Fig. 2. Neural Router architecture (single-broker view). Top (offline): subscriptions are embedded, clustered by semantic similarity, compressed via cover/merge, and assigned to LLM instances (OptimizeSubscriptions). Bottom (online): events are embedded and filtered to candidate clusters by cosine similarity to cluster centroids, then batched and packed into LLM prompts (MatchEvents). The LLM evaluates the semantic matching function 𝜇 (𝑒, 𝑠) and returns match results.
3.2
Architecture Overview
The Neural Router is a content-based pub/sub broker that replaces filter evaluation with LLMdriven semantic matching (Fig. 2). Two processing paths share a common embedding space. The subscription path (offline) embeds free-text subscription descriptions with a pre-trained encoder (e.g., Sentence-BERT [Reimers and Gurevych 2019]), clusters the embeddings into 𝑘 semantically coherent groups via 𝑘-means, compresses each cluster via cover/merge (subscriptions subsumed by a broader one are removed; similar subscriptions are combined), and assigns each cluster to an LLM instance based on QoE criteria such as accuracy and latency budget. The event path (online) embeds incoming events, routes them to the nearest cluster(s) by cosine similarity to cluster centroids, accumulates them into batches by time window or count threshold, and packs each batch with the cluster’s compressed subscription set into an LLM prompt; the LLM returns event-to-subscription matches. This two-level architecture (coarse-grained embedding clustering for scale, fine-grained LLM matching for precision) confines each LLM invocation to a semantically coherent subset of the subscription space and keeps prompts within context-window limits. Matching and delivery semantics. The matching function 𝜇 (𝑒, 𝑠) evaluates whether the semantic content of event 𝑒 satisfies the interest expressed in subscription 𝑠 rather than relying on topic labels — the canonical content-based pub/sub semantics, here delegated to an LLM. The current implementation provides at-most-once delivery; extending to at-least-once (with retry) or exactly-once (with deduplication) is a transport-layer concern and does not affect the matching algorithms. Because LLM inference is not bitwise deterministic even at temperature 0 (GPU floatingpoint non-determinism produces occasional variation), the matching function is not guaranteed idempotent across repeated invocations on the same input; we mitigate this with greedy decoding (temperature 0) and structured-output parsing, and deployments needing strict idempotency can cache results keyed on (event hash, subscription-set hash). 3.3
Algorithms
The Neural Router’s core function is semantic matching — deciding, for each event, which subscriptions it satisfies (forwarding matched content across a network of brokers belongs to the distributed
0:8
Lovén et al.
architecture and is outside the scope of this paper). We formalise the matching pipeline through three procedures whose processing flows mirror Fig. 2. Subscription Optimisation (Algorithm 2). OptimizeSubscriptions prepares the matching table. Given subscriptions S, an embedding model 𝜃 , the number of clusters 𝑘, and an LLM pool L, it (1) embeds every subscription, (2) partitions the embeddings into 𝑘 clusters via 𝑘-means, (3) for each cluster selects the LLM that maximises a quality-of-experience (QoE) score (combining accuracy, latency, and cost targets) and invokes LLM-assisted cover/merge compression (Algorithm 4), and (4) computes the cluster centroid for later event assignment. The QoE score per cluster 𝑐 and backend ℓ ∈ L𝑐 (the set of candidate backends for 𝑐) is a weighted scalarisation of three per-cluster, per-backend metrics, each min-max normalised across L𝑐 to neutralise absolute-scale dependence: QoE(𝑐, ℓ) = 𝛼 𝐹e1 (𝑐, ℓ) + 𝛽 𝐶e(𝑐, ℓ) + 𝛾 e 𝐿(𝑐, ℓ), 𝛼 + 𝛽 + 𝛾 = 1, 𝛼, 𝛽, 𝛾 ≥ 0. (1) e(𝑐, ℓ) is the min-max-normalised version of its raw counterpart over L𝑐 — Each component 𝑋 ˆ 𝐹 1 (𝑐, ℓ) for accuracy, 𝑝 (ℓ)𝑇¯ (𝑐, ℓ) for cost, and 𝑡¯llm (ℓ) for latency — with cost and latency inverted (lower-is-better) so all three lie in [0, 1] with higher = better, and 0/0 := 0.5 when a metric is constant across L𝑐 . Three weight presets operationalise operator preference: accuracy_first (𝛼, 𝛽, 𝛾) = (0.70, 0.15, 0.15), balanced (0.34, 0.33, 0.33), cost_first (0.15, 0.70, 0.15). This weighted-sum scalarisation places per-cluster backend selection in the QoS-aware service-composition tradition [Ardagna and Pernici 2007; Zeng et al. 2004], and inherits that tradition’s well-known caveat: a linear scalarisation can only reach Pareto-optimal backends on the convex hull of the accuracy/cost/latency trade-off surface; a backend that is Pareto-optimal but lies in a non-convex region of the front is unreachable by any weight setting (𝛼, 𝛽, 𝛾). On the two-tier gradient evaluated here the candidate front is effectively convex, so this does not bind; at larger backend pools a Chebyshev or 𝜀-constraint scalarisation would be needed to reach non-convex-front backends, which we leave to follow-on work. Calibration and assignment (Algorithm 1). The per-backend accuracy estimate 𝐹ˆ1 (𝑐, ℓ) is obtained through a one-shot calibration phase: each backend processes a 10 % held-out sample of cluster 𝑐’s events, and macro-F1, mean per-event latency 𝑡¯llm (ℓ), and expected token count 𝑇¯ (𝑐, ℓ) are recorded. Backend ℓ ∗ (𝑐) = arg maxℓ ∈ L𝑐 QoE(𝑐, ℓ) is selected per cluster. Backends with calibrated 𝐹ˆ1 = 0 across all clusters are filtered before normalisation to avoid trivial argmax pathologies on saturating denominators (this filter activates on the Qwen-1.5B A0 F1=0.000 cell in Section 5.5). The empirical QoE study under all three presets is reported in Section 5.6; analyses of round-robin as a calibration-free baseline and of QoE-optimised under varying calibration-sample fractions are also in that section. Algorithm 1 Calibrate and assign (QoE). 1: procedure CalibrateAndAssign(𝐶, L, 𝐸 cal, (𝛼, 𝛽, 𝛾)) 2: for 𝑐 ∈ 𝐶, ℓ ∈ L do (𝑐 ) Run ℓ on a 10% sample 𝐸 cal ⊂ 𝐸 assigned to 𝑐; ˆ ¯ 4: record 𝐹 1 (𝑐, ℓ), 𝑡 llm (ℓ), 𝑝 (ℓ)𝑇¯ (𝑐, ℓ). 5: end for 6: for 𝑐 ∈ 𝐶 do 7: L𝑐 ← {ℓ ∈ L : 𝐹ˆ1 (𝑐, ℓ) > 0} ee 8: Compute 𝐹e1, 𝐶, 𝐿 over L𝑐 per Eq. 1. 9: 𝑐.ℓ ← arg maxℓ ∈ L𝑐 [𝛼 𝐹e1 + 𝛽𝐶e + 𝛾 e 𝐿]. 10: end forreturn {𝑐.ℓ }𝑐 ∈𝐶 11: end procedure
3:
⊲ Filter saturating denominators
Neural Router: Semantic Content Matching for Agentic AI
0:9
Algorithm 2 Subscription optimisation. 1: procedure OptimizeSubscriptions(S, 𝜃, 𝑘, L, reunite) 2: v ← [ 𝜃 (𝑠) | 𝑠 ∈ S ] 3: 𝐶 ← KMeans(v, 𝑘) 4: for 𝑐 ∈ 𝐶 do 5: 𝑐.ℓ ← arg maxℓ ∈ L QoE(𝑐, ℓ)
𝑐.S ′ ← CoverAndMerge(𝑐.S, 𝑐.ℓ) 𝑐.v̄ ← mean{𝜃 (𝑠) | 𝑠 ∈ 𝑐.S ′ } end for if reunitethen Ð 10: 𝐶 ← ( 𝑐 ∈𝐶 𝑐.S ′, 𝑐 1 .ℓ) 11: end ifreturn 𝐶 12: end procedure 6: 7: 8: 9:
⊲ Embed all subscriptions ⊲ Cluster: 𝐶 = {𝑐 1, . . . , 𝑐𝑘 } ⊲ Select LLM for cluster ⊲ Compress subscriptions ⊲ Cluster centroid ⊲ Merge into single search space
An optional reunite flag (line 8 of Algorithm 2) merges all compressed clusters back into a single search space after per-cluster compression; its effect is evaluated in the ablation study (Section 4). Event Matching (Algorithm 3). MatchEvents evaluates the matching function 𝜇 for a batch of incoming events. Each event is embedded using the same model 𝜃 , then assigned to every cluster whose centroid similarity exceeds a threshold 𝜏. Events accumulate in per-cluster queues; for each non-empty queue, the compressed subscriptions and queued events are packed into an LLM prompt, and the cluster’s LLM returns the top-𝜅 matching subscriptions per event. Algorithm 3 Event matching. 1: procedure MatchEvents(E, 𝐶, 𝜃, 𝜏, 𝜅) 2: for 𝑒 ∈ E do
v𝑒 ← 𝜃 (𝑒) for 𝑐 ∈ 𝐶 do if cos(v𝑒 , 𝑐.v̄) ≥ 𝜏 then 𝑐.queue.append(𝑒) end if end for 9: end for 10: 𝑀←∅ 11: for 𝑐 ∈ 𝐶 where 𝑐.queue ≠ ∅ do 12: 𝑝 ← Pack(𝑐.S ′, 𝑐.queue, 𝜅) 13: 𝑀 ← 𝑀 ∪ InvokeLLM(𝑐.ℓ, 𝑝) 14: end forreturn 𝑀 15: end procedure 3: 4: 5: 6: 7: 8:
⊲ Embed event ⊲ Centroid similarity
⊲ Set of matching decisions ⊲ Build LLM prompt ⊲ (𝑒, 𝑠) pairs
LLM-Assisted Cover and Merge (Algorithm 4). Cover-AndMerge compresses the subscription set of a single cluster using two operations. Covering removes a subscription 𝑠 𝑗 whose semantics are entirely subsumed by a broader subscription 𝑠𝑖 ; merging replaces two overlapping subscriptions with a single combined entry that inherits both subscriber sets. Both operations aim to preserve recall: each merged subscription represents the union of its inputs’ subscribers, so an event matched to a merged subscription notifies all subscribers whose original subscription contributed to the merge. This recall-preservation guarantee is conditional on the matching
0:10
Lovén et al.
LLM engaging with the merged subscription set: at high cardinality, the LLM may collapse to a narrow prediction vocabulary and emit empty matches for many events — a regime we term empty-prediction collapse (used hereafter as the canonical name; “refusal” appears as descriptive shorthand for an empty structured-output prediction, distinct from the safety-tuning literature’s use for explicit task decline) — even when |S| eff fits within the context window (Section 5.4). We therefore refine the discrimination-capacity boundary |S| cross to depend on subscription-set composition (atomic vs. compound merged subscriptions), not cardinality alone; characterising this dependence quantitatively across LLMs and datasets is left to follow-on work. The subsumption and merge decisions are delegated to the cluster’s LLM via the prompt template in Suppl. Listing 1: the LLM receives the cluster’s subscriptions and identifies which pairs can be covered or merged. The algorithm applies the LLM’s decisions iteratively until a fixed point is reached (no further compression possible). In principle, SubsumptionTest and MergeTest can also be implemented with deterministic heuristics (e.g., embedding-distance thresholds); the LLM-assisted variant evaluated in this paper delegates both to the same prompt. Termination. Every applied cover removes one subscription and every applied merge replaces two subscriptions with one, so each iteration with changed=true strictly decreases |𝑆 | by at least one while no operation increases it; since |𝑆 | is a non-negative integer, CoverAndMerge executes at most |𝑆 | − 1 LLM rounds and is guaranteed to reach a fixed point, independent of the LLM’s proposals on any individual round. Algorithm 4 LLM-assisted subscription compression. 1: procedure CoverAndMerge(𝑆, ℓ) 2: repeat 3: changed ← false 4: (covers, merges) ← InvokeLLM(ℓ, CoverMergePrompt(𝑆)) 5: for (𝑠𝑖 , 𝑠 𝑗 ) ∈ covers do
𝑆 ← 𝑆 \ {𝑠 𝑗 }; changed ← true end for for (𝑠𝑖 , 𝑠 𝑗 ) ∈ merges do 𝑆 ← (𝑆 \ {𝑠𝑖 , 𝑠 𝑗 }) ∪ {Merge(𝑠𝑖 , 𝑠 𝑗 )}; end for until ¬ changed return 𝑆 12: end procedure 6: 7: 8: 9: 10: 11:
⊲ 𝑠𝑖 subsumes 𝑠 𝑗
changed ← true ⊲ Compressed subscription set
Design space. The algorithms above expose several configuration knobs that trade off matching accuracy against system cost: • Number of clusters 𝑘 controls subscription partitioning granularity. Larger 𝑘 yields smaller clusters (less LLM context per invocation, faster prompts) but may split semantically related subscriptions. • Cover/merge compression (enabled or disabled) reduces the token footprint of the subscription set at the risk of over-generalisation. • Reunite flag (reunite, line 8 of Algorithm 2) merges all compressed clusters back into a single search space after per-cluster compression, maximising LLM context at the cost of larger prompts. • Cosine threshold 𝜏 controls the precision–recall trade-off of the event-assignment prefilter. Lower 𝜏 sends events to more clusters (higher recall, higher cost); higher 𝜏 restricts assignment.
Neural Router: Semantic Content Matching for Agentic AI
0:11
• Event-side clustering (𝑘𝑒 ) optionally pre-groups events before matching, producing fewer but larger batches at the cost of coarser event–subscription granularity. • Top-𝜅 matches determines how many subscriptions the LLM returns per event. The ablation study in Section 4 systematically evaluates the contribution of each component. 3.4
Prompt Design
The Neural Router uses two prompt templates, reproduced verbatim in Suppl. App. A. The subscription optimisation prompt (Listing 1) instructs the LLM to identify which subscriptions within a cluster can be covered or merged; it expects structured JSON output listing the cover/merge decisions consumed by Algorithm 4. The event matching prompt (Listing 2) presents the compressed subscription set alongside a batch of events and asks the LLM to return the top-𝜅 semantically matching subscriptions per event, implementing 𝜇 from Section 3.1. Both templates use Python f-string placeholders for runtime injection of cluster subscriptions and event batches. 3.5
Cost Model
Each LLM invocation in the Neural Router consumes a prompt that must fit within the model’s context window. We derive how cover/merge compression affects the dominant cost: the number of LLM invocations during event matching. Context-window constraint. Let 𝑊 denote the context-window size (in tokens), 𝑡 inst the fixed token cost of the instruction template, 𝑡𝑠 the average token cost of one subscription, 𝑡𝑒 the average token cost of one event, and 𝑡 resp a token budget reserved for the model’s response. A single matching prompt for cluster 𝑐 must satisfy 𝑡 inst + |𝑐.S ′ | · 𝑡𝑠 + 𝑏 · 𝑡𝑒 + 𝑡 resp ≤ 𝑊 ,
(2)
where |𝑐.S ′ | is the number of compressed subscriptions in the cluster and 𝑏 is the number of events in the batch. The maximum batch size that fits alongside the subscriptions is therefore 𝑊 − 𝑡 inst − |𝑐.S ′ | · 𝑡𝑠 − 𝑡 resp 𝑏 max (𝑐) = . (3) 𝑡𝑒 Effect of cover/merge compression. Let 𝜌𝑐 = |𝑐.S ′ |/|𝑐.S| ≤ 1 denote the compression ratio for cluster 𝑐 (the fraction of subscriptions retained after cover/merge). Compression frees (1−𝜌𝑐 ) |𝑐.S| 𝑡𝑠 tokens, increasing the maximum batch size by (1 − 𝜌𝑐 ) |𝑐.S| 𝑡𝑠 Δ𝑏 (𝑐) = (4) 𝑡𝑒 additional events per invocation. This is the primary payoff of compression: it converts subscription token savings into higher event throughput per LLM call. Total matching cost. If 𝑚𝑐 events are assigned to cluster 𝑐 (via the centroid-similarity test in Algorithm 3), the number of LLM invocations for that cluster is 𝑚𝑐 𝐼𝑐 = . (5) 𝑏 max (𝑐) Í The total invocations across all active clusters are 𝐼 = 𝑐 ∈𝐶 𝐼𝑐 . With 𝑃 parallel LLM instances, the number of sequential rounds is 𝑅 = ⌈𝐼 /𝑃⌉. Matching latency. Let 𝑡¯llm denote the mean wall-clock time of a single LLM invocation (including prompt encoding, generation, and any network round-trip for cloud-hosted models). The end-to-end
0:12
Lovén et al.
matching latency for a batch of events is 𝐼 · 𝑡¯llm . 𝐿 = 𝑅 · 𝑡¯llm = 𝑃
(6)
When the context window is the binding constraint, compression reduces 𝐼 (fewer invocations) and batching increases 𝑏 max (more events per invocation), both directly reducing 𝐿. The per-event amortised latency is 𝐿/𝑚, which decreases with larger event batches. For a deployment with 𝑃 parallel instances and compressed subscriptions, the per-event latency scales as O (𝑡¯llm /(𝑃 · 𝑏 max )), making it tuneable through parallelism and compression. Configuration extremes. At |𝐶 |=1 (single cluster, 𝑘=1 or reunite=true), |𝑐.S ′ | is large and 𝑏 max small: every event matches against all subscriptions, with maximal LLM context. At |𝐶 |=𝑘 ≫ 1, each cluster carries |S ′ |/𝑘 subscriptions and 𝑏 max rises, but only clusters with queued events incur LLM calls. Event-side clustering (𝑘𝑒 >1) pre-groups events into larger batches at the cost of coarser event–subscription granularity. The ablation study (Section 4) evaluates these trade-offs. Worked example. The benefit of compression depends critically on the context-window size 𝑊 . Consider a fixed subscription set with 𝑘 = 10 clusters, 25 subscriptions each (|S| = 250), 𝑚 = 6,000 events uniform across clusters, and constants 𝑡 inst = 200, 𝑡𝑠 = 80, 𝑡𝑒 = 50, 𝑡 resp = 500. In the constrained regime (𝑊 = 4K, typical of edge models), the per-cluster batch size is 𝑏 max = 27 at 𝜌 = 1, requiring 𝐼 = 230 invocations; compression at 𝜌 = 0.6 frees enough tokens to lift 𝑏 max to 43, reducing invocations to 𝐼 = 140 (a 39% reduction). In the abundant regime (𝑊 = 128K, typical of cloud APIs), the entire subscription set consumes only a small fraction of the context window: 𝑏 max = 2,507 at 𝜌 = 1 and 2,523 at 𝜌 = 0.6, both yielding 𝐼 = 10 — the 16-event increase from compression is negligible. Worse, clustering splits subscriptions into 𝑘 groups, each requiring a separate LLM call: even with only one event per cluster, the minimum invocation count is 𝑘. Without clustering (A0), all subscriptions and events fit into 1–2 prompts. For example, A0 on D1 (19 subscriptions, 100 events) requires only 2 invocations totalling 7,514 prompt tokens, whereas A3 (with clustering and compression) requires 24 invocations totalling 11,534 prompt tokens. Crossover analysis. The cost model predicts a crossover window size 𝑊cross below which compression reduces invocations and above which the clustering overhead dominates. From Eq. (3), the per-cluster batch size is 𝑏 max (𝑐) = (𝑊 − 𝑡 inst − 𝑡 resp − 𝑡𝑠 |𝑐.S ′ |)/𝑡𝑒 . With 𝜌 = 1 and 𝑘 = 1 (no compression, no clustering) the per-prompt subscription footprint is 𝑡𝑠 |S|, while with compression at ratio 𝜌 and 𝑘 clusters the average per-cluster footprint is 𝑡𝑠 𝜌 |S|/𝑘. Setting 𝑏 max (𝑘 = 1, 𝜌 = 1) · 1 = 𝑏 max (𝑘, 𝜌) · 𝑘 (i.e., the total event-batching capacity matches across configurations) and solving for 𝑊 yields 1 − 𝜌/𝑘 𝑊cross = 𝑡 inst + 𝑡 resp + 𝑡𝑠 |S| · . (7) 1 − 1/𝑘 For the per-event token cost, we use the cost-validation production constants 𝑡𝑠 =80, 𝑡𝑒 =30 (Fig. 3a); the worked example above uses an alternative 𝑡𝑒 =50 to illustrate a more conservative event-token estimate (the qualitative picture is unchanged). With 𝑡 inst =200, 𝑡 resp =500, 𝜌=0.5, 𝑘=9: for |S|=19 (D1), 𝑊cross ≈ 700+80·19·0.94/0.89 ≈ 8K tokens; for |S|=201 (D2), 𝑊cross ≈ 700+80·201·0.94/0.89 ≈ 64K tokens. Modern cloud LLMs operate at 𝑊 ≥ 128K (with recent models reaching 200K), well above 𝑊cross for all three evaluation datasets. This explains the ablation finding that A0 dominates on cost: the context window is no longer the binding constraint, so the overhead of 𝑘 separate cluster prompts exceeds the savings from compression. For edge deployments with constrained local models (𝑊 ≤ 8K), the original compression benefit applies in full. Discrimination capacity. The crossover analysis above assumes the LLM’s matching accuracy is invariant to |S| as long as all subscriptions fit within the context window. This assumption conflates two resources: context capacity 𝑊 (how many tokens fit) and discrimination capacity 𝐷
Neural Router: Semantic Content Matching for Agentic AI
0:13
(how many subscriptions the LLM can evaluate concurrently with acceptable accuracy). The matching prompt requires the LLM to make |S| simultaneous binary relevance decisions over a shared attention pool, so the per-subscription attention budget is bounded by the model’s pool and discrimination degrades as decision load grows. This yields a second crossover point |S| cross , below which A0 achieves both cost and accuracy optimality, and above which LLM matching accuracy declines — potentially below independent pairwise baselines (TF-IDF, Sentence-BERT) whose per-pair decisions do not share attention. Crucially, |S| cross is model-dependent rather than a global scalar; the empirical evidence and the joint parameter-count × training-generation factor are reported in Sections 5.5 and 6.1. The two crossover points partition the deployment space into three regimes: • |S| < |S| cross and 𝑊 > 𝑊cross : A0 dominates on both cost and accuracy (D1, D3 across all evaluated models). • |S| < |S| cross and 𝑊 < 𝑊cross : compression pipeline reduces cost while preserving accuracy (edge deployments). • |S| > |S| cross : discrimination capacity is the binding constraint regardless of 𝑊 , but the binding strength is model-dependent. Frontier models may remain viable through partial escape, while resource-constrained models require alternative architectures (e.g., partitioning subscriptions to keep per-prompt |S ′ | ≤ 𝐷, or cascade approaches using embedding pre-filters). Full characterisation of the discrimination-capacity scaling law and its dependence on LLM architecture is left to follow-on work; here we identify the boundary qualitatively, validate it empirically at four open-weight data points and one closed-API single-seed reference (Sections 5 and 6), and observe that the four points are consistent with a joint parameter-count × training-generation factor rather than with parameter count alone. The single matched-scale cross-family pair (Mistral-7B vs. Qwen-2.5-7B) is suggestive but not definitive; broader cross-family validation is required before a closed-form law can be claimed. Token-based cost. When backends have different per-token prices, invocation count alone is insufficient. The per-event monetary cost is 𝐶 event = (𝑇prompt · 𝑝 in + 𝑇response · 𝑝 out )/𝑚
(8)
where 𝑇prompt,𝑇response are total prompt/response tokens across all invocations, 𝑝 in, 𝑝 out are pertoken prices, and 𝑚 is event count. For A0 on D1 (𝑇prompt =7,514, 𝑇response =2,029, 𝑚=100): Qwen 2.5 (Ollama) is $0/event (compute amortisation only); Claude 3 Haiku is ≈$0.023/1K; Claude Sonnet is ≈$0.028/1K. The blended-token list-price ratio is approximately 0 : 1 : 4 (Qwen:Haiku:Sonnet), but on this workload Sonnet’s prompt and response token counts are not 4× Haiku’s, so the realised per-event ratio is closer to 0 : 1 : 1.2. Cover/merge offline cost is amortisable: subscription updates occur far less frequently than event arrivals. 4
Experimental Setup
We evaluate the Neural Router along eight dimensions: (1) a component ablation study isolating the contribution of each design element, (2) comparison against embedding-similarity and keywordbased baselines, (3) parameter sensitivity analysis, (4) validation of the cost model derived in Section 3.5, (5) cross-dataset comparison (varied |S|, modality gap, and capability tier), (6) crossover validation under a context-constrained budget (Section 4.7), (7) a discrimination-capacity panel sweeping four open-weight backends on D2 (Section 4.8), and (8) evaluation of QoE-based heterogeneous backend assignment (Section 4.9). All configurations are run five times with different random seeds for 𝑘-means initialisation; we report means and 95 % confidence intervals throughout.
0:14
Lovén et al.
Table 1. Evaluation datasets. |E | = events, |S| = subscriptions, 𝑛¯𝑠 = mean labels per event, 𝑤¯ = mean words per event. Dataset
|E | |S|
𝑛¯𝑠
𝑤¯ Domain
CardiffNLP [Antypas et al. 2022] 6 000 19 1.6 30 Social media EUR-Lex [Chalkidis et al. 2021] 65 000 201 2.2 700+ EU legislation CASAS [Cook et al. 2013] 23 330 19 1.0 45 Smart home IoT
4.1
Datasets
We evaluate on three publicly available datasets (Table 1), chosen to represent the content diversity that agents encounter across the computing continuum: user-generated text at the application layer (D1), governance and regulatory documents (D2), and IoT sensor streams at the device layer (D3). Together, the datasets span a progression of semantic difficulty from keyword-friendly text matching to abductive reasoning over structured sensor data. In each case, documents or sensor event sequences serve as published events and label descriptions serve as natural-language subscriptions. The ground truth defines the matching function: event 𝑒 matches subscription 𝑠 if and only if 𝑠 is among 𝑒’s annotated labels. D1: CardiffNLP Tweet Topic [Antypas et al. 2022] pairs short English tweets with 19 coarse topics ("sports", "arts & culture", . . . ). D2: EUR-Lex (MultiEURLEX) [Chalkidis et al. 2021] uses the 201 EUROVOC level-2 subject matters as subscriptions; documents are full EU legislative texts averaging 700+ words, stress-testing the context-window constraint (Eq. (2)) and the cost model’s batch-size predictions. D3: CASAS Smart Home [Cook et al. 2013] provides ≈ 8 months of freeliving sensor traces from home hh113 (PIR motion, magnetic door, temperature), yielding ≈ 23 330 activity segments across 19 activity types (Cook, Eat, Sleep, etc.) after merging fine-grained variants. Each segment is templated into a natural-language event of the form "At {time}, sensor in {location} reported {value}."; subscription descriptions are generic activity descriptions ("Notify when the resident is cooking, involving kitchen activity. . . ") that do not reference sensor identifiers. This stresses the modality gap: sensor IDs and binary values (M003 ON, D001 OPEN) share no vocabulary with activity descriptions, and the LLM must perform abductive reasoning to bridge them. The ablation and baseline comparison (Sections 4.2 and 4.3) run on all three datasets; the parameter sensitivity sweeps (Section 4.4) use A3 on D1–D3 with 𝑘 matched to the label count where applicable. 4.2
Ablation Configurations
We define seven configurations that progressively enable each Neural Router component (Table 2). Each configuration is evaluated on a primary backend and complemented by a discriminationcapacity sweep on D2 across multiple open-weight scales and one cross-family model: • Qwen 2.5 7B (local, via Ollama): primary open-weight backend, evaluated on D1, D2, D3 for every ablation configuration. Zero API cost; inference latency ∼40–50 s per batch. • Qwen 2.5 1.5B, 32B (local, via Ollama): same-family scaling sweep on D2 to probe how the discrimination-capacity bound depends on parameter count. • Mistral 7B (local, via Ollama): cross-family open-weight model at parameter scale matched to Qwen 2.5 7B, used on D2 to disentangle parameter count from training-generation effects. • Claude Sonnet (Anthropic API): premium commercial backend. Moderate cost; inference latency ∼2–5 s per batch.
Neural Router: Semantic Content Matching for Agentic AI
0:15
• Claude 3 Haiku (Anthropic API): lightweight commercial backend. Lowest API cost; inference latency ∼1–3 s per batch. The four open-weight scales (Qwen 1.5B / 7B / 32B and Mistral 7B at matched scale) form the discrimination-capacity panel reported in Section 5.5; the Qwen 2.5 7B + Anthropic backends carry the main ablation, sensitivity, and scaling tables. For Sonnet, we use a single seed to control API cost. For D2 with Sonnet, we additionally cap at 5 000 stratified events (7.7 % of the full corpus), since the combination of long legal documents and a commercial API model at scale would be prohibitively expensive. For the Qwen-2.5-7B runs executed on a shared HPC partition with a 3-hour per-task walltime budget, events are stratified-subsampled to 1 000 (D1, D3) and 300 (D2). This is a compute-budget constraint, not an architectural one, and is uniform across configurations so withindataset comparisons remain apples-to-apples. The Haiku full-corpus evaluation establishes that the qualitative findings hold at full scale on the same dataset. Configurations A5 (event clustering) and A6 (no cosine filter) are evaluated on D1 and D3 only; on D2 the discrimination-capacity finding (Section 5.5) implies all configurations struggle equally because the binding constraint is the LLM’s per-subscription attention budget, so adding A5 and A6 measurements on D2 contributes negligible insight beyond what A0–A4 already establish. All configurations use Sentence-BERT (all-MiniLM-L6-v2) as the default embedding model unless otherwise noted in the sensitivity analysis (Section 4.4). The experiment pipeline is managed by DVC [Iterative, Inc. 2020] for full reproducibility; all stages, parameters, and intermediate results are version-controlled. LLM decoding and reproducibility. All LLM calls use temperature = 0 (greedy decoding) to minimise non-determinism. The matching and cover/merge prompts request structured JSON output, which is parsed deterministically. Despite greedy decoding, exact reproducibility is not guaranteed across LLM invocations: cloud API models may be updated by the provider, and local GPU inference can exhibit floating-point non-determinism across hardware or driver versions. We mitigate this by recording the API model snapshot identifiers (see Section 4.11) and by reporting results over multiple seeds, so that the variance attributable to 𝑘-means initialisation is captured explicitly. The five-seed protocol primarily controls for clustering randomness; LLM output variation at temperature = 0 is empirically negligible within a single model snapshot. Seed and subsample protocol. The ablation, cross-dataset, and discrimination-capacity tables use five 𝑘-means seeds {42, 123, 456, 789, 1024} for every (dataset, configuration) cell on the openweight Qwen and Mistral backends; Sonnet rows are single-seed by API-cost necessity and are marked indicative. The QoE perturbation (Section 5.6.1) and calibration-fraction (Section 5.6.2) experiments use 𝑛=15 seeds under a matched-pair regime with an on-disk LLM-call cache, so GPU and decoding nondeterminism cancel in perturbed-vs-baseline deltas; the wider-tier 7B/72B companion (Suppl. App. F) likewise uses 𝑛=15. Subsample caps (1,000 events for D1/D3 and 300 for D2 on Qwen-2.5-7B; 5,000 for Sonnet on D2) are compute-budget constraints applied uniformly within a dataset, so within-dataset comparisons remain matched-pair. Per-cell run provenance — SLURM job identifiers, completion timestamps, and regenerated-CSV paths — is recorded in the artefact’s operations log; the one dropped baseline-vs-injection cell in Section 5.6.1 (𝑛=14 for that comparison) is noted in place. Table 2 shows the seven configurations. A0 batches all subscriptions and events into a single LLM prompt with no clustering or compression — the accuracy ceiling and cost floor below the discrimination-capacity crossover. A1 clusters subscriptions into 𝑘 groups and routes events by centroid cosine similarity (𝜏) without cover/merge. A2 applies cover/merge to the full unpartitioned subscription set. A3 enables both clustering and compression. A4 adds the reunite flag, merging compressed clusters back into a single search space. A5 adds event-side clustering (𝑘𝑒 =5) on top of A3. A6 sets 𝜏=0 (all events to all clusters), isolating the cosine pre-filter’s contribution.
0:16
Lovén et al.
4.3
Evt . cl
Cos .
filte r
Reu
ust.
C&M
nite
Config
Clu st
er
Table 2. Ablation configurations. Default parameters: 𝑘=19, 𝜏=0.3, 𝜅=3.
A0: Raw LLM A1: Cluster only A2: C&M only A3: Clust. + C&M A4: + Reunite A5: + Evt. clust. A6: No cos. filter
– ✓ – ✓ ✓ ✓ ✓
– – ✓ ✓ ✓ ✓ ✓
– – – – ✓ – –
– – – – – ✓ –
– ✓ – ✓ – ✓ –
Baselines
We compare against seven baselines spanning keyword matching, embedding similarity, pairwise neural scoring, and zero-shot classification: BM25 [Robertson and Zaragoza 2009] as the canonical keyword baseline; Sentence-BERT cosine using the same encoder as the Neural Router (all-MiniLM-L6-v2), isolating the LLM’s contribution beyond embedding geometry; a crossencoder reranker (cross-encoder/ms-marco-MiniLM-L-6-v2) as a pairwise neural scoring model; zero-shot classification with DistilBART-MNLI-12-1 [Lewis et al. 2020; Yin et al. 2019] (valhalla/distilbart-mnli-12-1, distilled from facebook/bart-large-mnli; events classified against subscription descriptions as candidate labels via natural language inference [Laurer et al. 2024; Wang et al. 2018; Williams et al. 2018]); and GloVe, TF-IDF, Word2Vec as static embedding baselines retained for continuity. All baselines return a ranked subscription list per event and are evaluated at the same top-𝜅 cut-off as the Neural Router. 4.4 Parameter Sensitivity We sweep four parameters, varying one at a time with the others fixed at the A3 defaults: cluster count 𝑘 ∈ {1, 5, 10, 15, 19, 25, 30} (𝑘=19 matches the ground-truth topic count); cosine threshold 𝜏 ∈ {0.0, 0.1, . . . , 0.9}; top-𝜅 matches 𝜅 ∈ {1, 2, 3, 5, 7, 10}; and embedding model (all-MiniLM-L6-v2, all-mpnet-base-v2, e5-large-v2, bge-base-en-v1.5). 4.5 Cost Model Validation For every configuration in Sections 4.2 and 4.4, we record per-cluster compression ratio 𝜌𝑐 , measured invocations 𝐼 meas , prompt + response tokens per invocation, and per-step wall-clock times (embedding, clustering, cover/merge, cosine filtering, prompt packing, LLM inference, post-processing); we compare 𝐼 meas against the predicted 𝐼 pred from Eq. (5) (Fig. 3a). Cost is reported as tokens (prompt + response) per event; per-event monetary cost follows Eq. (8) with Qwen 2.5 7B at zero API cost, Claude 3 Haiku at $0.80/M input + $4.00/M output, and Claude Sonnet at $3.00/M input + $15.00/M output. At blended-token list price the per-token cost ratio is approximately 0 : 1 : 4 (Qwen:Haiku:Sonnet); the realised per-event cost ratio in Table 3 is closer to 0 : 1 : 1.2 on D1 because Sonnet’s prompt and response token counts are not 4× Haiku’s on this workload. 4.6
Scaling Analysis
We evaluate how the system behaves as the subscription set grows: empirically by subsampling and duplicating subscriptions to size |S| ∈ {50, 100, 250, 500, 1000, 2000} and measuring 𝐼 , 𝐿, 𝜌, F1 for
Neural Router: Semantic Content Matching for Agentic AI
0:17
A3; analytically by projecting 𝐼 and 𝐿 for |S| ∈ {5,000, 10,000, 50,000, 100,000} from the validated cost model with measured 𝜌 and 𝑡¯llm (Fig. 3c). 4.7
Crossover Validation
To validate the context-window crossover predicted in Section 3.5, we enforce 𝑊 = 4,096 tokens on the matching prompt and sweep |S| ∈ {50, 200, 2,000} on D1 via subsampling and duplication-withrename, comparing A0 (raw LLM, list truncated to fit) against A4 (full pipeline with CoverAndMerge). The budget is enforced by truncating the in-prompt subscription list before invoking the LLM, isolating the prediction from model-capacity confounds. We report F1, end-to-end latency, and per-event token cost; the crossover point is where A4 first matches or exceeds A0 in F1. This is distinct from the discrimination-capacity crossover |S| cross , which concerns accuracy at large |S| regardless of 𝑊 . Single seed on Qwen-2.5-7B at temperature 0; backends with native 𝑊 (≥ 32 K for local Qwen, ≥ 128 K for cloud APIs) are not context-constrained at |S| ≤ 2,000 and serve as upper-bound references. 4.8
Discrimination-Capacity Panel
The cost model’s discrimination-capacity crossover (Section 3.5) predicts that LLM matching accuracy declines when the subscription cardinality |S| approaches the model’s effective discrimination capacity. To probe how the boundary depends on the LLM, we evaluate four open-weight models on D2 (the dataset with |S| = 201, where the boundary is most consequential) under matched experimental conditions: • Same-family parameter sweep. Qwen 2.5 at three scales — 1.5B, 7B, 32B — with all other factors held constant. This isolates parameter count as the independent variable within a single training generation and tokenizer. • Cross-family control at matched scale. Mistral 7B Instruct at the same parameter count as the middle Qwen point, with the same prompt and decoding settings. Comparing Qwen 7B and Mistral 7B isolates training generation / instruction-tuning sophistication at fixed parameter count. All four models are evaluated on configurations A0 (raw LLM) and A1 (clustering only), which together span the relevant points on the cost-accuracy frontier on D2 (per the Section 5.1 ablation, A0 and A1 are the strongest configurations on D2 under the existing Qwen 7B baseline). Five seeds each, MAX_EVENTS=300 stratified subsample (matching the Qwen 7B D2 cap); 40 runs total. Where useful, the existing Haiku full-corpus and Sonnet 5,000-event subsample numbers are reported as upper-bound references against the open-weight panel. 4.9 QoE Heterogeneous Backend Assignment To explore the QoE-based backend assignment (Eq. (1)), we evaluate three assignment strategies on D1 with a Qwen-2.5 tier gradient (7B, 32B): homogeneous (all clusters to a single backend, the ablation default), round-robin (clusters cyclically assigned), and QoE-optimised (each cluster to the backend maximising QoE(𝑐, ℓ) after the calibration phase of Section 3). Calibration uses 10% of events per cluster per backend; the remaining 90% are used for evaluation. Three normalised weight configurations (sum to one) explore the accuracy-cost Pareto: accuracy-first (0.70, 0.15, 0.15), balanced (0.34, 0.33, 0.33), and cost-first (0.15, 0.70, 0.15). We additionally exercise the loop with two perturbations (topic-restricted calibration; latency injection) and a calibration-fraction sweep ∈ {0.05, 0.10, 0.20, 0.50, 0.80, 1.00} under matched-pair semantics, with all cells of a sweep sharing one SLURM job and an on-disk LLM-call cache keyed on (model, prompt) so GPU and decoding nondeterminism cancel in the perturbed-vs-baseline delta. Results in Sections 5.6, 5.6.1 and 5.6.2.
0:18
4.10
Lovén et al.
Metrics
Matching accuracy: precision, recall, F1 score, and false positive rate (FPR), computed perevent and macro-averaged. Let 𝐸 denote the event corpus, S the subscription set, 𝐺 (𝑒) ⊆ S the ground-truth subscription IDs for event 𝑒, and 𝑃 (𝑒) ⊆ S the predicted IDs returned by the matcher. ID-based per-event precision and recall are Precid (𝑒) = |𝑃 (𝑒) ∩𝐺 (𝑒)|/|𝑃 (𝑒)| and Recid (𝑒) = |𝑃 (𝑒) ∩ 𝐺 (𝑒)|/|𝐺 (𝑒)|; macro-averaged F1 is the harmonic mean per event averaged over 𝑒 ∈ 𝐸. A match is correct iff the returned subscription ID belongs to the event’s ground-truth set. Description-aware F1. When the subscription set contains multiple distinct IDs sharing one description (the duplication-with-rename construction used in Section 5.4 to test |S| > |S| native ), ID-based F1 mis-counts a CoverAndMerge correct merging of near-duplicates as missed matches. Let 𝑑 : S → D map each ID to its free-text description, and let 𝑃¯ (𝑒) = {𝑑 (𝑠) : 𝑠 ∈ 𝑃 (𝑒)} and 𝐺¯ (𝑒) = {𝑑 (𝑠) : 𝑠 ∈ 𝐺 (𝑒)} be the description-sets of prediction and ground truth. Description-aware per-event precision and recall are |𝑃¯ (𝑒) ∩ 𝐺¯ (𝑒)| |𝑃¯ (𝑒) ∩ 𝐺¯ (𝑒)| Precdesc (𝑒) = , Recdesc (𝑒) = , |𝑃¯ (𝑒)| |𝐺¯ (𝑒)| with the macro-averaged F1 defined identically to the ID-based variant. Compound merged IDs of the form id𝑖 +id 𝑗 produced by CoverAndMerge are split on + before applying 𝑑, so a merged prediction matches against any of the original descriptions it represents. Reduction-invariant statement: when every ID has a distinct description (i.e., 𝑑 is injective), description-aware F1 reduces to ID-based F1; the two metrics differ only on duplication-with-rename subscription sets. The synthetic-data invariant test tests/test_synthetic_data_metric_invariant.py in the artefact repository asserts this reduction together with the round-trip property "ID-based F1 underestimates description-aware F1 on duplicated sets". System performance: end-to-end matching latency 𝐿 (Eq. (6)), per-event amortised latency 𝐿/𝑚, throughput (events/second), and LLM invocation count 𝐼 . Cost: compression ratio 𝜌, tokens consumed, and monetary cost per 1 000 events. Statistical reporting: all metrics are reported as mean ± 95 % CI over 5 seeds. Pairwise significance is assessed via the Wilcoxon signed-rank test; with 𝑛=5 paired observations the minimum attainable two-sided 𝑝 is 0.0625, so directional consistency across seeds is the primary evidence and the 𝛼=0.05 rejection bar is reported for completeness only. Reproducibility apparatus. Five regression invariants in the artefact repository gate figure rendering against the headline claims: I1–I4 cover the F1, empty-prediction-rate, and cost-model claims; I5 asserts calibration–evaluation event-set disjointness, 𝐸 cal ∩𝐸 eval = ∅ per (cluster, backend), which is load-bearing for the perturbation and calibration-fraction experiments of Sections 5.6.1 and 5.6.2 (the sole exception is the frac=1.00 asymptotic-ceiling point, where disjointness is intentionally not imposed and the result is read as an upper bound). The invariants run as a blocking pytest stage in the DVC pipeline (dvc.yaml stage regression-invariants): a failing invariant fails the pipeline and blocks figure regeneration, so figures cannot drift from claims (Appendix G in the supplementary material). 4.11
Implementation and Environment
The Neural Router is implemented in Python 3.10 with embeddings via sentence-transformers (allMiniLM-L6-v2) and clustering via scikit-learn. LLM inference uses six backends via LiteLLM: Qwen 2.5 1.5B / 7B / 32B and Mistral 7B Instruct run locally via Ollama at the default Q4_K_M 4-bit GGUF quantisation (digests in ollama-digests.txt); Claude 3 Haiku (claude-3-haiku-20240307) and Claude Sonnet (claude-sonnet-4-20250514) run via the Anthropic API EU endpoint. The 7B Qwen variant is the primary open-weight backend across all ablations. Open-weight inference runs
Neural Router: Semantic Content Matching for Agentic AI
0:19
on CSC Puhti (NVIDIA V100 16 GB) for the main ablation table and CSC Mahti (NVIDIA A100 40 GB) for the cost-model validation re-run with per-cluster instrumentation (the additional logging needs the larger card’s memory) and for the 1.5B/32B Qwen and Mistral runs. Cross-hardware F1 was not separately validated; temperature-0 accuracy is FP-deterministic up to rounding noise. Reported 𝑡¯llm includes cloud API network round-trip for Haiku and Sonnet; local backends do not. Cloud model outputs may vary across provider updates, so results are indicative for the current model generation rather than exact reproducibility targets. Source code, experiment scripts, DVC pipeline definitions, regression tests (including the metric-invariant tests cited in Section 4.10), and the per-event prediction parquet files for the Section 5.4 sweep are at https://github.com/lloven/neuralrouter-experiments. 5 5.1
Results Ablation Study and Baseline Comparison
Table 3 reports matching accuracy and system cost on D1 for the seven ablation configurations defined in Section 4.2 alongside the seven model-independent baselines (Section 4.3); D2 and D3 ablation results appear in Suppl. Tab. 1, and the D2 open-weight discrimination-capacity panel in Table 5. Three patterns are visible. (i) A0 and A4 dominate the ablation frontier: A0 (raw LLM) attains F1=0.656±0.001 at 𝐼 =450 invocations, and A4 (full pipeline with CoverAndMerge) reaches 0.638±0.019 at 𝐼 =445, recovering nearly all of A0’s accuracy at the same invocation budget. (ii) Cluster-only variants (A1, A3, A5) trade an order of magnitude more invocations (∼ 4,600−4,700) for F1 ≈ 0.44, a strictly worse cost–accuracy point; A6 (no cosine filter) is dominated on every column. (iii) Every Neural Router configuration except A2 and A6 outperforms the strongest non-LLM baseline (DistilBARTMNLI zero-shot, F1=0.434) by ≥ 0.01 absolute F1, and A0/A4 do so by ≥ 0.20 absolute F1; the four cosine-baseline rows (Sentence-BERT, GloVe, TF-IDF, Word2Vec) span F1 ∈ [0.10, 0.42], confirming the semantic-matching gap that motivates the LLM-driven design. 5.2
Parameter Sensitivity
We sweep four pipeline hyperparameters one at a time on D1 with the Qwen-2.5-7B backend in the A3 (full pipeline) configuration; Suppl. Fig. 1 reports the four sweeps as a single multi-panel figure with panels (a)–(d). Panel (a), the cluster count 𝑘, gives a non-monotone F1 response: 𝑘=1 (degenerate single-cluster) and 𝑘=19 (one cluster per native subscription) both attain higher F1 than the intermediate values 𝑘 ∈ {5, 10, 15} at near-constant invocation cost. We retain 𝑘=19 because it cleanly separates clusters per topic, yielding the lowest false-positive rate. Panel (b), the cosine threshold 𝜏, shows F1 plateauing above 𝜏 ≈ 0.3 as the centroid-fallback path dominates; the effective operating range is 𝜏 ∈ [0.05, 0.2], where lower 𝜏 buys recall at the cost of additional invocations (at 𝜏=0, every event goes to every cluster, 𝐼 =301 vs. 𝐼 =31 at 𝜏=0.3). Panel (c) shows that 𝜅=3 is at the F1 plateau (matching 𝜅=5 within rounding); we adopt 𝜅=3 as the most parsimonious choice. Panel (d), the embedding-model response, spans F1 ∈ [0.263, 0.300] across four mainstream sentence-transformer encoders; the larger e5-large-v2 and BGE-base attain the highest F1 but at ∼ 10× the invocation count (their cosine-similarity distributions concentrate the assignment fallback through more clusters); we retain MiniLM-L6 as the lightweight default because it sits within 0.04 F1 of the best at ∼ 1/12 the cost. 5.3
Cost-Model and Scaling Validation
Figure 3 validates the cost model along three views. Panel a compares predicted to measured percluster invocations on D1 (one marker per (𝑐𝑜𝑛𝑓 𝑖𝑔, 𝑘) cell, with 𝐼 pred =⌈𝑚𝑐 /𝑏 max (𝑐)⌉): the median
0:20
Lovén et al.
Table 3. Ablation and baseline results on D1 (CardiffNLP, full 6,000-event corpus). Backend: Claude Haiku; F1 reported as mean±half-CI95 over 5 seeds, 𝑘=19, 𝜅=3. Best Neural Router F1 in bold. 𝐼 = LLM invocations, 𝜌 = compression ratio (n/a for baselines and uncompressed configs reported as 1.00), 𝐿 = end-to-end latency. Cost column from Section 3.5 worked example at Haiku rates. Results for D2 and D3 in Suppl. Tab. 1; the D2 open-weight discrimination-capacity panel appears in Table 5. Precision
Recall
F1
FPR
𝑰
𝝆
𝑳 (s)
$/1k evt
Neural Router ablation (Claude Haiku, D1) A0: Raw LLM 0.712 A1: Cluster only 0.552 A2: C&M only 0.308 A3: Clust. + C&M 0.551 A4: + Reunite 0.683 A5: + Evt. clust. 0.547 A6: No cosine filter 0.179
0.674 0.398 0.523 0.397 0.667 0.395 0.312
0.656±0.001 0.443±0.000 0.361±0.080 0.442±0.002 0.638±0.019 0.439±0.004 0.216±0.038
0.030 0.018 0.083 0.018 0.033 0.018 0.131
450 4,679 382 4,672 445 4,637 7,751
1.00 1.00 1.24 1.00 1.00 1.00 1.00
3.1 5.4 3.5 4.7 3.2 5.2 16.2
0.023 0.18 0.020 0.18 0.023 0.18 0.31
0.114 0.656 0.491 0.684 0.153 0.147 0.161
0.082 0.423 0.323 0.434 0.104 0.102 0.110
0.161 0.115 0.128 0.114 0.158 0.158 0.157
0 0 0 0 0 0 0
n/a n/a n/a n/a n/a n/a n/a
1.8 21.5 433.1 10,323 0.1 0.2 0.1
0 0 0 0 0 0 0
Configuration
Baselines (model-independent; D1) BM25 Sentence-BERT cosine Cross-encoder DistilBART-MNLI zero-shot GloVe cosine TF-IDF cosine Word2Vec cosine
0.070 0.334 0.258 0.340 0.084 0.084 0.090
predicted/measured ratio is 1.00 and 83% of the 𝑛=81 cells fall within the factor-of-two band. Stratified miss analysis (per analysis/cost_validation_stratified.py): all band-violations fall in the 𝑛=33 non-trivial cells (𝑚𝑐 >𝑏 max , where the ceiling is genuinely under test: 14 misses, 42%, median ratio 2.00), while the 𝑛=48 trivial cells (𝑚𝑐 ≤𝑏 max ) are exact and the model never underpredicts; the 83%-in-band headline thus aggregates an exact trivial stratum with a weaker 58%-inband non-trivial fit (Section 6.3). Panel b plots accuracy versus cost across all (𝑐𝑜𝑛𝑓 𝑖𝑔, 𝑏𝑎𝑐𝑘𝑒𝑛𝑑) cells on D1: each backend’s A0 and A4 attain the best F1 at the lowest invocation count; A6 (no cosine filter) is the most expensive configuration; A1/A3/A5 (multi-cluster routing) sit at moderateto-high cost; embedding baselines occupy the no-LLM band at the left edge. Panel c sweeps event count |E | ∈ {50, ..., 2,000} on D3 in configuration A3: F1 declines from 0.25 to below 0.02 as the per-event share of the 4,096-token budget shrinks (consistent with the discrimination-capacity argument of Section 3.5), while 𝐼 grows linearly (𝑅 2 ≥ 0.99); the dotted right-axis points project 𝐼 to |E | ∈ {5,000, ..., 100,000} from the linear fit. Latency breakdown. The latency decomposition reveals that LLM inference dominates endto-end matching time, accounting for >95 % of total latency across all configurations. Embedding (<0.5 s for 11K events), clustering (<0.1 s), and cosine pre-filtering (<0.01 s) are negligible. The practical implication is that latency optimisation should focus on LLM selection and batching (via the QoE-based cost model of Section 3.5), not on the embedding or clustering pipeline. 5.4
Crossover Validation
The cost model (Section 3.5) predicts that under a constrained context window, the compression pipeline (A4) preserves accuracy where raw prompting (A0) must drop subscriptions. We test this prediction on D1 with Qwen-2.5-7B at 𝑊 =4,096 tokens, sweeping |S| ∈ {50, 200, 2,000} via subsampling and duplication-with-rename, single seed (Fig. 4b). Because duplication-withrename produces multiple subscription IDs sharing one description, an ID-based F1 mis-counts
100
trivial (n=48) non-trivial (n=33)
100
101
Haiku Qwen-2.5-7B
0.5 0.0
101
Sonnet (1 seed) Baselines (I=0)
A4A0 A0 A4 A0 A4 A2 A5 A1 A2 A3 A2
103
A6 A1 A3 A3 A1A6 A5 A5 A6
F1 (empirical) I (empirical)
I (analytical)
104
0.2 0.0
LLM invocations I
101
0:21
Macro-F1
median ratio 1.00, 0 under-predictions non-trivial (n=33): median ratio 2.0
Macro-F1 (D1)
Predicted Ipred / cluster
Neural Router: Semantic Content Matching for Agentic AI
102 103
105
1) of events | | Measured Imeas / clusterLLM invocations I per run (log; baselines at I =Number
(a) Cost-model validation: pre- (b) Accuracy–cost trade-off on D1 (c) Event-count scaling on D3 with dicted vs. measured per-cluster in- across three backends and seven Qwen-2.5-7B in configuration A3. vocations on D1. configurations. Fig. 3. Cost-model and scaling validation. (a) Predicted vs. measured per-cluster LLM invocations across configs {A0, A1, A3} and 𝑘 ∈ {1, 2, 5, 10, 19}, 𝑛=81 markers (Qwen-2.5-7B on Mahti GPU and dry-run client). Predicted 𝐼 pred from Eq. (5) with production constants. The model is a conservative ceiling (median ratio 1.00, zero under-predictions): the grey wedge marks the one-sided within-2× region, trivial cells (𝑚𝑐 ≤𝑏 max , 𝑛=48) lie exactly on the line, and non-trivial cells (𝑛=33, median ratio 2.00) over-predict, 58% within 2×; 83% in-band overall. Stratified detail in body text. Both axes log. (b) F1 vs. invocation count 𝐼 (log) on D1; embedding baselines plotted at 𝐼 =1. Within each backend, A0 and A4 attain the best F1–𝐼 point and A6 the highest 𝐼 . (c) F1 (left) and 𝐼 (right, log) for |E | ∈ {50, ..., 2,000} at fixed ∼5–7 events/call; F1 declines as the per-event budget shrinks (Section 3.5). Dotted points project 𝐼 to |E | ∈ {5,000, ..., 100,000} from the linear fit.
CoverAndMerge’s correct merging of near-duplicates as missed matches. We therefore evaluate this experiment under the description-aware F1 defined formally in Section 4.10, which collapses both predictions and ground truth to description sets before computing precision and recall, so semantically equivalent merges count as hits.1 For A0, F1 plateaus at 0.43 once |S| ≥ 200: the𝑊 =4,096 truncation caps the in-prompt subscription count at ≈ 162, and matching against any 162-subscription subset of D1 yields the same accuracy. For A4, F1 decreases with |S| (0.367 → 0.067 → 0.038) where the cost model predicts increase. The decline is not a metric artifact: A4’s CoverAndMerge correctly compresses the duplicated set (𝜌 = 0.6 at |S| = 50, 𝜌 = 0.42 at |S| = 2,000), and the merged set’s effective size |S| eff stays in the 125–133 range across the high-|S| cells, comparable to A0’s |S| eff =162. Per-event analysis (Fig. 4a) reveals the failure mode: as |S| grows under A4, the LLM (i) emits an empty match for a growing fraction of events (refusal rate 19% → 31% → 54%) and (ii) draws its non-empty predictions from a shrinking subset of the active subscription set (167% → 66% → 34% of |S| eff , where values above 100% indicate the LLM also hallucinates IDs not in the set). Merged subscriptions with compound IDs are picked 38 times at |S|=50 and zero times at |S|=2,000. A0’s per-event statistics stay flat across |S| (≈ 20% refusal, ≈ 160% vocabulary used). The mechanism is therefore consistent with empty-prediction collapse and vocabulary narrowing (we use “refusal” descriptively for the empty structured-output predictions, distinct from the safety-tuning literature’s use for explicit task decline): faced with a subscription set that mixes atomic and compound merged subs at high cardinality, Qwen-2.5-7B narrows its prediction vocabulary and increasingly returns empty matches. Cross-seed reproducibility is not formally tested at this single seed; the per-event signature is consistent with the empty-prediction-and-vocabulary-narrowing pattern just described. The cost model’s worked example assumed CoverAndMerge preserves recall (Section 3.5); on subscription 1 An earlier version of this experiment used ID-based F1 and produced an apparent collapse that was traced to this metric-
invariant violation; the description-aware run reported here is the corrected measurement.
50
A0 A4
40 30 20
102
|S|
103
250
(b) Vocabulary coverage
Macro-F1
(a) Refusal rate
200 150 100
0.4
0.04
0.3
0.02
0.1
50 102
|S|
0.00
0.2 A0 F1 A4 F1 |S|·t_s = W (42)
102
103
A0 cost ($/1k evt) A4 cost ($/1k evt)
103
Subscription volume |S|
(a) Per-event mechanism: empty-prediction rate (left) and distinct-prediction vocabulary as a fraction of |S| eff (right) for A0 and A4 across |S|.
Per-event token cost ($/1k events)
Lovén et al.
Distinct predicted IDs / |S|_eff (%)
Empty-prediction rate (%)
0:22
0.02 0.04
(b) Aggregate F1 (left) and per-event token cost (right) across the same |S| sweep.
Fig. 4. Empirical crossover validation on D1 with Qwen-2.5-7B at 𝑊 =4,096, description-aware F1, single seed (subsampling and duplication-with-rename, evaluated under description-aware F1 to neutralise the duplication-with-rename metric-invariant). Panel (a) shows the per-event signature explaining panel (b)’s aggregate gap as empty-prediction collapse and vocabulary narrowing rather than mid-prediction compounddescription disambiguation; the vertical dotted line in (b) marks the analytical threshold |S|·𝑡𝑠 = 𝑊 . Numerical detail in the body text.
distributions whose redundancy yields a CoverAndMerge representative set near or beyond the LLM’s effective discrimination capacity, that assumption is empirically false. The mechanism is consistent with the discrimination-capacity finding (Section 6.1): an LLM faced with subscription descriptions that each combine several distinct concepts must perform a finergrained discrimination per matching decision than against unmerged subscriptions of the same total cardinality. The cost model accurately predicts invocation count per cluster (Fig. 3a) but not accuracy when CoverAndMerge produces compound descriptions. A refined model would condition the compression payoff on the redundancy structure of the subscription set: high-redundancy sets (where merging produces semantically clean representatives) should benefit; low-redundancy sets and adversarial duplication-with-rename distributions (where merging concatenates unrelated concepts) should fall back to A0. Identifying the redundancy threshold at which A4 ceases to dominate A0 is left to follow-on work. 5.5
Cross-Dataset Comparison and Discrimination-Capacity Panel
Table 4. Best Neural Router configuration per backend vs. the two strongest baselines: macro-F1 across the three datasets, best config in parentheses. † Sonnet single-seed, indicative. Full metrics (FPR, latency, all seven baselines) in Suppl. Tab. 1. D1 = CardiffNLP, D2 = EUR-Lex, D3 = CASAS. Method
D1
D2
D3
NR / Claude Haiku NR / Qwen-2.5-7B NR / Claude Sonnet†
0.656 (A0) 0.505 (A0) 0.717 (A0)
0.089 (A1) 0.045 (A1) 0.316 (A0)
0.401 (A4) 0.320 (A6) –
Sentence-BERT cosine TF-IDF cosine
0.423 0.102
0.154 0.162
0.225 0.268
Table 4 reports the best Neural Router configuration per backend and the two strongest baselines across the three datasets (full metrics, FPR, latency, and all seven baselines in Suppl. Tab. 1). On D1
Neural Router: Semantic Content Matching for Agentic AI
0:23
Table 5. D2 discrimination-capacity panel (|S|=201, 5 seeds; A0 = raw LLM, A1 = clustering only), F1 as mean±half-CI95 (A0 variance zero: temperature-0, seed-invariant). † Sonnet single-seed on a 5K subsample (indicative). Row blocks (top to bottom): Qwen-2.5 same-family sweep (1.5/7/32B), Mistral-7B at matched 7B scale, pairwise embedding baselines, closed-API frontier. Backend Qwen 2.5 1.5B Qwen 2.5 7B Qwen 2.5 32B Mistral 7B Instruct Sentence-BERT cosine TF-IDF cosine Claude Sonnet†
Params
A0 F1
A1 F1
1.5 B 7B 32 B 7B
0.000±0.000 0.002±0.000 0.127±0.000 0.039±0.000
0.001±0.001 0.045±0.016 0.116±0.022 0.080±0.015
– – –
0.154 0.162 0.316
0.154 0.162 –
(|S|=19, low semantic gap) and D3 (|S|=19, very high semantic gap requiring abductive reasoning), the best LLM configuration outperforms all baselines (Table 4). On D2 (|S|=201, moderate semantic gap), the picture is more nuanced: small open-weight models collapse below pairwise baselines, while larger or newer-generation models partially escape (Table 5). The four-model open-weight panel on D2 quantifies this gradient. The panel (Table 5) exhibits two structured patterns. First, same-family parameter scaling: within Qwen 2.5, raw-LLM (A0) F1 rises monotonically across 1.5B→7B→32B (a >60× jump from 7B to 32B). Second, at matched parameter scale, the more recently trained Mistral 7B reaches an order of magnitude above Qwen 2.5 7B. The pairwise embedding baselines sit above all four open-weight points (Qwen 32B is closest but does not surpass them); only the closed-API Sonnet reference (†, single-seed) is clearly above the pairwise band. All values in Table 5. These results refine the discrimination-capacity boundary identified in Section 3.5: |S| cross is not a global scalar but a model-dependent threshold determined by the joint of parameter count and training generation. On D2 specifically, |S| = 201 is well above |S| cross for Qwen 1.5B and 7B (deep collapse, F1 essentially zero); is partially binding for Mistral 7B and Qwen 32B (substantially lifted F1 relative to the smaller Qwen models, but still below the model-independent pairwise baselines); and is comfortably below |S| cross only for the closed-API frontier (Sonnet, the only point clearly above pairwise baselines in our panel). The capability-dependent gradient is consistent with the attention-budget mechanism but requires a second multiplicative factor — instruction-tuning sophistication / training generation — to account for the Mistral-vs-Qwen-7B gap at fixed parameter count. Statistical significance. A0 macro-F1 is a deterministic function of the matching prompt under temperature-0 decoding and is therefore identical across all five 𝑘-means seeds for every panel cell, so the variance reported in Table 5 for A0 reflects subscription-set partitioning only and collapses to zero for unclustered runs. The Wilcoxon signed-rank test on the three headline pairs at A0 (Mistral 7B vs. Qwen 7B, Qwen 32B vs. Qwen 7B, Qwen 1.5B vs. Qwen 7B) returns the minimum-power 𝑝=0.0625 at 𝑛=5 paired observations: the inter-seed differences are all of identical sign at every cell, so the test cannot distinguish them from any reproducible directional effect. The two headline gaps (Mistral-over-Qwen at matched 7B scale, Qwen-32B-over-7B same-family) therefore rest on consistent directionality across all five seeds rather than on the formal 𝛼=0.05 rejection. The A1 cells (where clustering randomness produces seed-to-seed variation) carry tighter Wilcoxon power once larger 𝑛 is collected; this is on the deferred-experiments list.
0:24
5.6
Lovén et al.
QoE Heterogeneous Backend Assignment
Suppl. Tab. 2 compares the three backend assignment strategies defined in Section 4.9 on CardiffNLP with two open-weight Qwen-2.5 tiers (7B and 32B). All four observations are clear from the table: (i) The bigger tier dominates on F1. Homogeneous Qwen-2.5-32B reaches F1=0.225 ± 0.019, ahead of Qwen-2.5-7B at F1=0.166 ± 0.023, at the cost of ∼ 2.5× longer wall-clock latency (359 s vs. 141 s on a 1,000-event subsample). (ii) Round-robin almost matches the bigger tier on F1 at 0.7× its latency. Round-robin (alternating clusters between Qwen-7B and Qwen-32B) yields F1=0.215 ± 0.006 at 𝐿=250 s, closing 83% of the 0.166→0.225 F1 gap while roughly halving the smaller-to-larger latency penalty. (iii) QoE-optimised differentiates assignments by weight preset, but does not outperform roundrobin in this campaign. The three weight presets produce three distinct per-cluster assignments (verified seed-by-seed; the bug-fixed implementation uses per-cluster min-max normalisation across candidate backends, so weights flip the argmax whenever F1 and cost or latency disagree on a cluster), but on D1 with a 10% calibration sample, the resulting F1 (0.179–0.192) sits below both round-robin and the larger homogeneous tier. The cost-first preset attains the highest F1 among QoE variants (0.192 ± 0.039) because it routes more clusters to the lower-token-count backend (Qwen-32B emits shorter responses than Qwen-7B in our prompt format), which happens to be both the cheaper and the more accurate option in this campaign. (iv) The bottleneck is calibration noise — a hypothesis the sweep below tests and bounds. Per-cluster F1 estimated from 10% of events (∼ 67 events per cluster × 2 backends) carries enough variance that the greedy argmax frequently selects the worse backend at the cluster level; round-robin avoids this by skipping the calibration step entirely. The natural conjecture — that QoE-optimised overtakes round-robin once a larger calibration sample drives per-cluster F1 standard error below the inter-backend gap — is tested directly by the calibration-fraction sweep of Section 5.6.2: across frac ∈ {0.05, . . . , 1.00} no fraction crosses 𝛼=0.05 (Friedman 𝑝=0.290), so the strong calibration-noise-limited claim is not empirically supported on this 7B/32B gradient. Where the QoE framework does pay off is the cost-side localisation established under perturbation (Section 5.6.1): separation from round-robin at the cost_first preset only. On this gradient round-robin is a strong calibration-free baseline. 5.6.1 Perturbation: workload shift and latency injection. We exercise the QoE loop on D1 with two perturbations under the matched-pair regime defined in Section 4.9 (𝑛=15 seeds, calibration fraction 0.10; Fig. 5). Topic-restricted calibration draws the calibration sample from three of nineteen CardiffNLP topics, simulating calibration-vs-deployment distribution shift; latency injection adds a 0.05 s delay to every LLM call for events with index ≥ 500. Matched-cell F1 deltas are exactly zero for the calibration-free strategies (Δ𝐹 1=0.000 for both homogeneous and round-robin, as theoretically required), and QoE-optimised moves in the predicted direction with Δ𝐹 1=−0.018 (95 % CI [−0.031, −0.005], paired Wilcoxon 𝑝=0.008), consistent with the calibration-quality hypothesis. For latency injection, the matched-cell latency delta on the mixed-backend strategies is Δ𝐿=+25.08 s (95 % CI [25.06, 25.10], 𝑛=14; one seed dropped due to a missing baseline-vs-injection cell, paired Wilcoxon 𝑝<10−3 ), exactly matching the expected 0.05 s × 500 post-injection events; matchedcell Δ𝐹 1 on the same comparison is 0.000 ([−0.017, +0.018]), confirming the perturbation cleanly isolates the latency dimension. Together the two perturbations close the “exercise-the-mechanism” gap. Stratified by weight preset, QoE-optimised separates from round-robin only at cost_first (F1 weight 0.15): paired Δ𝐹 1(QoE−RR)=+0.021 on baseline (one-sided Wilcoxon 𝑝=0.018, 11/4 positive seeds, 𝑛=15) and +0.019 under latency injection (𝑝=0.034, 𝑛=14); broken by topic-restricted calibration (𝑝=0.28). At accuracy_first and balanced, QoE does not separate from round-robin — the 7B/32B per-cluster F1 gap is too small for selective routing to outweigh calibration noise. Replacing the 32B tier with Qwen-2.5-72B (same 𝑛=15, frac=0.10) widens the homog F1 gap from 0.029 to 0.042; the 𝛼=0.05 separation does not generalise (accuracy_first leads with 11/4 positives,
Neural Router: Semantic Content Matching for Agentic AI
0:25
(b) Latency-injection L
0.18 0.16 homogeneous round_robin qoe_optimised
0.14 baseline
topic-restr.
L (s, matched-cell)
Macro-F1 (D1)
(a) F1 by perturbation
latency-inj.
25 20 15 10 5 0
qoe optimised
round robin
Fig. 5. Perturbation results on D1, Qwen-2.5 7B/32B tier gradient, calibration fraction 0.10, 𝑛=15 seeds. Left: per-(perturbation, strategy) macro-F1 with 95 % CIs. Right: matched-cell latency delta (latency-injection − baseline) on mixed-backend strategies recovers the 0.05 s × 500 event injection as +25 s. Calibration-free strategies show zero matched-cell F1 delta on topic-restricted by construction.
𝑝=0.126), but QoE-cost-first matches homog-72B within 0.012 F1 at Δ𝐿=−738 s, preserving the tunable tradeoff (full perturbation companion in Suppl. §F). 5.6.2 Calibration-fraction sweep. We test the calibration-noise-limited claim by sweeping the calibration sample fraction ∈ {0.05, 0.10, 0.20, 0.50, 0.80, 1.00} on D1 with the Qwen-2.5 7B / 32B tier gradient (Suppl. Fig. 2; matched-pair regime, 𝑛=15 seeds, balanced preset). Calibration-free strategies are flat and QoE-optimised’s mean trajectory is non-monotone (F1 0.150–0.165); the paired onesided Wilcoxon (QoE > round-robin) crosses no fraction (six-fraction Friedman 𝜒 2 =6.17, 𝑝=0.290; per-fraction 𝑝-values in Suppl. Fig. 2). No fraction crosses 𝛼=0.05: the calibration-noise-limited claim, in its strong form, is empirically unsupported on this tier gradient. The cost-side separations from Section 5.6.1 (QoE > round-robin only at cost_first) localise where the framework pays off. The QoE framework therefore contributes a tunable, weight-driven per-cluster routing mechanism that correctly differentiates assignments by operator preference; round-robin is a strong calibration-free baseline competitive with the larger homogeneous tier at 0.7× its latency. Across the perturbation (Section 5.6.1) and calibration-fraction (Section 5.6.2) experiments, QoE-overround-robin separation is localised at the cost_first preset and no calfrac crosses 𝛼=0.05; the calibration-sample-to-discrimination-gap ratio is supported as the operative knob in its weak (sign-of-effect, cost-side-localised) form on this tier gradient. 6
Discussion
The ablation confirms the cost model’s crossover prediction: at the evaluated volumes (|S|≤201) the simplest configuration A0 is the best Neural Router configuration (shifting to A1 on D2 for the open-weight panel/Haiku and A4 on D3 for Haiku; Sections 5.1 and 5.5). A0’s dominance is predicted, not a pipeline failure: above the context-window crossover compression can only remove information, and modern cloud LLMs (𝑊 ≥128K) sit above it for |S|≤201; the pipeline’s value is the sub-crossover regime (edge models, long |S|), where the empirical 𝑊 =4,096 sweep (Section 5.4) refines the picture — at high |S| on duplication-with-rename sets A4 falls below A0 via emptyprediction collapse, so the recall-preservation assumption is conditional on the LLM engaging with the merged set. Two readings follow. Backend choice dominates configuration choice where |S| binds: on the high-|S| D2 panel the cross-LLM F1 gap exceeds the within-LLM spread across non-dominated configurations (Section 5.5); there, select the LLM first and tune the pipeline second. Dataset characteristics modulate the LLM advantage beyond what the cost model predicts: D3 (|S|=19) succeeds across all backends while D2 (|S|=201) degrades model-dependently, so cardinality alone is not the whole story — the binding factor is |S| relative to each model’s discrimination capacity, which we analyse next.
0:26
6.1
Lovén et al.
Discrimination Capacity
The two-crossover framework of Section 3.5 (𝑊cross , |S| cross ) admits empirical localisation only for |S| cross : 𝑊cross is computed from the cost model, while |S| cross is model-dependent and must be characterised by sweep. The D2 panel (Section 5.5, |S|=201, four open-weight models) localises it along two axes — same-family parameter scaling (Qwen 2.5 1.5B→7B→32B) and training generation at matched scale (Mistral 7B vs. Qwen 2.5 7B), per-model F1 in Section 5.5 — with only the closedAPI frontier crossing the pairwise-baseline band. D3 has the highest semantic gap yet succeeds at |S|=19 across all backends, so semantic difficulty alone does not explain the D1→D2 gap: the binding factor is |S| relative to the model’s discrimination capacity. The underlying mechanism — a shared attention budget O (1/|S|) scaled by parameter count, compounded by instruction-tuning generation (so Mistral ≫ Qwen-7B at matched scale), which pairwise methods avoid because (𝑒, 𝑠) comparisons are independent — and the resulting deployment implications (model selection upstream of pipeline tuning; explicit 𝐷-management via subscription partitioning or embedding-prefilter cascades where no model fits below the boundary) are detailed in the electronic supplement. A quantitative scaling law for |S| cross is left to follow-on work. 6.2
The Accuracy–Latency Trade-off
LLM-based matching is more expensive per invocation than syntactic or embedding-distance filters; the Neural Router accepts this for semantic understanding, with the mitigation derived formally in Section 3.5 (Eq. (6)): cover/merge compression reduces invocations below the context-window crossover, batching amortises per-invocation overhead across 𝑏 max events, and parallelism across 𝑃 instances converts the chain into 𝑅=⌈𝐼 /𝑃⌉ rounds. The system targets regimes where per-batch latency on the order of seconds is acceptable (analytics dashboards, notification feeds, content recommendation, regulatory-alert dissemination, network-telemetry routing); rapid LLM-inference improvements [Kwon et al. 2023; Zheng et al. 2024] are narrowing the gap. 6.3
Limitations
Experimental scope. The matching engine is evaluated as offline top-𝜅 multi-label retrieval on static labelled corpora, single-broker, single-tenant; events do not arrive as a stream, subscriptions do not churn, and there is no broker-level concurrency or backpressure. The seven baselines target the matching task itself; we do not include a traditional CBPS broker (Siena, JEDI, OpenPubSub) in the panel because their structured-filter languages are not designed to match the modalitygap setting D3 stresses. All inference runs on CSC’s Mahti and Puhti A100/V100 partitions; the “computing-continuum” framing describes the design’s intended deployment span, not the experimental hardware. Generalisation to online broker integration, churning subscription sets, and heterogeneous edge silicon is a deployment-study agenda left to follow-on work. Inference latency and cost. Each LLM invocation adds latency and per-token cost; Section 6.2 and the token-count cost model (Eq. (8)) quantify the compression/batching/parallelism mitigation and let practitioners apply provider-specific pricing across the evaluated spectrum (zero-API-cost local Qwen/Mistral through Haiku/Sonnet). Cost-model fit on the non-trivial stratum. The 83% within-band headline (Section 5.3) aggregates a strong fit on trivial cells (𝑚𝑐 ≤𝑏 max ; 48/81, zero violations, model collapses to identity) with a weaker fit on the 33 non-trivial cells where the ceiling ⌈𝑚𝑐 /𝑏 max ⌉ is genuinely under test (14/33=42% violations). A variance-aware revision (predicting the per-cluster ratio variance) is the natural extension; we report the stratified breakdown for full disclosure. Modality, language, and assignment scope. The three datasets span social media, legal, and IoT smart home, but all use English text (sensor events are templated to text). Native multi-modal
Neural Router: Semantic Content Matching for Agentic AI
0:27
matching, non-English, and highly technical domains (medical, code) remain to be validated. The QoE comparison (Suppl. Tab. 2) is single-tenant, batch-evaluation, no-failure, so round-robin’s load-balancing weaknesses (skew, heterogeneous throughput, failure, SLAs) are not exercised; the per-cluster min-max normalisation (Eq. 1) is candidate-set dependent. The QoE mechanism assumes per-cluster accuracy, cost, and latency are measured by the operator from held-out calibration data, not self-reported by competing backend providers; the strategic regime in which a provider misreports to influence assignment is explicitly out of scope here and is a credibility-mechanism question (auditable logs, commitment devices) deferred to the service-economy treatment of [Lovén et al. 2026]. Benchmark contamination. D1 (CardiffNLP, 2022), D2 (EUR-Lex MultiEURLEX, 2021), and D3 (CASAS hh113, pre-2024) are all public datasets with release dates predating the training cut-offs of the LLMs evaluated here. Some fraction of the headline F1 numbers may therefore measure surface-level memorisation rather than zero-shot semantic matching. We do not run a closed-book probe in the current campaign; numbers in Table 3, Table 5, and the cross-dataset table in the supplementary material should accordingly be read as upper bounds for genuinely novel deployments. A held-out post-cut-off slice (e.g., post-2024 EU legislation with EUROVOC tags, or a synthetic CASAS template set on a held-out sensor schema) would tighten the external-validity argument and is on the deferred-experiments list. 6.4
Broader Implications and Future Work
The Neural Router shows LLMs can serve as practical matching engines for content-based pub/sub, not only as downstream consumers. As autonomous agents compose, negotiate, and consume services across device-edge-cloud environments [Saleh et al. 2025a], they cannot control the vocabulary or format of those services; the matching engine bridges vocabulary gaps via LLMdriven reasoning rather than pre-engineered taxonomies, with D3 instantiating the modalitygap regime where embedding baselines fail by construction. The most consequential extensions cluster around three themes: (i) a quantitative scaling law for |S| cross in parameters × training generation, validated across cross-family open-weight points (Gemma, Phi, future Llama), with discrimination-aware partitioning (|S ′ | ≤ 𝐷 per prompt) and embedding-prefilter cascades for high|S| regimes; (ii) online operation — incremental subscriptions, multi-modal extensions, distributed federation across administrative domains, and operational robustness against silent-snapshot rotation; (iii) convergence with service-economy mechanisms when the engine is offered as a priced service [Lovén et al. 2026] — formal capacity allocation, credibility mechanisms (auditable logs, commitment devices), and quality-of-experience menus that elicit private valuations. Together these extend the Neural Router from a single-broker prototype toward a deployable component of decentralised agentic service ecosystems. 7 Conclusion We introduced the Neural Router, a content-based publish/subscribe matching engine integrating LLMs into the matching loop, characterised by a two-crossover cost-accuracy model (contextwindow and discrimination-capacity), three composable algorithms (OptimizeSubscriptions, MatchEvents, CoverAndMerge), and a QoE-based heterogeneous backend assignment mechanism. Evaluation across three datasets (CardiffNLP, EUR-Lex, CASAS), six LLM backends (Qwen 2.5 1.5B/7B/32B, Mistral 7B, Haiku, Sonnet), and seven ablations against seven baselines confirms the cost model: raw LLM matching (A0) dominates above the context-window crossover (D1 Haiku F1 = 0.66); below it, a 𝑊 =4K sweep shows compression collapsing 0.37→0.04 as |S| grows 50→2,000 via empty-prediction collapse and vocabulary narrowing. The invocation-count prediction holds per-cluster on 𝑛=81 cells (median 1.00, 83% in band; misses concentrate in 33 non-trivial cells). The
0:28
Lovén et al.
D2 four-model open-weight panel reveals a structured discrimination-capacity escape consistent with a joint parameter-count × training-generation factor: Qwen 1.5B/7B collapse, Mistral 7B and Qwen 32B partially escape, only the closed-API frontier crosses the pairwise embedding band. The two-crossover model gives practitioners a principled cost-side tool plus an empirically characterised viability boundary; deployment validation, multi-modal extensions, distributed federation, market-mechanism convergence, and a quantitative |S| cross scaling law are the natural follow-ons. Acknowledgments This work was supported by Business Finland through the Neural Pub/Sub project (Diary No. 8754/31/2022), by the Research Council of Finland (Grant No. 362594), by the 6G Flagship program (Grant No. 369116), and by the Digital Twinning of Personal Area Networks for Optimized Sensing and Communication project (Diary No. 8782/31/2022). Data and Software Availability The datasets used in this study (CardiffNLP Tweet Topic, EUR-Lex MultiEURLEX, CASAS hh113) are publicly available from their respective sources cited in Section 4.1. The Neural Router source code, DVC pipeline definitions, experiment configurations, regression tests, and per-event prediction parquet files are available at https://github.com/lloven/neural-router-experiments. The repository is under access embargo during peer review; reviewers may obtain read access via the Guest Editors on request. Upon acceptance the repository will be made public and a Zenodo DOI minted from the frozen snapshot recorded in the artefact’s paper.tag file, replacing this URL at camera-ready. References Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Vítor Silva, Leonardo Neves, and Francesco Barbieri. 2022. Twitter Topic Classification. In Proceedings of the 29th International Conference on Computational Linguistics. 3386–3400. Danilo Ardagna and Barbara Pernici. 2007. Adaptive Service Composition in Flexible Processes. IEEE Transactions on Software Engineering 33, 6 (2007), 369–384. doi:10.1109/TSE.2007.1011 Pete Beckman, Jack Dongarra, Nicola Ferrier, Geoffrey Fox, Terry Moore, Dan Reed, and Micah Beck. 2020. Harnessing the Computing Continuum for Programming Our World. John Wiley & Sons, Ltd, Chapter 7, 215–230. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119551713.ch7 doi:10.1002/9781119551713.ch7 Sukanya Bhowmik, Muhammad Adnan Tariq, Jonas Grunert, Deepak Srinivasan, and Kurt Rothermel. 2018. Expressive content-based routing in software-defined networks. IEEE Transactions on Parallel and Distributed Systems 29, 11 (2018), 2460–2477. James Briggs and Aurelio AI. 2024. Semantic Router: Superfast AI Decision Making and Intelligent Processing of Multi-Modal Data. https://github.com/aurelio-labs/semantic-router. Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink: Stream and Batch Processing in a Single Engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015), 28–38. Antonio Carzaniga, David S. Rosenblum, and Alexander L. Wolf. 2001. Design and Evaluation of a Wide-Area Event Notification Service. ACM Transactions on Computer Systems 19, 3 (Aug. 2001), 332–383. Antonio Carzaniga and Alexander L. Wolf. 2003. Forwarding in a Content-Based Network. In Proceedings of ACM SIGCOMM 2003. Karlsruhe, Germany, 163–174. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX – A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP). 6974–6996. Diane J. Cook, Aaron S. Crandall, Brian L. Thomas, and Narayanan C. Krishnan. 2013. CASAS: A smart home in a box. IEEE Computer 46, 7 (2013), 62–69. doi:10.1109/MC.2012.328 Nathanaël Denis, Pierre Chaffardon, Denis Conan, Maryline Laurent, Sophie Chabridon, and Jean Leneutre. 2020. Privacypreserving content-based publish/subscribe with encrypted matching and data splitting. In SECRYPT 2020: 17th International Conference on Security and Cryptography, Vol. 3. ScitePress, 405–414. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June
Neural Router: Semantic Content Matching for Agentic AI
0:29
2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186. doi:10.18653/V1/N19-1423 Praveen Kumar Donta, Ilir Murturi, Víctor Casamayor Pujol, Boris Sedlak, and Schahram Dustdar. 2023. Exploring the Potential of Distributed Computing Continuum Systems. Computers 12, 10 (2023), 198. doi:10.3390/computers12100198 Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Kermarrec. 2003. The many faces of publish/subscribe. ACM Comput. Surv. 35, 2 (jun 2003), 114–131. doi:10.1145/857076.857078 Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. RouterBench: A Benchmark for Multi-LLM Routing System. In Agentic Markets Workshop at ICML. arXiv:2403.12031. Vatche Isahagian, Vinod Muthusamy, and Aleksander Slominski. 2023. Publish-subscribe with large language models: Improving expressiveness with natural language and generated custom notifications. In Proceedings of the 24th International Middleware Conference: Demos, Posters and Doctoral Symposium. 29–30. Iterative, Inc. 2020. Data Version Control (DVC). https://dvc.org. Open-source ML experiment management tool. Hans-Arno Jacobsen. 2009. Content-Based Publish/Subscribe. Springer US, Boston, MA, 464–466. doi:10.1007/978-0-38739940-9_1210 Jeffrey O. Kephart and David M. Chess. 2003. The Vision of Autonomic Computing. Computer 36, 1 (2003), 41–50. doi:10.1109/MC.2003.1160055 Henna Kokkonen, Lauri Lovén, Naser Hossein Motlagh, Abhishek Kumar, Juha Partala, Tri Nguyen, Víctor Casamayor Pujol, Panos Kostakos, Teemu Leppänen, Alfonso González-Gil, Ester Sola, Iñigo Angulo, Madhusanka Liyanage, Mehdi Bennis, Sasu Tarkoma, Schahram Dustdar, Susanna Pirttikangas, and Jukka Riekki. 2023. Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestration. arXiv:2205.01423 [cs.MA] Jay Kreps, Neha Narkhede, and Jun Rao. 2011. Kafka: A Distributed Messaging System for Log Processing. In Proceedings of the 6th International Workshop on Networking Meets Databases (NetDB). Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP). 611–626. doi:10.1145/3600006.3613165 Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2024. Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI. Political Analysis 32, 1 (2024), 84–100. doi:10.1017/pan.2023.20 Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7871–7880. Guoli Li and Hans-Arno Jacobsen. 2005. Composite Subscriptions in Content-Based Publish/Subscribe Systems. In Middleware 2005. Springer, 249–269. doi:10.1007/11587552_13 Mingdong Li, Qifeng Luo, Lu Wang, Ruisheng Shi, and Jinqiao Shi. 2020. Privacy-preserving content-based publish/subscribe service based on order preserving encryption. In Internet of Vehicles. Technologies and Services Toward Smart Cities: 6th International Conference, IOV 2019, Kaohsiung, Taiwan, November 18–21, 2019, Proceedings 6. Springer, 374–381. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638 Lauri Lovén, Roberto Morabito, Abhishek Kumar, Susanna Pirttikangas, Jukka Riekki, and Sasu Tarkoma. 2023. How Can AI be Distributed in the Computing Continuum? Introducing the Neural Pub/Sub Paradigm. arXiv preprint arXiv:2309.02058 (2023). Lauri Lovén, Alaa Saleh, Reza Farahani, Ilir Murturi, Miguel Bordallo López, Praveen Kumar Donta, and Schahram Dustdar. 2026. Real-Time AI Service Economy: A Framework for Agentic Computing Across the Continuum. arXiv preprint arXiv:2603.05614 (2026). Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 561–577. Nathalia Nascimento, Paulo Alencar, and Donald Cowan. 2023. Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems. In 2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOSC). 104–109. doi:10.1109/ACSOS-C58168.2023.00048 Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to Route LLMs from Preference Data. In Proceedings of the International Conference on Learning Representations (ICLR). arXiv:2406.18665.
0:30
Lovén et al.
Manish Parashar. 2025. Everywhere and Nowhere: Envisioning a Computing Continuum for Science. Computing in Science & Engineering 27, 1 (2025), 51–56. doi:10.1109/MCSE.2025.3543924 Shiyou Qian, Jiawei Xu, Jian Cao, Guangtao Xue, Junshen Li, and Wenyi Zhang. 2021. Fat Topic: Improving Latency in Content-Based Publish/Subscribe Systems on Apache Kafka. In Wireless Algorithms, Systems, and Applications: 16th International Conference, WASA 2021, Nanjing, China, June 25–27, 2021, Proceedings, Part I 16. Springer, 547–558. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2019), 3982–3992. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. In Foundations and Trends in Information Retrieval, Vol. 3. 333–389. Alaa Saleh, Roberto Morabito, Schahram Dustdar, Sasu Tarkoma, Susanna Pirttikangas, and Lauri Lovén. 2025a. Towards Message Brokers for Generative AI: Survey, Challenges, and Opportunities. Comput. Surveys 58, 1 (2025), 1–37. doi:10. 1145/3742891 Alaa Saleh, Sasu Tarkoma, Anders Lindgren, Praveen Kumar Donta, Schahram Dustdar, Susanna Pirttikangas, and Lauri Lovén. 2025b. Memindex: Agentic event-based distributed memory management for multi-agent systems. ACM Transactions on Autonomous and Adaptive Systems (2025). Alaa Saleh, Sasu Tarkoma, Susanna Pirttikangas, and Lauri Lovén. 2024. Publish/Subscribe for Edge Intelligence: Systematic Review and Future Prospects. Available at SSRN 4872730 (2024). Mazeiar Salehie and Ladan Tahvildari. 2009. Self-Adaptive Software: Landscape and Research Challenges. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 4, 2 (2009), Article 14. doi:10.1145/1516533.1516538 Sasu Tarkoma. 2012. Publish/Subscribe Systems: Design and Principles. Wiley, United States. doi:10.1002/9781118354261 Sasu Tarkoma, Roberto Morabito, and Jaakko Sauvola. 2023. AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systems. arXiv:2311.05842 [cs.NI] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 353–355. Chen Wang, Xunzhuo Liu, Yuhan Liu, Yue Zhu, Xiangxi Mo, Junchen Jiang, and Huamin Chen. 2025. When to Reason: Semantic Router for vLLM. In Workshop on ML for Systems at NeurIPS. arXiv:2510.08731. Minghe Wang, Trever Schirmer, Tobias Pfandzelter, and David Bermbach. 2024. ShutPub: Publisher-side Filtering for Content-based Pub/Sub on the Edge. In Proceedings of the 7th International Workshop on Edge Systems, Analytics and Networking. 13–18. Yijie Wang and Xingkong Ma. 2014. A general scalable and elastic content-based publish/subscribe service. IEEE Transactions on Parallel and Distributed Systems 26, 8 (2014), 2100–2113. Christian Wernecke, Helge Parzyjegla, Gero Mühl, Peter Danielis, Eike Schweissguth, and Dirk Timmermann. 2022. Evaluating P4-based Virtual Delivery Trees for Content-based Publish/Subscribe. In 2022 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN). IEEE, 78–84. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (2018), 1112–1122. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2019), 3914–3923. Tarek Zaarour, Anuraag Bhattacharya, and Edward Curry. 2022. OpenPubSub: supporting large semantic content spaces in peer-to-peer publish/subscribe systems for the internet of multimedia things. IEEE internet of things journal 9, 18 (2022), 17640–17659. Liangzhao Zeng, Boualem Benatallah, Anne H.H. Ngu, Marlon Dumas, Jayant Kalagnanam, and Henry Chang. 2004. QoS-Aware Middleware for Web Services Composition. In IEEE Transactions on Software Engineering, Vol. 30. 311–327. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems (NeurIPS).
A
Prompt templates
The Neural Router uses two prompt templates referenced from the body’s Prompt Design subsection (§3.4 in the body): Listing 1 is the subscription-optimisation (cover/merge) prompt consumed by
Neural Router: Semantic Content Matching for Agentic AI
0:31
Listing 1. Subscription-optimisation (cover/merge) prompt template. {cluster_subs} is replaced with the cluster’s |𝑐.S ′ | subscriptions formatted one per line as “[id] description”. The LLM returns JSON {"covers": [[i,j], ...], "merges": [[i,j], ...]}. Decoding: temperature 0, max_tokens 512. See CoverAndMerge (Algorithm 4 in the body). SYSTEM: You are a content-router optimiser. Given a list of natural-language subscription descriptions, identify pairs (i, j) where subscription i covers j (i.e., every event matching j also matches i) and pairs (i, j) where two subscriptions overlap enough to be merged into a single combined description. Return JSON only. USER: Subscriptions: {cluster_subs} Return JSON with keys "covers" and "merges", each a list of [i, j] index pairs. Use indices from the list above (1-based). Return {} if no pair qualifies. Listing 2. Event-matching prompt template. {active_subs} is the post-CoverAndMerge subscription set for the cluster (or the raw set, for A0); {event_batch} is the current batch of 𝑏 ≤ 𝑏 max events. The LLM returns JSON {"matches": [[event_idx, sub_id], ...]}, with up to 𝜅=3 subscriptions per event. Decoding: temperature 0, max_tokens scaled by batch size. See MatchEvents (Algorithm 3 in the body). SYSTEM: You are a content router. For each event below, return up to K subscription IDs whose descriptions match the event's content. Return JSON only; no commentary. USER: Subscriptions ([id] description): {active_subs} Events: {event_batch} Return JSON of the form {"matches": [[event_idx, sub_id], ...]} where event_idx is 1-based and sub_id is the [id] from the subscription list. At most K=3 subscriptions per event; omit events with no matching subscription.
the CoverAndMerge algorithm (Algorithm 4 in the body); Listing 2 is the event-matching prompt that implements the matching function 𝜇 from the body’s Problem Statement (§3.1) and is invoked by the MatchEvents algorithm (Algorithm 3 in the body). Both use Python f-string placeholders for runtime injection of cluster subscriptions and event batches. B
Parameter-sensitivity panel
Figure 6 accompanies the parameter-sensitivity discussion in §5.2 in the body. The four panels sweep, respectively, the cluster count 𝑘, the cosine pre-filter threshold 𝜏, the top-𝜅 cut-off, and the embedding-model choice on D1 with the Haiku backend in configuration A4.
5
10
15
0.3
6
Top- cut-off
8
(c) Top-𝜅 cut-off.
10
Macro-F1 (D1, A3)
Score
(b) Cosine threshold 𝜏.
F1 Precision Recall
4
0.00 0.25 0.50 0.75
Cosine threshold
(a) Cluster count 𝑘.
2
100
0.1 0.0
300 200
0.2
Number of clusters k 0.5 0.4 0.3 0.2 0.1 0.0
F1 Recall
LLM invocations I
38 36 34 32 30
LLM invocations I
0.5 0.4 0.3 0.2 0.1 0.0
Macro-F1 / Recall
Lovén et al.
Macro-F1
0:32
0.3
0.265
0.263
0.300
0.293
0.2 0.1 0.0
-L6 rge GE-base ase B MiniLM MPNet-b E5-la
Embedding model
(d) Embedding model choice.
Fig. 6. Parameter sensitivity on D1 (Haiku, A4): F1 vs. each parameter holding the others at the production defaults from §4.2 in the body. The plateau in 𝜏 above ≈ 0.3 motivates the discussion of cosine-prefilter geometry in §6.1 in the body.
C
Cross-dataset comparison table
Table 6 reports the best Neural Router configuration per backend against all seven baselines across the three datasets, complementing the dataset-specific narratives in §5.5 in the body and §6 in the body. D
QoE backend-assignment table
Table 7 reports the per-strategy F1, cost, and latency comparison summarised qualitatively in §5.6 in the body. E
Calibration-fraction sweep figure
F
Wider-tier-gradient perturbation
The body §5.6.1 reports a one-sentence summary of the Qwen-2.5 7B / 72B perturbation companion. We give the full table here so the body can be read without it. The campaign mirrors the 7B/32B perturbation design (baseline, topic_restricted_cal, latency_injection; calibration fraction 0.10; 𝑛=15 seeds; matched-pair LLM cache; balanced QoE) and replaces the 32B tier with Qwen-2.5-72B Q4 served across two A100 40 GB GPUs. The
Neural Router: Semantic Content Matching for Agentic AI
0:33
Table 6. Cross-dataset comparison: best Neural Router configuration per backend vs. baselines. F1 reported as mean±half-CI95 over 5 seeds with the winning config in parentheses; † Sonnet rows are single-seed and indicative only. D1 = CardiffNLP (19 topics, short text); D2 = EUR-Lex (201 subjects, long legal); D3 = CASAS smart home (19 activities, IoT sensor). Qwen-2.5-7B runs use the stratified-subsample event caps documented in §4.2 in the body; Haiku and pairwise baselines use the full corpus; Sonnet D2 uses the 5,000-event subsample. DistilBART-MNLI uses the full D1 corpus and stratified subsamples on D2 (|E |=200) and D3 (|E |=1,000). Sonnet D3 was not evaluated; see §6.3 in the body. F1 (best config) Method
D1
NR / Claude Haiku NR / Qwen-2.5-7B NR / Claude Sonnet†
D3
D1
D2
0.656±0.001 (A0) 0.089±0.020 (A1) 0.401±0.011 (A4) 0.030 0.019 0.076 0.505±0.000 (A0) 0.045±0.016 (A1) 0.320±0.001 (A6) 0.024 0.010 0.060 0.717 (A0) 0.316 (A0) – 0.038 0.015 –
3.1 1.9 5.3
11.4 1.7 12.0 43.6 14.7 –
BM25 Sentence-BERT cosine Cross-encoder DistilBART-MNLI zero-shot GloVe cosine TF-IDF cosine Word2Vec cosine
0.082 0.423 0.323 0.434 0.104 0.102 0.110
D2
𝑳 (s)
FPR
0.009 0.154 0.094 0.077 0.027 0.162 0.028
D3
0.130 0.225 0.176 0.094 0.079 0.268 0.077
D1
D2
D3
0.161 0.169 0.155 1.8 4.5 3.2 0.115 0.131 0.144 21.5 35.7 26.9 0.128 0.149 0.151 433.1 – – 0.114 0.022 0.156 10,323 678 413 0.158 0.166 0.158 0.1 0.4 0.3 0.158 0.130 0.138 0.2 0.6 0.5 0.157 0.166 0.158 0.1 0.5 0.3
Table 7. Heterogeneous backend assignment strategies on CardiffNLP, Qwen-2.5 tier sweep (tier_mid=Qwen2.5-7B, tier_large=Qwen-2.5-32B). All rows use the A3 configuration on a 1,000-event stratified subsample, mean over 5 seeds. Cost is the GPT-4o-mini-equivalent token cost per 1,000 events; for self-hosted Qwen tiers the actual deployment cost is more accurately approximated by latency 𝐿. The QoE-optimised rows use per-cluster min-max normalisation across candidate backends (§4.9 in the body). Strategy
F1 (mean)
$/1k evt
𝑳 (s)
Single-backend baselines Homogeneous (Qwen-2.5-7B) 0.166±0.023 Homogeneous (Qwen-2.5-32B) 0.225±0.019
0.017±0.000 141±5 0.015±0.001 359±84
Mixed-strategy assignment Round-robin QoE accuracy-first QoE balanced QoE cost-first
0.016±0.001 0.015±0.002 0.015±0.001 0.015±0.000
0.215±0.006 0.179±0.017 0.178±0.032 0.192±0.039
250±34 246±98 232±31 263±37
homogeneous F1 gap widens from 0.029 (7B/32B) to 0.042 (7B/72B; homog-mid 0.144±0.039 , homoglarge 0.186±0.064 ). Topic-restricted calibration. Matched-cell Δ𝐹 1(QoE-balanced)=−0.008 (𝑛=15, 95 % CI [−0.024, +0.008], paired Wilcoxon 𝑝=0.246). The calibration-quality response detected at 7B/32B (Δ𝐹 1=−0.018, 𝑝=0.008 in the body) is blunted at 7B/72B: the wider per-cluster F1 gap leaves the QoE argmax robust to a topic-biased calibration sample. The mechanism still operates — the sign is still negative — but the magnitude is no longer significant against the inflated inter-seed variance at the wider tier gradient. Latency injection. The matched-cell latency delta is Δ𝐿=+25.08 s as expected (25.08±0.03 s on round-robin and QoE-optimised, 𝑛=15). Matched-cell Δ𝐹 1(QoE-balanced)=+0.034 (𝑛=15, 𝑝=0.0001), unlike the clean isolation observed at 7B/32B (Δ𝐹 1=0.000). This is not a measurement artefact: QoE’s
Lovén et al.
Macro-F1 (D1)
0:34
0.18 0.16
homogeneous round_robin qoe_optimised
0.14 10 1
Calibration fraction
100
Fig. 7. Calibration-fraction sweep on D1, Qwen-2.5 7B / 32B tier gradient, matched-pair LLM cache, 𝑛=15 seeds throughout. Paired one-sided Wilcoxon (QoE > round-robin) returns 𝑝=0.555, 0.381, 0.640, 0.180, 0.738, 0.126 at frac=0.05–1.00; six-fraction Friedman 𝜒 2 =6.17, 𝑝=0.290. No fraction crosses 𝛼=0.05: the calibration-noiselimited claim does not hold in its strong form on this tier gradient. Full numerical detail in body §5.6.2.
three-component objective uses latency, and during calibration the injected 0.05 s per event ≥ 500 flows into the latency component of the per-cluster scalarisation. With min-max normalisation across two backends, the latency change is too small to flip the argmax at the 7B/32B latency ratio (12 vs 41 s baseline) but does flip borderline clusters at the 7B/72B ratio (12 vs 49 s). At this tier gradient, those flips happen to favour the better backend, so F1 rises with the injection. The reading: the “clean dimension isolation” of latency injection is contingent on the perturbation being below the resolution at which calibration argmax is sensitive, not a structural property of the QoE framework. Cost-and-latency tunability. QoE-cost-first matches homogeneous-72B’s F1 within 0.012 at Δ𝐿=−738 s relative to homogeneous-72B (matched-cell). The operator-tunable tradeoff is preserved across the wider gradient: QoE buys back almost all of the F1 ceiling at a fraction of the latency budget when the larger tier is genuinely costly to serve. G
Reproducibility apparatus: regression invariants
Five regression invariants are encoded as Python tests in the artefact repository (tests/test_figure_data_con tests/test_synthetic_data_metric_invariant.py, tests/test_calibration_split_invariant.py) and run on every figure-rendering pass. They constrain the relationship between the underlying CSVs and the manuscript figures, so that a quiet data-side change cannot silently mis-align with the body claims. I1 (cost-model crossover monotonicity — the𝑊 =4,096 context-budget crossover-validation diagnostic). For 𝑊 =4,096, A0 macro-F1 is non-decreasing in |S| until truncation engages, and constant thereafter. This is the operational signature of the cost-model crossover: below the truncation regime the prompt fits the budget and accuracy can only stay flat or improve as more subscriptions are seen; above it the prompt is truncated to the same prefix and accuracy plateaus. I2 (empty-prediction collapse — the duplication-with-rename discrimination diagnostic). The A4 empty-prediction rate is non-decreasing in |S| on duplication-with-rename subscription sets. This pins the discrimination-capacity diagnostic: as |S| grows under a fixed semantic complexity, the model increasingly emits empty predictions rather than a wrong answer. I3 (cost-model factor-of-two band — the predicted-vs-measured invocation-count validation). The cost-model factor-of-two band |𝐼 pred /𝐼 meas − 1| ≤ 1 holds for at least 80% of per-cluster cells. This formalises the predictive accuracy claim quoted in the body and breaks the build if a future re-run drifts the per-cluster distribution outside the headline tolerance. I4 (description-aware metric round-trip). The description-aware F1 reduces to ID-based F1 when every ID has a distinct description. This is the metric-invariant test: the description-aware
Neural Router: Semantic Content Matching for Agentic AI
0:35
refinement only adds resolving power when descriptions disambiguate IDs; in the trivially distinct case it must reproduce the ID-based F1 exactly. I5 (calibration–evaluation disjointness). For every (cluster, backend) cell in the QoE perturbation and calibration-fraction experiments, the calibration and evaluation event sets are disjoint, 𝐸 cal ∩ 𝐸 eval = ∅, with the single intentional exception of the frac=1.00 asymptotic-ceiling point (flagged in the body as an upper bound, not a generalisation estimate). This invariant guards the §5.6 perturbation and calibration-fraction conclusions against silent calibration–evaluation leakage at large calibration fractions. The five invariants are TAAS-RCR-aligned: they are mechanically checkable, archived alongside the data and code, and their failure modes correspond directly to claims a reviewer might want to verify quickly. H
Discrimination-capacity mechanism and deployment implications
This expands the discrimination-capacity finding of the body’s Discrimination Capacity subsection; the headline localisation (binding factor |S| relative to model discrimination capacity, two-axis D2 panel) remains in the body. Mechanism. Two coupled factors. Attention-budget: |S| simultaneous decisions over a shared attention mechanism gives O (1/|S|) per-subscription budget, analogous to “lost in the middle” [Liu et al. 2024]; parameter count enlarges the absolute pool. Instruction-following: newer instructiontuning regimes yield tighter structured-output behaviour, accounting for Mistral ≫ Qwen 7B at matched scale. Pairwise methods avoid both because (𝑒, 𝑠) comparisons are independent. The 𝜏 sensitivity (Suppl. Fig. 1(b)) shows the cosine pre-filter operating in a near-binary regime (𝜏 ∈ [0.05, 0.2]), so it cannot selectively route at high |S| either. Implications. Three takeaways: (i) model selection is upstream of pipeline tuning at |S| ∼ 200 (the 32B-over-1.5B and Mistral-over-Qwen gaps exceed any algorithmic-configuration gap on D2); (ii) parameter count and training generation contribute roughly independently within a 20× parameter range; (iii) where no model fits below the discrimination boundary, the architecture must manage 𝐷 explicitly via subscription partitioning (≤ 𝐷 candidates per prompt) or embeddingprefilter cascades.