Comparative Approaches to Agent Retrieval over Large Skill Libraries

Comparative Approaches to Agent Retrieval over Large Skill Libraries Indivara Kolluru Praetorian ikolluru@andrew.cmu.edu

Nathan Sportsman Praetorian nathan.sportsman@praetorian.com August 7, 2026

arXiv:2608.06196v1 [cs.AI] 6 Aug 2026

Abstract Agents backed by large skill libraries must decide which skills to load and in what order. Loading the entire library into context is expensive and provides no structure for autonomous sequencing. We study two systems for this problem over a corpus of 690 skills: a hybrid ranker combining lexical and dense-embedding retrieval for sparse, on-demand loading, and a typed knowledge graph encoding workflow relations such as prerequisites, data flow, and ordering. On a set of 117 realistic, nonechoing queries, the hybrid ranker retrieves the correct skill within the top five in 73.5%±8.0 of cases, leaving roughly a quarter of queries unserved. When used as the design intended (substituting graph neighbours for additional ranked results at matched token budget), the graph is significantly worse (−11.2 points, p = 0.0007). Its LLM-generated edge layer adds nothing over neighbours obtained free from a local embedding pass, and 73% of the queries the ranker misses are not reachable through the graph at all. We attribute this to a pre-filter topology bound. Because the graph’s candidate edges are drawn from the same embedding neighbourhood the ranker already searches, 98.6% of typed edges connect skills the ranker had already surfaced together. The graph can enrich relation semantics but cannot extend retrieval reach. We further show that evaluating on author-written queries overstates hit@5 by up to 44 points, which would have hidden these results entirely. Our contribution is a mechanistic account of why added structure does not improve retrieval over a strong ranker, and identify the conditions under which adding structural interdependence into the retrieval is optimal.

1

Introduction

1. The pre-filter topology bound (§5.3): a mechanistic, reproducible result that drawing a knowledge graph’s candidate edges from an embedding’s nearest-neighbours caps the graph at that embedding’s own topology; the language model can add semantics to an edge but cannot extend the graph’s reach.

Modern AI agents increasingly rely on libraries of reusable skills: self-contained instruction files, each describing how to perform one capability, loaded into context when needed. As libraries grow, and ours holds 690 skills across five departments, two questions arise: which skills to load, and in what order. The naive answer to the first question is to load everything, which in our corpus costs ∼46.9k tokens per task, paid on every invocation regardless of need. Any serious system must beat that cost, and doing so motivated this work. Our effort produced two systems aimed at the same goal of getting an agent the right skills without loading the library. The first, and the one we set out to build, is a typed knowledge graph over the library, encoding workflow structure that a flat ranked list cannot represent: which skill is a prerequisite for which, whose output feeds whose input, what comes before what. The second is a hybrid ranker: BM25 lexical scoring fused with dense embedding similarity, serving sparse ondemand retrieval. Both are outputs of one investigation, and this paper reports what we learned by building each and then comparing them. We situate both systems in the loading problem they address. An agent’s skill library is a set of SKILL.md files, each a capability described by a name and a oneline description in its frontmatter, plus metadata (department, category, tags, and hand-authored related-skill references). The library is reached through a gateway of meta-tools rather than by loading files directly, and there is no “load all skills” path: an agent would otherwise have to pull every skill’s discovery row into context, and that is the baseline both our systems improve on. Two access routes define the setting. In the pull route, the agent issues a query and receives a ranked list of candidate skills, resolving only those it needs; this is the route the ranker (§3) serves and the one this paper evaluates. A second push route, in which a per-turn hook injects an advisory list of candidate skills without loading them, exists in the same system but is not evaluated here. Both routes are query-conditioned: they answer “which skill matches this request?” Neither answers “given the skill I am now using, what comes next?”, the gap the knowledge graph (§2) is designed to fill, and the distinction the rest of the paper turns on. The paper follows one line of argument. We present the knowledge graph and its edge-generation pipeline (§2), then the hybrid ranker it is measured against (§3), then the head-to-head comparison and its explanation (§5), and finally what that explanation implies (§6). The thread connecting these sections is a single claim in two parts: retrieval is not solved, and the graph nonetheless does not improve it, for a structural reason we make precise.

2. Five converging measurements (§5.4): at matched token budget, substituting graph neighbours for additional ranked results is significantly worse (p = 0.0007); the LLM edge layer adds nothing over free embedding neighbours; 73% of missed queries are unreachable through the graph; search recovers 2× more often where the entry guess is wrong. 3. Author-written query sets inflate retrieval accuracy (§5.2): the same ranker and catalogue score 44 points higher (keyword) and 21 points higher (hybrid) on an author-written query set than on a nonecho set, a transferable methodological result. 4. The method and its cost (§2.2): LLM-atadd-time typed-edge generation, producing 1421 typed edges for ∼$2.70, hash-cached, cycle-safe, provenance-tagged, and reproducible, with an exhaustive 8.5% direction-error census. The method works; the ranker is simply the better fit for entry retrieval.

2

Approach: A Typed Knowledge Graph

We describe the graph’s edge model and its edgegenerating pipeline and report the generator’s coverage, cost, and edge precision.

2.1

The edge model

The graph has three edge layers, which differ in provenance and in how far we trust them. Curated edges are meaningful workflow relationships, hand-authored or LLM-generated. Derived edges come from each skill’s related metadata. Inferred similar to edges are embedding nearest-neighbours, the automatic backbone. The curated layer has eight typed relations. Each carries a traversal priority rather than a continuous weight: requires (→, prereq), specializes (→, ISA), feeds into (→, data flow), routes to (→, handoff), precedes (→, soft order), complements (↔), alternative to (↔), and verifies (→). An author writes one direction. The runtime indexes both endpoints, so the edge set never doubles. That rule avoids hand-mirroring every edge, an O(N 2 ) burden that stalled an earlier Go-based attempt after wiring only 16 of 134 skills. Four types (requires, specializes, feeds into, precedes) must stay acyclic.

Contributions. 2

In the current corpus, the curated and derived layers are both empty: only 1 of 689 skills declares related metadata, and no hand-authored edges exist. All active edges are LLM-generated (1421) or embedding-inferred (1626). The design ranks curated above llm, so a handauthored edge overrides a model’s guess; this is the mechanism that licences shipping the LLM layer at all. With zero curated edges, that override is implemented but unexercised. The similar to backbone links each skill to its nearest neighbours in embedding space. Write êa for the L2normalised embedding of skill a. Cosine similarity is then the dot product sim(a, b) = êa · êb .

Incremental cost. Each skill is keyed by sha256(description)[:16]. Let ∆ denote a set of skills added or edited, s a skill, hash(s) its current description hash, and hashprev (s) the hash recorded in the previous build artifact. The generator runs only for skills whose hash has changed, so the number of model calls is the count calls(∆) = { s : hash(s) ̸= hashprev (s) } ,

(3)

and adding one skill costs one call rather than a full rebuild. An edge depends on both endpoints, but we key it on only one. It can therefore hold a stale view of its far endpoint until that skill is itself reprocessed. The edges are advisory, and they heal on the next build.

(1)

Cycle-safety, provenance, determinism. A model can hallucinate a cycle, so we run a depth-first search and drop the back edges, separately for each acyclic type. Every edge carries provenance (empty for hand-authored, NK (a) = top- K b̸=a { b : sim(a, b) ≥ τ }, (2) llm, or a reserved learned). Only one step is nondeterministic, the LLM call. It writes its output to an yielding 1626 pairs over 690 nodes. Only id-pairs enter artifact, and the build then consumes that artifact deterthe committed graph. The scores stay in a seed file, so ministically, exactly as it consumes the embedding seeds. floating-point noise never churns the graph. Build results (corpus = 690). The generator produced 1421 typed edges with zero failed API calls in 2.2 Edge generation ∼15.5 minutes; validation dropped 113 hallucinated cyclic edges, which keeps the acyclic types acyclic. Type disWhy not learn edges from usage first? The best tribution: complements 390, feeds into 350, precedes evidence that two skills are related is that people use 242, routes to 128, requires 108, alternative to 101, them together. That was our first approach: read the or- specializes 89, verifies 13. (The shortlists came from ganisation’s version-control history, find sequences where a 639-era seed file, so about 50 newer skills received no one skill follows another, and draw an edge wherever edges. A refresh is pending; it will change coverage, not a pairing recurs. Two problems made it impractical. the results here.) Counting co-occurrence needs one place where all the sequences are logged, and none existed yet. And usage only Cost. At $1/M input and $5/M output tokens, the full shows what follows what, which suffices for precedes and backfill cost ∼$2.70, below the ∼$3 estimate; incremental complements, but a sequence never reveals that A recost is ∼$0.005 per edited skill. As §5.4 will show, this quires B, or that A is a special case of B. A new skill expenditure buys edges that add nothing over neighbours also has no history to learn from. We therefore switched obtained free from a local embedding pass. to asking a model when a skill is added, which needs no history and works from the first skill. We return to the usage approach in §6.6: the missing sink has since shipped, Precision. Because edges are predicted, not observed, and it is retained in the design as a future learned prove- we audited 48 (6 per type), judging each from both endpoints as Correct/Plausible/Wrong. Loose precision nance. (C+P) is 41/48 (∼85%); strict (C only) is 29/48 (60%). Those aggregates average over two populations that beCoarse-to-fine generation. When a skill is added, we have differently. Figure 1 separates them by relation first take its top-K=8 embedding neighbours; that is the type, six edges each. Read that way, the three relations coarse step. We then make one call to an inexpensive whose meaning does not depend on which endpoint comes model (claude-haiku-4-5) under forced tool use. It first (requires, specializes, complements) are entirely returns typed edges through an emit edges tool whose correct-or-plausible, while the three whose meaning does schema pins each type to the curated vocabulary; that is (precedes, verifies, routes to) fall to 67–83%, with the fine step. We keep only curated types between tar- feeds into and alternative to in between. The singleget and candidate, drop self-loops, and deduplicate. The number precision therefore understates the safe types and neighbours are drawn across all departments, preserving overstates the directional ones, and the split is ordered cross-domain pivots while bounding cost to a constant K. cleanly enough by directionality to suggest a systematic The neighbourhood NK (a) of skill a is the set of its K=8 highest-similarity neighbours whose similarity also meets a threshold τ = 0.5:

3

Correct

Plausible

Wrong

100% 100% 100% 83% 83% 83% 67% 67%

requires specializes complements feeds_into routes_to alternative_to precedes verifies 0

1

2 3 4 5 6 edges judged per type (6 sampled, 48 total)

7

8

Figure 1: Most edge types are accurate; direction-based types miss more. Per-type audit verdicts (n=48, six per type); percentages are the correct-or-plausible share.

error mode rather than sampling noise. §6.3 confirms this at population scale, where every contradiction category turns out to be a direction disagreement.

Skill Retrieval by Ranking

Having built the graph, we turn to the other product of this work, the hybrid ranker that serves the pull route introduced in Section 1. It is the baseline we measure the graph against. This section describes the ranker and the efficiency of sparse loading; the retrieval accuracy it achieves, and the headroom it leaves, are reported in §5.

tokens per task (context window cost)

3

3.1

What the ranker scores. The ranker scores catalogue entries of {id, kind, name, description, path}. It never returns path, so it scores each capability on name + description alone: one line of frontmatter. As noted in Section 4, the 690 full skill bodies are indexed by neither the ranker nor the edge generator; a sibling effort found that indexing bodies degrades retrieval under this short-context encoder.

The efficiency win, and the question it raises

105

46,915

104 ~99% cost

103 102

560 load everything search (naive) on demand

330 graph traversal

Figure 2: Tokens per task (690 nodes), logarithmic axis.

The payoff of sparse ranked loading is large and direct. The naive load-all harness costs 46,915 tokens per task (875 catalogue rows), while retrieving on demand costs ∼560, a 98.8% reduction. Figure 2 compares all three regimes on one logarithmic axis. Search on demand and graph traversal cost almost the same, and both cost two orders of magnitude less than loading everything, so the saving comes from loading less rather than from loading smarter. Paired with hit@5 of 0.735 on realistic queries (§5.1), the ranker is a deployable improvement over the naive baseline on its own, though it leaves real room for improvement. That efficiency sharpens the second question. Sparse loading solves cost, but the ranker still misses roughly a quarter of realistic queries, so there is retrieval headroom for added structure to capture. The knowledge graph

Three rankers. Keyword (BM25) [4] is dependencyfree and always available, strong on exact identifiers and rare tokens, and serves as an off-topic gate. Semantic uses MiniLM vector similarity and is strong on paraphrase. Hybrid fuses the two and is our production default. The embedding model is all-MiniLM-L6-v2 [3] at a pinned revision: 384-dim, mean-pooled, L2-normalised so cosine equals the dot product. It is the same model and revision the graph’s similar to layer uses, and the topology bound (§5.3) turns out to depend on that. This shared embedding is not incidental: both systems rest on the same similarity signal, which is exactly why one bounds the other. 4

encodes workflow relations a ranker does not represent, We evaluate on the retrieval set:1 117 task-to-skill cases and the rest of the paper tests whether that structure across roughly 15 domains, a catalogue of 875 entries, closes the gap. k=5. Its authors worded the queries deliberately not to echo the target skill’s description, tagged each direct, paraphrase, or indirect, and added an integrity test that 4 Dataset rejects echoing queries. Table 1 reports the result. The production hybrid ranker retrieves the correct skill into All measurements are taken on one skill library and one the top five for 73.5% of queries, and for only 62.8% when evaluation. the user describes a symptom rather than naming the skill. Retrieval is therefore not solved: roughly 31 of 117 Corpus. The library contains 690 skills across five de- queries are still yet to be answered. partments at the time of our edge-generation run; earlier retrieval measurements were taken at 639 skills, as the 5.2 Author-written query sets inflate recorpus grows by daily import. We state the corpus size trieval accuracy at each measurement rather than as a single figure. Each skill contributes one node; department, category, and tags The graph-boost result depends on measuring retrieval are node attributes, not edges. The full skill bodies are not accurately, which proved to be the harder problem. An indexed by either system; the ranker and the edge gener- earlier version of this work used a 37-query set written by ator both read only the one-line description. A sibling the same author who built the system. Evaluated with effort tested body indexing separately and found it de- the same ranker and catalogue, that set reports hit@5 of grades retrieval under this short-context encoder, though 0.946 for both keyword and hybrid, far above the noncontemporaneous work at larger scale reports the oppo- echo figures of 0.504 and 0.735. The gap is 44.2 points site [6]; we treat this direction as open (§6.6). for keyword and 21.1 points for hybrid. Figure 3 sets the two query sets side by side for each ranker. The gaps are Evaluation sets. We measure retrieval on two query unequal enough to change the ordering: on the authorsets, and the contrast between them is itself a find- written set the two rankers are level, and on the noning (§5.2). The primary set holds 117 natural-language echo set they separate by more than 23 points. Only the queries. Each maps to a verified expected skill and is queries change between the two conditions, so an evaluatagged direct, paraphrase, or indirect, and an integrity test tion built the natural way would have reported not merely rejects any query that echoes a skill’s description. We use inflated accuracy but a false equivalence between a lexical it for every headline retrieval number, and we credit the baseline and the production ranker. sibling effort that built it. The second is a 37-query set written by the authors while reading the corpus, reported only to quantify the inflation that author-written queries introduce. Edge precision is measured on a separate sample of 48 generated edges, six per type (§5).

Experiments and Results

We now compare the two systems. We first report retrieval accuracy (§5.1) and the inflation an author-written query set would have introduced (§5.2). We then present the pre-filter topology bound that structurally predicts the graph’s failure (§5.3), and finally test five graph usage modes against that prediction (§5.4).

5.1

hit@5 (correct skill in top 5)

5

author-written set (37 q) non-echo set (117 q)

100%

95% +44 pts

50%

0%

Retrieval accuracy

95%

74%

+21 pts

50%

keyword

hybrid

Figure 3: Author-written queries inflate retrieval accuracy. hit@5 on the author-written (37-query) and non-echo (117-query) sets; only the queries differ.

Table 1: Retrieval on the 117-query non-echo set. Hybrid leaves 26.5% of queries unserved at top-5.

The mechanism is description echo. Queries written Ranker

top-1

top-5 indirect top-5

keyword (BM25) 0.274 0.504 hybrid (default) 0.504 0.735

1 The 117-query non-echo evaluation set was constructed by a sibling team within the organisation. It is included in the repository; we did not control its construction but verified its integrity test.

0.419 0.628

5

while reading the corpus reuse the vocabulary of the descriptions being searched, and because BM25 scores exact term overlap, an echoing query approximates a primarykey lookup. This is why keyword collapses hardest when the echo is removed. We report both query sets rather than silently adopting the non-echo one, because the delta is itself the finding: a naive author-written evaluation would have reported near-perfect retrieval and hidden the headroom the graph then fails to capture.

5.3

(1.4%) fall outside, attributable to seed/corpus drift (inference, not measurement). Figure 5 shows that split. The 1.4% is not novel structure the model found; it is drift between the seed file and the current corpus. Because the candidate shortlist is the embedding top-K, a typed edge can only ever be proposed between skills the embedding layer already connected.

1.4% seed/corpus drift

The pre-filter topology bound

98.6% connect skills search already links

This is the paper’s core structural finding, and we present it before the graph usage experiments because it predicts their outcome. The coarse-to-fine design was presented as a cost bound; it is also a topology bound, not previously Figure 5: Share of the 1022 distinct typed pairs that also appear as similar to pairs. recognised. Typed edges do not change reachability. Adding all 1421 typed edges to the inferred backbone leaves the graph’s connectivity essentially unchanged. Table 2 gives the three layer sets side by side; the last two rows are the comparison that matters. Adding all 1421 typed edges to similar to leaves wired nodes at 594 (zero newly connected) and components at 112 (zero merged). The largest component is unchanged, and 3-hop mean reach rises 2.4%. Figure 4 shows the same three metrics before and after. An entire layer of 1421 edges is added, and the structure it was supposed to create does not appear.

Topology bound (general property of the method class). When LLM edge generation draws candidates from an embedding’s top-K neighbours, the resulting graph is confined to that embedding’s topology. The LLM can enrich semantics by assigning a typed relation to a pair, but it cannot add reach, because it is never shown a pair the embedding did not already surface. Any method that prefilters candidates by the same signal the ranker uses inherits this ceiling. The topology bound predicts two things: re-ranking can only reshuffle within the ranker’s own candidate set, so no retrieval improvement is mechanistically available; and graph neighbours at matched budget should underperform ranked results, because the graph draws from a subset of the same signal. The next section tests both predictions.

Table 2: Graph structure by layer set (reach over wired nodes, undirected). Adding the typed layer to similar to changes no connectivity metric.

layer set

wired

typed only 524/690 similar to 594/690 all layers 594/690 700 600

comps largest singles 3-hop 190 112 112

441 544 544

166 96 96

29.6 66.2 67.8

5.4

We test the graph in every plausible usage mode. Table 3 and Figure 6 report the full ladder; the graph cannot retrieve without a ranker, so every graph arm calls hybrid to pick its entry point.

no change 594 594

500 400

similarity edges only + all 1421 typed edges

300 200

no change 112 112

100 0

skills with at least 1 edge

Graph usage experiments

separate clusters

66.2

Table 3: Performance ladder on the 117-query non-echo set. Every graph arm uses hybrid internally for entry selection. arm

67.8

hit@5 note

graph alone (no ranker) 0.009 ≈chance hybrid@1 0.504 search, 1 cand hybrid@1 + graph (6) 0.632 search+graph hybrid@6 0.744 search, 6 cands hybrid@10 0.803 search, 10 cands

average 3-hop reach (nodes)

Figure 4: Connectivity of the inferred backbone before and after adding all 1421 typed edges.

The mechanism, measured. Of 1022 distinct typed Substitution at matched budget. The apples-topairs, 1008 (98.6%) are also similar to pairs; only 14 apples question is: given room for N candidates, should 6

chance

0.744

hybrid@6 hybrid@1 + graph nbrs (6)

0.632 0.504

hybrid@1 graph alone (no ranker)

spreading activation over the graph. Hybrid gains a single query in 117 (+0.85 points; 6 gained, 5 lost; McNemar p ≈ 1.000; 95% interval −4.8 to +6.5 points). This is a null: no evidence of an effect of either sign.

0.803

hybrid@10

0.009

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 hit@5 (fraction of queries where the correct skill appears in the top 5 results)

6

Discussion and Analysis

6.1

Why ranking is better at every usage mode

Three mechanisms explain why five independent measurements converge. First, search is query-conditioned; the graph is not. Search ranks 690 capabilities against the user’s actual task text; graph adjacency is fixed at build time. For “select the right skill for this task,” search has strictly more information. Second, the topology bound (§5.3) confines the graph to a strict subset of the ranker’s signal. The LLM edge layer adds zero reach (Table 2), zero retrieval value over free kNN (§5.4), and 73% of the queries the ranker misses are not reachable through the graph at all. A truncated view of a signal cannot beat the signal. Third, hybrid fusion already covers both failure modes the graph might have patched: the semantic channel covers paraphrase gaps and BM25 covers lexical drift, which is why keyword sheds 44 points when echo is removed but hybrid sheds only 21 (Figure 3). The residual headroom that a “graph rescues buried queries” narrative would require has already been absorbed by the fusion.

Figure 6: Performance ladder. At matched budget (6 candidates), graph neighbours significantly underperform additional ranked results.

they be N ranked results or 1 result plus N −1 neighbours? At budget 6, hybrid@1+graph(6) scores 0.632 against hybrid@6 at 0.744, a deficit of 11.2 points (McNemar p = 0.0007). The graph is significantly worse at matched budget on this corpus. The improvement over hybrid@1 (0.504 → 0.632) reflects the value of having any additional candidates, not the value of graph structure.

The LLM edge layer adds nothing over free neighbours. At budget 6, varying only which edge layer supplies the neighbours: similar to alone (free local kNN) scores 0.632; typed edges alone (the 1421 LLM-generated edges) score 0.607; both layers together score 0.632. The entire LLM edge-generation step (1421 edges, ∼$2.70, 6.2 Statistical power: the null is a bound, eight curated types) contributes zero measurable retrieval not a shortfall value over neighbours obtained free from a local embedding pass. This is the topology bound expressed as an The substitution arm is significant (p = 0.0007); it is the outcome rather than a structural statistic. re-ranking arm that requires care. Table 1 reports hit@5 of 0.735; its 95% half-width is ±8.0 points at n=117. On 73% of missed queries are unreachable. Of the 30 the re-ranking arm, the graph-boost difference is +0.9 queries hybrid@6 misses, we measure reachability from points (Newcombe 95% interval −4.8 to +6.5; exact Mchybrid’s top-1 result through the graph: 6 (20%) are Nemar p = 1.000). The interval excludes gains larger reachable within 1 hop, 2 (7%) at 3 hops, and 22 (73%) than roughly 6.5 points, so re-ranking cannot recover are unreachable within 3 hops. The headroom is struc- more than about a quarter of the 26.5-point headroom. turally out of the graph’s reach at any ranking or edge What the null supplies is an upper bound on the efquality; no amount of scoring improvement connects a fect, not a demonstration that the effect is exactly zero. Per-phrasing strata (n=30–44) are too small to support target that is not connected. any sub-group conclusion; the mechanism, not the sample size, is what makes the aggregate null strong. Where the entry guess is wrong, search recovers twice as often. Conditioning on whether hybrid’s top1 result is correct: when the entry is correct (n=59), 6.3 Direction-error census both arms score 1.000 trivially. When the entry is wrong The direction-reversal mode visible per type in Figure 1 (n=58), retrieving more ranked results recovers 48.3% of is confirmed at population scale. Of 1022 distinct pairs, cases while graph neighbours recover 25.9%. The graph 298 (29.2%) carry more than one type. Of those, 87 underperforms precisely where help is needed. (8.5% of all pairs) are contradictory. Twenty carry the same asymmetric type in both directions, which is an unRe-ranking (graph-boost) is a null. For complete- ambiguous error needing no semantic judgement. Every ness, we also test re-ranking the ranker’s output by contradiction category is a direction disagreement (e.g. 7

20 routes to-vs-routes to reversed; 17 feeds into-vsprecedes reversed), so the 48-edge audit and the 1022pair census agree on which relations fail and on how they fail. The implication for any product use is concrete: shipping “do X before Y ” guidance at 8.5% reversed is worse than shipping nothing, because a wrong prerequisite actively misleads.

6.4

and sufficiency experiment is the load-bearing missing test. A harness exists that measures next-skill invocation with and without typed data; the decisive experiment is buildable but needs runtime wiring and must first resolve a known backfire effect from the suggestion hook. Candidate generation beyond top-K is the only route that could give the graph reach the ranker lacks. A graph-native retrieval system: the topology bound shows that an embedding-confined graph cannot extend retrieval reach; a system that generates candidate edges from sources beyond the embedding kNN (e.g. cousage logs, dependency metadata, or cross-modal signals) could in principle break this ceiling and provide a retrieval mechanism that complements rather than duplicates the ranker. Disambiguation at scale: at larger corpus sizes where embedding neighbourhoods grow denser, clusters of near-identical skills may become harder for the ranker to distinguish; the disambiguation value of typed edges within such clusters warrants separate investigation. Body indexing under a long-context encoder: a sibling effort found that indexing bodies reduces retrieval under our 256-word-piece encoder, but Zheng et al. [6] report the opposite at scale, so we treat this direction as reopened rather than closed.

When the knowledge graph would be more beneficial

Everything above concerns retrieval. The graph’s remaining structural case is on ordering: the question of what skill follows the current one, which retrieval does not pose. Consider a task needing n skills in sequence. Repeated search succeeds with probability pn ; graph traversal succeeds with p · q n−1 , where q is per-hop edge accuracy. The graph wins iff q > p. The directional edge types that an ordering consumer would read run 0.67–0.83 accuracy, which straddles the p=0.735 threshold. Our numbers cannot settle whether traversal beats repeated search for ordering; the question requires a skill-to-skill sequence dataset that does not yet exist and an 8.5% directionerror rate that is unmet. This is future work, not a rescue for entry retrieval. The graph need not outperform search to be useful: 7 Related Work enriching resolve skill so that an agent loading a skill also receives its prerequisites, next-steps, and alternatives Two contemporaneous systems address our exact setis an additive change with no adoption cost, provided the ting, retrieval over a library of SKILL.md files, and reach direction-error rate is first reduced. positive conclusions where we reach a negative. SkillRouter [6] studies retrieval over ∼80K skills and reports that the full implementation body, not the front6.5 Threats to validity matter description, is the decisive signal: removing it We surface the threats a careful reviewer will raise. The degrades retrieval by 29–44 points across every method primary evaluation set has 117 queries; one query is they test. Their two-stage retrieve-and-rerank pipeline ∼0.85pp of hit@5. The non-echo set and its integrity reaches 74.0% top-1. Graph-of-Skills (GoS) [2] builds test were built by a sibling effort, not by us, which re- a typed graph over a SKILL.md library and retrieves moves the author-echo bias but means we did not control a dependency-aware bundle via hybrid semantic–lexical its construction. The precision audit behind Figure 1 seeding and reverse-weighted Personalized PageRank, reis n=48, single-judge, and description-only, which is why porting gains on SkillsBench and ALFWorld together the 87 structural contradictions (immune to this critique) with large token reductions. carry the weight. No latency was measured. We never The divergence from our result lies in candidate genertested sufficiency: every cost arm counts rows rather than ation. GoS derives its edges from dependency structure completed tasks. Our metric is single-gold hit@5; under in the skill packages, not from the retriever’s own emit, a graph that supplies a needed prerequisite but dis- bedding neighbourhood. Our generator does, and §5.3 places the gold skill from the top five scores as a loss. shows the consequence: 98.6% of typed pairs fall inside Finally, this is a single corpus, single organisation, single the similar to set the ranker already computes. More embedding model: no claim about other libraries, scales, broadly, graph-augmented retrieval has shown gains in or backends is supported. compositional and multi-hop settings [1, 5]; our result is notable precisely because it holds in a non-saturated single-hop regime, which the topology bound explains. 6.6 Future work In priority order. Co-usage telemetry: an ordered skill-call log has shipped to production; observed co-usage 8 Conclusion is orthogonal to the embedding and is the one remaining candidate signal that is not embedding-confined. It could We set out to improve how an agent selects skills from in principle break the topology bound. The sequence a large library, and built two systems toward that end. 8

Sparse ranked loading cuts context cost by roughly two orders of magnitude, a real and deployable gain. The introduction promised a two-part thread, and we close it here. Retrieval is not solved : on a non-echo set of 117 realistic queries the hybrid ranker reaches the correct skill for 73.5% of cases, leaving roughly a quarter unfulfilled. The graph does not close the gap, for a structural reason. Five measurements converge: at matched budget, graph neighbours are significantly worse than additional ranked results (p = 0.0007); the LLM edge layer adds nothing over free embedding neighbours; 73% of missed queries are unreachable through the graph; search recovers twice as often where the entry guess is wrong; and re-ranking with the graph is a null. The structural reason is that 98.6% of the graph’s edges connect skills the embedding layer had already connected, so the pre-filter that makes edge generation affordable also bounds the graph’s topology. The method works: it is cheap, reproducible, and reasonably accurate. What does not hold is the premise that an embedding-confined graph improves on a strong ranker at entry retrieval. Establishing that structurally, and identifying candidate generation beyond the embedding kNN and co-usage telemetry as the directions that remain open, is the contribution.

Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. [2] Dawei Liu, Zongxia Li, Hongyang Du, Xiyang Wu, Shihang Gui, Yongbei Kuang, and Lichao Sun. Graph-of-skills: Dependency-aware structural retrieval for massive agent skills. arXiv preprint arXiv:2604.05333, 2026. v3, 27 May 2026. Code and data: https://github.com/davidliuk/ graph-of-skills. [3] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of EMNLP-IJCNLP, 2019. [4] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3 (4):333–389, 2009. [5] Qiao Xiao, Haochen Shi, Yisen Gao, Wenbin Hu, Huihao Jing, Tianshi Zheng, Baixuan Xu, Ziheng Zhang, Weiqi Wang, Haoran Li, Jiaxin Bai, and Yangqiu Song. SING: Synthetic intention graph for scalable active tool discovery in LLM agents. arXiv preprint arXiv:2606.16591, 2026.

[6] YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuan Zhu, Baohua Dong, and Hangcheng Zhu. SkillRouter: Retrieve-and-rerank skill seRepository praetorian-inc/palatine, tag lection for LLM agents at scale. arXiv preprint skill-graph-paper-v2 (commit 17223a95). Every arXiv:2603.22455, 2026. Alibaba Group. figure reported here reproduces from that tag and from no other point in the repository: the work was developed as a stack of six pull requests, each contributing only its own edge layer, and three defects surfaced only when they were merged. Setup, environment pins, and each command with its expected output are in docs/skill-graph-reproduction.md at that tag. No measurement requires an API call. Two caveats recorded there are worth surfacing. The repository’s graph and gateway CI workflows trigger only on pull requests targeting main; every branch in the stack targeted a sibling, so neither suite ever ran on this work and the local runs are the sole evidence. And the committed gateway bundle must be rebuilt (make gateway-build) before any retrieval measurement, since a stale bundle causes the semantic and hybrid arms to skip silently rather than fail. The 117-case non-echo retrieval dataset, its phrasing taxonomy, and the description-only vs. description-plusbody ablation are prior work by a sibling effort and ship on main; only the graph arms are ours.

Reproducibility

References [1] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and 9

Related documents

• Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills#1069774
• SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents#819362
• SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents#809369
• Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skills#361525
• Imagine Before Retrieval: Prospective Skill Retrieval for LLM Agents#822079