arXiv:2606.05489v1 [cs.CV] 3 Jun 2026
LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval Shahrzad Esmat
Chaunté W. Lacewell
Sameh Gobriel
Iowa State University [email protected]
Intel Corporation [email protected]
Intel Corporation [email protected]
Nilesh Jain
Ali Jannesari
Intel Corporation [email protected]
Iowa State University [email protected]
Abstract Retrieval systems underpin modern AI applications — spanning visual search, recommendation engines, and multi-modal question answering. Modern multi-stage retrieval systems require the joint optimization of highly coupled parameters, yet traditional hyperparameter optimization (HPO) methods — including Tree-structured Parzen Estimators (TPE) and Gaussian Process Bayesian Optimization — rely on an independence assumption that fundamentally prevents them from navigating these coupled configuration spaces. We address this limitation with a phase-aware large language model (LLM) agent that conditions each proposal on its full optimization history, navigating the coupled parameter space across phasepartitioned exploration, exploitation, and fine-tuning stages. Evaluated on the HICO-DET human-object interaction retrieval benchmark using Intel VDMS (Visual Data Management System), our agent outperforms Optuna TPE by +33.3% and VDTuner by +34.2% under SIEVE (Safeguarded Index Evaluation of Vector-search Efficiency, a quality-constrained throughput metric), delivering a 15.3× throughput gain over UniIR. Validation across three benchmarks confirms that the agent’s advantage grows with the degree of parameter coupling: +33.3% on HICO-DET (high coupling), methods converge within 1% on GLDv2 (moderate coupling) and within 3.6% on SIFT1M (near-independent control). Cross-system validation on Milvus confirms the optimizer ranks first on all three datasets without modification, demonstrating transferability across vector database management system (VDBMS) platforms. These results position our agent as the method of choice for retrieval systems where parameter coupling is the dominant source of optimization complexity.
1
Introduction
The throughput and accuracy of a vector database management system (VDBMS) depend critically on approximate nearest neighbor (ANN) index configuration: on a production-scale deployment, the gap between default and optimized settings can exceed an order of magnitude in queries per second on identical hardware. These systems serve millions of queries daily across recommendation, visual search, and cross-modal content discovery. We study this problem in the context of Human-Object Interaction (HOI) retrieval — returning images of specific human-object interactions in response to natural language queries — an application with direct uses in video surveillance, assistive robotics, and interactive multi-modal search. Closing this performance gap demands an optimizer that treats parameters jointly, not one that treats them as independent.
Modern vector databases such as VDMS (Visual Data Management System) [24], Milvus [30], and GaussDB-Vector [28] combine two stages: a fast ANN search followed by a reranking step. The parameters of each stage are tightly coupled: reducing retrieval search depth (efSearch) forces the reranker to compensate with a higher fusion weight (𝛼) — a cross-stage dependency that parameter-independent optimizers cannot see. On HICO-DET, the gap between an unoptimized baseline and our agent-optimized configuration reaches 36.6× in throughput on a single A40 GPU node — the difference between 8 and 300 queries per second. No existing method is designed to close this throughput gap. On the LLM-guided configuration side, GPTuner [14] and 𝜆Tune [9] apply large language model (LLM) guidance to database knob tuning, but their knobs are mutually independent — no crossstage coupling exists. On the ANN index tuning side, VDTuner [34], Optuna TPE [1], grid and random search [3] all treat parameters independently, missing the coupling entirely. VDTuner has a further weakness: it performs less reliably when performance changes discontinuously at a quality threshold rather than gradually. We propose a phase-aware LLM agent that jointly optimizes ANN index and reranking parameters on Intel’s VDMS [24], and validate that it transfers without modification to Milvus [30]. At each step, the agent reviews all previous configurations and their results, identifies how parameters across stages interact, and decides whether to explore new regions or refine the best configuration found so far. No existing benchmark combines fine-grained compositional text queries with a large image corpus — standard ANN benchmarks use synthetic vectors or image-toimage similarity, while HOI datasets have only been used for detection [13, 16, 21, 36]. Alongside the agent, we introduce the first such benchmark built from HICO-DET [6]: its 600 verb–object categories (e.g., ride horse, hold umbrella) map directly to natural language queries, and its 90,641 annotated positive pairs provide retrieval ground truth without additional labeling. Under SIEVE, our joint quality-throughput optimization objective defined in Section 6.1, the agent achieves +33.3% over Optuna TPE and +34.2% over VDTuner [34] on HICO-DET. On simpler datasets — GLDv2 [32] (image-to-image) and SIFT1M [10] (pure ANN search) — this advantage diminishes proportionally to the degree of parameter coupling, confirming that LLM-guided optimization is most valuable when parameters interact across stages. Cross-system validation on Milvus [30] confirms the optimizer transfers to a second VDBMS without modification, with LLM ranking first on all three datasets.
Contributions. This paper makes the following contributions: • Phase-aware LLM agent for coupled ANN index optimization. We propose an LLM-guided optimizer that reads its own trial history to discover how index construction and reranking parameters must be configured together — a dependency that standard optimizers miss by treating parameters independently. On HICO-DET the agent achieves +33.3% over Optuna TPE, +34.2% over VDTuner [34], and +50.7% over GP-BO (Gaussian Process Bayesian Optimization) [26], with gains diminishing on simpler datasets where this inter-parameter dependency is absent. Cross-system transfer to Milvus [30] without modification confirms the improvement is structural, not incidental. • First text-query HOI retrieval benchmark. Existing ANN benchmarks use synthetic vectors or image-to-image similarity, neither of which tests optimization under real semantic queries — making it impossible to know whether smarter optimization actually matters in practice. We fill this gap by repurposing HICO-DET as a text-to-image retrieval challenge on a live vector database: 47,776 images queried by 600 natural-language verb– object phrases. The fine-grained compositional queries create the configuration-sensitive retrieval landscape that motivates our optimizer. • Single-index two-stage retrieval pipeline. We present a two-stage pipeline that pairs Contrastive Language-Image Pre-training (CLIP)-based ANN search with DINOv2 reranking, gaining +3.65 pp mean Average Precision (mAP) at under 1% extra latency. The design requires only a single ANN index, keeping the parameter space tractable for automated optimization while combining complementary visual and semantic cues. • Complexity-matched evaluation of when LLM reasoning pays off. We evaluate across three datasets of decreasing complexity — compositional HOI retrieval (HICO-DET), crossdomain landmark retrieval (GLDv2), and synthetic vector search (SIFT1M) — and show that LLM-guided gains shrink as the search space becomes simpler, giving practitioners a clear signal for when the added complexity is worth it. • SIEVE: a quality-constrained throughput objective for deployment-aware ANN evaluation. We propose SIEVE, a metric that enforces a minimum retrieval quality floor before rewarding throughput, matching the real deployment constraint that speed without accuracy is worthless. We show empirically that configurations ranked best by conventional smooth metrics routinely fall below this quality floor — meaning standard metrics actively mislead optimization toward configurations that would fail in production.
2 2.1
interaction classification in a cascade, extracting complementary strengths of one-stage and two-stage paradigms. HOI-CLIP (HOICLIP) [21] showed that vision-language pre-training transfers interaction knowledge efficiently, while Disentangled Pre-training for HOI (DPHoI) [16] exploited disentangled pre-training over interaction primitives. All operate in the detection paradigm — predicting (person, object, interaction) triples from an image — and the sole prior non-detection use of HICO-DET [12] employs spatial canvas composition with IoU matching rather than free-form text queries. Our novelty is specific to the intersection: general text-toimage retrieval benchmarks (COCO Captions [7], Flickr30k [35]) use open-domain descriptive captions; scene-graph retrieval uses structured graph predicates; compositional image retrieval pairs images with text modifications — none provides free-form verb–object text queries against a large HOI-annotated corpus with per-category retrieval ground truth, which is precisely the combination that stresses ANN index configuration and motivates our optimizer. CLIP [23] established joint vision-language embedding via contrastive pre-training on 400 million image-text pairs, enabling zero-shot text-to-image retrieval. DINOv2 [22] produces self-supervised visual features that complement CLIP’s languagealigned representations, capturing fine-grained visual structure suited to reranking. UniIR [31] unified multimodal retrieval tasks under a single model trained on M-BEIR, establishing a strong crossmodal baseline. LamRA [18] proposed a unified framework in which a large multi-modal model serves as both retriever and reranker across heterogeneous query types. We fill the orthogonal HOI retrieval direction and adopt CLIP and UniIR as primary baselines.
2.2
Vector Database Systems and ANN Indexing
The efficiency of HOI retrieval depends not only on the query models above but also on how the underlying approximate nearestneighbor indices are configured. ANN indexing algorithms span graph-based indices (e.g., HNSW [19] via hierarchical navigable small-world graphs, DiskANN [27] for billion-point SSD-resident datasets), inverted file structures with product quantization [10], and GPU-accelerated flat search [11]. ANN-Benchmarks [2] standardizes recall-versus-throughput evaluation across these algorithms; the NeurIPS’21 competition [25] established billion-scale baselines. Production VDBMSs integrate ANN indexing with metadata management and query processing. Milvus [30] supports multiple index backends with a cloud-native query layer. GaussDBVector [28] separates adjacency-list from vector storage and applies novel buffering strategies to sustain low-latency high-recall search at billion-vector scale. VSAG [38] exploits a subgraph inclusion property — stricter-parameter graphs are subgraphs of relaxed-parameter ones — enabling parameter tuning without index rebuilds. VDMS [24] co-locates graph-property metadata with vector data and exposes HNSW index parameters through a Python API, making it the natural platform for an optimizer that must rebuild and re-query the index at each evaluation step. Across all these systems, index configuration is left to manual tuning or blind search; automated optimization for multi-stage coupled parameter spaces remains an open problem.
Related Work Vision-Language Retrieval and HOI Detection
HOI detection has produced a series of strong transformerbased systems since HICO-DET [6] established the 600-category verb–object benchmark. Human-Object interaction TRansformer (HOTR) [13] introduced end-to-end interaction detection via encoder-decoder architectures, and Cascade Disentangling Network (CDN) [36] disentangles human-object pair detection from 2
2.3
3
Hyperparameter Optimization
Standard hyperparameter optimization (HPO) methods include random search [3], which provides a strong baseline under bounded evaluation budgets; Bayesian optimization with Gaussian Process surrogates [26], which applies acquisition functions to balance exploration and exploitation; and bandit-based scheduling such as Hyperband [15], which allocates evaluation budget adaptively. Auto-sklearn [8] combines Bayesian optimization with metalearning over prior tasks to warm-start configuration search. Optuna [1] implements Tree-structured Parzen Estimators for mixed-type spaces and serves as our strongest non-LLM baseline. For database management system (DBMS) tuning, OtterTune [29] pioneered ML-guided knob selection via Gaussian Process regression over workload representations. VDTuner [34] extends this approach to multi-objective Bayesian optimization across seven index types in Milvus, improving over manual baselines but operating on single-stage vector search without crossstage coupling. All of these methods — standard HPO [1, 3, 15, 26], AutoML [8], and system-specific Bayesian optimization [29, 34] — are designed for single-stage configuration spaces; none addresses the compensatory interactions that arise when index and reranking parameters are coupled across retrieval stages.
2.4
3.1
The HICO-DET Retrieval Benchmark Source Dataset
HICO-DET [6] contains 47,776 images (38,118 train / 9,658 test) annotated across 600 HOI categories with per-instance bounding boxes and interaction labels, where each category is a verb–object pair drawn from 117 action verbs and 80 COCO object classes. Three structural properties make HICO-DET particularly challenging for ANN index configuration optimization. First, compositional fine-grained discrimination: the 600 categories span 80 COCO object classes and 117 verb types; categories sharing the same object (e.g., ride horse, sit_on horse, straddle horse) differ only in the action verb, so single-encoder retrieval conflates them and the joint selection of ANN engine and Stage-2 fusion weight becomes critical. Second, long-tail relevance: category sizes are highly skewed, with the largest exceeding 1,000 positive images and the smallest having as few as 5; an optimizer that maximizes mean mAP without attending to per-category variance will exploit high-frequency categories while failing on rare but equally weighted queries. Third, a dual-encoder modality gap requires joint tuning: CLIP retrieves by language-aligned semantic similarity while DINOv2 captures fine-grained visual structure complementary to CLIP’s representations [22]; CLIP alone (Stage 1 without DINOv2 reranking) achieves lower mAP than the two-stage fusion, so the Stage-2 fusion weight 𝛼 is a data-dependent parameter that must be tuned jointly with the Stage-1 ANN configuration.
LLM-Guided System Configuration
Recent work has explored whether large language models can serve as optimization agents, bringing natural language reasoning to configuration search. OPRO [33] established that LLMs can act as black-box optimizers by conditioning successive proposals on a history of candidate solutions and their scores. Follow-up work extended this paradigm to ML hyperparameter optimization [17, 37]. GPTuner [14] applies LLM-guided Bayesian optimization to PostgreSQL knob tuning, extracting structured configuration knowledge from database documentation to guide search. 𝜆Tune [9] generates complete database configuration scripts via LLM, framing prompt construction as a cost-based optimization, and is evaluated across PostgreSQL and MySQL. Both outperform ML-based and RL-based baselines on DBMS tuning, yet both operate on flat, independently interpretable knob spaces where no cross-stage coupling exists by construction. Our work addresses what neither thread has reached: a coupled multi-stage parameter space where ANN retrieval depth and fusion reranking weight must be optimized jointly under a hard feasibility constraint that smooth surrogate models cannot represent. Three structural properties make directly applying OPRO [33] insufficient. First, the valid parameter set is conditional on engine type — the search space itself changes mid-optimization, violating OPRO’s fixed-candidate assumption. Second, the SIEVE feasibility cliff is discontinuous: plain history-conditioned proposals average over the cliff rather than committing to the feasible high-QPS side. Third, cross-stage (𝑘, 𝛼) coupling means the joint optimum is unreachable by treating parameters as independent marginals, as history-only proposals do. Our additions — phase structure, diagnostic guidance 𝑔(H ), and untried-value hints — directly target each gap (Section 6.7 brackets OPRO: removing history gives 0/3 seeds; retaining history without phases/hints upper-bounds OPRO at 299.5, 3/3 seeds).
3.2
Benchmark Construction
Retrieval Corpus. We use all 47,776 images as a single retrieval corpus. The original train/test split was designed for supervised detection training and has no meaning for retrieval evaluation, where the task is to rank corpus images given a query; merging both splits maximizes the per-category positive density needed for reliable mAP computation across all 600 categories. CLIP ViT-L/14 [23] embeddings are pre-computed offline using the official OpenAI weights and stored in VDMS as a 768-dimensional L2 DescriptorSet of 47,776 vectors. Query Design: One Canonical Query per HOI Class. We construct exactly one canonical natural-language text query per HOI category, yielding 600 queries. A single canonical query per category eliminates inter-query variance from paraphrase choice and ensures that each of the 600 HICO-DET categories contributes equally to the mean Average Precision score [20]. Each query follows the template “a person {verb} a {object}”, with the special case “a person near a {object}” for the no_interaction verb class. Queries are encoded with the same CLIP ViT-L/14 text encoder, producing 768-dimensional unit-norm text embeddings that reside in the same embedding space as the indexed image vectors. Ground Truth: Multi-Label Positive Structure. An image is relevant to a query if it carries the corresponding HOI annotation in the HICO-DET ground truth. A single image may be relevant to multiple queries (e.g., an image annotated with both ride motorcycle and hold phone contributes positives to two queries). Summed over all 600 queries, there are 90,641 positive (image, query) pairs. For each HOI category, the single image used as the DINOv2 reranking reference is excluded from relevance judgments (treated as a junk image), ensuring the reference cannot trivially self-retrieve; after 3
Table 1: Configuration space Θ (Definition 1); † conditional on engine; total |Θ|=102,144.
this exclusion the minimum per-category positive count is 5, so mAP is well-defined for all 600 queries.
3.3
Evaluation Protocol
We evaluate retrieval quality using mean Average Precision (mAP), Precision@10, Normalized Discounted Cumulative Gain (NDCG)@10, and Recall@10 over all 600 queries. mAP is our primary metric: it rewards systems that rank relevant images higher in the retrieved list and is the standard metric for ranked retrieval with multiple relevant documents per query [20]. Throughput is measured as batch QPS: 600 queries issued as a single batch to a live VDMS container, divided by total wall-clock time.The joint optimization objective is ( QPS(𝜃 ) if mAP(𝜃 ) ≥ 𝜏, Score(𝜃 ) = 𝜏 = 0.15. (1) 0 otherwise,
4.2
0.0, 0.05, . . . , 0.90 1, 3, 5, 10 first, centroid, diverse
19 4 3
5 4 7 5 8
102,144
Stage 1: ANN Candidate Generation
All 𝑁 CLIP ViT-L/14 image embeddings are pre-computed offline and stored in VDMS as a 768-dimensional L2 DescriptorSet. At query time, 𝑞ℎ is encoded by the frozen text encoder and submitted to VDMS as a FindDescriptor request. VDMS returns the top𝑘 candidate identifiers together with their squared L2 distances 𝛿 𝑐𝑗 = ∥c 𝑗 − q𝑐 ∥ 22 ; the candidate pool is formally T𝑘 (𝑞ℎ ) = 𝑗 ∈ [𝑁 ] : ∥c 𝑗 − q𝑐 ∥ 22 ≤ 𝛿 (𝑘 ) , (3)
Definition 3 (Optimization Problem). Given Θ (Definition 1), O (Definition 2), and a finite evaluation budget 𝑁 ∈ N, find
where 𝛿 (𝑘 ) denotes the 𝑘-th smallest squared L2 distance from q𝑐 to any corpus embedding. For FaissFlat, Eq. (3) is satisfied exactly; for FaissHNSWFlat and FaissIVFFlat, VDMS returns a high-recall approximation whose tightness is governed by the engine-specific search parameters. Three ANN engines expose qualitatively distinct accuracy– throughput operating points. FaissFlat: exact exhaustive L2 search; recall upper bound at lowest QPS. FaissHNSWFlat [19]: hierarchical navigable small-world graph; recall/latency controlled by graph connectivity 𝑀 and beam width efSearch (values in Table 1). The build-quality parameter efConstruction is fixed at 200 throughout HICO-DET and GLDv2 experiments; preliminary experiments confirmed that values above
(2)
subject to at most 𝑁 calls to O, where Score(𝜃 ) is defined by Eq. (1) (QPS(𝜃 ) if mAP(𝜃 ) ≥ 𝜏, else 0). We set 𝑁 =50 throughout, covering 0.05% of |Θ|.
4.1
3
𝑁 denote the corpus of 𝑁 =47,776 images, else 0. Let C = {(c𝑖 , d𝑖 )}𝑖=1 768 where c𝑖 ∈ R is the unit-normalized CLIP ViT-L/14 embedding and d𝑖 ∈ R1024 is the DINOv2 ViT-L/14-reg4 embedding of image 𝑖, and Iℎ ⊆ [𝑁 ] the annotated relevant set for HOI class ℎ. Given a text query 𝑞ℎ (e.g., “a person riding a horse”), the pipeline proceeds in two stages as illustrated in Figure 1. Stage 1 encodes 𝑞ℎ into a unit-norm vector q𝑐 ∈ R768 via the frozen CLIP ViT-L/14 text encoder and issues an ANN search over the VDMS CLIP DescriptorSet to retrieve a candidate pool T𝑘 (𝑞ℎ ) of cardinality 𝑘. Stage 2 reranks T𝑘 (𝑞ℎ ) using DINOv2 visual distances computed against a pre-built per-class reference rℎ , fusing both distance signals without any additional index. The seven parameters of 𝜃 — engine, 𝑀 or nlist, efSearch or nprobe, 𝑘, 𝛼, 𝑛 refs , ref _strategy — govern the accuracy–throughput trade-off across both stages and constitute the search space Θ (Table 1). The oracle cost per call is dominated by index construction (≈425–455 s) and 600 batch queries (≈35–99 s), yielding ≈8–9 minutes of wall-clock time per oracle evaluation.
Definition 2 (Evaluation Oracle). For any configuration 𝜃 ∈ Θ, the evaluation oracle O (𝜃 ) = mAP(𝜃 ), QPS(𝜃 ) is obtained by (i) rebuilding the VDMS ANN index under 𝜃 , and (ii) issuing all 600 HICO-DET text queries to the live container and recording mAP (Section 3) and batch QPS. The oracle is black-box: no closed-form gradient with respect to 𝜃 is available, and each call requires a full index rebuild followed by 600 retrieval operations.
4
𝑀 nlist efSearch nprobe 𝑘
Card.
FaissFlat, FaissHNSWFlat, FaissIVFFlat 8, 16, 32, 48, 64 64, 128, 256, 512 32–500 (7 vals) 4, 8, 16, 32, 64 50–1000 (8 vals)
Total |Θ|
Problem Statement
𝜃 ∈Θ
—
Stage 2 — DINOv2 reranking fusion weight 𝛼 ref. count 𝑛 refs ref. strategy —
Definition 1 (Configuration Space). Let Θ denote the joint configuration space of the two-stage pipeline (Section 4). Θ is the Cartesian product of seven parameter domains (engine; 𝑀, nlist; efSearch, nprobe; 𝑘; 𝛼; 𝑛 refs ; ref _strategy) enumerated in Table 1; |Θ| = 102,144.1
𝜃 ∗ = arg max Score(𝜃 ),
Notation Value Set
Stage 1 — ANN retrieval engine conn. depth† (HNSW) cluster count† (IVF) search beam† (HNSW) probed cells† (IVF) candidate pool
This threshold-gated objective rewards throughput only within the feasible region (mAP ≥ 𝜏): a configuration that sacrifices retrieval quality below 𝜏 scores zero regardless of how high its QPS is, while within the feasible region higher QPS is strictly better. 𝜏=0.15 is calibrated as the minimum acceptable quality floor: it lies between UniIR’s mAP (0.095, failing) and the unoptimized CLIP baseline (0.208, passing), ensuring optimized configurations retain at least the quality of the simplest non-trivial retrieval system; rankings are stable for 𝜏 ∈ [0.15, 0.20], and below 0.15, 𝑘=50 configurations with mAP ≈ 0.13 — which found high throughput but lack the compensating 𝛼=0.80 — become spuriously feasible.
3.4
Parameter
Two-Stage Retrieval Pipeline Oracle Instantiation and Notation
Oracle O (Definition 2) maps any 𝜃 ∈ Θ to a (mAP, QPS) pair; the optimizer maximizes Score(𝜃 ) (Eq. (1)): QPS(𝜃 ) if mAP(𝜃 ) ≥ 𝜏, 1 The LLM agent also proposes a constraint_strategy metadata pre-filter ( ∈ {none,
object, verb, object_and_verb}) outside Θ; all methods sample the full space, but the LLM converges to object in 66% of trials and uses it in all best configurations.
4
- CLIP Indexing -
CLIP ViT-L/14 Encoder
HICO-DET Images (47,776)
VDMS HNSW Index (CLIP vectors)
- HOI Query Encoding -
CLIP ViT-L/14 Encoder
Top-k Candidates (k optimized)
DINOv2 ViT-L/14 Reranker
Final Ranked List
(no VDMS — on-the-fly)
“person riding a horse”
query image features (DINOv2)
Indexing Time
Search Time
Figure 1: Two-stage retrieval pipeline (CLIP ANN retrieval + DINOv2 reranking). 200 yield negligible recall improvement at substantially higher build cost, so it is excluded from Θ. FaissIVFFlat [11]: inverted-file index; search probes nprobe of nlist Voronoi cells (values in Table 1). The candidate pool size 𝑘 (8 values: 50–1000, Table 1) determines the recall ceiling available to Stage 2 and is selected jointly with engine parameters. Batch QPS is defined as |𝑄 | QPS(𝜃 ) = , (4) 𝑇total (𝜃 ) where |𝑄 |=600 is the number of HOI text queries and 𝑇total (𝜃 ) is the elapsed wall-clock time for all queries issued in a single batch to the live VDMS container under configuration 𝜃 . This end-toend latency encompasses the VDMS round-trip for Stage 1, the in-memory reranking computation in Stage 2, and all inter-process communication overhead.
4.3
range (≈1.55–1.75 for unit-normalized 768-d embeddings), a signal that a global corpus statistic would collapse. The fused relevance score for candidate 𝑗 ∈ T𝑘 (𝑞ℎ ) is the convex combination [5] 𝑠 𝑗 = (1 − 𝛼) 𝑐ˆ𝑗 + 𝛼 𝑑ˆ𝑗 , (7) where 𝛼 ∈ {0.0, 0.05, . . . , 0.90} (19 values); lower 𝑠 𝑗 indicates higher predicted relevance. Setting 𝛼=0 degenerates to CLIP-only retrieval; 𝛼=1 to DINOv2-only reranking. The convex combination is preferred over reciprocal rank fusion because CLIP text queries on a unit-normalized 768-d space produce meaningful intra-pool distance values whose fine-grained ordering is destroyed when converted to ranks. The optimal 𝛼 is configuration-dependent and is learned jointly with Stage 1 parameters by the LLM optimizer (Section 5).
4.4
Stage 2: Feature Fusion and Reranking
Stage 2 reranks T𝑘 (𝑞ℎ ) using DINOv2 ViT-L/14-reg4 [22] visual features — capturing fine-grained appearance structure complementary to CLIP’s language-aligned semantics — without constructing any additional index. Each HOI class ℎ is represented by a visual reference vector rℎ ∈ R1024 , built offline as the mean DINOv2 embedding of a selected subset Sℎ ⊆ Iℎ : 1 ∑︁ rℎ = d𝑖 , (5) |Sℎ |
Í Let 𝝁𝑐ℎ = | I1 | 𝑖 ∈ Iℎ c𝑖 denote the visual centroid of HOI class ℎ ℎ Í in the CLIP space, and let 𝜎ℎ2 = | I1 | 𝑖 ∈ Iℎ ∥c𝑖 − 𝝁𝑐ℎ ∥ 22 denote the ℎ intra-class visual variance. For any query vector q′ ∈ R768 , the bias–variance decomposition of the expected squared L2 distance to a class member c𝑖 drawn uniformly from Iℎ yields E𝑖 ∥q′ − c𝑖 ∥ 22 = ∥q′ − 𝝁𝑐ℎ ∥ 22 + 𝜎ℎ2 . (8) |{z} | {z }
𝑖 ∈ Sℎ
intra-class variance centroid bias2 2 The variance term 𝜎ℎ is an intrinsic property of the class; only the bias term ∥q′ − 𝝁𝑐ℎ ∥ 22 depends on the query.
where 𝑛𝑟 = |Sℎ | ∈ {1, 3, 5, 10}. For 𝑛𝑟 =1 (the default), rℎ is the DINOv2 embedding of a single image from the HOI category. For 𝑛𝑟 >1, three strategies populate Sℎ : first takes the initial 𝑛𝑟 images in dataset order; centroid selects the 𝑛𝑟 images nearest to the Í DINOv2 class mean 𝝁𝑑ℎ = | I1 | 𝑖 ∈ Iℎ d𝑖 , anchoring rℎ at the visual ℎ core of the class; diverse applies greedy farthest-point sampling from the most central image. Reference construction requires <1 ms per class and does not contribute to 𝑇total in Eq. (4). CLIP squared L2 distances 𝛿 𝑐𝑗 = ∥c 𝑗 − q𝑐 ∥ 22 and DINOv2 L2 distances 𝛿 𝑑𝑗 = ∥d 𝑗 − rℎ ∥ 2 operate at different absolute scales; both are min-max normalized within T𝑘 (𝑞ℎ ) onto [0, 1]: 𝑚 𝛿𝑚 𝑗 − min𝑖 ∈ T𝑘 𝛿𝑖 𝛿ˆ𝑚 = , 𝑚 ∈ {𝑐, 𝑑}, (6) 𝑗 max𝑖 ∈ T𝑘 𝛿𝑖𝑚 − min𝑖 ∈ T𝑘 𝛿𝑖𝑚 + 𝜀 where 𝜀 = 10−10 , 𝑐ˆ𝑗 ≡ 𝛿ˆ𝑐 , and 𝑑ˆ𝑗 ≡ 𝛿ˆ𝑑 . Pool-local normalization 𝑗
Text Query Superiority: Centroid-Displacement Theory
CLIP’s contrastive training objective aligns the text embedding of a category description with the visual embeddings of all matching images [23]. For a compositionally structured category — e.g., “a person riding a horse” spanning diverse viewpoints, backgrounds, and subjects — the text embedding aggregates distributional statistics from many matched images and therefore lies near the visual centroid: q𝑐 ≈ 𝝁𝑐ℎ . The centroid bias is thus approximately zero: E𝑖 ∥q𝑐 − c𝑖 ∥ 22 ≈ 𝜎ℎ2 . (9) In contrast, any reference image c 𝑗 (𝑗 ∈ Iℎ ) lies at centroid displacement 𝜀 𝑗 = ∥c 𝑗 − 𝝁𝑐ℎ ∥ 2 > 0, incurring a strictly positive bias: E𝑖 ∥c 𝑗 − c𝑖 ∥ 22 = 𝜀 2𝑗 + 𝜎ℎ2 > 𝜎ℎ2 . (10)
𝑗
preserves the fine-grained relative ordering of CLIP distances within T𝑘 (𝑞ℎ ): intra-pool CLIP squared L2 distances span a narrow
The text query therefore achieves lower expected squared distance to every class member than any single reference image, by a margin 5
5.2
that shrinks as q𝑐 approaches 𝝁𝑐ℎ . This centroid-displacement advantage is monotonically amplified by intra-class visual diversity: the higher the variance 𝜎ℎ2 across HOI images (e.g., ride horse spans many settings and viewpoints), the larger the typical 𝜀 𝑗 and hence the larger the margin over any individual reference image. Under L2-based ANN retrieval, lower expected squared distance to class members translates directly to higher expected recall at any fixed 𝑘, and therefore to higher mAP. This result formally justifies the design choice of text queries for Stage 1 and additionally motivates the 𝑛𝑟 >1 reference aggregation in Stage 2: averaging 𝑛𝑟 randomly drawn class embeddings reduces the expected 𝐿2 centroid bias of rℎ √ by 𝑂 (1/ 𝑛𝑟 ) relative to a single reference image. Section 6 confirms this empirically: text queries outperform single-image reference queries by +2.32 pp mAP on HICO-DET, consistent with Eq. (9).
5
Let ℎ𝑡 = 𝜃 𝑡 , mAP(𝜃 𝑡 ), QPS(𝜃 𝑡 ) denote the evaluation record at iteration 𝑡 and define the search history trajectory H𝑡 = ℎ 1, ℎ 2, . . . , ℎ𝑡 , 𝑡 ≥ 0, H0 = ∅. (12) H𝑡 is the agent’s complete internal state: no surrogate model, gradient estimate, or kernel is maintained between iterations. At each iteration the agent receives a single structured prompt built from four components: (i) fixed system context encoding the VDMS pipeline architecture and per-engine parameter physics; (ii) the diagnostic class 𝑔(H𝑡 −1 ) (Section 5.2.1); (iii) the complete ordered history H𝑡 −1 as structured text; and (iv) dynamically computed untried-value hints (Section 5.2.2). The agent responds with a structured JSON object specifying the next 𝜃 𝑡 and a reasoning field that articulates its rationale (Figure 3). The necessity of history conditioning is verified in Section 6.7.
LLM-Guided ANN Hyperparameter Optimization
5.2.1 Diagnostic Guidance Function Let Score𝑡∗ denote the best Score seen through iteration 𝑡. The diagnostic class
The pipeline of Section 4 exposes a seven-parameter joint configuration 𝜃 ∈ Θ: engine, two engine-conditional index parameters, pool size 𝑘, fusion weight 𝛼, reference count 𝑛𝑟 , and reference strategy. Because engine choice renders some parameters inapplicable—𝑀 and efSearch are undefined for IVF; nlist and nprobe are undefined for HNSW—the joint space |Θ|=102,144 is engine-conditional and mixed-type. Each evaluation requires a full VDMS rebuild plus a 600-query pass (≈8–9 min per oracle call), ruling out exhaustive or surrogateheavy strategies that require thousands of evaluations. We address this with a sequential LLM optimizer (Figure 2) that exploits structural knowledge of Θ to allocate its 𝑁 -iteration budget efficiently.
5.1
excellent low_mAP 𝑔( H𝑡 ) = low_QPS below_best moderate
if Score(𝜃 𝑡 ) ≥ 0.98 Score𝑡∗ , if mAP(𝜃 𝑡 ) < 0.20 and Score(𝜃 𝑡 ) < 0.80 Score𝑡∗ , if mAP(𝜃 𝑡 ) ≥ 0.20 and QPS(𝜃 𝑡 ) < 100,
(13)
if Score(𝜃 𝑡 ) < 0.90 Score𝑡∗ , otherwise,
is injected into the prompt as a targeted natural-language directive. Excellent redirects to unexplored structural regions in early phases and shifts to incremental parameter micro-adjustment in Fine-Tuning; low_mAP directs the agent to increase 𝑘; low_QPS directs it to reduce 𝑘 (the primary QPS lever) or switch engines; below_best directs a return toward the best-known configuration with targeted adjustments; and moderate suggests coordinated adjustment of 𝑘 and 𝛼. All threshold values were fixed by empirical calibration on HICO-DET before any optimizer comparison was run.
Search Space and Coupling Structure
The configuration space decomposes as Θ = ΘHNSW ∪ ΘIVF ∪ ΘFlat,
History Trajectory and Prompt Generation
(11)
5.2.2 Untried-Value Hints and Duplicate Avoidance During EXPLOITATION and FINE-TUNING, the prompt additionally provides U𝑡 : untried 𝛼 values and (𝑛𝑟 , ref_strategy) pairs under the current best engine, derived from H𝑡 . These hints prevent the agent from re-proposing already-evaluated continuous-parameter combinations once the structural choice is settled. Each proposed 𝜃 𝑡 +1 is checked against H𝑡 before evaluation; if a duplicate is detected, the LLM is re-queried up to four times with the same prompt (a different response may arise from the model’s sampling temperature); if all four attempts remain duplicates, a uniformly random fallback is applied, guaranteeing loop progress. Each LLM call consumes approximately 2,000–5,000 tokens (growing with history; for larger 𝑁 , top-𝑘 history retention is a natural mitigation); the full 50-iteration run costs fewer than 250,000 tokens (<$0.20), negligible relative to the 6.5–7.5 h oracle cost.
where each component is defined by its engine-conditional index parameters together with shared parameters 𝑘, 𝛼, 𝑛𝑟 , and ref_strategy (full value sets in Table 1). FaissHNSWFlat contributes 𝑀 and efSearch; FaissIVFFlat contributes nlist and nprobe. The oracle O (𝜃 ) loads 𝜃 into a fresh VDMS instance, indexes the corpus, and evalu ates all |𝑄 |=600 HOI queries, returning the pair mAP(𝜃 ), QPS(𝜃 ) and Score(𝜃 ) = QPS(𝜃 ) if mAP(𝜃 )≥𝜏, else 0 (Eq. (1)). Two cross-parameter couplings invalidate the independence assumption shared by GP-BO [26] and Optuna TPE. First, pool size 𝑘 determines the candidate set passed from Stage 1 to Stage 2; increasing 𝑘 raises Stage 2 mAP recall but reduces throughput, so the joint (𝑘, 𝛼) frontier is not separable: optimizing 𝛼 at a suboptimal 𝑘 cannot reach the Score achievable by joint search. Empirically, at 𝑘=50 (QPS ≈ 295) the SIEVE Score collapses to 0 unless 𝛼 is simultaneously raised to compensate for reduced Stage-1 recall; the optimal 𝑘=50, 𝛼=0.80 configuration (Score ≈ 300) is reachable only through joint search—independent 𝛼-optimization at any fixed 𝑘 misses it. Second, engine selection sets the QPS range available to any efSearch sweep, so structural decisions must precede continuous refinement. These coupled dependencies motivate a phase-partitioned budget that resolves structural choices before continuous ones.
5.3
Phase-Aware Budget Partition
The 𝑁 -iteration budget is divided into three phases with boundaries
6
𝑡 exp = round(0.40 𝑁 ),
(14)
𝑡 expl = round(0.75 𝑁 ),
(15)
Hyperparameter Optimization Loop
Benchmarks SIFT1M
1,000,000 vectors 128-dim · L2 metric ANN benchmark
VDMS (Apptainer container)
Optimization Methods vectors LLM Agent
GLDv2
MiniMax-M2.1 (OpenRouter) Dedup guard · 4 retries Phase-aware prompting
1,580,470 vectors · 512-dim images IP metric · Landmark retrieval
⊕ Optuna TPE
GP-BO
Score = QPS if Recall@10 ≥ 0.90 else 0
Gaussian Process Bayesian Optim. Exp. Improvement
Vector Store (batch ≤ 50 per AddDescriptor)
Bayesian (Tree Parzen Est.) Direction: maximize
⊞ Grid Search
past (config, score) pairs
FaissFlat (exact · recall ceiling)
∼ Random Search
50 principled configs Deterministic
Uniform sampling
Score = QPS if mAP ≥ 0.20
FaissHNSWFlat ★ M · efC · efS
else 0
index config
HICO-DET
FaissIVFFlat (nlist / nprobe)
47,776 images · 600 HOI classes
CLIP ViT-L/14 + DINOv2 rerank
images Score = QPS if mAP ≥ 0.15 else 0
≡ Config Proposal
index + queries
{ engine: FaissFlat | FaissHNSWFlat } M ∈ [8,64] · efConstruction ∈ [100,500] · efSearch ∈ [16,512]
▶ Benchmark Evaluation
iteration i
① Build ANN index (batch=50) ② Run Nq queries ③ Measure QPS + Recall@10 / mAP
Legend ⊕ Adaptive optimizer ⊞ ∼ Baseline optimizer VDMS / system component Score feedback (LLM, Optuna, GP-BO)
↗ EXPLORATION (iter 1–20)
⊙ EXPLOITATION (iter 21–38)
∇ FINE-TUNING (iter 39–50)
feedback (Optuna, GP-BO)
(phase-aware adaptive prompting — LLM only)
★ Best Configuration
best result
argmax Score over all 50 iterations
◆ Score (SIEVE) QPS if metric ≥ τ else 0
feedback (LLM)
Figure 2: Optimization framework: five methods over 𝑁 =50 iterations (adaptive methods receive score feedback). Algorithm 1 LLM-Guided ANN Optimizer
defining Exploration (1 ≤ 𝑡 ≤ 𝑡 exp ), Exploitation (𝑡 exp < 𝑡 ≤ 𝑡 expl ), and Fine-tuning (𝑡 expl < 𝑡 ≤ 𝑁 ) — for 𝑁 =50: 20, 18, and 12 iterations respectively. The 0.40 boundary allocates sufficient budget to survey all three engine families before committing to a structural choice; the 0.75 boundary reserves 25% of the budget for Fine-Tuning, enough to sweep the 19 discrete 𝛼 values under the best-identified engine. The boundaries encode the structural-beforecontinuous principle: engine and 𝑀 determine the achievable QPS ceiling and must be resolved before 𝛼 and 𝑛𝑟 are refined—a conditional dependency that Optuna TPE, which models each parameter independently via univariate kernel density estimation, cannot exploit. The benefit of imposing this ordered structure over unphased search is quantified in the phase structure ablation in Section 6.7.
Require: Configuration space Θ; oracle O; budget 𝑁 Ensure: Best configuration 𝜃 ∗ and history H𝑁 1: H0 ← ∅; Score∗ ← 0; 𝜄 ← 0 2: 𝑡 exp ← round(0.40𝑁 ); 𝑡 expl ← round(0.75𝑁 ) ⊲ Eqs. (14)–(15) 3: for 𝑡 = 1, 2, . . . , 𝑁 do 4: 𝜙𝑡 ← Phase(𝑡, 𝑡 exp , 𝑡 expl ) ⊲ Expl. / Exploit. / Fine-Tuning 5: 𝑔𝑡 ← 𝑔 ( H𝑡 −1 ) ⊲ Eq. (13) 6: U𝑡 ← UntriedHints( H𝑡 −1 , 𝜙𝑡 ) ⊲ ∅ during EXPLORATION 7: 𝑐𝑡 ← ConsecAlphaOnly( H𝑡 −1 ) ⊲ anti-collapse count 8: for attempt = 1, . . . , 4 do ⊲ re-query; temp. diversity 9: 𝑝𝑡 ← BuildPrompt(𝜙𝑡 , 𝑔𝑡 , H𝑡 −1 , U𝑡 , 𝑐𝑡 , 𝜄 ) 10: 𝜃 𝑡 ← LLM(𝑝𝑡 ) ⊲ structured JSON response 11: if 𝜃 𝑡 ∉ {𝜃𝑠 : 𝑠 < 𝑡 } then 12: break ⊲ novel proposal accepted 13: end if 14: end for 15: if 𝜃 𝑡 ∈ {𝜃𝑠 : 𝑠 < 𝑡 } then ⊲ all 4 attempts duplicated 16: 𝜃 𝑡 ← RandomConfig(Θ) 17: end if 18: mAP𝑡 , QPS𝑡 ← O (𝜃 𝑡 ) 19: H𝑡 ← H𝑡 −1 ∪ { (𝜃 𝑡 , mAP𝑡 , QPS𝑡 ) } 20: 𝑠𝑡 ← QPS𝑡 · 1[mAP𝑡 ≥ 𝜏 ] ⊲ SIEVE Score (Eq. (1)), 𝜏=0.15 21: if 𝑠𝑡 > Score∗ then 22: Score∗ ← 𝑠𝑡 ; 𝜃 ∗ ← 𝜃 𝑡 ; 𝜄 ← 0 23: else 24: 𝜄 ←𝜄+1 25: end if 26: end for 27: return 𝜃 ∗ , H𝑁
Exploration (1 ≤ 𝑡 ≤ 20). Diversity-focused engine coverage: the excellent directive instructs the agent to sample structurally distinct (engine, 𝑀, 𝑘) combinations; the U𝑡 hint set is suppressed to prevent premature convergence. Exploitation (21 ≤ 𝑡 ≤ 38). U𝑡 is activated; excellent shifts to reinforcing the best-engine region while broadening (𝛼, 𝑛𝑟 ) coverage on the identified Pareto frontier. Fine-tuning (39 ≤ 𝑡 ≤ 50). Hints narrow to unexplored referencestrategy variants; proposals are restricted to coordinate-wise microadjustments of the current best (efSearch, 𝑛𝑟 ) for latency-targeted gains.
structural diversity an explicit part of the agent’s reasoning: an external hard-rejection filter would silently discard proposals and retry, consuming LLM calls without informing the model why its proposal was rejected; the prompt-embedded rule requires the agent to reason about and justify a structurally distinct alternative.Algorithm 1 gives the complete procedure. GP-BO uses GPSampler (Gaussian Process surrogate with Expected Improvement acquisition [26]) applied to our singleobjective Score. Optuna TPE fits univariate kernel density estimators over each parameter independently. Neither baseline encodes
Anti-collapse and stagnation constraints. Let 𝑐𝑡 count consecutive proposals in which only 𝛼 changed relative to the preceding configuration, and let 𝜄𝑡 count iterations elapsed without improvement to Score𝑡∗ . Two hard rules are encoded in the system prompt: (1) if 𝑐𝑡 ≥ 2, the next proposal must differ structurally—by changing engine, 𝑀, 𝑘, or 𝑛𝑟 —before any further 𝛼 refinement; (2) if 𝜄𝑡 ≥ 5, a stagnation escape directive overrides the current phase instruction and reverts to broad-coverage exploration. Embedding these rules in the prompt rather than as external rejection filters makes 7
HOI Query
HOI Query
Person: holding
Person: riding
Object: bottle
Object: horse
HOI Query ···
× 600
LLM User Message You are an expert VDMS Optimization Engine for HOI Retrieval. Iteration: 4 / 50 | OBJECTIVE: Score = QPS
if
Phase: EXPLORATION | Best Score so far: 136.1 mAP ( =0.15) else 0.0 maximize QPS s.t. mAP
Parameter Physics (190-line system context
0.15
key constraints summarised):
k_neighbors QPS (primary lever: k=50 295, k=100 145, k=200 58 QPS) efSearch: QPS-neutral in VDMS batch mode · M mAP ceiling · : discover empirically IVFFlat + constraint_strategy 'none' 0 candidates returned (always infeasible)
Diagnostic g( ) :
Class = low_mAP
[ mAP( ) = 0.000 < 0.20 and Score( ) = 0.0 < 0.80 × Score* = 108.9 ] IVFFlat returned mAP=0.000 (infeasible). Return to FaissHNSWFlat. Increase k or M.
History
(complete ordered trajectory
no truncation):
iter01:
HNSW M=64 efS=64 k=100 Score=136.1 mAP=0.209
=0.65 cs=object QPS=136.1
refs=10+centroid
iter02:
IVFFlat nlist=128 nprobe=8 k=100 =0.65 cs=none Score= 0.0 mAP=0.000 QPS=2267.6 (mAP <
refs=10+centroid infeasible)
iter03:
IVFFlat nlist=512 nprobe=64 k=100 =0.30 cs=none Score= 0.0 mAP=0.000 QPS=2281.6 (mAP <
refs=5+first infeasible)
Untried-value hints U :
Active during Exploitation & Fine-Tuning only (suppressed in Exploration phase). Each checked against ; duplicates trigger resubmission or random fallback.
MiniMax-M2.1 via OpenRouter
Structured JSON Response Iteration 4: { "engine": "FaissHNSWFlat", "params": {"M": 64, "efConstruction": 200, "efSearch": 64}, "k_neighbors": 50, "alpha": 0.70, "n_refs": 10, "ref_strategy": "centroid", "constraint_strategy": "object", new field (PMGD graph pre-filter) "reasoning": "k=50 QPS 295; M=64+cs=object keeps mAP =0.15." }
Figure 3: LLM prompt structure (parameter physics, history H𝑡 −1 , and phase-conditioned hints). the (𝑘, 𝛼) coupling or the engine-conditional structure of Θ; the LLM agent exploits both through its phase-partitioned prompt and guidance function. Unlike 𝜆-Tune [9], which chains LLM calls to translate a fixed workload description into knob recommendations, our agent maintains a live score trajectory H𝑡 and adapts its proposals iteration-by-iteration under the SIEVE constraint.
analogous categorical difficulty signal and retain their standard objectives. The LLM Agent + QDS best configuration uses FaissIVFFlat (nlist=128, 𝑘=500), achieving 1,362.0 QPS on average (3-seed mean). This throughput is specific to HICO-DET’s 47,776-image corpus: at this scale VDMS scans all nlist Voronoi cells during IVFFlat search regardless of nprobe, making IVFFlat equivalent to an exact flat scan over the quantized index. With only ≈373 vectors per cell on average, the total scan is fast enough to sustain > 1,000 QPS — a corpus-size effect that disappears at million-vector scale, where the same VDMS behavior collapses IVFFlat Score to ≈35 (verified on SIFT1M, which is why IVFFlat is excluded from the SIFT1M search space). The QDS result therefore characterizes the optimizer’s ability to discover non-obvious high-throughput configurations at 47K scale, not a general IVFFlat recommendation.
5.3.1 Difficulty-Weighted Objective (QDS) Because HOI queries vary widely in retrieval difficulty—some verb–object categories have hundreds of positive images while others have fewer than five—a uniform mAP treats all queries equally despite this imbalance. We therefore optionally upweight structurally hard queries via a Query Difficulty Score. For query 𝑗: gallery entropy Í 𝐻 𝑗 =− 𝑖 𝑝 𝑗𝑖 log 𝑝 𝑗𝑖 (𝑝 𝑗𝑖 =softmax(f𝑞⊤𝑗 f𝑑𝑖 /𝛽), 𝛽=0.01) measures the spread of CLIP mass over the gallery; centroid proximity CPR 𝑗 =f𝑞⊤𝑗 𝝁ˆ 𝑗 (cosine similarity to the unit-normalised visual centroid 𝝁ˆ 𝑗 ) measures how well the text query aligns with its HOIg 𝑗 (min-max normalised), class centroid. We define QDS 𝑗 =𝐻˜ 𝑗 −CPR g weight 𝑤 𝑗 =1+QDS 𝑗 ∈[1, 2], and replace the standard mAP with Í Í QDS-mAP= 𝑗 𝑤 𝑗 AP 𝑗 / 𝑗 𝑤 𝑗 as the oracle signal. QDS performance is reported in Table 3 and Table 4. QDS applies exclusively to HICO-DET: its semantic verb–object category structure admits per-query difficulty decomposition; GLDv2 (landmark-instance ground truth) and SIFT1M (exact 𝑘-NN ground truth) provide no
6 6.1
Experiments Experimental Setup
Datasets. We evaluate on three benchmarks. HICO-DET (Section 3): 47,776 images, 600 compositional text queries, 90,641 ground-truth positive pairs — a stringent testbed combining fine-grained HOI categories, long-tailed relevance distributions, and a cross-modal (text-to-image) retrieval setting. GLDv2 [32]: Google Landmarks Dataset v2, a large-scale image-to-image retrieval benchmark with 762,000 gallery images and 1,129 query 8
Best SIEVE Score (QPS)
images from the retrieval challenge evaluation split, spanning global landmarks, providing a 16× larger corpus for cross-domain generalization evaluation. SIFT1M [10]: a standard ANN benchmark of one million 128-dimensional SIFT feature vectors with 10,000 queries and exact nearest-neighbor ground truth (evaluated at Recall@10), providing a complexity-matched control — the index space reduces to a single-stage HNSW problem with no cross-stage coupling — to test whether the LLM’s advantage is specific to coupled search spaces. For HICO-DET, image embeddings are 768-dimensional CLIP ViT-L/14 vectors. For GLDv2, each image is represented by a 768-dimensional CLIP ViT-L/14 embedding and a 1024-dimensional DINOv2 ViT-L/14-reg4 embedding, both stored in VDMS. For SIFT1M, raw 128-dimensional SIFT vectors are stored directly in VDMS without any reranking stage; Score = QPS if Recall@10 ≥ 𝜏=0.90, else 0. Compared Methods. We compare five strategies, each for 𝑁 =50 iterations. LLM Agent (Ours): phase-conditioned prompt with full history H𝑡 −1 via MiniMax-M2.1/OpenRouter2 (seeds: 42, 99, 200). Backbone portability across GPT-4o-mini and Llama-3.3-70B is verified in Section 6.7. GP-BO: optuna.samplers.GPSampler (Gaussian Process surrogate, Expected Improvement [26]) (seeds: 42, 99, 200). Optuna TPE [1]: univariate kernel density estimators (seeds: 42, 99, 200). Random Search [3]: uniform random sampling (seeds: 42, 99, 200). Grid Search: lexicographic 1-D sweep (1 run). As system baselines we include UniIR [31], standalone CLIP ViT-L/14, standalone DINOv2 ViT-L/14-reg4, and CLIP+DINOv2 reranking, all under exact FaissFlat search (Section 6.2). Metrics. Retrieval quality: mAP, P@10, Recall@10 [20]. Throughput: batch QPS. Joint optimization objective: Score = QPS if mAP ≥ 𝜏 else 0, 𝜏=0.15 (Eq. (1)); for SIFT1M, Score = QPS if Recall@10 ≥ 𝜏 else 0, 𝜏=0.90. Implementation. All experiments run on the NCSA Delta GPU cluster (partition gpuA40x4): one NVIDIA A40 GPU (48 GB VRAM), 16 CPU cores (AMD EPYC 7763), and exclusive node access. VDMS is deployed as an Apptainer container (v2.2.1); all retrieval and optimization code is implemented in Python 3.12. For Optuna, we use the official optuna library [1] with the default TPE sampler. Each optimization iteration builds a complete ANN index from scratch and issues all 600 queries in a single batch, requiring approximately 8–9 minutes of wall-clock time. The full 50-iteration budget therefore demands approximately 6.5–7.5 hours of compute per method, making sample efficiency — how quickly each method reaches a high-scoring configuration — a practically meaningful criterion. 𝑁 =50 covers 0.05% of |Θ| (e.g., 63,840 HNSW configs on HICO-DET), totals ≈6.5–7.5 h of wall-clock compute per method, and matches the protocol of related LLM-guided and Bayesian optimization studies [14, 33].
6.2
iter 23 (LLM peak)
300 250 200 150 100 50 0
LLM Agent (Ours) Optuna TPE
5
10
15
20
25
Iteration
GP-BO Random Search
30
35
Grid Search
40
45
50
Figure 4: SIEVE Score vs. iteration on HICO-DET (±1𝜎; 3 seeds except Grid). mAP is 0.2763 — reflecting fine-grained categories such as ride horse, sit_on horse, and straddle horse that share the same object but differ only in verb. UniIR comparison. UniIR’s fine-tuned CLIP-SF achieves mAP = 0.095 in VDMS — a 2.5× drop from zero-shot CLIP (0.240; Table 2) — because multi-task fine-tuning on COCO-style pairs disrupts HOI semantic alignment for compositional queries. Under the SIEVE metric, all UniIR variants score 0 (mAP < 𝜏=0.15), failing to meet the minimum quality threshold; our unoptimized CLIP baseline already achieves SIEVE Score = 8.21 (QPS at 𝑘=500; Table 2). Modality complementarity and throughput. DINOv2 alone (mAP = 0.217) underperforms CLIP alone (0.240) by 2.3 pp, confirming language-aligned embeddings are essential; weighted fusion at 𝛼=0.5 achieves mAP = 0.276 (+3.65 pp, +15.2% relative) at only 0.6% extra latency. Reducing 𝑘 from 2000 to 500 raises QPS by 7.0× at 3.2 pp mAP cost, yielding a 7.0× Score gain — the non-linear asymmetry the LLM agent exploits by conditioning each proposal on the full history of observed outcomes.
6.3
End-to-End Optimization Performance
The LLM agent achieves mean Score = 300.3, outperforming every baseline: Optuna TPE (225.2, +33.3%), VDTuner [34] (223.8, +34.2%), GP-BO (199.3, +50.7%), Grid Search (183.0, +64.1%), and Random Search (111.4). This corresponds to a 15.3× throughput gain over UniIR (QPS = 19.67; SIEVE Score = 0 since mAP < 𝜏) and a 36.6× gain over the unoptimized CLIP FaissFlat reference (Score = 8.21). The LLM-optimal configuration — FaissHNSWFlat, 𝑀=64, efSearch=64, 𝑘=50, 𝛼=0.80, 𝑛𝑟 =5, centroid (Table 4) — achieves QPS = 300.5 while maintaining mAP = 0.160 just above the feasibility threshold 𝜏. The mAP of 0.160 falls below the unoptimized CLIP baseline (0.208) by design: SIEVE rewards throughput within the feasible region, so the agent trades excess mAP margin for higher QPS rather than maximizing mAP. GP-BO comparison. GP-BO achieves mean Score = 199.3, trailing the LLM agent by 50.7%, but with extreme seed-to-seed variance: best-scores span 110–299 across three seeds versus the LLM agent’s 299.9–300.5 (< 0.3% spread). The cause is structural: GP-BO’s smooth Gaussian Process surface cannot model the discontinuous feasibility boundary at 𝜏 — when some 𝑘=50 trials are feasible (Score ≈ 300) and others score 0 due to an insufficient 𝛼, the GP averages these into an expected value of ≈ 150 with high uncertainty, making 𝑘=50 appear less attractive than the consistently
System Baselines
Compositional difficulty. Even with exhaustive FaissFlat search and single-query DINOv2 reranking (𝑛 refs =1), the best achievable
3With pseudo-relevance feedback (𝑛
2 Estimated API cost: 50 iterations × ≈5,000 tokens/call ≈ 250K tokens total; at MiniMax-
refs =10), the FaissFlat oracle reaches mAP = 0.426 at 𝛼=0.80, confirming PRF and index tuning are complementary optimization axes.
M2.1 list pricing this amounts to under $0.20 per optimizer run. 9
Table 2: System baselines on HICO-DET (FaissFlat exact search; Score = QPS if mAP ≥ 𝜏, 𝜏=0.15). Method CLIP ViT-L/14 CLIP ViT-L/14 DINOv2 ViT-L/14-reg4 CLIP + DINOv2 rerank
𝛼
𝑘
mAP
P@10
R@10
Score (QPS)
0.0 0.0 1.0 0.5
500 2000 2000 2000
0.20787 0.23978 0.21655 0.27629
0.3370 0.3370 0.3232 0.3812
0.0967 0.0967 0.0800 0.1138
8.21★ 1.17 1.16 1.15
★ Optimizer reference baseline (SIEVE Score = 8.21, i.e. QPS since mAP ≥ 𝜏 ); 𝑘=500 trades mAP for higher QPS vs. 𝑘=2000.
Table 3: ANN optimizer comparison on HICO-DET (50 iters, 3-seed means; Grid: 1 run). ¶ GPSampler+EI [26]; § difficulty-weighted; † corpus-scale artifact (see Section 5.3.1); ‡ SIEVE Score = 0 (mAP < 𝜏=0.15); raw QPS = 19.67 / 19.46 respectively. Method
Score (QPS)↑
mAP↑
P@10↑
Iters→Best
0‡
UniIR CLIP-SF Large, image-only [31] UniIR CLIP-SF Large, multi-modal [31] Grid Search Random Search [3] Optuna TPE [1] GP-BO ¶ VDTuner [34]
0‡ 183.0 111.4 225.2 199.3 223.8
0.095 0.089 0.178 0.179 0.164 0.178 0.172
0.448 0.442 0.508 0.455 0.465 0.482 0.479
— — 41/50 29/50 41/50 39/50 24/50
LLM Agent (Ours) LLM Agent + QDS (Ours)§†
300.3 1362.0
0.161 0.175
0.438 0.292
23/50 39/50
Table 4: System efficiency on HICO-DET (47,776 images, 600 queries; best-configuration per representative seed; 3-seed mean Scores in Table 3). § difficulty-weighted; † corpus-scale artifact (see Section 5.3.1). Configuration
Engine
𝑘
Build(s)↓
CLIP(ms)↓
DINO(ms)↓
Lat(ms)↓
QPS↑
UniIR [31] (𝑘=10) CLIP-SF Large, image-only CLIP-SF Large, multi-modal
FaissFlat FaissFlat
10 10
427.6 427.7
50.8 51.4
— —
50.8 51.4
19.67 19.46
Exact-search baselines CLIP only (𝛼=0) CLIP only (𝛼=0) DINOv2 only (𝛼=1) CLIP+DINOv2 (𝛼=0.5)
FaissFlat FaissFlat FaissFlat FaissFlat
500 2000 2000 2000
433.3 431.2 432.9 430.1
121.0 850.0 851.6 855.2
0.03 0.09 5.18 4.82
121.1 850.1 856.8 860.0
8.21 1.17 1.16 1.15
Optimizer best configurations Grid Search Random Search [3] Optuna TPE [1] VDTuner [34] LLM Agent (Ours) LLM Agent + QDS (Ours)§†
HNSW 𝑀=64 HNSW 𝑀=8 HNSW 𝑀=16 HNSW 𝑀=64 HNSW 𝑀=64 IVFFlat 𝑛𝑙=128
100 100 50 50 50 500
13.2 11.7 13.3 13.4 12.6 17.2
5.40 6.95 3.19 3.23 3.21 0.35
0.04 0.13 0.11 0.08 0.08 0.34
5.46 7.13 3.33 3.35 3.33 0.74
183.0 140.2 300.7 298.9 300.5 1357.6
feasible 𝑘=100 region (Score ≈ 145). Without natural-language reasoning to diagnose why 𝑘=50 fails, GP-BO cannot locate the joint (𝑘, 𝛼) optimum; a constrained-BO variant would better model the 𝜏 cliff but not the cross-stage (𝑘, 𝛼) coupling, which requires joint history-conditioned reasoning.
final mean Score (225.2) falls below the LLM agent’s iteration-6 Score — a structural deficit, not a sample-size one. Convergence and sample efficiency. Figure 4 plots best-Scoreso-far mean ±1𝜎 curves. The LLM agent reaches Score = 295.4 at iteration 6, already exceeding Grid Search’s final mean (183.0) and well ahead of Optuna TPE (mean = 148.5) and Random Search (79.4) at the same iteration. A Bartlett variance test (𝑝=1.6×10−4 ) confirms the reliability gap: LLM 𝜎=0.38 versus Optuna TPE 𝜎=65.4 (bimodal: two seeds at Score ≈ 188, one at ≈ 301).
Optuna TPE comparison. Optuna TPE achieves Score = 225.2, trailing the LLM agent by 33.3% — the largest consistent gap among adaptive baselines. The cause is structural: TPE fits independent kernel density estimators over each parameter, so it cannot discount the FaissIVFFlat region without a joint conditional model over engine type; all three seeds encounter FaissIVFFlat configurations (mAP = 0, Score = 0) within the first ten iterations, biasing the KDE mass toward the infeasible high-throughput region. The phase-structured budget directly counters this by resolving engine selection during Exploration (iterations 1–20) before any continuous (𝛼, 𝑛𝑟 ) refinement — the same separation whose removal costs −0.3% in the phase ablation (Table 8), confirming it as the mechanism rather than a coincidence. As a result, Optuna TPE’s
Non-obvious operating point. Under the SIEVE objective, the optimal region is 𝑘=50: FaissHNSWFlat with 𝑘=50 achieves QPS ≈ 300 but mAP sits at the edge of feasibility (≈ 0.16, just above 𝜏=0.15). This boundary-adjacent optimum is structurally non-trivial: without the compensating combination of 𝛼=0.80 and 𝑛𝑟 =5 centroid references, 𝑘=50 falls below 𝜏 and scores 0. The LLM agent locates this joint (𝑘, 𝛼, 𝑛𝑟 ) operating point because its history-conditioned reasoning explicitly tracks the coupling: after observing that 𝑘=100 configurations are feasible at moderate 𝛼, it probes 𝑘=50 with compensating 𝛼 — an inference that Optuna 10
Table 5: Optimizer comparison on GLDv2 (762K gallery, 1,129 queries, 50 iterations, 3-seed means; Grid: 1 run). Method
Score (QPS)↑
mAP↑
P@10↑
QPS↑
Iters→Best
GP-BO [26] Random Search [3] Optuna TPE [1] Grid Search VDTuner [34]
56.57 265.47 272.35 272.98 273.91
0.1892 0.1594 0.1610 0.1615 0.1593
0.2235 0.1949 0.1963 0.1965 0.1953
56.57 265.47 272.35 272.98 273.91
35/50 30/50 37/50 6/50 43/50
LLM Agent (Ours)
271.45
0.1613
0.1961
271.45
23/50
Table 6: Optimizer comparison on SIFT1M (1M vectors, 10K queries, 50 iters, 3-seed means; Grid: 1 run; Score = QPS if Recall@10 ≥ 0.90). Method
TPE’s independent marginals over 𝑘 and 𝛼 cannot make, explaining why the LLM reaches the 𝑘=50 feasible region on all three seeds while GP-BO succeeds on only one. Across all 188 logged 𝑘=50 configurations, 𝛼 <0.70 yields only 46.2% feasibility (mean Score = 129.1) versus 84.2% at 𝛼≥0.70 (mean Score = 242.6) — a 113-point gap marginal optimizers cannot reliably navigate. Failure modes. Grid Search reaches its best Score of 183.0 at iteration 41 — well below the LLM optimum (300.3) — because its one-dimensional lexicographic sweep cannot jointly probe the (𝑘=50, 𝛼=0.80) operating point: sweeping 𝑘 and 𝛼 independently misses the coupled interaction that elevates Score from 183 to 300, even though M = 64 is individually explored from iteration 5.Random Search’s mean Score (111.4) reflects frequent infeasible trials (score = 0); the feasible configurations it finds tend toward large-𝑘 safe regions (Score ≈ 140) rather than the tight 𝑘=50 optimum that requires coupled (𝑘, 𝛼) inference to locate reliably.
6.4
Score (QPS)↑
Recall@10↑
Iters→Best
GP-BO [26] Random Search [3] VDTuner [34] Optuna TPE [1] Grid Search
755.4 1143.6 1150.9 1160.7 1174.4
0.9342 0.9268 0.9290 0.9120 0.9146
18/50 32/50 32/50 24/50 5/50
LLM Agent (Ours)
1184.5
0.9069
21/50
the GP surface predicted it lay in a risky low-mAP zone given nearby 𝑘=50 trials with Score = 0. Sample efficiency. Grid Search finds its best Score at iteration 6 (3seed mean); LLM Agent converges at iteration 23, VDTuner at iteration 43. The compressed performance landscape at 762K scale means high-QPS HNSW configurations are accessible to all methods, and the LLM’s history-conditioned reasoning confers no systematic advantage — consistent with the complexity-proportional prediction.
6.5
Complexity-Matched Validation: SIFT1M
SIFT1M [10] (1M 128-d SIFT vectors, 10,000 queries) provides a complexity-matched control: the optimizer tunes only 𝑀, efConstruction, efSearch, and 𝑘 — four uncoupled HNSW parameters with no reranking stage or fusion weight 𝛼. We evaluate five methods (LLM Agent, Optuna TPE, Grid, Random, and GP-BO) under the same 50-iteration budget. Overall performance. All methods except GP-BO converge to FaissHNSWFlat with efSearch∈ {24, 32} as the optimal operating point. 3-seed mean Scores are 1184.5 (LLM Agent), 1174.4 (Grid, 1 run), 1160.7 (Optuna), 1150.9 (VDTuner), and 1143.6 (Random) — a spread under 3.6%. GP-BO collapses to a 3-seed mean of 755.4 (−36.2% vs. LLM Agent): its acquisition surface converges on 𝑘=30 configurations that maximize raw throughput but oscillate around the 𝜏=0.90 recall threshold, with high seed-to-seed variance (471–1174) and no recovery across the remaining budget. Complexity-proportional advantage. The result confirms the paper’s central theoretical prediction: the LLM agent’s advantage is complexity-proportional.On HICO-DET, the 7-parameter coupled space (efSearch–𝑘–𝛼 interaction across stages) rewards historyconditioned architectural reasoning; removing the cross-stage coupling, as on SIFT1M, eliminates the advantage because the residual 4-parameter ANN search space (engine, 𝑀, efSearch, 𝑘) is shallow enough for random and grid search to locate the optimum within the same 50-iteration budget. SIFT1M has no 𝛼 parameter, so cross-stage coupling is absent by construction; the remaining 𝑀–efSearch interaction is monotone (𝜌 (𝑀, Score)=+0.287; 𝜌 (efSearch, Score)=+0.243) and BO-tractable without joint historyconditioned reasoning. Notably, sample-efficiency rankings invert: Grid Search reaches its best Score at iteration 5 vs. iteration 21 for the LLM agent, confirming that architectural history confers no navigational advantage on the uncoupled 4-parameter space.
Cross-Domain Generalization: GLDv2 Landmark Retrieval
We evaluate on GLDv2 [32] (762,000 gallery images, 1,129 queries) to test cross-domain generalization. The corpus is 16× larger than HICO-DET, the retrieval task shifts from text-to-image to imageto-image (removing the text-query centroid advantage), and each image is represented by CLIP ViT-L/14 and DINOv2 ViT-L/14-reg4 embeddings in VDMS under the same SIEVE objective (Score = QPS if mAP≥ 𝜏=0.15, else 0). Overall performance. With 3-seed means, VDTuner leads at 273.91 QPS, narrowly ahead of Grid Search (272.98), Optuna TPE (272.35), LLM Agent (271.45), and Random Search (265.47). The spread among the top-4 adaptive methods is under 1% — substantially compressed relative to HICO-DET (33.3%) — consistent with the lower coupling on this dataset. The LLM Agent’s fourth-place VDMS result (271.45) falls within this < 1% convergence band and is consistent with 3-seed measurement variance; it is not a structural loss. GP-BO collapses to a 3-seed mean of 56.6 QPS, replicating the cliff failure from HICO-DET. Each iteration completes in ≈5 min (≈4 h total), slightly faster than HICO-DET. GP-BO cliff effect on GLDv2. GP-BO achieves a 3-seed mean of 56.6 QPS (vs. ≈ 273 for the top methods), the same structural failure as on HICO-DET but more acute. The SIEVE objective imposes a hard feasibility cliff at 𝜏=0.15: configurations with mAP just below this threshold score 0 regardless of QPS. GP-BO’s smooth Gaussian Process surface cannot model this discontinuity — it interpolates over the cliff, assigning moderate expected value to boundary-adjacent configurations instead of committing to the high-QPS side. As a result, GP-BO locked onto 𝑘=150 for 40 of 50 iterations (maximizing mAP at the cost of QPS), never probing 𝑘=100 — a high-scoring setting (mAP≈0.165, QPS≈273) — because
6.6
Cross-System Generalizability (Milvus)
To verify that results are not VDMS-specific, we replicate all five methods on Milvus [30] under identical protocols (50 iterations, 3 seeds, same dataset splits and 𝜏 thresholds, zero code changes 11
to the optimizer). VDMS’s PMGD backend incurs per-query property-graph overhead that caps HNSW throughput near 300– 400 QPS at 47K-vector scale; Milvus’s GPU-accelerated engine sustains 7,000+ QPS on the same corpus. Table 7 therefore reports within-system relative rankings — the comparison tests optimizer transferability, not cross-backend QPS parity — and the LLM agent ranks first on all three datasets across all nine seeds. On HICO-DET the margin over VDTuner is +7.0% (7272 vs. 6798). On GLDv2 the margin widens to +13.3% (6375 vs. 5624): Milvus’s ≈20× higher absolute QPS at 762K scale reinstates (𝑘, 𝛼) coupling absent in the lower-QPS VDMS parameter landscape. GP-BO collapses (−70.5% vs. LLM), replicating the cliff failure on VDMS. On SIFT1M the gap compresses to +5.8% (41 847 vs. VDTuner’s 39 540), matching the complexity-proportional scaling seen on VDMS. Table 7: Cross-system results on Milvus (3-seed means, 50 iters). HICO-DET↑
GLDv2↑
SIFT1M↑
Random Search [3] Optuna TPE [1] GP-BO [26] VDTuner [34]
4504 5368 5449 6798
2632 4792 1882 5624
31,134 35,330 35,051 39,540
LLM Agent (Ours)
7272
6375
41,847
Method
6.7
upper-bounding OPRO [33], which retains history but lacks both). The non-monotonicity (w/o both: 298.2, 1/3≥299 > w/o history alone: 296.9, 0/3≥299) arises because history without phase boundaries anchors the agent to its plateau, while removing both restores structural exploration that occasionally reaches Score ≥ 299. Table 8: Mechanistic ablations on HICO-DET (3-seed means, 50 iters). Method LLM Agent (Full) w/o History conditioning w/o Phase structure w/o Both
Ablation Study
Best SIEVE Score (QPS)
7
299
298 LLM Agent (Full) w/o History w/o Phase w/o Both
294 10
20
30
Iteration
40
300.3 296.9 299.5 298.2
Gain
𝜎
Seeds ≥ 299
Δ@50
+4.3 +0.1 +1.3 +0.5
0.4 3.2 0.1 0.9
3/3 0/3 3/3 1/3
— −1.1% −0.3% −0.7%
Conclusion and Outlook
Two-stage ANN retrieval pipelines impose conditional dependencies — engine-type gating, cross-stage search-depth–fusion-weight coupling, and a discontinuous feasibility cliff — that parameterindependent HPO methods model incorrectly by treating jointly coupled parameters as independent marginals. We introduced SIEVE, a quality-gated throughput objective, and a phase-aware LLM agent (Exploration → Exploitation → Fine-tuning) that navigates coupling by conditioning each proposal on the full accumulated optimization history. On HICO-DET the agent outperforms the best non-LLM baseline by +33.3% and achieves 3/3-seed reproducibility (vs. 1/3 for Optuna TPE and GP-BO); on GLDv2 and SIFT1M performance converges (adaptive methods except GP-BO within 1% and 3.6% respectively), confirming that LLM optimization advantage scales with configuration-space coupling complexity. Future work includes scaling to billion-vector corpora and extending the phase-aware framework to multi-objective settings where latency, throughput, and quality targets are jointly negotiated.
3/3 0/3 3/3 1/3
296
296.0 296.8 298.1 297.6
Backbone portability. A cross-backbone check confirms that the phase-aware protocol is model-agnostic: GPT-4o-mini (seed 42, Score = 297.44) and Llama-3.3-70B (Score = 294.20) fall within 0.95% and 2.0% of the MiniMax-M2.1 mean (300.3), respectively. Phase boundary sensitivity. To assess sensitivity to the phase-boundary thresholds (𝑡 exp, 𝑡 expl ), we ran two additional configurations at seed 42: equal-thirds (0.33/0.66, Score = 299.73) and late-exploitation (0.50/0.80, Score = 302.12), versus the default (0.40/0.75, Score = 300.54). All three fall within ±0.53% of the default, confirming that the optimizer is insensitive to moderate boundary shifts and that the phase-structure benefit reported above holds across schedule variants.
Parameter ablations (𝛼, 𝑘, 𝑛𝑟 , engine). 𝛼=0.5 raises mAP from 0.240 to 0.276 (+3.65 pp) at 0.6% extra latency; 𝛼 ∈(0, 1) strictly dominates both pure encoders. Increasing 𝑘 500→2000 improves mAP but collapses Score 8.21→1.17 (7× throughput penalty); the agent discovers the non-obvious optimum 𝑘=50, 𝛼=0.80, 𝑛𝑟 =5 centroid (Score ≈ 300, mAP = 0.160, just above 𝜏). The centroid reference strategy (minimizes 𝜀 𝑗 , Eq. (10)) outperforms first and diverse; the agent selects HNSW 𝑀=64 on all seeds while Optuna spreads over 𝑀 ∈{16, 32, 48} (mean 225.2), confirming the value of joint engine-type conditioning. History 302 conditioning and phase structure.
300
Score@10 Score@50↑
50 Full Hist. Phase Both
Figure 5: Ablation: Best-Score-so-Far (BSF) convergence (left) and per-seed final Score at 𝑁 =50 (right).
Acknowledgments
Figure 5 and Table 8 isolate each architectural component across 3 seeds, revealing that the primary effect of history conditioning is not a marginal Score improvement but a binary reliability switch: removing it causes complete stagnation after iteration 5 (+0.1 pts gain over 45 remaining iterations) and 0/3 seeds reach Score ≥ 299, versus 3/3 for the full agent (𝜎: 3.2 → 0.4). Without the (𝜃, Score) trajectory the agent cannot commit to the (𝑘=50, 𝛼=0.80) joint operating point, mirroring the GP-BO cliff failure of Section 6.3.Phase structure has a secondary effect: removing it reduces the mean to 299.5 (−0.3%) but 3/3 seeds still cross the threshold, confirming that phase ordering accelerates convergence rather than enabling it (and
This work used NCSA Delta GPU at NCSA through allocation CIS240646 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program [4], which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
References [1] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631, 2019. 12
[25] H. V. Simhadri, G. Williams, M. Aumüller, M. Douze, A. Babenko, D. Baranchuk, et al. Results of the NeurIPS’21 challenge on billion-scale approximate nearest neighbor search. In Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pages 177–189. PMLR, 2022.
[2] M. Aumüller, E. Bernhardsson, and A. Faithfull. ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems, 87:101374, 2020. [3] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281–305, 2012.
[26] J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, pages 2960–2968, 2012.
[4] T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns. ACCESS: Advancing innovation: NSF’s advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing (PEARC ’23), pages 173–176, Portland, OR, USA, 2023. ACM.
[27] S. J. Subramanya, Devvrit, R. Kadekodi, R. Krishnaswamy, and H. V. Simhadri. DiskANN: Fast accurate billion-point nearest neighbor search on a single node. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[5] S. Bruch, S. Gai, and A. Ingber. An analysis of fusion functions for hybrid retrieval. ACM Transactions on Information Systems, 42(1):20:1–20:35, 2024.
[28] J. Sun, G. Li, J. Pan, J. Wang, Y. Xie, R. Liu, and W. Nie. GaussDB-Vector: A largescale persistent real-time vector database for LLM applications. Proceedings of the VLDB Endowment, 18(12):4951–4963, 2025.
[6] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng. Learning to detect human-object interactions. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 381–389, 2018.
[29] D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang. Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD), pages 1009–1024, 2017.
[7] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00325, 2015. [8] M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, pages 2962–2970, 2015.
[30] J. Wang, X. Yi, R. Guo, H. Jin, P. Xu, S. Li, et al. Milvus: A purpose-built vector data management system. In Proceedings of the 2021 International Conference on Management of Data (SIGMOD), pages 2614–2627, 2021.
[9] V. Giannakouris and I. Trummer. 𝜆 -tune: Harnessing large language models for automated database system tuning. Proceedings of the ACM on Management of Data, 3(1), 2025.
[31] C. Wei, Y. Chen, H. Chen, H. Hu, G. Zhang, J. Fu, A. Ritter, and W. Chen. UniIR: Training and benchmarking universal multimodal information retrievers. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[10] H. Jégou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117– 128, 2011.
[32] T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 — a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2575–2584, 2020.
[11] J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2021.
[33] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. Large language models as optimizers. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
[12] M. Kilickaya and A. W. Smeulders. Structured visual search via compositionaware learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1700–1709, 2021.
[34] T. Yang, W. Hu, W. Peng, Y. Li, J. Li, G. Wang, and X. Liu. VDTuner: Automated performance tuning for vector data management systems. In Proceedings of the 40th IEEE International Conference on Data Engineering (ICDE), pages 4357–4369, 2024.
[13] B. Kim, J. Lee, J. Kang, E.-S. Kim, and H. J. Kim. HOTR: End-to-end humanobject interaction detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 74–83, 2021. [14] J. Lao, Y. Wang, Y. Li, J. Wang, Y. Zhang, Z. Cheng, W. Chen, M. Tang, and J. Wang. GPTuner: A manual-reading database tuning system via GPT-guided Bayesian optimization. Proceedings of the VLDB Endowment, 17(8):1939–1952, 2024.
[35] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67– 78, 2014.
[15] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1–52, 2018.
[36] A. Zhang, Y. Liao, S. Liu, M. Lu, Y. Wang, C. Gao, and X. Li. Mining the benefits of two-stage and one-stage HOI detection. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 17209–17220, 2021.
[16] Z. Li, X. Li, C. Ding, and X. Xu. Disentangled pre-training for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[37] M. R. Zhang, N. Desai, J. Bae, J. Lorraine, and J. Ba. Using large language models for hyperparameter optimization. arXiv:2312.04528, 2023. [38] X. Zhong, H. Li, J. Jin, M. Yang, D. Chu, X. Wang, et al. VSAG: An optimized search framework for graph-based approximate nearest neighbor search. Proceedings of the VLDB Endowment, 18(12):5017–5030, 2025.
[17] S. Liu, C. Gao, and Y. Li. Large language model agent for hyper-parameter optimization. arXiv:2402.01881, 2024. [18] Y. Liu, Y. Zhang, J. Cai, X. Jiang, Y. Hu, J. Yao, Y. Wang, and W. Xie. LamRA: Large multimodal model as your advanced retrieval assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4015–4025, 2025. [19] Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020. [20] C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. [21] S. Ning, L. Qiu, Y. Liu, and X. He. HOICLIP: Efficient knowledge transfer for HOI detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23507–23517, 2023. [22] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. [23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021. [24] L. Remis and C. W. Lacewell. Using VDMS to index and search 100M images. Proceedings of the VLDB Endowment, 14(12):3240–3252, 2021. 13