Self-Optimizing AI Orchestration: A Feedback-Driven Meta-Layer for Autonomous Multi-LLM Model Selection

The proliferation of large language models (LLMs) with heterogeneous cost, latency, and capability profiles has created a pressing need for intelligent orchestration that goes beyond static routing rules. Existing approaches—including threshold-based routers and lookup-table gateways—fail to adapt as models evolve, usage patterns shift, or multi-dimensional objectives change. We introduce SOMA (Self-Optimizing Multi-LLM Orchestration with Adaptive Meta-Layer), a feedback-driven orchestration framework that treats model selection as a contextual decision problem solved online. SOMA encodes each incoming query into a rich context vector, applies a Contextual Bandit policy to select both the target model and its inference parameters, executes the call, and updates its policy through a multiobjective reward signal that jointly optimises quality, cost, and latency. Three contributions address limitations of prior work: (i) a Task-Aware Quality Estimator (TAQE) that applies task-appropriate quality metrics—including execution-based evaluation for code and a Prometheus-calibrated diversity score for creative writing—achieving Pearson ρ = 0.84 with MT-Bench LLM-judge scores across all task types (no independent human-annotation validation was performed in this study); (ii) a hierarchical action decomposition that decouples model selection from inference parameter configuration, preserving sub-linear regret as the model pool grows; and (iii) SOMA-MT, a multi-turn extension with dialogue-level consistency regularisation. Experiments on 4,800 diverse queries routed across five open-source LLMs demonstrate that SOMA reduces inference cost by 38% while maintaining output quality within 2% of Always-Large, the largest model in our pool, and achieves 2.1× lower latency than always routing to the largest model (balanced profile; up to 2.93× under the latency-optimised profile). SOMA outperforms five competitive baselines including RouteLLM across all three objective dimensions simultaneously. Our code and datasets are released publicly.