Configuration-Driven Dynamic API Routing for Resilient Service Integrations Nataraj Agaram Sundar eBay Inc.
arXiv:2605.26404v1 [cs.DC] 26 May 2026
Abstract
Traditional application-level resilience mechanisms remain necessary. Timeouts prevent unbounded waiting. Retries mask transient failures. Circuit breakers stop a failing dependency from consuming resources. Bulkheads isolate pools so one integration cannot exhaust capacity for unrelated workflows. These mechanisms, popularized in production engineering practice, are important local controls [1, 2]. However, they are incomplete for multiprovider workflows. A circuit breaker can stop requests from going to a failing vendor, but it does not decide which alternative vendor is most appropriate for a given operation, region, cost profile, compliance requirement, or current latency distribution. A retry policy can reattempt work, but it cannot know that another provider has better recent completion rate in a specific country or carrier segment. A static failover flag can switch traffic, but it often requires an operator to diagnose the outage, change configuration, and later remember to restore traffic after recovery. This paper argues for an explicit decision layer between application code and third-party providers. The decision layer treats provider selection as a continuously evaluated routing problem. Instead of encoding vendor preference in application logic, an operation such as SEND_SMS, VERIFY_PHONE, PAYMENT_AUTH, or TOKENIZE_CARD is mapped to a pluggable factor list: a declarative set of hard gates and weighted scoring functions. Gates eliminate providers that are currently unsafe or ineligible, for example because a circuit is open, a region is unsupported, quota is exhausted, or a policy constraint applies. Scores rank the remaining providers using normalized live signals such as recent completion rate, tail latency, per-request cost, and incident penalty. A telemetry pipeline computes these signals from event streams emitted by every third-party call. A router then selects the provider at request time, applies protection controls, emits outcome events, and closes the feedback loop. The approach is motivated by practical experience with high-scale external-integration systems. In one productioninspired case study, a global marketplace used SMS-based verification for login and registration. A primary SMS vendor experienced recurring outages while a secondary vendor existed but required manual failover. The architecture described here replaced manual switching with automatic provider selection based on live completion-rate metrics. Although implementation details and exact pro-
Modern online services rely on third-party APIs for authentication, payments, communication, identity verification, fraud detection, observability, and fulfillment. These dependencies are outside the direct operational control of the application owner and may experience regional outages, throttling, latency spikes, quota exhaustion, or behavior changes that surface as user-visible failures. This paper presents configuration-driven dynamic API routing, an architecture for resilient third-party service integration based on pluggable factor lists, real-time telemetry, circuit breakers, bulkhead isolation, and a closed-loop decision engine. A factor list defines operation-specific hard gates and weighted scoring functions that evaluate candidate providers using live metrics, regional policy constraints, quota state, latency, cost, and incident signals. The router separates routing policy from application code, allowing operators to adapt vendor selection at runtime without redeploying applications. We formalize the factor-list model, describe a request-time routing algorithm, present the event pipeline that computes sliding-window provider health metrics, and analyze failover behavior under degraded-provider scenarios. We also describe an anonymized SMS verification case study in which manual vendor switching was replaced by automated routing driven by completion-rate telemetry. Keywords: dynamic API routing, resilience engineering, third-party APIs, fault tolerance, circuit breakers, bulkhead isolation, telemetry, service reliability, configuration-driven systems, software engineering.
1
Tejas Morabia eBay Inc.
Introduction
The reliability of a modern application is no longer determined only by the reliability of its own code. A user request may synchronously or asynchronously depend on payment processors, messaging providers, identity-verification systems, fraud services, geocoding services, notification platforms, and other third-party APIs. These services may publish service-level agreements, but the consuming platform does not control their deployments, capacity events, throttling policies, regional carrier behavior, or incident response. The result is a recurring reliability pattern: the application is healthy, yet the user-visible workflow fails because an external dependency has degraded. 1
duction values are anonymized, the case study illustrates a general pattern: resilience improves when fallback is not merely present, but automated, observable, and governed by runtime policy. This paper makes four contributions:
availability, not only static priority. A provider that is preferred globally may be poor in a specific region, degraded for a carrier, or temporarily rate-limited. R3: Policy-code separation. Operators should change routing policy without redeploying application binaries. This is important during incidents, launches, vendor migrations, and regional expansions. It also reduces the risk that emergency fixes become hard-coded and forgotten. R4: Operation-specific semantics. Different operations require different selection criteria. SMS sending emphasizes completion rate, latency, regional coverage, and cost. Payment authorization may emphasize authorization success, fraud tooling, card-network behavior, and compliance. Identity verification may emphasize jurisdictional constraints and vendor-specific coverage. A single global provider ranking is insufficient. R5: Observability and explainability. Every routing decision should be auditable. Engineers should be able to answer why a provider was selected, which gates eliminated alternatives, which scores dominated, and how the decision affected downstream outcomes. R6: Stability. A router must avoid flapping between providers due to noisy metrics. Stability requires sliding windows, minimum sample thresholds, hysteresis, cooldown periods, circuit half-open behavior, and safe default policies. R7: Graceful degradation. When all providers are degraded, the system should fail in a controlled way: return a typed error, trigger delayed retry or queueing, use an alternate verification channel if available, or shed low-priority traffic. It should not produce an unbounded cascade.
1. It formalizes pluggable factor lists as an operationspecific abstraction for configuration-driven third-party API routing. 2. It describes a layered architecture that combines protection controls, dynamic routing, event-stream telemetry, and runtime configuration into a closed-loop resilience platform. 3. It provides a request-time routing algorithm with gates, normalized weighted scoring, hysteresis, circuit state, and fallback behavior. 4. It presents an anonymized production-inspired case study and an analytical evaluation of failover dynamics in SMS verification routing. The intended audience is practitioners and researchers working on distributed systems, software reliability, API platforms, service orchestration, and fault-tolerant application architecture. The paper does not claim that dynamic routing replaces strong provider due diligence, contract negotiation, incident management, or application-level correctness. Rather, it shows how to make provider redundancy operationally useful when third-party failures are frequent enough, user impact is high enough, and manual intervention does not scale.
2
2.1
Availability Motivation
Problem Statement and RequireIf a user-visible workflow depends on m independent serments vices in series, and each service i has availability Ai , the idealized end-to-end availability is
Consider a platform that exposes a user-facing workflow W requiring one or more external operations. Each operation o has a set of candidate providers Vo = {v1 , . . . , vn }. For example, an SMS verification operation may have providers that differ by price, carrier reach, country coverage, latency, deliverability, throttling limits, compliance constraints, and recent incident history. The application must choose a provider for each request while satisfying both correctness and service objectives. The central problem is not simply failover. In practice, failover is only one subproblem inside provider orchestration. A resilient third-party integration platform should meet the following requirements. R1: Fault containment. A slow or failing provider must not exhaust application resources, thread pools, connection pools, queues, or retry budgets. Failures should be isolated by operation and provider. This motivates circuit breakers, bounded timeouts, rate limits, and bulkheads. R2: Runtime-adaptive provider selection. Provider choice should respond to recent performance and
Aserial =
m Y
Ai .
(1)
i=1
Even with Ai = 0.999, three serial dependencies yield approximately 0.997 availability, and five yield approximately 0.995. This multiplication is only an approximation: real failures are not always independent, and traffic can be asynchronous or partially degraded. Nevertheless, the equation captures a practical concern: each external dependency consumes part of the workflow’s reliability budget. Redundancy can improve availability if alternative providers fail independently and the platform can route to a healthy provider. For two providers with availabilities A1 and A2 , ideal parallel availability is Aparallel = 1 − (1 − A1 )(1 − A2 ).
(2)
However, this bound is achieved only if detection, routing, and fallback are automatic and sufficiently fast. If provider 2
switching requires manual intervention, then user impact is governed not only by provider availability but also by alerting delay, diagnosis time, configuration propagation, and operator response.
2.2
Common gates include circuit closed, region supported, quota available, provider enabled, compliance allowed, credential valid, and maintenance window inactive. Gate order is useful for explainability but not required for correctness unless a gate has side effects, which should generally be avoided.
Why Static Failover Is Insufficient
A static primary-secondary configuration handles simple outages but fails under nuanced conditions. A provider may be partially degraded for one geography but healthy elsewhere; a low-cost provider may be appropriate for normal traffic but inappropriate during peak login events; a vendor may return success codes while downstream delivery silently degrades; or a secondary provider may have worse latency but higher completion rate in a specific window. Static failover collapses these signals into a single binary preference. Configuration-driven dynamic routing preserves the ability to encode default preference while allowing live metrics and policy constraints to override that preference when conditions change.
3
3.2
Provider Score
For each eligible provider v ∈ Co (x, t), the router computes a weighted score: Scoreo (v, x, t) =
q X
wk · nk (sk (v, x, t)),
(5)
k=1
P where wk ≥ 0, k wk = 1, and nk normalizes the metric orientation. For a higher-is-better metric such as completion rate, nk may be the clipped value itself. For a lower-is-better metric such as latency or cost, nk may be transformed as
System Model nk (y) = 1 −
We model a third-party API routing platform as a tuple
min(max(y, Lk ), Uk ) − Lk , Uk − Lk
(6)
P = (O, V, G, S, M, C, R),
(3) where Lk and Uk are configured lower and upper bounds. where O is a set of operations, V is a set of providers, G More advanced implementations may use percentiles, pieceis a set of gate functions, S is a set of scoring functions, wise functions, logistic transforms, or risk-adjusted scores. The selected provider is M is a set of live metrics, C is a configuration store, and R is the request-time routing function. R(o, x, t) = arg max Scoreo (v, x, t), (7) For each operation o ∈ O, the configuration store prov∈Co (x,t) vides a factor list Fo . A factor list contains three classes of configuration: subject to stability controls such as hysteresis. If C (x, t) is o
1. Eligibility gates Go = {g1 , . . . , gp }, where gj (v, x, t) ∈ empty, the platform invokes an operation-specific fallback: {0, 1} determines whether provider v is allowed for enqueue for delayed retry, use a lower-fidelity channel, return a typed error, or shed traffic. request context x at time t. 2. Scoring factors So = {s1 , . . . , sq }, where each sk (v, x, t) maps provider state and context to a normalized score in [0, 1].
3.3
Closed-Loop Control
Every provider attempt produces an event e containing 3. Control parameters, such as metric refresh interval, at least operation, provider, region, timestamp, outcome, minimum sample count, cool-down duration, circuitlatency, status code, timeout flag, retry count, and cost. breaker thresholds, tie-breaking policy, default provider, The event stream updates live metrics M . The router and fallback behavior. consumes M through a bounded-lag cache. The resulting The request context x may include operation name, user loop is: region, market, tenant, risk tier, request priority, carrier or payment-network segment, compliance attributes, and trafrequest → decision → provider attempt fic class. The router does not need every attribute for ev(8) ery operation; the factor-list abstraction allows operation→ event → metric → next decision. specific use of context. This loop is deliberately simple. It is not intended to be a fully autonomous optimization system with unconstrained objective functions. The factor list keeps operational intent The initial candidate set for operation o is Vo ⊆ V . Gates explicit: engineers decide which metrics matter, what reduce this set to eligible providers: constraints must never be violated, and how aggressive the Co (x, t) = {v ∈ Vo | ∀gj ∈ Go , gj (v, x, t) = 1}. (4) router should be during degradation.
3.1
Candidate Set
3
4
Architecture
specify behavior when the configuration store is unavailable. A safe default is to continue using the last known Figure 1 shows the layered architecture. The application good configuration for a bounded interval while alerting calls a stable internal API for an external operation. The operators. resilience platform resolves the operation’s factor list, evaluates provider candidates, invokes the selected provider 4.3 Decision Locality through an isolated integration adapter, and emits telemeThe routing decision can be made centrally or locally. A try. The architecture separates four responsibilities. Protection layer. This layer contains circuit breakers, central service simplifies policy enforcement and auditing timeouts, rate limits, retry budgets, and bulkhead isolation. but adds a dependency to every request. A local library or It protects the platform from local resource exhaustion sidecar reduces network hops but requires careful rollout and prevents a degraded provider from consuming capacity and consistent configuration. A hybrid approach is often practical: local request-time decisions use cached metrics that should remain available for other operations. Decision layer. This layer loads operation-specific and configuration, while centralized analytics computes factor lists, evaluates gates, normalizes scores, applies sliding-window metrics and publishes compact health sumhysteresis, and selects a provider. The decision layer can maries. be implemented in-process for low latency, as a sidecar, or as a dedicated routing service. The main requirement is 5 Pluggable Factor Lists that the application observes a stable operation API while provider policy remains externally configurable. A pluggable factor list is an ordered, declarative configuraTelemetry layer. This layer emits structured events tion that defines how an operation chooses among providers. for every attempt and aggregates them into sliding-window The term pluggable emphasizes that factors can be added, metrics. It should capture both technical metrics (la- removed, reordered, or retuned without changing applicatency, HTTP status, timeout, exception type) and busi- tion code. The term factor list emphasizes that provider ness outcome metrics (SMS completion, verification suc- selection is composed from independent, inspectable decicess, payment authorization, identity-verification pass/- sion factors rather than a monolithic rule. fail). Business outcome metrics are often more valuable Listing 1 shows a simplified factor list for SMS sending. than transport-level success because a provider can return The example uses three gates and four scoring factors. HTTP 200 while the user-visible workflow still fails. Completion rate has the highest weight because the operaAutomation/configuration layer. This layer stores tion’s primary objective is successful verification. Latency factor lists, thresholds, feature flags, regional overrides, and cost are important but secondary. The incident penalty and emergency disables. Configuration changes should be captures recent provider-level instability that may not yet versioned, auditable, and gradually rolled out. The layer be fully reflected in completion rate. is the mechanism that decouples operational policy from code deployment. Listing 1: Example factor list for SEND_SMS.
4.1
operation: SEND_SMS version: 2026-01-01.1 refresh_interval_seconds: 30 metric_window_seconds: 300 minimum_samples: 100 hysteresis: switch_margin: 0.05 cooldown_seconds: 120 providers: - vendor_a - vendor_b - vendor_c gates: - name: provider_enabled - name: circuit_breaker_closed - name: supports_region - name: quota_available - name: compliance_allowed scores: - name: completion_rate_5m weight: 0.50 direction: higher_is_better - name: p95_latency_5m weight: 0.25 direction: lower_is_better lower_bound_ms: 300 upper_bound_ms: 5000 - name: cost_per_request
Provider Adapter Boundary
Each provider should be encapsulated behind an adapter that translates platform-level requests into providerspecific API calls and normalizes provider responses into a common outcome model. This boundary is important for routing because scores should compare semantically equivalent outcomes. For SMS verification, for example, the router should compare send accepted, delivery confirmed, verification complete, timeout, and user failure using a consistent event vocabulary. Without a normalized outcome model, the decision layer may optimize inconsistent signals.
4.2
Configuration Propagation
The router may cache factor lists for performance, but configuration propagation must be bounded. A common pattern is to combine short TTL caches with explicit invalidation events. Emergency disables should propagate faster than normal tuning changes. The design should also 4
Application / Workflow Service Internal Operation API Resilience platform Config Store Protection Layer
Timeouts Circuit Breakers Bulkheads
Decision Trace
Decision Layer
Provider Adapter
Gates and Scores Hysteresis
Normalize Requests and Results
Event Stream
Metric Aggregators
Metrics Cache
Provider A
Provider B
Provider C
Figure 1: Layered architecture for configuration-driven dynamic API routing. should use minimum sample counts and cool-down intervals. A provider should not be excluded merely because one request failed; it should be excluded when a configured failure condition is met with sufficient confidence.
weight: 0.15 direction: lower_is_better - name: recent_incident_penalty weight: 0.10 direction: lower_is_better fallback: strategy: typed_error_or_delayed_retry preserve_decision_trace: true
5.1
5.2
Scores
Scores express preferences among eligible providers. Unlike gates, scores can trade off competing objectives. For example, one provider may be cheaper but slower, while another has higher completion rate but higher cost. Weighted scores let operators encode the relative importance of these attributes for each operation. The most important design principle is that scores should align with user-visible outcomes. For SMS verification, transport success is not enough; verification completion is a better metric because it reflects whether the user actually received and used the code. For payments, authorization success and downstream settlement behavior may matter more than API latency alone. For identity verification, accuracy, coverage, and compliance may dominate cost.
Gates
Gates are hard constraints. A provider that fails a gate is excluded from the candidate set regardless of its score. Gates should be used for conditions that must not be traded off against other metrics. Typical gates include: • Circuit state: exclude a provider whose circuit is open for the operation or region. • Coverage: exclude a provider that does not support the user’s country, carrier, payment method, or verification type. • Compliance: exclude a provider that is not allowed for the data class, jurisdiction, tenant, or contractual requirement.
5.3
Runtime Overrides
• Quota and throttling: exclude a provider with exhausted daily quota, active rate-limit response, or in- Factor lists should support scoped overrides. An override sufficient remaining capacity. may apply to a region, tenant, traffic class, operation, or • Operational control: exclude a provider under main- experiment cohort. For example, a provider may be distenance, manually disabled, or isolated by incident com- abled only for one country; a higher-cost provider may be preferred for account recovery but not marketing nomand. tifications; or a new provider may receive 5% of traffic Because gates have binary effect, they require careful de- for controlled ramp-up. Scoped overrides prevent global sign. A gate using noisy metrics may cause traffic to shift changes from solving a local issue by creating a broader too aggressively. Gates based on sliding-window failures incident. 5
route(request r): o = r.operation x = build_context(r) F = config_cache.factor_list(o) Ms = metrics_cache.snapshot(o, x.scope)
Incoming Request
Load Factor List
eligible = [] trace = new_decision_trace(r.id, F.version, Ms. timestamp)
Enumerate Providers
for provider v in F.providers: gate_result = evaluate_gates(F.gates, v, x, Ms) trace.add_gate_result(v, gate_result) if gate_result.passed: score = 0 for factor f in F.scores: raw = read_metric_or_default(f, v, x, Ms) normalized = normalize(f, raw) score += f.weight * normalized trace.add_score(v, f.name, raw, normalized) eligible.append((v, score))
All Gates Pass? no
Exclude Provider
yes
Read Live Metrics
Normalize Values
Weighted Score
if eligible is empty: return fallback_or_typed_error(r, trace)
Hysteresis
selected = argmax_score(eligible) selected = apply_hysteresis(selected, previous_choice( o, x), F)
Select Provider
result = protection_layer.invoke(selected, r) emit_attempt_event(r, selected, result, trace) return result
Protected Invoke
Emit Event + Trace
6.1
Hysteresis and Flapping Control
Figure 2: Gate-and-score decision flow for pluggable factor A naive router switches to the provider with the highest instantaneous score. This can produce flapping when two lists. providers have similar scores or metrics are noisy. Hysteresis introduces a switching margin δ: the router changes 5.4 Explainability from current provider vc to challenger vn only if Every decision should produce a decision trace. A trace records candidate providers, gate outcomes, normalized scores, final provider, factor-list version, and metric snapshot timestamp. Traces are invaluable during incidents because they convert routing from a black box into a debuggable process. They also enable offline analysis: engineers can replay historical events against proposed factor-list changes before applying them to production. Figure 2 illustrates the decision flow expressed by a factor list.
Score(vn ) > Score(vc ) + δ.
(9)
A cool-down period can further prevent repeated switching. For operations with high user impact, the router may require a challenger to maintain superiority for multiple consecutive metric windows.
6.2
Circuit Breakers
Circuit breakers integrate with gates. A provider circuit transitions among closed, open, and half-open states. In the closed state, requests are allowed. In the open state, 6 Routing Algorithm the provider fails the circuit gate and receives no normal traffic. In the half-open state, the router allows a small Listing 2 shows request-time provider selection. The algoprobe volume to test recovery. Successful probes close the rithm assumes that configuration and metrics are available circuit; failed probes reopen it. Circuit state can be tracked from local caches. Cache misses or stale metrics are hanper provider, operation, and region to avoid unnecessarily dled by operation-specific defaults. The router first loads disabling a provider globally. the factor list, enumerates candidate providers, evaluates gates, computes normalized scores, applies hysteresis, and invokes the selected provider through the protection layer. 6.3 Retries and Hedging Retries should be bounded and coordinated with routing. Retrying the same provider during an outage can amplify
Listing 2: Request-time routing algorithm. 6
App
Config
Router
Metrics
Provider
• request_id, operation, provider, region, and tenant or traffic class.
Stream
request
factor list
• start_time, end_time, latency, timeout flag, retry count, and circuit state.
snapshot
• transport outcome: HTTP status, exception category, provider error code, and rate-limit indicator.
gates + score + hysteresis API attempt
response normalized result
• business outcome: accepted, delivered, verification completed, authorized, declined, or equivalent domain result.
event + trace
Figure 3: Simplified request path and feedback from • cost and quota counters when available. provider attempts. • factor-list version and decision trace identifier. Figure 4 shows the feedback loop from request attempts to metrics and back to routing decisions. The event stream may be implemented with Kafka or another durable streaming system. Aggregators compute sliding-window statistics and publish compact summaries to a metrics store or cache consumed by routers. Kafka is commonly used for this class of log-oriented processing because it provides durable, partitioned streams and supports independent consumers [4].
load and delay failover. A safer policy is to retry only idempotent or explicitly safe operations, use small retry budgets, and allow retry-to-alternate-provider when the operation semantics permit it. For latency-sensitive readlike operations, hedged requests can reduce tail latency, but they increase provider load and cost; therefore they should be controlled by policy and reserved for operations where duplicate attempts are safe [3].
6.4
Tie Breaking
7.1
Tie breaking should be deterministic within a short window to avoid random oscillation. Options include sticky provider by user or tenant, weighted randomization proportional to score, priority order after scoring, or least-recentlyused balancing. For SMS verification, user-level stickiness can be useful because send and verify-complete events may need to remain associated with the same provider.
6.5
The router usually needs recent, localized metrics rather than global averages. For provider v, operation o, and region r, a completion-rate metric over a window W can be computed as Completed(v, o, r, W ) . Attempted(v, o, r, W ) (11) This metric should include minimum-sample checks. A provider with two successes out of two attempts should not necessarily outrank a provider with 9,900 successes out of 10,000 attempts. Confidence-aware scoring can downweight low-volume samples, or a gate can require minimum observations before a metric is trusted. Latency metrics should use percentiles such as p95 or p99 rather than averages. Tail latency matters because user-visible workflows are often governed by the slowest dependency [3]. Cost metrics may be static by contract or dynamic by region, carrier, volume tier, or retry behavior. CompletionRate(v, o, r, W ) =
Failover Latency Bound
The time between provider degradation and routing change is bounded by detection, event publication, aggregation, metric-cache refresh, and decision-cache propagation: Tf ailover ≤ Tdetect + Tpublish + Taggregate + Tref resh + Tdecision .
(10)
In many implementations, Tdecision is negligible compared with metric-window and refresh intervals. The largest design tradeoff is between sensitivity and stability. Short windows detect failures quickly but increase false positives; longer windows reduce noise but prolong user impact. Factor lists should make these windows explicit per operation.
7
Sliding-Window Metrics
7.2
Outcome Alignment
Transport metrics can be misleading. For example, an SMS vendor may return an accepted status while messages are delayed by a carrier route. A payment provider may return a syntactically successful response with a decline reason that reflects downstream network issues. Therefore the pipeline should distinguish attempt success from workflow success. In verification systems, a robust metric is often a completion rate linking send attempts to successful verification within a time window.
Event Pipeline and Metrics
The decision layer is only as reliable as its telemetry. Each provider attempt should produce a structured event, not merely a log line. The event should be machine-readable, schema-versioned, and correlated with both request and provider identifiers. A minimal event schema includes: 7
Factor List Config
API Requests
Dynamic Router
Third-Party Providers
Attempt + Outcome Events
Decision Traces
Metrics Cache
Sliding-Window Aggregators
Event Stream
Figure 4: Telemetry and control loop.
7.3
Metric Freshness Telemetry
and
Degraded 8.1
Telemetry itself can degrade. If the event stream lags, the router may make decisions based on stale data. The metric cache should expose freshness timestamps, and factor lists should define stale-data behavior. Conservative behavior may prefer a stable default provider when metrics are stale; aggressive behavior may preserve the last known good ranking for a bounded interval. In all cases, stale metrics should be visible in decision traces and operational dashboards.
7.4
Anonymized SMS Verification Scenario
The motivating deployment is a global marketplace platform that used SMS verification for login and registration. A primary SMS provider served most traffic. A secondary provider had already been integrated but was used mainly as a backup. During primary-provider incidents, engineers manually switched traffic after alerts and diagnosis. This process had three limitations: 1. User impact began immediately when the primary provider degraded, while failover occurred only after human intervention.
Replaying Decisions
Because routing decisions are configuration-driven, his- 2. The manual switch was coarse-grained and operationally expensive. It did not naturally account for regional torical event streams can be replayed against proposed partial failures or rapid recovery. factor-list changes. Replay enables offline evaluation before rollout: engineers can ask whether a new latency weight would have shifted traffic during a past outage, 3. Restoration required another manual action, creating risk of leaving traffic on a nonpreferred provider longer whether a gate would have excluded too many providers, than necessary. or whether hysteresis would have prevented flapping. This is one of the main advantages of declarative routing policy The platform implemented the architecture described in over hard-coded conditional logic. this paper. Every SEND and VERIFY_COMPLETE operation emitted an event. Aggregators computed live completionrate metrics per provider and relevant scope. The SMS 8 Evaluation and Case Study service cached these metrics and selected the provider with No proprietary production data, vendor names, user data, the strongest completion-rate signal, subject to gates such as circuit state, regional support, and quota availability. or internal system identifiers are disclosed. When the primary provider degraded, the score fell and A complete evaluation of dynamic API routing should intraffic shifted to the healthier provider. When the primary clude both production evidence and controlled experiments. provider recovered and sustained better metrics, traffic Production evidence demonstrates operational relevance shifted back according to hysteresis and cool-down policy. but may contain proprietary data. Controlled experiments are reproducible but may simplify real provider behavior. The reported outcome was a shift from reactive manual This section combines an anonymized production-inspired failover to automated routing. In the observed class of case study with an analytical failover model. vendor-switching events, on-call intervention for routine 8
Table 1: Operational comparison for the SMS verifica- Table 2: Model-based sensitivity of failures to failover tion scenario. Exact production values are intentionally delay for a 10-minute primary-provider outage. Values are anonymized; the table reports qualitative effects and mea- expected failed requests per 1,000 requests/minute. surable dimensions. Strategy Dimension
Manual Failover
Dynamic Routing
Trigger
Alert and operator diagnosis Manual configuration change Usually coarsegrained Manual restoration Incident notes and logs Required during vendor incidents Proportional to detection and response delay
Metric and gate evaluation Automatic requesttime selection Operation, region, provider, or cohort Automatic after sustained recovery Decision trace per request Eliminated for routine vendor switching Bounded by telemetry and refresh interval
Switch action Scope Recovery Auditability On-call load User impact
No failover Manual failover Static monitor failover Dynamic telemetry routing Ideal instant switch
Failover Delay
Expected Failures
10.0 min 8.0 min 2.0 min 0.5 min 0.0 min
9,500 7,620 1,980 595 100
only objective. A system that switches too quickly on noisy signals may flap and degrade user experience. The goal is not minimum possible detection time but a stable detection window aligned to the operation’s error budget.
8.3
Experimental Evaluation Plan
A deployable evaluation should measure the following diprovider switching was eliminated, and no widespread user- mensions: visible verification disruption was observed during complete provider outages because failover occurred before broad • Completion rate: percentage of workflows that reach the user-visible success state. verification failures accumulated. The architecture also enabled continuous optimization because provider choice could track recent completion rate rather than a static • Failover latency: time from provider degradation to traffic shift. priority order.
8.2
• Tail latency: p95 and p99 latency before, during, and after degradation.
Analytical Failover Model
To reason about failover behavior, consider a simplified outage of duration D minutes. Requests arrive at rate λ per minute. The primary provider’s success probability during the outage is pf , while the secondary provider’s success probability is ps . A manual system switches after Tm minutes; a dynamic system switches after Td minutes, where Td is governed by the telemetry and refresh bound in Section 6. Expected failed requests during the outage are approximately:
• Provider stability: number of provider switches per hour and flapping incidents. • Cost impact: blended cost per successful workflow. • Operator burden: number of manual interventions, pages, and incident tasks. • Explainability: percentage of decisions with complete traces and current metric snapshots.
E[F ailures] = λ (T · (1 − pf ) + (D − T ) · (1 − ps )) , (12) where T = min(D, Tm ) for manual failover and T = min(D, Td ) for dynamic routing. The difference is driven by Tm − Td . When pf is low, ps is high, and traffic volume is large, even a few minutes of faster failover can prevent a large number of user-visible failures. Table 2 illustrates the effect using normalized assumptions rather than production values: D = 10 minutes, pf = 0.05, ps = 0.99, and equal request volume per minute. The table reports failures per 1,000 requests per minute. The values are not claims about a specific deployment; they show the sensitivity of user impact to failover delay. The model highlights two engineering implications. First, integrating a secondary provider is not sufficient; the operational value of redundancy depends on the speed and accuracy of switching. Second, failover speed is not the
Controlled tests should include full outage, partial regional degradation, rate limiting, increased latency without errors, stale telemetry, configuration-store unavailability, and provider recovery. These tests are important because third-party incidents rarely present as clean binary failures.
9
Limitations Concerns
and
Operational
Dynamic API routing is not a universal solution. It introduces additional components, configuration, observability requirements, and failure modes. The architecture is most appropriate when traffic volume, user impact, regional variance, or provider instability justify the complexity. 9
Primary Preferred
user request. If live metrics are unavailable, it should use cached summaries or safe defaults.
T6
T1
T5
Secondary Preferred
T3
T4
9.5
Degraded Mode
T2
Probe Primary
T1: degradation or open circuit; T2: cooldown and probes allowed; T3: sustained recovery; T4: probe failure; T5: all providers fail gates; T6: provider recovery.
Figure 5: Provider preference state machine for the SMS verification case. Transition labels are defined in the figure legend.
9.1
Correlation of Provider Failures
The availability gains of multi-provider routing assume that provider failures are at least partially independent. In reality, providers may share cloud regions, carrier routes, identity networks, certificate authorities, DNS dependencies, or regulatory constraints. A router should not blindly assume independence. Provider diversity analysis and chaos-style exercises are needed to understand commonmode failures.
9.2
Metric Quality
Bad metrics produce bad routing. Completion rate may be delayed because verification completion occurs minutes after SMS send. Latency percentiles may be distorted by low sample counts. Cost may not reflect volume-tier contracts. Provider error codes may be inconsistent. Metric design therefore requires domain knowledge. The platform should expose confidence, sample size, freshness, and schema version alongside each metric.
9.3
Configuration Risk
Configuration-driven systems can fail through bad configuration. A malformed factor list, overly aggressive threshold, incorrect regional override, or accidental global disable can cause an outage. Safe configuration management requires validation, policy linting, staged rollout, versioning, rollback, and ownership. A factor list should be treated like production code, even though it is not deployed through the same binary-release process.
9.4
Latency Overhead
A router adds work to the request path. The overhead can be minimized with local caches, precomputed metrics, compact factor lists, and asynchronous telemetry. However, the decision path must be bounded. The router should not synchronously query a slow analytics store for every
Ethical and Compliance Considerations
Provider routing may affect data residency, user privacy, regulatory obligations, and fairness across regions. A lowercost provider should not be selected if it violates jurisdictional constraints or degrades service for a protected user segment. Compliance gates should be hard constraints, not scoring preferences. Decision traces should be retained according to privacy and data-governance policies.
9.6
When the Architecture Is Overkill
For a small application with one stable provider, low traffic volume, and tolerable manual recovery, the architecture may not be justified. Simpler patterns such as explicit timeouts, retries, and a manually controlled fallback flag may be sufficient. The proposed platform is best suited for highscale systems where third-party variance is operationally significant and where automated recovery materially improves user experience or business continuity.
10
Related Work
The architecture builds on several established areas of distributed systems and production engineering. Production resilience patterns. Circuit breakers, bulkheads, timeouts, and fail-fast behavior are widely used resilience patterns in production systems [1]. Site Reliability Engineering frames reliability work around service-level objectives, error budgets, monitoring, incident response, and automation [2]. Dynamic API routing complements these practices by making third-party provider selection an explicit control point. Tail latency and request duplication. Dean and Barroso show that tail latency dominates user experience in large-scale services and discuss techniques such as hedged requests [3]. Third-party routing must consider similar tail effects, but duplicate attempts may have side effects or cost implications. Therefore hedging and retries require operation-specific policy. Load balancing and traffic management. Systems such as Maglev demonstrate reliable software load balancing at scale [5]. Consistent hashing addresses distribution and remapping problems in distributed caching and storage [6]. The present work differs by focusing on semantic provider selection for external APIs, where metrics include business outcome, compliance, quota, cost, and regional capability rather than only server load. Streaming telemetry. Kafka introduced a distributed messaging system for log processing and durable event streams [4]. In the proposed architecture, event streams are used not only for offline analytics but also for operational feedback into routing decisions.
10
Autonomic and self-managing systems. Autonomic computing proposed systems that monitor, analyze, plan, and execute adaptations with human intent captured as policy [7]. Configuration-driven API routing follows a similar control-loop pattern but constrains the adaptation surface to provider eligibility and ranking for well-defined operations. Service mesh and API gateways. Service meshes and API gateways provide traffic management, retries, circuit breaking, observability, authentication, and policy enforcement. However, many deployments focus on service-to-service traffic within an organization’s infrastructure. The factor-list approach is specialized for external providers whose behavior includes business-level outcomes, contractual quotas, regional coverage, and provider-specific semantics.
support. Their practical perspectives on resilience, userverification workflows, incident response, and large-scale third-party integrations helped shape the architecture and lessons discussed in this paper.
References [1] M. T. Nygard. Release It! Design and Deploy Production-Ready Software. Pragmatic Bookshelf, 2nd edition, 2018. [2] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, editors. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016. [3] J. Dean and L. A. Barroso. The tail at scale. Communications of the ACM, 56(2):74–80, 2013.
[4] J. Kreps, N. Narkhede, and J. Rao. Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB Workshop, 2011. Third-party APIs are a critical part of modern application workflows, yet they create a reliability boundary that [5] D. E. Eisenbud, C. Yi, C. Contavalli, C. Smith, the application owner does not control. Static failover, R. Kononov, E. Mann-Hielscher, A. Cilingiroglu, B. retries, and timeouts are necessary but insufficient when Cheyney, W. Shang, and J. D. Hosein. Maglev: A fast provider behavior varies by region, quota, cost, latency, and reliable software network load balancer. In 13th and business outcome. This paper presented configurationUSENIX Symposium on Networked Systems Design and driven dynamic API routing as a practical architecture for Implementation (NSDI 16), pages 523–535, 2016. resilient third-party service integration. The central abstraction is the pluggable factor list: an [6] D. Karger, E. Lehman, T. Leighton, M. Levine, D. operation-specific configuration that combines hard eligiLewin, and R. Panigrahy. Consistent hashing and ranbility gates with weighted scoring functions. By separating dom trees: Distributed caching protocols for relieving routing policy from application code, the platform can hot spots on the World Wide Web. In Proceedings of adapt provider selection at runtime without redeployment. the Twenty-Ninth Annual ACM Symposium on Theory By feeding the router from event-stream telemetry, deciof Computing, pages 654–663, 1997. sions reflect recent provider behavior rather than static priority alone. By combining dynamic routing with circuit [7] J. O. Kephart and D. M. Chess. The vision of autonomic computing. Computer, 36(1):41–50, 2003. breakers and bulkheads, the system contains local failures while making global provider selection data-driven. The anonymized SMS verification case study illustrates the operational value of the approach. A workflow that previously depended on manual vendor switching can be transformed into an automated, observable, and continuously optimized routing process. The broader lesson is that redundancy becomes resilience only when the system can detect degradation, select a healthy alternative, and recover safely without relying on human intervention for routine provider incidents. Future work includes confidence-aware scoring under low sample volume, automated factor-list verification, replaybased policy testing, multi-objective optimization with explicit error budgets, and standardized decision traces for third-party API orchestration.
11
Conclusion
Acknowledgements The authors thank the Payment and Buyer Risk partners at eBay for their collaboration, feedback, and operational 11