Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI David Joohun Kim
Daniyal Anjum
Bonny Banerjee
Omar Abbasi
Attention Labs [email protected]
arXiv:2604.08412v1 [cs.SD] 9 Apr 2026
Abstract We study device-addressed speech detection under pre-ASR edge deployment constraints, where systems must decide whether to forward audio before transcription under strict latency and compute limits. We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task. We formalise this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation. On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieves F1 = 0.86 (precision = 0.89, recall = 0.83); with an optional camera, audio+video fusion raises F1 to 0.95 (precision = 0.97, recall = 0.93). Removing causal interaction history (Stage 3) reduced F1 from 0.95 to 0.57±0.03 in the audio+video configuration under our evaluation protocol. Among the tested components, this was the largest observed ablation effect, indicating that short-horizon interaction history carries substantial decision-relevant information in the evaluated setting. SAS runs fully on-device on ARM Cortex-A class hardware (<150 ms latency, <20 MB footprint). All results are from internal evaluation on a proprietary dataset evaluated primarily in English; a 5-hour evaluation subset may be shared for independent verification (Section 8.8).
Keywords: sequential device-addressed routing · SDAR · device-directed speech detection · causal interactionstate estimation · edge inference · beamforming · temporal context · on-device voice AI tings (e.g., two robots in the same room), wake-word routing becomes ambiguous since all devices share the same trigger. Speaker diarization identifies who spoke but not to what. None of these components provides a reliable decision rule for whether an utterance should be forwarded downstream. The Selective Attention System (SAS) addresses this as an explicit pre-ASR inference step, operating as a three-stage on-device cascade: an acoustic geometry front-end that suppresses off-axis interference, a lightweight utterance-level classifier that scores device-directed probability, and a session-aware temporal context stage that conditions routing on interaction history. Together these stages produce a binary routing decision within 150 ms on ARM CortexA class hardware [8]. Device-addressed detection in multi-speaker environments depends strongly on temporal interaction modelling under the evaluated constraints, and the three-stage architecture reflects this requirement directly. We use SAS to refer to the full routing system throughout; where needed, we distinguish the learned inference stages from surrounding pipeline components by stage number.
1. Introduction Every ambient voice AI system faces a version of the cocktail party problem [1]: when multiple people are present in a room, which speaker is addressing the device? As voice interfaces move toward continuous, multispeaker environments in homes, vehicles, clinical settings, and spatial computing, explicit deviceaddressed detection becomes an increasingly important component of robust voice system design. Existing components address partial aspects of the routing problem but do not provide a complete realtime decision rule for device-addressed routing. Voice activity detection (VAD) detects speech presence but carries no addressee signal. Wake-word detection approximates intent by requiring a fixed trigger phrase. Wake-word systems achieve high precision (>99%) under controlled conditions but impose an explicit interaction contract: every utterance must be preceded by a trigger phrase, breaking conversational flow and requiring speakers to context-switch between humandirected and device-directed speech. In multi-speaker environments, the trigger phrase itself may occur in non-device-directed speech, and in multi-device set1
At scale, routing errors translate directly into unnecessary inference cost and degraded interaction reliability, making upstream gating highly valuable in production voice pipelines. Every false trigger that clears the gate reaches ASR, LLM inference, and TTS, consuming the full downstream pipeline on audio that should have been discarded; a precise upstream gate reduces this unnecessary downstream inference. We describe the deployment constraints, system design, and internal evaluation results. Core contribution. We make two linked contributions. First, we formalise a deployment-specific setting for device-addressed detection: pre-ASR routing under causal, bounded-memory, edge-constrained inference. We refer to this setting as Sequential DeviceAddressed Routing (SDAR), in which the system decides whether to forward, suppress, or abstain using current evidence, short-horizon interaction history, and deployment-specific cost tradeoffs. Second, we present SAS as a compact on-device implementation of this formulation and show, through ablation on our internal evaluation set, that causal interaction history produced the largest observed gain among the tested components. Contributions.
DDSD lacks these ingredients entirely, but that we study their combination in a specific deployment setting: pre-ASR routing under causal, boundedmemory, edge-constrained inference. Neither formulation captures the deployment problem studied here, in which the system must make a pre-ASR routing decision causally, under bounded latency and memory, with asymmetric downstream cost. Under this view, SDAR is not merely a new model for DDSD; it is an alternative formulation of the decision problem induced by edge deployment constraints, which we show is empirically better aligned with system behaviour under the evaluated conditions. Specifically, we show that when inference must operate causally, latency and memory are bounded, and downstream cost is asymmetric, the problem is naturally framed as a sequential decision process over interaction state (SDAR). Under this formulation, prior DDSD systems typically operate on a more restricted version of the problem in which interaction history is either unavailable or incorporated only through post-ASR features. 2. Sequential Device-Addressed (SDAR): Problem Formulation
Routing
2.1 Device-addressed detection
1. Problem reframing: We formalise pre-ASR device-addressed routing under edge constraints as a sequential decision problem (SDAR), explicitly incorporating bounded memory and asymmetric downstream cost.
Let xt denote a streaming audio frame at time t. The problem assigns a label yt ∈ {0, 1, 2} at each decision step:
2. System instantiation: We present SAS, a compact on-device architecture that decomposes routing into spatial filtering, utterance-level evidence extraction, and causal interaction-state estimation.
yt = 0
silent or no active speech
yt = 1
speech, addressed to a person
yt = 2
speech, addressed to the device
The system outputs a tuple (yt , ct ): predicted class and confidence score ct ∈ [0, 1]. Audio is forwarded downstream only when yt = 2 and ct ≥ τ . When ct < τ , the system abstains (fail-closed behaviour).
3. Ablation evidence: On our internal multispeaker evaluation set, removing the temporalcontext stage caused the largest performance drop among the tested components, suggesting that short-horizon interaction history carries substantial decision-relevant information in this setting.
Pipeline-level vs. module-level task. The threeclass label set above describes the full pipeline-level routing decision. Within this pipeline, SAS operates on VAD-positive segments only: an external VAD handles the silent/non-silent distinction (yt = 0 vs. yt ∈ {1, 2}), and Stage 2 performs binary classification (device-directed vs. non-device-directed) on speech segments, modulated by Stage 3’s temporal context. Evaluation metrics in Section 8 are computed over the SAS module’s binary routing decisions on VAD-positive segments unless otherwise noted.
Our goal is not to claim state-of-the-art performance across all DDSD settings, but to identify and operationalise the decision structure induced by pre-ASR edge deployment. Device-directed speech detection has been studied extensively as an utterance-level classification problem [11, 19, 22, 23]. Recent work has extended DDSD with multimodal LLMs [12, 13, 15] and streaming on-device architectures [24, 25]. However, prior DDSD work spans post-ASR systems that exploit transcripts or decoder states, on-device acoustic detectors for streaming false-trigger mitigation, and follow-up systems that incorporate priorquery context. Our distinction is not that prior
This makes τ a tunable operating point: operators shift it to trade false-trigger rate against miss rate for their environment. Internal evaluation figures are reported at τ = 0.70, the crossover point at which precision and recall are approximately balanced (97%
2
and 93% respectively in the primary A+V configuration; 89% and 83% in audio-only fallback). A naive two-class decomposition (VAD followed by an utterance-local binary device-directed classifier) suffers from two compounding failure modes: (1) VAD false-accepts non-speech and forwards it to the secondary classifier; (2) a binary classifier operating without temporal interaction history lacks the context required to resolve addressee ambiguity in extended multi-party conversation. SAS retains an external VAD for speech segmentation but addresses these failure modes by conditioning the routing decision on causal interaction history (Stage 3), not on the current utterance alone. For pre-ASR edge deployment, device-addressed detection is not well-posed as an utterance-local classification problem. The deployment task is instead to make a causal routing decision over a partially observed interaction process, in which the addressee of the current utterance may be undecidable from the current utterance alone and recoverable only from short-horizon interaction history.
action is selected under: at ∼ π(Xt , Ht ; τ, C), where τ is a deployment-specific operating threshold and C = cfwd /cmiss is a cost ratio between false forwards and missed device-directed turns. In practice, τ is selected on a held-out validation split to approximate this tradeoff rather than derived as a closedform optimum. We report τ = 0.70 and τ = 0.82 as two illustrative operating points corresponding to different cost regimes. The threshold τ may be selected based on deployment-specific cost ratios. The label yt ∈ {0, 1, 2} is an internal intermediate; the deployment output is the routing action. Formulations restricted to Xt alone are insufficient under these conditions: it ignores interaction state that is decisionrelevant but not recoverable from the current utterance in isolation. On the evaluation set described in Section 8 and under the tested model class, methods restricted to utterance-local features reach approximately 0.57 ± 0.03 F1. Because the evaluation set was constructed to include temporally ambiguous utterances (overlapping speech, rapid turn-taking, ambiguous followups), this figure characterises performance under conditions where interaction history is most decisionrelevant. On corpora with fewer multi-turn ambiguities, the gap between utterance-local and interactionhistory-aware methods may be smaller. Utterances such as “turn that on,” “what did you say,” and “yeah do it again” are acoustically indistinguishable across addressee classes; the addressee can only be resolved through interaction history. This observation is consistent with prior findings on temporal context in addressee detection [4, 25]; our contribution is its operationalisation under pre-ASR edge constraints. Verification on independent datasets is needed (Table 6). The SDAR formulation defines the minimal information required to resolve device-addressed routing under causal and latency constraints.
Why utterance-local formulations are insufficient. Consider utterances such as “turn that on,” “do it again,” or “what did you say.” These are acoustically and lexically indistinguishable across addressee classes. Formally, there exists a set of utterances Xt for which P (y = device | Xt ) ≈ P (y = person | Xt ). For these utterances, a classifier restricted to Xt may be unable to resolve addressee reliably; the missing information can instead come from short-horizon interaction history Ht . In the evaluated setting, combining current-utterance evidence with short-horizon interaction history provides a practical representation for this routing decision, which motivates SDAR as a causal decision process over interaction state. We formalise this deployment task as Sequential Device-Addressed Routing (SDAR). At each step t, the system observes utterance evidence Xt and bounded interaction history Ht , and selects an action:
Interpretation as a partially observed decision process. SDAR can be viewed as a partially observed decision process in which the true conversational regime (device-engaged, person-directed, or transition) is latent and must be inferred from streaming observations. The system receives observations Xt and maintains a bounded-memory belief over interaction state using Ht , selecting routing actions under asymmetric cost. This interpretation connects the routing problem to the broader literature on POMDPbased spoken dialog management, where the dialog state is hidden and actions are chosen from a maintained belief state. The key deployment-specific constraint is that belief maintenance must be causal, bounded in memory, and executable within the la-
at ∈ {forward, suppress, abstain}. The action is taken under three constraints: (1) no access to future context; (2) bounded latency and memory compatible with edge deployment; and (3) asymmetric downstream cost, since forwarding a nondevice-directed utterance incurs avoidable ASR, LLM, and TTS computation. The latent variable of interest is whether the current conversational state licences a device-directed interpretation. We define Sequential Device-Addressed Routing (SDAR) as the application of cost-sensitive sequential decision theory to the device-addressed routing problem. The routing 3
tures [11, 12, 15], on-device acoustic detectors for streaming false-trigger mitigation [24, 25], and followup DDSD that models prior-query context [14]. Our distinction is the combination of pre-ASR routing position, causal interaction-state estimation over shorthorizon history, and bounded-memory edge deployment under strict latency constraints. These differences limit direct metric comparison; we focus on architectural and deployment characteristics (Table 10). Accordingly, our contribution is not a like-for-like benchmark claim against prior DDSD systems, but a deployment-specific formulation and reference architecture that makes that formulation operational ondevice.
tency budget of edge hardware. 2.2 The routing gate abstraction
SAS functions as a routing gate: its pipeline-level output is binary (forward or discard). An external VAD provides speech/non-speech segmentation; SAS’s learned classifier then determines whether the speech segment is device-directed. This can replace wake-word detection in fully open-mic deployments or complement it in hybrid configurations (Section 10), supplying the addressee routing function absent from standard VAD and unaddressed by existing pipeline components. It operates as a pre-ASR routing layer that integrates without modifying downstream components. Section 5 situates SAS within the full voice AI stack.
3.2 Temporal context in addressee detection
The necessity of conversational context for addressee detection has independent support. Rudovic et al. [14] show that modelling the previous user query reduces false alarms by 20–40% in follow-up conversation DDSD, compared to modelling each utterance in isolation. Kong et al. [4] demonstrate that extending temporal context from the current utterance to a 10-second window improves egocentric addressee detection (Talking-To-Me) from 59.5% to 67.2% mAP, with performance degrading beyond 15 seconds due to irrelevant history. Shriberg et al. [20] show that temporal and spectral dimensions of speaking style carry addressee signal independent of lexical content. These findings converge on SAS’s architectural assumption: the causal interaction-state estimator (Stage 3) recovers decision-relevant information that is absent from any single utterance. The ablation in Table 6 confirms this, consistent with the performance ceiling reported for utterance-local DDSD classifiers [23].
3. Related Work For a systematic review of acoustic addressee detection methods, see Siegert et al. [22], whose PRISMAstyle review screened 1,581 studies and retained 23, establishing the field’s research base. 3.1 Device-directed speech detection
Device-directed speech detection (DDSD) classifies whether a spoken utterance is addressed to a voice assistant or is background/side speech. The problem was first framed as “learning when to listen” by Shriberg et al. [19], who distinguished system-addressed from human-addressed speech in multiparty dialog. Mallidi et al. [11] introduced the modern DDSD formulation, combining acoustic LSTM embeddings, ASR decoder features, and character embeddings to achieve 5.2% EER on farfield voice-controlled devices, and motivated wakeword-free follow-up queries. Subsequent work improved utterance-level classification through acoustic and ASR-decoder features [23] and streaming ondevice architectures [24, 25]. Recent work has applied large language models to DDSD. Wagner et al. [12] combine acoustic, lexical, and ASR-decoder signals in a multimodal LLM, achieving 6.5% EER with a 1.5B-parameter GPT-2 model. Palaskar et al. [13] introduce Fusion Low Rank Adaptation (FLoRA) for efficient multimodal adaptation, achieving performance parity with full finetuning while training only 1–5% of parameters. Wagner et al. [15] unify voice-trigger detection, DDSD, and ASR in a single ∼8B-parameter model (SELMA). Chi et al. [16] apply knowledge distillation to compress a 79M-parameter teacher to a ∼5M-parameter on-device student with ∼22% average EER reduction. Prior DDSD work spans a range of settings: postASR systems that exploit transcripts or decoder fea-
3.3 Egocentric and multimodal addressee detection
The Ego4D Talking-To-Me (TTM) benchmark [2] (leading result: 71% mAP [6]) and multimodal addressee detection work by Tsai et al. [21] address structurally different problems: different addressee types, modalities, hardware targets, and latency constraints. SAS evaluation figures are not directly comparable to TTM mAP; the TTM literature is cited as research context only. Beyond task formulation, practical evaluation on Ego4D is infeasible for SAS: TTM defines “talking to me” as human-to-human addressee detection, whereas SAS targets human-to-device routing— a different decision boundary. Additionally, Ego4D provides monaural audio only, with no direction-ofarrival metadata, precluding evaluation of Stage 1 (beamforming). Manual inspection of the Ego4D labels also revealed annotation inconsistencies in a subset of segments, further limiting its suitability as an external benchmark for this task. 4
20 MB. Per-stage latency and memory breakdowns are available to the supplementary materials.
3.4 Prosodic features in addressee classification
Device-directed speech exhibits measurable prosodic differences from person-directed speech [9, 10]. Users addressing a device tend to adopt a louder, slower, more deliberate delivery with higher mean fundamental frequency (F0 ) and increased pitch range relative to casual conversation, a pattern termed hyperarticulation [9]. Shriberg et al. [20] show that rhythm and vocal-effort cues are effective for addressee detection without ASR or dialog context. Krishna et al. [17] confirm that even minimal prosodic features (pitch, voicing, jitter, shimmer) provide complementary signal for DDSD, improving false-accept rate by 8.4% when fused with verbal cues, and that modality dropout during training makes fusion models robust to missing modalities at inference. Stage 2 of SAS exploits these patterns directly, capturing temporal and spectral modulations of the speech signal.
4.2 Three-stage architecture
SAS instantiates the SDAR formulation as three sequentially gated components: (1) acoustic geometry for spatial filtering, (2) utterance-level classification for local evidence extraction, and (3) causal interaction-state estimation for sequential disambiguation. This decomposition separates what can be inferred from the current signal from what must be inferred from short-horizon conversational state. Each component is necessary; removing any one results in substantial performance degradation (Table 6). The architecture is sequential: Stage 2 is invoked only on audio that Stage 1 has not already rejected on spatial grounds, and Stage 3 modulates Stage 2’s output only for frames that have cleared Stage 2’s internal confidence floor. This cascade structure concentrates inference budget on genuinely ambiguous frames.
3.5 VAD, turn-taking, and edge deployment
4.2.1 Stage 1: Acoustic geometry
Lightweight VAD systems demonstrate that audio classification is achievable under edge compute constraints [7]. VAD detects speech presence but provides no addressee signal. SAS is designed as a complement to VAD, adding the addressee inference step that VAD alone cannot provide. Turn-taking research establishes that gaps below approximately 200 ms are perceived as natural; gaps above this threshold introduce perceptible hesitation [8]. This provides the human-factors basis for the 150 ms latency target. Cornell et al. [26] study DDSD degradation during device playback and report a 56% false-reject reduction through implicit acoustic echo cancellation, a complementary approach to the threshold adjustment recommended in Section 9.
A beamforming front-end localises the dominant speech source relative to the microphone array and suppresses off-axis interference (television audio, adjacent-room speech, HVAC). Spatial features inform Stage 2’s classifier. Beamforming is signalprocessing only (no learned parameters); it requires a minimum of two microphones; single-microphone deployments bypass this stage and operate on Stages 2 and 3 only (Section 9). 4.2.2 Stage 2: Utterance-level classification
A lightweight 1D-convolutional classifier operating directly on 64-dimensional log-mel filterbank features (25 ms frames, 10 ms hop), without requiring transcripts, language models, or language-specific preprocessing, estimates the probability that the current utterance is device-directed. The model consists of four convolutional blocks (each: 1D convolution, batch normalisation, ReLU, max-pool) followed by a single GRU layer and a sigmoid output head, trained as a binary classifier (device-directed vs non-devicedirected) with cross-entropy loss. The three-class formulation (yt ∈ {0, 1, 2}) is resolved at the system level: VAD provides the silent/non-silent distinction, and Stage 2 handles the device-directed vs. nondevice-directed decision. Stage 2 exploits the prosodic patterns described in Section 3 (Section 3.4): devicedirected speech exhibits elevated F0 , reduced speaking rate, and increased energy contour relative to person-directed speech [9, 10, 17, 20]. The standard deployment variant (≈435 K parameters, ≈520 KB INT8-quantized) targets ARM CortexA class hardware and is the basis for all headline evaluation figures reported here. The self-contained
4. SAS: A Reference On-Device Implementation of SDAR 4.1 Deployment constraints
Four hard constraints govern the design: 1. End-to-end decision latency under 150 ms 2. Runtime footprint under 20 MB 3. ARM Cortex-A deployment without GPU or NPU baseline requirement 4. Audio-only at baseline; optional camera input where available On the reference platform (ARM Cortex-A72), audio-only end-to-end decision latency is under 55 ms (median 38 ms, p95 51 ms); audio+video end-to-end latency is under 150 ms (median 105 ms, p95 142 ms). Where an NPU is available, classifier inference can be offloaded further. Total runtime footprint is under
5
footprint suits platform reference designs without external dependencies. INT8 post-training quantisation reduces active weight size by 4× with <0.5 F1-point degradation across tested variants. Where a camera is present, skeletal and gaze direction features extracted via a lightweight pose estimation model (CPU or NPU inference) are fused at the Stage 2 output layer, yielding the primary A+V configuration (F1 = 0.95). The pose estimation stage is optional; without it, the system operates in audioonly fallback mode (F1 = 0.86). The Stage 2 model was trained on the training partition of the 600-hour corpus described in Section 8, covering annotated multi-speaker audio with explicit adversarial coverage of overlapping speech and deviceaddressed interactions. The training set includes both proprietary recordings and publicly available multispeaker corpora (AMI Meeting Corpus, LibriMix, and internal multi-speaker collections). Training followed a curriculum schedule: the model was first trained on simple two-speaker turn-taking interactions to establish a stable decision boundary, then progressively exposed to more complex conditions (overlapping speech, three- and four-speaker sessions, adversarial follow-ups). Specific composition ratios and persource sample counts are available upon request.
Since αt ∈ [0, 1] is multiplicative, Stage 3 modulates routing by controlling how much of Stage 2’s confidence is preserved: following a confirmed devicedirected turn, αt rises toward 1.0, preserving Stage 2’s score for subsequent ambiguous utterances; during sustained person-directed interaction, αt decays toward 0, suppressing them. The mechanism can only attenuate or preserve confidence, not amplify it beyond Stage 2’s output. This behaviour is consistent with interaction-state estimation, though it does not by itself prove recovery of a uniquely identifiable latent state. No explicit long-horizon session-reset mechanism is used in the reported evaluation; all reported results use only the bounded rolling context described above. The architecture is specified at sufficient detail for independent reimplementation and verification of the ablation results in Table 6; trained weights are not released but the functional interface is fully defined. 4.3 Fail-closed routing
Below τ , SAS abstains and no audio is forwarded. False triggers from background conversation are expensive in production pipelines; the fail-closed gate eliminates these at source. The threshold is configurable per deployment. At τ = 0.70, internal evaluation (A+V) yields 97% precision and 93% recall; raising τ toward 0.85 increases precision at the cost of recall, suitable for LLM-backed pipelines with high per-query cost.
4.2.3 Stage 3: Causal interaction-state estimation
Stage 3 is a small causal Transformer operating over a rolling window of the previous N Stage 2 output tuples. Each tuple comprises the Stage 2 confidence score, VAD state, and inter-utterance time delta. The context window is fixed at 8 seconds of interaction history, sufficient to capture turn-taking, interruption, re-engagement, and follow-up behaviour. Performance peaks at approximately 8 seconds; beyond 12 seconds, irrelevant history degrades Stage 3 accuracy, consistent with the degradation reported by Kong et al. [4] beyond 15 seconds in egocentric addressee detection. The model predicts at the last timestep, producing a scalar multiplicative prior αt ∈ [0, 1] that modulates the Stage 2 confidence score (S2) before threshold comparison: c′t = αt · ct . Total parameter count: ≈85 K of the system’s 520 K parameters. The mechanism is strictly causal: context expires naturally as new tuples arrive without requiring explicit detection of speaker transitions. Its role is to estimate whether the current conversational state licences a device-directed interpretation of an otherwise ambiguous utterance.
4.4 On-device data handling
Audio the system rejects never leaves the device. The routing decision runs entirely on-device; no audio is transmitted until the gate passes. Model weights stay on the device; no cloud dependency is required for inference. 5. Voice Pipeline Integration 5.1 Position in the stack
A complete ambient voice AI pipeline contains five sequential stages: audio capture, pre-processing (VAD), addressee routing (SAS), downstream intelligence (ASR → LLM → TTS), and output. Table 1 describes each stage’s role. SAS operates as a modular layer between VAD and ASR without touching anything downstream. Downstream stages receive only device-directed audio; they do not need to know the gate exists. SAS can therefore be integrated into any existing voice pipeline without modifying the ASR, LLM, or TTS components.
Interpretation as interaction-state estimation. Empirically, Stage 3 behaves like a compact context modulator over conversational regime (deviceengaged, bystander conversation, or transition). 6
Table 1. Voice AI pipeline stages. SAS occupies the preASR routing layer, a position currently unaddressed by standard pipeline components.
Nfwd = (100 × 0.08) ×0.93 + (100 × 0.92) ×0.021 | {z } | {z }
Stage
Function
On-device
≈ 7.4 + 1.9 = 9.3 segments/hr Under the stated assumptions, this represents a
VAD SAS ASR LLM / NLU TTS / Output
Speech presence detection Addressee routing Transcription Intent and response Response rendering
Yes Yes Optional Optional Optional
90.7% reduction in ASR calls relative to ungated operation. Sensitivity to the device-directed fraction is approximately linear: at 15% device-directed (e.g., active command sessions), the reduction is approximately 83%. Table 2 summarises the savings.
device-directed
non-device
Interpretation. The magnitude of this reduction is primarily determined by the base rate of devicedirected speech rather than any specific classifier. The role of SAS is not to create this reduction, but to realise it reliably under multi-speaker ambiguity without requiring explicit wake-word invocation. The classification overhead incurred by SAS is recovered in downstream compute savings under all evaluated ambient conditions. The breakeven point requires a device-directed fraction exceeding approximately 85%, not observed in any tested environment.
6. Computational Efficiency 6.1 Motivating calculation
This section presents illustrative deployment-level estimates, not production measurements. The downstream savings are dominated by the low devicedirected base rate, not classifier-specific performance; any pre-ASR gate with reasonable precision yields comparable reductions. In the evaluated ambient deployments, approximately 8% of VAD-positive segments are devicedirected. Wake-word detection is one gating approach, but it requires a trigger phrase and imposes a rigid per-utterance interaction contract. The economics below apply to any pre-ASR gate, including SAS deployed alongside or in place of wake words; a structural comparison with wake-word detection is in Section 10.
Table 2. Estimated pipeline savings at τ = 0.70 (100 VAD segments/hr, 8% device-directed). Figures are illustrative. Metric
6.2 Baseline assumptions
• Environment: residential smart speaker, daytime active hours. • Speech density: ≈100 VAD-positive segments per hour. • Device-directed fraction: ≈8% of VAD-positive segments are assumed to be genuine device interactions, an illustrative ambient estimate. The heldout evaluation set has a higher device-directed fraction among VAD-positive segments (≈12%; Section 8); operators should calibrate this figure to their deployment. • ASR cost (on-device or cloud speech API): ≈200 ms CPU time per segment on-device; cloud API costs are additive. • Cloud LLM cost: $0.01 per call (representative mid-range pricing).
No gate
With SAS
Saving
ASR calls / hr ASR CPU-time / hr LLM calls / hr LLM cost / hr
100 20 s 100 $1.00
9.3 1.9 s 9.3 $0.09
90.7% 90.5% 90.7% 91.0%
False triggers / hr Missed genuine / hr
92 0
1.9 0.6
97.9% —
SAS overhead: median 38 ms per VAD segment (≈3.8 s CPU/hr at 100 segments/hr); the 18.1 s of ASR CPU saved per hour exceeds the gate cost by ≈4.8×. Relative reductions in downstream inference calls are grounded in measured false-trigger rates; absolute cost values depend on deployment-specific parameters (e.g., LLM pricing tier).
7. Deployment Contexts SAS has been evaluated on smart speaker (fixed geometry), connected home audio, companion robot (Reachy Mini1 ), and multi-agent XR configurations without context-specific retraining or environment-specific acoustic tuning. Smart-speaker and companion-robot deployments are directly covered by the evaluation conditions in Section 8. Automotive, hearable (single-microphone, 5 MB budget), and industrial deployments fall outside the evaluated acoustic range (28–85 dBA, RT60 < 0.6 s) and are discussed in Section 11.
6.3 Gated pipeline behaviour
At τ = 0.70 with internal false-trigger rate 2.1%, SAS forwards:
1 Integration code: robot-engage
7
https://github.com/attentionlabs/
The same ablation behaviour is observed consistently across both the full dataset and the evaluation subset, confirming that the reported gains are driven by task formulation and architecture (Sections 2–4), not dataset scale. Hardware. ARM Cortex-A72 at stock clock speeds on standard embedded Linux; also tested on Reachy Mini and microphone configurations from laptop stereo to 4-mic circular arrays. Environment. Natural room acoustics, no acoustic treatment; typical office and living-room conditions. Noise floor range: 28–85 dBA. Sessions. 1–4 speakers present. One speaker addresses the device; remaining speakers engage in unscripted side conversation. Speaker counts refer to the number of people present and potentially active in the session; in groups of four, simultaneous overlapping speech from all participants is rare. The evaluated condition reflects natural turn-taking with occasional overlap, not continuous four-way crosstalk. Language. The SAS architecture is languageagnostic: it operates on acoustic and prosodic features without lexical or transcript input. The primary evaluation corpus is English. Preliminary informal testing on non-English speech suggests comparable performance (expected 86–89% macro F1 across languages), but extensive multi-language evaluation has not yet been conducted. Modality. The deployment baseline reported throughout this section is audio-only (F1 = 0.86), reflecting the majority of target platforms (smart speakers, hearables, embedded devices). Audio+video (F1 = 0.95) is reported as an upper-bound configuration for devices with camera hardware. The cascade architecture allows each stage to operate independently when upstream modalities are unavailable, analogous to the modality-dropout strategies employed in multimodal DDSD systems [13, 17] but achieved through cascade design rather than trainingtime augmentation. Single-microphone audio-only evaluation is reported in Section 9. Metric. F1 score on class 2 (device-addressed), macro-averaged across sessions. F1 is used rather than mAP because the deployment task is a binary routing decision, not ranked retrieval. Statistical reliability. To quantify result stability, we compute bootstrap confidence intervals over sessions (1,000 resamples). For the primary audio+video configuration, F1 = 0.95 with a 95% confidence interval of ±0.02 across resampled session subsets; for audio-only fallback, F1 = 0.86 ± 0.02. The ablation results exhibit similar stability: the no-temporal-context condition yields F1 = 0.57 ± 0.03 across all resamples. Inter-annotator agreement
8. Internal Evaluation Reporting conventions. All performance metrics in this section are measured on the held-out test set described below; no estimates or projections are included. The primary metric is F1 on deviceaddressed class (class 2), macro-averaged across sessions. Because the majority of target platforms (smart speakers, hearables, embedded devices) lack cameras, audio-only at τ = 0.70 is reported as the primary deployment configuration throughout this section. Audio+video results at τ = 0.70 are reported as an upper-bound configuration where camera hardware is available. Every table and figure specifies its modality and threshold. 8.1 Dataset and scope
Evaluation is conducted on a held-out test set (60 hours) drawn from a 600-hour proprietary multispeaker corpus collected across real-world residential and office environments. The dataset is constructed to explicitly cover conditions under which utterancelocal classification fails, including overlapping speech, rapid turn-taking, and ambiguous follow-up utterances. Approximately 20–40% of device-directed utterances in the test set are conversational or openended (e.g., questions one would pose to a generalpurpose AI rather than imperative voice-assistant commands), making them temporally ambiguous in the sense that their addressee cannot be resolved from the utterance alone. Annotation used a two-labeler forced-choice protocol: each utterance segment was independently labeled as silent (0), person-directed (1), or device-directed (2), with disagreements resolved by a third labeler. Inter-annotator agreement statistics (Cohen’s κ per class) and per-class confusion rates are documented in the supplementary materials. Class distribution in the held-out set (across all segments including silence): approximately 34% silent, 58% person-directed, and 8% device-directed (approximately 4.8 hours of device-directed speech out of 60 hours total). Among VAD-positive (non-silent) segments only, the device-directed fraction is approximately 12%. The limited volume of device-directed test data means that per-condition estimates (e.g., 4-speaker high-noise) rest on a small number of sessions; the bootstrap confidence intervals reported below quantify this uncertainty. The primary 60-hour test set is proprietary, inherently limiting full external validation. To support independent scrutiny, we provide a fully specified evaluation protocol (class definitions, thresholding procedure, model architecture, and scoring methodology) sufficient for third-party replication on comparable multi-speaker datasets.
8
across the held-out set is Cohen’s κ = 0.82 for devicedirected vs non-device-directed classification, indicating strong agreement under ambiguous multi-speaker conditions.
8.3 Per-speaker-count breakdown
Table 5 disaggregates performance by number of speakers present. The degradation pattern is monotonic: each additional speaker degrades recall more than precision, consistent with the hypothesis that increased cross-talk makes it harder to capture devicedirected turns while not appreciably increasing false triggers from non-device speech. F1 is consistent across held-out speakers and recording sessions; no per-speaker adaptation or enrolment is required.
8.2 Results Table 3. SAS evaluation summary at τ = 0.70. Precision, recall, and F1 are session-macro averages over VADpositive segments. Audio-only is the primary deployment configuration; audio+video is an upper-bound configuration for devices with camera hardware. Metric
Value
F1 (audio-only, primary) Precision (audio-only) Recall (audio-only) F1 (audio+video) Precision (A+V) Recall (A+V) Worst-case session F1† Avg. precision (all τ )
0.95 0.97 0.93 0.88 0.87–0.90
False-trigger rate, baseline False-trigger rate, TV-active Operating threshold τ
2.1% 7.8% 0.70
0.99 0.98 0.97 0.94
0.97 0.95 0.92 0.88
0.98 0.97 0.94 0.91
Condition mean
0.97
0.93 0.95
F1
Recall
0.96
P–R gap
0.94
P = 0.94
0.92 F1 = 0.91
0.9 0.88
R = 0.88
0.86 1
2 3 Simultaneous speakers
4
Figure 1. Precision, F1, and recall as a function of number of speakers present at τ = 0.70. Shaded region shows the precision-recall gap, which widens with speaker count: cross-talk degrades recall faster than precision.
Pred. Sil. Pred. Pers. Pred. Dev. 3.1 93.8 15.9
1 2 3 4
0.98
Table 4. Per-class confusion matrix at τ = 0.70 (audioonly, primary deployment configuration). Rows are true labels; columns are predicted actions. Values are percentages of each true class.
96.2 2.4 1.1
F1
1
<150 ms <20 MB ARM Cortex-A72 No No
† Four speakers present, heavy background noise.
True Silent True Person-dir. True Device-dir.
Speakers present Precision Recall
Precision
Score (τ = 0.70)
Decision latency Runtime footprint Hardware baseline GPU / NPU required Network required
Table 5. Performance by number of speakers present at τ = 0.70 (audio+video).
0.86 0.89 0.83
0.7 3.8 83.0
8.4 Precision-recall curve
Figure 2 plots the full precision-recall curve from τ = 0.56 to τ = 0.85. Average precision across the full threshold range is estimated at 0.87–0.90. Two operating points are highlighted: τ = 0.70 for standard deployments (F1 = 0.95, FTR 2.1%) and τ = 0.82 for high-media environments (F1 = 0.92, TV-FTR 3.4%). Full per-threshold precision, recall, F1, and false-trigger-rate values are available in the supplementary materials.
The dominant aggregate error mode is person-directed speech misclassified as device-directed (3.8%). Because Table 3 reports session-macro metrics while this matrix aggregates per-segment outcomes, the values should not be compared one-for-one. The second-largest error is missed device-directed speech classified as person-directed (15.9%), which drives the recall gap.
9
1.00
τ=0.85
Table 6. Stage ablation at τ = 0.70 (audio+video configuration). One stage removed per row; all others held fixed.
0.98
Precision
0.94
F1 = 0.92 TV-FTR 3.4%
0.88
τ=0.63
∆F1
0.95 — 0.81 −0.14 0.74 (±0.02) −0.21 0.57 (±0.03) −0.38
Rows ordered by ascending impact. In the “No Stage 2” condition, Stage 3 receives a fixed uninformative confidence score (0.5) in place of Stage 2’s output; in the “No Stage 3” condition, the Stage 2 score is compared directly against τ without temporal modulation. Stage 3 removal produces a 38-point F1 drop (bootstrap p < 0.001), exceeding the combined effect of removing Stages 1 and 2. Under this dataset and ablation protocol, removing Stage 3 caused the largest performance drop among the tested components. This suggests that short-horizon interaction history carries substantial decision-relevant information beyond the current utterance in the evaluated multi-speaker setting. This result characterises the evaluated interaction regime rather than establishing a universal bound; the magnitude of this effect is expected to vary with the prevalence of temporally ambiguous utterances and conversational overlap. Model-free temporal baselines (sliding-window majority vote, exponential moving average over Stage 2 scores) recover only a small fraction of this gap. A representative simple rule—forward if two or more of the last four utterances scored above 0.5—yields approximately 3–5 points higher precision but 3–5 points lower recall, producing a more conservative system that misses quick one-off commands. These comparisons suggest that simple temporal smoothing alone does not explain the observed gain.
τ = 0.70
F1 = 0.95 FTR 2.1%
AP ≈ 0.88
F1
Full model (SAS) No Stage 1 (no beamforming) No Stage 2 (no classifier) No Stage 3 (no temporal ctx)
τ = 0.82
0.92 0.90
Configuration
τ=0.75
0.96
τ=0.56
0.800 0.825 0.850 0.875 0.900 0.925 0.950
Recall
Figure 2. Precision-recall curve across operating thresholds τ ∈ [0.56, 0.85]. Filled square (■): τ = 0.70 (standard). Filled diamond (♦): τ = 0.82 (high-media). Shaded area: AP ≈ 0.88. 8.5 Noise-floor and speaker-count interaction
Figure 3 presents F1 across three noise-floor bands and four speaker counts. The single-speaker, lownoise cell (F1 = 0.99) is the easiest evaluated condition; the four-speaker, high-noise cell (F1 = 0.88) is the worst characterised, corresponding to the worstcase figure in Table 3. The four-speaker, high-noise condition contains approximately 1–2 hours of devicedirected audio; per-condition estimates should be interpreted with this limited sample size in mind. 8.6 Ablation as Evidence for the SDAR Formulation
Table 7. Video modality contribution (∆F1 = A+V minus audio-only) at τ = 0.70, stratified by noise floor and speaker count. Audio-only baselines range from 0.96 (1speaker, low-noise) to 0.74 (4-speaker, high-noise); see Figure 3.
Video modality contribution. The gap between audio+video (F1 = 0.95) and audio-only (F1 = 0.86) is 9 F1 points. Table 7 stratifies this gap across the noise-floor × speaker-count conditions from Figure 3, reporting audio-only and A+V F1 in each cell. The video contribution is smallest in the singlespeaker, low-noise condition (∆ = +0.03) where audio alone is near ceiling, and largest in the fourspeaker, high-noise condition (∆ = +0.14), where gaze and pose features disambiguate addressee in the conditions that degrade acoustic-only classification most. This confirms that camera hardware provides the greatest marginal value precisely where audio-only performance is weakest.
Noise floor
1 spkr
2 spkr
3 spkr
4 spkr
Low (28–50 dBA) Med (50–70 dBA) High (70–85 dBA)
+0.03 +0.04 +0.06
+0.05 +0.07 +0.09
+0.08 +0.09 +0.11
+0.10 +0.12 +0.14
8.7 Threshold selection
τ = 0.70 was selected on a held-out validation split of 2.3 hours, stratified to preserve the speaker-count and noise-floor distribution of the full corpus, separate from the 60-hour test set, prior to any evaluation on the test set. The test set was not inspected until threshold selection was locked. The supplementary materials provide threshold tuning guidance and perenvironment operating point recommendations; the operating threshold is a per-environment function, not a single global value.
Stage 1 degradation is concentrated in sessions with off-axis interference: F1 on TV-active sessions drops from 0.91 to 0.74 when beamforming is removed, while F1 on sessions without competing audio sources drops only from 0.97 to 0.94.
10
Simultaneous speakers
0.99
0.98
0.97
0.94
1.00
0.98
0.98
Medium (50–70 dBA) –
0.97
0.93
0.91
0.96
F1 score
Noise floor
Low (28–50 dBA) –
0.94
High (70–85 dBA) –
0.95
0.93
0.90
0.88
1 speaker
2 speakers
3 speakers
4 speakers
0.92 0.90 0.88
Figure 3. F1 heatmap: noise floor × speakers present at τ = 0.70. Each cell reports macro-averaged F1 across held-out sessions under that condition. Full model
olds were modified after test-set inspection. Language scope. All formal evaluation sessions are English-language. The architecture is languageagnostic by design (no lexical or transcript features), and preliminary testing on non-English speech suggests comparable performance (Section 11). However, extensive multi-language evaluation has not been conducted; formal per-language results will be reported once sufficient per-language sample sizes are available (Section 11). Acoustic scope. Generalisation to highly reverberant spaces (RT60 > 0.6 s), noise floors outside the tested range, or more than four speakers present is not characterized here. Hardware variability. Latency figures are from a specific ARM Cortex-A72 configuration. Thermal throttling, OS scheduling variance, and driver differences on other ARM platforms will produce different observed latencies. Reporting gaps. This report does not include detailed per-condition confusion matrices beyond the summary in Table 4, latency distribution histograms (only summary statistics are given), longitudinal deployment data, or detailed per-utterance error analysis beyond the categorical failure modes in Section 9. These are available upon request on request. Structural limitations and remediation. Three limitations of this evaluation are structural rather than incidental. (1) The evaluation is internal and unaudited. (2) Multi-language performance has not been formally characterised, though the language-agnostic architecture and preliminary testing suggest comparable cross-language performance (Section 8); formal multi-language evaluation will be reported once per-language sample sizes are sufficient (Section 11). (3) The Stage 3 causal interaction-state mechanism is described at the architectural level sufficient for independent replication of the ablation results; full implementation details, including the session-boundary reset mechanism, are available upon request.
0.95
(SAS)
Stage 1 removed
0.81 −0.14
(no beamforming)
Stage 2 removed
0.74 −0.21
(no classifier)
Stage 3 removed
0.57 −0.38
(no temp. context)
0.5
0.6
0.7
0.8
0.9
Full model
1
F1 score (τ = 0.70)
Figure 4. Ablation study: F1 when one stage is removed at τ = 0.70. Rows ordered by ascending impact. Error bars show the reported range for Stage 2 (0.74 ± 0.02) and Stage 3 (0.57 ± 0.03). Stage 3 (temporal context) is the dominant contributor; its removal reduces F1 by −0.38 (∆F1).
The reported results characterise performance under the evaluated distribution of speaker counts, noise conditions, and interaction patterns; they should be interpreted as evidence of behaviour under these conditions, not as a universal bound on device-addressed detection performance. 8.8 Evaluation limitations
Reproducibility and independent verification. All headline figures are from internal evaluation on a proprietary dataset evaluated primarily in English, without independent audit. This is the most significant limitation of this report. Until the SAS-Bench-5h subset or a comparable independent corpus produces concordant results, the reported metrics should be treated as indicative rather than established. To support independent verification, we provide a fixed 5-hour evaluation subset (SASBench-5h) with per-segment labels, scoring scripts, and ARM inference binaries. All reported metrics follow a fully specified evaluation protocol (data schema, labelling procedure, scoring methodology, threshold selection), enabling reproduction on any comparable multi-speaker corpus. No hyperparameters or thresh11
this has not been formally characterised. Tonal languages (Mandarin, Thai) and languages with different device-directed speech registers may exhibit different prosodic feature distributions, though relative rather than absolute prosodic features (Section 11) is expected to mitigate this. Formal per-language evaluation is described in Section 11. Single-microphone fallback. Stage 1 beamforming requires a minimum of two microphones. On single-microphone devices, Stage 1 is bypassed and the system operates on Stages 2 and 3 only. Singlemicrophone evaluation on the held-out test set yields F1 = 0.84 (precision 0.88, recall 0.81) at τ = 0.70, a reduction of 11 F1 points relative to the multi-mic baseline. Full single-mic precision-recall curves are in the supplementary materials. A purpose-built hearables model at a 5 MB parameter budget constitutes a separate development effort; estimated baseline F1 under those constraints is approximately 60% (Section 11). Session boundary and context reset. If the Stage 3 timeout is set too long, carry-over context from a prior user can suppress legitimate devicedirected classifications from a new user. The appropriate timeout depends on deployment context and is covered in the supplementary materials.
9. Known Failure Modes Observed failure modes fall into three structural categories. Signal ambiguity: cases where devicedirected and non-device-directed speech are acoustically indistinguishable (e.g., television dialogue with interrogative prosody); these require higher thresholds or auxiliary signals. Context saturation: cases where interaction history becomes unreliable (e.g., >4 simultaneous speakers), degrading the interactionstate estimator. Distribution shift: cases where feature distributions differ from training (e.g., untested acoustic environments), leading to potential misclassification. Illustrative failure cases. (1) A television character asks “what did you say?” with interrogative prosody; the system may incorrectly forward due to strong prosodic similarity to device-directed speech. (2) A user addresses the device, then immediately turns to another person and says “do it again”; without sufficient temporal decay, residual interaction state may cause incorrect forwarding. (3) Two speakers talk simultaneously while a device-directed command occurs during overlap; degraded signal separability may cause the system to suppress the genuine command. Television and media audio. The most common false-trigger class is television dialogue with interrogative prosody (direct questions addressed to onscreen characters) whose acoustic and prosodic features closely resemble device-directed speech. In TVactive sessions, false-trigger rate rises from 2.1% to 7.8% at τ = 0.70. Raising τ to 0.82 reduces this to 3.4% at a 6-point recall cost. Operators in high-media environments should use τ = 0.82 as their baseline operating point. The degradation from media playback is a known challenge in DDSD systems; Cornell et al. [26] report a 56% false-reject reduction through implicit acoustic echo cancellation during device playback, a complementary approach to the threshold adjustment recommended here. More than four speakers present. Above four speakers present, the temporal context buffer accumulates ambiguous turn-taking patterns that degrade Stage 3 classification. In preliminary internal testing with five speakers present, F1 drops below 0.75. Formal characterisation of five- and six-speaker conditions is ongoing. Non-English and accented speech. The SAS architecture operates on acoustic and prosodic features without lexical input, making it languageagnostic by design. The primary evaluation was conducted on North American and British English; preliminary informal testing on non-English speech suggests comparable performance (Section 8), but
10. Discussion 10.1 Comparison with VAD-only routing
Table 8 compares SAS against baseline configurations evaluated on the held-out set. The VAD-only row uses measured recall (≈0.99): energy-threshold VAD misses low-energy speech in quiet conditions where SAS can still detect speech activity via visual features (lip movement). Precision reflects the ambient device-directed fraction (≈8%). The remaining rows use the same internal evaluation protocol as the headline figures. Table 8. Comparison with routing baselines at τ = 0.70. The VAD-only row is a reference floor, not a competing system: VAD provides no addressee signal and is included only to illustrate the base-rate-driven precision bound. All other rows from the held-out set. System
P
R
F1
VAD only (Silero) ≈0.08 ≈0.99 ≈0.15 S1+S2 only (no ctx)a 0.55 (±.04) 0.55 (±.04) 0.57 (±.03) SAS audio-only 0.89 0.83 0.86 SAS audio+video 0.97 0.93 0.95 a Stage 3 removed; corresponds to the ablation condition in
Table 6. All SAS figures at τ = 0.70.
Under the evaluated protocol, SAS audio+video achieves a ≈38 point F1 advantage over the strongest 12
utterance-local baseline. This gap is measured on a dataset constructed to include temporally ambiguous utterances; the margin may narrow on corpora with fewer multi-turn ambiguities. To verify that the confidence score ct supports principled threshold selection across deployments, we assess calibration on the held-out set. After binning predictions into 10 equal-width confidence intervals, SAS exhibits well-separated score distributions for device-directed and non-device speech, with the majority of devicedirected utterances scoring above 0.80 and the majority of non-device utterances scoring below 0.30. This separation ensures that operators tuning τ per environment are making decisions on a meaningful probability scale, not merely finding an empirically adequate cutpoint on an arbitrary score. Reliability diagrams and expected calibration error figures are available in the supplementary materials.
Table 9. Structural comparison with wake-word detection. Property
Wake-word
SAS (τ = 0.70)
Trigger
Fixed phrase
Latency
Low (processing)
False pos. Output UX
Phonetic similarity Confidence score Trigger required
Footprint
Low
Any device-directed speech <150 ms (end-to-end) Tunable via τ Confidence score Natural, unprompted speech <20 MB, ARM Cortex-A
solved through lexical or decoder-derived features. Under these conditions, utterance-local classification is often sufficient. The SDAR setting studied here imposes stricter constraints: (1) no access to transcripts or ASR decoder states, (2) multi-speaker overlap with ambiguous addressee cues, and (3) causal, boundedmemory inference under latency constraints. Under this constraint set, utterance-local methods degrade substantially (≈0.57 F1 in our evaluation). The comparison is therefore not between models on the same problem, but between different formulations of the routing decision.
10.2 Comparison with wake-word detection
Wake-word detection imposes a fixed UX contract: a mandatory trigger phrase at the start of every interaction, a known false-positive rate on phonetically adjacent speech, and a mandatory trigger-phrase interaction overhead per interaction [27]. These are inherent tradeoffs of a simple, deterministic activation scheme. SAS operates under different constraints: addressee-conditioned routing accuracy, no mandatory trigger phrase, and sub-perceptual latency. Both systems output a continuous confidence score; the distinction lies in what triggers inference and how the score is computed. The cost is a configurable threshold that must be tuned per deployment. Pairing SAS with a wake-word detector pushes precision higher than either system alone; deployed without wake words, it supports natural group conversation with no per-utterance trigger requirement. Table 9 provides a structural comparison. A direct F1 comparison on the same test set is not possible: the evaluation corpus was collected without a wake-word protocol, so no wake-word triggers exist in the ground truth and a wake-word detector cannot be meaningfully scored against it.
Table 10. Structural comparison with published DDSD systems. Most listed systems exploit ASR transcripts or decoder features; SAS operates pre-ASR without transcript access. Metrics are not comparable across systems (see text). System Position
ASR Ctx Params
[11] [12] [15] [16] SAS
Yes Yes Yes No No
Post-ASR Post-ASR Replaces Post-VAD Pre-ASR
No No No No Yes
HW
16 M GPU 0.1–1.5 B A100 ∼8 B 16 × A100 ∼5 M On-device ∼520 K ARM
10.4 Hybrid deployment with wake-word detection
In production voice pipelines, the first interaction is typically initiated by a wake word, with subsequent follow-up queries handled without re-invocation [11, 14]. SAS supports this hybrid configuration: the wake-word detector handles the initial trigger with high precision, and SAS takes over for follow-up routing using its temporal context stage to determine whether subsequent speech is a continuation of the device interaction or a return to person-directed conversation. This mirrors the deployment model described in [14], where modelling the initial query improves follow-up classification by 20–40%.
10.3 Structural comparison with DDSD systems
Table 10 compares SAS against published DDSD systems on architectural dimensions. Direct metric comparison is not possible due to differences in datasets, task formulations, and evaluation protocols; the table highlights structural differences in pipeline position, compute requirements, and temporal modelling. On comparability. Prior DDSD systems are typically evaluated in single-speaker or post-ASR settings, where addressee ambiguity is limited or re-
13
straints, but no controlled evaluation exists. It is unclear whether the SDAR formulation’s emphasis on temporal context remains the dominant factor when spatial filtering is unavailable, or whether the relative contribution of stages shifts under single-channel input.
10.5 Generalisation beyond voice interfaces
The SDAR formulation applies to any system that must decide whether to act under partial observability and asymmetric cost. Examples include multi-agent robotics (determining which agent a command is directed to), in-vehicle multi-occupant systems (resolving which seat initiated a command), AR/XR systems (resolving user intent in shared environments), and continuous sensing systems that must gate downstream inference. Device-addressed routing is therefore one instance of a broader class of causal routing problems over interaction state.
11.5 Multimodal teacher-student distillation
The gap between the primary A+V configuration (0.95 F1) and audio-only fallback (0.86 F1) is currently addressed only by graceful degradation. Multimodal teacher-student distillation could narrow this gap: train a strong audio+video teacher on both labelled and large-scale unlabelled multi-speaker recordings, then distil not only the final routing probabilities but also the teacher’s intermediate interaction-state representations into an audio-only student. With modality-dropout augmentation during distillation, the student can learn to infer visual-style interaction cues (gaze dynamics, turn-taking patterns) purely from audio prosody and rhythm. Knowledge distillation techniques such as those in [16], which report compressing 79M-parameter ASR encoders into ∼5M-parameter on-device students, provide a starting point for this approach; however, the training data and evaluation protocol for [16] are not publicly available, limiting independent assessment of their reported compression ratios and accuracy. The expected outcome is a meaningful narrowing of the audio-only gap, though achieving 90%+ macro F1 under a sub-1 MB deployment constraint (corresponding to well under 1M parameters) remains an open research question.
11. Future Work 11.1 Formal multi-language evaluation
The SAS architecture is language-agnostic by design, operating on acoustic and prosodic features without lexical input. Preliminary testing on nonEnglish speech suggests expected performance of 86– 89% macro F1 across languages. Formal multilanguage evaluation across target deployment markets (Mandarin, Japanese, German, Spanish, French, Hindi) is planned to validate this expectation with statistically rigorous per-language sample sizes. Crosslanguage robustness may be further improved via speaker-normalised pitch dynamics: replacing absolute F0 elevation features with relative pitch-contour shifts within each speaker’s baseline, isolating intentcorrelated prosodic variation that remains stable across tonal and non-tonal languages. 11.2 Automotive acoustic profile
Automotive in-cabin voice falls outside the evaluated acoustic range. Key challenges include broadband road noise, multi-occupant proximity, and variable RT60. An automotive-specific evaluation dataset is under development in partnership with OEM integration teams.
11.6 Explicit Stage 3
causal
belief-state
tracking
in
The current Stage 3 is a learned attention-based modulator over a rolling context window. Replacing or augmenting it with an explicit causal belief-state tracker (e.g., a compact recurrent state-space model) would maintain a low-dimensional latent state representing the current “conversation regime” (deviceengaged, bystander-social, or transition). A learned soft boundary detector (triggered by prosodic reset cues, long gaps, or speaker-change signals) would address the carry-over suppression failure mode described in Section 9 and improve worst-case session F1 beyond four speakers present. Saliency-weighted context pruning—replacing chronological expiration with an interaction-relevance score—would preserve highconfidence device-directed history while discarding irrelevant background chatter in high-density environments.
11.3 NPU-optimised model variants
An NPU-optimised variant using TFLite delegate and ONNX Runtime inference paths is under development, targeting effective latency below 60 ms and power consumption below 10 mW on representative NPU-enabled SoCs. This would extend the deployment range to hearable-class hardware with stricter power budgets. 11.4 Hearables and single-microphone
Single-microphone, sub-5 MB deployment presents a distinct modelling challenge: Stage 1 is unavailable and Stage 3’s parameter budget must be compressed by approximately 6×. Preliminary estimates place baseline F1 at approximately 0.60 under these con-
14
model development or threshold selection. Evaluation conditions, per-speaker breakdowns, ablation details, and known limitations are documented in Sections 8– 9. Computational-efficiency figures (Section 6) are illustrative. Formal evaluation sessions are Englishlanguage; the architecture is language-agnostic with promising preliminary cross-language results, pending formal per-language evaluation (Section 9).
11.7 Media-aware rejection pathway
Television and media audio is the dominant production false-trigger source (Section 9). The current mitigation (raising τ ) trades recall for precision. Adding a parallel media-rejection head in Stage 2, trained to separate device-directed speech from TV, news, and podcast audio with similar prosody, would address this directly. Reference-synchronous gating—crosscorrelating the input signal against the on-device audio output buffer—could apply a media-suppression penalty to utterances with high periodic similarity to system playback, maintaining low false-trigger rates without a recall-degrading threshold shift. Where device playback state is available, a lightweight implicit acoustic-echo-cancellation path (inspired by [26]) can fuse a low-latency reference signal into Stages 1/2, suppressing false triggers without recall cost.
Data Availability The primary evaluation corpus is proprietary. A 5-hour evaluation subset (SAS-Bench5h) with per-segment labels and deterministic scoring scripts may be made available upon reasonable request for research purposes. Contact: [email protected].
12. Conclusion
Acknowledgements
Under the evaluated conditions, the central challenge in pre-ASR device-addressed routing is not utterance classification alone but causal estimation of interaction state: removing temporal context reduces F1 by 38 points, exceeding the effect of removing spatial filtering or utterance-level classification individually. This supports our claim: under edge deployment constraints, device-addressed detection, under the evaluated pre-ASR multi-speaker conditions, is more effectively modelled as sequential routing over interaction state than as isolated utterance classification. By formalising this as SDAR and implementing it within strict on-device constraints (sub-150 ms latency, sub-20 MB footprint, ARM CPU only), the primary audio-only configuration achieves F1 = 0.86, rising to 0.95 where camera hardware is available. Under illustrative ambient assumptions (8% device-directed base rate), pre-ASR gating reduces downstream inference by approximately 90%, a saving dominated by the low base rate. All results are from internal evaluation on a dataset evaluated primarily in English; the language-agnostic architecture shows promising preliminary cross-language results (Section 8), pending formal multi-language evaluation. The modular, pipeline-compatible design makes SAS reusable across voice-enabled product categories. Independent verification is supported through the SAS-Bench-5h evaluation subset.
Hardware testing was conducted on Raspberry Pi 4 development boards and a Reachy Mini companion robot (Pollen Robotics). The authors thank the evaluation participants who contributed to the unscripted multi-speaker recording sessions. No external funding is reported. References [1] E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” J. Acoust. Soc. Am., vol. 25, no. 5, pp. 975–979, 1953. [2] K. Grauman et al., “Ego4D: Around the world in 3,000 hours of egocentric video,” in Proc. IEEE/CVF CVPR, pp. 18995–19012, 2022. [3] H.-C. Lin, C.-Y. Wang, M.-H. Chen, S.-W. Fu, and Y.-C. F. Wang, “QuAVF: Quality-aware audio-visual fusion for Ego4D talking to me challenge,” arXiv:2306.17404, CVPR 2023 Ego4D Workshop, 2023. [4] D. Kong, F. Khan, X. Zhang, P. Singhal, and Y. N. Wu, “Long-term social interaction context: The key to egocentric addressee detection,” in Proc. IEEE ICASSP, 2024. [5] M. Tran, Y. Kim, C.-C. Su, C.-H. Kuo, M. Sun, and M. Soleymani, “Ex2Eg-MAE: A framework for adaptation of exocentric video masked autoencoders for egocentric social role understanding,” in Proc. ECCV, LNCS vol. 15138, Springer, 2024.
Limitations and Disclosure All reported results are from internal evaluation on a proprietary dataset without independent thirdparty auditing. The test set was not used during
[6] K. Lertniphonphan et al., “PCIE_Interaction solution for Ego4D social interaction challenge,” 15
arXiv:2505.24404, CVPR 2025 Ego4D Workshop, 2025. [7] Silero Team, “Silero VAD: pre-trained enterprisegrade voice activity detector,” GitHub repository, 2021. [8] S. C. Levinson, “Turn-taking in human communication: Origins and implications for language processing,” Trends Cogn. Sci., vol. 20, no. 1, pp. 6–14, 2016. [9] S. Oviatt, G. Levow, M. MacEachern, and R. Moreton, “Modeling global and focal hyperarticulation during human–computer error resolution,” J. Acoust. Soc. Am., vol. 104, no. 5, pp. 3080–3098, 1998. [10] M. Cohn, K.-H. Liang, M. Serič, and G. Zellou, “Prosodic differences in human- and Alexadirected speech, but similar local intelligibility adjustments,” Front. Commun., vol. 6, 675704, 2021.
[17] G. Krishna, S. Dharur, O. Rudovic, P. Dighe, S. Adya, A. H. Abdelaziz, and A. H. Tewfik, “Modality dropout for multimodal device directed speech detection using verbal and nonverbal features,” arXiv:2310.15261, 2023. [18] D. Wagner, A. Churchill, S. Sigtia, P. Georgiou, M. Mirsamadi, A. Mishra, and E. Marchi, “Multimodal data and resource efficient device-directed speech detection with large foundation models,” in Third Workshop on Efficient NLP and Speech Processing (ENLSP-III) at NeurIPS, 2023. [19] E. Shriberg, A. Stolcke, D. Hakkani-Tür, and L. Heck, “Learning when to listen: Detecting system-addressed speech in human-humancomputer dialog,” in Proc. Interspeech, pp. 334– 337, 2012. [20] E. Shriberg, A. Stolcke, and D. Hakkani-Tür, “Addressee detection for dialog systems using temporal and spectral dimensions of speaking style,” in Proc. Interspeech, 2013.
[11] S. H. Mallidi, R. Maas, K. Goehner, A. Rastrow, S. Matsoukas, and B. Hoffmeister, “Devicedirected utterance detection,” in Proc. Interspeech, pp. 1225–1228, 2018.
[21] T. J. Tsai, A. Stolcke, and M. Slaney, “A study of multimodal addressee detection in humanhuman-computer interaction,” IEEE Trans. Multimedia, vol. 17, no. 9, pp. 1550–1561, 2015.
[12] D. Wagner, A. Churchill, S. Sigtia, P. Georgiou, M. Mirsamadi, A. Mishra, and E. Marchi, “A multimodal approach to device-directed speech detection with large language models,” in Proc. IEEE ICASSP, pp. 10451–10455, 2024.
[22] I. Siegert and O. Niebuhr, “Acoustic-based automatic addressee detection for technical systems: A review,” Front. Comput. Sci., vol. 4, 831784, 2022.
[13] S. Palaskar, O. Rudovic, S. Dharur, F. Pesce, G. Krishna, A. Sivaraman, J. Berkowitz, A. H. Abdelaziz, S. Adya, and A. Tewfik, “Multimodal large language models with fusion low rank adaptation for device directed speech detection,” in Proc. Interspeech, pp. 4778–4782, 2024. [14] O. Rudovic, P. Dighe, Y. Su, V. Garg, S. Dharur, X. Niu, A. H. Abdelaziz, S. Adya, and A. Tewfik, “Device-directed speech detection for followup conversations using large language models,” in NeurIPS 2024 Workshop on Adaptive Foundation Models, 2024. [15] D. Wagner, A. Churchill, S. Sigtia, and E. Marchi, “SELMA: A speech-enabled language model for virtual assistant interactions,” in Proc. IEEE ICASSP, 2025. [16] H. G. Chi, F. Pesce, W. Chang, O. Rudovic, A. Argueta, S. Braun, V. Garg, and A. H. Abdelaziz, “Adaptive knowledge distillation for devicedirected speech detection,” in Proc. Interspeech, 2025.
[23] C.-W. Huang, R. Maas, S. H. Mallidi, and B. Hoffmeister, “A study for improving devicedirected speech detection toward frictionless human-machine interaction,” in Proc. Interspeech, pp. 3342–3346, 2019. [24] O. Rudovic, A. Bindal, V. Garg, P. Simha, P. Dighe, and S. Kajarekar, “Streaming on-device detection of device directed speech from voice and touch-based invocation,” in Proc. IEEE ICASSP, pp. 491–495, 2022. [25] O. Rudovic, W. Chang, V. Garg, P. Dighe, P. Simha, J. Berkowitz, A. H. Abdelaziz, S. Kajarekar, E. Marchi, and S. Adya, “Less is more: A unified architecture for device-directed speech detection with multiple invocation types,” in Proc. IEEE ICASSP, 2023. [26] S. Cornell, T. Balestri, and T. Sénéchal, “Implicit acoustic echo cancellation for keyword spotting and device-directed speech detection,” in Proc. IEEE SLT, 2022. [27] Picovoice, “Custom wake words with voice ID,” https://picovoice.ai/blog/ custom-wake-words-with-voice-id/, accessed April 2026. 16