1
SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios
Abstract—Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning based systems. Sound source localization (SSL) has achieved remarkable success with deep learning, yet most methods localize all active sources without selectivity. Conversely, target sound extraction (TSE) extracts sources using multimodal prompts but typically fails to preserve the multichannel spatial information required for accurate localization. To bridge this gap, we formulate the task of prompt-guided selective target sound localization and propose SelectTSL, an end-to-end architecture that localizes only the user-specified target in multi-source acoustic scenes. Specifically, we design an target-aware selective localization strategy that employs a Prompt-Guided Selective Attention Module (PGSA) to generate prompt-informed embeddings. These embeddings guide an inter-channel phase difference (IPD) enhancer to refine raw phase cues, fusing with target magnitudes to jointly estimate direction of arrival (DoA) and target-source cardinality (i.e., the number of target sound sources). This coupled design effectively focuses on the user-specified target spatial cues for selective localization and also handles time-varying numbers of target sources. Extensive experiments on both synthetic data and real-world recordings demonstrate that our proposed method consistently outperforms other baselines and exhibits robust generalization to real acoustic environments. Dataset and code will be released.
I. I NTRODUCTION OUND source localization (SSL), which estimates the spatial location of acoustic events, supports a wide range of array-based audio and speech applications. For instance, in smart speakers, SSL enables direction-aware target speech enhancement and far-field automatic speech recognition (ASR) by steering microphone-array beamformers [1]. In hearing aids, it supports directional noise reduction via spatial filtering [2]. Despite remarkable progress of SSL via signal processing (e.g., generalized cross-correlation with phase transform (GCC-PHAT) [3], multiple signal classification (MUSIC) [4]) or deep neural networks (e.g., convolutional recurrent neural networks (CRNNs) [5]), existing methods are inherently semantic-blind, treating all acoustic sources as generic signals without identity awareness: all active sources are localized without selectivity, including interfering sounds. This differs from human auditory perception where listeners can selectively attend to a target sound when competing speakers and background noise exist, i.e., the “cocktail party problem” [6]. As illustrated in Fig. 1(a), conventional SSL systems indiscriminately localize both target and irrelevant sources (e.g., dog barks and noise), yielding non-selective direction-ofarrival (DoA) trajectories and preventing users from directing the system to focus on a specific target such as speech.
S
Conventional SSL (Semantic-Blind)
Ours (SelectTSL: Prompt-guided selective)
Interference (Dog barks)
Target (Speech)
Noise Target Trajectory
Interference Trajectory
Interference (Dog barks)
Target (Speech)
Noise Target Trajectory
Prompt Input
TSE-DOAnet
arXiv:2607.02343v1 [cs.SD] 2 Jul 2026
Ziyang Jiang, Student Member, IEEE, Yu Chen, Zexu Pan, Member, IEEE, Xinyuan Qian, Senior Member, IEEE, Bowen Xing, Ivor W. Tsang, Fellow, IEEE, Xu-Cheng Yin, Senior Member, IEEE, Haizhou Li, Fellow, IEEE
“locate the speech”
Dual-channel microphone
Another Speech
Localizes ALL sources
Selective DoA Estimation
Localizes ONLY the target source Output: Selective SSL
Output: Non-selective
(a)
Interference Trajectory
(b)
Fig. 1. Illustration of conventional semantic-blind SSL and our proposed prompt-guided SelectTSL: (a) Conventional SSL localizes all active sources, yielding non-selective DoA trajectories, whereas (b) SelectTSL uses a text and audio prompt (“locate the speech”) to focus on the target speech which only provides its corresponding DoA trajectory.
To associate semantics with location, sound event localization and detection (SELD) methods [7], [7], [8] detect and localize all events of known classes. However, they perform passive scene analysis that lacks an interactive mechanism to filter sources based on user intent [9]. Conversely, in the field of target sound extraction (TSE), methods [10]–[13] extract specific sound sources guided by audio or text prompts. Nevertheless, TSE focuses more on waveform reconstruction process rather than spatial sensing, thus spatial cues like interchannel phase difference (IPD) are often degraded for precise SSL. This leads to a fundamental mismatch: TSE is aware of what to extract but not where to localize, whereas conventional SSL knows where sources are but not which semantic target to follow. Therefore, it remains an open problem to build a unified framework that is both semantic-aware and locationaware. In this paper, we propose the SelectTSL network. Unlike standard approaches, SelectTSL is an end-to-end localization framework, where prompt-guided target extraction guides subsequent spatial estimation. In particular, a Prompt-Guided Selective Attention (PGSA) module, conditioned by multimodal prompts (text or audio), acts as a semantic filter to purify target-consistent features from the audio mixture. These features are then fused with spatial cues (e.g., IPD) and fed into a dedicated DoA estimator. This design decouples semantic selection from spatial estimation, enabling robust localization of user-specified targets even in noisy and multisource environments. Our contributions are listed as: 1) We formulate a prompt-guided selective target sound
2
localization task in dynamic, noisy and multi-source environments, where sources are moving and the number of active target sources is unknown and varies over time. The goal is to localize only the user-specified target(s) while suppressing interference. 2) We propose SelectTSL, an end-to-end framework that supports both text and audio prompts. A prompt-guided PGSA module produces extraction-informed embeddings (EIEs) that condition an IPD Enhancer to refine spatial phase cues, which are fused with target magnitudes for DoA estimation. 3) We handle unknown and time-varying active targets via a lightweight cardinality head that jointly predicts DoA heatmaps and frame-level target-source counts, enabling stable localization and tracking under moving speakers and intermittent activity. 4) Extensive experiments on synthetic and real data demonstrate consistent improvements over strong baselines in localization and tracking, and confirm robust generalization under realistic room acoustics. II. R ELATED W ORK Conventional SSL approaches excel at estimating spatial direction but are typically prompt-agnostic, localizing all active sources indiscriminately. Conversely, TSE provides controllable target selection via auxiliary cues to reconstruct clean sound. However, it prioritizes signal fidelity over spatial consistency, often distorting the spatial information needed for precise localization. Prompt-guided selective target sound localization bridges the gap between semantic-aware extraction and spatial-aware localization, effectively unifying these two objectives. Despite its promise, existing studies in this emerging field are still limited by unimodal prompting (text or audio only) and restricted evaluation scenarios (e.g., handling at most a single target per query, with limited evaluation in moving-source and noisy environments). The following section reviews literature in TSE, SSL, and prompt-guided localization. The significant methods summarized in Table I. A. Target Sound Extraction (TSE) TSE aims to extract a specific type of sound source (e.g., speech, music, or other acoustic events) from a mixture, conditioned on explicit auxiliary cues. Existing methods have explored auxiliary cues ranging from visual signals [14]–[17], spatial information [18], [19], reference audio [10], [20]–[22] and semantic text descriptions [11], [23]–[25]. While visual cues are highly effective in speech-oriented settings by leveraging articulatory motions (e.g., lip/face movements) [14], [15], they are not generally applicable to arbitrary acoustic events. Spatial cues used as auxiliary prompts often require additional priors (e.g., array geometry or target direction) beyond the mixture audio, and are thus beyond our scope. Therefore, we focus on text and audio cues in the remainder of this section. For text-prompted extraction, the core mechanism involves mapping a natural language caption (e.g., “piano”, “glass
breaking”) into a semantic embedding space, utilizing pretrained contrastive models such as Contrastive LanguageAudio Pretraining (CLAP) [24]. This embedding then acts as a condition to steer the separation network, predicting a timefrequency mask or waveform residual that extracts the source matching the description [11], [25], [26]. In contrast, audio-prompted extraction utilizes a reference audio clip as the enrollment cue. This mechanism generally operates in two distinct modes based on the granularity of the guidance: (i) query by example, where the reference shares the same semantic class as the target but differs in instance (e.g., using a generic dog bark to extract a specific dog) [27]; and (ii) target enrollment, commonly used in TSE, which utilizes a clean reference utterance to guide the extraction of the target speaker [10], [20], [22]. Architecturally, these prompt encoders are integrated into separation backbones, such as Conv-TasNet [28], DualPath Recurrent Neural Network (DPRNN) [29] and TFGridNet [30], via varying fusion mechanisms, ranging from simple feature-wise linear modulation (FiLM) to more complex cross-attention layers. However, while prompt-guided TSE offers fine-grained controllability over which source to recover, it is primarily optimized for signal reconstruction fidelity (e.g., scale-invariant signal-to-noise ratio (SI-SNR)) rather than spatial consistency. Unless explicitly designed to preserve inter-channel phase consistency, the reconstruction objective of TSE tends to distort the spatial cues essential for localization. In essence, TSE determines what to extract, yet remains agnostic to where the source is located. B. Sound Source Localization (SSL) In contrast to TSE that controls which source to extract, SSL addresses where sources are by estimating DoA from multichannel audio inputs [31]. Recent advances have largely adopted deep learning to map time-frequency (TF) and spatial features to DoA estimates. In parallel, the vision community has studied audio-visual sound source localization in videos, where visual context is used to select and localize the sounding object [32]–[34]. Contemporary systems generally follow a common pipeline: a feature extraction backbone followed by a task-specific head. Despite this shared framework, these methods differ primarily in two aspects: (i) the choice of input representations, ranging from raw short-time Fourier transform (STFT) spectrograms to spatially explicit cues such as IPD, inter-channel level difference (ILD), covariance eigenvectors, and generalized cross correlation (GCC) [7], [35]; and (ii) the modeling strategy for these representations, spanning from bandwise processing that preserves narrow-band spatial cues to full-band mechanisms that capture cross-frequency correlations [36], [37]. We organize this subsection along these two aspects to explain the sources of recent performance gains. Typical SSL models map TF features to DoA via classification or regression heads using deep neural backbones, such as convolutional neural networks (CNNs), CRNNs, Conformer, and Transformer [7], [38]–[41]. Beyond raw STFT features, many systems explicitly encode spatial cues. For example,
3
TABLE I BASELINE TAXONOMY WITH TRACK - SLOT ( TRACK - WISE ) PRIORITY. C ATEGORIES MARKED† ARE NON – TRACK - WISE ( NO FIXED - TRACK OUTPUTS ).
Category
Model
Task family
Input / Feature
Target
Track-wise
Multi-ACCDoA IPDNet MIMO-DoAnet EINV2 embed-ACCDoA DiffTrack-DoA SALSA / SALSA-Lite SE-ResNet NGCC-SELD CST-Former DCASE25
SELD SSL SSL SELD SELD SSL SELD SELD SELD SELD SELD
Mch. STFT Mch. STFT Mch. STFT FOA CLAP-aug. feat. Mch. STFT SALSA / SALSA-Lite ResNet/SE enc. NGCC + SELD pipe. CST attention Stereo (2-ch) audio
Multi-ACCDoA DoA (via DP-IPD) Multi-out DoA Track-wise SELD ACCDoA DoA + diff. track loss DoA + act. DoA + act. DoA + act. DoA + act. DoA + act.
SELD+DoA†
SWG Former SELDnet SELDT
SELD SELD SELD(T)
Transformer mod. Mel/FOA Transformer SELD
DoA + act. ACCDoA DoA + act.
Pure DoA†
FN-SSL SRP-DNN SALADnet
SSL SSL SSL
FB+NB fusion DP cues + SRP feat. FOA
DoA (via DP-IPD) (1src) SRP spectrum Attn. DoA
Prompt-based†
KeywordLoc Class-cond. SELD Text-Queried SEL LocSelect GCC-Speaker
SSL (query) SELD (query) SEL (query) SSL (query) SSL (query)
Mch. feat. + kw cue Mch. feat. + 1hot cls (FiLM) GCC-PHAT + txt emb. 2-ch spec. + enroll sp. GCC-PHAT + spk-wt
kw-cond. spk DoA cls-cond. (DoA+act.) txt-cond. DoA/traj enroll-cond. spk DoA spk DoA
Abbrev. Mch.=multichannel; FOA=First-order Ambisonics; FB/NB=full-/narrow-band; STFT=short-time Fourier transform; CLAP=Contrastive Language-Audio Pretraining; ACCDoA=activity-coupled Cartesian DoA; DP-IPD=direct-path inter-channel phase difference; SRP=steered response power; NGCC=neural generalized cross-correlation; SE=squeeze-and-excitation; enc.=encoder; feat.=feature; mod.=module; pipe.=pipeline; act.=activity; diff.=differentiable; aug.=augmented; spk=speaker; spec.=spectrogram; kw=keyword; cls=class; txt=text; traj=trajectory; 1hot=one-hot; wt=weighting; † Non–track-wise (no fixed-track outputs).
SALSA augments log-spectrograms with the normalized principal eigenvector of the spatial covariance [35], and SALSALite uses normalized IPD [42]. Spatial correlation features such as GCC-PHAT and its neural variant neural GCCPHAT (NGCC-PHAT) are also commonly used in recent SSL systems [3], [43], [44]. In addition to refined inputs, recent work improves robustness by modeling direct-path cues and cross-frequency structure. For example, FN-SSL and IPDNet use full- and narrow-band fusion to estimate direct-path interchannel phase difference (DP-IPD) and infer DoA (e.g., via template matching) [36], [37]. SRP-DNN learns direct-path delays and phase, and injects them into steered response power (SRP)-style spectra to refine spatial peaks [45]. Moreover, MIMO-DoAnet predicts source-wise pseudo-spectra to reduce reliance on post-hoc heuristics [46]. Beyond DoA estimation, SELD extends the SSL objective by jointly estimating the sound event type. In this case, SELD models can be used as SSL baselines by solely using its localization branch. For instance, ACCDoA and MultiACCDoA [47], [48] use activity-coupled Cartesian DoA (ACCDoA) vectors as regression targets, with auxiliary duplicating permutation invariant training (ADPIT) to handle sound overlaps of the same class. Another line of work adopts explicit track slots with permutation-invariant training. Representative methods such as EIN [49] employ separate attentionbased tracks and have been widely adopted in challenge systems [40], [50]. Variants mainly differ in encoders/backbones
(e.g., SALADnet [38], SwG-former [39], CST-Former [41]) and feature fusion strategies [51]. The Detection and Classification of Acoustic Scenes and Events (DCASE) 2025 Task 3 baseline further shows an audio-only stereo setting with a CRNN and a Multi-ACCDoA-like output [52]. While the aforementioned methods focus on frame-level DoA estimates, moving-source scenarios require associating predictions over time. For example, SELDT [53] forms trajectories by linking frame-wise source activity and DoA estimates. Differentiable tracking [54] further integrates identity assignment (e.g., Hungarian matching) into training to optimize sequence-level objectives. Although effective for trajectory estimation, such track-based designs inherently rely on a fixed number of output slots and lack a mechanism for user-specified target selection. This limitation motivates prompt-guided localization reviewed below. C. Prompt-guided Localization Prompt-guided localization conditions a DoA estimator on a user-specified cue, enabling it to localize the queried target selectively rather than all active sources indiscriminately. Existing work has explored this idea through two modalities: text-based conditioning for semantic descriptions and audiobased conditioning for acoustic enrollment. On the text side, early work already explored text-like cues in the form of fixed keywords. Keyword-based speaker localization [55] introduces the task of localizing the speaker
4
who uttered a predefined trigger phrase (e.g., a wake word) in the presence of overlapping speech. Beyond fixed keywords, class-conditioned localization provides a related formulation where the query is a target event class rather than free-form text. A class-conditioned SELD framework [56] utilizes onehot class indicators as conditioning cues, injecting them into an ACCDoA-based baseline via FiLM layers. This design enables the model to selectively estimate the DoA of the queried class while treating concurrent events from other classes as interference during training. More recently, SEL [57] extends text conditioning to free-form captions and benchmarks fusion schemes that combine textual embeddings (e.g., CLAP, BERT, or FlanT5) with spatial audio features such as GCC-PHAT for selective DoA prediction and trajectory estimation. For audio-based localization, LocSelect [58] studies target speaker localization given an enrollment utterance of the same target speaker. It first predicts a speaker-dependent spectrogram mask to suppress interferers and then estimates the target DoA from the filtered spectrogram using a long shortterm memory (LSTM)-based localizer [58]. Alternatively, approaches like GCC-Speaker [59] adapt classical spatial features directly, learning speaker-dependent weights for GCCPHAT to enhance selectivity in multi-speaker scenarios. Overall, existing prompt-guided localization studies are predominantly unimodal, utilizing either text or audio cues but not both within a unified model. Moreover, evaluations are often confined to simplified settings, with limited coverage of moving targets, noisy mixtures, or prompts that correspond to multiple targets. III. TASK F ORMULATION Let us consider a dual-channel recording setup. The signal received at the m-th microphone (m ∈ 1, 2), denoted by xm (t), is modeled as the sum of J source signals convolved with their respective room impulse responses (RIRs), together with additive ambient noise: J(t) X sj (t) ∗ hm,j (t, θj (t)) + nm (t), (1) xm (t) = j=1
where sj (t) is the j-th source signal, J(t) is the number of active sources at time t, nm (t) is the additive noise at the mth microphone, and hm,j (t, θj (t)) represents the RIR between the j-th source and the m-th microphone, which is critically dependent on the time-varying DoA of the source θj (t). ⋆ indicates convolution. Our goal is to learn a prompt-guided localization model Mψ , parameterized by ψ, designed to estimate the spatial trajectory of a target source specified by a multimodal prompt. Unlike traditional SSL systems, the proposed SelectTSL is conditioned on a user-provided cue. It maps the dual-channel mixture signal x(t) = [x1 (t), x2 (t)]T , together with an auxiliary audio cue xcue (t) or a textual prompt xtext , to two corresponding frame-level predictions: T ×Θ • The DoA Posteriorgram, P̂DoA ∈ [0, 1] out denotes frame-level DoA confidence maps, where Tout is the number of output frames after temporal alignment and Θ = 180 is the number of azimuth bins discretized from 0◦ to 179◦ at 1◦ resolution. Due to front-back ambiguity
in symmetric dual-microphone arrays, predictions are restricted to a 180◦ range, and the multi-label formulation allows multiple active bins per frame with continuous confidence values. T ×3 • The Source Cardinality, P̂card ∈ [0, 1] out denotes frame-level cardinality predictions, a probability distribution over the presence of 0, 1, or 2 active target sources. We optimize the model parameters ψ to jointly predict the target sources’ DoA posteriorgram P̂DoA and frame-level cardinality P̂card , as formally defined by: (P̂DoA , P̂card ) = Mψ (x(t)|xcue (t), xtext ).
(2)
IV. S ELECT TSL A RCHITECTURE SelectTSL is an end-to-end framework for prompt-guided selective target sound localization, in which a PGSA module generates extraction-informed embeddings from a dualchannel mixture and a DoA estimator jointly predicts framelevel DoA and source cardinality. The overall architecture is shown in Fig. 2 and is described as follows. A. Prompt-Guided Selective Attention Module (PGSA) We introduce the PGSA module, which acts as a promptguided selective filter. It leverages multimodal prompts (text or audio) to extract target-specific representations from the input mixture. 1) Audio Encoder: Given a dual-channel signal xm (t) for m ∈ {1, 2}, we compute complex spectrograms Sm (τ, ν) by short-time Fourier transform (STFT) using a Hann window, where τ ∈ {1, . . . , Tin } and ν ∈ {1, . . . , F } represent time frames and frequency bins, respectively. We extract • Magnitude cues: Am (τ, ν) = |Sm (τ, ν)|, which are encoded by a Conv1D followed by a ReLU module into (enc) Hm ∈ RDemb ×Tin : H(enc) = ReLU(Conv1D(Am )). m •
(3)
Spatial cues: IPD and inter-channel level difference (ILD):
IPD(τ, ν) = ∠S1 (τ, ν) − ∠S2 (τ, ν), (4) |S1 (τ, ν)| + ϵ . (5) ILD(τ, ν) = log |S2 (τ, ν)| + ϵ 2) Prompt Encoders: We employ a frozen CLAP model to extract audio cue xcue (t) and text xtext : caudio = MLPcue (CLAPaudio (xcue (t))) ∈ R256 ,
(6)
ctext = MLPtext (CLAPtext (xtext )) ∈ R256 .
(7)
Here, MLPcue and MLPtext serve as projection layers. During training, we freeze the CLAP encoders and optimize only the projection layers. We then form a unified guidance vector cfused = [caudio ; ctext ] ∈ R512 . In our model, the audio prompt xcue (t) is taken as the first 1 s of a 6 s clip, and the remaining 5 s of that clip is used as the target segment for selection and localization. 3) Fusion Layer: We condition the encoded magnitude (enc) features Hm using a two-stage FiLM cascade driven by cfused : FiLM(h, c) = γ(cfused ) ⊙ h + β(cfused ), (8)
5
Sem head
Over-and-Add
Inter-block
Intra-block
Chunking
Semantic Branch
Aligner
BiGRU
✅ ❌ ❌
SE
Norm&ReLU
DWConv1d
IPDenh
Pre-proj
Ac head
×
Projection
DoA(°) 180
PRTM
×2
FiLM
DoA
(c) Progressive Refinement Temporal Modeling Module (PRTM)
Acoustic Branch
Cross-Attention
Norm
Angle Refiner
time
Fuse head & IPD Delta
Extraction Network
Card Head
0
(b) IPD Enhancer
Fusion Layer
Nsources ={0, 1, 2, ...}
90
DWS Conv2d
(a) Fusion Layer and Extraction Network
Progressive Refinement Temporal Modeling Module (PRTM)
IPDenh
Semantic Branch
Fusion Layer
Projection & Concat
Audio Cue Encoder
Text Encoder
"Locate the cat"
Extraction Network
ReLU& Conv
ILD
TCNBlocks
ILD
❄️
Text Cue
Acoustic Branch
cos&sin
❄️
Audio Cue
DoA Estimator
IPD Enhancer
IPD
FRBlocks
Audio Encoder
Prompt-Guided Selective Attention Module (PGSA) C &S 0 0
Gated-FiLM
Mixed Audio
TCN block d=1 TCN block d=2
Forward GRU
TCN block d=4
Backward GRU
TCN block d=8
Fig. 2. Overall architecture of SelectTSL. The PGSA module outputs the target magnitude spectrogram and a stack of EIEs H(eie) (from the Extraction Network). The DoA estimator enhances spatial cues with these EIEs and aggregates semantic & spatial information for localization and cardinality estimation.
where γ(·) and β(·) are learnable functions that map the conditioning vector to scale and shift parameters. First, cfused (enc) modulates the full-dimensional Hm , followed by a projection. Then, a downsampled guidance modulates the projected (fused) features, yielding Hm . (fused)
into overlap4) Extraction Network: We segment Hm ping chunks of length Lchunk with 50% overlap, and process them using Ndprnn dual-path blocks (intra-/inter-chunk RNNs from DPRNN [29]). Inside the stack, we periodically inject cross-attention between the current audio features and the semantic guidance cfused . Let H ∈ RT ×Dh denote the framelevel features in a block and cfused ∈ RDc the guidance vector. We broadcast the guidance across time as C̃ ∈ RT ×Dc and compute cross-attention: Q = C̃WQ , Q
K = HWK ,
Dc ×dk
K
V
V = HWV ,
(9)
Dh ×dk
with W ∈ R and W , W ∈ R . The attention weights and residual update are QK⊤ , H ← H + αV. (10) α = softmax √ dk After the DPRNN stack, we project its output to a soft mask Mm , which filters the encoder features and a decoder reconstructs the target magnitude: Ĥm = Mm ⊙ H(enc) m ,
Âm = MLPdec (Ĥm ).
(11)
We use the extracted target features Â1 ∈ RF ×T for DoA estimation, where F and T denote frequency bins and frames. From the last KEIE DPRNN blocks (before overlap-andadd), we export extraction-informed embeddings (EIEs). Each block’s features are linearly projected to a TF feature map in
RF ×Tin , as illustrated in the Extraction Network (bottom-left) (eie) of Fig. 2. Averaging across the two channels yields Hk ∈ RF ×Tin , k = 1, . . . , KEIE . Stacking forms (eie) (eie) H(eie) = stack H1 , . . . , HKEIE ∈ RKEIE ×F ×Tin . (12) These maps emphasize TF regions that are relevant for extracting the target and serve as guidance for IPD enhancement. B. DoA Estimator The DoA estimator consists of two modules: an IPD Enhancer that refines spatial cues, and a Progressive Refinement Temporal Modeling Module (PRTM). Although the PGSA module outputs Âm for each channel m, these per-channel magnitude features are highly redundant. Therefore, we use only Â1 as the magnitude input to the DoA estimator. The IPD Enhancer takes the mixture spatial cues (IPD, ILD), the extracted target features Â1 from the PGSA module, and the H(eie) from the Extraction Network. It then outputs the enhanced IPD IPDenh . The PRTM operates on Â1 , IPDenh , and ILD to produce (i) a frame-level DoA posteriorgram over Θ azimuth bins and (ii) a categorical distribution over the source-count set N = {0, 1, . . . , Nmax } (with Nmax = 2 in our experiments), where Nmax is the maximum number of simultaneously active target sources. 1) IPD Enhancer: We refine the mixture IPD with an IPD Enhancer that is explicitly conditioned on both low-level spatial cues and high-level EIEs (Fig. 2). Acoustic branch. The input to the acoustic branch comprises Â1 , C0 = cos(IPD), S0 = sin(IPD), and ILD. These features are concatenated to form a 4-channel spectrogram
6
representation, which is subsequently processed by an acoustic feature encoder (denoted as ac head). This module employs a lightweight depthwise-separable Conv2d (DWS Conv2d) to efficiently capture fine-grained spectro-spatial features. Semantic branch. Operating in parallel, this branch exclusively processes H(eie) from the Extraction Network. It is encoded by a semantic head into a conditioning representation that drives a gated FiLM modulation. Specifically, global pooling is employed to derive channel-wise scale and shift parameters, while a concurrent spatial gate modulates the acoustic features to enable spatially selective IPD enhancement. Fusion and residual prediction. Let Aac denote the acoustic features produced by the acoustic branch and Asem the semantic features produced by the semantic head. We fuse them by applying FiLM conditioning followed by a gated modulation: Ã = 1 + g(Asem ) ⊙ FiLM(Aac , Asem ), (13) where g(·) is a spatial gating head that modulates the FiLMconditioned features to enable spatially selective IPD enhancement. A fuse head then predicts cosine–sine residuals (IPD deltas), and the enhanced IPD is recovered by the fourquadrant inverse tangent atan2(·, ·): C0 = cos(IPD), S0 = sin(IPD), (∆C, ∆S) = ∆IPD (Ã),
(14)
F ×Tin
IPDenh = atan2 S0 + ∆S, C0 + ∆C ∈ R
.
2) Progressive Refinement and Temporal Modeling (PRTM) Module: For each frame τ , we concatenate the features: zτ = Â1 (:, τ ); IPDenh (:, τ ); ILD(:, τ ) . (15) Stacking over time gives Zin ∈ RTin ×Cfeat . The sequence is then processed by a PRTM module. First, the input is projected and processed by a stack of Feature Refinement Blocks (FRBs), whose composite operation is denoted as R(·). This module progressively fuses three distinct cues: target magnitude, enhanced IPD, and ILD. Each FRB is implemented using depthwise-separable 1D convolutions, squeeze-and-excitation modules, and residual connections. This progressive fusion projects the inputs into a unified feature space where spatial and spectral evidence are tightly coupled. On top of this fused representation, a dilated temporal convolutional network (TCN) and a bidirectional gated recurrent unit (BiGRU) focus on modeling longer-range temporal patterns: H(temp) = BiGRU TCN R(Linear(Zin )) ∈ RTin ×2Dgru . (16) A temporal alignment layer adjusts the frame rate to obtain H(aligned) = T H(temp) ∈ RTout ×2Dgru . (17) The alignment layer T (·) compresses the sequence to a fixed length Tout , so that the prediction heads operate at the same temporal resolution as the labels. 3) Prediction Heads: The aligned features H(aligned) ∈ R are decoded by the Angle Refiner (Fig. 2), which is implemented as two parallel heads. Let projDoA (·) and projcard (·) denote two learned frame-wise affine projections Tout ×2Dgru
that output logits, applied row-wise to H(aligned) : (18a) P̂DoA = σ projDoA H(aligned) ∈ [0, 1]Tout ×Θ , P̂card = softmax projcard H(aligned) ∈ [0, 1]Tout ×|N | , (18b) where projDoA : R2Dgru → RΘ and projcard : R2Dgru → R|N | . During inference, we strictly couple the two predictions frame-wise. First, the number of active sources is estimated as n̂t = arg maxn∈N P̂card (t, n). Then, the DoA estimates are derived from the top-n̂t peaks of P̂DoA (t, :). Specifically, peak picking involves: (i) circular Gaussian smoothing on P̂DoA (t, :) to mitigate jaggedness while handling the 0◦ -180◦ wrap-around; (ii) identifying strict local maxima that exceed an adaptive threshold τt = max(µt + σt , τmin ) (where µt , σt are the frame-wise mean and std, and τmin = 0.3); and (iii) selecting the n̂t highest peaks as the final DoA predictions. 4) Training Objective: The network is jointly trained with three loss terms. The separation loss is the negative SI-SNR averaged over the two channels: 1 (19) Lsel = − SI-SNR(ŝ1 , s1 ) + SI-SNR(ŝ2 , s2 ) . 2 For localization, we use a Binary Cross Entropy (BCE) for frame-level DoA estimation and a Cross Entropy (CE) for source-count prediction: LDoA = BCE P̂DoA , P⋆DoA , Lcard = CE P̂card , n⋆ , (20) where P⋆DoA ∈ [0, 1]Tout ×Θ is a soft DoA label built from ground-truth (GT) azimuths, and n⋆ ∈ {0, 1, 2}Tout is the frame-wise source-count label. The total loss is a weighted sum of separation, DoA and cardinality terms: L = ωsel Lsel + ωDoA LDoA + ωcard Lcard ,
(21)
where we fix ωsel = 1 and tune the other weights to balance the influence of each task’s gradient. This is crucial as the losses operate at different scales, especially near convergence. The selection loss Lsel (negative SI-SNR) typically converges to a magnitude on the order of O(101 ). In contrast, LDoA is a BCE averaged over Tout ×Θ bins. Due to the sparse nature of the DoA target (at most 2 of 180 bins are active), a converging model predicts near-zero loss for ≈ 99% of the outputs. Consequently, the average loss LDoA becomes very small, e.g., O(10−1 ) or O(10−2 ). To prevent the DoA gradient from vanishing and ensure its influence remains comparable to the separation term (i.e., ωDoA LDoA ≈ ωsel Lsel ), we need to compensate for this scale mismatch. Therefore, ωDoA is chosen to be on the order of O(102 ). V. E XPERIMENTAL SETTING A. Dataset Our model is trained and evaluated on a synthesized dataset that covers a wide range of acoustic conditions. The synthesis process combines clean source signals with simulated dynamic RIRs and ambient noise. a) Sound Sources: To ensure diversity in our target sound events, we collected source signals from several public datasets. Specifically, speech signals were collected from the
7
chosen to cover the main indoor archetypes represented in the dataset, span distinct surface materials and room volumes (rock and plastic-coated surfaces, carpet vs. hard floors, glass Item Value partitions), and include both trajectory types provided by the Hours: 288.9 / 31.1 / 18.1 Simulated dataset (tr/val/tt) dataset (circular in Bomb shelter/Gym/PB132/PC226 and linClips: 208,008 / 22,392 / 13,032 ear in SA203/SC203/SE203/TB103/TC352). Although TAUTAU-SRIR: 9 rooms; Real dataset (tr/val/tt) Clips: 72,000 / 9,000 / 9,000 clips SRIR provides discrete source positions along circular/linear trajectories, we render each 5-s evaluation segment with a target categories 397 single spatial room impulse response (SRIR), i.e., a measured Prompt text and/or 1 s audio cue multichannel RIR with fixed source–microphone geometry, Mixture 1–2 targets/mixture; active/frame selecting the symmetric mic pair from the tetrahedral array ∈ {0, 1, 2} to obtain two channels. The goal is to ensure diverse reverNoise signal-to-noise ratio (SNR) beration and reflection patterns while keeping the evaluation ∼ U([−5, 5]) dB compact and non-redundant. Clean 5-s utterances from LibClip format 5 s, dual channel, 16 kHz Azimuth bins A=180 over [0, 180) riSpeech [60] are convolved with the selected two-channel Labeling RIRs (symmetric mic pair 0 and 2) and mixed with background Label rate 15 Hz (Tout =75) noise at SNR uniformly sampled from [−5, 5] dB; angles are expressed in the two-microphone frame by subtracting the LibriSpeech corpus [60]. Musical instrument sounds were array baseline and folding to [0◦ , 180◦ ). collected from the CC-Music Pianos [61] and GuitarSet datasets [62]. For a broader range of general acoustic events, B. Implementation details we utilized samples from AudioSet [63] and WavCaps [64]. With a Hann STFT at 16 kHz (nfft = 1024, hop = 256) we b) Noise Sources: A collection of noise recordings obtain F = 513 and Tin = 251; in the audio encoder, Demb = was used to create realistic and challenging noisy mixtures. 256; in the PGSA module, K = 80, Ndprnn = 6, Lchunk = These were sourced from multiple datasets, including MS128, Dh = Dc = dk = 64, KEIE = 2; in the DoA estimator, SNSD [65], WHAM! [66], ESC-50 [67], UrbanSound [68], Cfeat = 2180, Ctcn = 256, Dgru = 256 (thus 2Dgru = 512), QUT-NOISE [69] and Musan [70]. Tout = 75, Θ = 180, Nmax = 2. For training, we use up to c) Data Simulation: We generated all training, valida- 200 epochs with early stopping (patience = 10) and Adam tion, and test mixtures synthetically. The process relies on optimization with lr = 5 × 10−4 , and gradient clipping with dynamic RIRs generated using the GPURIR library [71]. For clip norm = 1. For the loss weights in the training objective, each simulated scenario, we define a room with dimensions we use (ω , ω sel DoA , ωcard ) = (1, 100, 1) in all experiments. of 4 × 4 × 2 meters and a reverberation time (T60 ) of 0.2 seconds. A dual-channel microphone array is positioned at C. Baseline Methods coordinates [1.9, 2.0, 1.0] and [2.1, 2.0, 1.0], with the interWe categorize the baselines into four types: (1) Track-wise microphone distance of 20 cm. methods (IPDNet [37], EINV2 [51], embed-ACCDoA [74], To simulate moving sources, we generate 5-s trajectories SALSA-Lite [42], the DCASE 2025 Task 3 baseline (denoted within a frontal azimuth range of 180◦ and uniformly resample as DCASE25 below) [52]), which allocate a fixed number of them at 15 Hz, yielding Tout = 75 discrete target directions output tracks for DoA/activity trajectories; (2) SELD+DoA per clip. Motivated by the DCASE SELD benchmarks, we methods (SELDnet [7], SELDT [53]), which output peruse a label frame rate coarser than the acoustic feature frame class predictions without explicit track control; (3) Pure DoA rate to shorten sequences while retaining sufficient temporal methods (SRP-DNN [45], FN-SSL [36]), which estimate detail [72]. At 15 Hz, a source traversing 180◦ in 5-s moves DoA from spatial features without activity branches; and (4) by at most 2.4◦ between consecutive labels, which is much Prompt-based methods, where we compare the text-queried smaller than the 20◦ angular tolerance adopted in the DCASE SEL model [57] against our proposed SelectTSL. SELD localization metrics [8], [72]. For fair comparison, we adapt all baselines to our binaural Each mixture is generated with one or two moving sound two-channel setup while preserving their original prediction sources. Since these sources may be intermittently silent, the heads and training objectives as much as possible. Specifically, number of active sources in any given time frame is zero, we adapt the baselines as follows: (i) multi-microphone or one, or two. These moving sources are created by convolving FOA inputs are reduced to the L-R pair, consistent with the clean source signals with their corresponding dynamic stereo/variable-array settings in prior work [37], [52], [75]; (ii) RIRs. Finally, a randomly selected noise signal is added to the standard FOA features are replaced by two-microphone spatial mixture at SNR uniformly sampled from the range of [−5, 5] cues (cos(IPD), sin(IPD), ILD, and GCC-PHAT), following dB. In total, we generated 288.9 hours of audio for the training established feature substitution protocols used in DCASE set, 31.1 hours for the validation set, and 18.1 hours for the baselines and SALSA-lite [42], [72], [76]; (iii) the geometry test set. is constrained to the horizontal plane (0◦ –180◦ azimuth) to red) Real Recordings: We use a subset of nine rooms move front-back ambiguity [52], [75]; (iv) original prediction from TAU-SRIR [73]: Bomb shelter, Gym, PB132, PC226, heads and losses are preserved with minimal changes, in line SA203, SC203, SE203, TB103, and TC352. This subset is with community practice and variable-array SSL designs [37], TABLE II DATASET STATISTICS AND KEY SETTINGS .
8
[72] and (v) inference outputs are mapped to a common framewise DoA heatmap format for fair comparison [42], [77]. D. Metrics We report static frame-level metrics (MAE, Precision, F1, Recall) and dynamic trajectory-level metrics (MOTA∗ , DetA, OSPA-T). The evaluation is based on framewise DoA matching. For each frame t, let Pt = {θ̂t,k } denote the set of predicted azimuth hypotheses and Gt = {θt,k } the annotated references. We match Pt to Gt via a Hungarian assignment that minimizes the circular distance on [0, 180◦ ), i.e., d◦ (θ̂, θ) = min |θ̂ − θ|, 180◦ − |θ̂ − θ| . A match is a TP if its error ≤Pδ; unmatched predictions/references P P are FP/FN. Let P TP = t TPt , FP = t FPt , FN = t FNt , TP , recall and GT = t |Gt |. We report precision P = TP+FP TP 2P R R = TP+FN and F1 = P +R . MAE (Mean Angular Error): X X 1 MAE = P d◦ (θ̂, θ), t |Mt | t (θ̂,θ)∈Mt
where Mt denotes the Hungarian assignment between Pt and Gt in frame t. MOTA∗ (Multiple Object Tracking Accuracy; ID-agnostic): We follow [54] but omit the ID–switch penalty; since all DoA hypotheses extracted under the same prompt belong to a single class, we report an ID-agnostic variant. P
+ FPt ) t (FN Pt
FN + FP =1− . |G | GT t t DetA (Detection Accuracy): TP . DetA = TP + FP + FN OSPA-T (Optimal Sub-Pattern Assignment [78]). Let dc (θ̂, θ) = min{d◦ (θ̂, θ), c} and nt = max(|Pt |, |Gt |). The frame score is 1 X OSPAt = dc (θ̂, θ) + c(FPt + FNt ) , nt ∗
MOTA = 1 −
(θ̂,θ)∈Mt
where the cutoff c in [78] is set to 15◦ , and FPt = |Pt |−|Mt |, FNt = |Gt | − |Mt |. The sequence score averages over all frames: T 1X OSPA-T = OSPAt . T t=1 VI. E XPERIMENTS A. Main Results Table III reports the performance of all baselines across static frame-level metrics (MAE, Prec., F1, Recall) and dynamic trajectory-level metrics (MOTA∗ , DetA, OSPA-T). For readability, we discuss results by the four baseline families defined in Sec. V-C. To disentangle architectural effects from input conditioning, we report results under three input conditions: Mix, Sel-Joint, and Clean. Specifically, Mix denotes the original stereo mixture with an unknown number of active sources (0/1/2) plus noise, fed directly to the baseline without any extraction front-end. Sel-Joint denotes end-to-end training where the baseline is preceded by the same PGSA module
as ours and takes the concatenation of the PGSA-enhanced magnitude and the original Mix phase as inputs, forming a two-channel representation. Clean is a clean condition where the input contains only the target source(s) consistent with the prompt, with no interfering sound sources and no noise. 1) Track-wise baselines: Within the track-wise family (IPDNet, EINV2, embed-ACCDoA, SALSA-Lite, DCASE25), only IPDNet and EINV2 remain effective in the Mix condition. IPDNet attains an MAE of 4.60◦ and an F1 of 0.24, whereas EINV2 achieves a higher F1 of 0.40 but at the cost of a much larger MAE of 11.37◦ . This apparent trade-off (high F1 but large MAE) is consistent with the DP-IPD and ACCDoA formulations. When the target is active, the per-track activity outputs are frequently triggered, leading to many detections and thus a higher F1. However, in the Mix condition the predicted DoAs are biased toward a direction between the target and interfering sources, which can still fall within the evaluation tolerance while increasing the average angular error. The remaining track-wise baselines have almost zero F1 and negative or near-zero MOTA∗ on Mix, and only become viable once Sel-Joint or Clean inputs simplify the scene. Under Clean inputs, IPDNet and EINV2 achieve MAE ≈ 1.4◦ and MOTA∗ ≈ 0.66–0.69, indicating that trackwise modeling can work once the prompt-based target is extracted. However, these systems are optimized to explain all active sources rather than follow a single target: DP-IPD based IPDNet treats any source at a given direction similarly, and ACCDoA-style systems (EINV2, SALSA-Lite, embedACCDoA, DCASE25) only weakly encode the prompt. In complex mixtures, tracks tend to follow non-target sources with similar geometry or break the target trajectory into short, fragmented segments. Our model directly targets the key weakness of generic track-wise designs: weak prompt binding that leads to drift and trajectory fragmentation in complex mixtures. We enforce target selectivity with a prompt-based PGSA front-end, whose EIEs condition the IPD Enhancer so that the refined spatial cues are biased toward the promptspecified source. On top of these target-dominant cues, we predict a per-azimuth DoA posteriorgram together with a cardinality head, avoiding slot-based track competition and stabilizing tracking under time-varying activity, which yields much higher MOTA∗ on Mix (Table III). 2) SELD+DoA (non-track-wise): The SELD+DoA baselines (SELDnet, SELDT) predict frame-level activity and DoA per class without explicit tracks. In the Mix condition they are highly sensitive to interference: SELDnet attains MAE ≈ 15.8◦ , SELDT MAE ≈ 22.1◦ , and both have MOTA∗ ≤ 0 with very poor detection. With Sel-Joint and Clean inputs, their static metrics improve substantially (MAE ≈ 6.6◦ ), consistent with prior SELD results on less cluttered scenes. Trajectory-level metrics improve much less. Even in Clean, SELDnet remains below MOTA∗ = 0.12 and SELDT only reaches MOTA∗ ≈ 0.19, suggesting that temporal inconsistency in the frame-level DoA predictions still leads to fragmented trajectories. In comparison, SELDT is clearly stronger on trajectory-level metrics, confirming that its trackingoriented objectives help stabilize trajectories. However, both models are trained to explain all event classes rather than a
9
TABLE III S TATIC METRICS ARE FRAME - LEVEL ( PEAK - MATCHING ); DYNAMIC METRICS ARE TRAJECTORY- LEVEL . MAE IS THE MEAN ANGULAR ERROR ON TRUE ∗ POSITIVES . P REC . DENOTES PRECISION . MOTA IS ID- AGNOSTIC ( NO ID- SWITCH PENALTY ). OSPA-T IS REPORTED IN DEGREES WITH p=1 AND c=15. P REC ./F1/R ECALL /MOTA∗ /D ETA ARE REPORTED AS PERCENTAGES . † N ON – TRACK - WISE : THE MODEL CANNOT SPECIFY THE NUMBER OF TRACKS WHEN PRODUCING D OA ESTIMATES . G LOBAL BEST RESULTS ARE IN BOLD ; C ATEGORY BEST RESULTS ARE UNDERLINED .
Category
Model
Input
Static DoA (frame-level)
Dynamic metrics (trajectory-level)
MAE (°) ↓ Prec. (%) ↑ F1 (%) ↑ Recall (%) ↑ MOTA∗ (%) ↑ DetA (%) ↑ OSPA-T (°) ↓ IPDNet
Mix Sel-Joint Clean
4.60 4.17 1.40
74.69 78.00 82.62
23.63 42.69 81.07
15.62 29.39 90.83
21.88 21.10 69.33
27.03 27.14 72.69
11.09 10.23 4.55
EINV2
Mix Sel-Joint Clean
11.37 4.18 1.45
47.27 69.12 76.25
40.36 63.99 84.80
31.16 49.50 95.52
7.91 41.04 65.77
25.28 52.40 73.62
11.26 7.66 4.15
embed-ACCDoA
Mix Sel-Joint Clean
20.02 6.28 6.51
38.33 59.96 60.00
1.17 34.20 39.83
0.59 23.93 29.80
-0.36 7.95 9.93
0.59 20.63 24.86
14.74 10.86 9.88
SALSA-Lite
Mix Sel-Joint Clean
19.66 4.66 4.34
36.22 70.12 72.60
1.36 31.53 37.08
0.69 20.34 24.89
-0.53 11.67 15.50
0.68 18.72 22.76
14.72 11.46 10.78
DCASE25
Mix Sel-Joint Clean
17.15 5.71 5.80
43.34 63.57 64.40
0.69 35.07 42.06
0.35 24.21 31.23
-0.11 10.34 13.97
0.35 21.26 26.63
14.77 10.79 9.72
SELDnet
Mix Sel-Joint Clean
15.83 6.62 6.34
42.62 57.66 60.76
1.51 35.18 41.36
0.77 25.31 41.36
-0.27 6.72 11.10
0.76 21.34 26.08
14.71 10.54 9.65
SELDT
Mix Sel-Joint Clean
22.12 6.66 6.63
31.81 61.53 62.96
2.97 48.26 53.17
1.56 39.71 46.01
-1.78 14.88 18.94
1.51 31.81 36.21
14.58 8.40 7.49
SRP-DNN
Mix Sel-Joint Clean
6.24 5.56 5.97
59.91 63.83 62.86
31.34 26.49 38.63
21.22 16.71 27.89
6.98 7.24 11.41
18.58 15.27 23.94
11.21 11.99 10.17
FN-SSL
Mix Sel-Joint Clean
7.01 5.83 5.92
70.23 82.56 79.72
17.41 35.12 33.82
19.08 22.98 24.29
18.23 23.10 21.13
17.87 25.13 23.78
10.09 7.82 8.84
SEL
Mix Sel-Joint Clean
5.37 2.78 2.84
59.82 83.01 82.91
30.45 49.50 50.23
20.42 44.17 43.88
6.70 16.25 17.14
7.96 23.71 21.17
13.14 7.96 8.01
Ours
Mix
0.98
98.28
95.67
93.20
91.57
91.70
2.08
Track-wise
SELD+DoA†
Pure DoA†
Prompt-based
single prompt-based target. When multiple sources overlap, they distribute energy across several classes and directions instead of committing to a single, stable trajectory for the target. Consequently, even with Sel-Joint or Clean inputs their MOTA∗ remains far below both the best track-wise baselines and our SelectTSL, which explicitly enforces selectivity and temporal consistency with respect to prompt. 3) Pure DoA (non-track-wise): The pure DoA baselines (SRP-DNN, FN-SSL) operate without any sound event detection (SED) head or explicit tracks, estimating DoA directly from binaural phase cues. SRP-DNN uses a causal CRNN to estimate a frame-wise spatial spectrum on a fixed azimuth grid, followed by peak-picking to obtain the DoA. In our binaural horizontal setup this yields very similar localization across input conditions (MAE = 6.24◦ / Mix, 5.56◦ / SelJoint, 5.97◦ / Clean) and only modest changes in MOTA∗ (≈ 0.07/0.07/0.11). This limited variation reflects the fixed spatial grid and the absence of explicit temporal modeling: cleaner magnitudes sharpen the peaks but only mildly affect their centers, while framewise fluctuations still lead to frag-
mented trajectories. FN-SSL also estimates DP-IPD but uses full-band / narrowband fusion to exploit cross-frequency and temporal structure. From Mix to Sel-Joint and Clean, its MAE improves from 7.01◦ to 5.8◦ , and MOTA∗ increases from 0.18 to 0.23 and 0.21, yielding a more favorable static–dynamic trade-off than SRP-DNN. Nevertheless, FN-SSL remains a purely geometric, single-stage DoA estimator that treats all spatial peaks in the mixture as equally plausible, without any mechanism to focus on the target source. Consequently, it significantly underperforms our SelectTSL integrating the PGSA front-end. This performance gap across both static and trajectory-level metrics confirms that geometry alone is insufficient for robust, prompt-based localization and tracking. 4) Prompt-based: The prompt-based baseline SEL [57] conditions localization directly on a text description of the target event. It takes multichannel audio and a text query, fuses an audio representation with a CLAP-based text embedding, and predicts a discretized azimuth distribution per frame. Under the Mix condition, SEL already benefits from text conditioning
10
but remains limited by interference and ambiguous audio–text correspondence (MAE = 5.37◦ , MOTA∗ = 0.07). Prepending the PGSA module (Sel-Joint) or switching to Clean inputs substantially strengthens frame-level localization (MAE ≈ 2.8◦ in both cases) and improves trajectory metrics (MOTA≈0.16–0.17). However, SEL predicts a single azimuth distribution per frame for a given text embedding. When more than one source matches the query, it often focuses on the dominant target or averages multiple directions, leading to missed or unstable secondary trajectories, consistent with the original benchmark. Our model is explicitly designed to handle up to at least two concurrent target sources per text query. Using the raw Mix as input, it achieves an MAE of 0.98◦ and a MOTA∗ of 0.92, substantially outperforming SEL even when SEL is given Sel-Joint or Clean inputs. We attribute these gains to the combination of a prompt-based PGSA front-end and a per-azimuth DoA heatmap that represents multiple peaks per frame, rather than collapsing them into a single discrete angle. This yields much sharper and more stable trajectories, especially in frames where two target sources overlap. B. Ablation and Design Analysis of SelectTSL 1) System-level architectural ablations: We perform system-level ablations to quantify the contribution of key design choices in SelectTSL (Table IV), including: (i) coupling between selection and DoA (conditioning on extracted target magnitude vs. mixture magnitude), (ii) spatial cues to the DoA estimator (IPD+ILD, w/o ILD, w/o IPD, or none), (iii) the number KEIE of DPRNN blocks used to form EIEs, (iv) the DoA module design (with/without cross-attention and FuseLayer), (v) the cardinality head, and (vi) the training scheme (end-to-end vs. two-stage). In each row, only the factor in the “Setting” column is changed and all other components follow the main configuration. Across (i)–(ii), both target-dependent conditioning and explicit multichannel cues are indispensable. Even with EIEs disabled (KEIE =0), conditioning on the selected target clearly outperforms the Mix→DoA variant using mixture magnitude (MAE 1.21◦ vs. 3.32◦ , MOTA∗ 0.88 vs. 0.49). Removing ILD already degrades performance, while removing IPD or all spatial cues leads to large errors (MAE ≥ 3.89◦ ) and much lower MOTA∗ , confirming the central role of phasebased spatial information in reverberant multi-source scenes. For temporal and structural choices (iii)-(vi), using EIEs from only a few DPRNN blocks is sufficient: our default KEIE =2 performs best, KEIE =1 remains close, whereas KEIE =3 or 4 monotonically worsen MAE and MOTA∗ , suggesting that incorporating EIEs from more blocks mainly injects noise. Removing cross-attention in the Extraction Network hurts trajectory metrics more than replacing the Fusion Layer with simple concatenation, since cross-attention is where the prompt interacts with the mixture features. Removing it weakens target emphasis and increases interference, which hurts temporal association. The cardinality head is crucial for stable tracking: removing it sharply reduces MOTA∗ (0.89→0.58) and increases OSPA (3.1◦ →6.3◦ ). Finally, the
TABLE IV A BLATION STUDY OF S ELECT TSL (MAE, F1, MOTA∗ , OSPA-T; ↓ LOWER IS BETTER , ↑ HIGHER IS BETTER ). F1 AND MOTA ∗ ARE REPORTED AS PERCENTAGES . T HE F ULL MODEL ROW IS SHOWN AS A REFERENCE AND IS NOT CONSIDERED WHEN HIGHLIGHTING BEST RESULTS . B EST RESULTS WITHIN EACH ABLATION GROUP ARE HIGHLIGHTED IN BOLD . Setting
MAE (°) ↓ F1 (%) ↑ MOTA∗ (%) ↑ OSPA-T (°) ↓
Full model (SelectTSL)
0.98
95.67
91.57
2.08
Coupling: w/o EIEs (KEIE =0) Coupling: Mix→DoA
1.21 3.32
93.58 65.92
87.93 49.16
2.72 7.82
Spatial: w/o ILD Spatial: w/o IPD Spatial: w/o spatial cues
1.79 3.89 4.54
87.18 60.43 54.52
77.27 43.30 37.48
4.14 8.39 9.20
EIE blocks: 1 EIE blocks: 3 EIE blocks: 4
1.43 1.95 2.25
91.68 84.98 81.79
88.76 73.88 69.20
3.11 4.50 4.97
Backbone: w/o cross-attn Backbone: w/o FuseLayer
1.50 1.08
91.04 94.65
83.55 89.84
3.24 2.40
Cardinality: w/o head
2.91
73.57
58.19
6.33
Training: separate Sel/DoA
1.28
90.16
81.64
3.56
two-stage training variant lags behind end-to-end optimization (MAE 1.28◦ , F1 0.90, MOTA∗ 0.82), indicating that joint training is needed for PGSA to produce EIEs that are aligned with the DoA objective. 2) Prompt-only ablations: To disentangle the roles of text and audio cues at the fusion layer, we run prompt-only ablations where the mixture is always encoded but one conditioning branch is masked to an all-zero embedding, forcing the model to rely solely on the remaining modality (Table V). For text-only conditioning, we keep the text branch active and zero out the audio embedding. We use Qwen2.5-7B [79] to generate paraphrases for each class and group them into three bands by CLAP similarity to the canonical caption: high, medium, and low ([0.85, 1.00], [0.70, 0.85), [0.50, 0.70)). Performance degrades monotonically as similarity decreases: relative to the canonical “Text-Only” caption (MAE = 1.12◦ , MOTA∗ = 0.86), the highest band already worsens to MAE = 1.94◦ /MOTA∗ = 0.74, and the lowest band falls to MAE = 3.48◦ /MOTA∗ = 0.43, indicating that semantically close prompts are crucial when no audio cue is available. For audio-only conditioning, we mask the text embedding and feed a 1 s audio cue through the audio-prompt branch. We compare four strategies: taking the first 1 s from the class clip (Audio-Only / Front), sampling a random 1 s segment (PosRand), time-stretching the first 1 s (Aug-Front), and taking the first 1 s from another clip of the same class (XClip-Front). Using the clip onset yields the best audio-only performance (MAE = 1.57◦ , MOTA∗ = 0.81); random or cross-clip cues substantially degrade it (MAE ≈ 2.30◦ , MOTA∗ ≈ 0.51), with simple time-stretch in between. This suggests that under audioonly conditioning, the model is most sensitive to temporal misalignment and identity mismatch, while moderate temporal deformation is less harmful. 3) DoA estimator ablations: We ablate the proposed DoA estimator to assess the IPD Enhancer, the semantic–acoustic branches, and the feature refinement block (FRB). Table VI summarizes MAE, frame-level F1, MOTA∗ , and OSPA-T; below we focus on MAE and MOTA∗ as representative static and trajectory metrics.
11
TABLE V J OINT ABLATION OVER TEXT AND AUDIO PROMPTS UNDER SINGLE - MODALITY CONDITIONING AT THE FUSION LAYER . F1 AND MOTA∗ ARE REPORTED ON A 0–1 SCALE . Modality
Setting / Similarity MAE (°) ↓ F1 ↑ MOTA∗ ↑ OSPA-T (°) ↓
Text-only
Text-Only [0.85, 1.00] [0.70, 0.85) [0.50, 0.70)
1.12 1.94 2.19 3.48
0.93 0.86 0.80 0.65
0.86 0.74 0.65 0.43
2.95 4.37 5.82 8.22
Audio-Only Pos-Rand Audio-only Aug-Front XClip-Front
1.57 2.30 1.92 2.29
0.89 0.71 0.78 0.70
0.81 0.52 0.62 0.51
3.67 7.28 5.93 7.46
TABLE VI A BLATION ON D OA ESTIMATOR ARCHITECTURE . F1 AND MOTA∗ ARE REPORTED ON A 0–1 SCALE . Setting
MAE (°) ↓ F1 ↑ MOTA∗ ↑ OSPA-T (°) ↓
Baseline and IPD / semantic–acoustic design (A1–A4) Full (ours) 0.98 0.96 A1: w/o IPD Enhancer 2.10 0.83 A2: semantic branch only 1.40 0.91 A3: acoustic branch only 1.62 0.89 A4: direct IPD input 2.71 0.73
0.92 0.69 0.84 0.80 0.53
2.08 5.02 3.32 3.78 6.50
FRB design (B1–B7) B1: w/o FRB B2: FRB depth ×1 B3: FRB depth ×3 B4: FRB depth ×4 B5: FRB conv-only, ×2 (no SE) B6: FRB SE-only, ×2 (no conv) B7: FRB w/o residual, ×2
0.38 0.79 0.79 0.69 0.68 0.72 0.72
7.90 3.71 3.81 4.91 4.95 4.50 4.62
3.80 1.52 1.37 2.12 2.13 1.96 1.93
0.64 0.84 0.83 0.83 0.83 0.85 0.85
For the IPD Enhancer and semantic–acoustic design (A1–A4), removing the IPD Enhancer (A1) causes a marked degradation relative to the full model (MAE = 0.98◦ → 2.10◦ , MOTA∗ = 0.92 → 0.69), showing that denoising and refining raw IPD is critical for robust localization. Using only the semantic branch (A2) or only the acoustic branch (A3) is also suboptimal: both fall behind the full system, and semantic-only outperforms acoustic-only, indicating that text-conditioned embeddings carry strong class cues but still benefit from being fused with spatial features. Replacing enhanced IPD with direct IPD input (A4) yields the worst performance in this group (MAE = 2.71◦ , MOTA∗ = 0.53), further highlighting the importance of the IPD enhancement module. For the FRB variants (B1–B7), the results show that iterative refinement is necessary but must be carefully structured. Completely removing FRB (B1) severely hurts performance (MAE = 3.80◦ , MOTA∗ = 0.38), indicating that a single pass through the backbone is insufficient. Varying FRB depth (B2–B4) suggests that the default depth ×2 strikes the best balance: shallower or deeper configurations do not improve MAE or MOTA∗ and can even degrade them. Ablating internal components (B5–B7) by removing the convolution, SE branch, or residual connection again worsens both metrics, confirming that the full FRB with all three components is most effective. 4) Cardinality head ablations: As described in Section IV-B, we decouple DoA estimation and source-count prediction: the DoA head produces a frame-level posteriorgram over Θ azimuth bins, while a separate cardinality head predicts a distribution over N = {0, 1, 2}. At inference, we take n̂t and select the top-n̂t local maxima of P̂DoA (t, :) as DoA estimates (“Ours” in Table VII), so the cardinality head provides a discrete prior on the number of sources without explicitly
TABLE VII A BLATION ON HOW THE CARDINALITY HEAD INTERACTS WITH THE D OA HEAD . A LL VARIANTS SHARE THE SAME BACKBONE AND CARDINALITY CLASSIFIER ; ONLY THE WAY CARDINALITY INFORMATION IS USED DIFFERS . F1 AND MOTA ∗ ARE REPORTED ON A 0–1 SCALE . Setting Ours (top-n̂ peaks) Card-head attention Embed (concat to DoA head)
MAE (°) ↓ F1 ↑ MOTA∗ ↑ OSPA-T (°) ↓ 0.98 1.87 1.52
0.96 0.85 0.91
0.92 0.72 0.83
2.08 4.67 3.45
injecting cardinality embeddings into the DoA representation. To test whether tighter coupling helps, we compare this design with two alternatives that feed cardinality information back into the DoA head. In the “Card-head attention” variant, the predicted count is mapped to an embedding that queries a multi-head attention block over DoA features, and the attention output is added via a residual connection. In the “Embed” variant, the 3-way cardinality probabilities are passed through a small MLP to produce an embedding that is concatenated with the DoA head input. Table VII shows that both variants are worse than the simple top-n̂ peak selection, despite using more parameters: MAE increases from 0.98◦ to 1.52◦ /1.87◦ and MOTA∗ drops from 0.92 to 0.83/0.72. Compared with the “no cardinality head” configuration in Table IV, all three designs benefit from having a dedicated source-count predictor, but directly injecting cardinality embeddings into DoA features tends to distort spatial structure. The decoupled design with top-n̂ selection therefore offers the best overall trade-off while keeping the architecture and decoding rule simple. C. Robustness and Generalization 1) Varying acoustic complexity: To study robustness under different acoustic conditions, we define three difficulty levels (A/B/C) by gradually enlarging the room and widening the range of reverberation time T60 ; the exact parameter ranges are given in Table VIII. Level A uses compact rooms with short T60 , Level B moderately increases both, and Level C spans the largest, most asymmetric rooms and the longest, most variable reverberation. As shown in Table VIII, performance degrades steadily from A to C. MAE increases from 2.36◦ to 3.03◦ and MOTA∗ drops from 0.61 to 0.46, with similar trends for the other static and trajectory-level metrics. Larger, more asymmetric rooms introduce stronger spatial ambiguities, and longer, more variable T60 smears binaural cues, together defining the robustness envelope of our model under increasing acoustic variability. 2) Varying motion dynamics: To evaluate robustness under different motion dynamics, we define four speed buckets (A– D) that control the instantaneous angular velocity of the sources. Bucket A corresponds to slow motion within ±5◦ , bucket B to moderate motion within ±15◦ , and buckets C and D to increasingly rapid motion within ±30◦ and ±50◦ , respectively (Table IX). Table IX shows that tracking performance degrades as motion becomes faster and more irregular. MAE remains low for slow and moderate motion (1.20◦ in A, 1.14◦ in B) but rises to 1.55◦ and 2.32◦ in buckets C and D, while MOTA∗ drops from 0.96 to 0.53 with similar declines in the other
12
TABLE VIII P ERFORMANCE AND ACOUSTIC CONFIGURATION ACROSS DIFFICULTY LEVELS . ROOM DIMENSIONS Lx , Ly , Lz AND REVERBERATION TIME T60 ARE SAMPLED FROM THE RANGES SHOWN FOR EACH LEVEL .
Setting
Lx (m)
Ly (m)
Lz (m)
T60 (s)
MAE (◦ )
Prec.
F1
Recall
MOTA*
DetA
OSPA-T (◦ )
Main
4.0
4.0
2.0
0.20
0.98
0.98
0.96
0.93
0.92
0.92
2.08
A B C
[3.6, 4.6] [3.8, 6.0] [3.5, 7.0]
[3.6, 4.6] [3.8, 5.2] [3.5, 6.0]
[1.8, 2.3] [2.2, 3.0] [2.2, 3.0]
[0.20, 0.35] [0.25, 0.55] [0.20, 0.65]
2.36 2.65 3.03
0.90 0.88 0.83
0.78 0.72 0.68
0.69 0.61 0.58
0.61 0.52 0.46
0.64 0.56 0.52
5.82 6.82 7.21
TABLE IX P ERFORMANCE ACROSS DIFFERENT MOTION SPEED BUCKETS . Bucket Range (◦ ) MAE Prec. A B C D
±5 ±15 ±30 ±50
1.20 1.14 1.55 2.32
0.98 0.97 0.95 0.88
F1
Recall MOTA* DetA OSPA-T (◦ )
Setting
0.98 0.95 0.88 0.72
0.98 0.93 0.82 0.61
01 bomb shelter 02 gym 03 pb132 04 pc226 05 sa203 06 sc203 08 se203 09 tb103 10 tc352
2.82 2.01 0.82 1.90 6.89 2.39 0.79 3.37 2.63
Mean
2.62
0.96 0.90 0.78 0.53
0.96 0.91 0.78 0.57
1.63 2.29 3.98 6.91
TABLE X A BLATION AND ROBUSTNESS UNDER DIFFERENT MOTION AND PROMPT CONDITIONS . F1 AND MOTA ∗ ARE REPORTED ON A 0–1 SCALE .
Category
Setting
Movement
stat–stat mov–stat stat–mov mov–mov
r=30% No-prompt r=50% r=70%
TABLE XI P ERFORMANCE ON THE TAU-SRIR REAL - ROOM EVALUATION SET.
MAE (°) ↓ F1 ↑ MOTA∗ ↑ OSPA-T (°) ↓ 0.15 0.40 0.97 0.98
0.99 0.98 0.96 0.96
0.98 0.96 0.92 0.92
0.45 0.70 2.02 2.08
1.00 2.22 2.53
0.96 0.83 0.78
0.92 0.67 0.60
1.20 3.32 4.55
metrics. Higher angular velocities induce larger frame-toframe direction changes, breaking the temporal smoothness the model relies on, so gradual or moderate movement is handled well whereas abrupt high-speed rotations remain challenging. 3) Robustness to motion and prompt conditions: We further test robustness when spatial dynamics are present and when text prompts may be missing (Table X). For motion, we independently control movement of the background and the target, yielding four regimes: stat–stat (static noise, static source), mov–stat (moving noise, static source), stat–mov (static noise, moving source), and mov–mov (both moving), with all other factors fixed. For prompts, we create no-prompt clips where the caption is removed but the target remains in the mixture, and vary the no-prompt proportion r ∈ {30%, 50%, 70%}. In the movement group, the fully static case (stat–stat) achieves the best performance (MAE = 0.15◦ , MOTA∗ = 0.98), background motion alone has only a mild impact (mov– stat), while moving the target (stat–mov, mov–mov) increases MAE to about 0.97◦ and reduces MOTA∗ to about 0.92. In the no-prompt group, performance degrades as the no-prompt proportion increases: MAE rises from 1.00◦ at r = 30% to 2.53◦ at r = 70%, while MOTA∗ drops from 0.92 to 0.60. Even so, the model remains usable when captions are frequently absent, showing that it can still localize and track targets based primarily on audio cues. 4) Real-world data: We further assess generalization on the real-room subset of TAU-SRIR, using the same protocol as in simulation. As summarized in Table XI, the mean performance across rooms is MAE = 2.62◦ , MOTA∗ = 0.77, and OSPA-T
MAE (◦ ) Prec.
OSPA-T (◦ )
F1
Recall MOTA* DetA
0.86 0.87 0.93 0.89 0.81 0.89 0.79 0.85 0.87
0.89 0.91 0.95 0.92 0.83 0.91 0.81 0.88 0.90
0.92 0.94 0.97 0.95 0.85 0.94 0.83 0.91 0.93
0.77 0.81 0.91 0.84 0.64 0.82 0.61 0.75 0.79
0.80 0.83 0.91 0.85 0.70 0.84 0.68 0.78 0.82
2.92 2.47 1.25 2.09 4.26 2.29 4.74 3.04 2.62
0.86
0.89
0.92
0.77
0.80
2.85
= 2.85◦ , indicating strong transfer to measured spaces. Room-level trends correlate with the measured room attributes and trajectory design. PB132 is a small carpeted classroom with circular trajectories, implying shorter effective reverberation and a stable direct-to-reverberant ratio (DRR); this yields easy association and correspondingly strong scores (MAE 0.82◦ , MOTA∗ 0.91, OSPA-T 1.25◦ ). SA203 is a lecture hall with an inclined floor and linear trajectories at multiple ranges, which introduce stronger early/late reflections and larger DRR variation, leading to increased angular bias and track fragmentation (MAE = 6.89◦ , MOTA∗ = 0.64, OSPAT = 4.26◦ ). SE203 (a large classroom with hard floor and linear trajectories) shows a different failure mode: framewise azimuths remain sharp (MAE = 0.79◦ ), but repeated crossings and specular clutter cause association breaks, resulting in the highest OSPA-T (4.74◦ ). Overall, the model maintains accurate DoA estimates across diverse real rooms, with performance variations that can be explained by room geometry and trajectory complexity. VII. V ISUALIZATION To better understand the effect of different prompt designs, we visualize the frame-wise DoA posteriors produced by four separately trained models with different prompt inputs: (i) a Full model that uses both the audio cue and the text prompt, (ii) a Text-only model, (iii) an Audio-only model that only uses the audio cue, and (iv) a No-prompt model. For each row in Fig. 3, all four models are evaluated on the same mixture recording, and the rightmost panel overlays their decoded DoA trajectories together with the GT. In the first row, only the Full model is able to follow the challenging non-stationary trajectory, while the Text-only and Audio-only models fail to provide a reliable track and the No-prompt model produces a completely wrong trajectory. In the second row, both the Full and Text-only models closely
13
Fig. 3. Heatmap visualization of frame-wise DoA posteriors from four separately trained models under different prompt configurations. For three representative target events (rows), the first four columns correspond to the Full (text+cue), Text-only, Audio-only, and No-prompt models, all evaluated on the same mixture. The rightmost column shows the estimated and GT DoA trajectories. Warmer colors indicate higher posterior probability.
Fig. 4. Polar visualization of temporal-spatial DoA trajectories. The radial axis encodes time (0–75 frames) and the angular axis encodes DoA (0◦ -180◦ ).
match the ground truth, whereas the Audio-only and Noprompt models exhibit large deviations and fragmented tracks. In the third row, the Full and Audio-only models roughly capture the motion pattern, while the Text-only and No-prompt models again fail to track the source. These qualitative results demonstrate that text and audio prompts provide complementary information, and that jointly exploiting both leads to the most robust localization performance. We also visualize estimated DoA trajectories in polar coordinates in Fig. 4, where the radial axis encodes time (0–75 frames) and the angular axis encodes DoA (0◦ -180◦ ). Panel (a) shows a dual-source recording with wide angular motion. The model follows the target across the full range. Panel (b) presents a single-source sequence with smooth motion, where predictions form a continuous track well aligned with the ground truth. Panel (c) shows a two-source sequence with silent periods that create three disjoint active segments. The model quickly re-locks onto the correct direction whenever the target becomes active again. These examples illustrate that the model can handle dual-source, single-source, and silent-period
cases while maintaining temporally consistent trajectories. VIII. C ONCLUSION We presented SelectTSL, a framework for prompt-guided selective target sound localization that leverages promptbased selectivity to learn target-aware representations for dualchannel DoA estimation. By leveraging text and audio prompts and a target-aware multi-spatial-cue representation, our model can selectively estimate the DoA of the target source. Experiments show consistent improvements on both static framelevel and dynamic trajectory-level metrics over competitive baselines. Future work will extend SelectTSL toward unified multimodal prompting (e.g., incorporating visual prompts and scene context) to facilitate robust deployment in real-world environments. R EFERENCES [1] J. Benesty, J. Chen, and Y. Huang, Microphone array signal processing. Springer, 2008.
14
[2] R. Hëb-Umbach, T. Nakatani, M. Delcroix, C. Boeddeker, and T. Ochiai, “Microphone array signal processing and deep learning for speech enhancement: Combining model-based and data-driven approaches to parameter estimation and filtering [special issue on model-based and data-driven audio signal processing],” IEEE Signal Processing Magazine, vol. 41, no. 6, pp. 12–23, 2025. [3] C. Knapp and G. Carter, “The generalized correlation method for estimation of time delay,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 24, no. 4, pp. 320–327, 2003. [4] R. Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE transactions on antennas and propagation, vol. 34, no. 3, pp. 276– 280, 1986. [5] S. Chakrabarty and E. A. Habets, “Multi-speaker doa estimation using deep convolutional networks trained with noise signals,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 1, pp. 8–21, 2019. [6] E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” J. Acoust. Soc. Am., vol. 25, no. 5, pp. 975–979, 1953. [7] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 1, pp. 34–48, 2018. [8] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, “Overview and evaluation of sound event localization and detection in dcase 2019,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 29, pp. 684–698, 2020. [9] A. Senocak, H. Ryu, J. Kim, T.-H. Oh, H. Pfister, and J. S. Chung, “Toward interactive sound source localization: Better align sight and sound!” IEEE Trans. Pattern Anal. Mach. Intell., 2025. [10] Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno, “VoiceFilter: Targeted voice separation by speaker-conditioned spectrogram masking,” INTERSPEECH, pp. 2728–2732, 2019. [11] X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Separate what you describe: Language-queried audio source separation,” in INTERSPEECH, 2022, pp. 1801–1805. [12] K. Saijo, J. Ebbers, F. G. Germain, S. Khurana, G. Wiehern, and J. Le Roux, “Leveraging audio-only data for text-queried target sound extraction,” in ICASSP. IEEE, 2025, pp. 1–5. [13] M. Kim, R. Mira, H. Chen, S. Petridis, and M. Pantic, “Contextual speech extraction: Leveraging textual history as an implicit cue for target speech extraction,” in ICASSP. IEEE, 2025, pp. 1–5. [14] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” IEEE Trans. Pattern Anal. Mach. Intell., 2018. [15] K. Li, F. Xie, H. Chen, K. Yuan, and X. Hu, “An audio-visual speech separation model inspired by cortico-thalamo-cortical circuits,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 10, pp. 6637–6651, 2024. [16] Z. Pan, M. Ge, and H. Li, “Usev: Universal speaker extraction with visual cue,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 30, pp. 3032–3045, 2022. [17] R. Tao, X. Qian, Y. Jiang, J. Li, J. Wang, and H. Li, “Audio-visual target speaker extraction with reverse selective auditory attention,” IEEE Trans. on Audio, Speech, and Language Processing, 2025. [18] R. Gu, L. Chen, S.-X. Zhang, J. Zheng, Y. Xu, M. Yu, D. Su, Y. Zou, and D. Yu, “Neural spatial filter: Target speaker speech separation assisted with directional information.” in INTERSPEECH, 2019, pp. 4290–4294. [19] R. Gu, S.-X. Zhang, Y. Zou, and D. Yu, “Towards unified all-neural beamforming for time and frequency domain speech separation,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 31, pp. 849– 862, 2022. [20] K. Žmolı́ková, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. Černockỳ, “Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 4, pp. 800–814, 2019. [21] C. Xu, W. Rao, E. S. Chng, and H. Li, “Spex: Multi-scale time domain speaker extraction network,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 28, pp. 1370–1384, 2020. [22] K. Liu, Z. Du, X. Wan, and H. Zhou, “X-sepformer: End-to-end speaker extraction network with explicit optimization on speaker confusion,” in ICASSP. IEEE, 2023, pp. 1–5. [23] H.-W. Dong, N. Takahashi, Y. Mitsufuji, J. McAuley, and T. BergKirkpatrick, “Clipsep: Learning text-queried sound separation with noisy unlabeled videos,” arXiv preprint arXiv:2212.07065, 2022. [24] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion
and keyword-to-caption augmentation,” in ICASSP. IEEE, 2023, pp. 1–5. [25] H. Ma, Z. Peng, X. Li, M. Shao, X. Wu, and J. Liu, “Clapsep: Leveraging contrastive pre-trained model for multi-modal query-conditioned target sound extraction,” IEEE Trans. on Audio, Speech, and Language Processing, 2024. [26] X. Liu, Q. Kong, Y. Zhao, H. Liu, Y. Yuan, Y. Liu, R. Xia, Y. Wang, M. D. Plumbley, and W. Wang, “Separate anything you describe,” IEEE Trans. on Audio, Speech, and Language Processing, 2024. [27] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “Zero-shot audio source separation through query-based learning from weakly-labeled data,” in AAAI, vol. 36, no. 4, 2022, pp. 4441–4449. [28] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time– frequency magnitude masking for speech separation,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 27, no. 8, pp. 1256–1266, 2019. [29] Y. Luo, Z. Chen, and T. Yoshioka, “Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,” in ICASSP. IEEE, 2020, pp. 46–50. [30] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “Tf-gridnet: Integrating full-and sub-band modeling for speech separation,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 31, pp. 3221–3236, 2023. [31] P.-A. Grumiaux, S. Kitić, L. Girin, and A. Guérin, “A survey of sound source localization with deep learning methods,” J. Acoust. Soc. Am., vol. 152, no. 1, pp. 107–151, 2022. [32] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon, “Learning to localize sound sources in visual scenes: Analysis and applications,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 5, pp. 1605–1619, 2019. [33] H. Xuan, Z. Wu, J. Yang, B. Jiang, L. Luo, X. Alameda-Pineda, and Y. Yan, “Robust audio-visual contrastive learning for proposal-based self-supervised sound source localization in videos,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 7, pp. 4896–4907, 2024. [34] Z. Song, J. Zhang, Y. Wang, J. Fan, and Z. Zhang, “Enhancing sound source localization via false negative elimination,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 10 499–10 514, 2024. [35] T. N. T. Nguyen, K. N. Watcharasupat, N. K. Nguyen, D. L. Jones, and W.-S. Gan, “Salsa: Spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 30, pp. 1749–1762, 2022. [36] Y. Wang, B. Yang, and X. Li, “Fn-ssl: Full-band and narrow-band fusion for sound source localization,” in INTERSPEECH, 2023, pp. 3779–3783. [37] ——, “Ipdnet: A universal direct-path ipd estimation network for sound source localization,” IEEE Trans. on Audio, Speech, and Language Processing, 2024. [38] P.-A. Grumiaux, S. Kitić, P. Srivastava, L. Girin, and A. Guérin, “Saladnet: Self-attentive multisource localization in the ambisonics domain,” in IEEE Workshop Appl. Signal Process. Audio Acoust. IEEE, 2021, pp. 336–340. [39] W. Huang, Q. Huang, L. Ma, and C. Wang, “Swg-former: A slidingwindow graph convolutional network for simultaneous spatial-temporal information extraction in sound event localization and detection,” arXiv preprint arXiv:2310.14016, 2023. [40] J. Hu, Y. Cao, M. Wu, Q. Kong, F. Yang, M. D. Plumbley, and J. Yang, “Sound event localization and detection for real spatial sound scenes: Event-independent network and data augmentation chains,” arXiv preprint arXiv:2209.01802, 2022. [41] Y. Shul and J.-W. Choi, “Cst-former: Transformer with channel-spectrotemporal attention for sound event localization and detection,” in ICASSP. IEEE, 2024, pp. 8686–8690. [42] T. N. T. Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W.-S. Gan, “Salsa-lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays,” in ICASSP. IEEE, 2022, pp. 716–720. [43] A. Berg, J. Engman, J. Gulin, K. Aström, M. Oskarsson, and B. Sony Europe, “The lu system for dcase 2024 sound event localization and detection challenge,” DCASE2024 Challenge, Tech. Rep, Tech. Rep., 2024. [44] A. Berg, J. Engman, J. Gulin, K. Åström, and M. Oskarsson, “Learning multi-target tdoa features for sound event localization and detection,” arXiv preprint arXiv:2408.17166, 2024. [45] B. Yang, H. Liu, and X. Li, “Srp-dnn: Learning direct-path phase difference for multiple moving sound source localization,” in ICASSP. IEEE, 2022, pp. 721–725. [46] H. Yin, M. Ge, Y. Fu, G. Zhang, L. Wang, L. Zhang, L. Qiu, and J. Dang, “Mimo-doanet: Multi-channel input and multiple outputs doa
15
network with unknown number of sound sources,” arXiv preprint arXiv:2207.07307, 2022. [47] K. Shimada, Y. Koyama, N. Takahashi, S. Takahashi, and Y. Mitsufuji, “Accdoa: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,” in ICASSP. IEEE, 2021, pp. 915–919. [48] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, “Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training,” in ICASSP. IEEE, 2022, pp. 316–320. [49] Y. Cao, T. Iqbal, Q. Kong, Y. Zhong, W. Wang, and M. D. Plumbley, “Event-independent network for polyphonic sound event localization and detection,” arXiv preprint arXiv:2010.00140, 2020. [50] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” in ICASSP. IEEE, 2021, pp. 885–889. [51] D. Mu, Z. Zhang, and H. Yue, “Mff-einv2: Multi-scale feature fusion across spectral-spatial-temporal domains for sound event localization and detection,” arXiv preprint arXiv:2406.08771, 2024. [52] K. Shimada, A. Politis, I. R. Roman, P. Sudarsanam, D. Diaz-Guerra, R. Pandey, K. Uchida, Y. Koyama, N. Takahashi, T. Shibuya et al., “Stereo sound event localization and detection with onscreen/offscreen classification,” arXiv preprint arXiv:2507.12042, 2025. [53] S. Adavanne, A. Politis, and T. Virtanen, “Localization, detection and tracking of multiple moving sound sources with a convolutional recurrent neural network,” arXiv preprint arXiv:1904.12769, 2019. [54] ——, “Differentiable tracking-based training of deep learning sound source localizers,” in IEEE Workshop Appl. Signal Process. Audio Acoust. IEEE, 2021, pp. 211–215. [55] S. Sivasankaran, E. Vincent, and D. Fohr, “Keyword-based speaker localization: Localizing a target speaker in a multi-speaker environment,” in INTERSPEECH, 2018. [56] O. Slizovskaia, G. Wichern, Z.-Q. Wang, and J. Le Roux, “Locate this, not that: Class-conditioned sound event doa estimation,” in ICASSP. IEEE, 2022, pp. 711–715. [57] J. Zhao, X. Qian, Y. Xu, H. Liu, Y. Cao, D. Berghi, and W. Wang, “Text-queried target sound event localization,” in EUSIPCO. IEEE, 2024, pp. 261–265. [58] Y. Chen, X. Qian, Z. Pan, K. Chen, and H. Li, “Locselect: Target speaker localization with an auditory selective hearing mechanism,” in ICASSP. IEEE, 2024, pp. 8696–8700. [59] G. Li, W. Xue, W. Liu, J. Yi, and J. Tao, “Gcc-speaker: Target speaker localization with optimal speaker-dependent weighting in multi-speaker scenarios,” in ICASSP. IEEE, 2023, pp. 1–5. [60] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP. IEEE, 2015, pp. 5206–5210. [61] M. Zhou, S. Wu, S. Ji, Z. Li, and W. Li, “A holistic evaluation of piano sound quality,” in National Conf. on Sound and Music Technology. Springer, 2023, pp. 3–17. [62] Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “Guitarset: A dataset for guitar transcription.” in ISMIR, 2018, pp. 453–460. [63] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in ICASSP. IEEE, 2017, pp. 776–780. [64] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 32, pp. 3339– 3354, 2024. [65] C. K. Reddy, V. Gopal, R. Cutler, E. Beyrami, R. Cheng, H. Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun et al., “The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,” arXiv preprint arXiv:2005.13981, 2020. [66] G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. L. Roux, “Wham!: Extending speech separation to noisy environments,” arXiv preprint arXiv:1907.01160, 2019. [67] K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in ACM Int. Conf. Multimedia. ACM Press, pp. 1015–1018. [68] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in ACM Int. Conf. Multimedia, Orlando, FL, USA, Nov. 2014, pp. 1041–1044. [69] D. Dean, S. Sridharan, R. Vogt, and M. Mason, “The qut-noisetimit corpus for evaluation of voice activity detection algorithms,” in
INTERSPEECH. International Speech Communication Association, 2010, pp. 3110–3113. [70] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” arXiv preprint arXiv:1510.08484, 2015. [71] D. Diaz-Guerra, A. Miguel, and J. R. Beltran, “gpurir: A python library for room impulse response simulation with gpu acceleration,” Multimedia Tools and Applications, vol. 80, no. 4, pp. 5653–5671, 2021. [72] A. Politis, S. Adavanne, and T. Virtanen, “A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection,” arXiv preprint arXiv:2006.01919, 2020. [73] A. Politis, S. Adavanne, T. Virtanen, E. Fagerlund, A. Koskimies, A. Hakala, and A. Gohar, “TAU Spatial Room Impulse Response Database (TAU-SRIR DB),” Apr. 2022. [74] K. Shimada, K. Uchida, Y. Koyama, T. Shibuya, S. Takahashi, Y. Mitsufuji, and T. Kawahara, “Zero-and few-shot sound event localization and detection,” in ICASSP. IEEE, 2024, pp. 636–640. [75] J. Wilkins, M. Fuentes, L. Bondi, S. Ghaffarzadegan, A. Abavisani, and J. P. Bello, “Two vs. four-channel sound event localization and detection,” arXiv preprint arXiv:2309.13343, 2023. [76] Y. Zhang, S. Wang, Z. Li, K. Guo, S. Chen, and Y. Pang, “Data augmentation and class-based ensembled cnn-conformer networks for sound event localization and detection,” Proc. DCASE, vol. 2021, 2021. [77] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen, “Starss22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,” arXiv preprint arXiv:2206.01948, 2022. [78] Á. F. Garcı́a-Fernández, A. S. Rahmathullah, and L. Svensson, “A metric on the space of finite sets of trajectories for evaluation of multi-target tracking algorithms,” IEEE Trans. on Signal Processing, vol. 68, pp. 3917–3928, 2020. [79] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan, “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024.