Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement

[2608.14812] Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Electrical Engineering and Systems Science > Audio and Speech Processing arXiv:2608.14812 (eess) [Submitted on 14 Aug 2026] Title: Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement Authors: Tongtao Ling , Zhong-Qiu Wang View a PDF of the paper titled Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement, by Tongtao Ling and Zhong-Qiu Wang View PDF HTML (experimental) Abstract: Audio-visual speech enhancement (AVSE) aims at extracting target speech from multi-speaker mixtures by exploiting visual cues. Although recent studies have reported strong performance on simulated datasets, the performance, however, often drops dramatically when they are applied to real-world audio-visual recordings. To bridge this gap, the Real-World AVSE Challenge held in the ISCSLP 2026 conference calls for participants to design a practical solution for AVSE under real-world conditions, where speaker overlap, acoustic interferences, room reverberation and visual degradations naturally co-exist. In our submission to the challenge, we propose a decoupled separation-then-association approach. It consists of two stages: a separation stage in which a trained, audio-only model (i.e., not using visual cues) is used to separate input multi-speaker mixture to individual speaker signals, followed by an association stage, where an audio-visual CLIP model is used to identify the separated speech signal with the highest similarity with the target speaker's facial video via cross-modal similarity matching. Evaluation results on the challenge dataset show the effectiveness of our proposed approach. Subjects: Audio and Speech Processing (eess.AS) Cite as: arXiv:2608.14812 [eess.AS] (or arXiv:2608.14812v1 [eess.AS] for this version) https://doi.org/10.48550/arXiv.2608.14812 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Tongtao Ling [ view email ] [v1] Fri, 14 Aug 2026 18:30:13 UTC (103 KB) Full-text links: Access Paper: View a PDF of the paper titled Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement, by Tongtao Ling and Zhong-Qiu Wang View PDF HTML (experimental) TeX Source view license Current browse context: eess.AS < prev | next > new | recent | 2026-08 Change to browse by: eess References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement#1035617
• Grounded Decoding for Autoregressive Speech Enhancement via Adaptive Code-Space Grounding and Local LLM Refinement#927546
• Gradient Phonetic Structure in Bilingual Speech: A Multidimensional Analysis of L1 and L2 Production#286166
• FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching#1155133
• Perceptually Motivated Alignment and Interpolation of Pitch-Aligned Time-Frequency Representations#1115927