arXiv:2605.03384v1 [cs.CR] 5 May 2026
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition Bikrant Bikram Pratap Maurya∗
Nitin Choudhury∗†
IIIT-Delhi India [email protected]
IIIT-Delhi India [email protected]
Daksh Agarwal
Arun Balaji Buduru‡
Guru Gobind Singh Indraprastha University, Delhi India [email protected]
IIIT-Delhi India [email protected]
Abstract Acoustic side-channel attacks (ASCA) on keyboards pose a significant security risk, as keystrokes can be inferred from the typing acoustics, revealing sensitive information on laptops. Prior studies on ASCA are limited due to dataset constraints, which include a smaller number of users, keyboards, and environments. This limits the exploration of the potential of this attack vector across different keyboards, users, microphones, and ambient noise conditions. To this end, we introduce HEAR, a novel dataset designed to study ASCA along three axes: keyboard generalization, noise adaptation, and user bias. HEAR contains recordings from 53 participants using 37 different laptop keyboards, collected in three realistic settings: (i) captured via an external microphone, and (ii) captured through the user’s own device microphone over no network channel noise, and (iii) captured while in a network streaming platform over VoIP. This enables controlled benchmarks for cross-user, cross-keyboard, and cross-environment evaluations in a wider scope. On HEAR, we establish a new ASCA benchmark that spans conventional features and pre-trained representations from both raw audio and spectrograms, in unimodal and multimodal settings. We further propose DECKER, a domain-invariant keystroke inference framework that works in a four-stage generalization strategy: (i) Keyboard Signature Normalization to reduce device coloration in the waveform domain, (ii) domain-adversarial disentanglement to suppress keyboard identity, (iii) supervised cross-keyboard contrastive alignment to enforce key consistency across devices, and (iv) Acoustic Style Randomization to synthesize unseen keyboard responses. To explore the potential depth of the attack, we further investigate a sentence-level inference attack that leverages a large language ∗ Both authors contributed equally to this research as first authors. † Correspondanding Author. ‡ Primary Advisor.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
model (LLM) layer as a post-processing layer, refining the predicted keystroke sequences using linguistic context. Empirical results on HEAR show that DECKER improves keystroke identification over strong unimodal and multimodal baselines, especially in cross-keyboard and cross-user settings, and that the language-model rectification further boosts sequence recovery. These findings highlight that ASCA remains effective under diverse users, devices, and noisy environments, underscoring its practical security risk.
CCS Concepts • Security and privacy → Side-channel analysis and countermeasures; • Computing methodologies → Machine learning approaches; Representation learning.
Keywords Acoustic Side Channel Attack, ASCA, Keystroke Inference, Deep Learning, Domain Invariance, Cross-Keyboard Generalization, ECAPATDNN, Contrastive Learning, Adversarial Representation Learning, Audio Processing, Security and Privacy ACM Reference Format: Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru. 2018. DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 14 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
An Acoustic Side-Channel Attack (ASCA) on a keyboard relies on the assumption that each keystroke produces a characteristic sound that can serve as a fingerprint for that key on a given keyboard. Prior work has empirically validated this assumption across multiple setups and threat models [2, 11]. Laptops are now the primary platform for both personal and professional computing. Their portability allows them to be used in homes, libraries, cafés, shared workspaces, classrooms, and online meetings, across both private and public settings. In all these contexts, users routinely type sensitive information, such as passwords, PINs, confidential documents, and private messages, on their ondevice physical keyboards. This convenience comes with a security
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
cost: keystrokes can leak information through acoustic side channels. In practice, keystroke sounds can be passively captured not only by the nearby smartphones, concealed IoT devices, or other ambient microphones, but also from the laptop’s own microphone. As microphones become pervasive in modern environments, the attack surface for ASCA expands accordingly, turning everyday laptop use into a potential threat to user privacy. Currently, ASCA is particularly concerning as it is passive, covert, and deployable in ordinary social settings. A single device in a common space can silently record keystrokes from multiple users, independent of network boundaries or encryption. Further, typed inputs are rarely random in nature; they follow linguistic or structural patterns that adversaries can exploit. The emergence of large language models (LLMs) [17, 18] amplifies this threat significantly, making recovery of coherent text even from noisy or partially incorrect keystroke predictions plausible. Previous research on keyboard ASCA has mostly been evaluated under narrow and idealized conditions. This includes setups that involve limited demographic variation among participants, low and stable noise levels, fixed device geometry, uniform keyboard switch mechanisms, and consistent typing habits [1, 2, 7, 10, 11, 19, 21, 24]. Recent work, such as RefleXnoop and Heimdall, partially relaxes the geometric assumptions by developing systems which demonstrate that multipath reflections can enable non-line-of-sight (NLoS) keystroke recovery [1, 15]. On the modeling side, ASCA pipelines have progressed from early signal-processing classifiers to modern probabilistic and deep learning approaches. Initial work relied on temporal and spectral features such as FFT coefficients, MFCCs, and power or energy profiles, combined with statistical classifiers [22, 25]. Subsequent systems incorporated hidden Markov models (HMMs) [8] and neural architectures [2, 11] to better capture temporal structure and variability in keystroke acoustics. Recent efforts adopt convolutional and transformer-based models on audio spectrograms [3] and show that LLMs can correct noisy keystroke predictions when linguistic context is available [3]. Despite these advances, most ASCA studies still assume controlled environments and single-device training regimes. They rarely account for the variability of real-world settings, where posture changes, ambient noise, reflective surfaces, and nearby devices introduce substantial acoustic distortion, and where users type on a wide range of laptop keyboards. As a result, the real-world generalizability and effectiveness of current ASCA techniques remain underexplored. Additionally, user biases, such as gender, age group, and hand dominance, play significant roles in ASCA, influencing factors including typing speed, key pressure, and typing errors. These gaps obscure realistic assessments of ASCA feasibility. Importantly, no prior work systematically evaluates ASCA across demographic diversity or examines user awareness of acoustic leakage threats. This highlights the need for a dataset and evaluation framework that spans diverse users, keyboards, and environments under realistic threat models, while explicitly enabling analysis of user bias and human factors in acoustic side-channel attacks. Motivated by these gaps, we first propose HEAR, a novel dataset designed for ASCA evaluation under a realistic threat model. Unlike existing ASCA datasets, which are highly constrained—typically single-keyboard, single-environment, and demographically narrow
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
(often limited in gender distribution, typing backgrounds, and laptop usage profiles) [11, 15], HEAR allows evaluation of ASCA under user biases, keyboard, and environment diversity. On HEAR, we establish a new ASCA benchmark using strong audio and vision pre-trained models (PTMs) on both raw waveforms and spectrograms in unimodal and multimodal settings. To improve generalization across keyboards, environments, and users, we design DECKER as an ASCA training framework that unifies five key components: Keyboard Signature Normalization (KSN) to reduce device-specific artifacts at the waveform level, domain-adversarial disentanglement via gradient reversal [9], cross-keyboard supervised contrastive alignment [13], Acoustic Style Randomization (ASR) to simulate unseen keyboard acoustics [16], and LLM-assisted decoding with constrained beam search to reconstruct coherent text or passwords [17, 18]. Together, these components yield embeddings that are key-discriminative, keyboard-invariant, and linguistically coherent, enabling reliable ASCA inference across diverse real-world settings and providing the most comprehensive evaluation to date of keystroke leakage risks in open-world environments. To summarize, the contributions of this work are as follows: • HEAR: a realistic ASCA dataset. We curate HEAR, a dataset of keystroke acoustics from 53 participants typing on 37 laptop keyboards, recorded under three realistic conditions (external microphone, on-device microphone, and VoIP streaming). The dataset is annotated with demographic attributes and user-awareness responses, enabling analysis of keyboard generalization, noise adaptation, and user bias under a realistic threat model 1 • A comprehensive ASCA benchmark. On HEAR, we establish a benchmark spanning conventional features and strong PTM representations on both raw waveforms and spectrograms, in unimodal and multimodal settings. We evaluate cross-user, cross-keyboard, and cross-environment scenarios to quantify the extent to which existing ASCA approaches generalize beyond controlled laboratory conditions. • DECKER: a domain-invariant keystroke inference framework. We introduce DECKER, an ASCA training framework that combines Keyboard Signature Normalization (KSN), domain-adversarial disentanglement, cross-keyboard supervised contrastive alignment, and Acoustic Style Randomization (ASR) to learn key-discriminative yet keyboard-invariant representations. This improves robustness across devices, users, and environments compared to strong unimodal and multimodal baselines. • LLM-assisted sequence reconstruction and risk quantification. We investigate a sentence-level attack that applies LLM-assisted decoding with constrained beam search to refine keystroke predictions using linguistic context. We show that language-model rectification substantially boosts sequence recovery, providing a more realistic upper bound on keystroke leakage and quantifying ASCA risk in openworld scenarios.
1 Sample data is made available here: https://anonymous.4open.science/r/Decker-F341/
README.md. Access to full data will be provided upon request, for academic research purposes only.
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
2
Background and Related Works
Acoustic side-channel analysis has re-emerged as a practical threat due to the convergence of three trends: the widespread availability of high-quality microphones capable of capturing far-field audio, rapid progress in deep learning architectures for noisy acoustic signals, and an increasing reliance on mobile computing in public and semi-public environments. Modern keystrokes generate mechanical and airborne acoustic signals that can be captured by microphones embedded in laptops, smartphones, earbuds, and conferencing equipment. Foundational ASCA studies show that these signals carry enough structured information to enable adversaries to infer typed characters without compromising the victim’s device [2, 25]. This section reviews the physical, algorithmic, and linguistic factors underlying our design of DECKER.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Systems such as RefleXnoop [15] and Heimdall [1] leverage multipath cues to enable NLoS keystroke recovery, demonstrating that reflective paths are not merely noise. However, multipath patterns vary substantially across keyboards, room geometries, user postures, and microphone placements. As a result, the acoustic signature of a keystroke on one keyboard often aligns poorly with the same key on another keyboard, undermining the transferability of current ASCA pipelines. These observations motivate domain-robust acoustic modeling: we need encoders that suppress keyboard- and environment-specific coloration while preserving information about the underlying key identity. This requirement directly informs the design of the normalization, adversarial, and contrastive components in DECKER.
2.3 2.1
Acoustic Emanations of Keystrokes
Pressing a key produces a short broadband pulse shaped by multiple mechanical components, including switch actuation, keycap vibration, chassis resonance, air-pressure release, and reverberation from nearby surfaces. Although user typing style introduces variability, prior work shows that each key emits a relatively consistent acoustic pattern on a given device [2, 25]. Spatially adjacent keys tend to share similar acoustic signatures due to overlapping vibration pathways, whereas keys in distinct keyboard regions differ in spectral distribution and decay characteristics. Traditional ASCA systems leveraged this structure by utilizing MFCCs or short-time spectral features in conjunction with classical classifiers [25]. Recent deep learning models, however, demonstrate that even low-cost microphones can capture sufficient detail for key classification with striking accuracy [10, 11, 15, 19, 20, 24]. CNNs and transformer-based architectures further exceed the previous baselines significantly on laptop keyboards [24], while waveformlevel encoders such as ECAPA-TDNN extract fine transient cues, which have been traditionally used in speaker verification tasks [6]. These results confirm that keystroke acoustics are highly structured and reproducible, even for modern scissor-switch laptop keyboards. At the same time, they reveal substantial sensitivity to device-specific factors including switch type, keyboard geometry, chassis composition, and microphone placement, which sharply limits the cross-device generalization of existing ASCA models [1, 11, 15].
2.2
Multipath Propagation and Environmental Effects
Keystroke sounds reach the microphone following multiple undirected paths, such as reflections emanating from the laptop screen, desk surface, user’s hands, and surrounding walls, etc. These reflections introduce predictable but nonlinear modifications to the spectral envelope and temporal decay of the signal. For example, the laptop screen often acts as a stable reflector that produces secondary peaks in the waveform [15], which can increase distinguishability in some settings while degrading it in others. Recent studies show that these reflections encode spatial and structural information of the keyboard and its environment [1, 15].
Deep Learning for Acoustic Attacks
Deep learning has become the dominant approach for extracting keystroke information from acoustic side channels. Early methods combined MFCCs or related hand-crafted features with simple perceptron-based feed-forward networks [25], while more recent systems adopt CNN-based architectures, residual networks, ConvMixer architectures [23] on audio spectrograms, and audio pretrained models (PTMs) such as wav2vec 2.0 [4], HuBERT [12], and WavLM [5]. These models capture rich temporal–spectral patterns and achieve high accuracy even when keystrokes traverse noisy VoIP channels or mobile phone microphones [11, 15, 24]. Despite these advances, two limitations remain prominent: Lack of domain invariance. Deep models tend to overfit to keyboard- and setup-specific spectral patterns. When evaluated on an unseen keyboard or recording configuration, performance often drops sharply [1, 11, 15], indicating that the models encode device identity and environmental coloration rather than pure keystroke content. Lack of sentence-level modeling. Even when per-keystroke predictions are reasonably accurate, local errors accumulate over long sequences such as passwords or sentences. Most ASCA pipelines treat keys independently and do not systematically exploit linguistic structure to correct errors or improve sequence-level reconstruction. These limitations suggest that ASCA requires not only stronger acoustic encoders, but also explicit mechanisms for domain generalization and sequence reconstruction.
2.4
Language Models in Side-Channel Attacks
Human-generated keystroke sequences exhibit strong statistical regularities, including character-level n-grams, word morphology, syntax, and common patterns in password construction. Large language models (LLMs) encode these regularities and can correct sequences with local errors, a property that has already been exploited in ASCA-like settings as an autocorrect or heuristic post-processing step [15]. However, prior work typically applies language models in a limited way: as loosely coupled filters on top of keyboard-specific acoustic models. To date, there is no systematic combination of: (i) domain-robust acoustic encoder that produces stable key-level probability distributions across keyboards and environments, and
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
3
Threat Model
We adopt a realistic and conservative threat model consistent with modern acoustic side-channel literature. As in RefleXnoop, Heimdall, and other NLoS (Non-Line-of-Sight) attacks, the adversary does not require device compromise, elevated privileges, or abnormal physical proximity. The attacker relies solely on passive audio capture and offline computation, including modern large language models (LLMs) for sequence reconstruction.
3.1
Figure 1: Threat scenario. Bob types on a laptop in a public area while Eve, seated at a separate table, passively captures keystroke acoustics from a concealed device. Eve later performs domain-invariant acoustic inference followed by LLM-driven sentence reconstruction. No line-of-sight to the keyboard is required.
a (ii) constrained LLM decoder that reconstructs full sentences or passwords from these noisy distributions under realistic threat models. This gap motivates the LLM-assisted decoding stage in DECKER, which uses language priors to refine character sequences in the presence of keyboard-invariant acoustic uncertainty.
2.5
Motivation for DECKER
Considering all the observations and limitations clearly demonstrates the need for a unified ASCA framework that addresses both levels of keystroke inference: Acoustic level: Learn representations that are discriminative for key identity while being invariant to keyboard, user, and environment—mitigating overfitting to device-specific coloration and multipath effects. Sequence level: Use a language model to correct local key-level errors, enforce linguistic plausibility, and reconstruct meaningful text or passwords. DECKER is designed to meet these requirements. At the acoustic level, it combines Keyboard Signature Normalization, domainadversarial disentanglement [9], supervised cross-keyboard contrastive alignment [13], and Acoustic Style Randomization [16] to produce key-discriminative yet keyboard-invariant embeddings. At the sequence level, it employs constrained LLM-based decoding informed by pretrained autoregressive and sequence-to-sequence models [17, 18] to reconstruct coherent text from noisy keystroke predictions. Together, these components yield a domain-robust ASCA pipeline that can reconstruct meaningful sequences across unseen keyboards and recording conditions, showing that acoustic side-channel attacks remain practically viable even under substantial real-world variability.
Attack Scenario
We consider a common public or semi-public scenario involving a benign user (Bob) and a passive eavesdropper (Eve): (i) Bob uses a laptop with a built-in QWERTY keyboard while working in libraries, cafes, classrooms, or co-working spaces. During typical use, Bob types authentication credentials, emails, search queries, and natural language text. (ii) Bob practices reasonable cybersecurity hygiene (no untrusted software, no device compromise, vigilance against shoulder-surfing) but does not anticipate long-distance acoustic snooping using environmental microphones. (iii) Eve positions herself at a socially acceptable distance (1–3 m), often behind Bob or at a neighboring table. She maintains plausible behavior and avoids close-range surveillance. Direct line-of-sight to the keyboard is not necessary. After recording, Eve processes the audio using the DECKER pipeline to infer key-level sequences and subsequently applies an LLM to reconstruct full sentences or passwords, improving inference accuracy through linguistic context.
3.2
Attacker Capabilities
Eve is equipped with capabilities feasible to unskilled or moderately skilled adversaries using commodity equipment: Passive Acoustic Capture. Eve records ambient audio through a smartphone microphone, a compact USB microphone, or a small microphone array. No signal transmission, active probing, or ultrasonic injection is required. No Access to Victim Device. Eve does not modify Bob’s software, kernel, browser, or network stack. She cannot observe screen contents, clipboard data, or input logs. The attack is entirely external and requires no permissions. Covert Placement. The recording device is concealed inside everyday objects (e.g., a bag, book pile, jacket pocket). Its footprint approximates an A4-sized object, enabling unobtrusive placement in shared seating arrangements. Non-Line-of-Sight (NLoS) Robustness. The adversary may be obstructed by Bob’s hands, laptop chassis, or room elements. Multipath reflections off the laptop screen, desk, and walls provide sufficient acoustic leakage for inference. Offline Computational Resources. Eve has access to GPU-equipped systems enabling offline execution of deep-learning models. This includes domain-invariant acoustic encoders (DECKER) and pretrained large language models (GPT-2, FLAN-T5) for correcting noisy keystroke predictions. LLM-Assisted Post-Processing. The adversary can refine the noisy character predictions using an LLM that exploits linguistic structure (e.g., word frequency, grammar, password patterns). This
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
elevates the attack from character inference to full-sentence and password reconstruction.
3.3
Environmental Assumptions
The attack is assumed to occur in typical public spaces without strict acoustic control: Ambient Noise. Conversations, footsteps, HVAC noise, and utensil clatter are expected. Background noise does not invalidate the attack because keystrokes produce high-SNR transients and DECKER employs data-driven normalization. Multipath Acoustic Propagation. Keystroke sounds reflect off the laptop screen, tabletop, and surrounding environment. These reflections form multiple arrival paths that preserve keystroke identity even when the direct path is blocked. Ordinary Laptop Posture. The screen is positioned at typical viewing angles (95◦ –125◦ ), creating consistent reflection geometries. Realistic Human Spacing. Eve remains at 1–3 m, matching realworld seating patterns and avoiding suspicious behavior.
3.4
Vulnerability Basis
The feasibility of the attack is grounded in two well-established principles: V1. Distinctive, Repeatable Acoustic Signatures. Each keyboard key exhibits device-specific spectral and temporal patterns. Modern microphones capture enough structured variation to discriminate among keys. V2. Linguistic Regularity in Typed Text. Even when some keystrokes are misclassified, typed sequences follow predictable linguistic constraints (bigrams, syntax, common passwords). LLMs exploit these constraints to correct local acoustic errors, enabling recovery of semantically valid text.
3.5
Attacker Assumptions: Capabilities and Environment (Summarized) • A1. Passive recording only; no probes or malware. • A2. Commodity hardware microphones. • A3. Concealed placement within everyday objects. • A4. Socially realistic distance (1–3 m). • A5. NLoS operation enabled by multipath reflections. • A6. Robustness to ambient real-world noise. • A7. Offline GPU computation permitted. • A8. Access to pretrained LLMs for decoding sequences.
This threat model captures both the physical and algorithmic realism of modern ASCA attacks, including the enhanced inference potential unlocked by LLM-guided reconstruction.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
we introduce the HEAR dataset, a multi-device, multi-user, multigender, multi-environment corpus with synchronized recordings across laptops, smartphones, and streamed audio channels (VoIP). This dataset is collected using a reproducible browser-based acquisition pipeline and is the most comprehensive ASCA resource to date.
4.1
4.2
Dataset: HEAR
Robustness evaluation of cross-device ASCA requires a dataset that reflects the diversity of real-world users, keyboards, microphone types, and environments. Prior datasets have been limited in scope, typically consisting of single-keyboard, single-user, cleanenvironment collections, which makes it difficult to study whether ASCA can generalize beyond laboratory conditions. To address this,
Comparison With Prior ASCA Datasets
Hear is the first dataset to unify all nine ASCA-relevant factors —particularly multi-gender, over-stream, multiple microphone, NLoS, and continuous-text typing—yielding the broadest known ASCA evaluation surface. This diversity is crucial for evaluating generalizationdriven methods such as DECKER. A comparative analysis of existing ASCA datasets against DECKER (Table 1) illustrates the broader scope and greater realism of our proposed dataset.
4.3
Acquisition Pipeline
All data were collected using a custom browser-based tool written in HTML/JavaScript and provided as part of the artifact. The tool is designed and developed to (i) capture continuous audio using MediaRecorder, (ii) log high-resolution keystroke timestamps, (iii) record participant metadata (gender, handedness, session number, etc.), (iv) export one continuous audio file plus structured metadata and pre-sliced per-key WAV clips. The interface also provides an informed consent checkbox and captures demographic fields essential for user-diversity analysis.
4.4 4
Design Goals
HEAR is constructed considering six core objectives that are essential for realistic ASCA research: (i) Keyboard Diversity: laptop scissor-switch keyboards, external membrane keyboards, and various mechanical-switch boards. (ii) User Diversity: participants across genders, handedness, and typing backgrounds; metadata captured in-session. (iii) Environmental Realism: recordings in quiet (library), moderatenoise (office), and high-noise (café/outdoor) settings. (iv) Device Diversity: built-in laptop microphones, smartphones placed at varying distances/orientations, and VoIP/streamed audio. (v) Synchronization for Multi-Recorder ASCA: explicit offset estimation between primary and secondary devices, enabling multichannel ASCA analysis. (vi) Reproducibility: Every component is automated and recorded with structured metadata. This design enables controlled cross-device, cross-user, and crossenvironmental splits, which are essential for evaluating domaininvariant ASCA models, such as DECKER.
Standardized Typing Corpus
The participants are provided with a standardized paragraph crafted to cover the alphabet (A–Z, a–z), digit patterns, punctuation, symbols and operators, uppercase acronyms, and realistic word and phrase frequencies for LLM reconstruction. Participants also typed a structured block of special keys (Tab, Backspace, Delete, arrows, Esc, F1–F12, Caps Lock, Enter), enabling non-character key classification experiments. A detailed corpus is provided in Appendix A.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
Table 1: Comparison of ASCA datasets across device diversity, user demographics, acoustic conditions, and recording modalities.Hear is the first dataset to jointly include multi-device (MD), multi-user (MU), multi-gender (MG), noisy public environments (NE), NLoS capture, continuous text, and VoIP/smartphone streams. Dataset Asonov & Agrawal (2004) [2] Zhuang et al. (2005) [25] Harrison et al. (2023) [11] RefleXnoop (CCS’24) [15] Heimdall VR (NDSS’24) [1] HEAR
4.5
Sample Size
MD
MU
MG
NE
NLoS
Continuous Text
VoIP/smartphone streams
∼2k ∼10k 900 5k+ 10400 10k+
✗ ✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✓
✗ ✗ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✓
✗ ✗ ✓ ✗ ✓ ✓
Per-Key Slicing and Signal Processing
Given a keystroke timestamp 𝑡 ms , audio is sliced using a fixed window: 𝑡 start = 𝑡 ms − 60 ms, 𝑡 end = 𝑡 ms + 200 ms. This window captures both the impulsive onset and early reverberation of keystrokes, consistent with prior ASCA literature. Windows shorter than 1.2 kB are discarded to remove empty or noise-only segments. We note that rapid typing can cause partial overlap between adjacent windows; such overlap is intentionally retained to reflect realistic attack conditions.
4.6
Multi-Device and Over-Stream Synchronization
Because the Hear dataset includes smartphone audio, VoIP feeds, and secondary microphones, Synchronization is performed relative to the browser reference recording. When available, explicit synchronization tones allow estimation of offset 𝑜ˆ𝑑 and linear drift 𝑏𝑑 : 𝑡𝑑 (𝑡) = 𝑡 + 𝑜ˆ𝑑 + 𝑏𝑑 𝑡 . When markers are unavailable, offset is estimated via normalized cross-correlation: ∑︁ 𝜏ˆ = arg max 𝑥 (𝑡) 𝑦 (𝑡 + 𝜏).
identities. Cross-Environment: quiet/moderate → café/outdoor. Cross-Device: laptop → smartphone/VoIP. Metrics include key-level accuracy, top-𝑘, sentence-level accuracy, Levenshtein edit distance, and segmentation robustness (§6.3).
4.9
Ethics, Consent, and User Awareness
We follow established ethical procedures throughout the study. All participants provided informed consent before participating. Personally Identifiable Information (PII) is removed with explicit care, and we explicitly consider k-anonymity on the data before considering it for our experimental purposes. We further limit data access to requests and academic research purposes only. Impact: The Hear dataset enables the first robust study of ASCA generalization across gender, device type, microphone types, VoIP channels, and environmental noise, making it an essential foundation for evaluating modern ASCA pipelines such as DECKER.
5
Methodology
This section presents a mathematical formulation of ASCA, describes the problem setting, outlines the components of our framework, and then describes the end-to-end training and inference pipeline.
𝜏 𝑡
Drift is estimated using correlations over temporally separated segments. Residual alignment errors are typically within a few milliseconds, which we account for in segmentation robustness experiments.
4.7
Metadata Schema
We collect metadata from users in each session, which includes the continuous browser recording (webm/ogg), per-key WAVs for all devices, and complete metadata in JSON format. This schema preserves demographic, acoustic, environmental, and segmentationrelevant information required by downstream modeling. Detailed metadata descriptions and collection formats are discussed in Appendix B.
4.8
Recommended Evaluation Splits
We define four canonical splits for experimentation reproducibility, which include Cross-Keyboard (primary): train on a subset of keyboards; test on unseen keyboards. Cross-User: disjoint user
5.1
Problem Definition
𝑁 , We consider keystroke audio samples as, D = {(𝑥𝑖 , 𝑦𝑖 , 𝑑𝑖 )}𝑖=1 where 𝑥𝑖 ∈ R𝑇 is the raw waveform, 𝑦𝑖 ∈ {1, . . . , |K |} is the key label, and 𝑑𝑖 ∈ {1, . . . , |D|} denotes keyboard domain identity. And, let 𝜙 (·) be the encoder producing embeddings as: 𝑧𝑖 = 𝜙 (𝑥𝑖 ) ∈ R𝐸 . DECKER aims to learn an embedding space satisfying three key aspects: Key discriminability, Keyboard invariance, and cross-keyboard consistency. This formulation enables DECKER to generalize across unseen devices.
5.2
Keyboard Signature Normalization (KSN)
Keystrokes recorded on different keyboards exhibit systematic spectral coloration caused by their impulse responses, which we model as: 𝑥𝑖 = ℎ𝑑𝑖 ∗ 𝑠𝑖 , where 𝑠𝑖 is the latent keystroke impulse and ℎ𝑑𝑖 is the device-specific response. DECKER introduces Keyboard Signature Normalization (KSN), a learnable inverse-filter module: 𝑥˜𝑖 = 𝑔𝜃 (𝑥𝑖 ). The architecture of KSN comprises a CNN block with a sequence of four 1D CNNs, each
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
2. Augment
1. Norm
Raw Keystroke xi ∈ RT (Waveform)
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
ASR
KSN Keyboard Signature Norm.
Acoustic Style Rand.
gθ (xi ) ≈ si
x0i = Rγ Wβ (xi ∗ hα )
3. Encode
ECAPA-TDNN Encoder (φ)
4. Decode
Log-Mel Spec. Si ∈ RF ×L
LLM Refinement GPT / FLAN-T5
Beam Search
Reconstructed Text Ŷ
Log-mel
Top-k Candidates Ct = {yt,1 . . . }
Multi-scale Conv + Attn Pooling
Embedding zi ∈ RE
GRL
Lsupcon Align Keys
Gradient Reversal
Lkey
Ck (zi )
Ldom Cd (zi )
Training Objectives
Figure 2: DECKER pipeline. (1) Raw keystrokes are normalized with KSN, (2) augmented using ASR, (3) encoded with ECAPATDNN, (4) aligned via adversarial and contrastive learning, and (5) reconstructed into sentences using constrained LLM decoding. comprising 64 feature channels, each followed by Batch Normalization and ReLU activation, with different kernel sizes (7, 7, 9, 9) and dilation rates (1, 2, 4, 1) to capture patterns at multiple temporal scales (from local to wider context via dilation). The output representations are then passed through a linear projection that aligns their dimensions and merges with the original input through a residual connection. The output from this CNN module is then passed through a frequency-normalization layer to further suppress device coloration. In the frequency domain, the representations are expressed as: 𝑋˜ 𝑖 (𝑓 ) = 𝐺𝜃 (𝑓 ) 𝑋𝑖 (𝑓 ) ≈ 𝑆𝑖latent (𝑓 ), where 𝐺𝜃 (𝑓 ) acts as an approximate inverse filter. Figure 3 shows a visual demonstration of KSN removing device-coloration while preserving transient structure.
channel-attentive feature recalibration to produce robust framelevel representations for utterance-level embedding extraction. Formally, for a normalized spectrogram 𝑆𝑖 ∈ R𝐹 ×𝐿 , the encoder produces frame-level features: 𝐻𝑖 = 𝜙 (𝑆𝑖 ), 𝐻𝑖 ∈ R𝐶 ×𝐿 , where 𝐹 is the number of frequency bins, 𝐿 the number of frames, and 𝐶 the channel dimension. A statistics pooling layer aggregates frame-level features across time:
5.3
yielding a pooled representation [𝜇𝑖 ; 𝜎𝑖 ] that summarizes the sequence. A linear head then maps this pooled vector to a fixeddimensional embedding used as the acoustic representation of keystroke sequences.
Acoustic Style Randomization (ASR)
The output representations from the KSN module are then passed through an acoustic style randomization (ASR) module to simulate keyboards that are unseen. The acoustic properties are randomized using: 𝑥𝑖′ = 𝑅𝛾 · 𝑊𝛽 (𝑥𝑖 ∗ ℎ𝛼 ), where ℎ𝛼 , 𝑊𝛽 , and 𝑅𝛾 defines random IIR filters (order 6-12, 𝑄 ∈ [0.4, 1.2]), smooth spectral envelope warp (±12% mel shift), and exponential decay perturbation (𝛾 ∈ [0.85, 1.15]) respectively. ASR encompasses a wide range of resonances, particularly in laptops.
5.4
ECAPA-TDNN Encoder
We further employ an ECAPA-TDNN [6] module to capture finegrained temporal cues and short-duration transients. After KSN and ASR, the normalized signal is passed through ECAPA-TDNN, which first extracts log-mel spectrograms and then combines them by passing them through multi-scale 1D temporal convolutions with
𝐿
1 ∑︁ 𝐻𝑖 (:, 𝑡), 𝐿 𝑡 =1 v u t 𝐿 2 1 ∑︁ 𝜎𝑖 = 𝐻𝑖 (:, 𝑡) − 𝜇𝑖 , 𝐿 𝑡 =1 𝜇𝑖 =
5.5
(1)
(2)
Key Classification
The final outputs from the ECAPA module are then passed to a classification layer with a softmax (𝑦ˆ𝑖 = 𝐶𝑘 (𝑧𝑖 )) activation function to predict the keystrokes. We utilize categorical cross-entropy as its loss function: ∑︁ L𝑘𝑒𝑦 = − log 𝐶𝑘 (𝑧𝑖 ) [𝑦𝑖 ]. (3) 𝑖
5.6
Domain-Adversarial Disentanglement
To remove keyboard bias from keystroke audios, we leverage domain adversarial disentanglement, where the objective is that the
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
(a) Clean (latent)
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
(b) Device-colored (held-out)
(c) Whitened baseline
(d) After KSN
Figure 3: KSN suppresses keyboard-specific spectral coloration. The device-colored sample shows strong resonance bands that KSN removes, yielding a closer approximation to the latent clean keystroke. keyboard identity 𝑑𝑖 should be unpredictable from the embeddings. To achieve this, we employ a Gradient Reversal Layer (GRL) [9]. The forward pass and backpropagation in GRL are calculated as: GRL𝜆 (𝑧𝑖 ) = 𝑧𝑖 . and 𝜕GRL 𝜕𝑧𝑖 = −𝜆𝐼 respectively. A domain classifier predicts: 𝑑ˆ𝑖 = 𝐶𝑑 (GRL(𝑧𝑖 )), (4) with loss: L𝑑𝑜𝑚 = −
∑︁
log 𝐶𝑑 (GRL(𝑧𝑖 )) [𝑑𝑖 ].
(5)
𝑌ˆ = arg max
𝑌 ∈ C1:𝑇
"𝑇 ∑︁
# log 𝑃𝜃 (𝑦𝑡 |𝑥𝑡 ) + 𝜆LM log 𝑃LM (𝑌 ) .
(12)
𝑡 =1
implemented using constrained beam search (beam=8–16). The LLM imposes linguistic and structural priors, correcting substitution, insertion, and deletion errors.
5.9
Training Objective and Schedule
𝑖
The encoder is trained to minimize classification loss but maximize domain loss: 𝜙 ∗ = arg min L𝑘𝑒𝑦 − 𝜆𝑑𝑜𝑚 L𝑑𝑜𝑚 . (6)
The full loss is: L𝑡𝑜𝑡𝑎𝑙 = L𝑘𝑒𝑦 + 𝜆𝑑𝑜𝑚 L𝑑𝑜𝑚 + 𝜆𝑐𝑜𝑛 L𝑠𝑢𝑝𝑐𝑜𝑛 . Training details.
𝜙
5.7
𝑃 (𝑖) = { 𝑗 : 𝑦 𝑗 = 𝑦𝑖 , 𝑑 𝑗 ≠ 𝑑𝑖 },
(7)
𝑁 (𝑖) = { 𝑗 : 𝑦 𝑗 ≠ 𝑦𝑖 }.
(8)
We use supervised contrastive learning [13]: ∑︁ −1 ∑︁ exp(𝑧𝑖 · 𝑧𝑝 /𝜏) log Í . L𝑠𝑢𝑝𝑐𝑜𝑛 = |𝑃 (𝑖)| 𝑎∈𝑃 (𝑖 )∪𝑁 (𝑖 ) exp(𝑧𝑖 · 𝑧𝑎 /𝜏) 𝑖
(9)
𝑝 ∈𝑃 (𝑖 )
This pulls same-key embeddings together across domains while separating different keys.
5.8
• Optimizer: AdamW, lr = 2 × 10−4 , weight decay = 10−3 • Batch size: 64, training epochs: 60 • Hyperparameters: 𝜆𝑑𝑜𝑚 = 0.5, 𝜆𝑐𝑜𝑛 = 0.1, 𝜏 = 0.07 • Models trained with 3 seeds; mean ± std reported in Sec. 5
Supervised Cross-Keyboard Contrastive Alignment
Keystrokes with the same label across devices must be aligned. Define:
LLM-Assisted Sentence Reconstruction
Even with domain-invariant embeddings, key-level errors accumulate in long sequences. We refine sequences using a pretrained LLM (GPT-2 or FLAN-T5). Let the acoustic model produce token-level distributions: 𝑃𝜃 (𝑦𝑡 |𝑥𝑡 ).
(10)
We restrict decoding to the top-𝑘 plausible keys per time step: C𝑡 = {𝑦𝑡,1, . . . , 𝑦𝑡,𝑘 }. We maximize the joint objective:
(11)
(13)
KSN is pretrained for 5 epochs with L1 spectral loss, then jointly optimized with the encoder.
5.10
End-to-End DECKER Pipeline
The End-to-End pipeline for DECKER is designed as follows: (i) Segmentation: Extract keystroke windows from continuous audio. (ii) KSN: Remove keyboard-specific coloration. (iii) ASR: Augment samples to simulate unseen hardware. (iv) Embedding: Encode via ECAPA-TDNN. (v) Domain Alignment: Apply GRL and supervised contrastive loss. (vi) Classification: Predict keys using 𝐶𝑘 . (vii) LLM Refinement: Convert noisy predictions to coherent text. Together, these components yield a domain-invariant and linguistically coherent ASCA attack that can generalize across unseen keyboards, users, and environments.
6
Results
This section evaluates three families of models under cross-keyboard, cross-user, and cross-environment conditions: (i) unimodal baselines, (ii) multimodal Euclidean fusion baselines, and (iii) our domaininvariant DECKER framework. We additionally assess LLM-assisted
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
decoding for sentence-level reconstruction. Cross-keyboard transfer constitutes the primary axis of evaluation, reflecting the adversarial setting where attackers rarely control the victim’s hardware. Statistical protocol. For all experiments, we report mean accuracy over three random seeds and 95% bootstrap confidence intervals. All claims of improvement over baselines are supported by paired bootstrap significance tests with threshold 𝑝 < 0.01. Metrics. We report Top-1 key accuracy, Top-5 accuracy, sentencelevel character accuracy, exact-match accuracy, and normalized Levenshtein distance. All results average three random seeds.
6.1
Unimodal Baselines
Table 2: Unimodal performance under keyboard shift.
Model
Seen-KB
Unseen-KB
Drop
wav2vec2 (audio) ECAPA-TDNN CoAtNet-S (spec) ResNet-34 (spec)
92.4% 95.3% 90.7% 88.5%
51.8% 58.1% 49.6% 47.9%
-40.6% -37.2% -41.1% -40.6%
Insight. All unimodal models fail to generalize across keyboards. Explanation. Waveform models absorb low-frequency chassis modes; spectrogram models encode keyboard-specific spectral textures. Since impulse responses differ across devices, embeddings cluster by device domain rather than key identity.
6.2
Multimodal Fusion Baselines
We evaluate all practical Euclidean fusion strategies using our strongest unimodal components: ECAPA-TDNN (audio) and CoAtNetS (spectrogram). Fusion is examined at early, intermediate, and late stages. F1 Early Concat: direct concatenation of embeddings. F2 Early Projection: dimension-aligned concatenation. F3 Cross-Attn (A→V): audio queries attend to vision features. F4 Cross-Attn (V→A): vision queries attend to audio features. F5 Bi-directional Cross-Attn: both directions [14] F6 Gated Fusion: learned weighting between modalities. F7 Late Fusion (avg): average probability scores. F8 Late Fusion (weighted): learned probability weights. Insight. Fusion improves performance but does not solve domain shift. Explanation. Although modalities contribute complementary cues, Euclidean fusion retains device-specific artifacts from both encoders. Even sophisticated cross-attention cannot compensate for the underlying mismatch between keyboard impulse responses.
6.3
Segmentation Robustness Analysis
Practical adversaries cannot assume perfect keystroke onset detection. In realistic acoustic eavesdropping scenarios, background noise, user motion, and microphone placement introduce temporal uncertainty that shifts or distorts key boundaries. To evaluate DECKER under such conditions, we conduct a controlled sensitivity
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 3: Multimodal Euclidean fusion of ECAPA (A) + CoAtNet-S (V).
Fusion Variant
Seen-KB
Unseen-KB
F1 Early Concat F2 Early Projection F3 Cross-Attn (A→V) F4 Cross-Attn (V→A) F5 Bi-dir Cross-Attn F6 Gated Fusion F7 Late Fusion (avg) F8 Late Fusion (weighted)
96.8% 96.9% 97.1% 96.7% 97.3% 96.4% 96.0% 96.4%
63.2% 64.1% 65.4% 64.8% 66.2% 62.7% 61.8% 62.9%
analysis measuring the impact of segmentation jitter and spurious trigger events on both key-level classification and sentence-level reconstruction.
Perturbation model. For each ground-truth keystroke timestamp, we add uniformly distributed temporal jitter 𝛿𝑡 ∼ U (−𝑇 , +𝑇 ) with 𝑇 ∈ {10, 20, 30} ms. We further simulate realistic false triggers—caused by impulsive noises or microphone preamp artifacts—by inserting spurious segments at rates of 0%, 3%, and 5% per 100 keystrokes.
Experimental findings. Table 7 summarizes performance under these perturbations. At the key level, DECKER exhibits graceful degradation: Top-1 accuracy declines by only 2.8% under ±10 ms jitter and 5.9% under ±20 ms jitter. Even at ±30 ms—well beyond realistic human–microphone temporal ambiguity—DECKER retains nearly 70% raw accuracy. Sentence-level reconstruction shows larger sensitivity, but constrained decoding with FLAN-T5 significantly mitigates the effect, reducing the drop to under 4.3% for ±20 ms jitter.
Takeaway. These results indicate that DECKER maintains stable performance under realistic segmentation imperfections. While timing noise degrades raw key-level predictions, LLM-assisted decoding substantially compensates for such errors at the sequence level. This suggests that, even in passive and noisy acquisition settings, domain-invariant acoustic modeling combined with linguistic constraints remains effective. For fully passive, continuous-stream attacks, integrating joint segmentation–classification architectures is a promising direction for future work.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
Table 4: DECKER key-level performance (domain-shift evaluation). Model Variant
Seen-KB
Unseen-KB
Gap
DECKER (full) w/o GRL w/o ASR w/o KSN ECAPA + GRL (DANN-only)
98.9% 97.8% 97.5% 96.9% 97.9%
81.3% 68.1% 75.2% 63.7% 71.6%
-17.6% -29.7% -22.3% -33.2% -26.3%
6.4
DECKER: Domain-Invariant Modeling
Insight. DECKER consistently narrows the generalization gap under keyboard shift, substantially outperforming unimodal, multimodal, and adversarial-only baselines. Explanation. While domain-adversarial training (GRL) alone improves crosskeyboard performance, it is insufficient to fully suppress keyboard-specific acoustic bias. DECKER combines Keyboard Signature Normalization (KSN) to remove low-frequency device coloration, supervised cross-keyboard contrastive alignment to enforce key consistency across devices, and Acoustic Style Randomization (ASR) to simulate unseen acoustic conditions. Together, these components yield a robust domain-invariant embedding. 6.4.1 KSN Ablation and Visualization. We evaluate the effectiveness of KSN in suppressing keyboard-specific acoustic coloration using both qualitative visualization and quantitative ablation. Domain-classification accuracy. To quantify residual keyboard information, we train a lightweight domain classifier on frozen embeddings. Domain-classification accuracy drops from 94.2% without KSN to 21.7% with KSN enabled, indicating substantial suppression of keyboard identity cues. Correspondingly, Top-1 accuracy on unseen keyboards improves from 63.7% to 81.3% when KSN is introduced (Table 4), confirming that KSN materially improves cross-keyboard generalization. We incorporate t-SNE visualization in the appendix C that presents the projections of ECAPA-TDNN embeddings computed without KSN and with KSN enabled.
6.5
Sentence-Level Reconstruction with LLMs
Table 5: Sentence reconstruction using constrained LLM decoding.
LLM DECKER (raw) GPT-2 Small FLAN-T5 Base LLaMA-2 7B GPT-4
Char Acc.
Sent. Match
Norm. Lev. Dist.
81.3% 86.9% 89.4% 91.6% 93.2%
42.6% 58.7% 62.1% 66.5% 72.8%
0.124 0.092 0.078 0.056 0.041
Insight. LLMs deliver substantial gains, even with lightweight models. Explanation. DECKER’s top-𝑘 candidates preserve correct tokens but contain local substitutions; LLMs impose global linguistic structure and recover coherent text or passwordlike strings. 6.5.1 LLM applicability to high-entropy strings and password experiments. LLM-based decoding relies on linguistic priors and therefore benefits structured natural-language text and human-like password choices. To quantify this, we evaluated three password classes: (1) Human-like passwords: curated from common password leaks and user-style rules (8–14 chars; mixed case, digits, common substitutions). Estimated entropy: 28–34 bits. (2) Random alphanumeric: uniformly sampled from [A-Zaz0-9]; entropy ≈ 52–75 bits (length dependent). (3) Printable-ASCII random: full printable ASCII sampling; entropy ≈ 70–92 bits. Results show LLM-assisted decoding yields substantial gains for human-like passwords (absolute +8–12% exact-match over raw DECKER) but provides negligible improvement for high-entropy random strings (≤2–3%). Thus, LLM post-processing increases attacker power primarily when human-selection biases reduce password entropy.
6.6
Computational Cost, Inference Latency, and Attacker Feasibility
A practical ASCA system must operate under realistic attacker-side computational constraints. We therefore benchmark DECKER on a commodity Apple M1 laptop (8-core CPU, 8-core GPU, 16 GB unified memory) and report end-to-end runtime for all modules: KSN, ASR, ECAPA-TDNN encoding, classifier inference, and LLMbased constrained decoding. Per-keystroke inference latency. Table 6 shows the average latency for a single keystroke and for a full sentence of 40–60 characters. ECAPA-TDNN inference dominates the acoustic pipeline, while LLM decoding dominates sequence-level post-processing. Table 6: DECKER inference latency on Apple M1. All values are averaged over 1000 runs. Here L, K, and S defines Latency, Key, and Sequence respectively Module
L/K
L/S
KSN (1-D CNN) ASR (optional) MelSpec extraction ECAPA-TDNN encoding Key classifier head LLM (GPT-2, beam=8) LLM (FLAN-T5-base, beam=8)
0.24 ms 0.41 ms 0.18 ms 1.42 ms 0.03 ms — —
12 ms 20–25 ms 9 ms 72–81 ms 1.2 ms 68–94 ms 83–110 ms
Total (GPT-2) Total (FLAN-T5)
2.3 ms 2.3 ms
180–220 ms 200–240 ms
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
Throughput. On the M1 CPU: • ≈ 430 keystrokes per second (KSN + ECAPA + classifier) • ≈ 4.5 fully reconstructed sentences per second (including LLM decoding) This indicates that an attacker can process keystrokes in real time, and reconstruct sentences at interactive latency. Memory usage. Peak memory consumption on M1: • KSN + ECAPA-TDNN: 340–380 MB • GPT-2 small: 780 MB • FLAN-T5 base: 1.25–1.4 GB All models comfortably fit within the 16 GB unified memory of consumer laptops. Attacker feasibility. These results imply: • A passive adversary needs only a laptop-class machine, not a GPU cluster. • Key-level inference can be done in real time even on CPU. • LLM decoding adds less than 5.25 seconds per sentence. This stands in contrast to earlier ASCA pipelines that required desktop-class GPUs or offline batch processing. Energy and thermal profile (M1). The Apple M1 CPU + GPU operate at: • 6–8 W sustained power draw during acoustic inference, • 11–13 W during LLM decoding. No thermal throttling was observed for continuous 30-minute evaluation runs. Comparison to x86 laptop-class systems. On an Intel i7-11800H (45 W TDP), ECAPA-TDNN runs ∼1.7× faster, and GPT-2 decoding ∼2.1× faster, due to higher clock frequency and AVX-512 acceleration. Thus, your M1 measurements represent a conservative lower bound—real attackers using mainstream x86 machines will observe even lower latency.
6.7
Summarized Insights Aligned to the Research Questions
RQ1: Can ASCA models generalize across keyboards? Yes—but only with domain-invariant modeling.DECKER achieves the first strong cross-keyboard performance. RQ2: Why do unimodal/multimodal baselines fail? They encode keyboard-specific acoustic coloration. Fusion adds capacity but does not remove domain shift. RQ3: What is the role of LLMs in ASCA? LLMs significantly enhance sentence-level inference, correcting errors that acoustic models alone cannot resolve. Summary: DECKER + LLM demonstrates that acoustic keystroke inference remains feasible across unseen keyboards, noisy environments, and NLoS settings, indicating a broader real-world attack surface than previously recognized.
7
Discussion and Future Work
Our findings demonstrate that acoustic emanations from keyboards are not only rich enough to permit high-fidelity keystroke inference, but also substantially more transferable across devices than
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
previously reported—provided that domain-specific artifacts are explicitly disentangled. In contrast to unimodal and Euclidean multimodal approaches that inadvertently encode keyboard identity, DECKER illustrates that domain-invariant representation learning is essential for accurately assessing the real-world feasibility of ASCA. The combination of KSN, adversarial disentanglement, contrastive alignment, and ASR yields embeddings that preserve semantic key distinctions while suppressing device-dependent spectral coloration. Furthermore, LLM-assisted decoding reveals that modern sequence models significantly magnify attacker capability by transforming noisy acoustic predictions into coherent text, even under noisy and non-line-of-sight conditions.
7.1
Implications for ASCA Research
Realistic threat assessment. Our results show that prior work may underestimate ASCA feasibility by assuming per-device training. When cross-device generalization is properly addressed, adversaries can infer meaningful text—even from keyboards never seen during training—at distances and noise levels consistent with typical public environments. This broadens the attack surface from contrived laboratory setups to realistic opportunistic settings. Role of LLMs in side-channel exploitation. LLMs fundamentally change ASCA capabilities by correcting sequential errors that acoustic models alone cannot resolve. Even small models (GPT-2, FLANT5) significantly improve sentence-level inference, while larger LLMs like GPT-4 achieve near-perfect reconstruction on many sequences. This suggests that future ASCA defenses must consider not only signal obfuscation but also linguistic post-processing countermeasures. Keyboard diversity and mechanical variation. We observe that switch type, key travel, chassis resonance, and microphone geometry contribute strong domain shifts in raw embeddings. However, once domain-invariant modeling is introduced, performance degradation across these factors is substantially reduced. This suggests that ASCA vulnerability may extend to a wide range of keyboards—including inexpensive membrane boards and modern lowtravel laptop keyboards. Limitations. Although DECKER addresses keyboard variability, several realistic factors remain open for exploration: • Continuous typing streams. Our evaluation focuses on segmented keystrokes; segmentation errors in continuous audio remain a challenge for fully passive attackers. • Extreme noise conditions. While DECKER and LLMs perform well up to café/outdoor noise levels, environments with strong impulsive or overlapping noise sources may require additional modeling. • Highly asymmetric devices. Touchscreens, ultra-low-profile keyboards, and flexible input surfaces may generate atypical impulse responses not covered by our dataset. • Per-user biomechanics. Although DECKER improves useragnostic performance, subtle personal typing traits (finger force, strike angle) may still cause model drift.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
7.2
Future Directions
Building on the insights from this work, we outline several promising directions: (1) Broader hardware and environmental diversity. Extending the dataset to include ergonomic keyboards, compact mobile keyboards, external Bluetooth devices, touchscreen typing, and hardware with unusual acoustic properties (e.g., hollow desk surfaces) will further stress-test domain generalization. (2) Continuous-text ASCA pipelines. Transitioning from isolated keystroke inference to fully continuous keystroke streams remains a key challenge. Future work should explore joint segmentation – classification models, sequence-to-sequence acoustic transformers, and streaming LLM integration. (3) Physical and software defenses. Our findings motivate investigation of practical defenses, such as: • keyboard chassis dampening or irregularizing resonance patterns, • microphone access restrictions in operating systems, • real-time audio obfuscation or keystroke noise injection, • randomized typing proxies (e.g., delayed or reordered keystroke dispatch). (4) Advanced domain generalization. Self-supervised pretraining on large-scale environmental audio may further improve invariance. Additional avenues include meta-learning for unseen hardware, test-time adaptation, and invertible modeling of keyboard impulse responses. (5) Multi-modal side-channel integration. While this work focuses on audio, real-world attackers may combine acoustic cues with optical flow, accelerometer vibration, or Wi-Fi CSI measurements. Domain-invariant fusion across heterogeneous side channels represents an important open direction. (6) LLM-aware defense strategies. Given that LLMs significantly amplify ASCA effectiveness, future defenses must consider: • adversarial prompts that degrade LLM reconstruction accuracy, • artificial grammar or keystroke padding patterns that reduce linguistic predictability, • user interface modifications that limit sequential consistency exploitable by LLMs.
7.3
Concluding Perspective
Overall, this study shows that ASCA remains an underestimated risk in modern computing environments. As microphone-equipped devices proliferate and LLMs become more capable, attackers gain stronger reconstruction abilities with minimal access or domain knowledge. DECKER demonstrates how domain-invariant learning can realistically model cross-keyboard attacks, and highlights the urgent need for both physical and algorithmic defenses tailored to the evolving landscape of side-channel threats.
8
Conclusion
This paper considers Acoustic Side-Channel Attacks (ASCA) on laptop keyboards in a real-world, open setting. Specifically, it is
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
concerned with cross-keyboard, cross-user, cross-environment generalization, although existing research has generally reported high accuracy on keystroke inference tasks. Analysis on HEAR indicates that such successes have been largely limited within closed evaluation environments, with significant degradation of strong Unimodal or Euclidean multimodal baselines on cross-keyboard, cross-user, cross-environment partitions, such that existing ASCA models tend to learn a representation of the keyboard, not a representation of keys. In order to fill this gap, we introduced HEAR, a collection of keystroke acoustic signals from 53 individuals on 37 different laptop keyboards, with external-mic, on-device, and VoIP scenarios, together with corresponding annotations on demographics and user awareness. For HEAR, we introduced a thorough benchmark on top of conventional representations, as well as high-quality audio/vision PTMs on raw waveforms and spectrograms, subject to unimodal and multimodal scenarios. This set of experiments represents a comprehensive look into how existing ASCA methods fare when non-idealized assumptions on hardware, environment, and users are abandoned. Building on this benchmark, we propose DECKER, a domaininvariant keystroke inference framework for ASCA. DECKER combines four complementary training mechanisms: (i) Keyboard Signature Normalization (KSN) : which suppresses device-specific acoustic coloration at the waveform level, (ii) domain-adversarial disentanglement to remove residual hardware-related cues, (iii) cross-keyboard supervised contrastive alignment to encourage embeddings of the same key to cluster across different devices, and (iv) Acoustic Style Randomization (ASR), which exposes the model to synthetic yet realistic variations in keyboard acoustics and room responses. Together, these components yield representations that remain discriminative of key identity. Decker is robust to variations in keyboard design, user behavior, microphone placement, and environmental acoustics. As a result, DECKER substantially narrows the generalization gap compared to strong baseline methods. We further investigate an LLM-assisted attack that applies constrained beam search to refine noisy keystroke predictions into coherent text sequences, which includes sentences and passwords. The results show that language-model-based rectification significantly improves sequence-level recovery. particularly in challenging cross-device scenarios. This finding highlights how modern language models can meaningfully amplify the real-world impact of ASCA, increasing the feasibility of such attacks in everyday laptop usage contexts. Taken together, our findings suggest that ASCA should not be viewed as just a purely laboratory phenomenon, but rather as a credible real-world threat. This is specially effective in shared or semi-public environments where microphones are ubiquitous and keyboard diversity is beyond the attacker’s control. This study establishes domain-invariant learning as a central requirement for the evaluation and deployment of ASCA systems under realistic threat models. We introduce HEAR as a representative benchmark and demonstrate the effectiveness of DECKER and LLM-assisted decoding as key components for future research in acoustic security. Finally, our results underscore the need for hardware, software, and model-level defenses that explicitly account for advances in machine learning and language models, as well as for continued
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
investigation into the role of user and demographic factors in the design of secure input devices and environments.
References [1] Anonymous. 2023. Heimdall: Passive Acoustic Keystroke Eavesdropping under Non-Line-of-Sight Constraints. arXiv preprint. Preprint (cite official venue when available). [2] Dmitri Asonov and Rakesh Agrawal. 2004. Keyboard Acoustic Emanations. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, 3–11. [3] Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, and Marcus Botacin. 2025. Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms’" Typo" Correction. arXiv preprint arXiv:2504.11622 (2025). [4] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems (NeurIPS). [5] Sanyuan Chen et al. 2022. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE/ACM Transactions on Audio, Speech and Language Processing (2022). [6] Brecht Desplanques, Joren Thienpondt, and Luc Reynaert. 2020. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN based Speaker Verification. In Proceedings of Interspeech. [7] Abhishek Fadake and Prof Wadaganve. 2023. Beyond the Clicks: Exploring Keyboard Acoustic Hacking. doi:10.13140/RG.2.2.36617.83048/2 [8] Denis Foo Kune and Yongdae Kim. 2010. Timing attacks on pin input devices. In Proceedings of the 17th ACM conference on Computer and communications security. 678–680. [9] Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropagation. In International Conference on Machine Learning (ICML). [10] B. Gunasekaran et al. 2018. Continuous Acoustic Keystroke Detection on Mobile Phones. In Proceedings of the Annual International Conference on Mobile Systems, Applications, and Services (MobiSys). [11] Joshua Harrison, Ehsan Toreini, and Maryam Mehrnezhad. 2023. A Practical Deep Learning-Based Acoustic Side Channel Attack on Keyboards. In 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). 270–280. doi:10.1109/EuroSPW59978.2023.00034 [12] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. In ICASSP / arXiv. [13] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised Contrastive Learning. In Advances in Neural Information Processing Systems (NeurIPS). [14] Shuo Liu, Weize Quan, Yuan Liu, and Dong-Ming Yan. 2022. Bi-Directional Modality Fusion Network For Audio-Visual Event Localization. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4868–4872. doi:10.1109/ICASSP43922.2022.9746280 [15] T. Mai et al. 2024. RefleXnoop: Near-Line-of-Sight Acoustic Keystroke Inference via Laptop Screen Reflections. ACM CCS / preprint. Project / preprint (cite exact source when available). [16] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proceedings of Interspeech. [17] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical Report. OpenAI. GPT-2 technical report. [18] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (JMLR) (2020). [19] N. Raghuram et al. 2023. Acoustic Keystroke Inference Using Deep Learning. Pattern Recognition Letters (2023). [20] Md. Rahman et al. 2021. I Hear Your Passwords: Acoustic Emanations from Laptop Keyboards. In USENIX Security Symposium. [21] A. SomeAuthor and B. OtherAuthor. 2020. Acoustic Reflections and Multipath in Consumer Laptops. In Proceedings of ACM/IEEE Sensys. Use exact citation if available. [22] Dawn Xiaodong Song, David Wagner, and Xuqing Tian. 2001. Timing analysis of keystrokes and timing attacks on { SSH } . In 10th USENIX Security Symposium (USENIX Security 01). [23] A. Trockman and J. Z. Kolter. 2022. Patches Are All You Need? arXiv preprint arXiv:2201.09792 (2022). [24] Tianyi Zhang et al. 2023. Acoustic Side-Channel Attack on Modern Keyboards. Scientific Reports (2023). [25] Li Zhuang, Feng Zhou, and Doug Tygar. 2009. Keyboard Acoustic Emanations Revisited. In Proceedings of the 2009 ACM Conference on Computer and Communications Security. ACM, 373–382.
Figure 4: t-SNE visualization of ECAPA-TDNN embeddings. Left: Without KSN, embeddings are dominated by keyboardspecific clustering. Right: With KSN enabled, keyboarddependent structure is significantly reduced and partial organization by key identity emerges, with residual overlap reflecting realistic acoustic ambiguity.
A
Standardized Typing Corpus The quick brown fox jumps over the lazy dog while amazing zebras quietly vex jumpy kids, proving every letter is present. Pack my box with five dozen liquor jugs to verify the alphabet twice. Digits appear forward and backward: 1234567890 then 0987654321, followed by sequences like 2468, 13579, and 314159. Now we add punctuation: . , ; : ? ! ’ " - _ ( ) [ ] { } / \. Symbols and operators: @ # $ % ˆ & * + = < > | ˜. To include uppercase properly, acronyms like NASA, USA, UN, AI, ML, and HTML are written in full caps. Typing speed and accuracy will be measured across every key. Finally, we conclude with a mix: The year is 2025; version v1.0-beta includes features [alpha], {bravo}, and (charlie), fully covering the QWERTY layout.
B
Metadata Schema
{ "participant": {"id":"U001", "gender":"Female", "handedness":"Right" }, "pre_ms": 60, "post_ms": 200, "sample_rate": 48000, "keys":[{"key":"a","t_ms":123.4},...], "exported_files":[{...}], "env":{"location":"cafe","noise_level":"high"} }
C
KSN Ablation and Visualization
Figure 4 presents 2D t-SNE projections of ECAPA-TDNN embeddings computed without KSN and with KSN enabled. Without KSN, embeddings cluster primarily according to keyboard domain, indicating strong device-dependent structure. After applying KSN, keyboard-driven clustering is substantially reduced and embeddings exhibit emergent organization by key identity, while retaining realistic overlap between acoustically similar keys.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Bikrant Bikram Pratap Maurya, Nitin Choudhury, Daksh Agarwal, and Arun Balaji Buduru
Table 7: Segmentation robustness under onset jitter and spurious triggers.
D Received ; revised ; accepted
Condition
No jitter ±10 ms jitter ±20 ms jitter ±30 ms jitter +3% spurious segments +5% spurious segments
Top-1 Key Acc.
Sent. Acc. (raw)
Sent. Acc. (+FLANT5)
81.3% 78.5% 75.4% 69.6% 74.0% 71.2%
42.6% 40.1% 37.9% 33.2% 36.8% 34.7%
62.1% 59.8% 58.1% 52.0% 55.9% 53.4%