Sema: Semantic Transport for Real-Time Multimodal Agents Jiaying Meng
Bojie Li
Unaffiliated
Pine AI than a human. Two intrinsic properties of agent models create an opportunity to rethink media transport:
arXiv:2604.20940v1 [cs.MM] 22 Apr 2026
Abstract Real-time multimodal agents transport raw audio and screenshots using networking stacks designed for human receivers, which optimize for perceptual fidelity and smooth playout. Yet agent models act as event-driven processors with no inherent sense of physical time, consuming task-relevant semantics rather than reconstructing signals in real time. This fundamental difference shifts the transport goal from the technical problem of signal fidelity (Shannon–Weaver Level A) to the semantic problem of meaning preservation (Level B). This mismatch imposes significant overhead. In visual pipelines, screenshot upload accounts for over 60% of end-to-end action latency on constrained uplinks, and in voice pipelines, conventional transport carries massive redundancy, sending 43–64× more data than needed to maintain task accuracy. We present Sema, a semantic transport system that combines discrete audio tokenizers with a hybrid screen representation (lossless accessibility-tree or OCR text, plus compact visual tokens) and bursty token delivery that eliminates jitter buffers. In simulations under emulated WAN conditions, Sema reduces uplink bandwidth by 64× for audio and 130–210× for screenshots while preserving task accuracy within 0.7 percentage points of the raw baseline.
1
• Semantic Requirements. Unlike human senses that require rich perceptual signals, agent models effectively process compact, discrete semantic information represented as tokens. Recent work demonstrates this by using such discrete representations internally: for audio, end-to-end speech models like CosyVoice [12] and Qwen3-Omni [48] use audio tokens; for vision, specialized visual tokenizers such as FlexTok [2] and structured screen representations via accessibility trees [15]. These token representations act as information bottlenecks [38], retaining only task-relevant semantics while discarding perceptual redundancy. This creates an opportunity to move the generation of these representations to the client, transforming them from a model-internal optimization into a transport-layer optimization that significantly reduces uplink payloads. • Event-time Tolerance. Unlike human perception that demands continuous playout, agent components process data as ordered event sequences, independent of wallclock playback timing. This decoupling applies across modalities. For vision, VLMs process screenshots as discrete events at action steps. For audio, on the user-toagent path, LLMs process tokens in bursts without requiring isochronous audio streams; on the agent-to-user path, TTS systems generate speech in discrete batches, naturally buffering playback. This tolerance allows us to abandon the strict jitter-buffering mechanisms required for human listening, simplifying the transport stack.
Introduction
AI agents are increasingly multimodal, maintaining persistent connections that transport rich media streams to the cloud. Whether capturing continuous audio for voice interaction (ChatGPT [24], Doubao [4], Limitless [19], Plaud [29], Looki [23]) or streaming screenshots for computer use (Atlas [25], Comet [28], Doubao Phone Assistant [5], OpenClaw [26]), these agents rely on networking stacks designed to transport raw audio and images between the user’s device and LLM servers on the cloud. These networking stacks inherit two core assumptions from human-facing real-time communication (RTC) systems. First, assuming high sensitivity to signal distortion, these stacks employ perceptual codecs like Opus [39] for audio and WebP for images to optimize for high-fidelity reconstruction. Second, assuming intolerance of timing irregularities, they mandate jitter buffers and playout schedulers to ensure continuous media playout by smoothing network variations. However, neither of these assumptions hold when the receiver is an agent model (the AI agent’s backend LLM) rather
The mismatch between current networking stacks and these intrinsic properties imposes significant overhead. In controlled measurements (Sec. 2.1), we find that screenshot upload alone accounts for over 60% of end-to-end action latency in computer-use pipelines at 5 Mbps uplink. This inefficiency stems from solving the wrong problem: conventional transport optimizes for faithful signal reproduction (Shannon–Weaver Level A), whereas agent models only require meaning preservation (Level B). A shift to semantic transport unlocks massive gains: by replacing perceptual codecs with compact semantic representations, we can reduce uplink bytes by ∼130–210× for screenshots and ∼64× for voice, all while preserving downstream task accuracy. To realize this semantic transport paradigm in practice, we present Sema, a semantic transport system that exploits 1
Meng and Li Raw Compressed 10
Sema
6
105 104 103 10
2
Audio (3s)
Screenshot (1080p)
Screenshot Latency (ms)
Uplink Bytes per Turn
both semantic requirements and event-time tolerance. On the uplink, Sema replaces raw media with client-side semantic tokenization. For vision, it employs a hybrid screen representation that combines lossless structured text (via accessibility trees [15]) with compact visual tokens [2, 30], yielding ∼3–5 KB per screenshot. For audio, it uses a discrete speech tokenizer [52] to produce 50–75 token IDs/s (∼500– 750 bps). On the server-side, these multimodal inputs are reconstructed for the downstream model. On the downlink, Sema relocates the vocoder from server to client. In summary, we make three contributions:
Raw+Compress (WebP) Sema (encode + transfer)
6000 5000 4000 3000 2000 1000 0
1
2
5
10 20
50 100
Uplink Bandwidth (Mbps)
Figure 1: Per-turn uplink Figure 2: Screenshot bytes (log scale). latency vs. uplink bandwidth. This massive compression preserves downstream ASR accuracy because the first RVQ layer retains linguistic content [9], effectively isolating semantics from acoustic details.
(1) We characterize the bandwidth and latency overhead of raw-media transport in multimodal agent pipelines, quantifying the gap between perceptual rate-distortion and task-oriented rate-distortion [34], which defines the minimum bit rate needed at a given quality target. (2) We design Sema, a transport system built on two principles for model-centric receivers: (a) semantic requirements, employing client-side hybrid tokenizers to minimize uplink payloads; and (b) event-time tolerance, employing bursty token delivery to eliminate jitter buffers. (3) We evaluate Sema via WAN simulation, demonstrating 130–210× uplink reduction for screenshots and 64× for audio, while maintaining task accuracy within 0.7 percentage points of the raw baseline.
2.2
Why Semantic Transport Works
The magnitude of these reductions stems not from more aggressive lossy compression but from optimizing a fundamentally different distortion metric: task accuracy rather than signal reconstruction. Redefining Fidelity: Task Accuracy vs. Perceptual Quality. Perceptual codecs (Opus, WebP) minimize distortion as perceived by human senses, but agent models have fundamentally different needs. An ASR or multimodal model for voice interaction needs linguistic tokens (phonemes, words, intent), not waveform quality; a VLM for computer-use tasks needs spatial layout and text content, not pixel-level texture. By compressing media into tokenized representations that preserve only what the downstream model uses, while discarding perceptual details irrelevant to the agent task, we achieve the 64–210× bandwidth ratios shown in Figures 1–2 without sacrificing task accuracy. Moreover, because compression efficiency correlates with model capability [11, 17], the semantic capacity of a fixed physical link grows as tokenizer models improve, which is a scaling property no conventional codec can match. Decoupling Time: Event Sequences vs. Continuous Playout. Human-facing RTC stacks deliver audio continuously at playout rate and use jitter buffers to smooth timing variation, machinery that exists solely because human perception is intolerant of discontinuities. Agent pipelines invert this: AI models are processing-time consumers [1] that consume ordered token sequences with no internal clock, so delivery jitter produces no perceptual artifact at the receiver. On the user-to-agent path, ASR and LLM decoders recover meaning from context even when tokens arrive in bursts. On the agent-to-user path, TTS models generate speech in discrete batches of several seconds [48], creating a natural playout cushion. Neither direction requires dedicated jitter-buffering machinery [46]. We quantify this tolerance in §4.
2 Background and Motivation 2.1 The Transport Bottleneck Current multimodal agent pipelines rely on networking stacks that transport raw or perceptually compressed media streams. We quantify the cost of this design by measuring the uplink overhead for the two primary modalities: visual streams and audio streams. Visual Streams. In visual-centric tasks like computer-use, the agent captures a 1080p screenshot at each action step and uploads it as a WebP quality-80 image to a remote VLM. Figure 1 compares per-turn uplink bytes on a log scale: raw PNG requires ∼950 KB and WebP ∼700 KB, whereas Sema’s hybrid semantic representation reduces this to ∼3–5 KB, a ∼130– 210× reduction. Figure 2 translates this gap into latency. At 5 Mbps uplink, the WebP upload takes ∼1.1 s, accounting for over 60% of end-to-end action latency. With semantic transport, the combined encode-and-transfer time stays below 100 ms even at 1 Mbps, effectively eliminating the upload bottleneck. Audio Streams. In voice interaction tasks, speech is typically encoded with Opus at 32 kbps. In contrast, a semantic layer using SpeechTokenizer [52] produces just 50 discrete tokens/s (∼62.5 B/s), achieving a ∼64× reduction (Figure 1). 2
Sema : Semantic Transport for Real-Time Multimodal Agents
Audio
Screenshot
Audio
Opus
WebP
Speech Tokenizer
Audio
Vision
Audio
Screenshot
Strcutured Text Stream
Visual Tokenizer
Vision
Payload Data
Payload Data
Uplink: Client-Side
Uplink: Client-Side Tokenizers
Upload
~12-700 KB/turn
Upload
Vocoder Actions Audio Wave
~2-5KB/turn
Payload Data
Payload Data Model ASR/VLM/LLM+TTS
Lightweight Vocoder
CNN Decoder
Audio
Vision
Model: ASR/VLM/LLM+TTS Audio Tokens
Audio
Vision
Server-Side
Actions
Server-Side Reconstruction Download
Download
~32kbps
Audio Wave
Actions
~500-700bps
Audio Tokens
Downlink: Client-Side
1500×1280 embedding (∼7.3 MB in fp32). Transmitting the model’s internal embeddings would therefore increase network cost, not decrease it. Sema instead interposes a discrete tokenizer on the client and a matching reconstruction engine on the server: the tokenizer maps media to a handful of compact codebook indices (a few hundred bytes per 1024 px tile, ∼62 B/s for speech), which are transmitted and then decoded back into a form the model’s native encoder can accept. This design keeps the model’s own encoder intact while shrinking the network payload by orders of magnitude. Figure 3 shows the Sema architecture. On the uplink (user to agent), client-side tokenizers (§3.1) compress audio and screenshots into compact token sequences. The server-side reconstructor (§3.2) converts these tokens back into the format each downstream model expects. On the downlink (agent to user, §3.3), the model produces speech tokens that a lightweight client-side vocoder decodes into audio. Both directions share a single lightweight framing protocol: each frame carries a header (modality tag, codebook ID, token count, sequence number, timestamp) followed by bit-packed codebook indices or CUP-style [15] compact text, and the receiver demultiplexes by modality tag. Model Compatibility. Voice pipelines span cascaded (ASR, LLM, TTS), end-to-end speech-to-speech [10], and omnimodal (Qwen3-Omni [48], Step-Audio [36]) designs; computeruse pipelines send screenshots to a VLM [6, 16]. Sema applies at two tiers: (1) semantic transport, with client-side tokenization and server-side reconstruction, works with any model that accepts images and text (all open-weight VLMs, Whisper [31], and proprietary APIs); (2) transport optimization, using compact payloads and batch-mode delivery without jitter buffers, is model-agnostic and benefits any pipeline that produces or consumes media in discrete batches. As tokenizers and downstream models evolve, Sema swaps components without protocol changes: the frame header’s codebook ID negotiates the active tokenizer per session.
Actions
Vocoder Audio Wave Audio
Downlink: Client-Side Synthesis
Traditional
SEMA
Figure 3: Architectural comparison. (a) Traditional pipelines send perceptually coded media over the network (∼12–700 KB per turn) on both paths. (b) Sema tokenizes on the client (§3.1), reconstructs on the server (§3.2), and decodes speech tokens via a client-side vocoder on the downlink (§3.3). Both directions share a lightweight token framing protocol.
3
Sema Design
Why tokenize, not encode? A common misconception is that multimodal large language models consume all modalities as discrete tokens. In practice, this is true only for text. For images, models such as Qwen2.5-VL [44] and LLaVA [20] pass raw pixels through a Vision Transformer (ViT) encoder that produces a sequence of continuous embeddings, one per image patch; for audio, models such as Whisper [31] use a mel-spectrogram encoder. These encoders are not tokenizers: their output is a high-dimensional continuous embedding, not a compact codebook index. Critically, the embedding is typically larger than the raw input, for example, a ViT-L encoder maps a 1024×1024 image (∼3 MB raw) to 1024 patch embeddings of dimension 1024 (∼4 MB in fp32), and a Whisper encoder maps 3 s of 16 kHz audio (∼96 KB PCM) to a
3.1
Uplink: Client-Side Tokenization
Audio. The audio tokenizer runs a discrete speech tokenizer (SpeechTokenizer [52] or the first RVQ layer of EnCodec [9]) on the client. It produces 50–75 discrete token IDs per second from a codebook of size 1024, yielding a bandwidth of 500– 750 bps compared to Opus’s 32 kbps, a 43–64× reduction. Encoding is lightweight: SpeechTokenizer processes 1 s of audio in ∼15–60 ms depending on hardware. Only the first RVQ layer is used because agent tasks require linguistic content but not acoustic fidelity. Modern end-to-end models (CosyVoice [12], Step-Audio [36], Qwen3-Omni [48]) use discrete speech tokens internally and can serve as drop-in replacements. 3
Meng and Li
to the downstream model [49, 53]. On the audio path, the server handles two scenarios: for models requiring waveforms, it employs a lightweight vocoder to reconstruct audio from codebook embeddings (∼5–10 ms); for native multimodal models, it forwards discrete tokens directly. On the vision path, the server first decodes visual tokens via a singlepass CNN, which takes about ∼30 ms using Layton [30]. It then overlays Set-of-Marks annotations derived from the structured text stream. This composite input, consisting of an annotated image for spatial context and structured text for precise element identity, is subsequently fed to the VLM. Since the structured text stream ensures accurate content identity, the visual stream only needs to convey spatial context, relaxing the requirement for pixel-perfect reconstruction. Overall, the entire reconstruction process adds only ∼30–35 ms of server-side latency.
Vision (hybrid screen representation). For screenshots, neither a visual tokenizer nor a structured text representation is sufficient alone. Visual tokenizers (Layton [30], FlexTok [2]) produce extremely compact codes, but small labels, button captions, form content, and text details are lost or garbled after decoding [7, 33], making text-heavy tasks such as form filling or reading notifications fail. Conversely, the accessibility tree [15] captures text and element identity perfectly but discards visual layout, so tasks that depend on visual context cannot be solved from text alone. Sema therefore combines the two streams so that each compensates for the other’s weakness. Structured text stream. The client reads the platform accessibility API (macOS Accessibility, Windows UI Automation, Linux AT-SPI, or the web DOM) [15] to obtain element types, labels, coordinates, and states in a compact text format (e.g., [e2] button "Back" @132,52 32x32 [click]). This yields ∼2–5 KB per screen with zero encoding compute, because the accessibility tree is maintained by the OS. When the accessibility tree is unavailable or incomplete (canvas-rendered UIs, games, remote desktop), Sema falls back to lightweight on-device OCR (Apple Vision framework, PaddleOCR; ∼20– 50 ms, ∼1–3 KB). Visual token stream. To capture visual semantics (spatial layout, iconography, colors) beyond the text stream, the client employs a lightweight tokenizer such as Layton [30] (256 tokens per 1024 px tile) or FlexTok [2] (8–128 variablelength tokens), compressing each tile to ∼200–500 B at ∼40 ms per tile on consumer desktop GPU and ∼30–150 ms on mobile via CoreML [8]. Non-square resolutions (e.g., 1080p, 720p) are handled via tiling: each screenshot is split into 1024 px square regions matching the tokenizer’s native input, following the patch strategy used by production vision APIs; a 1080p screenshot yields 2 tiles (∼800 B total) encoded in a single batched pass, and the structured-text stream carries coordinates in the original pixel space so tile boundaries do not affect grounding. Combined payload. By fusing these streams, Sema reduces the per-screenshot payload to approximately ∼3–5 KB, which is orders of magnitude smaller than a standard ∼700 KB WebP image. This massive reduction preserves utility because the structured text provides lossless element identity while visual tokens supply necessary spatial context. Our evaluation in §4.4 confirms this synergy. Visual tokens alone yield only 75.5% accuracy on text-heavy tasks, but adding the text stream restores performance to 93.3%, within 0.7 pp of the raw baseline (94.0%).
3.2
3.3
Downlink: Client-Side Synthesis
The downlink carries two types of model output: audio for voice pipelines and action commands (e.g., click coordinates, typed text) for computer-use pipelines. Action commands are already compact text (tens of bytes per action), so they need no special optimization in either pipeline. For audio, the key change is relocating the vocoder from server to client. In a traditional pipeline, the server’s TTS module generates speech tokens, a server-side vocoder (e.g., HiFi-GAN [18]) converts them to a waveform, and the waveform is encoded with Opus (∼32 kbps) for delivery. Sema skips server-side vocoding and Opus encoding: the TTS module, either diffusion-based or multi-codebook autoregressive [48], produces discrete speech tokens in batches of 3–5 s and sends them directly to the client (∼500–750 bps). A lightweight client-side vocoder then decodes the tokens into audio locally. Because each batch amounts to only a few hundred bytes delivered in a single burst every few seconds, no dedicated jitter buffer is needed.
4 Evaluation 4.1 Setup We conduct preliminary evaluation of Sema through simulation: each pipeline component (tokenizer, network transfer, server reconstruction) is modeled using measured percomponent latencies and payload sizes, composed under emulated network conditions. An end-to-end implementation is future work. Tokenizers. For audio we use SpeechTokenizer [52] (first RVQ layer, 50 tokens/s, codebook 1024); CosyVoice [12] is a drop-in alternative at 25 Hz. For vision we use Layton [30] (256 tokens per 1024×1024 image with a 1–2 step consistency decoder); FlexTok [2] (variable 8–128 tokens) is cited as an alternative that adapts the token budget to image complexity.
Server-Side Reconstruction
The Sema server accepts hybrid token payloads and reconstructs multimodal input before feeding both image and text 4
Sema-Adaptive
Uplink Bandwidth (Mbps) 6000
5000 4000 3000 2000
5
100 20
Model Server Transfer Encode
5000 4000 3000 2000 1000
1000 5
10
20
50
100
R+C S-H S-A
2
R+C S-H S-A
0 1
R+C S-H S-A
0
1
Task Accuracy (%)
Raw+Compress Sema-Hybrid
6000
Latency (ms)
Pre-inference Latency (ms)
Sema : Semantic Transport for Real-Time Multimodal Agents
95 90
Acceptable threshold
85 80
Voice Navigation Vision-Text Sema-Adaptive (Text)
75 70 102
Uplink Bandwidth (Mbps)
103
104
105
106
Uplink Bytes per Turn
Figure 4: Pre-inference latency (en- Figure 5: Latency breakdown by Figure 6: Rate–accuracy tradeoff code + transfer + server decode, ex- pipeline stage at three uplink band- across methods and modalities (log cluding constant model inference) vs. widths. x-axis). uplink bandwidth (RTT = 50 ms). Table 1: Per-turn uplink bytes (median). Compression ratios are relative to Raw+Compress.
Models. We use open-weight models: Whisper-large-v3 [31] for ASR (via vocoder-reconstructed audio) and Qwen2.5-VL7B [44] for computer-use action prediction (via reconstructed annotated images + structured text). Workloads and metrics. (1) Voice: 200 turns from LibriSpeech test-clean [27], simulated turn-taking; metric is ASR word-error rate (WER). (2) Vision-navigation: 100 webbrowsing tasks from the navigation subset of OSWorld [6], targeting layout actions (clicking buttons, navigating menus); metric is exact-match action accuracy against ground-truth click/type events. (3) Vision-text: 50 tasks requiring reading small on-screen text or filling forms, drawn from the productivity subset of OSWorld; metric is task success rate (correct field values submitted). Baselines. (a) Raw: WebRTC for audio, HTTP upload for PNG screenshots. (b) Raw+Compress: Opus at 32 kbps for audio, WebP quality-80 for 1080p screenshots (∼700 KB). (c) Sema-Static: always uses semantic transport (visual tokens only, no structured text). (d) Sema-Hybrid: structured text (accessibility tree or OCR) + visual tokens for screenshots, discrete tokens for audio.
4.2
Method
Raw (PCM / PNG) Raw+Compress Sema-Static (tokens) Sema-Hybrid
96 KB 12 KB 188 B 188 B
Ratio
950 KB 700 KB 1× (ref) 832 B 64× / 841× 3–5 KB 64× / 130–210×
1 Mbps the gap widens: Raw+Compress takes ∼5.6 s while Sema stays below 100 ms. The breakdown (Figure 5) confirms that network transfer dominates Raw+Compress at low bandwidth, while Sema shifts the bottleneck to a combined ∼70–100 ms encode/decode cost. Beyond ∼80 Mbps, raw upload becomes fast enough that the encoding overhead makes Sema’s advantage marginal. For voice, the latency gap is narrower because audio payloads are already small. A 3 s Opus turn (12 KB) transfers in ∼96 ms at 1 Mbps; Sema’s 188 B payload transfers in <2 ms, but adds ∼45–180 ms client encode and ∼8 ms server vocoder decode, yielding a net saving only below ∼2 Mbps. At constrained uplinks (mobile, IoT), the 64× bandwidth reduction still translates to meaningful latency and capacity gains.
Bandwidth Reduction
Table 1 reports per-turn uplink bytes. Discrete audio tokens achieve a 64× reduction over Opus. Sema-Static achieves ∼840× over WebP for screenshots (2 tiles at 1080p); SemaHybrid trades some of that ratio (130–210×) for lossless text fidelity via structured text. These values report applicationlayer payload bytes; the shared token-frame header adds only 17 B per frame.
4.3
Audio Screen(3 s turn) shot
4.4
Rate–Accuracy Tradeoff
Figure 6 plots uplink bytes per turn against downstream task accuracy. The Voice, Navigation, and Vision-Text series each contain raw, compressed, and semantic transport variants. For voice and navigation, Sema points fall in the Pareto-optimal region: near-baseline accuracy at orders-ofmagnitude lower bandwidth. For vision-text, the visual-tokensonly point (∼832 B) drops to 75.5%, but adding lossless structured text (∼5 KB) recovers accuracy to 93.3%, within 0.7 pp of the 94.0% raw baseline. Figure 7 breaks down accuracy by category. On voice, SpeechTokenizer increases WER from 2.7% to 4.1%, a modest degradation acceptable for conversational agents. On navigation, action accuracy is within 2 pp of raw. Sema-Static
Latency Improvement
Figures 4 and 5 show end-to-end latency for visual navigation tasks across 1–100 Mbps uplink. At 5 Mbps, Raw+Compress takes ∼1.1 s per 1080p screenshot. Sema-Hybrid with an accessibility tree totals ∼75 ms (encode ∼40 ms + transfer ∼5 ms + server decode ∼30 ms); with OCR fallback, ∼105 ms. At 5
Meng and Li
95 90 85 80 75
6
5
5
4
4
3
WER MOS
3
2
2 MOS < 3.0 (unacceptable)
70 65 60 Voice
Navigation
Text
1
1
0
0 0
100
200
300
400
Stream RTC (no buf) Stream RTC (40ms buf) Batch TTS 3s Batch TTS 5s
Gap Rate (%)
Sema-Hybrid Sema-Adaptive
MOS (1-5)
Raw+Compress Sema-Static
WER (%)
Task Accuracy (%)
100
50 40 30 20 10
500
Delivery Jitter Std. Dev. (ms)
0
0 100 200
500
750
1000
Delivery Jitter Std. Dev. (ms)
Figure 7: Task accuracy by workload Figure 8: Uplink jitter tolerance: Figure 9: Downlink gap rate vs. delivcategory (95% CIs). WER (left) vs. MOS (right). ery jitter. drops 12–18 pp on text-heavy tasks [7, 33]; Sema-Hybrid recovers near-baseline accuracy via lossless structured text.
4.5
image and structured text, compatible with any VLM (openweight or proprietary API). For audio, models that natively accept discrete tokens can bypass reconstruction entirely. Scaling with model progress. Conventional codecs are static engineering artifacts whose efficiency is bounded by fixed source-coding limits. Semantic transport inherits ML scaling laws: as tokenizers improve, the transport layer compresses further automatically. FlexTok [2] already achieves FID < 2 with 8–128 tokens; Layton [30] reconstructs 1024 px images from 256 tokens; structured screen parsers (CUP [15]) continue to advance. Each gain in model-side representation directly reduces network cost. From model compression to transport compression. The ML community has built increasingly compact representations of multimodal data, including accessibility trees [15], discrete speech tokens [12, 36], and compact visual tokens [2, 30]. Sema’s contribution is not a new tokenizer; it is relocating existing model-side compression to the client and designing the systems infrastructure to make this practical for real-time Internet transport. Future work. Our evaluation is simulation-based; an end-toend prototype is needed to validate compute budgets on heterogeneous clients, tail latency under real networks, and loss resilience beyond the structured-text fallback (FEC across RVQ layers, selective retransmission remain open). Baselines use standard WebP/Opus; RoI-WebP or HEVC screen-content extensions could narrow the margin but still target pixellevel distortion, leaving the orders-of-magnitude gap intact.
Event-Time Tolerance
Section 2.2 argues that agent models consume ordered event sequences with no internal clock, so delivery jitter should not degrade task performance. For vision this holds trivially: VLMs process each screenshot as a discrete event at each action step, so inter-frame timing variation has no effect on model input. Audio is the more demanding case because it is inherently temporal; we validate both directions below. Uplink (user to agent). Figure 8 compares the effect of delivery jitter on two receivers: an ASR model (WER) and a human listener (MOS via PESQ [32]). At 200 ms jitter, MOS drops from 4.3 to 2.6 (40% degradation) while WER increases only from 2.7% to 3.2% (18% relative). At 500 ms jitter, MOS collapses to 1.8 (below acceptable quality) while WER remains at 3.8% (still usable). The gap confirms that ASR models recover meaning from linguistic context despite signal-level artifacts. Downlink (agent to user). Figure 9 measures the playout gap rate as a function of inter-batch delivery jitter. With 3 s TTS batches, playout remains gap-free up to ∼500 ms of jitter; with 5 s batches, up to ∼1000 ms. By contrast, a traditional RTC system exhibits gaps under as little as 50 ms of jitter when jitter buffers are removed.
5
Discussion
6
Text fidelity and client encoding cost. The hybrid representation eliminates the primary weakness of pure visual tokenization by transmitting text losslessly via the accessibility tree or OCR. DeepSeek-OCR [22] (97% accuracy at 10× compression) and OCR-VQGAN [33] continue to close the remaining gap for pure visual encoders. All client-side encoding costs (§3) are sub-150 ms, keeping the total encodeplus-transfer budget well within interactive latency targets. Model compatibility. Sema’s reconstruct-then-feed approach is model-agnostic: the server produces a standard annotated
Related Work
AI-oriented RTC. Wu et al. [46] share our vision, proposing Context-Aware Video Streaming (CLIP-guided H.265 RoI, no jitter buffers), but dismiss client-side token streaming as infeasible, arguing MLLMs require continuous embeddings and discrete tokens incur prohibitive quantization loss. Sema is precisely that demonstration: a reconstruct-then-feed server sends discrete tokens on the wire yet decodes them back to continuous pixels before the VLM’s native ViT, leaving the model interface untouched; residual loss is absorbed by a lossless accessibility-tree stream which is critical for text 6
Sema : Semantic Transport for Real-Time Multimodal Agents
understanding. Sema reaches 93.3% vs. 94.0% raw on the hardest text-heavy workload (§4.4). Semantic communication. DeepSC [47] introduced deep semantic coding for text; subsequent work extended the principle to dynamic visual data [51] and developed codingtheoretic formulations [35]. Token Communications [41] and VLF-MSC [42] apply multimodal tokens to the semantic communication setting. This literature primarily targets wireless links and joint source-channel coding; Sema applies the same principle to Internet transport for agent pipelines: transmit meaning, not pixels or waveforms. Discrete tokenizers. Audio RVQ tokenizers (SoundStream [50], EnCodec [9], SpeechTokenizer [52]) compress speech into discrete codebook indices, enabling Sema to transmit only the first (semantic) layer. End-to-end models (CosyVoice [12], Step-Audio [36], Qwen3-Omni [48]) build on this foundation. Visual tokenization descends from VQ-VAE [40] and VQGAN [13]; recent work (FlexTok [2], Layton [30], OCRVQGAN [33]) dramatically improves fidelity at low token counts. Conventional codecs (HEVC [37], screen-content extensions [21], Salsify [14], learned compression [3]) all optimize for pixel-level reconstruction. Sema shows that discrete tokenizers developed for model efficiency also serve as transport-layer compressors when computed on the client. Agent workloads and screen representations. WebVoyager [16] and OSWorld [6] benchmark multimodal agents on GUIs. OmniParser [53], Set-of-Marks [49], GUI-Actor [45], and AgentOCR [43] develop compact screen representations that reduce model context and improve grounding. CUP [15] standardizes accessibility-tree encoding across platforms. Sema extends these techniques from a model-efficiency role to a transport-layer role by computing them on the client.
7
Conclusion
When the receiver is a model rather than a human, two transport axioms change: perceptual fidelity gives way to task-relevant meaning, and wall-clock continuous delivery gives way to event-time bursty delivery. Sema acts on both: it relocates semantic tokenization to the client and delivers tokens in bursts rather than streams. In simulations, Sema reduces uplink bandwidth by 64× for audio and 130–210× for screenshots while maintaining task accuracy within 0.7 percentage points of the raw baseline.
References [1] Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, and Sam Whittle. 2015. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. Proceedings of the VLDB Endowment 8, 12 (2015), 1792– 1803. 7
[2] Roman Bachmann et al. 2025. FlexTok: Resampling Images into 1D Token Sequences of Flexible Length. In International Conference on Machine Learning (ICML). Variable 1–256 tokens per image, FID < 2 at 8–128 tokens on ImageNet, rectified flow decoder. [3] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational Image Compression with a Scale Hyperprior. In International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=rkcQFMZRb [4] ByteDance. 2024. Doubao App: Real-Time Voice Assistant. https: //www.doubao.com/. 159M MAU; end-to-end speech-to-speech with interrupt support. [5] ByteDance. 2025. Doubao Phone Assistant. https://o.doubao.com/. OS-level mobile agent with GUI simulation, cross-app automation, and on-device memory; launched on Nubia M153. [6] Ruisheng Cao, Jixuan Chen, Zhoujun Cheng, Toh Hua, Fangyu Lei, Xiaochuan Li, Yitao Liu, Silvio Savarese, Dongchan Shin, Tianbao Xie, Caiming Xiong, Yiheng Xu, Tao Yu, Danyang Zhang, Siheng Zhao, Victor Zhong, and Shuyan Zhou. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems 37. 52040–52094. doi:10.52202/079017-1650 [7] Chameleon Team. 2024. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint arXiv:2405.09818 (2024). [8] Pavan Kumar Anasosalu Chen et al. 2025. FastVLM: Efficient Vision Encoding for Vision Language Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Apple. FastViTHD hybrid encoder, 85x faster TTFT, CoreML/MLX checkpoints. [9] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2023. High fidelity neural audio compression. Transactions on Machine Learning Research (2023). [10] Alexandre Défossez, Laurent Music, Thomas Harlé, Jade CopetKanishk Shao, Gabriel Synnaeve-Yossi Adi-Neil Zeghidour-Robin Moras, and Robin San Roman. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037 (2024). [11] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Shane Legg, and Joel Veness. 2024. Language Modeling Is Compression. In International Conference on Learning Representations (ICLR). [12] Zhihao Du et al. 2024. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. arXiv preprint arXiv:2412.10117 (2024). Dual-level RVQ: HuBERT semantic + ECAPA-TDNN acoustic, 25 Hz token rate. [13] Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming Transformers for High-Resolution Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12873–12883. https://openaccess.thecvf.com/ content/CVPR2021/html/Esser_Taming_Transformers_for_HighResolution_Image_Synthesis_CVPR_2021_paper.html [14] Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S. Wahby, and Keith Winstein. 2018. Salsify: Low-Latency Network Video through Tighter Integration between a Video Codec and a Transport Protocol. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). 267–282. https://www.usenix. org/conference/nsdi18/presentation/fouladi [15] Piotr Grzegorzek et al. 2026. Computer Use Protocol. https://github. com/computeruseprotocol/computeruseprotocol. 59 ARIA-derived roles, 15 canonical verbs, compact text encoding with ∼97% token reduction vs JSON.
Meng and Li
[16] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 6864–6890. doi:10.18653/v1/ 2024.acl-long.371 [17] Yuzhen Huang, Jinghan Li, Bangzheng Chen, Shijie Xiao, Zefan Wang, Shuai Feng, Bo Li, Xin Xie, Deqiang Ye, and Wenhan Zhang. 2024. Compression Represents Intelligence Linearly. arXiv preprint arXiv:2404.09937 (2024). [18] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. Advances in Neural Information Processing Systems 33 (2020), 17022–17033. [19] Limitless AI. 2024. Limitless AI Pendant. https://www.limitless.ai/. [20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems 36. [21] Shan Liu, Xiaozhong Xu, Shawmin Lei, and Kevin Jou. 2015. Overview of HEVC Extensions on Screen Content Coding. APSIPA Transactions on Signal and Information Processing 4 (2015), e10. doi:10.1017/ATSIP. 2015.11 [22] Yutao Liu et al. 2025. DeepSeek-OCR: Contexts Optical Compression. arXiv preprint arXiv:2510.18234 (2025). DeepEncoder (380M params), 97% OCR accuracy at 10x compression, 100 tokens/page. [23] Looki. 2025. Looki Multimodal AI Wearable. https://www.looki.ai/. [24] OpenAI. 2024. ChatGPT Advanced Voice Mode. https://openai.com/ index/chatgpt-can-now-see-hear-and-speak/. [25] OpenAI. 2025. ChatGPT Atlas. https://chatgpt.com/atlas/. [26] OpenClaw. 2025. OpenClaw: Persistent Client-Side Computer Use Agent. https://github.com/openclaw. [27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5206–5210. [28] Perplexity. 2025. Comet Browser: a Personal AI Assistant. https: //www.perplexity.ai/comet. [29] Plaud. 2024. Plaud NotePin: AI Wearable for Voice. https://www.plaud. ai/. [30] Yiming Qu, Sijie Li, Kecheng Yan, Jiaqi Wu, Kai Li, Jie Du, Shurun Xie, Xiu Luo, Jian Nie, et al. 2025. Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens. arXiv preprint arXiv:2503.08377 (2025). [31] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. International Conference on Machine Learning (ICML) (2023). [32] Antony W. Rix, John G. Beerends, Michael P. Hollier, and Andries P. Hekstra. 2001. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2 (2001), 749–752. [33] Juan A. Rodriguez et al. 2022. OCR-VQGAN: Taming Text-withinImage Generation. arXiv preprint arXiv:2210.11248 (2022). [34] Claude E. Shannon. 1959. Coding theorems for a discrete source with a fidelity criterion. IRE National Convention Record 7, 4 (1959), 142–163. [35] Yulin Shao, Qi Cao, and Deniz Gündüz. 2024. A Theory of Semantic Communication. IEEE Transactions on Mobile Computing 23, 12 (2024), 12211–12228. doi:10.1109/TMC.2024.3406375 [36] Step-Audio Team. 2025. Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction. arXiv preprint arXiv:2502.11946
(2025). 130B-param, dual tokenization: Paraformer (16.7 Hz) + CosyVoice (25 Hz). [37] Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (2012), 1649–1668. doi:10.1109/TCSVT.2012.2221191 [38] Naftali Tishby, Fernando C. Pereira, and William Bialek. 1999. The information bottleneck method. arXiv preprint physics/0004057 (1999). [39] Jean-Marc Valin, Koen Vos, and Timothy B. Terriberry. 2012. Definition of the Opus Audio Codec. In RFC 6716, IETF. [40] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems 30. https://papers.nips.cc/paper/7210neural-discrete-representation-learning [41] Various. 2025. Token Communications: A Unified Framework for Cross-modal Context-aware Semantic Communications. arXiv preprint arXiv:2502.12096 (2025). [42] Various. 2025. VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System. arXiv preprint arXiv:2511.10074 (2025). [43] Various. 2026. AgentOCR: Reimagining Agent History via Optical Self-Compression. arXiv preprint arXiv:2601.04786 (2026). 50%+ token reduction, 95% task performance, 20x rendering speedup. [44] Peng Wang et al. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025). 600M-param custom ViT, Naive Dynamic Resolution, Window Attention, M-RoPE. [45] Yue Wang et al. 2025. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents. arXiv preprint arXiv:2506.03143 (2025). [46] Jiangkai Wu, Zhiyuan Ren, Liming Liu, and Xinggong Zhang. 2025. Chat with AI: The Surprising Turn of Real-Time Video Communication from Human to AI. In Proceedings of the 24th ACM Workshop on Hot Topics in Networks (HotNets). doi:10.1145/3772356.3772390 [47] Huiqiang Xie, Zhijin Qin, Geoffrey Ye Li, and Biing-Hwang Juang. 2021. Deep Learning Enabled Semantic Communication Systems. IEEE Transactions on Signal Processing 69 (2021), 2663–2675. doi:10. 1109/TSP.2021.3071210 [48] Jinze Xu et al. 2025. Qwen3-Omni Technical Report. arXiv preprint arXiv:2509.17765 (2025). Thinker-Talker MoE, Audio Transformer encoder, multi-codebook speech tokens, 211ms first-packet latency. [49] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2024. Set-of-Mark Visual Prompting for GPT-4V. In European Conference on Computer Vision (ECCV). [50] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022. SoundStream: An End-to-End Neural Audio Codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022), 495–507. doi:10.1109/TASLP.2021.3129994 [51] Hongwei Zhang, Shuo Shao, Meixia Tao, Xiaoyan Bi, and Khaled B. Letaief. 2023. Deep Learning-Enabled Semantic Communication Systems With Task-Unaware Transmitter and Dynamic Data. IEEE Journal on Selected Areas in Communications 41, 1 (2023), 170–185. doi:10.1109/JSAC.2022.3221991 [52] Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zou, and Xipeng Xie. 2024. SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models. In International Conference on Learning Representations (ICLR). [53] Yadong Zheng, Jianwei Zhang, Xinya Li, and Yu Lu. 2025. OmniParser V2: Turning Any LLM into a Computer Use Agent. arXiv preprint (2025).
8