Token-Native Storage: Read and Write in your Agent's Language

Token-Native Storage: Read and Write in your Agent’s Language Kumar Shivendu1 1

Qdrant , mail@kshivendu.dev, kumar.shivendu@qdrant.com August 4, 2026

arXiv:2608.02376v1 [cs.DB] 3 Aug 2026

Abstract Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and language-model agents) work in token IDs, not characters, so every access pays to translate between the two. As agents become the primary readers and writers of stored text, we argue for token-native storage: keep the text as the model’s own byte-pair-encoding (BPE) token IDs. This is both smaller and faster. Packing r50k IDs as uint16 already beats UTF-8 by 2.25× on English with no compression, and an entropy coder reaches 3.30×. Across six tokenizers and three corpora (English, code, Hindi), compressing token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers tokens by merge order, not frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the entropy coder’s ratio while decoding ∼7× faster, a one-line change we ask AI labs to make when they publish vocabularies. And because a model reads token IDs, not text, a token-native store hands them over directly instead of re-tokenizing on every read, ∼10–600× faster. The only barrier is that sharing token IDs requires a common tokenizer, which is not always true across model families yet, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.

1

Introduction

Search and database engines store text as UTF-8 bytes,1 a human-facing format. But a retrieval or agentic system rarely reads those bytes as UTF-8: the moment a model touches the text it converts the bytes to token IDs, and converts them back while writing. The infrastructure therefore keeps the same content in two representations and pays for both: storage for the UTF-8 bytes, and a tokenization pass on every read. We argue for a different default for agents: token-native storage. Store the text as the serving model’s own BPE token IDs, the representation every consumer (embedder, reranker, LLM) already needs. Agents are becoming the primary readers and writers of stored text, and will soon account for more of it than humans: retrieval-augmented generation, chat history, and the model’s own training/finetuning data. For them, a UTF-8 store forces a translation on every access, while a token-native store lets them read and write in token IDs directly, at higher compression and throughput (Section 5.4). Translation then happens only at the edges, where the system meets a human. 1

UTF-8 is the near-universal default. The argument holds for any character encoding a database uses (ASCII, Latin-1, UTF-16), and all ratios are measured over UTF-8.

1

Beyond removing that translation cost, a tokenizer that covers the script compresses for free. A BPE token is a learned unit of language, so packing token IDs into fixed-width integers beats UTF-8 before any compression algorithm runs (Section 5). Two further contributions distinguish this work from the off-the-shelf compression it builds on: • BPE token IDs are not frequency-ordered. BPE assigns IDs in merge-discovery order rather than by usage. Re-ranking IDs by corpus frequency lets a variable-length integer codec (streamvbyte) recover most of the entropy coder’s ratio at ∼7× faster decode (Section 4). This motivates a near-free change to how AI labs publish tokenizers: emit IDs in frequency order (Section 7). • The latency win is an agent-workload win. When the reader or writer is a model, tokenization is mandatory. Token-native storage serves the IDs that already exist instead of re-tokenizing on every read, ∼10–600× faster (Section 5.4). Compressing token IDs with off-the-shelf coders is prior art. The nearest is Kalcher (2026), who also reorders IDs by frequency but runs a general compressor over them to squeeze the smallest file. We optimize instead for a live database: +freq keeps only streamvbyte, so reads decode in microseconds, and we argue AI labs should publish the ordering so no user has to re-derive it. Our broadest contribution is the framing itself: token IDs as the read/write form of a model-facing database, with per-item insert, read, and update, not a write-once training file or a compression step. The only limitation is portability: token IDs are shared only when reader and writer use the same tokenizer, and model families do not use a common one yet. We give a database interface for using token-native storage today, and argue that portability across models needs standardization (Section 7).

2

Related Work

Token-native storage sits next to three lines of work. Storing token IDs, but as a static training file. Storing token IDs to avoid recomputing them is already common in training. NVIDIA’s Megatron-Core stores a tokenized corpus as a memorymapped .bin of token IDs with a .idx of document offsets, so training never re-tokenizes (NVIDIA, 2024). But those IDs are a write-once training input, read only by the model. We make token IDs the read/write representation of a database or search engine, where individual items are inserted, read, updated, and deleted on demand. Each item is served either as IDs to a model or detokenized to characters for a human. Removing the two representations, but with lossy ones. A line of work attacks the same waste by making retrieval and generation share one learned representation: continuous memory tokens (He et al., 2025), a single forward pass over both tasks (Zhang et al., 2024), or learned discrete codes (Chen et al., 2025). These are lossy and model-specific. Token-native storage keeps the tokenizer’s own IDs, which decode back to the exact source text a store must return. Compressing token IDs. Compressing token-ID streams is established: neural arithmetic coding (Lester et al., 2024), zstd over packed BPE IDs for stored LLM prompts (Ulla, 2026), and frequency-ordered IDs with a varint before a general compressor (Kalcher, 2026). Kalcher keeps that compressor, so every read must decompress before it reaches token IDs, the cost we remove on the 2

agent path. We benchmark his best variant (zstd-22, since LZMA adds little ratio at far more cost): our entropy coder edges it on natural language, our dictionary on code, and +freq keeps most of its ratio with no compressor and an order of magnitude faster decode. Across all three, tokenization is only a preprocessing step for compression. Positioning. None of these proposes token IDs as the canonical stored form of text that models read and write, with on demand read/write access to individual items, an exactly-invertible vocabulary, and fast decode on read and nearly free write from models.

3

Token-Native Storage

UTF-8 store

token-native store

decompress + tokenize UTF-8 text token IDs on disk ∼235 µs, every read

model

serve directly

token IDs on disk

0.4–25 µs

model

o200k, ∼1.6–3.4× compression vs UTF-8

Figure 1: Every model read from a UTF-8 store pays a decompress-and-tokenize cost (∼235 µs), while a token-native store serves the IDs the model already uses (0.4–25 µs). A UTF-8 store keeps UTF-8, so every read a model performs must decompress and tokenize the text back. Writes mirror this: a model emits token IDs, which a UTF-8 store must first detokenize.

3.1

Step 1: store the token IDs

On write, tokenize the text with the serving model’s BPE tokenizer (Sennrich et al., 2016) and store the integer token-ID sequence. On read, return the IDs to a token-consuming client or detokenize for a human. When the writer is itself a model, the IDs already exist (the sampler produced them). The tokenizer is a collection-level choice, since a shared vocabulary and frequency table make sense for one tokenizer at a time. Text may be read by models with different tokenizers (an embedder, a reranker, an LLM). Store the IDs in the one that reads and writes most, usually the serving LLM. A consumer with a different tokenizer detokenizes and re-tokenizes, exactly as it does with UTF-8 today, so token-native storage is never worse. Full portability is the ecosystem’s job (Section 7). Tokenization compresses before any algorithm runs. Here’s a way to understand it. A BPE token covers about three-quarters of an English word on average (a common rule of thumb) and a word is about six UTF-8 bytes (5 character average + 1 space), so 6 B/word × 34 word/token = 4.5 B/token vs. 2 B/token ⇒ ∼2.25 × . | {z } | | {z } {z } text

uint16 ID

BPE

We use OpenAI’s tiktoken tokenizers (OpenAI, 2022): r50k (50,257 IDs, uint16), cl100k (100,277) and o200k (200,019). The larger two exceed uint16 but fit in 24 bits, so 3-byte packing (not uint32) is the practical raw representation. Even Gemma’s 256,000-token vocabulary uses only 18 of 24 bits. 3

3.2

Step 2 (optional): compress the token IDs

A byte codec cannot know that 0x017F is token 383. Two token-ID compressors, both optional and interchangeable: • ANS (asymmetric numeral systems). An entropy coder (Duda, 2013) (via constriction, Bamler, 2022) over the IDs, with a Laplace-smoothed static unigram table trained once on a corpus and shared across documents. A per-document table compresses better but must travel with each document (∼900 bytes per 512-token chunk), erasing the gain. Best ratio, slower decode. • Frequency-remap + streamvbyte (+freq). A lighter option (Section 4): most of ANS’s ratio at a fraction of the decode cost. Both are lossless: the token IDs are recovered exactly, and the tokenizer maps them back to text. Raw packing already compresses on its own (Section 5), so the coder is optional.

4

The frequency-ordering fix

BPE numbers its vocabulary inefficiently. Each new token gets the next integer ID as it is discovered during merging, so IDs reflect merge order, not usage: a common token can sit at ID 40,000 while a rare one sits at ID 400. Variable-length integer codecs (streamvbyte, LEB128) spend fewer bytes on small integers, so this numbering leaves compression on the table for free. Kalcher (2026) also reorders BPE IDs by frequency, but only as a preprocessing step for a general compressor. We instead store the reordered IDs and decode them with streamvbyte alone, skipping the general compressor for a faster decode. The +freq method re-ranks IDs by corpus frequency (most frequent token gets the smallest ID), then packs with streamvbyte, a SIMD varint codec. On English (C4), it reaches 2.60× (median, r50k), between raw packing (2.25×) and ANS (3.30×), while decoding ∼7× faster than ANS (Section 5.4). The re-rank does all the work: streamvbyte on merge-order IDs falls just below raw uint16 packing (2.13× vs 2.25×, since merge-order IDs are large), and only re-ranking by frequency lifts it to 2.60×. The fix generalizes beyond our codec: any downstream integer compressor benefits, at no modeling cost. This motivates a one-line request to tokenizer authors: publish vocabularies with IDs in frequency order (Section 7). Absent that, users can re-rank on their own corpus and beat the vendor’s assigned token IDs.

5

Evaluation

We evaluate on three real, held-out, non-overlapping corpora: English (C4) (Raffel et al., 2020)2 , Python code (codeparrot-clean)3 , and Hindi (Wikipedia)4 . We use 512-token chunks (a standard embedding chunk size), 40 test chunks per domain, with bootstrap confidence intervals. Training (ANS table, zstd dictionary, frequency ranks) uses only each domain’s train split.5 Every method 2

https://huggingface.co/datasets/allenai/c4 https://huggingface.co/datasets/codeparrot/codeparrot-clean 4 wikimedia/wikipedia, config 20231101.hi: https://huggingface.co/datasets/wikimedia/wikipedia 5 The tables generalize and do not overfit. Measured on the train split instead of held-out test, +freq and +ANS match within ∼2%, while +dict scores up to ∼9% higher because its corpus-trained dictionary partially memorizes training substrings. Train performs slightly better on average, so the held-out ratios we report are the conservative ones. 3

4

is lossless. Code and benchmarks are available.6 Figure 2 previews where each method lands on compression ratio and read latency.

Figure 2: Token-native methods form the ratio-latency frontier (English, o200k, the tokenizer GPT-4o-class models use). Read latency to token IDs (log x) against compression ratio, with a line through the token-native Pareto frontier. Every byte codec sits near ∼235 µs because a model must re-tokenize the text on read, while token-native methods return IDs in microseconds. +freq is the fast, high-ratio recommended default.

5.1

Compression ratio

A tokenizer that covers the script compresses for free (Table 1). Raw token-ID packing reaches 2.25× on English and 2.55× on Hindi, ahead of LZ4, gzip, and zstd, with no algorithm running. Used off its script a tokenizer does poorly and can even expand (r50k raw is 0.84× on Hindi), which an entropy coder or a script-appropriate tokenizer repairs. Compressing the token IDs then matches or beats every byte codec (Figure 3). On natural language it wins outright: a static entropy coder over the IDs (+ANS) reaches 3.30× on English and 5.90× on Hindi, past the strongest byte codec zstd –train. Code is the exception, because its heavy repetition is something an order-0 coder cannot exploit. There a corpus-trained dictionary over the packed IDs (+dict, 3.55×) wins instead, though such dictionaries are corpus-specific and ship with the data, unlike the shared tokenizer. The coders trace a ratio/decode-speed frontier (Figure 2) that token-native storage can pick any point on. +freq gives most of ANS’s ratio at the fastest decode: frequency-ordered IDs packed with streamvbyte and no general compressor. Order-0 ratios barely move with chunk size, so the tokennative ratio advantage is clearest at realistic 512-token chunks (Figure 4) while its latency advantage holds at every size. The full grid across tokenizers, corpora, and sizes is in the repository. 6

https://github.com/kshivendu/token-storage

5

Table 1: Median compression ratio vs. raw UTF-8, 512-token chunks, static tables trained on each corpus’s train split and measured on held-out test (C4 English / codeparrot / Hindi Wikipedia). Byte codecs (top) are tokenizer-independent. Token-native methods (bottom) are shown per tokenizer. Kalcher (+zstd) runs a general compressor over the frequency-ordered token IDs. +lz4 and +dict apply LZ4 and a corpus-trained zstd dictionary to the packed token IDs, the token-domain parallels of the byte LZ4 and zstd –train rows, evaluated on held-out test. Bold marks the best per language: +ANS on English and Hindi, +dict on code, where cross-file repetition lets the dictionary help. Method LZ4 gzip -9 zstd -19 brotli q11 zstd –train

English

Code

Hindi

1.27 1.92 1.94 2.57 2.72

1.76 2.46 2.45 2.87 3.47

1.52 2.38 2.42 2.89 4.52

r50k cl100k o200k r50k cl100k o200k r50k cl100k o200k raw +lz4 +freq +ANS Kalcher (+zstd) +dict

5.2

2.25 2.34 2.60 3.30 3.09 3.15

1.55 1.80 2.69 3.37 3.19 2.85

1.59 1.82 2.73 3.40 3.22 2.85

1.07 2.10 1.41 2.58 3.01 3.27

1.49 2.03 2.51 3.19 3.35 3.36

1.49 2.02 2.50 3.16 3.35 3.55

0.84 1.34 1.33 2.97 2.71 4.33

0.86 1.32 2.04 3.48 3.04 4.31

2.55 2.57 4.47 5.90 4.80 4.65

Generality across tokenizers

The effect is not an OpenAI or vocab-size artifact. On English, six modern tokenizers (r50k, cl100k, o200k, Qwen2.5, DeepSeek-V2, Gemma-2) with static ANS land in a tight 3.30–3.40× band. Latency is stable across tokenizers too: +freq reads in ∼4 µs and +ANS in ∼30 µs, with only raw packing sensitive to vocabulary (uint16 reinterpret for r50k, 3-byte unpack for the larger two).

5.3

The tokenizer drives most of the gain

Most of the gain is the tokenizer, not the coder. Feeding a coder token IDs instead of UTF-8 bytes is what lifts the ratio: ANS over raw UTF-8 bytes reaches only 1.76× on English, but over tokens it jumps to 3.30×. LZ-family coders (zstd, +lz4) gain the same way, though less, because BPE already collapses each frequent character sequence into one short token, leaving fewer long repeats in the stream for their substring matching to find.

6

Figure 3: Median compression ratio over UTF-8 by method, one bar per corpus. Token-native methods (colored) use o200k. Per-tokenizer numbers are in Table 1.

Figure 4: Compression ratio vs. chunk size (o200k, English). Order-0 token methods (raw, +ANS, +freq) stay flat because per-token entropy is additive. LZ-family methods (byte codecs, +dict, +lz4) climb by exploiting cross-chunk redundancy: a corpus-trained zstd –train catches +freq only at 4096 tokens, while +ANS and +dict stay on top at every size. The token-native advantage is largest at the realistic 512-token chunk.

5.4

Latency: an agent-workload win

Tokenization is expensive (∼235 µs per fresh 512-token chunk with tiktoken o200k on a single core, against ∼8 µs for LZ4 compress), and it is paid per query: each retrieved chunk is re-tokenized before the model can use it.7 The community is working on tokenizer performance, yet the cost can never reach zero, and it hurts every read. When the primary reader/writer is a model, tokenization is mandatory, so the only question is whether you pay it every time or once. • Write: an LLM emits token IDs already, so token-native storage packs them in microseconds, 7

Timed on distinct chunks with the tokenizer. The cost varies little across tokenizers: within ∼1.2× for r50k, cl100k, and o200k. Detokenization is 2–6× cheaper.

7

whereas any utf-8 store must first detokenize and then compress. • Read : token IDs are returned directly in microseconds, while UTF-8 text must be decompressed and tokenized back for the model (∼235 µs per chunk). Example: A search query returns multiple records: at 100 chunks per query, with a UTF-8 store, the system spends ∼24 ms of serial CPU re-tokenizing text for the model. Table 2: Median agent write / read latency per 512-token chunk (English, µs). Token-native methods (bottom) serve IDs directly, staying in the µs range across all three tokenizers. Byte codecs (top) instead re-tokenize on read (∼237 µs) and detokenize before compressing on write, shown at the o200k basis. Bold marks the fastest token-native write and read. Method

write (µs) read (µs) r50k cl100k o200k r50k cl100k o200k

LZ4 gzip-9 zstd-19 zstd –train brotli-q11

53 76 260 410 2828

raw 0.5 +lz4 2.6 +freq 2.9 +ANS 3.8 77 +dict Kalcher (+zstd) 73

6.0 9.5 3.1 4.2 214 83

238 245 241 240 246 5.8 9.4 3.1 4.2 210 77

0.4 2.1 3.6 25 4.2 27

5.3 7.1 4.1 30 9.5 31

5.0 7.1 4.0 32 9.3 29

+freq is the fast and recommended default: it decodes ∼7× faster than +ANS for about 20% less compression on English. +dict pays more on write, but it’s good with compression for repetitive corpora such as code.

Figure 5: The tokenization cost moves towards human reads and writes (log scale, +freq on the agent path). The panels mirror: an agent reads a token-native store in microseconds but re-tokenizes UTF-8, while a human’s costs flip. An agent works with token IDs, so token-native storage is fast for it and a UTF-8 store slow. But a human works with text, so the costs flip: a token-native store tokenizes what a human writes (∼235 µs) and detokenizes when a human reads (∼51 µs). 8

Token-native storage is therefore the right default when agents dominate the reads and writes, the common case for agentic systems (Figure 5). Agents produce most of the stored text while humans type short instructions, and one agentic search or RAG answer reads hundreds of chunks before responding. The saving can compound across the pipeline. A retrieval stack tokenizes the same text repeatedly: once to embed a document at ingest, again to rerank the candidates, and again for every agent that reads a retrieved chunk. When the database, the embedding model, the reranker, and the agents all speak token IDs, the system tokenizes once as a human submits text and detokenizes once as a human reads the answer. Every internal hop passes IDs through unchanged. The system pays the tokenize/detokenize cost only at its human/frontend interface.

5.5

Downstream effects of using Token IDs instead of UTF-8 characters

Text is one of the heaviest fields in a record: thousands of characters each consuming a byte, while other fields take only a few bytes. That is why using token IDs instead of characters pays off even more. Every copy of that text shrinks by the same factor (snapshots, backups, write-ahead logs, replicas, network), so tokenization also makes everything faster and cheaper. At 1 B documents (1000-word average, each having 5 chars average + 1 space), raw UTF-8 is 6.0 TB, r50k raw 2.7 TB, r50k+ANS 1.8 TB (∼$336/mo saved on SSD8 ). This also makes I/O-bound work 3.3× faster because less data has to be moved.

6

Interface

We need to do a one-time setup for a collection-level tokenizer: PUT /collections/documents/index { "schema": { "text": { "type": "token", "tokenizer": "o200k" } } }

Inserts accept token IDs (could be directly from a model) or a plain string (the engine tokenizes): PUT /collections/documents/points { "points": [{ "id": 123, "vector": [...], "payload": { "text": [1858, 6427, 20272, ...] } }] }

// or a string

Reads ask per field for tokens or text. Default "text" keeps existing clients unchanged. An LLM pipeline asks for "tokens" and skips detokenization: POST /collections/documents/points/search { "vector": [...], "limit": 10, "with_payload": { "text": "tokens" } } // Response: {"results": [{"id": 1, "text": [1858, 6427, 20272, 318, 257, 6997, ...]}]}

7

What the ecosystem needs

Token-native storage is production-ready within a single-model stack, but its full value needs three ecosystem changes. 8

The 4.2 TB saved over UTF-8, at ∼$0.08 per GB-month for general-purpose SSD (AWS EBS gp3). Rates from Napkin Math, https://github.com/sirupsen/napkin-math.

9

1. A standardized, published tokenizer vocabulary (an ASCII/UTF-8 for tokens), so IDs are portable across models and versions rather than a per-vendor, per-version lock-in. Most families open-source their tokenizers (Anthropic and Google’s proprietary models are the notable exceptions). Embedding models have started converging on LLM tokenizers too, for example OpenAI’s text-embedding-3 reusing cl100k and Jina’s embeddings-v4 moving to a BPE tokenizer. 2. Token-ID-native inference APIs that accept and return token IDs directly, so a store hands IDs to the model and takes back the IDs it emits with no detokenize-then-retokenize round trip. Open stacks already do this: vLLM accepts pre-tokenized prompt_token_ids and can return token IDs on its OpenAI-compatible endpoints.9 The closed source AI labs should do the same. 3. Publish IDs in frequency order (Section 4). It costs the AI labs almost nothing to re-rank a vocabulary by frequency after training, and it gives every downstream integer compressor free ratio, at no per-application effort.

8

Limitations

• Shared vocabulary. The IDs are portable only for consumers using the same tokenizer, which model families do not always share (embedding models often use WordPiece or SentencePiece). The fix is standardization, and it has already started (Section 7). • Tokenizer coverage. A tokenizer with poor coverage of a script splits it into many short tokens, so its raw IDs can exceed raw UTF-8 (r50k on Hindi, 0.84×). UTF-8 itself spends multiple bytes per character on such scripts. A tokenizer that covers the script turns that inefficiency into a larger win (o200k, 2.55× raw on Hindi). Even with a mismatched tokenizer, a coder recovers most of the ratio (r50k on Hindi, 1.33× +freq and 2.97× +ANS). • Corpus-specific tables. The ANS table and frequency ranks can degrade in case of high domain mismatch and they can be different across collections. • Integrated deployment. The results here are component-level. The natural next step is an integrated DB+LLM benchmark of end-to-end performance and cost analysis.

9

Conclusion

Models are increasingly becoming the primary users of our infrastructure. In agentic systems, they already read and write more than humans. But they work with token IDs instead of UTF-8 characters. Storing text as the serving model’s BPE token IDs gives free compression and on applying token ID compression, it surpasses all byte-level compression algorithms. It also makes both reads and writes from the model significantly faster. The bottleneck is mainly standardization. Token-native storage already pays off today when the store and its model share one tokenizer. The benefits only grow as the rest of the stack adopts that tokenizer: once embedders, rerankers, and other model families read the same IDs, one stored copy serves them all, with no vendor or model-family lock-in. All it takes is a shared vocabulary, ideally in frequency order, the same kind of agreement that once gave text characters ASCII and UTF-8. 9

Token-ID input (TokensPrompt): https://docs.vllm.ai/en/stable/api/vllm/inputs/. Returning token IDs to avoid retokenization drift: https://vllm.ai/blog/2025-10-22-agent-lightning.

10

References R. Bamler. Understanding entropy coding with asymmetric numeral systems and the constriction library. https://github.com/bamler-lab/constriction, 2022. J. Chen, X. Jiang, Z. Wang, et al. UniSearch: Rethinking Search System with a Unified Generative Architecture. arXiv:2509.06887, 2025. J. Duda. Asymmetric numeral systems. arXiv:1311.2540, 2013. J. He, R. He Bai, S. Williamson, J. Z. Pan, N. Jaitly, and Y. Zhang. CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning. arXiv:2511.18659, 2025. M. Kalcher. Frequency-Ordered Tokenization for Better Text Compression. arXiv:2602.22958, 2026. B. Lester, J. Lee, A. Alemi, et al. Training LLMs over Neurally Compressed Text. arXiv:2404.03626, 2024. NVIDIA. Megatron-Core Data Preparation: the IndexedDataset format. Developer documentation, 2024. https://docs.nvidia.com/megatron-core/. OpenAI. tiktoken: a fast BPE tokenizer. https://github.com/openai/tiktoken, 2022. C. Raffel, N. Shazeer, A. Roberts, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (C4). JMLR, 21(140), 2020. arXiv:1910.10683. R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. In ACL, 2016. arXiv:1508.07909. A. Ulla. LoPace: A Lossless Optimized Prompt Accurate Compression Engine for LLM Applications. arXiv:2602.13266, 2026. J. Zhang, C. Peng, M. Sun, et al. OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs. arXiv:2409.05152, 2024.

11

Related documents

• COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents#1115830
• Semantic Uncertainty Quantification Needs Factual Equivalence#1129572
• Language and Rules#470504
• Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents#263294
• To Save the Semantic View: An Argument for Returning to Suppes' Interpretation#713199