FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning Ali Sarabadania,∗, Massimo Piccardia , Mehran Abolhasana , Farzad Tofigha , Ian Zhoua , Justin Lipmana a Cyber Digital Centre, University of Technology Sydney,

Abstract

arXiv:2610.22906v1 [cs.CL] 16 Oct 2026

Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such as natural language processing (NLP) and multimodal analysis. For this reason, this paper presents a novel small language model that leverages byte-level text and multimodal data processing, enhanced by a linear attention mechanism, to enable manageable Federated Learning of NLP and multimodal analysis tasks. The proposed model, nicknamed FederatedBytes, manages to process text and any other data types, including images, by treating its input as a mere sequence of bytes, while its lean linear attention mechanism allows abating the overall memory and computational footprint. To prove the effectiveness of the proposed model we have carried out an extensive performance evaluation on sentiment analysis tasks in multilingual and multimodal settings over the Multilingual Amazon Reviews and ScienceQA datasets. The experimental results show that FederatedBytes has been able to outperform comparable models such as ByT5 and LSTM, achieving higher accuracy while significantly reducing memory and communication overheads. Keywords: Federated Learning, small language models, byte-level transformers, tokenizer-free, linear attention, multimodal classification. 1. Introduction Federated learning (FL) is a machine learning approach that allows training global machine learning models from multiple clients in a decentralized environment, while preserving the data privacy of the clients McMahan et al. (2017). This makes FL ideal for multi-client settings where both a global model and data privacy are required, such as in digital agriculture, healthcare, mobile communications, electric vehicles and many other use cases where data is sensitive, network bandwidth and computational resources are limited, and data sources are inherently distributed. For instance, a collaborative agricultural project might require training a global model for optimal harvesting by combining data from multiple farms, each with potentially different languages, data types (e.g., text reports, images and sensor readings), and low-resource edge devices. Among contemporary AI workloads, natural language processing (NLP) plays a special role for its ability to automatically extract useful information from text written in a multitude of languages. The field of NLP has recently been revolutionized by the widespread adoption of language models such as ChatGPT, Gemini and Claude, that have been successfully applied to a range of tasks such as sentiment classification, machine translation, document summarization, question answering and reasoning. ∗ Corresponding author

Email address: ali.sarabadani@student.uts.edu.au (Ali Sarabadani)

However, these models typically rely on a processing step known as tokenization, which entails breaking the input text into predefined units roughly corresponding to individual words or part thereof. Tokenization is a substantial process requiring large, language-specific or multilingual vocabularies, struggling with out-of-vocabulary words, and hindering the use of non-textual inputs such as images and videos. These challenges are particularly acute for edge devices where computational resources are intrinsically limited, and magnified by FL environments with their heavy communication requirements over networks with limited bandwidth. To mitigate vocabulary issues, byte-level language models such as ByT5 Xue et al. (2022) and bGPT Wu et al. (2024) have recently been proposed to operate directly on raw UTF8 bytes, effectively removing the tokenizer and using a fixed vocabulary of only 256 entries. While this improves flexibility, the existing token-free models are not optimized for tight memory and computational constraints of edge devices in a decentralized, multimodal setting. For this reason, in this paper we propose FederatedBytes - a novel byte-level, small language model specifically designed for federated learning. By operating directly on raw UTF-8 bytes, FederatedBytes eliminates the need for tokenization, naturally handling any language or data format that can be represented as a sequence of bytes. In addition, its architecture incorporates a linear attention mechanism, which abates the quadratic complexity of a standard Transformer (the core module of all language models), making it feasible to process long sequences on resource-constrained devices. Our work is motivated by the need for a versatile, effi-

cient, and private model for multi-client industrial applications, where previous models such as ByT5 and bGPT prove inadequate due to their high resource consumption. To this aim, the novelty of FederatedBytes lies in its unique combination of tokenizer-free byte-level processing, a memory-efficient linear attention mechanism, small size, and seamless integration into a privacy-preserving federated learning framework. Overall, this paper makes a threefold contribution:

for tokenization, making them inherently multilingual, robust to out-of-vocabulary words and noise, and multimodal. Recent research by Meta AI into byte-level Transformers Pagnoni et al. (2024) has further validated the scalability and potential of this approach. A critical gap, however, is that these powerful models have been designed and evaluated almost exclusively in centralized training settings, overlooking the challenges of deploying them in a privacy-preserving, federated context. Addressing the computational demands of Transformers, especially on long byte sequences, has been another key research goal. The standard self-attention mechanism has a quadratic time and memory complexity with respect to the input length, which is a major bottleneck. This has led to the development of efficient, linearized attention mechanisms such as Linformer and Performer Katharopoulos et al. (2020), and memory-efficient architectures such as the Longformer Beltagy et al. (2020). These innovations make it possible to efficiently process long inputs, a crucial capability for a byte-level model. For the reader’s reference, Table 1 situates our work with respect to these key research threads, while Table 2 provides a detailed comparison of specific token- and byte-level models.

• We propose FederatedBytes, a byte-level, small language model tailored for federated NLP capable of handling multilingual and multimodal data on edge devices. • We integrate into the model an efficient linear-attention mechanism that significantly lowers memory usage and extends the context window for processing long byte sequences - a critical requirement in many real-world applications. • We perform ample experiments and comparisons on the Multilingual Amazon Reviews and ScienceQA multimodal datasets, showing that FederatedBytes has been able to achieve superior accuracy compared to an established byte-level model such as ByT5 and the LSTM, while substantially reducing training time, communication and computational overheads.

3. The FederatedBytes Architecture Synthesizing the insights from the related work in Tables 1 and 2, we have identified a clear gap: existing frameworks effectively address federated learning, byte-level processing, or efficient attention in isolation, but no single architecture unifies all three. Recent byte-level models are too computationally heavy for federated edge devices, while existing efficient FL models rely on tokenization, sacrificing the robustness of raw byte processing. To bridge this gap, we propose FederatedBytes, a unified framework designed to bring the resilience of byte-level processing to the resource-constrained edge. In this section, we provide a high-level description of the architecture. As summarized in Figure 1, the model operates directly on raw byte sequences and incorporates linear complexity attention mechanisms to handle large inputs efficiently. To align with the logical flow of the architecture, the pipeline is organized into three main sequential stages:

2. Related Work Our work is positioned at the intersection of three active research areas: federated learning (FL) for natural language processing (NLP), byte-level text processing, and the development of efficient Transformer architectures. While significant progress has been made in each domain individually, FederatedBytes is the first model to merge these concepts into a unified framework designed for private, efficient, and multilingual NLP on decentralized data. The main advantage of federated learning is that it aggregates locally-trained models without the need to centralize the raw data, and its baseline aggregation mechanism is Federated Averaging (FedAvg) McMahan et al. (2017). Early largescale deployments such as Gboard’s prediction of the next-word demonstrated the practicality of FL for NLP tasks on real devices Hard et al. (2018). More recently, frameworks such as FedNLP Lin et al. (2021) have standardised comparisons of FL optimisers on Transformer-based tasks, highlighting the challenges that data heterogeneity poses for attention-based models such as Transformers. This has spurred research into personalised variants, such as FedTP Zhang et al. (2022), which adapts self-attention mechanisms in each client to counter the drift caused by non-IID (i.e., non independent and identicallydistributed) data. However, these frameworks predominantly operate on tokenized text, inheriting the limitations of predefined vocabularies. In parallel, byte-level models have emerged as a powerful alternative to token-based NLP approaches. By operating directly on raw byte sequences, models such as ByT5 Xue et al. (2022) and byte-level GPT (bGPT) Wu et al. (2024) eliminate the need

• Stage 1 (Input Processing: Patching and Hierarchical Encoding): Conversion of multilingual text into UTF8 byte streams, followed by segmentation and aggregation via a hierarchical convolutional encoder to reduce sequence length. • Stage 2 (Memory-Efficient LLM Processing): Encoding of patch representations via a Transformer Encoder augmented with linear-map memory attention (Performer or Linformer) to optimize computational resources. • Stage 3 (Prediction & Classification): Aggregation of latent representations via global pooling followed by supervised classification. 2

Table 1: High-level comparison of key token- and byte-level models.

Model/Framework

Data Granularity

Learning Paradigm

Context Handling

Primary Contribution

FedNLP Lin et al. (2021) ByT5 (Xue et al., 2022) Longformer (Beltagy et al., 2020) BLT (Pagnoni et al., 2024)

Token-Level Byte-Level Token-Level Byte-Level

Federated Centralized Centralized Centralized

Standard (O(L2 )) Standard (O(L2 )) Efficient (Linearized) Standard (O(L2 ))

Benchmarking FL for token-based NLP. State-of-the-art among early byte-level text processing. Efficiently processing long token sequences. Scaling byte-level models for large datasets.

FederatedBytes (Ours)

Byte-Level

Federated

Efficient (Linearized)

Unified framework for private, efficient, byte-level FL.

Table 2: Detailed comparison of specific models and methods. This table analyzes the key innovations and suitability for federated learning of the models discussed. While powerful, centralized models such as ByT5 and BLT are not inherently designed for the communication and resource constraints of FL.

Model/Method

Key Innovation

Primary Use Case

Suitability for Federated Learning

FedAvg McMahan et al. (2017) Decentralized model ag- General-purpose FL gregation

Foundational, but modelagnostic and can be inefficient for large models. Multilingual NLP in cen- Low. High computational tralized settings and memory requirements; not optimized for communication efficiency. Generative modeling on Low. Similar resource conraw bytes straints to ByT5; large model size poses communication challenges. Foundational research in Very Low. Designed for large byte-level models large-scale, centralized data centers, making it impractical for on-device FL.

ByT5 Xue et al. (2022)

Encoder-decoder Transformer on byte streams

bGPT Wu et al. (2024)

Decoder-only Transformer on byte streams

BLT (Pagnoni et al., 2024)

Scaling byte-level models to very large parameter counts

FederatedBytes (Ours)

Efficient attention on Multilingual and multi- High. Explicitly designed to byte streams within a modal NLP on decentral- be resource-efficient, reduce federated framework ized data communication, and operate without a central data store.

Unlike standard decoder-only architectures, our model utilizes bidirectional encoding to capture full sequence context (noncausal). The architecture is composed of modular components explicitly configured for the resource constraints of federated clients.

3.1. Input Processing: Patching and Hierarchical Encoding The first stage prepares raw text for the model by converting it into a compressed vector representation. Raw text inputs x are converted into byte sequences b = {b1 , b2 , . . . , bL }, where each bi is a byte value corresponding to the UTF-8 encoding. Each byte bi is then mapped to a continuous vector embedding ei ∈ Rd , which will later be used for processing. To process these sequences efficiently, we employ a fixed patching strategy augmented by a hierarchical encoder. The byte sequence is divided into patches of size P s (defaulting to 128). While simple averaging is supported, our primary configuration utilizes a Hierarchical Patch Encoder. This module applies a 1D Convolution over the patch dimension followed by a ReLU activation and an adaptive pooling layer to aggregate P s embeddings into one. This step significantly optimizes computational resources. By reducing the effective sequence length from L to L/P s , the memory complexity is reduced from O(L2 ) to O((L/P s )2 ) before the main Transformer layers process the data.

Core Components and Hyperparameters The model follows a standard Transformer topology. Based on our primary experimental configuration, the key hyperparameters are a hidden dimension d = 768, 12 attention heads, a depth of 12 layers, and a dropout rate of p = 0.1. The model’s structure, in order of execution, is: • (embed): An Embedding layer mapping byte values to dense vectors (d = 768). • (patch_encoder): A hierarchical convolutional block that aggregates raw byte embeddings into patch vectors. • (pos_emb): Learnable positional embeddings added to the patch vectors to capture sequence order.

3.2. Memory-Efficient LLM Processing The core of our framework is the FederatedBytes, a custom Transformer Encoder designed for memory efficiency.

• (att): A stack of memory-efficient Transformer Encoder blocks. Each block utilizes linear attention mechanisms 3

Figure 1: End-to-end FederatedBytes Architecture. The pipeline operates in three sequential stages tailored for efficient federated learning: Stage 1 (Input Processing): Raw multilingual text is encoded into UTF-8 bytes. These bytes are compressed into patch embeddings using a Hierarchical Patch Encoder (1D Convolution → ReLU → Adaptive Pooling) to reduce sequence length. Stage 2 (Memory-Efficient Processing): The patch sequence is processed by a Transformer Encoder utilizing the Performer mechanism. This approximates the full attention matrix using Orthogonal Random Features (ORFs) Yu et al. (2016), achieving linear O(L) time and space complexity. Stage 3 (Prediction & Classification): The encoded representations are aggregated via global mean pooling, normalized via BatchNorm, and passed through a linear classification head to generate final class logits.

To address this, we integrate the Performer mechanism Choromanski et al. (2020), which utilizes the kernel trick to linearize attention. Instead of computing the massive L × L attention matrix, Performer approximates the softmax kernel using a random feature map ϕ(·). This allows us to decompose the attention equation using the associative property of matrix multiplication:

to perform contextual feature extraction without quadratic memory costs. • (pool): A global mean pooling operation that aggregates the sequence of hidden states into a single vector representation. • (bn): A BatchNorm1d layer applied to the pooled features for normalization.

      V  Attention(Q, K, V) ≈ ϕ(Q) ϕ(K)⊤ |{z} |{z} |{z} 

• (dp): A Dropout layer for regularization. • (clf): A Linear layer acting as the classification head.

L×m

m×L

(2)

L×d

By changing the order of multiplication, we avoid constructing the L × L matrix entirely. Instead, we first compute ϕ(K)⊤ V, resulting in a small m × d summary matrix, and then multiply by ϕ(Q). The overall complexity drops from O(L2 d) to O(Lmd), where m is the number of random features (m ≪ L).

Memory-Efficient Attention The computational core of the model is its attention mechanism. In a standard Transformer, the attention score is calculated as: ! QK ⊤ Attention(Q, K, V) = softmax √ V (1) dk

Matrix Decomposition Sample: To visualize this reduction, consider the dimensionality change during the forward pass:

where Q, K ∈ RL×d are the query and key matrices. The term QK ⊤ produces an L × L matrix. This standard self-attention presents a significant bottleneck for long sequences due to its quadratic space and time complexity, O(L2 ). For example, for our byte sequences where L can reach 16,384, this matrix creates an imposing memory footprint.

Standard:

4

. " #  ..    ... ... × V  ... ...   | {z }  ..  . L×L (Bottleneck)

“This product is amazing and works perfectly!”

Performer:

 .    .    .  h ϕ(Q) ×  . . .     ..   . | {z } | L×m

 .   .  i  .  ϕ(K)⊤ . . . × V     ..  . {z }

We trace the transformation of this input through the three architectural stages defined in Section 3. 4.1.1. Stage 1: Input Processing (Patching and Averaging) Byte Encoding. The raw text is first converted into a sequence of UTF-8 integers, one per character:

m×d (Compact Context)

Our implementation specifically employs Orthogonal Random Features (ORFs) Yu et al. (2016) to construct ϕ(·). Unlike standard random Fourier features which are drawn independently, ORFs impose an orthogonality structure on the random projection weights. This theoretical guarantee reduces the variance of the approximation, providing a more stable estimate of the softmax kernel and encouraging consistent convergence during federated training rounds.

  84 100   97 “This product...” →   97 115  116

105 115 32 99 116 32 97 122 105 100 32 119 112 101 114 121 33

112 105 110 111 102

 114 111  115 32  103 32   114 107  101 99  

Purpose: To transform raw linguistic input into a numerical substrate suitable for arithmetic processing.

3.3. Prediction and Classification

Padding and Patching. The byte sequence of the review (44 bytes) is padded with the special token ρ = 256 up to a fixed length (L = 16, 384 in our case). We then segment this sequence into 128 patches of size P s = 128 each. Unlike the previous small-patch example, the larger patch size in our architecture means that the entire 44-byte review fits within the first patch, p1 , filled to its full length by padding tokens:

Once the attention mechanism has efficiently encoded the long byte sequence into a set of context-aware patch representations, the architecture must synthesize this information for the downstream task (i.e., classification in our case). In this final stage, the sequence of N = L/P s encoded features, H ∈ RN×d , is first compressed via global mean pooling to produce a single, fixed-size context vector h pooled . To ensure training stability and prevent overfitting—crucial in federated settings where client data can be sparse—this pooled vector undergoes batch normalization and dropout before the final projection. The output probabilities P ∈ RK for K classes are then computed as:   P = softmax Wo · dropout(BN(h pooled )) + bo

104 117 109 110 32 108

[84, 104, . . . , 119, 111, 114, 107, 256, 256, . . . , 256] | {z } | {z } 44 text bytes

84 padding tokens

The second patch (p2 ) and all subsequent patches comprise purely padding tokens. Purpose: Standardize sequence length and create structured patch inputs for the Transformer. Padding ensures all inputs are of uniform length, while the use of token 256 prevents conflict with valid UTF-8 byte values (0–255).

(3)

where Wo and bo denote the weights and biases of the final linear classification head. The entire network is trained end-toend by minimizing the cross-entropy loss between P and the ground-truth labels in the chosen training set.

Tabular View of Patch Content. Since the model utilizes a patch size of P s = 128, the entire example text fits within the first patch. Table 5 visualizes the first 32 active bytes of Patch 1. The remaining 96 indices (32–127) of this patch are filled with the padding token ρ = 256.

4. Methodology: Model, Data, and Training Having defined the architectural components of FederatedBytes, we now detail the specific implementation and experimental framework used to evaluate its efficacy. Our approach realizes this architecture as a lightweight, byte-level Transformer Encoder optimized for efficient training in resourceconstrained settings. The methodology encompasses three core components: (1) a hierarchical input representation that processes raw text as compressed byte patches; (2) a memoryefficient model implementation utilizing bidirectional attention with linear complexity; and (3) a rigorous federated training pipeline that transforms these patch representations into robust classification outputs.

Table 3: Byte content of the first 32 positions in Patch 1. T (84) d (100) a (97) a (97)

h (104) u (117) m (109) n (110)

i (105) c (99) a (97) d (100)

s (115) t (116) z (122) ␣(32)

␣(32) ␣(32) i (105) w (119)

p (112) i (105) n (110) o (111)

r (114) s (115) g (103) r (114)

o (111) ␣(32) ␣(32) k (107)

Embedding and Hierarchical Encoding. Once the input text is segmented into fixed-size byte patches, the next step is to project these raw integers into a continuous latent space. Each byte integer bi within a patch is first mapped to a dense embedding vector ei ∈ R768 . However, processing every individual byte embedding would be computationally prohibitive. To reduce sequence length while retaining local context, we utilize a Hierarchical Patch Encoder. Unlike simple averaging, this module applies a

4.1. Data Pipeline with Worked Example To illustrate the pipeline, we use the running example of a short Amazon review: 5

Figure 2: The FederatedBytes Inference Pipeline. An illustrative walkthrough of the running example (“This product is amazing...”) as it passes through the three stages of the model architecture. Stage 1: Input text is patched and processed by the Hierarchical Encoder (Conv1d) to reduce sequence length. Stage 2: Patches are contextualized via Performer linear attention. Stage 3: Features are pooled for final classification.

Purpose: To reducing sequence dimensionality and summarize local information within each patch via learned feature extraction, enabling efficient processing of contexts up to 16,384 bytes in size.

Table 4: Example embeddings (first 4 dimensions) for the first 8 bytes of Patch 1.

Byte

e(1)

e(2)

e(3)

e(4)

84 (T) 104 (h) 105 (i) 115 (s) 32 (space) 112 (p) 114 (r) 111 (o)

0.12 -0.21 0.19 0.03 0.00 0.44 -0.09 0.28

-0.08 0.15 -0.11 0.48 0.00 -0.05 0.25 -0.14

0.33 0.05 0.29 -0.22 0.00 0.19 -0.16 0.07

0.44 -0.36 0.07 -0.15 0.00 0.09 0.38 -0.21

4.1.2. Stage 2: Memory-Efficient Processing Contextualization. Having transformed the raw byte stream into a sequence of dense patch representations, the architecture now shifts to modeling the relationships between them. The sequence of patch vectors is processed by the Performer attention mechanism. This efficient linear attention allows the model to learn dependencies between distant parts of the byte stream—for instance, relating the patch containing the sentiment-bearing word “amazing” to the entity “product” appearing much earlier in the sequence. Table 5 illustrates hypothetical attention weights between the first three patches, demonstrating how the model attends to relevant context regardless of distance:

learned transformation to the patch. Table 4 illustrates the embeddings for the first few bytes of a patch before this aggregation takes place. We aggregate the sequence of embeddings within the patch (128 × 768 → 1 × 768) using a 1D Convolution followed by adaptive pooling. Formally, for the k-th patch matrix E (k) , the encoded representation h(k) is computed as:    h(k) = Pool σ Conv1d(E (k) )

Table 5: Hypothetical attention distribution across the first 3 patches.

Patch 1 Patch 2 Patch 3

(4)

where σ is the ReLU activation function. For this example, the resulting encoded vector begins:

Patch 1

Patch 2

Patch 3

0.60 0.20 0.10

0.25 0.65 0.30

0.15 0.15 0.60

Purpose: To capture long-range semantic relationships and global context across the various byte patches without incurring

h(k) = [0.07, 0.05, 0.08, 0.03, . . . ] 6

4.1.3. Stage 3: Prediction & Classification Pooling and Classification. Once the sequence has been contextualized, the final stage is to synthesize this information into a decision. The encoded features H ∈ R128×768 are aggregated via global average pooling across the sequence dimension, reducing the input to a single global context vector of size 768.

of size B = 2. Due to the memory footprint of the Performer attention map on large inputs (L = 16, 384), this micro-batching is essential to prevent Out-Of-Memory (OOM) errors while maintaining high client concurrency. We utilize gradient accumulation (128 steps) to achieve an effective batch size of 2 × 128 = 256. We assume our example is the first sample in the batch (index = 0) and has ground-truth label y0 = 1 (positive class in a binary set). Table 6 summarizes this sample.

p = mean(H) = [0.12, −0.05, 0.07, 0.19, . . . ] ∈ R768

Table 6: Sample input data for our example review within a training microbatch.

the quadratic cost of standard attention.

Data Element Input Patches (X0 ) Label (y0 )

This vector is then regularized and projected to the output space. It passes through a Batch Normalization layer, a Dropout layer, and finally the linear classifier to produce raw logits for the target classes (two in this case, as an example of binary classification): ŷ = model.clf(p)

→

Purpose: To break the data into manageable pieces, allowing the simulation to fit 100 distinct client models within the VRAM constraints of a single A100 GPU.

[1.45, −0.62]

Step 2: Forward Pass and Logit Generation. The batch is then fed into the model. Table 7 shows the tensor for our specific review sample as it is transformed through each layer of the FederatedBytes. Note the dimension’s increase to 768 (hidden size) and the sequence length of 128 patches. Purpose: To compute the probability of all classes in our class set by passing the input through the model’s layers, transforming raw bytes into a final logit vector that represents the classification score for each class.

Applying the softmax function yields the final class probabilities: P(positive) ≈ 0.888,

Sample Values for Review (i = 0) First patch: [84, 104, 105, 115, 32, 112, 114, 111, . . . , 256] (128 integers) 1

P(negative) ≈ 0.112

Consequently, the model correctly predicts the review sentiment as positive. Purpose: To aggregate distributed information across all patches into a fixed-size representation and generate the final predictive probability scores.

Step 3: Loss Computation. The discrepancy between the model’s logits and the ground-truth label is quantified using the standard Cross-Entropy Loss. For our example, the model produced logits [−0.62, 1.45], and the true label is 1. Therefore:

4.2. Training Process With the complete forward propagation pipeline—from raw byte patching to final classification—fully established, we now turn to the optimization procedure used to learn the model weights. The model is trained using a supervised learning regimen designed to minimize the error between predicted logits and ground-truth labels. This process involves iterating over the training dataset in batches, computing the crossentropy loss, and updating the model’s approximately 30 million and 100 million parameters in FederatedBytes-30m and FederatedBytes-100m variants, respectively, via backpropagation. To ensure training stability and convergence, we utilize the proven AdamW Loshchilov & Hutter (2019) optimizer with weight decay and mixed-precision gradients. In order to minimize the loss, the training process aims to maximize the probability assigned to the ground-truth class label for all samples in the training set (ideally approaching values of one). The following steps detail the flow of a single training step, using our running example of the positive Amazon review to illustrate the transformation of data.

Probabilities = softmax([−0.62, 1.45]) ≈ [0.112, 0.888] L0 = − log(Probability of true class) = − log(0.888) ≈ 0.119 The final loss is averaged across the batch. Purpose: To calculate a single scalar value that measures the model’s error on the sample, providing a differentiable objective signal for optimization. Step 4: Backpropagation and Weight Update. The batch loss is used to compute gradients for all trainable parameters via loss.backward(). Because the physical batch size (B = 2) is restricted by the A100 simulation constraints, we perform gradient accumulation for 128 steps before the AdamW optimizer updates the model’s weights. Purpose: To decouple the logical batch size from physical memory limitations, ensuring stable convergence for the 100client simulation despite the small micro-batch size. 4.3. Evaluation Protocol To assess the model’s generalization capabilities on unseen data, a comprehensive evaluation is performed on the test set of each dataset. The process mirrors the training forward pass but does not perform any weight updates.

Step 1: Batch Preparation (Simulation Scaling). To facilitate the large-scale simulation of 100 federated clients on a single NVIDIA A100 GPU, we adopt a strict memory-constrained training regimen. The training data is loaded in micro-batches 7

Table 7: Tensor transformation for one sample during the forward pass.

Stage of Processing 1. Input Patch

Data Shape ‘(128,)‘

2. Embedding

‘(128, 768)‘

3. Hierarchical Encoding 4. Attention 5. Global Pooling 6. Classification

‘(768,)‘ ‘(128, 768)‘ ‘(768,)‘ ‘(2,)‘

Sample Value / Representation (first 4 dims shown) ‘[84, 104, 105, 115, ...]‘ (A single patch of 128 byte integers)    0.12 −0.08 0.33 . . . −0.21 0.15 0.05 . . .  (Matrix of 128 embedding vectors)   . .. .. . .  .. . . . ‘[0.07, 0.05, 0.08, 0.03, . . . ]‘ (Encoded vector for the patch via Conv1d) ‘[0.09, -0.01, 0.15, 0.02, . . . ]‘ (Sequence of 128 contextualized vectors) ‘[0.12, -0.05, 0.07, 0.19, . . . ]‘ (Mean of all 128 patch vectors) ‘[-0.62, 1.45]‘ (Final 2-dim output logits for the sample)

Purpose: To gather all necessary outputs to perform a comprehensive, dataset-wide performance analysis, which is far more reliable than single-batch metrics.

Step 1: Inference Mode and Batch Processing. The model is set to inference mode (model.eval()) and gradient calculations are disabled. The evaluation data, including our example review, is processed in batches. (For simplicity, in Table 8 we show the same example used to illustrate the training, but in reality the training and test sets are always disjointed.)

Step 4: Comprehensive Metric Calculation. Using the complete, aggregated lists, a suite of metrics is calculated. Our example, having been correctly classified (y0 = 1, ŷpred,0 = 1), contributes positively to the overall accuracy.

Table 8: Sample input data for our example review within an evaluation batch.

Data Element Input Patches (X0 ) Label (y0 )

• Accuracy: The total number of matches between yall and ŷall , divided by Neval .

Sample Values for Review (i = 0) First patch: [84, 104, 105, 115, 32, . . . ] 1

• Weighted F1-Score: Calculated from yall and ŷall to account for class imbalance.

Purpose: To ensure deterministic model outputs by disabling the Dropout stochastic layer, and skip the unnecessary gradient computation.

• AUC: Calculated by comparing the true labels yall against the probabilities derived from ŷlogits, all . Purpose: To provide a robust quantitative assessment of the model’s performance, covering its correctness (Accuracy), balanced performance (F1-Score), and discriminative power (AUC) on unseen data.

Step 2: Prediction Generation. A forward pass, identical to the one used in training, generates the logits. The final prediction is determined by applying an argmax operator. For reference, let’s assume that the model, after training, produces slightly different logits for our sample:

4.4. Federated Learning Framework Having established the training routine and evaluation metrics for a single FederatedBytes model, in this section we describe how multiple such models are orchestrated across a decentralized network to form the complete training pipeline. To implement and simulate the federated learning process, we utilize the popular Flower framework Flower (2023). Unlike centralized training, where data is aggregated, our framework distributes the models to the data. The workflow consists of three iterative steps:

Table 9: Prediction generation for our review sample during evaluation.

Stage Final Pooled Vector Generated Logits (ŷlogits,0 ) Probabilities Prediction (ŷpred,0 )

Sample Value [0.15, −0.09, 0.11, 0.23, . . . ] [−1.15, 2.50] softmax([−1.15, 2.50]) = [0.025, 0.975] argmax([−1.15, 2.50]) = 1 (Positive)

Purpose: To obtain the model’s final, discrete predictions and class probabilities for each sample in the unseen test set.

1. Global Initialization. A global model instance with parameters θ0 is initialized on the central server. In our simulation, the server distributes this initial state to a subset of available clients Cr selected for the current round r.

Step 3: Dataset-Wide Result Aggregation. The prediction, true label, and logits for our sample are appended to lists that collect the results for the entire evaluation dataset (i.e., the test set) of size Neval . Table 10 illustrates the lists.

2. Local Training. Each selected client k retrieves the global parameters θt and trains a local instance of the FederatedBytes on its private partition of the dataset Dk . The client performs E local epochs (set to 5 in our experiments) to minimize the cross-entropy loss:

Table 10: Appending our example’s results to the aggregated evaluation lists.

Aggregated List All Labels (yall ) All Predictions (ŷall ) All Logits (ŷlogits, all )

State After Processing Our Sample [1, . . . ] [1, . . . ] [[−1.15, 2.50], . . . ]

θkt+1 ← LocalUpdate(θt , Dk )

(5)

This step utilizes the same byte-level patching and Performer attention mechanisms described in Section 3. 8

3. Federated Averaging (Model Aggregation). Once local training is complete, clients upload their updated model parameters θkt+1 to the server. The server aggregates these updates using the Federated Averaging (FedAvg) strategy McMahan et al. (2017). The new global model θt+1 is computed as the weighted average of the client models: θt+1 =

K X Nk

N k=1

θkt+1

amount of statistical noise is added to the model parameters. This makes it computationally infeasible for an adversary to determine whether any particular individual’s data was part of the training process, thus protecting against threats like membership inference attacks. • Secure Communication Channels: All communication between client devices and the aggregation server is encrypted using Transport Layer Security (TLS). This secures the data in transit, ensuring both the confidentiality and integrity of the model updates. It provides a crucial line of defense against man-in-the-middle (MITM) attacks, eavesdropping, and data tampering during transmission.

(6)

P where Nk is the number of samples on client k, and N = Nk is the total number of samples across participating clients. 4. Simulation Environment and Scalability Analysis. To validate this architecture at scale, we employ the flwr[simulation] engine backed by Ray1 . We adopt a progressive scaling strategy, systematically increasing the client cohort size K ∈ {5, 20, 50, 100} to record convergence behavior and system stability under varying levels of concurrency. This setup enables the simulation of up to 100 distinct clients with heterogeneous data partitions on a single NVIDIA A100 GPU, utilizing dynamic resource management to efficiently schedule and process concurrent client workloads.

• Robustness to Adversarial Attacks: Beyond privacy, security also involves ensuring the integrity of the final model. FederatedBytes can incorporate robust aggregation algorithms (e.g., using medians or trimmed means instead of a simple average) to defend against model poisoning attacks, where malicious clients attempt to corrupt the global model by sending faulty updates. 5. Experimental Setup

4.5. Security and Privacy Considerations Security and privacy are foundational to FederatedBytes’ architecture. The framework implements a multi-layered, defense-in-depth strategy to protect data and model integrity, adhering to the core principles of privacy-preserving machine learning:

We have conducted a series of experiments to evaluate the performance of our proposed byte-level model. We have experimented with two variants, FederatedBytes-Base, which adopts a standard attention mechanism, and FederatedBytesPerformer, which adopts a linear attention, and two different sizes, nicknamed FederatedBytes-30M and FederatedBytes100M with reference to their approximate parameter size. We have compared our models against established token-level and byte-level baselines of relatively small size (< 500M parameters): 1) GPT-2 Medium Radford et al. (2019); 2) ByT5 Xue et al. (2022); and 3) a 2-layer LSTM Hochreiter et al. (1997). We have first evaluated all the models in a centralized training setting (a single model trained on the entire training set), and then benchmarked the top-performing models in a simulated federated learning (FL) environment.

• Data Privacy by Design: The cornerstone of the framework’s privacy model is data localization. Raw, sensitive data never leaves the user’s device. Only abstract model parameters (e.g., weights or gradients), which are learned from the local data, are shared for aggregation. This approach inherently prevents data leakage from a central server and aligns with modern data protection regulations such as the General Data Protection Regulation (GDPR). • Secure Aggregation: While individual model updates do not contain raw data, they can still inadvertently leak information about the training data of a specific device. To mitigate this risk, FederatedBytes is designed to use secure aggregation protocols Bonawitz et al. (2017). These cryptographic methods allow the central server to compute the sum of all model updates from participating devices without being able to inspect any single, unencrypted update. This ensures that an honest-but-curious server cannot reverse-engineer a specific user’s contribution.

5.1. Datasets We have used two diverse datasets to evaluate model performance in different languages and modalities: • Amazon Multilingual Reviews: A sentiment classification dataset. We have used the subsets for Amazon (en), Amazon (de), and Chinese (zh). The task is binary classification, where reviews with 4 or 5 stars are labeled as positive (1) and all others as negative (0). Table 12 details the specific splits.

• Differential Privacy: To provide formal, mathematical privacy guarantees against powerful inference attacks, clientside differential privacy can be applied Dwork et al. (2006). Before an update is sent, a carefully calibrated

• ScienceQA: A multimodal, multi-class questionanswering dataset. Each entry consists of a question, multiple-choice answers, context (lecture, hint), and an optional image. The model must process both the unified text and the image (if present) to select the correct answer

1 https://github.com/ray-project/ray

9

Table 11: Overview of the Datasets Used in the Experiments.

Property

Multilingual Amazon Reviews

ScienceQA

Task Domain Classes Size Languages Modality Source

Binarized 5-Class Sentiment Analysis Product Reviews 5 (1-5 Stars) 200k samples per language English, German, Chinese Text Prettenhofer & Stein (2010)

Multimodal Question Answering Scientific Questions Multiple Choice (varies) 21k samples English Text + Image Lu et al. (2022)

Truncation

Note that, for Amazon Reviews, capping the input sequence length to 1024 bytes would truncate approx. 4% of the samples, while a 2048-byte cap would limit truncation to < 1% of the samples. Therefore, such cappings would have a limited impact on the measured performance.

• Model Size (GB): The disk footprint of the model, calculated as the sum of memory required for all parameters P ( p.nelement() × p.element_size()), reported in Gigabyte (GB).

Table 12: Amazon Reviews Dataset (3 Languages).

Language

# Train

# Val

# Test

Pos/Neg Ratio

English German Chinese

200,000 200,000 200,000

5,000 5,000 5,000

5,000 5,000 5,000

40.0% / 60.0% 40.0% / 60.0% 40.0% / 60.0%

• Total Time (minutes): The total wall-clock time in minutes for a single experiment run, including all training epochs and the final evaluation. • GPU and CPU RAM Used (GB): The peak increase in system GPU and CPU RAM (in Gigabytes) observed during the model’s training and evaluation. This is measured by taking a snapshot of ‘psutil.virtual_memory().used’ before and after the run.

index. This dataset tests the model’s ability to handle mixed-modality byte streams. Table 11 provides further details of the datasets. 5.2. Evaluation Metrics

• Network Footprint (MB): A simulated metric for federated learning overhead. We report Total Sent and Total Received (S/R), where each is calculated as: Total Sent/Received = Nclients_per_round × Model Size (MB) × Nrounds .

5.2.1. Model Performance Metrics • Accuracy: The percentage of correctly classified samples, Predictions calculated as Correct Total Predictions . • F1-Score (Weighted): The weighted average of the F1score for each class, accounting for label imbalance. This provides a robust measure of classification performance beyond simple accuracy.

5.3. Baseline Models We compare our proposed models against three comparable baselines representing token-level, byte-level, and recurrent sequential approaches:

• AUC (Area Under the ROC Curve): The Area Under the Receiver Operating Characteristic curve (i.e., the false positive/true positive trade-off for various values of the decision threshold). For binary tasks, this is the standard AUC. For multi-class tasks, we report the one-vs-rest (OvR) weighted AUC, which evaluates the model’s ability to discriminate between each class and the rest.

• GPT-2 Medium: A standard GPT-2 model with a classification head. It operates on BPE-tokenized text, with a maximum sequence length of 1024 tokens. We adopted this model as an instance of small-sized transformer models that employs conventional tokenization. • ByT5: The ‘google/byt5-small’ model with no tokenizer with a classification head. This model operates directly on UTF-8 bytes and serves as a powerful byte-level baseline.

5.2.2. Resource & Efficiency Metrics In addition to the accuracy, in a federated learning environment it is fundamental to measure the computational efficiency of an approach. For this, we employ the following metrics:

• LSTM: A 2-layer LSTM model with a byte-level embedding (257 tokens, including a pad token). It processes sequences of up to 1024 bytes.

• Model Parameters (M): The total count of all trainable parameters in the model, reported in millions. This is calculated by summing ‘p.numel()’ for all parameters ‘p’ in the model.

Table 13 provides a comparison of the trainable parameter counts for the models evaluated in our experiments. 10

AUC of 80.50 pp vs 58.45 pp for ByT5, in less than half the runtime and with 1/10 of the GPU memory footprint. For the ScienceQA dataset, only ByT5 and FederatedBytes2 have been able to perform the task in the first place, and FederatedBytes outperformed By5 in every metric. At their turn, the baseline LSTM and the FederatedBytes-Base model have struggled to match the performance of the more complex models.

Table 13: Model Size (Number of Trainable Parameters).

Model ByT5 (small) GPT-2 Medium LSTM (2-layer) FederatedBytes-Base FederatedBytes-Performer-30M FederatedBytes-Performer-100M

Hidden Dim

Param Count

512 768 512 512 512 512

300M 124M 4.3M (approx.) 18.6 (approx.) 30M (approx.) 100M (approx.)

6.2. Federated Learning Results We benchmarked the scalability and robustness of FederatedBytes (30M and 100M variants) against the ByT5 baseline in a simulated federated learning setting on the Amazon (en, de, zh) datasets. The experiments ranged from 5 to 100 participating clients to evaluate performance under increasing model fragmentation. As shown in Table 15, FederatedBytes has demonstrated a superior scalability:

5.4. Proposed Model The proposed model leverages a hierarchical patch-based embedding system which permits much longer contexts (up to 16,384 bytes) with comparable or smaller overall model size. We have implemented it in two variants: • FederatedBytes-Base: This model uses a standard 2-layer ‘TransformerEncoder’ to process the byte patches, and leverages a conventional quadratic attention.

• Resistance to Model Collapse: In high-concurrency settings (100 clients), the ByT5 baseline has consistently collapsed, with AUC dropping to near-random performance (≈ 50−52%). In contrast, FederatedBytes-Performer-30M has maintained robust learning, achieving AUC scores of 65.96% (en), 72.87% (zh), and 66.71% (de). Detailed results are shown in Figure 3.

• FederatedBytes-Performer: This model replaces the standard transformer encoder with a 4-layer ‘Performer’ (using the FAVOR+ linear attention Likhosherstov et al. (2022)) to more efficiently handle long patch sequences. We have implemented this model in two sizes, 30M and 100M parameters, respectively.

• Efficiency Gains: FederatedBytes-Performer-30M has reduced training time by approximately 10× compared to ByT5 (e.g., 56 min vs. 558 min for 100 clients), while consuming significantly less CPU and network bandwidth and maintaining a peak memory footprint of only 152 MB compared to ByT5’s 3.4 GB.

All the hyperparameters used for our experiments are detailed in Table A.16 in the Appendix. 6. Results In this section, we present the results of our centralized and federated experiments, focusing on the trade-offs between token- and byte-level processing.

• Generalization: Interestingly, the federated FederatedBytes-Performer model frequently has outperformed its centralized counterpart (e.g., 78.8% AUC vs 71.9% for the centralized model on Amazon (en)), suggesting that the federated averaging process acts as an effective regularizer for our byte-level architecture.

6.1. Centralized Results In the experiments carries out in the centralized setting, all models were trained as single models over the full training sets. Table 14 reports the results, which include metrics to evaluate both the effectiveness and the efficiency of the compared models. To begin with, Table 14 shows that the conventional tokenlevel GPT-2 model has been able to achieve the strongest accuracy metrics on all the text-only tasks (Amazon (en,de,zh)). However, this model is approximately 3.5× larger than the largest of our models, FederatedBytes-Performer-100M, has occupied ∼8× the GPU memory (15.6 GB vs 1.96 GB) and has proved ∼3× slower to run (20-21 minutes vs 7.7). By contrast, our FederatedBytes-Performer has still been able to achieve a remarkable performance on the text tasks, reaching 80.50 AUC pp on Amazon (en), with a much smaller computational footprint, while at the same time being able to process a multimedia dataset such as ScienceQA. FederatedBytes has also compared very favorably with ByT5, proving more accurate and more efficient in every metric. For instance, on the Amazon (en) dataset FederatedBytes-Performer-100M has achieved an

• Multimodal Performance: FederatedBytes-Performer has displayed interesting multimodal capabilities on ScienceQA, reporting an AUC of 62.60% vs. ByT5’s 53.40%. We also observe that the 30M parameter model has achieved a higher AUC than the 100M model as the number of clients increases. This occurs because dividing the data among many clients leaves each client with an increasingly small dataset (|Dlocal |), and the large 100M model is too complex for such small data (|θ| ≫ |Dlocal |) and ends up “memorizing” local noise (overfitting). When these noisy updates are then combined, they conflict and degrade the global model. In contrast, the 30M model has fewer parameters which helps prevent it from memorizing noise and learn more stable, general patterns that improve the overall accuracy. 2 Note that we have used a 38M parameter version for this task, that corresponds to the 30M text-alone version augmented with the image input.

11

Table 14: Centralized performance of single models on the Amazon and ScienceQA datasets. Note: The top AUC scores within the Byte-Level model category are highlighted in bold. Text-Level baselines updated with GPT-2 Medium results.

Language

Model

Byte-Level Models Amazon (en) FederatedBytes-Base FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5 LSTM

Model Params

Model Size (GB)

Acc (%)

F1 (%)

AUC (%)

Mem (CPU/GPU)

Time (min)

18.6M 32.5M 101.3M 300.8M 4.3M

0.07 0.12 0.38 1.12 0.02

64.00 73.60 70.40 59.80 59.80

60.26 73.76 70.56 44.76 44.94

65.78 79.68 80.50 58.45 50.08

0.00 GB / 1.56 GB 0.00 GB / 0.69 GB 0.05 GB / 1.96 GB 0.00 GB / 19.99 GB 0.01 GB / 0.28 GB

2.84 5.53 7.67 17.31 4.21

Amazon (de)

FederatedBytes-Base FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5 LSTM

18.6M 32.5M 101.3M 300.8M 4.3M

0.07 0.12 0.38 1.12 0.02

65.00 72.50 75.50 59.90 59.80

64.54 72.76 75.68 44.98 44.76

67.99 80.71 82.79 66.53 50.29

0.02 GB / 1.56 GB 0.00 GB / 0.69 GB 0.00 GB / 1.96 GB 1.63 GB / 19.99 GB 0.00 GB / 0.28 GB

2.83 5.54 7.66 17.30 4.22

Amazon (zh)

FederatedBytes-Base FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5 LSTM

18.6M 32.5M 101.3M 300.8M 4.3M

0.07 0.12 0.38 1.12 0.02

65.60 73.60 71.50 59.80 59.80

65.91 73.77 71.77 44.76 44.76

72.14 79.95 80.34 69.29 50.08

0.02 GB / 1.56 GB 0.66 GB / 0.68 GB 0.07 GB / 1.95 GB 0.15 GB / 19.99 GB 0.06 GB / 0.28 GB

2.83 5.62 7.76 17.29 4.19

ScienceQA

FederatedBytes-Performer (38M) ByT5

38.8M 299.6M

0.15 1.14

39.00 26.00

36.17 24.77

62.60 53.40

2.42 GB / 0.15 GB 0.26 GB / 3.37 GB

1.84 13.60

Token-Level Models (upper-bound reference for text tasks) Amazon (en) GPT-2 Medium 355M Amazon (de) GPT-2 Medium 355M Amazon (zh) GPT-2 Medium 355M

1.32 1.32 1.32

88.80 80.17 77.77

88.78 80.28 77.70

94.84 88.76 84.54

0.8 GB / 15.16 GB 0.8 GB / 15.16 GB 0.8 GB / 15.16 GB

20.81 20.39 21.40

Figure 3: FederatedBytes Scalability and Efficiency Analysis. Top: FederatedBytes maintains high AUC scores while ByT5 collapses as client counts increase. Bottom: FederatedBytes demonstrates significantly lower computational overhead, network traffic, and memory footprint compared to ByT5. FederatedBytesPerformer-30M demonstrates superior efficiency, maintaining competitive AUC scores with significantly lower training time, CPU memory, and network consumption compared to the larger ByT5 model, especially as the number of clients increases. Data points represent averages across Amazon (en), (zh), and (de) datasets.

12

Table 15: Federated learning results comparing FederatedBytes-Performer-30M, FederatedBytes-Performer-100M, and ByT5. The highest AUC for each comparison is bolded. CPU memory reflects the active process usage (total) rather than differential deltas. Dataset

Amazon (en)

Amazon (de)

Amazon (zh)

Model

Clients

Acc (%)

F1 (%)

AUC (%)

Time (min)

CPU (GB)

GPU (GB)

Net (GB)

Size (GB)

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

5 5 5

59.70 59.80 59.80

44.71 44.76 44.75

76.79 74.80 60.61

2.27 7.55 22.32

0.13 0.17 0.28

0.46 0.72 1.71

2.27 7.55 22.32

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

10 10 10

59.80 59.80 59.80

44.76 44.75 44.76

72.04 72.04 56.54

5.67 18.87 55.81

0.13 0.17 0.27

0.46 0.72 1.71

5.67 11.32 55.81

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

20 20 20

59.80 59.80 59.80

44.76 44.76 44.75

66.70 68.21 49.66

11.33 37.74 111.62

0.13 0.17 0.27

0.46 0.72 1.71

11.33 37.74 111.62

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

50 50 50

40.20 59.80 59.80

44.75 44.75 44.76

64.95 63.87 50.70

28.33 94.34 279.06

0.13 0.17 0.27

0.46 0.72 1.71

28.33 94.34 279.06

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

100 100 100

59.80 59.80 59.7

44.76 44.75 45.24

65.96 59.32 51.24

56.67 188.69 558.12

0.13 0.17 0.27

0.46 0.72 1.71

56.67 188.69 558.12

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

5 5 5

63.40 59.80 62.30

52.93 44.76 50.90

80.25 79.77 65.53

2.27 7.55 22.32

0.13 0.17 0.28

0.46 0.72 1.71

2.27 7.55 22.32

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

10 10 10

59.80 45.90 59.80

44.76 35.18 44.76

77.65 75.40 60.84

5.67 18.87 55.81

0.13 0.17 0.27

0.46 0.72 1.71

5.67 18.87 55.81

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

20 20 20

66.40 59.80 59.80

59.59 44.76 44.76

74.05 75.86 54.94

11.33 37.74 111.62

0.13 0.17 0.27

0.46 0.72 1.71

11.33 37.74 111.62

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

50 50 50

60.70 59.80 59.80

47.59 44.76 44.76

68.14 63.78 51.77

28.33 94.34 279.06

0.13 0.17 0.28

0.46 0.72 1.71

28.33 94.34 279.06

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

100 100 100

59.80 44.90 59.80

44.76 34.11 44.76

70.43 66.71 52.20

56.67 188.69 558.12

0.13 0.17 0.28

0.46 0.72 1.71

56.67 188.69 558.12

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

5 5 5

61.60 59.80 59.90

49.02 44.75 44.98

79.56 75.15 66.84

2.27 7.55 22.32

0.13 0.17 0.28

0.46 0.72 1.71

2.27 7.55 22.32

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

10 10 10

59.80 63.7 59.80

44.76 54.00 44.75

77.15 76.69 59.72

5.67 18.87 55.81

0.14 0.17 0.27

0.46 0.72 1.71

5.67 15.10 55.81

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

20 20 20

40.10 64.20 59.80

44.75 54.93 44.76

70.65 75.83 51.55

11.33 37.74 111.62

0.13 0.17 0.28

0.46 0.72 1.71

11.33 30.19 111.62

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

50 50 50

40.10 50.10 59.80

44.75 43.71 44.75

73.86 76.12 50.38

28.33 94.34 279.06

0.14 0.17 0.28

0.46 0.72 1.71

28.33 37.74 279.06

0.11 0.38 1.12

FederatedBytes-Performer-30M FederatedBytes-Performer-100M ByT5

100 100 100

40.20 59.80 59.80

44.75 44.76 44.76

72.87 68.20 49.72

56.67 188.69 558.12

0.14 0.17 0.28

0.46 0.72 1.71

56.67 188.69 558.12

0.11 0.38 1.12

13

7. Conclusion

McMahan, B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In AISTATS.

This paper has proposed FederatedBytes, the first byte-level transformer model expressly engineered for federated learning environments. Byte-level models are particularly attractive for federated learning as they are able to dispense with memoryoccupying language-specific vocabularies, remove the “vocabulary leak” privacy gap, and process the diverse type of data that may originate from different clients. By adopting the memorysparing Performer attention, we have been able to mitigate the quadratics cost of standard self-attention, permitting efficient processing on resource-constrained edge devices. Looking ahead, we aim to extend FederatedBytes with deeper multimodal capabilities, scale federated training to larger client populations and longer contexts, and incorporate personalized optimization methods to better accommodate diverse user data, all the while preserving privacy and communication efficiency.

Hard, A., et al. (2018). Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604. Gillick, D., Brunk, C., Vinyals, O., & Subramanya, A. (2016). Multilingual language processing from bytes. In NAACLHLT (pp. 1296–1306). Xue, L., et al. (2022). ByT5: Towards a token-free future with pre-trained byte-to-byte models. TACL, 10, 291–306. Wu, S., Tan, X., Wang, Z., Wang, R., Li, X., and Sun, M. (2024). Beyond Language Models: Byte Models are Digital World Simulators. arXiv:2402.19155. :contentReference[oaicite:0]index=0 Vaswani, A., et al. (2017). Attention is all you need. In NeurIPS (pp. 5998–6008).

8. Acknowledgment

Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020). Transformers are RNNs: Fast autoregressive transformers with linear attention. In ICML.

The authors are grateful to the Food Agility Cooperative Research Centre and Robert Bosch (Australia) Pty Ltd for their generous support toward this research. This project was supported by funding from Food Agility CRC Ltd, funded under the Commonwealth Government CRC Program. The CRC Program supports industry-led collaborations between industry, researchers and the community.

Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. Choromanski, K., et al. (2021). Rethinking attention with performers. In ICLR. Bonawitz, K., et al. (2017). Practical secure aggregation for privacy-preserving machine learning. In CCS (pp. 1175– 1191).

References Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Comput. 9, 1735–1780.

Maas, A. L., et al. (2011). Learning word vectors for sentiment analysis. In ACL (pp. 142–150).

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019. Language models are unsupervised multitask learners. OpenAI URL: https://cdn.openai. com/better-language-models/language_models_ are_unsupervised_multitask_learners.pdf.

Prettenhofer, P., & Stein, B. (2010). Cross-language text classification using structural correspondence learning. In ACL (pp. 1118–1127). Munkhdalai, T., Faruqui, M., & Gopal, S. (2024). Leave No Context Behind: Efficient linear Context Transformers with Infini-attention. arXiv:2404.07143v1.

Dwork, Cynthia and McSherry, Frank and Nissim, Kobbi and Smith, Adam (2006). Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference (TCC), pp. 265–284.

Lu, P., et al. (2022). Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35, 2507–2521.

Likhosherstov, V., Choromanski, K., Dubey, A., Liu, F., Sarlos, T., & Weller, A. (2022). FAVOR+: Sharp attention kernel approximations via new classes of positive random features. In International Conference on Learning Representations.

Wang, S., et al. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.

Yu, F. X., Suresh, A. T., Choromanski, K., Holtmann-Rice, D., & Kumar, S. (2016). Orthogonal random features. In Advances in Neural Information Processing Systems.

Xue, L., et al. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.

Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.

Anon (2020). Optimizing Multilingual LLMs via Federated Learning: A Study of Linguistic Composition in Clients. arXiv preprint arXiv:2010.11934. 14

Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., & others (2020). Rethinking attention with performers. In International Conference on Learning Representations. Pagnoni, A., Pasunuru, R., Rodriguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, L., Weston, J., Zettlemoyer, L., Ghosh, G., Lewis, M., Holtzman, A., and Iyer, S. (2024). Byte Latent Transformer: Patches Scale Better Than Tokens. arXiv:2412.09871. :contentReference[oaicite:0]index=0 Flower (2023). The Flower Federated Learning Framework. Wolf, T., et al. (2020). Huggingface Model Cards. Table A.16: Key hyperparameters and configuration settings used in the experiments.

Lu, P., et al. (2022). ScienceQA: A Multi-Modal QuestionAnswering Benchmark. NeurIPS. Lin, B. Y., et al. (2021). FedNLP: Benchmarking federated learning methods for natural language processing tasks. NAACL. Zhang, H., et al. (2022). FedTP: Federated learning with transformer personalization. arXiv preprint arXiv:2211.01572. Appendix A. Hyperparameters The main hyperparameters for the proposed model are reported in Table A.16.

Parameter

Value

General Optimization Random Seed Optimizer Learning Rate Weight Decay Dropout

42 AdamW 3.0 × 10−5 0.01 0.1

Training Dynamics Micro-Batch Size (B) Gradient Accumulation Steps Effective Batch Size Centralized Epochs

2 128 256 15

Federated Learning (Simulation) FL Strategy FedAvg Total Clients (K) {5, 20, 50, 100} Client Fraction (Fit) 0.5 Local Epochs (E) 5 FL Rounds (T ) 5

Appendix B. GPT-2 Model Configuration and Model Card Our GPT-2 baseline follows the HuggingFace gpt2 recipe: • Number of Layers: 12 • Hidden Size (d_model): 768

Model Architecture (FederatedBytes) Hidden Size (d) 768 Attention Heads 12 Encoder Layers 12 Max Sequence Length (L) 16,384 bytes Patch Size (P s ) 128 bytes Hierarchical Encoder Conv1d + Pool Attention Type Performer (ORFs)

• Number of Attention Heads: 12 • Intermediate Feed-Forward Size: 3072 • Vocabulary Size: 50,257 • Context window: 1024 tokens Model Card: In alignment with Wolf et al. (2020b), we provide brief model documentation: Use Cases: General-purpose language modeling, next-token prediction, and fine-tuning for classification tasks. Limitations: Prone to generate plausible-sounding but factually incorrect or biased text; context window limited to 1024 tokens in the base model. License: Licensed under the MIT or Apache-2.0 (check original GPT-2 license). Appendix C. Federated Training Algorithm Algorithm 1 outlines the overall federated training process. 15

Algorithm 1 FederatedBytes Training Procedure (Simulation) 1: Input: Global model θ0 , Learning rate η, Clients K, Rounds T , Local Epochs E, Micro-batch B, Accumulation Steps M, Patch size P s 2: Output: Final global model θT 3: for round t = 0, . . . , T − 1 do 4: Server selects subset of clients S t ⊆ {1, . . . , K} 5: Server broadcasts θt to clients k ∈ S t 6: for each client k ∈ S t in parallel do 7: θk ← θ t ▷ Initialize local model 8: gaccum ← 0 ▷ Reset gradient accumulator 9: for epoch e = 1, . . . , E do 10: for step j, (X, Y) in Dk (Micro-batch size B) do 11: // Step 1: Byte-Level Processing 12: b ← UTF8Encode(X) 13: bpad ← Pad(b, Lmax ) ▷ Pad to 16,384 bytes 14: // Step 2: Patching & Hierarchical Encoding 15: P ← Segment(bpad , size = P s ) ▷ Create 128-byte patches 16: Z ← Embed(P) ▷ Map bytes to vectors 17: Zenc ← Pool(σ(Conv1d(Z))) ▷ Hierarchical Encode 18: // Step 3: Efficient Encoding 19: H ← PerformerAttention(Zenc ) ▷ Linear O(L) Attention 20: Ŷ ← Classifier(MeanPool(H)) 21: // Step 4: Optimization with Accumulation 22: L ← CrossEntropy(Ŷ, Y) ▷ Accumulate 23: gaccum ← gaccum + M1 ∇θk L Gradients 24: if ( j mod M == 0) or (end of data) then 25: θk ← θk − η · AdamW(gaccum ) 26: gaccum ← 0 ▷ Reset accumulator 27: end if 28: end for 29: end for 30: Client sends θk to Server 31: end for 32: // Server Aggregation (FedAvg) P 33: θt+1 ← k∈S t PNNk j θk 34: end for

16

Related documents

• FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks#1071357
• Federated Art Chase: Privacy Leakage and Data Poisoning in Federated Fine-Tuning#1356883
• Federated Art Chase: Privacy Leakage and Data Poisoning in Federated Fine-Tuning#1348768
• A multi-objective approach for communication reduction in federated learning under devices heterogeneity constraints#983903
• CLIN-FL: Cost-efficient Load-balancing for In-Network Federated Learning with non-IID data#1358518