SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization
Seyed Mahmoud Sajjadi Mohammadabadi 1 Xiaolong Ma 2 Lei Yang 1 Feng Yan 3 Junshan Zhang 4
Abstract
natural language processing, computer vision, and a range of other domains. However, adapting these models to downstream tasks remains resource-intensive. Full fine-tuning, which updates all model parameters, demands considerable computational, memory, and storage resources [Houlsby et al., 2019]. Parameter-Efficient Fine-Tuning (PEFT) techniques address this challenge by freezing the backbone and updating only a small set of task-specific parameters. For example, adapter modules insert compact trainable layers into each network block [Houlsby et al., 2019]; prefix-tuning optimizes a continuous prompt of only ∼0.1% of the model’s parameters [Li & Liang, 2021]; and Low-Rank Adaptation (LoRA) injects low-rank update matrices into each layer [Hu et al., 2021]. These methods achieve performance comparable to fully fine-tuned models while updating less than 1% of the model’s parameters.
arXiv:2604.08368v1 [cs.LG] 9 Apr 2026
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (SubspaceOriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model’s singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.
Despite these parameter savings, the cumulative communication and storage costs of PEFT modules remain a critical bottleneck in many real-world scenarios, particularly as foundation models continue to scale [Wolf et al., 2020]. In distributed scenarios (e.g., federated learning), these adapters must be communicated and stored across multiple devices or nodes, leading to significant overhead [Wolf et al., 2020]. Communication and storage overhead increase with the number of PEFT modules, as many fine-tuned adapters are saved and frequently transmitted or synchronized, thus turning millions of adapter parameters into a major bottleneck, particularly in bandwidth-limited or memoryconstrained environments such as edge devices or federated learning systems [Gao & Zhang, 2024; Wang et al., 2025]. The resulting communication and storage costs (i.e., the number of adapter parameters that must be transmitted and stored) can lead to slower training, increased energy consumption, and reduced scalability, highlighting the need for more efficient adapter compression techniques.
1. Introduction Foundation models (i.e., large-scale pretrained transformer architectures) have catalyzed substantial progress across
To address this, several methods decouple tunable parameters from adapter rank and model dimensions: NOLA [Koohpayegani et al., 2024] expresses LoRA’s matrices as linear combinations of random basis matrices, training only the coefficients; VeRA [Kopiczko et al., 2023] uses shared frozen random vectors with small learned scaling vectors; and SVFT [Lingam et al., 2024] constructs a basis from
1
University of Nevada, Reno, Reno, NV, USA 2 Argonne National Laboratory, Lemont, IL, USA 3 University of Houston, Houston, TX, USA 4 University of California, Davis, Davis, CA, USA. Correspondence to: Seyed Mahmoud Sajjadi Mohammadabadi <[email protected]>. Preprint. April 10, 2026.
1
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Figure 1. Overview of SOLAR. Given fine-tuned adapters (A, B), SOLAR projects them onto structured subspaces derived from the pretrained model’s SVD. A seeded pseudo-random generator (seeded with a known value) deterministically creates the basis matrices. Top-k coefficients α and β are selected under a budget to reconstruct à and B̃, while the bases are never stored or transmitted. Only the coefficients α, β, and the seed need to be communicated or stored.
• We provide a formal theoretical analysis that bounds the reconstruction error. Our proof decomposes the total error into the original training error and a controllable compression error, which can be minimized by tuning SOLAR’s hyperparameters (N and k). • We demonstrate through extensive experiments that SOLAR reduces adapter sizes by up to 98% while preserving the performance of the original LoRA adapters. Our results show competitive accuracy across a wide range of vision and language tasks using ViT, GPT-2, and LLaMA models.
singular vectors of pretrained weights and learns a sparse combination during fine-tuning. However, random bases not aligned with the model or task may reduce representational efficiency, and methods such as [Kopiczko et al., 2023; Lingam et al., 2024; Koohpayegani et al., 2024] are not post-hoc, as they modify the training process and cannot compress adapters already trained—creating a need for a flexible, training-free compression utility. In this paper, we propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a novel post-training compression method for PEFT adapters. SOLAR exploits the empirical structure of adapter updates by reparameterizing them as linear combinations of structured, randomized basis matrices. It is model-agnostic and applicable posttraining without modifying the fine-tuning process. The main contributions of this work are as follows:
2. Proposed Method: SOLAR We propose a post-training compression strategy that serves as a modular add-on for compressing PEFT-based updates. It introduces no training overhead and is compatible with LoRA [Hu et al., 2021], QLoRA [Dettmers et al., 2023], Compacter [Karimi Mahabadi et al., 2021], and NOLA [Koohpayegani et al., 2024], operating post-hoc by taking the final trained adapter matrices as input. SOLAR applies to Orthogonal Finetuning (OFT) [Qiu et al., 2023] and variants [Liu et al., 2023], compressing ∆W = (R−I)W via its SVD-based subspace without altering the orthogonal parameterization. By exploiting the underlying low-rank structure of updates, SOLAR significantly reduces both communication and storage costs in distributed or resource-limited settings.
• We leverage the observed subspace similarity between the foundation model’s weights (W ) and the task-specific update (∆W ) to create a more compact and efficient adapter representation. By expressing ∆W as a sparse combination of basis vectors, our method effectively decouples the adapter’s final size from the model’s architecture. • We develop a three-step framework for post-hoc adapter compression that involves: 1) constructing a basis pool of size N by perturbing the foundation model’s singular vectors with random noise, 2) performing a sparse selection of the most significant basis vectors to meet a budget k, and 3) reconstructing the adapter using only the selected coefficients and a single random seed. 2
SOLAR: Subspace-Oriented Latent Adapter Reparametrization ⊤ and ∥VW V∆W ∥2F , where U and V contain the left and right singular vectors from the SVD of each matrix [Hu et al., 2021]. Under this perspective, the model’s response to updates is well-approximated by a first-order expansion: f (ξ; W + ∆W ) ≈ f (ξ; W ) + ⟨∇f (ξ; W ), ∆W ⟩, where f is the model, ξ is input data, and ∇W f (ξ; W ) denotes the gradient of the foundation model’s output. This implies that ∆W lies in a low-curvature (and hence low-dimensional) subspace defined by W ’s parameter space (see Section 3.4 for empirical evidence). Thus, projecting ∆W into the subspace of W enables an efficient and compact representation that can be sparsified with minimal information loss.
2.1. Problem Formulation Transformer-based models parameterize attention and MLP layers using full-rank weight matrices W ∈ Rm×n . Recent PEFT methods, such as LoRA [Hu et al., 2021], decompose the task-specific update ∆W as ∆W = BA, where A ∈ Rr×n , B ∈ Rm×r , and r ≪ min(m, n). This reduces the trainable parameters from mn to r(m + n), yielding a mn compression ratio of r(m+n) . While effective, LoRA’s fixedrank formulation limits its flexibility. Alternatives, such as NOLA [Koohpayegani et al., 2024], leverage random projections to approximate ∆W , but often require large basis sets to sufficiently capture the relevant directions. To address this challenge and enhance compression further, we formulate the problem as minimizing the approximation loss between ∆W and its compressed counterpart ∆W̃ subject to a strict communication (or storage) budget: min ∥∆W − ∆W̃ ∥2F ,
s.t. ∥∆W̃ ∥0 ≤ k,
Building on these insights, we design a three-stage compression framework (Figure 1). First, we construct a randomized basis set aligned with the foundation model (Section 2.2.1). Next, we select a sparse set of bases to approximate the projected update (Section 2.2.2). We then reconstruct the update using a budget-aware combination of selected components (Section 2.2.3).
(1)
∆W̃
where ∥ · ∥F denotes the Frobenius norm, and ∥ · ∥0 representsPthe number of non-zero elements (i.e., ∥X∥0 ≜ P m n i=1 j=1 I{Xij ̸= 0}). The parameter k specifies the total budget.
2.2.1. S TEP 1: S UBSPACE -O RIENTED R ANDOMIZED BASIS S ET We construct a basis set from the foundation model’s parameter space via SVD of the model weight, W = U ΣV T , where U ∈ Rm×m and V ∈ Rn×n are orthonormal, and Σ ∈ Rm×n is diagonal. This decomposition enables a basis naturally aligned with the directions of task-specific updates ∆W . Unlike methods such as NOLA [Koohpayegani et al., 2024] relying on unstructured random bases, our foundationaligned directions allow a more compact representation of ∆W .
Building on the LoRA formulation, we approximate the individual factors A and B, aiming to find compressed counterparts Ã, B̃ such that: min ∥BA − B̃ Ã∥2F ,
s.t. ∥Ã∥0 ≤ kA ,
Ã,B̃
∥B̃∥0 ≤ kB ,
(2)
kA + kB = k,
To enrich the expressive power of this subspace, we construct randomized basis matrices by perturbing slices of the singular vectors: n oNA (i) MA = MA = V [:, Ii ] + ϵi , i=1 (3) n oNB (j) MB = MB = U [:, Jj ] + ϵj ,
where kA and kB represent budgets for à and B̃, respectively. This problem is challenging: counting the number of nonzero elements is non-convex, sparse element selection is combinatorial, and excessive sparsity may degrade accuracy. Achieving high compression without task performance loss thus requires careful subspace design and adaptive optimization.
j=1
where Ii and Jj are randomly sampled index sets, NA , NB are the number of basis candidates for A and B, respectively, and ϵi , ϵj are random matrices with each entry drawn i.i.d. from N (0, 1). These basis sets form a flexible pool of candidates for approximation.
2.2. Method: Subspace-Oriented Randomized Basis, Sparse Selection, and Reconstruction To solve (2), we propose SOLAR. A key insight motivating our approach is that ∆W predominantly resides in the subspace spanned by W , particularly in LoRA-based finetuning, where constraining the rank r ≪ min(m, n) forces ∆W to concentrate its variation along specific directions of W [Hu et al., 2021]. This alignment (i.e., the overlap in the principal directions of W and ∆W ) has been observed empirically and explained theoretically via neural tangent kernel (NTK) theory [Jacot et al., 2018; Malladi et al., 2023; Seleznova et al., 2023]. The left- and ⊤ right-singular alignments are measured as ∥UW U∆W ∥2F
2.2.2. S TEP 2: S PARSE S ELECTION OF BASES To enable more compact approximations, the LoRA update ∆W = BA is first projected into the subspace of W . Given the singular value decomposition W = U ΣV T , this projection is defined as ∆WProj = U T ∆W V = (U T B)(AV ) = BProj AProj , where AProj = AV and BProj = U T B represent the update components expressed in the basis of W . This 3
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
transformation retains all information when W is full-rank, and is particularly effective when ∆W is already aligned with the foundation subspace, a property commonly observed in LoRA-based fine-tuning. Under this projection, the update becomes ∆W = U ∆WProj V T . This approach leverages the inherent alignment between W and ∆W , enabling more efficient approximations with fewer basis elements than methods such as NOLA, which rely on unstructured random projections. Specifically, we approximate the projected LoRA factors AV and U T B using sparse linear combinations of the basis matrices:
min AV − α
NA X
β
We assume that (A1) the model is initialized with spectral initialization; (A2) the optimal update is low-rank; (A3) the change in the model’s weights from fine-tuning is wellbehaved according to the generation process in [Zhang et al., 2025a]; and (A4) the singular values of the projected update matrix exhibit Fast Spectrum Decay. These assumptions are well-established and frequently utilized in the literature for convergence analyses, as in previous works, such as [Zhang et al., 2025a; Martinsson & Tropp, 2020]. Theorem 1 [SOLAR Reconstruction Error Bound] Let ∆W ∗ be the optimal low-rank adapter, ∆W be the adapter learned via fine-tuning, and ∆W̃ be the adapter reconstructed by SOLAR. Under assumptions (A1)–(A4), the expected total error is bounded by h i E ∥∆W̃ − ∆W ∗ ∥F ≤ C1 + C2 , where C1 captures the fine-tuning error (depending on learning rate, training steps, and spectrum of ∆W ∗ ; see Appendix A), P 12 q rA 2 (∆W ) and C2 = 1 + NA −r σ + t t>r −1 A A 1 q P 2 rB 2 1 + NB −r + t>rB σt (∆W ) B −1 P 12 2 , where σt (∆W ) is the t-th sint>k σt (∆W ) gular value of the fine-tuned update ∆W , and rA , rB denote the effective ranks after moving to the random basis space. The SOLAR reconstruction error has two parts: the fine-tuning error (C1 ) and the compression error (C2 ). The compression error decreases with larger basis pools (NA , NB ) and higher sparsity budget (k). Details are in Appendix A.
2 (i)
αi MA
NB X
s.t. ∥α∥0 ≤ kA ,
,
i=1
min U T B −
2.3. Theoretical Analysis of Reconstruction Error
F
(4)
2 (j)
βj M B
j=1
,
s.t. ∥β∥0 ≤ kB .
F
A two-step strategy is employed to solve these NP-hard problems efficiently. The first step computes the unconstrained least squares solution to obtain coefficients α∗ and β ∗ . The second step applies hard thresholding to retain only the topk entries by magnitude based on the budgets kA and kB . 2.2.3. S TEP 3: B UDGET-AWARE R ECONSTRUCTION The approximated model update is then reconstructed using the selected topk bases, resulting in à and B̃ for A and B, respectively:
3. Experiments
! A≈
X
(i) αi∗ MA
V T,
We evaluate SOLAR through extensive experiments in three domains: 1) image classification with ViT-B/L in few-shot and full-data settings (Section 3.1); 2) instruction tuning on LLaMA-3 models using Alpaca and MMLU (Section 3.2); and 3) language generation with GPT-2 on E2E NLG (Section 3.3). Across all settings, SOLAR matches LoRA and NOLA in accuracy while reducing adapter size by up to 98%, offering a lightweight representation for model adaptation.
i∈SA
B≈U
X
(5)
(j)
βj∗ MB ,
j∈SB
where SA and SB are the selected topk index sets. Because the update reconstruction is performed within the subspace defined by W , this step ensures strong alignment with taskrelevant directions. The reconstruction balances accuracy and compression, with the sparsity budgets kA and kB controlling the number of active basis.
3.1. SOLAR on Vision Transformers We conduct few-shot image classification experiments using ViT-B and ViT-L [Dosovitskiy et al., 2020] foundation models, initialized with either supervised or self-supervised [He et al., 2022].
Adaptive Compression. SOLAR enables flexible allocation of sparsity budgets kA and kB , adapting to system constraints such as memory, storage, or bandwidth. This allows deployment on resource-constrained devices, with adapter size dynamically adjustable post-training. For instance, a server can send a compact adapter to low-memory clients and a richer version to more capable devices.
Experimental Setup. We compare SOLAR against LoRA [Hu et al., 2021] and NOLA [Koohpayegani et al., 2024]. Experiments are conducted on ViT-Base (ViT-B) and ViT4
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 1. Top-1 classification accuracy (%) of ViT-B and ViT-L on benchmark datasets under two settings: (1) few-shot (10 samples/class, 25 epochs) and (2) full-data (5 epochs). Results report mean ± std over 5 runs. SOLAR is applied with configuration method(N →k) , where N and k are in thousands. Model
Method
# Param
10
CIFAR-10 Full
ViT-B
Full-FT LoRA (r=4) NOLA SOLARr=4(4→1.6) SOLARNOLA(4→1.2)
86M 74K 48K 41K 32K
91.1±.8 92.3±.6 92.2±.6 92.3±.7 92.1±.7
94.6±.5 98.3±.2 94.7±.5 98.3±.4 94.5±.3
78.2±.7 81.8±.8 81.3±.8 81.5±.7 81.1±.6
87.7±.3 90.3±.4 86.6±.4 89.8±.2 85.4±.3
65.8±.9 72.4±.7 72.6±.5 71.8±.6 72.5±.6
85.2±.4 87.6±.3 85.9±.2 87.0±.5 85.4±.3
78.1±1.0 77.9±.9 78.4±.7 77.9±.8 78.3±.8
85.4±.6 88.8±.4 82.8±.5 87.9±.4 82.3±.5
ViT-L
Full-FT 303M LoRA (r=4) 197K LoRA (r=2) 98K NOLA 96K SOLARr=4(4→1.6) 82K SOLARr=2(1→0.3) 50K SOLARNOLA(4→1.2) 64K
90.2±.9 97.1±.5 96.6±.4 96.0±.8 97.0±.5 96.1±.8 95.8±.9
94.1±.6 98.7±.1 98.7±.1 97.4±.6 98.5±.3 98.2±.4 97.0±.4
86.2±.7 88.1±.7 88.0±.6 87.8±1.0 87.9±.8 87.4±.9 87.7±.8
87.7±.5 92.4±.3 92.9±.3 89.3±.5 91.4±.4 90.0±.5 89.3±.4
73.9±.8 81.8±.7 82.1±.7 82.5±.8 76.8±.7 77.0±.8 82.1±.7
85.5±.4 89.8±.2 90.0±.2 86.7±.4 87.1±.4 86.8±.6 86.6±.3
80.8±1.1 84.4±.8 83.8±.7 84.3±.9 78.7±.7 76.4±.9 84.1±.8
89.2±.6 91.8±.5 90.4±.3 86.7±.6 88.6±.5 87.6±.6 86.4±.6
10
Large (ViT-L) architectures. Supervised ViT models pretrained on ImageNet-21k [Deng et al., 2009] are obtained from Google’s official releases via the Hugging Face repository [Wolf et al., 2020; Research, 2025], and MAE models pretrained on ImageNet-1K are sourced from the Timm library [Wightman, 2025]. All experiments run on a single NVIDIA RTX 4090 GPU using PyTorch [Paszke, 2019] and HuggingFace libraries. In SOLAR, the compressed representation consists of (i) a random seed to regenerate the basis vectors, (ii) an encoded list of selected basis indices, and (iii) their coefficients. Reported trainable parameters include both projection coefficients and overhead (i.e., seed and index encoding). The MLP classifier head is dataset-specific and excluded from the parameter count unless noted.
CIFAR-100 Full
10
Food-101 Full
10
T-ImageNet Full
sample four training splits per dataset and report mean top-1 accuracy on the test split (or validation for ImageNet-1k). Experiments are repeated with different random seeds, and learning rates are tuned per dataset and model. Additional details are in the appendix. Results and Performance Analysis. We evaluate SOLAR on various vision benchmarks using foundation models, with results in Table 1. In the tables, configurations are denoted as SOLARmethod(N →k) , indicating that SOLAR is applied to a NOLA or LoRA model trained with rank r, using N bases per matrix (N = NA = NB ) and selecting the top-k bases by significance, where N and k are given in thousands. SOLAR consistently achieves competitive top-1 accuracy in few-shot (10 samples per class) and full-data settings while requiring far fewer trainable parameters than LoRA and NOLA. On ViT-B and ViT-L, SOLAR matches LoRA’s performance using up to 74% fewer parameters. For instance, applied to a LoRA (r = 2), bases NA = NB = 4000, and topk = 1600, SOLAR reduces fine-tuned parameters from 98K to 25K while maintaining comparable accuracy.
Evaluation Benchmarks. We fine-tune on standard image classification datasets: CIFAR-10 [Krizhevsky et al., 2009], CIFAR-100 [Krizhevsky et al., 2009], Food-101 [Bossard et al., 2014], Tiny-ImageNet [Le & Yang, 2015], ImageNet1K [Deng et al., 2009], Oxford Pets [Parkhi et al., 2012], SUN397 [Xiao et al., 2010], and CUB-200-2011 [Welinder et al., 2010]. Comparison Methods. We compare SOLAR with several baselines: Full Fine-Tuning (Full-FT), LoRA [Hu et al., 2021], and NOLA [Koohpayegani et al., 2024]. In Full-FT, all backbone parameters are updated. For LoRA, we apply low-rank adapters to the attention Query projection matrices, with a rank of 4 for ViT-B and either 1 or 4 for ViT-L. For NOLA, following [Koohpayegani et al., 2024], adapters are inserted into MLP layers using 1000 random basis vectors for each of the A and B matrices. All models are trained with cross-entropy loss. For full-data settings, we train 5 epochs with batch size 128; for few-shot settings (10 samples per class), 25 epochs with batch size 16, emphasizing low-data efficiency relevant to real-world and distributed scenarios. To account for variance from limited data, we
Beyond parameter reduction, SOLAR improves storage efficiency. Table 2 reports mean and standard deviation over 5 runs on four additional datasets using ViT-B, quantifying the bit-level footprint assuming 32-bit precision during training. We apply 8-bit quantization to SOLAR after topk parameter selection. While LoRA (r = 1) requires 74KB of adapter parameters, SOLAR reduces this to 8KB (89% reduction). These extreme compressions incur only minor accuracy drops, showing SOLAR enables fine-grained control of model size to meet strict constraints and offers a flexible tradeoff between footprint and performance. In addition to reducing parameter and storage footprints, SOLAR remains highly robust under quantization. As shown in Table 3, reducing coefficient precision from 32-bit to 4-bit 5
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 2. Additional evaluation on vision datasets using ViT-B. The table shows bit-level representation footprint (32-bit baseline) and Top-1 accuracy. All models are trained for 10 epochs. Method
Byte Footprint
Oxford Pets
SUN397
CUB-200
ImageNet-1K
LoRA (r=1) NOLA SOLARr=1(2→0.2)
74KB 48KB 8KB (89% ↓)
93.0±.3 90.4±.5 92.6±.4
74.3±.2 61.7±.4 73.9±.2
84.7±.2 79.4±.4 84.2±.3
81.5±.4 77.4±.3 81.3±.2
Table 3. Effect of quantization on SOLARr=4(4→1.6) performance. Table 5. Model representation efficiency for LLaMA models. SOViT-L-MAE fine-tuned on CIFAR-10. LAR compresses LoRA adapter updates across various model sizes. Method Quant. Accuracy Byte Footprint SOLAR
32-bit 16-bit 8-bit 4-bit
86.7±-.3 86.5±-.3 85.9±-.4 84.8±-.6
Model
319KB 166KB 89KB 50KB
LLaMA-3.2 1B
LLaMA-2 13B (4-bit)
Method
LoRA NOLA SOLAR LoRA NOLA SOLAR r=8 1000 r = 8(4 → 1.2) r=1 1000 r = 1(1 → 0.3)
# Params
852K
64K
81K (90% ↓)
Val Loss 1.51 MMLU Acc 30.1
1.87 25.9
1.52 28.3
819K 140K 1.05 54.5
1.29 51.8
51K (94% ↓) 1.05 54.5
Table 4. Effect of rank and adapter placement in SOLARr=4(4→1) . Accuracy (%) on CIFAR-100 using ViT-B. Rank
Q
K
V
QV
QKV
1 2 4 8 16
87.0 87.5 87.8 88.1 87.9
85.5 85.7 86.1 86.0 86.0
86.6 87.4 87.5 87.4 87.1
88.3 88.6 89.0 89.1 89.0
90.1 90.5 90.6 90.7 90.6
Table 6. Performance and parameter efficiency on E2E NLG using GPT-2 Small and Medium. All methods use rank-4 adapters applied to the Query and Value projections. Method
GPT-2 Small MET # Params
GPT-2 Medium MET # Params
Full-FT LoRA (r=4) NOLA
28.4 29.7 29.1
46.2 47.2 46.8
124M 147K 48K
355M 393K 350K
SOLAR (r=4, 1 → 0.3) 29.7 15K (90% ↓) 46.4 30K (92% ↓) SOLAR (r=1, 0.1 → 0.1) 26.1 4K (97% ↓) 44.8 9K (98% ↓)
incurs less than a 2% accuracy drop on ViT-L-MAE (CIFAR10, 10-shot). We further evaluate the effect of adapter rank and placement (Table 4), observing that performance improves with rank up to 8 (with higher ranks requiring more time to converge), and that the Query (Q) projection yields the highest gains.
NOLA [Koohpayegani et al., 2024]. LoRA uses rank r = 8 for LLaMA-3 1B and r = 1 for the 13B model. NOLA follows its original configuration, with 1000 random basis vectors per matrix [Koohpayegani et al., 2024]. For the 13B model, we apply 4-bit quantization to all methods (LoRA, NOLA, and SOLAR). The reported trainable parameters include learned coefficients and overhead for basis indexing. All experiments use gradient checkpointing, and learning rates are tuned separately per model and method to ensure a fair comparison.
3.2. SOLAR on LLaMA Experimental Setup. We apply SOLAR to LLaMA-3 models of size 1B–13B. All models are fine-tuned using adapters in the query and value projections across all transformer layers. For the 1B model, we use LoRA with rank 8; for the 31B model, we use LoRA with rank 1. To reduce GPU memory usage for large-scale models, we quantize the 13B model using 4-bit NF4 quantization through the BitsAndBytes library [Dettmers et al., 2021; Dettmers, 2025]. Further implementation details and hardware configurations are provided in the Appendix.
Results and Performance Analysis. Table 5 reports results across model sizes. SOLAR matches LoRA in Alpaca validation loss and MMLU [Hendrycks et al., 2020] accuracy while reducing trainable adapter parameters by up to 94%. For example, on LLaMA-3.2 13B, SOLAR cuts the adapter size from 819K to 51K without accuracy loss.
Evaluation Benchmarks. All models are fine-tuned on the Stanford Alpaca [Taori et al., 2023] dataset for instructionfollowing and evaluated on its validation loss. We also assess generalization to out-of-distribution tasks using the MMLU benchmark [Hendrycks et al., 2020].
3.3. SOLAR on GPT-2 Experimental Setup. We evaluate our method on GPT-2 [Radford et al., 2019] base and medium models fine-tuned on the E2E NLG dataset [Novikova et al., 2017] using LoRA. The models are trained for 5 epochs using a batch size of 8 and a learning rate of 0.1. LoRA is applied to the self-
Comparison Methods. We compare SOLAR with PEFT baselines, including LoRA [Hu et al., 2021] and 6
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Figure 3. Performance vs. Cost: On ViT-B (r = 4), SOLAR demonstrates a trade-off between parameter count and performance, achieving strong results with far fewer parameters than LoRA. Figure 2. Subspace similarity between the W and ∆W matrices (Q, K, V) from the first layer of the ViT-B model using LoRA with rank r = 4.
Table 7. Runtime Overhead: LoRA (10 epochs) vs. SOLAR posttraining on ViT-B. SOLAR adds under 2% total overhead.
Dataset
attention Query and Value projection, with a rank of r = 4. After training, we apply SOLAR to compress the LoRA adapter updates.
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet ImageNet-1K
Evaluation Benchmarks. We use the E2E NLG dataset to evaluate generative quality. Generated outputs are assessed using METEOR [Banerjee & Lavie, 2005] metric. We report LoRA, NOLA, and SOLAR performance.
SOLAR (s)
Overhead (%)
1176 1165 3480 2081 56634
14 14 67 15 155
1.19 1.20 1.92 0.72 0.27
Table 8. Runtime Overhead: LoRA (10 epochs) vs. SOLAR posttraining on ViT-B across vision datasets. Times in seconds.
Results and Performance Analysis. Table 6 summarizes results on the E2E NLG dataset using GPT-2 Small and Medium models. SOLAR achieves competitive METEOR scores compared to LoRA and NOLA, while substantially reducing adapter size. On GPT-2 Medium, SOLAR reduces adapter representation size from 393K (LoRA) to 30K parameters with minimal performance loss. Applied to rank-1 LoRA, it achieves a 98% reduction, demonstrating strong compression capability.
Dataset
LoRA
SOLAR
Overhead (%)
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet ImageNet-1K
1176 1165 3480 2081 56634
14 14 67 15 155
1.19 1.20 1.92 0.72 0.27
and the selected top-k components. Each LoRA matrix A and B requires 4 × 768 = 3072 parameters. We observe that increasing k improves expressiveness. Moreover, a larger basis pool enhances performance by increasing the likelihood of capturing directions aligned with the fine-tuned model subspace. As shown in Figure 3, larger pools yield higher accuracy by enabling more precise reconstruction. This trade-off confirms Theorem 1: increasing N or sparsity k reduces compression error C2 .
3.4. Discussion and Analysis on SOLAR Performance and Efficiency Subspace Analysis. We analyze the subspace similarity between the foundation model’s weights W and the LoRA update ∆W with rank r = 4 (see Figure 2). Let ⊤ ⊤ W = UW ΣW VW and ∆W = U∆W Σ∆W V∆W denote their SVDs. To quantify subspace alignment, we define the (i) ⊤
LoRA (s)
(j)
similarity function as ϕ(W, ∆W, i, j) = ∥UW U∆W ∥2F , (i) (j) where UW and U∆W are matrices formed by the i and j left singular vectors. Figure 2 shows that the fine-tuned model emphasizes directions already present in the foundation model, supporting prior observations that LoRA updates lie in low-dimensional, structured subspaces [Hu et al., 2021; Zhang et al., 2025b]. SOLAR exploits this alignment in its basis pool, explaining its performance advantage over NOLA.
SOLAR Overhead and Runtime Efficiency. As a posttraining method, SOLAR introduces negligible runtime overhead and does not interfere with fine-tuning. For instance, fine-tuning LLaMA-3.2 1B with LoRA on Tiny-ImageNet took 2081 seconds, while SOLAR, including random basis generation, convex least-squares solving, and topk selection, took only 15 seconds (under 0.72% of training time). These operations are computationally lightweight, as shown in Table 8, confirming SOLAR’s practical efficiency.
Effect of Basis Pool Size and Communication Budget. To evaluate SOLAR’s trade-off, we analyze basis pool size
Limitations and Future Work. As a post-hoc method, SO7
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
PEFT Compression Techniques. To mitigate PEFT costs, pruning [Han et al., 2024; Ilhan et al., 2024] and quantization [Chen et al., 2024; Hubara et al., 2021] have been explored. These reduce model size but require careful tuning or retraining, are less effective under severe bandwidth limits, and are mainly optimized for full-model compression, limiting applicability to adapters. Adapter updates are highly redundant and lie in low-dimensional subspaces [Hu et al., 2021; Yadav et al., 2023; Wu et al., 2024], motivating post-training compression. Methods like ComPEFT [Yadav et al., 2023], BitDelta [Liu et al., 2024], Delta-CoMe [Ping et al., 2024], and DeltaZip [Yao et al., 2025] compress adapter weights after fine-tuning but rely on heuristics, taskspecific tuning, or training integration, reducing flexibility. Other approaches alter fine-tuning itself: VeRA [Kopiczko et al., 2023] employs a shared random basis, SVFT [Lingam et al., 2024] learns sparse coefficients for an SVD-based basis, and EigenLoRAx [Kaushik et al., 2025] builds a PCA basis from many pre-trained adapters. In contrast, SOLAR is a post-hoc, training-free utility that compresses any adapter, providing a complementary plug-and-play solution.
LAR’s performance is limited by the base adapter, and its hyperparameters (N and k) may need per-task tuning to optimize the compression-accuracy trade-off. While it shows strong results on vision and language tasks, its effectiveness on other modalities (audio, time series, or multimodal data) remains untested. Future work will extend SOLAR to these areas and evaluate its performance in other environments.
4. Background and Related Works Transformers in NLP and Vision. Transformers [Vaswani et al., 2017], are now the standard in NLP for modeling longrange dependencies via self-attention [Raiaan et al., 2024]. Models such as LLaMA [Touvron et al., 2023], BERT [Devlin et al., 2019], and GPT [Radford et al., 2018] build on this structure to achieve strong results across diverse benchmarks. In vision, ViT [Dosovitskiy et al., 2020] treats image patches as tokens, making Transformers a unifying backbone across modalities. Parameter-Efficient Fine-Tuning (PEFT). As transformers scale, task-specific fine-tuning becomes computationally intensive. PEFT methods mitigate this by updating only a subset of parameters. LoRA [Hu et al., 2021] introduces trainable low-rank matrices per layer, typically modifying <1% of weights, while NOLA [Koohpayegani et al., 2024] re-parameterizes these as linear combinations of random bases, decoupling parameters from rank and architecture. Yet PEFT gains often fall short in deployment, especially on edge, mobile, and federated settings with communication and storage bottlenecks. Adapting GPT-2 (117M) on-device may still require gigabytes of transfer and petaflop-scale computation per round [Wang et al., 2025], with updates taking seconds to transmit and hours to process on lowpower hardware (e.g., Jetson TX2).
5. Conclusion Adapter-based fine-tuning methods such as LoRA significantly reduce the cost of adapting large models. However, in distributed and on-device settings, communication and storage overheads remain a major bottleneck. To address this, we introduce SOLAR, a lightweight post-training compression method that reparameterizes adapter updates as sparse combinations of structured basis vectors aligned with the foundation model’s latent subspace. SOLAR substantially reduces adapter size and transmission cost without altering the training process or model architecture.
Challenges of PEFT. As models grow, adapter overhead scales rapidly. Even modest adapters (e.g., 7M parameters for a 7B model at rank 16) accumulate significant costs across users, tasks, or training rounds [Xu et al., 2023b]. A 1% adapter for LLaMA-2 70B adds 700M parameters; for GPT-3 (350B), 3.5B—tens of gigabytes in FP32. Such costs are infeasible in personalized or federated settings, where hundreds of adapters may be exchanged or stored per user [Zhang et al., 2024]. While PEFT leverages the low intrinsic dimensionality of task adaptation [Hu et al., 2021], deployment remains inefficient. It has been shown that BERT fine-tuning on MRPC [Dolan & Brockett, 2005] requires only 1,861 degrees of freedom out of 110M, highlighting redundancy in full-rank updates [Aghajanyan et al., 2020]. Yet even small adapters impose substantial overhead on massive models [Xu et al., 2023a; Lialin et al., 2023]. Hence, the true bottleneck is adapter size, not fine-tuning efficiency [Jie et al., 2023], motivating flexible post-training compression to reduce footprint without altering training.
References Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020. Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446–461. Springer, 2014. Chen, M., Shao, W., Xu, P., Wang, J., Gao, P., Zhang, 8
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
K., and Luo, P. Efficientqat: Efficient quantizationaware training for large language models. arXiv preprint arXiv:2407.11062, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790– 2799. PMLR, 2019.
Dettmers, T. Bitsandbytes: 8-bit optimizers and quantization. https://github.com/TimDettmers/ bitsandbytes, 2025. Accessed: 15-May-2025.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021.
Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pp. 4466–4475. PMLR, 2021.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115, 2023.
Ilhan, F., Su, G., Tekin, S. F., Huang, T., Hu, S., and Liu, L. Resource-efficient transformer pruning for finetuning of large models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16206–16215, 2024.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186, 2019.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Dolan, B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005.
Jie, S., Wang, H., and Deng, Z.-H. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17217–17226, 2023.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Karimi Mahabadi, R., Henderson, J., and Ruder, S. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022–1035, 2021.
Gao, C. and Zhang, S. Q. Dlora: Distributed parameterefficient fine-tuning solution for large language model. arXiv preprint arXiv:2404.05182, 2024. Halko, N., Martinsson, P.-G., and Tropp, J. A. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
Kaushik, P., Vaidya, A., Chaudhari, S., and Yuille, A. Eigenlorax: Recycling adapters to find principal subspaces for resource-efficient adaptation and inference. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 649–659, 2025.
Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024.
Koohpayegani, S. A., Navaneet, K., Nooralinejad, P., Kolouri, S., and Pirsiavash, H. Nola: Compressing lora using linear combination of random basis. ICLR 2024, 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009, 2022.
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023. 9
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Ping, B., Wang, S., Wang, H., Han, X., Xu, Y., Yan, Y., Chen, Y., Chang, B., Liu, Z., and Sun, M. Delta-come: Training-free delta-compression with mixed-precision for large language models. arXiv preprint arXiv:2406.08903, 2024.
Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B. Controlling textto-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320– 79362, 2023.
Lialin, V., Deshpande, V., and Rumshisky, A. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647, 2023.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
Lingam, V. C., Neerkaje, A., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J., Choi, E., Dimakis, A., Bojchevski, A., and Sanghavi, S. Svft: Parameter-efficient fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 37:41425–41446, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Liu, J., Xiao, G., Li, K., Lee, J. D., Han, S., Dao, T., and Cai, T. Bitdelta: Your fine-tune may only be worth one bit. Advances in Neural Information Processing Systems, 37:13579–13600, 2024.
Raiaan, M. A. K., Mukta, M. S. H., Fatema, K., Fahad, N. M., Sakib, S., Mim, M. M. J., Ahmad, J., Ali, M. E., and Azam, S. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE access, 12:26839–26874, 2024.
Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng, S., et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243, 2023.
Research, G. Vision Transformer Models on Hugging Face. https://huggingface.co/google, 2025. Accessed: 06-May-2025.
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pp. 23610– 23641. PMLR, 2023.
Seleznova, M., Weitzner, D., Giryes, R., Kutyniok, G., and Chou, H.-H. Neural (tangent kernel) collapse. Advances in Neural Information Processing Systems, 36:16240– 16270, 2023.
Martinsson, P.-G. and Tropp, J. A. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403–572, 2020.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model, 2023.
Mhanna, E. and Assaad, M. Countering the communication bottleneck in federated learning: A highly efficient zero-order optimization technique. Journal of Machine Learning Research, 25(418):1–53, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Novikova, J., Dušek, O., and Rieser, V. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.09254, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498–3505. IEEE, 2012.
Wang, S., Liu, J., Xu, H., Yan, J., and Gao, X. Efficient federated fine-tuning of large language models with layer dropout. arXiv preprint arXiv:2503.10217, 2025.
Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. Caltech-ucsd birds 200. 2010. 10
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Wightman, R. timm: PyTorch Image Models. https://github.com/huggingface/ pytorch-image-models/tree/main/timm, 2025. Accessed: 06-May-2025. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45, 2020. Wu, T., Wang, J., Zhao, Z., and Wong, N. Mixtureof-subspaces in low-rank adaptation. arXiv preprint arXiv:2406.11909, 2024. Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010. Xu, L., Xie, H., Qin, S.-Z. J., Tao, X., and Wang, F. L. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148, 2023a. Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. Qa-lora: Quantizationaware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717, 2023b. Yadav, P., Choshen, L., Raffel, C., and Bansal, M. Compeft: Compression for communicating parameter efficient updates via sparsification and quantization. arXiv preprint arXiv:2311.13171, 2023. Yao, X., Hu, Q., and Klimovic, A. Deltazip: Efficient serving of multiple full-model-tuned llms. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 110–127, 2025. Zhang, C., Long, G., Zhou, T., Zhang, Z., Yan, P., and Yang, B. When federated recommendation meets cold-start problem: Separating item attributes and user interactions. In Proceedings of the ACM Web Conference 2024, pp. 3632–3642, 2024. Zhang, Y., Liu, F., and Chen, Y. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. arXiv preprint arXiv:2502.01235, 2025a. Zhang, Y., Liu, F., and Chen, Y. One-step full gradient suffices for low-rank fine-tuning, provably and efficiently. arXiv preprint arXiv:2502.01235, 2025b.
11
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Appendix A. Proof of Theorem 1 Let ∆W ∗ ∈ Rm×n denote the optimal adapter for the downstream task, ∆W the adapter obtained by LoRA fine-tuning, f the SOLAR reconstruction. Let ∆Wproj denote the projection of ∆W onto the SOLAR bases (i.e., bases that are and ∆W constructed from the SVD of the foundation model’s weights, combined with randomized perturbations). Our proof relies on the following standard assumptions from the literature on parameter-efficient fine-tuning and randomized numerical linear algebra: (A1) Spectral Initialization: The LoRA adapter matrices A and B are initialized using the spectral initialization strategy from Zhang et al. [2025a]. (A2) Low-Rank Update: The optimal task-specific update ∆W ∗ is approximately low-rank, with rank r∗ < min{m, n} [Zhang et al., 2025a]. (A3) Well-Behaved Data: The training data follows the generation process outlined in Zhang et al. [2025a], where input features are drawn from an isotropic sub-Gaussian or Gaussian distribution. (A4) Fast Spectrum Decay: The projected update matrix ∆Wproj exhibits spectral decay, meaning its tail singular values are small [Martinsson & Tropp, 2020]. First, we decompose the total error using the triangle inequality. The total error, ∥∆W̃ − ∆W ∗ ∥F , is the distance between the SOLAR-reconstructed adapter and the optimal adapter. This is bounded by the sum of the Training Error and the Compression Error: ∥∆W̃ − ∆W ∗ ∥F ≤ ∥∆W̃ − ∆W ∥F + ∥∆W − ∆W ∗ ∥F | {z } | {z } Compression Error
(6)
Training Error
Here, the first term, ∥∆W̃ − ∆W ∥F , is the compression error introduced by SOLAR’s approximation. The second term, ∥∆W − ∆W ∗ ∥F , is the training error from the underlying LoRA fine-tuning process itself. We will bound each term separately. The analysis of the training error for LoRA adapters is non-trivial and has been extensively studied. We directly leverage the results from Zhang et al. [2025a], showing that under Assumptions (A1)-(A3), LoRA trained with gradient descent converges to the optimal low-rank adapter ∆W ∗ . Their analysis provides the following bound on the training error after t steps: t √ ηλr∗ ∗ ∗ ∥∆W − ∆W ∥F ≤ 2r 1 − λr∗ , (7) 64κ where r∗ is the rank of the optimal update ∆W ∗ , κ is its condition number, λr∗ is its r∗ -th singular value, and η is the learning rate. This bound, derived under the specified spectral initialization and data concentration assumptions, demonstrates that the fine-tuned adapter ∆W gets exponentially closer to the optimal adapter ∆W ∗ as training progresses. SOLAR reconstructs the adapter as a sparse coefficientization over these perturbed bases: f = ∆W
NB X NA X
(i)
(j)
β i α j MB MA .
(8)
i=1 j=1
Following the randomized rangefinder formulation [Halko et al., 2011; Martinsson & Tropp, 2020], we construct the sketch matrices for both the column and row spaces of the LoRA-style adapter update ∆W as YB = ∆W ⊤ ΩB ∈ Rn×NB .
YA = ∆W ΩA ∈ Rm×NA ,
(9)
Each column of YA represents the action of ∆W on a random probe vector drawn from the right-basis pool ΩA , effectively sampling the column space of ∆W . Similarly, each column of YB captures random projections of the row space of 1
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
∆W . These sketches compactly encode the dominant directions of ∆W without explicitly computing its singular value decomposition. (i)
(j)
The Gaussian perturbations in MA = V:,Ii + ϵi and MB = U:,Jj + ϵj play an important theoretical and practical role. First, they ensure that the composite sketching matrices ΩA and ΩB satisfy the sub-Gaussian concentration and Johnson–Lindenstrauss properties required for the probabilistic error bounds in randomized numerical linear algebra [Halko et al., 2011]. Second, adding small isotropic noise expands the effective span of the sampled singular directions, preventing over-alignment with any single dominant mode and improving numerical stability when the singular spectrum of ∆W decays slowly. Finally, this perturbation acts as a regularizer that mitigates sampling bias inherited from the foundation model’s specific singular subspace, ensuring broader coverage of the subspace where fine-tuned updates lie. We then compute orthonormal bases for the column spans of these sketches: QA = orth(YA ) ∈ Rm×qA ,
QB = orth(YB ) ∈ Rn×qB ,
(10)
where rA = rank(QA ) ≤ min(m, NA ),
rB = rank(QB ) ≤ min(n, NB ).
By construction, range(QA ) = range(YA ) and range(QB ) = range(YB ). In the terminology of randomized numerical linear algebra, this process corresponds to the rangefinder step, which identifies low-dimensional subspaces that approximate the dominant column and row spaces of ∆W . Finally, we define the two-sided (bi-rangefinder) projection as ⊤ PNA ,NB (∆W ) := QA Q⊤ A ∆W QB QB .
(11)
This projection provides a low-rank approximation to ∆W using orthonormal subspaces inferred from randomized sketches. Geometrically, PNA ,NB (∆W ) captures the principal subspace of ∆W identified by ΩA and ΩB , offering an efficient surrogate for the optimal SVD-based projection U1 U1⊤ ∆W V1 V1⊤ while retaining probabilistic error guarantees [Halko et al., 2011; Martinsson & Tropp, 2020]. We bound the bi-projection error by splitting it into two one-sided parts using projector non-expansiveness (∥QA Q⊤ A X∥F ≤ ∥X∥F ): ⊤ ⊤ ⊤ ⊤ ∥∆W − QA Q⊤ A ∆W QB QB ∥F ≤ ∥∆W − QA QA ∆W ∥F + ∥QA QA (∆W − ∆W QB QB )∥F ⊤ ≤ ∥∆W − QA Q⊤ A ∆W ∥F + ∥∆W − ∆W QB QB ∥F .
(12)
Each addend is a standard one-sided rangefinder error. By Theorem 10.5 of Halko et al. [2011] (Frobenius form) with oversampling NA > rA + 1 and NB > rB + 1, !12 12 X rA ⊤ 2 E ∥∆W − QA QA ∆W ∥F ≤ 1 + σt (∆W ) , (13) NA − rA − 1 t>r A
⊤ E ∥∆W − ∆W QB QB ∥F ≤ 1 +
rB NB − rB − 1
21
!12 X
σt (∆W )2
.
Combining equation 12–equation 14 yields the expected two-sided projection error bound: 12 X !12 21 rA rB 2 σt E ∥∆W − PNA ,NB (∆W )∥F ≤ 1 + + 1+ NA − r A − 1 NB − r B − 1 t>r A
(14)
t>rB
!12 X
σt2
.
(15)
t>rB
(When desired, power iterations can be incorporated on either side to sharpen the spectral decay and constants [Halko et al., 2011; Martinsson & Tropp, 2020].) After projection, SOLAR enforces sparsity by retaining only the top-k basis pairs in equation 8. Let the singular values of PNA ,NB (∆W ) be {σ̃t }, we have: !12 X 2 f − PN ,N (∆W )∥F ≤ ∥∆W σ̃t . (16) A
B
t>k
2
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Moreover, orthogonal projections are contractions in Frobenius norm and cannot increase tail energy, hence X X σ̃t2 ≤ σt (∆W )2 . t>k
(17)
t>k
Adding and subtracting PNA ,NB (∆W ) and using equation 15–equation 17, we obtain f − ∆W ∥F ≤ E ∥∆W − PN ,N (∆W )∥F + E ∥∆W f − PN ,N (∆W )∥F E ∥∆W A B A B 1 ! 12 X 21 X !12 2 rA rB 2 σt + 1+ σt2 ≤ 1+ NA − rA − 1 N − r − 1 B B t>rA t>rB !21 X + . σt2
(18)
(19)
t>k
Combining the decomposition with equation 19 and the LoRA training bound equation 7, we conclude !21 !12 12 X 12 X rA rB f − ∆W ∗ ∥F ≤ 1 + E ∥∆W + 1+ σ2 σ2 t
NA −rA −1
|
t
NB −rB −1
t>rA
t>rB
{z
projection error
!12 X
+
σt2
+
t>k
|
{z
}
√
t r∗ λr ∗ . 2r∗ 1 − ηλ 64κ | {z }
(20)
training error
}
sparsification error
Each term in equation 20 can be driven to zero under mild conditions: (i) the projection error vanishes as NA , NB grow so that rA , rB reach the true (or effective) rank of ∆W (then the corresponding spectral tails are zero); (ii) the sparsification error vanishes when k exceeds the numerical rank of PNA ,NB (∆W ); and (iii) the training error decays to zero as t → ∞ under (A1)–(A3) by equation 7. Consequently, with sufficient sampling (NA , NB ), sparsity budget (k), f − ∆W ∗ ∥F → 0. E ∥∆W
B. Implementation Details All models are implemented using PyTorch [Paszke, 2019], with HuggingFace Transformers [Wolf et al., 2020] for LLaMA and GPT-based models, and Timm [Wightman, 2025] for ViT-based vision backbones. Training and evaluation are performed on NVIDIA A100 and RTX 4090 GPUs. For all vision experiments, we use ViT-B and ViT-L as base encoders. For language models, we use GPT-2 and LLaMA-3 (1B, 3B, 8B). LoRA is applied to the query and value projections. SOLAR operates post-training by compressing the PEFT adapter matrices. All experiments are conducted under a fixed random seed for reproducibility. The implementation code for S OLAR, along with scripts used to reproduce the experiments, is included in the supplementary material and also available at https://github.com/mahmoudsajjadi/SOLAR.
C. Dataset Details We summarize dataset statistics in Table 9, including number of training samples and class counts. We summarize dataset statistics used in the LLM experiments in Table 10, covering instruction tuning (Section 3.2) and language generation tasks (Section 3.3). The table includes the number of training samples, average sequence lengths, and the model-specific context in which each dataset is used in the experiments.
D. Representation Cost Details: Parameters and Storage To quantify SOLAR’s compression benefit, we detail the number of adapter parameters and byte-level footprint across ViT-B, ViT-L, LLaMA, and GPT-2 models. We compare LoRA, NOLA, and SOLAR under adapter rank (r = 4). Tables 11 through 16 provide full parameter breakdowns. Byte-level analysis is presented in Table 14. 3
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 9. Dataset statistics used in experiments. Each dataset includes the number of training samples and classes.
Dataset CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet ImageNet-1K
Training Samples
Number of Classes
50,000 50,000 75,750 100,000 1,281,167
10 100 101 200 1,000
Table 10. Dataset statistics in LLM experiments.
Dataset
Samples
Avg. Seq. Length
Context
Stanford Alpaca MMLU E2E NLG
52,000 15,858 42,000
∼256 tokens ∼200 tokens ∼35 tokens
LLaMA-3 instruction tuning LLaMA-3 Generalization evaluation GPT-2 generation fine-tuning
ViT. For vision backbones, Table 11 and Table 12 report the number of representation parameters for query projections (Q) and classifier heads. In the experiments presented in the main paper, the classifier head parameters are excluded from comparison since they are identical across all methods following [Koohpayegani et al., 2024]. NOLA’s parameter footprint for MLP projections is shown in Table 13 (following the setup in [Koohpayegani et al., 2024]). Byte-level storage comparisons across quantization, used to produce Table 2 and Table 3 in the main paper, are provided in Table 14. Table 11. Number of representation parameters for ViT-B (Rank = 4). Each row reports the parameter count for query projections and the classifier head using SOLAR and LoRA across different datasets. The classifier head parameter count is shared across methods and is computed as (num_classes × 768 + num_classes). For SOLAR, the query projection count corresponds to: number of layers × (topk coefficients for A + topk coefficients for B + encoded basis for A + encoded basis for B) +1 (seed value). All SOLAR rows follow the form N → topk where N is the original subspace size. For LoRA, the query projection count corresponds to: number of layers × (input dimension × rank for A + rank × output dimension for B), where rank is 4.
Method
Dataset
SOLAR
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet
LoRA
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet
Query (Q) 12 ×
(1600 + 1600) + 4000+4000 32
Classifier Head
41,401 41,401 41,401
10 × 768 + 10 = 7,690 100 × 768 + 100 = 76,900 101 × 768 + 101 = 77,669 200 × 768 + 200 = 154,000
12 × [(768 × 4) + (4 × 768)] = 73,728 73,728 73,728 73,728
10 × 768 + 10 = 7,690 100 × 768 + 100 = 76,900 101 × 768 + 101 = 77,669 200 × 768 + 200 = 154,000
+ 1 = 41,401
LLMs. For language models, parameter counts for adapter layers are detailed in Table 15 for LLaMA and in Table 16 for GPT-2 variants.
E. Additional Experimental Results This section provides supplementary experimental results to further validate the claims made in the main paper. We present detailed performance metrics for additional model scales and include a crucial ablation study that compares SOLAR against a parameter-matched LoRA baseline. E.1. Performance on Intermediate-Scale LLaMA Models Table 17 extends our analysis to the LLaMA-3.2 3B and LLaMA-3.1 8B models, demonstrating SOLAR’s consistent efficiency and performance on intermediate-scale architectures. The results show that SOLAR maintains the performance of 4
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 12. Number of representation parameters for ViT-L (Rank = 4). Each row shows the parameter counts for Query projections and the classifier head using SOLAR and LoRA across different datasets. The classifier head parameter count is shared across methods and is calculated as (num_classes × 1024 + num_classes).
Method
Dataset
SOLAR
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet
LoRA
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet
Query (Q) 24 ×
(500 + 500) + 1000+1000 32
Classifier Head
25,501 25,501 25,501
10 × 1024 + 10 = 10,250 100 × 1024 + 100 = 102,500 101 × 1024 + 101 = 103,625 200 × 1024 + 200 = 204,800
24 × [(1024 × 4) + (4 × 1024)] = 196,608 196,608 196,608 196,608
10 × 1024 + 10 = 10,250 100 × 1024 + 100 = 102,500 101 × 1024 + 101 = 103,625 200 × 1024 + 200 = 204,800
+ 1 = 25,501
Table 13. Number of representation parameters for ViT-B (Rank = 4). Each row shows the parameter counts for MLP projections (for NOLA) and classifier head across datasets. The classifier head parameter count is shared across methods and is calculated as (num_classes × 768 + num_classes).
Method
Dataset
NOLA
CIFAR-10 CIFAR-100 Food-101 Tiny-ImageNet
MLP
Classifier Head
12 × 2 × 2 × 1000 + 1 = 48,001 48,001 48,001 48,001
10 × 768 + 10 = 7,690 100 × 768 + 100 = 76,900 101 × 768 + 101 = 77,669 200 × 768 + 200 = 154,000
the original LoRA adapters while achieving parameter reductions of over 90%. E.2. Compression of Adaptive-Rank PEFT Methods (AdaLoRA) To evaluate SOLAR on more recent PEFT methods, we applied it to AdaLoRA, which produces adaptive-rank adapter matrices (A and B). SOLAR compresses these trained adapters post-hoc, using an initial rank of r = 8 and a target average rank of r = 1 on LLaMA-3.2 3B and LLaMA-2 13B. As shown in Table 18, SOLAR significantly reduces adapter parameters while preserving MMLU performance. E.2.1. E XPERIMENTS WITH 2-B IT Q UANTIZATION To further validate SOLAR’s robustness to aggressive quantization, we conducted additional experiments with 2-bit quantization on LLaMA-2 13B and LLaMA-3.1 8B. The results, summarized in Table 19, confirm that SOLAR remains effective while drastically reducing parameter counts. E.3. Extreme Compression In this section, we report additional experiments demonstrating SOLAR’s ability to achieve extreme compression while retaining competitive accuracy. These results complement the main paper by highlighting scenarios where communication and storage constraints are especially strict (e.g., distributed or on-device learning). Table 20 shows evaluations on four vision datasets using ViT-B under different compression budgets. We quantify the bit-level representation footprint assuming 32-bit precision during training and apply 8-bit quantization to the SOLAR coefficients after top-k selection. Compared to LoRA (r = 1), SOLAR reduces the adapter footprint by up to 99% (from 74KB to 0.4KB) with only minor drops in accuracy. These results illustrate that SOLAR enables fine-grained tradeoffs between accuracy and storage cost under extreme compression budgets. 5
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 14. Byte-level footprint of representation parameters for ViT-B and ViT-L using LoRA and SOLAR. Each value reflects the total number of bytes required to store adapter updates (excluding classifier heads). For LoRA, storage is computed as: number of layers × rank × output dimension for B + input dimension × rank for A × precision in bytes (e.g., 4 bytes for 32-bit float). For SOLAR, storage is computed as: number of layers × topk coefficients for A + topk coefficients for B + encoded basis vectors for A + encoded basis for B × precision in bytes, plus 1 byte to store a random seed. For example, the row "500 → 50" denotes that 500-dimensional subspaces are sparsified to top-k = 50 coefficients, with encoded bases represented at 1 bit per element (8 elements per byte).
Method
Representation Footprint (Bytes) 12 × [(768 × 768)] × 4 = 73,728 × 1) + (1500 12 × (50 + 50) + 8 × 1 + 1 = 1,951 12 × (10 + 10) + 100 × 1 + 1 = 391 8
LoRA (r=1) SOLAR for ViT-B 8Bit (r=1, 500 → 50) SOLAR for ViT-B 8Bit (r=1, 100 → 10) LoRA (r=4) SOLAR for ViT-L 32Bit (r=4, 4000 → 1600) SOLAR for ViT-L 16Bit (r=4, 4000 → 1600) SOLAR for ViT-L 8Bit (r=4, 4000 → 1600) SOLAR for ViT-L 4Bit (r=4, 4000 → 1600)
24 ×[(1024 × 4) + (4 × 1024)] × 4 = 786,432 4000 24 × (1600 + 1600) + 32 × 4 + 1 = 319,201 24 × (1600 + 1600) + 4000 16 × 2 + 1 = 165,601 4000 24 × (1600 + 1600) + 8 × 1 + 1 = 88,801 24 × (1600 + 1600) + 4000 × 0.5 + 1 = 50,401 4
Table 15. Number of representation parameters for LLaMA-3 models using LoRA, NOLA, and SOLAR. Each row reports total adapter parameters for attention projections (Q and V for LoRA and NOLA; Q and K for SOLAR). Output heads and MLP layers are frozen. For LoRA, the parameter count is computed as: number of layers × input dimension × rank for B + rank × output dimension for A + . Due to differing dimensions between A and B in LoRA, the table computes the contributions for Q and V projections separately. For NOLA, it is computed as: number of layers × 2 × (number of random basis vectors), assuming separate basis sets for A and B. For SOLAR, the count is: number of layers × 2 × topk coefficients for B + topk for A + encoded bases for B + encoded bases for A , plus 1 byte to communicate or store the shared seed.
Model (Rank)
Configuration
Total Parameters
LLaMA-3.2 1B (r=8) NOLA SOLAR (r=8,4K → 1.2K)
16 layers (Q, V) 16 layers (Q, V) 16 layers (Q, V)
16 × [(2048 × 8 + 8 × 2048) + (2048 × 8 + 8 × 512)] = 851,968 16 × 2 × (1000 + 1000) = 64,000 16 × 2 × 1200 + 1200 + 4000 + 1 = 80,801 32
LLaMA-3.2 3B (r=1) NOLA SOLAR (r=1,1000 → 150)
28 layers (Q, V) 28 layers (Q, V) 28 layers (Q, V)
28 × [(3072 × 1 + 1 × 3072) + (3072 × 1 + 1 × 1024)] = 286,720 28 × 2 × (1000 + 1000)= 112,000 28 × 2 × 150 + 150 + 1000 + 1 = 18,551 32
LLaMA-3.1 8B (r=1) NOLA SOLAR (r=1, 1000 → 300)
32 layers (Q, V) 32 layers (Q, V) 32 layers (Q, V)
32 × [(4096 × 1 + 1 × 4096) + (4096 × 1 + 1 × 1024)] = 425,984 32 × 2 × (1000 + 1000)= 128,000 32 × 2 × 300 + 300 + 1000 + 1 = 40,401 32
F. Scalability to Larger Vision Models To validate that SOLAR remains effective and computationally tractable on larger-scale models, we conducted experiments on the ViT-G/14 architecture. This model is substantially larger than the ViT-B/L backbones used in our main experiments, providing a strong test of scalability. We fine-tuned a ViT-G/14 model on the full CIFAR-10, CIFAR-100, Food-101, and T-ImageNet datasets using a LoRA adapter with rank r = 4. We then applied SOLAR with a basis pool of 8,000 vectors, selecting the top 4,000 coefficients to form the compressed adapter. As shown in Table 21, SOLAR successfully preserves the performance of the original LoRA adapter with negligible accuracy drops, while reducing the adapter’s parameter count by 31% (from 492K to 340K). This result demonstrates that SOLAR’s core mechanisms—including SVD extraction and sparse reconstruction—scale effectively to larger models without sacrificing compression efficiency or task performance. 6
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 16. Number of trainable adapter parameters for GPT-2 models using LoRA, NOLA, and SOLAR. Each row reports the total number of parameters added to the query and value projections (Q and V). All configurations freeze the output heads and MLP layers. For LoRA, the parameter count is computed as: number of layers × 2 × input dimension × rank for B + rank × output dimension for A . For NOLA, the parameter count is: number of layers × 2 × (number of random basis vectors), assuming separate basis sets for Q and V. For SOLAR, the parameter count is: number of layers × 2 × topk coefficients for B + topk coefficients for A + encoded bases for B + encoded bases for A , plus 1 for the shared seed.
Model (Rank)
Configuration
Total Parameters
GPT-2 Small (r=4) NOLA SOLAR (r=1, 1000 → 300) SOLAR (r=1, 100 → 90)
12 layers (Q, V) 12 layers (Q, V) 12 layers (Q, V) 12 layers (Q, V)
12 × 2 × (768 × 4 + 4 × 768) = 147,456 12 × 2 × (1000 + 1000) = 48,000 12 × 2 × 300 + 300 + 1000 32 + 1 = 15,150 12 × 2 × 90 + 90 + 100 32 + 1 = 4,396
GPT-2 Medium (r=4) NOLA SOLAR (r=4, 1000 → 300) SOLAR (r=4, 100 → 90)
24 layers (Q, V) 24 layers (Q, V) 24 layers (Q, V) 24 layers (Q, V)
24 × 2 × (1024 × 4 + 4 × 1024) = 393,216 350,000 [Koohpayegani et al., 2024] 24 × 2 × 300 + 300 + 1000 32 + 1 = 30,301 24 × 2 × 90 + 90 + 100 32 + 1 = 8,791
Table 17. Model representation efficiency for LLaMA 3B and 8B models. For the 8B model, all methods use 4-bit quantization, making the LoRA baseline equivalent to QLoRA.
Model
LLaMA-3.2 3B
LLaMA-3.1 8B (4-bit)
Method
LoRA NOLA SOLAR LoRA NOLA SOLAR r=1 1000 bases SOLARr=1(1K→0.1K) r=1 1000 bases SOLARr=1(1K→0.3K)
# Params
287K
112K
16K (94% ↓)
425K
128K
40K (91% ↓)
Val Loss 1.02 MMLU Acc 54.0
1.31 52.7
1.04 54.0
0.89 60.9
1.01 56.1
0.90 60.9
F.1. Ablation Study: Budget-Matched LoRA Comparison To further validate the efficiency of our compression strategy, we conduct an ablation study directly comparing SOLAR to a budget-matched LoRA baseline, as suggested by reviewer feedback.[1] This comparison is critical to demonstrate that SOLAR’s benefits extend beyond mere parameter reduction and offer a more effective performance-compression trade-off than simply training a lower-rank adapter from scratch. As shown in Table 22, fine-tuning a LoRA adapter with a reduced rank (r=2) to match the parameter count of the compressed SOLAR adapter results in a significant performance degradation across all tasks. In contrast, SOLAR, when applied to the higher-performing LoRA (r=4) adapter, successfully preserves task accuracy while achieving a comparable parameter budget. This highlights that SOLAR retains the expressive power of the original higher-rank adapter, a feat not achievable by simply reducing the rank during training. All experiments were conducted on the full datasets using the ViT-B backbone, with results reported as the mean accuracy over five independent runs to ensure statistical robustness.
G. Comparison with Simple SVD Truncation To compare against simple post-hoc SVD truncation, we evaluate SOLAR’s performance against SVD applied directly to the LoRA update ∆W . Since the LoRA adapter ∆W already has rank r, SVD only provides compression if the truncation rank is set lower than r. We use an initial LoRA rank of r = 4 and truncate the SVD to rank 1. In contrast, SOLAR achieves a much smaller footprint by reparameterizing the update in the foundation model’s subspace. The results are summarized in Table 23.
H. Application to Federated Learning One of the motivations for developing SOLAR is to reduce communication overhead in distributed learning scenarios, such as Federated Learning (FL). In typical FL setups, clients fine-tune a model on their local data and transmit the resulting 7
SOLAR: Subspace-Oriented Latent Adapter Reparametrization Table 18. SOLAR applied to AdaLoRA adapters on intermediate-scale LLaMA models.
Method
# Params (Adapter) MMLU Accuracy
AdaLoRA (Baseline, 3B) SOLAR (on AdaLoRA, 3B) AdaLoRA (Baseline, 13B) SOLAR (on AdaLoRA, 13B)
305K 16K 871K 16K
54.8% 54.7% 57.9% 57.7%
Table 19. 2-bit quantization experiments comparing LoRA (QLoRA) and SOLAR.
Method
Quantization # Params MMLU Acc
LoRA (QLoRA) - LLaMA-2 13B SOLARr=1(1K→0.3K) - LLaMA-2 13B LoRA (QLoRA) - LLaMA-3.1 8B SOLARr=1(1K→0.3K) - LLaMA-3.1 8B
2-bit 2-bit 2-bit 2-bit
410K 51K 363K 40K
53.1 53.1 58.4 58.4
model updates (e.g., LoRA adapters) to a central server for aggregation. As highlighted by recent work [Mhanna & Assaad, 2024], communication—not computation—is often the primary bottleneck. Transmitting full adapters from thousands of clients can generate enormous data transfer loads. For example, in an FL setup with 10,000 clients—1,000 participating in each of 10 training rounds—transmitting 74 KB LoRA adapters per client would amount to 740 GB of total data transfer. SOLAR addresses this challenge as a lightweight, post-hoc compression utility. After local training, each client can compress its adapter with SOLAR before transmission. The server then receives only the sparse coefficients and a random seed, drastically reducing per-client communication costs. To demonstrate SOLAR’s effectiveness in distributed settings, we simulated a 10-client FL environment. We compare a baseline where clients transmit full LoRA adapters with a scenario where clients transmit SOLAR-compressed adapters. Each client fine-tunes a ViT-B model on CIFAR-10 with LoRA (r = 4), under two data distribution scenarios: an IID baseline and a non-IID distribution generated via a Dirichlet process with a concentration parameter of 0.5. The simulation runs for 30 communication rounds, with one epoch of local training per client per round. As shown in Table 24, the performance gap between full LoRA adapters and SOLAR-compressed adapters is minimal in both IID and non-IID settings. This demonstrates that SOLAR’s compression does not disproportionately harm aggregation performance, even under significant data heterogeneity. Our experiment confirms that SOLAR can serve as a post-training, plug-and-play module to reduce communication costs in standard FL frameworks without requiring complex changes to the aggregation strategy.
8
SOLAR: Subspace-Oriented Latent Adapter Reparametrization
Table 20. Evaluation of extreme compression on ViT-B. We report bit-level representation footprint (32-bit baseline) and top-1 accuracy over 5 runs. All models are trained for 10 epochs.
Method
Byte Footprint
Oxford Pets
SUN397
CUB-200
ImageNet-1K
LoRA (r=1) SOLAR (r=1, 500 → 50) SOLAR (r=1, 100 → 10)
74KB 2KB (97% ↓) 0.4KB (99% ↓)
93.0±0.5 91.2±0.6 90.3±0.7
74.3±0.3 72.4±0.4 72.4±0.5
84.7±0.4 81.4±0.5 81.3±0.6
81.5±0.6 80.7±0.4 80.6±0.5
Table 21. Scalability of SOLAR on the ViT-G/14 model. Results show top-1 accuracy (%) on full datasets.
Method LoRA (r = 4) SOLAR (r = 4, 8K → 4K)
# Params
CIFAR-10
CIFAR-100
Food-101
T-ImageNet
492K 340K (31% ↓)
99.4 99.4
94.6 94.5
91.2 91.2
92.8 92.8
Table 22. Comparison of SOLAR with a budget-matched LoRA (r=2) baseline on ViT-B. While LoRA (r=2) has a similar parameter count to the compressed SOLAR adapter, it shows a clear performance degradation. SOLAR maintains performance comparable to the original, higher-rank LoRA (r=4).
Method
#Params
CIFAR-10
CIFAR-100
Food-101
T-ImageNet
LoRA (r = 4) LoRA (r = 2)
74K 37K
98.3 97.1
90.3 89.0
87.6 85.5
88.8 87.4
SOLAR (r = 4, 4K → 1.6K) SOLAR (r = 4, 4K → 0.8K)
41K 22K
98.3 97.0
89.8 89.0
87.0 85.2
87.9 87.4
Table 23. Comparison of SOLAR and simple SVD truncation against standard LoRA adapters on multiple vision datasets. The table reports classification accuracy and the corresponding byte footprint of the adapter parameters after compression. SOLAR consistently reduces the parameter size while preserving or improving performance.
Method LoRA (r = 1) LoRA (r = 4) SVD truncation on LoRA SOLAR on LoRA (r = 1) SOLAR on LoRA (r = 4)
Byte Footprint
Oxford Pets
SUN397
CUB-200
ImageNet-1K
74KB 297KB 74KB 8KB 8KB
93.0 94.2 92.7 92.6 93.9
74.3 75.6 73.3 73.9 75.0
84.7 86.0 83.6 84.2 85.4
81.5 82.8 80.8 81.3 82.4
Table 24. Performance of SOLAR on ViT-B under IID and non-IID data distributions in a simulated 10-client federated learning environment.
Method LoRA (r = 4) SOLAR (r = 4, 4K → 2K)
# Params
CIFAR-10 (IID)
CIFAR-10 (non-IID)
74K 51K (31% ↓)
93.7 93.2
87.4 86.7
9