ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces Xun Dong1 Yibo Xu1 Naigang Wang2 Xin Li1 Penghang Yin1 Zi Yang1,† 1 University at Albany, SUNY 2 IBM T. J. Watson Research Center {xdong5, yxu25, xli48, pyin, zyang8}@albany.edu [email protected]
Abstract
to backpropagation, activation storage, and optimizer states. Zeroth-order (ZO) optimization provides a promising forward-only alternative, estimating update directions using only loss evaluations and avoiding backpropagation (Malladi et al., 2023). Despite its memory advantage, ZO finetuning suffers from high gradient-estimation variance when perturbations are sampled in the full parameter space. MeZO (Malladi et al., 2023) showed that in-place ZO-SGD can fine-tune very large models with inference-level memory, but later studies identified slow convergence and instability as major limitations. While ZO optimization offers an appealing alternative, matching FO optimization in terms of convergence and accuracy is still challenging. The core issue is that gradients estimated from function queries often have high variance, and this variance worsens in higher-dimensional problems (Duchi et al., 2015; Nesterov and Spokoiny, 2017; Liu et al., 2018). Subsequent work has improved ZO fine-tuning through better optimizers and more structured perturbations, including ZO-Adam and momentum variants (Zhang et al., 2024b), ZOMuon (Lang et al., 2026), curvature-aware preconditioning (Zhao et al., 2025), low-rank perturbations (Yu et al., 2024; Chen et al., 2025; Lin et al., 2026), and sparse perturbations (Liu et al., 2026). A particularly effective direction is lowrank or subspace-based ZO perturbation, which reduces the perturbation dimension to improve estimator stability. LOZO (Chen et al., 2025) and SubZero (Yu et al., 2024) restrict perturbations to low-rank matrices to reduce variance. AGZO (Lin et al., 2026) further incorporates activation information into low-rank ZO perturbations, showing that activation-aware directions can provide stronger update signals. ZO-Act uses activation information in a different way. Instead of only guiding the construction of perturbation directions, it uses activations to define a fixed low-dimensional parame-
arXiv:2607.01125v1 [cs.LG] 1 Jul 2026
Zeroth-order (ZO) optimization enables finetuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed lowdimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.
1
Introduction
Fine-tuning large language models (LLMs) (Houlsby et al., 2019; Hu et al., 2021; Gurses et al., 2025; Dettmers et al., 2023) has become a standard technique for adapting pretrained models to downstream tasks. However, as LLMs scale to billions of parameters, conventional first-order finetuning becomes increasingly memory-intensive due †
Corresponding author
1
terization. Specifically, each adapted weight update is represented by a frozen activation-informed basis and a small trainable coefficient matrix. This turns ZO fine-tuning into explicit subspace optimization, reducing the perturbation dimension, enabling standard momentum-based optimizers such as Adam, avoiding full-weight perturbation materialization, and naturally supporting quantized backbones by keeping the original weights frozen. We therefore propose ZO-Act, a one-shot activation-informed ZO fine-tuning method. For each adapted linear layer, ZO-Act computes a lowrank basis from input activations using a calibration batch, freezes this basis throughout fine-tuning, and optimizes only the corresponding coefficient matrix. This one-shot fixed-parameterization design provides a simple forward-only alternative to perturbation-based low-rank ZO methods while preserving the memory advantages needed for large and quantized LLM adaptation.
its effectiveness for both full-precision and quantized forward-only fine-tuning.
2
Related Work
Zeroth-order fine-tuning. MeZO (Malladi et al., 2023) adapts simultaneous perturbation stochastic approximation to language model fine-tuning by estimating an update direction from two forward losses evaluated under opposite random perturbations. Its in-place implementation avoids storing explicit perturbation vectors and therefore substantially reduces training memory. Nevertheless, MeZO samples perturbations in the full parameter space, so the resulting estimator remains highly noisy in billion-dimensional models. A number of recent works improve this basic ZO fine-tuning pipeline through optimizer design, curvature information, and structured parameter selection. Zhang et al. (Zhang et al., 2024b) provide a systematic study of ZO variants for LLM adaptation, including momentum, Adam-style updates, conservative updates, block-wise descent, and hybrid ZO-FO training. HiZOO (Zhao et al., 2025) estimates diagonal Hessian information and uses it to precondition ZO updates, addressing the heterogeneous curvature of LLM loss landscapes. ZO-Muon (Lang et al., 2026) combines matrixstructured ZO updates with Muon-style optimization to improve the stability and effectiveness of LLM fine-tuning. These methods demonstrate that reducing estimator noise and improving update geometry are crucial for making ZO fine-tuning competitive.
• We propose ZO-Act, a one-shot activationinformed ZO fine-tuning method that uses input activations to define a fixed lowdimensional subspace. ZO-Act freezes the activation basis and optimizes only lightweight coefficient matrices, turning full-weight perturbation into explicit coefficient-space optimization. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with standard momentumbased optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping the low-bit weights frozen.
Structured ZO perturbations. Another closely related line of work reduces the perturbation dimension by imposing structured perturbations. SMeZO (Liu et al., 2026) reduces the update dimension by selecting a sparse set of sensitive parameters. LOZO (Chen et al., 2025) and SubZero (Yu et al., 2024) perform ZO optimization in randomly generated low-dimensional subspaces. ZO-Muon (Lang et al., 2026) further exploits low-rank structure by combining subspace gradient orthogonalization with Muon-style matrix updates, improving the stability and effectiveness of ZO fine-tuning. AGZO (Lin et al., 2026) incorporates activation information into the construction of low-rank ZO perturbations, showing that activation-aware directions can provide more informative update signals than purely random perturbations. ZO-Act is most closely related to this subspace-based line of work,
• We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that, by perturbing the k-dimensional coefficients instead of the d-dimensional weights, it reduces both the variancedependent convergence term and the finitedifference error of the ZO estimator. The analysis also makes the resulting trade-off explicit: ZO-Act obtains lower-variance estimation at the cost of a controlled subspace approximation bias, which is mitigated by the low-rank structures of LLM activations and gradients. • We demonstrate that ZO-Act consistently improves over strong ZO baselines on Llama-38B, OPT-13B, and INT4 Llama-3-8B across language understanding, question answering, and commonsense reasoning tasks, showing 2
ZO-Act. Motivated by the above observation, we restrict zeroth-order perturbations to the dominant right singular subspace of the input activations. For each linear layer, we first compute a fixed activation-informed basis from a calibration batch, and then perform zeroth-order optimization only over a low-dimensional coefficient matrix. Consider a linear layer with pretrained weight W ∈ Rm×n . Let Vr ∈ Rm×r be the top-r right singular vectors of the input activation matrix X. We parameterize the weight update as
but differs in how the subspace is used. Existing methods primarily use low-dimensional subspaces to construct more effective perturbation directions, whereas ZO-Act uses input activations to define a fixed update parameterization. The activationinformed basis is computed once and kept frozen, and fine-tuning is performed by optimizing only the lightweight coefficient matrices.
3
Methodology
Motivation. Consider a standard linear layer with input dimension m and output dimension n, Y = XW, where X ∈ Rb×m , W ∈ Rm×n , and Y ∈ Rb×n . Let gY ∈ Rb×n denote the gradient of the final loss with respect to the layer output Y. Then the gradient with respect to the weight matrix W is gW = X⊤ gY . Let X = UDV⊤ be the thin SVD of X, where the diagonal entries of D are the singular values in descending order. Denote the top-r components by Ur := U:,:r ,
Vr := V:,:r ,
∆W = Vr B, where B ∈ Rr×n is the only trainable parameter for this layer. The effective weight is therefore Weff = W + Vr B. The pretrained weight W and the activation basis Vr are both frozen throughout training, while B is initialized as zero. At each training step, we sample a random perturbation in the low-dimensional coefficient space:
Dr := D:r,:r .
Z ∼ N (0, I),
The weight gradient can be written as
Z ∈ Rr×n .
The corresponding weight-space perturbation is
gW = VDU⊤ gY .
∆WZO = Vr Z.
If the singular values of X decay rapidly, then gW can be well approximated by its rank-r truncation:
Given a perturbation magnitude µ > 0, the forwardperturbed effective weight is
gW ≈ Vr Dr U⊤ r gY .
+ Weff = W + Vr (B + µZ).
This suggests that the dominant components of the weight gradient lie in the subspace spanned by the top right singular vectors of the input activation matrix, namely Vr . The same intuition can also be understood from the perspective of weight perturbation. Suppose we perturb the weight matrix along the activationinformed subspace: ∆W = Vr R, R ∼ N (0, I). Then the induced perturbation on the layer output is
For an input activation X, the perturbed forward pass can be written as Y+ = XW + (XVr )(B + µZ). Let L+ denote the loss from the forwardperturbed model and let L denote the unperturbed loss on the same mini-batch. The forwarddifference zeroth-order estimator for B is L+ − L bB = Z. g µ With q independent perturbation directions, we average the estimates:
∆Y = X∆W = UDV⊤ Vr R = Ur Dr R.
q
Thus, perturbing W within the span of Vr directly targets the dominant activation directions of the layer. In contrast, perturbations outside this subspace are largely suppressed by the small singular values of X and have limited effect on the layer output. Therefore, restricting zeroth-order perturbations to the Vr subspace preserves the most effective perturbation directions while substantially reducing the perturbation dimension.
1 X L(j,+) − L (j) bB = g Z . q µ j=1
We then update B using a first-order optimizer such bB . Since as Adam with the estimated gradient g the perturbation is sampled in Rr×n rather than Rm×n , the effective perturbation dimension is reduced from mn to rn, where r ≪ m. The whole ZO-Act algorithm is summarized in Algorithm 1 and Algorithm 2. 3
Algorithm 1 Subspace Initialization
range of U introduces a subspace approximation bias: ZO-Act can only reduce the component of the gradient captured by the activation-informed update subspace. We first examine the variance of the estimated gradient in the coefficient space. Let gt = ∇ϕ(βt ). For a single Gaussian direction zt ∼ N (0, Ik ), the one-sided estimator is
Require: Pretrained model, calibration batch Bcal , rank r 1: Run one forward pass on Bcal and collect input activations Xℓ for each linear layer ℓ. 2: for each linear layer ℓ with weight Wℓ ∈ Rmℓ ×nℓ do 3: Compute the top-r right singular vectors of Xℓ : ⊤ Xℓ ≈ Uℓ,r Dℓ,r Vℓ,r .
4: Freeze the pretrained weight Wℓ and the basis Vℓ,r . 5: end for 6: return Frozen bases Vℓ,r .
gbt =
Algorithm 2 ZO-Act
For small µ, the leading term is at = ⟨gt , zt ⟩zt . This term is unbiased, since E[at ] = gt . Its second moment is
Require: Data D, pretrained weights {Wℓ }ℓ , rank r, perturbation scale µ, learning rate η, queries q, steps T 1: Sample calibration batch Bcal ⊂ D. 2: Obtain frozen subspaces {Vℓ,r }ℓ using Algorithm 1 with Bcal and rank r. 3: Initialize the trainable matrix Bℓ ← 0 ∈ Rr×nℓ for all ℓ. 4: for t = 1, . . . , T do 5: Sample mini-batch Bt ⊂ D. 6: Evaluate L with Wℓ,eff = Wℓ + Vℓ,r Bℓ . bBℓ ← 0 for all ℓ. 7: Initialize g 8: for j = 1, . . . , q do (j) 9: Sample Zℓ ∼ N (0, I) for all ℓ. (j,+) 10: Evaluate L(j,+) with Wℓ,eff = Wℓ + Vℓ,r (Bℓ
E∥at ∥2 = E[⟨gt , zt ⟩2 ∥zt ∥2 ] = (k + 2)∥gt ∥2 . Therefore, E∥at − gt ∥2 = E∥at ∥2 − ∥gt ∥2 = (k + 1)∥gt ∥2 . With q independent perturbation directions, define P (j) āt = q −1 qj=1 at . Then
(j)
+µZℓ ). 11: a(j) ← (L(j,+) − L)/µ. b Bℓ ← g bBℓ + a(j) Z(j) 12: g ℓ for all ℓ. 13: end for b Bℓ ← g bBℓ /q for all ℓ. 14: g b Bℓ . 15: Update each Bℓ with an optimizer using g 16: end for 17: return Fine-tuned model with Wℓ,eff = Wℓ + Vℓ,r Bℓ .
4
ϕ(βt + µzt ) − ϕ(βt ) zt . µ
E[āt ] = gt ,
E∥āt − gt ∥2 =
k+1 ∥gt ∥2 . q
Thus, the leading variance of the ZO gradient estimator scales linearly with the perturbation dimension and decreases as 1/q with multi-query averaging. We now analyze the convergence of ZO-Act on the objective ϕ(β) = F (θ0 + U β). The analysis follows the standard analysis framework for gradient descent with a zeroth-order gradient estimator.
Theoretical Analysis
We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space. Let F (θ) denote the fine-tuning objective, where θ ∈ Rd collects the weight parameters of all adapted linear layers. As described in Section 3, ZO-Act freezes the pretrained weights and the activation-informed bases, and optimizes only the coefficient matrices k {Bℓ }L all trainable coeffiℓ=1 . Let β ∈ R collect P cient matrices, where k = L ℓ=1 rnℓ . Equivalently, there exists a fixed activation-informed embedding U such that θ(β) = θ0 + U β, where θ0 denotes the frozen pretrained weights. Thus, ZO-Act optimizes the restricted objective ϕ(β) = F (θ0 + U β), rather than the full objective F (θ) over all weight entries. This restricted formulation creates a variance– bias trade-off. Because ZO-Act samples perturbations only in the k-dimensional coefficient space, the variance-dependent term in the ZO estimator is governed by k, rather than the full weight dimension d. At the same time, restricting updates to the
Theorem 1 (Informal convergence of ZO-Act). Let ϕ(β) = F (θ0 + U β) be the objective optimized by ZO-Act, where β ∈ Rk collects all trainable coefficient matrices. Assume that F is LF -smooth and ϕ is lower bounded by ϕinf k . When ZO-Act applies gradient descent using a q-query one-sided Gaussian zeroth-order estimator with perturbation scale µ in the coefficient space, the iterates satisfy k L 1 + ϕ(β0 ) − ϕinf F k q 1 2 E∥∇ϕ(βt )∥ ≤ O T t=0 T + O L2F µ2 k3 . T −1 X
The proof is shown in Appendix A.1. Theorem 1 highlights the main optimization benefit of ZO-Act. Full-weight ZO corresponds to the special case P k = d, where d = L m n ℓ=1 ℓ ℓ is the full adapted weight dimension. ZO-Act instead perturbs only 4
PL the coefficient matrices, giving k = ℓ=1 rnℓ . Since r ≪ mℓ , we have k ≪ d, so the variancedependent term that governs the convergence of ϕ is substantially reduced, yielding a lower-variance and more stable ZO estimator. The second term is the finite-difference error induced by the one-sided estimator with nonzero perturbation scale µ. Under smoothness-only assumptions, this term scales as O(L2F µ2 k 3 ). For fullweight ZO, the same term scales as O(L2F µ2 d3 ). Since ZO-Act uses the much smaller coefficient dimension k ≪ d, it can dramatically reduce this finite-difference noise. Thus, the low-dimensional activation-informed parameterization reduces both the leading variance term and the higher-order finite-difference error, making the ZO estimator substantially more stable. This improvement comes with a subspace approximation bias. ZO-Act optimizes the restricted objective ϕ(β) = F (θ0 + U β), rather than directly optimizing F (θ) over the full adapted weight space. Therefore, ZO-Act can only reduce the component of the full gradient that lies in the activationinformed update subspace. Let PU = U U ⊤ be the projector onto this subspace. The theorem controls the projected component through ∇ϕ(βt ) = U ⊤ ∇F (θt ), while the residual term ∥(I − PU )∇F (θt )∥2 measures the bias introduced by restricting updates to the activation-informed subspace. Thus, ZO-Act trades a controlled subspace bias for a much lower-variance zeroth-order estimator. In large language model fine-tuning, this subspace bias is often mitigated by the lowrank structure of adaptation. Effective task-specific updates are known to be highly structured (Zhao et al., 2024) and are often well captured by lowdimensional parameterizations such as LoRA (Hu et al., 2021; Gurses et al., 2025). ZO-Act leverages this phenomenon in a data-informed way by choosing the subspace from dominant input activation directions, where layerwise gradients are expected to concentrate; we verify this gradient concentration empirically in Section 5.3. The method can thus preserve important update directions while greatly reducing ZO estimation noise. To further understand the subspace bias, consider a least-squares layerwise approximation. Let R = Y − XW0 be the residual target. The full update problem minimizes ∥X∆W − R∥2F , while ZO-Act restricts the update to ∆W = VB and minimizes ∥XVB − R∥2F . If PX and PXV denote the projectors onto col(X) and col(XV), respec-
tively, then the restricted optimality gap is ϕ⋆V − f ⋆ = ∥(PX − PXV )R∥2F . P If X = i σi ui vi⊤ and V = Vr contains the top r right singular vectors of X, then col(XVr ) = span{u1 , . . . , ur }. Therefore, ϕ⋆V − f ⋆ =
X
2 ∥u⊤ i R∥2 .
i>r
Thus, the approximation bias is small when the residual target R has little energy along the discarded left singular directions of the activation matrix. When the activation spectrum is concentrated, the dominant left singular directions often capture the most influential output variations induced by input activations, making this residual energy small in practice. This supports the use of activationinformed low-rank subspaces: a small rank can preserve the main update directions while substantially reducing the variance of zeroth-order perturbations.
5
Experiments
We evaluate ZO-Act on forward-only fine-tuning tasks for large language models. Our experiments cover three settings: language understanding and question answering, commonsense reasoning, and quantized LLM fine-tuning. For full-precision models, we consider Llama-3-8B (Grattafiori et al., 2024) and OPT-13B (Zhang et al., 2022). For quantized fine-tuning, we evaluate an INT4 quantized Llama-3-8B model, denoted as Llama-3-8B-w4, which is quantized by MagR (Zhang et al., 2024a). To ensure a fair comparison with existing ZO baselines, we match or slightly reduce the total number of forward passes used by ZO-Act compared with the corresponding baseline methods whenever possible. Unless otherwise specified, ZOAct uses Gaussian perturbations with q = 8 perturbation directions per update step and perturbation magnitude µ = 10−3 . We use rank r = 1 for fullprecision models and rank r = 32 for the INT4 quantized model. The rank-one subspace is sufficient for full-precision fine-tuning and gives the lowest perturbation dimension, while the quantized model benefits from a moderately larger subspace to compensate for the reduced capacity. We use the forward-difference estimator and optimize the lowrank coefficient matrices with Adam. Full details are shown in Appendix B. 5
Llama-3-8B
Method
OPT-13B
SST-2 RTE CB BoolQ WiC SQuAD SST-2 RTE CB BoolQ WiC SQuAD • Adam • LoRA MeZO S-MeZO HiZOO LOZO SubZero Subspace-MeZO AGZO ZO-Muon
96.0 95.0 92.7 92.1 93.5 92.5 92.1 92.3 93.4 94.3
92.0 92.0 80.9 73.2 74.4 69.6 69.7 69.6 75.1 69.6 66.8 69.6 71.4 67.9 68.6 69.6 82.7 71.4 81.2 69.6
86.6 86.4 76.7 80.5 80.0 79.4 82.0 80.0 84.5 82.9
72.6 70.7 57.8 56.9 59.7 55.8 58.8 62.9 62.4 65.2
90.4 89.4 86.7 87.5 87.3 89.0 88.3 84.5 90.3 88.2
95.3 94.8 91.4 90.4 92.1 91.7 92.1 91.7 89.8 92.5
80.9 94.6 78.3 69.6 66.1 66.0 63.5 69.6 69.3 69.6 70.4 69.6 71.8 71.4 70.7 71.4 67.5 66.1 72.9 71.4
83.5 80.2 66.1 66.4 67.6 71.9 70.8 68.1 68.1 72.4
66.3 64.3 59.4 58.8 59.4 60.2 60.8 61.7 56.0 61.7
89.5 88.0 81.8 80.8 82.1 84.9 84.5 83.5 85.4 84.5
ZO-Act Full Adam 94.6 ZO-Act Full SGD 94.0 ZO-Act INT4 Adam 93.5
87.0 89.3 83.8 91.1 83.8 82.1
85.1 83.8 83.8
69.1 64.7 60.7
89.1 90.7 88.2
94.0 92.4 –
74.0 75.0 64.6 71.4 – –
73.1 70.0 –
59.4 58.2 –
85.1 83.9 –
Table 1: Language understanding and question answering results of various ZO fine-tuning methods.
5.1
Language Understanding
informed space provides effective update directions and leads to robust forward-only fine-tuning performance across different model architectures. ZO-Act also remains effective when applied to the INT4 quantized Llama-3-8B model. Although the INT4 results are generally lower than the fullprecision ZO-Act results, the performance remains comparable to or stronger than many full-precision ZO baselines. These results suggest that ZO-Act can preserve strong task adaptation ability even when the pretrained model is quantized, supporting its suitability for memory-constrained fine-tuning. Finally, comparing ZO-Act with Adam and SGD highlights the importance of optimizing the explicit coefficient matrices with a momentum-based optimizer. Replacing Adam with SGD leads to noticeably worse performance on most tasks, especially on OPT-13B. This confirms one practical advantage of the ZO-Act parameterization: by exposing lightweight trainable coefficient matrices, it allows standard optimizers such as Adam to be directly applied to ZO gradient estimates.
Models and Datasets. Following the setting of Lang et al. (2026), ZO-Act is evaluated on standard language understanding and question answering tasks. The evaluation includes six benchmarks: SST-2, RTE, CB, BoolQ, WiC, and SQuAD in SuperGLUE (Wang et al., 2019). F1 score is reported for SQuAD, and accuracy is reported for the remaining tasks. Experiments are conducted on two full-precision LLMs, Llama-3-8B and OPT-13B, as well as an INT4 quantized Llama-3-8B model. Baselines. We compare ZO-Act with representative zeroth-order fine-tuning methods, including MeZO (Malladi et al., 2023), S-MeZO (Liu et al., 2026), HiZOO (Zhao et al., 2025), LOZO (Chen et al., 2025), SubZero (Yu et al., 2024), SubspaceMeZO (Lang et al., 2026), AGZO (Lin et al., 2026), and ZO-Muon (Lang et al., 2026). The baseline results, except for AGZO, are taken from the ZOMuon paper. The AGZO results are obtained using our implementation.
5.2
Results. Table 1 reports the language understanding and question answering results. ZO-Act with Adam achieves strong performance across both model families. On Llama-3-8B, ZO-Act obtains the best full-precision ZO result on SST-2, RTE, BoolQ, and WiC, and achieves especially large gains on RTE and CB compared with prior ZO baselines. For example, compared with ZO-Muon, ZOAct improves RTE from 81.2 to 87.0 and CB from 69.6 to 89.3. On OPT-13B, ZO-Act also achieves the best full-precision ZO result on SST-2, RTE, CB, and BoolQ, and remains competitive on WiC and SQuAD. These results show that the activation-
Commonsense Reasoning
Models and Datasets. We further evaluate ZOAct on full-data commonsense reasoning tasks. The benchmark consists of eight tasks: ARCChallenge, ARC-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and WinoGrande (Sakaguchi et al., 2021). We finetune Llama-3-8B on the combined training data and report accuracy on each evaluation task individually, as well as the average accuracy. All tasks are evaluated under an open-ended generation protocol 6
Method
ARC-c ARC-e BoolQ HellaS OBQA PIQA SIQA WinoG Average
Zero-shot
10.0
12.5
61.9
12.3
23.4
47.4
2.2
0.0
21.2
LOZO AGZO SubZero HiZOO ZO-Muon
51.9 65.0 60.2 60.3 61.4
69.6 83.3 78.3 79.6 78.0
62.6 64.5 62.5 63.6 63.0
48.3 70.7 68.0 65.6 63.6
53.0 60.8 53.6 58.2 50.8
70.2 79.9 76.7 73.1 76.7
54.5 63.5 59.2 58.6 57.3
50.6 57.7 52.6 53.5 53.6
57.6 68.2 63.9 64.1 63.1
ZO-Act ZO-Act INT4
68.7 60.5
85.9 80.3
64.7 62.2
76.4 69.9
65.6 60.2
79.6 77.0
67.3 61.8
58.4 53.5
70.8 65.7
Table 2: Commonsense reasoning results on Llama-3-8B.
with answer-match scoring. Baselines. We compare ZO-Act with representative ZO baselines, including LOZO, AGZO, SubZero, HiZOO, and ZO-Muon. ZO-Act and ZOMuon are trained for 20k steps using forward differences with q = 8, corresponding to 9 forward passes per step, including one unperturbed forward pass and eight perturbed forward passes. The remaining baselines are trained for 100k steps with 2 forward passes per step. Thus, ZO-Act uses no more total forward evaluations than these baselines. Results. Table 2 reports the commonsense reasoning results on Llama-3-8B. ZO-Act achieves the best average performance among all evaluated ZO methods and obtains the top result on nearly all sub-tasks. Compared with prior baselines, ZO-Act shows consistent improvements across the benchmark, indicating that the activation-informed coefficient space provides effective update directions beyond the tasks considered in Table 1. When ZO-Act is applied to the INT4 quantized model, it remains competitive with strong full-precision ZO baselines, despite some degradation relative to the full-precision ZO-Act model. This suggests that the same ZO-Act algorithm can be effectively used on quantized models while retaining strong adaptation performance. 5.3
Figure 1: Stability and gradient alignment of activationinformed subspaces at epoch 3 of Llama-3-8B RTE full fine-tuning. We show the q-projection layers at decoders 0, 15, and 31.
responding activation basis recomputed at checkpoint t. We measure the stability of the activation ∥V⊤ Vℓ,0 ∥2
subspace by ℓ,t r F . We also measure the fraction of full-gradient energy captured by the fixed ∥VT Gℓ,t ∥2
F ℓ,0 initialization subspace: ∥G , where Gℓ,t is 2 ℓ,t ∥F the full weight gradient of layer ℓ at checkpoint t. Figure 1 (a) shows the results at the final checkpoint. The activation subspace remains stable across all inspected q-projection layers. For r = 1, the overlap is close to one at all three depths, indicating that the dominant activation direction changes little during fine-tuning. For r = 32, the overlap is slightly lower but still remains high. The fixed activation subspace also captures meaningful gradient energy. The results are shown in Figure 1 (b). Even with r = 1, the initialization subspace captures a clear portion of the fullgradient energy, showing that the dominant activation direction is already gradient-aligned. This
Stability and Gradient Alignment of Activation Subspaces
ZO-Act computes the activation-informed basis once at initialization and keeps it fixed during finetuning. To validate this design choice, we perform a diagnostic study using a separate full first-order fine-tuning run of Llama-3-8B on RTE for three epochs. We inspect the query projection layers at depths 0, 15, and 31. For each layer ℓ, let Vℓ,0 ∈ Rmℓ ×r denote the top-r right singular vectors of the input activation matrix at initialization, and let Vℓ,t denote the cor7
Rank
Llama-3-8B
OPT-13B
Method
Runtime (min)
Memory (GB)
1 32 128 256
87.0 83.0 84.5 80.9
74.0 68.2 71.6 72.6
LOZO SubZero AGZO ZO-Muon HiZOO
44.5 45.5 47.4 42.6 94.0
16.1 16.1 16.1 19.0 30.0
ZO-Act
42.0
16.1
Table 3: Effect of subspace rank on RTE test accuracy.
is important for ZO-Act, since the rank-one setting uses the smallest perturbation dimension and therefore benefits most from variance reduction. When the rank is increased to r = 32, the captured gradient energy further increases, indicating that additional activation directions provide broader gradient coverage. Together, these results demonstrate that a very small activation subspace can already identify useful update directions. 5.4 Effect of Subspace Rank
Table 4: Runtime and memory on Llama-3-8B.
evaluated methods while matching the lowest memory usage. Its memory is the same as LOZO, SubZero, and AGZO, and lower than ZO-Muon and HiZOO. The one-shot subspace initialization adds negligible overhead, taking only 1.5 seconds before fine-tuning. This indicates that the activationinformed coefficient-space design introduces little practical overhead while retaining the memory efficiency of forward-only fine-tuning. Moreover, when ZO-Act is applied to the INT4 quantized model, the memory usage is further reduced to 5.8GB, demonstrating its advantage for low-bit fine-tuning.
We further study the effect of the activation subspace rank on RTE. Table 3 reports the test accuracy of ZO-Act with different ranks on Llama-3-8B and OPT-13B. Interestingly, the rank-one subspace achieves the best performance on both models. Increasing the rank does not necessarily improve the final test accuracy. These results suggest that the dominant activation direction already captures a highly effective update subspace for ZO fine-tuning. The effect of rank is governed by two opposing forces: larger ranks improve subspace coverage and expressiveness, but also increase the coefficientspace perturbation dimension and hence the variance and difficulty of zeroth-order optimization. This explains why the accuracy is non-monotonic in the rank rather than uniformly decreasing: very small ranks may underfit the update subspace, intermediate ranks can be variance-dominated, and the adaptive scaling of Adam partially compensates for the added variance at larger ranks. The rank-one subspace nonetheless attains the best accuracy on both models while using the smallest perturbation dimension. In our main experiments, we therefore use a small activation-informed rank, which provides strong performance while preserving the variance-reduction advantage of ZO-Act. 5.5
6
Conclusion
We proposed ZO-Act, a one-shot activationinformed zeroth-order fine-tuning method for large language models. ZO-Act uses input activations to construct a fixed low-dimensional subspace and optimizes only lightweight coefficient matrices within this subspace. By reducing the perturbation dimension, this design lowers the variance of ZO gradient estimation, enables standard momentum-based optimizers, improves convergence, and naturally supports quantized LLM fine-tuning. Our theoretical analysis shows that ZO-Act improves the stability of zeroth-order optimization by reducing the variance-dependent convergence term and the finite-difference error. It also clarifies the main trade-off introduced by the activationinformed subspace: ZO-Act gains more stable coefficient-space optimization while relying on the selected subspace to capture useful update directions. Empirically, ZO-Act achieves strong performance on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B across language understanding, question answering, and commonsense reasoning tasks, consistently improving over strong ZO fine-tuning baselines. These results suggest that activationinformed coefficient-space optimization is an effective and practical approach for forward-only finetuning of both full-precision and quantized LLMs.
Efficiency Comparison.
We compare the practical efficiency of ZO-Act with representative ZO fine-tuning baselines in Table 4. The comparison is conducted on the RTE task using Llama-3-8B, and all methods are evaluated under the same total forward-query budget of 4500 forward passes. ZO-Act achieves the fastest runtime among the 8
7
Limitations
this increased accessibility as largely beneficial, but note that, as with any fine-tuning technique, it could in principle be used to adapt models toward harmful ends.
Although ZO-Act substantially improves over existing ZO fine-tuning baselines, it still does not fully close the gap to first-order fine-tuning. In particular, first-order methods can often achieve stronger final performance because they use exact backpropagation gradients rather than noisy zerothorder estimates. ZO-Act reduces the variance of ZO estimation by restricting perturbations to an activation-informed coefficient space, but the updates are still based only on forward loss evaluations and therefore remain less informative than full gradients. ZO-Act also remains slower than standard firstorder fine-tuning in terms of wall-clock training time. Each ZO update requires multiple forward evaluations to estimate a gradient direction, while first-order methods obtain gradients through a single forward-backward pass. As a result, ZO-Act is most useful in settings where backpropagation is memory-prohibitive, unavailable, or difficult to support, such as inference-oriented or quantized deployment environments. Improving the runtime efficiency and further narrowing the final performance gap between ZO and FO fine-tuning remain important directions for future work.
8
References Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. 2025. Enhancing zeroth-order finetuning for language models with low-rank structures. In International Conference on Learning Representations, volume 2025, pages 62581–62607. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), pages 2924–2936. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Ethical Considerations
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
This work focuses on improving the optimization efficiency of zeroth-order fine-tuning for large language models and does not introduce new ethical risks beyond those already associated with finetuning pretrained LLMs. All experiments are conducted on publicly available models (Llama-3-8B, OPT-13B) and standard public benchmarks (SuperGLUE and commonsense reasoning datasets), used in accordance with their respective licenses and intended research use. We do not collect any new data or involve human subjects. ZO-Act is a general-purpose fine-tuning method and inherits the limitations and potential harms of its underlying pretrained models, including possible biases, factual errors, and harmful generations. Because ZO-Act adapts models using only forward loss evaluations, it does not mitigate or amplify these issues by design; practitioners should apply standard safety and bias evaluations before deploying any adapted model. By lowering the memory cost of fine-tuning and supporting quantized backbones, ZO-Act may broaden access to LLM adaptation on resource-constrained hardware. We view
John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. 2015. Optimal rates for zeroorder convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Selcuk Gurses, Aozhong Zhang, Yanxia Deng, Xun Dong, Xin Li, Naigang Wang, Penghang Yin, and Zi Yang. 2025. Diablo: Diagonal blocks are sufficient for finetuning. arXiv preprint arXiv:2506.03230. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR.
9
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Ziming Yu, Pan Zhou, Sike Wang, Jia Li, and Hua Huang. 2024. Subzero: Random subspace zerothorder optimization for memory-efficient llm finetuning.
Yicheng Lang, Changsheng Wang, Yihua Zhang, Mingyi Hong, Zheng Zhang, Wotao Yin, and Sijia Liu. 2026. Powering up zeroth-order training via subspace gradient orthogonalization. arXiv preprint arXiv:2602.17155.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800.
Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, and Hong Xu. 2026. Agzo: Activation-guided zerothorder optimization for llm fine-tuning. arXiv preprint arXiv:2601.17261.
Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. 2024a. Magr: Weight magnitude reduction for enhancing post-training quantization. Advances in neural information processing systems, 37:85109–85130.
Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. 2018. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In International Conference on Artificial Intelligence and Statistics, pages 288–297. PMLR.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. 2026. Sparse mezo: Less parameters for better performance in zerothorder llm fine-tuning. Advances in Neural Information Processing Systems, 38:21139–21169.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, and 1 others. 2024b. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391.
Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor Tsang. 2025. Second-order finetuning without pain for llms: A hessian informed zeroth-order optimizer. In International Conference on Learning Representations, volume 2025, pages 43496–43520.
Yurii Nesterov and Vladimir Spokoiny. 2017. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.
A
Proofs and Analysis
A.1
Formal Statement and Proof of Theorem 1
Theorem 2 (Convergence of ZO-Act). Let ϕ(β) = F (θ0 + U β), where β ∈ Rk collects all trainable coefficient matrices. Assume that F is LF -smooth, U ⊤ U = I, and ϕ is lower bounded by ϕinf k . At iteration t, ZO-Act samples q independent Gaussian (j) directions zt ∼ N (0, Ik ) and uses the one-sided estimator
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 4463–4473.
q
gbt =
(j)
1 X ϕ(βt + µzt ) − ϕ(βt ) (j) zt . q µ j=1
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
Suppose the update is βt+1 = βt − ηb gt . If 0<η≤
10
1
8LF 1 + k+1 q
,
√ E∥b̄t ∥ ≤ (LF µ/2) Mk . Thus,
then T −1
4 ϕ(β0 ) − ϕinf 1 X k E∥∇ϕ(βt )∥2 ≤ T ηT
E⟨gt , gbt ⟩ = E⟨gt , āt ⟩ + E⟨gt , b̄t ⟩
≥ ∥gt ∥2 − ∥gt ∥E∥b̄t ∥ 1 1 2 2 2 ∥gt ∥ + (E∥b̄t ∥) ≥ ∥gt ∥ − 2 2 2 2 L µ 1 = ∥gt ∥2 − F Mk . 2 8
t=0
5 + L2F µ2 k(k + 2)(k + 4). 8 gives 1 8LF 1+ k+1 q
In particular, choosing η = T −1 X
32LF 1 E∥∇ϕ(βt )∥2 ≤ T t=0
1 + k+1 q
ϕ(β0 ) − ϕinf k
Next, we upper bound the second moment of gbt . For the leading Gaussian term,
T
(j)
E∥at ∥2 = E[⟨gt , z⟩2 ∥z∥2 ] = (k + 2)∥gt ∥2 .
5 + L2F µ2 k(k + 2)(k + 4). 8
(j)
Therefore, since the at ’s are independent and each has mean gt ,
Proof. Since F is LF -smooth and U ⊤ U = I, the restricted objective ϕ(β) = F (θ0 + U β) is also LF -smooth. Indeed, ∇ϕ(β) = U ⊤ ∇F (θ0 + U β), and hence
E∥āt ∥2 = ∥Eāt ∥2 + E∥āt − Eāt ∥2 1 (j) = ∥gt ∥2 + E∥at − gt ∥2 q k+1 = 1+ ∥gt ∥2 . q
∥∇ϕ(β) − ∇ϕ(β ′ )∥ ≤ LF ∥β − β ′ ∥. (j)
Let gt = ∇ϕ(βt ). For each Gaussian direction zt , define
For the finite-difference remainder, Jensen’s inequality gives q
(j)
(j)
gbt
=
ϕ(βt + µzt ) − ϕ(βt ) (j) zt , µ gbt =
1 q
q X
E∥b̄t ∥2 ≤
j=1
Using ∥āt + b̄t ∥2 ≤ 2∥āt ∥2 + 2∥b̄t ∥2 , we obtain L2 µ2 k+1 2 E∥b gt ∥ ≤ 2 1 + ∥gt ∥2 + F Mk . q 2
(j)
gbt .
j=1
By LF -smoothness of ϕ, for any z,
Let Ak = 1 + (k + 1)/q, then we have
ϕ(βt + µz) = ϕ(βt ) + µ⟨gt , z⟩ + Rt (z), |Rt (z)| ≤
µ2
LF 2
L2 µ2 1X (j) E∥bt ∥2 ≤ F Mk . q 4
E∥b gt ∥2 ≤ 2Ak ∥gt ∥2 +
∥z∥2 .
L2F µ2 Mk . 2
By smoothness of ϕ, the update βt+1 = βt − ηb gt satisfies
(j) (j) (j) (j) We write gbt = at + bt where, at = (j) (j) (j) (j) Rt (zt ) (j) ⟨gt , zt ⟩zt , and bt = zt . µ (j) The remainder term satisfies ∥bt ∥ ≤ P (j) (j) (LF µ/2)∥zt ∥3 . Let āt = q −1 qj=1 at and P (j) b̄t = q −1 qj=1 bt , so that gbt = āt + b̄t .
LF η 2 ∥b gt ∥2 . 2 Taking conditional expectation and substituting the two bounds above gives L2F µ2 1 2 Et [ϕ(βt+1 )] ≤ ϕ(βt ) − η ∥gt ∥ − Mk 2 8 L2F µ2 LF η 2 2 + 2Ak ∥gt ∥ + Mk . 2 2 ϕ(βt+1 ) ≤ ϕ(βt ) − η⟨gt , gbt ⟩ +
We first lower bound the expected descent direc(j) (j) tion. Since zt ∼ N (0, Ik ), we have E[at ] = gt , and hence E[āt ] = gt . Moreover, q
LF µ 1X (j) E∥b̄t ∥ ≤ E∥bt ∥ ≤ E∥z∥3 . q 2
Rearranging,
j=1
Et [ϕ(βt+1 )] ≤ ϕ(βt ) −
η
− LF η 2 Ak ∥gt ∥2
2 ηL2F µ2 L3 η 2 µ2 + Mk + F Mk . 8 4
It holds E∥z∥3 ≤ (E∥z∥6 )1/2 and E∥z∥6 = k(k + 2)(k + 4). We define Mk = k(k + 2)(k + 4), Then 11
If 0 < η ≤ 8LF1 Ak , then η/2 − LF η 2 Ak ≥ η/4. Also, since Ak ≥ 1, we have LF η ≤ 1/8, and hence ηL2F µ2 L3F η 2 µ2 Mk ≤ Mk . 4 32 Therefore, 5ηL2F µ2 η Et [ϕ(βt+1 )] ≤ ϕ(βt ) − ∥gt ∥2 + Mk . 4 32 Taking total expectation and summing from t = 0 to T − 1, we get T −1
5ηL2F µ2 ηX E∥gt ∥2 ≤ ϕ(β0 )−E[ϕ(βT )]+ Mk T. 4 32 t=0
Since ϕ(βT ) ≥ ϕinf k , dividing by ηT /4 yields T −1
4(ϕ(β0 ) − ϕinf 1 X k ) 5 2 2 E∥∇ϕ(βt )∥2 ≤ + LF µ Mk . T ηT 8 t=0
This proves the theorem.
B
Experiment Setups
In all experiments, ZO-Act is applied to all linear layers except the embedding layers and taskspecific linear heads. Language understanding and question answering. Following the setting of Lang et al. (2026), ZO-Act is evaluated on six standard language understanding and question answering benchmarks: SST-2, RTE, CB, BoolQ, WiC, and SQuAD in SuperGLUE (Wang et al., 2019). F1 score is reported for SQuAD, while accuracy is reported for all other tasks. Experiments are conducted on two full-precision LLMs, Llama-3-8B and OPT-13B, as well as an INT4 quantized Llama-3-8B model. Following prior ZO fine-tuning work (Malladi et al., 2023), we randomly sample 1,000 training examples and 1,000 test examples for each task, and use the same prompts as MeZO (Malladi et al., 2023). Commonsense reasoning. The commonsense reasoning benchmark consists of eight tasks: ARCChallenge, ARC-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019), and WinoGrande (Sakaguchi et al., 2021). ZO-Act is fine-tuned on the combined training data using Llama-3-8B, and performance is reported as accuracy on each individual task as well as the average accuracy across all tasks. The hyperparameters are reported in Table 5, Table 6, and Table 7. 12
CB
RTE
WiC
SST-2
BoolQ
SQuAD
Commonsense
Rank 1 Optimizer Adam / SGD LR (Adam) 5 × 10−5 5 × 10−5 4 × 10−5 3 × 10−5 5 × 10−5 2.5 × 10−5 LR (SGD) 3 × 10−6 3 × 10−6 3 × 10−6 3 × 10−6 3 × 10−6 5 × 10−6 Scheduler constant with warmup Weight decay 0 µ 10−3
Adam 2 × 10−5 –
Table 5: Hyperparameters for ZO-Act on Llama-3-8B.
CB
RTE
WiC
SST-2
BoolQ
SQuAD
Rank 32 Optimizer Adam LR (Adam) 6 × 10−5 5 × 10−5 5 × 10−5 3 × 10−5 3 × 10−5 3 × 10−5 Scheduler constant with warmup Weight decay 0 µ 10−3
Commonsense
2 × 10−5
Table 6: Hyperparameters for ZO-Act on INT4 quantized Llama-3-8B.
CB
RTE
WiC
SST-2
BoolQ
SQuAD
Rank 1 Optimizer Adam / SGD LR (Adam) 4 × 10−5 2.5 × 10−5 2 × 10−5 2.5 × 10−5 2 × 10−5 2.5 × 10−5 LR (SGD) 7 × 10−7 1 × 10−6 1 × 10−6 1 × 10−6 1 × 10−6 1 × 10−6 Scheduler constant with warmup Weight decay 0 µ 10−3
Table 7: Hyperparameters for ZO-Act on OPT-13B.
13