arXiv CS open access

Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

F INE -T UNING L OW-B IT M ODELS WITH G RADIENT IN Q UANTIZED C ODE S PACE Shiguang Wu1 , Zhouchen Lin2 , Quanming Yao1 Department of Electronic Engineering, Tsinghua University 2 School of Intelligence Science and Technology, Peking University wsg23@mails.tsinghua.edu.cn, qyaoaa@tsinghua.edu.cn

arXiv:2608.30908v1 [cs.LG] 31 Aug 2026

A BSTRACT Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodeS consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.

I NTRODUCTION

Large language models (LLMs) are increasingly deployed in low-precision formats such as INT4, NF4, and MXFP4 to reduce storage, memory traffic, and inference cost (Xiao et al., 2023; Lin et al., 2024). These savings matter not only for a shared backbone, but also when many task- or user-specific checkpoints must be stored under a fixed memory budget. We study a strict fully low-bit setting in which the targeted weights of every adapted checkpoint remain in the target quantized representation, with only datatype-permitted scale metadata and no high-precision residual adapter at inference time (Jeon et al., 2025; Malinovskii et al., 2024). Under this constraint, fine-tuning is no longer optimization in a nearly continuous floating-point space; it is optimization over quantization codes and scales. One practical strategy retains continuous optimization. Quantized-backbone adapter methods train high-precision LoRA-style parameters and either keep them at inference time or merge and requantize them afterward (Dettmers et al., 2023; Li et al., 2024; Loeschcke et al., 2024; Xu et al., 2024). This is effective when mixed-precision deployment or a separate conversion stage is acceptable. Under the stricter setting above, however, retaining the adapter violates the deployment constraint, while merging and requantizing can make the evaluated low-bit state differ from the state optimized during training. Quantization-aware and compressed-representation methods move optimization closer to the final checkpoint (Jeon et al., 2025; Malinovskii et al., 2024), but still leave a fundamental question: how should first-order information be expressed when the actual decision variable is a quantization-code index? Direct discrete and zeroth-order approaches instead evaluate candidates in or near the quantized model family (Zhou et al., 2025; Shang et al., 2026; Xu et al., 2026), but random or population proposals can require many forward evaluations in high-dimensional code spaces. The central challenge is to combine deployment-faithful selection with first-order directional guidance. To be deployment-faithful, we optimize over quantized code coordinates rather than weight. To be efficient, we try to find more useful information beyond zero-order. Based on this principle, we propose GradCodeS, a gradient-guided search method. We first identify that mapping ordinary 1

≠

GSM8K accuracy (%)

+ %&* (. *. !(%&*) = $" ## ∇" ##

$$ #

GradCodeS QLoRA-16A retains FP16 adapter

16-bit SFT

PV-Tuning 35 merge + requantize -13.4 pts

30 Base-4

weight space

code space

QLoRA-4Merge

Fully 4-bit

Llama-3.2-1B-Instruct

4-bit + FP16 adapter Unquantized 16-bit

(a) Geometry mismatch between weight and code spaces. (b) Accuracy under different deployment regimes.

Figure 1: Motivation for deployment-faithful code-space optimization. (a) Group scales and nonuniform codebook gaps distort local geometry, so a weight gradient does not identify the best discrete code move. (b) In the Llama-3.2-1B GSM8K example, continuous adaptation either retains an FP16 adapter or loses 13.4 points after QLoRA requantization; GradCodeS improves the fully 4-bit result while retaining a deployable checkpoint. gradient in weight space to code space is generally not the steepest descent direction in code space, as illustrated in Figure 1(a), thus introducing a code-coordinate surrogate gradient by codebook continuation, serving as first-order information in code space with theoretical support. GradCodeS uses code-coordinate surrogate gradient descent to provide reference and then samples multiple nearby code candidates biased toward that reference and selects an update only according to its realized deployed loss. Thus every selected iterate is a valid low-bit checkpoint. This rule is parameterization-agnostic and datatype-agnostic: it can be instantiated on full weights, integrated with structured parameterizations such as LoRA, combined with joint scale optimization, and paired with different quantization datatypes. Our contributions can be summarized as follows: • We derive a code-coordinate surrogate gradient and establish its local steepest-descent interpretation under codebook continuation. • We develop a guided candidate-sampling algorithm that combines first-order guidance with lossbased selection over deployable low-bit candidates. • We evaluate GradCodeS across three task families, multiple quantization datatypes, and both full and structured code parameterizations, observing consistent gains in the fully low-bit setting.

R ELATED W ORK

Continuous low-bit adaptation. Low-bit adaptation commonly retains continuous optimization. Quantized-backbone parameter-efficient methods train high-precision adapters, improve their quantized initialization, or design updates that can later be merged (Dettmers et al., 2023; Li et al., 2024; Xu et al., 2024; Loeschcke et al., 2024). They are effective when mixed-precision inference or a separate conversion stage is acceptable, but under strict fully low-bit deployment the optimized and deployed states may differ because an adapter remains at inference time or a merged update must be requantized. Quantization-aware and compressed-representation methods, including L4Q and PVTuning, optimize closer to the final quantized checkpoint (Jeon et al., 2025; Malinovskii et al., 2024). Their focus is efficient optimization of differentiable or structured compressed representations. Our complementary question is how a weight gradient should be expressed in ordered code coordinates without treating the resulting continuous reference as an accepted update. Direct low-bit optimization. This approach avoids a separate deployment conversion. QuZO and QZO use zeroth-order estimates for low-precision or quantization-aware fine-tuning, while QES applies evolution strategies to quantized parameters (Zhou et al., 2025; Shang et al., 2026; Xu et al., 2026). These methods reduce or avoid backpropagation and evaluate search information through forward perturbations or population fitness. Although candidate quality can therefore be measured in or near the quantized model family, proposal efficiency remains difficult in a high-dimensional 2

space under a finite evaluation budget. GradCodeS instead spends one backward pass to obtain an analytical local signal, uses it only to guide where discrete search looks, and still selects updates by the realized loss of deployable low-bit candidates. Gradient-informed discrete proposals. A derivative can shape a discrete proposal without being followed deterministically. Representative samplers construct local, Langevin-style, Newton-style, or importance proposals from gradients with respect to discrete inputs (Grathwohl et al., 2021; Zhang et al., 2022; Sun et al., 2023a; Xiang et al., 2023; Liu et al., 2023); related optimization methods bias random search with surrogate gradients or adapt discrete sampling to combinatorial objectives (Maheswaranathan et al., 2019; Sun et al., 2023b; Li & Zhang, 2025). These works motivate separating proposal construction from state selection, but they do not resolve two features of low-bit fine-tuning. First, the available derivative is with respect to recovered weights, whereas the decision variable is a code index whose geometry is distorted by group scales and non-uniform codebook gaps. Second, our goal is loss minimization over a finite candidate set rather than preservation of a stationary distribution. GradCodeS addresses these differences through a quantization-specific code-coordinate signal, proposal-only gradient guidance, and loss-based selection of deployable states.

P ROBLEM S ETUP

Fix a target quantization datatype q, specified by its bitwidth bq , numerical codebook Cq , grouping structure Gq , and admissible scale space Sq . Its ordered codebook is (q)

(q)

Cq = {c1 , . . . , cKq },

(q)

c1 ≤ · · · ≤ cKq ,

Kq ≤ 2bq .

The indices follow numerical order and need not match hardware bit-pattern order. For a weight matrix W ∈ Rdout ×din , let [Kq ] := {1, . . . , Kq } and Zq := [Kq ]dout ×din . A code matrix Z ∈ Zq selects (q) values entrywise as Cq (Z) = [cZij ]ij , where ψ(i, j) ij denotes the matrix whose (i, j)-th element is ψ(i, j). Each entry belongs to a group gq (i, j) ∈ Gq with scale metadata s = {sg }g∈Gq ∈ Sq . Defining Sq (s) = [sgq (i,j) ]ij , the deployed weight is cq (Z, s) = Sq (s) ⊙ Cq (Z), W

(1)

where ⊙ is element-wise multiplication. This representation covers the evaluated datatypes NF4, INT4, and MXFP4; their datatype-specific components are given in Appendix B. For a model, let L be the set of targeted quantized linear layers. We use Z and s for their layer-wise collections and let Qq denote the product of the corresponding layer-wise feasible code and scale c q (Z, s) for the recovered weights. Parameters outside L are left unquantized. sets. We use W Given an initial feasible state (Z0 , s0 ), a target distribution D, a model f , and a loss function ℓ, we formulate fully low-bit fine-tuning as h i c q (Z, s) , y . min(Z,s)∈Qq L(Z, s) := E(x,y)∼D ℓ f x; W (2) Unlike approaches that optimize a continuous auxiliary state and quantize only after training, (2) is defined directly over deployable states. Here Z is discrete, while s is continuous in the default setting but constrained to its admissible space. Since q is fixed, we henceforth omit datatype annotations (q) c is c (Z, s) = S(s) ⊙ C(Z). Bold W from Kq , ck , Cq , Zq , Sq , and Sq . Thus, for a single matrix, W reserved for the full collection of targeted weights, while L denotes the corresponding objective.

M ETHOD

To optimize (2), GradCodeS separates gradient guidance from discrete state selection. Section 4.1 constructs a geometry-aware code surrogate gradient, and Section 4.2 establishes its local and discrete first-order properties. Section 4.3 then incorporates this signal into a guide–sample–evaluate–select procedure that accepts updates according to the realized loss of deployable low-bit states. 3

4.1

C ONSTRUCTING THE C ODE S URROGATE G RADIENT

The ordinary weight gradient ∇W c L is not directly a signal for the discrete code index because group scales and local codebook gaps rescale a unit code move. Thus in general, the direction in code space corresponding to the steepest descent direction in weight space is not the steepest descent direction in code space, as illustrated in Figure 2. For a matrix A, write A+ = [max{Aij , 0}]ij and A− = [max{−Aij , 0}]ij . Let Z ± = clip(Z ± 1, 1, K) and define the forward and backward gaps G+ (Z) = cZ + − cZij ij , G− (Z) = cZij − cZ − ij . ij

The quantities above characterize the local geometry of the two neighboring code moves. To use a weight gradient for discrete search, we need a code-coordinate signal that reflects this geometry; Definition 1 constructs such a signal. Definition 1 (Local effective step and code surrogate gradient). Given a quantized state (Z, s) with fixed codebook C, let σ = sign(∇W c L(Z, s)) and define the descent-aligned effective step by + − D Z, s, ∇W c L(Z, s) = S(s) ⊙ G (Z) ⊙ σ− + G (Z) ⊙ σ+ . The directional code surrogate gradient is then ∇Z L(Z, s) = ∇W (3) c L(Z, s) ⊙ D Z, s, ∇W c L(Z, s) . Figure 2 gives a simplified illustration of Definition 1. It uses an identity codebook, whose adjacent gaps are all one, so the effective step reduces to D = S(s). The example therefore shows even without nonuniform codebook gaps, accounting for the deployed-weight change of each code coordinate yields a substantially better descent direction. Appendix C derives this construction from the local piecewise-linear continuation. Let C(Z) = Z, S(s) = (0.1, 10), L(Ŵ ) = Ŵ1 + 0.6Ŵ2 . ∇W c L = (1, 0.6), so mapped code move (to achieve ∇W c L) is (−10, −0.06); while D = S(s) and ∇Z L = (0.1, 6). For an equal code-space step ∆Z = ϵd with ∥d∥2 = 1: Unit direction d Loss change ∆L √ Mapped weight gradient (−10, −0.06)/ √100.0036 −0.136ϵ −6.001ϵ Definition 1 surrogate (−0.1, −6)/ 36.01 Result. The effective-step correction yields about 44× more loss decrease for the same code-space step. Direction

Figure 2: A two-coordinate illustration of the effective-step correction in Definition 1. 4.2

F IRST-O RDER P ROPERTIES

The results below summarize the overall argument. Lemma 1 gives the surrogate a local continuous interpretation. Proposition 1 establishes the central discrete connection by showing exact firstorder consistency for feasible one-level code moves. Corollary 1 then converts this identity into a realized-loss bound used by the candidate-search analysis in Proposition 2. Lemma 1 (Local steepest-descent interpretation). Let DZ,s := D Z, s, ∇W c L(Z, s) . Define the local directional surrogate function of L(Z, s) by extending it to real-valued interpolation ∆Z ∈ Rdout ×din by c (Z, s) + DZ,s ⊙ ∆Z . L̄Z,s (∆Z) = L W Then ∇∆Z L̄Z,s (∆Z) ∆Z=0 = ∇Z L(Z, s). Consequently, whenever ∇Z L(Z, s) ̸= 0, the direction −∇Z L(Z, s) is the steepest descent direction of L̄Z,s at ∆Z = 0 under the Frobenius norm. Lemma 1 gives −∇Z L a precise but deliberately local interpretation: it is the steepest direction of the active continuous surrogate, not an automatically valid code update. We next connect its score to an actual one-level code move. Proposition 1 (One-step consistency). Let ∆Z ∈ {−1, 0, 1}dout ×din be a feasible one-step code move such that Z + ∆Z ∈ Z, ∆Z ⊙ ∇W c L(Z, s) ≤ 0 entrywise. Then the surrogate linear term matches the exact first-order deployed-weight change: D E c c ⟨∇Z L(Z, s), ∆Z⟩F = ∇W . (4) c L(Z, s), W (Z + ∆Z, s) − W (Z, s) F

Algorithm 1 GradCodeS: Guide–Sample–Evaluate–Select in Quantized Code Space 1: Input: state (Z, s); codebook C; data D; feasible sets (Z, S); step sizes (ηZ , ηs ); proposal p;

candidates M ; iterations T . 2: for t = 1, . . . , T do 3: Sample mini-batch Bt ∼ D b B (Z, s)) 4: Update scales by s ← ΠS (s − ηs ∇s L t b 5: Guide: compute ∇Z LBt (Z, s) using (3), then Z ref using (5) ref 6: Sample: sample M code candidates {Z m }M ) m=1 from p(· | Z m b B (Z , s) for each candidate 7: Evaluate: compute L t b e 8: Select: Z ← arg minZ∈{Z,Z 1 ,...,Z M } LBt (Z, s) e 9: end for c (Z, s) = S(s) ⊙ C(Z) 10: Return: W

Proposition 1 provides the central discrete connection: by selecting the adjacent gap in the descent direction, the surrogate score exactly matches the first-order deployed-weight change of every feasible gradient-aligned one-level move. Because the network loss is nonlinear, the next result controls the discrepancy from the realized loss. Corollary 1 (One-step smooth-loss upper bound). Under the assumptions of Proposition 1, if L is c (Z, s), then β-smooth as a function of the deployed weight in a neighborhood of W L(Z + ∆Z, s) ≤ L(Z, s) + ⟨∇Z L(Z, s), ∆Z⟩F +

2 β c c (Z, s) . W (Z + ∆Z, s) − W 2 F

Corollary 1 bounds the realized loss change by the surrogate prediction plus a curvature penalty; a move is guaranteed to decrease the loss when its negative surrogate term dominates this penalty. Thus the signal is first-order faithful for feasible local moves, while candidate evaluation accounts for nonlinear and higher-order effects. This bound underlies Proposition 2; proofs of the present results are provided in Appendix D. 4.3

G RADIENT-G UIDED D ISCRETE O PTIMIZATION

The results in Section 4.2 show how to rank feasible local code moves. We now turn this signal into a complete procedure that alternates differentiable scale updates with deployment-faithful code search. The surrogate gradient guides where to search, while the deployed loss determines what to accept. Performing gradient descent with the code surrogate gradient (3) gives a real-valued reference point in code space: Z ref = Z − ηZ ∇Z L(Z, s), (5) where ηZ > 0 is the step size. Because Z ref is generally non-integer, it only centers a proposal over valid codes. The guide–sample–evaluate–select pipeline draws M ≪ |Z| candidates with greater mass near Z ref and retains the lowest-loss state. Appendix E gives an efficiently sampled coordinate-factorized proposal. b B (Z, s) denotes the average loss over mini-batch B, an unbiased estimator of L(Z, s). In practice, L Each iteration first updates the differentiable scales by projected gradient descent with Z fixed and then searches the codes at the updated scales. Algorithm 1 summarizes the procedure; ΠS projects onto the admissible scale space, and selection ties retain the current Z. The next result isolates the role of gradient guidance in the deterministic full-batch code-search substep; it is conditional on the proposal mass placed near the reference step. Proposition 2 (Guided best-of-M descent). Fix the scales s, let qt be the proposal mass of gradientaligned one-step feasible updates ∆ satisfying ∥∆ + ηZ ∇Z L(Zt , s)∥F ≤ cηZ ∥∇Z L(Zt , s)∥F , and c (Zt + ∆, s) − W c (Zt , s)∥F ≤ κ∥∆∥F . If c ∈ [0, 1) and suppose that Corollary 1 applies with ∥W βκ2 ηZ 2 α := 1 − c − 2 (1 + c) > 0, then M independent candidates followed by the selection rule in Algorithm 1 satisfy E[L(Zt+1 , s) | Zt ] ≤ L(Zt , s) − αηZ 1 − (1 − qt )M ∥∇Z L(Zt , s)∥2F . 5

The factor 1 − (1 − qt )M is the probability that at least one candidate lies in the specified nearreference set, showing how proposal concentration and candidate count control expected progress. The proof is provided in Appendix F. Operational distinction. Continuous or STE-style approaches use a gradient to produce an update that is retained, projected, or requantized, whereas zeroth-order and evolutionary approaches preserve discrete evaluation but rely on sampled losses to determine where to search. GradCodeS separates these roles: a quantization-aware backward signal shapes a proposal over valid codes, while realized discrete losses select the accepted deployable state. Section 5 evaluates the two practical consequences of this separation: deployment-faithful performance and more effective candidate search.

E XPERIMENTS

We evaluate GradCodeS to address three questions: whether GradCodeS improves performance while remaining in the fully 4-bit deployment space; whether the results are robust across data regimes, code parameterizations, and quantization datatypes; and whether gradient guidance improves candidate quality and end-to-end search efficiency. 5.1

E XPERIMENTAL S ETTINGS

We evaluate Qwen3-0.6B, Llama-3.2-1B and 3B-Instruct on GSM8K, AlpacaEval, and MASSIVE en-US, covering arithmetic reasoning, instruction following, and structured semantic parsing. We quantize all transformer-block linear layers while leaving the token embedding and LM head unquantized. Under matched data and evaluation protocols, we compare unquantized references (Base-16 and 16-bit SFT), mixed-precision adapter methods (QLoRA-16A and QA-LoRA), and fully 4-bit baselines (Base-4, QLoRA-4Merge, LoQT, L4Q, PV-Tuning, QuZO, QZO, and QES). We evaluate both full-matrix and low-rank code parameterizations of GradCodeS; complete dataset, metric, quantization, and baseline details are provided in Appendix A. All results are obtained with three independet runs with different random seeds, reported in meanstd . 5.2

M AIN R ESULTS

Tables 1 and 2 report results across arithmetic reasoning, instruction following, and structured semantic parsing. Across both backbones and all three tasks, GradCodeS is the strongest fully 4-bit method. Direct 4-bit quantization causes a substantial drop, and the competing fully 4-bit methods recover this loss to varying degrees; GradCodeS provides the most consistent recovery across the reported metrics. We skip relatively weak baselines on 3B model due to computation budget. The comparison between QLoRA-16A and QLoRA-4Merge isolates the deployment-state mismatch. QLoRA-16A retains a high-precision adapter at inference time, whereas QLoRA-4Merge collapses that adapter into the backbone and requantizes the resulting weights. The degradation after this conversion, also highlighted in Figure 1(b), supports deployment-faithful optimizing rather than relying on an optimize-then-quantize pipeline. Within the fully 4-bit regime, both GradCodeS parameterizations are effective: the LoRA variant is generally strongest, while the full-matrix variant remains competitive. We examine their dependence on training-set size in Section 5.3. Compared with 16-bit SFT, the best GradCodeS variant obtains higher GSM8K accuracy and stronger MASSIVE results, while the 16-bit reference remains strongest on Qwen AlpacaEval and is marginally higher on Llama AlpacaEval. We interpret the gains with low-bit codes as regularization in Section 5.3, not as a general claim that lower precision is universally superior. Deployment cost view. Figure 1(b) complements the task tables by plotting Llama-3.2-1B-Instruct GSM8K accuracy across deployment regimes. The comparison is computed on the target quantized modules and excludes modules intentionally left unquantized. GradCodeS attains the best accuracy on the fully 4-bit deployment line, whereas mixed-precision methods retain additional high-precision adapter parameters. Structured prediction. Table 2 evaluates a more application-like structured prediction setting. GradCodeS gives the best fully 4-bit results on both backbones and all three metrics. Exact Match 6

Table 1: Results on GSM8K and AlpacaEval. Values are means over three independent runs, with standard deviations shown as subscripts. The second column reports the final deployment regime: 16 denotes unquantized 16-bit deployment, Mixed denotes a 4-bit backbone with high-precision adapter parameters, and 4 denotes fully 4-bit deployment for the quantized modules. Deployment bits

Method

Llama-3.2-1B-Inst.

Llama-3.2-3B-Inst.

GSM8K Acc.

Qwen3-0.6B Alpaca WR

GSM8K Acc.

Alpaca WR

GSM8K Acc.

Alpaca WR

Base-16 16-bit SFT

16 16

37.450.00 39.650.54

base 88.262.16

36.320.00 38.930.63

base 56.642.36

56.940.00 67.100.28

base 78.572.86

QLoRA-16A QA-LoRA

Mixed (4.08) Mixed (4.05)

39.570.47 36.130.62

54.902.87 53.343.12

38.210.51 34.150.74

53.892.74 49.233.18

65.050.39 -

68.263.12 -

Base-4 QLoRA-4Merge LoQT L4Q PV-Tuning QuZO QZO QES

4 4 4 4 4 4 4 4

27.140.00 26.460.71 30.940.58 29.890.67 38.700.49 29.450.84 29.760.76 28.800.92

26.581.94 38.352.76 44.672.63 40.282.91 70.513.21 34.353.47 38.993.14 41.133.36

25.850.00 24.790.66 29.040.61 31.060.57 36.920.45 26.540.78 27.600.81 26.390.85

46.162.42 33.683.55 48.852.81 51.212.66 52.631.84 41.573.26 39.393.41 38.123.67

49.130.00 49.130.52 63.080.43 -

35.942.68 36.203.07 62.362.79 -

GradCodeS (LoRA) GradCodeS (Full)

4 4

45.720.42 43.680.38

82.242.57 83.092.48

40.520.47 41.630.58

54.661.83 56.342.31

67.080.31 66.390.35

69.523.26 72.043.14

requires the entire semantic frame to be correct, while Slot F1 measures the quality of the predicted slot structure. The gains on both metrics indicate that deployment-faithful code updates preserve structured output behavior after quantization. Table 2: MASSIVE en-US test results. Values are means over three independent runs, with standard deviations shown as subscripts. Models are fine-tuned on the English-US training split and evaluated on the English-US test split. EM denotes exact semantic-frame match. Method Base-16 16-bit SFT QLoRA-16A QA-LoRA

Deployment bits

Qwen3-0.6B

Llama-3.2-1B-Inst.

Llama-3.2-3B-Inst.

Intent

Slot F1

Intent

Slot F1

Intent

Slot F1

0.000.00 57.920.36

0.000.00 83.950.22

3.380.00 65.310.44

0.000.00 63.060.41

0.130.00 85.570.29

1.390.00 73.390.38

0.070.00 66.160.33

3.030.00 88.970.24

3.630.00 75.210.35

Mixed (4.08) 57.160.42 Mixed (4.05) 50.170.58

83.490.31 80.770.47

66.740.48 61.190.62

63.390.37 54.100.55

85.810.26 84.300.35

74.130.43 65.410.66

65.230.39 -

88.060.28 -

75.140.41 -

0.610.00 0.670.11 72.090.51 72.030.56 82.920.37 57.300.72 59.790.64 52.530.81

3.970.00 3.610.18 45.830.78 46.020.74 66.290.53 38.550.91 42.500.83 34.411.02

0.000.00 0.000.00 52.140.59 53.430.64 64.500.38 46.350.77 45.180.81 43.560.85

0.030.00 0.030.02 75.270.46 75.880.49 86.310.31 64.170.68 62.500.74 58.350.79

0.620.00 0.480.14 49.900.82 58.620.69 74.880.46 46.800.88 41.330.95 39.810.97

0.000.00 0.000.00 65.890.35 -

1.950.00 1.950.13 86.950.27 -

3.320.00 3.290.21 76.200.42 -

16 16

Base-4 QLoRA-4Merge LoQT L4Q PV-Tuning QuZO QZO QES

4 4 4 4 4 4 4 4

0.000.00 0.000.00 51.420.63 48.270.71 58.710.45 31.260.86 30.060.79 29.250.93

GradCodeS (LoRA) GradCodeS (Full)

4 4

65.880.34 85.530.28 76.600.39 67.370.31 86.890.33 76.250.37 70.480.27 90.970.19 79.850.31 66.610.32 86.850.24 75.880.41 69.170.28 88.130.21 78.740.34 67.260.29 87.530.25 78.240.36

5.3

ROBUSTNESS ACROSS DATA , PARAMETERIZATIONS , AND DATATYPES

Data size and code parameterization. The comparison with 16-bit SFT suggests that the restricted code-space parameterization may provide useful regularization when fine-tuning data are limited. We treat this as an empirical hypothesis rather than a consequence of the local optimization theory. Table 3 examines the hypothesis by comparing the original 7.0k-example GSM8K training set with a 167k-example training set formed by adding 160k MetaMathQA examples (Yu et al., 2024). With 7.0k examples, GradCodeS (LoRA) performs best; with 167k examples, 16-bit SFT becomes strongest. This result is consistent with, but does not prove, a regularization benefit from the low-rank code parameterization in the smaller-data regime. 7

Table 3: GSM8K test accuracy (%) across training-set sizes and parameterizations on Llama-3.2-1BInstruct. Values are means over three independent runs, with standard deviations shown as subscripts. Method

Parameterization

7.0k examples

167k examples

16-bit SFT

LoRA (rank 8)

38.90.6

55.90.7

GradCodeS

LoRA (rank 8) Full matrix

40.50.6 41.60.5

54.30.6 54.50.5

Quantization datatype. Based on the unified datatype-specific recovery map in (1), GradCodeS does not rely on a single quantization datatype. Different datatypes change the codebook C and scale grouping, while the search rule requires only ordered code levels and the local effective-step map in Definition 1. Table 4 shows that GradCodeS can be applied to NF4, INT4, and MXFP4. NF4 and MXFP4 are stronger than INT4 in this setting, and all three variants remain competitive with the adaptive PV-Tuning baseline. These results establish compatibility across the tested datatype-specific geometries without attributing the performance differences to any single codebook property. Table 4: Results of GradCodeS across quantization datatypes on Llama-3.2-1B-Instruct. Values are means over three independent runs, with standard deviations shown as subscripts.

5.4

Method

Datatype

GSM8K ACC.

AlpacaEval WR.

MASSIVE EM.

GradCodeS

NF4 INT4 MXFP4

41.60.5 38.90.6 41.10.4

56.31.8 52.02.4 57.12.2

69.20.3 65.10.5 68.80.4

PV-Tuning

Adaptive

36.90.5

52.61.8

64.50.4

S EARCH E FFICIENCY AND C OMPONENT A NALYSIS

End-to-end efficiency. Figure 3(a) compares wall-clock time against the best test accuracy reached so far. GradCodeS is not the cheapest possible step: it uses one backward pass to build the guided reference and then evaluates multiple discrete candidates. The benefit is that these evaluations are targeted. The baselines improve quickly but plateau below the final GradCodeS accuracy, while GradCodeS reaches a stronger result within the displayed time range. This supports the intended trade-off of spending one gradient computation to reduce wasted discrete search. QZO LoQT PV-Tuning GradCodeS (Ours)

sample nearest projection guide (high-precision) best

1.4 1.2

1.0 Loss

Best Test Performance So Far (%)

0.8

0.6

0.4

Wall-clock Time (hours)

0.2

(a) Wall-clock time versus the best test accuracy reached so far, evaluated once per epoch.

0.4

0.6

0.8

1.0 Distance

1.2

1.4

1.6

(b) Candidate loss versus distance from the guided reference in one sampling step.

Figure 3: Training efficiency and candidate-set quality for Llama-3.2-1B-Instruct on GSM8K. Candidate-set quality. Figure 3(b) gives a candidate-level view of one illustrative sampling step. Candidate loss tends to be lower near the guided reference, but the relationship is imperfect. This is why GradCodeS does not simply project to the nearest code point: it keeps several nearby candidates reachable and lets their realized discrete losses determine the accepted update. 8

Table 5: Candidate-selection statistics for different samTable 5 reports the empirical selection pling distributions under the same candidate budget. Valprobability and selected mean loss under ues are means over three independent runs, with standard the same candidate budget. The guided deviations shown as subscripts. sampler is selected substantially more often than Gaussian or one-hop sampling, inSampling Sel. Prob. Sel. Mean Loss dicating that gradient guidance improves Gaussian 5.83% 0.2955 candidate-set quality rather than merely 1-Hop Neighbor 38.16% 0.2963 increasing the number of forward evalu∇Ŵ L-Guided 48.62% 0.2951 ations. And the proposed code surrogate ∇Z L-Guided (Ours) 67.97% 0.2949 gradient (∇Z L)-guided sample better candidates than mapped ordinary weight gradient (∇Ŵ L)-guided, indicating it is a more effective first order signal. Together, Figure 3(b) and Table 5 are consistent with the proposal-guidance role analyzed in Proposition 2, without directly testing its candidate-count dependence. Component ablations. We ablate three components of the default GradCodeS configuration: gradient guidance, code-index updates, and scale updates. The w/o guide variant samples uniformly from a neighborhood of the current code without conditioning on (5). The ∇Ŵ L-guide replace code surrogate gradient with mapped ordinary weight gradient. The w/o scale update variant freezes group-wise scales and optimizes only codes, while the w/o code update variant freezes the indices and optimizes only group-wise scales.

61.0

66.4

54.9

60 55

52.6

52.8

54.1

54.5 30.3

30.0

69.2

56.3 33.1

35.0 32.5

52.1

37.5

MASSIVE EM

37.8

40.0

39.0

Performance (%)

42.5

AlpacaEval WR

41.6

GSM8K Acc.

27.5

50 GradCodeS

w/o guide

∇Ŵ L-guide

w/o scale update (code only)

w/o code update (scale only)

Figure 4: Component ablations for GradCodeS (LoRA) on Llama-3.2-1B-Instruct. Figure 4 shows that all three components contribute. Both removing gradient guidance and replacing ∇Z L signal with ∇Ŵ L consistently reduce performance, supporting the use of the code surrogate gradient to shape the proposal. Freezing either codes or scales causes larger task-dependent drops, indicating that both variables are important to the complete optimization procedure.

C ONCLUSION

We presented GradCodeS, a gradient-guided code-search method for fine-tuning low-bit model. The key idea is to analyze quantization geometry to find the steepest descent direction in code space, and use such first-order information to shape a candidate-sampling distribution, while letting the realized discrete loss decide which state to select to be deployment-faithful. Across reasoning, instructionfollowing, and structured language-understanding tasks, GradCodeS consistently improves fully 4-bit adaptation and remains compatible with different codebooks, low-rank code parameterizations, and alternating scale updates. The local analysis further clarifies why guided best-of-M code search can dominate both blind discrete search and single-point projected updates. An important next step is to scale the method to larger backbones and richer structured candidate-sampling families while preserving the same deployment-faithful update principle.

R EFERENCES Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pp. 10088–10115, 2023. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023. Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris J. Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 3831–3841, 2021. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Hyesung Jeon, Yulhwa Kim, and Jae-Joon Kim. L4Q: Parameter efficient quantization-aware finetuning on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2002–2024, 2025. Muheng Li and Ruqi Zhang. Reheated gradient-based discrete sampling for combinatorial optimization. Transactions on Machine Learning Research, 2025. Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoftQ: LoRA-fine-tuning-aware quantization for large language models. In International Conference on Learning Representations, 2024. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, 2024. Meng Liu, Haoran Liu, and Shuiwang Ji. Gradient-guided importance sampling for learning binary energy-based models. In International Conference on Learning Representations, 2023. Sebastian Loeschcke, Mads Toftrup, Michael J. Kastoryano, Serge Belongie, and Vésteinn Snæbjarnarson. LoQT: Low-rank adapters for quantized pretraining. In Advances in Neural Information Processing Systems, volume 37, pp. 115282–115308, 2024. Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl-Dickstein. Guided evolutionary strategies: Augmenting random search with surrogate gradients. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 4264–4273, 2019. Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtárik. PV-Tuning: Beyond straight-through estimation for extreme LLM compression. In Advances in Neural Information Processing Systems, volume 37, pp. 5074–5121, 2024. Sifeng Shang, Jiayi Zhou, Chenyu Lin, Minxian Li, and Kaiyang Zhou. Fine-tuning quantized neural networks with zeroth-order optimization. In International Conference on Learning Representations, 2026. Haoran Sun, Hanjun Dai, Bo Dai, Haomin Zhou, and Dale Schuurmans. Discrete langevin samplers via wasserstein gradient flow. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206, pp. 6290–6313, 2023a. Haoran Sun, Katayoon Goshvadi, Azade Nova, Dale Schuurmans, and Hanjun Dai. Revisiting sampling for combinatorial optimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 32859–32874, 2023b. 10

Yue Xiang, Dongyao Zhu, Bowen Lei, Dongkuan Xu, and Ruqi Zhang. Efficient informed proposals for discrete distributions via Newton’s series approximation. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206, pp. 7288–7310, 2023. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023. Yinggan Xu, Kajetan Schweighofer, Risto Miikkulainen, and Xin Qiu. Quantized evolution strategies: High-precision fine-tuning of quantized LLMs at low-precision cost. arXiv preprint arXiv:2602.03120, 2026. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. QA-LoRA: Quantization-aware low-rank adaptation of large language models. In International Conference on Learning Representations, 2024. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, 2024. Ruqi Zhang, Xingchao Liu, and Qiang Liu. A Langevin-like sampler for discrete distributions. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 26375–26396, 2022. Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. QuZO: Quantized zeroth-order fine-tuning for large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5341–5359, 2025.

E XPERIMENTAL D ETAILS

A.1

M ODELS AND E VALUATION TASKS

Models. We evaluate three instruction-tuned decoder-only language models, Qwen3-0.6B and Llama-3.2-1B and 3B-Instruct. Compact backbones are a useful stress test for fully low-bit adaptation: they are realistic for mobile, edge, and other memory-constrained deployment scenarios, and they make it easier to attribute performance differences to the optimization method rather than to very large overparameterized capacity. For each backbone, we compare the original 16-bit checkpoint, the directly quantized 4-bit checkpoint, mixed-precision adapter baselines, and fully 4-bit fine-tuned variants under the same data and evaluation protocol. Datasets and metrics. We evaluate on three datasets covering arithmetic reasoning, general instruction following, and mobile-style language understanding. For GSM8K, models are fine-tuned on the official training split and evaluated on the official test split; we report answer accuracy after extracting the final numerical answer. For Alpaca, models are fine-tuned on Alpaca Cleaned and evaluated with AlpacaEval; we report win rate under the same evaluator for all methods. For MASSIVE en-US, models are fine-tuned on the English-US training split and evaluated on the English-US test split. We formulate MASSIVE as structured semantic parsing and report Exact Match, Intent Accuracy, and Slot F1. A.2

Q UANTIZATION P ROTOCOL

Following the block-wise setup in Section 3, we quantize all linear parameters in the transformer blocks, including the attention and MLP projection matrices in every layer, while leaving the token embedding and LM head unquantized, following common LLM quantization practice (Dettmers et al., 2023; Frantar et al., 2023; Lin et al., 2024). We use the same target modules, quantization format, training data, and evaluation scripts whenever applicable. The reported 4-bit scores reflect the actually deployable checkpoint rather than the relaxed training state. A.3

BASELINES

We compare with three groups of baselines. A method is considered fully low-bit deployable only if inference does not require high-precision adapters or residual branches for the quantized modules. A method is considered mixed if it uses a quantized backbone but keeps high-precision inference-time adapter parameters. We also include unquantized references to separate adaptation quality from deployment cost. Unquantized references. • Base-16 evaluates the original 16-bit checkpoint without task fine-tuning. • 16-bit SFT fine-tunes 16-bit LoRA adapters on the 16-bit backbone and serves as a high-precision parameter-efficient adaptation reference (Hu et al., 2022). Mixed-precision baselines. • QLoRA-16A freezes a 4-bit backbone and fine-tunes a 16-bit LoRA adapter, giving a strong low-memory training baseline whose deployed model still contains high-precision adapter weights (Dettmers et al., 2023). • QA-LoRA uses quantization-aware group-wise low-rank adapters to improve compatibility between LoRA adaptation and quantized deployment, but optimizes continuous adapter variables (Xu et al., 2024). Fully 4-bit baselines. • Base-4 directly evaluates the quantized 4-bit checkpoint before adaptation. • QLoRA-4Merge follows QLoRA training, then merges the learned adapter into the backbone and requantizes the merged weights to 4-bit for inference (Dettmers et al., 2023). • LoQT trains low-rank adapters for quantized weights and periodically merges them into quantized full-rank weights; we evaluate its final merged-and-requantized checkpoint (Loeschcke et al., 2024). • L4Q is a representative projection/STE-style low-bit fine-tuning baseline (Jeon et al., 2025). 12

• PV-Tuning optimizes a compressed representation and alternates quantized variables with auxiliary continuous quantities (Malinovskii et al., 2024). • QuZO, QZO, and QES represent deployment-faithful discrete or zeroth-order low-bit optimization baselines (Zhou et al., 2025; Shang et al., 2026; Xu et al., 2026). Our method, GradCodeS, searches over low-bit code states directly. Every selected update is a deployable low-bit state, so the final model does not require a 16-bit adapter or high-precision residual branch at inference time. We evaluate both the full-matrix code update, GradCodeS (Full), and the low-rank code parameterization from Appendix G, GradCodeS (LoRA).

DATATYPE -S PECIFIC I NSTANTIATIONS OF THE U NIFIED Q UANTIZATION M ODEL

The common update rule depends on each datatype only through its ordered codebook, scale grouping, and recovery map. Table 6 summarizes these components for the evaluated datatypes. Datatype

Codebook

Scale/group structure

NF4 INT4 MXFP4

4 4 4

16 16 16

Non-uniform Uniform E2M1-style

Group-wise scaling Group-wise scaling Block/microscaling

Table 6: Datatype-specific components used by the unified quantization representation. (q)

(q)

For datatype q, let Cq = {c1 , . . . , cKq }, let gq (i, j) denote the scale group of entry (i, j), and let Sq (s) = [sgq (i,j) ]ij . The deployed weight is cq (Z, s) = Sq (s) ⊙ Cq (Z), W with datatype-specific code and scale constraints. A feasible code move changes an entry by (q) cq,ij = sg (i,j) c(q) ∆W − c q Zij +∆Zij Zij .

(6)

Thus the effective step automatically incorporates the datatype-specific codebook gap and scale. Here datatype-agnostic refers to the common optimization rule, not to identical feasible spaces; every candidate is still evaluated through the appropriate recovery map in (6).

D ERIVATION OF THE C ODE S URROGATE G RADIENT

c The chain rule would suggest ∇Z L = ∇W c L ⊙ ∇Z W , but the codebook lookup has no standard derivative. For an actual code-coordinate move ∆Z ∈ Zdout ×din , the deployed-weight change is exactly c (clip(Z + ∆Z, 1, K), s) − W c (Z, s) = S(s) ⊙ C(clip(Z + ∆Z, 1, K)) − C(Z) . W We therefore extend C(·) piecewise linearly around the current code, using the adjacent gaps G+ (Z) and G− (Z) from Section 4.1 as directional slopes. For ∥∆Z∥∞ ≤ 1, this continuation satisfies c (clip(Z + ∆Z, 1, K), s) − W c (Z, s) = S(s) ⊙ G+ (Z) ⊙ (∆Z)+ − G− (Z) ⊙ (∆Z)− . W For a descent-aligned move, a negative weight-gradient entry calls for a forward code move, whereas a positive entry calls for a backward code move. Writing σ = sign(∇W c L), the corresponding nonnegative directional slope is S(s) ⊙ G+ (Z) ⊙ σ− + G− (Z) ⊙ σ+ , which is precisely the effective step D Z, s, ∇W c L in Definition 1. Applying the chain rule to this local continuation yields ∇Z L(Z, s) = ∇W c L(Z, s) ⊙ D Z, s, ∇W c L(Z, s) , as stated in (3). For a uniform codebook and interior coordinates, the adjacent gaps coincide, so D reduces to the quantization gap multiplied by S(s). 13

P ROOFS FOR S ECTION 4.2

Restatement of Lemma 1. Define the local directional surrogate function of L(Z, s) by extending it to real-valued interpolation ∆Z ∈ Rdout ×din by c (Z, s) + D Z, s, ∇ c L(Z, s) ⊙ ∆Z . L̄Z,s (∆Z) = L W W Then ∇∆Z L̄Z,s (∆Z) ∆Z=0 = ∇Z L(Z, s). Consequently, whenever ∇Z L(Z, s) ̸= 0, the direction −∇Z L(Z, s) is the steepest descent direction of L̄Z,s at ∆Z = 0 under the Frobenius norm. Proof of Lemma 1. By definition, c (Z, s) + D Z, s, ∇ c L(Z, s) ⊙ ∆Z . L̄Z,s (∆Z) = L W W Since D Z, s, ∇W c L(Z, s) is fixed at the current point, the map c (Z, s) + D Z, s, ∇ c L(Z, s) ⊙ ∆Z ∆Z 7→ W W is affine, and its differential in direction H is D Z, s, ∇W c L(Z, s) ⊙ H. Therefore, by the chain rule, c ∇∆Z L̄Z,s (∆Z) = ∇W c L W (Z, s) + D Z, s, ∇W c L(Z, s) ⊙ ∆Z ⊙ D Z, s, ∇W c L(Z, s) . Evaluating at ∆Z = 0 gives ∇∆Z L̄Z,s (∆Z) ∆Z=0 c = ∇W c L W (Z, s) ⊙ D Z, s, ∇W c L(Z, s) = ∇W c L(Z, s) ⊙ D Z, s, ∇W c L(Z, s) = ∇Z L(Z, s), which proves the exact-gradient claim. For the steepest-descent statement, consider the first-order model m(∆Z) = L̄Z,s (0) + ⟨∇Z L(Z, s), ∆Z⟩F . For any ∆Z with ∥∆Z∥F ≤ r, Cauchy–Schwarz gives ⟨∇Z L(Z, s), ∆Z⟩F ≥ −∥∇Z L(Z, s)∥F ∥∆Z∥F ≥ −r∥∇Z L(Z, s)∥F . Equality is attained at ∆Z = −r

∇Z L(Z, s) , ∥∇Z L(Z, s)∥F

whenever ∇Z L(Z, s) ̸= 0. Hence −∇Z L(Z, s) is the steepest descent direction of L̄Z,s at ∆Z = 0 under the Frobenius norm. Restatement of Proposition 1. Let ∆Z ∈ {−1, 0, 1}dout ×din be a feasible one-step code move such that Z + ∆Z ∈ Z, ∆Z ⊙ ∇W c L(Z, s) ≤ 0 entrywise. Then the surrogate linear term matches the exact first-order deployed-weight change: D E c c ⟨∇Z L(Z, s), ∆Z⟩F = ∇W . c L(Z, s), W (Z + ∆Z, s) − W (Z, s) F

Proof of Proposition 1. We first prove the exact first-order identity D E c c ⟨∇Z L(Z, s), ∆Z⟩F = ∇W . c L(Z, s), W (Z + ∆Z, s) − W (Z, s) F

Fix one coordinate (i, j). If ∆Zij = 0, both sides contribute zero. If ∆Zij = 1, feasibility gives Zij < K, and gradient alignment c L(Z, s))ij ≤ 0. When (∇W c L(Z, s))ij < 0, the gives (∇W definition of D Z, s, ∇W L(Z, s) selects the forward gap, so c cij (Z + ∆Z, s) − W cij (Z, s) = Sij (s) cZ +1 − cZ W ij ij = d+ ij (Z, s) = D Z, s, ∇W c L(Z, s) ij ∆Zij . If instead (∇W c L(Z, s))ij = 0, then both first-order contributions are zero. If ∆Zij = −1, feasibility gives Zij > 1, and gradient alignment gives (∇W c L(Z, s))ij ≥ 0. When (∇W c L(Z, s))ij > 0, the definition of D Z, s, ∇W L(Z, s) selects the backward gap, so c cij (Z + ∆Z, s) − W cij (Z, s) = Sij (s) cZ −1 − cZ W ij ij = −d− ij (Z, s) = D Z, s, ∇W c L(Z, s) ij ∆Zij . Again, if (∇W c L(Z, s))ij = 0, both first-order contributions are zero. Therefore, for every coordinate, c c (∇W c L(Z, s))ij Wij (Z + ∆Z, s) − Wij (Z, s) = (∇W c L(Z, s))ij D Z, s, ∇W c L(Z, s) ij ∆Zij . Summing over coordinates yields D E c c ∇W = ∇W c L(Z, s), W (Z + ∆Z, s) − W (Z, s) c L(Z, s) ⊙ D Z, s, ∇W c L(Z, s) , ∆Z F F

= ⟨∇Z L(Z, s), ∆Z⟩F , which proves (4). Restatement of Corollary 1. Under the assumptions of Proposition 1, if L is β-smooth as a c (Z, s), then function of the deployed weight in a neighborhood of W L(Z + ∆Z, s) ≤ L(Z, s) + ⟨∇Z L(Z, s), ∆Z⟩F +

2 β c c (Z, s) . W (Z + ∆Z, s) − W 2 F

Proof of Corollary 1. If L is β-smooth as a function of the deployed weight in a neighborhood of c (Z, s), the descent lemma gives W E D c (Z + ∆Z, s) ≤ L W c (Z, s) + ∇ c L W c (Z, s) , W c (Z + ∆Z, s) − W c (Z, s) L W W F

2 β c c (Z, s) . + W (Z + ∆Z, s) − W 2 F

By Proposition 1, D E c c c ∇W = ⟨∇Z L(Z, s), ∆Z⟩F . c L W (Z, s) , W (Z + ∆Z, s) − W (Z, s) F

Substituting this identity together with c (Z, s) , L(Z, s) = L W

c (Z + ∆Z, s) L(Z + ∆Z, s) = L W

yields L(Z + ∆Z, s) ≤ L(Z, s) + ⟨∇Z L(Z, s), ∆Z⟩F + as claimed. 15

2 β c c (Z, s) , W (Z + ∆Z, s) − W 2 F

A C ONCRETE FACTORIZED C ANDIDATE -S AMPLING D ISTRIBUTION

To instantiate the local candidate-sampling rule in Section 4.3, define the neighborhood around the guided reference Z ref by N (ρ) (Z ref ) = Z ′ ∈ Z | ∥Z ′ − Z ref ∥∞ ≤ ρ , where ρ ∈ Z+ is the code search radius. For efficient sampling in high-dimensional spaces, we sample each matrix entry (i, j) ∈ [1, dout ] × [1, din ] independently from a discrete scalar distribution pij (u | Z ref ), which jointly induces Ydout Ydin m p(Z m | Z ref ) = pij (Zij | Z ref ). i=1

j=1

We choose pij (u | Z ref ) = P

ref ϕ(|u − Zij |)

ref |) ϕ(|v − Zij (ρ) v∈Nij (Z ref )

(ρ) ref Nij (Z ref ) = u ∈ [K] | |u − Zij |≤ρ

where ϕ(x) is a monotonically decreasing function, such as (x + ε)−1 with ε > 0 or exp(−x2 ), so that code states closer to Z ref receive larger coordinatewise probability mass.

E XPECTED D ESCENT OF THE C ODE -S EARCH S UBSTEP

For the proof of Proposition 2, we analyze the deterministic full-batch code-search substep of Algorithm 1 with a fixed scale s. At iteration t, define the code-space surrogate gradient and continuous reference step by gt = ∇Z L(Zt , s),

¯ t = −ηZ gt , ∆

¯ t. Ztref = Zt + ∆

For a sampled candidate state Ztm , let m ∆m t = Zt − Zt

denote its code update relative to the current state. The proof uses the following assumptions. Assumption A1 (smoothness on admissible sampled moves). There exist constants β > 0 and κ > 0 such that, for every iteration t and every candidate update ∆ considered below, Corollary 1 applies and c (Zt + ∆, s) − W c (Zt , s) ≤ κ∥∆∥F . W (7) F

Assumption A2 (near-reference proposal mass). Fix some c ∈ [0, 1). For each iteration t, define the good-update set n o Gt = ∆ ∆ is gradient-aligned and one-step feasible, and ∥∆ + ηZ gt ∥F ≤ c ηZ ∥gt ∥F . Let qt denote the total proposal mass of candidate states whose update belongs to Gt : X qt = p(Z ′ | Ztref ). ′ ′ Z : Z −Zt ∈Gt

Proof of Proposition 2. Fix an iteration t. If ∆ ∈ Gt , then by definition of Gt , ⟨gt , ∆⟩ = ⟨gt , −ηZ gt ⟩ + ⟨gt , ∆ + ηZ gt ⟩ ≤ −ηZ ∥gt ∥2F + ∥gt ∥F ∥∆ + ηZ gt ∥F ≤ −(1 − c)ηZ ∥gt ∥2F , and also ∥∆∥F ≤ ∥ηZ gt ∥F + ∥∆ + ηZ gt ∥F ≤ (1 + c)ηZ ∥gt ∥F . Combining this with (7) gives c (Zt + ∆, s) − W c (Zt , s) W 16

2 F

2 ≤ κ2 (1 + c)2 ηZ ∥gt ∥2F .

Therefore, by Corollary 1, βκ2 ηZ 2 L(Zt + ∆, s) ≤ L(Zt , s) − 1 − c − (1 + c) ηZ ∥gt ∥2F = L(Zt , s) − αηZ ∥gt ∥2F . 2 Now let

n o Et = at least one of the M sampled candidates has update in Gt .

By independence of the M samples, Pr(Et | Zt ) = 1 − (1 − qt )M . On the event Et , the sampled set contains a candidate whose loss is at most L(Zt , s) − αηZ ∥gt ∥2F . On Et , the best-of-M rule therefore returns a candidate whose loss is no larger than this value. On the complementary event Etc , the strict accept-if-improving rule gives L(Zt+1 , s) ≤ L(Zt , s). Combining the two cases yields the pointwise bound L(Zt+1 , s) ≤ L(Zt , s) − αηZ ∥gt ∥2F 1Et . Taking conditional expectation gives E[L(Zt+1 , s) | Zt ] ≤ L(Zt , s) − α ηZ 1 − (1 − qt )M ∥gt ∥2F , which proves the claimed one-step descent inequality.

L OW-R ANK C ODE U PDATE

c (Z), The core search rule described above only requires a code matrix Z ∈ Z to induce weight W and a differentiable loss to compute (3). As a result, the same rule can also be used with other parameterizations. Low-rank parameterizations are widely used in LLM fine-tuning to reduce the number of optimization variables and improve parameter efficiency. We apply this principle in code space by parameterizing a local code update as a low-rank decomposition with two integer-valued factors: ∆Z(A, B) = AB ⊤ , A ∈ Zdout ×R , B ∈ Zdin ×R . This reduces the number of searched coordinates from dout din to R(dout + din ). Unlike ordinary LoRA, A and B are not inference-time residual adapters; after training, only the deployed code state clip(Z + ∆Z(A, B), 1, K) needs to be stored. The factor-wise guidance follows from the same surrogate chain rule. Given ∇Z L by (3) at the induced code state, we use ∇A L = ∇Z L B,

⊤

∇B L = (∇Z L) A.

We form guided references for A and B, sample integer-valued factor candidates, and select by the deployed loss after merging the induced code update.

Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space

Analyze & explore related research in Conceptio

Related documents