Is Domain Adaptation Always Helpful? A Frozen-Backbone Study of Cross-Domain Sentiment Transfer Phat Tran
Artin Lahni Pranav Kulkarni Oregon State University
Yaolun Zhang
arXiv:2607.05937v1 [cs.CL] 7 Jul 2026
{tranphat, lahnia, kulkarnp, zhanyaol}@oregonstate.edu
els (PLMs) used as frozen feature extractors, with lightweight task heads trained on top of their embeddings [1], [2]. As these backbone models have scaled from hundreds of millions to billions of parameters, their reported performance on standard benchmarks has grown correspondingly impressive [3]. However, this progress raises a critical measurement problem: at scale, PLMs may be exposed to benchmark evaluation examples during pretraining, making it difficult to distinguish genuine generalization from memorization. When a frozen backbone has already seen SST-2 [4] or Financial PhraseBank [5] during pretraining, high zero-shot accuracy reflects data contamination rather than genuine generalization. This makes it fundamentally unclear whether explicit domain adaptation techniques, methods that align a model’s representations across source and target domains, are truly helpful, or whether they merely compensate for contamination artifacts introduced by smaller, less-contaminated backbones. We address this question directly: “When does domain adaptation help for sentiment transfer?” To reduce reliance on potentially contaminated benchmark memorization, we use the Qwen3-Embedding series (0.6B, 4B, and 8B parameters) [6], an embedding family that provides a controlled scale progression for studying frozen-backbone transfer. This yields a setting in which the effects of backbone scale and domain specialization can be compared more directly. On top of these frozen embeddings, we train a lightweight MLP adapter using three complementary domain adaptation objectives: DomainAdversarial Neural Networks (DANN) [7], Maximum Mean Discrepancy (MMD) [8], and Supervised Contrastive Learning (SCL) [9]. We train on consumer product and restaurant reviews (Yelp Reviews [10] and Amazon Polarity [10], [11]) and evaluate zeroshot transfer to movie reviews (SST-2 [4], [12]) and financial news (Financial PhraseBank [5]), with no
Abstract—Sentiment analysis with frozen pretrained language model (PLM) backbones has become a common paradigm, yet the practical benefit of explicit domain adaptation remains unclear, particularly when backbones encode varying degrees of target-domain knowledge. We present a preliminary case study evaluating a controlled family of frozen embedding backbones (Qwen3-Embedding 0.6B, 4B, 8B), alongside RoBERTa-base and FinBERT. We train a lightweight MLP adapter on consumer reviews using Domain-Adversarial Neural Networks (DANN), Maximum Mean Discrepancy (MMD), and Supervised Contrastive Learning (SCL), and evaluate transfer to movie reviews (SST-2) and a heavily restricted subset of financial news (Financial PhraseBank). Within this constrained sample, we observe two distinct transfer patterns. On SST-2, domain adaptation provides negligible gain regardless of scale. On the financial subset, explicit domain adaptation appears to recover substantial performance for small general-purpose backbones. Notably, we find that adversarial alignment (DANN) is associated with degraded performance for domain-specialized backbones like FinBERT, consistent with erosion of pre-existing domain-specific structure, whereas supervised contrastive loss appears to preserve it. These preliminary findings suggest that the efficacy of explicit domain adaptation is highly contingent on whether the frozen backbone already possesses target-domain coverage. Index Terms—Sentiment analysis, domain adaptation, pretrained language models, frozen backbones
I. Introduction Sentiment analysis, the task of classifying the subjective polarity of text, is a cornerstone application of natural language processing (NLP), powering everything from product recommendation systems to financial market monitoring. Modern approaches increasingly rely on large pre-trained language mod-
1
the field shifted decisively toward transferring representations from large pre-trained language models [2]. A common instantiation uses BERT-style encoders as frozen feature extractors, where a simple probe or classification head is trained on top of fixed representations [13]. While this probing paradigm reveals what information is encoded in pretrained representations, it also introduces a measurement risk: as model scale grows, benchmark examples may appear in pretraining corpora, making it difficult to distinguish genuine generalization from memorization [14]–[16]. Our work is motivated by this risk. We use a controlled family of frozen Qwen3-Embedding backbones at multiple scales, together with RoBERTa and FinBERT baselines, to study when domain adaptation improves transfer under fixed representations.
target labels used during training. Our experiments reveal two qualitatively distinct transfer regimes, clarifying when domain adaptation is and is not worth applying. On SST-2, which shares informal opinionated language with the source domains, all backbone configurations achieve strong zero-shot performance (macro F1 of 0.85 to 0.91), and adding any combination of DA losses provides negligible additional benefit. On Financial PhraseBank, however, the domain shift is severe: the 0.6B backbone without DA achieves only 0.309 F1 , while the full DANN and MMD combination recovers 0.637. Critically, we find that a domain-specialized backbone (FinBERT) reaches 0.902 F1 without any adaptation, but its performance is actively harmed by adversarial alignment (DANN), which erodes the domain-specific structure that makes it valuable. Together, these findings suggest a unified principle: explicit domain adaptation is necessary and effective only when the frozen backbone lacks coverage of the target distribution, and it becomes harmful when applied against a backbone whose pretraining already encodes target domain structure. This work makes the following contributions: • We present an empirical case study of domain adaptation with frozen sentiment backbones using Qwen3-Embedding models of different scales, together with RoBERTa and FinBERT baselines. • We observe that adaptation has little measurable effect on close-domain transfer to SST-2, but substantially improves cross-domain transfer to Financial PhraseBank in our setting, with gains up to 32.8 macro F1 points on a 93-example evaluation set. • We find that adversarial alignment can harm domain-specialized backbones such as FinBERT, while supervised contrastive learning achieves the best financial result observed in our experiments. • We propose a preliminary, exploratory guideline: use distribution-matching adaptation when the backbone lacks target-domain coverage, and use contrastive refinement when target-domain structure is already present, pending validation on larger benchmarks.
B. Unsupervised Domain Adaptation for NLP Unsupervised domain adaptation (UDA) for NLP addresses the challenge of transferring a model trained on labeled source text to an unlabeled target domain [17]. The dominant family of approaches encourages domain-invariant representations through adversarial training, of which DANN [7] is the canonical example, using a gradient reversal layer to make the feature extractor indistinguishable across domains. A complementary family uses distribution-matching losses such as MMD [8], which minimizes a kernelbased distance between source and target feature distributions without requiring an explicit discriminator. However, theoretical analysis of MMD for domain adaptation has shown that naive distribution matching can degrade intra-class discriminability [18], a finding consistent with our own ablation results showing that adding contrastive loss on top of DANN and MMD hurts performance for small-capacity backbones. Crucially, all of these prior works apply adaptation while jointly fine-tuning the backbone, making it difficult to disentangle adaptation signal from backbone drift [19]. Our two-stage frozen pipeline isolates this variable by design. C. Contrastive Learning for Representation Alignment SCL [9] trains representations by pulling same-class samples together and pushing apart cross-class pairs using temperature-scaled cosine similarity. When applied to domain adaptation, contrastive objectives are appealing because they can sharpen class boundaries without requiring target labels [20]. Prior work on
II. Related Work A. Sentiment Analysis with Pre-trained Language Models Early neural approaches to sentiment analysis relied on task-specific architectures trained from scratch, but
2
backbone itself. We evaluate five backbone configurations. From the Qwen embedding family, we include Qwen3-Embedding-8B (8B parameters, 4096dimensional output), Qwen3-Embedding-4B (4B parameters, 2560-dimensional output), and Qwen3Embedding-0.6B (0.6B parameters, 1024-dimensional output) [6]. In addition, we consider two widely used encoder baselines: RoBERTa-base (125M parameters, 768-dimensional output) [2] and FinBERT (110M parameters, 768-dimensional output) [1], a BERT variant further pre-trained on financial corpora. All backbones are kept fully frozen during training. Embeddings are pre-computed once per dataset split and cached, so no gradients flow into the backbone at any stage.
aspect-based sentiment adaptation [21] shows that combining adversarial and contrastive losses improves robustness to domain shift. We build on this intuition but uncover an important asymmetry: for domainspecialized backbones such as FinBERT, contrastive loss is the single most effective component (+0.076 macro F1 ), while DANN appears to suppress domainspecific information that makes those representations valuable. This interaction between backbone specialization and adaptation-objective type has not been characterized in prior frozen-backbone settings. D. Pseudo-Labeling in Semi-Supervised Adaptation Pseudo-labeling augments supervised training with high-confidence model predictions on unlabeled target data, providing a weak supervision signal across domain boundaries. Chen et al. [22] show that pseudolabel guided UDA of contextual embeddings outperforms strong baselines on both named entity recognition and sentiment analysis, using masked language modeling on target text to bootstrap label assignment. Compared to that approach, we treat pseudo-labeling as a supplementary signal activated at epochs 4 and 5 (of 5 total) with a high confidence threshold (0.95), layered on top of the primary DANN, MMD, and contrastive objectives rather than as a standalone adaptation mechanism. This conservative design ensures the adapter has converged sufficiently on source data before target predictions become reliable enough to incorporate, preventing confirmation bias from corrupting adaptation in early training.
B. Adapter Architecture The adapter is a two-layer MLP projection head followed by a linear classifier, with approximately 6.3M trainable parameters, regardless of the backbone. The projection maps the backbone embedding dimension to a shared 1024-dimensional space via two blocks of Linear → LayerNorm → GELU → Dropout(0.1). The classifier maps to 512 dimensions before producing binary logits. A separate domain discriminator, consisting of Linear(1024 → 512) → ReLU → Dropout(0.1) → Linear(512 → K), is used for the DANN objective. C. Domain Adaptation Losses We study three domain adaptation objectives combined with standard cross-entropy. The base supervised objective is standard cross-entropy on labeled source samples:
III. Methodology We adopt a two-stage pipeline that decouples representation learning from domain adaptation. A frozen pre-trained backbone encodes each input text into a fixed-length vector. A lightweight adapter is then trained on top of these frozen embeddings using labeled source data and domain adaptation losses applied to unlabeled target samples. Fig. 1 illustrates the overall pipeline. Target-domain embeddings are used with stop-gradient to ensure that domain-alignment losses do not update the frozen backbone.
C n X 1X yic log ŷic LCE = − n i=1 c=1
(1)
where yic is the ground-truth label indicator and ŷic is the predicted probability for class c ∈ {0, 1}. a) DANN: A gradient reversal layer (GRL) negates gradients flowing from the domain discriminator into the feature extractor, encouraging domaininvariant representations:
A. Frozen Backbone
LDANN =
Freezing the backbone is a deliberate design choice: it eliminates representation drift during adaptation and allows performance differences across domains to be attributed to the adapter and domainadaptation objectives rather than to changes in the
n 1X Ldom (D(GRL(fi )) , di ) n i=1
(2)
where fi is the projected feature for sample i, di ∈ {0, . . . , K−1} is its domain label, D(·) is the domain discriminator, and GRL negates gradients during backpropagation with scale λDANN , which warms
3
Frozen
Source Text
Projection Head
Classifier
Linear ↓
1024→512→2
d
es ∈ R
Yelp / Amazon
Frozen Backbone
src emb.
Target Text
et ∈ Rd
SST-2 / Financial
tgt emb. Frozen
Trainable
LMMD
LayerNorm ↓ GELU ↓ 1024-dim
Qwen3 / RoBERTa / FinBERT
LCE
Ltotal Domain Discriminator
LDANN
GRL + MLP
LCon
target (unlabeled)
Fig. 1: Overview of the frozen-backbone domain adaptation pipeline. Source and target texts are encoded by a fixed pretrained backbone and mapped into a shared adapter space. The classifier is trained with source-label cross-entropy, while DANN, MMD, and supervised contrastive losses encourage domain alignment and class separation. Only the adapter, classifier, and domain discriminator are updated during training.
up linearly over the first 500 steps to prevent early instability. The inner loss follows the same crossentropy form as Equation (1), applied over K domain classes rather than sentiment classes. b) MMD: Maximum mean discrepancy with a Gaussian kernel (σ=1.0) minimizes the distance between source and target feature distributions at each training step: n m 1X 1 X LMMD (S, T ) = ϕ(si ) − ϕ(tj ) n i=1 m j=1
predictions become reliable enough to incorporate, preventing confirmation bias from corrupting adaptation in early training. E. Training Details All adapters are trained for 5 epochs with AdamW (lr=2×10−4 , weight decay 0.01) [23], cosine learning rate schedule with 6% warmup, gradient clipping at 1.0, and bfloat16 mixed precision. Batch size is 32 with gradient accumulation over 2 steps. Following standard practice in multi-objective domain adaptation [7], [8], the loss weights are set to λDANN =0.1, λMMD =0.05, and λCon =0.05, balancing the contribution of each alignment objective relative to the primary cross-entropy supervision signal. All experiments use a fixed random seed of 42 and run on a single NVIDIA GPU.
2
(3) H
where ϕ is the feature map induced by the Gaussian kernel, S and T are the source and target feature sets, and n, m are their respective batch sizes. c) Supervised Contrastive Loss: Same-class feature pairs are pulled together and cross-class pairs pushed apart using temperature-scaled cosine similarity (τ =0.07), using source labels only: X −1
Feature Space After Domain Adaptation t-SNE by Domain
exp(zi · zp / τ ) log P LCon = (4) |P (i)| p∈P (i) a̸=i exp(zi · za / τ ) i∈I
t-SNE by Sentiment yelp amazon sst2 financial
10
X
where P (i) is the set of same-class indices in the batch, z are ℓ2 -normalized features from the projection head, and τ = 0.07 is the temperature. The total loss is:
5
5
0
0
5
5
10
10
15
40
20
0
20
40
Class 0 Class 1
10
15
40
20
0
20
40
Fig. 2: t-SNE of adapter features (Qwen3-0.6B), colored by domain (left) and sentiment (right), showing domain-mixed yet class-separated representations.
L = LCE + λDANN LDANN + λMMD LMMD + λCon LCon (5) D. Pseudo-Labeling
IV. Experiments
At epochs 4 and 5 (of 5 total), high-confidence target predictions (softmax confidence ≥ 0.95) are incorporated into training with their predicted labels, providing weak supervision on the target domain. This conservative schedule ensures the adapter has converged sufficiently on source data before target
A. Setup a) Datasets: We train on two labeled source domains (Yelp Reviews and Amazon Polarity) and evaluate zero-shot transfer to two target domains (SST-2 and Financial PhraseBank), with no target
4
Domain Transfer - Evaluation Metrics yelp
0.953
amazon
0.929
0.929
0.929
0.8
sst2
0.953
1.0
0.893
0.893
0.893
0.7
financial
labels used during training. Dataset statistics are shown in Table I. We convert Yelp and Financial PhraseBank to binary sentiment by excluding their neutral classes: 3-star reviews for Yelp and neutral examples for Financial PhraseBank. We note that DANN, MMD, and pseudo-labeling draw unlabeled target features from the same pool of examples later used for zero-shot evaluation, since Table I shows no separate target training split. This is standard practice in transductive unsupervised domain adaptation, but it means our reported “zeroshot” results reflect transductive rather than inductive generalization, and should not be interpreted as performance on data unseen at training time. b) Main Pipeline vs. Ablation Protocol: For the main results in Table II, we train the full pipeline with all three DA losses. We first use SST-2 as the adaptation target and then adapt to Financial PhraseBank, following a curriculum intended to stabilize training on the harder financial domain. For the ablation study (§IV-C), each DA configuration is trained from scratch with Financial PhraseBank as the sole target domain. This provides a cleaner measure of each component’s individual contribution.
0.953
0.699
0.689
0.701
accuracy
macro_f1
weighted_f1
0.9
0.6 0.5
Fig. 3: Domain transfer heatmap for Qwen3-0.6B, with rows as source domains and columns as evaluation domains. Off-diagonal entries show zero-shot performance.
C. Ablation Study To isolate each component’s contribution, we train the adapter once for each of six DA configurations per backbone, targeting only the financial domain from scratch. Table III reports macro F1 on Financial PhraseBank. Because this evaluation set contains only 93 examples, single-run macro F1 differences of a few points should be interpreted with caution. We report multi-seed variance only for the 8B backbone due to compute constraints, and treat other single-run comparisons in this section as suggestive rather than conclusive. For SST-2, we ran the six-condition ablation across all four backbone families, as shown in Table IV. Every backbone exhibits the same flat pattern: all conditions fall within 0.006 of each other per model, the plain baseline is best or tied-best in every column, and the full combination is at or below baseline in all cases. This finding holds across a 13× range of model scale and across both general-purpose and domainspecialized backbones, confirming that DA provides negligible benefit on close-domain transfer regardless of capacity or objective. The financial ablation reveals three distinct patterns across backbone types. a) Small general-purpose backbones (0.6B): Without DA, the adapter generalizes poorly to financial text (0.309), reflecting the backbone’s limited financial pretraining. DA provides substantial recovery: DANN+MMD together yield the largest gain (+0.328 over baseline). However, adding contrastive
B. Main Results Table II reports macro F1 for all backbone–domain combinations under the full DA pipeline. Results reveal two distinct transfer regimes. On SST-2, performance is stable across all models (0.85– 0.91), with the gap between the smallest and largest models only 6.5 points. Consumer reviews and movie reviews share informal, opinionated language, so even compact backbones generalize effectively. The financial domain shows a different pattern: Qwen3Embedding models span a 25.5-point range (0.689 to 0.944) that tracks closely with model scale, while RoBERTa (0.748) and FinBERT (0.731) fall in between, despite being substantially smaller than the Qwen3-Embedding models. Notably, FinBERT underperforms RoBERTa on the financial domain under the full pipeline despite its financial pretraining. As the ablation study suggests, this is not due simply to a lack of financial representations, since its no-DA baseline performs considerably better, but rather because the combined domain adaptation losses in the full pipeline, particularly DANN, degrade the domain-specific features that make FinBERT effective.
5
Domain
Role
Train
Eval
Pos/Neg (%)
Yelp Reviews Amazon Polarity SST-2 Financial PhraseBank
Source Source Target Target
31,880 40,000 — —
3,997 5,000 872 93
46 / 54 51 / 49 51 / 49 61 / 39
Avg. Len
Source
128 76 20 24
[10] [10], [11] [4] [5]
TABLE I: Dataset statistics. Yelp counts are after binarization, with 3-star neutral reviews removed. Financial PhraseBank counts are after removing the neutral class. Pos/Neg shows class balance (%). Avg. Len is the mean word count per sample. Target labels are held out during training. Model
Macro F1
Params
Qwen3-Emb-0.6B Qwen3-Emb-4B Qwen3-Emb-8B RoBERTa-base FinBERT
0.6B 4B 8B 125M 110M
Accuracy
Yelp
Amazon
SST-2
Fin.
Yelp
Amazon
SST-2
Fin.
0.953 0.972 0.976 0.951 0.921
0.929 0.948 0.950 0.918 0.891
0.893 0.908 0.913 0.848 0.854
0.689 0.793 0.944 0.748 0.731
0.953 0.972 0.976 0.951 0.922
0.929 0.948 0.950 0.918 0.891
0.893 0.908 0.913 0.849 0.854
0.699 0.796 0.946 0.774 0.731
TABLE II: Macro F1 and accuracy across all backbone–domain combinations in the full DA pipeline. Source domains use in-domain test splits, and target domains are evaluated in a zero-shot setting. Condition Baseline + DANN + MMD + Contrastive DANN + MMD All three
D
M
C
✓ ✓ ✓ ✓ ✓
✓ ✓
✓
0.6B
4B
8B†
FinBERT
0.309 0.389 0.442 0.549 0.637 0.537
0.632 0.708 0.728 0.613 0.665 0.689
0.421 0.428 0.331 0.399 0.412 0.428
0.902 0.796 0.923 0.978 0.891 0.763
TABLE III: Financial domain ablation (macro F1 ). D = DANN, M = MMD, C = Contrastive. Each condition trains from scratch on financial as the sole target. †: 8B values are means across 3 seeds, with high variance (std up to 0.118) due to the small 93-sample eval set.
loss to DANN+MMD reduces performance (0.537 vs. 0.637), suggesting a negative interaction when all three components operate simultaneously on a lowcapacity backbone. b) Larger general-purpose backbones (4B): The stronger backbone produces a better no-DA baseline (0.632). DANN and MMD each improve performance individually (+0.075 and +0.096, respectively), while contrastive loss alone slightly reduces performance (0.019). DANN+MMD reaches 0.665, which is lower than MMD alone, indicating that the two alignment losses do not combine additively in this setting. The full combination is the best configuration for 4B (0.689), suggesting that contrastive regularization becomes beneficial at this capacity only when combined
with the alignment losses. c) Large general-purpose backbones (8B): At 8B scale, all six conditions cluster between 0.33 and 0.43, averaged across three seeds, with no condition clearly dominant. The high variance (std up to 0.118) and the small 93-sample financial evaluation set prevent drawing strong conclusions from these ablation values. Notably, a large gap exists between the ablation baseline (0.421) and the 8B main result (0.944). Unlike smaller models, the 8B backbone appears to benefit substantially from the curriculum training protocol, which adapts first to SST-2 and then to Financial PhraseBank. In contrast, the fresh-start financial ablation appears unable to fully exploit the backbone’s representation quality. We therefore treat the 8B fi-
6
Condition Baseline + DANN + MMD + Contrastive DANN + MMD All three
D
M
C
✓ ✓ ✓ ✓ ✓
✓ ✓
✓
0.6B
4B
8B†
FinBERT
0.878 0.877 0.881 0.881 0.876 0.875
0.928 0.910 0.908 0.921 0.915 0.921
0.918 0.919 0.921 0.919 0.919 0.915
0.856 0.848 0.841 0.835 0.850 0.828
TABLE IV: SST-2 ablation (macro F1 ). D = DANN, M = MMD, and C = Contrastive. Bold marks the best condition per backbone. The plain baseline leads or ties in every column, and DA shows no consistent benefit on close-domain transfer.
nancial ablation as indicative rather than conclusive. d) Domain-specialized backbones (FinBERT): FinBERT’s no-DA baseline (0.902) is the highest cold-start result in our study, confirming that financial pretraining provides strong prior knowledge. The component interactions differ sharply from those observed for general-purpose backbones. DANN is actively harmful (-0.106): gradient reversal forces the adapter to make representations less distinguishable across financial and non-financial domains, erasing the domain-specific structure that makes FinBERT valuable. MMD provides a small benefit (+0.021). Contrastive loss produces the largest single-run improvement (+0.076) in our ablation, reaching 0.978 on the 93-example evaluation set, the best financial result we observe across configurations and models. This result has not been validated across multiple seeds and should be treated as preliminary. It sharpens class separation within the existing financial representation space without disrupting domain structure. The full combination (0.763) performs worse than no domain adaptation, driven by the destructive effect of DANN.
Confusion Matrix - sst2
0
350 387
41
300
1
True
250 200 52
392
150 100
0
Predicted
1
50
(a) SST-2
0
Confusion Matrix - financial 24
12
40 35 30
True
D. Discussion
1
Our results show that the benefit of domain adaptation is determined by one factor: whether the frozen backbone already covers the target distribution. On SST-2, consumer reviews and movie reviews share informal opinionated language and similar lexical sentiment cues, so all backbone configurations produce representations that are already linearly separable on the target. In this regime, DA objectives have little distribution gap to close, and the added regularization from DANN and MMD marginally degrades class boundaries rather than improving them, explaining the flat ablation pattern across all conditions.
25 16
41
20 15
0
Predicted
1
(b) Financial
Fig. 4: Confusion matrices for Qwen3-0.6B on zeroshot transfer targets.
7
alters adapter convergence, a dynamic that warrants isolated study in future work.
Financial PhraseBank presents a fundamentally different setting. Its formal, entity-centric language relies on domain-specific vocabulary that is largely absent from consumer review corpora, placing source and target representations in different regions of feature space. For small general-purpose backbones, explicit distribution matching with DANN and MMD appears to bridge this gap, recovering up to 32.8 macro F1 points for the 0.6B backbone in our single-run evaluation. This is consistent with the regime these objectives were designed for. The FinBERT results reveal a fundamental incompatibility between adversarial alignment and frozen domain-specialized backbones. DANN is designed to encourage a jointly trained feature extractor to discard domain-specific structure. When the backbone is frozen, however, that structure is fixed and cannot be re-learned within the backbone. Gradient reversal therefore forces the adapter to suppress informative financial features, weakening the specialized representation manifold and reducing macro F1 by 0.106. Supervised contrastive loss avoids this failure mode. Instead of aligning source and target distributions, it sharpens sentiment boundaries within the existing representation space while preserving financialdomain structure. This yields 0.978 macro F1 , the best result we observe across all configurations. These observations suggest a candidate selection heuristic worth further validation: use distribution-matching DA when the backbone lacks target-domain coverage, and use contrastive refinement when target-domain structure is already present. This exploratory setup is subject to several important limitations. First, our focus on binary sentiment and product/restaurant source domains means these trends may not generalize to multi-class tasks or other text types. Second, domain adaptation loss weights are held fixed across all configurations, which precludes model-specific optimization. Third, our Financial PhraseBank evaluation set is relatively small, which induces high variance in some configurations. For example, the 8B model has a standard deviation of 0.118 across seeds. As a result, these results should be interpreted as directional trends rather than definitive benchmarks. Finally, the main pipeline’s sequential curriculum, which adapts to SST-2 before financial text, confounds direct comparison with our fresh-start ablations. The substantial performance divergence between these two settings for large-capacity models suggests that sequential target exposure significantly
V. Conclusion This work examines when domain adaptation improves sentiment transfer with frozen pre-trained backbones. By keeping the backbone fixed and training only a lightweight adapter, we isolate the effect of adaptation objectives across backbone scale and domain specialization. Our observations point to two transfer regimes. On SST-2, the close distributional proximity to the source domains makes explicit adaptation unnecessary across all backbone scales in our setting. On Financial PhraseBank, distribution-matching objectives recover substantial performance for general-purpose backbones, while supervised contrastive learning is the safest and most effective objective for domainspecialized backbones such as FinBERT in our experiments. Taken together, these case-study observations support a backbone-aware DA selection strategy that matches the adaptation objective to the representation gap rather than applying a fixed recipe. However, confirming this as a general strategy will require validation on larger, multi-seed benchmarks and additional source-target domain pairs. References [1] Dogu Araci. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv preprint arXiv:1908.10063, 2019. [2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692, 2019. [3] Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li. How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence. In Forty-second International Conference on Machine Learning, 2025. [4] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, 2013. [5] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts. Journal of the Association for Information Science and Technology, 65(4):782–796, 2014.
8
[6] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176, 2025. [7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res., 17(1), 2016. [8] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25):723–773, 2012. [9] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661–18673, 2020. [10] Xiang Zhang, Junbo Zhao, and Yann LeCun. Characterlevel Convolutional Networks for Text Classification. In Advances in Neural Information Processing Systems, volume 28, 2015. [11] Julian McAuley and Jure Leskovec. Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, pages 165–172, 2013. [12] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A MultiTask Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, 2018. [13] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, 2019. [14] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Benchmark Probing: Investigating Data Leakage in Large Language Models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2024. [15] Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey, 2024. [16] Yifan Sun, Han Wang, Dongbai Li, Gang Wang, and Huan Zhang. The Emperor’s New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination. In Forty-second International Conference on Machine Learning, 2025. [17] Alan Ramponi and Barbara Plank. Neural Unsupervised Domain Adaptation in NLP—A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838–6855, 2020. [18] Wei Wang, Haojie Li, Zhengming Ding, and Zhihui Wang. Rethink Maximum Mean Discrepancy for Domain Adaptation, 2020. [19] Jiuming Qin, Che Liu, Sibo Cheng, Yike Guo, and Rossella Arcucci. Freeze the Backbones: a Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-Training. In ICASSP 2024 - 2024 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1686–1690, 2024. [20] Walter Simoncini, Andrei Bursuc, Spyros Gidaris, and Yuki M Asano. No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [21] Joris Knoester, Flavius Frasincar, and Maria Mihaela Truşcǎ. Domain Adversarial Training for Aspect-Based Sentiment Analysis. In Web Information Systems Engineering – WISE 2022, pages 21–37, 2022. [22] Tianyu Chen, Shaohan Huang, Furu Wei, and Jianxin Li. Pseudo-Label Guided Unsupervised Domain Adaptation of Contextual Embeddings. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 9–15, 2021. [23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
9