Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
arXiv:2607.18725v1 [cs.CL] 21 Jul 2026
Shaswata Mitra∗ , Subash Neupane† , Trisha Chakraborty‡ , Himanshu Tripathi§ , Sudip Mittal¶ , Aritran Piplai∗∗ , Shahram Rahimi∥ ∗§¶∥ The University of Alabama – {∗ smitra3, § htripathi, ¶ sudip.mittal, ∥ shahram.rahimi}@ua.edu † Meharry Medical College – † [email protected] ‡ Mississippi State University – ‡ [email protected] ∗∗ The University of Texas at El Paso – ∗∗ [email protected] Abstract—Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledgefocused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rankcorrelation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines. Index Terms—Large Language Models, Retrieval-Augmented Generation, Cybersecurity, Model Selection, Fine-tuning, Question-Answering
I. I NTRODUCTION Large Language Models (LLMs) have produced remarkable advances in Natural Language Processing (NLP), demonstrating unprecedented language understanding and generation capabilities. Organizations across many industries now deploy LLMs as domain-specific Question Answering (QA) systems. In cybersecurity, a field tied to critical infrastructure and national security, incorrect model responses can trigger attacks, facilitate breaches, or expose sensitive information, making reliable deployment especially consequential [1]. Effective cybersecurity QA requires a model to possess domain vocabulary, specialized knowledge, and the ability to synthesize heterogeneous retrieved information. Consider the query: “Can the LangChain vulnerability affect my system?” Answering correctly demands both general knowledge of LangChain vulnerability classes and context-specific awareness of the
user’s deployed version. These two information types are qualitatively different, yet both are necessary for a correct, non-leaking response. Because cybersecurity knowledge evolves continuously and labeled data are scarce, practitioners frequently couple a small, deployable LLM with Retrieval-Augmented Generation (RAG) [2] and, where data permit, fine-tune the model to the deployment scope. This raises a practical question that precedes deployment: which small model should be selected, and whether fine-tuning will help at all. Answering empirically by fine-tuning every candidate is expensive in both compute and time. We therefore ask whether a lightweight, pre-adaptation diagnosis can guide this choice. We introduce FiT, a task-oriented diagnostic framework that characterizes a small LLM’s domain understanding and contextualization ability for knowledge-intensive QA. Whereas existing cybersecurity benchmarks such as CYBERBENCH [3] and SecureBERT [4] report static, pre-trained performance, to the best of our knowledge FiT is the first framework to diagnose a small LLM’s cybersecurity QA suitability before adaptation and to characterize how fine-tuning reshapes that suitability. Rather than positing FiT as a general predictive benchmark, we use it as a lens to study how small models behave before and after two distinct fine-tuning regimes, so that organizations can screen candidates and anticipate the direction of post-tuning change. FiT decomposes suitability into three aspects (Vocabulary, Knowledge, Contextualization), and we instantiate the study in cybersecurity1 . Concretely, we investigate three research questions: RQ1. Can a lightweight, pre-adaptation diagnosis characterize a small LLM’s suitability for cybersecurity QA along vocabulary, knowledge, and contextualization, without the cost of fine-tuning each candidate? • RQ2. How do knowledge-focused and instruction-focused fine-tuning regimes alter these capabilities in small LLMs, and do they help uniformly? • RQ3. Do pre-fine-tuning FiT scores anticipate post-finetuning behavior closely enough to guide model selection and avoid unnecessary tuning? •
1 Code and data: github.com/shaswata09/FiT
Generative AI Lifecycle
Define Problem
FiT Applicability
Prompt Engineering Choose Model
Evaluate
Optimize and Deploy
Fine-Tuning
Build LLMpowered Application
Align With Human Feedback
Scope and Selection Prompt (𝓟)
Vocabulary (𝑵𝒊 )
Can langchain vulnerability affect my system?
langchain, vulnerability, affect, System,…,
Adapt and Align Model Domain Knowledge (K d )
Retrieved Domain Information (𝓖𝒅𝒊 )
Retrieved Specific Information (𝑺𝒅𝒊 )
CVE-2023-36189: Description: SQL injection vulnerability in langchain before v0.0.247 allows a remote attacker…
`langchain==0.0.270`: Used for generation of document embeddings for text, enhancing information retrieval capabilities …
What is Langchain? 1. Operating System 2. LLM Framework 3. Machine Learning 4. Blockchain Ans: 2
TASK - 1
Application Integration Completion (𝓒) using Domain and Specific Information (𝓖𝒅𝒊 ∪ 𝑺𝒅𝒊 ) … your system is running langchain version 0.0.270, which is not mentioned...
TASK - 2 TASK - 3
Fig. 1. Implementation scope of FiT in the generative-AI life-cycle (colored area implies applicability) with an example of our three evaluation tasks. By aligning the tasks in a complete process, we visualize the propagation of the domain suitability required to generate Completion (C) for a given Prompt (P).
In addressing the above research questions (RQs), we make the following contributions: • Addressing RQ1, we propose FiT, a diagnostic decomposition of cybersecurity QA suitability into vocabulary, knowledge, and contextualization, with a metric for each. • Addressing RQ2, we present an empirical study of how knowledge-focused and instruction-focused fine-tuning trade off these capabilities in small (7B) LLMs, including an abstention-driven inversion of knowledge rankings that we quantify with rank-correlation analysis. • Addressing RQ3, we show that retrieval-grounded contextualization is robust to fine-tuning, and derive practical guidance for model selection in dynamic, low-data domains such as cybersecurity. The rest of the paper is organized as follows. Section II formulates the problem. Section III reviews background and related work. Section IV details the FiT tasks. Sections V and VI present the experiment, findings, and limitations, followed by concluding remarks. II. P ROBLEM F ORMULATION In this section we define the problem and its foundations. We first describe the implementation scope, then the evaluation tasks; Table I summarizes the notation used throughout. Fig. 1 provides a visual reference for the scope and tasks. In a typical knowledge-intensive, critical-domain QA task using an LLM with RAG, the objective is to generate a relevant completion (C) for a given prompt (P) without disclosing sensitive information. Irrespective of the deployment domain, two types of information are primarily required to generate C. One is domain-specific information (Gid ) relevant to P; the other is contextual or specific information (Sid ) needed to
TABLE I D ESCRIPTION OF N OTATIONS . Notation P {N d ∈ N } {Kd ∈ K} {Gid | Gid ∈ G d } {Sid | Sid ∈ S d } C Nid , Ci L Φ(· | P)
Description User Input Prompt Domain-specific Vocabulary Domain-specific Knowledge Domain Information for P Specific Information for Gid ∪ P Completion for P given (Gid ∪ Sid ) | Kd Ground-truth (expected) outputs LLM under evaluation Task suitability score over P
contextualize Gid for P. The LLM then combines its domain vocabulary (N d ) and knowledge (Kd ) to produce C. To assess an LLM’s contextualization ability within this scope, we adopt a process-oriented decomposition into three tasks, each addressing a distinct capability: 1) Vocabulary. We assess familiarity with domain vocabulary via a keyword-recognition task that instructs the LLM to identify important keywords (Nid ) in P. 2) Knowledge. We assess domain knowledge via a multiple-choice QA task probing the LLM’s domain understanding (Kd ). 3) Contextualization. We assess whether the LLM can comprehend and tailor Gid in light of Sid to generate C for P, without leaking unnecessary information. This decomposition lets us characterize an LLM’s suitability for critical-domain QA from a relevancy and reliability
standpoint. We further analyze how each capability changes after fine-tuning, in order to identify systematic patterns of behavioral change. As we show, these patterns are regimedependent, and understanding them helps practitioners decide whether and how to fine-tune a given model and curate data accordingly. III. P RELIMINARIES The application of pre-trained LLMs in specialized domains has been an active research area [5]. We briefly review the prerequisite background and related developments. A. LLM, RAG, and Fine-tuning LLMs have advanced NLP through transformer architectures [6] that offer remarkable parallelization [7]. Pre-trained on massive Internet text with large parameter counts, these models exhibit strong learning capabilities, yet they can produce plausible-but-inaccurate predictions and struggle on problems requiring specialized domain knowledge. Reported reasons [8] for the failure of general-purpose LLMs in closed domains include a deficit in domain knowledge (lack of exposure to a specialized domain), outdated information (a training cutoff that omits post-training developments), and forgetting (catastrophic forgetting [9] during additional training, where prior knowledge is lost). To mitigate knowledge deficiency for domain-specific tasks, an additional knowledge-ingestion step is required. The two most common approaches are Retrieval-Augmented Generation (RAG) and fine-tuning. RAG, introduced by Lewis et al. [2], leverages an external knowledge base (a document corpus, a structured database, or any source of domain information) to overcome the knowledge limitations of pre-trained LLMs. Given an input query, the RAG architecture retrieves the most relevant passages and integrates them into the input, supplying the LLM with additional context. As models grow in size, updating all parameters becomes computationally demanding and cost-prohibitive, particularly under limited hardware. This motivates parameter-efficient tuning methods that target strong task performance while minimizing the number of tunable parameters. Representative efforts include adapter-based [10], prompt-based [11], LoRA [12], QLoRA [13], and hybrid [14] approaches. B. LLM Benchmarking Numerous frameworks evaluate general and domain-specific language models across NLP tasks. Some, such as GLUEX [15] and PromptBench [16], assess general capabilities including robustness to out-of-distribution and adversarial inputs, while KOLA [17] evaluates knowledge and reasoning. Domain-specific benchmarks also exist: MultiMedQA targets medical QA, and MATH [18] evaluates mathematical reasoning. In cybersecurity, multi-task benchmarks include CyberBench [3] and SecureBERT [4] for sentiment analysis and NER. These efforts largely report static, pre-trained performance. FiT is complementary: rather than ranking models
on a fixed leaderboard, it decomposes suitability into processaligned capabilities and uses them to study how those capabilities shift under fine-tuning, an aspect existing cybersecurity benchmarks do not address. IV. F I T F RAMEWORK We now describe the three FiT tasks used to characterize a small LLM’s suitability for domain-specific QA with RAG, exemplified in Fig. 2. A. Task 1: Vocabulary Assessment Ensuring a model understands domain vocabulary is a prerequisite for domain-specific QA: it lets the model comprehend the input and respond in compatible terms. We assess this via a Keyword Recognition (KR) task, an NLP task that identifies important entities in unstructured text. The LLM is instructed to identify the keywords in the input prompt (P). This probes two aspects: the number of correct keyword identifications (a proxy for domain-oriented linguistic understanding, since vocabulary differs sharply across domains, e.g., medical versus cybersecurity), and recognition of instruction-relevant terms (a proxy for understanding the task itself). Formally, let P = {Pi }ni=1 be the set of prompts, d the domain, and N d the domain vocabulary. For each Pi there exists a gold keyword set Nid . The vocabulary score Φ(N d | P) is Φ(N d | P) =
n 1X ∆ L(Pi ), Nid n i=1
(1)
where ∆(·, ·) is the F1 score between the predicted and gold keyword sets, L is the model, and n > 0. Task-1: Vocabulary Assessment (N d ) Instruction: Print the keywords from the following ... Pi : Can langchain vulnerability affect my system? Nid : langchain, vulnerability, affect, system
B. Task 2: Knowledge Analysis Precisely defining “knowledge” is a philosophical question beyond our scope; we instead quantify domain knowledge through a multiple-choice question-answering (MCQ) task. If a model comprehends the question, it can select the correct option, providing a measurable proxy for parametric knowledge. Formally, let P = {Pi }ni=1 be a set of MCQ problems, each with m options, and let γi be the correct answer for Pi . Writing âi = L(Pi ) for the model’s selected option and ⊮[·] for the indicator function, the knowledge score is the accuracy n
Φ(Kd | P) =
1X ⊮[âi = γi ] , n i=1
(2)
where âi is the model’s selected option, γi the correct option, m the number of choices per item, and n > 0. We note in advance that this score conflates two distinct behaviors when a model declines to answer: a wrong selection
Start Task 1 Task 2 User Prompt
Keyword Request
LLM
Keywords
End
MCQ Answer
Domain Specific MCQ Domain Infor mation Retr iever Specific Infor mation Retr iever
Task 3
Relevant Domain Information
Contextual Completion
LLM Domain + Specific Relevant Information
Specific (Relevant) Information
LLM
Contextulization Query
Fig. 2. Flowchart of the FiT evaluation process. The three tasks align with the deployment scope: Task 1 assesses vocabulary, Task 2 assesses domain knowledge, and Task 3 assesses contextualization for relevant, reliable generation.
and an abstention both reduce accuracy. We exploit this distinction in Section VI. d
Task-2: Knowledge Analysis (K ) Instruction: Answer the correct choice for the question ... Pi : A hash function guarantees the integrity of a message. It guarantees that the message has not been 1: Replaced 2: Overviewed 3: Changed 4: Violated γi : (3) Changed
C. Task 3: Contextualization Analysis Contextualization is the ability to understand and generate text based on the relationships among multi-faceted information. For example, “What potential impact could CVE2023-3894 have on the integrity of our TOML configuration data?” requires jointly reasoning over the CVE and the referenced TOML server. In knowledge-intensive QA, RAG bridges the model’s knowledge deficiency; contextualization lets the model produce relevant, reliable answers from retrieved multifaceted information while following instructions. This ability is critical: in scenarios involving recommendation or privacy, domain information must be tailored to the specifics of the request, and without accurate contextualization a model may provide misinformation or surface inappropriate detail. We therefore adopt contextualized RAG [19] as our final task. Comparing the generated response with a Subject-MatterExpert (SME) reference, we report two RAGAS metrics, answer correctness and semantic similarity, to assess, respectively, how factually and contextually correct the answer is and how closely it tracks the expert reference. Formally, let P = {Pi }ni=1 be domain questions, Gid the domain information and Sid the specific information for Pi , and Ci the expert answer. The contextualization score is
n
Φ(C | P) =
1X Ω L(Pi | Gid ∪ Sid ), Ci n i=1
(3)
where Ω(·, ·) is the RAGAS scoring function (answer correctness or semantic similarity), L(Pi | Gid ∪Sid ) is the model’s completion conditioned on the retrieved context, and n > 0. We are careful not to interpret semantic similarity as a direct measure of information leakage; it captures fidelity to the expert reference, and we treat it as a reliability signal rather than a security guarantee (see Section VI). Task-3: Contextualization (C) Instructions: Given the following retrieved knowledge, answer ... Gid : CVE ID: CVE-2023-36189 Description: SQL injection vulnerability in langchain before v0.0.247 allows a remote attacker to obtain ... CVE ID: CVE-2023-36188 Description: An issue in langchain v.0.0.64 allows a remote attacker to execute arbitrary code via the PALChain ... Sid : ‘langchain==0.0.270‘: Used for generation of document embeddings for text, enhancing information retrieval capabilities ... Pi : Can langchain vulnerability affect my system? Ci : Langchain has multiple reported vulnerabilities ... your system is running langchain version 0.0.270, which is not mentioned to be vulnerable in the provided information...
V. E XPERIMENT & E VALUATION We describe the dataset, infrastructure, and evaluation protocol. Cybersecurity, an information-critical domain, serves as our case study. A. Data Description and Preparation We construct one dataset per task. Task 1 uses 50 cybersecurity questions paired with expert-annotated gold keywords. Task 2 uses the computer-security subset of MMLU [20]
TABLE II F I T RESULTS FOR PRE - TRAINED , KNOWLEDGE - FOCUSED (F INETUNED -1), AND INSTRUCTION - FOCUSED (F INETUNED -2) MODELS . TASK 1: KEYWORD - RECOGNITION F1 ; TASK 2: MCQ ACCURACY; TASK 3: RAGAS SEMANTIC SIMILARITY (S IM ) AND ANSWER CORRECTNESS (C OR ). C ELL SHADING INDICATES CHANGE RELATIVE TO THE PRE - TRAINED VALUE ( RED : DECREASE , GREEN : INCREASE ). VALUES ARE FROM A SINGLE FINE - TUNING RUN PER REGIME AND SHOULD BE READ AS INDICATIVE ( SEE S ECTION VI).
Model gpt-3.5-turbo llama2-7b mistral-7b prometheus-7b westlake-7b westseverus-7b
Task 1 F1 0.85 0.62 0.47 0.76 0.77 0.74
Pretrained Task 2 Task 3 Acc Sim Cor 0.76 0.92 0.77 0.51 0.91 0.78 0.59 0.90 0.72 0.75 0.92 0.73 0.71 0.92 0.74 0.72 0.89 0.69
Task 1 F1 – 0.35 0.27 0.65 0.59 0.69
(100 MCQ items). Task 3 draws on two repositories: a domain information repository (G d ) built from NIST [21], and a QA-specific information repository (S d ) curated from an organization-specific infrastructure wiki; since such infrastructure detail is sensitive, we substitute synthetic data of the same form. We then author 50 questions that require both repositories to answer, each with an SME ground-truth completion. For fine-tuning we prepare two datasets: a knowledge-focused set (Finetuned-1), built by generating QA pairs from the Cisco Talos corpus [22], and an instruction-focused set (Finetuned2), drawn from a training split of the Task-3 evaluation data with explicit instructions to abstain when uncertain. All curated datasets and fine-tuned checkpoints will be released. We report dataset sizes explicitly since, given the modest n, individual numeric differences should be read as indicative rather than significant (Section VI). B. Experiment Infrastructure We evaluate five open-weight 7-billion-parameter, 4-bitquantized QA models: Llama-2-7b2 , Mistral-7b3 , Prometheus7b4 , WestLake-7b5 , and WestSeverus-7b6 . We also include GPT-3.5-Turbo7 as a strong reference point (pre-trained only; we do not fine-tune the API model). These models were chosen deliberately: their pre-training predates much of the evolving threat intelligence (recent CVEs, advisories, infrastructure detail) used here for fine-tuning and evaluation, which reduces data-contamination risk and lets us attribute gains to retrieval and fine-tuning rather than prior exposure. For retrieval we use ChromaDB8 as the vector store. Experiments ran on an Intel i9-12900 with an NVIDIA GeForce RTX™ 3090 Ti and 128 GB RAM. All models were decoded greedily (temperature 0) for deterministic, comparable outputs; the 4bit quantization is held fixed across models so comparisons are like-for-like, though quantization may shift absolute scores. 2 huggingface.co/meta-llama/Llama-2-7b-chat-hf 3 huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 4 huggingface.co/AiMavenAi/AiMaven-Prometheus 5 huggingface.co/senseable/WestLake-7B-v2 6 huggingface.co/FelixChao/WestSeverus-7B-DPO-v2 7 platform.openai.com/docs/models/gpt-3-5-turbo 8 trychroma.com
Finetuned-1 Task 2 Task 3 Acc Sim Cor – – – 0.32 0.87 0.75 0.39 0.86 0.74 0.61 0.85 0.73 0.65 0.85 0.73 0.66 0.89 0.76
Task 1 F1 – 0.48 0.43 0.55 0.73 0.73
Finetuned-2 Task 2 Task 3 Acc Sim Cor – – – 0.31 0.92 0.79 0.18 0.86 0.76 0.16 0.86 0.75 0.08 0.93 0.79 0.12 0.91 0.78
TABLE III I NTER - RATER AGREEMENT (F LEISS K APPA ) BETWEEN THE TWO CYBERSECURITY SME S ACROSS MODELS . Model gpt-3.5-turbo llama2-7b mistral-7b prometheus-7b westlake-7b westseverus-7b
Kappa (K) 0.861 0.845 0.782 0.864 0.944 0.868
Standard Error 0.080 0.084 0.082 0.077 0.081 0.078
C. Evaluation Protocol We combine quantitative and qualitative evaluation. The quantitative evaluation uses the RAGAS [23] framework for Task 3; the qualitative evaluation uses two cybersecurity SMEs to judge generated responses. Results appear in Tables II–III. 1) Quantitative Evaluation: For Task 1 (KR) we report F1 between predicted and gold keywords. Task 2 (MCQ) is scored by accuracy. For Task 3 we report RAGAS answer correctness and semantic similarity. We use RAGAS rather than BLEU [24] or ROUGE [25] because the latter are tailored to machine translation and summarization and correlate poorly with answer correctness in QA. 2) Qualitative Evaluation: Two SMEs assessed FiT’s contextual responses on a 5-point Likert scale [26], from 1 (“factually incorrect and contextually irrelevant”) to 5 (“factually accurate and contextually relevant”). Inter-rater agreement, measured by Fleiss’ κ [27] (Table III), was strong for most models (gpt-3.5-turbo 0.861, llama2-7b 0.845, prometheus-7b 0.864, westlake-7b 0.944, westseverus-7b 0.868) and moderate for mistral-7b (0.782). This agreement establishes the reliability of the SME judgments used to ground the Task-3 references. D. Fine-tuning We fine-tune with QLoRA [13], a PEFT [28] method, under both the knowledge-focused and instruction-focused regimes. Hyper-parameters were held constant across regimes (rank 64, batch size 4, 5 epochs). Each regime was run once per model; we therefore frame the resulting numbers as indicative
and analyze patterns of change rather than individual cell differences. VI. F INDINGS AND L IMITATIONS
recall for caution, a property that is harmful for closedbook MCQ but potentially desirable for retrieval-grounded deployment, where unsupported answers are a liability. This aligns with the abstention behavior described in [29].
A. Fine-tuning does not uniformly help small models Across Table II, both fine-tuning regimes degrade vocabulary (Task 1) and parametric knowledge (Task 2) for every model relative to its pre-trained baseline. In a domain like cybersecurity, where prompts routinely contain newly disclosed terms and CVEs, knowledge-focused tuning (Finetuned-1) cannot keep pace: it lowers keyword F1 for all models (e.g., Llama-2 0.62 → 0.35) and reduces MCQ accuracy in parallel. The practical implication is that, absent abundant and current labeled data, a well-chosen pre-trained model paired with RAG is often the safer choice. B. The two regimes trade off differently but predictably in direction
D. Retrieval-grounded contextualization is robust Task-3 correctness is remarkably stable across all models and conditions (range 0.69–0.79), and semantic similarity is uniformly high (∼ 0.85–0.93). When relevant context is retrieved and supplied, model choice and fine-tuning have little effect on contextual correctness. We therefore caution against interpreting similarity as an information-leakage metric: it is saturated and non-discriminative here, and reflects fidelity to the expert reference rather than the presence or absence of sensitive disclosure. The practical takeaway is that, for contextual correctness in this setting, investment in retrieval quality yields more than investment in fine-tuning. E. Selection guidance
TABLE IV S PEARMAN RANK CORRELATION (ρ) BETWEEN PRE - TRAINED AND POST- FINE - TUNING SCORES ACROSS THE FIVE OPEN - WEIGHT MODELS . H IGH POSITIVE ρ INDICATES PRESERVED RANKINGS ; NEGATIVE ρ INDICATES INVERSION . Task Task 1 (vocabulary) Task 2 (knowledge) Task 3 (correctness)
Pre → Finetuned-1
Pre → Finetuned-2
0.60 0.82 0.70 −0.60 not interpretable (range 0.69–0.79)
To quantify how rankings move, we compute Spearman rank correlations between pre-trained and post-tuned scores (Table IV). Vocabulary rankings are well preserved under both regimes (ρ = 0.60 and 0.82), and knowledge rankings are preserved under knowledge-focused tuning (ρ = 0.70). Strikingly, knowledge rankings invert under instruction-focused tuning (ρ = −0.60): the strongest pre-trained knowledge models (WestLake, WestSeverus) become the weakest on Task 2 (accuracy 0.08 and 0.12). This inversion is the key reason a naive “best-before-equals-best-after” heuristic fails, and why FiT must be read as a regime-aware diagnostic rather than a monotonic predictor. C. The instruction-tuning collapse is abstention, not knowledge loss The Task-2 collapse under Finetuned-2 is an artifact of the abstention instruction, not erasure of knowledge. Two pieces of evidence support this. First, the instruction-focused data explicitly directed the model to withhold an answer when uncertain, making conservatism the trained behavior. Second, and more tellingly, the same instruction-tuned models retain or slightly improve contextual answer correctness (Task 3, e.g., WestLake 0.74 → 0.79) even as their standalone MCQ accuracy falls to near zero. A model that has truly lost knowledge could not answer correctly when that knowledge is retrieved; these models can. Instruction tuning thus trades parametric
Taken together, these patterns make FiT useful as a preadaptation screen: weak pre-trained models (e.g., Mistral on vocabulary) remain weak after tuning, and the strongest knowledge models survive knowledge-focused tuning. Where instruction-focused tuning is planned, practitioners should expect, and can pre-empt, an abstention-driven collapse in closed-book knowledge, and should evaluate such models in their retrieval-grounded configuration rather than on standalone knowledge probes. F. Threats to Validity Construct. Our metrics are proxies. Task 2 accuracy conflates a wrong answer with a deliberate abstention; we crosscheck against retrieval-grounded correctness (Task 3), but the proxy is imperfect. RAGAS similarity captures fidelity to the expert reference, not information leakage, so we read it only as a reliability signal. Internal. Each regime was run once with fixed hyper-parameters and greedy decoding, so regime effects are not separated from run-to-run variance; Finetuned1 uses Cisco Talos while Task 2 uses MMLU, so part of the Task 2 drop may be distribution shift rather than a general fine-tuning effect; fixed 4-bit quantization keeps comparisons like-for-like but may shift absolute scores; and using models whose pre-training predates the evaluation content reduces, but does not eliminate, contamination risk. External. We cover five open-weight models at one 7B scale plus an API reference in a single domain, so patterns may not transfer across scales, architectures, or domains [3]; Task-3 specific data are synthetic for confidentiality and the threat-intelligence snapshot is fixed in time. Conclusion. Datasets are modest (n = 50/100/50) and we report no significance tests, so numeric differences are indicative; in particular, the Spearman values in Table IV are computed over five models and are descriptive of direction rather than statistically significant. A larger model pool and repeated runs would be needed to test these trends formally.
VII. C ONCLUSION Cybersecurity is tied to critical infrastructure, making the reliable deployment of LLMs in this domain consequential. We presented FiT, a task-oriented diagnostic that decomposes cybersecurity QA suitability into vocabulary, knowledge, and contextualization, and used it to study how five small LLMs behave under knowledge-focused and instruction-focused finetuning. Fine-tuning did not uniformly help: it degraded parametric capabilities in every model, knowledge-focused tuning preserved relative rankings, and instruction-focused tuning collapsed measured knowledge through abstention while leaving retrieval-grounded contextualization intact, a regimedependent trade-off we quantified with rank-correlation analysis. These results indicate that, in dynamic, low-data domains, a well-chosen pre-trained model paired with strong retrieval often strikes a better balance than fine-tuning, and that preadaptation diagnosis can guide model selection and reduce unnecessary tuning cost. E THICS S TATEMENT Our study uses datasets that contain no sensitive information. To obtain cyber-threat intelligence, specifically Common Vulnerabilities and Exposures (CVEs), we used web crawlers issuing API calls within the limits set by authorized sources. We anonymized our human evaluators and ensured no personally identifiable information was disclosed. We confirm that our research aligns, to the best of our knowledge, with the IEEE Code of Ethics. ACKNOWLEDGMENT This work was supported by the National Science Foundation under Grant No. 2611682. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of their institution or the National Science Foundation. R EFERENCES [1] M. Dahl, V. Magesh, M. Suzgun, and D. E. Ho, “Hallucinating law: Legal mistakes with large language models are pervasive.” https://hai.stanford.edu/news/hallucinating-law-legal-mistakeslarge-language-models-are-pervasive, 2024. [2] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., “Retrievalaugmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020. [3] Z. Liu, J. Shi, and J. F. Buford, “Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity,” 2024. [4] E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domainspecific language model for cybersecurity,” in International Conference on Security and Privacy in Communication Systems, pp. 39–56, Springer, 2022. [5] P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin, “Generating fake cyber threat intelligence using transformer-based models,” in 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–9, IEEE, 2021. [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [7] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Computing Surveys, vol. 56, no. 2, pp. 1–40, 2023.
[8] C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, et al., “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,” arXiv preprint arXiv:2310.07521, 2023. [9] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521– 3526, 2017. [10] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International conference on machine learning, pp. 2790–2799, PMLR, 2019. [11] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691, 2021. [12] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter efficient tuning of pre-trained models using dynamic searchfree low-rank adaptation,” arXiv preprint arXiv:2210.07558, 2022. [13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024. [14] Y. Mao, L. Mathias, R. Hou, A. Almahairi, H. Ma, J. Han, W.-t. Yih, and M. Khabsa, “Unipelt: A unified framework for parameter-efficient language model tuning,” arXiv preprint arXiv:2110.07577, 2021. [15] L. Yang, S. Zhang, L. Qin, Y. Li, Y. Wang, H. Liu, J. Wang, X. Xie, and Y. Zhang, “Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,” arXiv preprint arXiv:2211.08073, 2022. [16] K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, N. Z. Gong, Y. Zhang, et al., “Promptbench: Towards evaluating the robustness of large language models on adversarial prompts,” arXiv preprint arXiv:2306.04528, 2023. [17] J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li, et al., “Kola: Carefully benchmarking world knowledge of large language models,” arXiv preprint arXiv:2306.09296, 2023. [18] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al., “Large language models encode clinical knowledge,” Nature, vol. 620, no. 7972, pp. 172– 180, 2023. [19] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llmintegrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp. 79– 90, 2023. [20] CAIS, “Measuring massive multitask language understanding.” huggingface.co/datasets/cais/mmlu, 2024. [21] NIST, “National vulnerability database.” nist.gov, 2024. [22] Cisco, “National vulnerability database.” talosintelligence.com, 2024. [23] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “Ragas: Automated evaluation of retrieval augmented generation,” arXiv preprint arXiv:2309.15217, 2023. [24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002. [25] L. C. Rouge, “A package for automatic evaluation of summaries,” in Proceedings of Workshop on Text Summarization of ACL, Spain, vol. 5, 2004. [26] I. E. Allen and C. A. Seaman, “Likert scales and data analyses,” Quality progress, vol. 40, no. 7, pp. 64–65, 2007. [27] M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012. [28] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al., “Parameter-efficient fine-tuning of largescale pre-trained language models,” Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023. [29] J. Xin, R. Tang, Y. Yu, and J. Lin, “The art of abstention: Selective prediction and error regularization for natural language processing,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1040– 1051, 2021.
Task 1 Pretrained
Task 3 Pretrained (Correctness)
(b)
(a)
(b)
Task 3 Pretrained (Similarity)
(c)
(c)
Fig. 3. Performance of pre-trained models on Tasks 1 and 3. (a) F1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions. Task 1 Finetuned-1
Task 3 Finetuned-1 (Correctness)
(d)
(b) (e)
(d)
(e)
Task 3 Finetuned-1 (Similarity)
(c)
(f)
(f)
Fig. 4. Performance of Finetuned-1 (knowledge-focused) models on Tasks 1 and 3. (a) F1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions.
Task 1 Finetuned-2
(g)
Task 3 Finetuned-2 (Correctness)
Task 3 Finetuned-2 (Similarity)
(b) (e)
(f)
(h)
(i)
Fig. 5. Performance of Finetuned-2 (instruction-focused) models on Tasks 1 and 3. (a) F1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions.