Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports Ahmed Ryan∗ , Saad Sakib Noor† , Md Erfan∗ , Shaswata Mitra∗ , Sudip Mittal∗ , and Md Rayhanur Rahman∗ ∗ The University of Alabama, {aryan9, merfan}@crimson.ua.edu, {smitra3, sudip.mittal, mrahman87}@ua.edu
arXiv:2606.18166v1 [cs.CR] 16 Jun 2026
† The University of Dhaka, [email protected]
Abstract—Classifying Cyber Threat Intelligence (CTI) using MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is essential for proactive defense, but historically required extensive human effort. Pre-Large Language Model (LLM) automation sped up this process, but could not resolve the complex language and multi-step attack patterns found in unstructured CTI reports. LLMs addressed previous limitations by using contextual reasoning to understand unstructured text. However, current evaluations rely on simplified, single-technique sentences that ignore the complexity of real-world CTI reports, which often leads to inflated performance results. Consequently, the baseline performance of open-source LLMs on complex unstructured CTI reports remains unevaluated. To address this gap, we constructed a ground-truth dataset of 2,076 humanannotated sentences (1,281 technique-positive, 795 negative) from 83 complex unstructured CTI reports. These sentences were mapped to 114 unique ATT&CK techniques using a six-phase annotation process, achieving κ = 0.68 inter-annotator agreement. Using this dataset, we evaluated seven open-source LLMs ranging from 8B to 236B parameters across prompt strategy and temperature configurations. The highest-performing LLM achieved a micro-averaged F1 score of 0.22, establishing the empirical baseline for multi-label ATT&CK classification on complex unstructured CTI. Parameter size showed a statistically significant positive correlation with F1 score. Prompt strategy and temperature produced no statistically significant gains across model configurations. These results indicate that current opensource LLMs are insufficient for production-grade ATT&CK classification. The dataset, benchmark, and findings provide a reproducible foundation for future CTI research. Index Terms—Cyber Threat Intelligence, MITRE ATT&CK, Large Language Models, Multi-Label Classification, Natural Language Processing, Cybersecurity Automation.
I. I NTRODUCTION Cyber threats impose financial and operational damage on organizations worldwide, with annual costs projected to reach $15.63 trillion by 2029 [1]. To contain this threat, organizations collect and classify Cyber Threat Intelligence (CTI): structured, actionable knowledge about adversary behavior derived from raw threat data. Classifying CTI enables Security Operations Centers (SOCs) to shift from reactive threat blocking to proactive threat anticipation. The industry structures this classification process around established frameworks, most prominently MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK), alongside Cyber Kill Chain, Diamond Model, and VERIS [2]–[5]. The classification of CTI at the ATT&CK (see Section II) technique level remains difficult and time-consuming to per-
form effectively at scale. Analysts who manually map CTI reports to ATT&CK techniques face a complex 211-class taxonomy, sentences that contain multiple overlapping behaviors, and a volume of data that is impossible to process by hand [6], [7]. For example, consider a typical Digital Forensics and Incident Response (DFIR) sentence: “Upon execution, the macro dropped a VBScript payload which initiated a PowerShell instance to download a secondary module.” Labeling this sentence requires identifying both the Command and Scripting Interpreter (T1059) and Ingress Tool Transfer (T1105) techniques from a list of 211 options. Earlier automation using pre-Large Language Model (LLM) machine learning models processed data faster, but struggled to understand complex language or multi-step attack patterns, resulting in lower accuracy [7]–[9]. Therefore, pre-LLM tools could not accurately and efficiently process the high volume of CTI reports. LLMs address one of the core limitation of prior automation: the capability to understand complex meaning and context within unstructured text [10]. Because of this reasoning capability, LLMs are good at classifying ATT&CK techniques, and initial evaluations show they work well for simple, single-technique examples [11]. However, existing evaluations rely on simple, single-action sentences (e.g., “The adversary used cmd.exe to execute a batch script”) that map to only one technique. These sentences fail to reflect the complex, multi-label nature of real-world CTI reports [12], [13]. Constructing a multi-label ground-truth dataset at the technique level requires adversarial domain expertise, involves solving classification across 211 different techniques, and is extremely time-consuming. This difficulty explains why very few such datasets exist in academic research [11], [14]. As a result, the performance of open-source LLMs on multi-label ATT&CK classification over unstructured CTI reports remains unevaluated. Establishing this performance baseline would provide security practitioners empirical evidence to evaluate LLM integration within threat ingestion pipelines, provide organizations data to guide resource allocation, and provide researchers a reference point for future CTI automation work. The goal of this paper is to establish a baseline performance for open-source LLMs on multi-label ATT&CK classification through a manually constructed ground-truth dataset and a systematic evaluation of seven open-source models. We
investigate the following research questions: RQ1 : How can unstructured CTI reports be annotated to construct a multi-label ATT&CK ground-truth dataset? RQ2 : What is the baseline multi-label classification performance of open-source LLMs on unstructured CTI reports across different inference configurations? RQ3 : How do inference settings (parameter size, prompt strategy, and temperature) correlate with multi-label classification performance of LLMs? To address these questions, we implemented an experimental design spanning dataset construction, model evaluation, and statistical analysis. For RQ1 , we applied a six-phase annotation strategy to extract and label sentences from unstructured DFIR reports. For RQ2 , we evaluated seven open-source LLMs across N -shot, Chain-of-Thought (CoT), and temperature variations, and compared the highest-performing configuration against a Retrieval-Augmented Generation (RAG)enabled pipeline. For RQ3 , we applied Spearman’s rank correlation to F1 scores to identify statistically significant relationships between inference settings and classification performance. This study produces three primary contributions: (a) Dataset: A ground-truth dataset of 2,076 sentences from 83 unstructured CTI reports, annotated across 114 ATT&CK techniques. (b) Performance Benchmark: A systematic evaluation of seven open-source LLMs across 53 inference configurations, establishing a baseline for multi-label ATT&CK classification. (c) Statistical Analysis: A correlation analysis measuring the impact of parameter size, prompt strategies, and temperature on multi-label classification performance of LLMs. The remainder of this paper is organized as follows. Section II defines key concepts. Section III reviews related work. Section IV details the methodology. Section V presents the findings. Section VI discusses the findings. Section VII addresses threats to validity. Section VIII concludes the paper. II. K EY C ONCEPTS We describe several key concepts in this section. ATT&CK Tactics and Techniques: ATT&CK is a globallyaccessible knowledge base of adversary tactics and techniques based on real-world observations. Tactics represent the adversary’s high-level tactical goals, such as achieving persistence or exfiltration, while techniques describe the specific methods used to attain those objectives [2]. Inference Settings and Configuration: Inference settings control how an LLM processes input and produces output. The settings are: (a) reasoning strategy (Chain-of-Thought: Yes/No) [15], (b) prompt strategy (Zero-Shot/Few-Shot) [16], and (c) decoding parameter (Temperature=0/0.5). Together they make an inference configuration such as, No CoT + Shot=3 + Temperature=0.5. Full Factorial Design: A systematic experimental methodology that tests every possible combination of independent variables to measure both individual main effects and complex interaction effects [17].
Retrieval-Augmented Generation (RAG): An architectural framework that enhances LLM responses by dynamically retrieving relevant information from an external vector database at runtime [18]. III. R ELATED W ORK We describe several related works in this section. Dataset Construction and Annotation: Existing public CTI datasets lack the complexity of real-world, messy data [14]. MITRE’s Threat Report ATT&CK Mapper (TRAM) restricts annotations to the 50 most prevalent techniques and trains on simplified procedure examples rather than unsanitized incident reports [19]. While Legoy et al. [20] annotated 1,490 CTI reports, their analysis focused on 12 broad categories (tactics) rather than granular sentence-to-technique mappings. CTIHAL [21] provides sentence-level labels for 81 reports, but only includes 116 annotated sentences because it ignores sentences without techniques. In contrast, our work uses 2,076 annotated sentences from 83 complex incident reports mapped to 114 ATT&CK techniques. Our dataset includes both technique-positive (1,281) and negative (795) sentences, requiring an LLM to distinguish between relevant and irrelevant information. Our dataset is ≈ 18 times larger than CTIHAL by annotation count and the largest of its kind to date. Pre-LLM Tactic, Technique, and Procedure (TTP) Extraction: TTPDrill [22] relies on 392 hand-crafted rules which face limitations when parsing the varied and unpredictable sentences in security reports. AttacKG [23] achieves strong technique-graph recall on 16 labelled reports but requires template-aligned input. SMET [24] and SecureBERT [25] are trained on rigid, pre-categorized data. Consequently, these encoder-only models cannot extract the full vocabulary of attack techniques from natural, open-ended text. Rahman and Williams [8] showed that prior extraction models lack a standardized benchmark, leaving the performance of modern LLMs on raw incident reports unevaluated. LLM-Based Classification and Benchmarks: CTIBench [11] is an existing benchmark, evaluating multiple LLMs across five CTI tasks. However, it treats technique extraction as a sub-task and does not evaluate how different prompt structures affect model performance. RAM [7] applies LLMs to structured SIEM rules rather than unstructured CTI. FALCON [26] similarly applies LLMs to autonomous CTI mining but targets Intrusion Detection System (IDS) rule generation rather than multi-label technique classification. Mezzi et al [27] showed that LLMs performance are inconsistent when processing fulllength reports. Prior research has not thoroughly evaluated whether increasing an LLM’s parameter size improves its extraction performance. Our paper fills this gap by evaluating seven open-source models and the impact of parameter size on classification performance. IV. M ETHODOLOGY We detail the experimental design and evaluation procedures for RQ1 through RQ3 in the following sections.
A. RQ1 : Dataset Construction We detail the dataset construction methodology in the following sections. 1) Data Collection: We sourced 83 reports (April 2020–February 2025) from The DFIR Report [28], which publishes detailed analysis of cyber incidents. We scraped the text using Scrapy and BeautifulSoup and parsed the content with Newspaper3k [29]–[31]. The full CTI reports contain raw log dumps, hex code, firewall configurations, and exhaustive lists of Indicator of Compromises (IoCs). The “Case Summary” section in the reports presents the attack lifecycle in natural language. Hence, we scraped only the “Case Summary” section of the reports to minimize noise (e.g. log dumps, hex code etc.). We parsed the complete text of the report only when the “Case Summary” section was unavailable. After parsing, we extracted 2,076 sentences. 2) Data Annotation: The first and second authors independently annotated 41 and 42 reports, respectively (comprising 898 and 1,067 sentences), under the supervision of the last author. To define a clear boundary for the classification task, we labeled each sentence against the 211 techniques from MITRE ATT&CK Enterprise v17.1 (August 2025), omitting the more granular sub-techniques. We carried out the annotation in six phases: (i) Independent Labeling, where the annotators labeled the allocated reports independently; (ii) Initial Cross-Labeling, where we computed Cohen’s Kappa (κ) [32] per technique to account for chance agreement on a random 10% sample (195 sentences), which was collected from both annotators; (iii) Relabeling, where the annotators re-evaluated 981 low-agreement sentences (κ < 0.7), split 460/521 between first and second annotators; (iv) Secondary Cross-Labeling, where we checked 10% of the data relabeled in step-iii (98 sentences) to verify κ improvement; (v) Dispute Resolution, where we resolved conflicts via reconciliation meeting; and (vi) Dataset Aggregation, where we merged data of from all phases into the final ground-truth dataset. Following the six-phase process, we reached a mean interannotator agreement of κ = 0.68 across 114 techniques (Landis and Koch: substantial agreement [33]), with 65 techniques (57.0%) exceeding the κ ≥ 0.7 quality threshold. The lower agreement on the remaining 49 techniques (43.0%) highlights the inherent difficulty of this multi-label classification task. B. RQ2 : Performance of LLMs 1) Experimental Design: We implemented a full factorial design (see Section II) across three inference settings. (i) NShot: Zero-Shot (N = 0) establishes pre-trained capability baselines; Three-Shot (N = 3) assesses in-context learning while preventing context saturation and diminishing returns. (ii) Reasoning: CoT prompting forces step-by-step deduction to improve multi-stage reasoning. (iii) Temperature: T = 0.0 versus T = 0.5 evaluates the trade-off between deterministic and creative responses [15], [16], [34].
2) Prompt Strategy: In each prompt, we defined the LLM as a “Cybersecurity Threat Analysis Expert” and included: (a) the list of 211 ATT&CK techniques to prevent hallucinated techniques; (b) instructions to ignore sub-techniques; and (c) for CoT runs, a reasoning example to guide the LLM. 3) Inference: We selected open-source LLMs for two reasons: data privacy and operational cost. Our strategy avoids the expense of proprietary LLMs and resolves the organizational reluctance to share sensitive incident reports with external LLM providers. We selected the LLMs mentioned in Table I to encompass a range from 8B to 236B parameters. This selection allowed us to evaluate how parameter scale influenced performance [35]. All the LLMs support ≥128k-token context windows to process CTI sentences alongside the 211-class ATT&CK taxonomy without truncation [36]. The Llama-3.1, Gemma, and DeepSeek (Mixture of Experts (MoE)) families represent distinct architectural paradigms [36]–[38]. We ran all the LLMs at 4-bit quantization (Q4_K_M) to control quantization variance while balancing efficiency and reasoning capability. TABLE I LLM S AND I NFERENCE S PECIFICATIONS
ID
LLM Name
Params
Context
DS236 GP120 LLM70 GMM27 GPT20 GMM12 LLM08
DeepSeek-V2.5 [39] GPT-OSS [40] Llama 3.1 Instruct [41] Gemma 3 [42] GPT-OSS [43] Gemma 3 [44] Llama 3.1 Instruct [45]
236B 120B 70B 27B 20B 12B 8B
160k 128k 128k 128k 128k 128k 128k
4) RAG Comparison: Given the dynamic nature of CTI and the static knowledge cutoffs inherent to LLMs, we introduce a RAG pipeline to evaluate the efficacy of external knowledge grounding against standard inference. The pipeline operates in three phases: (i) ATT&CK documentation is parsed into chunks and embedded in a Facebook AI Similarity Search (FAISS) [46] vector database; (ii) each DFIR sentence is embedded at runtime to retrieve the top-5 relevant technique definitions; (iii) retrieved context is appended to the prompt, grounding the LLM in current documentation instead of parametric memory. We measure the precision, recall, and F1 ∆ between standard and RAG-enabled inference. C. RQ3 : Impact of Inference Settings 1) Inference Settings: We executed a full factorial design over four inference settings: (a) LLM Parameter Size (mentioned in Table I); (b) N-Shot (N=0 vs. N=3); (c) Reasoning (No CoT vs. CoT); (d) Temperature (T=0.0 vs. T=0.5). 2) Performance Metrics: The technique label distribution in our annotated dataset is heavily skewed: a small number of techniques account for the majority of labeled sentences (see Section V-A4). Macro average indicates how well a classifier performs on rare techniques, whereas micro average indicates how well a classifier performs on all the technique instances. As micro average reflects operational coverage, we report micro-averaged precision, recall, and F1 score as the
evaluation metrics for the ATT&CK technique classification task. 3) Correlation Analysis: We applied Spearman’s rank correlation (ρ) [47] to assess monotonic relationships (i.e. two variables change together but not necessarily at a constant rate) between each inference setting and F1 score without assuming normality, with significance evaluated at α = 0.05 (null hypotheses rejected only if p ≤ 0.05). V. F INDINGS The following sub-sections present our empirical results. We first characterize the annotated dataset and the distribution of ATT&CK techniques in CTI reports (RQ1 ). We then evaluate the LLMs of all inference configurations, compare their performance against a RAG guided inference, and analyze qualitative failures (RQ2 ). Finally, we measure the statistical impact of parameter size, prompt strategy, reasoning, and temperature on classification performance (RQ3 ). A. Findings of RQ1 : Dataset Creation The final dataset comprises 2,076 sentences mapped to 114 unique ATT&CK techniques, totaling 2,028 technique occurrences across 211 potential candidates. Each entry facilitates sentence-level multi-label classification, enabling the extraction of ATT&CK techniques from unstructured CTI reports. The reliability of these labels is validated by a mean inter-annotator agreement of κ = 0.68 (substantial agreement), which confirms the ground-truth quality despite the inherent complexity of multi-label classification. Consequently, this dataset provides a foundation for evaluating LLMs in multilabel CTI classification domain. TABLE II DATASET S TATISTICS S UMMARY Statistic
Value
Total Sentences Unique sentences Vocabulary (Unique Words) Mean Words per Sentence Labeled sentences (>= 1 technique) Label space size |L| Total Label Instances Mean Labels per Labeled Sentence Multi-Label Sentences Singleton Labels Imbalance Ratio Normalized Shannon Entropy Gini Coefficient
2,076 2,021 4,001 17.8 ± 13.2 1,281 (61.7%) 114 2,028 1.58 515 (40.2% of labeled) 27 229.0 0.802 0.693
360 words, typically corresponding to command listings or indicator descriptions. The dataset consists of 38,401 number of words in total and the number of unique words is 4,001. The lexical diversity, also referred to as Type-Token Ratio (TTR), is 0.104. The value indicates that approximately 10.4% of the text consists of unique words, while remaining 89.6% are repetitions. The low TTR reflects the repetitive nature of incident reports. 3) Technique Label Space: The annotators applied 114 unique ATT&CK technique labels and assigned the labels to the sentences a total of 2,028 times. The sentences are dominated by post-compromise behavior (e.g. execution, lateral movement, credential access, command and control, and impact). The ten most frequent labels account for approximately 50% of all annotations: Command and Scripting Interpreter (11.3%), Remote Services (9.2%), Ingress Tool Transfer (6.3%), Data Encrypted for Impact (4.5%), OS Credential Dumping (4.1%), Application Layer Protocol (3.1%), Obfuscated Files or Information (3.0%), Impair Defenses (2.9%), Process Injection (2.8%), and Phishing (2.7%). 4) Class Imbalance and Long-Tail Distribution: The most frequent technique appears 229 times more than the rarest, with 56 (49.1% of 114 techniques) containing five or fewer instances, including 27 singletons (23.7% of 114 techniques), which appeared exactly once. The disparity is reflected by a mean imbalance ratio (Mean Imbalance Ratio (MIR)) of 82.64 (far exceeding the 1.0 of a balanced dataset) and a Gini coefficient of 0.693, confirming that a few techniques dominate the total occurrences. Conversely, a normalized Shannon entropy of 0.802 indicates that the labels are spread across a wide variety of unique categories. Overall, the dataset exhibits a severe long-tail distribution, indicating high class imbalance. 5) Multi-Label Cardinality: In the dataset, a single sentence may describe more than one ATT&CK technique simultaneously (e.g. a sentence describing a malicious LNK execution chain may receive both User Execution, and Command and Scripting Interpreter). Multi-label cardinality measures the average number of labels assigned to a single sentence. Of the 1,281 labeled sentences, 515 (40.2%) carry two or more techniques. The mean cardinality of the label is 1.58, with a maximum of 9 labels in a sentence (see Table III). TABLE III D ISTRIBUTION OF L ABELS PER L ABELED S ENTENCE Labels per sentence
1) Volume and Label Coverage: The dataset contains 2,076 sentences, of which 1,281 (61.7%) carry at least one technique label. The remaining 795 (38.3%) form a negative class that do not map to any technique. Including the negative class is essential to evaluate LLMs tendency to generate false positives during the classification process. 2) Sentence-Level Properties: The sentences are technical and dense with named entities (i.e. binaries, protocols, paths, command-line fragments). The sentences have a mean length of 17.8 words (standard deviation: 13.2), and 95% are 31 words or shorter. A small number of outliers reach up to
1 2 3 4 5–9
Sentences
Share of labeled (%)
766 354 19 40 12
59.8 27.6 8.5 3.1 0.9
B. Findings of RQ2 : Performance of LLMs 1) Inference: Table IV reports micro and macro precision, recall, and F1 score for the 7 open-source LLMs across 53 inference configurations. We use micro F1 score as the primary metric for comparison as mentioned in Section IV-C2.
The overall performance ceiling is low. Across all configurations, the maximum micro F1 score is 0.22, produced by DS236 at (T=0.0, Shot=3, CoT=Yes). The models separate into three clusters. DS236, LLM70, and GP120 form the upper cluster within a F1 score of 0.20 to 0.22, with all five topranked configurations combining CoT prompting with either DS236 or LLM70. GMM27 and GMM12 occupy the middle cluster at F1 score = 0.17 and 0.13 respectively. GMM12 is notable because micro F1 score varies by ≤ 0.01 across all eight configurations, indicating invariance to prompt changes. LLM08 and GPT20 form the lower cluster, with GPT20 reaching micro F1 score ≤ 0.03 in five of eight configurations: the LLM classifies almost no sentences correctly under most settings, with all five bottom-ranked rows in Table IV belonging to GPT20. Two LLMs fail in opposite ways, showing two different types of errors. GMM12 produces micro recall in the range [0.21, 0.22] against micro precision in the range [0.09, 0.10] (a recall-to-precision ratio of approximately 2.2) indicating the LLM over-predicts, generating false positives. GPT20 exhibits the inverse pattern: micro precision of [0.12, 0.27] against micro recall of [0.00, 0.04], indicating the LLM underpredicts, the prediction is correct between 12% and 27% of the time but detects at most 4% of the techniques. The three configuration dimensions affect performance differently. Temperature variations (0.0 to 0.5) shift micro F1 score by ≤ 0.01 across all seven models, confirming that temperature has negligible effect on technique classification performance at this range. Few-shot prompting (N=3) raises micro F1 score by 0.02−0.05 for DS236, GP120, and LLM08, but produces changes of ≤ 0.01 for GMM12, GMM27, and LLM70, and unstable changes of −0.01 to +0.04 for GPT20. The findings demonstrate that in-context examples benefit some models but not all. CoT prompting raises micro F1 score by 0.02 − 0.05 for DS236 and LLM70, although the comparison for LLM70 is partial due to omitted baseline configurations. It produces negligible or negative changes for the remaining five models, indicating that step-by-step reasoning provides a measurable advantage only at the largest parameter scales in this study. Under macro F1 score, the ranking shifts: LLM70 leads at 0.15, followed by DS236 (0.14), GP120 (0.13), GMM27 (0.12), GMM12 (0.11), LLM08 (0.10), and GPT20 (0.03). The reordering of DS236 and LLM70 indicates that DS236 correctly identifies common techniques better than rare techniques, while LLM70 maintains consistent accuracy across both common and rare techniques. 2) RAG Comparison: The RAG evaluation grounds LLM70 (T=0.0, Shot=3, CoT=No) in ATT&CK documentation retrieved at inference time (see Table V). Under this configuration, LLM70 reaches micro F1 score of 0.32, which exceeds the maximum non-RAG micro F1 score of 0.22 in Table IV (DS236 at T=0.0, Shot=3, CoT=Yes) by 0.10 points. The improvement is primarily caused by micro recall, which moves from 0.23 to 0.41 (∆ = +0.18), a 1.78x factor. While we cannot isolate RAG’s contribution mathematically
TABLE IV P ERFORMANCE E VALUATION OF O PEN -S OURCE LLM S ACROSS C ONFIGURATIONS (T OP 5 IN G REEN , W ORST 5 IN R ED ) LLM
T
N-Shot
CoT
Precision
Recall
F1 Score
Micro
Macro
Micro
Macro
Micro
Macro
DS236
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.25 0.22 0.22 0.21 0.22 0.20 0.22 0.21
0.08 0.12 0.11 0.14 0.08 0.12 0.11 0.14
0.13 0.21 0.18 0.23 0.12 0.20 0.17 0.22
0.08 0.14 0.12 0.15 0.08 0.13 0.12 0.15
0.17 0.21 0.20 0.22 0.16 0.20 0.19 0.21
0.08 0.12 0.11 0.14 0.08 0.12 0.11 0.14
GMM12
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.10 0.10 0.09 0.10 0.09 0.10 0.09 0.10
0.08 0.09 0.08 0.09 0.08 0.09 0.08 0.09
0.22 0.22 0.21 0.22 0.21 0.22 0.21 0.22
0.15 0.15 0.14 0.15 0.14 0.15 0.14 0.15
0.13 0.13 0.13 0.13 0.13 0.13 0.13 0.13
0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.11
GMM27
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.12 0.13 0.14 0.13 0.12 0.13 0.13 0.13
0.10 0.11 0.11 0.11 0.10 0.11 0.11 0.11
0.22 0.23 0.23 0.23 0.22 0.23 0.23 0.23
0.15 0.16 0.15 0.16 0.15 0.15 0.15 0.16
0.16 0.17 0.17 0.17 0.16 0.16 0.17 0.17
0.11 0.12 0.12 0.12 0.12 0.12 0.12 0.12
LLM08
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.07 0.06 0.11 0.08 0.07 0.06 0.08
0.06 0.08 0.06 0.09 0.05 0.07 0.07
0.13 0.16 0.10 0.16 0.11 0.14 0.14
0.09 0.11 0.07 0.12 0.08 0.10 0.10
0.09 0.08 0.11 0.11 0.08 0.09 0.10
0.07 0.09 0.06 0.10 0.06 0.08 0.08
LLM70
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.21 0.19 0.17 0.20 0.18 0.17
0.08 0.14 0.14 0.08 0.13 0.14
0.13 0.24 0.26 0.13 0.23 0.25
0.09 0.17 0.18 0.09 0.16 0.17
0.16 0.21 0.21 0.16 0.20 0.20
0.08 0.14 0.15 0.09 0.14 0.15
GP120
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.21 0.26 0.20 0.21 0.21 0.23 0.20 0.20
0.10 0.07 0.13 0.13 0.10 0.07 0.13 0.13
0.15 0.11 0.19 0.20 0.14 0.11 0.19 0.20
0.10 0.07 0.12 0.14 0.10 0.07 0.13 0.14
0.17 0.15 0.19 0.20 0.17 0.15 0.19 0.20
0.09 0.07 0.12 0.13 0.09 0.07 0.12 0.13
GPT20
0.0 0.0 0.0 0.0 0.5 0.5 0.5 0.5
0 0 3 3 0 0 3 3
No Yes No Yes No Yes No Yes
0.12 0.21 0.24 0.27 0.13 0.20 0.19 0.22
0.00 0.00 0.01 0.00 0.01 0.01 0.03 0.01
0.00 0.00 0.02 0.00 0.01 0.01 0.04 0.02
0.00 0.00 0.01 0.00 0.01 0.01 0.03 0.01
0.00 0.01 0.03 0.00 0.02 0.03 0.06 0.04
0.00 0.00 0.01 0.00 0.01 0.01 0.03 0.01
Note: Three configurations omitted due to unrecoverable failures during inference.
(because this comparison is made between LLMs of different size and CoT reasoning), the takeaway is clear. Anchoring a model to external knowledge provides larger performance gain (∆ = +0.10) than any configuration change (i.e. temperature, prompt, reasoning) evaluated in this study. The maximum performance change achieved by configuration change in nonRAG evaluation is ∆ = 0.05 evident in LLM70. 3) Failure Samples: Qualitative analysis indicates that failures in mapping sentences to the ATT&CK taxonomy stem
TABLE V RAG C ONFIGURATION P ERFORMANCE (T = 0.0, 3-S HOT, N O C OT) Configuration RAG
Precision
Recall
F1 Score
Micro
Macro
Micro
Macro
Micro
Macro
0.27
0.27
0.41
0.33
0.32
0.28
from keyword over-reliance, multi-stage context, and insufficient domain knowledge. These failures manifest as Type I (false positive) and Type II (false negative) errors, which are shown below. False Positives: LLMs generate false positives by misinterpreting keywords, hallucinating techniques from benign descriptions, or substituting specific actions with incorrect technical equivalents (see Table VI). False Negatives: LLMs miss techniques by over-relying on literal taxonomy keywords, lacking the domain knowledge to synthesize multi-stage behaviors, or misaligning actions due to semantic overlap (see Table VII). C. Findings of RQ3 : Impact of Inference Settings We analyze the influence of parameter size, prompt strategy, reasoning approach, and generation temperature on micro precision, recall, and F1 score to isolate which factors improve ATT&CK mapping. 1) Parameter Size: Table VIII indicates that parameter size positively correlates with classification performance, with DS236 (236B) achieving the maximum mean F1 of 0.20. However, performance does not scale monotonically relative to parameter size; for instance, LLM70 (70B) outperformed the larger GP120 (120B). Furthermore, the 20B model (GPT20) demonstrated a performance deficit (F1 score = 0.02), showing that architectural or training factors can override the influence of parameter size. Despite these variations, the Spearman rank correlation confirms a statistically significant relationship between parameter size and F1 scores in this CTI classification task (ρ = 0.85, p = 0.014). 2) Zero-Shot vs. Few-Shot Prompting: Table IX shows a marginal increase in mean performance metrics when transitioning from zero-shot (N=0) to few-shot (N=3) prompting. Mean precision increased from 0.16 to 0.17, while mean F1 score increased from 0.13 to 0.15. However, the Spearman rank correlation indicates that the relationship is weak and statistically insignificant (rho = 0.17, p = 0.22). Since the p-value exceeds the α = 0.05 significance threshold, the null hypothesis cannot be rejected; the increase in F1 scores cannot be attributed to the number of contextual examples provided in the prompt. 3) CoT Prompting: Table IX shows that CoT prompting primarily influences recall, which increased from 0.15 to 0.17, while mean precision remained constant at 0.16. The increase in recall resulted in a marginal mean F1 score increase from 0.13 to 0.14. However, the Spearman rank correlation indicates that this relationship is weak and statistically insignificant (ρ = 0.13, p = 0.34). As the p-value exceeds the significance threshold (α = 0.05), the increase in F1 scores cannot be attributed to the use of CoT reasoning.
4) Temperature: Table IX shows that T=0.0 and T=0.5 yield identical aggregate scores across micro precision (0.16), recall (0.16), and F1 score (0.14). The correlation is functionally zero (ρ = 0.00, p = 0.95), confirming that temperature adjustment within this range has no statistically significant monotonic relationship with micro F1 score and does not influence CTI classification performance for this CTI classification task. VI. D ISCUSSION The strongest non-RAG configuration in this study, DS236 with T=0.0, 3-shot, and CoT, reached micro F1 score = 0.22. The RAG-enabled evaluation using LLM70 improved this to F1 score = 0.32, driven by a recall increase from 0.23 to 0.41. These figures sit below the values from earlier tools, including TTPDrill (0.82) [22], TTPHunter (0.88) [48], TTPXHunter (0.97) [49], and AttacKG (0.79) [23]. The gap is explained by methodological differences rather than capability differences. Those tools restrict evaluation to the top-50 ATT&CK techniques, train on ATT&CK procedure descriptions, and show report-level scores. Our evaluation spans 211 technique space, uses independently authored CTI reports, and operates at the sentence level under multi-label cardinality score of 1.58. When prior work is evaluated under similar conditions as ours, the performances drop and look similar to our results. Büchel et al. [50] reports a maximum F1 score of 0.71 (RoBERTa on TRAM2) and 0.63 (CySecBERT on AnnoCTR). Nguyen et al. [51] measured the TRAM/SciBERT performance at F1 ≈ 0.40 under class imbalance. Haque et al. [52] reported that LLM’s highest accuracy varied by report, peaking at 0.786 for the SolarWinds incident but reaching only 0.549 for XZ Utils. Mezzi et al. [27] concluded that LLMs cannot guarantee sufficient performance on real-sized reports, even after finetuning. CTIBench [11] places GPT-4 at micro F1 in the mid0.30s and Llama-3-8B near 0.15 on the CTI-ATE subtask. Our RAG F1 score of 0.32, obtained with a 70B open-source model at 4-bit quantization, is consistent with this range. The RAG improvement of 0.10 F1 score, dominated by recall, aligns with Fayyazi et al. [53], who reported promptonly Samples F1 score = 0.60 against exact-URL RAG F1 = 0.95 and similar-procedure RAG F1 = 0.68. Morbiato et al. [13] demonstrated a further ∆ = +3.8% F1 score with hierarchical retrieval over flat RAG. These results indicate that RAG addresses the recall gap on rare techniques but does not resolve precision-related errors (i.e. flagging the wrong techniques), which Haque et al. [52] attribute to siblingtechnique confusion in 33.3% of misclassifications. Our correlation analysis quantifies an effect which the prior literature has suggested but has not measured: parameter size is the only statistically significant predictor of F1 score (ρ = 0.85, p = 0.014) for this CTI classification task, while N-shot, CoT, and temperature are rather insignificant predictors. Inferencetime prompt tuning will not resolve the F1 score deficit. Closing the gap requires external knowledge and domainspecific training.
TABLE VI FALSE P OSITIVES IN LLM T ECHNIQUE E XTRACTION
LLM DS236
Context (Quote) “... see Defender Control turning off Defender via Local Group Policy Editor”
LLM08
“...ran a second batch file...performed a nslookup for each host” “This was likely to evade detection logic based around command line execution”
LLM70
Ground Truth Impair Defenses, Domain Policy Modification System Network Configuration Discovery None
Prediction Modify Registry, System Binary Proxy Execution Account Discovery, Process Discovery Hijack Execution Flow
Error Analysis Missed the defense impairment and policy modification. Misinterpreted network enumeration as other reconnaissance. Hallucinated based on the keywords “evade” and “execution”.
TABLE VII FALSE N EGATIVES IN LLM T ECHNIQUE E XTRACTION
LLM GMM27 DS236 GPT20
Context (Quote) “The threat actor then staged a ransomware binary on each of the hosts” “...campaign was likely delivered via an email, with a link, causing the executable’s download when clicked” “We assess with high confidence that the delivery was via email”
TABLE VIII I MPACT OF PARAMETER S IZE ON M EAN M ICRO F1 S CORE
LLM DS236 GP120 LLM70 GMM27 GPT20 GMM12 LLM08
Parameters 236B 120B 70B 27B 20B 12B 8B
Mean Micro F1 0.20 0.18 0.19 0.17 0.02 0.13 0.09
Note: Each mean micro F1 score is aggregation of 8 inference configuration. The statistical tests were conducted on all 53 configurations.
TABLE IX I MPACT OF H YPERPARAMETERS ON LLM P ERFORMANCE
Precision
Recall
F1 Score
0 3
0.16 0.17
0.15 0.17
0.13 0.15
CoT
No Yes
0.16 0.16
0.15 0.17
0.13 0.14
Temperature
0.0 0.5
0.16 0.16
0.16 0.16
0.14 0.14
Parameter
Setting
N-Shot
VII. T HREATS TO VALIDITY Internal Validity. Human annotation is subject to interpretive variance. We mitigated this through a six-phase crosslabeling protocol with per-technique Cohen’s κ, achieving κ = 0.68 across 114 techniques, which falls within the substantial agreement range of Landis and Koch [33]. The RAG configuration uses LLM70 without CoT, while the strongest non-RAG configuration uses DS236 with CoT, introducing confounding between retrieval grounding, model size, and reasoning prompts. External Validity. The dataset originates from The DFIR
Ground Truth Ingress Tool Transfer Phishing, User Execution Phishing
Prediction Data Staged None None
Error Analysis Confused the word “staged” with the exfiltration technique. Failed to synthesize the multistage delivery process. Missed the contextual implication of email delivery.
Report and may not generalize to vendor advisories, SOC logs, or alternative CTI sources. Results are tied to ATT&CK Enterprise v17.1 and may shift with framework revisions. All LLMs were executed at Q4_K_M 4-bit quantization, which may introduce degradation relative to full-precision inference. Closed-source LLMs were excluded by design. Construct Validity. Micro F1 score weights all classification errors equally and ignores the hierarchical structure of ATT&CK. Haque et al. [52] report that 33.3% of LLM misclassifications involve sibling techniques sharing the same tactic. Hierarchy-aware metrics such as hF1 [54], set-based hierarchical scoring [55], and CoPHE [56] would assign partial credit for correct-tactic, wrong-technique predictions and produce higher scores.
VIII. C ONCLUSION We established an empirical baseline for open-source LLMs on ATT&CK technique classification using a manually annotated dataset of 2,076 sentences from 83 unstructured CTI reports. Across 53 configurations of 7 LLMs spanning 8B to 236B parameters, the strongest non-RAG configuration reached micro F1 score = 0.22, and a RAG-enabled pipeline reached 0.32. Spearman correlation analysis identifies parameter size as the only statistically significant predictor (ρ = 0.85 , p = 0.014) of F1 score, while prompt strategy, reasoning, and temperature show no significant effect. The findings align with recent evaluations of LLMs for CTI and demonstrate that retrieval grounding, rather than prompt tuning, is the productive direction for closing the remaining gap. Future work should evaluate hierarchical RAG, parameter-efficient fine-tuning, and hierarchy-aware metrics on the released dataset, and should extend sentence-level extraction toward mining temporal and sequential attack patterns [57], [58].
R EFERENCES [1] Statista Research Department, “Estimated cost of cybercrime worldwide 2018–2029,” https://www.statista.com/forecasts/1280009/ cost-cybercrime-worldwide, 2026. [2] B. E. Strom et al., “MITRE ATT&CK: Design and philosophy,” The MITRE Corporation, Tech. Rep. MP180360R1, 2018. [Online]. Available: https://attack.mitre.org/docs/ATTACK_Design_ and_Philosophy_March_2020.pdf [3] E. M. Hutchins et al., “Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains,” Lockheed Martin Corporation, White Paper, 2011. [Online]. Available: https://www.lockheedmartin.com/content/dam/lockheed-martin/ rms/documents/cyber/LM-White-Paper-Intel-Driven-Defense.pdf [4] S. Caltagirone et al., “The diamond model of intrusion analysis,” Center for Cyber Threat Intelligence and Threat Research, Technical Report ADA586960, 2013. [Online]. Available: https://apps.dtic.mil/sti/ citations/ADA586960 [5] Verizon Risk Team, “The vocabulary for event recording and incident sharing (VERIS) framework,” https://verisframework.org, 2010. [6] P. Wang et al., “Research on discovery and mapping of ATT&CK tactics and techniques by cyber threat intelligence based on BERT-TextCNN,” IEEE Access, 2026. [7] P. N. Wudali et al., “Rule-ATT&CK mapper (RAM): Mapping SIEM rules to TTPs using LLMs,” arXiv preprint arXiv:2502.02337, 2025. [8] M. R. Rahman and L. Williams, “From threat reports to continuous threat intelligence: A comparison of attack technique extraction methods from textual artifacts,” 2022. [9] M. R. Rahman et al., “What are the attackers doing now? automating cyberthreat intelligence extraction from text on pace with the changing threat landscape: A survey,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–36, 2023. [10] W. X. Zhao et al., “A survey of large language models,” 2023. [11] M. T. Alam et al., “CTIBench: A benchmark for evaluating LLMs in cyber threat intelligence,” in Proc. NeurIPS 2024, Datasets and Benchmarks Track. Curran Associates, Inc., 2024, pp. 50 805–50 825. [12] Center for Threat-Informed Defense, “TRAM: Threat report ATT&CK mapper,” https://github.com/center-for-threat-informed-defense/tram, 2020. [13] F. Morbiato et al., “Hierarchical RAG for adversarial technique annotation,” 2026. [14] M. Tamanna et al., “What are adversaries doing? automating tactics, techniques, and procedures extraction: A systematic review,” 2026. [15] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Proc. NeurIPS 2022. Curran Associates, Inc., 2022, pp. 24 824–24 837. [16] T. B. Brown et al., “Language models are few-shot learners,” in Proc. NeurIPS 2020. Curran Associates, Inc., 2020, pp. 1877–1901. [17] D. C. Montgomery, Design and Analysis of Experiments, 10th ed. Hoboken, NJ: John Wiley & Sons, 2020. [18] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Proc. NeurIPS 2020. Curran Associates, Inc., 2020, pp. 9459–9474. [19] Y. Schwartz et al., “LLMCloudHunter: Harnessing LLMs for automated extraction of detection rules from cloud-based CTI,” in Companion Proc. ACM WWW 2025. ACM, 2025. [20] V. Legoy et al., “Automated retrieval of ATT&CK tactics and techniques for cyber threat reports,” arXiv preprint arXiv:2004.14322, 2020. [21] S. Della Penna et al., “CTI-HAL: A human-annotated dataset for cyber threat intelligence analysis,” 2025. [22] G. Husari et al., “TTPDrill: Automatic and accurate extraction of threat actions from unstructured text of CTI sources,” in Proc. ACSAC 2017. ACM, 2017, pp. 103–115. [23] Z. Li et al., “AttacKG: Constructing technique knowledge graph from cyber threat intelligence reports,” in Proc. ESORICS 2022. Springer, 2022, pp. 589–609. [24] B. Abdeen et al., “SMET: Semantic mapping of CVE to ATT&CK and its application to cybersecurity,” in Proc. DBSec 2023. Springer, 2023, pp. 243–260. [25] E. Aghaei et al., “SecureBERT: A domain-specific language model for cybersecurity,” in Proc. SecureComm 2022. Springer, 2023, pp. 39–56. [26] S. Mitra et al., “FALCON: Autonomous cyber threat intelligence mining with LLMs for IDS rule generation,” 2025.
[27] E. Mezzi et al., “Large language models are unreliable for cyber threat intelligence,” 2025. [28] The DFIR Report, “The DFIR report — real intrusions by real attackers,” https://thedfirreport.com, 2024. [29] Zyte and Scrapy Developers, “Scrapy: An open source and collaborative framework for extracting data from websites,” https://scrapy.org, 2024. [30] L. Richardson, “Beautiful Soup: A python library for pulling data out of HTML and XML files,” https://www.crummy.com/software/ BeautifulSoup/, 2024. [31] L. Ou-Yang, “Newspaper3k: Article scraping and curation,” https:// github.com/codelucas/newspaper, 2024. [32] J. Cohen, “A coefficient of agreement for nominal scales,” Educ. Psychol. Meas., vol. 20, no. 1, pp. 37–46, 1960. [33] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977. [34] A. Holtzman et al., “The curious case of neural text degeneration,” in Proc. ICLR 2020, 2020. [Online]. Available: https://openreview.net/ forum?id=rygGQyrFvH [35] J. Kaplan et al., “Scaling laws for neural language models,” 2020. [36] A. Dubey et al., “The Llama 3 herd of models,” 2024. [37] Gemma Team et al., “Gemma 2: Improving open language models at a practical size,” 2024. [38] DeepSeek-AI, “DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model,” 2024. [39] ——, “DeepSeek-V2.5 model repository,” https://huggingface.co/ deepseek-ai/DeepSeek-V2.5, 2024. [40] OpenAI, “GPT-OSS-120B model repository,” https://huggingface.co/ openai/gpt-oss-120b, 2024. [41] Meta AI, “Meta-Llama-3.1-70B-Instruct model repository,” https:// huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct, 2024. [42] Google DeepMind, “Gemma-3-27B model repository,” https://huggingface.co/google/gemma-3-27b-it, 2024. [43] OpenAI, “GPT-OSS-20B model repository,” https://huggingface.co/ openai/gpt-oss-20b, 2024. [44] Google DeepMind, “Gemma-3-12B model repository,” https://huggingface.co/google/gemma-3-12b-it, 2024. [45] Meta AI, “Meta-Llama-3.1-8B-Instruct model repository,” https:// huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct, 2024. [46] Meta Platforms, Inc., “Faiss: A library for efficient similarity search and clustering of dense vectors,” https://faiss.ai, 2024. [47] C. Spearman, “The proof and measurement of association between two things,” Am. J. Psychol., vol. 15, no. 1, pp. 72–101, 1904. [48] N. Rani et al., “TTPHunter: Automated extraction of actionable intelligence as TTPs from narrative threat reports,” in Proc. ACSW 2023. ACM, 2023, pp. 126–134. [49] ——, “TTPXHunter: Actionable threat intelligence extraction as TTPs from finished cyber threat reports,” Digit. Threats Res. Pract., vol. 5, no. 4, pp. 1–19, 2024. [50] M. Büchel et al., “SoK: Automated TTP extraction from CTI reports,” in Proc. USENIX Security 2025. USENIX Association, 2025, pp. 4621– 4641. [51] H. C. Nguyen et al., “Towards effective identification of attack techniques in cyber threat intelligence reports using large language models,” 2025. [52] M. N. Haque et al., “Beyond single reports: Evaluating automated ATT&CK technique extraction in multi-report campaign settings,” in Proc. ASE 2026. IEEE/ACM, 2026. [53] R. Fayyazi et al., “Advancing TTP analysis: Harnessing the power of large language models with retrieval augmented generation,” 2024. [54] S. Kiritchenko et al., “Learning and evaluation in the presence of class hierarchies: Application to text categorization,” in Proc. Canadian AI 2006. Springer, 2006, pp. 395–406. [55] A. Kosmopoulos et al., “Evaluation measures for hierarchical classification: A unified view and novel approaches,” Data Min. Knowl. Discov., vol. 29, no. 3, pp. 820–865, 2015. [56] M. Falis et al., “CoPHE: A count-preserving hierarchical evaluation metric in large-scale multi-label text classification,” in Proc. EMNLP 2021. Association for Computational Linguistics, 2021, pp. 907–912. [57] M. R. Rahman et al., “Mining temporal attack patterns from cyberthreat intelligence reports,” Knowl. Inf. Syst., vol. 67, no. 10, pp. 8941–8981, 2025. [58] ——, “ChronoCTI: Mining knowledge graph of temporal relations among cyberattack actions,” in Proc. IEEE ICDM 2024. IEEE, 2024, pp. 420–429.