TTP RINT: Evidence-Grounded TTP Extraction via Diverge-then-Converge Verification Yutong Cheng1 , Changze Li1 , Raihan Sultan Pasha Basuki2 , Qian Cui3 , Wei Ding3 , Peng Gao1 1 Virginia Tech 2 Universitas Ary Ginanjar, Jakarta, Indonesia 3 Amazon
Abstract
arXiv:2605.25836v1 [cs.CR] 25 May 2026
Extracting MITRE ATT&CK techniques from cyber threat intelligence (CTI) reports is an open-set, multi-label problem requiring both high recall (not missing techniques) and high precision (not hallucinating unsupported ones). Existing methods—rule-based, supervised, and LLM-based—struggle to achieve both: rule-based and supervised approaches lack generalizability across diverse attack descriptions, while LLM-based approaches that couple candidate generation and validation within a single inference step suffer from limited recall and precision simultaneously. We propose TTP RINT, which addresses this challenge through a diverge-then-converge design inspired by how human analysts work: first extracting broadly, then verifying rigorously. In the divergent phase, reports are decomposed into atomic behaviors and candidate techniques are proposed broadly. A deterministic span localization stage then anchors each candidate to a specific evidence window in the source text. A convergent verification stage retains only candidates supported by both the localized evidence and the authoritative MITRE definition. We contribute two evaluation resources—a cleaned TRAM benchmark (TRAM-Clean) and a new annotated dataset (TTP RINT-Bench)— to address known annotation noise in existing benchmarks and elevate the task to documentlevel TTP extraction. On TRAM-Clean and TTP RINT-Bench, TTP RINT achieves 76.48% and 87.39% macro-F1 respectively, outperforming the leading baseline by 63.5% and 29.4%. A multi-backbone analysis across six LLMs and a threshold sensitivity study further demonstrate generalizability across model choices and provide practical guidance for parameter selection.
1
Introduction
Indicators of Compromise (IoCs) such as malicious IP addresses, file hashes, and domain names have long served as the primary currency of cyber threat
intelligence (CTI). However, IoCs are inherently ephemeral: adversaries can rotate infrastructure, recompile binaries, and register new domains with minimal effort, rendering specific indicators obsolete within days (Büchel et al., 2025; Huang et al., 2024b). In contrast, tactics, techniques, and procedures (TTPs) describe how attackers achieve their objectives—the methods they use to gain initial access, move laterally, escalate privileges, and exfiltrate data—and remain operationally valuable long after individual IoCs have expired (Husari et al., 2017). The MITRE ATT&CK framework provides a standardized ontology for cataloging these TTPs, and automatically extracting them from unstructured CTI reports has become a central task in threat intelligence analysis (Strom et al., 2020). However, automatically extracting MITRE ATT&CK techniques from CTI reports faces several challenges. These reports are lengthy, loosely structured documents in which multiple attack techniques may be described across dozens of paragraphs, often without explicit mention of MITRE identifiers. TTPs are frequently embedded in narrative descriptions of adversary behavior and must be inferred from context rather than matched by keywords (Husari et al., 2017; Büchel et al., 2025). As a result, TTP extraction is an open-set, multi-label problem: a model must simultaneously achieve high recall (not missing techniques that are genuinely described) and high precision (not hallucinating techniques that lack textual support). Balancing these two objectives is the core challenge. Existing methods approach this task from three paradigms, each with characteristic limitations. Rule-based systems such as TTPDrill (Husari et al., 2017) offer interpretability but depend on manually constructed ontologies that cannot keep pace with evolving attack descriptions. Supervised neural models, including LADDER (Alam et al., 2023) and MITREtrieval (Huang et al., 2024b), learn contextual patterns from labeled data but require
Figure 1: The diverge-then-converge principle behind TTP RINT. Existing single-pass methods couple precision and recall within one inference step. TTP RINT explicitly separates them: the divergent phase (top) decomposes a CTI report into atomic behaviors and proposes a broad set of candidate techniques, maximizing coverage. Evidence grounding (center) anchors each candidate to a localized sentence window in the original report, restricting downstream reasoning to verifiable textual support. The convergent phase (bottom) filters candidates through a verification threshold τ , retaining only techniques that are explicitly supported by the grounded evidence.
substantial annotation effort and degrade under distribution shift to new report styles or technique vocabularies (Büchel et al., 2025). With the rapid advancement of large language models (LLMs), growing interest has turned to applying them to CTI extraction and analysis. CTINexus (Cheng et al., 2025) and AttacKG+ (Zhang et al., 2025) leverage LLMs to construct CTI knowledge graphs from reports, with AttacKG+ producing TTP labels as an intermediate product within a broader multi-layered extraction pipeline. However, no existing work focuses on applying LLMs to extract TTPs from CTI reports as a standalone task—despite this being a critical need in the CTI analyst workflow. High-recall, precise TTP extraction directly informs detection rule authoring, threat actor profiling, and defensive prioritization; missed techniques leave blind spots in defensive coverage, while hallucinated ones waste analyst effort on nonexistent threats. Applying LLMs to TTP extraction is challenging because the task is inherently a bi-objective optimization problem: maximizing coverage over
all techniques described in a report while precisely mapping each behavior to the correct MITRE ATT&CK identifier. When addressed through single-pass prompting, these two objectives are tightly coupled, introducing two compounding failure modes. First, the model may sacrifice coverage for confidence, omitting behaviors that are genuinely present but less salient within a dense, multi-technique passage. Second, even when a behavior is correctly extracted, a surface similarity mismatch—a gap between the operational language used in CTI reports and the canonical terminology in MITRE ATT&CK definitions—degrades the accuracy of the behavior-to-identifier mapping. For example, a report stating that “the malware writes itself to the Run registry key to survive reboots” describes T1547 (Boot or Logon Autostart Execution)1 , yet the phrasing shares almost no lexical overlap with the technique name or its official definition, causing the model to select a superficially closer but incorrect technique. To address these challenges, we propose 1
https://attack.mitre.org/techniques/T1547/
TTP RINT, a four-stage pipeline that explicitly separates hypothesis generation from hypothesis verification through a diverge-then-converge design. In the divergent phase, TTP RINT decomposes a CTI report into atomic attack behaviors and proposes a broad set of candidate MITRE ATT&CK techniques for each behavior, deliberately favoring coverage over confidence. A deterministic span localization stage then anchors each candidate to a specific sentence window in the original report, restricting all downstream reasoning to verifiable textual evidence rather than the full document. In the convergent phase, a verification stage evaluates each candidate against both its localized evidence span and the official MITRE ATT&CK technique description, retaining only those that exceed a confidence threshold τ . This design decouples the two failure modes identified above: the divergent phase eliminates omission risk by treating extraction as a standalone objective, while the convergent phase resolves surface similarity mismatch by cross-referencing evidence against authoritative definitions. The threshold τ gives practitioners direct control over the precision–recall trade-off through a single interpretable parameter, and every final prediction is traceable to its source passage. To comprehensively benchmark TTP RINT’s performance in TTP extraction, we contribute two evaluation resources. First, we systematically clean the most widely used benchmark, TRAM (Ross and Lasky, 2023), a sentence-level TTP evaluation dataset originally designed to fine-tune a BERT-based (Devlin et al., 2019) NER model. TRAM contains a significant number of annotation errors, including both false positives and false negatives; the corrected version, TRAM-Clean, provides a reliable baseline for comparison with prior work. However, sentence-level evaluation alone cannot assess a system’s ability to extract TTPs from complete reports, where techniques are distributed across dozens of paragraphs and must be identified without pre-segmentation. To evaluate this more realistic and challenging setting, we construct TTP RINT-Bench, a new document-level benchmark of 150 CTI reports that requires extracting TTPs directly from full, unstructured documents. Both datasets will be released to support further research and evaluation on automated TTP extraction. Our contributions are as follows:
• We identify precision–recall coupling as a structural limitation of single-pass TTP extraction and propose the diverge-then-converge principle, which explicitly separates hypothesis generation from evidence-grounded verification. • We instantiate this principle in TTP RINT, a four-stage pipeline featuring deterministic span localization that anchors every prediction to source evidence, enabling full traceability and fine-grained precision–recall control via a single threshold parameter τ . • We contribute two evaluation resources—a cleaned version of the TRAM benchmark (TRAM-Clean) and a new document-level dataset (TTP RINT-Bench)—that address annotation noise in existing benchmarks and elevate the task from passage-level to report-level extraction. • We conduct a comprehensive evaluation on two distinct benchmarks, including ablation studies isolating the contribution of each pipeline stage and a multi-backbone analysis across six LLMs demonstrating modelagnosticism. TTP RINT achieves a Macro-F1 of 76.48% on TRAM-Clean and 87.39% on TTP RINT-Bench, outperforming the leading baseline by 63.5% and 29.4% respectively.
2
Related Work
TTP Extraction from CTI Reports. Automatically extracting MITRE ATT&CK techniques from CTI reports has been studied through several lines of work. Early approaches such as TTPDrill (Husari et al., 2017) rely on rule-based matching using dependency parsing and predefined ontologies. These methods are easy to interpret but struggle when the same technique is described in different ways or only implied in the text. Later work moves toward supervised neural models. Systems such as EXTRACTOR (Satvat et al., 2021), AttacKG (Li et al., 2022), LADDER (Alam et al., 2023), and MITREtrieval (Huang et al., 2024b) typically treat the problem as classification or alignment over ATT&CK techniques using fine-tuned transformer models. While they capture context better than rule-based methods, they depend on labeled data and often do not generalize well across different report styles (Büchel et al., 2025). More recently, large language models (LLMs) have been
applied to CTI analysis tasks. AttacKG+ (Zhang et al., 2025) and CTINexus (Cheng et al., 2025) focus on CTI knowledge graph construction, where AttacKG+ treats TTP labeling as one layer of a multi-layered output that also includes behavior graphs and state summaries, with its technique identifier mapping pre-extracted structured behavior graphs to ATT&CK labels rather than extracting techniques from raw report text. In contrast, TTP RINT targets the problem setting where a system must extract a precise and comprehensive set of TTPs directly from raw CTI reports. LLM-Based Structured Extraction and Verification. Similar challenges appear in other domains such as medical coding, where long documents must be mapped to large label spaces. Recent approaches break the task into steps such as evidence extraction and candidate ranking (Baksi et al., 2025; Motzfeldt et al., 2025), suggesting that a single LLM call is often insufficient. More broadly, LLM-based information extraction has been explored for relation extraction and document-level extraction tasks (Swarup et al., 2025; Bhattacharyya et al., 2025), as well as multi-stage pipelines that incorporate validation mechanisms for structured data extraction (Aggarwal et al., 2025). To reduce hallucinations, methods such as Chain-of-Verification (Dhuliawala et al., 2024) and Self-Refine (Madaan et al., 2023) introduce additional reasoning or refinement steps. These approaches operate after generation and rely on the model to correct itself, which can be unreliable without external grounding (Huang et al., 2024a). In contrast, TTP RINT restricts the input by localizing evidence spans with deterministic algorithms and then verifies candidates against MITRE ATT&CK definitions through confidence-calibrated verification, producing evidence-grounded, source-anchored TTP extraction outputs.
3
TTP RINT Design
As shown in Figure 2, TTP RINT operationalizes the diverge-then-converge principle through a fourstage pipeline organized into two phases. The divergent phase—behavior extraction (§3.1) and TTP proposal (§3.3)—prioritizes recall by decomposing the report into atomic behaviors and generating a broad set of candidate techniques. The convergent phase—span localization (§3.2) and TTP verification (§3.4)—prioritizes precision by
grounding each candidate to localized textual evidence and discarding those that lack support. We detail each stage in the following sections. 3.1
Behavior Extraction
The first stage decomposes a raw CTI report into a set of atomic attack behaviors. Given an input document D, we prompt an LLM to produce a list of minimal, self-contained descriptions of adversarial actions that are explicitly mentioned in the text. Each behavior bi isolates a single attack action while explicitly preserving critical technical artifacts—tool names, command strings, registry paths, infrastructure identifiers, or protocol details. This preservation is essential to prevent context collapse, where the LLM abstracts away specific artifacts during extraction, producing generic behavior descriptions that are insufficient for accurate downstream technique identification. This decomposition addresses a fundamental challenge in document-level TTP extraction: a single CTI report may describe dozens of distinct attack actions spanning multiple tactics, and directly prompting an LLM to enumerate all relevant techniques from the full document leads to omissions, conflation, and loss of specificity. By isolating individual behaviors, this stage creates atomic units that can each be independently grounded and classified in subsequent stages. 3.2
Span Localization
This stage is the architectural keystone of TTP RINT: it bridges the divergent and convergent phases by establishing the extractive evidence link that all downstream predictions must pass through. For each behavior bi ∈ B, a deterministic span localization module identifies the most relevant contiguous sentence window si in D that supports bi . The localization algorithm operates in three steps. First, bag-of-words overlap scoring: for each candidate sentence window of length up to ℓmax sentences, we compute the fraction of normalized behavior tokens that appear in the window. Second, frequency-aware weighting: tokens that appear in more than a top-p fraction of the corpus are downweighted to prevent common terms from dominating the score. Third, n-gram matching: bigram and trigram overlap is computed as a secondary signal to reward contiguous phrase matches that indicate specific technical content (e.g., tool names, file paths). The window with the highest combined score is selected as the evidence span si . Rather
Convergent Phase: Filter by Evidence (Precision ↑)
Divergent Phase: Maximize Coverage (Recall ↑)
① Behavior Extraction
③ TTP Proposal
② Span Localization
④ TTP Verification
Verified TTP
Atomic Attack Behaviors
Hypothesis Generation: propose broadly, defer judgement
Figure 2: TTP RINT pipeline overview. The pipeline is organized into two phases. The divergent phase (Stages ➀ and ➂) maximizes candidate coverage by decomposing a CTI report into atomic attack behaviors and proposing multiple candidate MITRE ATT&CK techniques per behavior. The convergent phase (Stages ➁ and ➃) filters candidates through evidence grounding: span localization anchors each behavior to a specific sentence window via deterministic lexical matching, and TTP verification retains only candidates whose confidence exceeds a threshold τ after cross-referencing the localized evidence against authoritative MITRE ATT&CK definitions.
than receiving the full document D as context, the TTP proposal and verification stages receive only the localized span si . This input-level restriction limits the information available to the LLM, ensuring that proposed techniques are grounded in a specific, identifiable passage. 3.3
TTP Proposal
Given a behavior bi and its localized evidence span si , the TTP proposal stage generates a small set of candidate MITRE ATT&CK techniques. We prompt an LLM to propose up to k parent-level technique identifiers that plausibly correspond to the observed behavior, conditioned on both bi and si . This design draws inspiration from Best-ofN sampling (Gao et al., 2023; Wu et al., 2024), which improves generation quality by producing N independent samples and selecting the highestscoring one. However, independent sampling tends to cluster around the model’s mode, yielding nearduplicate candidates. We instead instruct the LLM to propose k distinct techniques within a single generation, explicitly encouraging diverse interpretations of the same behavior and improving coverage of the candidate space. The prompt constrains outputs to valid technique identifiers present in the provided MITRE ATT&CK corpus. By separating proposal from verification, TTP RINT avoids the premature filtering that characterizes single-step approaches. 3.4
posed for behavior bi , the verification module provides the LLM with three inputs: the behavior text bi , its evidence span si , and the official MITRE ATT&CK description of technique tj . The model is instructed to determine whether the technique is explicitly supported based solely on the provided evidence, without relying on external knowledge or assumptions beyond what is stated in si . By providing the canonical definition, we standardize the verification criterion: the model assesses whether the evidence span describes an action that matches the official technique semantics. The verifier outputs a presence confidence score cver j ∈ [0, 1] for each candidate. Techniques satisfying cver ≥ τ j are retained as final predictions. The threshold τ serves as an explicit, tunable precision–recall knob: higher values of τ yield fewer but more confident predictions (favoring precision), while lower values admit more candidates (favoring recall). We set τ = 0.7 as a default based on our threshold sensitivity analysis (§4.5), which identifies this value as the optimal operating point.
4
Evaluation
We evaluate TTP RINT along four dimensions: overall effectiveness against baselines (4.2), the individual contribution of each pipeline stage (4.3), robustness across LLM backbones (4.4), and a detailed analysis of the verification threshold’s impact on precision–recall trade-offs (4.5).
TTP Verification
The final stage evaluates whether each proposed technique is genuinely supported by the localized evidence. For every candidate technique tj pro-
4.1
Evaluation Setup
Datasets. We evaluate on two datasets. TRAMClean is our corrected version of the TRAM
Method TTPDrill One-Shot GPT-4o CoT GPT-4o TTP RINT
TRAM-Clean TTP RINT-Bench Type Prec. Rec. F1 Prec. Rec. F1 Rule 1.49 5.03 2.19 4.00 35.72 7.16 LLM 9.56 10.51 9.66 62.99 52.80 44.23 LLM 12.54 14.59 12.99 69.90 50.70 58.02 LLM 74.59 82.10 76.48 82.15 95.50 87.39
Table 1: Document-level results on TRAM-Clean and TTP RINT-Bench. Best results in bold.
dataset (Ross and Lasky, 2023), whose original annotations were designed to fine-tune a BERT-based NER model and contain substantial false positives and false negatives. We manually re-annotated the dataset under a blind protocol. TTP RINT-Bench is a new document-level benchmark of 150 full CTI reports drawn from 12 vendor and journalism sources published between 2022 and 2025. It covers 125 unique MITRE ATT&CK techniques, including 66 rare techniques (fewer than 5 occurrences) that support long-tail evaluation. Each report carries sentence-level annotations produced under double-annotation with adjudication by 3 qualified annotators, achieving Cohen’s κ = 0.76. Full source list, annotation procedure, and interannotator agreement are in Appendix B. Baselines. We compare TTP RINT against baselines from two methodological families. For rulebased approaches, we include TTPDrill (Husari et al., 2017), which uses dependency parsing with a manually constructed threat-action ontology and BM25 scoring. For LLM-based approaches, we include two direct prompting baselines: One-Shot, which classifies each sentence with one in-context example, and CoT, which adds explicit chain-ofthought reasoning to the same prompt. These two baselines are particularly informative because they isolate the effect of reasoning depth without architectural decomposition, directly testing our hypothesis that single-pass inference couples precision and recall. All baselines are evaluated using official implementations or faithful reproductions. We exclude supervised neural models as these methods require substantial labeled training data, and our evaluation datasets cannot support a fair comparison under their expected training regime. Metrics. TTP extraction is a multi-label prediction problem: each report may map to multiple techniques, and models must identify the complete set. We evaluate at the document level by aggregating predictions across the full report and comparing the resulting technique set to the ground truth. Fi-
nal scoresP are macro-averages over all documents: 1 F1 = |D| d∈D F1d . 4.2
RQ1: Overall Effectiveness
Table 1 reports document-level results. TTP RINT achieves the highest macro-F1 on both datasets by a wide margin: 87.39% on TTP RINT-Bench (+29.37 over CoT GPT-4o at 58.02%) and 76.48% on TRAM-Clean (+63.49 over CoT GPT-4o at 12.99%). Both precision and recall improve simultaneously—on TTP RINTBench, recall rises from 50.70% to 95.50% and precision from 69.90% to 82.15%. The same method ranking holds on TRAM-Clean, indicating the gains generalize across source distributions and annotation quality. The two single-pass LLM baselines exhibit the failure modes identified in Section 1. One-Shot GPT-4o achieves 52.80% recall and 62.99% precision (F1 44.23%), reflecting incomplete behavior extraction and surface similarity mismatch respectively. Adding chain-of-thought (CoT GPT4o) raises precision to 69.90% but recall stays at 50.70%, indicating deeper reasoning alone cannot recover missing behaviors. TTPDrill trails all LLM baselines on TTP RINT-Bench (F1 7.16%, precision 4.00%, recall 35.72%). Its manually constructed ontology cannot capture the paraphrases and implicit descriptions common in modern CTI reports (Büchel et al., 2025). 4.3
RQ2: Ablation Study
Setup. To quantify the contribution of each pipeline stage, we evaluate four variants of TTP RINT on TTP RINT-Bench: (1) the full pipeline; (2) w/o Behavior Extraction, where the raw document is passed directly to span localization without decomposition; (3) w/o Span Localization, where TTP proposal and verification operate over the full document rather than a localized evidence window; and (4) w/o Verification, where all proposed candidates above a minimal confidence threshold are accepted without the verification stage.
100
High Recall
Balanced
High Precision
Score (%)
80
60
40
20
Precision Recall Macro-F1
τ=0.7
0 0.4
0.5
0.6
0.7
0.8
0.9
Verification threshold τ
(a) Ablation on TTP RINT-Bench (GPT-4o). Darker segments show (b) Effect of τ on TTP RINT-Bench (GPT-4o). Macro-F1 the drop (∆) from the full pipeline. peaks at τ = 0.7.
Figure 3: Ablation and threshold sensitivity analysis on TTP RINT-Bench (GPT-4o).
Results. As shown in Figure 3a, each ablation produces a distinct degradation pattern. Removing behavior extraction causes the largest recall drop (∆ = −19.2%, from 95.50% to 76.30%) while precision remains relatively stable (∆ = −4.9%): without decomposition into atomic units, techniques embedded in complex paragraphs are missed, but the convergent phase still filters unsupported candidates. Removing verification causes the largest precision drop (∆ = −26.6%, from 82.15% to 55.55%) and the largest overall F1 degradation (∆ = −23.0%), as all proposed candidates are accepted without evidence-based scrutiny. Recall also decreases (∆ = −16.1%), reflecting the loss of MITRE definition cross-referencing that helps disambiguate borderline candidates. Removing span localization degrades both precision and recall substantially (∆ = −8.9% and −18.3%, respectively), yielding an F1 drop of −13.2%. Without evidence grounding, proposal and verification operate over the full document, increasing both hallucination risk and context dilution. Taken together, the ablation reveals that the stages are not merely additive but mutually reinforcing. Verification improves precision as expected (∆ = −26.6%), but it also contributes meaningfully to recall (∆ = −16.1%) through MITRE definition cross-referencing that disambiguates borderline candidates. Span localization degrades both metrics when removed, confirming that constraining the LLM’s input context benefits not only precision (by excluding irrelevant passages) but also recall (by reducing noise that causes the verifier to reject valid candidates). This cross-phase interaction explains why TTP RINT’s overall gains exceed the
Backbone GPT-4o GPT-4o-mini Claude Sonnet 4.6 Gemini 2.5 Flash Qwen 3 235B DeepSeek v3.2
Type Closed, large Closed, small Closed, large Closed, small Open, large Open, large
Prec. 82.15 61.31 65.09 46.62 57.81 67.05
Rec. 95.50 57.49 80.12 94.41 80.91 77.94
F1 87.39 58.01 71.07 61.62 66.59 71.04
Table 2: Effect of LLM backbone on TTP RINT-Bench. All pipeline hyperparameters are held constant.
sum of what any single stage contributes in isolation: evidence grounding makes both upstream proposals and downstream verification more effective. 4.4
RQ3: LLM Backbone Effectiveness
Setup. To test whether TTP RINT’s gains come from the pipeline architecture rather than a specific LLM, we instantiate the full pipeline with multiple backbones spanning two axes: model family (closed-source vs. open-source) and scale. Closed-source models include GPT-4o, GPT-4omini, Claude Sonnet 4.6, and Gemini 2.5 Flash. Open-source models include Qwen 3 235B and DeepSeek v3.2. All pipeline hyperparameters (ℓmax , τ , temperature) are held constant across backbones; see Appendix C for the complete configuration. Results. Table 2 reports results for each backbone on TTP RINT-Bench. Two key findings emerge. First, every backbone instantiated within TTP RINT substantially outperforms the best single-pass baseline (CoT GPT-4o at 58.02% F1), confirming that the pipeline architecture provides consistent gains regardless of the underlying model. Even the smallest closed-source model (Gemini
2.5 Flash) and the open-source alternatives achieve macro-F1 scores well above the single-pass ceiling, demonstrating that the diverge-then-converge decomposition is the primary driver of improvement, not raw model capability. Second, while a performance gap exists between large and small models, this gap is substantially smaller than the gap between single-pass and pipeline-based approaches. GPT-4o achieves the highest F1 as the default backbone (87.39%), with Claude Sonnet 4.6 and DeepSeek v3.2 following closely (71.07% and 71.04%, respectively). GPT-4o-mini and Gemini 2.5 Flash, despite being smaller models, still achieve F1 scores of 58.01% and 61.62%—both matching or exceeding CoT GPT-4o’s 58.02%. This confirms that a smaller model with the pipeline architecture can match or outperform a larger model without it. The strong performance of open-source backbones (Qwen 3 235B and DeepSeek v3.2) has practical implications: organizations that cannot send sensitive CTI reports to external APIs due to data sovereignty or confidentiality requirements can deploy TTP RINT with a locally hosted backbone and retain the vast majority of performance gains. The architecture is effectively model-agnostic. 4.5
RQ4: Threshold Sensitivity
Figure 3b plots precision, recall, and macro-F1 as a function of τ . Three findings emerge. First, the frontier is smooth and monotonic: as τ increases from 0.30 to 0.95, precision rises steadily while recall falls, with no abrupt jumps or inversions. This indicates that the verification scores cver j are well-calibrated—small changes in τ produce proportional changes in system behavior, making the threshold a reliable and predictable control parameter in practice. Second, the curve reveals a clear F1-optimal region. Macro-F1 peaks at τ = 0.7 (87.39%), confirming the default value used throughout RQ1–RQ3. The F1 surface is relatively flat in the neighborhood of the optimum: for τ ∈ [0.55, 0.75], macro-F1 remains within approximately 3% of the peak. This robustness to moderate mis-specification of τ is a desirable property for practical deployment where per-dataset tuning may not be feasible. Third, the frontier reveals three natural operating regimes for different deployment scenarios (shaded regions in Figure 3b). In the high-recall regime (τ ≤ 0.525), the system achieves recall above 98% with precision around 63–72%, suitable for SOC alert triage
where missing a technique is costlier than investigating a false alarm. In the balanced regime (0.525 < τ ≤ 0.75), macro-F1 is maximized, with the optimal point at τ = 0.7 delivering 82.15% precision and 95.50% recall—appropriate for general-purpose CTI analysis. In the highprecision regime (τ > 0.75), precision exceeds 90% with recall still above 75%, appropriate for automated response workflows where every prediction must be highly reliable before triggering defensive actions. Practitioners can select their operating point based on organizational risk tolerance by adjusting a single parameter, without modifying any pipeline component or rerunning inference.
5
Conclusion and Future Work
We introduced the diverge-then-converge principle for TTP extraction and instantiated it in TTP RINT, a four-stage pipeline that decouples recall-oriented candidate generation from precision-oriented evidence-grounded verification. On TRAM-Clean and TTP RINT-Bench, TTP RINT achieves 76.48% and 87.39% macro-F1 respectively, outperforming the strongest baseline by over 29% on TTP RINTBench while improving both precision and recall simultaneously. Ablations confirm the architecture’s design: divergent components control recall, convergent components control precision, and a single threshold τ provides continuous, inference-free control over the tradeoff. Results hold across six LLM backbones— including open-source models suitable for airgapped deployments—demonstrating that the gains are architectural, not model-dependent. Every prediction is traceable to a localized evidence span, supporting analyst auditability. We will release both evaluation resources, TRAM-Clean and TTP RINT-Bench, to facilitate future research on TTP extraction. We plan to package TTP RINT as a modular LLM skill—a self-contained capability that can be invoked by broader LLM-based CTI analysis pipelines as a callable component. This would allow downstream systems performing tasks such as threat actor profiling, attack graph construction, or automated report generation to obtain structured, evidence-grounded TTP extractions, lowering the barrier to integrating high-quality TTP extraction into end-to-end threat intelligence workflows.
Limitations TTP RINT targets English-language CTI reports published by major security vendors and threat intelligence outlets, which is the dominant medium for technical adversary descriptions and the natural use case for analyst tooling. Extension to nonEnglish CTI or to adjacent document genres such as incident response reports and malware analysis writeups would require source-specific annotation and is left to future work. The verification stage relies on the official MITRE ATT&CK technique descriptions as its canonical reference; where the underlying taxonomy itself contains semantically adjacent techniques (e.g., within Discovery or Defense Evasion), TTP RINT surfaces candidates faithfully but cannot resolve ambiguity that is structurally present in the source ontology. Finally, our evaluation measures extraction quality against expert-annotated ground truth; the impact of TTP RINT on downstream operational metrics in a live SOC—triage time, detection coverage, attribution confidence—requires controlled deployment studies, which fall outside the scope of a benchmark-driven contribution.
Ethics Statement TTP RINT assists defensive analysts in identifying adversary behaviors described in publicly published CTI reports authored by security vendors and journalists; the MITRE ATT&CK framework and the source reports are already accessible to both defenders and adversaries, so the net effect of automation is to reduce the analytical burden on defenders rather than to enable new offensive capabilities. The evaluation datasets contain no private user data, and victim-identifying information from source reports is retained only where required for technical TTP labeling. For organizations handling sensitive threat intelligence, our multi-backbone analysis demonstrates that TTP RINT retains its performance gains with open-source LLMs hosted locally, removing any need to transmit report content to third-party providers.
References Vikram Aggarwal, Jay Kulkarni, Aditi Mascarenhas, Aakriti Narang, Siddarth Raman, Ajay Shah, and Susan Thomas. 2025. Information extraction from fiscal documents using llms. arXiv preprint arXiv:2511.10659.
Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2023. Looking beyond iocs: Automatically extracting attack patterns from external cti. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID ’23, page 92–108, New York, NY, USA. Association for Computing Machinery. Krishanu Das Baksi, Elijah Soba, John J Higgins, Ravi Saini, Jaden Wood, Jane Cook, Jack I Scott, Nirmala Pudota, Tim Weninger, Edward Bowen, and Sanmitra Bhattacharya. 2025. MedCodER: A generative AI assistant for medical coding. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 449–459, Albuquerque, New Mexico. Association for Computational Linguistics. Aniket Bhattacharyya, Anurag Tripathi, Ujjal Das, Archan Karmakar, Amit Pathak, and Maneesh Gupta. 2025. Information extraction from visually rich documents using LLM-based organization of documents into independent textual segments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17241–17256, Vienna, Austria. Association for Computational Linguistics. Marvin Büchel, Tommaso Paladini, Stefano Longari, Michele Carminati, Stefano Zanero, Hodaya Binyamini, Gal Engelberg, Dan Klein, Giancarlo Guizzardi, Marco Caselli, Andrea Continella, Maarten van Steen, Andreas Peter, and Thijs van Ede. 2025. Sok: automated ttp extraction from cti reports are we there yet? In Proceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA. USENIX Association. Yutong Cheng, Osama Bajaber, Saimon Amanuel Tsegai, Dawn Song, and Peng Gao. 2025. Ctinexus: Automatic cyber threat intelligence knowledge graph construction using large language models. In 2025 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3563–3578, Bangkok, Thailand. Association for Computational Linguistics. Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR. Jie Huang, Xinyun Chen, Linan Jiao, and Min-Yen Kan. 2024a. Large language models cannot self-correct
reasoning yet. In The Twelfth International Conference on Learning Representations (ICLR).
and philosophy. Technical Report MP180360R1, The MITRE Corporation.
Yi-Ting Huang, R. Vaitheeshwari, Meng-Chang Chen, Ying-Dar Lin, Ren-Hung Hwang, Po-Ching Lin, Yuan-Cheng Lai, Eric Hsiao-Kuang Wu, ChungHsuan Chen, Zi-Jie Liao, and Chung-Kuan Chen. 2024b. Mitretrieval: Retrieving mitre techniques from unstructured threat reports by fusion of deep learning and ontology. IEEE Transactions on Network and Service Management, 21(4):4871–4887.
Anushka Swarup, Tianyu Pan, Ronald Wilson, Avanti Bhandarkar, and Damon Woodard. 2025. LLM4RE: A data-centric feasibility study for relation extraction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6670– 6691, Abu Dhabi, UAE. Association for Computational Linguistics.
Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017. Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. In Proceedings of the 33rd Annual Computer Security Applications Conference, ACSAC ’17, page 103–115, New York, NY, USA. Association for Computing Machinery. Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022. Attackg: Constructing technique knowledge graph from cyber threat intelligence reports. In Computer Security – ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I, page 589–609, Berlin, Heidelberg. Springer-Verlag. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Andreas Geert Motzfeldt, Joakim Edin, Casper L. Christensen, Christian Hardmeier, Lars Maaløe, and Anna Rogers. 2025. Code like humans: A multi-agent solution for medical coding. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22612–22627, Suzhou, China. Association for Computational Linguistics. James Ross and Jackie Lasky. 2023. Our tram large language model automates ttp identification in cti reports. https://medium.com/mitre-engenui ty/our-tram-large-language-model-automat es-ttp-identification-in-cti-reports-5 bc0a30d4567. Medium article; Center for ThreatInformed Defense / MITRE Engenuity. Kiavash Satvat, Rigel Gjomemo, and V.N. Venkatakrishnan. 2021. Extractor: Extracting attack behavior from threat reports. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 598–615. Blake E. Strom, Andy Applebaum, Doug P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. 2020. MITRE ATT&CK: Design
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724. Yongheng Zhang, Tingwen Du, Yunshan Ma, Xiang Wang, Yi Xie, Guozheng Yang, Yuliang Lu, and EeChien Chang. 2025. Attackg+: Boosting attack graph construction with large language models. Comput. Secur., 150(C).
A
Prompt Templates
This appendix documents the complete prompt templates used in TTP RINT and its baselines. All templates use Jinja2 syntax; variables enclosed in {{ }} are substituted at runtime. We organize prompts by pipeline stage (§A.3–§A.5) and include baseline prompts (§A.1) for reproducibility. Throughout this appendix, system prompts are shown in blue, user prompts in green.
A.1
Baseline: One-Shot GPT-4o System Prompt — One-Shot GPT-4o
You are a cybersecurity analyst extracting MITRE ATT&CK techniques from a CTI report. Rules: - Extract every ATT&CK technique that is explicitly stated or strongly supported by the report content. - You must strictly base your judgment on the report. - Output only ATT&CK technique IDs in parent-technique format: TXXXX, no child technique like TXXXX.XXX. - You should select TTP IDs that are valid MITRE ATT&CK technique IDs - Deduplicate technique IDs. - If no supported techniques are present, return an empty list. Return strict JSON with this schema: { "ttp_ids": ["TXXXX", "TYYYY"] }
User Prompt — One-Shot GPT-4o Extract all MITRE ATT&CK technique IDs from the following CTI report. Return only a JSON object that matches the required schema. CTI report: {{ report_text }}
A.2
"behavior": "short behavior summary", "evidence": "brief supporting evidence from the report", "mapped_ttp": "TXXXX"
}
} ], "ttp_ids": ["TXXXX", "TYYYY"]
User Prompt — CoT GPT-4o Read the CTI report below and reason step by step before producing the final ATT&CK technique list. Requirements: - Only include techniques supported by the report. - Normalize all outputs to parent-technique IDs in the form TXXXX. - Return only a JSON object matching the required schema. CTI report: {{ report_text }}
A.3
Stage ➀: Behavior Extraction System Prompt — Behavior Extraction
You are a cybersecurity expert. Your task is to accurately extract attack behaviors from CTI reports. Requirements: 1. Each extracted attack behavior must be a very brief sentence, closely matching the original wording 2. Reuse technical terms and verb phrases from the original text as much as possible 3. If the original text contains specific strings (domains , hashes, paths, command lines), prioritize preserving them in behavior_text 4. Each behavior corresponds to one or more sentences in the original text 5. You must output strict JSON format, no explanatory text allowed, no extra fields allowed Output format (strict JSON, no other text): { "behaviors": [ { "behavior_text": "A brief sentence describing the attack behavior" } ] }
Baseline: CoT GPT-4o User Prompt — Behavior Extraction
System Prompt — CoT GPT-4o You are a cybersecurity analyst extracting MITRE ATT&CK techniques from a CTI report.
Please extract all attack behaviors from the following CTI report:
Use a careful step-by-step reasoning process: 1. Identify concrete attacker behaviors described in the report. 2. Map each supported behavior to the most appropriate ATT &CK technique. 3. Check whether there are any omitted TTPs that can be strongly supported by the report. 4. Remove unsupported mappings and deduplicate technique IDs.
{{ raw_text }}
Rules: - Extract every ATT&CK technique that is explicitly stated or strongly supported by the report content. - You must strictly base your judgment on the report. - Output only ATT&CK technique IDs in parent-technique format: TXXXX, no child technique like TXXXX.XXX. - You should select TTP IDs that are valid MITRE ATT&CK technique IDs - Deduplicate technique IDs. - If no supported techniques are present, return an empty list. Return strict JSON with this schema: { "reasoning_steps": [ {
Please output strict JSON format according to the requirements, do not add any explanatory text.
A.4
Stage ➂: TTP Proposal System Prompt — TTP Proposal
You are a cybersecurity expert. Your task is to recommend the most relevant TTPs (Techniques only, NOT subtechniques) from the MITRE ATT&CK framework based on the given attack behavior and evidence sentences. Requirements: 1. You must output strict JSON format, no explanatory text allowed 2. You must output exactly N different TTP candidates (N ={{ n }}) 3. Each candidate must contain ttp_id (format like "T1059 ", NOT "T1059.003") and propose_confidence (a float between 0-1) 4. IMPORTANT: You must ONLY use parent techniques (e.g., " T1059"), NOT sub-techniques (e.g., "T1059.003"). Subtechniques are NOT allowed.
5. You should select TTP IDs that are valid MITRE ATT&CK technique IDs (parent techniques only) Output format (strict JSON, no other text): { "ttp_candidates": [ { "ttp_id": "T1059", "propose_confidence": 0.85 }, { "ttp_id": "T1071", "propose_confidence": 0.75 } ] }
User Prompt — TTP Proposal Recommend {{ n }} MITRE ATT&CK TTP IDs. Behavior: {{ behavior_text }} Evidence: {{ evidence_sentences }} Output JSON (only parent techniques, NOT sub-techniques): { "ttp_candidates": [ {"ttp_id": "T1059", "propose_confidence": 0.85}, {"ttp_id": "T1071", "propose_confidence": 0.75} ] }
A.5
Stage ➃: TTP Verification System Prompt — TTP Verification
You are a cybersecurity expert. Your task is to determine whether each MITRE ATT&CK TTP is present/supported in the evidence based on the given attack behavior and evidence sentences. Important constraints: 1. You must strictly base your judgment on the evidence sentences (best_span_sentences) and TTP definition, and you are not allowed to make inferences based on common knowledge 2. If the key points described in the TTP's definition are not clearly reflected in the evidence sentences, the confidence must be low (close to 0) 3. You must output strict JSON format, no explanatory text allowed 4. Each TTP's presence_confidence must be a float between [0,1] 5. IMPORTANT: Some of the given TTPs may be misleading or not actually present in the evidence. You must carefully and accurately judge whether each TTP is truly supported by the evidence sentences, avoiding false positives. Confidence calibration rules (required): - 0.00-0.09: No supporting evidence, or evidence contradicts the TTP definition. - 0.10-0.29: Very weak support (only vague/indirect lexical overlap, missing core TTP behavior). - 0.30-0.49: Partial support (some relevant action is present, but key required elements are missing). - 0.50-0.69: Moderate support (core behavior is plausible in evidence, but important ambiguity remains). - 0.70-0.89: Strong support (The behavior described in the evidence sentence is consistent with the TTP definition, but the evidence may lack a small number of characteristics present in the definition). - 0.90-1.00: Very strong support (The behavior described in the evidence sentence is exactly the same as the TTP definition, with little differences). Scoring discipline: - Base confidence on evidence quality and completeness, not on prior likelihood. - Prefer conservative scoring when evidence is ambiguous. - Use two-decimal precision (e.g., 0.73). Output format (strict JSON, no other text): {
}
"ttp_verifications": [ { "ttp_id": "T1059", "presence_confidence": 0.75 } ]
User Prompt — TTP Verification Based on the following attack behavior and evidence sentences, determine whether each TTP is supported in the evidence: Attack Behavior: {{ behavior_text }} Evidence sentences (you must strictly base your judgment on these sentences): {{ evidence_sentences }} TTPs to verify: {% for ttp_info in ttp_info_list %} {{ loop.index }}. TTP ID: {{ ttp_info.ttp_id }} definition: {{ ttp_info.contents_preview }} {% endfor %} Please output presence_confidence for each TTP, indicating the degree to which the TTP is supported in the evidence sentences.
B
Dataset Annotation Protocol
This appendix provides the full annotation methodology for both evaluation resources introduced in Section 4: the cleaned TRAM benchmark (TRAMClean) and the new TTP RINT-Bench dataset. Section B.1 describes the systematic cleaning protocol applied to the original TRAM annotations. Section B.2 describes the end-to-end protocol used to construct TTP RINT-Bench from scratch. Section B.3 reports inter-annotator agreement statistics for both datasets. B.1
TRAM-Clean: Annotation Cleaning Protocol
Background and motivation. The TRAM dataset (Ross and Lasky, 2023) consists of 150 CTI reports with sentence-level annotations covering the 50 most frequently observed MITRE ATT&CK techniques. Its annotations were originally produced to fine-tune a BERT-based NER model and were not designed as a gold-standard evaluation benchmark. Our preliminary review identified two systematic classes of annotation error that materially distort evaluation: • False positives (over-annotation): technique labels assigned to sentences that describe generic system activity, reconnaissance context, or background narrative rather than a concrete attacker action matching the labeled technique’s definition. These inflate measured precision and artificially penalize conservative models.
• False negatives (under-annotation): sentences that clearly describe a MITRE ATT&CK technique but carry no annotation, typically because the technique is expressed through paraphrase, implicit context, or toolspecific terminology rather than the keyword patterns that dominated the original annotation pass. These depress measured recall and disadvantage models with strong contextual understanding. Annotator selection and training. All cleaning was performed by 3 annotators with verifiable backgrounds in both cybersecurity operations and the MITRE ATT&CK framework (threat intelligence analysts with operational SOC experience and familiarity with the MITRE ATT&CK knowledge base). Before the main annotation, annotators completed a calibration phase on a held-out set of 10 reports not drawn from TRAM, during which disagreements were discussed and resolved to establish shared interpretation norms. Annotators were explicitly instructed to use the official MITRE ATT&CK technique descriptions as the sole reference for deciding whether a technique label is warranted; colloquial associations or keyword matches were not considered sufficient justification. Blind protocol. To prevent any system’s predictions from influencing ground-truth corrections, annotators had no access to the output of any TTP extraction system—including TTP RINT—during the cleaning process. The only reference materials permitted were the original CTI report and the ATT&CK knowledge base.
tation errors or insufficient technical content were flagged for exclusion. 2. Adjudication. All report-level exclusion decisions, all F LAG decisions, all R EMOVE decisions where annotators disagreed, and all proposed A DD cases were brought to a joint adjudication session. Each contested case was resolved by majority vote among the three annotators, with the requirement that the deciding vote be accompanied by a citation to the relevant passage in the official MITRE ATT&CK technique description. 3. Consistency audit. A final pass checked that all annotations for the same technique were applied consistently across the retained reports (e.g., if a particular phrasing was accepted in one report, equivalent phrasings in other reports were audited for corresponding labels). Scope and outcome. The quality audit was applied to a subset of the original TRAM dataset. Due to the labor-intensive nature of manual annotation review, we sampled 21 reports from the full 150report corpus for cleaning, prioritizing reports that span diverse threat actors, attack campaigns, and technique distributions to ensure broad coverage. After cleaning, these 21 reports yield 2,066 sentences with 665 technique labels across 37 unique techniques, constituting the TRAM-Clean evaluation set. Table 3 summarizes the composition of TRAM-Clean.
Procedure. The quality audit proceeded in three passes. 1. Independent review. Each annotator independently reviewed every report in the original TRAM dataset, assessing both reportlevel quality (sufficient technical depth, coherent narrative, absence of systematic annotation artifacts) and label-level correctness. For each sentence–technique pair, the annotator recorded one of three decisions: K EEP (the annotation is correct), R EMOVE (false positive), or F LAG (uncertain, requires discussion). Annotators also recorded any unannotated sentences they believed should carry a label (A DD). Reports with pervasive anno-
Statistic
Value
Sentences in TRAM-Clean Technique labels in TRAM-Clean Unique techniques Avg. sentences per report Avg. labels per sentence
2,066 665 37 98.4 1.23
Table 3: Composition of TRAM-Clean after quality audit of the original TRAM dataset.
B.2
TTP RINT-Bench: New Dataset Annotation Protocol
Source selection. TTP RINT-Bench was constructed to complement TRAM-Clean by providing evaluation data with higher diversity and no legacy annotation noise. We collected 150 CTI reports published between 2022 and 2025 from 12 sources, including vendor threat intelligence blogs (Palo Alto Networks Unit 42, CrowdStrike,
Dataset TRAM-Clean (contested cases) TTP RINT-Bench (pre-adjudication)
Cohen’s κ
Macro-F1
Scope
0.74 0.76
78.3 81.2
3 pairs 3 pairs
Table 4: Inter-annotator agreement for TRAM-Clean and TTP RINT-Bench. κ ≥ 0.60 is conventionally interpreted as substantial agreement.
Bitdefender Labs, Bitdefender Business Insights, ESET WeLiveSecurity, Zscaler ThreatLabz, Forcepoint X-Labs, LevelBlue SpiderLabs), infrastructure provider security blogs (Cloudflare), and independent security journalism outlets (Krebs on Security, The Hacker News, Security.com Threat Intelligence). Source selection criteria were: (1) the report describes a concrete attack campaign or adversary behavior at sufficient technical depth for MITRE ATT&CK labeling; (2) the report has not appeared in any existing TTP extraction benchmark; and (3) the source covers a range of adversary sophistication levels, targeted sectors, and geographic contexts to avoid systematic bias. Reports shorter than 15 sentences or longer than 300 sentences were excluded to control for lengthrelated variability in evaluation metrics. Preprocessing. Each report was segmented into sentences using a rule-based sentence splitter calibrated on cybersecurity text (handling abbreviations such as “e.g.”, “Fig.”, and version strings that trigger false sentence boundaries in generic tools). Code blocks, tables, and structured IOC lists were preserved as single units and treated as individual “sentences” for annotation purposes, since they frequently encode technique-relevant evidence. Document metadata (title, publication date, author) was stripped before annotation to prevent anchoring bias. Annotation unit. Annotators labeled at the sentence level: for each sentence, they assigned zero or more parent-level MITRE ATT&CK technique IDs (format: TXXXX). Sub-techniques (TXXXX.XXX) were not used; if a sub-technique was the most precise match, annotators recorded the corresponding parent technique. This design is consistent with TRAM and enables fair comparison across datasets. Annotator pool and qualification. 3 annotators participated in TTP RINT-Bench annotation. Each annotator satisfied the following criteria: • Holds a cybersecurity qualification (GCTI, OSCP, or equivalent industry experience in
threat intelligence analysis) or has authored peer-reviewed research in threat intelligence or intrusion analysis. • Has demonstrated familiarity with the MITRE ATT&CK framework assessed by a qualification test requiring correct labeling of 30 curated example sentences at ≥80% accuracy. Annotators were compensated at standard institutional rates and were not involved in the development of TTP RINT. Annotation procedure. The annotation process followed a four-phase workflow: 1. Guideline review. Before annotation began, all annotators received a 20-page annotation guideline document covering: (a) the scope and structure of the MITRE ATT&CK framework; (b) decision criteria for assigning vs. withholding technique labels; (c) worked examples of boundary cases (implicit descriptions, generic system activity, compound behaviors); and (d) instructions for handling uncertain cases. 2. Pilot round. Each annotator independently labeled a common set of 5 pilot reports. Pilot annotations were compared to a consensus set prepared by the authors, and annotators with agreement below 0.60 Cohen’s κ on the pilot received targeted feedback before proceeding to the main annotation. 3. Double-annotation. Each report in the main annotation batch was independently labeled by exactly two annotators. Annotators worked without visibility into each other’s labels. 4. Conflict resolution. Sentence–technique pairs where the two annotators disagreed were resolved by a third annotator acting as adjudicator. The adjudicator had access to both annotators’ labels but not their rationales, and was required to produce a decision with an explicit citation to the relevant ATT&CK technique description. If the adjudicator disagreed
Backbone GPT-4o GPT-4o-mini Claude Sonnet 4.6 Gemini 2.5 Flash Qwen 3 235B DeepSeek v3.2
API Identifier gpt-4o gpt-4o-mini claude-sonnet-4-6 gemini-2.5-flash qwen3-235b-a22b deepseek-chat (v3)
Architecture Dense Dense Dense Dense MoE MoE
Parameters Context Undisclosed 128K Undisclosed 128K Undisclosed 200K Undisclosed 1M 235B (22B active) 131K 671B (37B active) 128K
Table 5: LLM backbone specifications. “Context” denotes the maximum context window in tokens. MoE = Mixture-of-Experts; active parameters are those used per forward pass.
with both primary annotators, all three labels were reviewed jointly and resolved by majority vote. Annotation interface. Annotation was conducted in a custom web interface built on top of Label Studio. The interface displayed the full report alongside the ATT&CK technique browser, allowing annotators to look up technique definitions without leaving the annotation environment. Each sentence was presented in its document context (three preceding and three following sentences were visible but not annotatable) to assist with coreference resolution and implicit behavior identification. Quality control. To detect annotation drift over time, each annotator re-labeled a random sample of 5% of their previously completed sentences midway through the annotation campaign (cross-over check). Labels that differed from the original by more than 10% were flagged for review. In addition, 15 “honeypot” sentences with known ground truth (verified by three senior annotators) were inserted into each annotator’s queue; any annotator whose honeypot accuracy fell below 85% had their batch re-reviewed. B.3
Inter-Annotator Agreement
We report inter-annotator agreement (IAA) using two complementary metrics. Cohen’s κ measures pairwise agreement on the binary presence/absence decision for each technique at the sentence level, accounting for chance agreement. Macro-averaged F1 treats one annotator’s labels as predictions and the other’s as ground truth, averaged over all technique classes with at least one occurrence; this is more informative than κ for highly imbalanced label distributions typical of CTI annotation. Table 4 reports IAA for both datasets. For TRAM-Clean, we report the pairwise κ between annotators computed over the set of contested cases (i.e., cases that reached the adjudication stage). For TTP RINT-Bench, we report the average pairwise
κ over all double-annotated pairs prior to adjudication. Both datasets achieve substantial inter-annotator agreement (κ > 0.70), confirming that the annotation task is well-defined and reproducible under our protocol. The slightly higher agreement on TTP RINT-Bench (κ = 0.76) compared to TRAM-Clean contested cases (κ = 0.74) is expected: TRAM-Clean agreement is measured only over cases that reached adjudication (i.e., the hardest subset), while TTP RINT-Bench reports pre-adjudication agreement across all doubleannotated pairs. The macro-F1 agreement (78.3% and 81.2%) provides additional assurance that consistency holds across rare techniques, not just frequent ones—an important property given that TTP RINT-Bench covers 66 techniques with fewer than 5 occurrences. We also analyzed disagreement patterns to identify systematically difficult annotation cases. The most common sources of disagreement were: • Compound behaviors: sentences describing two distinct attack actions (e.g., credential access followed by lateral movement) where annotators differed on which technique was primary. • Implicit techniques: sentences that strongly imply a technique through contextual inference (e.g., describing the effect of a persistence mechanism without naming the mechanism itself), where annotators differed on whether implication was sufficient for labeling. • Tactic–technique boundary: sentences that clearly indicate a tactic (e.g., exfiltration) but do not specify the technique used, leading to disagreement between annotators who labeled the most likely technique and those who withheld the label for lack of explicit support.
These patterns motivated the annotation guideline’s explicit treatment of each case and informed the calibration examples used in annotator training.
C
LLM Backbone Configuration
Table 5 summarizes the LLM backbones used in the multi-backbone analysis (§4.4). For each model, we report the API identifier or model version used in our experiments, the architecture type, publicly available parameter counts, and context window size. Shared hyperparameters. All pipeline hyperparameters are held constant across backbones to isolate the effect of the underlying model. We set the span localization window ℓmax = 5 sentences, the number of TTP proposals per behavior k = 5, and the corpus frequency cutoff for token down-weighting at the top-p = 0.01 fraction. The verification threshold is τ = 0.7 throughout. For LLM inference, we use each provider’s default API parameters (temperature, top-p, etc.) without modification. Hyperparameter
Value
Temperature Verification threshold τ Max span window ℓmax (sentences) TTP proposals per behavior k Top-p frequency cutoff
0.0 0.7 5 5 0.01
Table 6: Shared pipeline hyperparameters used across all backbone experiments.
All closed-source models were accessed via their respective commercial APIs. Open-source models (Qwen 3 235B and DeepSeek v3.2) were accessed via third-party API providers; as noted in Section 4.4, these models can alternatively be deployed locally for organizations with data sovereignty requirements.