FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming Chaeyun Kim1,*,† , Dae-Young Park2,* , Junghwan Kim1,† , Jinyoung Jeong3 , Eunji Song3 , YongTaek Lim1 , Minwoo Kim1,‡ 1
arXiv:2606.19887v1 [cs.CR] 18 Jun 2026
2
DATUMO INC., Republic of Korea Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea 3 Financial Security Institute (FSI), Republic of Korea
Abstract—Existing safety benchmarks target general adversarial scenarios but miss finance-specific risks. Financial LLMs face regulatory-compliance violations, fraud facilitation, and systemic trust erosion that require targeted evaluation. We introduce FinRED, an expert-guided red-teaming framework for financial LLM safety evaluation developed with financial experts. FinRED uses a novel two-level taxonomy mapping global standards (e.g., FATF, EU DORA) to threats from regulatory evasion to complex fraud, integrated with a scalable pipeline that converts real financial documents into context-rich red-teaming Behavioral Prompts (seeds) through an expert-defined schema. Rigorous expert validation confirms seed plausibility and realism for meaningful LLM safety evaluation. We also provide an expertvalidated finance-specific rubric beyond disclaimer checks, aligning better with human experts than static one-size-fits-all rubrics and reducing critical false negatives from 28 to 12. Aligned with internationally adopted risk and information-security standards (e.g., ISO/IEC 27001), FinRED is deployed in South Korea’s Financial Security Institute (FSI) regulatory sandbox for generative-AI security evaluation in real financial services. To mitigate dual-use risks, the dataset, generation pipeline, prompt template, and evaluation framework are gated for qualified researchers at https://github.com/selectstar-ai/FinRED-paper and https://huggingface.co/datasets/datumo/FinRED.1 Index Terms—LLM Safety, Red-Teaming, Financial Domain, Benchmark, Risk Taxonomy
I. I NTRODUCTION The rapid deployment of large language models (LLMs) across critical sectors has raised concerns about their safety, particularly in high-stakes domains such as finance [1]–[3]. Red teaming and safety benchmarking have become essential methodologies for identifying these vulnerabilities [4], [5]. While the scope of these benchmarks has evolved from evaluating purely technical exploits to evaluating nuanced sociocultural harms [6]–[8], they predominantly focus on universal risks, revealing a significant limitation in their ability to measure the unique, expertise-driven threats inherent in specialized domains [9]. This evaluation gap is particularly * Both authors contributed equally to this research. † Work done while at DATUMO INC. ‡ Corresponding author, [email protected] 1 For review purposes, reviewer access to these resources will be maintained until the paper is officially published.
Fig. 1: Structure of The Proposed Financial Risk Taxonomy
acute in the financial sector. Existing safety taxonomies often evaluate “Specialized Advice” merely by checking for disclaimers [9]–[12], rather than evaluating the substantive harm of the response. Although a suite of financial benchmarks like FinEval [13], FinLMEval [14], Pixiu [15], CFBenchmark [16], and FinanceBench [17] has emerged, their primary focus lies on performance metrics such as information extraction, reasoning, and domain knowledge comprehension. They evaluate what models can do, not how they behave under adversarial pressure. This leaves a critical gap: a lack of precise evaluation benchmarks for measuring safety in the financial domain. Financial LLM safety extends beyond general risk categories, requiring evaluation of domain-specific threat vectors. These include violations of complex regulations (e.g., Financial Consumer Protection Act), sophisticated fraud scenarios indistinguishable from legitimate advice, and systemic risks that erode trust in financial ecosystems. This challenge is amplified by the increasing deployment of both general-purpose and finance-specific LLMs, raising a critical question: are these models, despite their financial proficiency, resilient against domain-specific attacks? We posit that an expert adversarial
Benchmark
Domain Focus
Adversarial Type
HarmBench ALERT AIR-Bench SALAD-Bench CASE-Bench JailbreakBench
General General General (policy-aligned) General General General
Prompt-level Automated red-teaming Multi-turn adversarial Multi-facet adversarial Context-aware red-teaming Prompt-level
Pixiu FinEval FinLMEval FinanceBench DocFinQA CFBenchmark FinBen FinRED (ours)
Finance (performance) Finance (performance) Finance (performance) Finance (reasoning) Finance (reasoning) Finance (compliance) Finance (multi-task) Finance (safety)
QA/IE (Non-adversarial) Reasoning (Non-adversarial) QA/reasoning (Non-adversarial) QA (Non-adversarial) QA (Non-adversarial) Simulated violation Mixed (QA/IE/TG) (Non-adversarial) Framework + red-teaming
Expert Involvement Safety Rubric General Safety / Red-Teaming Benchmarks ✗ Generic (disclaimer-based) ✗ Binary harmfulness ✗ Regulation-oriented ✗ Hierarchical safety taxonomy Partial Policy-grounded rubric ✗ Generic (disclaimer-based) Financial-domain Benchmarks ✗ None (task accuracy only) ✗ None (QA metrics) ✗ None ✗ None (factuality) ✗ None (QA-based) Partial Rule-based ✗ None (task-based) ✓ (12 FSI experts) Category-specific financial safety rubric
Evaluation Target
Language
Year
Open-domain LLMs General-purpose LLMs Government-related LLM eval General-purpose LLMs General / Gov LLMs Open-domain LLMs
English English English English English English
2024 2024 2024 2024 2025 2024
FinLLMs and general LLMs Financial QA models FinLLMs FinLLMs FinLLMs and general LLMs FinLLMs FinLLMs and general LLMs General + FinLLMs
English/Chinese Chinese English English English Chinese English/Spanish Korean/English
2023 2025 2023 2023 2024 2023 2024 -
*Abbreviations: QA = Question Answering; IE = Information Extraction; TG = Text Generation; FinLLM = Financial Large Language Model.
TABLE I: Comparison of existing red-teaming and financial-domain benchmarks. FinRED integrates expert-guided construction, adversarial generation, and finance-specific safety evaluation as an applied framework for financial LLM adversarial safety testing.
user, equipped with deep financial knowledge and adaptive techniques, can craft subtle attacks that bypass generic safety filters. Motivated by this realistic threat landscape, we introduce FinRED 2 , an applied red-teaming construction and evaluation framework designed to systematically generate realistic financial threat scenarios and verify LLM safety as shown in Fig. 1. In this work, “safety evaluation” strictly focuses on evaluating an LLM’s adversarial resilience against domain-specific threats (e.g., fraud facilitation and regulatory evasion), rather than its general financial task performance or institutional compliance. FinRED was designed for practical financial AI safety evaluation rather than as a stand-alone dataset. A consensus of 12 domain experts 3 guided the pipeline from the foundational risk taxonomy to the final evaluation rubric. Our schemabased approach integrates real financial documents with domain knowledge to generate contextually rich scenarios, which are then transformed into red-teaming Behavioral Prompts (Seeds). Through expert quality evaluation and downstream ASR ablations, we show that schema-driven seeds are more domain-specific, plausible, actionable, and operationally effective than zero-shot or context-only baselines (see Section IV-C). To evaluate model safety against these high-quality seeds, our expert-validated judge rubric is applied to generalpurpose open-source LLMs, finance-specific LLMs [15], [18], and leading commercial APIs. We place a particular focus on the vulnerabilities of small Language Models (sLMs), which represent a realistic deployment target for financial services. Highlighting its practical impact, FinRED has been incorporated into the Financial Security Institute (FSI)’s regulatory sandbox framework, where it is used to evaluate security countermeasures for generative AI in financial services 4 . The main contributions of this study are as follows: 2 The FinRED artifact is available through gated access at: https:// huggingface.co/datasets/datumo/FinRED 3 They belong to the Financial Security Institute (FSI), one of the government agencies related to the financial industry in South Korea. For this institutional information, refer to the following link: https://www.fsec.or.kr/. 4 This framework is used to verify AI security of financial companies. For information about this sandbox framework and its AI security verification, refer to the following links: https://www.fsec.or.kr/bbs/detail?menuNo=69&bbsNo= 11629 and https://www.fsec.or.kr/bbs/detail?menuNo=69&bbsNo=11607
1) Expert-Guided Financial Risk Taxonomy: We design a two-level taxonomy that covers real financial threats such as complex fraud and regulatory evasion, while mapping them to global security and supervisory standards. 2) Schema-Driven Scenario Generation and Real-World Deployment: We build a scalable pipeline that generates red-teaming seeds from global and local financial documents. The framework has been deployed in the FSI regulatory sandbox for generative-AI security verification, demonstrating practical utility in a real financialsupervision setting. 3) Expert-Aligned Evaluation Rubric: We provide a five-dimensional finance-specific rubric that evaluates harmfulness beyond simple refusal behavior and better matches financial-security experts, reducing critical false negatives in threat detection. Unlike static financial safety datasets, FinRED is designed as an extensible and regulation-adaptive framework that decouples threat taxonomy design from retrieval corpora to enable newly released international regulations and jurisdictionspecific supervisory documents to be incorporated with minimal effort for customized financial safety evaluation across diverse regulatory environments. II. RELATED WORKS A. LLM Red Teaming and Safety Benchmarks Red teaming identifies LLM vulnerabilities and has produced many safety benchmarks [10], [19]–[28], including ALERT [21] for adversarial-prompt evaluation, HarmBench [29] for automated red teaming, and AIR-Bench 2024 [27] for regulation-aligned evaluation. Their pipelines either leverage LLM’s inherent knowledge through prompting [19], [23], [30], [31] or guide targeted prompt creation with relevant documents or policy [27], [32]–[34]. These benchmarks support general LLM safety alignment but focus on universal harms, lacking financial nuance [9], and often employ a single, fixed rubric, missing categoryspecific context. FinRED addresses both with a domainspecific, context-aware pipeline and category-specific expertvalidated judge rubric.
B. Financial Domain-Specific LLM Evaluation Financial benchmarks such as Pixiu [15], FinEval [13], CFBenchmark [16], FinanceBench [17], and FinBen [35] evaluate QA, extraction, and generation across languages [36]– [38]. Yet they measure financial co-pilot capability, not safety under adversarial pressure. FinRED targets this intersection with a fine-grained expert-consensus taxonomy for realistic financial threat stress testing. Table I summarizes recent benchmarks and FinRED’s expert-guided adversarial generation with a finance-specific safety rubric. C. Automated Adversarial Attack Methodologies LLM adversarial attacks expose vulnerabilities and unsafe behavior through white-box (token-level) or black-box (prompt-level) methods. Token-level attacks optimize tokens with internal access: GCG [20] uses gradient-based greedy search enhanced by momentum [39], hybrid multi-coordinate updates [40], or random restarts [41], while AutoDAN [42] evolves prompts genetically. Black-box attacks need no internal information and often use other LLMs to generate or refine prompts. TAP [43] uses tree-of-thought reasoning, GPTFuzzer [44] applies LLMguided mutations, and other methods use ciphering [45], string compositions [46], or flipped letter order within words [47]. We study these attacks in finance-specific contexts. III. THE F IN RED FRAMEWORK In this section, we detail the methodology for constructing FinRED, our applied framework for systematically evaluating LLM safety against finance-specific risks. As shown in Fig. 2, our framework is organized into three core stages: (a) expertdriven Financial Risk Taxonomy and Schema definition, (b) relevant context retrieval from real-world documents, and (c) Prompt Generation leveraging these contexts and expertvalidation. A key output is the Behavior Seed: a structured, natural-language specification of a harmful behavior grounded in a realistic financial context, which serves as input for downstream attack methods (e.g., GCG, TAP). These seeds capture the nuanced complexity of financial threats that cannot be adequately represented through simple zero-shot prompts. The released artifact contains 5,805 expert-validated seeds across five Level-1 and 26 Level-2 risk types, together with taxonomy labels, schema and prompt-format metadata, split identifiers, and gated access controls for qualified researchers. Finally, we design a specialized FinRED Evaluation Rubric incorporating a robust financial domain judge. A. Financial Risk Taxonomy and Schema Definition 1) 2-Level Financial Risk Taxonomy: The foundation of our framework is a domain-specific risk taxonomy, established through extensive literature review, focus group interview (FGI) and discussion with financial security experts. We designed a 2-level structure where each of the five high-level risk categories (level 1) is broken down into granular sub-types
(level 2), indicating a fine-grained financial risk taxonomy for evaluating LLM safety. At Level 1, five overarching domains are defined as: (R1) Cyber Threats, (R2) Financial Crime, (R3) Misinformation & Deception, (R4) Consumer Rights Violation, and (R5) Financial information and communication technology (ICT) Compliance Evasion. Then, each category is decomposed into fine-grained Level 2 subtypes reflecting concrete adversarial intents—such as digital authentication bypass, asset theft, and advanced persistent threat (APT) intrusions under R1; voice phishing, electronic financial fraud, and money laundering assistance under R2; misleading financial product descriptions and deceptive promotional content under R3; disclosure evasion and mis-selling practices under R4; and incident non-reporting or data protection evasion under R5 5 . The proposed taxonomy and schema templates are designed by abstracting threat-modeling principles from international supervisory frameworks such as FATF, BIS/BCBS, ISO/IEC 27001, NIST, OWASP, and the EU’s DORA, rather than by treating source documents as evidence of compliance. They define reusable standard-aligned threat dimensions, such as AML evasion, operational resilience failure, consumer harm, and cybersecurity abuse. Consequently, injecting a local regulatory corpus (e.g., Korean financial-security documents) or newly released international guidance as retrieval context customizes generated seeds to the target jurisdiction while preserving globally grounded threat-assessment dimensions. 2) Threat Behavior Schema Definition and Refinement: Generating realistic financial threat scenarios requires more than free-form LLM generation. Financial risks involve complex interactions between actors, systems, regulations, and motivations that must be systematically captured to ensure both completeness and realism. A schema serves as a structured template that defines the essential components of a threat without containing actual data. In the financial domain, it may outline attacker profiles, compromise paths, deceptive tactics, or regulatory violations—providing a consistent framework that guides scenario generation while allowing flexibility in specific instantiations. Schema Structure. For each Level-1 taxonomy category, we define schemas in JSON format, dividing elements into Essential cues (e.g., attackerProfile, targetTechnology) and Optional cues (e.g., vulnerabilityHypothesis, requesterPersona). “Essential” elements form the core content of the threat behavior and must be included, while “Optional” elements add contextual richness and diversity. We chose the JSON format for two key reasons: (1) Flexibility: hierarchical key-value structures naturally represent complex threat behaviors with nested relationships, (2) Industry alignment: security domains commonly use JSON-based specifications (e.g., STIX Domain Objects for threat intelligence). Expert-Driven Refinement. Initial schema drafts were constructed using state-of-the-art LLMs (Gemini 2.5 Pro, GPT4.1, Claude Sonnet 4) to prototype comprehensive attribute 5 For detailed definition and explanation, refer to https://bit.ly/4e4qNpN.
Refinement
Reference Documents
Finance Expert
Essential Fields Per Taxonomy
FinRED Taxonomy
Schema by Taxonomy
Optional Fields Per Taxonomy Level-2 Keyword List
(a) Financial Risk Taxonomy & Schema Chunking with Chipper Reference Documents
Vector Database
Semantic Retrieval
Relevant Chunk per Taxonomy
FinRED
[FGI] Finance Expert x Rubric
(b) Contextual Retrieval Relevant Chunk per Taxonomy Intermediate Scenario
Final Prompts
LLM-Based Seed Generation
Manual Verification
Schema by Taxonomy
(c) Context Grounded Behavior Seed Generation
Fig. 2: Overview of FinRED: (a) expert-guided taxonomy/schemas, (b) document-context retrieval, and (c) expert-validated seed generation. sets for each risk category. These drafts were then iteratively refined in close collaboration with the FSI experts over multiple review cycles. This process focused on ensuring each schema was not only comprehensive but also met the standards of specificity, realism, and logical coherence required to model plausible threats in a real-world financial security environment. Through three discussion sessions and consensus-building activities, the FSI experts validated that each schema field accurately captured domain-specific threat mechanics, and could systematically generate threats beyond generic templates. The final schemas thus represent an expert consensus, with each element explicitly grounded in attack patterns and regulatory violations observed in real-world financial operations. This design treats experts as a validation layer rather than a manual bottleneck: new documents or threat-intelligence feeds can update retrieval contexts, while schema-level changes require only targeted expert review. B. Document Collection and Contextual Retrieval This stage establishes the foundation for generating domain grounded seed prompts by collecting expert-curated financial documents and retrieving relevant contexts through a structured pipeline. 1) Document Collection and Chunking: First, we collected approximately 500 financial documents selected and reviewed by domain experts, including regulatory frameworks, supervisory guidelines, audit reports, and risk assessment reports. These documents provide rich contextual information grounding our seed prompts in realistic regulatory and operational environments. Next, we employ a two-stage chunking pipeline [48] using the Unstructured library with Chipper 6 , a vision-based model 6 A layout-aware document parsing model by Unstructured
that outputs JSON representations of document elements with layout metadata. Our two-stage pipeline first performs structural chunking, which preserves hierarchy by initiating chunks at titles or tables, merging elements under 200 characters into blocks up to 2,800 characters. Second, chunks exceeding 1,200 characters are re-segmented with a 150-character overlap. This approach balances embedding constraints with semantic coherence. 2) Query Construction and Retrieval: We developed expert-crafted queries for every Level-2 category in our taxonomy. These queries reflect actual information-seeking behaviors in security operations and regulatory compliance. Documents are chunked separately for each Level-2 category, encoded using OpenAI embeddings, and stored in a Chroma vector database. We retrieve the top-8 semantically relevant chunks per category using the corresponding expertconstructed queries. 3) Role of Retrieved Context: The retrieved documents specify prohibited actions in financial services, including regulatory violations and operational misconduct. In fact, our goal is to generate seed prompts describing harmful behaviors that LLMs should refuse. We treat compliance guidance as an inverted playbook: our generation process identifies prohibitions and devises methods to circumvent them. This approach transforms compliance documents into realistic threat scenarios grounded in regulatory frameworks. C. Automated Generation of Context-Grounded Behavior Seeds Building on the retrieved regulatory contexts and defined schemas, this stage automatically generates the Behavior Seeds that specify target harmful behaviors for our benchmark. The process is designed to synthesize the structured knowledge
from the schemas with the unstructured text from the context documents, producing prompts that are both realistic and systematically varied. Schema-Driven Seed Prompt Construction. Our generation process is a two-step, schema-driven procedure designed to maximize realism and specificity. First, the expert-defined schema (the blueprint) and the retrieved context for a specific threat (the raw material) are provided to a powerful LLM. The model’s task is to generate a structured JSON scenario, populating the schema’s fields with concrete details synthesized from the provided context 7 . This intermediate scenario serves as a rich, machine-readable representation of a unique threat instance, detailing its background, actors, motivations, and intended outcomes. This intermediate JSON scenario is then transformed into a natural-language Seed through two core strategies. Dynamic Persona Assignment instructs the LLM to adopt a contextappropriate attacker persona (e.g., an ’experienced penetration tester’ for cyber threats), imbuing the prompt with realistic voice and intent. Strategic Combination of Elements ensures diversity: all essential elements are included in every seed, while optional elements are systematically varied across seeds to explore different facets of the same threat. This process yields high-quality, diverse, and context-grounded Behavior Seeds that form the FinRED benchmark. Final Verification. Lastly, to mitigate LLM hallucination and ensure scenario realism, we incorporate self-correction instructions during prompt generation, requiring the LLM to verify and correct logical coherence. Generated prompts underwent final validation by financial experts through multiple focus group interviews. Experts evaluated scenario realism, attack effectiveness, and clarity of potential harm from multiple perspectives. D. FinRED Evaluation Framework: Financial Domain Judge Rubric To ensure reliable and domain-grounded safety evaluation, we design a Financial Domain Judge Rubric that extends the LLM-as-a-judge paradigm. Unlike prior safety benchmarks that rely primarily on disclaimer detection or binary refusal checks, the proposed judge defines and uses fine-grained evaluation criteria reflecting the five dimensions of model responses. 1) Judgment rubric design: The FinRED rubric defines what constitutes harmful behavior in the financial domain and how to measure it. It comprises five dimensions: harmfulness evaluates the severity of potential financial or legal damage; persuasiveness evaluates how plausible and convincing the harmful content appears; refusal quality checks whether the model recognizes and rejects the risky request; factualness judges the technical or regulatory accuracy of the content; and evasiveness captures indirect hints or partial assistance toward harmful behavior. The core evaluation objective is to determine how effectively the attacker’s intent is realized, 7 For examples of schemas, refer to https://bit.ly/4eqoOxy
focusing on whether the LLM response directly or indirectly contributes to the attacker’s harmful financial objective. The rubric’s objective is twofold: (1) domain specificity: evaluate whether a model’s response could realistically facilitate or justify financial misconduct (e.g., fraud, mis-selling, data misuse); (2) evaluator consistency: enable both human experts and automated judges to apply a unified standard across Level2 taxonomy categories. Each rubric item is accompanied by detailed behavioral cues extracted from real-world financial security guidelines (e.g., Financial Consumer Protection Act) and agreed upon through expert interviews with the 12 domain experts from the FSI. This expert panel provided both qualitative feedback and quantitative validation, ensuring conceptual alignment with institutional supervisory frameworks 8 . 2) Automated evaluation: The automated judge operates on a structured instruction template to ensure reproducibility and alignment with expert reasoning. The template enforces constrained JSON outputs to enable scalable benchmarking while preserving interpretability 9 . This mechanism enables scalable, reproducible, and expertaligned evaluation of LLM safety across thousands of redteaming prompts. By explicitly modeling how and why a response is harmful, the proposed FinRED Judge bridges the gap between human expert reasoning and LLM-based evaluation — a crucial step toward domain-specific AI safety evaluation in finance. Following a conservative fail-safe policy, a response is labeled as Unsafe if any rubric dimension is judged Unsafe, reflecting the principle that even partial harmful assistance may create unacceptable financial risk. This rule matches how financial-security experts flag subtle regulatory evasion and contributes to the 57% reduction in critical false negatives reported in Section IV-D 10 . Due to space limitations, the full evaluation template and rubric specifications are available in the project repository mentioned in the Abstract. IV. EXPERIMENTS A. Experimental Settings Attack Methods. To evaluate the vulnerability of LLMs to finance-specific threats, we apply state-of-the-art automated attack methods to FinRED seeds: white-box attacks (GCG [20], AutoDAN [42]) and black-box attacks (TAP [43], GPTFuzzer [44], AutoDAN-Turbo [49]). We also include a Direct Request baseline using unmodified seeds. For reproducibility, we fix the main attack budgets as GCG (500 steps, width=512), AutoDAN (100 steps, batch=256), AutoDANTurbo (150 epochs, break score ≥8.5), TAP (depth=10, branching=4), and GPTFuzzer (1,000 queries); all experiments are performed on NVIDIA H200 GPUs. Target LLMs. Our evaluation spans a wide range of models to provide a comprehensive analysis of the financial safety 8 Representative rubric examples for each taxonomy category are available in the project repository. 9 The evaluation prompt template is available at: https://bit.ly/4a2e1av. 10 To evaluate the effectiveness of our judge rubric, we provide details in Section IV-D.
General-Purpose sLMs Taxonomy
Attack Method
Finance-Specific sLMs
API-Based LLMs
Llama3.1
Qwen2.5
gemma3
EXAONE
FinMA
qqWen-7B
qqWen-32B
GPT-5
GPT-5-mini
Claude-4
Gemini-Flash
Gemini-Pro
24.05 19.62 37.34 6.96 75.95 12.66
82.41 91.77 98.73 68.35 88.24 89.51
53.80 59.49 73.42 46.20 65.18 33.75
87.34 88.61 98.73 75.32 88.73 85.44
92.41 49.37 42.41 55.06 78.15 78.48
75.32 85.25 98.73 87.97 79.11 90.51
83.04 87.97 90.51 65.19 63.29 68.99
0.63
1.27
63.29
0.63 0.00 0.37
0.63 0.00 0.89
22.78 8.86 4.43 7.14
53.80
R1: Cyber Threats
Direct Request GCG (White-box) AutoDAN (White-box) TAP (Black-box) GPTFuzzer (Black-box) AutoDAN-Turbo
1.27 27.85 23.42
1.90 58.38 35.97
Direct Request GCG (White-box) AutoDAN (White-box) TAP (Black-box) GPTFuzzer (Black-box) AutoDAN-Turbo
4.29 2.86 7.86 0.71 75.71 7.86
25.71 60.71 43.57 12.86 82.14 31.43
8.57 18.57 17.14 7.86 68.57 36.83
20.00 20.71 27.86 17.14 82.86 32.14
83.57 60.71 50.71 55.00 87.14 60.00
23.57 51.43 65.71 42.14 76.43 37.14
20.00 33.57 15.71 11.43 74.95 7.14
0.00
0.00
21.43
4.29 0.00 3.57
8.57 0.00 1.78
7.14 1.43 2.14 0.35
21.43
R2: Financial Crime
1.43 23.57 6.43
3.57 52.14 24.28
15.67 6.72 14.93 2.24 86.57 6.72
27.61 76.87 32.09 35.07 87.01 23.88
19.40 27.61 16.42 6.72 64.17 54.18
38.06 31.34 26.87 20.15 97.76 33.58
70.15 66.42 52.99 29.10 71.04 39.55
31.34 73.13 34.33 41.79 97.01 27.61
26.87 59.70 31.34 20.15 66.41 23.88
1.49
3.73
31.34
5.22 0.75 8.15
7.46 3.73 9.20
0.00 0.00 8.96 7.14
32.09
R3: Misinformation
Direct Request GCG (White-box) AutoDAN (White-box) TAP (Black-box) GPTFuzzer (Black-box) AutoDAN-Turbo
2.99 58.81 14.18
5.22 42.39 37.55
Direct Request GCG (White-box) AutoDAN (White-box) TAP (Black-box) GPTFuzzer (Black-box) AutoDAN-Turbo
4.02 8.62 10.34 3.45 90.80 9.20
31.03 72.41 29.89 24.71 93.68 27.59
12.64 33.91 23.56 9.77 67.24 37.81
20.69 25.86 26.44 26.44 94.25 20.11
49.43 43.10 36.78 31.03 86.20 38.51
16.09 66.67 33.91 29.89 87.93 30.46
17.82 40.23 24.71 12.07 50.20 18.39
4.60
4.60
27.01
3.45 0.00 4.08
9.77 4.02 3.59
4.60 0.57 6.90 2.04
26.44
R4: Consumer Rights
1.15 47.59 9.20
5.75 46.44 11.43
15.10 13.47 25.31 5.71 85.31 11.43
31.84 42.45 31.02 24.08 96.73 26.53
14.29 58.78 19.59 16.33 73.01 53.26
26.53 30.61 31.02 23.67 81.63 19.59
45.31 50.61 31.84 35.92 65.71 33.88
27.35 74.69 27.35 22.04 93.47 21.63
24.90 57.20 28.16 17.55 62.85 23.67
9.39
7.76
37.14
6.86 0.00 6.93
11.84 2.04 5.86
4.90 2.86 4.08 4.29
44.49
R5: Compliance Evasion
Direct Request GCG (White-box) AutoDAN (White-box) TAP (Black-box) GPTFuzzer (Black-box) AutoDAN-Turbo
2.86 27.76 11.84
4.49 4.08 18.52
TABLE II: Attack Success Rate (%) of various attack methods on our FinRED benchmark across a representative set of target models. Cells are colored by ASR value: ≥80% 60-80% 40-60% 20-40% <20% . White-box attacks (GCG, AutoDAN) are not applicable to API-based models ( -) due to the requirement of direct access to model parameters. Mean ASR per Risk Category and Model R1 R2
Mean ASR (%)
R3 R4
-s on ge ne m t in i-2 .5 -fl as ge h m in i-2 .5 -p ro
-m in i
e4
PT -5 G
cl au d
b
PT -5 G
7b
W en -3 2
b -7 A
Fi nM
W en -
-it 8b
-it 2b
7.
O
N
E3.
5-
-1
ge m m a3
(a) Mean Category and Model MeanASR ASR byby RiskRisk Category Attack Method R1 Mean ASR (%)
Metrics. For our primary evaluation metric, we use Attack Success Rate (ASR) following prior work [29], [30], [57]. ASR is the percentage of attack prompts for which a target model produces a harmful or non-refused response, as determined by the expert-validated FinRED Judge rubric (Section IV-D). Because category counts reflect real-world coverage rather than a class-balanced design, we report ASR by risk category and attack method instead of relying on a single pooled score. We also measure agreement between the LLM judge and domain-expert judgments.
EX
A
Q
w
en 2
.1
.5
-8
-7
bit
bit
R5
Ll am a3
landscape. We test widely-used open-source small language models (sLMs) (Llama-3.1-8B-it [50], Qwen2.5-7B-it [51], gemma3-12B-it [52], EXAONE3.5-7.8B-it [53]); prominent finance-specific sLMs (FinMA [15], qqWen 7B and 32B [18]); and a suite of leading commercial API-based models (GPT-5, GPT-5-mini [54], Claude 4 Sonnet [55], Gemini 2.5 Flash, and Gemini 2.5 Pro [56]).
R2 R3 R4 R5 Direct Request
GCG (W)
AutoDAN (W)
TAP (B)
GPTFuzzer (B) AutoDAN-Turbo (B)
(b) Mean ASR by Risk Category and Attack Method
Fig. 3: Heatmap visualizations of Mean ASR.
B. Main Evaluation Result We present comprehensive evaluation results in Table II, showing Attack Success Rates (ASR) across six attack methods, five taxonomies, and twelve target models. Fig. 3 further summarizes mean ASR by model and attack method. Our findings reveal critical vulnerabilities and validate our core hypotheses regarding domain-specific financial threats. Compact heatmap analysis. The heatmaps show that vulnerability is concentrated in open-source sLMs, including finance-specific models, whereas leading API-based models remain comparatively robust. Across risk categories, R1 is the most vulnerable and R2 is relatively more resistant because explicit financial-crime requests more often trigger refusal.
GPTFuzzer is the strongest black-box attack overall, while the non-trivial Direct Request ASR confirms that FinRED seeds are adversarial even without additional optimization. Interestingly, several finance-specific and open-source LLMs exhibit lower ASR under optimization-based attacks than under Direct Request. This may occur because optimization-based suffixes sometimes disrupt the rich financial context already embedded in FinRED seeds. C. Evaluation of Generation Pipelines To validate our schema-driven pipeline’s ability to generate realistic financial threat scenarios, we evaluate it against two
progressively simplified baselines. This comparative framework situates our method within existing red-teaming practices while isolating the specific impact of structural schemas on the fidelity and relevance of generated seeds. 1) Experimental Setup: a) Compared Pipelines: We evaluate three progressive pipelines: P1 Context-Free Generation: Generates prompts using only a query and task definition, representing zero-shot approaches that rely on an LLM’s internal knowledge [19], [23], [30], [31]. P2 Context-Aware Direct Generation: Augments the P1 setup with domain-specific documents as context. This reflects advanced red-teaming methods [27], [32]–[34] and serves as our model without schema-driven step. P3 Ours (Schema-Driven Generation): Our full proposed pipeline, which incorporates context-driven schemabased behavior generation process. b) Evaluation Process: We conducted a blind study with 12 financial security experts11 . We sampled approximately 90 prompts from each of the three pipelines, for a total of 270 unique prompts. These prompts were randomly shuffled and presented to the evaluators, with the source pipeline concealed. For each risk category (R1-R5) prompts, evaluators provided a rating on a 0-5 Likert scale across three quality criteria: • Financial Risk Alignment: How well the prompt reflects a nuanced, domain-specific financial risk. • Threat Plausibility: The realism and logical coherence of the scenario, including the actor, motivation, and method. • Specificity & Actionability: How clearly the prompt requests a concrete, actionable, and malicious deliverable. Threat Plausibility
Specificity & Actionability
Average Score
Financial Risk Alignment
Pipeline 1
Pipeline 2
Ours (FinRED)
Fig. 4: Human Evaluation Results on Comparing Seed Quality across Three Pipelines. 2) Results and Analysis: The human evaluation in Fig. 4 confirms a progressive improvement from context-free generation (P1), to context-aware generation (P2), to our schemadriven pipeline (P3). P3 achieves the highest scores across financial risk alignment, threat plausibility, and specificity/actionability, showing that schemas structure retrieved regulatory context into concrete and realistic threat narratives. Beyond expert-rated quality, we additionally conduct an operational ASR ablation on the same 270-prompt subset (90 11 Measures to mitigate potential confirmation bias are detailed in Section V-C
prompts per pipeline). P3 produces the highest ASR across model families, with average ASR of 58.05% for generalpurpose sLMs, 70.28% for finance-specific sLMs, and 44.44% for API-based LLMs, while P1/P2 remain lower across the evaluated settings. This confirms that the schema-driven design improves not only expert-rated seed quality but also the ability of the framework to expose real model failures. Table III provides a compact visual summary. We attribute this ASR improvement to the schema-driven design, which exposes latent vulnerabilities by explicitly modeling attacker goals and domain-specific threat elements. TABLE III: Operational ASR ablation summary for generation pipelines. Pipeline P1 (Context-free) P2 (Context-aware) P3 (Ours)
General sLMs
Financial-specific sLMs
API LLMs
41.32 49.86 58.05
52.47 61.73 70.28
28.91 36.58 44.44
D. Agreement Rates of The Proposed Rubric To validate the reliability and alignment of the proposed FinRED Judge rubric, we conducted an empirical human–LLM consistency analysis using evaluations from twelve domain experts at the FSI. Each expert independently reviewed the <red-teaming prompt, LLM response> pairs generated from the FinRED and provided binary safety labels (Safe / Unsafe). The same dataset was simultaneously evaluated by both the baseline HarmBench rubric 12 and our FinRED Judge to compare agreement rates (in terms of alignment) with domain expert (human) judgment. A total of 130 prompt–response pairs were sampled across all level-2 taxonomy categories (five samples per category, covering R1– R5). Each pair consisted of a final seed red-teaming prompt and an LLM output randomly drawn from either Gemma or Exaone models to ensure model diversity. Human labels were aggregated via majority voting to form the ground truth, and performance was evaluated in terms of agreement rate with these expert judgments. 1) Inter-expert Agreement: To further verify the internal consistency of expert judgments used as the ground truth, we analyzed pairwise agreement among the twelve FSI experts. As shown in Fig. 5, most pairwise agreement rates exceed 0.8, reflecting a strong consensus. This internal alignment validates the robustness of our human-annotated baseline, and confirms that the superior performance of the FinRED Judge is measured against a reliable expert standard. 2) Human–Model Agreement Results: Fig. 6 compares domain expert (human)–model agreement between the baseline HarmBench rubric and our proposed FinRED Judge across financial risk domains. FinRED improves agreement from 76.92% to 88.46% (+11.54 points), with a paired ttest confirming significance (p = 0.0024, 95% CI [4.17%, 12 As a widely recognized benchmark in LLM safety research, HarmBench rubric serves as our primary baseline.
κ = 0.79 and α = 0.82, supporting substantial inter-rater reliability. The detailed per-question FGI results are provided in the project repository mentioned in the Abstract.
Fig. 5: Inter-expert Agreement Heatmap Illustrating Pairwise Consistency among Twelve FSI Domain Experts.
Fig. 6: Domain Expert (Human)–LLM Agreement on Two Judge Rubric across Financial Risk Domains 18.91%]) and McNemar’s test supporting different error patterns (p = 0.0041). Errors decrease from 30 to 15, recall improves from 0.73 to 0.88, and Cohen’s κ rises from 0.47 to 0.68. Most importantly for adversarial safety testing, critical false negatives drop from 28 to 12 (57% reduction). We use an “Unsafe if any rubric item is Unsafe” decision rule intentionally: in financial compliance settings, even partial harmful assistance embedded in an otherwise cautious answer can create unacceptable operational risk. V. E XPERT VALIDATION AND R ELIABILITY A. Validation of Financial Risk Taxonomy Following social science research [58], [59], we conducted a FGI and two discussion sessions with 12 financial security experts to evaluate representational adequacy, inter-expert consistency, structural balance, and practical granularity of the taxonomy. Experts provided five-point Likert responses with qualitative feedback, and we computed Cohen’s κ [60] and Krippendorff’s α [61] for agreement beyond chance. Across the four dimensions, experts reported substantial-tohigh agreement (75.0–91.7%), mean Likert scores of 4.20– 4.59, and reliability of κ = 0.73–0.83. They confirmed the taxonomy’s representational breadth, interpretive consistency, structural balance, and practical applicability, while suggesting minor refinements such as clarifying overlap between R2. Financial Crime and R4. Consumer Rights Violation, simplifying low-frequency R5 subtypes, and expanding consumer-facing R4 entries. Per-dimension results in Table IV yield overall
B. Validation of Judge Rubric In addition to the output-level agreement study in Section IV-D, twelve FSI experts reviewed representative rubric criteria across all five Level-1 risk domains. As shown in Table V, experts reported strong agreement that the rubric captures domain-specific harmfulness more effectively than conventional disclaimer-based rubrics (mean = 4.47, SD = 0.43), with substantial reliability (κ = 0.79, α = 0.81). Minor revisions focused on ambiguous partial-compliance and mixed-intent responses. C. Experimental Controls Against Confirmation Bias To mitigate confirmation bias, we used a source-concealed evaluation protocol even though the same 12 domain experts participated in schema refinement and prompt evaluation. Evaluators received randomized prompts without sourcepipeline identifiers (P1/P2/P3), the schema-design and evaluation tasks were separated temporally and cognitively, and final labels or scores were determined through majority voting and cross-checked with inter-rater reliability metrics. These controls do not replace a fully independent panel, but they reduce recognition and preference effects under the practical scarcity of certified financial-security evaluators. In addition, although all experts were affiliated with the FSI, their expertise spans financial fraud prevention, cybersecurity, regulatory compliance, consumer protection, and ICT supervision to provide diverse perspectives across the taxonomy categories. Furthermore, the domain experts were not asked to evaluate outputs generated from scenarios that they had individually authored. The taxonomy construction, schema refinement, prompt evaluation, and rubric validation activities were conducted as separate tasks with different objectives and at different stages of the study. During evaluation, experts received only the generated prompts and model responses, without access to generation metadata, source pipeline information, or prior annotations. Consequently, evaluation decisions were based solely on the observed prompt–response pairs rather than prior knowledge of the generation process. VI. CONCLUSION This paper presents FinRED, an expert-guided construction and evaluation framework for evaluating LLM safety in the financial domain. FinRED bridges general safety evaluation and applied financial risk assessment through a two-level taxonomy, a schema-driven seed generation pipeline, reproducible attack configurations, and a finance-specific judge rubric validated by domain experts. Its successful deployment in the FSI regulatory sandbox shows that FinRED can function beyond an academic dataset as a standardized security validation pipeline that financial institutions can use before deploying LLMbased services. FinRED will be continuously updated with emerging financial threats and evolving regulations, supporting trustworthy, regulation-aligned, and safe financial AI systems.
TABLE IV: Summary of Expert Agreement Metrics for the Financial Risk Taxonomy (FGI Results) Agreement (%)
Likert Score (SD)
Cohen’s κ
Krippendorff’s α
(1) Representational Adequacy (Q1)
83.3
4.46 (0.49)
0.78
0.8
(2) Inter-Expert Consistency (Q2)
83.3
4.32 (0.52)
0.81
0.82
(3) Structural Bias / Balance (Q3)
75.0
4.2 (0.55)
0.73
0.76
(4) Practical Applicability (Q4)
91.7
4.59 (0.44)
0.83
0.84
Overall Mean
83.3
4.39 (0.5)
0.79
0.81
Evaluation Dimension (Q)
Agreement (%)
Likert Score (SD)
Cohen’s κ
Krippendorff’s α
Clarity of Criteria Consistency across Domains Domain-Specific Adequacy Practical Applicability
91.7 83.3 83.3 91.7
4.62 (0.41) 4.45 (0.48) 4.41 (0.46) 4.39 (0.44)
0.82 0.78 0.79 0.8
0.84 0.80 0.81 0.82
Overall Mean
87.5
4.47 (0.43)
0.79
0.82
Evaluation Dimension
TABLE V: Expert Agreement for FinRED Judge Rubric
E THICS S TATEMENT This work studies adversarial financial-risk scenarios to improve the safety of LLMs in financial contexts. We acknowledge that red-teaming benchmarks can be dual-use: prompts designed to expose unsafe model behavior may also resemble malicious requests. Accordingly, FinRED is intended strictly for controlled safety evaluation, model security evaluation, and defensive research, and the dataset is distributed through gated access for qualified researchers. The benchmark construction process avoids real personal data, customer records, confidential supervisory materials, and directly actionable operational instructions. The domain experts participated in validation and rubric evaluation for research purposes, and their feedback is summarized only in aggregate form. We encourage users of FinRED to apply appropriate access controls, monitoring, and responsible disclosure practices when evaluating models with potentially harmful financial prompts. ACKNOWLEDGMENT We would like to express our sincere gratitude to the twelve domain experts at the Financial Security Institute (FSI): Songi Hwang, Ju-hyun Lee, Yoon-hee Lee, Yu-ri Lee, Jeong-won Lim, Kwang-yong Lee, So-young Yoo, Jung-sik Shin, Sanghwa Lee, Jong-yeop Lee, Kyunggyu Kim, Young-mook Kang. Their professional insights, domain knowledge and detailed reviews significantly improved the quality and credibility of our work. R EFERENCES [1] Z. Shi, Y. Wang, F. Yin, X. Chen, K.-W. Chang, and C.-J. Hsieh, “Red teaming language model detectors with language models,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 174–189, 2024. [2] M. Russinovich, A. Salem, and R. Eldan, “Great, now write an article about that: The crescendo {Multi-Turn}{LLM} jailbreak attack,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 2421– 2440.
Key Expert Feedback Summary Taxonomy effectively captures domain-specific risk spectrum; suggested clearer distinction between Financial Crime and Consumer Rights Violation domains. High consistency in category interpretation; proposed simplifying redundant subtypes in ICT compliance domain. Generally balanced across technical and regulatory risks; minor overemphasis on cybersecurity (R1) relative to consumer protection (R4). Strong practical relevance; JSON-schema structure enables automated scenario generation aligned with financial information sharing system (FISS) information framework. Substantial consensus among 12 FSI experts, confirming the taxonomy’s validity, reliability, and practical utility.
[3] D.-Y. Park, “Graph-theoretical approach to enhance accuracy of financial fraud detection using synthetic tabular data generation,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 5467–5470. [4] M. Nagireddy, B. Guillén Pegueroles, and I. Baldini, “Dare to diversify: Data driven and diverse llm red teaming,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6420–6421. [5] Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Redteaming llm agents via poisoning memory or knowledge bases,” Advances in Neural Information Processing Systems, vol. 37, pp. 130 185– 130 213, 2024. [6] Y. J. Ong, J. P. Gala, S. An, R. Moore, and D. Jadav, “Exploring vulnerabilities in llms: A red teaming approach to evaluate social bias,” in IEEE International Congress on Intelligent and Service-Oriented Systems Engineering, 2024. [7] T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing, “Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity,” arXiv preprint arXiv:2301.12867, 2023. [8] T. Gillespie, R. Shaw, M. L. Gray, and J. Suh, “Ai red-teaming is a sociotechnical system. now what?” arXiv e-prints, pp. arXiv–2412, 2024. [9] S. Ghosh, H. Frase, A. Williams, S. Luger, P. Röttger, F. Barez, S. McGregor, K. Fricklas, M. Kumar, K. Bollacker et al., “Ailuminate: Introducing v1. 0 of the ai risk and reliability benchmark from mlcommons,” arXiv preprint arXiv:2503.05731, 2025. [10] L. Li, B. Dong, R. Wang, X. Hu, W. Zuo, D. Lin, Y. Qiao, and J. Shao, “Salad-bench: A hierarchical and comprehensive safety benchmark for large language models,” arXiv preprint arXiv:2402.05044, 2024. [11] Y. Wang, H. Li, X. Han, P. Nakov, and T. Baldwin, “Do-not-answer: Evaluating safeguards in llms,” in Findings of the Association for Computational Linguistics: EACL 2024, 2024, pp. 896–911. [12] T. Yuan, Z. He, L. Dong, Y. Wang, R. Zhao, T. Xia, L. Xu, B. Zhou, F. Li, Z. Zhang et al., “R-judge: Benchmarking safety risk awareness for llm agents,” arXiv preprint arXiv:2401.10019, 2024. [13] X. Guo, H. Xia, Z. Liu, H. Cao, Z. Yang, Z. Liu, S. Wang, J. Niu, C. Wang, Y. Wang et al., “Fineval: A chinese financial domain knowledge evaluation benchmark for large language models,” in Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2025, pp. 6258–6292. [14] Y. Guo, Z. Xu, and Y. Yang, “Is chatgpt a financial expert? evaluating language models on financial natural language processing,” arXiv preprint arXiv:2310.12664, 2023. [15] Q. Xie, W. Han, X. Zhang, Y. Lai, M. Peng, A. Lopez-Lira, and J. Huang, “Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance,” Advances in Neural Information Processing Systems, vol. 36, pp. 33 469–33 484, 2023. [16] Y. Lei, J. Li, D. Cheng, Z. Ding, and C. Jiang, “Cfbenchmark: Chinese financial assistant benchmark for large language model,” arXiv preprint arXiv:2311.05812, 2023. [17] P. Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen, “Financebench: A new benchmark for financial question answering,” arXiv preprint arXiv:2311.11944, 2023. [Online]. Available: https://arxiv.org/abs/2311.11944 [18] B. R. Hogan, W. Brown, A. Boyarsky, A. Schneider, and Y. Nevmy-
vaka, “Technical report: Full-stack fine-tuning for the q programming language,” arXiv preprint arXiv:2508.06813, 2025. [19] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models, 2022,” URL https://arxiv. org/abs/2202.03286, vol. 15, 2022. [20] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [21] S. Tedeschi, F. Friedrich, P. Schramowski, K. Kersting, R. Navigli, H. Nguyen, and B. Li, “Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming,” arXiv preprint arXiv:2404.08676, 2024. [22] S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri, “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,” arXiv preprint arXiv:2406.18495, 2024. [23] Y. Mou, S. Zhang, and W. Ye, “Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types,” Advances in Neural Information Processing Systems, vol. 37, pp. 123 032–123 054, 2024. [24] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” arXiv preprint arXiv:2310.03693, 2023. [25] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang, “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,” Advances in Neural Information Processing Systems, vol. 36, pp. 24 678–24 704, 2023. [26] Y. Wang, H. Li, X. Han, P. Nakov, and T. Baldwin, “Do-notanswer: A dataset for evaluating safeguards in llms,” arXiv preprint arXiv:2308.13387, 2023. [27] Y. Zeng, Y. Yang, A. Zhou, J. Z. Tan, Y. Tu, Y. Mai, K. Klyman, M. Pan, R. Jia, D. Song et al., “Air-bench 2024: A safety benchmark based on risk categories from regulations and policies,” arXiv preprint arXiv:2407.17436, 2024. [28] A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins et al., “A strongreject for empty jailbreaks,” Advances in Neural Information Processing Systems, vol. 37, pp. 125 416–125 440, 2024. [29] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” Proceedings of Machine Learning Research, 2024. [30] P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer et al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 55 005–55 029, 2024. [31] X. Li, J. Ding, C. Peng, B. Zhao, X. Gao, H. Gao, and X. Gu, “Safegenbench: A benchmark framework for security vulnerability detection in llm-generated code,” arXiv preprint arXiv:2506.05692, 2025. [32] Y. Lu, J. Cheng, Z. Zhang, S. Cui, C. Wang, X. Gu, Y. Dong, J. Tang, H. Wang, and M. Huang, “Longsafety: Evaluating long-context safety of large language models,” arXiv preprint arXiv:2502.16971, 2025. [33] H. Xu, W. Zhang, Z. Wang, F. Xiao, R. Zheng, Y. Feng, Z. Ba, and K. Ren, “Redagent: Red teaming large language models with contextaware autonomous language agent,” arXiv preprint arXiv:2407.16667, 2024. [34] G. Sun, X. Zhan, S. Feng, P. C. Woodland, and J. Such, “Case-bench: Context-aware safety benchmark for large language models,” arXiv preprint arXiv:2501.14940, 2025. [35] Q. Xie, W. Han, Z. Chen, R. Xiang, X. Zhang, Y. He, M. Xiao, D. Li, Y. Dai, D. Feng et al., “Finben: A holistic financial benchmark for large language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 95 716–95 743, 2024. [36] S. Kaur, C. Smiley, A. Gupta, J. Sain, D. Wang, S. Siddagangappa, T. Aguda, and S. Shah, “Refind: Relation extraction financial dataset,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 3054– 3063. [37] Z. Chen, W. Chen, C. Smiley, S. Shah, and et al., “Finqa: A dataset of numerical reasoning over financial data,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. [Online]. Available: https://arxiv.org/abs/2109.00122
[38] V. Reddy, R. Koncel-Kedziorski, V. D. Lai, and et al., “Docfinqa: A long-context financial reasoning dataset,” in Proceedings of ACL 2024 (Short Papers), 2024. [Online]. Available: https://huggingface.co/ datasets/kensho/DocFinQA [39] Y. Zhang and Z. Wei, “Boosting jailbreak attack with momentum,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. [40] X. Jia, T. Pang, C. Du, Y. Huang, J. Gu, Y. Liu, X. Cao, and M. Lin, “Improved techniques for optimization-based jailbreaking on large language models,” arXiv preprint arXiv:2405.21018, 2024. [41] J. Hayase, E. Borevković, N. Carlini, F. Tramèr, and M. Nasr, “Querybased adversarial prompt generation,” Advances in Neural Information Processing Systems, vol. 37, pp. 128 260–128 279, 2024. [42] X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” arXiv preprint arXiv:2310.04451, 2023. [43] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” Advances in Neural Information Processing Systems, 2024. [44] J. Yu, X. Lin, Z. Yu, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253, 2023. [45] Y. Yuan, W. Jiao, W. Wang, J.-t. Huang, P. He, S. Shi, and Z. Tu, “Gpt4 is too smart to be safe: Stealthy chat with llms via cipher,” arXiv preprint arXiv:2308.06463, 2023. [46] B. R. Huang, “Plentiful jailbreaks with string compositions,” arXiv preprint arXiv:2411.01084, 2024. [47] Y. Liu, X. He, M. Xiong, J. Fu, S. Deng, and B. Hooi, “Flipattack: Jailbreak llms via flipping,” arXiv preprint arXiv:2410.02832, 2024. [48] A. J. Yepes, Y. You, J. Milczek, S. Laverde, and R. Li, “Financial report chunking for effective retrieval augmented generation,” arXiv preprint arXiv:2402.05131, 2024. [49] X. Liu, P. Li, E. Suh, Y. Vorobeychik, Z. Mao, S. Jha, P. McDaniel, H. Sun, B. Li, and C. Xiao, “Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms,” arXiv preprint arXiv:2410.05295, 2024. [50] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [51] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al., “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [52] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé et al., “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [53] L. Research, S. An, K. Bae, E. Choi, K. Choi, S. J. Choi, S. Hong, J. Hwang, H. Jeon, G. J. Jo et al., “Exaone 3.5: Series of large language models for real-world use cases,” arXiv preprint arXiv:2412.04862, 2024. [54] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [55] Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 268232499 [56] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [57] Y. Chen, H. Gao, G. Cui, F. Qi, L. Huang, Z. Liu, and M. Sun, “Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp,” arXiv preprint arXiv:2210.10683, 2022. [58] R. A. Krueger, Focus groups: A practical guide for applied research. Sage publications, 2014. [59] D. W. Stewart and P. N. Shamdasani, Focus groups: Theory and practice. Sage publications, 2014. [60] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960. [61] K. Krippendorff, “Reliability in content analysis: Some common misconceptions and recommendations,” Human communication research, vol. 30, no. 3, pp. 411–433, 2004.