Beyond the Hype: Evaluating LLM Integration and Practical Limitations in Security Operation Centers

arXiv:2608.17154v1 [cs.CR] 17 Aug 2026

Beyond the Hype: Evaluating LLM Integration and Practical Limitations in Security Operation Centers Elnaz Rabieinejad1 , Ali Dehghantanha1 , Fattane Zarrinkalam2 , Sarina Dastgerdy1 1 Cyber Science Lab, Canada Cyber Foundry, University of Guelph, Guelph, ON, Canada 2 College of Engineering, University of Guelph, Guelph, ON, Canada {erabiein,adehghan, sdastge}@uoguelph.ca, fzarrink@uoguelph.ca Abstract

tioned as a promising response, with the ability to summarize heterogeneous incident data, generate human-readable explanations, and produce draft responses in accessible terms [31]. This capability goes beyond detection: LLMs can translate technical evidence into narratives and candidate next steps that analysts can review, refine, and communicate. Given analyst fatigue and cybersecurity skill shortages, this automation potential is attractive to many organizations. A recent Gartner survey reflects growing industry interest in Generative Artificial Intelligence (GenAI), with approximately 55% of companies experimenting with such technologies; however, only about 10% report production deployment, suggesting persistent concerns about operational reliability and trustworthiness [7]. Most prior work on LLMs in cybersecurity focuses on capability evaluations, benchmark-style studies, or broad surveys of opportunities and risks. By contrast, less is known about how LLMs are used in real SOC workflows, which workflowgrounded failure modes arise when outputs are treated as investigative leads or draft artifacts, and what organizational readiness is needed to verify outputs and mitigate hallucination risk in daily practice. In operational SOC settings, failures may be difficult to detect quickly: models can produce plausible but incorrect statements, invalid logic, or hard-to-verify explanations. Even infrequent errors can erode trust and create costly rework because practitioners must validate outputs before acting on them. Deployment is further constrained by governance requirements such as auditability, change control, and approval processes, as well as by limited access to specialized datasets, rapidly aging threat data, and computational demands [9]. In this work, we investigate how SOC practitioners perceive the utility, limitations, and mitigation strategies associated with LLM-based security tools, and we characterize when perceived benefits are bounded by failure modes and verification burden. Our study is guided by three research questions:

Large Language Models (LLMs) are increasingly being explored within Security Operation Centers (SOCs) to support text-heavy analytical work such as alert contextualization, incident summarization, and drafting investigative artifacts. Despite this interest, practitioners describe critical operational concerns, most notably hallucinations (plausible but incorrect outputs), opaque reasoning, and the verification effort required to safely use model-generated content in security workflows. In this paper, we present findings from semi-structured interviews with 20 SOC practitioners spanning frontline analysts, SOC managers, and tool developers. Participants report perceived time savings for low-stakes tasks that are quickly verifiable (e.g., summarizing logs or drafting initial investigative leads), but they consistently frame LLM outputs as preliminary drafts and suggestions rather than decision-grade conclusions. Participants also describe limited trust in LLMs for high-stakes security decisions due to unreliable outputs and unclear model reasoning, and they report relying primarily on ad-hoc verification norms and continuous human oversight rather than standardized mitigation procedures. Based on these interview-grounded accounts, we introduce a maturity rubric to characterize readiness for LLM integration and outline a research agenda emphasizing auditability and transparent explanation mechanisms to support safer adoption in SOC workflows.

1

Introduction

Security Operations Centers (SOCs) face sustained overload from the growing volume and complexity of alerts, logs, and incident artifacts. Traditional Machine Learning (ML) techniques can support narrow classification and anomaly detection tasks such as intrusion detection, malware classification, and threat correlation [20]. However, these approaches often stop at labeling events as malicious or benign and do not consistently provide the contextual, human-readable explanations needed for fast, defensible analyst decisions under time pressure. Large Language Models (LLMs) have recently been posi-

• RQ1: Where and how are LLMs currently integrated 1

into SOC workflows?

The remainder of this paper is organized as follows. In Section 2, we review relevant literature. Section 3 details our research methodology. We then present findings addressing our research questions in Sections 4, 5, and 6, respectively. Section 7 discusses implications, limitations, and actionable recommendations for future research and practice. Finally, Section 8 concludes the paper.

• RQ2: What are the perceived benefits and practical challenges associated with using LLMs in security operations? • RQ3: To what extent are SOCs organizationally prepared to identify, understand, and mitigate LLM hallucination risks?

2

To answer these questions, we conducted semi-structured interviews with 20 security practitioners across SOC ecosystems, including front-line analysts, SOC managers, tool developers, and researchers exploring LLM integrations. Participants described use cases, perceived benefits, failure modes, and the verification and mitigation practices they rely on in daily work. This scoping decision was explicit rather than post-hoc. Among the failure modes practitioners encounter, hallucination is uniquely dangerous in SOC workflows because, unlike adversarial manipulation or syntax errors, hallucinated facts and misleading investigative direction can appear plausible enough to pass initial scrutiny and silently propagate into triage checklists, detection drafts, and incident reports. We therefore scoped RQ3 around hallucination because assessing readiness to manage this risk required practitioner-grounded interview evidence that capability benchmarks alone cannot provide. Scope/Non-goals. We study non-adversarial operational reliability failures when SOC practitioners use LLMs as assistants in routine workflows. Here, a failure mode is any plausible but wrong, incomplete, misleading, or hard-to-verify output that increases workflow risk through rework, mis-triage, delay, or incorrect reporting. Our taxonomy is grounded in participant-reported day-to-day use, including hallucinated facts, invalid logic, misleading direction, interpretation errors, and verification burden, under the assumption that analysts remain accountable decision-makers. We do not evaluate adversarial attacks or attack-surface effectiveness; these are discussed only as context because they were not directly measured in our interviews. Contributions. This paper makes four contributions grounded in SOC practice:

Related Work

Recent work frames LLMs in cybersecurity as a double-edged capability, offering defensive benefits while introducing new attack risks. Focusing on operational SOC environments, we review two strands of literature: defensive SOC-oriented uses and offensive/AI-native threats, then highlight the evaluation and operational gaps motivating RQ1–RQ3.

2.1

Defensive Uses of LLMs in Cybersecurity

A growing body of research explores how LLMs can support security analysis tasks that are language-heavy and contextdependent, including security investigation assistance, narrative summarization, and explanation. For example, HuntGPT integrates anomaly detection and explainable AI with an LLM to support threat hunting and analyst-facing reasoning over signals [2]. Survey work similarly highlights how LLMs are being explored across defensive tasks such as analysis, triage, and knowledge extraction [31, 33]. More recently, SOC-specific literature has begun to organize these capabilities around concrete SOC workflows. Habibzadeh et al. provide a comprehensive SOC-focused survey, structuring LLM use cases across core functions such as alert triage, incident response support, log analysis, and threat intelligence, while emphasizing practical constraints including data sensitivity, evaluation realism, and reliability [8]. Complementing surveys, early system-oriented studies examine how LLMs can be embedded into SOC toolchains. Singh et al. evaluate multi-model LLM integration with SIEM workflows for Tier-1 triage decisions, illustrating potential efficiency gains while underscoring the need for careful validation when LLM outputs affect operational prioritization [28]. Tseng et al. target a persistent SOC bottleneck, the repetitive analysis of natural-language CTI reports, and propose an LLM-based agent to automate key CTI analysis steps, aiming to reduce analyst overhead in intelligence-driven workflows [29]. Together, these works motivate our focus on where and how LLMs are integrated in real SOC practice (RQ1) and what benefits and frictions practitioners perceive (RQ2). A second line of work focuses on developer- and codecentric security workflows. Empirical evaluations show that code assistants can introduce security risks in practice: users may accept insecure suggestions, misunderstand generated code, or over-trust outputs under time pressure [21, 26]. Together, these studies suggest that LLMs can be valuable as assistive tooling, but benefits are tightly coupled to verification practices, user expertise, and workflow design.

• a role-stratified empirical mapping of where LLMs enter SOC workflows and which tasks practitioners delegate versus retain (RQ1); • a transcript-derived taxonomy of LLM output failure modes in SOC settings (FM1–FM7) with role coverage and salience (RQ2); • a hallucination-mitigation maturity rubric with observable criteria and a participant-level distribution (RQ3); • and a decision-grade, verification-first synthesis matrix connecting SOC task types, failure modes, verification burden, maturity prerequisites, and safe integration patterns. 2

2.2 Offensive Uses and AI-native Attack Classes

manipulation is discussed only as context. The goal is to support responsible deployment by helping teams manage hallucination-related risks in line with security requirements.

Prior work shows that LLM-integrated systems can be vulnerable to adversarial manipulation (e.g., malicious prompting and knowledge-source manipulation) and that LLMs may lower the cost of generating persuasive malicious content. We cite this literature to situate the broader risk landscape, not as phenomena we empirically test. For instance, prompt injection has been formalized and benchmarked as a control-plane manipulation in LLM applications [17]. RAG increases reliance on external corpora [15], enabling knowledge-corruption attacks such as PoisonedRAG that can steer generations by poisoning parts of the retrieval corpus under certain conditions [34]. Other work highlights LLM-enabled offensive content generation at scale (e.g., phishing) [24], jailbreak-style robustness failures across models [27], and risks in agentic toolusing configurations in constrained settings [6]. In contrast, our interview-based evidence focuses on non-adversarial operational reliability failures in SOC workflows, incorrect or weakly grounded outputs, prompt-to-prompt variability, and the verification burden required to safely use LLM-generated drafts and preliminary leads, treating adversarial risks as background motivation rather than a measured outcome or contribution.

2.3

3

Methodology

To investigate our research questions, we conducted semistructured interviews with 20 cybersecurity professionals engaged in SOC activities. Our goal was to capture diverse perspectives on the practical use, perceived benefits, challenges, and organizational preparedness regarding the integration of LLMs in security operations. Participants were specifically selected for their direct experience in using, managing, or researching security tools within operational SOC environments.

3.1

Semi-Structured Interviews

Each participant completed a 30-minute semi-structured online interview conducted in English. Interviews consisted of three thematic sections, directly aligned with our research questions. Initially, participants described the integration and use of LLMs within their SOC workflows (RQ1). Subsequently, they discussed the comparative benefits and limitations of LLM-based tools versus alternative methods (RQ2). Finally, we examined participants’ perceptions of organizational readiness and strategies employed to address hallucinations in LLM outputs, specifically highlighting trust impacts and mitigation approaches (RQ3). To maintain consistency, we developed and adhered to a standardized interview protocol, covering introductions, consent processes, clear and non-leading questions, and postinterview compensation procedures.Before data collection, the interview guide was independently reviewed by a cybersecurity professional external to the author team and with SOC-relevant experience, in order to assess question clarity, terminology, and domain appropriateness; the guide was revised accordingly. In addition, we conducted a single pilot interview using the interview guide before formal data collection, and used it to refine question wording, flow, and timing.

Positioning of Our Study

Existing work has examined where LLMs may support defenders, including SOC surveys and early SIEM/CTI automation prototypes [8, 28, 29], as well as the risks these systems may introduce. However, this literature largely emphasizes technical solutions and controlled benchmarks, while giving less attention to practitioner experience, perceptions, and organizational preparedness. Prior qualitative SOC studies also describe workflows and adoption barriers, but often stop short of producing LLM-specific, workflow-grounded guidance. Our study extends this literature by contributing three practitioner-derived artifacts for deployment decisions: (i) a failure-mode taxonomy, (ii) a hallucination-mitigation maturity rubric, and (iii) a SOC Integration Constraint Matrix linking task types to failure modes, verification burden, maturity prerequisites, and safer integration patterns. Although concerns such as hallucinations, prompt sensitivity, verification burden, and the need for human oversight are consistent with prior literature, our contribution is to show how they appear in SOC workflows specifically, where outputs are produced under time pressure, must be traceable to evidence such as logs or threat-intelligence sources, and are constrained by governance requirements. In this setting, the question is not only whether LLM outputs can be wrong, but which tasks can remain safely draft-oriented, which failures are hardest to detect quickly, and what controls are needed before broader deployment. Our empirical claims are limited to non-adversarial operational reliability failures in SOC workflows; adversarial

3.2

Participant Recruitment

Participants were recruited from June to December 2024 through professional networks and platforms, notably LinkedIn, and via personal referrals. Eligible candidates had at least one year of practical SOC experience, were 18 years or older, and actively engaged with cybersecurity operations. We did not require explicit expertise in AI or LLMs, allowing us to include varied perspectives reflective of the broader SOC landscape. This recruitment strategy, similar to prior qualitative studies in cybersecurity [1, 5, 20] which had below 20 participants, do not claim our findings are generalizable given the sample size. Instead, we use the data to highlight salient emerging themes and concepts. All interviews were audiorecorded with participant consent and transcribed verbatim by a GDPR-compliant professional service. Each participant received a $50 CAD electronic gift card upon completion. 3

Table 1: Code saturation by participant batch.

18 to 9, primarily consisting of niche job roles or specific tool names rather than new conceptual challenges.In the final Interview Batch New Primary New Cumulative % Final batch (P16–P20), no new codes or dimensions were added to Codes Subcodes Codes Codebook the codebook, confirming that our sample of N = 20 was suffiP01–P05 14 108 122 82% cient to capture the breadth of practitioner experiences in this P06–P10 0 18 140 94% regional SOC ecosystem. We classify participants as LLM P11–P15 0 9 149 100% users if they reported any workplace LLM use (including P16–P20 0 0 149 100% occasional or low-intensity use). Participants who reported no workplace LLM use are labeled Not applicable because the mitigation maturity rubric is not applicable to them. Statements from participants without approved workplace LLM Participant demographics are summarized in Table 4 in Apuse are treated as perceptions or indirect exposure and are not pendix D, highlighting a wide range of roles, experiences, and used to infer operational deployment. organizational contexts represented in our sample. For all thematic findings, the individual participant (N = 3.3 Data Analysis 20) serves as the primary unit of analysis. While participants represent 13 distinct organizations, their reported experiences Interview recordings were transcribed verbatim by a General reflect individual roles and specific sub-teams. Consequently, Data Protection Regulation (GDPR)-compliant professional maturity levels and usage patterns are classified at the parservice and analysed using codebook thematic analysis folticipant level to capture the heterogeneity of practice within lowing Braun and Clarke’s phases (familiarisation, generatlarger organizations. Where a metric is reported at the oring codes, searching/reviewing themes, defining/naming, reganizational level (e.g., demographics in Section 4.1), it is porting) [4]. Two researchers independently coded an initial explicitly noted. subset (5 interviews) to develop a shared codebook, compared Researcher positionality. The research team has experience interpretations, and reconciled discrepancies. Inter-rater reliin cybersecurity research and SOC-adjacent problem settings, ability (IRR) was then estimated on held-out batches every which informed the framing of the interview protocol and few interviews using Cohen’s κ, with interpretation followinterpretation of practitioner terminology. To reduce intering Landis and Koch [14]. Coding proceeded iteratively with pretive bias, we used independent coding, iterative codebook periodic codebook updates; when agreement fell below our refinement, and consensus-based reconciliation throughout threshold (κ < 0.80), we resolved differences by discussion, the analysis. refined code definitions, and retro-fit earlier transcripts as needed. After five coding rounds, agreement stabilised at 3.4 Limitations κ > 0.80 across thematic categories. Cohen’s κ computed Our study has several limitations typical of qualitative work. at the subcode level on held-out transcripts improved across First, while semi-structured interviews enabled depth, rerounds from κ = 0.81 to 0.88 [Round 1: 0.81; Round 2: 0.83; sponse detail varied because follow-up probing was not perRound 3: 0.85; Round 4: 0.86; Round 5: 0.88]. The most fectly uniform across participants; core questions were covfrequent disagreements were boundary cases between data ered consistently, but some themes have richer narrative evileakage/model elicitation vs. prompt/model misuse. Consisdence than others. As with most interview-based qualitative tent with our recruitment rationale, we monitored both code studies, our findings reflect participant accounts rather than and meaning saturation during analysis and observed no new direct observation of workflows, creating the possibility of substantive codes after Interview 15, continuing to N=20 to self-reporting and social desirability effects. strengthen meaning saturation. We define saturation as no new sub-themes or dimensions emerging in the last five transcripts, Second, the sample is geographically concentrated (19/20 with no changes to code definitions [10]. The final codebook, in Ontario, Canada; 1 in California, USA). Although particexemplar quotes, and IRR statistics appear in Appendix C ipants represented organizations of varied sizes (Table 4), (Table 2 and Table 3). we do not claim global representativeness, nor did we systematically control for SOC maturity beyond self-described According to Table 1 in Appendix E, to demonstrate the roles and contexts. The results should therefore be read internal robustness of our thematic discovery, we conducted as an in-depth regional snapshot of early LLM use and as a saturation audit by tracking the first appearance of each practice-informed hypotheses, not sector-wide benchmarks. code within our participant batches. We defined saturation Gender was collected as a demographic variable (16 male, 4 as the point where a batch of five interviews yielded no new female) but was omitted from Table 4 to preserve participant top-level primary codes and fewer than 5% new subcodes. anonymity given the small sample size; the gender imbalance All 14 primary thematic categories (e.g., Challenges & toward male participants may limit the breadth of perspectives Limitations, Adaptability, Comparison to Rule-Based) were captured. established within the first five interviews. Between Batch 2 and Batch 3, the emergence of new subcodes dropped from Third, confidentiality and NDAs meant we did not re4

quire disclosure of LLM vendors, model versions, deployment mode (hosted vs. on-prem), or fine-tuning, so we cannot separate model/deployment-specific effects from broader adoption patterns; our findings capture use patterns and practitioner perceptions, not comparative performance across models. Fourth, the findings are time-stamped to mid–late 2024, before more integrated agentic workflows and widespread grounding became common; while tools may evolve quickly, SOC governance and verification typically change more slowly, so we frame our themes as a baseline for future longitudinal study. In addition, our coding framework allowed failure modes to co-occur through multi-label coding, but co-occurrence reporting was not a pre-specified objective of this paper. We therefore do not quantify co-occurrence frequencies here, and leave systematic analysis of the most common co-occurring pairs for future work. Finally, we do not measure operational KPIs (e.g., incident response time, FP/FN reduction, detection lift); claims are limited to interview-grounded descriptions of practice and perceived utility, usually conditional on verification.

trusted outputs) as the central lens for understanding readiness and trust in this integration. Our results reflect two kinds of interview-grounded evidence: (i) reported practices (how participants say they currently use LLMs in SOC workflows, including what artifacts they generate and how they validate them), and (ii) perceptions (participants’ reported benefits, costs, and trust judgments). We do not evaluate decision-grade operational outcomes (e.g., detection lift, false positive reduction, or incident response time reduction). Accordingly, terms such as “effectiveness” and “efficiency” in this section refer to participant-reported perceived utility and workflow support, typically conditional on verification.

4.1 Participant Background and Demographics Table 4 summarizes demographics and professional profiles for 20 participants spanning diverse cybersecurity roles (frontline analysts, SOC managers, threat-response engineers, data scientists, and senior leadership) with 1.5–25 years of experience. We classified participants as SOC professionals if their primary work involved day-to-day SOC operations (e.g., triage, incident response, detection engineering, or SOC management), and as security researchers if they primarily worked in research/R&D but had direct SOC workflow involvement within the past 12 months; purely academic researchers without operational SOC engagement were excluded. Participants’ expertise ranged from defensive-only to combined offensive/defensive practice, with varied methodological preferences (rule-based, AI-driven/ML/LLM, or hybrid) and wideranging AI experience from conceptual familiarity to handson model development. The sample was geographically concentrated (19/20 Ontario, Canada; 1 California, USA) and drawn from 13 organizations spanning in-house enterprise SOCs, public-sector SOCs, MSSPs/providers, and security product vendors across all scale bands—small (50–249), medium (250–999), large (1,000–9,999), and very large (≥ 10,000); to preserve anonymity, we report scale bands rather than exact headcounts, and no single organization dominated the sample.

3.5 Analytic Triangulation and Robustness Checks We employed two internal robustness checks to ensure our findings were not biased by our initial analytical lenses: Negative-Case Analysis: While most participants reported a heavy verification burden, outliers like P12 and P17 highlighted a critical boundary condition: LLM utility is highly task-dependent. In code-centric generative tasks, such as drafting filtering logic or Python scripts, practitioners experienced minimal friction because outputs are rapidly falsifiable through technical tools like execution tests and syntax checks. Conversely, skepticism remained high for investigative or narrative-driven tasks—such as incident reasoning or attribution—where errors often appear plausible and are difficult to verify quickly, necessitating much heavier human judgment. Sensitivity Analysis: To confirm that our conclusions are not an artifact of presentation structure, we re-aggregated the coded segments using two alternative organizational lenses: (i) a task/workflow lens (Section 4) and (ii) an evaluation-criteria lens (Section 5). We treat the task/workflow lens as primary, and use evaluation criteria only as a supporting organization for how participants justified trust and perceived utility. Across both lenses, the core conclusion remained unchanged: perceived LLM utility is highest in text-centric, draft-oriented tasks, while adoption is bottlenecked by verification burden and limited trust in decision-grade outputs.

4

4.2 Common Use-Cases and Contextual Dependencies Across participant categories, four use-cases appeared consistently: log summarization, initial incident investigation support, documentation drafting, and detection rule refinement. Rather than treating these as task types alone, we characterize each as a workflow segment with an identifiable entry point, decision gate, and transition condition, based on participant accounts. Log summarization. Entry was volume-driven: analysts invoked LLMs when a log artifact exceeded what could be parsed manually at triage speed (e.g., P09, P10). After prompting for a summary, they applied a rapid plausibility check, comparing the output against a spot-check of the raw logs, be-

Results: Tool Usage (RQ1)

In this section, we first provide an overview of participants’ professional backgrounds and then present findings on how LLM-based security tools are integrated into SOC workflows, using LLM output failure modes (i.e., incorrect or un5

fore deciding to escalate, close, or continue. The falsification condition is fast and concrete (“is this entity in the raw log?”), which explains why participants reported low verification burden for this task. Triage guidance. A second entry point was unfamiliarity with an alert type. Analysts (e.g., P13) described prompting for first-pass investigation steps, then filtering the output against internal playbooks, retaining steps supported by available telemetry and discarding those that contradicted knownsafe behavior or lacked log evidence. The LLM output served as a brainstorm scaffold, not an executable runbook; each suggested step required independent justification before execution. Detection rule drafting. Engineers (e.g., P04, P12) invoked LLMs after a detection gap was already identified, providing a natural-language description or a suspicious log excerpt and receiving a candidate query or rule fragment. The immediate decision gate was syntactic: linting or sandbox execution to confirm operator and field validity. Rules that passed moved to backtesting against historical traffic before peer review and production promotion. Failures, including the FM2 case where an LLM invented nonexistent operators (P04), triggered manual correction or prompt revision. CTI and report synthesis. Researchers (e.g., P07, P19) invoked LLMs after aggregating intelligence from multiple feeds, prompting for TTP extraction or a structured summary. The decision gate was cross-referencing: extracted indicators were checked against internal threat intelligence platforms or MITRE ATT&CK before operational use. P07 described constraining prompts to provided source text to reduce inferencedriven fabrication. A consistent structural pattern emerged across all four cases: the LLM is invoked at a specific handoff point in an existing workflow, and its output passes through a humanoperated decision gate before any downstream action. The feasibility of that gate, and therefore the practical verification burden, depends on whether the output type admits a fast, concrete falsification condition. This structure motivates the verification burden and maturity prerequisite mappings in the SOC Integration Constraint Matrix (Appendix F).

5

security tasks [32]. Participants consistently described perceived workflow benefits in some SOC tasks, particularly where outputs were draft-oriented and quickly verifiable. 5.1.1

A dominant perception among participants (13 of 20) was that LLMs can provide useful workflow support in text-intensive SOC tasks by reducing the time and manual effort associated with summarizing logs and reports, drafting standardized detection content, and producing documentation. Participant P09, for example, described rewriting detection rules with LLM assistance as “much faster” than manual drafting, while still treating the output as a draft requiring review. Participants further suggested that automating routine language-related tasks could allow analysts to redirect attention toward more complex activities requiring human expertise. We interpret these accounts as participant-reported perceived utility and workflow support, rather than as direct evidence of measured performance improvement or operational productivity gains. 5.1.2

Emerging Use in Detection Engineering Support

A subset of participants described using LLMs for detection engineering support, where models help analysts draft and refine artifacts that may later feed existing detection workflows. Reported uses included extracting salient fields from log excerpts, drafting triage checklists, and generating candidate query/rule fragments or investigative steps. A smaller subset discussed experimental pilots in which LLMs proposed candidate detection ideas from log snippets, but these outputs were consistently framed as suggestions that require rigorous human validation and conventional backtesting/review before any operational adoption. Overall, participants viewed LLMs as accelerating early-stage drafting and hypothesis generation, not replacing detection pipelines or analyst judgment. Three participants also mentioned experimental deployments where LLMs directly suggested detections from log excerpts or suspicious data points; again, these suggestions were universally described as needing strict human verification prior to acceptance. Together, the findings indicate a cautious but growing integration of LLMs into SOC detection work, balancing the speed benefits of automation with ongoing accuracy concerns and the continued necessity of analyst oversight, with an envisioned model where LLM tools augment rather than replace human analysts.

Results: Perception (RQ2)

We asked participants to describe their experiences and perceptions regarding the integration of LLM-based tools into SOC workflows, specifically focusing on the perceived benefits and challenges. From the thematic analysis of our interviews, five distinct evaluation criteria emerged consistently: Effectiveness, Usability, Efficiency, Adaptability, and Trust. Below, we detail the participants’ insights with respect to these criteria, emphasizing particularly prominent themes.

5.1

LLMs Excel in Text-Centric Tasks

5.2

Usability

Usability refers to the ease with which SOC analysts can effectively interact with and configure LLM-based tools within their existing workflows [35]. While participants generally appreciated the intuitive nature of chat-like interfaces compared to traditional, specialized ML toolchains, they also highlighted critical usability challenges, particularly related to prompt engineering and transparency of outputs.

Effectiveness

Effectiveness refers to an LLM’s capacity to generate accurate and contextually relevant outputs that support operational 6

5.2.1 Accessible Interfaces Accelerate Onboarding Participants widely recognized the accessibility and intuitive use of conversational interfaces, particularly highlighting widely available platforms such as ChatGPT. Seven participants explicitly praised user-friendly features such as conversational histories, natural language prompts, and proactive disclaimers about potential inaccuracies. Another four participants reported primarily interacting with LLM outputs via integrated, vendor-based security platforms (e.g., Splunk, Microsoft Defender), appreciating the seamless embedding of LLM functionalities within familiar security dashboards. A frequently cited advantage (n=3) was the minimal technical setup required to initiate interaction with LLM-based tools. Participants described significant ease in formulating initial queries, detection rules, or summaries without needing custom code. For example, P12 emphasized this usability benefit by stating: “I don’t need to code intricate filters; I just describe what I want, and the tool drafts it.” The approachable interface reduces barriers to adoption, enabling rapid onboarding and reducing barriers to drafting and exploratory tasks (e.g., writing initial summaries or first-pass rule/query drafts), especially for analysts who can rapidly validate outputs.

only hampers trust in LLM-generated results but can also slow incident response times, limiting overall operational efficiency. In summary, while the initial usability of LLM-based tools is positively perceived, practical adoption is significantly influenced by challenges around effective prompt engineering and output transparency. Addressing these issues through improved user training, clearer explanatory mechanisms, or enhanced interface designs could substantially enhance the usability and operational utility of these tools in SOC environments.

5.2.2 Challenges with Prompt Engineering Despite initial ease of use, eight participants highlighted prompt engineering as a key usability challenge. They noted that vague or inadequately specified prompts frequently resulted in overly general or irrelevant outputs, requiring repeated adjustments or clarifications. Participant P07 emphasized the effectiveness of incorporating real-world examples or log excerpts directly into prompts, stating that this practice significantly improved output accuracy. This finding indicates an inherent tension between the superficial simplicity of LLM interfaces and the nuanced skills required to craft effective prompts. While the chat-based interaction model appears straightforward, achieving consistent, relevant results heavily depends on analyst domain knowledge and precise query formulation. Thus, successful integration and sustained usability of LLM-based tools depend substantially on analysts’ expertise and their familiarity with effective prompting techniques.

A majority (n=13) of participants emphasized the substantial time-savings enabled by LLM assistance in routine, languageintensive SOC tasks such as log summarization, incident documentation, and security briefings. For instance, P16 noted that email security summaries previously taking “an hour or two to compile” were now generated “within minutes.” Three participants explicitly mentioned workload reductions, indicating that LLMs effectively reduced the burden of mundane tasks, enabling them to focus more on complex investigations. Despite these advantages, participants acknowledged that effective prompt engineering was still necessary to ensure coherence and relevance in LLM-generated outputs. These accounts suggest participants perceived workload relief in specific tasks, especially where outputs were quickly verifiable.

5.3

Efficiency

Efficiency refers to how LLM-based solutions help analysts complete security tasks with minimal time, data, and computational overhead [32]. Participants consistently highlighted reported perceived efficiency gains—particularly in automating repetitive tasks—but also emphasized the importance of balancing these gains against verification overhead. 5.3.1

5.3.2

Time-Saving and Workload Reduction for Repetitive Tasks

Automation Potential Beyond Text Summarization

Beyond text-based tasks, five participants recognized the potential of LLMs to facilitate semi-automated scripting and investigation workflows. Participant P17, for example, described using LLM-generated Python code snippets for rapid log analysis, reducing manual coding effort in their workflow, conditional on validation. Similarly, participant P13 mentioned that LLMs effectively drafted “first-pass triage steps,” enabling the team to concentrate on deeper forensic analysis. Nevertheless, all participants using such semi-automated workflows stressed the essential role of human review due to the persistent risk of inaccuracies and logical errors in automated outputs. This underscores an emerging hybrid model, where LLMs accelerate initial task stages but require analyst-led validation, balancing efficiency gains with accuracy requirements.

5.2.3 Nontransparent Reasoning Hinders Adoption Seven participants raised significant usability concerns related to the opacity of LLM reasoning processes. Unlike traditional rule-based detection methods, where triggered rules explicitly clarify why certain outputs are generated, LLM outputs are often presented without clear justification or traceable reasoning. Participant P05 described this concern vividly: “With rule-based systems, at least we know which rule triggered. The LLM just gives an answer, and I’m not always sure why.” Such opacity frequently compels analysts to conduct supplementary manual verification—examining logs, consulting historical records, or otherwise attempting to infer the rationale behind generated outputs. This lack of transparency not 7

5.3.3 Verification Overhead May Erode Efficiency Gains Despite general satisfaction with accelerated content generation, several participants expressed concerns about the associated verification burden. Participant P04 stated explicitly, “We’ll save an hour drafting rules but spend half an hour verifying every line.” This overhead becomes especially prominent in critical situations—such as near-real-time incident response or high-visibility reporting—where thorough verification is essential. Many participants described cross-referencing LLM outputs with internal procedures or historically validated rules to mitigate risks. This verification necessity highlights a fundamental aspect of LLM deployment in security: although the speed advantages are clear, ongoing human oversight remains crucial. Thus, participants generally perceived efficiency benefits as contingent upon their capacity to quickly identify and correct occasional inaccuracies, particularly in low-risk contexts.

5.4

noting that continuously fine-tuning models to emerging threats is expensive, time-consuming, and frequently limited by internal expertise. 5.4.3 Organizational Barriers and Data Policies Beyond technical considerations, seven participants noted significant organizational barriers affecting adaptability. Challenges included compliance regulations, data governance policies, vendor approval processes, and leadership hesitancy. Participant P15 characterized deploying an internal LLM as a “multi-layer labyrinth” requiring extensive approvals from Legal, Privacy, and Compliance teams. Conversely, two participants noted smoother adoption experiences when senior management explicitly supported AI initiatives, facilitating faster piloting and experimentation. Budget constraints further impacted adaptability: Participant P07 observed that frequent model fine-tuning was costly, causing some organizations to rely solely on general-purpose, vendor-trained models. Thus, practical adaptability hinges not only on technical flexibility but also on organizational structures, resource availability, and policy alignment.

Adaptability

Adaptability describes an LLM-based tool’s capability to seamlessly integrate into existing organizational processes and respond effectively to varying operational contexts [32]. While participants recognized the general adaptability of LLMs, they highlighted notable limitations when encountering novel or highly specialized security scenarios.

5.5

Trust

Trust refers to users’ confidence in the reliability and factual accuracy of LLM outputs, particularly regarding resistance to inaccuracies and fabricated information [32]. Participants consistently identified trust as a critical factor influencing LLM adoption in SOC environments.

5.4.1 Flexibility Across Data Sources and Tasks Several participants (n=6) positively assessed the adaptability of LLMs across diverse SOC tasks and data sources, including asset risk scoring, summarizing alerts, and synthesizing vulnerability metrics. Participant P17, for example, successfully integrated LLM outputs with internal classification data, noting that the model “shifts easily between different environments if given the right inputs.” However, two participants clarified that effective adaptation often required significant domain-specific input and careful data curation from knowledgeable analysts. This suggests that LLM adaptability relies heavily on ongoing analyst input, highlighting the importance of domain expertise and proactive context provisioning in sustaining successful deployment.

5.6 Taxonomy of LLM Output Failure Modes We move beyond treating hallucination as a single issue by defining a practitioner-grounded taxonomy of LLM output failure modes observed in SOC work, based entirely on our interview corpus. We focus on failures that produce incorrect, ungrounded, invalid, or hard-to-verify outputs that could be adopted in routine SOC tasks (e.g., detection drafting, interpreting suspicious activity, investigative guidance). The taxonomy reflects practitioner-described breakdowns and verification burdens, and intentionally excludes adversarial threats (e.g., prompt injection) except where participants reported first-hand incidents. Using coded segments about trust failures and incorrect outputs, we identify seven failure modes and assign severity via a consequence-if-acted-upon rubric: Low (wasted time/minor confusion), Medium (misdirected triage/investigation and forensic drag), and High (plausible missed detections, incorrect escalations, or unsafe actions such as blocking legitimate traffic). These severities represent conceptual potential consequences, not observed incident impact in our study. Appendix Table 5 (Appendix E) reports the taxonomy with inclusion/exclusion criteria and representative evidence. For reproducibility, we use thematic segments (sentence groups describing a specific failure event) as the unit of analysis and apply multi-label coding for compound failures (e.g., FM1 + FM5). Appendix Table 9 (Appendix E.1) summarizes

5.4.2 Limits with Novel or Specialized Threats Despite overall versatility, four participants expressed concerns regarding LLM performance deterioration when dealing with novel or specialized threats, such as zero-day vulnerabilities lacking historical references. Participant P19 described that “when it’s a fresh zero-day or we have minimal intel, the LLM is forced to guess, and sometimes that guess is offbase.” Consequently, in highly specialized scenarios, LLM predictions often become error-prone. Participants indicated two primary strategies to mitigate this issue: quickly supplementing models with specialized threat data or reverting to manual investigations. However, three participants raised concerns about resource constraints, 8

coverage and role-based salience across participants. Collectively, this taxonomy clarifies that what practitioners describe as hallucination in SOC settings spans several distinct failure modes, each with different operational outcomes. This structured approach enables more precise recommendations by tying mitigations (e.g., schema validation for FM2 or RAG for FM1) to specific failure classes rather than to an unquantified notion of trust.

readiness conditions in early SOC adoption, rather than as a missing case or lack of relevance. Operational risk model.To make our security framing explicit and falsifiable from interview evidence, we operationalize hallucination risk as a set of observable outcome types for an LLM output used during a SOC task. We treat an output as a failure when it exhibits one or more of the following: (i) fabricated facts/entities (e.g., fictitious incident details or nonexistent IOCs/TTPs), (ii) schema- or tool-invalid artifacts (e.g., invalid detection-rule syntax or invented operators), or (iii) misleading procedural recommendations (e.g., incorrect first-pass triage steps or investigative guidance that increases verification burden or steers analysis incorrectly). A failure becomes operationally consequential (i.e., “successful” in risk terms) when it is plausible enough to be acted upon or treated as a lead, when it causes wasted analyst time through rework and verification, or when it propagates error into an incorrect investigative step, draft rule, or report content. Importantly, we do not measure incident impact or business loss, and we do not evaluate attacker adaptation or adversarial robustness; instead, we report practitioner-described failure modes and the mitigation practices they use in operational settings. Table 6 in Appendix E summarizes these outcome types, the operational consequences that make them risky in practice, and the mitigation practices reported by participants.

5.6.1 Essential Role of Human Oversight Nearly all participants (n=15) explicitly reinforced the necessity of ongoing human oversight. While many participants described perceived workflow benefits in specific tasks from LLM assistance, participants insisted that human judgment was indispensable for verifying outputs. Participant P08 succinctly stated:“an LLM is an assistant, never the final authority,” reflecting a prevailing sentiment that analysts must thoroughly vet LLM-generated detections, summaries, and recommendations. This emphasis on human oversight aligns with an emerging best practice in cybersecurity: combining AI-driven acceleration with expert human scrutiny. Participants described a model of use where any potential workflow benefit depends on keeping analysts in the loop to validate outputs and prevent errors from propagating. In summary, although participants acknowledged the perceived operational usefulness provided by LLMs, they remain cautious about fully entrusting AI with critical security decision-making tasks. Near-term trust appears fundamentally dependent on robust oversight and validation procedures capable of managing the inherent uncertainty in current LLM technologies.

6

6.1 General Observations About Hallucinations Participants consistently recognized hallucination as a unique operational failure mode distinct from traditional false positives generated by rule-based systems or conventional ML models. Specifically, they noted that LLMs can fabricate extensive narratives—such as incident details, fictitious indicators of compromise, or nonexistent tactics, techniques, and procedures (TTPs)—posing a significant risk to SOC workflows. Participant P06 captured this sentiment succinctly: “Most of us know LLMs can make things up, but we’re not sure how to systematically prevent it. Our current approach is basically: ‘keep your eyes open, and hope no one junior takes it at face value.” Although participants universally refrained from letting LLM outputs directly influence high-stakes decisions without human oversight, the absence of automated or robust verification processes often led to manual, resourceintensive triage efforts. These accounts map directly to the operational outcome types in Table 6, reinforcing that the risk we study is grounded in observable workflow failures rather than adversarial threat coverage.

Results: Readiness (RQ3)

In this section, we examine how participants described the readiness of their SOC workplace contexts to manage hallucination-related reliability risks associated with LLM outputs in SOC workflows, drawing on the failure-mode taxonomy introduced in Section 5.6 (Table 5 in Appendix E). Our findings reflect participant-reported practices (unit of analysis: participants, N=20), not organization-level maturity estimates. While participants widely acknowledged these risks as a core barrier to trustworthy integration, they described a fragmented set of readiness practices, with limited formalized countermeasures and heavy reliance on human validation. Non-adoption as a boundary condition.In our corpus, non-adoption was analytically meaningful rather than a simple absence of use. Participants who reported no approved workplace LLM use, or who described non-adoption in their teams, associated that boundary with concerns about output reliability, the absence of approved pathways for handling sensitive data, governance restrictions on how such tools could be used in operational environments, and uncertainty that any efficiency gains would justify the added verification burden. We therefore interpret non-adoption as evidence about bounded

6.2 Specific Tactics for Handling Hallucinations Participants reported employing several distinct tactics—ranging from informal manual verification to experimental automated solutions—to address hallucination risks. Below, we summarize these strategies according to 9

6.2.5 Confidence Factors and Cross-Model Checks Few participants explored advanced validation methods involving confidence scores or multiple model cross-checks (n=2). One participant integrated LLM-generated confidence metrics with SIEM metadata, while another advocated parallel querying of multiple LLMs to detect contradictory outputs. For instance, participant P17 stated: “If my local LLM says an IP is malicious but ChatGPT says it’s benign, I investigate further.” Although such redundancy occasionally identified inaccuracies, participants acknowledged this approach as resource-intensive and not foolproof against consistent hallucinations across models.

their prevalence and maturity. 6.2.1 No Formal Mitigation Method Over one-third (n=8) of participants reported that, in their immediate work setting, they did not observe any formalized strategy for managing hallucinations. Instead, hallucinations were handled ad hoc, relying on analyst judgment and manual cross-checks. Participant P14 exemplified this approach: “If we see something that doesn’t align with the logs or feels off, we just know to question it.” Participants attributed this absence of structured methods to limited organizational resources, risk aversion toward autonomous AI, and uncertainty regarding effective mitigation practices. Consequently, hallucination identification depended largely on analyst vigilance and intuition rather than defined protocols, further increasing senior analysts’ workload and verification burden.

6.2.6 Alignment with Established Standards One participant explicitly attempted to validate LLM outputs by referencing recognized security frameworks, such as MITRE ATT&CK or NIST guidelines. If the LLM-generated information lacked identifiable links to these standards, the analyst treated the output as suspect. Nevertheless, this method required ongoing manual verification, as noted by the participant: “I still had to check if the MITRE technique ID it gave me was made up.”

6.2.2 Heavy Reliance on Human Oversight The most common approach reported (n=10) involved extensive reliance on human oversight. Analysts frequently crossreferenced suspicious LLM-generated outputs against internal knowledge bases, log repositories, and external threat intelligence platforms. Participant P09 summarized this practice: “We don’t have an automated safety net, so we treat the LLM output as a lead, not as a conclusion.” Analysts described themselves as human validators, responsible for manually verifying and correcting questionable outputs, a process participants acknowledged was resource-intensive and reduced the efficiency benefits initially anticipated from LLM-based automation.

6.2.7 Structured Output Validation Two participants described early-stage initiatives aimed at validating LLM outputs against authoritative data schemas or internal databases. One employed scripts that automatically cross-referenced IP addresses, hosts, or file hashes with trusted internal databases. Another converted LLM responses into structured JSON for automated schema validation, preventing syntax errors or nonexistent operators from infiltrating SOC workflows. While these structured validations effectively captured basic syntax-level errors, participants admitted deeper factual inaccuracies still required human scrutiny.

6.2.3 Prompt Refinement Three participants explicitly discussed prompt refinement as an effective strategy to proactively minimize hallucinations. Rather than allowing LLMs unrestricted freedom in generating narratives, these participants carefully crafted prompts to constrain responses—for instance, using specific instructions such as bullet points or restricting content solely to provided data. Participant P07 noted a marked reduction in errors when instructing the model explicitly: “I told the LLM, ‘never assume anything we haven’t provided in the logs,’ and the error rate plummeted.” However, participants acknowledged that this approach increased analysts’ cognitive overhead, as effective prompt engineering required specialized skill sets beyond typical SOC analyst competencies.

6.3 Hallucination Mitigation Maturity Rubric We define a Hallucination Mitigation Maturity Rubric to replace informal readiness assessments with a transcriptevidence–based classification of hallucination-mitigation practices (Table 7, Appendix E). Using participants as the unit of analysis (N=20), we coded each participant’s described practices for constraining, validating, and governing LLM outputs (e.g., prompt constraints, review gates, automated checks) and assigned them to the highest maturity level they explicitly described as a routine practice (personally or as an org/team procedure). The rubric spans four levels: L0 = adhoc individual verification; L1 = shared-but-undocumented norms (e.g., routine manual cross-checking); L2 = at least one documented, consistently used preventive guardrail (e.g., prompt libraries/templates or evidence-only prompting); and L3 = at least one proactive, auditable control beyond manual review (e.g., automated validation scripts, schema validation, formal Legal/Compliance sign-off gates, or audit logging of verification). Participants with no active workplace LLM use (so mitigation could not be assessed) are labeled Not appli-

6.2.4 Self-Correcting Loops A minority of participants (n=2) reported experimental attempts at employing self-correcting loops, where the LLM reviews and assesses its outputs to identify inconsistencies or contradictions. Although these methods occasionally flagged logical errors, participants remained skeptical of relying solely on a single model for self-validation, cautioning that “it can just hallucinate a second time.” However, they suggested that combining self-correcting loops with RAG from trusted sources might offer future potential. 10

7

cable and excluded from preparedness claims. Applying this rubric revealed a distribution skewed toward lower maturity levels: Level 1 was most prevalent (30%, n=6), followed by Level 0 (20%, n=4). Fewer participants reported practices consistent with Level 2 (15%, n=3) or Level 3 (20%, n=4). Three participants (15%, n=3) reported no workplace LLM use and are therefore labeled Not applicable (Table 7). Each level reflects the share of participants whose transcripts described practices consistent with that level; it should not be interpreted as an organization-level readiness estimate across the 13 organizations.

Discussion

This study examined LLM integration into SOCs, offering insights into practitioners’ perceptions and usage. While LLMs show potential for automating text-intensive tasks, our findings reveal significant concerns about trustworthiness, transparency, and hallucination. Some observations are likely time-sensitive (e.g., specific product integrations or grounding features). However, the time-stable issues raised by participants—verification overhead, responsibility for outputs, and where LLM suggestions can safely enter workflows—are shaped by organizational process and remain relevant even as tools evolve. We discuss key implications, contextualize findings within existing research, and provide recommendations for deploying LLM-driven security solutions. Several of our findings are consistent with broader literature on LLM reliability and AI-assisted workflows, particularly concerns around hallucinations, prompt sensitivity, verification burden, and continued reliance on human oversight. What appears especially salient in SOC settings, however, is the combination of these issues with operational time pressure, the need to trace claims back to logs, rules, or threat-intelligence evidence, and governance constraints on how model-generated content may enter detection and response pipelines [3, 19, 25]. These factors help explain why participants were more comfortable using LLMs for quickly verifiable, draft-oriented tasks than for investigative reasoning or higher-stakes decision support. How to use the artifacts in practice. The artifacts support three decision points: engineers/leads use the Constraint Matrix to choose safe SOC use cases, management uses the maturity rubric to gate deployment based on guardrails, and analysts use the failure-mode taxonomy to guide verification and escalation of LLM outputs during investigations.

6.4 Stratified Analysis: Operational Perspectives by Role To provide actionable design implications, we conducted a stratified analysis of our findings across four key professional roles identified in our metadata: Front-line Analysts, Technical Engineers, Security Managers, and Tool Developers/Researchers. Table 8 summarizes how these roles differ in their usage, risk perceptions, and mitigation priorities. The study concludes that a one-size-fits-all AI strategy is ineffective for security centers, as different roles demand distinct tool functionalities. To succeed, designers must address three key areas: balancing analyst requirements for speed and sourcing against managerial needs for policy compliance and privacy; distinguishing between tasks suitable for AI—like generative report drafting where verification is easy—and high-stakes investigative reasoning where AI opacity remains a barrier; and providing role-specific features, such as side-byside data comparison for analysts and audit trails for leadership. These specialized focus areas are essential for navigating the current challenges and the future evolution of AI-human collaboration in security operations.

7.1 Addressing Practical Barriers: Data, Integration, and Organizational Hurdles

6.5 Decision-Grade Synthesis: SOC Integration Constraint Matrix

Observed practice: Participants described that adoption is often constrained by data governance, integration overhead, resourcing, and deployment costs, which can stall pilots before full operational integration. Expressed need: They want clearer compliance pathways and reliable internal pipelines that make SOC-specific grounding feasible without relying on informal, case-by-case exceptions. We propose: Treat LLM adoption as a data-and-governance program, not only a model-selection decision. Near-term efforts should prioritize building auditable data pipelines that capture and govern logs, threat intelligence feeds, and analyst annotations under explicit access and usage rules, in close coordination with legal and compliance stakeholders. Related decentralized log-analysis research illustrates one potential architecture for addressing this tension. Federated LogTracer extracts contextual relationships and identifies malicious logs while retaining sensitive telemetry locally rather than consolidating it on a central analysis server [22]. Although our interview study does not evaluate this architecture, it demonstrates

This subsection introduces a single actionable artifact for SOC designers: the SOC Integration Constraint Matrix (Appendix F, Table 10). The matrix summarizes design constraints by mapping common SOC task types to (i) dominant failure modes (FM1–FM7), (ii) verification burden (low/medium/high), (iii) minimum mitigation maturity required (L0–L3), and (iv) a safe integration pattern (e.g., chat-only drafts, template+schema validation, gated deployment). It is populated solely from coded interview evidence: two coders independently performed the mappings and resolved disagreements via the same consensus protocol as thematic coding; unclear cases are marked Insufficient Evidence rather than extrapolated. The matrix highlights a key boundary: tasks with quickly falsifiable outputs (e.g., scripts/rules) can be integrated with lower verification effort, while narrative/investigative decision-support requires higher maturity controls and stronger gating because errors are harder to detect quickly. 11

how SOC-specific contextualization can be pursued under privacy and data-locality constraints. For organizations exploring model localization, we propose evaluating iterative refinement approaches (e.g., human-in-the-loop updating/active learning) [18] and efficiency-oriented fine-tuning strategies (e.g., fractional fine-tuning) [12] as options to reduce operational cost and privacy friction, while remaining explicit that our interview study does not assess their effectiveness or costbenefit in situ. Finally, we propose that executive sponsors and SOC leadership define upfront the data rights, privacy policies, and infrastructure commitments required for pilots to transition into sustained operational tools.

triage, cross-checks against SOC systems, structured outputs for validation, and dual-control for higher-impact actions, as practical ways to bound incorrect outputs without removing human judgment. Expressed need: They want these practices to be repeatable and auditable, rather than dependent on individual vigilance. We propose: Codify these guardrails into runbooks and tooling: (i) adopt an evidence-only policy for prompts and responses; (ii) maintain a curated prompt/checklist library for common triage and drafting tasks; (iii) require structured outputs with schema validation and explicit provenance fields (e.g., referenced log IDs, CTI citations); (iv) apply dualcontrol review for high-impact actions or changes derived from LLM suggestions; and (v) instrument lightweight verification logs that record what was checked and on which evidence [11, 16]. These are process and tooling directions grounded in participant-described practice; we do not claim they eliminate failure modes, particularly under novelty or time pressure.

7.2 Mitigating Hallucinations in High-Stakes Security Tasks Observed practice: Participants primarily managed incorrect outputs through human-in-the-loop verification and crossreferencing against logs, indicators, and internal knowledge. Expressed need: They want mitigations that reduce verification drag while making it clearer when an output is grounded versus speculative. We propose: Evaluate layered mitigation designs that align with these needs: (i) grounding outputs using retrieval over curated internal/CTI sources (e.g., RAG) as a way to attach provenance to claims [15]; (ii) redundancy or cross-checking across models or conventional systems to flag inconsistencies [30]; and (iii) monitoring approaches that periodically re-validate high-impact outputs as context and threat information shift [13]. These are proposed design directions motivated by participant reports; our study does not empirically validate their effectiveness.

7.5 The Impact of Organizational Maturity on LLM Integration Observed practice: Participants described substantial heterogeneity: smaller teams often rely on commodity interfaces for general assistance, while larger enterprises/MSSPs described more structured (but still experimental) integration within SIEM/XDR toolchains under stricter governance. This pattern is consistent with CISO-level evidence showing that enterprise GenAI readiness is generally stronger in upstream governance than in runtime assurance, where limited telemetry and operational procedures leave detection largely humanin-the-loop [23]. Expressed need: Organizations want solutions that match their maturity constraints, low-maturity SOCs need usable guardrails without specialized AI expertise, while highermaturity SOCs need auditability and policy alignment. We propose: Develop maturity-tailored configurations aligned with the rubric (Table 7). For lower-maturity environments, prioritize safe defaults and bounded workflows (e.g., evidence-only prompting, structured templates, and clear escalation boundaries) that reduce reliance on ad hoc judgment. For higher-maturity environments, prioritize API-driven integrations that support governance controls (e.g., access restrictions, audit logging, and approval gates) and allow controlled experimentation under compliance constraints. We also propose SOC-facing AI maturity benchmarks that help organizations self-assess readiness for higher-autonomy use (e.g., expected verification overhead and accountability requirements) without implying that any specific configuration is validated in this study.

7.3 Designing Actionable Explanations and Workflow Integration Observed practice: Participants preferred concise outputs that could be quickly checked against familiar SOC artifacts (e.g., detection rules, known indicators, and raw logs), and were less receptive to long narratives when verification was unclear. Expressed need: They want explanations that are operationally actionable and easy to validate under time pressure. We propose: Design interfaces around verification-first explanations: provide a short triage-ready summary plus a drilldown view that links claims to concrete evidence (e.g., referenced log lines, query results, or CTI items) and maps key elements to recognized frameworks (e.g., MITRE ATT&CK) when applicable. We also propose lightweight feedback channels (e.g., flag/annotate outputs and capture “what evidence was used”) so organizations can iteratively align outputs with local workflows without assuming that feedback automatically improves reliability.

7.4 Operationalizing Human Factors: Codifying Analyst Resourcefulness

8

Observed practice: Participants described a set of operatorled guardrails, evidence-only prompting, checklist-driven

Our study of 20 security practitioners suggests that LLMs are being cautiously integrated into SOC workflows primarily as 12

Conclusion

APPENDIX B. Open Science

supplementary tools. Participants report perceived time savings for text-heavy tasks (e.g., summarization and drafting), typically conditional on substantial verification. However, participants describe limited trust in LLMs for high-stakes security decisions due to unreliable outputs and opaque reasoning. Participants also seldom described standardized, documented mitigation procedures, relying instead on ad-hoc verification norms and continuous human oversight. Overall, these accounts point to a hybrid reality where LLM outputs serve as preliminary leads or drafts that must be corroborated using existing SOC tooling and expert judgment. Using a maturity rubric applied at the participant level, we find readiness is uneven and often reflects informal practices rather than governed processes. Future work should examine how vendorhosted deployment models shape privacy, data-residency, and auditability constraints in production SOC environments.

To support transparency during double-blind review, we provide an anonymized materials package at https://anon ymous.4open.science/r/llm-soc-usability-artif acts-52E4/ (interview guide, codebook, and participantfacing documents). Materials are scrubbed of institutional identifiers and author metadata to preserve double-blind review. Due to ethics and re-identification risk in operational security settings, we do not release raw audio/transcripts, organizational metadata, participant-level coding vectors, or any verbatim transcript excerpts; exemplar snippets in the paper are paraphrased/abstracted. We therefore emphasize process auditability via shared instruments/codebook and make derived outputs explicit and checkable within the paper.

APPENDIX A. Ethical Considerations

[1] Bushra A Alahmadi, Louise Axon, and Ivan Martinovic. 99% false positives: a qualitative study of {SOC} analysts’ perspectives on security alarms. In 31st USENIX Security Symposium (USENIX Security 22), pages 2783– 2800, 2022.

References

This study received institutional ethics approval (protocol details withheld for double-blind review). All participants provided informed consent and spent approximately 35 minutes in total. Relevant stakeholders include participants, their organizations, the broader SOC community and end users, vendors, and potential attackers. Key risks include re-identification, disclosure of operationally sensitive practices, reputational or organizational harm, misuse of findings to evade detection, and misinterpretation that amplifies analyst displacement narratives. We mitigated these risks through data minimization and conservative reporting: no PII was retained in the research dataset; individual and organization names were not kept beyond scheduling; scheduling and remuneration emails were stored separately and deleted after compensation; transcripts were anonymized and stored securely. In reporting, we avoid operationally enabling details, paraphrase rather than quote when sensitivity or identification risk is non-trivial, and release only decision-guiding artifacts (taxonomy, rubric, constraint matrix) rather than operationally prescriptive instructions. Overall, we judge that the benefits of workflow-grounded guidance for safer, more verifiable LLM use in SOCs outweigh residual risks, which are further reduced by not releasing identifying details, operationally sensitive specifics, or raw transcripts. More explicitly, the stakeholders in this work include participants, their organizations, vendors, the broader SOC community, downstream users of SOC outputs, and potential attackers. The main publication risk is not only re-identification, but also that practitioner accounts could be overgeneralized in ways that normalize premature or weakly governed deployment. We therefore frame the contribution as decision-guiding rather than deploymentprescriptive, and we judge publication to be warranted because practitioner-grounded evidence is needed to support safer, more governable LLM adoption in SOC environments.

[2] Tarek Ali and Panos Kostakos. HuntGPT: Integrating machine learning-based anomaly detection and explainable AI with large language models. 2023. URL: https://arxiv.org/abs/2309.16021. [3] Priyanka Badva, Kopo M Ramokapane, Eleonora Pantano, and Awais Rashid. Unveiling the {HunterGatherers}: Exploring threat hunting practices and challenges in cyber defense. In 33rd USENIX Security Symposium (USENIX Security 24), pages 3313–3330, 2024. [4] Virginia Braun and Victoria Clarke. Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77–101, 2006. [5] Constanze Dietrich, Katharina Krombholz, Kevin Borgolte, and Tobias Fiebig. Investigating system operators’ perspective on security misconfigurations. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 1272–1289, 2018. [6] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. LLM agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144, 2024. URL: https://arxiv.org/abs/2404.08144. [7] Gartner, Inc. Gartner Poll Finds 55% of Organizations are in Piloting or Production Mode with Generative AI, 2023. Accessed: 2025-03-18. URL: https://www.ga rtner.com/en/newsroom/press-releases/2023 -10-03-gartner-poll-finds-55-percent-of-o rganizations-are-in-piloting-or-productio n-mode-with-generative-ai. 13

[8] Ali Habibzadeh, Farid Feyzi, and Reza Ebrahimi Atani. Large language models for security operations centers: A comprehensive survey. arXiv preprint, 2025. URL: https://arxiv.org/abs/2509.10858, arXiv:2509 .10858.

[17] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831– 1847, Philadelphia, PA, August 2024. USENIX Association. URL: https://www.usenix.org/conferenc e/usenixsecurity24/presentation/liu-yupei.

[9] Mohammed Hassanin and Nour Moustafa. A comprehensive overview of large language models (llms) for cyber defences: Opportunities and directions. arXiv preprint arXiv:2405.14487, 2024.

[18] Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yun Wang, Peng Zhang, Tun Lu, Toby Jia-Jun Li, and Dakuo Wang. Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks. arXiv preprint arXiv:2311.09825, 2023.

[10] Monique M. Hennink, Bonnie N. Kaiser, and Vincent C. Marconi. Code saturation versus meaning saturation: How many interviews are enough? Qualitative Health Research, 27(4):591–608, 2017. doi:10.1177/1049 732316665344.

[19] William P Maxam III and James C Davis. An interview study on {Third-Party} cyber threat hunting processes in the {US}. department of homeland security. In 33rd USENIX Security Symposium (USENIX Security 24), pages 2333–2350, 2024.

[11] Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL: https://aclanthology.org/P17- 1141/, doi:10.18653/v1/P17-1141.

[20] Jaron Mink, Hadjer Benkraouda, Limin Yang, Arridhana Ciptadi, Ali Ahmadzadeh, Daniel Votipka, and Gang Wang. Everybody’s got ml, tell me what else you have: Practitioners’ perception of ml-based security tools and explanations. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2068–2085. IEEE, 2023. [21] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of GitHub Copilot’s code contributions. arXiv preprint arXiv:2108.09293, 2021. URL: https://arxiv.org/abs/2108.09293.

[12] Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. arXiv preprint arXiv:2410.08020, 2024.

[22] Elnaz Rabieinejad, Ali Dehghantanha, Fattane Zarrinkalam, and Jeff Schwartzentruber. United we log, divided we identify: A decentralized approach for automated log analysis. In Applied Cryptography and Network Security Workshops, volume 15655 of Lecture Notes in Computer Science, pages 70–88. Springer, 2026. doi:10.1007/978-3-032-01823-6_5.

[13] Ms G Illakiya, GV Jeeshitha, R Manjushree, and P Harshini. Ai based hallucination detector. International Research Journal on Advanced Science Hub, 6(12):381–389, 2024. [14] J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics, pages 159–174, 1977.

[23] Elnaz Rabieinejad Balagafsheh, Ali Dehghantanha, and Fattane Zarrinkalam. “Org-Wide, We’re Not Ready”: C-Level Lessons on Securing Generative AI Systems. In 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2026.

[15] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 9459–9474, 2020.

[24] Sayak Saha Roy, Poojitha Thota, Krishna Vamsi Naragam, and Shirin Nilizadeh. From chatbots to phishbots?: Phishing scam generation in commercial large language models. In 2024 IEEE Symposium on Security and Privacy (SP), 2024. URL: https://sp2024.iee e-security.org/accepted-papers.html.

[16] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020.

[25] Aakanksha Saha, James Mattei, Jorge Blasco, Lorenzo Cavallaro, Daniel Votipka, and Martina Lindorfer. Expert insights into advanced persistent threats: analysis, 14

attribution, and challenges. In 34th USENIX Security Symposium (USENIX Security 25), pages 2185–2204, 2025.

Security 25), pages 3827–3844, 2025. URL: https: //www.usenix.org/conference/usenixsecurity 25/presentation/zou-poisonedrag.

[26] Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. Lost at c: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium (USENIX Security 2023), pages 2205– 2222, 2023. URL: https://www.usenix.org/confe rence/usenixsecurity23/presentation/sandov al.

[35] Alexandra Zytek, Dongyu Liu, Rhema Vaithianathan, and Kalyan Veeramachaneni. Sibyl: Understanding and addressing the usability challenges of machine learning in high-stakes decision making. IEEE Transactions on Visualization and Computer Graphics, 28(1):1161–1171, 2021. APPENDIX C. Interview Protocol and Codebook This appendix presents the interview guide and the codebook used for thematic analysis, including frequencies and descriptions. Tables 2 and 3 summarize the analytical framework and recurring themes.

[27] L. Schwinn, D. Dobre, S. Günnemann, and G. Gidel. Adversarial attacks and defenses in large language models: Old and new threats. In NeurIPS 2023 Workshop on ICBINB (I Can’t Believe It’s Not Better), 2023. https://openreview.net.

C.1

Semi-Structured Questions

The following prompts guided all interviews; follow-ups were used to probe for detail. Questions align with the dimensions in Tables 2 and 3.

[28] Yuvraj Singh, N. D. Patel, and Shishir Kumar Shandilya. Enhancing security operations center efficiency through multi-model integration of large language models and siem systems. Research Square preprint, 2024. doi: 10.21203/rs.3.rs-5615639/v1.

1. Can you describe your current role and responsibilities within the SOC?

[29] PeiYu Tseng, ZihDwo Yeh, Xushu Dai, and Peng Liu. Using llms to automate threat intelligence analysis workflows in security operation centers. arXiv preprint, 2024. URL: https://arxiv.org/abs/2407.13093, arXiv:2407.13093.

2. Have you had any experience using LLM-based tools in your SOC? 3. What are the primary benefits you have observed from using LLMs in your SOC?

[30] Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A Malin, and Sricharan Kumar. Sac3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. arXiv preprint arXiv:2311.01740, 2023.

4. What challenges or limitations have you encountered when using LLMs? 5. How do LLMs compare to traditional rule-based systems in performance and usability?

[31] Jie Zhang, Haoyu Bu, Hui Wen, Yu Chen, Lun Li, and Hongsong Zhu. When LLMs meet cybersecurity: A systematic literature review. In arXiv preprint arXiv:2405.03644, 2024.

6. How adaptable do you find LLMs to different security environments and threats?

[32] Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5):593, 2021.

8. Have you encountered hallucinations (incorrect or misleading information)?

7. How often do you update or retrain your LLM models?

9. Are you familiar with hallucination mitigation techniques?

[33] Wei Zhou, Xiaogang Zhu, Qing-Long Han, Lin Li, Xiao Chen, Sheng Wen, and Yang Xiang. The security of using large language models: A survey with emphasis on chatgpt. IEEE/CAA Journal of Automatica Sinica, 2024.

10. How intuitive do you find the user interface of LLMbased tools? 11. How do you provide feedback to LLM-based tools, and how is it incorporated?

[34] Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG: Knowledge corruption attacks to Retrieval-Augmented generation of large language models. In 34th USENIX Security Symposium (USENIX

C.2

Interview Codebook

The codebook, developed from analyses of participant interviews, appears in Tables 2 and 3. 15

APPENDIX D. Participants’ Demographic and Professional Information This section details the profiles of the 20 participants, including their roles, experience levels, and primary security domains. APPENDIX E. Thematic Frameworks and Taxonomy This appendix consolidates the paper’s derived artifacts for auditability. We include (i) the failure-mode taxonomy with inclusion/exclusion rules (Table 5), (ii) hallucination-risk outcome types and reported mitigations (Table 6), (iii) the participant-reported maturity rubric with brief exemplars (Table 7), and (Iv) a role-stratified summary of risks and implications (Table 8). We include short de-identified exemplars illustrating the transcript evidence used to map maturity levels in Table 7: Level 0 relies on individual intuition and occasional “spotchecking”; Level 1 reflects shared informal norms where LLM outputs are treated as leads and manually cross-checked; Level 2 uses documented, reusable prompt templates/libraries with evidence-only prompting; and Level 3 implements governed, auditable controls such as redundancy, structured outputs, and workflow-integrated validation steps.

E.1

Taxonomy Coverage and Role Distribution

APPENDIX F. SOC Integration Constraint Matrix This appendix reports the decision-grade SOC Integration Constraint Matrix referenced in the main text. The matrix maps SOC task types to dominant failure modes (FM1–FM7), expected verification burden (VB), minimum maturity prerequisites (MP), and recommended safe integration patterns (SIP), using IE where interview evidence is insufficient.

16

Table 2: Interview Codebook — Coded Descriptions of Interview Themes Primary Code

Subcode

Freq.

Description

Domain

IT Cybersecurity

11/20

Organizations engaged in threat hunting, rule formulation, or comprehensive SOC operations. Entities specializing in network infrastructure and related security services. Organizations operating within legal frameworks or law practices. Institutions in the public or governmental domain focused on governance and risk management. Organizations in the telecommunications industry delivering or securing network services. Entities within the sports or entertainment sectors that maintain dedicated cybersecurity functions. Organizations in the financial sector including banks and financial institutions. Organizations in the retail food industry with in-house cybersecurity operations.

Technical Task

Non-Technical Task

Task Output Recipient

Current Tools

Network Solutions

2/20

Legal Public Sector

1/20 1/20

Telecommunication

1/20

Sports/Entertainment

1/20

Finance

3/20

Food Retailing

1/20

Threat Hunting

6/20

SOC Analysis SOC Leadership Technical Account Management

3/20 1/20 1/20

Information Security Analysis Anomaly Detection Threat Intelligence Detection Engineering Cyber Operations Management

1/20 1/20 3/20 4/20 3/20

Threat Intelligence Leadership

1/20

Incident Response / DFIR Security Platform Management

7/20 2/20

Security Advisory Model Tuning

1/20 1/20

Dashboard Review Security Research Playbook Development Incident Command Solution Architecture CCR Team Management Data Science in Security

1/20 3/20 1/20 1/20 1/20 1/20 1/20

Management/Coordination Reporting/Governance Stakeholder Interaction

2/20 2/20 2/20

Security Training

2/20

Internal Security Team Site Reliability Engineering Executive Management Platform Engineering

12/20 1/20 1/20 1/20

Datacenter Administration R&D Team Threat/Triage Response Clients Senior Management/Directors Vendors/MSSP

1/20 1/20 1/20 4/20 6/20 4/20

Systematic search for indicators of compromise or anomalous patterns within logs or network traffic. First-line triage and investigation of security alerts. Oversight and coordination of SOC analysts and daily operations. Provision of technical support and consultation regarding security services. Routine evaluation of potential security incidents. Identification of deviations from normal operational behavior. Collection and analysis of adversary tactics, techniques, and procedures. Development and maintenance of detection rules and logic. Strategic management of cybersecurity teams and incident response protocols. Leadership roles that combine operational and research-based threat analysis. Execution of incident response and digital forensics procedures. Administration of security tools such as firewalls, endpoint detection, and cloud security systems. Provision of expert guidance on cybersecurity matters. Customization and refinement of detection models for specific environments. Analysis of aggregated security data and performance dashboards. Investigation of emerging threats and advanced security methodologies. Creation and revision of standardized incident response procedures. Coordination of security incident management. Design and implementation of security systems and solutions. Supervision of teams focused on containment and threat mitigation. Application of data-driven models and AI techniques in cybersecurity. Oversight of security teams, budget management, and vendor relations. Preparation of reports, policies, and ensuring regulatory compliance. Communication with internal and external stakeholders regarding security issues. Delivery of cybersecurity awareness and training programs. SOC analysts and internal staff who receive and act upon security alerts. Technical teams responsible for maintaining large-scale systems. Senior leadership reviewing high-level security summaries. Engineering teams responsible for system infrastructure and platform maintenance. Management of data center operations. Research and development groups focused on security innovation. Specialized teams for threat identification and incident response. External customers receiving security solutions and advisories. Organizational leadership involved in strategic security decisions. Third-party security providers or managed service partners.

Web Security Solutions

12/20

HDR Systems Security Gateways ETL Pipelines Rule Engines Large-Scale Search Platforms Prisma SIEM/XDR Tools

1/20 1/20 1/20 1/20 1/20 1/20 7/20

DLP Solutions Vulnerability Management Cloud Databases Cisco XDR AI-Based Security Tools Forensic Tools Federated Search Threat Intelligence Platforms Anomaly Detection ML Tools Custom Security Tools LLM-Based Security Tools Security Awareness Tools

2/20 1/20 1/20 1/20 2/20 4/20 1/20 5/20 2/20 1/20 2/20 1/20

Next-generation firewalls, endpoint detection and response systems, and cloud security platforms. Host detection and response solutions. Perimeter devices controlling network data flow. Systems for extracting, transforming, and loading security logs. Platforms utilizing manually crafted rules or signatures. Tools designed for indexing and searching extensive datasets. Commercial security solutions referenced for specific functionalities. Integrated platforms for log analysis and threat detection (e.g., CrowdStrike, Microsoft Defender). Systems for preventing data leakage. Tools such as Qualys for vulnerability assessment. Databases employed for storage of security data. Cisco’s extended detection and response suite. Security solutions incorporating artificial intelligence techniques. Software for digital forensics and incident investigation. Unified search systems for aggregating security data. Commercial or public platforms providing curated threat intelligence. Machine learning solutions for identifying anomalous patterns. In-house developed tools for security analysis. Solutions that leverage LLMs for security applications. Systems for managing and organizing security training programs.

Level of LLM Usage

Continuous Use Occasional/Individual Use Not Utilized

11/20 6/20 3/20

Regular and integrated use of LLM-based tools in daily operations. Sporadic use by individual analysts. Organizations that do not employ in-house LLM-based solutions.

Benefits of LLM

Efficiency/Time-Saving Task Automation Support for Lateral Thinking Workload Reduction Enhanced Documentation Data Analysis Simplification Decoding Obfuscated Code Process Mapping Summarization of Texts Standardized Communication Knowledge Augmentation

13/20 5/20 2/20 3/20 3/20 1/20 1/20 1/20 1/20 4/20 1/20 2/20

Automates repetitive tasks, thereby increasing operational efficiency. Enables partial or complete automation in various security processes. Facilitates creative approaches and alternative analytical perspectives. Alleviates routine operational burdens. Improves the quality and consistency of security documentation. Assists in processing and analyzing large datasets. Translates complex technical information into more accessible language. Aids in the interpretation of ambiguous or complex scripts. Summarizes and clarifies complex operational processes. Condenses extensive logs and reports into concise summaries. Ensures uniformity in internal communications. Provides rapid access to best practices and reference information.

17

Table 3: Interview Codebook — Coded Descriptions of Interview Themes Primary Code

Subcode

Freq.

Description

Challenges & Limitations of LLM

Hallucination Risk Overly General Output Lack of Formalized Prompt Guidelines Inability to Maintain Context Variability in Responses Misclassification Errors Requirement for High-Quality Data Inadequate Contextual Coverage Limitations in Context Length Verification Overhead Limited Retraining Frequency Insufficient LLM Expertise High Time Investment Resource Limitations Suboptimal for Incident Response Latency Issues Opaque Decision Processes

11/20 1/20 4/20 1/20 1/20 6/20 2/20 2/20 1/20 4/20 1/20 2/20 1/20 3/20 1/20 2/20 1/20

Potential for generating plausible but inaccurate responses. Insufficient detail or task-specific precision. Absence of standardized prompt design protocols. Difficulties in sustaining topic focus during interactions. Inconsistent outputs from identical inputs. Occurrence of false positives and false negatives. Necessity for curated and standardized data inputs. Omission of critical contextual information. Challenges with processing multi-step or voluminous data. Need for manual cross-verification of model outputs. Infrequent updates leading to potential model staleness. Deficit in specialized skills for managing LLMs. Significant effort required for setup and ongoing management. Budgetary or computational constraints affecting model fine-tuning. Limited efficacy in real-time incident response scenarios. Delays in model inference impacting time-sensitive tasks. Lack of transparency in the model’s internal reasoning.

Factor to Choose LLM

Trustworthiness Task Suitability Available Support Ease of Use Performance Evaluations Time Efficiency Scalability Accuracy

3/20 2/20 1/20 3/20 1/20 6/20 2/20 5/20

Confidence in the reliability of the LLM’s outputs. Preference for language-centric applications. Access to technical support or vendor assistance. Low adoption friction and minimal user effort. Positive industry benchmarks and reviews. Demonstrated improvements in operational speed. Ability to manage increased data volumes or incident frequency. Proximity to ground truth with minimal misclassifications.

Adaptability

General Adaptability High Adaptability Limitations for Novel Attacks Inapplicability for Statistical Analysis Proficient in Data Analysis

3/20 4/20 3/20 1/20 3/20

Limited for High-Certainty Tasks Infrastructure Constraints Dependence on Data Recency High Alignment Effort

1/20 4/20 1/20 7/20

Reduced Effort Compared to Traditional ML Uncertain Adaptability

2/20

Effective performance with standard threat patterns and data. Superior flexibility when provided with high-quality data and context. Reduced efficacy with emerging or obscure threat vectors. Unsuitability for tasks requiring precise numerical computations. Capable of processing logs, IPs, and other structured data for threat detection. May require supplementary tools for exact computations. Usage limited by compliance or environmental factors. Performance contingent on the freshness of the input data. Significant organizational effort needed for policy and procedural alignment. Lower implementation burden when using pre-trained models.

1/20

Potential applicability in niche domains such as digital forensics.

LLM Application at Work

Investigative Assistance Structuring Unstructured Data Script/Regex Generation Drafting Investigation Steps Documentation Transactional Risk Scoring Log Analysis IoC Extraction Malware Detection Support Gap Analysis Risk Assessment Insights Automated Pentesting Detection Suggestions Communication Enhancement Alert Summarization BI Integration Contextual Information Provision Specialized BI Integration Custom Detection Evaluation

3/20 1/20 – 5/20 4/20 1/20 6/20 3/20 2/20 1/20 1/20 1/20 3/20 1/20 2/20 1/20 2/20 1/20 3/20

Provides guidance and preliminary steps for incident investigations. Organizes and clarifies unstructured textual information. Automates creation of small code snippets or expressions. Generates initial checklists and procedures for incident response. Summarizes and standardizes security documentation. Integrates vulnerability and risk data for assessment. Efficient summarization and analysis of large-scale log data. Identifies and extracts indicators of compromise. Assists in the classification or analysis of potential malware. Evaluates the comprehensiveness of detection strategies. Provides summarized risk evaluations for security assets. Generates preliminary exploit paths or test scenarios. Proposes modifications or enhancements to detection logic. Improves clarity in written reports and communications. Aggregates and condenses multiple alerts into coherent summaries. Integrates LLM outputs with business intelligence dashboards. Supplies additional context to support investigative decision-making. Facilitates import of summarized data into BI tools. Assesses and validates custom detection filters.

Comparison to Rule-Based

Superior for Language-Heavy Tasks Time Efficiency Opaque Model Operation Enhanced Pattern Recognition Complexity in Tuning Reduced Coding Requirements Flexible Analytical Perspectives Accessibility Across Experience Levels Minimal AI Reliance

2/20 3/20 1/20 1/20 1/20 1/20 2/20 1/20 3/20

Excels in processing and analyzing unstructured textual data. Achieves faster processing with reduced manual intervention. Operates as a black box relative to transparent rule-based systems. Capable of detecting subtle patterns beyond predefined rules. More challenging to modify compared to manually adjustable rules. Minimizes the need for extensive programming in certain tasks. Evaluates data from multiple viewpoints beyond fixed patterns. Usable by personnel with varying levels of expertise. Some users prefer limited reliance on AI compared to rule-based methods.

Effectiveness Factors

Not Evaluated Feedback Mechanism Customized Benchmarking Consistency Verification Algorithmic Trustworthiness Operational Time Efficiency Industry Feedback Standard ML Metrics Business Impact Metrics Contextual Enrichment Manual Validation

2/20 1/20 1/20 1/20 1/20 1/20 2/20 7/20 1/20 1/20 12/20

Absence of formal evaluation metrics. Incorporates user feedback for continuous model improvement. Utilizes tailored test scenarios for performance evaluation. Ensures stable outputs across repeated queries. Subjective evaluation of model reliability. Reduction in incident response times. Based on external reviews and best practices. Employs precision, recall, and related metrics. Evaluates performance based on customer satisfaction. Provides supplementary context to aid decision-making. Reliance on expert review to confirm LLM outputs.

LLM Explanations & UI

Reliance on Vendor Interface Open Source GPT Interface Scope for Improvement Lack of Transparency Indeterminate UI Evaluation

4/20 7/20 4/20 1/20 3/20

Utilizes the vendor-provided user interface for interaction. Employs a chat-based interface with transparency features. Indicates potential for enhanced integration and error flagging. Perceived opacity in the model’s decision-making process. Uncertainty in assessing interface clarity.

Feedback in UI

In-Tool Feedback Mechanisms Absence of Formal Feedback Submission Alternative Feedback Channels Lack of Structured Feedback Regular Feedback Meetings

4/20 1/20 2/20 4/20 5/20

Features for user corrections and annotations within the interface. No direct feedback channels established with model developers. Utilization of external communication methods (e.g., Slack). Absence of a systematic approach for gathering user input. Periodic review sessions with vendors or partners.

Handling Hallucinations

Absence of Formal Methodology Prompt Refinement Human Oversight Use of Confidence Metrics Self-Correction Mechanisms Retrieval-Augmented Generation Standard Reference Alignment Iterative Output Consistency Cross-Model Comparison Hard Validation Tools Cross-Referencing Techniques Schema-Based Validation

8/20 3/20 10/20 1/20 2/20 1/20 1/20 1/20 1/20 1/20 1/20 1/20

Reliance on ad hoc approaches for mitigating fabricated outputs. Modification of prompts to reduce model guesswork. Continuous expert review of outputs to ensure accuracy. Incorporation of confidence scoring for validation purposes. Implementation of internal re-evaluation of outputs. Grounding outputs using verified external sources. Labeling outputs with established standards for verification. Repetition of processing to confirm stable results. Comparison of outputs from multiple models for consistency. Use of specialized tools for output verification. Matching outputs against known data sources. Validation of outputs against predefined schemas.

Frequency of LLM Retraining

Uncertain No Retraining Periodic Retraining (1–3 Months) Conditional Retraining Infrequent Retraining Event-Driven Updates

9/20 2/20 2/20 2/20 1/20 3/20

Frequency of model updates remains unclear. Use of pre-trained models without further retraining. Regular updates on a short-term basis. Updates deployed based on performance improvements. Occasional model updates. Updates triggered by significant incidents or anomalies.

18

Table 4: Overview of Participants’ Demographic and Professional Information ID

Job Role

Industry

Experience (years)

Security Domain

Detection Method

AI Expertise

Location

P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20

Security Analyst Security Manager Director, Cybersecurity Operations Threat Response & Detection Engineer Solution Architect Security Analyst Data Scientist Security Researcher Security Analyst Security Analyst SOC Manager SOC Manager Security Analyst Technical Lead Cyber Crisis Response (CCR) Manager Security Analyst Technical Account Manager Security Analyst Director, Threat Intelligence SOC Team Lead

Cybersecurity Cybersecurity Food Retail Cybersecurity Network Solutions Finance Cybersecurity Cybersecurity Cybersecurity Telecommunications Finance IT IT IT Finance Cybersecurity Computer & Network Security Legal Cybersecurity IT

3 20 17 14 22 7 6 15 4 3 8 25 3 10 8 1.5 5 4 10 5

Defensive Offensive & Defensive Offensive & Defensive Defensive Defensive Defensive Defensive Defensive Defensive Defensive Offensive & Defensive Offensive & Defensive Defensive Defensive Defensive Defensive Defensive Defensive Defensive Defensive

Both Rule-based Both ML Rule-based Rule-based ML Both Both Both Both ML Rule-based Both Rule-based Rule-based Both Rule-based Both Both

Practical Application Conceptual Understanding Practical Application Practical Application Methodological Knowledge Practical Application Practical Application Practical Application Practical Application Practical Application Practical Application Practical Application Methodological Knowledge Practical Application Conceptual Understanding Conceptual Understanding Practical Application Conceptual Understanding Practical Application Practical Application

Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA Ontario, CA California, USA Ontario, CA Ontario, CA Ontario, CA

Table 5: Transcript-derived taxonomy of LLM output failure modes in SOC workflows, with inclusion/exclusion criteria and severity mapping (consequence-if-acted-upon). Class FM1: Fabricated security facts

FM2: Invalid detection logic / syntax errors FM3: Misleading investigative direction FM4: Incorrect interpretation yielding FP/FN FM5: Opaque / unverifiable claims

FM6: Guessing under novelty / sparse intel FM7: Verification overhead eroding efficiency

Inclusion (what it is) Output asserts specific “facts” (e.g., incident details, IOCs, TTPs) that are not grounded in provided logs/CTI/internal knowledge bases. Output produces nonexistent operators, invalid query/rule syntax, or wrong rule constructs that appear plausible and may be copied into tooling. Output recommends incorrect next steps, prioritization, or triage checklists that steer analysts away from evidence or waste effort. Output misinterprets suspicious data and produces false positives/negatives in labeling, explanation, or triage conclusions. Output provides an answer without traceable rationale (no “why,” no linkage to rule triggers/log lines), increasing verification load and slowing response. Under novel threats (e.g., fresh zero-days) or minimal context, output fills gaps by inference and can become off-base or overconfident. The cost of checking output meaningfully cancels time saved, particularly under time pressure or high-stakes reporting.

Exclude (boundary rule) Clearly labeled hypotheses (“might be...”) with explicit evidence pointers or requests for additional data. Pure formatting/style suggestions that do not change semantics or executable behavior.

Severity High

Benign brainstorming explicitly separated from action steps (e.g., alternatives framed as optional).

Medium

P09: “We treat the LLM output as a lead, not as a conclusion.”

Cases where the model refuses to conclude and asks for more evidence/context. Outputs that explicitly cite evidence (log lines, query results, KB entries) or provide reproducible reasoning steps. Outputs that explicitly decline to speculate and request specialized threat intelligence or additional telemetry. Routine quick checks in lowstakes tasks where verification is minimal and not burdensome.

High

P03, P11, P19: Participants reported persistent interpretation errors where models mislabeled benign patterns as malicious. P05: “With rule-based systems, at least we know which rule triggered. The LLM just gives an answer, and I’m not always sure why.” P19: “when it’s a fresh zero-day or we have minimal intel, the LLM is forced to guess, and sometimes that guess is off-base.”

19

High

Medium

Medium

Low

Representative excerpt(s) P06: “Most of us know LLMs can make things up, but we’re not sure how to systematically prevent it... hope no one junior takes it at face value.” P04: an LLM “invented operators” in detection rule syntax, requiring extensive manual correction.

P04: “We’ll save an hour drafting rules but spend half an hour verifying every line.”

Table 6: Hallucination-risk outcome types observable in our interview corpus. Outcome type (failure) Fabricated facts / entities

Schema-invalid / tool-invalid artifacts Misleading procedural recommendations

Operational consequence (what “risk” means) Plausible misinformation (e.g., fictitious incident narratives, nonexistent IOCs/TTPs) increases verification burden and may misdirect investigation if treated as a lead. Invalid detection content (e.g., rule syntax) causes rework, delays, and can introduce errors into SOC tooling pipelines. Incorrect “what to do next” guidance (e.g., firstpass triage steps) can waste time, increase cognitive load, or propagate error into the workflow.

Example evidence (participants) P06 (fabrication risk; lack of systematic prevention); Section 6.1 examples of fictitious IOCs/TTPs. P04 (“invented operators” in detection rule syntax). P13 (LLM drafted “first-pass triage steps”); participants reporting need for review due to inaccuracies/logical errors.

Mitigation practices reported Evidence-based verification against logs/CTI; treat output as a lead not a conclusion (P09); strict human review and sign-off (P08). Structured output constraints; schema validation; dual-control review for rule changes; human correction loops. Prompt constraints to avoid assumptions (P07); human oversight as final authority (P08); cross-checking via redundancy (e.g., multi-model comparison, P17).

Table 7: Participant-reported maturity rubric for LLM mitigation in SOC workflows (N=20), assigned from interview transcripts (Section 6.3). Not applicable = no workplace LLM use Maturity Level Level 0: Ad-hoc Level 1: Informal Norms Level 2: Tactical Managed Level 3: Governed/Systemic Not applicable

Observable Criteria No shared policy; verification is up to individual intuition; analysts question it only if it feels off. Shared human-in-the-loop culture; informal "never trust" agreements; manual cross-referencing is standard. Documented prompt libraries; task-specific templates; evidence-only prompting requirements. Automated validation scripts; schema validation; formal Legal/Compliance sign-off gates. Organizational readiness could not be assessed due to lack of active LLM usage.

Severity Mapping None; reliance on individual vigilance. Reactive human oversight. Documented preventive guardrails. Proactive, auditable technical controls. N/A

Representative P-IDs P02, P05, P14, P16 P01, P09, P10, P11, P12, P20 P03, P07, P08 P04, P15, P17, P19 P06, P13, P18

Table 8: Stratified analysis of LLM integration and operational risks across SOC roles. Role

Primary Tasks

Critical Failure Mode (Taxonomy) FM1/FM3: Fabricated facts and misleading investigative direction.

Mitigation Priority

Design Implication

Front-line Analysts (e.g., P01, P09, P10, P20)

Log summarization, commandoutput synthesis, initial triage.

Human-in-the-loop: Treating output as a “lead” rather than a conclusion.

Detection scripting, regex generation, logic translation.

FM2: Invalid syntax/operators that break tool ingestion.

Security Managers (e.g., P03, P11, P12, P19)

SIEM strategy, unified dashboards, automated reporting.

FM4: Interpretation errors leading to incorrect escalations.

Structured validation: Schema checks (JSON/YAML) and dualcontrol rule review. Governance gates: Legal/Privacy sign-offs and policy-driven oversight.

Tool Developers/Researchers (e.g., P07, P08)

RAG implementations, prototype development, report synthesis.

FM5/FM6: Opaque reasoning and speculative guessing under novelty.

Conciseness & Citations: Focus on succinct summaries with direct links to raw log evidence to minimize verification drag. Syntax & Schemas: Prioritize structured output formats and automated linting to prevent toolchain corruption. Auditability: Implement verification logs and tiered data classification to manage compliance hurdles. Transparency: Develop execution-trace dashboards and tiered explanation designs to demystify model rationale.

Technical Engineers (e.g., P04, P05, P14, P17)

20

Architectural controls: Retrieval-Augmented Generation (RAG) and multi-model cross-checks.

Table 9: Taxonomy Coverage and Role Distribution (N = 20) Failure Mode

Primary Roles

Salience Summary

FM1: Fabricated Facts

Mentions 11

Analysts

FM2: Invalid Logic FM3: Misleading Direction

4 5

Engineers Analysts

FM4: Incorrect Interp. FM5: Opaque Reasoning

6 7

Analysts/Managers Analysts/Managers

FM6: Novelty/Sparse Intel

4

Researchers

FM7: Verification Overhead

4

Analysts

Triage validity concern. Toolchain disruption. Forensic drag/wasted time. Missed alert risk. Verification bottleneck. Zero-day overconfidence. Value-prop erosion.

Table 10: SOC Integration Constraint Matrix. Rows are SOC task types; columns map each task to dominant failure modes (FM1–FM7), expected verification burden (VB), minimum maturity prerequisite (MP; L0–L3), and a recommended safe integration pattern (SIP). Use IE where transcript evidence is insufficient to justify a mapping. (FM definitions and L0–L3 rubric are provided elsewhere in the paper.) SOC task type Summarization

Dominant failure modes FM1, FM5, FM7

VB Med

MP L1–L2

Detection drafting (rules/queries)

FM2, FM5, FM7

Med

L2–L3

Triage guidance (next steps / prioritization)

FM3, FM4, FM1, FM6

High

L2–L3

CTI synthesis (narrative intel, attribution context)

FM1, FM5, FM6

High

L3

Script generation (automation/snippets)

FM2 ± FM5

Low

L1–L2

21

Safe integration pattern (SIP) Chat-only drafts: treat output as a lead, not a conclusion; require evidence pointers (e.g., log lines, ticket IDs, or source references); include standard uncertainty disclaimers; keep human sign-off. Template + schema validation: structured output (YAML/JSON), linting/unit checks, and peer review; Gated deployment: promote to production only via CI + approval (dual-control). Chat-only assist: brainstorming with explicit uncertainty + alternatives; prohibit auto-execution; enforce “human final authority.” IE for fully automated triage without human sign-off. Gated deployment: retrieval-grounded generation over trusted sources; require citations/evidence trail; cross-check workflow; audit logging for provenance and edits. Chat-only drafts + execution tests: run/verify in sandbox; lint + unit tests before reuse; optional templates for common tasks; never auto-run on production endpoints.

Related documents

• ENGINEERING BUSINESS TRUST INFRASTRUCTURE FOR INTEGRATING LEGAL, CYBERSECURITY, COMPLIANCE, DATA, AND CORPORATE GOVERNANCE ACROSS U.S. ENTERPRISES#438366
• Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching#822442
• CyberPersistBench: Evaluating LLM-Based Cyber Attackers on Installation and Persistence#1131774
• CyberPersistBench: Evaluating LLM-Based Cyber Attackers on Installation and Persistence#1122220
• Enhancing Cybersecurity through AI and ML: Strategies, Challenges, and Future Directions#671773