arXiv CSopen access

A Sociotechnical, Practitioner-Centered Approach to Technology Adoption in Cybersecurity Operations: An LLM Case

arXiv:2604.21679v1 [cs.CR] 23 Apr 2026

A Sociotechnical, Practitioner-Centered Approach to Technology Adoption in Cybersecurity Operations: An LLM Case Francis Hahn∗

Mohd Mamoon∗

Alexandru G. Bardas

University of South Florida

University of Kansas

University of Kansas

Michael Collins

Daniel Lende

Xinming Ou

Univ. of Southern California - ISI

University of South Florida

University of South Florida

S. Raj Rajagopalan Resideo Technologies

Abstract

Traditionally, a key challenge for SOC analysts has been determining whether an alert or finding is a false positive or a true positive [4–6]. This often requires navigating vast, distributed data across multiple platforms and reaching conclusions under strict time constraints [7, 8]. In the context of this high-stress work environment, introduction of any new technology is often viewed with great suspicion. A new tool for SOC could result in disruption of the existing workflow with ultimately little or no real efficiency gain [2]. There has been increasing interest in applying generative AI (GenAI) such as large language models (LLMs) in cybersecurity [9, 10]. In fact, some leading companies are already pushing out products and services that purportedly incorporate GenAI [11–14]. Practitioners’ reception to such technology has been cautious [15], due to well-known problems such as hallucinations and incorrect/inconsistent answers. Despite long-standing efforts to improve SOC effectiveness and reduce burnout [16] through improved processes, tools, and training, research consistently identifies a persistent gap between academic work and operational SOC practice (e.g., [17], [18]). Ethnographic studies show that tools designed without accounting for analysts’ experiential and iterative reasoning often fail in practice, leading to limited adoption or abandonment [19]. This reflects a recurring mismatch between practitioners’ needs and researchers’ assumptions. We draw on Nonaka’s SECI model of knowledge conversion (Socialization, Externalization, Combination, Internalization) [20] and its adaptation to cybersecurity fieldwork through Tool-Oriented SECI [21]. We examine how LLMbased companion systems fit within and extend this knowledge conversion cycle. In particular, these systems generate new explicit artifacts that must undergo trust calibration before analysts internalize and incorporate them in workflow. As part of this effort, we embedded two computer science PhD students in a major commercial organization. Through participant observation and sustained fieldwork, we identified where explicit and tacit knowledge resides and co-developed LLM-based companion tools with SOC analysts to meet operational needs. Using a sociotechnical lens, we analyzed

Technology for security operations centers (SOCs) has a storied history of slow adoption due to concerns about trust and reliability. These concerns are amplified with artificial intelligence, particularly large language models (LLMs), which exhibit issues such as hallucinations and inconsistent outputs. To assess whether LLM-based tools can improve SOC efficiency, we embedded two PhD researchers within a multinational company SOC for six months of ethnographic fieldwork. We identified recurring challenges, such as repetitive tasks, fragmented/unclear data, and tooling bottlenecks, and collaborated directly with practitioners to develop LLM companion tools aligned with their operational needs. Iterative refinement reduced workflow disruption and improved interpretability, leading from skepticism to sustained adoption. Ethnographic analysis indicates that this shift was enabled by our sociotechnical co-creation process consistent with Nonaka’s SECI model. This framework explains the common challenges in traditional SOC technology adoption, including workflow misalignment, rigidity against evolving threats and internal requirements, and stagnation over time. Our findings show that the co-creation approach can overcome these old barriers and create a new paradigm for creating usable technology for cybersecurity operations.

1

Introduction

Security Operations Centers (SOCs) are foundational to modern information security across industry, academia, and government. They serve as the operational hub for monitoring network activity, defending against cyber threats, and supporting regulatory compliance [1, 2]. Effective SOC operations rely on the coordinated interaction of people, processes, and technology, working together to enable timely detection, investigation, and response to security incidents [3]. ∗ Francis Hahn and Mohd Mamoon contributed equally and share first

authorship.

1

adoption patterns and iterative refinement driven by practitioner feedback, resulting in a AI companion platform that supported multiple security tools for the analysts. This iterative engagement mirrored and extended the SECI cycle. Embedded participation surfaced tacit workflow expectations (Socialization); reflection through discussions and interviews highlighted pain points and challenges (Externalization); tool development incorporated these insights into usable tools (Combination); and deployment enabled trust-calibrated internalization within operational routines (Internalization). Over the six-month engagement, our student researchers participated in on-boarding and training, attended meetings, and collaborated closely with SOC analysts and managers. This sustained presence enabled us to observe workflows, tool usage, communication patterns, and breakdowns from an insider perspective. Sustained day-to-day interaction [22] fostered trust, and provided access to candid operational insights typically unavailable to external researchers. By applying a sociotechnical perspective to the challenges faced by cybersecurity professionals, we extend prior work through the explicit use of co-creation as a structured method for tool development and integration. Accordingly, the central research theme of this work can be summarized as a practitioner centered, sociotechnical investigation of how LLM based tools can be co-created, trusted, and embedded into cybersecurity operations to improve workflow efficiency while preserving privacy, security, and operational reliability. Through participant observation, fieldwork, and cocreation, we show that a sociotechnical approach grounded in direct collaboration with cybersecurity professionals improves both the adoption and operational impact of LLMs. Our findings demonstrate how social observation and technical development are tightly integrated. Our contributions can be summarized as follows: • A structured co-creation framework for developing LLMbased tools with cybersecurity practitioners in live operational environments. The approach integrates fieldwork, participant observation, and tool development to improve operational relevance and impact. • Illustrate how thematic analysis supports the systematic identification of workflow pain points, operational challenges, and analyst reception to the process. • Provide insights into adoption and long-term sustainability through a staged integration strategy that incrementally built trust and operational alignment within an industry SOC. • Extend Tool-Oriented SECI by identifying generative recombination and trust-calibrated internalization as defining characteristics of LLM-mediated knowledge conversion.

2

tools for security operations, and analyst-centered work on trust, explainability, and workflow integration. Embedded Sociotechnical Studies in SOCs. Prior research at the intersection of cybersecurity, sociotechnical systems, and researcher-practitioner collaboration has consistently documented gaps between academic security research and operational practice [23]. Ethnographic studies show that SOC work is practice-driven, socially organized, and shaped by informal coordination as much as by technical tooling [24]. Sundaramurthy et al. [24], drawing on a 3.5-year multi-site anthropological study, model SOC work as an activity-theoretic system shaped by tensions among tools, procedures, and organizational expectations. Their findings emphasize that meaningful innovation requires embedded researchers who understand analysts’ tacit knowledge and operational constraints. Previous work also identifies sociotechnical barriers to AI adoption in cybersecurity. Roch et al. [15] show that although experts are willing to use AI for repetitive and dataintensive tasks, adoption is limited by concerns about trust, transparency, determinism, and data protection. Analysts favor gradual trust-building through low-risk task delegation and calibrated autonomy. Similarly, Fung et al. [25] emphasize that trust, workflow alignment, and secure, closed deployments are critical for successful integration. These studies highlight that effective AI adoption in security contexts depends on trust calibration and organizational fit. Beyond SOCs, Tuladhar et al’s [26] insider ethnography illustrates how secure practices become organizational norms only when learned in context and reinforced through hands-on collaboration with security advocates. Vielberth et al.’s [27] systematic review similarly finds that SOC research remains fragmented, often relying on interviews or isolated case studies and lacking holistic sociotechnical frameworks. Greig et al. [28] further document persistent misalignments between formal security policies and everyday practice. Collectively, this literature establishes that organizational cybersecurity policies are enacted through people’s workflows, stories, and peer norms rather than through tools alone. However, while these studies richly characterize SOC work, they do not examine how new AI tools are shaped through co-creation or integrated into live workflows of cybersecurity professionals over time. As a result, we lack empirical understanding of how sociotechnical dynamics influence the adoption of LLM companions in operational security settings. AI/LLM Tools for Security Operations. Recent research has explored how AI, and more recently LLMs, can support tasks across the security operations lifecycle. Prior work has largely focused on applying machine learning to detectioncentric problems such as alert ranking, anomaly detection, and log analysis, with mixed success in operational settings. Srinivas et al. [29] presents a comprehensive taxonomy of AI applications in SOCs, cataloging use cases ranging from log summarization and alert triage to vulnerability management. Despite this breadth, they highlight persistent chal-

Related Work

This section reviews prior work across three areas: embedded sociotechnical studies of SOC practice, AI and LLM-based 2

lenges including interpretability, robustness, hallucinations, and legacy system integration that continue to limit operational fit. Binbeshr et al. [30] similarly note that while MLdriven approaches dominate the literature and promise improved detection accuracy, but their deployment remains constrained by data quality, explainability, and trust. Empirical studies consistently show that practitioners view AI systems as augmentative rather than autonomous. Mink et al. [31] find that analysts prefer ML tools that complement existing rule-based workflows and provide contextual explanations that support verification, rather than opaque predictions. In the LLM context, Singh et al. [32] reported from a 10-month longitudinal study of 45 analysts that GPT-4 was primarily used as an on-demand sensemaking aid for interpreting commands, scripts, and telemetry rather than as a decision authority. These findings suggest that LLMs are most valuable when embedded into analysts’ reasoning processes, rather than positioned as replacements for expert judgment. Recent systems demonstrate the growing interest in LLMbased security tooling. Kramer et al. [33] examined the use of LLMs for incident response summarization. Through controlled experiments with 18 analysts and 50 real-world incidents, they show that fully autonomous LLM-generated summaries frequently omit critical details or introduce factual inaccuracies, but that collaborative, analyst-in-the-loop use can improve readability and consistency while reducing effort. Importantly, their work evaluates LLMs primarily as post-hoc summarization aids, and focuses on controlled experimental settings rather than live, evolving SOC workflows. Other work explores LLMs as task-oriented security assistants. Deng et al.’s PentestGPT [34] demonstrates LLM driven orchestration of penetration testing workflows by chaining tools and reasoning steps, while Shashwat et al. [35] explore early applications for security analysis and triage. Although these systems highlight LLM flexibility as reasoning and orchestration layers, they are largely evaluated in isolated or simulated settings, with limited focus on organizational constraints, analyst trust, or sustained adoption. Microsoft’s Copilot for Security further illustrates the scalability of LLM-based approaches, reporting improvements in aggregate triage and remediation metrics [11]. However, such evaluations largely emphasize system-level performance and leave unanswered questions about how these tools reshape individual analyst workflows, decision-making practices, and accountability in day-to-day operations. Across this literature, LLMs are shown to be promising sensemaking and productivity aids. Yet none of these studies attempted to investigate LLMs’ adoption in real operational environments through zero-proximity observations and active researcher interventions. Analysts and Sociotechnical Factors. A body of research emphasizes the importance of analyst-centered design, trust, and workflow integration in security tooling. Rastogi et al. [36] show that generic explainability techniques often fail to

meet analysts’ needs in high-pressure SOC contexts, where time constraints and alert volume demand concise, actionable explanations. Nyre-Yu et al. [37] report that a deployed explainable-AI tool saw limited use and did not improve analyst accuracy, underscoring the importance of aligning system outputs with users’ roles, responsibilities, and trust thresholds. These findings echo broader human-AI collaboration research, which emphasizes maintaining user authority, calibrating trust, and embedding AI outputs into established practices rather than displacing them. However, prior work typically examines trust and explainability as static design properties, rather than as outcomes negotiated through sustained use and organizational context. Our Work. In contrast to prior work, our study integrates embedded ethnography with the co-creation and in-situ deployment of LLM-based companion tools within a live SOC. This design enables us to examine how analysts perform their work, and how LLM tools are iteratively developed, their outputs negotiated, validated, and internalized within operational workflows over time. By tracing this integration process, we address a gap in SOC and AI-for-security research, which often evaluates systems outside the contexts in which they must ultimately function.

3

Methodology

This study examines the design and integration of LLM-based companion tools for domain-specific security tasks within a live SOC. Using a sociotechnical approach, researchers were embedded within an industry SOC to understand workflows, norms, and operational constraints through fieldwork and enculturation [38]. Co-creation with practitioners provided context-rich feedback that guided iterative development. SECI-Informed Research Design. Our methodological approach was informed by Nonaka’s SECI model [20, 39] of knowledge conversion and its tool-oriented adaptation for cybersecurity fieldwork [21], see Figure 1. Rather than treating ethnographic observation and tool development as separate activities, we structured our field engagement as an iterative cycle of knowledge conversion mediated by tool building. Overall, (i) fieldworkers acquired tacit operational knowledge through embedded participation (Socialization: tacit→tacit), (ii) surfaced and articulated implicit pain points and challenges (Externalization: tacit→explicit), (iii) formalized these challenges into usable tools (Combination: explicit→explicit), and (iv) evaluated how these tools shaped practice through deployment and iterative adoption (Internalization: explicit→tacit). Ethnographic Approach. Participant observation, fieldwork, and co-creative engagement were central to our ethnographic approach. Through close engagement, we examined how SOC analysts perform their work, integrated tools in their workflows, and coordinate within their environment. By partic3

Social Acceptance by the Community of Practice

Apprenticeship

Socialization

Internalization

Tacit Knowledge

relied on the IA’s organizational knowledge and hierarchical position to navigate internal dynamics, communicate goals with management and team members, and identify key pain points and high-priority issues.

Combination

Explicit Knowledge

Models, algorithms, and tools

SECI Phase 2: Externalization (Tacit→Explicit). To externalize tacit understanding acquired through the embedding, the fieldworkers conducted structured reflection and reconstruction of analyst workflows. This reflexive step was supported by (i) workflow walkthroughs (step-by-step demonstrations across vendor platforms and internal tools), (ii) debrief discussions following observations and demo sessions, and (iii) targeted questioning to surface day-to-day challenges and pain points. Most important, the fieldworker participated in the same tasks as the other analysts. This hands-on experience is necessary to the reflection and reconstruction process. Externalization outputs included written workflow narratives, reconstructed investigative models (e.g., query-building sequences, cross-system data access patterns), and explicit articulation of “pain points” such as repetitive log correlation, and manual cross-referencing across systems.

Externalization Questioning, Reflection, and Reconstruction

Figure 1: Tool-Oriented SECI Model (based on [21]) – SECI adaptation for cybersecurity fieldwork where ethnographic insight is translated into operational tools that embed explicated tacit knowledge back into analyst workflows. ipating in peripheral tasks [22] and encountering situated challenges [40], we gained firsthand insight into operational constraints. These experiences, combined with interviews and informal discussions, provided a context-rich understanding of the daily challenges analysts face in security operations. Beyond observation, fieldworkers sought participation in analyst tasks. This included requesting access to relevant databases, vendor platforms, and APIs, and submitting proposals to deploy LLM-based tools within the organization’s infrastructure. This provided hands-on exposure to the technical and institutional constraints shaping daily operations. Iterative Feedback Loop. A key component of our ethnographic methodology was a structured feedback-loop process (Figure 2) that incorporated insights gathered during demonstrations, shadowing, and live tool use. The loop enabled continuous refinement of both the tools and fieldworker’s understanding of analyst work. During each iteration, they adapted and tuned LLM configurations to specific use cases and conducted comparative evaluations across models to determine suitability prior to deployment. SECI Phase 1: Socialization (Tacit→Tacit). During the first month of engagement, the fieldworkers sought to develop an external perspective on the security team while gradually building trust through sustained social interaction. Because the environment was fully remote, traditional forms of participant observation, such as informal conversations or spontaneous workplace interactions, were absent. Instead, observation was conducted through Microsoft Teams meetings, screen sharing, and text-based communication. Organizational artifacts, including hierarchy maps, were used to understand reporting structures, identify stakeholders, and coordinate engagement across geographically distributed teams. The Internal Advocate (IA). The engagement began by connecting with members across the organization, facilitated by an internal advocate (IA) who provided legitimacy to the fieldworker’s presence and helped expand their access within the organization [41, 42]. Early in the embedding, the researchers

SECI Phase 3: Combination (Explicit→Explicit). In our fieldwork, the combination phase manifests as the creation of AI companions that incorporate the externalized tacit knowledge. Drawing on insights from participant observation and fieldwork, the fieldworkers identified several LLM-supported tools to address tedious, repetitive tasks involving widely distributed or difficult-to-interpret data within the SOC and pentesting teams. These tools targeted Root Cause Analysis (RCA), Asset Discovery, and Query Building. Development progressed iteratively, with incremental demonstrations during team stand-ups to share updates, gather feedback, and discuss security implications and future design directions. Collecting Data. The systematic recombining of explicit artifacts was enabled by document collection from the SOC analyst team at different stages of tooling co-creation. In the initial stages the collected data existed as static resources, including RCA playbooks and query examples from the security vendor. After evaluating initial tool performance, data collected was supplemented with custom examples derived from fieldwork insights. In the final phase, analysts contributed realworld examples directly. Combination consisted of systematically recombining explicit artifacts collected from the field (playbooks, internal documents) with explicit reconstructions produced during Externalization (prompt templates, query patterns, RCA reports, and so on). SECI Phase 4: Internalization (Explicit→Tacit). The internalization process is realized through deployment and itearative co-adaptation of the developed AI companions. Our co-creation approach followed an iterative design process grounded in continuous engagement with security team members, drawing on their tacit knowledge and making it explicit through discussion and collaboration [43]. Regular testing and demo sessions were used to evaluate usefulness, identify 4

Months 4 to 6

Months 1 to 3 Socialization

Externalization

Combination

Initial Engagement

Hands-On Experience

Tool Development

User Feedback

(Duration: 1-2 month) • Handle incidence response tickets • Develop code for API systems • Engage with sec. vendor support teams • Write AI internal proposals to allow for tool deployment • Perform web API testing on training ground and organizational targets

(Duration: 1-3 months) • Develop tools using feedback gained over the course of engagement • During initial tool development have weekly stand-ups with small cohort of members to help direct development • Select target use-cases found during fieldwork for teamwide demonstration

(Duration: 2 months) • Demonstrate to the entire team and have group-wide discussion of use cases and features they like and would like to see • Take feedback and iterate on the development of the tools • Shadow team members as they use the tools; discuss how it fits into their workflow

(Duration: 1 month) • Engage with various security teams • Discuss goals&outcomes with mgmt.

Participant Observations (Duration: 1 month) • Shadow teams to understand workflows • Show general tool to begin conversations

Internalization

Figure 2: Feedback Loop – Six-month embedded fieldwork and co-creation timeline showing how ethnographic immersion, hands-on SOC work, and iterative user feedback formed a continuous feedback loop for tool design and refinement limitations, and refine features for integration into daily workflows. This deliberate co-creative process ensured that the resulting tools reflected both the fieldworker’s insights and the practical expertise of cybersecurity professionals. In this study, co-creation is a process in which researchers and cybersecurity professionals jointly develop, evaluate, and refine LLM-based tools while simultaneously synthesizing knowledge through close, ongoing collaboration [44, 45]. Participant Recruitment. Participants included 18 SOC analysts, including CISO and the SOC director, and members of adjacent security teams (e.g., vulnerability management and pentesting). Sampling was purposive and within the constraints of organizational access, prioritizing roles directly involved in alert handling, query construction, and tool use. Thematic Analysis. Both fieldworkers independently coded field notes, meeting transcripts, feedback, and observational artifacts using an inductively developed codebook that was iteratively refined throughout the analysis. They iteratively applied the codebook until no new codes and themes emerged [46]. The codebook is included in the appendix.

4

siloed data sources, and ad-hoc task assignments. Many tooling constraints were rooted in organizational policies that dictated how cybersecurity professionals were required to perform their work. Throughout the embedded study, the fieldworkers identified several recurring workflow gaps, including: 1. Developing automation workflows that leverage organizational data not accessible to the vendor’s platform. 2. Handling large quantities of data, logs or source code, that is often obfuscated or unintelligible. 3. Building complex query statements for the platform. 4. Handling distributed data across multiple platforms. 5. Maintaining the security vendor’s visibility of assets. Root Cause Analysis (RCA). RCA is an event that takes place between the SOC, legal, management, and any other team which have been the victim of an incident which resulted in the loss, alteration, or unintended addition of data to the organization’s data stores. This has been marked as a time critical event which involves participants with a widerange of technical background, perspectives, objectives, and involvement in the incident. Through our discussions with both the legal team and the SOC we identified RCA as a strong use case where LLMs could assist. The idea to use LLMs to assist with RCA was shared by everyone due to the time critical nature of the event and the need to summarize the data being analyzed for the various perspectives and objectives. Due to the sensitivity of the data, it was often not something the security platform could assist with directly, but would be involved for collecting or searching for data. The RCA tool was designed to assist with gaps (1), (2), and (4).

LLM Opportunity Identified through Ethnographic Study

By working alongside the organization’s security team, the PhD researchers learned operational norms, practices, and constraints necessary for developing impactful AI- and LLMbased tools. At the outset, the organization’s exposure to AI was largely limited to vendor-provided modules. To acquire the situated knowledge required for effective tool design, we positioned fieldworkers as learners in close collaboration with experienced analysts. Through the ethnographic study, we identified a number of pain points in the operation that offer opportunities for LLM adoption.

4.1

Query Builder. During the fieldwork the embedded researchers noticed a need for assistance with building query statements to search for log and event information when performing incident response for the alerts they were assigned using the security vendor’s security information and event management system (SIEM). During the time of the fieldwork, the security vendor released an AI-powered query builder for assisting with performing query statement construction. How-

Tooling Gaps

Across discussions, a consistent theme emerged: limitations and bottlenecks caused by the security vendor’s platform, 5

Supply Ticket or Alert Description

Process Relevant Log Data

Perform Nmap Scan Perform VirusTotal Scan

Evidence-Free Summarization

Evidence-Based Summarization

LLM

LLM

Figure 3: Demonstration SOC Tool Workflow – End-to-end pipeline showing how alert inputs are processed, enriched with external scans, and summarized by an LLM with and without supporting evidence.

5

ever, due to limited visibility into critical organizational event logs and the inability to supply this information, the query builder was only able to assist in constructing query statements similar to examples found in online documentation. Discussions with analysts revealed that their experiences with the tool were similar. This led the fieldworkers to consider developing an internal query-building tool capable of leveraging user-supplied examples and storing them for future use, creating a living database that would expand the available knowledge base over time. The query builder was initially designed to address gaps (1) and (3), and during the iterative refinement phase, it was extended to address gap (2).

Co-Creating LLM Solutions

We examined AI/LLM tool adoption by co-creating tools alongside analysts and observing how they were incorporated into practice. Tools were developed in response to analysts’ expressed needs and field observations, then iteratively demonstrated during weekly or bi-weekly sessions. This process enabled analysts to guide development without adding to their workload or disrupting existing workflows. Over a six month period, we developed three security tools and one integrated security platform. To supply the tools with the explicit information of the SOC analysts we relied on RAG (Retrieval-Augmented Generation) [47,48], a technique in LLM application development where documents that have been turned into vector embeddings are appended to the query prompt to add additional context to the LLM during inference. We use RAG and the associated vectorstores for serving the SOC-specific data to the LLM during response generation. This section focuses on the platform and three tools deployed within the SOC (see Figure 4), highlighting how early workflow disruptions led to non-adoption and how iterative refinement, grounded in analyst feedback, ultimately supported sustained integration.

Asset Discovery. Following the demonstration of the model SOC tool, discussions with a vulnerability management analyst surfaced interest in the system’s data processing pipeline, particularly how structured data from Excel files was parsed, aggregated, and prepared for LLM-based summarization. These discussions prompted closer examination of the organization’s asset discovery workflow. The workflow involved polling the security vendor’s API to retrieve agent status information, providing visibility into organizational assets. Retrieved data was then correlated with records from internal databases (e.g., Azure and ServiceOne) and consolidated into remediation emails sent to asset owners to address reporting inconsistencies. To develop a grounded understanding of this process, fieldworkers transitioned from observation to direct task execution, including implementing API interactions and working with the relevant databases. This hands-on engagement enabled end-to-end understanding of the workflow and informed subsequent tool design decisions. Participant observation and fieldwork revealed several recurring challenges. Data was highly distributed across multiple systems. It was siloed and often inconsistent. Analysts needed to perform recurring API calls to an application at fixed intervals. They also had to aggregate and summarize the collected information in order to produce routine reports and track remediation actions for identified issues. These observations led to discussions about designing and developing an asset discovery tool to address gaps (1), (2), (4), and (5).

5.1

The LLM Security Platform

An initial demonstration of the security tools sparked discussion of not wanting the tools to be widely dispersed or distributed across multiple platforms or domains. To achieve this design requirement LangChain framework was used which has a feature called the ReAct chain [49] which provided tool dispatching using LLM-based reason and action generation using supplied context for registered tools. Two issues were observed during the development stage: (1) due to the demand for GPUs supplied by our cloud service provider we were only able to obtain a 16GB slice of a NVIDIA A10 GPU, and (2) during initial development we observed highly variable ReAct responses between two models, Mistral:7B [50] and Llama3.2:8b [51]. 6

Role

Task

Pain Points

SOC Analyst

Root Cause Analysis

Tight time requirements to parse various amounts of distributed data. Can be challenging to decide on root cause when human error is involved. Can cause scheduling conflict if analysis needs to be conducted in parallel with existing tasks.

SOC Analyst

Query Building

There are of a variety of query languages for SIEMs. The SIEM used by the organizations provides example, but they are reported and observed to be limited in-scope. A query builder was provided by the security vendor for their platform, however due to the lack visbility of specific organizational resources limits the creation of readily available query statements.

Vulnerability Management

Asset Discovery

Requires multiple highly repitive tasks to complete by analyzing distributed data sources. A highly critical and time sensitive task. Requires the use of API code to collect necessary reporting data.

Table 1: Operational Tasks and Pain Points – SOC roles, recurring tasks, and workflow bottlenecks identified through embedded fieldwork, which informed the selection of initial targets for LLM-based tool development. Response Time Testing. The impact of RAG parameters on performance was evaluated by varying k_fetch (1, 3, 5) and k_retrieval. Increasing both parameters improved response quality but also increased latency. For the evaluated dataset, setting both values to 3 provided a balance between accuracy and response time, producing outputs that effectively incorporated retrieved context.

These observed issues led to us performing a comparative test of the LLM-platform to decide which model would perform best under our constrained environment while ensuring consistency of output using the ReAct Chain. To facilitate trust in the tools, the LLM was designed to perform correct and predictable tool dispatching while producing meaningful and usable outputs. The tests were conducted using a Macbook Pro M2 32GB and led to the use of the Mistral:7B model. Although the hardware resources were fairly modest, the needs oriented design and practical utility of the platform proved to be a decisive factor in the SOC environment. The result of the comparative tests can be found here.

5.2

5.3

Agent Design

There were four agents used in this work, three of the agents were explicit and one of the agents was implicit. The agents were designed to sub-divide the task, otherwise know as task decomposition [52], the three explicit agents we utilized were (1) the asset discovery agent, (2) the query builder agent, and (3) the MCP agent. The one implicit agent was (4) the ReAct agent which was used to decide which of the three explicit agents were to be used. Query Builder. The tool was designed to assist with constructing query statements by allowing a locally run model within the organization’s infrastructure that wouldn’t violate confidentiality requirements as stated by the AI governance. The initial design of this tool was to take user prompt for a specified query statement request and using a RAG system which would access a vectorstore populated with examples of query statements as the knowledge base it would inform the LLM how to predict which tokens to use for generating query statements. A key feature for using RAG and vectorstores for this generation approach is that the knowledge base vectorstore would grow through repeated usage, this feature was discussed numerous times with analysts on how it would provide the analysts a growing database of shared experience. The process for vectorstore retrieval worked by taking the query prompt and retrieve the three most relevant documents

Model Testing

We evaluated four models, Llama 3.2 3B (Meta), Mistral 7B (HuggingFace), Qwen3 8B (HuggingFace), and GPT-OSS 20B (OpenAI), to assess reliability, tool-calling accuracy, and response time. Five representative queries were used to test correct tool invocation and end-to-end latency. To stress-test tool selection, we introduced additional tools with similar contextual prompts. We also evaluated performance under varying RAG configurations by adjusting k_fetch and k_retrieval. Tool Call Testing. Tool-calling accuracy varied across models. GPT-OSS 20B correctly selected the intended tool in Query 1, while other models misrouted to Document Analysis. All models correctly invoked the MCP agent for Query 2, with Mistral 7B achieving the lowest response time. For Query 3, all models selected correctly, but GPT-OSS 20B exceeded the 20-minute runtime limit and was terminated. In Query 4, all models performed correctly except Llama 3.2 3B, which again misrouted. For Query 5, all models correctly utilized RAG. Notably, GPT-OSS 20B occasionally relied on pre-training knowledge instead of the RAG system. 7

Organization’s Data

artifact through file upload or paste mode. If the input is structured (JSON/YAML), the tool normalizes it into a consistent schema (e.g., event type, timestamp, users, IPs, host/asset identifiers, process and URLs/domains). If the input is unstructured, the system preserves the raw logs and uses them directly as the incident representation. To ground outputs in organizational guidance, the tool supports an optional RAG workflow. Analysts can upload playbooks, policies, and prior RCA documents, which are chunked and indexed into a persistent Chroma [54] vectorstore using local embeddings. For each incident, the tool constructs a retrieval query from available indicators and performs hybrid similarity search to select top-k context chunks, which are embedded into the RCA prompt with citation anchors. The RCA generator produces a structured report containing: (1) what happened, how, and why; (2) ranked contributing factors with explicit evidence; (3) mitigations scoped by time horizon (0–24h, 1–7d, >7d); (4) data gaps and assumptions; and (5) advisory triage tags and suggested MITRE ATT&CK [55] mappings. Prompt constraints require the model to reuse concrete indicators from the incident artifact and to state “Not available in provided data” rather than inferring missing fields. This was done to keep the outputs in check and avoid hallucinations. Finally, the tool logs key interaction events (input ingestion, retrieval metadata, generation outputs, and analyst feedback) to an append-only JSONL audit trail to support iterative evaluation and refinement.

SOC Analyst Query

LLM ReAct Chain Agent

Vector Store

Query Builder Agent

Asset Discovery Agent

LLM

LLM MCP Server

Usable Output Figure 4: LLM Security Platform – The platform receives an analyst’s query and uses an LLM with a ReAct-based agent to determine the appropriate tool or action. Selected agents interact with organizational data and vector stores via the Model Context Protocol (MCP), producing a structured, usable output for the analyst. to the user prompt from the vectorstore and supply it to the query. The LLM would then generate the query statement and supply it to the user. Asset Discovery. The asset discovery tool was designed to automate the task of detecting which assets on the organization network were not properly reporting to the security vendor’s platform. To accomplish this, the tool was designed to accept a user query for asset discovery. It then issued an API call to the API tool registered with the MCP server in order to retrieve information about the asset’s reporting history. It would return back this information and then process the log data to determine the reporting status, checkpoints based on status history were created to prevent creating duplicate flaggings. The tool would then take the assets name from the API data and make another API call via the MCP server to access the datastores which contained the assets name, system information, and corresponding asset owner information. The LLM would use all of the collected information to summarize and draft an email for all assets currently marked as “not reporting” or “stale” and wait for for the user to respond with modifcation requirements or to send.

5.4

5.5

Early Integration Challenges

The initial development of the query builder used examples for constructing query statements from the security vendors website, where the examples were obtained using a webscraper and then processed into a uniform format, within that set of examples there was one hand-crafted example by the embedded researchers based on their experience for conducting an investigation of an alert regarding excessive login failures. Initial Demonstration and Use. The initial demonstration to the SOC team used both security-vendor examples and a researcher-crafted example. After deployment, we observed that LLM queries not informed by the researcher-crafted example produced lower-quality outputs, leading to negative feedback and workflow disruption. Analysts reported similar issues with the vendor-provided query builder, indicating that the examples used to guide query construction were critical to output quality. Another issue that was observed during the analysts initial use of the query builder tool was that it was incapable of ingesting alert specific information, this resulted in query statements that had generic or hallucinated values within the parameter fields. It was possible to provide these values through the prompt window, but it was noted that supplying log information and prompting was undiserable. A pain point for the RCA tool was the limited access to

Root Cause Analysis

The RCA tool was implemented to help analysts convert heterogeneous incident artifacts (JSON/YAML objects or raw log text) into structured, SOC-ready incident narratives. To satisfy confidentiality and governance requirements, all generation is performed using locally hosted Ollama [53] models so incident data remains within the organization’s infrastructure. The RCA pipeline begins by ingesting a single incident 8

historical RCA reports. Formal RCA documents are rarely produced, typically only for severe or escalated incidents. This limited access constrained our ability to ground early prototypes in real-world reference material or to train and validate the tool against a large corpus of completed analyses.

were understanding and describing the security implications of deploying the MCP with access to internal API calls that allowed for observing which assets on the organization’s network were properly reporting to the security vendor’s platform and which assets presented a blindspot. The non-technical aspects were the planning and design of a proposal which described how we would mitigate and prevent actions that would allow the LLM and the MCP server to be used to violate security, privacy, and trust. This led to various discussions with the AI governance on how we would use traditional methods of authentication and authorization for securing the use of the LLM and MCP server. While these discussions did not lead to an additional approved proposal the analysts were still using the tool with performance issues from the GPU limitations and it being a manual process due to the lack of API support from the MCP server. Improvements to the RCA Tool. Development for the RCA Tool was through an iterative refine-and-test cycle. The prototypes were deployed within the SOC, and analysts used them on real incidents over multiple iterations. After each trial, feedback was gathered through observations, think-aloud sessions, and debrief interviews, then rapidly incorporated changes. Each iteration brought the tools closer to the SOC’s needs. By the second demo, the analyst’s feedback had grown more positive, they reported that LLM suggestions were becoming more relevant and saw clear time-saving benefits, coupled with remaining requests like improving the UI/UX and adding some guardrails for the LLM outputs. This participatory refinement aligned the solutions with user expectations.

Limited Benefits. Development of the asset discovery tool was disruptive by a challenge in policy. As development of the asset discovery tool progressed, there was a need for the use of emerging technologies and development methods. In the case of the asset discovery tool we found after initial development that providing the information to the LLM manually, did not aid the analysts in meaningful way. The quantity and repetiveness for the asset discovery workflow in a sense demanded automation and to achieve this it required the LLM to access the data provided by the API autonomously. However, our initial proposal to the organization’s AI governance team did not include accessing resources from assets beyond what was available to the server which hosted the LLM. This presented a pause in development and led to a slow down in the tools adoption while a proposal and policy was developed for autonomous internal-API calling by an LLM.

5.6

Workflow Aligned Iterative Refinement

Feedback from tool usage and observations during analyst shadowing informed the refinement of features. This increased analyst control over data supplied during prompting, incorporated workflow-derived query examples, and guided discussions around policies and requirements for securely enabling autonomous LLM interactions with internal APIs.

6

Improvements to the Query Builder. The first step taken towards improving the query builder tool was adding in an additional text window for supplying logs, this fufilled the request made by the analysts to seperate where to supply logs and the prompt. This modification added the supplied text to the data pipeline for the RAG system thus using the data supplied as an additional document to retrieve from for future use. The next step was adding analyst specific example documents to the RAG vectorstore. However, getting the documents took time, and getting the examples from the analysts took more time because this was an added task to their already busy routine. These changes prompted discussions with the SOC director, during which the proposed template and its benefits for providing examples were presented. Following this, analysts were instructed to contribute workflow-derived examples. Together, these changes were associated with higher observed adoption and were reported as substantially less disruptive than earlier iterations. Analysts reported more frequent use of the tool to support their work and demonstrated increased receptivity to follow-up feedback and data collection.

Findings Shaping an Extended SECI Model

This section presents qualitative findings that inform an extension of the Tool-Oriented SECI model to an LLM-Oriented SECI framework, highlighting how LLM companions reshape knowledge conversion and tool integration in practice.

6.1

Qualitative Findings

We present qualitative findings from ethnographic engagement with SOC analysts, showing how operational constraints, co-creation, and trust shaped LLM tool adoption, design, and workflow-aligned use as analyst-enablers. We use pseudonyms such as “P4” and “P11” to refer to analysts who participated in the research. Ethnographic Insight Reveals Operational Constraints. Early immersion revealed that the core challenges were not simply a lack of automation, but fragmentation across tools, teams, and objectives. P4 explained how analysts routinely moved between multiple platforms to complete a single task: using the SIEM for vulnerability data, consulting external sources for risk scores and remediation guidance, reviewing official CVE databases for documentation, and manually cor-

Improvements to the Asset Discovery Tool. Improving the asset discovery tool was a different challenge, that was a mix of technical and non-technical aspects. The technical aspects 9

relating information across these distributed systems. These disconnects between platforms and processes created cognitive strain and increased workload. P11, a senior security executive, commented that the fieldworkers were having “conversations that should be happening were often not.” Analysts were focused on immediate incident response, and were not able to make long term design decisions that could improve productivity. P8 noted “any kind of automation that reduces the workload of analysts is welcome.” These observations showed that the real need was not full automation of analyst work, but targeted support that reduced cognitive load and reclaim analyst workhours. The goal became to streamline information gathering, correlation, and summarization so analysts could focus on higher-value tasks. This ethnographic grounding helped avoid premature automation and positioned LLM integration as a tool to strengthen analysts’ capabilities, as an analyst enabler.

tives must (1) be replicable and capable of being institutionalized, and (2) produce outputs that yield corporate value through tools that are usable in the operational environment.” Analysts reported reduction in investigative time and cognitive load, particularly in consolidating distributed data sources. P5 stated, “We use these things, when we need to use it, it helps us learn more,” reflecting companion-style usage rather than dependence. Similarly, P8 noted, “Good to get an outside perspective, tools are not the regular tools, these are some additional tools from what they get from the vendors. [Vendor] has added a query generator, this tool seems to be better than [Vendor]’s.” The RCA tool was well received by the analysts with P7 emphasising “RCA tool saves a lot of time. I start my investigation with this to get an approach for the investigation, understand the root cause of the alert. I have been using it regularly. I am a new recruit and I believe this can also help in training new recruits who are learning how to start the investigations.” The asset discovery also integrated well with the analyst workflows, "Has had two successful uses based on past usage of other users. Changed the asset device name after finding something someone else used.". The use of RAG for query builder drew positive reception as P9 mentioned “a benefit to the team as a whole by allowing the team to learn from each other” by providing a growing database of shared experience. P5 reinforced this point, noting that “Everyone is using LLM to increase efficiency but the analysts cannot put alerts into ChatGPT so homegrown LLM tools like these will really help.” Following the conclusion of the embedding, we conducted follow-up discussions to evaluate the sustained use and integration of the tools within analysts’ workflows. P7 noted that the tool helped with “reducing the lag from having taken a month break for the holidays”. P5 explained, “I don’t use this tool everyday, but when I need it, I use it. It helps with things I don’t know”. These reflections indicate that the tools were adopted and retained as practical resources that supported analysts’ ongoing operational needs. New Ideas. The analysts came up with more ideas for new tools and new features in existing tools, which was another metric of success of the ethnographic co-creation approach led by trust building. Some ideas that came up were: “a tool that could tell us that this alert also came on 11 June then the analyst can go back and look how it was handled back then and it would save him a lot of time” and “pattern recognition in the alerts, if the SOC receives a particular type of alert weekly or biweekly or if there is a pattern to that they could probably look into it. Another idea is about false positives, if there is any way to automate the process where a tool gives out a probability that there is a X% chance this is a FP.”

Co-Creation makes Tacit Knowledge Explicit. Interest in AI was strong across teams, with pentesting, phishing detection, incident reporting, vulnerability triage, and workflow automation identified as potential LLM use cases. Through brainstorming sessions and workflow sketching, we combined the analysts’ domain knowledge with the fieldworkers’ technical expertise to formulate solutions that were both useful and technically feasible. Our decision to develop the RCA, Query Builder, and Asset Discovery tools were well received by many of the analysts despite the initial set of issues identified. This process externalized tacit knowledge embedded in social interactions and informal coordination. For example, analysts emphasized the importance of reducing user communication delays during investigations, stating that “reducing that time is critical.” Iterative refinement incorporated this feedback, resulting in features that generated richer contextual data. Early usability challenges were addressed through UI refinements, including the addition of tool tips and tutorials, enabling analysts to use the tools more effectively. Through this iterative loop, the tools adapted to real-world practice, strengthening analysts’ sense of ownership and involvement. P8 remarked that they had “never seen a project where [outsiders] come in and build tools with us according to our needs,” underscoring how co-building distinguished this approach from vendor-driven deployment. Trust as an enabling factor. Adoption depended on institutional trust along with usability. Early in the engagement, the internal advocate (IA) played a critical mediating role, lending legitimacy to the fieldworkers’ presence. The importance of this role became apparent when simply referencing the IA or including them in communications was sufficient to secure approvals, drawing on the trust the IA had already established within the organization. Over time, analysts engaged directly with the fieldworkers, indicating transferred trust.

6.2

AI enablement and workflow integration. P11 articulated the parameters of a successful endeavor: “successful initia-

LLM Oriented SECI

We observed a recurring pattern that extends tool-oriented knowledge conversion in two important ways described below. 10

Social Acceptance by the Community of Practice (Verifiability)

Apprenticeship

Socialization

Internalization

Tacit Knowledge

Human Validation/ Verification

viously exist in fixed form. This shifted the role of the “tool” from a passive container of explicit knowledge to an active coproducer. It transformed the Combination phase from a static consolidation process into a dynamic, iterative mechanism embedded directly within everyday workflow execution.

Combination

Explicit Knowledge

SOC LLM Companion

Trust Calibrated Internalization. However, the presence of generative outputs altered the trajectory from explicit artifact to tacit practice. In the tool-oriented SECI model, once a tool is deployed and demonstrates operational utility, its outputs are gradually incorporated into routine practice through repeated use. Explicit artifacts stabilize, become familiar, and are internalized as part of analysts’ tacit knowledge. In our study, this progression was neither automatic nor linear. LLM-generated artifacts did not flow directly into tacit practice. Instead, they were subjected to iterative cycles of human validation, contextual verification, and selective adoption. Analysts treated model outputs as provisional hypotheses rather than authoritative results. Before integration into workflow routines, each artifact had to pass through structured and informal evaluative checks. Specifically, analysts assessed outputs along three dimensions. First, interpretability: could the result be quickly inspected and mentally simulated against known ground truth or expected patterns? Second, reliability: did the model produce stable and consistent outputs across similar inputs, or did responses vary unpredictably? Third, workflow fit: did the artifact reduce cognitive load and investigative friction, or did it introduce additional verification overhead that offset its efficiency gains? Only after satisfying these criteria did explicit LLMgenerated artifacts transition toward internalization. Even then, internalization was conditional and partial. Analysts often embedded the outputs, as structured starting points that accelerated subsequent reasoning. In this sense, generative explicit knowledge required trust calibration before it could become tacitly embedded. The internalization phase was therefore mediated by analyst judgment, domain expertise, and situated context, rather than by repeated tool exposure alone. All in all, our findings suggest that LLM companion systems reshape the Tool-Oriented SECI Model. Tacit knowledge is externalized through co-creation; operational constraints are incorporated into the tools; the LLM generates new explicit artifacts through recombination; analysts evaluate them; and validated outputs are internalized into tacit knowledge.

Generative Recombination (e.g., RCA Reports, LEQL Queries, Asset Owner) Questioning, Reflection, and Reconstruction Externalization

Figure 5: LLM-Oriented SECI Model – Extension of the tool-oriented SECI framework in which the red components denote the proposed update, introducing LLM-based companion systems that support generative recombination, and human validation & verification.

While our findings align with the general trajectory of tacit knowledge being surfaced, incorporated into tools, and integrated into practice, the introduction of LLM-based companion systems changed how explicit knowledge was produced and how it entered routine workflow as shown in Figure 5. Generative Recombination: LLMs as Source of new Explicit Knowledge. In tool-oriented SECI [21], tools function primarily as outputs of explicit knowledge. Once tacit workflow insights are formalized, they are embedded into algorithms, tools, or models. The tool then operationalizes this explicit knowledge. In this formulation, the tool is a stabilizing mechanism: it stores, enforces, and scales previously articulated logic. LLM-based systems extended and added a new dimension to this paradigm. We did not simply encode workflow logic into static scripts. Instead, we populated vector stores with organization-specific query examples, investigation artifacts, playbooks, prior RCA reports, and structured documentation. This enabled RAG, allowing the system to ground its outputs in locally validated explicit materials rather than relying solely on base model priors. However, unlike traditional tools, these LLM-based systems did not merely store or execute explicit logic. They actively generated new structured explicit artifacts. The examples, included the (i) RCA reports which synthesized alert data into structured summaries, (ii) Query builder outputs constructed from contextual prompts and retrieved examples, and (iii) Asset discovery outputs that consolidated API responses and prior documentation into analyst-ready representations. These outputs are not static rule executions. They were newly generated structured artifacts derived from recombining existing explicit materials (logs, documentation, prior examples, retrieved context, playbooks, etc). We characterize this as a generative recombination layer within the explicit domain. Extending the SECI Combination phase (explicit→ explicit), the LLM dynamically recombined stored explicit artifacts via RAG to produce candidate explicit outputs that did not pre-

7

Discussion and Limitations

We examine LLM adoption in cybersecurity operations through a sociotechnical, practitioner-centered lens. Our findings show that integration challenges extend beyond technical performance. They mostly involve how LLM-generated artifacts are validated and verified within existing workflows. Adoption emerged through co-creation, with analysts shaping tool behavior and integration rather than passively consuming 11

outputs. By adopting tool-oriented SECI in our fieldwork, we show that LLM-based companion systems extend knowledge conversion by introducing generative recombination and trust-calibrated internalization in the transition from explicit to tacit knowledge.

through use. Our findings suggest that in AI-augmented environments this transition is mediated rather than direct. Because LLM outputs are probabilistic and dynamically generated, they are treated as provisional. Before becoming tacit practice, they must be validated and verified for interpretability, reliability, and workflow fit. Human-AI Knowledge Loop. Accoring to the tool-oriented SECI model, tools are the final outcome of explicit knowledge. In our setting, LLM companions acted as generative participants in explicit knowledge production: through RAG, stored examples and documentation were recombined into new artifacts. This introduces a hybrid human–AI knowledge loop: Tacit expertise surfaces through collaboration; workflow expectations are operationalized within LLM-based systems (e.g., vector store and RAG); the tool generates explicit artifacts; analysts validate and refine them; and ultimately validated outputs become embedded into tacit workflow. All in all, this indicates that LLM systems should not be evaluated solely as automation tools. They can function as dynamic collaborators in explicit knowledge production. However, internalization remains human-centered and institutionally constrained. This explains both early skepticism and eventual selective stabilization observed in our fieldwork.

Methodological Scope, Limitations, and Future Work. This study is based on an embedded engagement within a single SOC, which may limit generalizability to organizations with different structures, maturity levels, or governance constraints. Tool evaluation emphasized workflow alignment, perceived usefulness, and longitudinal adoption rather than controlled performance benchmarks across diverse incident types; thus, we do not claim universal performance superiority of LLM-based tools. Additionally, because LLM capabilities and AI governance frameworks are rapidly evolving, adoption dynamics may shift over time. Future research could include multi-site longitudinal studies across diverse SOC environments to examine how organizational maturity, governance structures, and risk tolerance shape trust calibration and cocreative adoption dynamics. Comparative analysis can help distinguish between context-dependent settings and aspects generalizable across settings of LLM Tool-Oriented SECI. A Sociotechnical Approach to Tool Building. By performing an ethnographic study and co-creating tools alongside cybersecurity professionals, the fieldworkers were able to build tools that the analysts both understood and trusted. The fieldworkers were able to engage with the analysts frequently and perform the tasks of their job to get unique depth of understanding how they perform their work and understand their workflows. The act of socializing in a zone of proximal development enabled us to gather highly detailed tacit knowledge made explicit which was used to inform the LLM models specifically on the use-cases necessary for their operational environment. LLM adoption in SOCs is difficult because generative outputs are probabilistic and sometimes inconsistent. This amplifies existing concerns about trust and reliability. Our findings indicate that sociotechnical co-creation enhanced the adoption pathway of LLM in security operations. Importantly, LLM-based tools were not deployed as static solutions but they evolved through iterative refinement driven by practitioner feedback. Analysts actively shaped when, and how LLM outputs were generated and accepted, shifting adoption from top-down vendor driven approach towards collaborative knowledge conversion. The SECI framework guided this LLM integration in ways that preserved analyst judgment while enabling effective augmentation.

8

Conclusion

This study examined the integration of LLM-based companion systems within a security operations center through a sociotechnical, practitioner-centered approach. Adoption emerged through co-creation: analysts actively shaped tool behavior, constraints, and deployment boundaries, ensuring alignment with operational requirements. By leveraging and extending Tool-Oriented SECI, we demonstrate that LLM systems introduce two critical dynamics into knowledge conversion: generative recombination and trust-calibrated internalization. Our extension to LLM Tool-Oriented SECI highlights that in generative AI contexts, tools not only incorporate explicit knowledge but also actively produce new explicit artifacts that reshape how tacit expertise develops over time.

9

Ethics Statement

This study involved security analysts and managerial staff and was conducted in accordance with established ethical standards. Research activities took place between July and November 2025 and were approved by the relevant Institutional Review Boards (IRBs). Written consent was waived; participants provided informed oral consent and received IRBapproved documentation describing the study, procedures, and their right to withdraw. LLM Usage Considerations. LLMs were used solely for editorial purposes, and all outputs were reviewed by the authors for accuracy and originality.

LLMs extend the SECI Model. While LLM-based companion tools generated queries, RCA drafts, and remediation messages, these artifacts did not automatically become part of SOC practice. Instead, they were incorporated only after processes of validation and governance negotiation. In tool-oriented SECI, internalization (explicit→tacit) occurs as explicit artifacts are absorbed into routine work 12

10

Acknowledgments

[7] S. Tariq, M. Baruwal Chhetri, S. Nepal, and C. Paris, “Alert fatigue in security operations centres: Research challenges and opportunities,” ACM Computing Surveys, vol. 57, no. 9, pp. 1–38, 2025. [Online]. Available: https://dl.acm.org/doi/10.1145/3723158

This work is supported by the National Science Foundation under awards no. 2143393, no. 2235102, and the Office of Naval Research under award no. N00014-23-1-2538. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of these agencies. We are grateful to our organizational collaborators for welcoming us into their environments and for their partnership in enabling this work through shared learning and co-creation.

[8] L. Kersten, K. Beelen, E. Zambon, C. Snijders, and L. Allodi, “A field study to uncover and a tool to support the alert investigation process of tier-1 analysts,” Proc. of USEC, pp. 1–15, 2025. [Online]. Available: https://www.ndss-symposium.org/wp-content/uploads /usec25-34.pdf [9] G. de Jesus Coelho da Silva and C. B. Westphall, “A survey of large language models in cybersecurity,” 2024. [Online]. Available: https://arxiv.org/abs/2402.16968

References [1] S. C. Sundaramurthy, A. G. Bardas, J. Case, X. Ou, M. Wesch, J. McHugh, and S. R. Rajagopalan, “A human capital model for mitigating security analyst burnout,” in Proceedings of the Eleventh USENIX Conference on Usable Privacy and Security, ser. SOUPS ’15. USA: USENIX Association, 2015, p. 347–359. [Online]. Available: https://www.usenix.org/conferenc e/soups2015/proceedings/presentation/sundaramurthy

[10] H. Xu, S. Wang, N. Li, K. Wang, Y. Zhao, K. Chen, T. Yu, Y. Liu, and H. Wang, “Large language models for cyber security: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2405.04760 [11] S. Freitas, J. Kalajdjieski, A. Gharib, and R. McCann, “Ai-driven guided response for security operation centers with microsoft copilot for security,” in Companion Proceedings of the ACM on Web Conference 2025, ser. WWW ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 191–200. [Online]. Available: https://doi.org/10.1145/3701716.3715209

[2] S. C. Sundaramurthy, M. Wesch, X. Ou, J. McHugh, S. R. Rajagopalan, and A. G. Bardas, “Humans are dynamicour tools should be too,” IEEE Internet Computing, vol. 21, no. 3, pp. 40–46, 2017. [Online]. Available: https://ieeexplore.ieee.org/document/7927884/

[12] B. Edelman, J. Bono, S. Peng, R. Rodriguez, and S. Ho, “Randomized controlled trial for copilot for security (whitepaper),” Jan. 2024. [Online]. Available: https://www.microsoft.com/content/dam/microsoft/fin al/en-us/microsoft-product-and-services/microsoft-d ynamics-365/pdf/Microsoft-Copilot-for-Security-pro ductivity-findings-Whitepaper-Jan2024.pdf

[3] C. Zimmerman, 11 Strategies of a World-Class Cybersecurity Operations Center. MITRE Corporation, 2022. [Online]. Available: https://www.mitre.org/sites/ default/files/2022-04/11-strategies-of-a-world-class-c ybersecurity-operations-center.pdf [4] B. A. Alahmadi, L. Axon, and I. Martinovic, “99% false positives: A qualitative study of {SOC} analysts’ perspectives on security alarms,” in 31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 2783–2800. [Online]. Available: https://www.usenix.o rg/conference/usenixsecurity22/presentation/alahmadi

[13] Rapid7, “Rapid7’s AI engine supercharges security operations with generative AI,” Jun. 2024. [Online]. Available: https://www.rapid7.com/about/press-relea ses/rapid7s-ai-engine-supercharges-security-operation s-with-generative-ai/

[5] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 672–681. [Online]. Available: https: //petertsehsun.github.io/soen7481/papers/icse13b.pdf

[14] ReliaQuest, “Reliaquest launches first autonomous, self-learning AI agent for security operations,” Sep. 2024. [Online]. Available: https://reliaquest.com/new s-and-press/reliaquest-launches-first-autonomous-sel f-learning-ai-agent-for-security-operations/ [15] N. Roch, H. Sievers, L. Schöni, and V. Zimmermann, “Navigating autonomy: Unveiling security experts’ perspectives on augmented intelligence in cybersecurity,” in Twentieth Symposium on Usable Privacy and Security (SOUPS 2024). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 41–60. [Online]. Available: https://www.usenix.org/conference/soups2024/presen tation/roch

[6] J. Smith, B. Johnson, E. Murphy-Hill, B. Chu, and H. R. Lipford, “Questions developers ask while diagnosing potential security vulnerabilities with static analysis,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 248–259. [Online]. Available: https://dl.acm.org/doi/pdf/10.1145/2786805.2786812 13

[16] K. Thimmaraju, S. I. Rispens, and G.-J. Ahn, “Human performance in security operations: a survey on burnout, well-being and flow state among practitioners,” in Proc. 2025 Workshop on Security Operations Center Operations and Construction (WOSOC 2025), 2025, pp. 2–4. [Online]. Available: https://www.ndss-symposium .org/wp-content/uploads/wosoc25-final2.pdf

in Human-Centered cybersecurity,” in Twentieth Symposium on Usable Privacy and Security (SOUPS 2024). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 567–586. [Online]. Available: https://www.us enix.org/conference/soups2024/presentation/haney [24] S. C. Sundaramurthy, J. McHugh, X. Ou, M. Wesch, A. G. Bardas, and S. R. Rajagopalan, “Turning contradictions into innovations or: How we learned to stop whining and improve security operations,” in Twelfth Symposium on Usable Privacy and Security (SOUPS 2016). Denver, CO: USENIX Association, Jun. 2016, pp. 237–251. [Online]. Available: https: //www.usenix.org/conference/soups2016/technical-ses sions/presentation/sundaramurthy

[17] F. B. Kokulu, A. Soneji, T. Bao, Y. Shoshitaishvili, Z. Zhao, A. Doupé, and G.-J. Ahn, “Matched and mismatched SOCs: A qualitative study on security operations center issues,” in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 1955–1970. [Online]. Available: https: //doi.org/10.1145/3319535.3354239

[25] C. Fung, E. Zeng, and L. Bauer, “Adopting AI to protect industrial control systems: Assessing challenges and opportunities from the operators’ perspective,” in Twenty-First Symposium on Usable Privacy and Security (SOUPS 2025), 2025, pp. 555–573. [Online]. Available: https://www.usenix.org/system/files/soups 2025-fung.pdf

[18] C. Catal, A. Ozcan, E. Donmez, and A. Kasif, “Analysis of cyber security knowledge gaps based on cyber security body of knowledge,” Education and Information Technologies, vol. 28, no. 2, p. 1809–1831, Aug. 2022. [Online]. Available: https: //doi.org/10.1007/s10639-022-11261-8

[26] A. Tuladhar, D. Lende, J. Ligatti, and X. Ou, “An analysis of the role of situated learning in starting a security culture in a software company,” in Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021). USENIX Association, Aug. 2021, pp. 617–632. [Online]. Available: https://www.usenix.org/conferenc e/soups2021/presentation/tuladhar

[19] R. Werlinger, K. Hawkey, D. Botta, and K. Beznosov, “Security practitioners in context: Their activities and interactions with other stakeholders within organizations,” International Journal of Human-Computer Studies, vol. 67, no. 7, pp. 584–606, 2009. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1 071581909000354

[27] M. Vielberth, F. Böhm, I. Fichtinger, and G. Pernul, “Security operations center: A systematic study and open challenges,” IEEE Access, vol. 8, pp. 227 756–227 779, 2020. [Online]. Available: https: //api.semanticscholar.org/CorpusID:230513062

[20] I. Nonaka, “A dynamic theory of organizational knowledge creation,” Organization Science, vol. 5, no. 1, pp. 14–37, 1994. [Online]. Available: https://josephma honey.web.illinois.edu/BA504_Fall%202008/Uploade d%20in%20Nov%202007/Nonaka%20(1994).pdf

[28] A. Greig, K. Renaud, and S. Flowerday, “An ethnographic study to assess the enactment of information security culture in a retail store,” in 2015 World Congress on Internet Security (WorldCIS), 2015, pp. 61–66. [Online]. Available: https://ieeexplore.ieee. org/document/7359415

[21] S. C. Sundaramurthy, J. McHugh, X. S. Ou, S. R. Rajagopalan, and M. Wesch, “ An Anthropological Approach to Studying CSIRTs ,” IEEE Security & Privacy, vol. 12, no. 05, pp. 52–60, Sep. 2014. [Online]. Available: https://doi.ieeecomputersociety.org/10.110 9/MSP.2014.84

[29] S. Srinivas, B. Kirk, J. Zendejas, M. Espino, M. Boskovich, A. Bari, K. Dajani, and N. Alzahrani, “AI-augmented SOC: A survey of LLMs and agents for security automation,” Journal of Cybersecurity and Privacy, vol. 5, no. 4, 2025. [Online]. Available: https://www.mdpi.com/2624-800X/5/4/95

[22] J. Lave and E. Wenger, Situated Learning: Legitimate Peripheral Participation, ser. Learning in Doing: Social, Cognitive and Computational Perspectives. Cambridge University Press, 1991. [Online]. Available: https://www.cambridge.org/highereducation/books/si tuated-learning/6915ABD21C8E4619F750A4D4AC A616CD#overview

[30] F. Binbeshr, M. Imam, M. Ghaleb, M. Hamdan, M. A. Rahim, and M. Hammoudeh, “The rise of cognitive SOCs: A systematic literature review on AI approaches,” IEEE Open Journal of the Computer Society, vol. 6, pp. 360–379, 2025. [Online]. Available:

[23] J. M. Haney, C. C. IV, and S. M. Furman, “Towards bridging the Research-Practice gap: Understanding Researcher-Practitioner interactions and challenges 14

https://www.computer.org/csdl/journal/oj/2025/01/108 58372/23VPu8d631m

/edited-volume/3397/The-Encultured-BrainAn-Intro duction-to

[31] J. Mink, H. Benkraouda, L. Yang, A. Ciptadi, A. Ahmadzadeh, D. Votipka, and G. Wang, “Everybody’s got ML, tell me what else you have: Practitioners’ perception of ML-based security tools and explanations,” in 2023 IEEE Symposium on Security and Privacy (SP), 2023, pp. 2068–2085. [Online]. Available: https://ieeexplore.ieee.org/document/10179321

[39] I. Nonaka and G. von Krogh, “Perspective—tacit knowledge and knowledge conversion: Controversy and advancement in organizational knowledge creation theory,” Organization Science, vol. 20, no. 3, pp. 635–652, 2009. [Online]. Available: https://pubsonline.i nforms.org/doi/10.1287/orsc.1080.0412 [40] L. A. Suchman, Plans and situated actions: An inquiry into the idea of human-machine communication. University of California, Berkeley, 1984. [Online]. Available: https://www.cs.colby.edu/courses/J16/cs267 /papers/Suchman-PlansAndSituatedActions.pdf

[32] R. Singh, S. Tariq, F. Jalalvand, M. B. Chhetri, S. Nepal, C. Paris, and M. Lochner, “LLMs in the SOC: An empirical study of human-AI collaboration in security operations centres,” 2025. [Online]. Available: https://arxiv.org/abs/2508.18947

[41] Y. Engeström and A. Sannino, “Studies of expansive learning: Foundations, findings and future challenges,” Introduction to Vygotsky, pp. 100–146, 2017. [Online]. Available: https://www.sciencedirect.com/science/arti cle/abs/pii/S1747938X10000035

[33] D. Kramer, L. Rosique, A. Narotam, E. Bursztein, P. G. Kelley, K. Thomas, and A. Woodruff, “Integrating large language models into security incident response,” in Twenty-First Symposium on Usable Privacy and Security (SOUPS 2025), 2025, pp. 133–148. [Online]. Available: https://www.usenix.org/system/files/soups 2025-kramer.pdf

[42] J. Van Maanen, Tales of the field: On writing ethnography. University of Chicago Press, 2011. [Online]. Available: https://press.uchicago.edu/ucp/boo ks/book/chicago/T/bo11574153.html

[34] G. Deng, Y. Liu, V. Mayoral-Vilches, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, and S. Rass, “PentestGPT: Evaluating and harnessing large language models for automated penetration testing,” in 33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 847–864. [Online]. Available: https://www.usenix.org/c onference/usenixsecurity24/presentation/deng

[43] P. Ehn, “Work-oriented design of computer artifacts,” Ph.D. dissertation, Arbetslivscentrum, 1988. [Online]. Available: https://www.diva-portal.org/smash/get/diva2: 580037/fulltext02.pdf [44] K. Anderson, M. Foster, C. Freeman, and I. Scott, “A multifaceted intervention to reduce inappropriate polypharmacy in primary care: research co-creation opportunities in a pilot study,” The Medical journal of Australia, vol. 204, pp. S41–S, 04 2016. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.5 694/mja16.00125

[35] K. Shashwat, F. Hahn, X. Ou, D. Goldgof, L. Hall, J. Ligatti, S. R. Rajgopalan, and A. Z. Tabari, “A preliminary study on using large language models in software pentesting,” in Workshop on SOC Operations and Construction (WOSOC), March 2024. [Online]. Available: https://arxiv.org/abs/2401.17459

[45] T. Greenhalgh, C. JACKSON, S. Shaw, and T. Janamian, “Achieving research impact through co-creation in community-based health services: Literature review and case study: Achieving research impact through co-creation,” The Milbank Quarterly, vol. 94, pp. 392–429, 06 2016. [Online]. Available: https://pubmed .ncbi.nlm.nih.gov/27265562/

[36] N. Rastogi, S. Pant, D. Dhanuka, A. Saxena, and P. Mairal, “Too much to trust? measuring the security and cognitive impacts of explainability in AI-driven SOCs,” 2025. [Online]. Available: https://arxiv.org/abs/2503.02065 [37] M. Nyre-Yu, E. Morris, M. R. Smith, B. Moss, and C. Smutz, “Explainable AI in cybersecurity operations: Lessons learned from xAI tool deployment,” Proceedings 2022 Symposium on Usable Security, 2022. [Online]. Available: https://api.semanticscholar.org/Co rpusID:253156531

[46] K. Panahi, S. Robertson, Y. Acar, A. G. Bardas, T. Kohno, and L. Simko, “"but they have overlooked a few things in Afghanistan:" an analysis of the integration of biometric voter verification in the 2019 afghan presidential elections,” in 33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 2047–2064. [Online]. Available: https://www.usenix.org/conferenc e/usenixsecurity24/presentation/panahi

[38] D. Lende and G. Downey, Eds., The encultured brain: an introduction to neuroanthropology. MIT Press, Jan. 2012. [Online]. Available: https://direct.mit.edu/books 15

[47] S. Gupta, R. Ranjan, and S. N. Singh, “A comprehensive survey of retrieval-augmented generation (RAG): Evolution, current landscape and future directions,” arXiv preprint arXiv:2410.12837, 2024. [Online]. Available: https://arxiv.org/abs/2410.12837 [48] M. Rahman, K. O. Piryani, A. M. Sanchez, S. Munikoti, L. De La Torre, M. S. Levin, M. Akbar, M. Hossain, M. Hasan, and M. Halappanavar, “Retrieval augmented generation for robust cyber defense,” Pacific Northwest National Laboratory (PNNL), Richland, WA (United States), Tech. Rep., 2024. [Online]. Available: https: //www.pnnl.gov/main/publications/external/technical_ reports/PNNL-36792.pdf [49] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The eleventh international conference on learning representations, 2022. [Online]. Available: https://arxiv.org/abs/2210.0 3629 [50] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, P. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, T. Lavril, T. Lacroix, and et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023. [Online]. Available: https://arxiv.org/abs/2310.06825 [51] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. AlDahle, A. Letman, A. Mathur, and et al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783 [52] L.-W. Ku, A. F. Martins, and V. Srikumar, “Findings of the association for computational linguistics: Acl 2024,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024. [53] Ollama. Ollama. Accessed: 2026/02. [Online]. Available: https://ollama.com/ [54] Chroma. Chroma. Accessed: 2026/02. [Online]. Available: https://www.trychroma.com/ [55] MITRE. Mitre attack. Accessed: 2026/02. [Online]. Available: https://attack.mitre.org/

16

A

Codebook

Table 2: Thematic Structure: High-Level Themes, Definitions, and Associated Codes Theme

Definition

Codes

Ethnographic Insight

Embedded anthropological approach actively reexamined operational pain points, surfaced tacit constraints, and supported the co-creation of LLM tools that were successfully adopted in practice.

anthropology; academics; interdisciplinary; research questions; knowledge transfer; gap between academic and industry people; fieldworker’s research background; internal advocate; CISO; soc director; upper management; long term goals of org; objectives of org; organisation.

Operational and Structural Constraints

Persistent workflow bottlenecks, expertise gaps, and infrastructural limitations created cognitive and temporal strain within security operations, generating the need for augmentation rather than automation replacement.

challenges; lack of time and expertise for tool building; different roles. different expertise.; task allocation; pentesting; rapid7; RCA; table top exercises; side projects; critical thinking; familiarise with organisation; problem identification; security tool gap; distributed data; disparate workflows; cognitive friction; infrastructure latency; communication latency ; environmental mismatch

Co-Creation as Tacit Knowledge Externalization

Through participant observation, shadowing, demos, and iterative refinement, tacit operational expertise was made explicit and embedded into tool design, transforming socially situated knowledge into structured technical artifacts.

participant observation; fieldwork; talking to people; tacit knowledge; requirement gathering; idea formulation; demo; feedback on project; feedback on tools; tool building; solution; collaboration; socialization; discussing workflow; human nonintelligible information; data confidence; work enablement; co-building; reclaiming work hours

Governance, Control, and Human Authority

AI integration was negotiated through governance constraints, security assurances, and preservation of analyst authority. Adoption depended on bounded autonomy, human-in-the-loop positioning, and institutional trust-building.

concerns with LLM; human in the loop; trust building; self manageable tools; automation; analyst enabler; concerns with LLM; human in the loop; trust building; self manageable tools; automation; analyst enabler.

Situated AI Enablement and Workflow Integration

Adoption emerged when LLM systems aligned with existing workflows, reclaimed work hours, and demonstrated utility without disrupting operational practice. Tools were embraced when positioned as enablers embedded within routine work.

AI; AI adoption; LLM; tool; tool usage; positive reception; solution.

17