arXiv:2606.21963v1 [cs.AI] 20 Jun 2026
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale Jia Li∗
Wenyuan Ma∗
Ting Peng
[email protected] The Chinese University of Hong Kong Hong Kong, Hong Kong
[email protected] Tencent Inc. Shenzhen, China
[email protected] Tencent Inc. Shenzhen, China
Haibin Zheng
Yuetang Deng†
[email protected] Tencent Inc. Shenzhen, China
[email protected] Tencent Inc. Shenzhen, China
Abstract Diagnosing mobile crashes in ultra-large-scale industrial applications is a formidable challenge due to the sheer volume of code, the complexity of mixed-language environments, and the inability to reproduce failures locally. Traditional static analysis struggles with scalability, while existing LLM-based agents often rely on reproducible environments unavailable in post-mortem scenarios. We present Holmes, a multi-agent system that automates root cause analysis by synthesizing multimodal runtime signals—stack traces, logs, and thread states—to reconstruct failure contexts without reproduction. Holmes introduces a hierarchical Retrieve-ExploreReason architecture that leverages low-level artifacts (e.g., registers, assembly) to bridge the semantic gap between open-source business logic and closed-source system frameworks. By dynamically compressing the search space using runtime clues, Holmes precisely navigates 70-million-line codebases to identify non-local defects. Evaluated on real-world crashes from WeChat, Holmes achieves 87.6% accuracy in function-level fault localization and reduces average investigation time by over 98% (to ∼77 seconds), demonstrating its effectiveness in transforming labor-intensive debugging into an efficient verification workflow.
CCS Concepts • Software and its engineering → Software testing and debugging; • Computing methodologies → Natural language processing.
Keywords Automated Crash Diagnosis, Multi-Agent Systems, Large Language Models, Fault Localization, Mobile Applications ∗ Both authors contributed equally to this research. † Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
ACM Reference Format: Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, and Yuetang Deng. 2026. Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale. In . ACM, New York, NY, USA, 9 pages. https://doi.org/ 10.1145/nnnnnnn.nnnnnnn
1
Introduction
Modern mobile applications have evolved into complex ecosystems serving billions of users. In WeChat, the client generates millions of crash reports daily. While automated services cluster crashes based on stack trace similarity, they lack automated root cause analysis (RCA) and actionable fix suggestions. Developers must manually map runtime symptoms to static code defects, a daunting task in ultra-large-scale industrial repositories. Bridging the gap between runtime data and source code requires deep domain knowledge and hours of investigation. Internal data shows that complex crash clusters typically require 2 to 3 hours of ticket handling time to reach an actionable diagnosis (Time-to-Insight), involving labor-intensive correlation of multimodal clues across stacks, logs, thread states, user trajectories, and source code. In large industrial repositories, the gap between dynamic symptoms and static logic widens. Traditional static analysis [3] is often impractical at the scale of 70 million lines of code due to the computational cost of global inter-procedural analysis. Recent LLM-based approaches [6] using pre-built call graphs face high maintenance costs and limited scalability in systems with over 6 million functions. Furthermore, state-of-the-art software engineering agents [15–17] typically rely on reproducible environments and failureinducing tests, which are unavailable for post-mortem diagnosis of mobile crashes. Given the diversity of user environments and privacy constraints that prevent data access, local sandboxes cannot replicate the transient states of millions of devices. This necessitates a post-mortem reasoning paradigm capable of navigating massive repositories to localize root causes using only sparse, read-only artifacts, without requiring a reproducible environment. Despite the promising reasoning capabilities of LLMs, automated diagnosis in ultra-large-scale industrial environments faces significant hurdles. First, existing methods typically rely on single-modal analysis, focusing either on logs [7, 9, 11, 13] or code [2, 4–6, 14].
Conference’17, July 2017, Washington, DC, USA
This isolation fails to synthesize the spatial, temporal, and concurrency clues essential for complex crashes. Second, current techniques struggle with the semantic gap in mixed-source environments. Most approaches assume full source visibility and hit dead ends when execution flows into closed-source binaries, unable to trace logic across system boundaries. Third, navigating massive search spaces for non-local root causes is computationally prohibitive; standard retrieval struggles to locate distant defects without overwhelming context windows. Finally, the lack of low-level runtime signals (e.g., registers, memory snapshots) limits hypothesis verification in non-reproducible, post-mortem scenarios. In this paper, we propose Holmes, a multi-agent system that formulates crash diagnosis as an agentic reasoning task. To overcome single-modal and scalability limitations, Holmes performs joint reasoning over the full spectrum of dynamic runtime signals available in system dumps—stack traces, time-ordered log events, and concurrent thread states. This multimodal synthesis not only reconstructs the full failure context but also aggressively compresses the code search space, enabling precise navigation to non-local root causes within 70 million lines of code. Crucially, Holmes integrates low-level artifacts like registers and assembly to bridge the semantic gap between closed-source system frameworks and opensource business logic, enabling hypothesis verification across these boundaries. By adopting a hierarchical Retrieve-Explore-Reason architecture, Holmes synthesizes these heterogeneous clues to deliver industrial-grade localization, explanation, and fix suggestions in approximately one minute. We evaluated Holmes using a dataset of 73 real-world crash reports sampled from the WeChat iOS production environment. Senior developers defined a taxonomy and annotated the dataset with ground-truth fault locations and root causes. On this complex dataset, Holmes achieves 87.6% pass@1 accuracy for function-level fault localization and 65.7% for root-cause identification, with a mean latency of 168.5 seconds (compared to 77 seconds average in large-scale production). By synthesizing multimodal evidence to navigate millions of functions, Holmes reduces investigation time by over 98%, shifting the workflow from manual investigation to efficient verification. Contributions. This paper makes the following technical contributions: • Multimodal Agentic Diagnosis Framework. We propose Holmes, a multi-agent system that formulates crash diagnosis as a collaborative reasoning task. By synthesizing heterogeneous signals from stacks, logs, threads, and code, it overcomes the limitations of single-modal approaches and reconstructs complex failure contexts. • Hybrid-Source Reasoning with Low-Level Artifacts. We introduce a method to bridge the semantic gap in mixed-source environments by integrating registers, memory snapshots, and assembly code. This enables seamless reasoning across the boundary between open-source business logic and closed-source system frameworks. • Scalable Navigation via Dynamic Compression. We present a retrieval-augmented strategy that leverages dynamic runtime signals to aggressively compress the search space. This allows
Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, and Yuetang Deng
precise navigation to non-local root causes within a 70-millionline repository within minutes, without relying on expensive global static analysis. • Industrial-Scale Evaluation. We evaluate Holmes on realworld crashes from WeChat. Results show it achieves 87.6% accuracy in function-level localization and reduces investigation time by over 98% (to ∼77 seconds), demonstrating its effectiveness and efficiency in a high-volume production environment.
2
Methodology
Holmes is designed for industrial crash diagnosis in mixedlanguage codebases (C/C++/ObjC/Swift), leveraging multi-modal artifacts including stacks, logs, code, and rich metadata. It adopts a hierarchical Retrieve-Explore-Reason architecture (Figure 1) with three layers: (1) Parallel Context Retrieval for evidence collection; (2) Agentic Code Exploration for deep root cause search; and (3) Synthesis & Reasoning for final diagnosis. This design ensures focused data extraction before global synthesis.
2.1
Layer 1: Parallel Context Retrieval
Crash Summary In the production environment, crash reports are captured on user devices and uploaded to a central server. The diagnosis workflow is initiated by the Dispatcher, a core component of the Holmes framework responsible for orchestrating the multi-agent collaboration. Upon receiving a raw crash report, the Dispatcher first parses the raw data to extract three primary artifacts: the crashing thread’s stack trace, the full log stream, and relevant source/assembly code. Remaining key fields (e.g., exception type, user click paths, register states, memory stats) are normalized into metadata (Table 1). This metadata serves as stable grounding signals injected into downstream agents to constrain and calibrate subsequent evidence extraction. 2.1.1 Agent 1: The Stack Code Retriever (Focus: Static Stack). This agent provides the baseline code context for the crashing thread by taking the crashing thread stack trace as input and outputting initial stack code snippets (source code or assembly). Its primary role is to collect the code executed at the time of the crash before deep exploration, performing retrieval only without conducting reasoning. To implement this, the agent extracts code for all functions in the crashing thread’s stack trace using a dual-mode retrieval policy based on frame type: it retrieves source code (e.g., MessageMgr.mm) for internal business logic to support understanding, and assembly code (e.g., libobjc.A.dylib) for system frameworks when source code is unavailable, ensuring complete visibility into the execution path across system boundaries. 2.1.2 Agent 2: The Log Miner (Focus: Runtime Logs). This agent identifies time-ordered events relevant to the crash by processing the Crash Summary and Raw Log Stream to output filtered key log snippets. Unlike naive approaches that simply truncate logs to a fixed size (e.g., the last 200 lines), which risks missing early precursors, we capture a broader 1-minute temporal window and employ a semantic Map-Reduce strategy to distill relevant signals. The implementation begins by building a Crash Summary (including crash metadata, crashing thread context, and key stack symbols) as an anchor. To mitigate the lost-in-the-middle phenomenon, the system
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale
Conference’17, July 2017, Washington, DC, USA
[ Crash Summary ]
Layer 1: Parallel Context Retrieval Crash Event
Agent 3: Code Explorer
Agent 1: Stack Code Retriever
Dispatcher Parse Payload & Generate Summary
Identify Missing Links
Agentic Search
Retrieve Source/Assembly on Crashing Thread Stack Trace
Gen Func Signatures
🧐 Reasoning
Batch Fetch & Update
Accumulated Code Context
Layer 2: Agentic Code Exploration [ Expanded Code Context ]
Agent 2: Log Miner Chronological Log Distillation by Semantic Map-Reduce
Layer 3: Synthesis & Reasoning [ Stack Code ] [ Key Logs ]
[ Crash Summary ]
Dynamic Prompt Composition
Role + Blame-Frame + Artifact Aggregation
Agent 3: Thread Inspector
[ Thread Status ]
Attention Guidance
Relevance Scoring (MapReduce) Top-K Threads
Blame-Frame Localization
Agent 5: Lead Analys
🏁 Final Diagnostic Report Example: MagicBrush Race Condition Category: Concurrency Issue Root Cause: Race Condition Defect Location: • File: MBBizSystem+PublicService.m m • Function: MBBizSystem(PublicService) unbindAllPublicServiceForBiz • Line: 149 Fix: Add NSRecursiveLock to protect shared resource
[ Crash Summary ]
Figure 1: System architecture of the Holmes framework. It illustrates the tripartite workflow: parallel context retrieval via specialized agents (Layer 1), iterative logic navigation across massive codebases (Layer 2), and multimodal evidence synthesis for final diagnosis (Layer 3). Table 1: Metadata fields extracted from crash reports. Field
Explanation
Value
Basic Info
Exception, OS, Device info
User Click Path Crashed Thread Registers Memory Snapshots VM Summary
View controller trajectory Top frame of crashing thread Key CPU registers (pc, lr, sp) Relevant variable values Virtual and resident memory
Exception: EXC_BAD_ACCESS; Process: WeChat (9.8.241); Device: iPhone12,1; OS: iOS 16.3.1 MultiSelectContactsVC → MinimizeVC → NewMainFrameVC Thread 0: com.apple.main-thread; Top: __CFStringAppendBytes → ... pc: 0x1c5565d60; lr: 0x1910dc01c5533c64; sp: 0x16f9afdf0 x1 (0x117aba2ee): migration_info; Virtual: 9.7G / Resident: 391.4M; Stack: 94.2M
splits the 1-minute log window into 8k-token chunks and processes them with four parallel workers. These workers are explicitly instructed to retain only log lines that share causal dependencies (e.g., resource allocation, state transitions) with the entities identified in the Crash Summary. Finally, these filtered segments are aggregated into a single timeline of relevant events.
2.1.3 Agent 3: The Thread Inspector (Focus: Concurrency State). The Thread Inspector detects cross-thread interference, such as deadlocks and race conditions, by analyzing the Crash Summary and Full Thread Dump to output the top 2 high-relevance thread stacks. Since analyzing all threads uniformly introduces noise and exceeds context limits, we formulate thread analysis as a relevance ranking problem using a Map-Reduce strategy. The system groups the full thread dump into chunks (approx. 8k tokens) with atomic thread boundaries. Six parallel workers assign relevance scores to each thread against the Crash Summary in the Map Phase, and the results are merged and sorted based on these scores in the Reduce Phase to retain the most relevant threads.
2.2
Layer 2: Agentic Code Exploration
A crash stack trace often reflects the observed failure, while the underlying cause may reside in asynchronous callbacks or in earlier execution paths. Scalability Challenge. In a 70M LOC repository (6M+ functions), maintaining a global static Call Graph (CG) is impractical. Internal benchmarks show generating an inter-procedural CG requires >12 hours and >500GB memory, blocking CI integration. Similarly, RAG approaches [8, 12] face scalability hurdles, with index construction taking >7 days and struggling with updates. Furthermore, dynamic dispatch in ObjC/C++ renders static graphs inherently incomplete. Agentic Search Paradigm. Instead of pre-built graphs, Holmes adopts an agentic search paradigm supported by a lightweight function-to-file-path index for on-demand navigation. Initial indexing takes ∼3.3 hours (6GB footprint), with incremental updates taking only ∼10 minutes. Building on this foundation, the layer consumes the Crash Summary together with the Initial Context (Stack Code Snippets from Agent 1, Key Logs from Agent 2, and Relevant Threads from Agent
Conference’17, July 2017, Washington, DC, USA
3) and incrementally produces an Expanded Code Context that accumulates the retrieved function implementations. The core mechanism is a loop (maximum 3 iterations) that progressively expands the context: (1) Context Assembly: Formats accumulated evidence; (2) Reasoning & Decision: The LLM identifies missing logical links and outputs target function signatures; (3) Batch Retrieval: Fetches implementations in parallel; and (4) State Update: Accumulates new snippets. The loop terminates when no new targets are found.
2.3
Layer 3: Synthesis & Reasoning
Acting as the lead analyst, this layer is responsible for global evidence synthesis and final report generation. It takes the Crash Summary, Expanded Code Context, Stack Code Snippets, Key Logs, Relevant Threads, and the Blame-Frame Prompt as input, and produces a Final Diagnostic Report containing the root cause, defect localization, evidence chain, and fix suggestions. The workflow consists of four steps: (1) Attention Guidance (Blame-Frame Localization), which predicts a single suspicious file:line anchor to force reasoning to stay anchored; (2) Dynamic Prompt Composition, which aggregates refined outputs and conditionally appends specialized sections; (3) Heuristic Rule Injection, guided by internal statistics showing that memory errors account for over 50% of crashes, injects a minimal set of domain rules (e.g., checking low addresses to distinguish null pointers from wild pointers) to resolve specific ambiguities; and (4) Final Reasoning, which, instead of relying on the LLM’s inherent capabilities alone, leverages the structured, semantically aligned evidence prepared by the previous layers to enable rigorous cross-modal consistency checks and generate the final report.
3
Experiment
To validate Holmes’ effectiveness in an industrial setting, we conducted a comprehensive evaluation using real-world crash reports labeled by senior developers. Crash Taxonomy Definition To ensure a rigorous and standardized evaluation, we established a taxonomy of crash categories and root causes. This taxonomy is derived from the Common Weakness Enumeration (CWE) framework and refined based on the statistical distribution of failure modes in our production environment. These categories are detailed in Table 2.
3.1
Dataset & Ground Truth
We collected a dataset of 73 crash reports from the WeChat iOS client. Sampling: Given the labor-intensive nature of manual root cause analysis, we constructed a focused dataset of 73 cases. We performed stratified sampling from the top 500 most frequent crash clusters. This type-based stratified sampling approach preserves the true distribution of the most frequent failure modes in the production environment. Diversity: The dataset spans multiple critical chains covering diverse modules ranging from low-level infrastructure (network, storage, kernel) to high-level UI and business logic (multimedia, mini-programs). Based on the 73 valid cases used for calculation, the failure modes are distributed as follows: Memory Errors (53.4%),
Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, and Yuetang Deng
Logic Errors (23.3%), External & Environmental Issues (13.7%), Resource Management Issues (4.1%), and Concurrency Issues (5.5%). This distribution reflects the high prevalence of memory safety issues in large-scale codebases using C++ and Objective-C. Labeling: To prevent bias, the labeling process was conducted independently of and prior to the Holmes diagnosis. Each report was manually analyzed by two senior developers using a crossvalidation process to establish the Ground Truth for the category, root cause, and defect location (file/function/line). All three localization labels refer to the defect-causing (most attributable) location as identified by developers (i.e., the file/function/line where the defect was introduced or that is causally responsible for the crash), rather than the patch location. Disagreements were resolved through discussion to ensure high-quality labels.
3.2
Accuracy Results
We define Accuracy as the percentage of cases where the AIgenerated output matches the ground truth. We evaluated accuracy across five dimensions. The results on our dataset (𝑁 = 73) are shown in Table 3. Holmes demonstrates robust fault localization, the most laborintensive debugging phase. High Function (87.6%) and File (90.4%) Accuracy confirm that the retrieval system effectively narrows the search space to the correct code region. Comparable Line Accuracy indicates precise defect pinpointing once the region is locked. Category Accuracy (79.4%) remains strong, effectively distinguishing defect types despite occasional confusion between memory and logic errors. While Root Cause Accuracy (65.7%) is lower due to semantic ambiguities (e.g., null vs. garbage pointers), high localization accuracy ensures developers are guided to the correct verification region. Regarding efficiency, the average diagnostic latency of 168.5 seconds (approx. 2.8 minutes) on this complex dataset is higher than the production average (77s) due to deeper exploration, yet still represents an order-of-magnitude improvement over manual debugging. 3.2.1 Per-Category Accuracy Breakdown. To provide a finer-grained view of Holmes’ diagnostic capabilities across the taxonomy defined in Table 2, we report per-category accuracy in Table 5 and per-root-cause accuracy in Table 6. Category-level analysis. Holmes achieves near-perfect classification for Memory Errors (97.4% category accuracy), which constitute the majority of crashes (53.4%). Concurrency and Resource Management categories, though small in sample size (𝑛 = 4 and 𝑛 = 3), are diagnosed with 100% accuracy across all metrics, demonstrating Holmes’ strength in leveraging thread-level and runtime evidence. Performance is weaker on Logic Errors (58.8%) and External & Environmental issues (30.0%), where the system tends to misclassify them as memory errors due to similar crash signatures (e.g., assertion failures manifesting as SIGABRT). Root-cause-level analysis. At the fine-grained root cause level (Table 6), Holmes excels at structurally distinctive failure modes: Garbage Pointer (100%), Race Condition (100%), and Unrecognized Selector (100%) are identified with perfect root cause accuracy. For Null Pointer Dereference—the most frequent root
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale
Conference’17, July 2017, Washington, DC, USA
Table 2: Detailed crash taxonomy and root cause definitions. Category
Root Cause
Core Description
Memory Error
Null Pointer Dereference Garbage Pointer Invalid Free Buffer Overflow Stack Overflow
Attempting to read or write memory pointed to by a NULL or nullptr pointer. Accessing memory that has been freed (dangling pointer) or uninitialized (wild pointer). Attempting to free a pointer not allocated by malloc/new or already freed. Writing data beyond the boundaries of an array or buffer. Infinite recursion or allocating excessively large local variables on the stack.
Logic Error
Division by Zero Unhandled Exception Assertion Failure Unrecognized Selector Type Conversion Failure Null Parameter
Denominator is zero in a mathematical operation. Program throws an exception without any try-catch block to handle it. Condition of assert() macro is false, causing program termination. Dynamic language (e.g., ObjC) calls an unimplemented method on an object. Runtime type casting check fails (e.g., in Swift). Passing a null value to a function that expects a non-null argument.
Concurrency
Race Condition Deadlock
Multiple threads access shared data concurrently without synchronization. Two or more threads waiting for each other to release resources.
External & Env
Improper API Usage Corrupted Input
Violating usage contracts when calling third-party libraries or system APIs. Crash during parsing of malformed or corrupted files/data streams.
Resource Mgmt
Main Thread Violation Object Lifecycle Resource Leak Disk Space Exhaustion Out of Memory
Executing UI operations on a non-main thread. Improper timing of object creation/destruction, or internal state inconsistency. Leaking file descriptors or contexts. Storage operations fail due to insufficient space. Allocation failure due to insufficient memory.
Table 3: Evaluation of Holmes accuracy across five dimensions (𝑁 = 73).
Table 4: Ablation study results comparing Holmes with variants and baseline. Model / Variant
Metric (Definition) Category Accuracy (Correct classification of crash type) Root Cause Accuracy (Identification of fundamental reason) Function Accuracy (Identifying exact defect function) File Accuracy (Identifying correct source file) Line Accuracy (Pinpointing defect-causing source line)
Accuracy 79.4% 65.7% 87.6% 90.4% 87.6%
cause (𝑛 = 22)—root cause accuracy is 72.7% while function localization reaches 95.5%, indicating that even when the specific root cause label is debated, the defect location is correctly identified. The weakest root causes are Assertion Failure (33.3%), Improper API Usage (37.5%), Corrupted Input (0%), Invalid Free (0%), and Null Parameter (0%). These share a common challenge: the crash symptom is indirect, requiring deeper semantic understanding of API contracts or data provenance that goes beyond what stack traces and code can directly reveal. Notably, even for these difficult cases, function localization accuracy remains substantially higher (66.7–100%), confirming that Holmes successfully guides developers to the correct code region even when the precise root cause taxonomy label is contested.
3.3
Comparative Analysis & Ablation Study
To justify the multi-agent design, we compare Holmes with a commonly used paste-the-stack-into-a-chatbot baseline and conduct
Holmes (Full) w/o Low-level Artifacts w/o Attention Guidance w/o Thread Agent w/o Log Agent w/o Code Explorer Vanilla DeepSeek-V3.1
Function Accuracy
Root Cause Accuracy
87.6% (64/73) 84.2% (61/73) 81.5% (59/73) 78.1% (57/73) 83.6% (61/73) 68.5% (50/73) 42.5% (31/73)
65.7% (48/73) 57.5% (42/73) 56.2% (41/73) 63.0% (46/73) 52.1% (38/73) 60.3% (44/73) 35.6% (26/73)
ablation studies (Table 4). We report Pass@1 for function-level fault localization and root-cause identification based on our labeled dataset. Baseline: We compared Holmes against Vanilla DeepSeekV3.1 (Zero-shot). The Vanilla baseline uses only raw crash stacks and exceptions, lacking repository retrieval, logs, or thread data. We excluded other state-of-the-art tools (e.g., SWE-agent [17], RepoGraph [14]) and standard RAG approaches [8, 12] because they are inapplicable to our industrial setting: they either require reproducible environments (unavailable in post-mortem scenarios) or cannot scale to 70-million-line codebases due to the prohibitive cost of global graph or vector index construction. Ablation Variants: w/o Log Agent (no runtime logs); w/o Thread Agent (no cross-thread evidence); w/o Code Explorer (stack-visible code only); w/o Attention Guidance (no file:line anchor). w/o Low-level Artifacts (no registers/memory/assembly).
Conference’17, July 2017, Washington, DC, USA
Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, and Yuetang Deng
Table 5: Per-category accuracy breakdown (𝑁 = 73). 𝑛: number of cases per category. Category
𝑛
Memory Error Logic Error External & Env Concurrency Resource Mgmt
39 17 10 4 3
Cat. Acc. RC Acc. Func. Acc. 97.4% 58.8% 30.0% 100% 100%
76.9% 47.1% 30.0% 100% 100%
94.9% 82.4% 60.0% 100% 100%
File Acc. Line Acc. 94.9% 82.4% 80.0% 100% 100%
89.7% 82.4% 80.0% 100% 100%
Table 6: Per-root-cause accuracy for root causes with 𝑛 ≥ 2. Root Cause
𝑛
RC Acc.
Func. Acc.
Null Pointer Dereference 22 Garbage Pointer 10 Assertion Failure 9 Improper API Usage 8 Buffer Overflow 5 Race Condition 4
72.7% 100% 33.3% 37.5% 80.0% 100%
95.5% 90.0% 77.8% 75.0% 100% 100%
Analysis: Holmes (Full) balances high-recall retrieval with constrained reasoning. Multimodal data provides complementary causal views, while bounded exploration completes logical links missing from static snapshots, significantly reducing hallucinations. w/o Low-level Artifacts: Excluding low-level signals (registers, assembly) severs the link between high-level logic and binary execution states. This is particularly detrimental for mixed-source crashes (e.g., JNI, system frameworks), where register values are often the only clue to validate hypotheses, leading to a notable drop in root cause accuracy. w/o Attention Guidance: Performance drops confirm that blame-frame anchoring is a critical attention gate. Lacking file:line constraints leads to attention diffusion, prone to misattributing errors to irrelevant glue code. w/o Thread Agent: Missing thread data hinders the detection of concurrency interference (e.g., cross-thread race conditions, deadlock precursors). Since root causes are often non-local, removing this dimension causes the model to overfit to surface symptoms of the victim thread, thereby missing true concurrency accomplices. w/o Log Agent: Logs provide necessary runtime constraints (e.g., execution paths, I/O states, feature flags). Without logs, the model cannot reconstruct the scene, degrading reasoning to static guessing based on code, making it difficult to pinpoint specific root causes. w/o Code Explorer: A 19.1% drop in localization accuracy confirms that on-demand exploration is vital for reconstructing causal chains. Relying solely on stack code leads to context truncation, unable to trace upstream logic errors. Vanilla DeepSeek-V3.1: Performs worst, revealing the limitations of weak-evidence reasoning. Lacking repository context and runtime states, LLMs can only make probabilistic guesses, unable to establish reliable symptom-defect mappings. Conclusion: Experiments reveal that standalone LLM reasoning fails in industrial RCA due to the lack of runtime context. By dynamically integrating logs, threads, and code via multi-agent collaboration, Holmes reconstructs fragmented clues into causal chains, proving that full-stack evidence alignment is key to high-precision automated diagnosis.
Root Cause
𝑛
RC Acc.
Func. Acc.
Unrecognized Selector Unhandled Exception Corrupted Input Invalid Free Null Parameter Main Thread Violation
3 2 2 2 2 2
100% 50.0% 0.0% 0.0% 0.0% 100%
66.7% 100% 0.0% 100% 100% 100%
3.4
Performance Analysis
We analyzed Holmes’ performance on 39,795 production runs (Table 7, Figure 2). 3.4.1 Statistical Summary. Latency (Time Cost): With a median latency of 73.44s and mean of 77.02s (Std. Dev. 24.38s), Holmes reduces Time-to-Insight by >98% compared to manual investigation (2–3 hours). Input Tokens (Cost): Median input is stable at ∼232k tokens (Mean 267k, Std. Dev. 142k). The converted USD cost is ∼$0.13 per session. This indicates high cost-effectiveness for large-scale deployment. Output Tokens (Summarization): Output length is consistent (Mean 1,803, Median 1,768, Std. Dev. 320), summarizing contexts into ∼1,800 tokens. 3.4.2 Correlation. Pearson coefficients (Table 8) reveal no significant correlation between input tokens and latency (-0.01), confirming that the Map-Reduce architecture mitigates computational time costs of large contexts. Near-zero correlation between input and output tokens indicates output length depends on diagnostic complexity, not input volume. A moderate positive correlation (0.34) exists between latency and output tokens.
3.5
Improvement on the State of the Practice
Beyond accuracy metrics, we evaluated Holmes’ comprehensive impact on engineering efficiency, cost, and developer adoption. 3.5.1 Efficiency Gains. Manual Workflow: Traditional diagnosis involves symbolication, manual git grep across 70M+ LOC, and log filtering. Internal telemetry shows this takes 2–3 hours per complex crash. Holmes Workflow: The system generates a report in ∼77 seconds (Mean latency). Impact: A 98%+ reduction in “Time to Insight,” transforming developers from investigators to verifiers. 3.5.2 Cost Efficiency Analysis. Running Holmes is orders of magnitude cheaper than human effort. Resource Consumption: Due to the multi-agent parallel architecture, processing a single crash consumes an average of ∼267k input tokens (heavy log/code context) and ∼1.8k output tokens. ROI Calculation: Comparing the salary
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale
Response Latency (seconds) - Distribution
4000
6000
2000 1500 1000
2000
4000
2000
500 200
300
400
0
500
Response Latency (seconds) - Box Plot
400
400k
Input Tokens
500k
200
200k
300k
Input Tokens
400k
0
1k
2k
3k
4k
Output Tokens
5k
6k
Output Tokens - Box Plot 6k
300k 200k 100k
100
0
500k
Input Tokens - Box Plot
500
300
100k
Output Tokens
100
Response Latency (seconds)
Response Latency (seconds)
Output Tokens - Distribution
Frequency
6000
0
Input Tokens - Distribution
2500
Frequency
Frequency
8000
Conference’17, July 2017, Washington, DC, USA
5k 4k 3k 2k 1k 0
Figure 2: Statistical distribution of key performance metrics over 39,795 production runs. Table 8: Pearson correlation coefficients.
Table 7: Statistical distribution of performance metrics. Metric
Latency (s)
Input (Tokens)
Output (Tokens)
Mean Median Std. Dev.
77.02 73.44 24.38
267,194 231,575 141,970
1,803 1,768 320
cost of a senior engineer spending 2.5 hours (estimated > $70) versus the API/GPU cost of a session (∼$0.13), the cost reduction approaches 99%. This makes it economically viable to run Holmes on every new crash cluster. 3.5.3 Developer Adoption & Feedback. In a one-month pilot deployment with 45+ developers: Adoption: The active adoption rate for complex crash tickets has reached 92%. Feedback: One senior developer noted: “I used to dread assigning these Heisenbugs; now I check Holmes first.” Acceptance: Developers accepted the AI’s root cause analysis in 78% of the cases without needing further manual log inspection.
3.6
Case Study: Race Condition in MagicBrush Engine
We present a representative case involving a race condition in the MagicBrush engine to illustrate Holmes’s reasoning transparency. In this scenario, the application crashed with a SIGBUS error during module destruction. Holmes’s agents collaborated to reconstruct the failure: the Log Miner and Thread Inspector identified a temporal conflict between the crashing thread (attempting to post a message) and a background thread (destroying the shared service), while the Code Explorer confirmed that the shared
Correlation Pair Latency ↔ Output Tokens Latency ↔ Input Tokens Input Tokens ↔ Output Tokens
Coefficient 0.34 (Moderate positive) -0.01 (None) -0.00 (None)
dictionary bizToServiceMessages was accessed without synchronization. The Synthesis Layer correctly diagnosed an atomicity violation where an object was deallocated by the background thread while being accessed by the main thread, and suggested adding an NSRecursiveLock, which resolved the issue.
4 Discussion 4.1 Limitations & Failure Analysis Privacy-Induced Information Entropy Reduction: Strict privacy redaction (e.g., GDPR) removes sensitive data essential for diagnosis. In Case #10, Holmes misdiagnosed corrupted input as a generic memory error because the redacted message payload prevented tracing data dependencies. The Heisenbug Nature of Concurrency: Race conditions are often invisible in static snapshots. In Case #7, Holmes analyzed the victim thread in isolation, missing the temporal state inconsistency and misdiagnosing an assertion failure as a garbage pointer issue. Semantic Gap in Polyglot Runtimes: In hybrid stacks (e.g., JNI, FFI), semantic fidelity can be lost across boundaries. In Case #1, Holmes failed to interpret a high-level Swift exception and reverted to a low-level C++ error pattern (null pointer dereference). Non-Determinism of Memory Corruption: Diagnosing memory safety issues often requires allocation history, but heavy instrumentation tools like AddressSanitizer or Malloc Stack Logging
Conference’17, July 2017, Washington, DC, USA
are unavailable in production due to excessive overhead. In Case #12, Holmes observed only the crash consequence (invalid access) rather than the antecedent buffer overflow, causing a misdiagnosis.
Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, and Yuetang Deng
distributions, suggesting that the architectural constraints (e.g., blame-frame anchoring) effectively mitigate stochastic variations.
5 4.2
Lessons Learned
Architectural Determinism vs. Open-Ended Reasoning: Ablation studies show that unstructured reasoning leads to attention drift (e.g., removing blame-frame anchoring drops accuracy by 6.1%). The 3+1+1 architecture decomposes diagnosis into atomic, verifiable steps (Retrieval -> Exploration -> Synthesis), ensuring evidence chain integrity and reducing hallucinations compared to concatenating all artifacts into a single prompt. The Necessity of Semantic Filtering: Feeding raw logs (e.g., 10k lines) triggers ‘lost-in-the-middle’ issues. Semantic Map-Reduce is essential for RCA; filtering logs by relevance to the crash context improves the signal-to-noise ratio, enabling detection of subtle correlations. The Economic Value of Negative Feedback: In industrial settings, false positives are costlier than false negatives. We explicitly instruct agents to report “evidence not found” rather than forcing a plausible but wrong explanation. This conservative bias builds developer trust and makes diagnoses actionable.
4.3
Threats to Validity
Internal Validity. The primary threat to internal validity lies in the potential subjectivity of the ground truth labels. Although we employed a rigorous cross-validation process with two senior developers to annotate the root causes and defect locations, manual labeling inherently carries a risk of human error or bias. To mitigate this, we resolved disagreements through discussion and excluded cases that were ambiguous. External Validity. The results are based on data from a single large-scale application (WeChat iOS). While the dataset covers diverse modules (networking, storage, UI) and failure modes, the generalization to other platforms (e.g., Android, Server-side) remains to be fully verified. However, the core contribution—the 3-Stream reasoning paradigm—is platform-agnostic. By abstracting runtime artifacts into universal categories (Stack, Log, Thread) and decoupling retrieval from reasoning, Holmes can be adapted to other systems by simply integrating the respective symbolication toolchains (e.g., ProGuard/R8 mappings for Android, dSYM for iOS, DWARF for Linux), mitigating the risk of overfitting to a specific OS. Construct Validity. We used Pass@1 accuracy and Time-toInsight as primary metrics. While Pass@1 is a standard metric for fault localization, it may not fully capture the utility of the generated explanations or fix suggestions. A correct location with a misleading explanation could still hinder developers. To address this, we included a qualitative evaluation of the evidence chain and fix suggestions within our user study. Reliability Validity. Large Language Models exhibit inherent non-determinism. Even with a fixed temperature (0.6), Holmes may produce slightly different reasoning paths for the same input across runs. To ensure robustness, we evaluated the system on a large-scale production dataset (𝑁 = 39, 795) and observed stable performance
Related Work
We categorize the research landscape into three primary domains that address the challenges of automated diagnosis in large-scale systems.
5.1
Evolution of Crash Analysis: From Text Similarity to Semantic Reasoning
Industry tools such as Firebase Crashlytics and Sentry focus on crash clustering to handle large volumes of reports, and ReBucket [5] provides a basis for call stack similarity metrics. In log analysis, LogAnomaly [13] and LogPPT [11] leverage sequence modeling and prompt-based few-shot learning to detect anomalies. More recent work, including LogLLM [7] and FaithLog [9], further advances log understanding: LogLLM uses general-purpose LLMs to replace weaker pattern-matching pipelines, while FaithLog stresses diagnostic faithfulness, requiring the evidence chain to be verifiable against raw artifacts. However, most log-centric methods still treat logs as an isolated modality, lacking repository-scale, cross-modal reasoning over crash stacks and source code; consequently, they struggle to map runtime symptoms to specific defect-causing code locations. Holmes addresses this gap by combining an industrial Map-Reduce strategy with semantic filtering to integrate logs, stack traces, and code, bridging the semantic disconnect between runtime observations and navigation of repository-scale code.
5.2
Repository-Scale Navigation: Static Indexes vs. Agentic Exploration
Navigating massive codebases is a core challenge for SE agents. Approaches such as CodePlan [2], RepoGraph [14], and LocAgent [4] build fine-grained global indexes via static repository analysis to support more reliable context construction and cross-file reasoning. In this context, fine-grained global indexes refer to highresolution semantic structures, such as complete Abstract Syntax Trees (ASTs), symbol cross-references (def-use / call-site) graphs, and pre-computed inter-procedural control-flow / call relations. While powerful, maintaining such dense indexes for production environments with millions of functions is prohibitively expensive due to the immense performance overhead and the volatility of industrial codebases. Recent empirical studies [6] have attempted to enhance crash reports by utilizing LLMs to navigate codebases via such pre-built call graphs. This persistent reliance on static modeling reflects a broader limitation shared by traditional static analysis tools; for instance, Facebook Infer [3] identifies potential defects based on code patterns but frequently suffers from severe scalability bottlenecks and a lack of runtime perspective. In ultralarge-scale repositories, the computational complexity of global inter-procedural analysis becomes intractable, and the absence of dynamic context often leads to high false-positive rates. Holmes addresses these scalability and precision bottlenecks by replacing dense semantic indexing with an agentic exploration paradigm supported by a shallow, lightweight function-to-file-path lookup table. By acting as a human investigator equipped with on-demand
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale
retrieval tools, Holmes autonomously navigates a 70M LOC repository, delegating logical cross-referencing to the agent’s dynamic reasoning instead of a brittle pre-calculated graph.
5.3
The Runtime Dependency Bottleneck: Sandboxed Execution vs. Post-mortem Reasoning
Advancements in automated debugging generally fall into two categories: Workflow-based Fault Localization and Autonomous SE Agents. Traditional Spectrum-based Fault Localization (SBFL) [1, 10] relies on comparing execution traces from passing and failing tests. Modern LLM-based frameworks, such as Agentless [16] automate this navigation but still rely on a dynamic loop to verify hypotheses within a sandbox. Similarly, autonomous SE agents like SWE-agent [17] and Openhands [15] follow an active coding paradigm where agents plan, edit, and verify fixes within a runnable environment. A common limitation of these approaches is their heavy dependency on reproducible runtime environments and failure-inducing tests, which are often unavailable in production crash scenarios. Holmes addresses this post-mortem challenge without re-running the application or relying on trial-and-error verification. Instead, Holmes employs an investigative reasoning paradigm to reconstruct the failure context solely from sparse, readonly artifacts (such as logs and stack snapshots) within a frozen, massive repository.
6
Conclusion
We have presented Holmes, a multi-agent framework that addresses the scalability and reproducibility challenges of post-mortem crash diagnosis in ultra-large-scale industrial systems. By synthesizing multimodal runtime signals—from high-level logs to low-level register states—Holmes bridges the semantic gap in mixed-source environments and efficiently navigates 70M+ LOC repositories without reproduction. Empirical evaluation on WeChat demonstrates 87.6% fault localization accuracy and a 98% reduction in investigation overhead, shifting the workflow from manual debugging to efficient verification. While validated on iOS, the underlying paradigm is platform-agnostic. Future work will focus on deploying lightweight, specialized models on local devices to enhance privacy and latency, as well as integrating automated reproduction and fix generation to achieve a closed-loop self-healing ecosystem.
Conference’17, July 2017, Washington, DC, USA
References [1] Rui Abreu, Peter Zoeteweij, and Arjan J.C. van Gemund. 2007. On the Accuracy of Spectrum-based Fault Localization. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007). 89– 98. doi:10.1109/TAIC.PART.2007.13 [2] Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D. C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. 2023. CodePlan: Repository-level Coding Using LLMs and Planning. arXiv preprint arXiv:2309.12499 (2023). https://arxiv.org/abs/2309.12499 [3] Cristiano Calcagno, Dino Distefano, Jeremy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter O’Hearn, Irene Papakonstantinou, Jim Purbrick, and Dulma Rodriguez. 2015. Moving Fast with Software Verification. In NASA Formal Methods. Springer International Publishing, Cham, 3–11. [4] Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. 2025. LocAgent: GraphGuided LLM Agents for Code Localization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 8697–8727. doi:10. 18653/v1/2025.acl-long.426 [5] Yingnong Dang, Rongxin Wu, Hongyu Zhang, Dongmei Zhang, and Peter Nobel. 2012. ReBucket: A method for clustering duplicate crash reports based on call stack similarity. In 2012 34th International Conference on Software Engineering (ICSE). 1084–1093. doi:10.1109/ICSE.2012.6227111 [6] S. M. Farah Al Fahim, Md Nakhla Rafi, Zeyang Ma, Dong Jae Kim, Tse-Hsun, and Chen. 2025. Crash Report Enhancement with Large Language Models: An Empirical Study. arXiv preprint arXiv:2509.13535 (2025). https://arxiv.org/abs/ 2509.13535 [7] Wei Guan, Jian Cao, Shiyou Qian, Jianqi Gao, and Chun Ouyang. 2025. LogLLM: Log-based Anomaly Detection Using Large Language Models. arXiv preprint arXiv:2411.08561 (2025). https://arxiv.org/abs/2411.08561 [8] Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A. Rossi, Subhabrata Mukherjee, Xianfeng Tang, Qi He, Zhigang Hua, Bo Long, Tong Zhao, Neil Shah, Amin Javari, Yinglong Xia, and Jiliang Tang. 2025. Retrieval-Augmented Generation with Graphs (GraphRAG). arXiv preprint arXiv:2501.00309 (2025). https://arxiv.org/abs/2501.00309 [9] Minghua He, Tong Jia, Chiming Duan, Pei Xiao, Lingzhe Zhang, Kangjin Wang, Yifan Wu, Ying Li, and Gang Huang. 2025. Walk the Talk: Is Your Logbased Software Reliability Maintenance System Really Reliable? arXiv preprint arXiv:2509.24352 (2025). https://arxiv.org/abs/2509.24352 [10] James A. Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE ’05). Association for Computing Machinery, 273–282. doi:10.1145/1101908.1101949 [11] Van-Hoang Le and Hongyu Zhang. 2023. Log Parsing with Prompt-based Fewshot Learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2438–2449. doi:10.1109/ICSE48619.2023.00204 [12] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS ’20). Curran Associates Inc., 793. [13] Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, and Rong Zhou. 2019. Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). AAAI Press, 4739–4745. [14] Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2025. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. arXiv preprint arXiv:2410.14684 (2025). https://arxiv.org/abs/2410.14684 [15] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint arXiv:2407.16741 (2025). https: //arxiv.org/abs/2407.16741 [16] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489 (2024). https://arxiv.org/abs/2407.01489 [17] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 50528–50652. doi:10.52202/079017-1601