Can Large Language Models Generate Observability-Aware Code? Yongliang Tao1 , Hongyu Zhang1 , Pengfei Gao2 , Minghua Ma2 , Zhiyu Fan2 , Yu Kang2 Jue Zhang2 , Si Qin2 , Liqun Li2 , Qingwei Lin2 , Saravan Rajmohan2
arXiv:2607.05785v1 [cs.SE] 7 Jul 2026
1
Chongqing University, Chongqing, China 2 Microsoft
Abstract—Recent advances in coding agents have enabled the generation of increasingly complex software systems. While existing evaluations primarily focus on functional correctness, production systems must expose failure evidence to support observability. In this paper, we present a systematic study of observability in agent-generated systems. We examine whether agents can reconstruct source-level diagnostic semantics by restoring observability artifacts in 10 open-source and 8 industrial repositories. We also evaluate whether these artifacts translate into effective fault signals at runtime through 200 generated microservice systems deployed on Kubernetes with 13 injected faults. Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals (i.e., explicit, fault-specific evidence) at runtime. At the source level, agents partially recover observability artifacts but struggle to capture key diagnostic semantics. At runtime, generated systems expose fault signals for only a small fraction of failures (up to 13.99%), despite the presence of logging, suggesting that the generated observability artifacts may lack the failure-specific semantics needed to effectively expose faults. We further introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure, but the gains remain limited, indicating that the gap is not easily addressed. More broadly, our findings suggest that current evaluations focusing primarily on functional correctness may overlook observability as an important dimension of practical software quality. Index Terms—Coding Agent, Fault Signal, Observability
I. I NTRODUCTION Recent advances in coding agents, such as Copilot [1], Cursor [2], and Claude Code [3], have made it increasingly feasible to generate complete software systems. These systems may compile, deploy, and run successfully under normal workloads. However, software that is runnable is not necessarily software that is operable. When failures occur, developers need sufficient information to understand what happened. This shift from runnability to operability brings failure-time observability into focus. Beyond generating runnable code, agents should also generate observability artifacts (i.e., logs, traces, and metrics) that expose fault-indicating runtime evidence. In human-written software systems, even though developers may have a substantial understanding of the system, unexpected production failures still occur, making observability a fundamental need for debugging.
Coding agents further increase developers’ need for observability. While they can generate large amounts of code quickly, developers may not inspect or reason about all of the generated code in detail. As a result, developers may not fully understand the runtime behavior of the generated system. This creates knowledge debt: developers may need to maintain generated systems whose runtime behavior they do not fully understand. Knowledge debt makes observability more difficult to improve after systems are deployed. When developers lack comprehensive understanding of the runtime behavior of agentgenerated systems, they may lack the semantic context needed to determine what diagnostic semantics (i.e., failure-relevant context captured by observability artifacts) should be encoded. This is because observability design requires reasoning about potential failure modes and identifying relevant runtime states, which depends on a deep understanding of system behavior. Consequently, post-hoc instrumentation may fail to capture a failure-relevant context and, therefore, fail to expose the corresponding fault signal during failures. We therefore suggest that observability should be treated as a generation-time expectation for agent-generated code. As coding agents implement functionality, they should simultaneously generate observability artifacts, so that diagnostic semantics are encoded during code generation and the fault signal can be exposed when failures occur. This reframes observability from a post-hoc remediation activity into an integral part of code generation. Accordingly, this paper asks whether current coding agents generate code that is not only runnable but also observable under failure? To answer this question, we connect source-level diagnostic semantics (i.e., failure-relevant context captured by observability artifacts, such as error codes or request IDs) with the fault signal (i.e., explicit, fault-specific evidence available during failure-time execution, such as a service crash observed in logs). Source-level analysis evaluates whether generated observability artifacts encode meaningful diagnostic semantics, while runtime analysis evaluates whether the generated systems expose the corresponding fault signal under injected failures. Together, these two perspectives enable a comprehensive evaluation of observability in agent-generated code. Accordingly, we structure our study around the following research questions:
RQ1 (Static Observability Restoration): Can coding agents generate observability artifacts consistent with human-written observability artifacts? Method. We conduct a controlled source-level restoration study on 10 open-source and 8 industrial systems with mature human-written observability artifacts, covering 1,223 instances in total. We remove existing observability artifacts while preserving the original business logic, ask coding agents to restore them using the remaining repository context, and evaluate both the placement of generated observability artifacts and the diagnostic semantics they capture. Findings. Coding agents recover observability only partially. They are substantially better at identifying where observability artifacts should be placed than what diagnostic semantics should be recorded, revealing a persistent semantic observability gap even when the functional business logic is already available. RQ2 (Observability in Agent-Generated Microservice Systems): To what extent do end-to-end agent-generated microservice systems expose fault signals under realistic failure scenarios? Method. We generate 200 microservice systems from highlevel specifications using coding agents with SOTA LLMs, deploy them in Kubernetes, and inject 13 representative production faults, resulting in 1,615 failure instances. We collect runtime logs during each fault window and measure Fault Signals Rate (FSR), the fraction of injected faults that produce explicit fault signals in logs. Findings. Agent-generated systems produce fault signals for only a small fraction of failures, with FSR ranging from 4.95% to 13.99% across models. The limitation is not the absence of logging, but the systematic absence of explicit fault-indicating semantics and diagnostic context in the generated logs. RQ3 (Observability-Oriented Guidance): Can lightweight observability-oriented guidance (Skill) improve both static diagnostic semantics and runtime fault signals exposure? Method. We derive a lightweight observability-oriented skill from approximately 200 real-world commits, summarizing diagnosis-oriented practices (e.g., instrumentation placement and diagnostic data capture) as structured generation guidance. We then re-evaluate the enhanced agents under both the source-level restoration (RQ1) and runtime fault-injection (RQ2) settings. Findings. Observability-oriented guidance yields only marginal improvements in static diagnostic semantics, ranging from 0.004 to 0.012 across models. For runtime fault exposure, while FSR increases modestly for GPT-5.5 (+8.67 pp), the gains for Claude Opus 4.8 (+0.99 pp) and Gemini 3.5 Flash (+2.54 pp) remain minimal, leaving the overall FSR relatively low (up to 16.53%). These limited gains suggest that lightweight generation-time guidance may partially mitigate, but is unlikely to fully resolve, the semantic observability gap, indicating substantial room for more effective methods. The major contributions of the paper are as follows: • A multi-level framework for evaluating agentgenerated observability. We introduce a multi-level eval-
uation framework that examines agent-generated code from two complementary perspectives. Specifically, the framework evaluates, at the static level, the ability of coding agents to restore observability artifacts in terms of their placement and the diagnostic semantics they capture; at the dynamic level, it assesses whether the generated observability implementations can effectively expose fault signals under real system deployment and fault injection scenarios. • Empirical evidence of a semantic observability gap. Across 10 open-source and 8 industrial systems, agents recover human-written observability only partially: they are better at placing observability artifacts than recovering diagnostic semantics. In 200 generated microservice systems and 1,615 injected failure instances, logs are widely generated but often uninformative, failing to expose meaningful fault signals under failure conditions. As a result, only a small fraction of failures can be associated with actionable runtime evidence, indicating a substantial gap in failure-time observability. • Implications for observability-aware code generation. Our results show that current coding agents exhibit insufficient observability in generated code. Lightweight observability-oriented guidance improves fault signal exposure but still leaves a substantial gap, suggesting that future coding agents and benchmarks should treat observability as a first-class objective in code generation and evaluation. II. BACKGROUND AND M OTIVATION A. Agent-Generated Code Creates Knowledge Debt The increasing adoption of coding agents changes how developers acquire operational understanding of software systems. Traditionally, implementation and understanding are tightly coupled: by constructing software themselves, developers gradually build mental models of execution flow, dependency interactions, and potential failure modes. Coding agents weaken this coupling and further increase developers’ reliance on observability. While they can generate large amounts of code quickly, developers may not inspect or reason about all of the generated code in detail. As a result, they may not fully understand the runtime behavior of the generated system. We refer to this gap as knowledge debt: developers may need to maintain systems whose runtime behavior they do not fully understand, making diagnosing failures in production significantly more difficult. Observability has long been essential for production diagnosis. In the era of coding agents, however, runtime evidence increasingly compensates for reduced authorship-based understanding. Consequently, the key question is not only whether coding agents can generate functionally correct software, but whether they can also generate observability that supports effective failure-time diagnosis.
RQ1 Design: Static Observability Restoring Open Source Repos Industrial Repos
Agentic Regeneration
Observability Stripping
Ground Truth Comparison
Position F1 Score
KeyBag F1 Score
RQ2 Design: Observability in Agent-Generated Microservice Systems Vibe-coded microservices
Fault Injection
System Deployment
RQ3 Design: Observability-Oriented Guidance Failure Profile Extraction
Retest Enhanced Agent
Observability Skill
Analyse Runtime Logs
Retest
Fault Signal Rate
Fault Signal Rate Position F1 Score
KeyBag F1 Score
Fig. 1. Overview of Study
B. From Diagnostic Semantics to Fault Signals We study observability across two levels. At the source-code level, an observability artifact encodes diagnostic semantics: the failure-relevant context it is intended to capture, such as dependency states, error codes, request identifiers, state transitions, and variables that explain exceptional behavior. At the runtime level, observability should produce a fault signal: explicit, fault-specific evidence emitted during failuretime execution, such as a dependency becoming unavailable, a request exceeding its timeout, or an invalid state transition. In this study, observability refers to the ability of generated code to encode diagnostic semantics in source code and expose fault signals at runtime. These two levels are connected but not equivalent. Diagnostic semantics are a precondition for fault signals: instrumentation that does not encode failure-relevant context cannot expose such context at runtime. However, source-level semantics alone do not guarantee runtime evidence, because the relevant execution path must be exercised, the generated functionality must behave as intended, and the evidence must remain visible in logs. Thus, source-level analysis captures whether agents express meaningful diagnostic intent, while runtime analysis captures whether generated systems expose that intent under real failures. Evaluating coding agents across both levels is essential for understanding whether they can generate observability code that goes beyond functional correctness and exposes failurerelevant evidence in production-like settings. III. S TUDY D ESIGN A. Overview Our study investigates the following central question: to what extent can current coding agents generate code with effective observability? To answer this question, we design a three-stage evaluation pipeline that progressively examines observability from isolated capability analysis to realistic
runtime behavior and guided improvement, as illustrated in Figure 1. RQ1 (Static Observability Restoration). To isolate the underlying capability of coding agents in capturing diagnostic semantics without the confounding burden of business logic generation, we conduct a controlled source-level reconstruction study. We curate 1,223 instances from 10 open-source and 8 industrial repositories, remove their human-written observability artifacts, and task coding agents with reconstructing them. This controlled, repository-aware setting enables us to evaluate the agents’ capability to recover diagnosis-oriented observability artifacts independently of business logic synthesis. We assess reconstruction quality using Position F1 (placement accuracy) and KeyBag F1 (semantic overlap of essential diagnostic variables). RQ2 (Observability in Agent-Generated Microservice Systems). Building upon the capability analysis in RQ1, we investigate whether the observed source-level capability limitation is consistently or not reflected in realistic runtime behavior. We generate 200 microservice systems from highlevel specifications, deploy them in a Kubernetes environment, and inject 13 representative production faults, resulting in 1,615 failure instances. We analyze runtime logs using the Fault Signals Rate (FSR), which measures the proportion of injected faults that expose explicit fault signals. RQ3 (Observability-Oriented Guidance). Finally, we investigate whether structured diagnosis-oriented guidance can improve the observability capability of coding agents. We derive a lightweight observability skill from real-world observability-related commits and apply them during system generation. The enhanced agents are then re-evaluated under the same source-level restoration and runtime fault-injection settings. Together, these three research questions form a unified multi-level evaluation framework. RQ1 isolates the capability of coding agents to model diagnostic semantics in a con-
Evaluation Harness Coding Agent
Config
Generate Service
Runtime Environment
Runtime Execution
Fault Injection
Collect Runtime Signals
Fault Signal Rate
Deployment Build & Deploy to kubernetes
Continuous Requests
Fig. 2. Evaluation of Observability in Agent-Generated Microservice Systems
Fig. 3. Example of Static Observability Restoration
trolled setting. RQ2 evaluates whether this capability limitation is consistently reflected in the runtime fault-signal exposure of end-to-end generated systems. RQ3 investigates how lightweight diagnosis-oriented guidance can further improve runtime fault-signal exposure, providing insights into the development of future observability-aware coding agents. B. RQ1 Design: Static Observability Restoration Goal. RQ1 asks whether agents can restore human-written source-level observability at the code level, after the original observability statements are removed. 1) Dataset: We construct a dataset from both open-source and industrial repositories, including 10 GitHub repositories [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] (441 instances) and 8 industrial datasets (782 instances), resulting in 1,223 instances in total. The selected GitHub repositories are widely used microservice projects, generally with more than 1k stars, and are actively maintained by their communities. These projects exhibit sustained demand for observability improvements, where logging statements, telemetry instrumentation, and related observability artifacts are continuously introduced and refined to support operational diagnosis. The selected industrial repositories are maintained over extended periods (typically longer than one year), deployed in real production environments, and require observability improvements driven by incidents. In these settings, missing observability signals are often identified and introduced during incident diagnosis and operational maintenance.
2) Procedure: Instances of static observability restoration are extracted using language-specific parsers and retained if they contain at least one observability construct, including logging statements, tracing operations, metric updates, span attributes, or span events. We exclude trivial functions, test files, generated code, and build artifacts. For each instance, we remove all observability statements (e.g., logging and tracing) while preserving business logic and control-flow structures. The stripped implementation serves as the agent input, whereas the original implementation is retained as the ground truth (Figure 3). Importantly, the agent operates over the entire repository context, rather than a single function in isolation. This setting distinguishes coding agents from standalone LLMs, allowing them to exploit repository-level conventions such as logging styles, tracing schemas, and telemetry naming patterns. 3) Prompting Strategies: We evaluate three prompting configurations: Observability-blind: no explicit instruction on observability; to assess whether coding agents can proactively introduce missing observability artifacts without explicit guidance, reflecting their inherent observability awareness. Observability-hinted: explicit instruction to improve observability; to evaluate how explicit guidance influences the quality and completeness of generated observability artifacts. Few-shot: same-file examples of observability instances; to evaluate whether coding agents can learn and adopt repositoryspecific observability practices from in-context example. 4) Functional Scope: RQ1 is a static source-level restoration study. Generated code is not executed during this evaluation, and we do not assess whether the regenerated implementation preserves the original functional behavior. Instead, the evaluation focuses exclusively on whether coding agents recover human-written observability artifacts, including their placement and diagnostic semantics. 5) Evaluation Metrics: Because both the original and reconstructed implementations are available, we can evaluate observability reconstruction along two complementary aspects: where artifacts are placed and what they capture. a) Position F1 (Where to instrument): Position F1 measures whether generated observability appears at the same execution regions as the ground truth. We first align generated and ground-truth functions using non-observability statements as anchors. These anchors partition each function into buckets: before the first matched statement, between consecutive matched statements, and after the last matched statement.
Each bucket is represented by a binary indicator denoting whether it contains any observability statement. A bucket is counted as a true positive if both the ground truth and generated code contain observability in that aligned bucket, as a false positive if only the generated code does, and as a false negative if only the ground truth does. Let T Ppos , F Ppos , and F Npos denote these bucket-level counts. We compute: TP
TP
Position F1 =
2 · Position P · Position R Position P + Position R
pos pos Position P = T Ppos +F Ppos , Position R = T Ppos +F Npos .
This metric evaluates whether observability is restored at the correct workflow locations, independently of the exact diagnostic content. b) KeyBag F1 (What to capture): Position correctness alone does not guarantee that the restored observability captures the same diagnostic concepts. We therefore compute a token-level content metric over aligned buckets where both the ground truth and generated code contain observability. For each such bucket, we extract a bag of normalized lexical tokens from the observability statements. The extractor considers string literals, identifiers, attribute names, and keyword-argument names, then normalizes them by lowercasing, splitting on common separators such as dots, underscores, hyphens, slashes, and camel-case boundaries, and removing observability-framework stop words such as logger, span, and event. Let Tbgt and Tbgen be the resulting token sets for a comparable bucket b. We aggregate token-level true positives, false positives, and false negatives across all comparable buckets: T Pkey =
X
|Tbgt ∩ Tbgen |,
b
F Pkey =
X
|Tbgen \ Tbgt |, F Nkey =
b
X
|Tbgt \ Tbgen |.
b
We then compute: TP
TP
KeyBag F1 =
2 · KeyBag P · KeyBag R KeyBag P + KeyBag R
key key KeyBag P = T Pkey +F Pkey , KeyBag R = T Pkey +F Nkey
KeyBag F1 is therefore a lightweight lexical proxy for diagnostic-content agreement. It rewards generated observability that mentions the same operational concepts as the ground truth, while Position F1 separately accounts for missing or spurious instrumentation locations.
C. RQ2 Design: Observability in Agent-Generated Microservice Systems Goal. As the observability of systems generated by coding agents in practical settings remains largely unclear, RQ2 measures the Fault Signals Rate in agent-generated systems to assess whether generated systems can produce fault signals under faults. 1) Dataset: We collect 200 representative architecture specifications for cloud-native microservice applications, by abstracting architectures from real-world open-source and industrial systems, covering a diverse range of domains, including e-commerce, social networking, IoT telemetry, finance, healthcare, content management, SaaS platforms, travel, logistics, gaming, and observability/DevOps systems. Each architecture specification is provided to the coding agent as the implementation prompt, allowing the agent to automatically generate a complete deployable microservice system for subsequent runtime evaluation under fault injection. 2) Procedure: Figure 2 presents the end-to-end runtime fault-injection evaluation pipeline. For each architecture specification, the coding agent generates a deployable microservice implementation, which is subsequently executed within a standardized evaluation harness. The harness provides a unified runtime environment, deployment configuration, workload generation, and fault-injection infrastructure, ensuring that all generated systems are evaluated under the same settings. After deployment and successful health verification, each service is continuously exercised by the workload generator while representative production faults are injected using Chaos Mesh. Runtime logs are collected throughout each fault window and analyzed offline to determine whether the generated observability exposes diagnostic evidence corresponding to the injected fault. 3) Fault Injection: To evaluate runtime observability under realistic operating conditions, we inject representative production faults using Chaos Mesh [14] in a Kubernetes cluster. As summarized in Table I, our benchmark covers 13 fault primitives spanning service failures, dependency failures, network anomalies, resource contention, and time-related faults. These fault types simulate common failure scenarios encountered in cloud-native microservice systems, including service crashes, upstream failures, database and cache outages, queue disruptions, CPU contention, network corruption, and clock skew. Across the 200 generated systems, these configurations produce a total of 1,615 executable fault instances. 4) Evaluation Metrics: After fault injection, we allow sufficient time for fault propagation and collect runtime logs from all affected services. We focus on logs as the primary evaluation signal because they are the most widely used and consistently available observability artifacts produced by current coding agents, whereas traces and metrics are often incomplete or entirely absent in generated systems. As a result, logs provide the most reliable basis for assessing failure visibility. If even logs fail to expose fault signals effectively, this suggests that the observability of current coding agents is fundamentally limited.
TABLE I E VALUATED FAULT PRIMITIVES WITH INJECTION AND LOG EVIDENCE .
ID F01 F02 F03 F04 F05 F06 F07 F08 F09 F10 F11 F12 F13
Fault pod-kill network-delay upstream-fail upstream-slow db-down db-slow cache-down cache-slow queue-down queue-slow cpu-stress net-corrupt time-skew
Chaos Type PodChaos NetworkChaos HTTPChaos NetworkChaos PodChaos NetworkChaos PodChaos NetworkChaos PodChaos NetworkChaos StressChaos NetworkChaos TimeChaos
Injected Failure Kill service pod Service egress latency Upstream returns 503 Upstream latency Postgres pod kill Postgres latency Redis cache pod kill Redis cache latency Redis stream pod kill Redis stream latency CPU pressure Packet corruption Clock offset
Conservative Notion of Observability. It is important to clarify that our evaluation adopts a conservative notion of observability. We do not require logs to fully diagnose root causes or support complete Root Cause Analysis [15], [16], [17], [18], [19], [20] (RCA). Instead, a log is considered a fault signal only if it explicitly encodes failure-related semantics, such as dependency failures, timeout conditions, protocol errors, or other fault-specific indicators that can be directly attributed to the injected fault. Generic information such as request logs or standard status codes is not sufficient unless it explicitly reflects abnormal system behavior. This design intentionally avoids relying on post-hoc inference from runtime symptoms and ensures that detected signals correspond to explicit and actionable fault semantics. For each fault type, we define a set of fault-specific log evidence (see Table I) derived from the expected manifestations of the injected failure. Examples include timeout-related messages for network delays, connection-refused errors for service disruptions, and out-of-memory indicators for resource exhaustion. These signatures serve as a proxy for the observability of failures in logs. A fault instance is considered observable if at least one corresponding fault signal appears in the collected logs after fault injection. Based on this definition, we compute the Fault Signals Rate N , where NSignal denotes the number of (FSR) as F SR = NSignal fault fault instances that produce observable fault-related signals, and Nfault denotes the total number of injected fault instances Unlike traditional observability evaluations that focus on the quantity of generated logs, metrics, or traces, this evaluation measures whether generated observability artifacts actually expose runtime failures. A low FSR indicates that failures remain largely silent, preventing subsequent diagnosis regardless of the volume of generated observability artifacts. This evaluation enables us to quantify the practical effectiveness of observability in LLM-generated systems and to examine whether the observability artifacts introduced by coding agents are sufficient to surface failures in realistic deployment environments.
Summary of Fault-specific Log Evidence restart/startup evidence or restartCount increase timeout/latency/deadline evidence upstream/HTTP 503 upstream timeout/slow/deadline evidence postgres/connection refused/FATAL evidence db timeout/query latency/deadline evidence redis/connection reset/refused evidence redis timeout/slow/cache latency evidence stream/queue/redis connection failure evidence queue/stream timeout or slow publish/consume evidence CPU/load/slow processing/timeout evidence reset, broken pipe, network error evidence clock/time skew/token expiry/timestamp evidence
D. RQ3 Design: Observability-Oriented Guidance Goal. RQ3 investigates whether the static and runtime observability gaps identified in RQ1 and RQ2 can be mitigated through targeted, diagnosis-oriented observability guidance. Dataset and System. To ensure comparability with prior findings, we reuse the same set of application generation tasks and fault-injection scenarios from RQ1 and RQ2. By reusing the same workload, we can directly measure whether the proposed mitigation reduces the previously observed gaps. Procedure. We construct an observability skill by analyzing approximately 200 observability-related code commits collected from an internal failure repository. These commits were created by experienced engineers to diagnose and remediate production failures, providing practical examples of effective observability artifacts. They capture recurring diagnosis and remediation patterns, including dependency failures, timeouts, retries, and exception propagation. We abstract these recurring patterns into a set of lightweight, diagnosis-oriented principles. We then evaluate a skill-guided agent, where the observability skill is provided as structured generation guidance, and compare it against a baseline agent without such guidance. Both agents are tasked with generating or instrumenting services under the same workload and fault conditions. All other factors (model, prompts, and environment) are held constant to isolate the effect of the skill. Skill Content. Due to space constraints, we present a highlevel abstraction of the observability skill. The skill consists of diagnosis-oriented principles that guide the agent to: Identify service boundaries, responsibilities, dependencies, and operational contexts. • Instrument dependency calls and control-flow boundaries where failures or latency may occur. • Capture key diagnostic signals, including correlation IDs, status codes, latency, retry counts, timeout values, and exception causes. • Follow or establish consistent observability conventions for service, operation, dependency, and request context. • Ensure that instrumentation remains lightweight, avoids high-cardinality or sensitive data, and supports efficient •
(b) Position quality 0.8
4.9
0.6
4
2
2.2
Score
Statements / instance
(a) Obs. quantity
2.1
0.55
0.44
0.51
0.4 0.2
0
Ref.
Blind
0.0
Hinted
(c) KeyBag token quantity
Prec.
10
0.4
0
7.6
Ref.
Blind gen.
Hinted gen.
F1
Blind 0.49 0.46 Hinted
20 11.5
Rec.
(d) KeyBag quality
22.9
Score
KeyBag tokens / instance
0.54
Blind 0.69 Hinted 0.61
0.36
0.33
0.26 0.20
0.2
0.0
Prec.
Rec.
F1
Fig. 4. Quantity over Quality under observability-hinted prompt
diagnosis under failure scenarios. E. Implementation and Reproducibility We evaluate three closed-source frontier models: GPT5.5 [21], Claude Opus 4.8 [22], and Gemini 3.5 Flash [23]. All model calls are made through a unified agent interface based on GitHub Copilot SDK [24], using the model identifiers exposed by the provider at experiment time. For generatedservice experiments, all models share identical task descriptions, prompt templates, repository workspaces, and interaction protocols. The agent operates with repository access and automatic tool-permission approval, and execution traces are recorded for auditing. IV. S TUDY R ESULTS A. RQ1 Results: Static Observability Restoration To answer RQ1, we systematically investigate how coding agents perform under the three prompting strategies introduced earlier. We obtain the following findings. Finding 1: Implicit observability awareness exists, but artifact quality remains limited. As shown in Table II, coding agents demonstrate a degree of implicit observability awareness even under the observability-blind prompt, as they still recover a non-trivial portion of observability-related content. For instance, GPT-5.5 achieves a KeyBag recall of 0.458, while Claude Opus 4.8 reaches positional recall of 0.708, indicating that these models can recognize when observability artifacts are needed even without explicit guidance. However, this awareness does not translate into high-quality outputs. Under blind prompting, the KeyBag F1 scores remain low across all models (0.357 for GPT-5.5, 0.294 for Claude Opus 4.8, and 0.237 for Gemini 3.5 Flash), suggesting that the generated artifacts are frequently incomplete, noisy, or misaligned with the expected key elements.
Finding 2: Quantity over Quality under Explicit Instructions. As shown in Fig. 4, explicit instructions (i.e., observability-hinted prompts) primarily encourage increased generation rather than improved quality, compared to implicit instructions (i.e., observability-blind prompts). From the quantity perspective, Fig. 4(a) and (c) show that explicit instructions lead to substantially more generated statements (4.9 vs. 2.1) and KeyBag tokens (22.9 vs. 11.5), indicating clear over-generation. We measure quantity as the average number of generated observability statements and the number of corresponding KeyBag tokens per instance. However, this increased quantity does not translate into better quality. In Fig. 4(b) and (d), precision drops notably while recall improves only slightly. Specifically, KeyBag recall increases (0.488 vs. 0.458, +0.030), but precision decreases sharply (0.20 vs. 0.33, -0.13), resulting in a lower F1 score (0.26 vs. 0.36, -0.10). A similar pattern is observed for position quality, where F1 also declines (0.51 vs. 0.55, -0.05). This reveals a potential quantity–quality trade-off: explicit instructions push the model toward over-generation, improving coverage at the expense of correctness. Such over-generation not only reduces output quality but also introduces unnecessary computational and post-processing overhead, posing an additional burden on downstream systems. Finding 3: Contextual Few-Shot Improves Both Content and Placement Quality. As shown in Table II, Contextual few-shot prompting improves both the content quality and placement accuracy of generated observability artifacts. Compared to implicit instructions, few-shot prompting increases KeyBag F1 from 0.357 to 0.383 (+0.025) and Position F1 from 0.551 to 0.594 (+0.043). These improvements are primarily driven by substantial gains in recall (KeyBag: +0.114; Position: +0.168), while maintaining comparable precision. This suggests that few-shot examples help the model better identify relevant observability signals without introducing excessive noise. We attribute this improvement to the contextual grounding provided by in-context examples, which enables the model to better align with repository-specific observability practices and reduce ambiguity during generation. Finding 4: Placement is Easier than Content Generation. As shown in Table II, across all prompting strategies, Position F1 consistently exceeds KeyBag F1, suggesting that agents more reliably determine where to introduce observability than what observability artifacts to generate. For example, under implicit prompting, Position F1 (0.551) is significantly higher than KeyBag F1 (0.357), with similar gaps under explicit prompting (0.505 vs. 0.260) and few-shot prompting (0.594 vs. 0.383). This reveals a gap between structural understanding and content-level precision, suggesting that coding agents can capture structural signals more reliably than semantic requirements of observability content.
TABLE II P ERFORMANCE COMPARISON ACROSS PROMPTING STRATEGIES
Prompt
Model
Pos. P
Pos. R
Pos. F1
KeyBag P
KeyBag R
KeyBag F1
observability-blind prompt
GPT-5.5 Claude Opus 4.8 Gemini 3.5 Flash
0.544 0.554 0.277
0.606 0.708 0.333
0.551 0.580 0.278
0.331 0.270 0.233
0.458 0.412 0.262
0.357 0.294 0.237
observability-hinted prompt
GPT-5.5 Claude Opus 4.8 Gemini 3.5 Flash
0.442 0.426 0.269
0.695 0.842 0.438
0.505 0.528 0.307
0.204 0.243 0.176
0.488 0.476 0.288
0.260 0.291 0.197
few-shot examples prompt
GPT-5.5 Claude Opus 4.8 Gemini 3.5 Flash
0.530 0.568 0.409
0.774 0.816 0.578
0.594 0.633 0.448
0.334 0.371 0.349
0.571 0.518 0.446
0.383 0.398 0.367
TABLE III RQ2 RUNTIME OBSERVABILITY UNDER FAULT INJECTION .
Model GPT-5.5 Claude Opus 4.8 Gemini 3.5 Flash
Runnable Services
FSR
Subset FSR
151 / 200 154 / 200 136 / 200
4.95% 6.32% 13.99%
6.56% 8.28% 20.62%
Summary. Observability is not a simple instructionfollowing capability but a context-dependent generation problem. Coding agents show partial awareness of observability, but generated artifacts often lack precision and fail to record key diagnostic semantics, reflected in consistently low KeyBag F1 scores. This raises a key question: whether such source-level limitations persist in realistic settings, particularly in the ability of generated systems to expose fault signals during runtime? B. RQ2 Results: Observability in Agent-Generated Microservice Systems To answer RQ2, we evaluate runtime observability using the Fault Signal Rate (FSR), defined as the fraction of intended fault instances for which the generated system emits faultspecific log evidence during the fault window. Each model is evaluated on 1,615 intended fault instances. We report both the overall FSR and the Subset FSR (computed only over successfully executed services). Finding 1: Most injected faults remain unobservable. As shown in Table III, all models exhibit consistently low FSR, ranging from 4.95% to 13.99%. Even when restricting the evaluation to successfully executed services, the bestperforming model achieves only 20.62%, indicating that the majority of injected faults are not reflected as explicit fault signals in runtime logs. This result does not imply that generated systems lack logging. Many services emit runtime logs, including request processing events and status codes. However, these logs often do not encode fault-specific semantics that can be attributed to particular failure types. Thus, the limitation is not the absence
Fig. 5. Representative success and failure cases under the same evaluation. TABLE IV P ER -FAULT FSR U NDER FAULT I NJECTION .
Fault
Caught / Total
FSR
Difficulty
F03 (upstream fail) F01 (pod kill) F07 (cache down) F08 (cache slow)
14 / 51 122 / 600 55 / 321 46 / 321
27.45% 20.33% 17.13% 14.33%
Easy Easy Easy Easy
F10 (queue slow) F02 (network delay) F05 (db down) F09 (queue down)
18 / 198 53 / 600 32 / 420 15 / 198
9.09% 8.83% 7.62% 7.58%
Medium Medium Medium Medium
F06 (db slow) F04 (upstream slow) F12 (net corrupt) F11 (cpu stress) F13 (time skew)
25 / 420 3 / 51 10 / 555 8 / 555 7 / 555
5.95% 5.88% 1.80% 1.44% 1.26%
Hard Hard Hard Hard Hard
of logging, but the absence of explicit failure semantics in logs. Finding 2: Observable faults are primarily those with explicit error manifestations. FSR varies significantly across fault types (Table IV). Faults that naturally produce explicit errors—such as upstream failures (27.45%), pod crashes (20.33%), and cache outages (17.13%)—achieve substantially
TABLE V E FFECT OF OBSERVABILITY SKILL ON DIAGNOSTIC SEMANTICS AND FAULT SIGNALS . Model
Setting
Pos F1
KeyBag F1
FSR
GPT-5.5
w/o skill w/ skill Gain
0.594 0.597 ∆0.003
0.383 0.395 ∆0.012
4.95% 13.62% +8.67 pp
Claude Opus 4.8
w/o skill w/ skill Gain
0.633 0.649 ∆0.015
0.398 0.407 ∆0.009
6.32% 7.31% +0.99 pp
Gemini 3.5 Flash
w/o skill w/ skill Gain
0.448 0.452 ∆0.004
0.367 0.371 ∆0.004
13.99% 16.53% +2.54 pp
higher detection rates, as they directly manifest as connection failures, error responses, or service unavailability. As illustrated in our success cases (Figure 5), when such explicit errors occur, the generated logs successfully capture failure-related semantics (e.g., affected dependencies, concrete timeout messages), allowing the faults to be directly exposed as fault signals. In contrast, faults with weaker or more implicit effects—such as time skew (1.26%), CPU stress (1.44%), and network corruption (1.80%)—exhibit consistently low coverage. These faults do not inherently generate explicit error messages and can only be observed if the system proactively records additional runtime context. Finding 3: Generated logs lack diagnostic semantics. Although generated systems frequently emit runtime logs, these logs rarely include the information required to distinguish between different failure types. In particular, generated logs seldom explicitly encode affected dependencies, failure modes, or abnormal system conditions. This is clearly demonstrated in our failure cases (Figure 5): the system emits multiple runtime logs (e.g., request events, normal HTTP 201 status codes), but completely lacks explicit failure semantics (e.g., timestamp mismatches or clock drift). The logs exist in abundance, but they are semantically hollow regarding the actual fault. As a result, even when faults occur and logs are present, the available information is often insufficient to determine what kind of failure has happened. This observation is consistent with the pattern in Finding 2: faults that inherently produce explicit errors can still be detected, whereas faults requiring contextual interpretation remain unobservable because the generated logs fail to express the necessary failure-relevant information. Summary. The observability of microservices generated by current coding agents appears limited, with at most 13.99% of injected faults being exposed. This limitation does not seem to arise from the absence of logs, but is associated with the inability of logs to capture fault-relevant signals. This finding is consistent with the low KeyBag F1 scores observed in RQ1. C. RQ3 Results: Observability-Oriented Guidance RQ1 shows the limitations in generating diagnostic semantics, and RQ2 shows that generated systems exhibit low
FSR under failures. To examine whether such limitations can be easily mitigated, we evaluate whether or not an explicit observability skill can improve system observability under the same setting. The results are shown in Table V. Finding 1: Improvements in fault signals. Observability skill improves FSR across all evaluated coding agents. GPT5.5 increases from 4.95% to 13.62% (+8.67 pp), Claude Opus 4.8 from 6.32% to 7.31% (+0.99 pp), and Gemini 3.5 Flash from 13.99% to 16.53% (+2.54 pp). However, the overall FSR remains relatively low. Finding 2: Limited improvements in diagnostic semantics. In addition to FSR, both Pos F1 and KeyBag F1 show small but consistent improvements across models (see Table V), indicating that observability skill can partially enhance source-level diagnostic semantics. Summary. Overall, observability skill improves FSR, Pos F1, and KeyBag F1 to a limited extent. While these results suggest that skill-based guidance can enhance observability, the remaining gap indicates substantial room for future methods to further improve observability-aware code generation. V. D ISCUSSION AND T HREATS TO VALIDITY A. Discussion This study examines the observability of agent-generated microservice systems from two complementary perspectives. At the source-code level, observability restoration evaluates whether coding agents can recover the diagnostic semantics embedded in human-written observability code. The results show that agents are able to restore only part of these observability artifacts, indicating limited capability in encoding faultrelevant diagnostic semantics. At the system level, observability evaluation measures whether the generated systems expose actionable fault signals during failures. Despite producing runnable systems with abundant logging output, the generated systems expose fault signals for only a small fraction of injected faults. Taken together, these findings suggest that current coding agents remain insufficient in generating observability that effectively supports failure diagnosis. To investigate whether this limitation can be mitigated during generation, we introduce lightweight observabilityoriented skills. Although these skills consistently improve FSR, Position F1, and KeyBag F1, the overall gains remain modest. This suggests that simply augmenting prompts with observability-related guidance is insufficient to substantially improve diagnostic quality. Instead, generating effective observability appears to require deeper reasoning about failure propagation, runtime states, and diagnostic semantics, which current prompt-level interventions do not fully provide. Overall, our findings indicate that observability should be considered an independent capability of coding agents rather than a by-product of functional code generation. Future research may require integrating failure-aware reasoning or runtime feedback into the generation process, instead of relying solely on prompt engineering.
One possible explanation is that observability is rarely represented as an independent objective in the training data. In realworld software repositories, logging and other observabilityrelated code are typically developed together with functional code and are gradually refined during software evolution. Consequently, coding agents may learn where observability components are usually inserted, allowing them to reproduce common logging structures, while failing to infer what diagnostic information should be recorded for specific failures. Another possible explanation is that current coding agents primarily learn from static source code rather than runtime system behavior. Effective observability depends on understanding how failures manifest during execution and which runtime states provide actionable diagnostic evidence. Without deployment or execution feedback during training, agents may lack sufficient knowledge to reason about failure propagation, resulting in logs that execute correctly but rarely expose informative fault signals. B. Threats to Validity We have identified the following major threats to validity. Oracle-based FSR may miss valid fault signals. The definition of Fault Signals Rate (FSR) relies on fault-specific oracles to identify valid fault signals. Although the oracle is constructed based on failure-relevant patterns, it may not capture all possible forms of valid signals. In particular, some logs may contain partial or indirect information that does not satisfy the oracle criteria and is therefore counted as no signal. However, since the same oracle is consistently applied across all evaluated settings, and our goal is to measure the presence of actionable fault signals rather than arbitrary logging outputs, the relative comparisons and overall trends remain stable. Human-written observability does not represent an optimal upper bound. Our study derives diagnostic semantics (KeyBags) from human-written code. Many of these observability-related signals are introduced or refined after faults have occurred, reflecting practical debugging and operational processes. While such code captures realistic engineering practice, it does not guarantee complete or optimal observability. In some cases, human-written systems may still omit fault-relevant signals or include redundant logging. Therefore, our evaluation reflects alignment with practical engineering standards rather than an ideal ground truth. Importantly, coding agents still fail to reproduce these practical diagnostic patterns even under simplified settings such as observability reconstruction, suggesting that the observed limitation is unlikely to be caused by the choice of reference. Randomness of LLM-based generation may affect FSR. Coding agents based on large language models exhibit inherent randomness in generation. Variations in decoding may affect the placement and content of generated logs, which in turn can influence the observed FSR. To mitigate this effect, we follow previous works [25] and repeat each experiment five times, reporting the average results. We further note that LLM hallucination [26], [27] may introduce additional variability in generated logs; however, the consistency of trends across
repeated runs suggests that randomness does not materially affect our conclusions. VI. RELATED WORK Evaluation of Coding Agents: The rapid advancement of large language models has led to a wide range of benchmarks [28], [29], [30], [31], [32] for evaluating coding agents. Early studies primarily focused on function-level code generation, with datasets such as HumanEval [33] and TerminalBench [34] assessing functional correctness through unit test pass rates. These benchmarks emphasize whether generated programs execute correctly, but largely abstract away systemlevel complexity. More recent repository-level benchmarks such as SWE-Bench [35] extend evaluation to realistic software engineering tasks, where agents are required to resolve issues in full GitHub repositories. This line of work better captures engineering complexity, but evaluation remains largely centered on task completion or patch correctness. As a result, they provide limited insight into runtime behavior under failure conditions, particularly whether generated systems expose fault signals for debugging. This reveals a gap between functional correctness-oriented evaluation and observability in real-world deployment scenarios. Software Observability and Logging Practices: Software observability, particularly logging, is a fundamental practice for understanding and debugging complex systems. Prior work shows that effective logs should provide rich contextual information, such as variable states, dependency interactions, and error propagation paths, rather than simple execution traces [36]. As systems evolve toward distributed and microservice architectures, observability becomes increasingly critical, since failures often arise from cross-service interactions [37]. Moreover, logging is typically not designed upfront but evolves iteratively as developers encounter failures and augment instrumentation with additional diagnostic context. Log Analysis and Instrumentation: Automated log analysis has evolved from heuristic methods to recent deep learning and large language models, demonstrating strong semantic reasoning in Log Parsing [38], [39], [40], Log Anomaly Detection [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], and Root Cause Analysis [51], [52], [53]. Despite these advances, log analysis and instrumentation remain largely decoupled. Existing instrumentation approaches focus on surface-level patterns and rarely ensure that generated logs encode sufficient failure-relevant semantics for effective runtime diagnosis. As a result, whether automated coding agents can generate observability artifacts with rich diagnostic semantics remains an open question. VII. C ONCLUSION In this paper, we present a systematic study of observability in code generated by coding agents. The lack of diagnostic semantics observed in source-level observability restoration, together with the low FSR under fault injection, suggests that current coding agents generate code with limited observability. We further introduce a lightweight observability-
oriented guidance, which improves both diagnostic semantics and fault signals exposure. While these results demonstrate the potential of observability-oriented guidance, they also show that the improvements remain limited, and reveal substantial room for future research toward enabling coding agents to consistently generate observability-aware code. We call for future research to focus on enabling coding agents to generate observability-aware code that can consistently produce fault signals reflecting underlying failures. R EFERENCES [1] Microsoft, “Microsoft copilot,” https://copilot.microsoft.com, 2026. [2] Cursor, “Cursor,” https://cursor.com, 2026. [3] Anthropic, “Claude code,” https://www.anthropic.com/claude-code, 2026, accessed: 2026. [4] OpenTelemetry, “opentelemetry-demo,” gitHub repository. Available: https://github.com/open-telemetry/opentelemetry-demo. Accessed: July 2026. [5] GoogleCloudPlatform, “microservices-demo,” gitHub repository. Available: https://github.com/GoogleCloudPlatform/microservices-demo. Accessed: July 2026. [6] Delimitrou Lab, “Deathstarbench,” gitHub repository. Available: https: //github.com/delimitrou/DeathStarBench. Accessed: July 2026. [7] dotnet, “eshop,” gitHub repository. Available: https://github.com/dotnet/ eShop. Accessed: July 2026. [8] golevelup, “nestjs,” gitHub repository. Available: https://github.com/ golevelup/nestjs. Accessed: July 2026. [9] Robusta Dev, “robusta,” gitHub repository. Available: https://github.com/ robusta-dev/robusta. Accessed: July 2026. [10] microservices-demo, “microservices-demo,” gitHub repository. Available: https://github.com/microservices-demo/microservices-demo. Accessed: July 2026. [11] FudanSELab, “train-ticket,” gitHub repository. Available: https://github. com/FudanSELab/train-ticket. Accessed: July 2026. [12] Strapi, “strapi,” gitHub repository. Available: https://github.com/strapi/ strapi. Accessed: July 2026. [13] Vector.dev, “vector,” gitHub repository. Available: https://github.com/ vectordotdev/vector. Accessed: July 2026. [14] C. M. Authors, “Chaos mesh,” https://github.com/chaos-mesh/ chaos-mesh, gitHub repository, accessed July 2026. [15] Y. Sui, Y. Zhang, J. Sun, T. Xu, S. Zhang, Z. Li, Y. Sun, F. Guo, J. Shen, Y. Zhang et al., “Logkg: Log failure diagnosis through knowledge graph,” IEEE Transactions on Services Computing (TSC), 2023. [16] J. Huang, J. Liu, Z. Chen, Z. Jiang, Y. Li, J. Gu, C. Feng, Z. Yang, Y. Yang, and M. R. Lyu, “Faultprofit: Hierarchical fault profiling of incident tickets in large-scale cloud systems,” ser. ICSE-SEIP ’24. Association for Computing Machinery, 2024, p. 392–404. [17] Q. Zhang, T. Jia, Z. Wu, Q. Wu, L. Jia, D. Li, Y. Tao, and Y. Xiao, “Fault localization for microservice applications with system logs and monitoring metrics,” in 2022 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), 2022, pp. 149–154. [18] Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zeng, S. Ghosh, X. Zhang, C. Zhang, Q. Lin, S. Rajmohan, D. Zhang, and T. Xu, “Automatic root cause analysis via large language models for cloud incidents,” ser. EuroSys ’24, New York, NY, USA, 2024, p. 674–688. [19] D. Liu, C. He, X. Peng, F. Lin, C. Zhang, S. Gong, Z. Li, J. Ou, and Z. Wu, “Microhecl: High-efficient root cause localization in largescale microservice systems,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2021, pp. 338–347. [20] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, D. Liu, Q. Xiang, and C. He, “Latent error prediction and fault localization for microservice applications by learning from system trace logs,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 683–694. [21] OpenAI, “Openai official website,” https://openai.com, 2026. [22] Anthropic, “Claude,” https://www.anthropic.com/claude, 2026.
[23] G. DeepMind, “Gemini,” https://deepmind.google/technologies/gemini/, 2026. [24] GitHub, “Copilot sdk,” https://github.com/github/copilot-sdk, gitHub repository, accessed July 2026. [25] Y. Liu, Y. Ji, S. Tao, M. He, W. Meng, S. Zhang, Y. Sun, Y. Xie, B. Chen, and H. Yang, “Loglm: From task-based to instruction-based automated log analysis,” 2025. [Online]. Available: https://arxiv.org/abs/2410.09352 [26] B. Galitsky, A. Chernyavskiy, and D. Ilvovsky, “Truth-o-meter: Handling multiple inconsistent sources repairing llm hallucinations,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, p. 2817–2821. [27] C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13. [28] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” 2024. [Online]. Available: https://arxiv.org/abs/2403.07974 [29] H. Chen, C. Li, and J. Li, “Featbench: Towards more realistic evaluation of feature-level code generation.” [Online]. Available: https://arxiv.org/abs/2509.22237 [30] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 [31] J. Li, G. Li, Y. Zhao, Y. Li, H. Liu, H. Zhu, L. Wang, K. Liu, Z. Fang, L. Wang, J. Ding, X. Zhang, Y. Zhu, Y. Dong, Z. Jin, B. Li, F. Huang, and Y. Li, “Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,” 2024. [Online]. Available: https://arxiv.org/abs/2405.19856 [32] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3623316 [33] M. Chen, J. Tworek, H. Jun et al., “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107. 03374 [34] M. A. Merrill, A. G. Shaw, N. Carlini et al., “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” 2026. [Online]. Available: https://arxiv.org/abs/2601.11868 [35] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” 2024. [Online]. Available: https://arxiv.org/abs/2310. 06770 [36] Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie, “Where do developers log? an empirical study on logging practices in industry,” pp. 24–33, 2014. [Online]. Available: https://doi.org/10.1145/2591062.2591175 [37] U. Faseeha, H. Jamil Syed, F. Samad, S. Zehra, and H. Ahmed, “Observability in microservices: An in-depth exploration of frameworks, challenges, and deployment paradigms,” IEEE Access, vol. 13, pp. 72 011–72 039, 2025. [38] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in 2017 IEEE International Conference on Web Services (ICWS). IEEE, 2017, pp. 33–40. [39] Z. Jiang, J. Liu, J. Huang, Y. Li, Y. Huo, J. Gu, Z. Chen, J. Zhu, and M. R. Lyu, “A large-scale evaluation for log parsing techniques: How far are we?” in Proc. International Symposium on Software Testing and Analysis (ISSTA), 2024. [40] Z. Ma, A. R. Chen, D. J. Kim, T.-H. Chen, and S. Wang, “Llmparser: An exploratory study on using large language models for log parsing,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ACM, Apr. 2024. [Online]. Available: http://dx.doi.org/10.1145/3597503.3639150 [41] W. Guan, J. Cao, S. Qian, J. Gao, and C. Ouyang, “Logllm: Log-based anomaly detection using large language models.” [Online]. Available: https://arxiv.org/abs/2411.08561 [42] Y. Liu, S. Tao, W. Meng, F. Yao, X. Zhao, and H. Yang, “Logprompt: Prompt engineering towards zero-shot and interpretable log analysis,” in
2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2024, pp. 364–365. [43] J. Pan, W. S. Liang, and Y. Yidi, “Raglog: Log anomaly detection using retrieval augmented generation,” in 2024 IEEE World Forum on Public Safety Technology (WFPST), 2024, pp. 169–174. [44] X. Li, P. Chen, L. Jing, Z. He, and G. Yu, “Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020, pp. 92–103. [45] Q. Lin, H. Zhang, J.-G. Lou, Y. Zhang, and X. Chen, “Log clustering based problem identification for online service systems,” in Proceedings of the 38th International Conference on Software Engineering Companion, ser. ICSE ’16, 2016, p. 102–111. [46] Y. Huo, C. Lee, Y. Su, S. Shan, J. Liu, and M. R. Lyu, “Evlog: Identifying anomalous logs over software evolution,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 391–402. [47] J.-G. Lou, Q. Fu, S. Yang, Y. Xu, and J. Li, “Mining invariants from console logs for system problem detection.” in USENIX Annual Technical Conference, 2010, pp. 1–14. [48] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, p. 1285–1298. [49] H. Studiawan, F. Sohel, and C. Payne, “Anomaly detection in operating system logs with deep learning-based sentiment analysis,” IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 5, pp. 2136–2148, 2021. [50] L. Yang, J. Chen, Z. Wang, W. Wang, J. Jiang, X. Dong, and W. Zhang, “Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2021, pp. 230–231. [51] C. Lee, T. Yang, Z. Chen, Y. Su, and M. R. Lyu, “Eadro: An endto-end troubleshooting framework for microservices on multi-source data,” in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23, 2023, p. 1750–1762. [52] C. Zhang, X. Peng, C. Sha, K. Zhang, Z. Fu, X. Wu, Q. Lin, and D. Zhang, “Deeptralog: Trace-log combined microservice anomaly detection through graph-based deep learning,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 2022, pp. 623–634. [53] Y. Meng, S. Zhang, Y. Sun, R. Zhang, Z. Hu, Y. Zhang, C. Jia, Z. Wang, and D. Pei, “Localizing failure root causes in a microservice through causality inference,” in 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS), 2020, pp. 1–10.