arXiv CSopen access

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection Huanchi Wang, Zihang Huang, Yifang Tian, Kristina Dzeparoska, Hans-Arno Jacobsen and Alberto Leon-Garcia

arXiv:2605.22779v1 [cs.SE] 21 May 2026

Department of Electrical and Computer Engineering University of Toronto, Toronto, Ontario {huanchi.wang, zihang.huang, yifang.tian,kristina.dzeparoska}@mail.utoronto.ca, [email protected], [email protected]

Abstract—Production systems generate millions of log lines daily, yet most anomaly detectors operate at the session or window-level, flagging groups of lines rather than identifying the specific message responsible. This coarse granularity forces operators to inspect many routine lines per alert. Messagelevel detection offers finer granularity, but remains challenging. A single event template may correspond to both normal and anomalous messages, failures arise from heterogeneous subsystems, and line-level labeling at scale is impractical. Although large language models (LLMs) can reason over log semantics, applying them to every line is too costly for continuous monitoring. We present FAME (Failure-Aware Mixture-of-Experts), a labelefficient message-level mixture-of-experts framework that uses an LLM only once offline. We annotate at most K labeled lines per template to derive binary normal/anomaly indicators and representative examples. The LLM proposes a partition of templates into failure domains, and a certification step validates the proposal before training. FAME trains a lightweight router and domain experts that run on-premise and output anomaly predictions and failure-domain labels. On BGL, FAME achieves F1 = 98.16 at K = 100 reducing annotation effort by 76× and detects 86.3% of anomalies from unseen EventIDs. On Thunderbird, FAME reaches F1 = 99.95 with perfect recall. Index Terms—Log Anomaly Detection, Message-level Detection, Expert Routing, Few-shot Learning, Mixture-of-Experts, Large Language Models, On-premise Inference

I. I NTRODUCTION Modern distributed systems generate tens of thousands of log lines per hour across heterogeneous components such as kernels, storage controllers, schedulers, and application services [1]. These messages record execution events, state transitions, warnings, and errors, forming the primary foundation for runtime observability. Anomalies that signal hardware faults, process crashes, or service degradation are typically rare, often accounting for less than 5% of generated lines. This severe class imbalance makes fine-grained log anomaly detection a persistent operational challenge [2]. Most deployed log anomaly detectors operate at the window- or session-level, assigning a single anomaly label to a contiguous group of lines delimited by a session key or a fixed time window [3]–[5]. While computationally convenient, this granularity imposes substantial downstream cost. A windowlevel alert localizes suspicion to a region rather than a specific message, and dismissing a false alarm typically requires reviewing hundreds of routine log lines. In one production setting that motivates our work (hereafter “Company A”), each

WINDOW-LEVEL DETECTION VS. MESSAGE-LEVEL DETECTION (Coarse-grained) (Fine-grained) ... 2005.06.03 R02-M1-N0-C:J12-U11 2005-06-03-15.42.50.363779 R02-M1-N0-C:J12-U11 RAS KERNEL INFO instruction cache parity error corrected 2005.06.03 R27-M1-L3-U18-C 2005-06-03-15.47.20.852959 R27-M1L3-U18-C RAS LINKCARD INFO MidplaneSwitchController performing bit sparing on R27-M1-L3-U18-C bit 3 2005.06.03 R20-M1-N5-C:J17-U01 2005-06-03-15.51.25.712950 R20-M1-N5-C:J17-U01 RAS KERNEL INFO generating core.304 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.50.687138 R16-M1-N2-C:J17-U01 RAS KERNEL INFO 1 ddr errors(s) detected and corrected on rank 0, symbol 0, bit 3 2005.06.03 R13-M0-NA-C:J14-U01 2005-06-03-16.01.50.728205 R13-M0-NA-C:J14-U01 RAS KERNEL INFO 3450051 L3 EDRAM error(s) (dcr 0x0157) detected and corrected 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.50.945374 R16-M1-N2-C:J17-U01 RAS KERNEL INFO CE sym 0, at 0x0b8580c0, mask 0x10 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.51.141964 R16-M1-N2-C:J17-U01 RAS KERNEL INFO total of 1 ddr error(s) detected and corrected 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.08.01.248664 R16-M1-N2-C:J17-U01 APPREAD ... RAS APP FATAL ciod: failed to read message prefix on control stream (CioStream socket to 172.16.96.116:33399) 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.12.00.444970 R16-M1-N2-C:J17-U01 RAS KERNEL INFO ddr: activating redundant bit steering: rank=0 symbol=0 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.12.01.215908 R16-M1-N2-C:J17-U01 RAS KERNEL INFO ddr: excessive soft failures, consider replacing the card 2005.06.03 R26-M0-NC-I:J18-U11 2005-06-03-16.17.40.190183 R26-M0-NC-I:J18-U11 RAS APP FATAL ciod: Error loading /p/gb2/stella/RAPTOR/65641/raptor: invalid or missing program image ...

Single alert with multiple lines

... 2005.06.03 R02-M1-N0-C:J12-U11 2005-06-03-15.42.50.363779 R02-M1-N0-C:J12-U11 RAS KERNEL INFO instruction cache parity error corrected 2005.06.03 R27-M1-L3-U18-C 2005-06-03-15.47.20.852959 R27-M1L3-U18-C RAS LINKCARD INFO MidplaneSwitchController performing bit sparing on R27-M1-L3-U18-C bit 3 2005.06.03 R20-M1-N5-C:J17-U01 2005-06-03-15.51.25.712950 R20-M1-N5-C:J17-U01 RAS KERNEL INFO generating core.304 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.50.687138 R16-M1-N2-C:J17-U01 RAS KERNEL INFO 1 ddr errors(s) detected and corrected on rank 0, symbol 0, bit 3 2005.06.03 R13-M0-NA-C:J14-U01 2005-06-03-16.01.50.728205 R13-M0-NA-C:J14-U01 RAS KERNEL INFO 3450051 L3 EDRAM error(s) (dcr 0x0157) detected and corrected 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.50.945374 R16-M1-N2-C:J17-U01 RAS KERNEL INFO CE sym 0, at 0x0b8580c0, mask 0x10 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.01.51.141964 R16-M1-N2-C:J17-U01 RAS KERNEL INFO total of 1 ddr error(s) detected and corrected 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.08.01.248664 R16-M1-N2-C:J17-U01 APPREAD ... RAS APP FATAL ciod: failed to read message prefix on control stream (CioStream socket to 172.16.96.116:33399) 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.12.00.444970 R16-M1-N2-C:J17-U01 RAS KERNEL INFO ddr: activating redundant bit steering: rank=0 symbol=0 2005.06.03 R16-M1-N2-C:J17-U01 2005-06-03-16.12.01.215908 R16-M1-N2-C:J17-U01 RAS KERNEL INFO ddr: excessive soft failures, consider replacing the card 2005.06.03 R26-M0-NC-I:J18-U11 2005-06-03-16.17.40.190183 R26-M0-NC-I:J18-U11 RAS APP FATAL ciod: Error loading /p/gb2/stella/RAPTOR/65641/raptor: invalid or missing program image ...

Direct fault message localization

Fig. 1. Window-level vs message-level in log anomaly detection

false-positive window forces a manual review of roughly 300 interleaved lines, consuming on the order of 2.5 engineer-hours before clearance. Under realistic alarm rates, the cumulative review cost of false positives can exceed the investigation budget for genuine incidents. Figure 1 contrasts the two granularities. More broadly, in interleaved production streams where session boundaries are weak or unavailable [6], per-message output is often the only actionable granularity. We therefore target message-level (synonymously, line-level) anomaly detection, where each raw log message is individually classified as normal or anomalous. Three properties of production log streams make this task substantially harder than its windowlevel counterpart. 1) C1: Template ambiguity. A single parser-derived event template can host both benign and anomalous instances depending on parameter values. Template identity alone cannot separate normal from anomalous lines [7]. 2) C2: Heterogeneous failure modes. Production anomalies span memory errors, filesystem faults, network partitions, and software exceptions. A single global classifier must fit one decision boundary across all these populations, trading recall against precision under severe imbalance. 3) C3: Label scarcity. Labeling a production stream at message granularity requires reviewing millions of lines,

an annotation effort that is impractical in most operational settings. Each major family of prior work addresses some of these properties but leaves the combined problem open. Template and feature-based detectors [7]–[9] discard parameters and conflate benign and anomalous instances of the same template (C1). Neural sequence detectors [3]–[5], [10], [11] aggregate evidence over a session or window, producing group-level labels that do not localize the responsible line (C1, C2). LLM-based detectors [12]–[14] reason over log semantics effectively but incur prohibitive API cost, latency, and dataegress overhead at per-line production scale (C3). Section VI develops these comparisons in detail. We present FAME (Failure-Aware Mixture-of-Experts), a framework for message-level log anomaly detection built on a failure-aware mixture-of-experts (MoE) architecture. The core insight is that LLM reasoning is needed only once. During an offline setup phase, the LLM partitions parser-derived event templates into semantically coherent failure domains that group templates sharing a subsystem or failure mechanism. After this one-time partitioning, the LLM is never invoked again, and all deployment-time inference runs on lightweight on-premise models. FAME is structured as a sparse MoE with top-1 routing [15], [16], with two departures from the standard paradigm. First, experts are semantically grounded: each corresponds to a named failure domain proposed by the LLM, rather than emerging implicitly from end-to-end training. This decomposition addresses C2 by letting each expert fit a decision boundary over a coherent sub-population instead of the union of all failure types. Second, the architecture is asymmetric. Under K-shot sampling with a rare-anomaly prior, observing all-anomalous labels in a domain is statistically conclusive, while observing all-normal labels is not (Section II-C). FAME exploits this asymmetry: pure-anomaly domains are resolved by routing alone, while mixed domains employ independently calibrated BERT [17] classifiers. As a byproduct, the routing label tags every detection with its failure domain, providing a diagnostic signal for downstream triage. To control annotation cost (C3), FAME requires at most K labeled lines per event template. At the best result K = 100 on BGL [18], this yields 53,287 labeled lines, an 76× reduction relative to fully labeling the 4.7M-line offline region. On Thunderbird [19], where anomalous templates are largely disjoint from normal ones, FAME reaches F1 = 99.95. We report Thunderbird as a closed-world sanity check, with BGL as the primary evidence for the realistic intra-template-mixed regime. The main contributions are organized as follows: • Message-level detection with failure-domain routing. FAME produces both a binary anomaly decision and a named failure-domain label for every log line, enabling direct fault localization without session reconstruction. • Failure-aware asymmetric MoE. We formalize an asymmetric confidence property of K-shot sampling under the rare-anomaly prior, and use it to design an MoE in which pure-anomaly domains are resolved by routing alone while

mixed domains employ per-message classifiers. Expert boundaries are defined by a one-time LLM partitioning and validated deterministically against K-shot labels. • Fully on-premise inference with minimal labeling. After a single offline LLM invocation, all inference runs on local models, eliminating recurring API cost and data egress. At K = 100, FAME requires 76× fewer labels than full annotation and reaches F1 = 98.16 (BGL) and F1 = 99.95 (Thunderbird), processing up to 1.20M lines/hour on a single 4-GPU node at a one-time cost of $10.23 versus $6,698–$10,047 per run for frontier-LLM inference. Semantic routing further enables generalization to unseen templates, detecting 86.3% of anomalies from event types absent during setup. II. P RELIMINARIES A. Problem Formulation Message-level log anomaly detection takes as input a single raw log message x ∈ X and produces a binary prediction ŷ ∈ {0, 1}, where ŷ = 1 indicates an anomaly. We consider this task under a K-shot labeling regime: during an offline setup phase, messages are parsed into EventIDs e ∈ E (e.g., via Drain [20]), and at most K labeled messages are available per EventID, together with additional unlabeled offline data. Evaluation uses standard per-message precision, recall, and F1 . Beyond the binary decision, FAME additionally produces a routing label ĉ ∈ {1, . . . , C} identifying the assigned failure domain, a semantically coherent group of log events sharing a subsystem or failure mechanism. This routing output is a design goal of FAME (not part of the general task definition) and is evaluated qualitatively in Section IV. B. Sparse Mixture-of-Experts In a sparse mixture-of-experts (MoE) model [15], [16], a gating network routes each input to the top-k (typically k = 1) of C expert sub-networks, decoupling model capacity from per-example compute. The canonical recipe trains gate and experts jointly with a load-balancing loss to prevent expert collapse. A notable variant replaces the learned gate with deterministic domain identity. DEMix Layers [21] assign each expert to a known text genre and route by genre label, trading adaptive gating for interpretability and eliminating the collapse problem entirely. FAME follows this domain-grounded lineage, treating failure domains in logs as the analogue of text genres in DEMix, but introduces two modifications. 1) Decoupled, supervised training. The gate and selector are trained as classifiers over failure-domain labels. Each expert is then trained independently after routing is frozen, rather than jointly with the gate. 2) Asymmetric routing. Domains whose K-shot samples contain only anomalous evidence carry no trained classifier and are resolved by routing alone. Mixed domains each have an independently calibrated classifier. The statistical basis for this design choice is developed next.

2

C. Asymmetric Confidence under K-Shot Sampling

(A) Offiline Setup

Let an EventID e contain N lines, of which K are labeled, and let p ≪ 1 denote the prior probability that any individual line is anomalous.

Raw System Logs

Observation (Asymmetric K-Shot Confidence). Consider two extreme outcomes of K-shot sampling from e. All K labels are anomalous. The probability that K independent draws from a normal-dominated population are all anomalous is pK , which for any reasonable K (e.g., K ≥ 10) and p ≪ 1 is negligible. The unlabeled remainder therefore almost surely contains no normal lines, and routing alone suffices as the detection mechanism. All K labels are normal. Seeing K normal lines does not rule out the possibility that the EventID still contains a small number of anomalies that the sample simply did not hit. A trained classifier is needed to recover these hidden anomalies.

Drain3 Parsing K-Shot Sampling

LLM Semantic Expert Proposal

Partition Certification

Two-Phase Expert Training

Router Training

Threshold Calibration

Incoming Log Messages

The two cases are fundamentally asymmetric. An allanomalous K-shot sample is conclusive, while an all-normal sample is not. This asymmetry provides the statistical justification for FAME’s MoE design, in which pure-anomaly domains are resolved by routing while mixed domains require per-message classifiers. Section III develops the full pipeline.

Router

Experts

Gate

Pure-Anomaly Expert

Normal

Mixed Expert

Expert Selector

(B) Online Inference

Universal Normal Expert

Anomaly (Failure Domain)

Fig. 2. FAME system architecture. (a) Offline setup: raw logs are parsed by Drain3, K-shot labels are sampled, an LLM proposes a failure-domain partition that is then certified, and two-phase BERT experts are trained alongside a DistilBERT gate and selector. This stage is executed once. (b) Online inference: the trained router directs each incoming log line to the appropriate expert. All inference runs on-premise with no LLM dependency.

III. M ETHODOLOGY FAME operates in two stages (Figure 2). An offline setup stage, executed once, builds the entire detection pipeline: it parses raw logs into event templates, samples K-shot labels, partitions templates into failure domains, trains a lightweight router and per-domain experts, and calibrates decision thresholds. An online inference stage then routes each incoming log line through the trained pipeline with no further LLM involvement. The remainder of this section follows the offline pipeline in execution order, then describes the inference procedure. We extend the notation of Section II-A. Let {(xi , yi , ei )}N i=1 denote the chronologically ordered offline log lines, where ei ∈ E is the Drain-parsed EventID of line xi . The first 85% of lines form the offline region and the final 15% the held-out test set. Failure domains are indexed by c ∈ {1, . . . , C}, with the distinguished class u denoting UNIVERSAL_NORMAL. The certified partition is a deterministic mapping π : E → {1, . . . , C}, and each domain carries a type flag ρc ∈ {0, 1} (ρc = 1 for pure-anomaly, ρc = 0 for mixed). The PU-normal pool Inormal contains all offline lines not labeled anomalous in the K-shot sample.

K most recent labeled lines in the offline region to reduce temporal mismatch with deployment. These samples are split chronologically 80/20 per EventID into a training subset (used in Phase 2 fine-tuning, Section III-D) and a calibration subset (reserved for threshold calibration, Section III-E). From each EventID’s sample, we derive two binary signals, H AS N OR MAL S IGNAL and H AS A NOMALY S IGNAL, and extract at most one representative normal and one anomalous line. These signals and representatives are the only inputs to the LLM in the next stage. B. Failure-Domain Partitioning A single global classifier over the full log vocabulary faces the heterogeneous-failure-mode problem (C2): it must place one decision boundary across memory errors, filesystem faults, network events, and software exceptions simultaneously. FAME addresses this by decomposing the log space into semantically coherent failure domains, each narrow enough for a focused expert. The decomposition proceeds in two steps: an LLM proposes an initial grouping, and a deterministic certification procedure validates it against K-shot evidence. 1) LLM-Guided Semantic Expert Proposal: The LLM receives a per-EventID statistics table containing the total offline line count, the binary signals, and the representative lines described above. A structured prompt instructs the LLM to group EventIDs by underlying failure mechanism, ignoring boilerplate tokens such as severity labels, node identifiers,

A. Log Parsing and K-Shot Sampling Before any model is trained, raw messages must be abstracted into templates so that structural patterns become visible. We strip label tokens to prevent leakage, then apply Drain3 [20] with similarity threshold 0.5 and tree depth 4 (fixed across all datasets). Each message xi is mapped to an EventID ei that replaces instance-specific tokens with wildcards while preserving structural semantics. To control annotation cost (C3), we limit labeling to at most K lines per EventID, deterministically selecting the

3

and hexadecimal payloads. The LLM determinesSthe number C of groups C and outputs a partition E = c=1 Ec that must include one group named UNIVERSAL_NORMAL. Any unassigned EventID defaults to UNIVERSAL_NORMAL. The resulting partition is fixed and reused for all subsequent training and evaluation. 2) Feasibility Validation and Partition Certification: Because the LLM output is non-deterministic, we treat it as a proposal and subject it to deterministic feasibility validation and deterministic validation using the K-shot binary signals. For each failure-domain class c ̸= u:

used without subsampling. To address class imbalance, class weights are defined as

(i) Pure-anomaly expert (ρc = 1). All EventIDs in Ec satisfy H AS A NOMALY S IGNAL= 1 and H AS N ORMAL S IG NAL= 0. In this case, routing itself serves as the detector and no expert model is trained. We require pure-anomaly experts to be sufficiently distinct from the UNIVERSAL_NORMAL normal pool under TF-IDF cosine similarity. Otherwise the group is dissolved into UNIVERSAL_NORMAL. (ii) Mixed expert (ρc = 0). At least one EventID in Ec has H AS N ORMAL S IGNAL= 1. Both Phase 1 and Phase 2 training are applied. (iii) No anomaly signal. If all EventIDs in Ec satisfy H AS A NOMALY S IGNAL= 0, the group is dissolved into UNIVERSAL_NORMAL.

D. Two-Phase Expert Training

wc =

Ntotal , (C − 1) · countc

(1)

where countc is the number of training lines in expert c. Training stops when overall accuracy on the router validation split reaches 0.80. It also stops if this metric fails to improve for one epoch.

UNIVERSAL_NORMAL serves as both the default inference path for non-expert lines and a catch-all anomaly detector for anomalies missed by the gate. Each mixed expert with ρc = 0, including UNIVERSAL_NORMAL, undergoes both Phase 1 and Phase 2 training. Pure-anomaly experts with ρc = 1 require no expert model. 1) Phase 1: Domain-adaptive MLM pre-training: BERT’s original pre-training corpus is general web text, which is a poor lexical match for log syntax. To adapt the language model to log vocabulary, we continue pre-training bert-base-uncased with masked language modeling (masking rate 15%) [17]. The pre-training corpus for all experts is the UNIVERSAL_NORMAL PU-normal pool Inormal ∩ {i : ei ∈ Eu }, capped at 200,000 uniformly sampled lines. Using a shared normal-only corpus serves two purposes: it avoids contaminating the language model with anomalous patterns, and it allows all experts to share a single Phase 1 checkpoint rather than pre-training independently. 2) Phase 2: Supervised fine-tuning: A binary classification head is added on top of the BERT [CLS] representation. Following Sun et al. [23], we freeze the embedding layer and the first 10 of 12 transformer layers. We train only the top two layers and the head. For each mixed expert c, training uses labeled anomaly lines from the 80% K-shot training split together with normal lines drawn from the UNIVERSAL_NORMAL PUnormal pool. For UNIVERSAL_NORMAL, the anomaly set includes all expert EventIDs so that it can act as a fallback detector for misrouted anomalies. The normal pool is capped at 10× the anomaly count for UNIVERSAL_NORMAL and 20× for expert experts. We optimize Focal Loss with γ = 2 and α = 0.75, which emphasizes hard anomaly examples under severe class imbalance:

After certification, the partition is frozen as the mapping π together with the type flags {ρc }. C. Router Training With the partition certified, the next task is to learn a router that directs each line to the correct domain at inference time. The router is trained before expert models so that routing quality can be verified before the more expensive expert-training stage. Because the router evaluates every log line, both of its components are instantiated as lightweight distilbert-base-uncased classifiers [22]. 1) Gate: The gate gϕ : X → [0, 1] is a binary DistilBERT classifier that predicts whether a line belongs to any expert domain. Its labels are yigate = 1[π(ei ) ̸= u]. All expert lines are retained. UNIVERSAL_NORMAL lines are subsampled to a 3:1 ratio to reduce training cost. We finetune all DistilBERT layers with a linear head using Focal Loss with γ = 2 and α = 0.75. We fix the routing threshold at τgate = 0.5. Training stops once validation recall at this threshold reaches 0.95. This corresponds to fewer than 5% of expert-domain lines in the router validation split being misrouted to UNIVERSAL_NORMAL. Because the gate is trained on a subsampled class distribution but deployed on the natural distribution, deployment-time precision may differ from training estimates. This distribution shift is partially compensated by the UNIVERSAL_NORMAL expert, which acts as a fallback detector (Section III-F). 2) Selector: The selector hψ : X → {1, . . . , C} \ {u} is a multiclass DistilBERT classifier that routes expert-domain lines to a specific expert. All expert-domain training lines are

Lfocal (p̂, y) = −αt (1 − p̂t )γ log p̂t ,

(2)

where p̂t = p̂ if y = 1 and 1−p̂ otherwise, and αt = α if y = 1 and 1 − α otherwise. Small datasets with fewer than 4,000 total lines are trained for a fixed 500 gradient steps. AUROCbased early stopping is checked every 50 steps. Larger datasets use epoch-based training with validation AUROC and patience P2 = 3. The anomaly score is sc (xi ) = σ(fθc (xi )) ∈ [0, 1].

4

(3)

a Phase 2 training subset and a held-out calibration subset. We report Precision, Recall, F1, and AUROC as percentages on a 0–100 scale for readability; for example, 98.16 corresponds to 0.9816 in the standard [0,1] representation. Two decimal places are retained throughout because, at the scale of our test sets, a difference of 0.01 in reported F1 corresponds to roughly 70 individual lines, so small changes in false positives or false negatives remain meaningful and are not rounding noise. FAME and its ablations use the calibration protocol described in Section III. External score-based baselines are calibrated on the same held-out subset using validation-based threshold selection. We evaluate on BGL [18] and Thunderbird [19] datasets because both provide native line-level anomaly labels and represent two different regimes. BGL is harder because many templates mix normal and anomalous lines across heterogeneous subsystems. Thunderbird is easier because anomalous EventIDs are largely distinct from normal ones. Many widely used log benchmarks, such as HDFS, provide only sessionalor block-level labels and therefore do not support direct linelevel evaluation without extra relabeling assumptions. a) Blue Gene/L (BGL): The BGL dataset contains 4,747,963 log lines collected from an IBM Blue Gene/L supercomputer, of which 348,460 (7.3%) are labeled anomalous. The test partition contains 712,195 lines including 46,278 anomalies. b) Thunderbird: Thunderbird is a 5M-line subset of the Thunderbird supercomputer log, with 7.6% anomalous lines. The test partition contains 750,000 lines including 56,800 anomalies.

E. Threshold Calibration Each expert produces a continuous anomaly score that must be converted to a binary decision. For each mixed expert c, the decision threshold τc is calibrated on the held-out 20% K-shot calibration subset defined above. We select τc by maximizing F1 over 1,000 score percentiles, subject to recall ≥ 0.90. If fewer than 1,000 unique scores are available, we search over all unique scores. For UNIVERSAL_NORMAL, we additionally fuse the expert score su (xi ) with the gate output gϕ (xi ): s̃u (xi ) = σ(logit(su (xi )) + w · logit(gϕ (xi ))) ,

(4)

where both w and the final threshold are selected by grid search on the same calibration subset. Because this subset is small in the low-K regime, threshold estimates may be unstable. We revisit this effect in Section IV. F. Routing and Inference At inference time, each log line follows one of three paths:   if gϕ (xi ) < 0.5, 1[s̃u (xi ) ≥ τu ] ŷi = 1 if gϕ (xi ) ≥ 0.5 and ρc∗ = 1, (5)   1[sc∗ (xi ) ≥ τc∗ ] if gϕ (xi ) ≥ 0.5 and ρc∗ = 0, where c∗ = hψ (xi ) is the selector’s chosen domain. Lines routed to UNIVERSAL_NORMAL (gate score below 0.5) are scored by the fused model s̃u and flagged without a failuredomain label. Lines routed to a pure-anomaly domain are flagged immediately. Lines routed to a mixed domain are scored by the domain’s expert against its calibrated threshold. In the latter two cases, the routing label ĉi = c∗ is emitted alongside the binary decision, providing a failure-domain annotation for downstream triage.

B. Baseline Methods We compare FAME against three non-routed line-level baselines, six frontier LLMs evaluated as direct classifiers, a nonLLM grouping variant of FAME, and architectural ablations. All methods share the same Drain3 parsing and chronological test partition. a) Drain+Random Forest (RForest): A supervised EventID-only baseline using one-hot Drain3 EventIDs and a Random Forest model trained on the same K-shot labeled lines as FAME. b) TF-IDF+Isolation Forest (IForest): An unsupervised baseline trained on TF-IDF token bigram features over the PU-normal pool. c) SBERT+Logistic Regression (LR): A global semantic baseline using frozen Sentence-BERT embeddings of raw log lines, followed by logistic regression trained on the same Kshot labeled lines [24]. d) FAME w/ TF-IDF grouping: A non-LLM variant of FAME in which the offline expert partition is produced by TF-IDF similarity rather than by an LLM. e) Direct LLM Inference: We evaluate six frontier LLMs as direct log-line classifiers, representing the alternative of always-on LLM inference without a trained detection model. For each of three LLM families, GPT, Claude, and Gemini, we select one high-capability model and one cost-efficient model

IV. E VALUATION We evaluate FAME on two widely used system log benchmarks to answer six research questions: RQ1. Does K-shot expert routing outperform single-model and classical baselines? RQ2. How does FAME compare to direct LLM inference in detection performance and deployment cost? RQ3. How does detection performance vary with the annotation budget K? RQ4. How sensitive is FAME to the choice of LLM for semantic grouping? RQ5. What is the contribution of each design component? RQ6. How reliably does FAME route anomalies to their correct failure domain, and how does it handle unseen EventIDs? A. Evaluation Protocol All methods operate strictly at the message-level, without session aggregation or temporal context. Logs are split chronologically into an offline region and a held-out test region. The first 85% of lines form the offline region and the final 15% form the test set. Within the offline region, K-shot labeled lines are sampled per EventID and further split chronologically into

5

TABLE I M ESSAGE - LEVEL ANOMALY DETECTION PERFORMANCE AT K=100 ON BGL AND T HUNDERBIRD . LLM BASELINES USE MANY- SHOT IN - CONTEXT INFERENCE WITH ANOMALY- ONLY DEMONSTRATIONS . AUROC IS OMITTED FOR LLM S BECAUSE THEY PRODUCE HARD BINARY PREDICTIONS RATHER THAN CALIBRATED SCORES .

Method

Prec.

Blue Gene/L (BGL) Rec. F1 AUROC

Prec.

Thunderbird Rec. F1

AUROC

Statistical Baselines Drain+RForest TF-IDF+IForest SBERT+LR

99.91 17.13 49.89

20.19 71.84 96.09

33.59 27.67 65.68

60.14 81.14 96.64

100.00 38.48 73.48

100.00 52.99 91.95

100.00 44.58 81.68

100.00 93.79 99.32

FAME without LLM grouping FAME w/ TF-IDF grouping

97.63

88.80

93.01

99.97

97.41

100.00

98.69

100.00

Direct LLM Inference (in-context) GPT-5.4 GPT-5-mini Claude Sonnet 4.6 Claude Haiku 4.5 Gemini 3.1 Pro Gemini 3 Flash

92.32 80.33 92.25 94.02 92.32 84.52

100.00 100.00 100.00 100.00 100.00 100.00

96.01 89.09 95.97 96.92 96.01 91.61

– – – – – –

100.00 65.48 84.29 94.41 99.48 92.59

100.00 100.00 100.00 100.00 100.00 100.00

100.00 79.14 91.47 97.13 99.74 96.15

– – – – – –

Ablations Single BERT (P1+P2) Single BERT (P2 only) Single Qwen 3.5-0.8B (P2 only) No-Gate (symmetric routing) FAME (P2 only)

17.46 36.91 7.35 95.30 96.01

98.24 39.38 88.36 87.05 99.77

29.65 38.10 13.58 90.99 97.85

95.39 96.05 78.84 99.62 99.92

11.01 13.23 7.68 99.90 99.90

91.83 90.29 100.00 100.00 100.00

19.67 23.09 14.27 99.95 99.95

68.65 64.19 60.43 100.00 100.00

FAME (K=100)

98.18

98.14

98.16

99.94

99.90

100.00

99.95

100.00

decision boundary over many different anomaly types. FAME improves on this by splitting the log space into smaller expert domains. Thunderbird is an easier dataset as a closed-world control. Drain+RForest already reaches perfect performance, which means anomalous templates are largely separable from normal ones. SBERT+LR is also strong with F1 = 81.68. IForest is weaker, which suggests that simple unsupervised scoring is not enough even when the dataset is easier. FAME should therefore not be read as uniquely solving a hard detection problem on Thunderbird. Instead, it matches near-perfect performance on an easier benchmark while yielding much larger gains on BGL. The single-model neural ablations remain far below FAME on both datasets. On BGL, this supports the value of expert routing. On Thunderbird, the contrast with Drain+RForest shows that the main issue is not lack of signal in the data. The main issue is learning one well-calibrated global detector under strong class imbalance.

in the same family. The pairs are GPT-5.4 [25] with GPT5-mini, Claude Sonnet 4.6 [26] with Claude Haiku 4.5, and Gemini 3.1 Pro [27] with Gemini 3 Flash. Each log line is submitted individually to the LLM with a structured prompt asking for a binary normal or anomaly decision. For each log line, the LLM receives at least 1 anomalous sample per EventID as in-context examples alongside the line to classify. No fine-tuning is performed. f) Single BERT (Phase 1+2): A single BERT model trained with the same two-phase protocol as FAME but without routing or specialization. All log lines are modeled by one global expert. This ablation isolates the contribution of expert routing. g) Single BERT (Phase 2 only): FAME with Phase 1 MLM pre-training disabled. Phase 2 Focal Loss finetuning starts directly from pretrained bert-base-uncased weights without routing or specialization. This ablation isolates the contribution of domain-adaptive MLM initialization. h) Single Qwen 3.5 (Phase 2 only): To test whether the weakness of a single global detector is specific to BERT, we also evaluate Qwen 3.5-0.8B. The model is fine-tuned with LoRA under the same Phase 2 objective, using the top two layers and a linear head, but without expert routing.

D. Direct LLM Inference and Deployment Cost (RQ2) For direct many-shot in-context LLM inference, we cannot simply include the full training set as labeled example lines. The examples alone would exceed 2 million tokens, making this approach impractical and extremely expensive. Instead, we construct the prompt to preserve anomaly coverage while keeping the context compact. We include at least one anomalous example from every EventID that contains anomalies, so that the LLM sees the full range of anomalous EventID types present in the offline region. When the number of anomaly-containing EventIDs is smaller than 50, we sample additional anomalous labeled lines until reaching at least

C. Single-Model and Classical Baselines (RQ1) Table I shows different behavior on the two datasets. On BGL, FAME clearly outperforms all non-routed baselines. Drain+RForest reaches F1 = 33.59. TF-IDF+IForest reaches 27.67. SBERT+LR reaches 65.68 and has AUROC = 96.64. This means that a strong global semantic representation can rank anomalies well, but still struggles to place one global

6

BGL

F1

100

100.0 K=100

96 94 92

Recall 100

97.5

98

Score (%)

Thunderbird

Precision

95.0

98

92.5

96

90.0

94

5 10 25 50 100 200 400 K (labeled lines per template)

5 10 25 50 100 200 400 K (labeled lines per template)

5 10 25 50 100 200 400 K (labeled lines per template)

Fig. 3. K-sensitivity on BGL and Thunderbird with best result at each K.

Claude Sonnet 4.6

F1

100

Kimi 2.5

GPT 5.4

Precision

100

Gemini 3.1 Pro

Recall

100

Score (%)

98 96 94

95

95

90

90

92 90

BGL

Thunderbird

85

BGL

Thunderbird

85

BGL

Thunderbird

Fig. 4. LLM grouping sensitivity on BGL and Thunderbird (K=100). Each subplot shows F1, Precision, or Recall for all four LLMs.

50 demonstrations for a fair comparison across datasets and models. We also evaluated normal-only prompts and mixed prompts containing both normal and anomalous examples. Normal-only prompting performed substantially worse. Mixed prompting performed similarly to anomaly-only prompting but required a larger context. We therefore use anomalyonly demonstrations as the strongest compact baseline. This yields prompt sizes of 4,652 tokens/call for BGL, with 96 example lines, and 2,367 tokens/call for Thunderbird, with 50 example lines. These counts include a 500-token system prompt, labeled demonstrations, and the test line. For cost estimation, we assume a 10-token output per call.

performance. This is consistent with the structural separability of that dataset. Table II shows a study on the corresponding cost of incontext LLM inference at standard API pricing [28]–[30]. A single full BGL test-set run costs $10,047 with Claude Sonnet 4.6, $8,373 with GPT-5.4, and $6,698 with Gemini 3.1 Pro. Even lower-cost variants still require $837 to $1,675 per run. On Thunderbird, costs range from $5,438 for Claude Sonnet 4.6 and $4,532 for GPT-5.4 down to $453 for GPT-5-mini. These API charges recur for every evaluation and scale linearly with log volume. By contrast, FAME incurs a one-time setup cost and inference cost of $10.23, including 7 hours of offline training, a single LLM grouping call, and the 0.6 hours of inference-time cost on GCP.

Table I shows that on BGL, direct many-shot in-context LLM inference is strong but still below FAME in F1. All LLM baselines achieve perfect recall at the cost of lower precision, suggesting they flag most lines as anomalous rather than learning a well-calibrated decision boundary. FAME achieves a better precision-recall balance, reaching F1 = 98.16 with both high recall (98.14) and high precision (98.18). Among the evaluated models, Claude Haiku 4.5 performs best with F1 = 96.92, followed by GPT-5.4 and Gemini 3.1 Pro at F1 = 96.01, while FAME reaches F1 = 98.16. Lowercost variants are less consistent, with GPT-5-mini achieving F1 = 89.09 and Gemini 3 Flash reaching F1 = 91.61. On Thunderbird, several frontier LLMs achieve near-perfect or perfect

E. K-Sensitivity Analysis (RQ3) Figures 3 show F1, Precision, and Recall as a function of K. On BGL, the best F1 is achieved at K=100. The trend is non-monotone across budgets. At very small K, the calibration subset contains too few anomalies for reliable threshold selection, making the precision–recall tradeoff unstable. At K=100, FAME reaches its strongest balance of precision and recall, and performance at K=200 is essentially equivalent. A modest decline at K=400 is driven by a precision drop with recall maintained. This pattern is consistent with expert overfitting

7

level where calibration is most sensitive to the specific K-shot sample drawn. At larger budgets (K≥100), variability reduces further (F1 std ≤ 2.13), confirming that FAME converges to a stable operating regime with sufficient labeled data. The consistently high AUROC across all trials (≥ 99.00 at every K and seed) indicates that F1 variability reflects threshold calibration sensitivity rather than learning instability. The model’s discriminative ability is robust, while the specific Kshot sample drawn for calibration determines the precisionrecall operating point. At low K, this sensitivity is amplified because the small calibration subset makes threshold selection more dependent on which anomaly examples are sampled. At high K, it diminishes as the calibration pool grows more representative.

TABLE II E STIMATED COST OF MANY- SHOT IN - CONTEXT LLM INFERENCE ON EACH TEST SET AT STANDARD API PRICING , COMPARED WITH FAME’ S ONE - TIME SETUP AND INFERENCE COST. Model

BGL (712,195 lines)

Thunderbird (750,000 lines)

$10,047 $8,373 $6,698 $3,349 $1,675 $837

$5,438 $4,532 $3,626 $1,813 $906 $453

Claude Sonnet 4.6 GPT-5.4 Gemini 3.1 Pro Claude Haiku 4.5 Gemini 3 Flash GPT-5-mini

$10.23

FAME

TABLE III A NNOTATION COST VS . FULL LINE - LEVEL LABELING AT EACH K. K

5 10 25 50 100 200 400

BGL

F. LLM Grouping Sensitivity (RQ4)

Thunderbird

Labels

Reduction

Labels

Reduction

6,554 11,420 22,711 34,799 53,287 85,900 142,690

616× 353× 178× 116× 76× 47× 28×

3,714 6,213 11,885 17,636 23,503 31,276 44,539

1144× 684× 358× 241× 181× 136× 95×

Figure 4 compares four LLMs for the offline semantic grouping step under identical pipeline settings at K=100. We select these four LLMs to cover three distinct capability and deployment profiles for the offline grouping step. GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro represent the highcapability closed-model tier. Kimi 2.5 represents a competitive open-weights model, testing whether open-weights alternatives can match closed proprietary models for semantic grouping. On BGL, Claude Sonnet 4.6 gives the strongest result among the tested grouping configurations, with F1 of 98.16, followed by Gemini 3.1 Pro at 97.78 and GPT-5.4 at 94.10. Kimi 2.5 performs lowest at 92.55. These differences reflect different grouping strategies. Gemini 3.1 Pro produces only 4 experts and reaches near-perfect recall (Rec = 99.79), but generates 2,002 false positives. GPT-5.4 produces 8 experts and achieves high precision (95.97), but misses 3,563 anomalies. Kimi 2.5 is the most conservative grouping, with only 457 false positives but 6,047 missed anomalies. This means that coarser partitions tend to favor recall, while more conservative partitions may improve precision at the cost of missed anomalies. More broadly, these results suggest that multiple valid partitions exist. Performance differences mainly reflect partition granularity and EventID boundary choices rather than all-or-nothing success of the grouping step. In practice, the grouping stage should be viewed as an offline design choice where different reasonable partitions can be proposed by different LLMs or repeated runs, and FAME remains effective across this variation. On Thunderbird, all four LLMs produce 6-expert partitions covering the same five failure categories. Small F1 differences remain, but they come from minor EventID boundary decisions rather than large structural changes. This shows that grouping quality matters most on heterogeneous datasets such as BGL. Structurally separable datasets such as Thunderbird are much less sensitive to the choice of grouping mechanism. The non-LLM FAME variant with TF-IDF grouping in Table I reinforces this conclusion: the TF-IDF baseline remains strong on both datasets, reaching F1 = 93.01 on BGL and 98.69 on Thunderbird. This indicates that FAME does not rely on LLM-based grouping as a core mechanism. Instead,

at high K. Because K-shot sampling selects the most recent labeled lines per EventID, a larger K draws from a wider temporal window that may shift the learned anomaly distribution away from the test region. We therefore recommend K=100 as the practical operating point, where annotation efficiency and detection reliability are jointly optimized. For annotationconstrained deployments, K=25 requires only 22,711 labeled lines on BGL (a 178× reduction) and achieves F1 = 97.57, offering a practical alternative when labeling resources are limited. On Thunderbird, performance improves rapidly with K and becomes near-perfect from K=25 onward. Recall is 100.00 at every K, while precision and F1 rise sharply between K=5 and K=25 and then remain essentially saturated, with only negligible fluctuations at larger budgets. Even at K=5, FAME achieves recall = 100.00 with F1 = 95.50. This means that only a small amount of line-level labeling is needed on a structurally separable dataset. The labeling cost analysis in Table III shows the annotation savings directly. At K=100 on BGL, only 53,287 lines require human review, which is a 76× reduction relative to full linelevel annotation 4,035,768 lines. This setting also achieves the strongest overall F1. We completed a run-to-run standard deviation analysis across 5 independent trials per K, each using a different random seed for model initialization and K-shot sampling. Standard deviation in F1 remains below 3.50 across all budgets, confirming that FAME’s performance is stable rather than dependent on a favourable random initialization. The highest variability occurs at K=50 (F1 std = 3.50, Recall std = 5.31), consistent with the non-monotone behaviour at that budget

8

Overall, the dominant gain in FAME comes from the routed expert architecture rather than from backbone choice alone. The additional ablations support the same interpretation. Removing the gate reduces F1 to 90.99 on BGL and is driven by a recall drop from 98.14 to 87.05, as anomalies are more often misrouted as normal logs under symmetric selection in the presence of extreme class imbalance. This confirms that asymmetric routing, in which the gate pre-filters normal-domain traffic before the selector, is an important design choice in the routing architecture. FAME (Phase 2 only) removes Phase 1 from the full pipeline while retaining all other components. On BGL, this reduces F1 from 98.16 to 97.85, a small drop driven primarily by a precision decrease (98.18 → 96.01), while recall improves slightly (98.14 → 99.77). This shows that Phase 1 MLM acts mainly as a precision booster within FAME. It gives each expert a tighter normal-domain baseline that reduces false positives, while the routing architecture retains strong recall without it. On Thunderbird, removing Phase 1 has no measurable effect, with F1 = 99.95 in both cases. This interpretation is consistent with the TF-IDF grouping variant in Table I, discussed in RQ4, which shows that expert decomposition remains effective even without LLM-based partitioning. On Thunderbird, the ablations also remain far below FAME, but this result should be interpreted together with the perfect Drain+RForest score. The main lesson is not that Thunderbird is intrinsically difficult. Rather, global neural detectors remain poorly calibrated to this low-label setting, whereas FAME is better aligned with the separable structure of the dataset. Finally, we note that in both evaluations, the partition certification step (Section III-B2) did not dissolve any expert group proposed by the LLM. All proposed partitions passed the K-shot consistency checks without triggering the fallback mechanism, indicating that the certification step primarily serves as a safeguard against non-deterministic LLM behavior rather than as a correction mechanism that is frequently invoked.

TABLE IV E XPERT DOMAINS PROPOSED BY EACH LLM FOR BGL AT K=100. P = PURE - ANOMALY ( GATE = DETECTOR ) AND M = MIXED ; UNIVERSAL NORMAL IS OMITTED . LLM

Specialist Domain

T

F1

Claude 4.6

HARDWARE MACHINE CHECK INTERRUPT DDR MEMORY ERROR KERNEL PANIC AND FATAL TERMINATION CIOD IO DAEMON ERROR LUSTRE FILESYSTEM MOUNT ERROR NETWORK LINK AND SWITCH FAILURE PROGRAM INTERRUPT AND FP EXCEPTION BGLMASTER CONTROL DAEMON FAILURE POWER AND HARDWARE DEACTIVATION

M M P P P P M M P

98.16

MEMORY HARDWARE FAILURE KERNEL FATAL ERROR FILESYSTEM MOUNT FAILURE NETWORK COMMUNICATION ERROR MIDPLANE SWITCH FAILURE CONTROL SYSTEM FAILURE PROGRAM INTERRUPT EXCEPTION

M P P P P M M

92.55

HARDWARE MACHINE CHECK ERRORS INTERRUPT AND EXCEPTION FAULTS FILESYSTEM AND STORAGE FAILURES CIOD MAILBOX COMMUNICATION ERRORS RTS KERNEL PANIC AND ASSERTIONS NETWORK FABRIC ERRORS POWER AND SWITCH HARDWARE FAULTS SYSTEM SERVICE RUNTIME CRASHES

M M P P P P P M

94.10

HARDWARE MEMORY FAILURE NETWORK COMMUNICATION FAILURE FILESYSTEM IO FAILURE KERNEL SOFTWARE FAILURE

M P P M

97.78

Kimi 2.5

GPT 5.4

Gemini 3.1 Pro

the principal structural gain comes from expert decomposition, while LLM-based grouping serves as an offline quality booster on more heterogeneous datasets and yields semantically grounded failure domains. Table IV details the expert domains proposed by each LLM for BGL at K=100. Despite differences in naming conventions, all LLMs identify the same core failure categories. The main structural difference is expert granularity. Claude Sonnet 4.6 and Kimi 2.5 separate some failure types more finely, while Gemini 3.1 Pro merges them into broader groups. G. Ablation Study (RQ5) In the ablation study, all variants use the same calibration procedure as the full FAME pipeline in Section III-E to ensure a fair comparison. At K=100 on BGL, all singlemodel ablations perform far worse than FAME. Single BERT (Phase 1+2) reaches only F1 = 29.65. Removing Phase 1 increases this to 38.10, but recall drops to 39.38. Single Qwen 3.5-0.8B performs worse still under the same calibration setting, reaching F1 = 13.58. These results indicate that a single global detector does not produce anomaly scores strong enough to meet FAME’s recall target while maintaining competitive precision on BGL’s heterogeneous anomaly space. To better isolate backbone effects, we also removed the recall-floor constraint for the single-model baselines. Under this more permissive setting, Single BERT (Phase 2 only) improves to F1 = 48.91 with recall 37.35, and Single Qwen 3.5-0.8B improves substantially to F1 = 84.61 with recall 75.55. This shows that a stronger decoder-style backbone can improve the precision–recall tradeoff of a single global detector, but it still remains well below FAME’s F1 = 98.16 while increasing inference time by 3.5× relative to BERT.

H. Domain Routing Analysis (RQ6) We examine RQ6 by analyzing the BGL and Thunderbird test partitions. On BGL, 79.8% of anomalous test lines originate from EventIDs not matched by any Drain3 template extracted from the offline region, indicating substantial temporal concept drift between the training and test periods. FAME’s gate and selector generalize to these unseen templates by routing them to the semantically nearest known expert domain based on log content with only 0.7% of the logs routing to the fail safe UNIVERSAL_NORMAL. Of the 36,936 unknown-EventID anomalies, 86.3% are successfully detected. For the 20.2% of anomalous lines whose EventIDs are matched to known templates, all expert routing assignments match the LLM-proposed failure domain exactly. This confirms that when FAME routes a log line to a named expert, the assignment is reliable. On Thunderbird, all 56,800 test anomalies match known EventIDs, achieving a domainlabel precision of 100.00 overall. This confirms that Thun-

9

Case 1: Benign Log Processing

derbird is a closed-world benchmark where the offline and test distributions are fully aligned, which directly explains its near-perfect performance across all methods. In particular, Drain+RForest achieves perfect F1 = 100.00 on Thunderbird because all anomalous templates are seen during training and an EventID-level classifier is sufficient when the test distribution is fully covered. On BGL, Drain+RForest degrades to F1 = 33.59 because it has no mechanism to generalize beyond the training template set.

LOG INPUT 1131148621 ... RAS APP FATAL ciod: Error loading /p/gb1/bmiller/CALTECH/Thunder_compare/mdcask.opt: program image too big, 1380627184 > 532152320

DistilBert Gate Routing The inital Gate produces a low score of 0.1089. Since this is below the 0.5 threshold, the system identifies the message as non-suspicious.

Expert Assignment: UNIVERSAL_NORMAL Expert The log is routed to the ‘UNIVERSAL_NORMAL’ expert for final verification.

CLASSIFICATION: NORMAL

I. Inference Efficiency We documented per-line inference latency over 35 runs, with 5 runs for each K ∈ {5, 10, 25, 50, 100, 200, 400}. On a single GCP node with 4×T4 GPUs, 8 vCPUs, and 52 GB RAM, FAME averages 3.07 ms per line on BGL and 2.99 ms on Thunderbird. This corresponds to 1.17M and 1.20M lines/hour. Variance across runs is negligible, indicating that inference cost is effectively independent of K once the pipeline is trained.

The Universal Normal expert produces a score of 0.0046, resulting in a ‘NORMAL’ status and requiring no further engineer intervention.

Case 2: Failure-Domain Triage LOG INPUT KERNREC ... RAS KERNEL FATAL Error receiving packet on tree network, expecting type 57 instead of type 3 (softheader=00000001 ...) PSR0=00001f01 PSR1=00000000

DistilBert Gate Routing The Gate produces a score of 0.4790. Being below the 0.5 threshold, the system flags this as a NORMAL with low confidence.

Expert Assignment: UNIVERSAL_NORMAL Expert

V. C ASE S TUDY We present two inference-time edge cases to illustrate how FAME handles ambiguous log messages, as shown in Figure 5. Both logs contain the keyword FATAL, but they correspond to different outcomes. The first is normal and the second is anomalous. Although the message contains FATAL in the first case, it indicates that the job’s program binary exceeded the allowed size, which is an expected condition rather than a system failure. Both the gate and UNIVERSAL_NORMAL assign high confidence to the normal class. This shows that FAME does not rely on severity keywords alone. In the second case, the log is a true anomaly and its template appears only in the test set. The gate output gϕ (xi ) = 0.4790 reflects uncertainty between a failuredomain expert and UNIVERSAL_NORMAL. Despite this uncertainty, UNIVERSAL_NORMAL assigns a high anomaly score of 0.7107 and correctly detects the anomaly. This example shows that UNIVERSAL_NORMAL provides effective fallback behavior for unseen anomalous templates. Since the anomaly is detected by UNIVERSAL_NORMAL, FAME flags it as anomalous without assigning a failure-domain label.

The log is routed to the ‘UNIVERSAL_NORMAL’ expert for final verification.

CLASSIFICATION: ANOMALY (w/o Failure Domain) The Universal Normal expert produces a score of 0.7107, resulting in an 'ANOMALY' status and allowing for immediate, localized fault triage.

Fig. 5. Case study with ambiguous keyword ’FATAL’

model temporal structure within sessions to improve detection accuracy. LogBERT [3] is the most cited representative which applies masked language modeling to log sequences and emits an aggregate score over a session window. The contrast axis with FAME is granularity. LogBERT’s training objective and decision rule are session-aggregate by construction. Adapting it to per-message decisions is non-trivial because token scores within a line do not align with line boundaries, and the session aggregate cannot be back-projected to a single responsible message without an additional attribution model. FAME targets per-message decisions directly, making the two approaches complementary rather than directly comparable. LLM-based analysis and the AdaptiveLog comparable. LLMs have been applied to parsing through LUNAR [37] and CSLParser [38], to per-line detection through LogGPT [12], and to root-cause analysis through RCACopilot [13], AetherLog [14], LogKG [39], COMET [40], and causal approaches [41]. Continuous per-line LLM processing introduces practical considerations such as API cost, latency, and data handling. AdaptiveLog [42] is the closest hybrid in design philosophy. It routes high-confidence predictions to a small model and defers the uncertain residual to an LLM, which reduces but does not eliminate runtime LLM invocation. LasRCA [43] is a related variant. The crucial point for regulated deployments is that even a small deferred fraction still incurs per-call cost and, more importantly, exfiltrates the deferred lines to an external API. FAME instead uses an LLM only during an offline setup stage, while all inference runs on-premise, reflecting a different deployment design choice. MoE variants, the DEMix comparable, and label-efficient

VI. R ELATED W ORK Log parsing and classical detectors. Most pipelines parse raw messages into event templates that abstract instancespecific tokens. Drain [20] is the dominant online parser. Complementary parsers include IPLoM [31], Spell [32], and Logram [33], while He et al. [34] survey the space. Classical detectors then apply statistical models to template counts or token features, including PCA [8], Isolation Forest [9], SVM and clustering [7]. However, they treat all instances of a template as equivalent, which may limit their ability to distinguish normal and anomalous lines that share an EventID. Neural sequence models and the LogBERT comparable. DeepLog [4], LogAnomaly [5], HitAnomaly [10], LogRobust [11], AllInfoLog [35], SwissLog [6], and PreLog [36]

10

learning. Sparse MoE [16] and the foundational adaptivemixtures formulation [15] establish gated conditional-compute architectures, with most work targeting generative modeling. DEMix Layers [21] is the closest architectural analog to FAME. It uses per-domain experts trained separately from the routing, with hard routing based on domain identity, namely text genre in DEMix and failure domain in FAME. Recent work such as LogMoE [44] also explores a mixture-of-experts approach for sequence-level log anomaly detection, where lightweight experts trained on different source systems are combined with a gating mechanism to support cross-system generalization. FAME generalizes this pattern in two ways. (i) Failure domains are synthesized by an LLM and certified against K-shot signals rather than read from metadata. (ii) Pure-anomaly domains carry no trained expert at all, and routing alone serves as the detector. On supervision, selfsupervised LogBERT and DeepLog [3], [4], pretraining-based PreLog [36], and pseudo-label PLELog [45] reduce labels through different mechanisms. FAME instead spends a strict K-shot budget per template, achieving a 76× reduction relative to full line-level annotation on BGL. To our knowledge, no prior system jointly targets per-message detection, named failure-domain routing labels, K-shot supervision, and fully on-premise deployment.

propose a semantic partition of EventIDs into failure domains. After setup, every routing and detection decision is made by lightweight local models, with no recurring API calls, no log egress, and no dependence on external services at deployment time. Our results show that the main contribution is the architecture itself rather than reliance on an LLM. On BGL, FAME reaches F1 = 98.16 at K = 100, while the non-LLM variant with TF-IDF grouping still achieves a strong F1 = 93.01. This shows that expert decomposition is the dominant source of improvement, and that LLM-based grouping mainly strengthens an already effective design. The ablation study reinforces this conclusion. Replacing routed experts with a single global detector causes a large performance drop, which confirms that heterogeneous log anomalies are better handled as specialized sub-problems than by one global decision boundary. The results also show that FAME remains practical under tighter labeling budgets. Although K = 100 gives the best overall F1 on BGL, K = 25 already achieves F1 = 97.57 with only 22,711 labeled lines, which is a 178× reduction from full line-level labeling. This gives practitioners a useful tradeoff between annotation cost and detection performance. When labeling resources are limited, a smaller budget can still deliver strong message-level detection while preserving the same fully local inference pipeline. FAME demonstrates that lightweight per-template annotation, offline semantic partitioning, and specialized SLM inference can be combined in a practical on-premise pipeline for line-level log analysis. More broadly, FAME can serve as a foundation for richer incident analysis workflows. One natural next step is online adaptation under schema drift, where new templates are absorbed into existing experts or trigger new specialists, and only the affected components are retrained. Another is to model temporal structure over expert activations, which could support causal incident graphs rather than only per-line anomaly labels. The framework also creates a natural entry point for combining logs with complementary operational signals such as system metrics, especially in cases where disagreement between the two may help expose silent failures or noisy components.

VII. THREATS TO VALIDITY Internal validity. The K-shot sampling strategy selects the most recent labeled lines to approximate deployment conditions, but this recency bias may not reflect the full distributional range of each EventID. Threshold calibration relies on a held-out 20% subset of K-shot data. At low K, the small number of anomalies can lead to unstable precision–recall tradeoffs. FAME also commits to a fixed offline partition. If failure patterns evolve, this partition may become outdated and require re-running the offline setup. Finally, minor differences in baseline training or hyperparameter tuning may affect comparisons. External validity. Our evaluation uses BGL and Thunderbird, both drawn from supercomputer environments. Thunderbird’s structural separability may overestimate performance compared to noisier real-world systems. The framework also assumes K labeled lines per template, which may be difficult under high template turnover. For unseen EventIDs, failuredomain labels are assigned by semantic similarity and cannot be directly verified due to the lack of ground-truth failure types. In addition, different LLMs or prompts may produce different partitions, and real-world deployments may exhibit stronger concept drift than observed in our experiments.

R EFERENCES [1] J. Liu, J. Zhu, S. He, P. He, Z. Zheng, and M. R. Lyu, “Logzip: Extracting hidden structures via iterative clustering for log compression,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2019, pp. 863–873. [2] S. Petrescu, F. Den Hengst, A. Uta, and J. S. Rellermeyer, “Log parsing evaluation in the era of modern software systems,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 379–390. [3] H. Guo, S. Yuan, and X. Wu, “LogBERT: Log anomaly detection via BERT,” in Proceedings of the International Joint Conference on Neural Networks, 2021. [4] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 1285–1298. [5] W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun, and R. Zhou, “Loganomaly: Unsupervised detection

VIII. C ONCLUSION We presented FAME, a failure-aware mixture-of-experts framework for message-level log anomaly detection under limited line-level supervision. The main result of this work is that strong message-level detection can be achieved with a routed expert architecture that remains fully on-premise during inference. The LLM is used only once during offline setup to

11

of sequential and quantitative anomalies in unstructured logs,” in Proceedings of the International Joint Conference on Artificial Intelligence, 2019, pp. 4739–4745. [6] X. Li, P. Chen, L. Jing, Z. He, and G. Yu, “Swisslog: Robust anomaly detection and localization for interleaved unstructured logs,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 4, pp. 2762–2780, 2022. [7] V.-H. Le and H. Zhang, “Log-based anomaly detection with deep learning: How far are we?” in Proceedings of the 44th international conference on software engineering, 2022, pp. 1356–1367. [8] A. Lakhina, M. Crovella, and C. Diot, “Diagnosing network-wide traffic anomalies,” in Proceedings of ACM SIGCOMM, 2004, pp. 219–230. [9] A. Farzad and T. A. Gulliver, “Unsupervised log message anomaly detection,” ICT Express, vol. 6, no. 3, pp. 229–237, 2020. [10] S. Huang, Y. Liu, C. Fung, R. He, Y. Zhao, H. Yang, and Z. Luan, “Hitanomaly: Hierarchical transformers for anomaly detection in system log,” IEEE transactions on network and service management, vol. 17, no. 4, pp. 2064–2076, 2020. [11] X. Zhang, Y. Xu, Q. Lin, B. Qiao, H. Zhang, Y. Dang, C. Xie, X. Yang, Q. Cheng, Z. Li et al., “Robust log-based anomaly detection on unstable log data,” in Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, 2019, pp. 807–817. [12] J. Qi, S. Huang, Z. Luan, S. Yang, C. Fung, H. Yang, D. Qian, J. Shang, Z. Xiao, and Z. Wu, “Loggpt: Exploring chatgpt for log-based anomaly detection,” in 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), 2023, pp. 273–280. [13] Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zhu, A. Sailer, L. Lozano, C. Bansal, S. Rajmohan, and D. Zhang, “Automatic root cause analysis via large language models for cloud incidents,” in Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys), 2024, pp. 674–688. [14] T. Cui, R. Fu, C. Liu, Y. Ji, W. Gu, S. Zhang, Y. Sun, and D. Pei, “Aetherlog: Log-based root cause analysis by integrating large language models with knowledge graphs,” in 2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2025, pp. 49–60. [15] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, vol. 3, no. 1, pp. 79–87, 1991. [16] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, pp. 1–39, 2022. [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186. [18] A. Oliner and J. Stearley, “What supercomputers say: A study of five system logs,” in 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), 2007, pp. 575–584. [19] J. Zhu, S. He, P. He, J. Liu, and M. R. Lyu, “Loghub: A large collection of system log datasets for ai-driven log analytics,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 355–366. [20] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in Proceedings of IEEE International Conference on Web Services, 2017, pp. 33–40. [21] S. Gururangan, M. Lewis, A. Holtzman, N. A. Smith, and L. Zettlemoyer, “Demix layers: Disentangling domains for modular language modeling,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 5557–5576. [22] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. [23] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune BERT for text classification?” in China National Conference on Chinese Computational Linguistics, 2019, pp. 194–206. [24] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th

international joint conference on natural language processing (EMNLPIJCNLP), 2019, pp. 3982–3992. [25] OpenAI, “Gpt-5 technical overview,” https://openai.com, 2026, accessed: April 2026. [26] Anthropic, “Claude model documentation,” https://www.anthropic.com, 2026, accessed: April 2026. [27] G. DeepMind, “Gemini api documentation,” https://ai.google.dev, 2026, accessed: April 2026. [28] OpenAI, “Pricing — OpenAI Developer Platform,” https://openai.com/ api/pricing/, accessed: April 2026. [29] Anthropic, “Pricing — Anthropic Developer Documentation,” https:// www.anthropic.com/pricing, accessed: April 2026. [30] Google DeepMind, “Gemini Developer API Pricing,” https://ai.google. dev/pricing, accessed: April 2026. [31] A. A. Makanju, A. N. Zincir-Heywood, and E. E. Milios, “Clustering event logs using iterative partitioning,” in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, 2009, pp. 1255–1264. [32] M. Du and F. Li, “Spell: Streaming parsing of system event logs,” in 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016, pp. 859–864. [33] H. Dai, H. Li, C.-S. Chen, W. Shang, and T.-H. Chen, “Logram: Efficient log parsing using n n-gram dictionaries,” IEEE transactions on software engineering, vol. 48, no. 3, pp. 879–892, 2020. [34] S. He, P. He, Z. Chen, T. Yang, Y. Su, and M. R. Lyu, “A survey on automated log analysis for reliability engineering,” ACM computing surveys (CSUR), vol. 54, no. 6, pp. 1–37, 2021. [35] R. Xiao, H. Chen, J. Lu, W. Li, and S. Jin, “Allinfolog: Robust diverse anomalies detection based on all log features,” IEEE Transactions on Network and Service Management, vol. 20, no. 3, pp. 2529–2543, 2022. [36] V.-H. Le and H. Zhang, “Prelog: A pre-trained model for log analytics,” Proceedings of the ACM on Management of Data, vol. 2, no. 3, pp. 1–28, 2024. [37] J. Huang, Z. Jiang, Z. Chen, and M. Lyu, “No more labelled examples? an unsupervised log parser with llms,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 2406–2429, 2025. [38] W. Hong, Y. Wu, L. Zhang, C. Duan, P. Xiao, M. He, X. Yang, and Y. Li, “Cslparser: A collaborative framework using small and large language models for log parsing,” in 2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2025, pp. 61–72. [39] Y. Sui, Y. Zhang, J. Sun, T. Xu, S. Zhang, Z. Li, Y. Sun, F. Guo, J. Shen, Y. Zhang et al., “Logkg: Log failure diagnosis through knowledge graph,” IEEE Transactions on Services Computing, vol. 16, no. 5, pp. 3493–3507, 2023. [40] Z. Wang, J. Li, M. Ma, Z. Li, Y. Kang, C. Zhang, C. Bansal, M. Chintalapati, S. Rajmohan, Q. Lin et al., “Large language models can provide accurate and interpretable incident triage,” in 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2024, pp. 523–534. [41] M. Markakis, B. Youngmann, T. Gao, Z. Zhang, R. Shahout, P. B. Chen, C. Liu, I. Sabek, and M. Cafarella, “From logs to causal inference: diagnosing large systems,” Proceedings of the VLDB Endowment, vol. 18, no. 2, pp. 158–172, 2024. [42] L. Ma, W. Yang, Y. Li, B. Fei, M. Zhou, S. Li, S. Jiang, B. Xu, and Y. Xiao, “Adaptivelog: An adaptive log analysis framework with the collaboration of large and small language model,” ACM Transactions on Software Engineering and Methodology, 2025. [43] Y. Han, Q. Du, Y. Huang, J. Wu, F. Tian, and C. He, “The potential of one-shot failure root cause analysis: Collaboration of the large language model and small classifier,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 931–943. [44] J. Qi, Z. Luan, S. Huang, C. Fung, Y. Wang, A. Wang, H. Zhang, H. Yang, and D. Qian, “Logmoe: Lightweight expert mixture for crosssystem log anomaly detection,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025, pp. 330– 341. [45] L. Yang, J. Chen, Z. Wang, W. Wang, J. Jiang, X. Dong, and W. Zhang, “Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2021, pp. 230–231.

12