SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

arXiv:2607.24588v1 [cs.AI] 27 Jul 2026

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents Hang Ni

Weijia Zhang

Fan Liu

Artificial Intelligence Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China [email protected]

Mengqian Lu∗

Hao Liu

Department of Civil and Environmental Engineering The Hong Kong University of Science and Technology Hong Kong, China [email protected]

Artificial Intelligence Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China [email protected]

Abstract

CCS Concepts

Early warning of extreme weather is essential for mitigating the societal, economic, and environmental risks posed by hazardous weather events. However, expert-centered warning workflows are costly, labor-intensive, and difficult to scale throughout the warningto-action process. Although recent advances in Large Language Model (LLM) agents have enabled the automation of weatherrelated tasks, existing studies remain centered on isolated scientific tasks and overlook the chain of interdependent processes required for operational extreme-weather early warning. To bridge this gap, this study investigates automated end-to-end extreme-weather early warning through LLM agents. We first develop SIREN-Bench, a comprehensive benchmark comprising 600 question-answer instances across 19 tasks, and covering four individual warning procedures and an end-to-end warning chain. Evaluation on SIRENBench reveals substantial capability gaps in existing weather agent frameworks. This motivates us to develop SIREN, an experiencegrounded agent framework inspired by experts’ use of historical cases, which combines an agentic execution environment integrating heterogeneous weather evidence and tools with a family of agent harnesses that exploit historical cases through retrieval, skill distillation, and predictive modeling. Extensive experiments demonstrate that SIREN outperforms weather-agent baselines on both individual warning procedures and end-to-end warning chains.

• Computing methodologies → Intelligent agents; • Applied computing → Earth and atmospheric sciences.

∗ Also with Otto Poon Center for Climate Resilience and Sustainability, The World

Sustainable Development Institute. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX

Keywords Extreme-weather early warning, large language model agent ACM Reference Format: Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, and Hao Liu. 2018. SIREN: Towards End-to-End Extreme-Weather Early Warning with ExperienceGrounded LLM Agents. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 18 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

The increasing frequency, intensity, and compound impacts of extreme weather events are amplifying systemic risks to human societies, economic activities, and ecosystems [2]. In response, the United Nations (UN) and the World Meteorological Organization (WMO) have identified Early Warning Systems (EWS) as a cornerstone of disaster risk reduction and climate adaptation [35, 42]. An effective EWS is not merely a forecasting module but an operational warning-to-action chain that integrates hazard monitoring, event localization, impact-based risk assessment, warning communication, and preparedness decision-making [33]. Its objective is therefore to move beyond hazard-centered prediction toward actionable, impact-oriented warnings that enable timely interventions before hazardous weather develops into a severe disaster [6]. Implementing such an integrated warning pipeline remains challenging. In current practice, extreme-weather warning relies heavily on domain experts to interpret evolving atmospheric conditions, apply meteorological knowledge and prior experience, coordinate heterogeneous data sources and analytical tools, and synthesize intermediate evidence into warning and response decisions [27, 41]. Despite its effectiveness, this expert-centered workflow is costly,

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Four Pillars in Real Early Warning System (EWS) Pipeline

Pillar 1

Observations, Monitoring, Analysis, Forecasting

Event Characterization

Pillar 2

Disaster Risk Knowledge

Spatiotemporal Prediction

Pillar 3

Table 1: Comparisons of benchmark coverage, with EC, SP, IA, RD, and WC denoting five task categories, respectively. Pillar 4

Warning Dissemination & Communication

Preparedness & Response Capabilities

Impact Assessment

Responsive Decision-making

Four Task Procedures Formulated in our Extreme-weather Warning Problem

Figure 1: Connections to the EWS pipeline [33]. labor-intensive, difficult to scale, and vulnerable to cascading errors across interdependent stages. Existing software systems and machine learning (ML) models have improved individual components, such as detection, forecasting, and diagnostic analysis, but typically address isolated tasks, leaving human experts to integrate evidence and manage the end-to-end warning process. Recent advances in LLMs have enabled a new paradigm of data-driven LLM agents [12, 17], which can automatically orchestrate multistep reasoning, generate code, and use scientific tools to transform heterogeneous evidence into grounded analyses. These advances motivate us to formulate extreme-weather early warning as an agentic task in which LLM agents gather meteorological evidence, reason about hazard risks, and support operational decisions. Recent studies have investigated LLM agents for data-driven weather applications, including extreme-weather diagnosis [13, 34] and automated weather-science workflows [11, 15, 38, 39, 44]. However, these agents primarily focus on early-stage tasks, such as hazard analysis and forecasting, while largely omitting downstream impact assessment, warning communication, and response-oriented decision support. Consequently, existing benchmarks do not capture the full operational objectives of end-to-end extreme-weather warning. To address this gap, we construct SIREN-Bench1 , a comprehensive benchmark based on over 12 types of extreme-weather events across the U.S. region in 2021, comprising 600 questionanswer (QA) pairs. As shown in Figure 1, SIREN-Bench follows the warning-to-action perspective of EWS [33] and organizes extremeweather warning into four core task procedures: event characterization to interpret ongoing hazards, spatiotemporal prediction to forecast their evolution, impact assessment to estimate potential consequences, and responsive decision-making to translate evidence into warnings and interventions. It further integrates these individual capabilities into a warning chain task that requires agents to orchestrate the end-to-end operational pipeline. As summarized in Table 1, SIREN-Bench provides the broadest coverage by spanning all five task categories. Nevertheless, evaluations of five recent weather-agent frameworks reveal substantial capability gaps, indicating that existing agents remain far from operational readiness for both individual procedures and warning chains. In practice, extreme-weather analysis and warning decisions are rarely made in isolation. For example, estimating injuries caused 1 The benchmark is available at https://anonymous.4open.science/r/SIREN-5CE8/.

Benchmark

EC SP IA RD WC

ZephyrusBench [38] Jiang et al. [13] ClimaBench [39] Climate-Agent-Bench-85 [15] Tang et al. [34]

" " " " "

" % " " %

% % % % %

% % " % %

% % % % %

SIREN-Bench

" "

": covered; ": partially covered; %: not covered.

by a tornado requires not only analysis of the physical mechanisms of the ongoing event but also reference to prior cases with similar spatiotemporal or hazard characteristics. Existing weather agents primarily reason over the current meteorological state while underusing historical cases that can provide reusable operational experience through retrieval, adaptation, or task-specific distillation. To address this limitation, we introduce SIREN, an experiencegrounded agent framework for operational extreme-weather early warning. Specifically, SIREN is built on a comprehensive and executable agentic environment that provides diverse weather-oriented analytical tools and interfaces for multimodal evidence sources, including meteorological grids, mesoanalysis images, geospatial information, etc., thereby enabling LLM agents to ground their analyses and decisions in verifiable evidence rather than relying solely on parametric knowledge. To incorporate historical operational experience, we construct a historical case database from questionanswer pairs collected in earlier years and design a family of agents instantiated with diverse harnesses that capture complementary uses of historical cases in agentic reasoning [3, 10, 43]: (1) SIREN-Base does not use historical cases; (2) SIREN-RAG retrieves similar historical cases as direct analogical references for the current task; (3) SIREN-Skill performs deliberate rehearsal on informative cases and distills reusable procedural skills from their solution trajectories; and (4) SIREN-Modeling derives task-specific predictive indicators, constructs training and validation sets from the complete historical database, and develops ML models to support downstream analyses. Extensive experiments demonstrate that grounding agents in historical experience substantially improves operational performance on both individual procedures and end-to-end warning chains. Our main contributions are summarized as follows: (1) To the best of our knowledge, SIREN-Bench is the first benchmark for endto-end extreme-weather early warning, comprising 600 QA pairs across 12 types of extreme events; (2) SIREN is the first agent framework for end-to-end extreme-weather early warning, consisting of an agentic environment that integrates heterogeneous weather evidence and analytical tools, and a family of agents that leverage historical cases through complementary experience-grounded harnesses; and (3) extensive experiments demonstrate strong performance against competitive weather-agent baselines on both individual procedures and operational warning chains.

SIREN-Bench

This section presents SIREN-Bench, a benchmark for evaluating LLM agents in operational extreme-weather early warning.

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

2.1

Task Formulation

Each instance in SIREN-Bench is defined as a tuple 𝜏 = (𝑞, 𝑦), where 𝑞 denotes an extreme-weather analysis question and 𝑦 denotes the ground-truth answer or reference response. Given 𝑞, an LLM agent with policy 𝜋𝜃 produces an answer 𝑦ˆ = 𝜋𝜃 (𝑞) by retrieving evidence, invoking tools, reasoning over intermediate results, and synthesizing a final response. The objective is to generate an answer 𝑦ˆ that is accurate, grounded in evidence, and operationally useful.

2.2

Benchmark Overview

2.2.1 Overall Coverage. SIREN-Bench focuses on extreme-weather events in the U.S. in 2021. It contains 600 QA instances across five categories of extreme-weather analysis aligned with the EWS pipeline, covering 19 subtasks. These instances span 12 primary hazard families, including convective storms, tropical systems, tornadoes, floods, winter weather, high-wind events, visibility hazards, heat, fire, drought, marine hazards, and cold-weather events. They are also distributed across all months of the year, enabling the benchmark to capture seasonal variation in weather analysis. 2.2.2 Task Categories. Following the operational EWS framework [33], our dataset is organized into five task categories: (1) event characterization analyzes the nature of an ongoing hazard, including hazard type classification and event mechanism analysis, with 64 instances; (2) spatiotemporal prediction estimates where, when, and how an event will develop, including spatial localization and forecasts of event occurrence, severity, evolution, and duration, with 160 instances; (3) impact assessment evaluates potential event consequences, including the risks of injury, property damage, agricultural loss, and power outages, with 192 instances; (4) responsive decision-making converts analytical evidence into operational actions, e.g., warning decisions, alert generation, hazard mitigation, and disaster-recovery site selection, with 160 instances; and (5) warning chain integrates the preceding categories into an end-toend pipeline that reflects operational early-warning processes, with 24 instances. The first four task categories comprise 18 types of atomic subtasks, as summarized in Appendix A, while the fifth category, warning chain, constitutes a single end-to-end subtask, yielding 19 atomic subtasks in total.

2.3

Dataset Construction

SIREN-Bench is constructed through a four-stage pipeline: (1) raw data curation, (2) multi-source data preprocessing and alignment, (3) task instance generation, and (4) quality control. 2.3.1 Raw Data Curation. Our task instances are derived from three U.S. data sources related to extreme weather: (1) National Oceanic and Atmospheric Administration (NOAA) Databases [20]: The NOAA National Centers for Environmental Information (NCEI) maintains the Storm Events Database (SED) [22], which records various types of severe weather together with spatiotemporal information, event types (e.g., tornadoes and heatwaves), and impacts (e.g., numbers of injuries and deaths). These metadata support tasks such as extreme-weather type classification, spatiotemporal prediction, and impact assessment. In addition, the NOAA Storm Prediction Center (SPC) provides Mesoscale Discussions (MDs) [24], which are

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

specialized meteorological statements issued by forecasters to describe mesoscale conditions associated with hazardous weather. We collect these statements for extreme-weather mechanism analysis. (2) Federal Emergency Management Agency (FEMA) Databases [7]: FEMA provides datasets on disaster impacts and response decisions. We select records associated with extreme weather events for task construction. Specifically, these datasets support property risk assessment, public warning generation, and hazard mitigation decision-making. (3) Open Energy Data Initiative (OEDI) [37]: OEDI provides utility outage information resources [40] that link utility service areas to public outage information channels maintained by electric utilities. We use entries associated with extreme weather to construct the outage duration assessment task. 2.3.2 Multi-source Data Preprocessing and Alignment. The preprocessing pipeline first filters event records containing missing or anomalous entries to obtain a consistent subset of the raw data. We then map key metadata (e.g., timestamps, spatial identifiers, event types, etc.) from different sources to a shared event schema because these sources use heterogeneous field definitions, value formats, units, and naming conventions. This step retains source-specific details while normalizing the data for cross-source use. Constructing warning chain tasks requires identifying records that refer to the same extreme event across data sources and organizing them into four sequential task procedures. However, the spatiotemporal attributes of the same event often differ across sources. For example, SED may record a tornado as occurring on February 9, 2021, at 5:20 PM, whereas FEMA records the same event at 4:50 PM, making cross-source alignment nontrivial. To address this issue, we design a soft alignment strategy that matches events of the same type while allowing controlled spatiotemporal tolerance. For example, two events are considered matched when the difference between their recorded timestamps is within a predefined threshold or when their recorded locations are in neighboring counties. 2.3.3 Task Instance Generation. Based on the normalized task metadata, we construct formal QA instances in which each question contains three components: (1) task instruction, (2) event conditions, and (3) answer specification. The task instruction defines the objective of the subtask, e.g., identifying an event type or predicting an affected area, without prescribing an overly specific solution. The event conditions provide the event-specific inputs required to complete the task, such as the reference timestamp, affected regions, and event descriptions, while excluding ground-truth information to prevent label leakage. The answer specification constrains the output space according to the task requirements, e.g., categorical labels, numerical values, or open-ended responses. For chained tasks, the four question templates are arranged sequentially, and the final response combines the outputs of all four procedures for end-to-end evaluation. This design better reflects operational chains, in which an agent must analyze the same event context through multiple cascading tasks rather than solve isolated problems. 2.3.4 Quality Control. To ensure benchmark quality, we audit the generated instances from 2021 through a three-stage process involving LLMs and human experts. First, LLMs audit the instances according to predefined criteria: (1) whether the event conditions

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

are incomplete or prone to leakage and (2) whether the groundtruth answers or reference responses are implausible, ambiguous, or insufficiently informative. Low-quality instances are then discarded. Subsequently, to prevent frequent hazards or disaster-prone months from dominating the benchmark, we resample the filtered instances to promote diversity and balanced coverage across extreme event types and occurrence seasons. Finally, experts manually inspect the remaining instances to verify that they are plausible and representative of operational extreme-weather scenarios.

2.4

Evaluation Protocols

For task evaluation, we categorize task instances into four types based on their answer formats. Multiple-choice Classification (MC). MC tasks require the agent to select one or more labels from a closed option set, such as hazard types or hazard mitigation actions. We report the accuracy measure based on exact label matching. Numeric Regression (NR). NR tasks require a scalar value, such as event duration or societal impact. Given a prediction 𝑦ˆ and reference 𝑦, we first compute the relative error as RE = |𝑦ˆ − 𝑦|/|𝑦|. We then report the bounded relative score RS = 1/(1 + RE), where higher values indicate better numerical agreement. Geospatial Localization (GL). GL tasks evaluate open-form spatial answers, e.g., affected administrative areas and recovery-center addresses. For state-county localization, we use an adjacency-aware score that assigns partial credit to nearby locations: LS = 𝜆/(𝛿 state + 1) + (1 − 𝜆)/(𝛿 county + 1), where 𝜆 controls the credit assigned to state matching, and 𝛿 state and 𝛿 county denote the minimum hop distances in the state- and county-level adjacency graphs, respectively. For coordinate or address localization, we geocode the prediction and reference when necessary, compute the minimum haversine distance 𝑑 in kilometers between the predicted and reference points, and report SS = 1/(1 + 𝑑) together with the best-distance statistic. Open-ended Generation (OG). OG tasks require free-form operational text, such as descriptions of physical mechanisms, publicwarning instructions, or hazard mitigation responses. For most OG tasks, we extract a small set of essential points from the reference response and compute answer recall as Recall = |Phit |/|Pref |, where Pref is the set of essential reference points and Phit is the subset supported by the agent response. For physical-understanding tasks, which require more holistic judgment, we use an expert-style LLM evaluator to score the answer against the reference on a 0–10 scale, with validation of LLM–human alignment provided in Appendix I. For chained tasks, we do not assign a single monolithic metric. Instead, we parse the response into procedure-specific answers and evaluate each procedure using the corresponding MC, NR, GL, or OG metric. In addition to task-effectiveness metrics, we report two process-level measures for LLM agents: the completion pass rate CPR, which indicates whether the agent successfully completes the task with a valid answer, and the execution pass rate EPR, which indicates whether the agent completes the task without code-execution or post-check failures.

SIREN

This section presents SIREN, an agent framework for extremeweather early warning. As illustrated in Figure 2, the framework

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

comprises an executable agentic environment and a family of agents whose harnesses integrate historical cases through complementary experience-grounded reasoning mechanisms.

3.1

Agentic Environment

Operational early warning requires agents to track an evolving hazard from meteorological evidence to local impacts and response actions across interdependent stages, but existing weather-agent environments provide task-specific data and tools for isolated analysis or forecasting rather than an end-to-end workflow. We therefore construct a unified agentic environment spanning these operational procedures, denoted by E, which comprises multimodal evidence X, analytical tools T , and a guarded code execution server. 3.1.1 Multimodal Evidence Data. The environment provides access to four types of multimodal evidence X: (1) Meteorological grids comprise high-resolution numerical weather analysis and forecast fields over the contiguous U.S., derived from the NOAA High-Resolution Rapid Refresh system [19]. They provide quantitative atmospheric states (e.g., wind, temperature, and pressure) that serve as primary evidence for diagnostic reasoning and weather-field visualization. (2) Mesoanalysis images are operational severe-weather analysis graphics from the NOAA Storm Prediction Center Mesoanalysis archive [23], which summarizes surface objective analyses and short-range model fields as diagnostic maps for forecasters. They complement gridded variables by presenting mesoscale ingredients, instability patterns, and storm-environment structures in a form suitable for visual inspection by agents. (3) Geospatial information combines U.S. administrative boundary files from the Census Bureau TIGER/Line products [36] with map features from OpenStreetMap [26]. It anchors meteorological evidence to counties, populated places, transportation networks, and local facilities, thereby supporting localization and exposure-aware reasoning. (4) 𝐾 , where each Historical cases constitute a case base K = {K𝑘 }𝑘=1 case K𝑘 = (𝑞𝑘 , 𝑦𝑘 ) contains a warning question and the corresponding reference answer from a previous year. The event conditions associated with 𝑞𝑘 are retained in the question context, allowing agents to compare the current task with prior operational situations. 3.1.2 Analytical Tools. The environment exposes analytical tools T in seven high-level categories: (1) Evidence indexing provides controlled access to evidence sources, allowing agents to retrieve task-relevant data while maintaining a clear separation between reasoning and data storage; (2) Meteorological analysis derives physical diagnostics from weather fields (e.g., wind structures and thermodynamic summaries), converting raw variables into interpretable evidence; (3) Atmospheric forecasting provides access to advanced forecasting models, enabling agents to reason about future hazard evolution for operational decision-making; (4) Visual processing renders maps and gridded fields and supports lightweight image operations such as cropping, zooming, and annotation, enabling visual evidence to be inspected at the spatial scale required by each task; (5) Impact modeling estimates potential consequences from hazard intensity and exposed local assets, linking meteorological signals to operational risk assessment; (6) Geospatial normalization resolves place names, administrative units, and coordinates within a shared spatial frame, ensuring consistent location-based reasoning across

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Agentic Environment

Task Input

1) Multimodal Evidence Data Mesoanalysis Images

Meteorological Grids

Geospatial Information

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Agent Harnesses

SIREN-RAG Identify Key Points

Historical Cases

Retrieve Similar Cases

Analogies

SIREN-Skill

Data Querying Evidence Indexing

Meteorological Analysis

Impact Modeling Tool Invoking

Experience Grounding

2) Analytical Tools Atmospheric Forecasting

Geospatial Normalization

SIREN-Modeling

Visual Processing ML Development

3) Guarded Code Execution Server

Retrieve Practice and Rehearsal Cases Reflection Prepare Train/Val Data

Code Request Execution Observation

Update Skills Select and Develop Models

SIREN-Base Reason

Code

Observe

Final Answer

Skills

Model

Augment the Multi-turn Workflow

Figure 2: The overall framework of SIREN, which comprises an executable agentic environment and a family of agents instantiated by diverse experience-grounded harnesses. weather, map, and benchmark records; (7) ML development provides interfaces for training task-specific predictive models using training and validation samples constructed by agents. This category is enabled only for SIREN-Modeling, as discussed in Section 3.2.4. Detailed tool descriptions are provided in Appendix C. 3.1.3 Code Execution Server. LLM agents require executable computation to analyze large-scale evidence, but generated code should not directly access raw storage paths or tool implementations. We therefore route every Python snippet through a guarded code execution server that validates tool calls and path access, executes only approved operations in an isolated workspace, and returns structured observations to the agent. This design provides a controlled interface for data-intensive reasoning.

3.2

Agent Harness

We design a family of LLM agents that use E to solve operational extreme-weather warning tasks. The SIREN-Base agent follows a multi-turn code-execution workflow that alternates between code generation and observation-based reasoning before synthesizing the final solution. However, this agent relies solely on current meteorological conditions and cannot use precedents from historical cases with similar spatiotemporal contexts. To incorporate prior experience, we build three multi-stage harnesses on the base workflow, each corresponding to a representative strategy in recent datadriven agent research: case-level analogy [10], procedure-level skill accumulation [43], and data-level ML modeling [3]. These strategies yield SIREN-RAG, SIREN-Skill, and SIREN-Modeling, respectively. 3.2.1 SIREN-Base. SIREN-Base disables access to the historical case base K and relies only on the current question and event context. Formally, the agent 𝜋𝜃 follows a multi-turn workflow: {𝑟𝑡 , 𝑎𝑡 } ∼ 𝜋𝜃 (· | 𝑞, ℎ𝑡 ),

𝑜𝑡 = E (𝑎𝑡 ),

ℎ𝑡 +1 = ℎ𝑡 ∪ {𝑟𝑡 , 𝑎𝑡 , 𝑜𝑡 }. (1)

At each turn 𝑡 ∈ {1, . . . ,𝑇 }, the agent produces a reasoning trace 𝑟𝑡 and executable code 𝑎𝑡 based on the current trajectory ℎ𝑡 and

then receives an execution observation 𝑜𝑡 from the environment E. The trajectory ℎ𝑡 stores the intermediate outputs and observations accumulated up to turn 𝑡. After the interaction terminates, the agent produces the final task solution as 𝑦ˆ = 𝜋𝜃 (𝑞, ℎ𝑇 ). 3.2.2 SIREN-RAG. SIREN-RAG augments SIREN-Base with a retrieval harness for case-based analogy. Guided by a retrieval instruction 𝐼 rag , the agent first runs the code-execution loop with policy 𝜋𝜃 (· | 𝑞, ℎ𝑡 , 𝐼 rag ) to search K and identify a small set of cases Krag ⊂ K that provide useful analogies for the current task based on both task intent and event conditions. The retrieval phase terminates when the observation 𝑜𝑇rag at turn 𝑇rag returns Krag . The agent then resumes the base workflow with these cases as in-context references, following {𝑟𝑡 , 𝑎𝑡 } ∼ 𝜋𝜃 (· | 𝑞, ℎ𝑡 , Krag ) for 𝑡 > 𝑇rag , such that comparable historical situations, rather than parametric knowledge alone, inform the final answer. 3.2.3 SIREN-Skill. SIREN-Skill uses historical cases as practice tasks to acquire reusable solution procedures. The harness consists of three phases: retrieval, rehearsal, and target solving. First, a rehearsal-selection instruction 𝐼 reh guides the agent to retrieve Kreh = {K 𝑗 }𝑚 𝑗=1 ⊂ K, where the selected cases exercise reasoning patterns relevant to the current task. Second, for each case K 𝑗 = (𝑞 𝑗 , 𝑦 𝑗 ), the agent solves 𝑞 𝑗 without access to 𝑦 𝑗 , using a compact skill summary S 𝑗 that contains previously distilled textual guidance. After rehearsal, the reference answer is revealed only to update the skills, yielding S 𝑗+1 = 𝜋𝜃 (𝐼 skill, S 𝑗 , ℎ𝑇𝑗 𝑗 , 𝑦 𝑗 ), where 𝐼 skill specifies how to compare the rehearsal trajectory with the reference answer and distill reusable procedural guidance for skill refinement. Finally, the target task is solved with the accumulated skills under the policy 𝜋𝜃 (· | 𝑞, ℎ𝑡 , S𝑚+1 ), allowing prior cases to improve the solution procedure rather than serve as direct answer examples. 3.2.4 SIREN-Modeling. SIREN-Modeling introduces a modeling harness that converts historical cases into task-specific predictive

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Table 2: Performance across atomic procedures. The best method is shown in bold and the second best is underlined in italics. Event Characterization

Spatiotemporal Prediction

Impact Assessment

Responsive Decision-Making

Avg.

CPR

EPR

Avg.

CPR

EPR

Avg.

CPR

EPR

Avg.

CPR

EPR

Avg.

CPR

EPR

Zephyrus EWE ClimAgent ClimateAgent HVR-Met

0.453 0.422 0.438 0.456 0.403

1.000 1.000 1.000 1.000 1.000

1.000 0.922 0.797 0.797 0.891

0.199 0.218 0.224 0.260 0.241

1.000 0.994 0.800 1.000 1.000

0.981 0.956 0.969 0.838 0.969

Qwen3.7-Plus 0.461 1.000 0.448 1.000 0.449 1.000 0.461 1.000 0.479 1.000

1.000 0.984 0.984 0.927 1.000

0.252 0.221 0.268 0.269 0.254

1.000 1.000 1.000 1.000 0.994

1.000 0.981 0.994 0.994 0.988

0.329 0.318 0.335 0.351 0.342

1.000 0.998 0.944 1.000 0.998

0.995 0.969 0.962 0.906 0.976

SIREN-Base SIREN-RAG SIREN-Skill SIREN-Modeling

0.509 0.488 0.478 0.522

1.000 1.000 1.000 1.000

1.000 0.938 0.953 0.984

0.226 0.230 0.225 0.277

1.000 1.000 0.994 1.000

1.000 0.875 0.944 1.000

0.266 0.521 0.476 0.463

0.995 1.000 1.000 1.000

0.984 0.948 0.964 0.984

0.253 0.537 0.472 0.554

1.000 1.000 0.994 1.000

1.000 0.825 0.912 0.994

0.278 0.441 0.406 0.443

0.998 1.000 0.997 1.000

0.995 0.892 0.943 0.991

Zephyrus EWE ClimAgent ClimateAgent HVR-Met

0.516 0.475 0.481 0.484 0.263

1.000 1.000 1.000 0.969 0.625

1.000 0.906 0.641 0.812 0.812

0.216 0.248 0.245 0.185 0.180

0.938 0.975 0.994 0.887 0.750

0.988 0.988 0.925 0.875 0.931

GPT-5.4 mini 0.372 0.984 0.382 0.917 0.401 1.000 0.392 0.974 0.357 0.844

0.901 0.969 0.953 0.896 0.922

0.216 0.195 0.240 0.216 0.190

0.994 1.000 1.000 0.994 0.938

1.000 0.994 0.975 0.919 0.988

0.301 0.303 0.322 0.296 0.251

0.976 0.965 0.998 0.955 0.819

0.964 0.974 0.917 0.887 0.931

SIREN-Base SIREN-RAG SIREN-Skill SIREN-Modeling

0.512 0.516 0.391 0.472

1.000 1.000 1.000 1.000

1.000 0.922 0.953 1.000

0.227 0.285 0.247 0.238

1.000 1.000 1.000 1.000

1.000 0.925 0.925 0.994

0.391 0.530 0.427 0.527

1.000 1.000 1.000 1.000

0.990 0.896 0.953 1.000

0.224 0.363 0.361 0.331

1.000 1.000 1.000 1.000

1.000 0.950 0.919 1.000

0.313 0.414 0.355 0.386

1.000 1.000 1.000 1.000

0.997 0.922 0.936 0.998

Zephyrus EWE ClimAgent ClimateAgent HVR-Met

0.438 0.491 0.534 0.512 0.484

1.000 1.000 1.000 0.984 1.000

0.797 0.562 0.688 0.453 0.938

0.162 0.233 0.233 0.209 0.247

0.787 1.000 1.000 1.000 1.000

Gemini 3.1 Flash-Lite 0.944 0.363 1.000 0.613 0.384 1.000 0.600 0.375 1.000 0.512 0.397 1.000 0.800 0.381 1.000

0.891 0.932 0.870 0.807 0.984

0.214 0.201 0.209 0.191 0.237

1.000 1.000 1.000 1.000 1.000

0.956 0.900 0.994 0.938 0.988

0.274 0.303 0.307 0.300 0.316

0.941 1.000 1.000 0.998 1.000

0.913 0.793 0.809 0.722 0.929

SIREN-Base SIREN-RAG SIREN-Skill SIREN-Modeling

0.544 0.534 0.506 0.484

1.000 1.000 1.000 1.000

1.000 1.000 1.000 0.969

0.263 0.269 0.248 0.241

1.000 1.000 1.000 1.000

1.000 0.994 1.000 0.981

1.000 0.896 0.990 1.000

0.224 0.411 0.247 0.300

1.000 1.000 1.000 1.000

1.000 0.963 1.000 0.975

0.317 0.410 0.332 0.340

1.000 1.000 1.000 1.000

1.000 0.953 0.997 0.984

Method

evidence. Guided by a modeling instruction 𝐼 model , the agent constructs training and validation samples from K, selects an appropriate ML model using the enabled ML development tools, and follows {𝑟𝑡 , 𝑎𝑡 } ∼ 𝜋𝜃 (· | 𝑞, ℎ𝑡 , 𝐼 model ) until execution returns a trained predictor 𝑀 together with the observation at turn 𝑇mod , denoted by {𝑜𝑇mod , 𝑀 } = E (𝑎𝑇mod ). The subsequent solving phase follows the base workflow with 𝑀 as an additional predictive signal, using {𝑟𝑡 , 𝑎𝑡 } ∼ 𝜋𝜃 (· | 𝑞, ℎ𝑡 , 𝑀) for 𝑡 > 𝑇mod . This variant targets tasks in which recurring patterns across historical events provide useful statistical evidence for operational judgment.

Experiments

In this section, we evaluate the efficacy of SIREN agents for extremeweather warning from the perspectives of individual procedures and the end-to-end warning chains. Specifically, we compare SIREN agents with five strong weather-science baselines: Zephyrus [38], EWE [13], ClimAgent [39], ClimateAgent [15], and HVR-Met [34]. We instantiate all agents with three advanced API-based LLM backbones: Qwen3.7-Plus [29], GPT-5.4 mini [25], and Gemini 3.1 FlashLite [9]. Implementation details are provided in Appendix D.

4.1

Individual Procedure Evaluation

We evaluate individual task procedures from five perspectives: aggregate performance across task categories, fine-grained behavior on individual subtasks, robustness across event types, geographic variation across states, and temporal variation across months.

0.363 0.485 0.413 0.407

1.000 1.000 1.000 1.000

Overall

4.1.1 Aggregate Performance. Table 2 reports the aggregate results for the four individual task categories and three LLM backbones, yielding five main observations: (1) The benchmark remains challenging, with difficulty varying across operational procedures. The best Overall Score is only 0.443, with most results below 0.4. Event characterization is relatively tractable, whereas prediction is consistently the hardest category. Impact assessment and decisionmaking benefit more substantially from historical experience. (2) Grounding agents in historical experience substantially improves their operational capabilities. The best SIREN variant outperforms the strongest baseline by 26.2%, 28.6%, and 29.7% with Qwen, GPT, and Gemini, respectively. Together with the consistent gains of SIRENRAG, SIREN-Skill, and SIREN-Modeling over SIREN-Base, these results demonstrate the importance of historical experience for operational reasoning. (3) The execution environment itself provides a strong foundation for weather-oriented agents. Without historical cases, SIREN-Base attains Overall Scores of 0.313 and 0.317 on GPT and Gemini: it is within 0.009 of the strongest GPT baseline and narrowly exceeds the strongest Gemini baseline. Its lower Qwen score of 0.278 nevertheless indicates that the environment provides a competitive foundation but cannot replace the use of historical experience. (4) The three experience-grounded mechanisms exhibit distinct and complementary strengths. SIREN-RAG achieves the best performance with GPT and Gemini, whereas SIREN-Modeling ranks first with Qwen and is particularly competitive in prediction and decision-making. SIREN-Skill generally improves over SIREN-Base

Score

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents TU

SeD

EcI

HoI

EnI

AOD

RCL

0.5 0.0

Score

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 3: Subtask-level performance, with full subtask names and detailed descriptions provided in Appendix A.

Score

4.1.3 Event-Type Analysis. Figure 4 examines the aggregate performance of SIREN-RAG with Qwen3.7-Plus across extreme event families, with definitions of the 12 event families provided in Appendix B. Experience grounding transfers across diverse hazards rather than benefiting only frequent event families. Performance is strongest for marine hazards and remains competitive for convective storms, tropical systems, and several cold-season hazards. Fire, high-wind events, and tornadoes are more challenging, likely because their rapid development or localized impacts complicate case retrieval and evidence transfer. This variation highlights the need for stronger robustness across hazard-specific dynamics. Complete method comparisons are provided in Appendix E.

Score

4.1.2 Subtask-Level Analysis. Figure 3 presents a fine-grained comparison across the 18 atomic subtasks using Qwen3.7-Plus. Historical experience benefits all four operational stages, although no mechanism dominates every subtask. SIREN-RAG performs best on most impact-assessment tasks, whereas SIREN-Modeling performs particularly well on temporal and path prediction and several warning and resource-allocation tasks. SIREN-Skill remains competitive on selected impact and mitigation tasks, while the consistently weak hazard-mitigation results reveal a shared difficulty in deriving interventions from limited context. These differences further confirm the complementarity of the three experience mechanisms.

and Alaska are more challenging. The pattern does not follow a simple regional divide: performance varies substantially within the Gulf Coast and Southeast, while New York and New Jersey remain competitive in the Northeast. These differences likely reflect regional event compositions and the alignment of historical cases with local conditions, highlighting the need for stronger geographic robustness. Complete state-level and joint event–state comparisons are provided in Appendix F and Appendix G. 0.6 0.4 TX

Figure 5: Performance across states. 4.1.5 Temporal Analysis Across Months. Figure 6 shows the monthly performance of SIREN-RAG with Qwen3.7-Plus in each task category. Event characterization and impact assessment vary noticeably throughout the year, reflecting seasonal changes in event composition and the efficacy of analogous cases. Decision-making performs better during the middle and later parts of the year, whereas prediction is more stable but consistently more difficult than the other categories. Overall, experience grounding remains effective throughout the year, although the remaining variation highlights the need for stronger temporal robustness. Complete monthly comparisons are provided in Appendix H. 0.75

Score

but underperforms the other variants, suggesting that reusable skills preserve less task-specific information than analogies or predictive models. (5) Answer quality and execution reliability must be assessed separately. Across all backbones, the SIREN agents maintain CPR values from 0.997 to 1.000, while EPR values range from 0.892 to 1.000. Thus, experience grounding rarely prevents task completion, but some high-scoring variants still incur code-execution failures that impose reliability costs.

0.50 0.25 1

Event Character. Spatiotemporal Pred.

0.50 0.25

Impact Assessment Responsive Decision.

Figure 6: Performance across months. Conv. Trop. Torn.

Flood Winter Wind

Visib. Heat Fire

Drought Marine Cold

4.2

End-to-end Chain Evaluation

Figure 4: Performance across event types.

We further evaluate complete operational task chains that sequentially solve four individual procedures for the same event.

4.1.4 State-Level Analysis Across Regions. Figure 5 compares the performance of SIREN-RAG with Qwen3.7-Plus across twelve representative states. Michigan achieves the best performance, followed by strong results in North Carolina and Louisiana, whereas Florida

4.2.1 End-to-end Performance. Table 3 reports chain-level performance with Gemini 3.1 Flash-Lite. Chain-level performance is generally lower than isolated-procedure performance, confirming the greater difficulty of end-to-end warning chains. SIREN-RAG ranks

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Table 3: Operational-chain performance with EC, SP, IA, and RD denoting four task procedures, respectively. Method

Overall

Zephyrus EWE ClimAgent ClimateAgent HVR-Met

0.317 0.225 0.300 0.258 0.275

0.083 0.083 0.125 0.125 0.125

0.025 0.180 0.136 0.172 0.124

0.536 0.586 0.583 0.472 0.491

0.240 0.268 0.286 0.257 0.254

SIREN-Base SIREN-RAG SIREN-Skill SIREN-Modeling

0.325 0.342 0.233 0.258

0.083 0.167 0.125 0.125

0.107 0.275 0.252 0.305

0.558 0.731 0.602 0.332

0.268 0.379 0.303 0.255

Table 4: Cross-procedure dependency. Dependency

𝐸𝐶 →𝑆𝑃 𝑆𝑃 →𝐼𝐴 𝐼𝐴 →𝑅𝐷 (𝐸𝐶/𝑆𝑃 ) →𝐼𝐴 (𝐸𝐶/𝑆𝑃 /𝐼𝐴) →𝑅𝐷

Spearman 𝜌 0.196 0.226 0.282 0.589 0.254

first overall with a score of 0.379, outperforming the strongest baseline, ClimAgent, by 32.5%. SIREN-Skill ranks second with a score of 0.303. SIREN-RAG leads EC, SP, and RD, whereas SIREN-Modeling leads IA, demonstrating the complementary benefits of retrieved cases and learned predictive evidence. The weaker overall results of SIREN-Modeling indicate that historical experience must be used effectively rather than merely made available. 4.2.2 Cross-Procedure Dependency. Table 4 reports inter-stage dependencies for SIREN-RAG with Gemini 3.1 Flash-Lite. Spearman 𝜌 measures the association between each procedure, or its cumulative prefix, and the subsequent procedure across all operational chains. All five dependencies are positive, with the strongest association observed for 𝐸𝐶/𝑆𝑃 → 𝐼𝐴 at 0.589. The positive associations for 𝐼𝐴 → 𝑅𝐷 and 𝐸𝐶/𝑆𝑃/𝐼𝐴 → 𝑅𝐷 further indicate that stronger upstream performance tends to improve the final decision. These consistent dependencies confirm that performance propagates through the end-to-end warning chain, highlighting the importance of complete operational chains beyond isolated tasks.

Related Work

In this section, we review recent studies on benchmarks and LLM agents in weather science. Benchmarks for Weather Science. Early benchmarks focus on numerical forecasting and event detection [30–32], supporting meteorological prediction and hazard localization [1, 5, 21]. LLMoriented datasets extend evaluation to QA tasks, including weather reasoning [14, 18], anomaly interpretation [4], event forecasting [16], warning-message generation [8], and forecast reporting [45], but they typically provide the required inputs rather than requiring active evidence collection, tool use, and multi-stage reasoning. Agent benchmarks more closely reflect realistic workflows: ZephyrusBench [38] covers diverse weather-science problems, ClimaBench

[39] focuses on research-oriented climate analysis, and ClimateAgent-Bench-85 [15] evaluates executable workflows and report generation. Extreme-weather benchmarks further assess signal identification, physical explanation, and structured diagnosis [13, 34]. Nevertheless, these benchmarks remain focused on scientific analysis rather than operational weather applications, e.g., extremeweather early warning, which require impact-oriented assessment and warning decisions. LLM Agents for Weather Science. Recent LLM agents automate weather and climate workflows through tool use, code execution, and domain-specific data access. Zephyrus [38] provides unified Python interfaces to meteorological data, forecasting models, and simulations, while ClimAgent [39] supports planning and refinement for open-ended climate analysis. ClimateAgent [15] coordinates specialized agents to decompose climate questions and selfcorrect the resulting code, visualizations, and reports. For extremeweather diagnosis, EWE [13] combines knowledge-guided planning with meteorological tools, whereas HVR-Met [34] iteratively verifies anomalous signals and replans its reasoning.

Conclusion

In this paper, we formulate extreme-weather early warning as an agentic problem and construct SIREN-Bench, a benchmark that covers four atomic warning procedures and an end-to-end warning chain. To address this problem, we develop SIREN, an experiencegrounded agent framework that combines an execution environment that integrates heterogeneous weather evidence and analytical tools with a family of agents that leverage historical cases through retrieval, skill distillation, and predictive modeling. Experiments with multiple LLM backbones and strong weather-agent baselines demonstrate the effectiveness of SIREN on both atomic tasks and operational warning chains.

Limitations and Ethical Considerations Limitations. Our study is limited by (1) its focus on U.S. extremeweather events and selected data sources, which may restrict generalization across regions, periods, and hazard regimes; and (2) the gap between retrospective benchmark evaluation and real-world deployment, where data latency, missing data, evolving forecasts, and operational constraints may affect performance. Ethical considerations. Our study (1) uses only publicly available governmental and geospatial data in accordance with the stated access and attribution requirements, without identifiable private information or direct human-subject participation, and therefore did not require IRB review under our institutional policy; and (2) may be misused or produce incorrect warnings, public alarm, or resource misallocation, so SIREN is intended only to support qualified professionals who retain human oversight and decision authority.

Generative AI Usage Apart from the LLMs used as research components described in this paper, generative AI tools assisted with code generation, language polishing, and the creation of figure icons unrelated to the reported results. All generated content was reviewed by the authors, who independently verified the experimental results and conclusions and take full responsibility for the manuscript.

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

References [1] Saugat Adhikari, Da Yan, Zhe Jiang, Jiao Han, Zelin Xu, Yupu Zhang, Arpan Sainju, and Yang Zhou. 2025. Scaling Terrain-Aware Spatial Machine Learning for Flood Mapping on Large Scale Earth Imagery Data. ACM Trans. Spatial Algorithms Syst. 11, 2, Article 9 (April 2025), 29 pages. doi:10.1145/3703157 [2] Gustau Camps-Valls, Miguel-Ángel Fernández-Torres, Kai-Hendrik Cohrs, Adrian Höhl, Andrea Castelletti, Aytac Pacal, Claire Robin, Francesco Martinuzzi, Ioannis Papoutsis, Ioannis Prapas, et al. 2025. Artificial intelligence for modeling and understanding extreme weather and climate events. Nature Communications 16, 1 (2025), 1919. [3] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. 2025. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=6s5uXNWGIh [4] Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Wei Chen, Bing Zhu, Junwei Liang, and Zixuan Yuan. 2025. ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 5322–5333. doi:10.1145/3711896.3737406 [5] Shengchao Chen, Guodong Long, Jing Jiang, and Chengqi Zhang. 2024. Personalized adapter for large meteorology model on devices: towards weather foundation models. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2696, 47 pages. [6] Erin Coughlan De Perez, Kristoffer B Berse, Lianne Angelico C Depante, Evan Easton-Calabria, Elton Pierre R Evidente, Theodore Ezike, Dorothy Heinrich, Christopher Jack, Alfredo Mahar Francisco A Lagmay, Selma Lendelvo, et al. 2022. Learning from the past in moving to the future: invest in communication and response to weather early warnings to reduce death and damage. Climate Risk Management 38 (2022), 100461. [7] Federal Emergency Management Agency. 2026. OpenFEMA Data Sets. https: //www.fema.gov/about/openfema/data-sets. Accessed: July 6, 2026. [8] Giacomo Gonella, Gian Maria Campedelli, Stefano Menini, and Marco Guerini. 2026. CrisiText: A dataset of warning messages for LLM training in emergency communication. In Findings of the Association for Computational Linguistics: EACL 2026, Vera Demberg, Kentaro Inui, and Lluís Marquez (Eds.). Association for Computational Linguistics, Rabat, Morocco, 6657–6677. doi:10.18653/v1/2026. findings-eacl.350 [9] Google. 2026. Gemini 3.1 Flash-Lite. https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-flash-lite. Accessed: July 8, 2026. [10] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. DS-agent: automated data science by empowering large language models with case-based reasoning. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 668, 36 pages. [11] Zijie Guo, Jiong Wang, Fenghua Ling, Wangxu Wei, Xiaoyu Yue, Zhe Jiang, Wanghan Xu, Jing-Jia Luo, Lijing Cheng, Yoo-Geun Ham, et al. 2025. A selfevolving AI agent system for climate science. arXiv preprint arXiv:2507.17311 (2025). [12] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Robert Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Yongxin Ni, Zhibin Gou, Zongze Xu, Yuyu Luo, and Chenglin Wu. 2025. Data Interpreter: An LLM Agent for Data Science. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 19796–19821. doi:10.18653/v1/2025.findings-acl.1016 [13] Zhe Jiang, Jiong Wang, Xiaoyu Yue, Zijie Guo, Wenlong Zhang, Fenghua Ling, Wanli Ouyang, and Lei Bai. 2025. EWE: An Agentic Framework for Extreme Weather Analysis. arXiv preprint arXiv:2511.21444 (2025). [14] Soyeon Kim, Cheongwoong Kang, Myeongjin Lee, Eun-Chul Chang, Lee Jaedeok, and Jaesik Choi. 2026. K-MetBench: A Multi-Dimensional Benchmark for FineGrained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology. In Findings of the Association for Computational Linguistics: ACL 2026, Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 5574–5612. doi:10.18653/v1/2026.findings-acl.275 [15] Chenyue Li, Hyeonjae Kim, Wen Deng, Mengxi Jin, HUANG Wen, Mengqian Lu, and Binhang Yuan. 2026. ClimateAgent: Multi-Agent Orchestration for Complex Climate Data Science Workflows. Transactions on Machine Learning Research (2026). https://openreview.net/forum?id=XLWvXNumGa [16] Haobo Li, Zhaowei Wang, Jiachen Wang, Yueya Wang, Alexis Kai Hon Lau, and Huamin Qu. 2025. CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting. In Proceedings of the 2025 Conference on Empirical Methods in

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 17536–17562. doi:10.18653/v1/2025.emnlp-main.886 [17] Fan Liu, Xiaozhao Zeng, and Hao Liu. 2026. Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id= kZHSvETWdi [18] Chengqian Ma, Zhanxiang Hua, Alexandra Anderson-Frey, Vikram Iyer, Xin Liu, and Lianhui Qin. 2024. Weatherqa: Can multimodal language models reason about severe weather? arXiv preprint arXiv:2406.11217 (2024). [19] National Oceanic and Atmospheric Administration. 2026. High-Resolution Rapid Refresh (HRRR). https://rapidrefresh.noaa.gov/hrrr/. Accessed: July 7, 2026. [20] National Oceanic and Atmospheric Administration. 2026. National Oceanic and Atmospheric Administration. https://www.noaa.gov/. Accessed: July 6, 2026. [21] Hang Ni, Weijia Zhang, and Hao Liu. 2026. UniExtreme: A Universal Foundation Model for Extreme Weather Forecasting. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (Republic of Korea) (KDD ’26). Association for Computing Machinery, New York, NY, USA, 1101–1112. doi:10.1145/3770854.3780172 [22] NOAA National Centers for Environmental Information. 2026. Storm Events Database. https://www.ncei.noaa.gov/stormevents/. Accessed: July 6, 2026. [23] NOAA/NWS Storm Prediction Center. 2026. Storm Prediction Center Mesoscale Analysis Pages. https://www.spc.noaa.gov/exper/mesoanalysis/. Accessed: July 7, 2026. [24] NOAA/NWS Storm Prediction Center. 2026. Storm Prediction Center Mesoscale Discussions. https://www.spc.noaa.gov/products/md/. Accessed: July 6, 2026. [25] OpenAI. 2026. Introducing GPT-5.4 mini and nano. https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/. Accessed: July 8, 2026. [26] OpenStreetMap contributors. 2026. OpenStreetMap Data. https://www. openstreetmap.org/. Accessed: July 7, 2026. [27] Sally Potter, Sara Harrison, and Peter Kreft. 2021. The benefits and challenges of implementing impact-based severe weather warning systems: perspectives of weather, flood, and emergency management personnel. Weather, climate, and society 13, 2 (2021), 303–314. [28] Qwen Team. 2026. Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. https://qwen.ai/blog?id=qwen3.6-27b. Accessed: July 16, 2026. [29] Qwen Team. 2026. Qwen3.7-Plus: Multimodal Agent Intelligence. https://qwen. ai/blog?id=qwen3.7-plus. Accessed: July 8, 2026. [30] Evan Racah, Christopher Beckham, Tegan Maharaj, Samira Ebrahimi Kahou, Prabhat, and Christopher Pal. 2017. Extreme weather: a large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 3405–3416. [31] Nian Ran, Peng Xiao, Yue Wang, Wesley Shi, Jianxin Lin, Qi Meng, and Richard Allmendinger. 2025. HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=5AtlfHYCPa [32] Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sha. 2024. WeatherBench 2: A Benchmark for the Next Generation of DataDriven Global Weather Models. Journal of Advances in Modeling Earth Systems 16, 6 (2024), e2023MS004019. e2023MS004019 2023MS004019. arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2023MS004019 doi:10.1029/2023MS004019 [33] Markus Reichstein, Vitus Benson, Jan Blunk, Gustau Camps-Valls, Felix Creutzig, Carina J Fearnley, Boran Han, Kai Kornhuber, Nasim Rahaman, Bernhard Schölkopf, et al. 2025. Early warning of complex climate risk with integrated artificial intelligence. Nature Communications 16, 1 (2025), 2564. [34] Shuo Tang, Jiadong Zhang, Gengxian Zhou, Qizhao Jin, Qinxuan Wang, Yi Hu, Ning Hu, Hongchang Ren, Lingli He, Shiming Xiang, Jingtao Ding, Jian Xu, Jiaolan Fu, and Cheng-Lin Liu. 2026. HVR-Met: A Hypothesis-Verification-Replanning Agentic System for Extreme Weather Diagnosis. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=tb2aPbeg86 [35] United Nations. n.d.. Early Warnings for All. https://www.un.org/en/ climatechange/early-warnings-for-all. Accessed: July 27, 2026. [36] United States Census Bureau. 2025. TIGER/Line Shapefiles. https://www.census. gov/geographies/mapping-files/time-series/geo/tiger-line-file.html. 2025 release; accessed: July 7, 2026. [37] U.S. Department of Energy. 2026. Open Energy Data Initiative. https://data. openei.org/. Accessed: July 6, 2026. [38] Sumanth Varambally, Marshall Fisher, Jas Thakker, Yiwei Chen, Zhirui Xia, Yasaman Jafari, Ruijia Niu, Manas Jain, Veeramakali Vignesh Manivannan, Zachary Novack, Luyu Han, Srikar Eranky, Salva Rühling Cachay, Taylor BergKirkpatrick, Duncan Watson-Parris, Yian Ma, and Rose Yu. 2026. Zephyrus: An

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Agentic Framework for Weather Science. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=aVeaNahsID [39] Hao Wang, Jindong Han, Wei Fan, and Hao Liu. 2026. ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis. In Findings of the Association for Computational Linguistics: ACL 2026, Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 21219–21241. doi:10.18653/v1/ 2026.findings-acl.1067 [40] Jon Weers. 2015. Utility Outage Information. Open Energy Data Initiative (OEDI), National Renewable Energy Laboratory, https://data.openei.org/submissions/460. Published: August 28, 2015; accessed: July 6, 2026. [41] World Meteorological Organization. 2015. WMO Guidelines on Multi-hazard Impact-based Forecast and Warning Services. Technical Report WMO-No. 1150. World Meteorological Organization, Geneva, Switzerland. [42] World Meteorological Organization. n.d.. WMO and the Early Warnings for All Initiative. https://wmo.int/activities/early-warnings-all/wmo-and-earlywarnings-all-initiative. Accessed: July 27, 2026. [43] Zherui Yang, Fan Liu, Yansong Ning, and Hao Liu. 2026. EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Management. arXiv preprint arXiv:2606.03841 (2026). [44] Kaikai Zhang, Xiang Wang, Haoluo Zhao, Nan Chen, Mengyang Yu Jing-Jia Luo, Tao Song, and Fan Meng. 2026. TianJi: An autonomous AI meteorologist for discovering physical mechanisms in atmospheric science. arXiv preprint arXiv:2603.27738 (2026). [45] Zinan Zheng, Yang Liu, Nuo Chen, Juepeng Zheng, Hong Cheng, and Jia Li. 2026. WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=WlScVvMwpB

We use Qwen3.6-27B [28] with temperature 0 for all LLM-based evaluation components and apply the same evaluator to every method. The core prompts are provided in Appendix K.

Task Taxonomy

Table 5 summarizes the 18 atomic subtasks, their abbreviations, evaluation protocols, and representative question templates.

Extreme Weather Events

Table 6 defines the 12 event families covered by SIREN-Bench.

Analytical Tools

Table 7 lists the tools available in our agentic environment. Access to historical cases is restricted to experience-grounded variants, while ML tools are restricted to SIREN-Modeling. Notably, our agents use task-specific ML models (e.g., weather forecasting, decision-tree, and regression models) as intermediate reasoning tools rather than standalone predictors, while its ultimate goal is to solve practical and operational warning tasks that are not directly comparable to the isolated prediction objectives of conventional ML baselines.

Implementation Details

For each backbone, all compared methods use deterministic decoding with temperature 0. We use identical benchmark inputs and the same evaluation pipeline for our methods and the reproduced baselines. Given the scale of the benchmark and the number of evaluated methods and backbone models, we run each configuration once and report the resulting performance. All experience-grounded variants use the same pool of historical cases. SIREN-RAG retrieves up to six cases for the target task, whereas SIREN-Skill may select and execute up to three practice cases before producing an answer. Transient model-service failures, unsuccessful code or tool executions, and malformed action or answer formats trigger bounded retries with explicit error feedback. A run is considered unsuccessful if the corresponding retry budget is exhausted.

Comparative Results across Event Types

Figure 7 extends the aggregate event analysis to all methods and backbones. When averaged across the 12 event families, SIRENModeling performs marginally best with Qwen, whereas SIRENRAG leads clearly with GPT and Gemini. The ordering varies across individual hazards, indicating that retrieved analogues and learned predictive evidence provide complementary benefits rather than uniform, event-independent gains.

Comparative Results across States

Figure 8 shows that the benefits of historical experience extend broadly across geographic regions. SIREN-RAG achieves the highest mean across the selected states for all three backbones, narrowly outperforming SIREN-Modeling with Qwen and leading more clearly with GPT and Gemini. The remaining variation across states indicates that agent performance is not yet geographically robust across heterogeneous local conditions.

Comparative Results across Event Types and States

We report the results for Qwen3.7-Plus on the four representative event families with the largest sample counts among those spanning at least 12 states. For each family, all 12 most represented states contain nonzero samples. Figure 9–Figure 12 reveal substantial geographic variation within each event family. On average, SIREN-RAG performs best for convective storms and tropical systems, whereas SIREN-Modeling performs best for winter weather and tornadoes. The varying margins across states indicate that agent robustness depends jointly on event-specific dynamics and heterogeneous local conditions.

Comparative Results across Months

Figure 13 demonstrates that experience grounding remains effective throughout the annual cycle. SIREN-Modeling achieves the highest monthly performance with Qwen, whereas SIREN-RAG leads with GPT and Gemini. No baseline consistently dominates across all months, and the varying margins among the three SIREN mechanisms indicate uneven robustness to seasonal changes in event composition.

Human Evaluation

We validate both LLM-based evaluators using SIREN-RAG outputs for all queries in each relevant open-ended setting and across all three backbone models. The recall-based study covers Public Warning and Mission Assignment, whereas the 0–10 study covers Physical Understanding. For each response, annotators assess the analysis and score produced by the evaluator, independently assign a human score under the same guidelines, and reach a binary consensus judgment. We measure LLM–human alignment using (1) Agreement Rate, the percentage of evaluator judgments considered reasonable by the annotators, (2) Kendall Tau-b Rank

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 5: Atomic task taxonomy in SIREN-Bench. MC, NR, GL, and OG denote multiple-choice classification, numeric regression, geospatial localization, and open-ended generation, respectively. Task Procedure

Subtask

Event Characterization

Type Understanding Physical Understanding

Abbr. Eval. Representative Question TU PU

MC OG

Which extreme-weather event type best describes this record? What physical processes and forcing ingredients drive this event?

Spatiotemporal Prediction

Spatial Detection Temporal Prediction Severity Detection Path Prediction Duration Prediction

SD TP SeD PP DP

GL MC MC MC NR

Which state and county are currently experiencing this event? How many hours from the reference time until the event starts? What is the current severity level of this event? In which direction is this event expected to move? How long will this event last, in hours?

Impact Assessment

Agricultural Impact Economic Impact Human Impact Household Impact Infrastructure Impact Energy Impact

AI EcI HI HoI II EnI

NR NR NR NR NR NR

How much crop damage will this event cause, in USD? How much property damage will this event cause, in USD? How many injuries will this event cause? How many households require FEMA housing assistance? How much public-assistance funding is needed, in USD? How long will the county-level power outage last, in hours?

Alert Operation Decision AOD Public Warning PW Responsive Hazard Mitigation HM Decision-Making Mission Assignment MA Recovery Center Location RCL

MC OG MC OG GL

Which operation code should be applied to the current alert? What public response instruction should be issued for this event? Which hazard-mitigation projects should be selected? What mission-assignment response is needed for this event? Where should disaster-recovery centers be activated?

Table 6: Extreme-weather event families covered by SIREN-Bench. Event Type

Abbr.

Description

Convective Storms

Conv.

Tropical Systems Tornadoes Floods Winter Weather High-Wind Events Visibility Hazards Heat Fire Drought Marine Hazards Cold-Weather Events

Trop. Torn. Flood Winter Wind Visib. Heat Fire Drought Marine Cold

Thunderstorms producing hazards such as hail, lightning, heavy rain, or damaging convective winds. Hurricanes, tropical storms, tropical depressions, and related wind, rain, and surge hazards. Violently rotating columns of air extending from a convective cloud to the ground. Riverine, flash, coastal, or other inundation of normally dry areas. Snow, blizzard, sleet, ice, freezing rain, and other hazardous winter precipitation. Damaging non-convective winds, strong-wind episodes, and wind advisories. Fog, dust, smoke, or related conditions that substantially reduce visibility. Excessive-heat events and heat waves posing elevated thermal risk. Wildfires and fire-weather conditions that support ignition or rapid spread. Prolonged precipitation or moisture deficits that cause water stress. Hazardous marine winds, waves, thunderstorms, or other conditions over coastal waters. Extreme cold, wind chill, freeze, and related low-temperature hazards.

Correlation Coefficient, which measures agreement in pairwise ordering while accounting for tied scores, and (3) Spearman Rank Correlation Coefficient, which measures the monotonic association between the complete evaluator and human score rankings. Table 8 and Table 9 report the alignment results, indicating that both evaluators generally align with human rankings.

Expert-Reviewed Case Study

We further examine a representative Physical Understanding case for severe convection over the northeastern United States on July 6, 2021. With Qwen3.7-Plus, SIREN-RAG raises the normalized score from 0.4 for SIREN-Base to 0.8. It retrieves six historical analogues, including four convective cases and five with overlapping

regional conditions, then checks the analogy against target-time HRRR fields and meteorological diagnostics. As summarized below, the resulting explanation correctly connects a warm, moist, unstable environment and sufficient shear to organized convection, damaging downburst winds, and hail. A domain expert reviewed the complete input, observable tool-use trajectory, execution observations, and final answer, and confirmed that the evidence use and meteorological conclusion were professionally reasonable. Representative SIREN-RAG Physical Understanding Case Input. At 20:23 UTC on July 6, 2021, severe weather affected a corridor from eastern Pennsylvania and New Jersey through southern New

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Table 7: Enabled analytical tools in the agentic environment. Category

Tool

Description

Evidence Indexing

access_historical_cases access_hrrr_reanalysis access_osm_data access_run_artifact access_spc_mesoscale_images save_run_artifact

Retrieves percentile slices of prior-year cases for experience-grounded variants. Loads cached or newly fetched HRRR analysis fields as a reusable weather-data bundle. Retrieves indexed OpenStreetMap features or road networks for a target U.S. region. Reads a previously saved artifact through the controlled run-data boundary. Fetches SPC mesoscale-analysis imagery for a requested valid time and product set. Persists structured data, text, tables, or arrays for reuse within the current run.

apply_threshold_analysis compute_frontogenesis compute_layer_profile_metrics Meteorological compute_wind_diagnostics Analysis describe_meteorological_analysis_options inspect_gridded_data_bundle summarize_gridded_field

Creates a threshold mask and reports the fraction of a field meeting a specified condition. Computes a two-dimensional frontogenesis diagnostic from temperature and wind fields. Derives layer diagnostics such as bulk shear, lapse rate, helicity, and relative humidity. Computes wind speed, vorticity, and divergence from gridded wind components. Lists the supported meteorological diagnostics and threshold-analysis modes. Inspects a weather or diagnostic bundle and reports its available variables. Produces compact descriptive statistics for a gridded field or masked region.

Atmospheric Forecasting

access_model_forecast

Loads model forecast fields at a requested initialization time and lead time.

Visual Processing

annotate_image crop_image visualize_field_comparison visualize_gridded_field zoom_in_image

Adds points, boxes, and text labels to an image artifact. Extracts a rectangular image region using normalized coordinates. Renders two compatible fields side by side or as their difference. Renders a gridded weather or diagnostic field as a map image. Crops and enlarges a selected image region for closer inspection.

Impact Modeling

estimate_integrated_impact rank_impacted_regions summarize_impact_distribution

Combines hazard, exposure, and vulnerability inputs into an impact proxy. Orders regions by a selected impact-score field. Summarizes impact values with descriptive statistics and percentiles.

align_bbox_to_weather_grid build_region_grid_mask Geospatial list_supported_us_geospatial_layers Normalization lookup_point_location normalize_location_reference ML Development

describe_ml_development_options fit_vulnerability_curve predict_with_saved_model train_ml_model

Snaps a geographic bounding box to the enclosing HRRR grid cells. Rasterizes a U.S. administrative region onto the HRRR grid. Lists the prepared U.S. administrative layers available for spatial normalization. Resolves coordinates to prepared U.S. administrative regions. Matches a U.S. place reference to a canonical administrative feature. Lists supported ML problem types, input forms, and model families. Fits a vulnerability function relating hazard intensity to observed impact. Applies a trained model to inference data and optionally saves its predictions. Trains and evaluates a lightweight predictive model on agent-constructed data.

Table 8: LLM–human alignment performance for the recallbased evaluator (%). Agree. Rate

Kendall Tau

Spearman Coeff.

73.96

80.50

87.05

Table 9: LLM–human alignment performance for the 0–10 scale evaluator (%). Agree. Rate

Kendall Tau

Spearman Coeff.

66.67

72.48

78.01

England. Explain the meteorological setup and key forcing ingredients in one focused paragraph. Observable agent trajectory.

(1) Retrieve analogues. The agent calls access_historical_cases, scans all 1,897 eligible cases, and ranks them by event type, region, and task form. It retains six cases, including prior convective-wind episodes over Pennsylvania, New Jersey, New York, and southern New England. (2) Inspect target-time conditions. It calls access_hrrr_reanalysis and access_spc_mesoscale_images for the northeastern United States near 20 UTC. The returned HRRR fields show a warm surface environment, strong low-level moisture, and locally enhanced winds, while unavailable SPC panels are explicitly recorded rather than inferred. (3) Diagnose storm organization. It applies compute_layer_profile_metrics, compute_wind_diagnostics, and visualize_gridded_field. The observations include 850-hPa moisture up to 15.4 g kg −1 and 850–500-hPa bulk shear with a 95th percentile of 17.4 m s −1 . (4) Synthesize the evidence. The agent combines the regional analogues with the target-specific diagnostics, rather than copying a

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Score

Conv.

Trop.

Torn.

Flood

Winter

Wind

Visib.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Heat

Fire

Drought

Marine

Cold

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Heat

Drought

(a) Qwen3.7-Plus

Score

Conv.

Trop.

Torn.

Flood

Winter

Wind

Visib.

Fire

Marine

Cold

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Heat

Drought

(b) GPT-5.4 mini

Score

Conv.

Trop.

Torn.

Flood

Winter

Wind

Visib.

Fire

Marine

Cold

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 7: Method comparison across event types. Each subfigure uses one LLM backbone and each panel represents one event family.

historical answer, to identify organized multicell convection and its principal hazards. Output. The agent reports a warm and moist environment with substantial instability and locally stronger shear near mesoscale boundaries. It concludes that organized thunderstorms can produce damaging convective wind gusts and hail while moving into southern New England, with gradual weakening near the more stable coastal environment.

<CODE EXECUTION AND RESPONSE RULES> Available multimodal evidence: <EVIDENCE SUMMARY> Available analytical tools: <TOOL DESCRIPTIONS> Target question: <TARGET QUESTION>

Prompts

This section presents the prompts used throughout SIREN. They include general prompts for individual-task execution, experiencegrounding prompts for SIREN-RAG, SIREN-Skill, and SIREN-Modeling, operational-chain prompts for coordinating the sequential warning chain, and recovery prompts for handling execution, formatting, and timeout failures.

K.1

General Prompts

Individual Initial Prompt You are an extreme-weather early-warning agent that solves the target task through evidence-grounded reasoning and interactive code execution. <WORKFLOW AND ACTION RULES>

Task-specific guidance: <TASK GUIDANCE> Experience-grounding guidance, when enabled: <EXPERIENCE GUIDANCE> Accumulated skill guidance, when available: <ACCUMULATED SKILL GUIDANCE> Prior execution observations: <EXECUTION HISTORY> Available intermediate artifacts: <ARTIFACT SUMMARY>

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Score

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

(a) Qwen3.7-Plus

Score

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

(b) GPT-5.4 mini

Score

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 8: Method comparison across the 12 most represented states.

Score

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 9: State-level method comparison for convective storms with Qwen3.7-Plus.

Score

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 10: State-level method comparison for tropical systems with Qwen3.7-Plus.

Score

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 11: State-level method comparison for winter weather with Qwen3.7-Plus.

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Score

1.0

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY HI

0.5 0.0

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 12: State-level method comparison for tornadoes with Qwen3.7-Plus.

Score

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

0.50 0.25

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Aug

Oct

(a) Qwen3.7-Plus

Score

Jan

Feb

Mar

Apr

May

Jun

Jul

Sep

Nov

Dec

0.50 0.25

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Aug

Oct

(b) GPT-5.4 mini

Score

Jan

Feb

Mar

Apr

May

Jun

Jul

Sep

Nov

Dec

0.50 0.25

Zephyrus EWE

ClimAgent ClimateAgent

HVR-Met SIREN-Base

SIREN-RAG SIREN-Skill

SIREN-Modeling

Figure 13: Method comparison across calendar months. Workflow and Action Rules Follow an workflow.

iterative

reason–execute–observe

1. At each turn, briefly state what is known and what evidence is still needed. 2. If further evidence is required, provide a short plan and pseudocode, then take exactly one code-execution action. 3. Use the returned observation to revise the next action. Do not repeat an unsuccessful action without addressing its failure. 4. When the task depends on external evidence or analytical tools, do not produce the final solution before at least one successful execution observation. 5. Finish only when the accumulated evidence supports the requested answer. Preserve the exact answer type and format required by the target

question. 6. For an operational chain, solve only the current individual procedure and carry its completed answer forward before moving to the next procedure.

Code Execution and Response Rules Use only the documented analytical tools and the guarded code-execution interface. Do not access hidden storage paths or tool implementations. Inside an execution action, define a callable run(tools). Use code for data access, filtering, aggregation, statistical analysis, visualization, and task-specific computation. Return only compact textual findings and the images that must be inspected in the next reasoning turn. Keep large intermediate objects in reusable artifacts.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Valid execution response: <plan> A concise list of the remaining evidence-gathering steps. </plan> <pseudocode> A compact description of the next computation. </pseudocode> <execute> def run(tools): ... </execute> Valid final response: <solution> The final answer in the format required by the target question. </solution> Each response must contain exactly one action: either one execution action or one final solution. Never place executable code outside the execution block or a final answer outside the solution block.

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Historical case collection: <HISTORICAL CASES> 1. Inspect the complete historical collection relevant to the target subtask. 2. Rank cases by their usefulness for the target task. Consider task intent, event conditions, reference time, location, event type, question form, answer format, source variables, and available labels. 3. Select at most six cases. Preserve each selected case’s question, reference answer, and event metadata so that its analogy remains interpretable. 4. Use the selected cases as evidence and answer-format references. Do not copy a historical answer without checking it against the current event evidence. 5. Continue the evidence-grounded solving workflow with the selected cases in context.

Experience Guidance: SIREN-Skill Rehearsal

Observation Continuation Prompt

You are the rehearsal stage of SIREN-Skill. Historical cases are practice tasks for acquiring reusable solving procedures, not direct answer examples.

Continue solving the same extreme-weather task using the newest execution result.

Target question and event conditions: <TARGET TASK>

Latest observation: <LATEST OBSERVATION>

Historical case collection: <HISTORICAL CASES>

Remaining objective: <REMAINING OBJECTIVE>

Current skill guidance: <CURRENT SKILL GUIDANCE>

Accumulated skill guidance, when available: <ACCUMULATED SKILL GUIDANCE>

Experience Guidance: SIREN-RAG

1. Inspect the complete historical collection relevant to the target subtask and select at most three cases with the highest rehearsal value. 2. For each selected case, hide its reference answer and solve the rehearsal question using the same evidence-grounded workflow as the target task. 3. After completing the rehearsal, reveal the reference answer and invoke the Skill Refinement Prompt to compare the solution process with the reference. 4. Retain only reusable procedural guidance. Do not store event-specific answers or details that would not transfer to another case. 5. Repeat rehearsal and refinement for the selected cases, then solve the target task using the accumulated skill guidance.

You are the case-retrieval stage of SIREN-RAG. Use historical cases as analogical evidence for the target task.

Skill Refinement Prompt

Target question and event conditions: <TARGET TASK>

You are refining the reusable skill guidance of SIREN-Skill after one rehearsal case.

Relevant prior observations and artifacts: <EXECUTION HISTORY AND ARTIFACTS> Treat the latest observation as the newest evidence. Either gather the next necessary piece of evidence through one execution action or return the final solution when the evidence is sufficient.

K.2

Experience-Grounding Prompts

SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents

Existing skill guidance: <CURRENT SKILL GUIDANCE> Rehearsal question and event conditions: <REHEARSAL TASK> Rehearsal solution and execution observations: <REHEARSAL TRAJECTORY>

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

5. Apply the trained predictor to the current task and return its prediction, validation evidence, and the information needed to interpret the prediction. 6. Do not replace model training with nearest-neighbor retrieval, descriptive statistics, or heuristic scoring alone. Continue to the target-solving phase only after a trained and validated predictor is available.

Reference answer: <REFERENCE ANSWER>

K.3 1. Identify which reasoning steps, evidence choices, tool-use strategies, or answer-format decisions were effective. 2. Diagnose errors by comparing the rehearsal solution with the reference answer and the supporting evidence. 3. Add or revise only short, actionable guidance that can improve future tasks of the same kind. Keep one reusable procedure per item. 4. Preserve useful existing guidance and remove an item only when the rehearsal provides clear evidence that it is misleading. 5. Exclude the rehearsal’s final answer and event-specific facts from the refined skill. Return only the revised skill guidance.

Experience Guidance: SIREN-Modeling You are the modeling stage of SIREN-Modeling. Convert the historical case collection into a trained task-specific predictor that provides predictive evidence for the target task.

End-to-end Operational-Chain Prompts

Operational-Chain Initial Prompt You are solving an operational extreme-weather warning chain in one continuous conversation. Work on exactly one individual procedure at a time while preserving the shared event context. Full operational-chain question: <FULL CHAIN QUESTION> Individual procedures and required answer formats: <PROCEDURE DEFINITIONS> Current individual procedure: <CURRENT PROCEDURE> Available evidence and analytical tools: <EVIDENCE AND TOOLS> Task-specific and experience-grounding guidance: <TASK AND EXPERIENCE GUIDANCE> <WORKFLOW AND ACTION RULES> <CODE EXECUTION AND RESPONSE RULES>

Target question and event conditions: <TARGET TASK> Historical case collection: <HISTORICAL CASES>

Solve only the current individual procedure. Return its answer in the required format and do not assemble the final chain response until every procedure has been completed.

Available ML development tools: <ML TOOLS>

Operational-Chain Transition Prompt 1. Use the complete historical collection relevant to the target subtask. Construct explicit training and validation samples with task-appropriate inputs and targets. 2. Design features from historical cases and available meteorological, spatial, temporal, or impact evidence. Prevent target leakage. 3. Select an appropriate ML model, train it on the training split, and evaluate it on held-out validation samples with a task-appropriate metric. 4. If the requested answer is not directly learnable, define a learnable intermediate target that can be mapped to the required answer, then train and validate a model for that target.

The previous individual procedure is complete. Continue the same operational chain with the shared event context and prior answers in scope. Completed procedure answers: <COMPLETED PROCEDURE ANSWERS> Next individual procedure: <NEXT PROCEDURE> Guidance for the next procedure: <NEXT-PROCEDURE GUIDANCE>

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Use historical experience only for the current individual procedure. Solve the next procedure in its required answer format, preserve the completed answers, and defer final chain assembly until all procedures are complete.

K.4

Recovery Prompts

Execution Recovery Prompt The previous code execution failed. Correct the failure before continuing the task. Failed action and returned error: <FAILED EXECUTION> Earlier unsuccessful attempts: <RECOVERY HISTORY> Remaining objective: <REMAINING OBJECTIVE> Identify the likely cause, revise the plan, and produce one corrected execution action. Use documented tools only, repair invalid arguments or result structures, and change strategy when the same failure has already occurred. Do not return a final solution in this recovery turn.

Output-Format Recovery Prompt The previous response did not follow the required output format. Correct its format without redoing the analysis or introducing new facts. Required answer or action format: <REQUIRED FORMAT> Previous invalid response: <INVALID RESPONSE> Format error: <FORMAT ERROR> Return one corrected response that preserves the intended content and satisfies the required format exactly.

Timeout Recovery Prompt The previous code execution exceeded the time limit. Produce a faster evidence-gathering action. Timed-out action: <TIMED-OUT EXECUTION>

Hang Ni, Weijia Zhang, Fan Liu, Mengqian Lu, Hao Liu

Evidence already available: <AVAILABLE OBSERVATIONS> Remaining objective: <REMAINING OBJECTIVE> Reduce the scope of the next computation, avoid repeated work and broad scans, and reuse existing observations or artifacts. Return one revised execution action and do not return a final solution in this recovery turn.

Record · ID 405694 · SHA-256 3bec2a111f3a25cc
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.