Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
Gal Engelberg, Michael Arenzon, Leon Goldberg open.security (by Sola Security), Tel Aviv, Israel [email protected]
arXiv:2607.27288v1 [cs.CR] 29 Jul 2026
Abstract Enterprises are moving toward autonomous cyber defense: agentic AI that builds situational awareness of an organization’s security state and reasons from it to assessments, decisions, and actions. This rests on a holistic view of the enterprise’s security state, the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent can produce an answer but whether it should be trusted to. The field cannot yet answer this question. Real enterprise environments are private, crossvendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the environment data gap. We present Open Security Benchmark (OSB), a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment-a frozen, holistic view of the security state-and evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor’s native API over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation. K eywords Autonomous Enterprise Cyber Defense · Security Posture Management · Agentic AI · Benchmarking
1
Introduction
Autonomous cyber defense, systems that sense, reason about, and act on cyber threats with limited human intervention, is now an explicit goal in enterprise security, pursued across reviews of automated and autonomous cyber defense [1] and training environments for autonomous agents [2]. Yet every stage of the sense-reason-act loop presupposes one foundation: an accurate, current, holistic understanding of the organization’s own security state. Situational-awareness research makes this precise [3], adapted in security as cyber situational awareness [4]: an agent cannot act on what it cannot see. Establishing this self-knowledge is therefore a prerequisite for autonomous cyber defense. Security posture management, the continuous, holistic assessment of an organization’s defensive readiness and security state, supplies this situational-awareness foundation, on which any autonomous assessment or decision rests. This assessment spans multiple subdomains: identity and access [5], cloud and infrastructure configuration [6], data exposure and classification [7], application security and vulnerability state [8], third-party and SaaS integrations [9], and compliance alignment with regulatory and contractual obligations [10]. Catalogs like
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
NIST SP 800-53 [11], the CIS Critical Security Controls [12], and ISO/IEC 27001 [13] codify this work, and posture tooling from cloud auditors [14, 15] to SaaS scanners [16] operationalizes it across vendor stacks. Enterprises increasingly apply agentic AI to this work, with reported gains in coverage and analyst throughput [17]. As deployment scales, the binding question changes: an agent can produce an answer, but should we trust it, in this environment, with this data, given the evidence it produces and the ways it is known to fail? Answering this requires systematic, reproducible evaluation of agents on posture work, faithful to how it is done. Recent cybersecurity benchmarks advance LLM evaluation across adjacent capabilities [18, 19, 20], but each addresses a narrow slice in isolation (§2). More fundamentally, the field faces an environment data gap: enterprise environments are proprietary, span many vendors, and interlink their data, and while executable environments exist for offensive security tasks [21, 22], none surfaces a cross-vendor enterprise environment as a shared, queryable substrate for evaluating defensive posture-investigation agents end to end with gold posture answers. Practitioners therefore cannot compare agents on the work they must deploy them for, and researchers lack a stable target. We present Open Security Benchmark (OSB), a framework for evaluating agentic AI on security posture tasks 1 . OSB closes the environment data gap by surfacing curated enterprise environments as a single, end-to-end view of the security state. Its methodological commitment is a frozen environment that agents investigate through the surface they would use in practice: text-to-SQL over the relational snapshot, or each vendor’s native API over a served instance of the same environment. OSB aims at the full autonomouscyber-defense loop, up through automated remediation, but starts at the assessment stage. Assessment is the read-only work of surfacing and substantiating findings, on which every later stage builds (§7). We formulate posture investigation as an agentic, execution-grounded text-to-SQL task. Each instance is a pair (D, q): D is a read-only relational snapshot of the organization’s holistic security state (identities, cloud and infrastructure, identity providers, SaaS), and q is a natural-language security question. The agent must discover the tables in D relevant to q, compose read-only SQL queries that retrieve the supporting evidence, and synthesize a natural-language answer a. The answer is graded against a closed-form reference (a verdict, a closed set of matching subjects, and the determining configuration facts), so scoring is plan-independent: two semantically equivalent queries score alike, in the denotation-based tradition of semantic parsing that judges a parse by its answer, not its logical form [23]. This extends the schema-realistic, agentic text-to-SQL paradigm of the Spider and BIRD lineage [24, 25, 26] to defensive security work, casting each investigation as a reason-and-act loop [27] over a frozen environment behind a constrained tool interface [28]. A second modality keeps the same D, question, and answer criteria but changes the surface: the agent interrogates a served instance of the same environment over its vendors’ native APIs with real vendor tooling, and its recorded request trace stands in for the SQL the relational modality would produce (§3). We ground this general substrate in a concrete first domain. Identity security posture management (ISPM) is a natural start: cross-vendor by nature, data-rich, consequential, and newly equipped with public benchmark definitions we build on [29, 30]. The framework extends across posture subdomains under the same methodology. It also extends along the progression that autonomous cyber defense automates: assessment, risk assessment and prioritization [31], mitigation and remediation recommendation [32], remediation and response implementation [1, 2], and post-remediation verification via continuous monitoring [33]. It instantiates only the assessment stage (§7). This paper makes three contributions: (i) Open Security Benchmark, an evaluation framework whose five components (a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path) compose a standardized substrate for both relational and native-API investigation, with an explicit trust boundary; (ii) a curated data catalog of multi-vendor enterprise environments, packaged as synthetic-organization datasets that close the environment data gap; and (iii) initial use-case packs (ISPM Visibility and ISPM Cross-Vendor) that instantiate the framework end to end and open an extension path for community-authored packs across posture subdomains. The rest of the paper is structured as follows. Section 2 surveys related work and positions OSB against it. Section 3 presents the two investigation modalities and why text-to-SQL is standardized first. Section 4 describes the framework’s five components. Section 5 presents the environment datasets and Section 6 the benchmarking packs. Section 7 discusses broader implications and limitations, and Section 8 concludes.
1
OSB is open source at https://github.com/OpenSecurityAI/benchmark.
2
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
2
Related Work
AI evaluation in security and data systems has expanded to measure models’ reasoning, operational reliability, and alignment with real-world investigative workflows. Three threads dominate: text-to-SQL benchmarks that assess data-centric reasoning; cybersecurity benchmarks spanning SOC, CTI, vulnerability, and accessgovernance reasoning; and methodological work on automated evaluation. Open Security Benchmark draws on all three. The Spider benchmarks are the canonical foundation for natural-language database interfaces. Spider 1.0 [24] introduced cross-domain text-to-SQL over a static schema-query paradigm, scored by exact set match and, later, execution accuracy. BIRD [25] moved toward large, “dirty” databases demanding external-knowledge grounding and value semantics, close to the messy-enterprise character OSB targets. Spider 2.0 [26] embeds models in enterprise environments such as BigQuery and Snowflake, requiring multi-step workflows, error revision, and reasoning over code-schema dependencies. Together these mark a transition from scoring a single SQL output to evaluating the robustness and repairability of agentic workflows, the transition OSB follows for defensive security investigation. ExCyTIn-Bench [18] rewards step-by-step investigation in a simulated SOC aligned with a threat investigation graph, while CyberSOCEval [34] grounds evaluation in authentic threat reports and sandbox outputs. For cyber threat intelligence, CTIBench [35] mixes expert-validated answers with CVE/CWE mappings, and SEvenLLM-Bench [36] adds a bilingual setting with hybrid semantic scoring. SecLLMHolmes [37] benchmarks vulnerability detection over labeled C/C++ and Python code. General-purpose suites add breadth: CyberBench [38] spans classification, summarization, NER, and QA; CyberMetric [39] measures certification-level knowledge; and CyberSecEval 2 [20] evaluates insecure code generation, prompt-injection susceptibility, and code-interpreter abuse. Identity and access reasoning, central to modern posture, gained attention through OrgAccess [19], which builds synthetic hierarchies, permission sets, and RBAC rules to test whether models apply access-governance policies. Where OrgAccess tests RBAC application within synthetic structures, OSB targets operational visibility in realistic environments: enumerating identities, identifying misconfigurations, and evaluating authentication posture over production-shaped IAM, identity-provider (IdP), and SaaS data, within a substrate that generalizes beyond identity to other posture subdomains. The two Sola ISPM benchmarks [29, 30], which OSB packages as its first packs, established the single-platform visibility and cross-vendor correlation task families on which we build. Two adjacent lines make OSB’s gap concrete. First, executable agent environments have appeared for offensive security: Cybench [21] and, at scale, NYU CTF Bench [22] run agents over capture-the-flag tasks, and vulnerable-by-design cloud labs ship shareable misconfigured environments [40]. These are shared, executable substrates, but they target attack rather than defensive posture assessment and carry no gold posture answers for end-to-end scoring. Second, configuration-as-SQL tooling such as Steampipe [41] and CloudQuery [42] already exposes cloud, IdP, and SaaS configuration state as queryable relational tables, direct evidence that posture-as-SQL is a natural, increasingly common representation. But these are query engines and human-run audit tools, not agent benchmarks: they provide neither a fixed task suite with gold answers nor a scoring methodology for comparing agents. What OSB provides is the missing piece: a shared, defensively oriented, cross-vendor relational environment paired with gold posture answers and a reproducible scoring methodology for end-to-end agent evaluation. A complementary line addresses automated-evaluation methodology. G-Eval [43] uses LLMs as evaluators with chain-of-thought form-filling prompts that align well with human judgment; RAGAS [44] introduces reference-free metrics for context relevance and answer faithfulness. These inform how OSB constructs scoring: deterministic checks where exact agreement is meaningful, rubric-locked LLM judges where exact matching breaks down, and expert calibration to keep both honest. OSB draws on this trajectory in three ways: it adopts text-to-SQL over a frozen relational snapshot as a unifying paradigm, extending Spider and BIRD to defensive workflows; it integrates deterministic structural metrics, rubric-locked LLM-judge panels, and expert calibration as distinct, attributable axes rather than one aggregate; and it treats public comparability and private tenant evaluation as core properties via a shared substrate and explicit trust boundary. Where each prior benchmark addresses a narrow capability slice, OSB is a framework into which packs spanning identity, configuration, vulnerability, and broader posture subdomains plug under a shared methodology.
3
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
3
Investigation Modalities
Posture review is investigative: correlating data across many sources into a holistic view of the enterprise’s security state, then reasoning over it to surface and substantiate a finding. This situational awareness is the foundation of autonomous enterprise cyber defense, and an agent doing this work must act on the environment through some interrogation surface. OSB instantiates two investigation modalities over the same environment: relational queries and native vendor interfaces. An agent is thus measured on the surface it would actually use, while the environment, tasks, and answer criteria stay shared. An agent can interrogate an enterprise environment along at least three surfaces. Relational queries (textto-SQL over a centralized store, where each pivoted entity becomes a table, each cross-vendor relationship a join key, and each posture question a workload over that schema) are the surface studied by the Spider lineage [24, 26]. API and tool calls against live vendor endpoints are exercised by tool-using agents [27] and measured by API- and function-calling benchmarks [45, 46]. Shell and command-line invocation of vendor CLIs in a sandbox is studied under agent–computer interfaces [28]. Each route reaches the same finding but defines “done the work” differently; OSB implements the relational surface and the native-vendor surface, the latter spanning both direct API calls and vendor command-line tools. The relational modality is faithful to the investigative activity: posture analysts correlate across sources (joining, pivoting, filtering), and a relational query is a direct, legible encoding of that correlation. It is also, decisively for a benchmark, the surface whose reproducible target is cheapest to obtain and check, because posture state is naturally a relational snapshot that freezes immutably and admits cheap setequivalence checking. Reproducibility comes from freezing or recording the target, not from SQL as such: API-Bank drives simulated API executors, Gorilla scores against a static API database, and SWE-bench-style harnesses run agents in frozen container images with deterministic tests [46, 45, 28]. Text-to-SQL is effective precisely when posture-relevant configuration and identity state, not security telemetry, is available as centralized, normalized, queryable relational data: configuration-as-SQL engines such as Steampipe [41] and CloudQuery [42] already present cross-vendor configuration state as relational tables, whereas the Open Cybersecurity Schema Framework [47] and managed security lakes [48] normalize telemetry rather than the inventory and entitlement state OSB queries. The hard, often unsolved part is cross-vendor identity resolution: reconciling that a person in an HR system, an identity provider, and several clouds is one identity, which these engines rarely expose as clean foreign keys. OSB therefore keeps it in the task rather than assuming it solved (§6). The native-vendor modality measures the work as an operator actually does it, for the many settings where no consolidating relational layer exists. The same environment snapshot is compiled into a running world that serves the enterprise over its vendors’ native API surfaces; the agent interrogates it with real vendor tooling: the aws CLI and ordinary HTTP calls to the identity provider, productivity suite, source host, and HR system, authenticating as an operator would. Its only documentation is the pinned, real vendor API specifications, and it never sees the underlying tables: it observes the environment solely through computed, deterministic, read-only API responses, so reproducibility follows from the frozen snapshot behind a faithful emulator rather than from freezing a query result. Every request the agent issues is recorded to a request log that plays the role the query trace plays for the relational modality-the evidence the scorer reads. The native-vendor modality trades cheap set-equivalence for fidelity to how the work is really done, and needs no pre-consolidated schema. Both modalities share the scoring layer, the environment, and the answer criteria; they differ only in the interrogation surface and the trace it produces. Text-to-SQL is standardized first because its verifiable target is cheapest, and the metrics that inspect the query artifact-SQL quality and the structural table and join checks (§4.3)-are specific to it, while answer correctness and reasoning quality apply to both. Neither surface is a ceiling: because scoring is defined independently of the surface, further modalities (a graph surface for transitive access paths, cloud-posture-audit checks) plug in under the same methodology (§7).
4
Framework Components
Open Security Benchmark comprises five components that together implement the commitment argued in Section 3. A data layer (§4.1) surfaces the enterprise environment as a frozen, shared snapshot. A task and evaluation-set layer (§4.2) defines security questions and their expected structured answers. A scoring layer (§4.3) reports each criterion separately. A harness layer (§4.4) provides a small, fixed agent action space that makes every run auditable and reproducible. A bring-your-own path (§4.5) lets researchers and practitioners
4
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
use the framework from one substrate without compromising one another’s needs. This section describes each component at the level of design intent; concrete instantiations appear in Sections 5 and 6. 4.1
Enterprise Environment Data
Posture review runs against the operational state of an enterprise: identities, resources, role and group memberships, configuration settings, vulnerabilities, and the access paths that connect them. OSB’s data layer exposes this state as a curated relational database, under the entity-to-table mapping of Section 3 and for the reasons argued there. Vendor coverage spans the layers where posture risk concentrates (cloud and infrastructure, identity providers, SaaS suites, application and dependency inventories, data stores); the present packs instantiate a subset. Within each vendor, the schema captures the entities and relationships that rule sets such as Scout Suite [14] and ScubaGoggles [16] interrogate, plus the artifacts vendor best-practice guidance [49, 50] treats as auditrelevant. Across vendors, the joinable columns on which multi-stack reasoning depends-shared emails and usernames, group and resource identifiers, ownership fields-are documented but deliberately not wired as declared foreign keys, so the agent must infer which columns correspond and cross-vendor identity resolution stays part of the task (§6). This read-only relational data is the shared foundation, without precluding other substrates (log streams, graph telemetry, simulated endpoints); transitive, path-based questions (nested group membership, privilege-escalation chains) are weakest in flat SQL, so a graph surface in the spirit of identity attack-path tooling [51] may later serve them better (§7). A trust boundary separates what the agent observes (the schema and its contents) from what only the evaluator observes (the gold reference, the expected result, and the seed data binding). Datasets are produced by a generator pipeline that cross-checks each question against the data before release, but that pipeline sits outside the benchmark: the shipped artifact is the frozen snapshot plus the evaluator-only reference, not an authoring trace. Because the same contract holds whatever organization the data describes, a benchmark authored against one environment ports to another without re-engineering the methodology. 4.2
Tasks and Evaluation Sets
A task pairs a natural-language security question with a SQL-derivable structured answer (the (D, q) formulation of §1). That answer is either a list of entities with the fields an operator consumes or a posture verdict with the configuration facts behind it; for an enumeration question, the answer is the set of matching subjects, empty when none qualify. Because the answer is specified independently of the query that derives it, equivalent queries receive the same score. On the evaluator side, each task carries a gold reference that produces the expected answer against the same data the agent sees, and the trust boundary keeps this reference in a separate visibility class, accessible only to the scorer. An evaluation set is a coherent collection of tasks scoped to a domain or capability, such as identity hygiene, configuration drift, or cross-vendor reasoning. The contract is designed around three visibility classes: an open class for public comparison and ablation, a gated class that withholds the reference artifacts to protect leaderboard integrity, and a private class that keeps tasks and evaluators inside a tenant’s boundary. Open comparison and private bring-your-own evaluation ship today; the gated leaderboard channel is on the roadmap (§7). The same task definition serves public comparison and internal review without duplication. 4.3
Evaluation Metrics
OSB scores every run using evaluation criteria adopted from the cross-vendor ISPM benchmark [30], reusing its Appendix B rubric texts verbatim rather than re-deriving them. Scoring proceeds in a four-stage pipeline: execute every question, collect a per-question evidence bundle, apply a rubric-guided LLM-as-judge panel, and run a deterministic structural check. Judged metrics use a three-level scale (0, 0.5, 1), except the binary answer-verdict check. The judge design follows work on LLM-as-judge evaluation [43] and retrievalaugmented evaluation [44], applying rubric-locked panels with explicit thresholds rather than free-form scoring. The deterministic structural metrics add a verification layer immune to the semantic ambiguity of purely LLM-based judgment. The framework avoids collapsing these criteria into a single aggregate: one number obscures the source of failure, whereas separate metrics let teams attribute performance shifts to specific behaviors and read a scorecard in the language of the property they care about. Answer correctness is the primary measure, graded against the answer fields of the task’s ground-truth atom (§5): a verdict, a closed set of matching 5
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
Table 1: OSB’s evaluation criteria for posture investigation, with rubric texts adopted from the cross-vendor ISPM benchmark [30]. Family
Criterion
Scale
Type
Description
Answer quality
AnswerCorrectnessVsGT
0/.5/1
judged
Answer agrees with the groundtruth verdict, subjects, and facts. Primary yes/no verdict matches ground truth.
AnswerVerdictCorrectnessVsGT0/1 Reasoning ReasoningUtility
judged
0/.5/1
judged
Reasoning trace makes plausible progress toward the answer.
SQL qual- SQLSemanticAppropriateness 0/.5/1 ity
judged
SQL is a sensible, schemaconsistent strategy for the question. Share of the required tables/joins the agent used. Precision and F1 over tables/joins.
Structural
Tables/Joins Recall
[0,1]
det.
Tables/Joins Prec., F1
[0,1]
det.
subjects, and the determining configuration facts. Because the agent answers in free-form prose, it is scored by a rubric-locked judge that maps the answer onto that reference, absorbing benign variation such as how subjects are named or how a count is reported (“about 12” versus an exact 12) rather than requiring naive string equality. The other judged criteria assess the yes/no verdict, the utility of the reasoning trace, and the schema-consistency of the generated SQL, while the deterministic structural metrics measure whether the agent constructed the required relational path, the tables and joins the finding needs. All are compared against the ground-truth atom or its minimal-sufficient evidence set (the smallest set of tables and joins that suffices to derive the answer) rather than a particular query plan, so scores are plan-independent (§4.2); the criteria are complementary but not statistically independent: wrong tables tend to produce wrong answers, so we report the relationships among them rather than assume orthogonality. The answer-correctness, verdict, and reasoning criteria apply to both investigation modalities; the SQL-quality criterion and the structural table and join metrics are specific to the text-to-SQL modality, since they inspect the query artifact. In the native-API modality the same structural role is played by whether the agent’s recorded request trace reaches the endpoints and objects the finding requires (§3). Beyond the per-metric scores, OSB breaks every metric down within a run across the task taxonomy (complexity, focus, platform, and single- versus cross-platform scope; §6), and across environment variants (organizational scale today, and the broader matrix of vendor stack, schema variant, and noise as the catalog grows; §5) by comparing runs. Generalization and robustness are thus read from the same criteria across configurations rather than measured separately: an agent that exploits one configuration is distinguished from one that holds up across the matrix. This breakdown is a reporting practice, not an additional criterion. Scores come from a panel of two judges from different providers, paired to reduce the correlated and selfpreferring biases LLM judges are known to exhibit [52, 53]. Each task is run for several independent traces; every trace is scored by every judge; and a metric’s score is the mode of the pooled votes (majority voting over samples in the spirit of self-consistency [54]), ties broken toward the lower grade. Because frontier judges drift, scores are comparable only within a fixed judge configuration, and open-weights judges may be substituted for reproducibility or to keep a private tenant’s data in-boundary (§4.5). To expose scorer noise rather than hide it in a point estimate, every metric is reported with a bootstrap 95% confidence interval (resampling over tasks) and inter-judge agreement (Cohen’s κ [55]). Finally, the framework treats expert calibration as its anchor: automated scores are to be validated against expert-labeled subsets, with their agreement reported so scorer drift stays visible. 4.4
The Harness
The harness is the execution substrate that runs an agent against a task and produces the artifacts on which every metric is computed. A single principle governs its design: every action an agent takes must be observable and reproducible from the recorded run alone. This rules out hidden state, opaque tool calls, and free-form interaction patterns that cannot be replayed. 6
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
The agent acts through two tools. get-schema returns the schema documentation for the whole environment in a single view: every table with its columns, row counts, and a short note. It exposes column names but neither types nor foreign keys, so the agent must infer every join, within and across vendors, from the columns themselves (§4.1). run-query executes a single SQL statement and returns its result. Each task runs against a private, disposable copy of the snapshot, so although any statement is permitted, writes touch only that copy and the shared environment is never mutated: posture review is non-mutating by nature, and copy-isolation gives that guarantee without constraining the agent’s SQL. The surface is small enough that agent behavior maps cleanly onto measurable events: query iteration is the trajectory of exploratory and candidate queries, and answer verification is the presence and structure of a validating query. Beyond the tool surface, each run produces an evidence archive: an ordered trace of every model step, tool call, and result, written to disk (§7). The agent’s reasoning is interleaved with this trace as the text between its tool calls, and a reasoning journal is recovered from it for scoring. Because reasoning and actions share one ordered record, any claim a scorer makes is checkable against it: a reasoning-utility metric that flags an unsupported assertion can point to the surrounding tool calls, and a structural metric that detects a wrong join can point to the query in which the join appears. The harness is the only component the framework standardizes at the implementation level; the rest of the agent runtime (model serving, prompting strategy, orchestration loop) is the experimental variable. The framework already runs several agent runtimes over this one tool interface, and any model can be placed behind it, so cross-stack comparability follows from pinning what every agent must produce rather than how it is built.
4.5
Bring Your Own Agent or Model
The components above compose into one path for both researchers and practitioners. A researcher evaluating an agent or model points one of the framework’s supported runtimes at a published pack, runs it under the standard harness, and receives a scorecard across all criteria together with the full evidence archive. The pack is version-pinned and its evaluator artifacts are gated, so scorecards are comparable across runs on the same pack. An ablation is a re-run against that pinned pack with a single variable changed, whether the model, the prompt strategy, or the context configuration, and the resulting metric movements are attributable to it. Comparability comes from pinning the artifacts those choices must produce, not from enforcing a specific model or runtime. A practitioner evaluating an agent against a specific tenant follows a structurally identical path, with the same harness, contract, and criteria, but supplies a private pack: tenant-specific data and the success definitions (the expected answers) for the benchmark’s questions. Because the judged metrics send the agent’s evidence bundle to an LLM panel, a private pack can run that panel on self-hosted or open-weights judges inside the tenant boundary, so confidential posture data never leaves it, while the deterministic metrics require no external call at all. The metric layer accepts additional rubric-defined criteria for such packs (procedure alignment, control coverage, internal risk definitions), and the trust boundary keeps any private evaluator’s outputs out of public comparison, so private evaluation can be as opinionated as a tenant needs without affecting the public surface.
5
Environment Datasets
The data catalog closes the environment data gap with environment datasets of synthetic organizations: each a fully fictional enterprise with no real individuals, organizations, or credentials, every identifier (names, emails, logins, keys) fabricated on .example domains. Although every value is fabricated, the environments are synthesized from real ones: each vendor’s tables and columns follow that product’s real data model, and their data patterns (identity lifecycles, group and privilege structures, common misconfigurations, credential ages, and access distributions) mirror those seen in production enterprises. Tasks therefore exercise realistic posture reasoning while the snapshot exposes no real identity, credential, or organization. The datasets will be released under the Open Security AI organization on the Hugging Face Hub,2 each an immutable, content-addressed revision that pins exactly the bytes a run was reviewed against; the initial catalog is in preparation, and the counts below describe the reference snapshot.
2
https://huggingface.co/OpenSecurityAI
7
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
Table 2: Illustrative slice of the relational encoding. Vendor source
Example table
Representative columns
aws_iam_user
name, mfa_enabled, password_enabled, password_last_used access_key_id, user_name, status, create_date, last_used_date minimum_password_length, max_password_age, password_reuse_prevention
AWS aws_iam_access_key aws_iam_account_ password_policy okta_user
Okta
login, email, status, last_login, department name, status, authenticators
okta_mfa_policy Google Workspace
googleworkspace_ shared_drive_file
name, owner_email, permissions, trashed
HRIS (HiBob)
hibob_employee
email, fullname, status, lifecyclestatus, terminationdate
AWS Identity Center
aws_identitystore_user
id, name, display_name
Table 3: The reference environment at three organizational scales.
5.1
Scale
Employees
Total rows
Small Mid Large
∼75 ∼400 ∼2,000
∼1,600 ∼8,300 ∼36,000
Relational encoding for text-to-SQL
Each environment is a single read-only relational snapshot of an organization’s identity-relevant state across eight vendor-style sources-AWS (IAM and Identity Center), Azure Active Directory, Google Cloud Platform, GitHub, Google Workspace (with Drive), a human-resources information system, MongoDB Atlas, and Okta-comprising 44 relational tables in the reference release. Every vendor entity and its configuration lives in a table (Table 2); vendors are linked by joinable columns, shared identifiers documented in the schema but, per §4.1, deliberately not declared as foreign keys, so cross-vendor identity resolution remains part of the task. A shared email, for instance, links hibob_employee and okta_user to the same person in aws_identitystore_user, so a question like “terminated employees who still have cloud access” becomes a join over those tables filtered by hibob_employee.status. The two packs of §6 contribute 127 tasks, realized at every scale. A single environment cannot test generalization, so the reference organization is realized at three scales (small, mid, and large; Table 3), sharing task structure but spanning more than an order of magnitude in size, from ∼75 people and ∼1,600 rows to ∼2,000 people and ∼36,000 rows over the same 44 tables (roughly a 5x step between consecutive scales). This instantiates the scale axis of the environment matrix, so generalization and robustness (§4.3) are reported as coverage across scale rather than one number, distinguishing a scale-robust agent from one that exploits a small configuration’s idiosyncrasies. The broader matrix (vendor stack, schema variant, noise profile) is built into the data-layer contract as the natural direction of growth. Each dataset is a self-contained bundle: a manifest with the identity header, the SQLite snapshot, and a per-task ground-truth file. That file is a ground-truth atom: the answer fields (a verdict, a closed set of matching subjects, and the determining configuration facts) plus the tables and joins the finding touches. Time-relative questions (“inactive for 90 or more days,” “left the company but still has access”) anchor to a single publish date in the manifest, so look-back semantics belong to the data, not wall-clock time. Because each revision is immutable and self-contained, sharing no rows or answers with its predecessor, an environment can be rotated: regenerated and republished. Rotation is the lever against answer-level memorization, since memorizing one environment’s rows and answers yields no advantage on its successor. It does not defend against overfitting to the fixed public schema, task set, and solution strategies, which persist across rotations and scales; held-out task splits and a held-out environment generator (future work, §7) are the guard for that stronger notion. 8
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
5.2
The same environment, served natively
The relational snapshot is not the only way an agent meets this environment. For the native-API modality (§3), the same bundle-the manifest plus the identical SQLite snapshot-is compiled into a self-contained, runnable world that serves the enterprise over its vendors’ native API surfaces: the AWS control plane reached through the aws CLI, and the identity provider, productivity suite, source host, and HR system reached over their real HTTP APIs. Responses are computed from the same rows the SQL modality queries, so the two modalities share one ground truth; but the agent sees them only as vendor API results, authenticates with vendor-native credentials, and reads the pinned real vendor API specifications as its sole documentation, never seeing the tables. A subset of the eight sources is served this way today, and one content-addressed revision underlies both presentations, so a finding is defined once and can be posed either as a query workload or as a sequence of native API calls.
6
Use-Case Packs
A pack is the unit in which OSB is instantiated and extended: a data domain, an evaluation set, and the scoring configuration that binds them. The framework ships today with two identity-security packs, derived from the two public Sola ISPM benchmarks, over the environment datasets of Section 5. Together they comprise 127 tasks-77 in the visibility pack and 50 in the cross-vendor pack-each realized at all three environment scales. The visibility pack, derived from the single-platform Sola Visibility ISPM benchmark [29], evaluates foundational identity inventory and configuration-hygiene questions scoped to one vendor at a time. Representative questions include “Which active access keys are older than 90 days?” over AWS and “Which applications have mfa_required set to false or null?” over Okta (a synthetic-schema simplification; production Okta enforces multi-factor requirements through sign-on policies rather than a per-application flag). These are the questions a posture analyst asks first (who and what exists, and whether baseline hygiene controls are in place), and they establish whether an agent can enumerate identities and read configuration state correctly before any cross-vendor reasoning. The cross-vendor pack, derived from the Cross-Vendor Sola ISPM benchmark [30], evaluates federated correlation across vendor boundaries, where no shared schema or explicit foreign key connects the sources and the agent must reconstruct identity resolution paths itself. Representative questions include “Which terminated employees still have AWS single sign-on access?”, which correlates a human-resources record against cloud access, and “Which publicly shared Google Workspace files are owned by Okta users who have been deactivated?”, which joins an identity provider against a SaaS suite’s sharing state to surface an exposure set. These questions exercise the correlation that makes identity posture hard and that no single-vendor tool can answer. Both packs share a lean task taxonomy that drives every reported breakdown: complexity (easy, medium, hard); focus (authentication, authorization, and administration, the last covering lifecycle and hygiene such as offboarding and dormant credentials); and the concrete platforms a task’s ground truth touches. From the platforms, a single- versus cross-platform scope is derived. Reporting each metric across these dimensions turns a single score into a diagnosis: an agent may enumerate single-platform inventory well yet fail cross-platform correlation, and the taxonomy makes that visible. The pack is also the unit of community contribution: a new posture subdomain is a new domain and evaluation set composed with the existing scoring methodology, so the path from these two identity packs to packs for cloud configuration, vulnerability state, or data exposure is an authoring task, not a framework redesign.
7
Discussion
OSB is not a scoreboard-it is an engine that can both measure an agent and advance it toward autonomous enterprise cyber defense, and the mechanism for the second role is already present in the first. Every run writes an evidence archive to disk (§4.4): the agent’s actions through the get-schema and run-query tools, the intermediate results those queries returned, the reasoning interleaved between them, and the final answer, together with the scorecard derived from that record. The archive is browsable through the framework’s run viewer and stamped with the binary and dataset revision it was produced against, so a run can be reopened and read against the exact bytes it saw. Three caveats qualify it: the prompt is reconstructible from the pinned pack inputs rather than stored verbatim, a stored run can be reviewed again but not yet automatically re-scored, and very large tool outputs are truncated in the record. These are the same artifacts a training 9
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
loop consumes. OSB implements no training loop, but the pipeline that scores an agent today produces the versioned, contract-conformant, replayable data one would use to train its successor. One potential utility of OSB is therefore to advance agents, not only to score them: what makes the archive trainable rather than merely readable is that OSB already computes the signal a learner needs. An outcome the framework can check automatically-a matched subject set, a correct verdict-is a verifiable reward, the signal behind recent reasoning models [56, 57]. Where the outcome is partial-credit or rubric-graded rather than exactly checkable, a reward model learned from preferences or feedback supplies the same signal [58, 59]. The logged trajectories are the input distribution for offline reinforcement learning [60] and for imitation learning from the successful traces [61]. Filtering the runs OSB has already verified as correct and fine-tuning an agent on them closes the loop as bootstrapped self-improvement [62, 63], with OSB’s metric outputs entering these procedures as the scalar or vector reward. None of it is a separate system; it reuses the scorecard the evaluator already emits. Our work is nonetheless limited by its reliance on synthetic environments: they are shareable, reproducible, and privacy-safe, and because each is synthesized from real product data models and the behavioral patterns of production enterprises (identity lifecycles, group and privilege structures, common misconfigurations), the posture reasoning they demand carries over to real deployments; what a fabricated organization does not reproduce is a live tenant’s operational drift and incident telemetry. Future work includes growing the catalog and extending the surface. Because a shared benchmark advances a field only while it resists contamination [64, 65, 66], we will publish more synthetic organizations and use-case packs, broadening coverage across schemas, vendor stacks, and enterprise archetypes while hardening the target as current packs saturate [65]. The scoring core already spans text-to-SQL and the native-vendor modality, and further surfaces plug into it the same way: cloud-posture audits after CIS [12], Prowler [15], and Scout Suite [14], a graph surface for the transitive access-path questions that flat SQL handles awkwardly [51], and new packs for configuration, vulnerability, and data-exposure posture. Above all, because autonomy is graded rather than binary [67], the read-only assessment OSB scores today-grounded in situation awareness [3]extends up the ladder toward risk prioritization, remediation, and verification, the goal of autonomous cyber defense [1, 2], with the evaluation-to-training loop above carrying an agent from each rung to the next.
8
Conclusion
We introduced Open Security Benchmark, a framework for evaluating agentic AI on security posture tasks. OSB addresses the environment data gap by surfacing curated, synthetic-organization environments as a shared snapshot of the security state, and evaluating posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor’s native API over a served instance of the same environment. The design freezes the target once and grades answers against a known-correct reference. Around this commitment it assembles the five components of Section 4, from the data layer through the bring-your-own path, with a scoring layer that reports each criterion on its own axis. We instantiated the framework with two identity-security packs and a family of environment datasets spanning multiple scales, and charted its extension along the surface, subdomain, and automation axes. Our aim is not a single leaderboard but a shared substrate: one on which the community can author packs, compare agents on the work they are deployed to do, and turn every evaluation into data that advances the next agent. On such a substrate, trusting a posture agent becomes a matter of measurement rather than judgment, a foundation for extending trustworthy agentic AI to the situational-awareness core of autonomous enterprise cyber defense.
References [1] Sanyam Vyas, John Hannay, Andrew Bolton, and Pete Burnap. Automated cyber defence: A review. arXiv preprint arXiv:2303.04926, 2023. [2] Maxwell Standen, Martin Lucas, David Bowman, Toby J. Richer, Junae Kim, and Damian Marriott. Cyborg: A gym for the development of autonomous cyber agents. In IJCAI-21 1st International Workshop on Adaptive Cyber Defense, 2021. arXiv:2108.09118. [3] Mica R. Endsley. Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1):32–64, 1995. [4] Ulrik Franke and Joel Brynielsson. Cyber situational awareness - a systematic review of the literature. Computers & Security, 46:18–31, 2014. [5] RSA Security. Defining identity security posture management (ispm). https://www.rsa.com/ resources/reports/defining-identity-security-posture-management-ispm/, 2025. 10
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
[6] Cloud Security Alliance. Cloud controls matrix (ccm) v4. https://cloudsecurityalliance.org/ research/cloud-controls-matrix, 2021. [7] National Institute of Standards and Technology. Guide for mapping types of information and information systems to security categories. Technical Report NIST Special Publication 800-60 Vol. 1, Rev. 1, National Institute of Standards and Technology, 2008. [8] OWASP Foundation. OWASP top 10:2021 - the ten most critical web application security risks. https://owasp.org/Top10/, 2021. [9] National Institute of Standards and Technology. Cybersecurity supply chain risk management practices for systems and organizations. Technical Report NIST Special Publication 800-161 Rev. 1, National Institute of Standards and Technology, 2022. [10] National Institute of Standards and Technology. The NIST cybersecurity framework (CSF) 2.0. Technical Report NIST CSWP 29, National Institute of Standards and Technology, 2024. [11] Joint Task Force. Security and privacy controls for information systems and organizations. Technical Report NIST Special Publication 800-53, Revision 5, National Institute of Standards and Technology, 2020. [12] Center for Internet Security. Cis critical security controls, version 8. https://www.cisecurity.org/ controls/v8, 2021. [13] International Organization for Standardization. Iso/iec 27001:2022 - information security, cybersecurity and privacy protection - information security management systems - requirements. https://www.iso. org/standard/27001, 2022. [14] NCC Group. Scout suite: Multi-cloud security auditing tool. ScoutSuite, 2024.
https://github.com/nccgroup/
[15] Prowler. Prowler: Open source cloud security tool for aws, azure, gcp and kubernetes. https: //github.com/prowler-cloud/prowler, 2024. [16] Cybersecurity and Infrastructure Security Agency (CISA). Scubagoggles: Google workspace secure configuration baseline assessment tool. https://github.com/cisagov/ScubaGoggles, 2024. [17] Google Cloud. 2025 roi of ai in security report. https://cloud.google.com/resources/content/ roi-of-ai-security, 2025. [18] Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K. Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, and Anand Mudgerikar. Excytin-bench: Evaluating llm agents on cyber threat investigation. arXiv preprint arXiv:2507.14201, 2025. [19] Debdeep Sanyal, Umakanta Maharana, Yash Sinha, Hong Ming Tan, Shirish Karande, Mohan Kankanhalli, and Murari Mandal. Orgaccess: A benchmark for role based access control in organization scale llms. arXiv preprint arXiv:2505.19165, 2025. [20] Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161, 2024. [21] Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, and Percy Liang. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In International Conference on Learning Representations (ICLR), 2025. [22] Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. arXiv preprint arXiv:2406.05590, 2024. [23] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1533–1544, 2013. 11
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
[24] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. [25] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.-C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [26] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763, 2024. [27] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [28] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [29] Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, Reuven Elezra, Idan Pinto, Tal Moalem, Shmuel Cohen, and Yoni Weintrob. Sola-visibility-ispm: Benchmarking agentic ai for identity security posture management visibility. arXiv preprint arXiv:2601.07880, 2026. [30] Eden Yavin, Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, and Gal Baron. Cross-vendor sola ispm benchmark: Evaluating agentic ai for federated identity security reasoning. arXiv preprint arXiv:2606.02674, 2026. [31] National Institute of Standards and Technology. Guide for conducting risk assessments. Technical Report NIST Special Publication 800-30 Rev. 1, National Institute of Standards and Technology, 2012. [32] National Institute of Standards and Technology. Guide to enterprise patch management planning: Preventive maintenance for technology. Technical Report NIST Special Publication 800-40 Rev. 4, National Institute of Standards and Technology, 2022. [33] National Institute of Standards and Technology. Information security continuous monitoring (iscm) for federal information systems and organizations. Technical Report NIST Special Publication 800-137, National Institute of Standards and Technology, 2011. [34] Lauren Deason, Adam Bali, Ciprian Bejean, Diana Bolocan, James Crnkovich, Ioana Croitoru, Krishna Durai, Chase Midler, Calin Miron, David Molnar, Brad Moon, Bruno Ostarcevic, Alberto Peltea, Matt Rosenberg, Catalin Sandu, Arthur Saputkin, Sagar Shah, Daniel Stan, Ernest Szocs, Shengye Wan, Spencer Whitman, Sven Krasser, and Joshua Saxe. Cybersoceval: Benchmarking llms capabilities for malware analysis and threat intelligence reasoning. arXiv preprint arXiv:2509.20166, 2025. [35] Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence. arXiv preprint arXiv:2406.07599, 2024. [36] Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, Changyu Ren, and Zhoujun Li. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence. arXiv preprint arXiv:2405.03446, 2024. [37] Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Privacy (S&P), 2024. [38] Zefang Liu, Jialei Shi, and John F. Buford. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. In AAAI-24 Workshop on Artificial Intelligence for Cyber Security (AICS), 2024. [39] Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. Cybermetric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. arXiv preprint arXiv:2402.07688, 2024. [40] Rhino Security Labs. Cloudgoat: Rhino security labs’ “vulnerable by design” aws deployment tool. https://github.com/RhinoSecurityLabs/cloudgoat, 2024. 12
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
[41] Turbot. Steampipe: Zero-etl sql for cloud, saas, and infrastructure apis. https://steampipe.io, 2024. [42] CloudQuery. Cloudquery: Open-source cloud asset inventory. https://www.cloudquery.io, 2024. [43] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [44] Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023. [45] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. [46] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [47] OCSF Project. Open cybersecurity schema framework (ocsf). https://ocsf.io, 2024. [48] Amazon Web Services. Amazon security lake. https://aws.amazon.com/security-lake/, 2024. [49] Okta. Okta security best practices. https://help.okta.com/, 2024. [50] Google. Security checklist for medium and large businesses (100+ users). https://support.google. com/a/answer/7587183, 2024. [51] SpecterOps. Bloodhound: Active directory and azure attack-path management. https://github.com/ SpecterOps/BloodHound, 2024. [52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [53] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [54] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023. [55] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. [56] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645:633–638, 2025. [57] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. [58] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2203.02155. [59] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1706.03741. [60] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. [61] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627–635, 2011. arXiv:1011.0686. [62] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2203.14465. 13
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
[63] Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kiciman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: Llms can reward and refine their own reasoning for open-ended tasks. arXiv preprint arXiv:2506.13351v1, 2025. [64] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. [65] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1905.00537. [66] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. [67] Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 30(3):286–297, 2000.
14