PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems Suraj Ranganath Halıcıoğlu School of Data Science and Computing University of California, San Diego

Abstract

arXiv:2606.08481v1 [cs.LG] 7 Jun 2026

Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant Text2Cypher benchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a local benchmark-generation pipeline that turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combines schema profiling, reverse-query grounding, constrained generation, deterministic Cypher governance, execution validation, redaction, diversity controls, and a calibrated local LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative: zero-shot transfer is weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads.

Introduction

Property graphs are attractive in enterprise settings because the facts of interest are often relational paths: account transfers, identity entitlements, access chains, customer interactions, supplier dependencies, and fraud rings. Cypher gives analysts a compact language for these patterns. As LLMs be-

Anish Raghavendra Independent Researcher

come natural-language interfaces for graph analytics, an organization cannot evaluate a Text2Cypher system only on public schemas; it needs to know whether the model handles its labels, relationship directions, values, governance rules, and recurring operational questions. For evaluation, that privacy boundary matters as much as model accuracy. A static public dataset is useful for shared comparison, but it cannot contain a bank’s account taxonomy, an identity team’s permission graph, or the categorical values that make a query answerable. It also cannot change when the production schema changes. Industry teams therefore need something different: a repeatable way to turn a live property graph into a balanced, executable, privacy-aware NL-to-Cypher benchmark without sending sensitive schema or values to paid generation APIs. At first glance, benchmark generation looks like a natural job for a general AI agent: inspect the schema, draft Cypher, run it, fix mistakes, and write questions. That can work for a small oneoff study when a strong model has the right tools, prompts, and examples. It is a weaker recipe for enterprise refreshes. Graphs change as products ship, integrations are added, and analysts ask new questions; a benchmark factory has to run again at scale. Many deployments also prefer efficient local models, including quantized variants that run quickly on less hardware, because cost, latency, privacy, and availability matter. These models can often write useful individual queries, but they are less reliable at managing the full loop: grounding real values, preserving relationship directions, avoiding unsafe clauses, balancing categories and difficulty, rejecting ambiguous examples, and leaving evidence for review. We therefore make benchmark generation a constrained pipeline: the model handles language and Cypher generation, while deterministic graph checks make the process repeatable, scalable, and auditable.

Code and artifacts: https://github.com/suraj-ranganath/PIPE-Cypher/ Correspondence: [email protected]

PIPE-Cypher profiles a target graph, finds graph values that make candidate questions answerable, constrains a local generator, validates and repairs the generated Cypher, executes it, and then asks a local LLM judge to review only candidates that already have execution evidence. We treat Cypher correctness as something to check, not something to hope the prompt induces: relationship direction, read-only safety, exact literal use, categorical values, contextual return columns, and RETURN DISTINCT are enforced before examples are accepted. Contributions. We make four contributions: (1) a local-model workflow for generating private NLto-Cypher benchmarks from an organization’s own graph; (2) outcome-aware reverse grounding and Cypher-specific validators for read-only safety, relationship direction, exact literals, categorical values, contextual returns, and conservative rewrites; (3) a scaled public-proxy evaluation over FinBench, SNB, and ICIJ with ablations, judge calibration, redaction audits, and an 11-model local transfer study; and (4) reproducibility artifacts for onboarding, value sampling, benchmark refresh, evidence packaging, and appendix-level audit.

Related Work

Recent Text2Cypher resources, including Mind the Query (Chauhan et al., 2025), SyntheT2C (Zhong et al., 2025), Auto-Cypher (Tiwari et al., 2025), Text2Cypher (Ozsoy et al., 2025), CypherBench (Feng et al., 2025), and the public Text2Cypher-2024 corpus (Neo4j, 2024), show that good Cypher data needs schema grounding, execution checks, verification, and complexityaware evaluation. PIPE-RDF (Ranganath, 2026) makes a related benchmark-factory argument for RDF/SPARQL, using reverse querying, categorybalanced generation, retrieval, deduplication, execution validation, and deployment metrics. For enterprise Cypher benchmark generation, PIPECypher adds outcome-aware reverse grounding before natural-language realization, deterministic property-graph governance before export, local judge calibration after execution, explicit privacy and value policies, and provenance-rich refresh artifacts for organizations that need to benchmark their own graphs. Mind the Query is especially relevant because it reports an Industry Track Text2Cypher dataset with schema, runtime, value, and human logical

validation. PIPE-Cypher keeps that validation discipline but changes the object of study. We are not proposing one more static dataset or tuning corpus. We are proposing the process an organization would use to generate, refresh, audit, and redact a benchmark for its own graph, local model endpoint, and value policy. Text-to-SQL benchmarks such as Spider 2.0 (Lei et al., 2024) and BIRD (Li et al., 2023) have pushed text-to-query evaluation toward realistic database tasks and execution-based scoring, while execution-guided decoding shows why query execution is useful semantic feedback rather than a cosmetic check (Wang et al., 2018). Recent residual-skill Text-to-SQL work further shows that optimizing complementary agent skills on residual failures can improve selected accuracy across SQL dialects (Zhu et al., 2026). Graph query generation adds different failure modes: relationship direction can invert the meaning of a query, node and relationship properties live in different namespaces, and path-shaped questions can be syntactically valid while semantically ungrounded. CIKM AutoQuery (Zheng et al., 2024) motivates treating workload generation as a separate object of study from downstream model quality. LDBC FinBench (Qi et al., 2023) and SNB (Püroja et al., 2023) provide public graph workloads with financial and socialnetwork structure, while ICIJ Offshore Leaks gives an additional public finance/compliance onboarding check. For benchmark quality, lexical diversity alone is not enough. A question bank can use varied wording and still overuse the same graph values or the same Cypher template. PIPE-Cypher therefore combines text-generation diagnostics such as Distinct-n (Li et al., 2016) and self-BLEU-style redundancy checks (Zhu et al., 2018) with text-toquery structure metrics: schema coverage, relationship/property coverage, query-signature diversity, structural feature rates, and normalized entropy over graph/category/difficulty cells. For subset construction, we use an MMR-style novelty objective (Carbonell and Goldstein, 1998) over Cypher signatures, template families, structural substructures, schema atoms, values, and question tokens.

Method

PIPE-Cypher has six stages: schema profiling, workload planning, reverse Cypher grounding, constrained generation and repair, deterministic vali-

dation and execution, and LLM-judge review. The central design choice is simple: accepted examples should prove that they are answerable and safe. Prompts can ask a model to respect relationship directions or exact literals, but accepted examples must pass schema checks, parser-style structure extraction, live execution, and judge review. Schema profiling records labels, relationship types, properties, observed directions, and bounded low-cardinality categorical values. Workload planning targets eight categories that appear in operational graph analytics: simple retrieval, complex retrieval, simple aggregation, complex aggregation, boolean existence, negation/difference, path/temporal transaction, and ranking/top-k. Reverse grounding then runs read-only Cypher to find slot values that actually produce rows. This step avoids a common synthetic-data failure: a plausible-looking question that has no answer in the graph. The deterministic layer checks read-only safety, syntax shape, labels and properties from the schema, explicit relationship types, observed relationship directions, schema-provided categorical values, and non-empty execution where required. Live execution uses read-only credentials and readaccess sessions; token-level write rejection is only the first safety check. A lightweight Cypher analyzer extracts return aliases, variables, labels, relationship patterns, risky constructs, and rewrite skip reasons. This makes normalization auditable instead of a silent string edit. The direction gate reads both outgoing and incoming arrow syntax before schema checking, and rejects undirected relationship patterns when direction is required.

Implementation

PIPE-Cypher is a Python package built around a read-only graph client, a schema/value profiler, local model endpoints, Cypher governance, and export/audit tools. Before any model call, the profiler records the schema, relationship directions, and value samples the run may use. Generation, validation, judging, and export use that profile and execution traces rather than backend-specific objects. We use Neo4j for experiments, but only the graph client is backend-specific; the method targets Cypher over property graphs. Benchmark generation and judge review use a local Qwen3.5-9B endpoint (Qwen, 2026) behind a vLLM/OpenAI-compatible interface. Downstream evaluation uses 11 completed

locally served checkpoints from general instruction, code-tuned, Cypher-tuned, and Text2Cypher-tuned families. We keep these roles separate: one endpoint builds the benchmark, and the downstream study measures how other local models behave on the exported examples. All generation and evaluation stay inside the organization’s compute boundary without paid generation APIs. The graph profile makes graph-specific assumptions explicit. FinBench uses the public datagen snapshot export with typed node and relationship properties. SNB uses the official Neo4j/Cypher headers and read-query files. Live inspection also records bounded low-cardinality strings as categorical constraints. During FinBench import, we create rather than merge transaction relationships so repeated account-to-account events remain visible to path and aggregation queries. These choices determine answerability: a property may belong to a relationship rather than a node, a value may be unsafe to sample, and a relationship may only make sense in one direction. A company-owned onboarding run creates the same profile before the first LLM call. For generation, PIPE-Cypher starts from workload templates whose slots are filled by reversebinding queries. It can also use a mixed mode in which the LLM proposes additional templates after seeing proven workload seeds. Every run records both accepted and rejected candidates. Retrieved few-shot examples replace graph-specific values with typed placeholders, so the model sees the query structure without repeatedly seeing the same tenant values. A lightweight value grounder adds typed annotations for categorical values and reverse-bound entities, including punctuation variants, possessives, plurals, synonyms, name partials, and small typos. Inspired by Mind the Query’s prompt-setting analysis, PIPE-Cypher exposes prompt profiles for schema-only, instructions-only, examples-only, examples-plus-instructions, and full governed generation. We report a profile only when it passes the same target-size and evidence checks as the main run. For LLM-judge review, we do not send the entire schema when a query touches only a small part of it. The judge prompt includes the labels, relationship types, and properties mentioned by the candidate query, while deterministic validators still check against the full schema. This keeps local 9B prompts manageable on larger graphs without

Enterprise inputs • private property graph • read-only Cypher endpoint • domain terminology • local model endpoint

1. Schema and value profiling

2. Workload planning

Artifacts • labels / rel types • properties • directions • categorical values • schema fingerprint

Controls • graph targets • category targets • difficulty buckets • diversity caps • attempt budget

Repair and rewrite • add RETURN DISTINCT • normalize projections • reject risky CALL, UNION, writes, reserved variables, or parser-risky cases

Retrieval memory accepted examples with tenant values replaced by typed placeholders; nearest examples feed few-shot prompts without leaking raw values

3. Reverse grounding

4. Constrained generation

5. Cypher governance

6. Execution validation

7. Judge, audit, and export

Grounding • reverse bindings • entity/value slots • exact literal policy • placeholder examples

Local model • Qwen3.5-9B endpoint • schema-only prompts • Cypher constraints • local generation only

Deterministic gates • read-only safety • parser/AST features • schema vocabulary • direction/value checks

Live Cypher run • syntax/runtime checks • result samples • non-empty gate • empty-result diagnosis

Quality review • relevant schema + result sample • local LLM-judge JSON • ambiguity/alignment • human-audit • calibration

Top-up and refresh controller • rebalances missing cells • graph/category/difficulty • estimates capacity • starts scaled runs • from logged gaps

Benchmark package • NL-Cypher JSONL • train/dev/test splits • result samples • difficulty features • benchmark card • refresh manifest • redacted review export

Rejected-candidate ledger • validation issues • judge failures • duplicate/diversity blocks • repair decisions • empty-result classes • prompt profile / run ID

Figure 1: PIPE-Cypher benchmark generation pipeline. The key industry additions beyond static Text2Cypher dataset construction are privacy/value policies, reverse grounding, Cypher governance, execution diagnostics, local judge calibration, and benchmark export/refresh. Gate

Evidence checked

Failure mode caught

Read-only safety Schema validity

blocked Cypher write/admin tokens labels, relationship types, properties, categorical values observed relationship directions structure extraction, rewrite safety, RETURN DISTINCT quoted values use exact literals query runs and returns rows ambiguity, semantic alignment, schema use

destructive or operational queries hallucinated graph vocabulary or values

Direction validity Cypher analysis and normalization Question constraints Execution LLM judge

reversed or invalid traversals unsafe rewrites or duplicate/noisy rows fuzzy matching of exact user values syntactic validity without answerability executable but semantically weak examples

Table 1: PIPE-Cypher candidate acceptance gates.

weakening schema validation. The Cypher layer uses constraints drawn from production Text2Cypher work: schema-only prompting, exact matching, relationship direction discipline, RETURN DISTINCT, reserved variable rejection, categorical values, required contextual return columns, fuzzy value annotations, placeholderized retrieval examples, and parseraware rewrite boundaries. PIPE-Cypher records parser-style structure features and skips rewrites for risky constructs such as UNION, CALL, UNWIND, WHERE EXISTS, multiple WHERE clauses, or reserved variables. Before scaling a run, PIPE-Cypher checks that each target category has enough executable slot bindings; this is only a launch guard, and exported examples still pass validation, execution, diversity, and judge gates. For enterprise onboarding, PIPE-Cypher includes a deployment template for a company’s own graph: read-only credentials, local model endpoint, schema introspection, privacy policy, dry run, scaled run, audit, and redacted export. We validate this pattern on three public proxy graphs

rather than a proprietary tenant deployment. Configurable value-sampling policies decide which low-cardinality graph values may enter prompts. Redacted exports replace quoted literals, entity values, and string-valued result samples with stable placeholders for broader internal review. To support schemas beyond the built-in LDBC profiles, PIPE-Cypher derives relationship-count, anti-join, and top-k templates from observed labels, relationship directions, and safe low-cardinality properties, then grounds slot values with outcome-aware reverse Cypher.

Experiments

Research questions. We evaluate four questions that matter for an industry benchmark generator: RQ1, can a local-model pipeline produce a balanced executable benchmark over live property graphs? RQ2, do Cypher-specific validation and grounding steps make generation reliable at scale? RQ3, does the resulting benchmark expose meaningful downstream Text2Cypher failures rather than merely checking syntax? RQ4, can the same workflow onboard a new public enterprise-style

Graph

Candidates

Accepted

Acceptance

Categories at target

3,405 1,520 4,925

2,000 1,000 3,000

0.587 0.658 0.609

8/8 8/8 16/16

FinBench SNB Total

Table 2: Full live generation with local Qwen3.5-9B. Candidate counts include the initial sequential run and category-specific recovery top-ups.

graph without hard-coding FinBench or SNB? The generated benchmark contains 3,000 accepted examples: 2,000 from LDBC FinBench and 1,000 from LDBC SNB, balanced over eight workload categories. We report three completed FinBench/SNB ablation suites: target-50, corrected target-100, and a seed-17 target-50 repeat. Downstream Text2Cypher evaluation uses live execution accuracy and answer-set F1 as primary metrics; reference-based text metrics are supported for debugging only and reported in the appendix. We additionally report ICIJ Offshore Leaks as a third public finance/compliance onboarding proxy.

Results

RQ1: executable benchmark generation. The full live run produced 3,000 accepted examples from 4,925 candidates using local Qwen3.5-9B for generation and judging. Category-specific recovery top-ups filled the only under-target categories from the initial sequential run. Every exported example passed read-only, syntax, schema, execution, nonempty result, and judge gates. The accepted records are exported with stable identifiers, train/dev/test splits, result samples, gate metadata, aggregate statistics, and a manifest hash; the appendix gives the full artifact distribution. Diversity and residual concentration. We do not reduce diversity to one score. The full export has perfect category balance, near-perfect difficulty balance, 1,115 unique grounded entity values, and exact quotation of grounded values in 82.6% of examples with entity bindings. Query-signature diversity remains low because seeded graph-grounded templates carry much of the generation load. Table 3 shows that diversity is still governable after acceptance: at the same graph/category target, a selector using query signatures, template families, structural substructures, schema atoms, values, and question tokens improves structural coverage, adjusted Distinct-2, query-signature ratio, and property coverage. The result is a balanced executable benchmark, plus clear diagnostics for what should diversify during refresh.

Metric PIPE-Diversity index Unique query-signature ratio Structural substructures Adjusted Distinct-2 Property coverage

Random

Governed

0.557 0.062 97 0.266 0.407

0.575 0.135 134 0.284 0.426

Table 3: Diversity-governed selection at the same graph/category target. The selector improves structural, lexical, signature, and schema coverage after all quality gates have already passed; the appendix reports the full diversity audit, including residual template concentration.

RQ2: governed generation. Most full-run rejections come from duplicate/diversity controls or empty execution results. Only 2 of 4,925 candidates were schema-invalid after the Cypher checks. A rewrite audit found that all reported-run candidates were already identical to their normalized Cypher, so accepted examples do not depend on semantics-changing rewrites. The ablations should therefore be read as a reliability study of the execution-grounded core: in the target-100 suite, every non-unconstrained graph/setting cell reached all eight category targets, while unconstrained generation did not produce balanced executable coverage. Across the three evidence-ready suites, target-normalized coverage is 1.000 for every non-unconstrained cell. Judge calibration. An 80-row post-hoc human audit sampled accepted and rejected candidates across both graphs and all categories. Agreement is 80.0%, Cohen’s κ = 0.60, judge precision/specificity are 1.00, recall is 0.714, and no false accepts appear in the sample. The judge is conservative and protects accepted-example quality; human labels calibrate the gate but do not participate in generation. RQ3: downstream stress test. We evaluate downstream Text2Cypher by giving each model the schema text and the question, then scoring the generated query by live execution on FinBench or SNB. This is where the benchmark becomes useful: many outputs parse, mention plausible schema, and still answer the wrong graph question. The local Qwen3.5-9B baseline, for example, reaches 0.963 parse validity and 0.916 schema validity but only 0.189 exact execution accuracy on the 296-example held-out split. In an 11-model completed local transfer suite, zero-shot execution accuracy ranges from 0.000 to 0.203 (mean 0.036). Table 4 gives the control comparison. The primary few-shot gen-

eralization result is the scored no-signature control, which excludes exact query-signature matches and near-duplicate questions and raises mean accuracy to 0.200. Ordered and random same-category example banks reach 0.269/0.267, but we treat them as operational upper bounds because they often share query signatures. This split is important for industry use: the benchmark can be a hard modelevaluation set and, separately, a private example bank for schema-specific retrieval or adaptation. Condition

Sig.

Mean

Best

Zero-shot No-signature Ordered Random

– 0.000 0.866 0.854

0.036 0.200 0.269 0.267

0.203 0.828 0.993 0.986

Table 4: Eleven-model local downstream transfer controls on the 296-example held-out split. Sig. is the fraction of selected demonstrations sharing the test query signature; the no-signature row is the leakage-aware example-bank result, while ordered and random samecategory rows are operational upper bounds.

RQ4: third-graph onboarding. ICIJ onboarding reaches 800 accepted examples from 983 candidates on a 2.0M-node, 3.3M-edge public finance/compliance graph, with 100 examples in every category. This is evidence from a third public graph, not proof of private-tenant coverage. It is still important because it exercises schema-derived relationship-count, anti-join, and top-k templates beyond the two LDBC workloads.

Industry Use

Enterprise graphs are usually specialized artifacts, not generic benchmark schemas. They encode a company’s products, risk rules, permissions, data integrations, and analyst vocabulary. A Text2Cypher system is useful only if it works on the questions users actually ask of that graph. PIPECypher treats seed queries as a practical bridge from deployment to evaluation: when available, they can come from historical customer questions, analyst query logs, or agent tool calls triggered by user requests. The pipeline then expands those seeds into a balanced benchmark that tests the same kinds of operations the organization expects its agent to perform. This matters because the downstream stress test in Table 4 and Appendix Figure 11 shows that general Text2Cypher models often do not transfer cleanly to a new graph with its own schema.

The generated examples are therefore useful in two ways. First, they form a private held-out test set for measuring agent behavior under the organization’s schema, values, and safety rules. Second, accepted examples can become a schema-specific question– query bank for retrieval-augmented prompting; the no-signature few-shot control improves mean localmodel accuracy, while same-category banks give an operational upper bound (Appendix Tables 17–19). For more complex deployments, the same accepted pairs can also seed supervised adaptation, although we do not claim a tenant-specific fine-tuning result here. PIPE-Cypher is meant to be rerun when an enterprise graph changes. Schemas change as teams add products, ingest new sources, or encode new business logic, and static Text2Cypher corpora become stale quickly. Each example records the schema snapshot, graph profile, model identifier, validation gates, execution sample, judge scores, difficulty features, and source run, so the benchmark can be refreshed and audited. A deployment needs read-only graph credentials, schema introspection, a bounded value policy, a local endpoint, a dry run, scaled generation, judge calibration, and redacted export. Local inference keeps generation inside the compute boundary; if an organization later permits remote inference, the same value-sampling and redaction policies provide a safer artifact boundary. We validate this pattern on FinBench, SNB, and ICIJ while keeping the private-tenant gap explicit.

Conclusion

PIPE-Cypher reframes Text2Cypher benchmarking as a private, repeatable enterprise workflow. The main lesson is that generation improves when Cypher constraints become executable checks. Reverse grounding makes questions answerable. Deterministic validation catches unsafe or schemainvalid queries. Execution exposes empty or brittle candidates. Diversity diagnostics reveal concentration. A calibrated local judge adds a conservative semantic filter. Together these pieces produce a benchmark that is balanced, auditable, refreshable, and able to reveal downstream model failures that syntax-only evaluation would hide.

Limitations Execution validity does not guarantee semantic correctness. The completed 80-row, single-humanannotator calibration suggests a conservative judge

with no observed false accepts in the labeled sample, but the confidence interval is wider than the point estimate and larger multi-annotator audits may reveal additional failure modes. FinBench, SNB, and ICIJ are public enterprise-style proxies rather than a proprietary tenant graph, so we test the onboarding pattern but not every deployment constraint of a real organization. The full export is balanced by graph, category, and difficulty, but query-signature diagnostics still show template concentration from seeded, execution-grounded generation. PIPE-Cypher deliberately disallows or skips risky Cypher constructs such as writes, undirected relationships, UNION, CALL, UNWIND, and parser-risky rewrites. This is a safe benchmarkgeneration subset, not complete coverage of every production Cypher idiom. The downstream few-shot result should be read as graph-specific example-bank conditioning: the no-signature control is the leakage-aware result, while ordered and random same-category demonstrations are upperbound conditions that often share query signatures. Tenant-specific fine-tuning remains an engineering path enabled by the artifact, not a completed deployment claim here. The redaction audit checks exact residuals for known value-bearing strings, but it is not a full PII classifier and does not make schema names confidential.

aigentx. 2025b. llama-3.1-8b-instruct-cypher-mixedsamples. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06.

Ethics Statement

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, and 1 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.

PIPE-Cypher is designed for private benchmark generation under local-model deployment constraints. Organizations using it should restrict benchmark artifacts to authorized users, review sampled values for sensitive content, and document whether judge calibration relied on human annotators. For this study, one external human annotator labeled an 80-row post-hoc judge-calibration packet. The annotator was informed that the labels would be used for research calibration and paper reporting, and raw value-bearing annotation rows are not released. The protocol received an IRB exemption; identifying review-board details are omitted for double-blind submission. Human labels were never used as a generation gate.

References aigentx. 2025a. llama-3.1-8b-instruct-cypher. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06.

Azzedde. 2025. llama3.1-8b-text2cypher. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 335–336. ACM. Vashu Chauhan, Shobhit Raj, Shashank Mujumdar, Avirup Saha, and Anannay Jain. 2025. Mind the query: A benchmark dataset towards Text2Cypher task. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1890–1905, Suzhou, China. Association for Computational Linguistics. Yanlin Feng, Simone Papicchio, and Sajjadur Rahman. 2025. CypherBench: Towards precise retrieval over full-scale modern knowledge graphs in the LLM era. Preprint, arXiv:2412.18702.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and 1 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. Qwen2.5-coder technical report. Preprint, arXiv:2409.12186.

Matthew A. Jaro. 1989. Advances in record-linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Association, 84(406):414–420. Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, and Michalis Vazirgiannis. 2022. FrugalScore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1305–1318, Dublin, Ireland. Association for Computational Linguistics. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. Preprint, arXiv:2411.07763. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. Neo4j. 2024. text2cypher-2024v1: Consolidated textto-Cypher dataset. Hugging Face dataset. Neo4j. 2025a. text-to-cypher-Gemma-3-4B-Instruct2025.04.0. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Neo4j. 2025b. text2cypher-gemma-2-9b-it-finetuned2024v1. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Makbule Gulcin Ozsoy, Leila Messallem, Jon Besga, and Gianandrea Minneci. 2025. Text2Cypher: Bridging natural language and graph databases. In Proceedings of the Workshop on Generative AI and Knowledge Graphs, pages 100–108, Abu Dhabi, UAE. International Committee on Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. projectwilsen. 2024. llama3.1-8b-text2cypher-neo4jlive. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. David Püroja, Jack Waudby, Peter Boncz, and Gábor Szárnyas. 2023. The LDBC social network benchmark interactive workload v2: A transactional graph query benchmark with deep delete operations. Preprint, arXiv:2307.04820. Shipeng Qi, Heng Lin, Zhihui Guo, Gábor Szárnyas, Bing Tong, Yan Zhou, Bin Yang, Jiansong Zhang, Zheng Wang, Youren Shen, Changyuan Wang, Parviz Peiravi, Henry Gabb, and Ben Steer. 2023. The LDBC financial benchmark. Preprint, arXiv:2306.15975. Qwen. 2026. Qwen3.5-9B. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. Suraj Ranganath. 2026. PIPE-RDF: An LLM-assisted pipeline for enterprise RDF benchmarking. Preprint, arXiv:2602.18497. Saiprasanth15. 2024. llama3.1-8b-text2cypher-neo4jlive. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Aman Tiwari, Shiva Krishna Reddy Malay, Vikas Yadav, Masoud Hashemi, and Sathwik Tejaswi Madhusudhan. 2025. Auto-cypher: Improving LLMs on cypher generation via LLM-supervised generationverification framework. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2: Short Papers, pages 623–640, Albuquerque, New Mexico. Association for Computational Linguistics. tomasonjo. 2024. text2cypher-demo-16bit. Hugging Face model repository. Checkpoint source. Accessed: 2026-06-06. Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-SQL generation with execution-guided decoding. Preprint, arXiv:1807.03100.

William E. Winkler. 1990. String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage. In Proceedings of the Section on Survey Research Methods, pages 354–359. American Statistical Association. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, and 1 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. Xiuwen Zheng, Arun Kumar, and Amarnath Gupta. 2024. Generating cross-model analytics workloads using LLMs. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4303–4307. ACM. Zijie Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang. 2025. SyntheT2C: Generating synthetic data for fine-tuning large language models on the Text2Cypher task. In Proceedings of the 31st International Conference on Computational Linguistics, pages 672–692, Abu Dhabi, UAE. Association for Computational Linguistics. Jiongli Zhu, Haoquan Guan, Parjanya Prajakta Prashant, Nikki Lijing Kuang, Seyedeh Baharan Khatami, Canwen Xu, Xiaodong Yu, Yingyu Lin, Zhewei Yao, Yuxiong He, and Babak Salimi. 2026. Residual skill optimization for text-to-sql ensembles. Preprint, arXiv:2605.21792. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1097–1100. ACM.

Prior-Work Positioning

Table 5 expands the related-work comparison from the main paper. PIPE-Cypher’s contribution is the way answerability, governance, privacy, judging, and refresh are assembled into an organization-run benchmark factory. Prior mechanism

PIPE-Cypher delta

Template/LLM synthesis in SyntheT2C (Zhong et al., 2025) and AutoCypher (Tiwari et al., 2025) Execution validation in Auto-Cypher (Tiwari et al., 2025) and Mind the Query (Chauhan et al., 2025) Schema/value checks in Mind the Query (Chauhan et al., 2025) PIPE-RDF enterprise benchmark factory (Ranganath, 2026)

Outcome-aware reverse grounding binds slots through live Cypher before NL realization.

Human logical review in Mind the Query (Chauhan et al., 2025) Public static datasets and tuning corpora (Ozsoy et al., 2025; Feng et al., 2025; Neo4j, 2024)

Execution is one gate in a ledger with direction, read-only, value, non-empty, and judge evidence. Checks are packaged for private refresh with configurable value sampling and redacted exports. Adapts the factory framing from RDF/SPARQL to property graphs, where direction, relationship properties, and path semantics require Cypher-specific governance. Human labels calibrate a local LLM judge; they are not a generation gate. Organization-run benchmark factory with manifests, refresh, and provenance audits.

Experimental Setup and Exported Benchmark

Unless noted otherwise, the extended results use the same live FinBench/SNB export as the main results. Tables in this section pin down the scale, graph mix, split, and validation totals so the downstream and diversity analyses are tied to a single benchmark artifact. B.1

Value

Accepted examples Primary graph Secondary graph Categories Generation/judge model Execution backend Judge audit packet

3,000 LDBC FinBench, 2,000 examples LDBC SNB, 1,000 examples 8 balanced categories, 375 each Local Qwen3.5-9B Neo4j Community, two live databases 80 sampled accepted/rejected pairs

Table 6: Full live experimental setup used for the generated benchmark artifact.

Export artifact Live full benchmark

Table 5: Prior mechanism comparison. PIPE-Cypher combines outcome-aware grounding, deterministic Cypher checks, local judge calibration, privacy policy, refresh support, and audit logs so organizations can generate their own benchmarks rather than only consume static datasets.

Setting

Benchmark Artifact Summary

Tables 6, 7, and 8 give the run configuration, export manifest summary, and gate/distribution details for the 3,000-example benchmark.

Examples

FinBench

SNB

Train

Dev

Test

3,000

2,000

1,000

2,408

296

Table 7: Accepted live full benchmark package with stable IDs, gate metadata, result samples, statistics, and manifest hash cf274344be2abe7e.

Artifact property Categories FinBench/category SNB/category Difficulty split Used labels / rel. types Read/syntax/schema/exec/judge gates

Value 8 balanced categories 250 125 1,569 easy / 1,431 medium 16 / 17 3,000/3,000

Table 8: Distribution and gate summary for the exported full benchmark artifact.

Governed Generation Evidence

The central reliability question is whether reverse grounding and Cypher validation can fill every planned graph/category cell at target scale. C.1

Target-100 Stress Baseline

Figure 2 and Table 9 show the target-100 stress baseline. The unconstrained rows are deliberately harsh: raw local-model generations may look plausible, but they do not produce balanced, executable benchmark coverage. The governed variants fill every target cell on both graphs.

Acceptance rate

1.0

0.98

FinBench

SNB

0.98

0.96

0.99

0.98

0.97

0.99

0.96

0.97

Target-100 ablation gate rates

0.97

0.8

FinBench / Unconstrained LLM

1.000

0.806

0.723

0.557

0.6

FinBench / Reverse-only

1.000

0.982

FinBench / Validators+repair

1.000

0.960

FinBench / No retrieval

1.000

0.977

FinBench / No rewrite

1.000

0.969

FinBench / No LLM judge

1.000

0.985

FinBench / Full PIPE-Cypher

1.000

0.971

SNB / Unconstrained LLM

1.000

0.998

0.599

0.478

0.144

SNB / Reverse-only

1.000

0.998

0.976

SNB / Validators+repair

1.000

0.998

0.971

SNB / No retrieval

1.000

0.998

0.989

SNB / No rewrite

1.000

0.998

0.959

SNB / No LLM judge

1.000

0.998

0.966

1.00

0.47

0.4

0.98 0.2 0.03

0.0

Categories at target

1.00

0.96

1.00

0.94

0.8 0.6

0.92

0.4 0.25

0.2 0.00

0.0 LM

e ain

str

con

-on

e ers

v Re

air

rep

rs+

ato

lid Va

iev

retr

rite rew

LLM

ge Full

PIP

y E-C

Figure 2: Target-100 ablation yield on FinBench and SNB. Unconstrained local generation is reported as a stress baseline with explicit attempt accounting; the execution-grounded governed variants reach all eight workload-category targets on both graphs. This shows that the grounded governance core is reliable at filling the planned benchmark cells; it does not claim that each optional stage independently increases yield.

SNB / Full PIPE-Cypher

Graph

Unconstrained LLM Unconstrained LLM Reverse-only Reverse-only Validators+repair Validators+repair No retrieval No retrieval No rewrite No rewrite No LLM judge No LLM judge Full PIPE-Cypher Full PIPE-Cypher

FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB

Attempts

Records

Accepted

Acceptance

Categories at target

422 2,000 815 820 833 824 819 809 826 834 812 828 824 824

200 50 800 800 800 800 800 800 800 800 800 800 800 800

0.474 0.025 0.982 0.976 0.960 0.971 0.977 0.989 0.969 0.959 0.985 0.966 0.971 0.971

2/8 0/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8 8/8

Table 9: Live target-100 ablation evidence with local Qwen3.5-9B. Governed graph runs target 100 accepted examples per category; the unconstrained row is a stress baseline reported with explicit attempt accounting.

C.2

Gate-Level Quality

Yield alone is not enough. A benchmark factory must show which checks keep bad examples out. Table 10 and Figure 3 report read-only, syntax, schema, execution, and judge/post-hoc rates for the same target-100 suite. Read-only and syntax checks saturate once governance is active; execution and semantic judging expose the remaining differences. Setting

Graph

FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB FinBench SNB

Read-only

Syntax

Schema

Exec.

Judge/post-hoc

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

1.000 0.998 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.806 0.599 0.982 0.998 1.000 0.998 1.000 0.998 1.000 0.998 1.000 0.998 1.000 0.998

0.723 0.478 0.982 0.998 1.000 0.998 1.000 0.998 1.000 0.998 1.000 0.998 1.000 0.998

0.557 0.144 0.982 0.976 0.960 0.971 0.977 0.989 0.969 0.959 0.985 0.966 0.971 0.971

Table 10: Quality-gate rates for the live target-100 ablation suite. Rates are computed over all generated records in each graph/setting; for no-judge settings, the judge column is a post-hoc scoring diagnostic.

0.88

0.86

1.000

0.998

0.971

Read-only

Syntax

Schema

Exec.

Judge

Figure 3: Gate-rate heatmap for the audited target-100 ablation suite. Deterministic read-only and syntax gates are saturated; execution and judge rates expose the remaining quality differences across graph and pipeline variants.

C.3 Setting

0.90

jud

Gate pass rate

1.00

1.0

Repeated-Suite Stability

The same pattern holds across target sizes and seeds. The repeated-suite comparison normalizes by each suite’s planned target, so it measures stability rather than rewarding larger raw counts. Full PIPE-Cypher and the governed ablations reach complete target coverage on both graphs, which is the behavior we want from a repeatable benchmark generator. Setting

Graph

Suites

Target cov.

Acceptance

Cat. target

Exec.

Judge

No LLM judge No retrieval No rewrite Full PIPE-Cypher Reverse-only Validators+repair No LLM judge No retrieval No rewrite Full PIPE-Cypher Reverse-only Validators+repair

FinBench FinBench FinBench FinBench FinBench FinBench SNB SNB SNB SNB SNB SNB

3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3 3/3

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.994±0.008 0.967±0.031 0.969±0.023 0.975±0.016 0.994±0.011 0.987±0.023 0.982±0.015 0.993±0.004 0.983±0.021 0.987±0.014 0.989±0.011 0.987±0.014

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

1.000 1.000 1.000 1.000 0.994 1.000 0.996 0.996 0.996 0.996 0.996 0.996

0.994 0.967 0.969 0.975 0.994 0.987 0.982 0.993 0.983 0.987 0.989 0.987

Table 11: Target-size and repeated-seed ablation sensitivity. Target coverage normalizes accepted examples by each suite’s planned graph/category target, so target-50 and target-100 suites can be compared without treating larger raw counts as quality gains. Unconstrained local generation is excluded from this stability table and reported separately as the attempt-logged stress baseline in Table 9.

Three-suite ablation stability

Acceptance mean

1.00

Benchmark Artifact and Third-Graph Onboarding

D.1

Export Balance

0.98 0.96 0.94 0.92 FinBench

SNB

0.90

The exported benchmark makes scale and balance visible. Figure 6 shows the planned 2:1 FinBench/SNB mix, exact category balance, and near-even difficulty split. PIPE-Cypher produces a benchmark that is balanced by design rather than sampled opportunistically.

0.98 0.96 0.94 0.92 0.90 nly e-o

rs ve Re

l va

a rep

rs+ ato lid Va

ie retr

rite rew

LLM

ge jud Full

r he

E-C

PIP

Unconstrained LLM omitted: reported separately as a stress baseline with attempt accounting.

Figure 4: Three-suite ablation stability over the original target-50 suite, corrected target-100 suite, and seed-17 target-50 repeat. Target-normalized coverage stays at 1.000 for all non-unconstrained cells.

C.4

Full export category balance FinBench

350

Accepted examples

Target coverage mean

1.00

Difficulty

SNB

1750 1569

300

1250

250 200

1000

150

750

100

500 250

Rejected Candidate Taxonomy

lea oo

The rejection taxonomy shows where the remaining work occurs. After Cypher validation, schemainvalid candidates are rare. Most rejections are duplicate/diversity blocks from recovery or emptyresult candidates caught before export. This is the behavior an enterprise deployment wants: invalid or brittle examples remain in the ledger, not in the benchmark. Failure bucket

Count

Share of rejected

Diversity/duplicate control Empty result Judge semantic reject Execution error Schema invalid

1,366 486 67 4 2

0.710 0.252 0.035 0.002 0.001

Table 12: Failure taxonomy over full-run generation candidates before benchmark export. Accepted examples are excluded from the bucket shares.

Full-run rejection taxonomy before export

1431

1500

g xa ple

t. x re ple

o ati

th/t Pa

p em

in nk Ra

ple

Sim

g ag

ple

easy

ret.

medium

Sim

Figure 6: Full 3,000-example export distribution. The benchmark preserves the planned 2:1 FinBench/SNB graph mix while balancing all eight Cypher workload categories and maintaining a near-even easy/medium difficulty split.

D.2

Graph Scale and Schema Variety

Table 13 summarizes the graph sizes and live schema inventories before the third-graph onboarding audit. FinBench and SNB are the controlled LDBC workloads. ICIJ is the public finance/compliance graph we use to test whether the same onboarding code works outside the two original schemas. Figures 7–9 show the same schemas in graph form. Node boxes are labels; arrows are directed relationship families observed by introspection. Some arrows group repeated label-pair alternatives so the figure remains readable, while Table 13 gives the full inventory counts.

1366 (71.0%)

Diversity/duplicate control 486 (25.2%)

Empty result

Graph FinBench SNB ICIJ Offshore Leaks

67 (3.5%)

Judge semantic reject

Execution error

4 (0.2%)

Schema invalid

2 (0.1%)

200

400

600

800

1000

1200

1400

1600

Rejected candidates

Figure 5: Full-run rejection taxonomy before benchmark export. Most rejected candidates come from duplicate/diversity control during category recovery and from empty execution results; schema-invalid Cypher is rare after Cypher validation.

Nodes

Edges

Labels

Rel. types

Patterns

Node props

Rel. props

10,006 34,735 2,016,523

57,622 70,842 3,339,267

5 14 5

9 15 14

13 59 64

37 62 82

32 5 48

Table 13: Study graph size and schema inventory from live introspection. Patterns are directed startlabel/type/end-label triples; property counts are distinct label-property or relationship-type-property fields. ICIJ Offshore Leaks is used as a public third-graph onboarding audit beyond the two LDBC workloads.

Person

GUARANTEE

YT AI PS EO RP DE

UNT

Account

INVEST

Company

INVEST GUARANTEE

6 props

SIG N

LOA PLY_

_IN

9 props

Medium 5 props

Figure 7: FinBench schema graph used in the reported runs. The workload is centered on financial entities, account ownership, transaction/event relationships, loans, sign-in media, guarantees, and investments.

Forum

HAS_TAG CONT

AINE

HAS_

R_OF

LI KE S

HAS_MEMBER HAS_MODERATOR

Tag

TAG

Message Post Comment

HAS_TYPE

TagClass

IS_SUBCLASS_OF

REPLY_OF

HAS_INTEREST

ICIJ accepted examples by category

Accepted examples

WORK_AT STUDY_AT

IS_LOCATED_IN N

ED_I

Organisation Company University

100

IS_LOCATED_IN

Person

KNOWS

Place City Country Continent

IS_PART_OF

AT _LOC

19 props

of inte ficer_ rm of conn ediary ecte _o d_to f sa me _n am e_ as

Address registered_address

15 props

same_as same_address_as

Entity

connected_to underlying

registered_address st

_of ary edi of erm er_ o int ffic ted_t o nec n o c

19 props

officer_of

Other 14 props

Figure 9: ICIJ Offshore Leaks schema graph used for the third-graph onboarding audit. The schema is compact in node labels but large in edge volume: relationship families encode officer/entity/intermediary roles, registered addresses, identity-resolution links, and similarity links.

D.3

100

Remaining sparse-binding failures 100

100

bindings unavailable

141

60 40 bindings exhausted

ple

ICIJ Onboarding

Table 14 and Figure 10 report the third-graph result. ICIJ matters because it forced PIPE-Cypher to derive sparse-category templates from the schema instead of relying on preauthored FinBench/SNB templates. The run reaches all eight category targets while using the same validation and audit standards.

t. re

et xr ple

Sim

ple

xa ple

ole

tio

em h/t

t Pa

kin

n Ra

100

125

150

Rejected

Figure 10: ICIJ Offshore Leaks onboarding audit. Schema-derived sparse-category templates recover balanced target-100 coverage on a public finance/compliance graph beyond the two LDBC workloads.

D.4

same_company_as registered_address same_name_as same_as similar_company_as

29 props

Intermediary

100

same_as underlying similar

same_name_as similar same_intermediary_as

100

Figure 8: SNB schema graph used in the reported runs. Related labels are grouped into content, place, and organization families so all 14 labels and 59 directed label/type/label patterns remain readable; Table 13 gives the full inventory counts.

Officer

100

Sim

same_name_as same_id_as similar underlying

2,016,523 / 3,339,267 5 / 14 983 / 800 0.814 8/8 ready complex agg. 97, negation 28, ranking 98

Table 14: ICIJ Offshore Leaks third-graph onboarding audit. The public finance/compliance graph tests arbitrary-schema generation beyond the two LDBC study workloads; raw values remain outside the reported artifacts.

Target

O AT

Value

Graph nodes / relationships Labels / relationship types Generated / accepted Acceptance rate Categories at target Study audit Sparse schema-derived accepts

TRANSFER_TO WITHDRAW_TO

9 props NT

U ACCO OWN_

ICIJ onboarding property

Loan

APPLY_LOAN OWN_ ACCO

8 props

Category Crosswalk

Table 15 connects PIPE-Cypher’s eight categories to the simpler retrieval/aggregation/evaluationquery taxonomy used in Mind the Query. The crosswalk keeps the comparison clear while making our extra enterprise workload classes explicit: negation, temporal/path reasoning, and ranking/top-k. PIPE-Cypher category

Closest MTQ category

Added enterprise distinction

Simple retrieval Complex retrieval Simple aggregation Complex aggregation Boolean existence Negation/difference Path/temporal transaction Ranking/top-k

SR CR SA CA EQ CR CR/CA SA/CA

Direct node/edge lookup with exact filters. Multi-hop or multi-pattern retrieval. Single aggregation such as count/min/max/average. Grouped or multi-stage aggregation over graph neighborhoods. Precise yes/no or existence answer. Absence, anti-join, or difference query. Temporal or path-oriented transaction neighborhood. Ordered top-k query with explicit limit.

Table 15: Category crosswalk to Mind the Query. PIPE-Cypher keeps the familiar retrieval/aggregation/evaluation-query structure while adding enterprise workloads such as negation, temporal paths, and ranking.

Downstream Transfer and Example-Bank Utility

E.1

Transfer Controls

The downstream experiment tests whether the benchmark is useful for model evaluation. A useful enterprise benchmark should not merely reward syntactically valid Cypher. It should reveal when a model produces a query that parses, mentions plausible schema elements, and even executes,

but answers the wrong operational question. The Qwen3.5-9B result has exactly that shape: parse validity is 0.963 and schema validity is 0.916, while exact execution accuracy is only 0.189 on the full held-out split. The gap shows that PIPE-Cypher measures semantic graph-query competence rather than only query formatting. The multi-model transfer suite compares local general instruction, code-tuned, Cypher instruction, and Text2Cypher-finetuned models while keeping the schema prompt, evaluation backend, split, and execution metrics fixed. The exact checkpoint and adapter provenance is listed in Table 16. Figure 11 shows a sharp pattern: zero-shot transfer is weak, but accepted graph-specific examples can become a useful private question-answer bank. Demonstration-bank controls close much of the gap for compatible model families such as Qwen, Qwen-Coder, and one Gemma Text2Cypher LoRA. Several public fine-tuned checkpoints remain brittle under enterprise-style prompting. Table 18 reports checkpoint-level uncertainty for the same controls. The interval is deliberately conservative because the unit is model checkpoint rather than question row; it supports the narrower claim that example-bank gains are model-family dependent, not universal. Control Ordered same-category Scored no-signature Random same-category mean

Mean acc.

∆ vs zero

95% ∆ CI

Models improved

0.269 0.200 0.267

0.233 0.163 0.231

[0.000, 0.481] [0.000, 0.335] [0.000, 0.464]

3/11 3/11 3/11

Table 18: Model-level paired bootstrap uncertainty for downstream few-shot controls. The unit of resampling is the local checkpoint, not an individual question, so the interval is a conservative check on whether gains are broad across model families. Zero-shot mean execution accuracy is 0.036.

Table 19 gives the interpretation for the transfer result. Ordered and random same-category demonstrations are useful example-bank conditions. The scored no-signature condition is the stricter leakage-aware control.

Selection mode

Rows

Demos

Sig. match

High sim.

Mean sim.

Ordered No-signature Random

296 296 296

1480 1480 1480

0.866 0.000 0.854

0.389 0.000 0.409

0.825 0.585 0.824

Table 19: Few-shot leakage controls for downstream demonstration-bank evaluation. The held-out split has 0 exact train/test question overlaps and 289 train/test query-signature overlaps; selection-mode rates show how often retrieved demonstrations share the test query signature or exceed the 0.90 normalized-question similarity threshold.

Displayed model

Served checkpoint or adapter

Family

Citation

aigentx/Llama-3.1-8B Cypher LoRA aigentx/Llama-3.1-8B Cypher mixed LoRA Azzedde/llama3.1-8b-text2cypher Gemma-2-9B-IT neo4j/Gemma-2-9B Text2Cypher LoRA neo4j/Gemma-3-4B Text2Cypher projectwilsen/Llama-3.1-8B Text2Cypher LoRA Qwen2.5-Coder-7B-Instruct Qwen3.5-9B Saiprasanth15/Llama-3.1-8B Text2Cypher LoRA tomasonjo/text2cypher-demo-16bit

aigentx/llama-3.1-8b-instruct-cypher aigentx/llama-3.1-8b-instruct-cypher-mixed-samples Azzedde/llama3.1-8b-text2cypher google/gemma-2-9b-it neo4j/text2cypher-gemma-2-9b-it-finetuned-2024v1 neo4j/text-to-cypher-Gemma-3-4B-Instruct-2025.04.0 projectwilsen/llama3.1-8b-text2cypher-neo4j-live Qwen/Qwen2.5-Coder-7B-Instruct Qwen/Qwen3.5-9B Saiprasanth15/llama3.1-8b-text2cypher-neo4j-live tomasonjo/text2cypher-demo-16bit

Cypher LoRA Cypher mixed LoRA Text2Cypher fine-tuned General instruction Text2Cypher LoRA Text2Cypher fine-tuned Text2Cypher LoRA Code instruction General instruction Text2Cypher LoRA Text2Cypher fine-tuned

(Grattafiori et al., 2024; aigentx, 2025a) (Grattafiori et al., 2024; aigentx, 2025b) (Grattafiori et al., 2024; Azzedde, 2025) (Gemma Team et al., 2024) (Gemma Team et al., 2024; Ozsoy et al., 2025; Neo4j, 2025b) (Gemma Team et al., 2025; Ozsoy et al., 2025; Neo4j, 2025a) (Grattafiori et al., 2024; projectwilsen, 2024) (Hui et al., 2024) (Yang et al., 2025; Qwen, 2026) (Grattafiori et al., 2024; Saiprasanth15, 2024) (Grattafiori et al., 2024; tomasonjo, 2024)

Table 16: Downstream model provenance for the 11 completed local checkpoints in the transfer study. Original model-family papers are cited wherever available; Hugging Face repository citations are retained only to identify exact fine-tuned checkpoints or adapters without a separate paper.

Downstream execution accuracy under few-shot controls aigentx Cypher LoRA

0.00

aigentx mixed LoRA

0.00

Azzedde T2C

0.00

Gemma-2-9B-IT

0.00

Gemma-2 T2C LoRA

0.20

0.98

0.70

0.98

Gemma-3 T2C

0.00

projectwilsen LoRA

0.00

Qwen2.5-Coder

0.01

0.98

0.67

0.97

Qwen3.5-9B

0.19

0.99

0.83

0.99

Saiprasanth LoRA

0.00

tomasonjo T2C

0.00

Zero

Ordered

No-sig

Random mean

1.0

0.6

0.4

Execution accuracy

0.8

0.2

0.0

Figure 11: Execution accuracy for 11 completed local downstream models under zero-shot and few-shot control modes. The heatmap highlights both findings: schema-specific examples can sharply help compatible models, and several public fine-tuned checkpoints still fail under enterprise-style Cypher prompting.

Model

Tuning

Zero

Ordered

No-sig

Random µ

Random σ

Best

Cypher LoRA Cypher mixed LoRA Text2Cypher fine-tuned general instruction Text2Cypher LoRA Text2Cypher fine-tuned Text2Cypher LoRA code instruction general instruction Text2Cypher LoRA Text2Cypher fine-tuned

0.000 0.000 0.000 0.000 0.203 0.000 0.000 0.007 0.189 0.000 0.000

0.000 0.000 0.000 0.000 0.983 0.000 0.000 0.983 0.993 0.000 0.000

0.000 0.000 0.000 0.000 0.699 0.000 0.000 0.669 0.828 0.000 0.000

0.000 0.000 0.000 0.000 0.981 0.000 0.000 0.972 0.986 0.000 0.000

0.000 0.000 0.000 0.000 0.009 0.000 0.000 0.002 0.007 0.000 0.000

no gain no gain no gain no gain ordered (0.983) no gain no gain ordered (0.983) ordered (0.993) no gain no gain

Table 17: Few-shot demonstration-bank controls for local downstream Text2Cypher evaluation over completed 296-example local-model runs. Ordered uses the deterministic same-graph, same-category example bank; scored excludes exact query-signature matches and near-duplicate questions; random reports the mean and standard deviation across seeds 13, 17, and 23. “No gain” means no few-shot control exceeded that model’s zero-shot execution accuracy. Model-family papers and exact checkpoint sources are listed in Table 16.

E.2

Downstream Failure Modes

Downstream outcome Answer mismatch Execution failed Schema invalid Parse invalid

Count

Share of incorrect

125 79 25 11

0.521 0.329 0.104 0.046

Table 20: Downstream Text2Cypher failure taxonomy for local Qwen3.5-9B on the full exported test split. Shares exclude exact-answer matches.

Table 20 makes the downstream study actionable. An answer mismatch means the model produced executable Cypher for the wrong semantics, so better graph-specific examples, retrieval, or adaptation are the likely fixes. Execution failures point instead to unsupported operators, brittle literals, or missing repair rules. Parse and schema failures are the

errors deterministic guards should catch before a query ever reaches users. This is why the taxonomy appears next to the transfer results. Many incorrect outputs are not malformed queries; they run and return the wrong answer set. A syntax-only or schema-only benchmark would miss that failure. For a benchmark owner, the taxonomy is therefore a debugging interface: it shows whether the next improvement should target schema retrieval, prompt constraints, value grounding, operator repair, or tenant-specific adaptation.

Diversity and Cypher Strategy Audit

F.1

Diversity Diagnostics

Aggregate acceptance rates hide too much. The diversity and strategy diagnostics show whether the benchmark is balanced, which Cypher operators it exercises, and where residual concentration remains. Figure 12 shows the useful diversity picture: category, graph-category, and difficulty balance are strong by construction, while schema and value coverage remain quantities to monitor during refresh. Metric PIPE-Diversity index Question Distinct-1 Question Distinct-2 Question adjusted Distinct-2 Question self-BLEU-2 (sampled) Mean nearest-neighbor question Jaccard Unique query-signature ratio Top query-signature share Template-family entropy Operator-combination entropy Unique structural substructures Category normalized entropy Graph-category normalized entropy Difficulty normalized entropy Label coverage Relationship-type coverage Property-name coverage Unique grounded-value ratio Grounded values exactly quoted Aggregation / negation / ordering rates

Value 0.549 0.039 0.085 0.266 0.813 0.775 0.041 0.083 0.826 0.880 134 1.000 0.980 0.998 0.941 0.708 0.426 0.357 0.823 0.500 / 0.125 / 0.125

Table 21: Diversity diagnostics for the full exported benchmark. PIPE-Diversity is a geometric mean of lexical, query-template, structural, schema, value, and balance components; component rows are shown so the composite score does not hide residual concentration.

Full benchmark diversity components

Normalized score

0.8 0.69

0.64

0.6

a category-balanced dataset can still miss joins, paths, negation, ranking, or bounded-result patterns. The strategy matrix and downstream strategy outcomes show what the benchmark actually exercises.

0.99

1.0

0.73

0.55 0.45

0.4 0.19

0.2

0.0 PIPE-Diversity index

Lexical variety

Query-template variety

Structural coverage

Schema coverage

Value variety

Primary strategy

Balance coverage

Figure 12: Diversity diagnostics for the full 3,000example export. Category, graph-category, and difficulty balance are strong by construction; schema coverage and query-signature diversity expose remaining concentration from the seeded-template run.

F.2

Diversity-Governed Selection

Table 22 reports the first diversity-improvement pass. We select a target-50-per-graph/category subset from the 3,000 accepted examples and compare it with a hash-balanced random subset of the same size. The selector uses an MMR-style novelty score over query signatures, template families, structural substructures, schema atoms, entity values, and question tokens after all quality gates have passed. It improves the PIPE-Diversity index, unique querysignature ratio, unique structural substructures, adjusted Distinct-2, and property coverage while preserving a signature-disjoint 640/80/80 split. The mixed template-family entropy and self-BLEU result is useful rather than embarrassing: it shows that post-hoc selection can improve coverage, but stronger diversity requires oversampling or inducing more source templates during generation when a graph/category cell has only one or two viable signatures. Metric PIPE-Diversity index Unique query-signature ratio Top signature share (lower better) Template-family entropy Operator-combination entropy Unique structural substructures Question self-BLEU-2 (lower better) Adjusted Distinct-2 Property coverage

Random balanced

Diversity governed

∆

0.557 0.062 0.062 0.903 0.901 97.000 0.850 0.266 0.407

0.575 0.135 0.062 0.868 0.911 134.000 0.866 0.284 0.426

+0.017 +0.073 +0.000 -0.035 +0.010 +37.000 +0.016 +0.018 +0.019

Table 22: Balanced subset comparison at the same graph/category target. The diversity-governed selector applies MMR-style novelty over Cypher signatures, template families, structural substructures, schema atoms, values, and question tokens after quality gates have already passed; structural/schema gains are reported alongside residual template concentration.

F.3

Cypher Strategy Coverage

Strategy diagnostics complement category balance. Two questions can share a category while exercising different Cypher behavior. Conversely,

Aggregation Join-heavy Negation Order/rank Path Single hop

Examples

Rel. patterns

Return arity

Downstream exec. acc.

1125 375 375 375 375 375

0.375 0.125 0.125 0.125 0.125 0.125

1.42 2.33 1.89 1.87 1.37 1.00

1.00 2.33 2.84 3.41 3.00 2.33

0.396 0.000 0.000 0.000 0.000 0.324

Table 23: Cypher strategy diagnostics over the full 3,000-example export. Strategy tags are derived from generated Cypher structure rather than from category labels; downstream execution accuracy is reported on the full held-out test split when a strategy appears there.

Cypher strategy coverage by workload category

1.0

0.75

0.25

1.00

0.00

0.33

0.00

Complex agg.

0.00

1.00

0.00

Complex ret.

0.00

1.00

0.00

1.00

Negation

0.13

0.87

0.00

1.00

0.00

0.93

Path/temp.

0.63

0.37

0.00

1.00

0.00

1.00

Ranking

0.13

0.87

1.00

0.00

1.00

Simple agg.

1.00

0.00

1.00

0.00

Simple ret.

1.00

0.00

1.00

0.8

0.6

0.4

Share of category examples

Boolean

0.2

0.0

e gl

a eg gr

tio

ea -h

r/r de Or

th Pa

tio

a eg

io pt

n ou

d de

Figure 13: Cypher strategy coverage by workload category. Category balancing does not by itself guarantee operator coverage; the strategy matrix shows where the benchmark exercises single-hop retrieval, joins, aggregation, ordering, negation, path patterns, optional matches, and bounded-result queries.

Downstream outcomes by gold Cypher strategy 120

Exact Answer mismatch

111

Exec. failed Schema invalid

Parse invalid

Held-out examples

100

60 37

0 Ag

tio

gre

n-h

Joi

n tio

ran

er/

Ord

h Pat

gle

Sin

Figure 14: Downstream outcomes by gold Cypher strategy on the full held-out test split. The local Qwen3.5-9B baseline fails differently across strategies: aggregation mixes exact matches with execution failures, while join-heavy, negation, path, and ranking examples are dominated by semantically wrong executable queries or execution failures.

Supplementary Evaluator Metrics

Table 24 lists optional text-overlap metrics for debugging answer rendering and near-match behavior. These are not correctness metrics; the reported correctness results use execution accuracy and answerset F1. Metric

Supported use and primary citation

ROUGE-1/2/L

Reference overlap over serialized answer sets and query strings (Lin, 2004) Sentence-level reference overlap with brevity penalty (Papineni et al., 2002) Exact-token precision/recall with fragmentation penalty (Banerjee and Lavie, 2005) Optional contextual-embedding similarity (Zhang et al., 2020) Optional efficient learned NLG metric (Kamal Eddine et al., 2022) Token-count vector cosine similarity (Manning et al., 2008) Character-level near-match similarity (Jaro, 1989; Winkler, 1990) Raw and normalized string exact match (Rajpurkar et al., 2016)

BLEU METEOR BERTScore FrugalScore Cosine Jaro-Winkler Exact match

Table 24: Supplementary reference-based text metrics supported by the PIPE-Cypher evaluator. These metrics are useful for debugging answer rendering, paraphrase sensitivity, and near-match behavior, but they do not replace execution accuracy or answer-set F1 for Text2Cypher correctness. BERTScore and FrugalScore are optional integrations because they require additional metric/model packages.

Governance, Judge Calibration, and Deployment Details

The deployment details below matter because enterprise failures rarely come from a single model call. They come from missing safety boundaries, unclear value policies, weak audit trails, or no explanation for why an example was accepted. The validator cascade, prompt contracts, automation comparison, and judge audit describe the controls that make PIPE-Cypher a governed local workflow rather than a manual dataset-writing exercise. H.1

Validation Cascade

Table 25 lists the deterministic and judge gates in execution order. It shows how an enterprise deployment keeps unsafe, schema-invalid, empty, or semantically weak examples out of exported benchmarks.

Gate or ledger

Count

Denominator

Export accepted Read-only safety Syntax validity Schema/value validity Execution success LLM judge pass Rejected candidates logged

3,000 3,000 3,000 3,000 3,000 3,000 1,925

3,000 3,000 3,000 3,000 3,000 3,000 4,925

Table 25: PIPE-Cypher validation cascade for the full export and its logged candidate ledger. Unlike Mind the Query, human review is calibration-only.

H.2

Rewrite and Governance Audits

Table 26 addresses rewrite preservation directly. In the reported generation records, generated Cypher was already identical to normalized Cypher, so the accepted benchmark does not rely on a semanticschanging RETURN DISTINCT insertion or projection rewrite. Rewriting is still part of the pipeline, but in this run its role is conservative validation and logging rather than silent semantic alteration. Rewrite audit property

Value

Generation records audited Accepted records audited Records changed by normalization Accepted records changed RETURN DISTINCT insertions Accepted RETURN DISTINCT insertions Rewrite-skip reasons logged Live comparisons required Answer-set equality in comparisons

4,925 3,000 0 0 0 0 196 0 0

Table 26: Rewrite prevalence and impact audit over reported generation records. When no generated query differs from its normalized form, no live original/normalized re-execution is required for semantic drift.

Table 27 separates direction, schema/value, syntax/parser, and read-only issues across generation, ablation, and downstream prediction artifacts. The full generation records have no direction failures after validation. Downstream predictions still contain direction errors, which is exactly the kind of graph-specific failure a Cypher benchmark should expose.

Evidence source Full generation records Target-size ablations Downstream predictions Combined

Direction

Schema/value

Syntax/parser

Read-only

0 260 25 285

2 988 9 999

0 5 12 17

0 0 0 0

Table 27: Governance failure audit. Direction errors, schema/value errors, syntax/parser failures, and readonly violations are counted separately so the appendix shows which Cypher-specific gates do real work.

H.3

Gate-Impact Counterfactual

Table 28 summarizes the first gate that blocks each non-accepted candidate. This gives the counterfactual view missing from a yield-only ablation: it shows what kind of bad example would enter the benchmark if a deployment weakened duplicate/diversity controls, non-empty execution, judge review, schema validation, direction checking, or read-only safety.

First blocking gate

Candidates

Accepted Duplicate/diversity Empty result Judge reject Schema Execution failure

3,000 1,366 486 67 2 4

0.609 0.277 0.099 0.014 0.000 0.001

Redaction audit property

Value

Examples audited Sensitive values checked Examples with sensitive values Examples with residual raw values Residual raw-value matches Residual rate per checked value Unique placeholders Reused placeholders Max placeholder frequency

3,000 10,956 2,970 0 0 0.000 3,754 2,342 337

Table 29: Exact-match redaction audit over valuebearing benchmark surfaces. The audit checks entity bindings, quoted Cypher literals, reverse grounding literals, and string-valued result samples after applying the configured redaction policy.

H.5

Operational Accounting

Table 30 reports local-run accounting from completed generation records. These numbers are for deployment planning: acceptance rate and graph execution latency explain throughput bottlenecks without turning the analysis into a paid-API cost comparison. Scope Overall FinBench SNB

Records

Accepted

Acceptance

Exec. p50 ms

Exec. p95 ms

4,925 3,405 1,520

3,000 2,000 1,000

0.609 0.587 0.658

11.995 11.837 12.420

32.832 32.398 38.558

Table 28: Counterfactual first-blocking-gate audit over generation records. The table shows which failure class would enter the benchmark if that gate were removed or weakened.

Table 30: Operational accounting from completed generation records. These are local-run latency and acceptance diagnostics, not paid-API cost claims.

H.4

H.6

Privacy and Redaction Audit

Table 29 evaluates the redaction policy rather than merely describing it. The audit builds a sensitivevalue set from entity bindings, quoted Cypher literals, reverse-grounding literals, and string-valued execution samples. It then applies the configured redactor and exact-matches the raw values against the redacted question, Cypher, entity, and result fields. This does not replace a tenant’s PII classifier, but it gives reviewers a measurable privacy check for the value-bearing fields PIPE-Cypher itself creates.

Prompt Profiles and Contracts

Table 31 documents the prompt-profile factors used for ablation planning. The full prompt contracts follow immediately after the table, including the exact LLM-judge system prompt and user prompt template used for reported judge decisions. Profile

Added constraint

Intended evidence

Schema-only Instructions Examples Examples + instructions Full governed

Use only visible schema. Exact values, datatype rules, no nested aggregations. Placeholderized retrieved NL-Cypher pairs. Few-shot plus explicit rules. Production-derived Cypher hints, AST-safe rewrites, judge gate.

Baseline for schema-grounded prompting. Targets MTQ-style prompt refinements. Tests whether examples help without extra governance. Closest controlled analogue to MTQ Table 5. PIPE-Cypher production setting.

Table 31: Prompt profiles implemented for Mind-theQuery-style prompt-factorial evaluation. Results are reported only for completed, audited target-50-or-larger suites.

Prompt Contracts

PIPE-Cypher treats prompts as versioned implementation artifacts. The list summarizes the prompt contracts used for generation, repair, judging, and

downstream evaluation; hashes fingerprint the full prompt constants in the codebase. 1. Template generation. Stage: Workload proposal. SHA-256: 10b25b. Contract. Schema-only labels, relationships, properties, and categorical values; Realistic enterprise analyst wording; At most two typed slots and JSON-only output 2. Reverse binding. Stage: Graph grounding. SHA-256: 48e949. Contract. Read-only MATCH/WHERE/RETURN DISTINCT/LIMIT only; Slot variables named exactly as requested; Forward relationship directions from the schema 3. Cypher generation. Stage: Candidate query. SHA-256: 61c557. Contract. Only schema-visible constructs and observed directions; RETURN DISTINCT for set returns and exact equality for quoted values; Context columns, categorical hints, placeholderized retrieval, and no writes 4. Repair. Stage: Validation feedback. SHA256: 56c419. Contract. Preserve question intent while fixing validation or execution issues; Keep query read-only and schema-grounded; Return only corrected Cypher 5. LLM judge. Stage: Quality gate. SHA-256: 421c7b. Contract. Inputs include question, Cypher, relevant schema excerpt, execution rows, and validation summary; Strict JSON scores for ambiguity, semantic alignment, schema use, and difficulty; Categorical values constrain query literals, not observed result-row values; Pass only useful, unambiguous enterprise benchmark examples 6. Downstream Text2Cypher. Stage: Model evaluation. SHA-256: 4c07ff. Contract. Read-only Cypher only; Schemavisible constructs and exact direction preservation; RETURN DISTINCT, count/ranking rules, and no explanations

I.1

LLM Judge Prompt Used in Reported Runs

The judge runs only after deterministic validation and live execution. For each reviewed example, we fill the template below with the candidate question, Cypher, relevant schema excerpt, sampled execution rows, and validation summary. This is the prompt used for all reported LLM-judge decisions. System prompt: You are an expert benchmark engineer. Return strict JSON only. No markdown or extra text. User prompt template: You are judging whether an NL-to-Cypher benchmark example is acceptable for an enterprise benchmark. Graph schema: {schema} Question: {question} Cypher: {cypher} Execution sample: {rows} Validation summary: {validation} Return strict JSON with: - pass: boolean - ambiguity_score: number from 0 to 1, lower is better - semantic_alignment_score: number from 0 to 1 - schema_use_score: number from 0 to 1 - difficulty: one of easy, medium, hard - failure_reason: short string, empty if pass is true Pass only if the question is unambiguous, the Cypher answers it, the schema use is valid, and the result would be useful in an enterprise benchmark. - Categorical property values in the schema constrain literal values written in the Cypher query. - Do not reject because the execution sample returns a value that is absent from the categorical-value list; result rows are observed graph outputs. - If deterministic validation says schema use is valid, lower schema_use_score only when the Cypher itself uses nonexistent schema elements, invalid relationship directions, or invalid literal values.

I.2

Automation and Calibration

Table 32 gives the deployment contrast with Mind the Query, and Table 33 reports the completed human calibration packet. Human review has not disappeared from the research process. It has moved from a production gate to post-hoc calibration evidence. Human annotation protocol. One external annotator labeled the frozen 80-row audit packet after generation was complete. The packet sampled judge-accepted and judge-rejected candidates across FinBench/SNB categories. For each

row, the annotator inspected the NL question, Cypher, graph/category metadata, execution evidence, and judge decision, then filled a binary human_accept label and optional notes under the rubric in Appendix I: accept only if the question is clear and the Cypher is read-only, schema-grounded, directionally plausible, exact about quoted values, and semantically aligned with the question. The annotator’s labels calibrate the local judge and are reported only in aggregate. We do not report protected demographic attributes because a single annotator would be identifiable; recruitment and compensation details are recorded in the Responsible NLP checklist. The annotation protocol was determined exempt by an IRB. Dimension

Mind the Query

PIPE-Cypher

Generation review gate Human effort Private values Refresh Model endpoint

Manual logical review Reported 1,400 person-hours Public benchmark values Static dataset release Gemini in their reported pipeline

Deterministic gates + local LLM judge 80-row post-hoc judge calibration audit Configurable sampling and redacted export Rerunnable private benchmark factory Local Qwen3.5-9B endpoint

Table 32: Industry deployment contrast with Mind the Query. PIPE-Cypher focuses on private refreshable benchmark generation rather than a one-time public dataset.

Audit packet property Rows Human annotators IRB status Use of human labels Judge accept / reject FinBench / SNB rows Easy / medium rows Labeled rows Calibration status Judge-human agreement / κ Judge precision (95% CI) Judge recall (95% CI) False-accept rate (95% CI) False-reject rate (95% CI)

Value 80 1 external annotator exempt determination post-hoc calibration only 40 / 40 48 / 32 26 / 54 80 complete 0.800 / 0.600 1.000 (0.912–1.000) 0.714 (0.585–0.816) 0.000 (0.000–0.138) 0.286 (0.184–0.415)

Table 33: Post-hoc judge calibration packet coverage and judge-human agreement. Human labels calibrate the automated gate after generation and are not used as a generation gate.

Public Artifacts

The public code repository is available at https: //github.com/suraj-ranganath/ PIPE-Cypher. The public-proxy benchmark exports for FinBench/SNB and ICIJ Offshore Leaks are available at https://huggingface. co/datasets/suraj-ranganath/ PIPE-Cypher-benchmarks.

Representative Accepted Examples

The examples below are selected in stable identifier order from tracked evidence snapshots, one per graph/category cell when available. They show the NL question, accepted Cypher, structural tags, gate status, and a bounded execution-result sample. ICIJ examples are redacted before rendering, preserving query structure while removing value-bearing strings. 1. FinBench / Boolean Existence / easy. Question: Does account ’187743809466009406’ have any outgoing transfer? MATCH (src:Account {accountId: ’187743809466009406’}) -[:TRANSFER_TO]-> (:Account) RETURN DISTINCT COUNT(DISTINCT src) > 0 AS HasOutgoingTransfer

Structure: single_hop, aggregation. Relationships: TRANSFER_TO.

Relationships: OWN_ACCOUNT, TRANSFER_TO. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {AccountId: 4687402787162554854, AccountType: credit card, IsBlocked: False}; observed rows: 3 4. FinBench / Negation Difference / medium. Question: Which accounts owned by person ’Kant’ have not sent any transfers? MATCH (p:Person {personName: ’Kant’}) -[:OWN_ACCOUNT]-> (a:Account) WHERE NOT (a) -[:TRANSFER_TO]-> (:Account) RETURN DISTINCT a.accountId AS AccountId, a.accountType AS AccountType, a.isBlocked AS IsBlocked LIMIT 300

Structure: bounded_result.

join_heavy,

negation,

Gates: RO/Syn/Schema/Exec/Judge.

Relationships: OWN_ACCOUNT, TRANSFER_TO.

Result sample: {HasOutgoingTransfer: True}; observed rows: 1

Gates: RO/Syn/Schema/Exec/Judge.

2. FinBench / Complex Aggregation / medium. Question: What is the total transferred amount from accounts owned by person ’Gwar’? MATCH (p:Person {personName: ’Gwar’}) -[:OWN_ACCOUNT]-> (src:Account)-[t:TRANSFER_TO]-> (:Account) RETURN DISTINCT SUM(t.amount) AS TotalTransferredAmount

Structure: join_heavy, aggregation. Relationships: OWN_ACCOUNT, TRANSFER_TO. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {TotalTransferredAmount: 14972318.41}; observed rows: 1 3. FinBench / Complex Retrieval / easy. Question: Which accounts received transfers from accounts owned by person ’Zof’? MATCH (p:Person {personName: ’Zof’}) -[:OWN_ACCOUNT]-> (src:Account) -[:TRANSFER_TO]-> (dst:Account) RETURN DISTINCT dst.accountId AS AccountId, dst.accountType AS AccountType, dst.isBlocked AS IsBlocked LIMIT 300

Structure: join_heavy, bounded_result.

Result sample: {AccountId: 4739757132830738341, AccountType: merchant account, IsBlocked: False}; observed rows: 1 5. FinBench / Path Temporal / medium. Question: Which accounts can receive money within two transfer hops from accounts owned by person ’Sossamon’? MATCH (p:Person {personName: ’Sossamon’}) -[:OWN_ACCOUNT]-> (src:Account) -[:TRANSFER_TO*1..2]-> (dst:Account) RETURN DISTINCT dst.accountId AS AccountId, dst.accountType AS AccountType, dst.isBlocked AS IsBlocked LIMIT 300

Structure: join_heavy, path, bounded_result. Relationships: OWN_ACCOUNT, TRANSFER_TO. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {AccountId: 4687402787162554854, AccountType: credit card, IsBlocked: False}; observed rows: 4 6. FinBench / Ranking Topk / medium. Question: For accounts owned by person ’Barry’, which account sent the highest total transfer amount?

MATCH (p:Person {personName: ’Barry’}) -[:OWN_ACCOUNT]-> (src:Account)-[t:TRANSFER_TO]-> (:Account) WITH src, SUM(t.amount) AS totalAmount RETURN DISTINCT src.accountId AS AccountId, src.accountType AS AccountType, src.isBlocked AS IsBlocked, totalAmount ORDER BY totalAmount DESC LIMIT 1

Structure: join_heavy, aggregation, order_rank, bounded_result. Relationships: OWN_ACCOUNT, TRANSFER_TO. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {AccountId: 4732438783436260772, AccountType: internet account, IsBlocked: False}; observed rows: 1 7. FinBench / Simple Aggregation / easy. Question: How many accounts are owned by person ’Kaewsuktae’? MATCH (p:Person {personName: ’Kaewsuktae’}) -[:OWN_ACCOUNT]-> (a:Account) RETURN DISTINCT COUNT(DISTINCT a) AS AccountCount

9. ICIJ Offshore Leaks / Boolean Existence / medium. Question: Does offshore entity ’ENTITY_VALUE_1’ have a registered address? MATCH (e:Entity {name: ’ENTITY_VALUE_1’}) OPTIONAL MATCH (e) -[:registered_address]-> (addr:Address) RETURN DISTINCT COUNT(addr) > 0 AS HasRegisteredAddress

Structure: single_hop, aggregation, optional. Relationships: registered_address. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {HasRegisteredAddress: True}; observed rows: 1 10. ICIJ Offshore Leaks / Complex Aggregation / easy. Question: How many distinct officers are connected to entities in jurisdiction ’JURISDICTION_VALUE_1’? MATCH (o:Officer) -[:officer_of]-> (e:Entity {jurisdiction: ’JURISDICTION_VALUE_1’}) RETURN DISTINCT COUNT(DISTINCT o) AS OfficerCount

Structure: single_hop, aggregation. Relationships: officer_of.

Structure: single_hop, aggregation.

Gates: RO/Syn/Schema/Exec/Judge.

Relationships: OWN_ACCOUNT.

Result sample: {OfficerCount: 2}; observed rows: 1

Gates: RO/Syn/Schema/Exec/Judge. Result sample: {AccountCount: 1}; observed rows: 1 8. FinBench / Simple Retrieval / easy. Question: Which accounts are owned by person ’Barry’? MATCH (p:Person {personName: ’Barry’}) -[:OWN_ACCOUNT]-> (a:Account) RETURN DISTINCT a.accountId AS AccountId, a.accountType AS AccountType, a.isBlocked AS IsBlocked LIMIT 300

11. ICIJ Offshore Leaks / Complex Retrieval / easy. Question: Which officers share a registered address with offshore entity ’ENTITY_VALUE_1’? MATCH (e:Entity {name: ’ENTITY_VALUE_1’}) -[:registered_address]-> (addr:Address) <-[:registered_address]- (o:Officer) RETURN DISTINCT o.node_id AS OfficerId, o.name AS OfficerName, addr.address AS RegisteredAddress LIMIT 300

Structure: single_hop, bounded_result.

Structure: join_heavy, bounded_result.

Relationships: OWN_ACCOUNT.

Relationships: registered_address.

Gates: RO/Syn/Schema/Exec/Judge.

Result sample: {AccountId: 4732438783436260772, AccountType: internet account, IsBlocked: False}; observed rows: 1

Result sample: {OfficerId: OFFICER_ID_1, OfficerName: OFFICER_NAME_1, RegisteredAddress: ADDRESS_1}; observed rows: 8

12. ICIJ Offshore Leaks / Negation Difference / easy. Question: Which offshore entities in jurisdiction ’JURISDICTION_VALUE_1’ do not have a registered address? MATCH (e:Entity {jurisdiction: ’JURISDICTION_VALUE_1’}) WHERE NOT (e) -[:registered_address]-> (:Address) RETURN DISTINCT e.node_id AS EntityId, e.name AS EntityName, e.jurisdiction AS Jurisdiction LIMIT 300

Structure: bounded_result.

single_hop,

negation,

Relationships: registered_address. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {EntityId: ENTITY_ID_1, EntityName: ENTITY_NAME_1, Jurisdiction: JURISDICTION_1}; observed rows: 2 13. ICIJ Offshore Leaks / Path Temporal / medium. Question: Which officers share offshore entities with officer ’OFFICER_VALUE_1’, and when did each connection start? MATCH (src:Officer)-[srcRel:officer_of]-> (entity:Entity) <-[dstRel:officer_of](dst:Officer) WHERE trim(src.name) = ’OFFICER_VALUE_1’ AND dst <> src RETURN DISTINCT dst.node_id AS OfficerId, dst.name AS OfficerName, entity.name AS SharedEntityName, dstRel.start_date AS ConnectionStartDate LIMIT 300

Structure: node_scan, aggregation, order_rank, negation, bounded_result. Relationships: none. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {entityCount: 209634, jurisdiction: JURISDICTION_1}; observed rows: 10 15. ICIJ Offshore Leaks / Simple Aggregation / easy. Question: How many offshore entities are connected to officer ’OFFICER_VALUE_1’? MATCH (o:Officer) -[:officer_of]-> (e:Entity) WHERE trim(o.name) = ’OFFICER_VALUE_1’ RETURN DISTINCT COUNT(DISTINCT e) AS OffshoreEntityCount

Structure: single_hop, aggregation. Relationships: officer_of. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {OffshoreEntityCount: 1}; observed rows: 1 16. ICIJ Offshore Leaks / Simple Retrieval / easy. Question: Which offshore entities is officer ’OFFICER_VALUE_1’ connected to? MATCH (o:Officer)-[r:officer_of]-> (e:Entity) WHERE trim(o.name) = ’OFFICER_VALUE_1’ RETURN DISTINCT e.node_id AS EntityId, e.name AS EntityName, e.jurisdiction AS Jurisdiction, r.link AS Link LIMIT 300

Structure: single_hop, bounded_result. Structure: bounded_result.

join_heavy,

negation,

Relationships: officer_of. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {ConnectionStartDate: DATE_1, OfficerId: OFFICER_ID_1, OfficerName: OFFICER_NAME_1}; observed rows: 15 14. ICIJ Offshore Leaks / Ranking Topk / medium. Question: Which jurisdictions have the most offshore entities? MATCH (e:Entity) WHERE e.jurisdiction IS NOT NULL WITH e.jurisdiction AS jurisdiction, COUNT(DISTINCT e) AS entityCount RETURN DISTINCT jurisdiction, entityCount ORDER BY entityCount DESC LIMIT 10

Relationships: officer_of. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {EntityId: ENTITY_ID_1, EntityName: ENTITY_NAME_1, Jurisdiction: JURISDICTION_1}; observed rows: 1 17. SNB / Boolean Existence / medium. Question: Does person with id 6597069766828 like any post? MATCH (p:Person {id: 6597069766828}) OPTIONAL MATCH (p) -[:LIKES]-> (post:Post) RETURN DISTINCT COUNT(post) > 0 AS LikesAnyPost

Structure: single_hop, aggregation, optional. Relationships: LIKES. Gates: RO/Syn/Schema/Exec/Judge.

Result sample: {LikesAnyPost: True}; observed rows: 1 18. SNB / Complex Aggregation / medium. Question: How many distinct posts are in forums joined by person with id 6597069766845? MATCH (forum:Forum) -[:HAS_MEMBER]-> (p:Person {id: 6597069766845}) MATCH (forum) -[:CONTAINER_OF]-> (post:Post) RETURN DISTINCT COUNT(DISTINCT post) AS JoinedForumPostCount

Structure: join_heavy, aggregation. Relationships: HAS_MEMBER.

CONTAINER_OF,

Gates: RO/Syn/Schema/Exec/Judge. Result sample: {JoinedForumPostCount: 1}; observed rows: 1 19. SNB / Complex Retrieval / easy. Question: Which people are members of forums containing posts tagged ’Manuel_Noriega’? MATCH (forum:Forum) -[:HAS_MEMBER]-> (p:Person), (forum) -[:CONTAINER_OF]-> (post:Post) -[:HAS_TAG]-> (tag:Tag {name: ’Manuel_Noriega’}) RETURN DISTINCT p.id AS PersonId LIMIT 200

MATCH (src:Person {id: 4398046511136}) -[:KNOWS*1..2]-> (dst:Person) RETURN DISTINCT dst.id AS PersonId, dst.firstName AS FirstName, dst.lastName AS LastName LIMIT 200

Structure: single_hop, path, bounded_result. Relationships: KNOWS. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {FirstName: Rafael, LastName: Fernández, PersonId: 4398046511333}; observed rows: 25 22. SNB / Ranking Topk / medium. Question: Which city records are linked from the most organisation records through :IS_LOCATED_IN? MATCH (s:Organisation) -[:IS_LOCATED_IN]-> (e:City) WITH e, COUNT(DISTINCT s) AS relatedCount RETURN DISTINCT e.id AS TargetId, e.name AS TargetName, relatedCount ORDER BY relatedCount DESC LIMIT 10

Structure: single_hop, aggregation, order_rank, bounded_result. Relationships: IS_LOCATED_IN.

Structure: join_heavy, bounded_result.

Gates: RO/Syn/Schema/Exec/Judge.

Relationships: CONTAINER_OF, HAS_MEMBER, HAS_TAG.

Result sample: {TargetId: 164, TargetName: Kolkata, relatedCount: 130}; observed rows: 10

Gates: RO/Syn/Schema/Exec/Judge. Result sample: {PersonId: 4398046511220}; observed rows: 19 20. SNB / Negation Difference / easy. Question: Which person records are not linked from any message record through :HAS_CREATOR? MATCH (p:Person) WHERE NOT EXISTS((:Message) -[:HAS_CREATOR]-> (p)) RETURN DISTINCT p.id AS PersonId, p.firstName AS PersonFirstName, p.lastName AS PersonLastName

23. SNB / Simple Aggregation / easy. Question: How many posts are tagged ’Vietnam’? MATCH (post:Post) -[:HAS_TAG]-> (tag:Tag {name: ’Vietnam’}) RETURN DISTINCT COUNT(DISTINCT post) AS PostCount

Structure: single_hop, aggregation. Relationships: HAS_TAG.

Structure: single_hop, negation.

Gates: RO/Syn/Schema/Exec/Judge.

Relationships: HAS_CREATOR.

Result sample: {PostCount: 1}; observed rows: 1

Gates: RO/Syn/Schema/Exec/Judge. Result sample: {PersonFirstName: R., PersonId: 8796093022313, PersonLastName: Rao}; observed rows: 25 21. SNB / Path Temporal / easy. Question: Which people are within two knows hops of person with id 4398046511136?

24. SNB / Simple Retrieval / easy. Question: Which post IDs did person with id 4398046511124 like? MATCH (p:Person {id: 4398046511124}) -[:LIKES]-> (post:Post) RETURN DISTINCT post.id AS PostId LIMIT 200

Structure: single_hop, bounded_result. Relationships: LIKES. Gates: RO/Syn/Schema/Exec/Judge. Result sample: {PostId: 343597385744}; observed rows: 5

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Related documents