FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings

FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings Sasan Mansouri∗1 , Daniel Saad2 , Mark Wahrenburg2 , Manu Weissel2,3 , and Fabian Woebbeking4 1

University of Groningen, Groningen, Netherlands Goethe University Frankfurt, Frankfurt am Main, Germany 3 DataNXT GmbH, Frankfurt am Main, Germany 4 Halle Institute for Economic Research (IWH) and Martin Luther University Halle-Wittenberg, Halle (Saale), Germany 2

arXiv:2608.07400v1 [cs.AI] 7 Aug 2026

Abstract

document corpora (Lewis et al., 2020; Karpukhin et al., 2020), financial document question answering presents a fundamental challenge distinct from open-domain settings: regulatory disclosures are heavily templated. Standardized accounting conventions, risk-factor disclosures, and revenue-recognition notes read almost identically across competing firms and reporting periods. Consequently, the primary bottleneck in automated financial analysis is rarely answer composition, but evidence discrimination—distinguishing the target firm’s disclosure from plausible distractors, such as a competitor’s near-identical boilerplate or a prior period’s filing. For an analyst, auditor, or compliance reviewer, an ungrounded or misattributed answer carries limited practical utility and introduces severe operational risk; yet existing financial benchmarks (Z. Chen, W. Chen, et al., 2021; Zhu et al., 2021; Reddy et al., 2024) focus primarily on numerical calculation over provided snippets or end-to-end correctness, failing to isolate evidence retrieval and hard-negative suppression. Beyond boilerplate templating, document length and structural complexity pose additional hurdles. A single 10K filing frequently spans hundreds of pages of dense prose, financial tables, and footnotes, mixing accounting, legal, and forward-looking language that requires domain expertise to interpret. Furthermore, answers frequently depend on evidence spread across non-adjacent tables, footnotes, and narrative sections rather than a single paragraph. General-domain QA benchmarks (e.g., Rajpurkar et al., 2016; Kwiatkowski et al., 2019), by contrast, treat supporting evidence as a secondary annotation—inverting the requirements of financial domain QA, where an answer’s validity depends on strict provenance to its underlying disclosure. To address these challenges, we introduce FinRank, an evidence-grounded benchmark consisting of 1185 question– answer records manually authored over the 10-K and 10-Q filings of 22 companies across three sectors (pharmaceuticals, oil and gas, and automotive) covering filing years 2024–2025. Each record pairs a question and reference answer with gold supporting passages, filing and question-level metadata (topic, difficulty, reasoning type, evidence scope), and a hand-curated set of hard-negative

Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence. Similar facts and disclosures recur across sections of a filing, across reporting periods of the same firm, and across comparable firms. FinRank targets this provenance-sensitive retrieval problem by requiring systems to identify evidence for the intended entity, reporting period, and disclosure context. The benchmark contains 1185 manually authored question–answer records over the 10-K and 10-Q filings of 22 companies. Each record includes a reference answer, gold supporting passages, and hand-curated hard negatives drawn from confusable passages within filings, across reporting periods, and across comparable firms. FinRank evaluates passage retrieval, reranking, and hard-negative discrimination as separately measured tasks. Baseline results demonstrate the difficulty of this setting: among the evaluated systems, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billionparameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points, and pairwise accuracy falls by 13.0–20.5 percentage points when random negatives are replaced with the curated hard negatives. FinRank provides an evidence-first benchmark for developing financial question answering systems that are not only accurate but also grounded in the correct disclosure.

1

Introduction

Financial disclosures submitted to the U.S. Securities and Exchange Commission (SEC), such as Form 10-K and 10-Q filings, serve as the primary source of truth for corporate financial analysis, auditing, and regulatory compliance (Wu et al., 2023; Islam et al., 2023). While recent advances in retrieval-augmented generation (RAG) have enabled large language models (LLMs) to query extensive ∗ Corresponding author: s.mansouri@rug.nl

1

passages drawn from comparable filings. In a realistic and proactive-conversational reasoning. In all of these analyst workflow, a model must identify the passages that the supporting context is supplied, so retrieval is not evalactually support an answer, distinguish them from plausi- uated. A second line makes retrieval part of the task: ble distractors, and produce a faithful response; FinRank FinanceBench (Islam et al., 2023) scores open-book anevaluates the ranking and discrimination steps directly, swer correctness over whole filings; DocFinQA (Reddy et over a curated passage pool. Its construction criteria al., 2024) lengthens FinQA to full-document context; Fin(Section 3.2; Appendix A) jointly control topic coverage, TextQA (J. Chen et al., 2024) and T2-RAGBench (Strich qualitative versus quantitative reasoning, complexity, ev- et al., 2026) benchmark end-to-end RAG over financial idence scope, and sub-question decomposition, thereby prose and tables; and FinDER (Choi et al., 2025) conensuring that the benchmark evaluates financial reasoning tributes expert-written, abbreviation-heavy queries with and hard-negative discrimination rather than superficial annotated evidence, scored with RAGAS-style metrics lexical matching alone. FinRank is designed specifically (Es et al., 2024). The most recent entries extend the setfor retrieval-grounded financial QA over selected SEC ting further still, to agentic document- and chunk-ranking filings; its scope is defined by the included corporate dis- (Choi, Kwon, et al., 2025), multimodal visual citation (S. closures, and it is not intended for generating automated Zhao et al., 2025), and analyst-workflow evaluation across investment advice, valuations, or trading signals. within-disclosure, cross-entity, and longitudinal pathways In summary, our contributions are fourfold. First, we re- (Jiang et al., 2026). FinDER itself originated as the namelease FinRank1 as, to our knowledge, the first financial QA sake task of the ICAIF’24 FinanceRAG Challenge (Choi, benchmark to release a dedicated set of human-selected, Sohn, et al., 2024), which unified seven of these datasets semantically confusable hard negatives for each question, into a single NDCG@10 retrieval leaderboard. Beyond enabling direct comparison of discrimination against cu- document-centric RAG, recent work explores bypassing rated versus random distractors, complementing resources retrieval altogether by giving models direct access to cuthat focus on realistic queries (Choi et al., 2025) or open- rated vendor data through the Model Context Protocol, book correctness (Islam et al., 2023) without releasing which is effective for quantitative financial QA but weaker gold hard negatives. Second, we provide rich, stratifi- on qualitative questions (Mansouri et al., 2026); FinRank able annotations across four axes (document type, reason- targets the retrieval-based setting that remains necessary ing type, evidence scope, and difficulty) alongside gold when answers must be grounded in the filings themselves sub-question decompositions (query rewrite) for 69% of rather than a pre-structured data feed. records, enabling performance to be disaggregated along Across this body of work, evidence annotation identifies fine-grained dimensions obscured by aggregate metrics. what is relevant — as gold passages or, in FinAgentBench’s Third, we establish reference retrieval and reranking base- case, graded chunk relevance — but no benchmark ships, lines spanning sparse (TF-IDF, BM25), dense (bi-encoder), per question, a dedicated set of human-selected, semanand cross-encoder architectures, demonstrating through tically confusable hard-negative passages against which hard-versus-random negative contrasts that curated dis- discrimination can be scored directly against random distractors degrade model accuracy by 13.0–20.5 percentage tractors. FinRank fills exactly this gap (Table 1). Each points. Fourth, we support transparency, auditability, and question carries gold supporting passages and humanextensibility through documented construction criteria, selected distractors from comparable filings, together with a deterministic label-normalization and repair pipeline difficulty, reasoning-type, and evidence-scope metadata with a machine-readable change log, per-passage identi- and, for 69% of records, a gold sub-question decomposition, fiers and text hashes, a hard-negative taxonomy, and a so that retrieval, reranking, and hard-negative discriminatarget-versus-realized distributional audit. tion become separately measurable. The nearest resource

2

Related Work

2.1

Financial QA and RAG Benchmarks

with curated hard negatives, DocReRank (Wasserman et al., 2025), generates hard-negative queries to train page-image rerankers, and is neither a QA benchmark nor passage-level. Concurrent method-side work reaches the same diagnosis from the systems direction: FinCARDS (Zhou et al., 2026) recasts financial evidence selection as constraint satisfaction over entities, metrics, fiscal periods, and numeric spans, reranking within-filing BM25 candidates on FinAgentBench (Choi, Kwon, et al., 2025) and reporting lexical retrieval to be vulnerable to numeric drift, temporal misalignment, and boilerplate repetition. FinCARDS contributes a method and releases code but no evaluation data or negatives, and confines itself to intra-document retrieval within a single filing, noting that its effectiveness across multiple documents remains unevaluated. FinRank is complementary on both counts: it

Early financial QA benchmarks target numerical reasoning over a provided context. FinQA (Z. Chen, W. Chen, et al., 2021) pairs questions with executable reasoning programs over S&P 500 earnings-report excerpts; TATQA (Zhu et al., 2021) covers hybrid table–text reasoning; ConvFinQA (Z. Chen, Li, et al., 2022) extends FinQA to multi-turn dialogue; and MultiHiertt (Y. Zhao et al., 2022) and PACIFIC (Deng et al., 2022) push multi-table 1 Dataset, evaluation harness, repair log, and hard-negative taxonomy: https://github.com/datanxt/FinRank

2

supplies the annotated positives and curated distractors 3.2 Question-Generation Criteria that such constraint-aware rerankers are scored against, and its hard negatives are drawn predominantly from other The annotators worked from a shared criteria document filings, targeting precisely the cross-document setting left that fixes what a well-formed FinRank record must contain; we summarize it here and reproduce it in full in open there. Appendix A. The criteria define the unit of evidence (a passage: a self-contained span of filing text), separate 2.2 Retrieval, Reranking, and Hard Neg- qualitative reasoning (interpretation and judgment) from quantitative reasoning (numerical computation, itself split atives into metrics-generated, single-step, and compositional calBM25 (Robertson and Zaragoza, 2009) remains a strong culations), and require, for medium- and hard-difficulty sparse baseline; dense retrievers (Karpukhin et al., 2020; questions, a query rewrite that decomposes the question Reimers and Gurevych, 2019; Izacard et al., 2022) and into an ordered sequence of atomic sub-questions. cross-encoder rerankers (Nogueira and Cho, 2019) form the Coverage is controlled along three axes with explicit standard two-stage pipeline evaluated by general IR bench- targets. Topic: questions span eight filing-relevant domarks such as BEIR (Thakur et al., 2021) and MTEB mains (Company Overview, Financials, Footnotes, Gover(Muennighoff et al., 2023), with FinMTEB (Tang and nance, Accounting, Legal, Risk, and Shareholder Return), Yang, 2025) the financial counterpart. A long line of work each mapped to a distinct set of 10-K/10-Q disclosure shows that hard negatives (not random in-batch ones) are items. Complexity: a 30%/40%/30% Easy/Medium/Hard what most improve dense retrievers (Xiong et al., 2021; split, where Easy questions are direct factual lookups, Qu et al., 2021; Zhan et al., 2021). FinRank carries this Medium questions require comparison or trend analysis insight to the evaluation side: rather than mining hard across sections, and Hard questions require multi-section negatives to train a retriever, it releases human-curated synthesis. Evidence scope: a 40%/30%/30% single-/twohard negatives as a fixed benchmark asset and reports how /multi-passage split, so the benchmark is not dominated much they degrade ranking relative to random distractors by single-paragraph lookups. The criteria additionally in(Section 7). struct annotators to balance qualitative and quantitative questions. Section 4.2 reports how closely the released dataset meets each target. 2.3 Attribution and Faithfulness Fixing these criteria ex ante is what lets FinRank test Retrieval-augmented generation (Lewis et al., 2020) can more than surface lexical matching: a system must locate still hallucinate or misattribute, motivating metrics for the disclosure passages that support the reasoning type a citation quality (Gao et al., 2023), reference-free RAG question demands, not merely passages that share its voevaluation (Es et al., 2024), and atomic factual precision cabulary, and, for multi-passage questions, must integrate (Min et al., 2023). These score generated text; FinRank evidence spread across non-adjacent filing sections. is complementary, measuring the retrieval and reranking steps upstream through gold-versus-hard-negative passage labels, so that evidence attribution can be assessed before 3.3 Record Schema generation is introduced. The dataset is distributed as a single .jsonl file, one JSON object per line. Each record is flat at the top level, with the question, reference answer, optional 3 Dataset Construction query rewrite array, question- and filing-level metadata, and two nested arrays: passages (supporting passages, 3.1 Sources and Design Principles each with text and page number) and hard negatives (each additionally tagged with its originating ticker, FinRank aggregates 1185 records, each grounded in a 10-K year, and doc type). The full field-by-field schema is (annual) or 10-Q (quarterly) filing submitted to the SEC. given in Appendix B. The records were collected by trained annotators, each responsible for roughly 250 records, whose assignments together span the pharmaceutical, oil-and-gas, and automotive sectors (Section 3.5). The benchmark is organized around four principles: examples are grounded in primary disclosures rather than secondary summaries; every question is paired with supporting passages for evidencebased evaluation; hard negatives are included to support retrieval and reranking; and metadata exposes variation across sector, topic, reasoning type, filing year, and document type, so that performance can be analyzed beyond aggregate scores.

3.4

Supporting Passages and Hard Negatives

The passages array contains the passages from the underlying SEC filing that support the reference answer, each with a text field and a page number field; some questions are answered by a single passage, others require evidence drawn from multiple parts of the same filing. Supporting passages thus play two roles: they are ground-truth retrieval targets, and they constrain answer generation 3

Benchmark

Source documents

# Records

Retrieval

Hard neg.

Reranking

Rich metadata

FinQA (Z. Chen, W. Chen, et al., 2021) TAT-QA (Zhu et al., 2021) ConvFinQA (Z. Chen, Li, et al., 2022) FinanceBench (Islam et al., 2023) DocFinQA (Reddy et al., 2024) FinDER (Choi et al., 2025) FinAgentBench (Choi, Kwon, et al., 2025)

Earnings reports (S&P 500) Report table/text snippets Earnings reports (dialogue) 10-K/10-Q/8-K/earnings Full 10-K documents 10-K (S&P 500) 10-K/10-Q/8-K, calls, DEF-14A

8,281 16,552 14,115 10,231† 7,437 5,703 26,000§

– – – partial yes yes yes

– – – – – – –

– – – – – yes‡ yes§

– – – sector – topic –

FinRank (ours)

10-K/10-Q

1185

yes

yes

yes

full

Table 1: FinRank against representative financial QA and retrieval benchmarks. “Hard neg.” denotes per-question curated hard-negative passages; “Reranking” a gold-vs.-hard-negative reranking task; “Rich metadata” per-record difficulty, reasoning-type, and evidence-scope labels. † 150 questions are open-sourced. ‡ FinDER evaluates LLM reranking of retrieved passages but releases no curated hard-negative candidate set. § FinAgentBench provides 26K graded-relevance annotations for document- and chunk-ranking, without a per-question curated hard-negative set. To our knowledge, FinRank is the first financial QA benchmark to release per-question curated hard negatives and to score discrimination against curated versus random distractors directly. by providing the evidence to which a faithful generator should be grounded. Each entry in hard negatives additionally records the originating filing’s ticker, year, and doc type, so distractors may come from another company, another year, or a different filing type than the supporting evidence. We treat hard negatives as a first-class benchmark asset rather than as optional augmentation: the dataset is intended for explicit comparison of retrievers and rerankers on their ability to suppress these distractors.

3.5

comparable filings, most often a competitor in the same sector, and otherwise the same company in a different reporting period or filing type. Because hard negatives were chosen by a human reader searching for the most confusable evidence rather than sampled by a retrieval heuristic, they concentrate in the “same industry, different company” bucket that dominates Figure 1. This distribution is a deliberate annotation choice, and it is precisely the configuration that most often confuses lexical and embedding-based retrievers (Section 7). During collection the authors performed a sampled review of each student’s records and returned corrective feedback, so that systematic misreadings of the criteria could be caught before release. This review was applied to a sample rather than exhaustively, and we did not compute a formal inter-annotator-agreement statistic; both points are recorded as limitations in Section 9. After collection we applied a single deterministic normalization pass (Section 3.8; Appendix C) that harmonizes label surface forms, recovers the small number of placeholder values deterministically from the structured question id, and repairs transposed hard-negative metadata, preserving every raw value in a parallel <field> raw field.

Data Collection

FinRank was collected by five business students, each independently responsible for a set of companies within one of the three sectors, together spanning the pharmaceutical, oil-and-gas, and automotive sectors. All annotators worked from a shared question-generation criteria document (Section 3.2; reproduced in Appendix A) that fixes the topic taxonomy, the qualitative and quantitative reasoning categories, the complexity-level split, the passage-coverage split, and the query-rewrite rules. Every record was authored manually: students read the underlying 10-K and 10-Q filings directly and wrote each question, its reference answer, and, for medium- and hard-difficulty questions, the query rewrite decomposition into sub-questions by hand. No language model was used to generate questions, answers, or rewrites; the templated question id strings (for example, JNJ CompanyOverview Hard MultiPassage Quantitative 03) are the naming convention students followed from the criteria document, not an artifact of automated generation. Evidence was assembled by hand as well. For each question the responsible student located the supporting passage or passages in the source filing and transcribed them into the passages array with their page references. The student then selected the record’s hard negatives manually, drawing plausible-but-incorrect passages from

3.6

Retrieval Corpus Definitions

To make retrieval and reranking results on FinRank reproducible, we specify two evaluation regimes that are both reproducible from the JSONL alone, with no additional corpus required. Global pooled corpus C. C is the union of every non-empty supporting-passage text and hard-negative text appearing anywhere in FinRank, deduplicated on the raw passage text. Each entry is tagged with the (ticker, year, doc type, page number) metadata of the first record in which it appeared. For the released split, |C| = 5230 unique passages drawn from 1185 records, with on average 1.96 supporting passages and 5.08 hard nega4

tives per record. First-stage retrieval baselines (Section 7) 79.9% Same industry, rank all of C for each query and report Recall@k, MRR, different company n = 4,835 and nDCG@10 against the per-record gold positives. We 13.2% Same company, emphasize what this regime is: because C consists solely different year/form n = 796 of annotator-selected positives and curated distractors, re6.9% Same company, trieval over C is a curated passage-ranking stress test with same year & form n = 420 an elevated density of confusable passages, not retrieval Cross-industry <0.1% through the complete text of each filing. Scores on C are n=1 therefore not comparable to full-document retrieval set% of hard negatives (N = 6,052) tings, and extending FinRank with exhaustive filing-level chunks is left to a future release. Figure 1: Taxonomy of the 6021 hard negatives, by the relationship between each negative’s originating filing and In-record candidate set Lr . For each record r, its source record. A hard negative is most often a passage Lr = Gr ∪ HNr is the union of r’s supporting pas- from a competitor’s filing of comparable type and period, sages and its hard negatives. Reranking and hard- the configuration retrievers are most likely to confuse with negative-discrimination evaluations are restricted to Lr true evidence. and report MRR, nDCG@5, and pairwise accuracy on (positive, hard negative) pairs. This isolates ranking quality from first-stage recall and matches how hard negatives inline in a parallel <field> raw field, every change is enumerated in the released machine-readable repair log, and are intended to be used (Section 3.4). Studies using FinRank should state which regime is all distributional statistics in this paper are reported on being reported for each metric, and any corpus extension the canonical labels. beyond C (for example, a larger filing-derived pool) should be released alongside the results.

4

3.7

Dataset Analysis

Hard-Negative Construction Taxon- We summarize the composition of FinRank on the canonical normalized labels (Section 3.8), verified directly against omy

the released FinRank.jsonl. Figure 2 gives the full picEach hard negative carries the (ticker, year, ture along six axes; per-field distribution tables and the doc type) of its originating filing, so comparing it against raw-vs.-normalized counts are provided in Appendix C. the source record yields the four-bucket taxonomy in The 1185 records come from 22 companies in three Figure 1. Because negatives were selected by hand sectors: pharmaceuticals (499 records), oil and gas (438), (Section 3.5), 80% fall in the “same industry, different and automotive (248), with per-company counts ranging company” bucket: a competitor’s filing of comparable from 9 (XOM) to 142 (JNJ). Difficulty follows the intended type and period, the case retrievers are most likely to 30/40/30 Easy/Medium/Hard split (Figure 2b), and topics confuse with true evidence. A byte-equal audit identified are close to uniform (Figure 2f). Three axes are markedly 11 hard negatives that duplicated a supporting passage skewed: 88% of records come from 10-K filings and only of their own record; these, together with one degenerate 12% from 10-Qs; 75% of filings are from 2025 and 25% from entry, are removed from the release. The remaining 2024; and qualitative questions outnumber quantitative 442 hard negatives (∼7.3%) coincide with a supporting ones roughly two to one (68% vs. 32%). Evidence scope is passage of a different record, where the passage may spread across single- (34%), two- (40%), and multi-passage legitimately be relevant to more than one question. We (26%) records, so roughly two thirds of the benchmark release hn taxonomy.json so that users of hard-negative require integrating evidence from more than one part of a metrics can filter these overlaps. filing. These skews mean that aggregate metrics on FinRank 3.8 Official Label Set and Normalization primarily reflect performance on the dominant strata (10K, 2025, qualitative, pharma and oil-and-gas). We thereTo make evaluation reproducible without forcing every fore recommend that retrieval and answer-generation metconsumer to re-derive the same mapping, the released rics be reported per stratum, by document type, filing FinRank.jsonl carries the official benchmark labels: a year, reasoning type, and sector, in addition to in aggredeterministic, idempotent curation pass (Appendix C) gate, and we note that the 2024 and 10-Q partitions are title-cases difficulty, consolidates industry variants small and produce noisier per-stratum estimates. into three sectors with the oil-and-gas sub-industry surfaced in a new field, unifies topic, passage type, and 4.1 Data Quality and Normalization reasoning type typos, reconciles the Ford/F ticker split, and recovers literal placeholders from the structured The data as collected contains surface-level label inconsisquestion id. Every value the pass changed is preserved tencies (casing and spelling variants and a small number 5

(a) Sector

(b) Difficulty

37% 499 438

31%

42%

Pharma O&G

(c) Reasoning type

41%

484

68%

29%

21%

363

Auto

Easy Medium Hard

248

338

(d) Passage type

(e) Document type

35%

1,048

40%

409 472

26%

304

88%

12%

137 Single

Accounting Company Overview Financials Risk Legal Shareholder Return Footnotes Governance

Two

Multi

10-K

10-Q

(f) Topic

809

32%

376 Qualitative Quantitative

1,185 records 22 companies / 3 sectors 8topics 158 (13.3%) 153 (12.9%) 152 (12.8%) 150 (12.7%) 146 (12.3%) 143 (12.1%) 142 (12.0%) 141 (11.9%)

Figure 2: Composition of FinRank across the 1185 records: (a) sector, (b) difficulty, (c) reasoning type, (d) evidence scope (passage type), (e) document type, and (f) topic. Counts are on the canonical normalized labels. The dataset is deliberately near-uniform over topics (141–158 records each) but skewed along sector, document type, and reasoning type, skews that motivate stratified reporting. of literal template placeholders) and, in ∼0.3% of hardnegative entries, transposed metadata fields. A deterministic, idempotent curation pass (Section 3.8; Appendix C) resolves these into the canonical labels of the released FinRank.jsonl; original values are preserved in parallel <field> raw fields, every change is enumerated in the released repair log; duplicate passages within a record, degenerate hard negatives, and annotation-tool artifacts embedded in passage text are removed; and sixty-five records are excluded from the release entirely (reliance on a non-filing source, placeholder or synthetic content, or duplicate question–answer pairs), each enumerated in the repair log. Records whose stored labels disagree with their structured question id are flagged in the repair log for the audit discussed in Section 9.

dataset (Table 2, on canonical normalized labels). The complexity split is met almost exactly. Passage coverage deviates more: two-passage records are overrepresented by about ten percentage points, drawn from the single- and multi-passage buckets. Reasoning type is the largest departure from the criteria, at roughly two qualitative records per quantitative one. Appendix H reports cross-tabulations showing that this qualitative skew is present at every difficulty level.

5

Tasks

FinRank supports five evaluation tasks that mirror the stages of an analyst workflow over SEC filings: locating candidate evidence (retrieval), distinguishing it from plausible distractors (reranking), integrating evidence across 4.2 Methodology-to-Dataset Validation passages (multi-passage reasoning), producing an answer Because the question-generation criteria of Section 3.2 faithful to that evidence (answer generation), and mapping specify explicit target distributions over difficulty, passage a colloquial query to a retrievable form (query rewriting). coverage, and reasoning type, we can measure realized Retrieval tasks operate on the global pooled corpus C deviations between the stated targets and the released defined in Section 3.6; reranking and hard-negative dis6

Dimension

Target

Realized

Deviation

Complexity Easy Medium Hard

30% 40% 30%

30.6% 40.8% 28.5%

+0.6 pp +0.8 pp −1.5 pp

Passage coverage Single-passage Two-passage Multi-passage

40% 30% 30%

34.5% 39.8% 25.7%

−5.5 pp +9.8 pp −4.3 pp

Reasoning type (methodology calls for balance) Qualitative balanced 68.3% Quantitative balanced 31.7%

skewed ( ∼ 2:1 ) skewed

Table 2: Target distributions mandated by the FinRank question-generation criteria (Section 3.2) against realized values in the canonical normalized release. The 30/40/30 complexity target is met to within about one percentage point per bucket; the 40/30/30 passage-coverage target deviates most, with two-passage records over-represented; and reasoning type shows the largest gap, with qualitative records outnumbering quantitative ones roughly two to one. Consumers should treat these realized values as the authoritative distribution and stratify results accordingly. crimination operate on the per-record candidate set Lr dependencies and the reporting checklist we recommend from the same section. Table 3 summarizes the inputs, for future studies. outputs, suggested metrics, and possible baselines for each task.

6.1

Baselines

TF-IDF (scikit-learn Citation and attribution evaluation. Reranking We run seven systems: TfidfVectorizer, sublinear TF, cosine scoring); with hard negatives extends naturally to RAG-style ciBM25 (rank-bm25 BM25Okapi, default parameters, tation evaluation: a system that produces an answer toRobertson and Zaragoza, 2009); a dense retriever gether with one or more cited passages can be scored on (all-mpnet-base-v2, Reimers and Gurevych, 2019, whether the cited passages overlap with the gold supportcosine over L2-normalized embeddings); and a crossing passages and whether they avoid the hard-negative set. encoder reranker (ms-marco-MiniLM-L-6-v2, Nogueira This is particularly relevant in financial settings where and Cho, 2019), applied both over the top-20 dense auditability of generated answers is a practical concern. candidates from the global pool C and over the inrecord set Lr ; a stronger general-purpose embedder Multi-passage records and the query rewrite (bge-large-en-v1.5); a finance-adapted embedfield. Records with passage type Two-Passage or der (FinLang/finance-embeddings-investopedia); Multi-Passage (Section 4) provide the substrate for the and a 7B instruction-tuned embedder multi-passage reasoning task. The query rewrite array, (e5-mistral-7b-instruct, encoded per its referwhere present, supplies reference reformulations that can ence implementation), all under a uniform 512-token be used as targets for the query-rewriting task or as para- truncation. We additionally report a metadata-filtered phrase inputs for retrieval-robustness analyses. BM25 baseline: BM25 restricted to the corpus entries whose (ticker, year, doc type) match the query record’s source filing, quantifying the effect of metadata 6 Experimental Protocol and pre-filtering before semantic ranking. Baselines left to future work include an SEC-filing-adapted retriever, a Baselines closed-book LLM, an open-book LLM with gold passages, We report executed retrieval, reranking, and hard-negative and a full RAG pipeline (Lewis et al., 2020). discrimination baselines using deterministic, publicly available models; generative answer-quality evaluation is left 6.2 Splits and Ablations to future work. All reported metrics are produced by the released evaluation harness (baselines/) applied to The baselines here are computed over the entire the released FinRank.jsonl and are exactly reproducible dataset as reference numbers. We additionally refrom these artifacts. All baselines are evaluated on the lease five generalization splits as record-ID assignfull 1185-record dataset using the canonical normalized ments (baselines/splits.json): random (948/118/119 labels, and we fix seed=42 for the only non-deterministic train/dev/test), held-out-ticker (910/140/135), leave-onecomponent (random-negative sampling). Appendix G lists sector-out folds, year (test = 2024, 296 records), and 7

Task Answer tion

Inputs

Output

Metrics

genera- Question; passages Natural-language (gold or retrieved) answer

Passage retrieval Question; corpus

EM, token F1, ROUGE-L, BERTScore, faithfulness vs. provided passages passage Ranked candidate Recall@k, MRR, passages nDCG@k, Hit Rate

Possible baselines Closed-book LLM; openbook LLM; RAG

TF-IDF, BM25 (Robertson and Zaragoza, 2009); DPR (Karpukhin et al., 2020); SBERT (Reimers and Gurevych, 2019); hybrid Reranking with Question; in-record Ranking over the MRR, nDCG@k, pair- Cross-encoder rerankers hard negatives positives and hard union wise accuracy on (positive, (Nogueira and Cho, negatives hard negative) pairs 2019) Multi-passage Question; multiple Answer integrat- EM, F1, ROUGE-L, Long-context LLMs; reasoning supporting passages ing all relevant pas- BERTScore, evidence multi-passage RAG; sages coverage question decomposition Query rewriting Original question One or more refor- Semantic preservation LLM-based and mulations (downstream answer template-based rewritagreement); retrieval ers improvement (Recall@k, MRR)

Table 3: Tasks supported by FinRank, with inputs, outputs, suggested evaluation metrics, and possible baselines. document type (test = 10-Q, 137 records). We recommend that future studies report under at least one, noting that the year and 10-Q test partitions are small and yield noisier estimates. We stratify results along document type, reasoning type, passage type, and difficulty (Section 7), and report two design-validating contrasts: a query-rewrite ablation (question alone vs. question concatenated with its query rewrite sub-questions) and a hard-vs.-random negative contrast that tests whether the released hard negatives are in fact harder than arbitrary distractors.

query record’s filing eliminates 92.9% of that record’s hard negatives and shrinks the pool to ≈173 candidates on average, lifting BM25 from 32.1 to 55.0 Recall@10 (MRR 23.8 → 41.9). Because pooled entries carry first-occurrence metadata (Section 3.6), the filter excludes at least one gold passage for 33 records and every gold passage for 2; occurrence-level source attribution is left to a future release. Two readings follow. First, metadata filtering is genuinely powerful, and deployed filing assistants should use it; results on FinRank’s unfiltered pool characterize the complementary regime in which filing attribution is unavailable or unreliable (noisy metadata, cross-filing ques7 Results tions, multi-company corpora). Second, even under this filter, nearly half of the gold evidence is missed at k = 10: The retrieval, reranking, and hard-negative discrimination within-filing semantic discrimination, the setting probed tables below report numbers actually produced by the base- by the same-filing and same-company hard negatives, is lines of Section 6.1 on the entire 1185-record dataset. Num- not eliminated by metadata filtering. bers are emitted by baselines/make macros.py from the JSON output of the run scripts, and any rerun automatically refreshes them. Answer generation and faithfulness 7.2 Reranking over the In-Record Candiare left to future work (Section 6). date Set Table 5 reports reranking on the per-record candidate set Lr = Gr ∪ HNr (Section 3.6), isolating ranking quality Table 4 and Figure 3(A) report first-stage retrieval over the from first-stage recall. global pooled corpus C (Section 3.6; |C| = 5230). Each score is the mean across queries against the per-record 7.3 Hard-Negative Discrimination gold positives. Figure 3(B) reports pairwise accuracy across all Metadata-filtered BM25. Because most hard nega- (positive, hard negative) pairs per record under the intives originate from a different filing, a system that knows record ranking, contrasted with the same metric against the target filing could pre-filter candidates on metadata random negatives drawn uniformly from the pooled corpus before ranking. We quantify this directly: restricting C C excluding the record’s own positives (Section 6.2, seed to entries whose (ticker, year, doc type) match the 42, one random negative per curated hard negative; tied

7.1

Retrieval over the Global Pool

8

Recall (%)

A Retrieval over the global pool — Recall@k 50 40 30 20 10 0

TF-IDF BM25 Dense (all-mpnet)

Fin-BGE bge-large

R@1

e5-mistral-7B Dense+CE rerank

R@5

R@10

R@20

Pairwise accuracy (%)

B Hardness premium — pairwise accuracy, hard vs. random negatives 100 75

Hard negatives

+14.2

91.9

+13.0

Random negatives

+19.3

+20.5

93.7

87.8

90.8

77.8

74.8

74.5

70.3

TF-IDF

BM25

Dense

Fin-BGE

+17.5

+16.0

+15.9

76.3

80.2

76.1

bge-large

e5-7B

Cross-enc.

93.8

96.2

92.0

50 25 0

Gap = random-negative accuracy − hard-negative accuracy. A larger gap means the released hard negatives are genuinely harder.

Figure 3: Executed baselines on FinRank. (A) First-stage retrieval over the global pool: Recall@k for the evaluated systems. Even the strongest, a 7B instruction-tuned embedder, reaches only 44.8% Recall@10, and sub-billionparameter dense encoders gain little over BM25. (B) The hardness premium: pairwise ranking accuracy of each model against its curated hard negatives versus random negatives. The 13.0–20.5 point gap is the empirical evidence that the released hard negatives are genuinely harder than arbitrary distractors. Model TF-IDF BM25 Dense (all-mpnet-base-v2) + CE rerank of mpnet top-20 Dense (finance-adapted BGE) Dense (bge-large-en-v1.5) Dense (e5-mistral-7b-instruct)

Recall@1 Recall@5 Recall@10 Recall@20 MRR nDCG@10 7.7 8.3 6.5 9.6 5.2 8.8 10.1

23.7 24.5 21.5 27.5 16.5 25.9 33.1

32.8 32.1 31.8 35.2 22.4 35.6 44.8

41.9 40.1 42.0 42.0 29.8 44.6 55.0

23.9 23.8 22.2 26.9 16.5 25.9 31.0

21.9 21.8 20.4 24.6 14.8 23.8 29.7

Table 4: Retrieval performance on the global pooled corpus C (|C| = 5230, 1185 queries). All values are percentages averaged over queries.

7.4

scores count for the positive, and records are averaged with equal weight). Every model ranks a positive above a random negative 88–96% of the time but above a curated hard negative only 70–80% of the time, a 13.0–20.5 point drop. This gap is the empirical hardness premium of the released hard negatives, and it is the single result that most directly justifies shipping them as a first-class benchmark asset.

Stratified Performance and Ablations

Table 6 reports retrieval and reranking metrics stratified along four axes (document type, reasoning type, passage type, difficulty). Per-stratum partition sizes are emitted alongside the metrics in the underlying ablations.json. Selected cells are reproduced below; the full stratified breakdown is available in baselines/results/ablations.json. 9

Model

MRR nDCG@5

TF-IDF BM25 Dense (all-mpnet-base-v2) Dense (finance-adapted BGE) Dense (bge-large-en-v1.5) Dense (e5-mistral-7b-instruct) Cross-encoder over Lr

80.4 78.2 78.6 72.3 79.1 83.6 79.8

77.9 75.4 74.7 70.2 76.5 80.5 76.6

Table 5: Reranking performance on the in-record candidate set Lr .

Query-rewrite ablation. Table 7 compares retrieval performance on the subset of records with non-empty query rewrite lists (819 records; see ablations.json) under two conditions: the raw question alone, and the raw question concatenated with all listed rewrites.

7.5

Answer Generation and Error Analysis

FinRank is designed to support answer-generation evaluation (closed-book, open-book with gold passages, and full RAG, scored with EM, F1, ROUGE-L, BERTScore, and citation-faithfulness metrics (Gao et al., 2023; Es et al., 2024)) and a qualitative error analysis that separates retrieval failures (no supporting passage in the top-k), reranking failures (a hard negative preferred to a supporting passage), generation failures on correctly retrieved passages, and attribution failures. Because these require a generative model and a human-validated or LLM-judged correctness protocol, we leave them to future work and report only the executed retrieval and reranking results above; the evaluation protocol is specified in Section 6 so that such studies remain comparable.

instruction tuning help where wrong-register domain adaptation hurts, and SEC-filing-specific adaptation remains open. Third, and most directly tied to the benchmark’s design, the released hard negatives cost every model 13.0– 20.5 points of pairwise accuracy relative to random distractors; this hardness premium is the single number that justifies shipping curated hard negatives as a first-class asset. Fourth, the stratified breakdown exposes where aggregate scores hide weakness: 10-Q, multi-passage, and quantitative records score lowest, which is precisely why we recommend per-stratum reporting. The 10-Q gap should be read descriptively rather than causally: all 10-Q records come from the oil and gas sector and predominantly from 2025 (Appendix H), so document type is confounded with sector, filing year, and annotator, and disentangling these effects requires the controlled splits of Section 6.2. Why evidence grounding matters in finance. In audit, compliance, and investor-relations settings the value of an answer is tied to its provenance: a reviewer needs to know not just what a system claims but where the claim comes from, so that the underlying disclosure can be inspected. FinRank operationalizes this by making the identification of supporting passages a first-class evaluation target and by pairing it with hard negatives that stress the boundary between genuinely supporting evidence and plausible distractors. Because those distractors are most often a competitor’s comparable filing (Section 3.7), the benchmark rewards systems that disambiguate between lexically and topically similar disclosures, the failure mode most likely to matter in practice. Multi-passage records add a complementary axis, since financial conclusions frequently combine an MD&A narrative, a footnote, and a tabular disclosure.

Scope. FinRank is intended for evaluating evidence retrieval, reranking, and (in future work) answer generation for financial-analyst assistants, compliance QA tools, and 8 Discussion SEC-filing search interfaces. It is not designed or validated What the baselines reveal. Four observations stand for automated investment decisions, valuation, or trading out from Section 7. First, ranking within a curated pool of signals, and any deployment in high-stakes settings should confusable filing passages is hard: the strongest evaluated retain human oversight. system, a 7B instruction-tuned embedder, reaches 44.8% Recall@10 over the pooled corpus C, leaving more than half of the gold evidence outside the top ten. Because C 9 Limitations contains only annotated positives and curated distractors rather than the full text of each filing, these numbers Scale and skew. FinRank contains 1185 records over measure discrimination among confusable disclosure pas- 22 companies and three sectors, enough for ranking and sages, not end-to-end retrieval over complete documents; answer-quality comparisons, but small relative to openfull-document retrieval remains untested here. Second, domain QA benchmarks, and skewed across sector, filing encoder capacity dominates domain labels on this heavily year (75% from 2025), document type (87% 10-K), and reatemplated text: sub-billion-parameter encoders earn little soning type (roughly 2:1 qualitative). Aggregate metrics (mpnet ties BM25; bge-large gains 3.5 Recall@10 points), therefore reflect the dominant strata, per-stratum subsets a finance-adapted embedder tuned on consumer-finance can be noisy, and performance should not be assumed to text trails BM25 by 9.7 points, while the 7B instruction- transfer to companies, sectors, jurisdictions, or periods tuned e5-mistral gains 12.7 points over BM25 — scale and outside the release. 10

Axis

Stratum

BM25 R@10 Dense R@10 CE rerank R@10

Document type

10-K 10-Q

33.9 18.0

33.7 17.3

36.9 21.8

Reasoning

Qualitative Quantitative

34.7 26.5

35.2 24.4

38.2 28.8

Passage type

Single-Passage Two-Passage Multi-Passage

44.7 28.0 21.5

42.3 28.8 22.2

46.9 31.7 24.8

Difficulty

Easy Medium Hard

37.6 32.4 25.9

33.7 32.5 28.6

39.0 35.3 31.0

Table 6: Recall@10 stratified by document type, reasoning type, passage type, and difficulty. Per-stratum sample sizes are available in ablations.json. this paper reports only retrieval and reranking baselines; generative answer-quality results are left to future work, so no claim about generator performance on FinRank follows from it.

Model / condition Recall@10 (raw) Recall@10 (with rewrites) TF-IDF BM25 Dense

32.3 29.9 30.7

41.1 38.7 34.1

Table 7: Query-rewrite ablation on the subset of records with non-empty query rewrite lists. Because rewrites are annotator-authored decompositions written with the gold evidence in view, the right column should be read as an oracle upper bound on decomposition-assisted retrieval. The right column concatenates the original question with all listed rewrites; the left uses the question alone.

Single-annotator review. Records were authored manually by the student annotators and reviewed on a sampled basis by the authors during collection (Section 3.5); we did not re-verify every record or compute a formal interannotator-agreement statistic, because each record was authored by a single student rather than double-annotated. Correctness on FinRank is thus single-annotator with sampled review rather than consensus-adjudicated, and a stratified double-annotation study is the most important outstanding item for a future release. Label and hard-negative artifacts. Raw fields contain casing and spelling variants and a few placeholder values, which our normalization pipeline (Section 3.8) resolves; consumers who skip normalization may see spurious Unknown buckets. Separately, 442 hard negatives (∼7.3%) are byte-equal to a supporting passage of a different record, where the passage may legitimately be relevant to more than one question; the released hn taxonomy.json lets users filter these overlaps. Modality and generation. Tables and figures are represented as text, so evaluations requiring true multimodal reasoning over image-based tables are out of scope, and reference-based metrics may penalize answers that are correct but phrased differently from the reference. Finally,

10

Ethical Considerations

FinRank is derived from filings that companies have submitted to the SEC and made public; it contains no nonpublic personal information. As is inherent to SEC filings, passages may name executives, directors, and officers in their public corporate roles, including disclosures such as executive compensation that issuers are legally required to publish. FinRank is released under CC BY-NC 4.0 for non-commercial research use with attribution; commercial licensing is available from the authors (Appendix I). Because the dataset is skewed toward a few sectors, filing years, and document types (Section 4), models tuned or evaluated on it may reflect the language of the dominant strata, and cross-sector generalization should be tested explicitly rather than inferred from aggregate scores. Strong performance on FinRank does not establish suitability for investment decisions, valuation, or trading, and FinRank-aligned systems should complement rather than replace qualified human analysts. Even accurate retrieval and generation systems can mis-rank passages, omit relevant evidence, or produce plausible but incorrect answers, so generated answers should not be presented as investment advice and should carry inline citations to their supporting passages, supporting the audit and compliance review that financial disclosure requires.

11

Conclusion

We introduced FinRank, an evidence-grounded benchmark for financial question answering and retrieval over SEC 10-K and 10-Q filings. Across 1185 manually authored records spanning 22 companies, three sectors, and eight topics for filings from 2024–2025, each question is paired

11

with supporting passages from the underlying filing and with a curated set of hard negatives from comparable filings, making FinRank, to our knowledge, the first financial QA benchmark to release per-question curated hard negatives and to measure discrimination against curated versus random distractors directly. Among the evaluated baselines, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the curated evidence pool, sub-billion-parameter embeddings beat BM25 by at most 3.5 points on this templated text, and the released hard negatives cost every model 13.0–20.5 points of pairwise accuracy over random distractors. Future work includes broadening coverage to more sectors, companies, and filing years; a stratified double-annotation study with reported inter-annotator agreement; executing the generation and RAG protocols with LLM-based systems; and extending the schema and hard-negative construction to multimodal (tabular and graphical) evidence.

12

References Chen, Jian et al. (2024). “FinTextQA: A Dataset for Longform Financial Question Answering”. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, pp. 6025–6047. arXiv: 2405.09980 [cs.CL]. Chen, Zhiyu, Wenhu Chen, et al. (2021). “FinQA: A Dataset of Numerical Reasoning over Financial Data”. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Chen, Zhiyu, Shiyang Li, et al. (2022). “ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering”. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Choi, Chanyeol et al. (2025). “FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation”. In: arXiv preprint arXiv:2504.15800. Choi, Chanyeol, Jihoon Kwon, et al. (2025). “FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering”. In: Proceedings of the 6th ACM International Conference on AI in Finance (ICAIF ’25). ACM. arXiv: 2508.14052 [cs.IR]. Choi, Chanyeol, Jy-Yong Sohn, et al. (2024). The ACMICAIF’24 FinanceRAG Challenge. https://financerag.com/. Competition at the 5th ACM International Conference on AI in Finance (ICAIF ’24). Dataset: Linq-AI-Research/FinanceRAG on Hugging Face. Deng, Yang et al. (2022). “PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance”. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, pp. 6970–6984. arXiv: 2210.08817 [cs.CL]. Es, Shahul et al. (2024). “RAGAs: Automated Evaluation of Retrieval Augmented Generation”. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. St. Julians, Malta: Association for Computational Linguistics, pp. 150–158. arXiv: 2309.15217 [cs.CL]. Gao, Tianyu et al. (2023). “Enabling Large Language Models to Generate Text with Citations”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. arXiv: 2305.14627 [cs.CL]. Islam, Pranab et al. (2023). “FinanceBench: A New Benchmark for Financial Question Answering”. In: arXiv preprint arXiv:2311.11944. Izacard, Gautier et al. (2022). “Unsupervised Dense Information Retrieval with Contrastive Learning”. In: Transactions on Machine Learning Research (TMLR).

Jiang, Yidong et al. (2026). “Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings”. In: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’26). arXiv: 2602.07294 [cs.CL]. Karpukhin, Vladimir et al. (2020). “Dense Passage Retrieval for Open-Domain Question Answering”. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Kwiatkowski, Tom et al. (2019). “Natural Questions: A Benchmark for Question Answering Research”. In: Transactions of the Association for Computational Linguistics. Lewis, Patrick et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”. In: Advances in Neural Information Processing Systems (NeurIPS). Mansouri, Sasan et al. (2026). “Bypassing Document Ingestion: An MCP Approach to Financial Q&A”. In: arXiv preprint arXiv:2603.20316. arXiv: 2603.20316 [cs.IR]. Min, Sewon et al. (2023). “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. arXiv: 2305. 14251 [cs.CL]. Muennighoff, Niklas et al. (2023). “MTEB: Massive Text Embedding Benchmark”. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Dubrovnik, Croatia: Association for Computational Linguistics, pp. 2014–2037. arXiv: 2210.07316 [cs.CL]. Nogueira, Rodrigo and Kyunghyun Cho (2019). “Passage Re-ranking with BERT”. In: arXiv preprint arXiv:1901.04085. Qu, Yingqi et al. (2021). “RocketQA: An Optimized Training Approach to Dense Passage Retrieval for OpenDomain Question Answering”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp. 5835–5847. arXiv: 2010.08191 [cs.CL]. Rajpurkar, Pranav et al. (2016). “SQuAD: 100,000+ Questions for Machine Comprehension of Text”. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). Reddy, Varshini et al. (2024). “DocFinQA: A LongContext Financial Reasoning Dataset”. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Bangkok, Thailand: Association for Computational Linguistics, pp. 445–458. arXiv: 2401.06915 [cs.CL]. Reimers, Nils and Iryna Gurevych (2019). “SentenceBERT: Sentence Embeddings using Siamese BERTNetworks”. In: Proceedings of the 2019 Conference

13

on Empirical Methods in Natural Language Processing tational Linguistics, pp. 6588–6600. arXiv: 2206.01347 (EMNLP). [cs.CL]. Robertson, Stephen and Hugo Zaragoza (2009). “The Zhou, Yixi et al. (2026). “FinCARDS: Card-Based AnaProbabilistic Relevance Framework: BM25 and Beyond”. lyst Reranking for Financial Document Question AnIn: Foundations and Trends in Information Retrieval swering”. In: Findings of the Association for Com3.4, pp. 333–389. putational Linguistics: ACL 2026. San Diego, CaliforStrich, Jan et al. (2026). “T2-RAGBench: Text-and-Table nia, United States: Association for Computational LinBenchmark for Evaluating Retrieval-Augmented Genguistics, pp. 24836–24852. doi: 10 . 18653 / v1 / 2026 . eration”. In: Proceedings of the 19th Conference of the findings - acl . 1244. arXiv: 2601 . 06992 [cs.CL]. European Chapter of the Association for Computational url: https://aclanthology.org/2026.findingsLinguistics (Volume 1: Long Papers). Rabat, Morocco: acl.1244/. Association for Computational Linguistics, pp. 165–191. Zhu, Fengbin et al. (2021). “TAT-QA: A Question AnarXiv: 2506.12071 [cs.CL]. swering Benchmark on a Hybrid of Tabular and Textual Tang, Yixuan and Yi Yang (2025). “FinMTEB: Finance Content in Finance”. In: Proceedings of the 59th Annual Massive Text Embedding Benchmark”. In: Proceedings Meeting of the Association for Computational Linguisof the 2025 Conference on Empirical Methods in Natural tics (ACL). Language Processing. Introduces the Fin-E5 financeadapted embedding model. Suzhou, China: Association for Computational Linguistics, pp. 3620–3638. arXiv: 2502.10990 [cs.CL]. Thakur, Nandan et al. (2021). “BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models”. In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS). arXiv: 2104.08663 [cs.IR]. Wasserman, Navve et al. (2025). “DocReRank: Single-Page Hard Negative Query Generation for Training MultiModal RAG Rerankers”. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, pp. 8640–8658. arXiv: 2505.22584 [cs.CV]. Wu, Shijie et al. (2023). “BloombergGPT: A Large Language Model for Finance”. In: arXiv preprint arXiv:2303.17564. Xiong, Lee et al. (2021). “Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval”. In: International Conference on Learning Representations (ICLR). arXiv: 2007.00808 [cs.IR]. Zhan, Jingtao et al. (2021). “Optimizing Dense Retrieval Model Training with Hard Negatives”. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM. arXiv: 2104.08051 [cs.IR]. Zhao, Suifeng et al. (2025). “FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain”. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, pp. 4215–4249. arXiv: 2505.17471 [cs.CL]. Zhao, Yilun et al. (2022). “MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Compu-

14

A

Question Generation and Annotation Criteria

This appendix consolidates the full question-generation methodology that governs FinRank, complementing the summary in Section 3.2. The methodology fixes the criteria against which the released JSONL was constructed and against which a third party can audit or extend the benchmark.

A.1

Purpose

The methodology defines criteria for generating questions about financial filings so that the resulting question set provides systematic coverage across financial topics, reasoning types, and complexity levels. The intent is a balanced and thorough evaluation resource, not an arbitrary collection of natural-language prompts.

A.2

Core Definitions

Passage. A discrete section of text from a financial filing that can provide standalone context for answering a question. Passages are the unit of evidence and the unit of retrieval. Financial filing. An SEC document, specifically a Form 10-K (annual report) or Form 10-Q (quarterly report), containing a company’s financial and operational disclosures. Query rewrite. A decomposition of a complex question into a sequence of sub-questions that guide analytical reasoning and expose intermediate retrieval and reasoning steps.

A.3

Reasoning Types

Qualitative reasoning. Interpretation, explanation, or judgment questions that do not require arithmetic calculations (e.g. business strategy analysis, revenue-recognition judgments, litigation impact discussions). Quantitative reasoning. Any question requiring numerical computation, comparison, or mathematical analysis. Quantitative questions are further partitioned into three subtypes: 1. Metrics-generated: questions derived automatically from standardized financial ratios and templated calculations (e.g. current-ratio queries). 2. Single-step arithmetic: a single division, multiplication, addition, or subtraction. 3. Compositional calculations: multi-step arithmetic requiring two or more sequential operations.

A.4

Financial Topic Categories

The methodology partitions filing content into eight financial topic categories. For each topic the methodology supplies both qualitative and quantitative reasoning targets (Table 8).

A.5

Complexity-Level Definitions

Questions are distributed across three complexity levels with target shares 30% Easy, 40% Medium, 30% Hard. Easy (direct / factual). Simple factual retrieval from a filing section, straightforward numerical queries, basic definitional questions; minimal interpretation. Examples: “What was the company’s total revenue in 2023?”; “What is the company’s primary business segment?”; “When does the company’s fiscal year end?” Medium (analytical). Comparisons between periods or segments, basic trend analysis, percentage calculations, and cross-referencing between related filing sections. Medium-level questions ship with a multi-sub-question query rewrite. Example original question: “How did the company’s profitability change between 2022 and 2023, and what drove this change?” Rewrite (abridged): gross/operating/net margins for both periods, followed by an attribution sub-question for the year-over-year change. 15

Topic

Qualitative reasoning examples

Company Overview Financials

Business nature, strategy evaluation, M&A rationale Earnings quality, operating margin consistency Footnotes Accounting policy implications, lease classifications Governance Board composition, audit committee scope Accounting Revenue-recognition judgments, impairment triggers Legal Contingent liability discussions, litigation impacts Risk Risk sensitivity analysis, risk management evaluation Shareholder Re- Dividend policy rationale, buyback turn strategy

Quantitative reasoning examples YoY revenue growth, headcount changes, EBITDA margin EPS changes, debt-to-equity, free cash flow per share Depreciation schedules, stockcompensation expense Director independence %, executive pay ratios Amortization changes, deferred-tax calculations Legal provision amounts, legal cost trends Value-at-Risk, interest coverage ratios Total shareholder returns, dividend payout ratios

Table 8: Topic–reasoning matrix from the FinRank question-generation methodology. The matrix is prescriptive: it specifies the (topic, reasoning type) cells that the dataset is expected to populate. Hard (complex synthesis). Multi-step reasoning across multiple sections, complex synthesis, risk assessment using multiple data points, or strategic analysis combining various filing elements. Hard-level questions ship with three or more sub-questions that sequentially reduce the synthesis problem to factual queries. Example original question: “How do the company’s liquidity risks relate to their debt covenant requirements and upcoming maturities?” Rewrite (abridged): current, quick, and cash ratios; cash balances; specific covenant requirements; current covenant margins; 12- and 24-month maturities; final synthesis sub-question.

A.6

Query Rewrite Rules

Query rewrites serve four purposes: they break complexity into manageable components, ensure completeness of the reasoning chain, make the analytical process transparent, and allow evaluation of intermediate retrieval steps rather than only end-to-end answer correctness. The application rule is: • Easy questions: generally do not require rewrites. • Medium questions: require multiple atomic sub-questions. • Hard questions: require three or more sub-questions that sequentially decompose complex reasoning into factual queries.

A.7

Passage Coverage Requirements

Questions are also distributed by the number of passages required to answer them, with target shares 40% singlepassage (1 passage), 30% two-passage (exactly 2 passages), and 30% multi-passage (3 or more passages). The intent is to force the benchmark to include multi-passage synthesis, which more closely matches realistic financial analysis.

A.8

Implementation Guidelines

The methodology binds the above criteria into five implementation guidelines for any party generating FinRank-style questions: 1. Balance across topics: every topic category is represented. 2. Reasoning-type mix: balance between qualitative and quantitative questions. 3. Complexity distribution: adhere to the 30%–40%–30% split. 4. Passage requirements: follow the 40%–30%–30% split exactly. 5. Sub-question creation: generate the required rewrites for medium and hard questions. Realized deviations from these guidelines are reported in Table 2. 16

A.9

Related Resources

The methodology cites two prior financial-QA resources as conceptual context for FinRank-style question generation. FinanceBench (Islam et al., 2023) provides a comprehensive open-book QA test suite of 10,231 questions about publicly traded companies, and the FinDER dataset of Choi et al. (2025) contributes expert-generated query–evidence–answer triplets with realistic abbreviations and domain-specific language for RAG evaluation in finance. FinRank differs from both in foregrounding the retrieval and reranking sub-tasks with explicit hard negatives drawn from comparable filings (Section 2).

A.10

What the Methodology Does and Does Not Guarantee

The methodology provides structured generation criteria: it fixes what topics, reasoning types, complexity levels, passage counts, and rewrites a FinRank question must satisfy, and is therefore the source of truth for reproducibility and benchmark validity arguments. It does not by itself certify the correctness of any individual question, answer, supporting passage, or hard negative; it does not specify human validation, expert adjudication, or inter-annotator agreement statistics. We treat the methodology as evidence that the dataset’s generation procedure was deliberate and as the basis for the deviation reporting in Table 2, while recording label-correctness audits as outstanding items in Section 9.

B

Record Schema

Table 9 gives the top-level schema of a FinRank record. Field

Type

Description

id question id ticker industry question answer query rewrite topic

string string string string string string array string

reasoning type difficulty

string string

num passages passage type

string/int string

year doc type passages hard negatives

string/int string array array

folder name

string

UUID identifier of the record. Original question identifier inherited from the source folder. Stock ticker of the filing company. Industry classification associated with the filing company. Natural-language question. Reference answer text. Optional alternative formulations of the question. Topical category of the question (for example, Risk, Accounting, Governance). Qualitative or Quantitative reasoning required to answer. Easy, Medium, or Hard, with some casing variation in the raw data. Number of supporting passages required to answer. Single-, One-, Two-, or Multi-Passage indicator, with some spelling variation in the raw data. Filing year of the underlying document. SEC document type (10-K or 10-Q). Supporting passages with text and page reference. Hard negative passages with text, page reference, ticker, year, and document type. Source folder of the record.

Table 9: Top-level schema of a FinRank record. The nested passages and hard negatives objects, abbreviated above, are detailed in Tables 10 and 11. Field

Type

Description

text page number passage id text sha1

string string/int string string

Passage content drawn from the underlying SEC filing. Page reference within the source document. Stable identifier <record id>:p<index> (normalized release). SHA-1 of the whitespace-trimmed text (normalized release).

Table 10: Schema of an entry in the passages array.

17

Field

Type

Description

text page number ticker

string string/int string

year doc type passage id

string/int string string

text sha1

string

Hard-negative passage content. Page reference within the source document. Ticker of the company whose filing the hard negative is drawn from. Filing year of the source document. Source document type (for example, 10-K or 10-Q). Stable identifier <record id>:hn<index> (normalized release). SHA-1 of the whitespace-trimmed text (normalized release).

Table 11: Schema of an entry in the hard negatives array.

C

Canonical Label Normalization Mapping

Table 12 enumerates the canonical label normalizations applied in curating the released FinRank.jsonl (Section 3.8). Changed values are preserved inline in parallel <field> raw fields, and every individual change is enumerated in the released repair log.json. Field

Raw value(s)

Canonical value

reasoning type reasoning type reasoning type passage type passage type passage type passage type topic topic topic topic industry industry industry difficulty ticker ticker difficulty doc type year

Qaltitative, Qualtitative, Qalitative "Quantitative " (trailing space) "string" One-Passage, One Passage, OnePassage Tow-Passage, Two-Page Mutli-Passage, Multi Passage, Multi-passage "string" Foodnotes CompanyOverview ShareholderReturn Financial Pharmaceutical, Phamaceuticals Oil and Gas Oil & Gas {Midstream,E&P,...} lowercase / leading-space variants Ford "string" "string" multi-source compounds (2 records) "integer"

Qualitative Quantitative Recovered from question id Single-Passage (merged) Two-Passage Multi-Passage Recovered from question id Footnotes Company Overview Shareholder Return Financials Pharmaceuticals Oil & Gas Oil & Gas (sub-industry surfaced) Title-case Easy / Medium / Hard F Recovered from question id Recovered from question id Unknown Majority year of same-ticker, same-doc type records (flagged as inferred)

Table 12: Canonical label normalization mapping applied in curating the release. Changed values are preserved in parallel <field> raw fields. Beyond label mapping, the pipeline performs three record-level repairs, each preserved in a machine-readable repair log: (i) hard-negative metadata whose fields were transposed at entry time is restored by rule (a numeric ticker alongside an alphabetic page number is swapped back, 10 cases; a ticker/doc type rotation is reversed, 2 cases; one placeholder hard-negative year is inferred by ticker–document majority); (ii) the 11 hard negatives whose trimmed text duplicated a gold passage of their own record are removed, as are empty and within-record duplicate passages, and page references that are not page numbers are nulled; and (iii) every stored label is cross-checked against the structured question id, with the 56 disagreements shipped in the repair log as candidates for manual audit rather than silently overwritten. A final exclusion pass removes sixty-five records from the release: fifty-two whose question, answer, rewrite, or evidence involves a non-filing source (an academic journal article one annotator used as an analytical frame, detected both by download-watermark markers and by the article’s distinctive regression variables); three whose hard negatives are synthetic relevance rationales rather than filing passages; three with placeholder or empty question or answer text; four exact duplicates of another record’s question–answer pair; and three records left with fewer than four unique hard negatives. Each exclusion is recorded in repair log.json with its rule; the release-blocking checks are implemented in baselines/validate release.py. 18

D

Example Record

A sanitized and abbreviated example of a single FinRank record (long passages and the full hard-negative array are truncated). The example illustrates several data-quality observations from Section 4.1: a reasoning type typo (Qaltitative), a lower-case difficulty (easy), and inconsistent doc type casing between the supporting passages and the hard negatives. { "id": "97546a79-...-82c119a9c38c", "question_id": "JNJ_CompanyOverview_Easy_OnePassage_Qualitative_01", "ticker": "JNJ", "industry": "Pharmaceuticals", "question": "Which reportable business segments does Johnson & Johnson operate in as of FY2024?", "answer": "As of FY2024, Johnson & Johnson operates in two main segments: Innovative Medicine and MedTech ...", "query_rewrite": [], "topic": "Company Overview", "reasoning_type": "Qaltitative", "difficulty": "easy", "num_passages": "1", "passage_type": "One-Passage", "year": "2025", "doc_type": "10-K", "passages": [ { "text": "### Description of the company and business segments ...", "page_number": "22" } ], "hard_negatives": [ { "text": "#### Note 19: Segment Information ...", "page_number": "106", "ticker": "LLY", "year": "2025", "doc_type": "10-k" } ], "folder_name": "annotator_01" }

E

Reasoning-Type Examples

To make the conceptual range of the questions concrete, we reproduce one record per representative reasoning mode below, verbatim from FinRank.jsonl (excerpts truncated for space). Each shows the question, an excerpt of the reference answer, an excerpt of one supporting passage, and an excerpt of one hard negative. The hard negatives illustrate the dominant “same industry, different company” bucket of Section 3.7. Accounting interpretation JNJ, 2025 10-K · Accounting · Hard · Qualitative Q. How do Johnson & Johnson’s tangible and intangible assets contribute differently to the Company’s ability to generate revenue and sustain competitiveness, and what does this imply about future business risks and strategic priorities? Reference answer. Johnson & Johnson’s tangible assets, such as manufacturing facilities and equipment, are used efficiently, as seen in the higher PPE turnover ratio. This shows that the Company effectively leverages its physical asset base to support large-scale production and distribution. However, the balance sheet is far more heavily weighted toward in . . . Supporting passage (p. 44). Johnson & Johnson and subsidiaries consolidated balance sheets At December 29, 2024 and December 31, 2023 (Dollars in Millions Except Share and Per Share Amounts) (Note 1) 2024 2023 Assets Current assets Cash and cash equivalents . . .

19

Hard negative (PFE, 2025 10-K). Consolidated Balance Sheets Pfizer Inc. and Subsidiary Companies (MILLIONS, EXCEPT PER SHARE DATA) Assets As of December 31, 2024 2023 Cash and cash equivalents $ 1,043 $ 2,853 Short-term investments . . .

Risk-disclosure comparison JNJ, 2025 10-K · Risk · Hard · Qualitative Q. What types of costs does Johnson & Johnson report in its footnotes that illustrate risks from sudden market shifts, and how did the COVID-19 vaccine manufacturing exit generate such costs? Reference answer. In its footnotes, Johnson & Johnson discloses a variety of costs that arise when markets or business conditions change, such as litigation charges, restructuring costs, impairment of acquired assets, acquisition and integration expenses, divestiturerelated losses, and regulatory compliance charges. These items show how rapidly evolving m . . . Supporting passage (p. 88-89). (3) Innovative Medicine segment income before tax includes: • Acquired in-process research & development expense of $ 1.25billion to secure the global rights to the NM26 bispecific antibody (Yellow Jersey acquisition) • Monetizati . . . Hard negative (LLY, 2025 10-K). Numbers may not add due to rounding. (1) Jardiance revenue includes Glyxambi, Synjardy, and Trijardy XR. (2) Humalog revenue includes insulin lispro. (3) Basaglar revenue includes Rezvoglar. (4) Olumi . . .

Quantitative extraction JNJ, 2025 10-K · Financials · Hard · Quantitative Q. What was Johnson & Johnson’s free cash flow to the firm (FCFF) in 2024, based on the reported financials? Reference answer. Free cash flow to the firm is calculated by starting from operating profit after tax and adjusting for non-cash charges, capital expenditures, and working capital movements. The company reported EBIT of 20,804 million and with an effective tax rate of 15.7% this results in a NOPAT of 17,536 million. Depreciation and amortization added bac . . . Supporting passage (p. 45). Johnson & Johnson and subsidiaries consolidated statements of earnings (Dollars and Shares in Millions Except Per Share Amounts) (Note 1) 2024 2023 2022 Sales to customers $88,821 85,159 79,990 Cost of products sold 27,471 26,553 . . . Hard negative (LLY, 2025 10-K). ELI LILLY AND COMPANY AND SUBSIDIARIES ¡div align=’center’¿(Dollars in millions, except per-share data, and shares in thousands)¡/div¿ Year Ended December 31, 2024 2023 2022 Revenue (Note 2) $ 45,042. . . .

Business-segment identification JNJ, 2025 10-K · Company Overview · Hard · Qualitative Q. How has the company restructured its business areas in recent years, and what strategic rationale can be derived from its current focus on two segments? Reference answer. Johnson & Johnson has restructured its business in recent years by fully separating its Consumer Health segment through a three-step process: the Kenvue IPO in May 2023, an August 2023 exchange offer reducing ownership to 9.5%, and a debt-for-equity exchange in Q2 2024 that eliminated the remaining stake. While transition manufacturing an . . . Supporting passage (p. 50). On May 8, 2023, Kenvue, completed an initial public offering (the IPO) resulting in the issuance of 198,734,444 shares of its common stock, par value $0.01 per share (the “Kenvue Common Stock”), at an initial public offering of $2 . . . Hard negative (LLY, 2025 10-K). Divestitures Olanzapine Portfolio (including Zyprexa) In July 2023, we sold the rights for the olanzapine portfolio, including Zyprexa, to Cheplapharm Arzneimittel GmbH (Cheplapharm), a European compa . . .

Legal / regulatory interpretation ABBV, 2025 10-K · Legal · Hard · Qualitative Q. How do AbbVie’s inherited Allergan litigations, combined with the financial aftermath of divested assets and integration costs, illustrate the legal and strategic risks that follow from large-scale acquisitions? Reference answer. AbbVie’s acquisition of Allergan shows how legal exposures can extend well beyond the transaction date and shape the company’s financial risk profile. The company inherited hundreds of lawsuits, including opioid litigation and breast implant cases, where plaintiffs seek compensatory and punitive damages, medical monitoring, and other reme . . . Supporting passage (p. 96-97). Government Proceedings Lawsuits are pending against Allergan and several other manufacturers generally alleging that they improperly promoted and sold prescription opioid products. Approximately 435lawsuits are pending against All . . . Hard negative (JNJ, 2025 10-K). The Company is subject to significant legal proceedings that can result in significant expenses, fines and reputational damage. In the ordinary course of business, Johnson & Johnson and its subsidiari . . .

Shareholder-return analysis LLY, 2025 10-K · Shareholder Return · Hard · Quantitative Q. How closely does Eli Lilly’s stock performance align with its revenue and EPS growth trajectory, and what explains any divergence? Reference answer. Eli Lilly’s stock performance has outpaced both revenue and EPS growth in recent years. Revenue increased by 20% in 2023 and by 32% in 2024, while diluted EPS declined by 16% in 2023 due to higher internal and acquired R&D expenses before rebounding with 102% growth in 2024. By contrast, the stock delivered annualized gains of more than 4 . . . Supporting passage (p. 40). The following graph compares the return on Lilly stock with that of the Standard & Poor’s (S&P) 500 Stock Index and our peer group for the years 2020 through 2024. The graph assumes that, on the last business day of 2019, a person . . .

20

Hard negative (JNJ, 2025 10-K). Johnson & Johnson and subsidiaries consolidated statements of earnings (Dollars and Shares in Millions Except Per Share Amounts) (Note 1) 2024 2023 2022 Sales to customers $88,821 85,159 79,990 Cost o . . .

F

Example Prompt Templates

We provide example prompt templates for three evaluation modes; benchmark consumers should report the exact prompts used (including system prompts and any few-shot exemplars) alongside their results.

F.1

LLM-Based QA Prompt

You are a financial analyst answering a question about a company’s SEC filing. Use only the information in the provided passages. If the passages do not contain enough information to answer the question, respond that the answer is not available in the provided passages. Question: {question} Passages: {passages} Answer:

F.2

RAG-with-Citations Prompt

You are a financial analyst answering a question about a company’s SEC filing. Use only the information in the provided passages. For every claim in your answer, cite the passage that supports it using its number, in square brackets, e.g. [1], [2]. Do not include claims that are not supported by the provided passages. Question: {question} Passages: [1] {passage_1} [2] {passage_2} ... Answer with citations:

F.3

Hard-Negative Discrimination Prompt

You are evaluating which passages from a candidate set support the answer to a financial question about an SEC filing. Read the question and the candidate passages and assign each passage one of two labels: - SUPPORTING: the passage contains evidence that directly supports an answer to the question. - NON-SUPPORTING: the passage does not directly support an answer to the question, even if it is topically related. Return your decisions as a list of (passage_id, label) pairs. Question: {question}

21

Candidate passages: [1] {passage_1} [2] {passage_2} ... Decisions:

G

Reproducibility Notes

In addition to the checklist in Section 6, studies should release the exact split files (record ids per partition) rather than describing splits in prose only, report whether and how the normalization of Section 3.8 was applied (with raw and normalized counts for each affected field), report the verbatim prompts used by any LLM-based component, describe how the retrieval corpus was assembled (in-record passages and hard negatives only, or a broader corpus from the underlying filings), and document model versions, decoding settings, and hardware to support replication. Auditability of the question set. Future users should be able to regenerate or audit the dataset using the methodology document (Appendix A), the question-generation criteria (Section 3.2), the released FinRank.jsonl, the per-passage identifiers, text hashes, and inline <field> raw values it carries, the query rewrite arrays, the machine-readable repair log of the normalization pipeline, and the evaluation scripts in baselines/. Pairing the methodology with the realized distribution (Table 2) is what allows a third party to detect divergence from the documented criteria without having to rederive them from scratch; it makes the benchmark more transparent and reduces ambiguity in question design, while not substituting for the question-by-question correctness audit listed in Section 9.

H

Cross-Tabulation Tables

The following cross-tabulations of the FinRank metadata are useful for diagnosing where future model performance is likely to be most variable. All counts are computed on the canonical normalized labels (Section 3.8) by baselines/make crosstabs.py; per-row and per-column totals match the marginal distributions reported in Section 4. Industry

Accounting Company Overview Financials Footnotes Governance Legal Risk Shareholder Return Row total

Automotive Oil & Gas Pharmaceuticals

35 60 63

33 55 65

32 53 67

23 57 62

27 55 59

30 55 61

36 51 63

32 52 59

248 438 499

Column total

158

153

152

142

141

146

150

143

1185

Table 13: Cross-tabulation of industry by topic on the canonical normalized labels.

Industry

10-K 10-Q Row total

Automotive Oil & Gas Pharmaceuticals

248 301 499

– 137 –

248 438 499

Column total

1048

137

1185

Table 14: Cross-tabulation of industry by document type on the canonical normalized labels.

Observations. The (industry × topic) cross-tab (Table 13) shows that all three industries populate every topic bucket, with no empty cell; Automotive cells are smaller in absolute terms because Automotive has half as many records as Pharmaceuticals or Oil & Gas. The (year × doc type) cross-tab (Table 15) confirms that the 10-Q partition is concentrated in a single filing year, so per-year and per-doc type analyses are not independent. The (difficulty × reasoning type) and (passage type × reasoning type) cross-tabs (Tables 17, 16) expose where the roughly 2:1 22

Filing year

10-K 10-Q Row total

2024 2025

283 765

13 124

296 889

Column total

1048

137

1185

Table 15: Cross-tabulation of filing year by document type on the canonical normalized labels. The Unknown entry is a record whose raw doc type names multiple source documents and cannot be attributed to a single filing. Passage type

Qualitative Quantitative Row total

Single-Passage Two-Passage Multi-Passage

276 322 211

133 150 93

409 472 304

Column total

809

376

1185

Table 16: Cross-tabulation of passage type by reasoning type on the canonical normalized labels. Difficulty

Qualitative Quantitative Row total

Easy Medium Hard

229 348 232

134 136 106

363 484 338

Column total

809

376

1185

Table 17: Cross-tabulation of difficulty by reasoning type on the canonical normalized labels. The qualitative-toquantitative skew is present at every difficulty level. qualitative-to-quantitative skew of Section 4 is most pronounced; consumers who report difficulty- or passage-typestratified metrics should also report the corresponding qualitative/quantitative split rather than assume it matches the dataset-level marginal.

I

Dataset Card

Identity. Name: FinRank. Distribution: https://github.com/datanxt/FinRank. Distribution file: FinRank.jsonl, carrying the canonical labels of Section 3.8 with raw surface forms preserved inline in <field> raw fields. Contents. 1185 question–answer records derived from the 10-K and 10-Q filings of 22 publicly listed companies (three sectors, 8 topical labels, filing years 2024–2025). Each record contains a question, a reference answer, optional query rewrites, structured metadata, an array of supporting passages, and an array of hard negative passages. The release additionally includes a deduplicated global pooled retrieval corpus of 5230 unique passages (see Section 3.6). Intended use. Benchmarking financial information-retrieval and question-answering systems on SEC-filing prose, with explicit support for retrieval, reranking, hard-negative discrimination, multi-passage reasoning, query rewriting, and retrieval-augmented generation with citations. Prohibited use. Automated investment decisions, automated trading signals, real-time valuation, regulatory or compliance decisions, or any high-stakes financial action without independent human oversight. Outputs of any system evaluated on FinRank must not be presented as financial advice. Reference answers reflect a particular reading of forward-looking, ranged, or conditional disclosure language and are not guaranteed to be correct in all financial respects. Known issues. Label inconsistencies in the raw release (Section 4.1); distributional skews along industry, year, document type, and reasoning type (Section 4); hard-negative contamination of 442 HNs across 305 records (Section 3.7); no independent ground-truth audit performed in this paper, see Section 9. 23

Recommended evaluation practice. Use the released canonical labels for all metric computation; report retrieval metrics on the global pool C and reranking / hard-negative metrics on the in-record set Lr ; report stratified results in addition to aggregate results (see Section 7.4); report the random seed and the exact split used; and accompany generative answer-quality results with faithfulness and citation-accuracy numbers in addition to EM/F1/ROUGEL/BERTScore. License. FinRank is released under the Creative Commons Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0). Academic and educational use is permitted with attribution; commercial licensing is available from the authors on request. The underlying SEC filings are public records; FinRank redistributes only derived question–answer records, supporting passages, and hard negatives.

J

Baselines Reproduction

The retrieval, reranking, hard-negative, and ablation results in Section 7 are produced by the scripts in baselines/. The complete pipeline is: cd FinRank/baselines pip install -r requirements.txt python validate_release.py --data ../FinRank.jsonl python run_baselines.py --data ../FinRank.jsonl --out results/ python run_extra_retrievers.py --data ../FinRank.jsonl --out results/ python make_splits.py --data ../FinRank.jsonl --out splits.json python make_crosstabs.py --data ../FinRank.jsonl --out ../ python make_macros.py --data ../FinRank.jsonl \ --results results/results.json \ --repair-log ../repair_log.json --out _results_macros.tex Models and seeds: • Random seed: 42 (random-negative sampling). • Dense retriever: sentence-transformers/all-mpnet-base-v2 (∼420 MB). • Cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2 (∼80 MB). • Sparse: rank-bm25 BM25Okapi with default parameters; scikit-learn TfidfVectorizer with sublinear TF and unigram features. • Tokenization for sparse baselines: lowercase + punctuation stripping (regex [^a-z0-9]+). Runtime expectations: on a single A10G GPU the full harness (run baselines.py) completes in under five minutes and the additional dense retrievers (run extra retrievers.py, including the 7B e5-mistral) in roughly ten; on a CPU-only laptop the mpnet/cross-encoder harness takes about an hour, and the 7B model is impractical. Score matrices are cached to baselines/results/score cache.npz so repeated runs do not re-encode the corpus.

24

Related documents

• MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answering#131347
• FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability#1051685
• Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute#1014987
• A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models#1155904
• VERA-8B: Evidence-Grounded Audit Risk Reasoning from SEC Filings#618610