Detecting Where Effects Occur by Testing Hypotheses in Order

[2602.21068] Detecting Where Effects Occur by Testing Hypotheses in Order Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Statistics > Methodology arXiv:2602.21068 (stat) [Submitted on 24 Feb 2026 ( v1 ), last revised 14 Aug 2026 (this version, v4)] Title: Detecting Where Effects Occur by Testing Hypotheses in Order Authors: Jake Bowers , David Kim , Nuole Chen View a PDF of the paper titled Detecting Where Effects Occur by Testing Hypotheses in Order, by Jake Bowers and 2 other authors View PDF HTML (experimental) Abstract: Experimental evaluations of public policies often randomize a new intervention within many sites or blocks. After an overall statistically significant result is reported, the natural question from a policy maker is: \emph{where} did effects occur? Standard adjustments for multiple testing answer this question with little power because they ignore how the experiment is organized: blocks nest within cohorts, sites, and districts. We organize the hypotheses in the shape of a tree that follows this administrative structure and test them top-down, stopping at any branch where the null is not rejected. A stopping rule and valid tests at each node suffice for weak control of the family-wise error rate (FWER). Whether the unadjusted procedure also controls the FWER in the strong sense depends on an \emph{error load} computable from design quantities before any data are tested; when the load exceeds one, an adaptive $\alpha$-schedule, which we prove controls the FWER on regular and irregular trees without pruning, restores control. [Correction, August 2026: an anonymous referee identified errors in the previous version. The error loads reported in the paper were computed incorrectly; corrected loads exceed 1 in all 25 block-randomized MDRC education trials at the planning effect size $d = 0.20$, so the adjustment the earlier version claimed unnecessary is in fact required. The theorem claiming FWER control under branch pruning and its switching corollary are false as stated and are withdrawn; an exact counterexample attains FWER 0.063 at $\alpha = 0.05$. The detection comparison with the Hommel procedure is under recomputation. A correction notice on page 1 details what changes and what stands; a fully corrected version is in preparation.] Subjects: Methodology (stat.ME) ; Statistics Theory (math.ST); Applications (stat.AP) Cite as: arXiv:2602.21068 [stat.ME] (or arXiv:2602.21068v4 [stat.ME] for this version) https://doi.org/10.48550/arXiv.2602.21068 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Jake Bowers [ view email ] [v1] Tue, 24 Feb 2026 16:29:45 UTC (82 KB) [v2] Fri, 27 Feb 2026 13:57:19 UTC (83 KB) [v3] Sat, 13 Jun 2026 22:33:22 UTC (154 KB) [v4] Fri, 14 Aug 2026 20:54:53 UTC (159 KB) Full-text links: Access Paper: View a PDF of the paper titled Detecting Where Effects Occur by Testing Hypotheses in Order, by Jake Bowers and 2 other authors View PDF HTML (experimental) TeX Source view license Current browse context: stat.ME < prev | next > new | recent | 2026-02 Change to browse by: math math.ST stat stat.AP stat.TH References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• The use and limitations of null-model-based hypothesis testing#778840
• When statistical evidence is not specific enough#720158
• Causal patterns and adequate explanations#761462
• "The hypothesis becomes forced": Cultural narratives about deductive hypothesis testing influence scientists' self-perception and actions.#479097
• Finding the way in phenotypic space: the origin and maintenance of constraints on organismal form#760736