arXiv:2605.15714v1 [cs.SE] 15 May 2026
Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation
Sunil Kothari 1 Sumukha Sharma Thoppanahalli Chandramouli 1 Naman Khandelwal 2 Parth Kulshreshtha 3 Ashi Jain 3 Kriti Banka 2 Tanuja Chintada 2 Venkata Triveni 2 Gulipalli Praveen Kumar 2 Manish Mehta 1 Tao Liu 1 only economics (when they are equal), making Abstract timing a measurable design variable rather than a This position paper argues that the machine learnconfiguration afterthought. A survey of 47 recent ing community should prioritize early-stage qualpapers reveals that only 4% report when validaity assurance in annotation pipelines over the pretion occurs, a striking gap given timing’s demonvailing practice of late-stage validation. Data strated impact in adjacent fields. Without explicit quality bottlenecks increasingly limit foundation attention to QA timing, the community risks opmodel improvement, yet quality assurance retimizing validation methods while ignoring the search focuses almost exclusively on validation structural variable that may matter most. methods rather than validation timing. When valiActing on this position requires three steps: redation occurs—not merely what validation methsearchers should report QA timing configurations ods are employed—fundamentally determines alongside validation methods; annotation platboth error rates and annotation costs. This tempoforms should expose timing as a first-class paramral neglect is puzzling given the well-established eter; and the community should run controlled “shift-left” principle from software engineering, experiments that measure stage-specific detection where empirical studies demonstrate 4–100× cost rates directly, operationalizing the shift-left prinmultipliers for defects detected in later developciple for annotation pipelines. ment stages (Boehm, 1981; Shull et al., 2002). Annotation pipelines, we argue, exhibit analogous dynamics: errors caught before annotation begins 1. Introduction cost a fraction of those discovered after review cycles complete. We argue that quality assurance timing—specifically, whether validation occurs before annotation, after annotation, or after review—is a critical design variable that the machine learning community has systematically overlooked, and that early-stage QA is more costeffective than late-stage validation for most annotation workflows.
We propose a taxonomy of three QA trigger points—pre-annotation (T0 ), post-annotation (T1 ), and post-review (T2 )—that decompose annotation workflows into discrete validation opportunities. A parametric error-propagation model formalizes when timing affects final error rates (when stage-specific detection rates differ) versus Equal contribution 1 Centific AI Research, Redmond, WA, USA 2 Centific AI Research, Hyderabad, India 3 Centific AI Research, Chennai, India. Correspondence to: Sunil Kothari <[email protected]>, Sumukha Sharma Thoppanahalli Chandramouli <[email protected]>, Naman Khandelwal <[email protected]>, Parth Kulshreshtha <[email protected]>, Ashi Jain <[email protected]>, Kriti Banka <[email protected]>, Tanuja Chintada <[email protected]>, Venkata Triveni <[email protected]>, Gulipalli Praveen Kumar <[email protected]>, Manish Mehta <[email protected]>, Tao Liu <[email protected]>.
The data-centric AI paradigm has established that training data quality fundamentally constrains model performance (Ng, 2021; Sambasivan et al., 2021; Whang et al., 2023). Recent analyses have revealed pervasive label errors across major benchmarks: Northcutt et al. (2021b) identified error rates averaging 3.3% in ImageNet, CIFAR, and other widely-used test sets, while Beyer et al. (2020) found that 6% of ImageNet validation labels require correction. For video annotation pipelines processing thousands of hours of footage at frame rates of 16–30 fps, even small per-frame error rates compound into substantial quality degradation (Dave et al., 2020; Voigtlaender et al., 2019).
Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
The research community has responded with increasingly sophisticated validation methods. Confident learning enables statistical detection of label errors without ground truth
*
1
Title Suppressed Due to Excessive Size
(Northcutt et al., 2021a). Vision-language models such as GPT-4V and Qwen2-VL provide semantic verification capabilities (Wang et al., 2024; OpenAI, 2023). Multi-annotator consensus methods model annotator reliability and aggregate labels accordingly (Dawid & Skene, 1979; Raykar et al., 2010; Goh et al., 2022). Inter-annotator agreement metrics quantify annotation consistency (Artstein & Poesio, 2008; Cohen, 1960; Krippendorff, 2011).
Section 5 presents the full configuration taxonomy. Section 6 engages substantively with alternative views. Section 7 presents our call to action, Section 8 describes the limitations of this approach and Section 9 concludes.
2. QA Trigger Points: A Framework Before presenting our argument, we must establish precise terminology for discussing when quality assurance occurs in annotation pipelines. We introduce the concept of QA trigger points—discrete moments in the annotation workflow where validation can be invoked—and define three canonical trigger points that capture the structure of typical annotation pipelines.
These advances focus exclusively on what to validate and how to detect errors. Yet they share a critical blind spot: the question of when validation should occur in the annotation pipeline receives virtually no systematic attention. We surveyed 47 articles on CVPR, NeurIPS and ICML annotation quality (2022–2024) and found that only 2 articles (4%) explicitly reported the stage of the pipeline at which validation was applied.
2.1. Annotation Pipeline Structure Modern annotation pipelines for computer vision tasks typically proceed through three phases (Roh et al., 2019; Monarch, 2021). First, machine learning pre-annotation generates initial predictions (bounding boxes, segmentation masks, tracking identities) that serve as starting points for human annotators (Yao et al., 2012; Papadopoulos et al., 2017). Second, human annotation refines, corrects, or validates these predictions, with annotators adding missing objects, adjusting boundaries, and correcting labels (Su et al., 2012; Kovashka et al., 2016). Third, human review provides quality control, with reviewers accepting, rejecting, or requesting revisions to submitted annotations (Daniel et al., 2018; Vaughan, 2017).
This omission matters because the same validation method produces different outcomes depending on when it executes. Consider a video annotation pipeline in which machine learning pre-annotation produces bounding boxes with a 15% error rate. If quality assurance runs only after human review (late-stage), these errors propagate throughout the pipeline before detection, consuming annotator time on corrections that could have been prevented. If quality assurance runs before human annotation begins (early-stage), errors are caught before human effort is invested, potentially reducing both error rates and costs. Software engineering learned this lesson decades ago. The “shift-left” principle, formalized by Boehm (1981) and validated in subsequent empirical studies (Shull et al., 2002; Jones, 2008; McConnell, 2004), establishes that defects caught in early development stages cost substantially less to remediate than those caught later. Manufacturing quality management and ML technical debt research encode similar intuitions, which we examine in Section 3.
This three-phase structure creates natural boundaries where quality assurance can intervene. We formalize these as trigger points. 2.2. Trigger Point Definitions We define three QA trigger points, illustrated in Figure 1:
We hypothesize that annotation pipelines exhibit dynamics analogous to software development and manufacturing: errors detected early cost less to remediate than errors detected late. This paper does not prove this hypothesis empirically— such proof requires controlled experiments comparing identical validators across pipeline stages, which we have not conducted. Rather, we argue that the hypothesis is sufficiently plausible, and the current neglect of timing sufficiently systematic, that the research community should treat QA timing as a first-class research question.
Definition 1 (T0 : Pre-Annotation Trigger). The T0 trigger point occurs after machine learning pre-annotation completes but before human annotators begin work. At T0 , validation operates exclusively on machine predictions. QA agents at this stage can assess prediction confidence, detect systematic ML failures, verify coverage against expected object density, and flag anomalous predictions for human attention. The key characteristic of T0 is that errors caught here prevent downstream human effort. If a low-confidence ML prediction is flagged at T0 and routed for re-processing or special handling, annotators never invest time correcting an error that would otherwise propagate through the pipeline.
The remainder of this paper proceeds as follows. Section 2 introduces our taxonomy of QA trigger points and situates it within annotation workflow structure. Section 3 presents evidence that timing is currently invisible in both research and practice. Section 4 develops a parametric model clarifying when timing affects error rates versus only economics.
Definition 2 (T1 : Post-Annotation Trigger). The T1 trigger point occurs after human annotation submission but before review. At T1 , validation can compare human an2
Title Suppressed Due to Excessive Size
T0
ML Pre-annot.
T1
Annotation
Pre-annot. QA
Review
Post-annot. QA
T2
Output
Post-review QA
Figure 1. QA trigger points in annotation pipelines. T0 occurs after ML pre-annotation but before human work. T1 occurs after annotation but before review. T2 occurs after review. Each trigger point enables different validation capabilities and incurs different intervention costs. Table 1. Validation capabilities by trigger point. Each stage enables different analyses based on available information. Trigger
Enabled Capabilities
T0
ML confidence assessment, coverage validation, systematic failure detection, anomaly flagging
T1
Human-ML comparison, annotation error detection, temporal consistency, IAA computation (dual-annotator only), annotator metrics
T2
2008)—is only computable at T1 or later, because it requires completed annotations from multiple annotators. This makes IAA a timing-dependent quality signal: workflows that validate only at T0 cannot leverage agreement as a quality indicator.
3. Evidence That Timing Is Overlooked We present three categories of evidence that QA timing is systematically overlooked: a literature survey, an analysis of annotation platforms, and an examination of how adjacent fields treat analogous questions.
Reviewer decision auditing, bias detection, gold standard comparison, compliance documentation
3.1. Literature Survey We conducted a structured survey of annotation quality research. Using search terms “annotation quality,” “label quality,” “data validation,” and “annotation error detection,” we identified 127 candidate papers from CVPR, NeurIPS, and ICML proceedings (2022–2024). We retained the 47 papers with explicit methodology sections describing validation approaches.
notations against the ML baseline, detect annotation errors (spatial, label, temporal), and—for workflows with multiple annotators—compute inter-annotator agreement (IAA). The key characteristic of T1 is access to human judgment. Validation at this stage can leverage the comparison between ML predictions and human corrections to identify likely errors. However, human effort has already been invested; errors caught at T1 cannot prevent annotation labor, only review labor.
For each paper, we coded: (a) whether validation method characteristics were reported (precision, recall, accuracy); (b) whether computational requirements were reported; (c) whether the pipeline stage at which validation was applied was explicitly stated.
Definition 3 (T2 : Post-Review Trigger). The T2 trigger point occurs after reviewer assessment completes. At T2 , validation can audit reviewer decisions, detect systematic reviewer biases, compare final outputs against gold standards, and generate compliance documentation.
Table 2 summarizes our findings. While 100% of papers reported validation method characteristics and computational requirements, only 4.3% (2/47) explicitly stated when validation was applied. The two papers that reported timing— both comprehensive surveys by Klie et al. (2023) and Klie et al. (2024)—did so while analyzing annotation practices rather than proposing new methods.1
The key characteristic of T2 is finality. All human effort (annotation and review) has been invested; errors caught at T2 require full rework. However, T2 provides the most complete information for validation, including the full provenance chain from ML prediction through annotation to review decision.
The most comprehensive annotation error detection survey, Klie et al. (2023), reimplemented 18 error detection methods across 9 datasets. Every method focuses on post-annotation detection (T1 or T2 ); none evaluate pre-annotation prevention (T0 ). Klie et al. (2024), analyzing quality management practices across 591 NLP dataset papers, found that system-
2.3. What Each Trigger Point Enables and Precludes The trigger points differ not only in timing but in the validation capabilities they enable (Table 1).
1 Complete survey methodology and full paper list available in Supplementary Material.
Critically, inter-annotator agreement—a primary quality signal in many annotation workflows (Artstein & Poesio, 3
Title Suppressed Due to Excessive Size Table 2. Literature survey results: QA timing reporting by venue. Only 2 of 47 surveyed papers explicitly report when validation occurs in the annotation pipeline. Venue
Papers
Timing Reported
%
CVPR/ICCV NeurIPS ICML Other ML venues
12 11 8 16
0 0 0 2
0% 0% 0% 12.5%
Total
47
2
4.3%
ciples. Crosby’s 1-10-100 rule (Crosby, 1979) posits that prevention costs $1, detection costs $10, and correction after failure costs $100. Toyota’s production system, widely studied in operations research (Liker, 2004), emphasizes “building in quality” through early-stage defect prevention rather than end-of-line inspection. We hypothesize that annotation pipelines exhibit analogous dynamics. The conditions for shift-left to apply are: (1) errors can propagate through stages, (2) later detection requires more rework, and (3) detection capabilities exist at early stages. Annotation pipelines plausibly satisfy all three conditions: ML errors propagate to annotation and review; late detection requires discarding completed work; and confidence scores, coverage analysis, and anomaly detection are feasible at T0 .
atic quality processes “are only mentioned rarely.” 3.2. Platform Analysis We examined public documentation for six major annotation platforms: Scale AI, Labelbox, CVAT, Label Studio, Amazon SageMaker Ground Truth, and Appen. We assessed whether each platform exposes QA timing as a named, configurable parameter with documented tradeoffs.
However, we emphasize that this is a hypothesis, not a proven fact. The shift-left principle derives from software and manufacturing contexts that differ from annotation in important ways. Empirical validation—comparing identical validators across T0 , T1 , and T2 —is needed to establish whether the principle transfers.
Finding: No platform explicitly exposes “QA timing” as a first-class parameter. While platforms offer sophisticated multi-stage workflows, timing emerges implicitly from configuration choices rather than being named, documented, or optimized as a design variable.
4. When Does Timing Affect Outcomes? To reason precisely about timing effects, we develop a parametric error propagation model that clarifies when timing affects error rates versus only economics.
Labelbox supports up to 10 workflow stages with AutoQA nodes insertable at any stage—but documentation frames this as workflow configuration, not timing optimization. CVAT provides Ground Truth mode (post-annotation comparison) and Honeypot mode (validation items mixed into annotation queues)—the closest approximation to timing exposure, but without guidance on when each approach is optimal. Label Studio supports webhooks for TASK CREATED (T0 ) and ANNOTATION CREATED (T1 ), but native T2 support requires the Enterprise edition.
4.1. Model Setup Consider an annotation pipeline with ML pre-annotation error rate e0 (the fraction of ML predictions that are incorrect). Let dann denote the annotator’s natural detection rate (the probability an annotator notices and corrects an ML error without QA assistance), and drev denote the reviewer’s detection rate. Without any QA intervention, the final error rate is:
3.3. The Shift-Left Principle
enone final = e0 · (1 − dann ) · (1 − drev )
The systematic neglect of timing in annotation research contrasts sharply with adjacent fields where timing is a central concern. Software engineering’s shift-left principle, formalized by Boehm (1981) in his analysis of software economics, establishes that defect remediation costs increase as defects progress through development stages.
(1)
This represents errors that escape both annotator and reviewer detection. 4.2. QA at Different Trigger Points Now suppose QA is applied at a single trigger point with detection rate dTi . The final error rate becomes:
Empirical studies support this principle across multiple contexts. Shull et al. (2002) found defects cost 4–5× more to fix in testing than in design. Jones (2008) documented cost multipliers ranging from 1× to 100× depending on defect type and detection stage. McConnell (2004) synthesized industry data showing 10–25× cost increases for defects escaping to production.
QA at T0 only: 0 eTfinal = e0 · (1 − dT0 ) · (1 − dann ) · (1 − drev )
(2)
QA at T1 only: 1 eTfinal = e0 · (1 − dann ) · (1 − dT1 ) · (1 − drev )
Manufacturing quality management encodes similar prin4
(3)
Title Suppressed Due to Excessive Size
QA at T2 only: 2 eTfinal = e0 · (1 − dann ) · (1 − drev ) · (1 − dT2 )
tors work independently; a reviewer reconciles differences. This enables IAA computation but approximately doubles annotation cost.
(4)
4.3. Key Insight: Timing Effects Are Conditional
5.2. Configuration Space
Examining these equations reveals a critical insight: if detection rates are equal across stages (dT0 = dT1 = dT2 ), timing has no effect on final error rate.
For each workflow option, any non-empty subset of {T0 , T1 , T2 } can be activated, yielding 7 configurations per option (Table 3).
When detection rates are equal, the multiplicative structure ensures identical outcomes regardless of when QA is applied. The difference is only in where errors are caught, which affects economics (human effort invested before detection) but not final quality.
This taxonomy provides vocabulary for discussing QA design. Rather than describing a pipeline as having “standard quality controls,” practitioners can specify “A-1+2 configuration”—immediately communicating that QA runs at T1 and T2 with single annotators.
Timing affects error rates only when detection rates differ across stages. This can occur because: (a) ML errors may be more systematic and detectable at T0 than diverse human errors at T1 ; (b) comparing human annotations against ML baseline at T1 may reveal errors invisible to T0 analysis; or (c) different validation methods may excel at different stages.
6. Alternative Views We engage substantively with four credible counterarguments to our position. 6.1. “Validation Method Quality Dominates Timing”
We do not know which scenario reflects reality. Determining whether detection rates differ across stages—and in which direction—requires empirical studies that, to our knowledge, have not been conducted.
The counterargument: Investing in better detection methods—higher-precision VLMs, improved confident learning algorithms, more sophisticated IAA metrics— provides larger returns than timing optimization. A superior detector deployed at any stage will outperform an inferior detector at the “optimal” stage.
4.4. Economic Effects Are Always Present Even when detection rates are equal (and thus error rates are identical), timing affects costs. Let cann denote annotation cost per task and crev denote review cost per task.
Our response: We find this counterargument partially compelling. In regimes where detection rates vary dramatically across methods but minimally across stages, method improvement will dominate timing optimization. Our model confirms this: when dT0 = dT1 = dT2 , timing affects only economics, not error rates.
With T0 QA, errors flagged before annotation save cann +crev per correctly flagged error. With T2 QA, errors flagged after review save nothing—the work is already complete.
5. Configuration Taxonomy
However, we identify three limitations to this counterargument. First, method improvement and timing optimization are not mutually exclusive; the optimal strategy considers both. Second, timing optimization is often cheaper than method improvement—changing configuration requires no new model development. Third, the counterargument assumes detection rates are equal across stages, which is an empirical question, not an established fact.
The three trigger points, combined with workflow options (single vs. dual annotator), generate a space of 14 distinct QA configurations.
We do not claim timing dominates method quality. We claim timing deserves study alongside method quality, because we currently lack evidence to assess their relative importance.
5.1. Workflow Options
6.2. “The Shift-Left Principle May Not Transfer”
Option A (Single Annotator + Reviewer): One annotator produces annotations; a separate reviewer assesses quality. This is cost-efficient but provides no annotator comparison signal.
The counterargument: Software engineering and manufacturing findings may not transfer to annotation contexts. Software defects have different characteristics than annotation errors: software bugs can cascade unpredictably through code paths, while annotation errors are typically localized.
This economic effect is why shift-left matters even when final quality is unaffected. However, the economic benefit depends on QA precision: false positives at T0 that incorrectly flag correct predictions can increase costs by triggering unnecessary rework.
Option B (Dual Annotator + Reviewer): Two annota5
Title Suppressed Due to Excessive Size Table 3. QA configuration taxonomy. The 14 configurations represent every non-empty subset of {T0 , T1 , T2 } across two workflow options. Agent counts, cost multipliers, and quality ratings are illustrative—derived from team experience with video annotation involving bounding boxes and tracking, not empirical measurements. Actual values depend on annotation type, validator complexity, and platform overhead. The “Best For” column provides scenario-level guidance; specific deployment choices should account for domain constraints and quality requirements. Config
Cost
Quality
Best For
Option A: Single Annotator + Reviewer A-0 T0 5 1.0× A-1 T1 7 1.2× A-2 T2 8 1.3× A-0+1 T0 +T1 12 2.2× A-0+2 T0 +T2 13 2.3× A-1+2 T1 +T2 15 2.5× A-MAX T0 +T1 +T2 20 3.5×
Basic Good Good High Medium High Maximum
High-volume, low-stakes data; rapid prototyping Standard production pipelines Audit-heavy workflows Mid-stakes datasets; cost-quality balance Weak ML with high audit needs Compliance-driven workflows Single-annotator high-quality datasets
Option B: Dual Annotator + Reviewer (+IAA) B-0 T0 5 2.0×
Good
Research datasets; consensus matters but cost is constrained Standard research benchmarks Subjective tasks needing reviewer reconciliation High-quality benchmarks with cost discipline Specialized domains with weak ML Production datasets for high-stakes ML Safety-critical datasets (medical, autonomous driving)
B-1 B-2 B-0+1 B-0+2 B-1+2 B-MAX
Triggers
T1 +IAA T2 +IAA T0 +T1 +IAA T0 +T2 +IAA T1 +T2 +IAA T0 +T1 +T2 +IAA
Agents
8 9 13 14 17 22
2.4× 2.6× 4.4× 4.6× 5.0× 7.0×
High High High High Maximum Ultimate
Manufacturing defects involve physical materials with nonlinear failure modes, while annotation involves human judgment on defined tasks.
However, their finding addresses whether to add QA, not when to add it. Given that organizations will deploy QA— for compliance requirements, audit trails, or error detection— the timing question remains relevant. Moreover, T0 QA can inform instruction improvement by identifying systematic ML failures that annotators encounter, creating a feedback loop between QA and guideline refinement.
Our response: We acknowledge significant uncertainty about whether shift-left transfers. The principle applies when: (1) errors compound through stages, (2) later detection requires more rework, and (3) early detection is feasible. We believe annotation satisfies these conditions, but we have not proven it empirically.
The findings are complementary, not contradictory: improve instructions and optimize QA timing.
The counterargument strengthens our call for empirical research. If controlled experiments show that timing has minimal effect in annotation contexts—that the shift-left principle does not transfer—we will have learned something important about the disanalogy between annotation and other quality-critical processes. Either outcome advances the field.
6.4. “Empirical Evidence Is Insufficient” The counterargument: Position papers should be grounded in empirical findings. Advocating for attention to QA timing without empirical evidence that timing matters is premature. The shift-left analogy is speculative; the error propagation model uses assumed parameters; no controlled experiments compare validators across stages.
6.3. “Instruction Quality Matters More Than QA”
Our response: This is the strongest counterargument, and we substantially accept it.
The counterargument: Rädsch et al. (2024), analyzing 57,648 images across 924 annotators, found that “improving labeling instructions yields higher effects than adding QA steps.” This suggests upstream prevention (better guidelines) dominates downstream detection (any QA), making timing optimization within QA a second-order concern.
We have not proven that timing matters empirically. Our model is theoretical; our parameters are illustrative; our shift-left hypothesis is untested in annotation contexts. A skeptical reader could reasonably conclude that our position is premature.
Our response: We agree that instruction quality is critical and potentially underweighted in practice. The Rädsch et al. finding is important and should inform annotation pipeline design.
However, we offer a meta-argument: timing is currently invisible, and we cannot assess its importance without making it visible. The field cannot conduct meta-analyses of timing effects when timing is not reported. Our call for reporting 6
Title Suppressed Due to Excessive Size
costs nothing and enables future empirical assessment.
8. Limitations
If the research community begins reporting timing, conducts controlled experiments, and finds that timing effects are negligible, we will have learned something valuable. If timing effects are substantial, we will have enabled optimization that is currently impossible. Either outcome justifies the modest investment of treating timing as a reportable variable.
We acknowledge several limitations. First, our error propagation model is theoretical; the shift-left hypothesis for annotation remains untested. Second, our examples focus on video annotation with bounding boxes and tracking; generalization to other modalities requires validation. Third, agent counts, cost multipliers, and quality ratings in Table 3 are illustrative, not empirically derived. Fourth, our platform analysis is based on public documentation; enterprise features may provide timing configurability. Fifth, our survey may miss papers that mention timing incidentally; the 4% figure reflects explicit reporting.
7. Call to Action We call on four communities to take specific, actionable steps.
9. Conclusion
7.1. For Researchers Publishing Validation Methods
We have argued that quality assurance timing in annotation pipelines—the question of whether validation occurs at T0 (pre-annotation), T1 (post-annotation), or T2 (post-review)— deserves systematic study as a first-class research question.
We urge researchers to report QA timing configuration when publishing validation results. Specifically, state whether validation was applied to ML predictions (T0 ), human annotations before review (T1 ), or post-review outputs (T2 ). This requires minimal effort and enables future meta-analysis. Where resources permit, we encourage measuring whether validation methods perform differently at different stages, reporting results even if differences are minimal.
Our parametric model clarifies when timing affects error rates (when detection rates differ across stages) versus only economics (when detection rates are equal). The shift-left analogy provides theoretical grounding; empirical validation in annotation contexts remains future work.
7.2. For Dataset Creators
The evidence suggests timing is currently invisible: surveyed papers rarely report when validation occurs, and no major platform exposes timing as a named parameter. This invisibility prevents the field from assessing timing’s importance.
We urge dataset creators to document QA timing alongside methods. Dataset papers should specify not only what quality assurance was applied but when. For example: “Quality assurance included automated bounding box validation at T1 and expert review sampling at T2 , with no T0 validation of ML pre-annotations.” This is essential for reproducibility and for understanding dataset quality characteristics.
Our call to action is deliberately modest. We ask researchers to report timing when publishing validation results. We ask platforms to expose timing as a configurable parameter. We ask the community to conduct controlled experiments that would operationalize the shift-left principle in annotation contexts.
7.3. For Annotation Platform Developers We urge platform developers to expose timing as a named, configurable parameter. Users should be able to specify which validation methods run at T0 , T1 , and T2 , with documented tradeoffs for each choice. Platforms should provide guidance on latency implications, cost implications, and capability implications of timing choices.
If these efforts reveal that timing effects are minimal, we will have learned that annotation differs from software and manufacturing in important ways. If timing effects are substantial, we will have enabled optimization currently impossible. Either outcome advances the field beyond the current state where timing is simply ignored.
7.4. For the Research Community
References
We urge the research community to conduct controlled timing experiments. The highest-value contribution would be experiments that apply identical validators at T0 , T1 , and T2 on the same underlying data, with ground truth labels enabling measurement of true detection rates by stage. This would directly test whether the shift-left hypothesis holds for annotation.
Artstein, R. and Poesio, M. (2008). Survey article: Intercoder agreement for computational linguistics. Computational Linguistics, 34(4):555–596.
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., and van den Oord, A. (2020). Are we done with ImageNet? arXiv preprint arXiv:2006.07159. 7
Title Suppressed Due to Excessive Size
Boehm, B. W. (1981). Software Engineering Economics. Prentice-Hall.
Northcutt, C. G., Athalye, A., and Mueller, J. (2021b). Pervasive label errors in test sets. In NeurIPS Datasets & Benchmarks.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46.
OpenAI (2023). GPT-4V(ision) system card. Technical report. Papadopoulos, D. P., Uijlings, J. R., Keller, F., and Ferrari, V. (2017). Extreme clicking for efficient object annotation. In ICCV.
Crosby, P. B. (1979). Quality Is Free: The Art of Making Quality Certain. McGraw-Hill. Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., and Allahbakhsh, M. (2018). Quality control in crowdsourcing: A survey. ACM Computing Surveys, 51(1):1–40.
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Heller, N., Isensee, F., Kopp-Schneider, A., and Maier-Hein, L. (2024). Quality assured: Rethinking annotation strategies in imaging AI. In ECCV, pages 52–69.
Dave, A., Khurana, T., Tokmakov, P., Schmid, C., and Ramanan, D. (2020). TAO: A large-scale benchmark for tracking any object. In ECCV.
Raykar, V. C. et al. (2010). Learning from crowds. JMLR, 11:1297–1322.
Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates. J. Royal Statistical Society C, 28(1):20–28.
Roh, Y., Heo, G., and Whang, S. E. (2019). A survey on data collection for machine learning. IEEE TKDE, 33(4):1328–1347.
Goh, H. W., Tkachenko, U., and Mueller, J. (2022). CROWDLAB: Supervised learning for multi-annotator consensus. In NeurIPS Human in the Loop Learning Workshop, 2022.
Sambasivan, N. et al. (2021). “Everyone wants to do the model work, not the data work.” In CHI. Sculley, D. et al. (2015). Hidden technical debt in ML systems. In NIPS’15.
Jones, C. (2008). Applied Software Measurement. McGrawHill, 3rd edition.
Shull, F. et al. (2002). What we have learned about fighting defects. In IEEE International Symposium on Software Metrics.
Klie, J.-C., Webber, B., and Gurevych, I. (2023). Annotation error detection: Analyzing past and present. Computational Linguistics, 49:157–198.
Su, H., Deng, J., and Fei-Fei, L. (2012). Crowdsourcing annotations for visual object detection. In AAAI Workshop.
Klie, J.-C., Eckart de Castilho, R., and Gurevych, I. (2024). Analyzing dataset annotation quality management. Computational Linguistics, 50(3):817–866.
Vaughan, J. W. (2017). Making better use of the crowd. JMLR, 18(193):1–46.
Kovashka, A., Russakovsky, O., Fei-Fei, L., and Grauman, K. (2016). Crowdsourcing in computer vision. Found. Trends Comput. Graph. Vis., 10(3):177–243.
Voigtlaender, P. et al. (2019). MOTS: Multi-object tracking and segmentation. In CVPR. Wang, P. et al. (2024). Qwen2-VL: Vision-language model perception. arXiv:2409.12191[cs.CV].
Krippendorff, K. (2011). Computing Krippendorff’s alphareliability. Technical report, University of Pennsylvania.
Whang, S. E., Roh, Y., Song, H., and Lee, J.-G. (2023). Data collection and quality challenges in deep learning. VLDB Journal, 32:791–813.
Liker, J. K. (2004). The Toyota Way. McGraw-Hill. McConnell, S. (2004). Code Complete. Microsoft Press, 2nd edition.
Yao, A., Gall, J., Leistner, C., and Van Gool, L. (2012). Interactive object detection. In CVPR.
Monarch, R. M. (2021). Human-in-the-Loop Machine Learning. Manning Publications. Ng, A. (2021). MLOps: From model-centric to data-centric AI. DeepLearning.AI. Northcutt, C. G., Jiang, L., and Chuang, I. L. (2021a). Confident learning: Estimating uncertainty in labels. JAIR, 70:1373–1411. 8