arXiv CSopen access

Task Decomposition for Efficient Annotation

arXiv:2606.24734v1 [cs.CL] 23 Jun 2026

Task Decomposition for Efficient Annotation Nupoor Gandhi, Emma Strubell Carnegie Mellon University

Abstract High-quality annotations of structured representations are expensive to collect over large corpora. Manual annotation of structure is laborious, and model-based annotation, although cheaper to generate, requires expensive validation and potentially significant supervision to ensure that the annotation quality is strong enough to be useful downstream. In traditional annotation workflows, annotation of each complete example is performed end-to-end by a single annotator. However, structured annotation is complex, and each aspect of the task represents a unique challenge with an associated inferential load for a given annotator. Modern annotation projects can incorporate heterogeneous groups of annotators, including both models and human annotators with varying domain and linguistic expertise. It remains unclear, however, how to redesign annotation tasks in this setting, where efforts are discriminately allocated across heterogeneous annotators with respect to distinct annotation challenges. We propose to decompose annotation tasks into sub-tasks in order to reduce the aggregate inferential load of annotation projects. Inspired by the notion of centers from centering theory, we introduce a formal model of inferential load based on the degrees of freedom in the space of valid annotations. Using this model, we show that identifying these centers (i.e. salient anchor entities realized by annotation sub-tasks) constrains the output space complexity, and decompositions which isolate and advance center identification reduce the aggregate inferential load. We provide guidelines for decomposing complex structured annotation tasks, supported by examples demonstrating improved cost-efficiency from our prior work. Finally, we present a procedure for allocating subtasks across annotators to maximize quality under a fixed budget.

1

1

Introduction

Structured representations of text persist as a valuable modality for domain experts analyzing large corpora, where structuring refers to a mapping from text to a semantic, graphical representation of meaning composed of spanbound atomic units and relations, where the particular structure may be selected to support an application (Abend and Rappoport, 2017). Structure is necessarily grounded in text, which affords reliability in contrast with automating analyses completely (Maynez et al., 2020; Xiong et al., 2025), and structure is a compact, natural representation of the global properties of entities that experts value. Further, projection of structure onto text is a necessary step in inductive coding, which remains the standard for reliable qualitative methods (Glaser and Strauss, 2017). Rich structured annotation is historically prohibitively expensive to develop over large volumes of text. This is especially true of specialized domains (Kim et al., 2003; Li et al., 2016; Krallinger et al., 2015). It took thousands of hours to annotate concepts for the CRAFT biomedical article dataset (Bada et al., 2012). For the POLIANNA policy design dataset, Sewerin et al. (2023) report over 600 hours of annotation. The problem is compounded for high-dimensional, ambiguous tagsets or cases where documents are long and require expertise to interpret (Wright-Bettner et al., 2020). As LLMs and, more generally, neural architectures have improved in annotation capabilities, a natural approach to reducing costs is to substitute human annotators with models. In the conventional annotation paradigm, after agreement between annotators is established, each example is annotated in its entirety by individual annotators. To incorporate models as annotators, this paradigm can be adapted where models may substitute humans, potentially focusing on the subset of unambiguous examples (Li et al., 2023). Humans optionally annotate training examples which are used to develop the model (Rebholz-Schuhmann et al., 2010; Hahn et al., 2010). Alternatively, LLMs pre-annotate for the complete task, and human annotators optionally refine the pre-annotations (Naraki et al., 2024; Goel et al., 2023). Regardless of which annotation process is selected, human annotation effort is necessary for validation of the process. While LLM-generated datasets such as SynthIE (Josifoski et al., 2023), PileNER (Zhou et al., 2024), and NuNER (Bogdanov et al., 2024) are widely used, naive substitution of LLMs for human annotators does not necessarily transfer to extraction over specialized domains representing a conceptual distribution shift (Zhang et al., 2024; Golde et al., 2025), especially with respect to annotation specifications that require a substantial number of examples to 2

robustly adhere to, such as span boundary rules (Volkanovska, 2025). Such tasks exhibit high inferential load, with a vast space of valid annotations, where annotators make several decisions with associated dependencies in a single pass, some of which may require domain or linguistic expertise. Any limiting aspect of the task will bottleneck model performance. Accordingly, developing high-performing model-based annotation for specialized domains can be costly, making use of hundreds of annotated examples for finetuning (Zhou et al., 2024; Sainz et al., 2024). In this work, we propose to decompose tasks into sub-tasks that reflect distinct aspects of annotation complexity in order to more efficiently spend scarce annotation resources. In a heterogeneous, multi-annotator setting including both human and model annotators, we argue for (1) the re-design of annotation tasks to reduce the aggregate inferential load, and (2) the redistribution of annotator efforts to aspects of a task that can be performed efficiently for the given annotator. First, to reduce aggregate inferential load, decomposition of annotation tasks can isolate the identification of centers—salient discourse entities that are tracked and updated over the course of annotation, as characterized by (Grosz et al., 1995a)—since jointly inferring centers and their attributes expands the effective output space, whereas fixing centers apriori collapses this space and simplifies subsequent decisions. Second, decomposition allows for the redistribution of annotator efforts such that each atomic sub-task can be assigned to the least costly, adequate annotator, and expert effort can be reserved for the most demanding aspects of a task. We first specify the desiderata for annotation projects in heterogeneous multi-annotator settings (§2). Our primary contribution is a model of inferential load for structured representation annotation (§3) supporting task decomposition as a more efficient use of scarce annotation resources. With example annotation tasks, we illustrate the savings in aggregate inferential load from decomposition, which we support by highlighting empirical evidence from our own recent work that task decomposition is a more efficient use of annotation resources (§4). For brevity, we present only an overview of core results from our supporting works, and we encourage interested readers to refer to the original publications for further details. We additionally provide a procedure for allocating annotation sub-tasks to diverse sets of annotators such that the budget is satisfied and annotation quality is maximized (§6).

3

2

Design Objectives for Annotation Projects in Multi-Annotator Settings

We begin by reviewing the core goals of annotation projects in a setting with a heterogeneous set of annotators. Low Cost The cost of an annotation project depends both on the number of examples that need to be annotated and the inferential load required for each individual example. The inferential load of annotation depends on the complexity of an annotation structure, which is tied to utility, highlighting a trade-off between inferential load and downstream utility. For example, a large, granular set of ambiguous, higher order tags may be more useful than a small, less expressive set of simple tags, but at the same time exert a higher cognitive load on the annotator (Ma et al., 2023). For a given task, the inferential load for a human or model annotator is not necessarily the same, in absolute terms or relative to other tasks. For a given task, we can also expect the cost of model inference and training, if it is performed, to differ from the cost of human expert annotation. We can expect that for heterogeneous sets of annotators including humans and models, the relative costs of tasks will differ. To spend annotation resources efficiently, a given annotator should be assigned such that their efforts are concentrated on the aspects of a task that are cheap relative to other annotators. High Quality In addition to minimizing cost, it is necessary to establish both reliability, in that the task is well-defined and can be annotated consistently, and validity, in that that the task captures the truth about the phenomena we are interested in (Artstein and Poesio, 2008). Since annotators can share the same prejudices, strong agreement does not necessarily imply validity, so agreement is a necessary but insufficient condition. For example, Bayerl and Paul (2011) finds that expert annotators and non-expert annotators tend to have comparable agreement within-group, and poorer agreement for a mixed group of experts and non-experts. In practice, however, high agreement with an expert or consensus annotation is a widely-used heuristic for annotation quality. To show that the task is well-defined, consistency can be established between annotators over a sample of data. Consistency is a heuristic for shared understanding of guidelines across annotators, which is necessary to demonstrate that the task is well-defined, and that the proposed annotation procedure captures the purported phenomena documented in the annotation guidelines.

4

Standard measurements of consistency must be adapted in a heterogeneous, multi-annotator setting. Consistency between a pair of model annotators is not necessarily meaningful. For example, model annotators may share a similar training distribution, resulting in common biases and artificially high agreement (Szymanski et al., 2025; Bavaresco et al., 2025; Lu et al., 2025) with collapsed diversity in judgment (Messeri and Crockett, 2024). Agreement between a machine and human annotator is also not without limitations, as high agreement can suggest that the task is shallow or surface-driven and not necessarily well-defined. Historically, agreement between human experts is the standard approach to establish the clarity of annotation guidelines (Pustejovsky and Stubbs, 2012). Regardless of how agreement is measured, more informative signals may come from component-level consistency, since different parts of a task vary in their importance for downstream utility and in the extent to which they introduce disagreement with respect to the complete task, as seen in CRAFT concept annotation where unitization is a major bottleneck in annotator training (Bada et al., 2012). The extent to which task decomposition can reduce validation labor remains an open question that we leave for future investigation. In this work, we propose decomposition primarily as a vehicle for reducing inferential load and efficient distribution of annotation efforts.

3

Framework

Necessary steps to reduce the cost of high-quality annotation projects include minimizing the aggregate inferential load and efficiently allocating annotator efforts. In this work, we argue that decompositions can produce both these outcomes. First, with respect to aggregate inferential load, we will observe that the identification of centers, salient discourse entities that are realized over the course of annotation, is a primary source of complexity in structured annotation. Drawing on centering theory (Grosz et al., 1995a), which characterizes coherent discourse in terms of continuous centers and minimal shifts in attention, we model annotation as a process that requires tracking and updating such entities. When centers are not specified in advance, annotators must simultaneously identify and characterize them, along with any dependencies prescribed by the structure. This joint burden expands the effective output space, as the annotator considers each plausible center assignment alongside its set of attributes. In contrast, when centers are fixed, the output space collapses and subsequent decisions are constrained. In a decomposed setting, we

5

illustrate that the aggregate output complexity can be reduced by identifying centers in isolation of other aspects of the annotation task. Second, task decomposition allows for more efficient allocation of annotator effort, where sub-tasks are assigned to the least costly, adequate annotator. Sub-tasks represent focused, atomic tasks, with strictly fewer inferential leaps than the complete task. Under decomposition, impeding aspects of annotation can be isolated as sub-tasks, such that expensive annotator expertise is spent only on the aspects of the task where it is necessary. To analyze how annotation design shapes resource efficiency, we formalize the annotation process in a way that makes inferential load and effort allocation explicit. This framework enables a precise characterization of the annotation project design problem (§3.1), a model for quantifying inferential load (§3.2), and a concrete definition of task decomposition (§3.3).

3.1

Problem Formulation

Our objective is to design an annotation project that produces annotations of the highest possible quality over the complete corpus given an annotation task, a set of annotators, and a fixed budget. In designing an annotation task, we refer to two core mapping decisions. The first decision, is if and how to decompose the annotation task 𝑡 into sub-tasks 𝑡1 , 𝑡2 , . . . , 𝑡 𝑘 , where executing the sub-tasks is equivalent to executing the complete task. In the second decision, annotators 𝐴1 , . . . , 𝐴ℓ are assigned to sub-tasks, where each subtask is performed by one annotator. In this work, the decomposition and an annotator assignment is selected to maximize annotation quality, which we take to be agreement with expert human annotation of a sample of the corpus.

3.2

Model of Inferential Load

In this section, we propose to model inferential load as a function of three components: (1) the prerequisite information processing necessary to perform the annotation, (2) the space of decisions that the annotator considers, and (3) the quantity of salient entities realized by the annotation. The necessary information processing for annotation includes the document to be annotated and the task instructions, which we formalize as sets of span-bound units. The space of all possible annotations refers to the broad set of valid annotations given the parameters of a task, of which the annotator selects one. Finally, the process of annotation introduces salient entities from the task input, representing slots that the annotation task instructions expect to be filled by the output of annotation. To more formally measure these aspects of in-

6

ferential load, we develop a framework to analyze annotation in its complete or decomposed form.

Figure 1: We illustrate the atomic space for a coreference resolution task decomposition, where 𝑡1 is typed mention identification and 𝑡 2 is antecedent linking. The top row represents the atomic space of a single instance of task 𝑡1 , and the subsequent rows represent instances of the task 𝑡 2 . The dashed links between boxes denote the transfer of seed centers between sub-tasks. We denote the introduction of a new center in yellow. Each extracted mention represents a new center, and centers are introduced for each new entity created at the linking sub-task. See the graphical representation of this decomposition of coreference resolution in Figure 4.

3.2.1

Annotation Task

An annotation task is a transformation of a structured or unstructured input to a structured output using a set of instructions. More formally, an annotation task instance is a function that transforms a set of input atoms, or markable units, to a set of output atoms, or annotated units, using a set of map atoms, or mapping concepts. An atom represents a sequence of tokens or a substructure, with optional properties or characteristics. We define the three types of atoms below. 1. Map atoms: Representing the structure the document is mapped to, map atoms appear in the task instructions (e.g. class types for NER, relation types for relation extraction, event triggers for event argument

7

extraction). Map atoms may also be introduced from the input atoms while performing the task (e.g. a new active entity in coreference resolution). 2. Input atoms: The units to map primarily appear in the document (e.g. mentions for mention identification). The input atoms are restricted to the document context necessary to perform the annotation task, not necessarily the complete original input document. 3. Output atoms: The atoms representing the product of annotation (e.g. typed mentions, a pair of typed mentions and a relation type, linked coreferent mentions with a corresponding entity type). Figure 1 represents an example atomic space for a decomposed coreference resolution task. We additionally define two primitive annotation tasks in terms of these atomic spaces, where a task is a sequence of one or more of the following primitives performed in secession. 1. Atom selection: One or more subsets of contiguous atoms are selected from the set of input atoms. 2. Atom classification: One or more subsets of atoms are classified from the input atoms with respect to one or more map atoms. We also introduce centers, which are the entities salient to the annotation task appearing in both the annotated output atoms and the input atoms. Centers are the discourse entities realized during annotation, where the entities represent individuals, objects, or concepts that can be tracked over the course of annotation. In other words, centers represent the entities that annotators are tasked to identify or characterize. An annotation task may specify that entities are identified from an atomic space, in which case the resultant output atoms represent new centers (center identification task). Otherwise, the annotation task specifies that information about an entity or group of entities be updated, in which cases, the current set of centers is enriched (enrichment task). This may entail marking the precise span boundaries for a given mention vicinity, classifying the entity type of mention, or determining if the mention is referent with another mention. This notion of centers in part follows from centering theory (Grosz et al., 1995a) and focus spaces (Grosz and Sidner, 1986), where each annotation task instance is associated with a unique focus space containing salient entities. In our case, salient entities are those which are mentioned explicitly in the annotation output or are necessary for performing the annotation. Centers are not bound to a precise surface

8

form, and we expect that they evolve in atomic representation over the course of annotation. The notion of a center is more meaningfully fixed in the annotator’s mental model, where the centers are iteratively updated with new information for chains of dependent sub-tasks. Rather than consider a single center at a time, our model assumes that annotators consider multiple centers at a time, where there is an inferential cost incurred from introducing a new center to the set, signaling a shift in focus (Grosz and Sidner, 1986). 3.2.2

Features of Annotation Tasks Affecting Inferential Load

Input and Map Space Given the input atoms and map atoms, the fixed dimensionality of the atomic spaces can be computed. The input space represents the relevant context that the annotator must process in order to perform the annotation. The map space represents the active sub-structure that the input atoms can be projected onto, so the annotator necessarily considers the map space dimensionality. Accordingly, we assume both the input and map atomic space affect inferential load, as they represent the space of units and tools for transformation performed during annotation. Output Space The output atomic space represents the set of all possible output atom sequences that are valid given the annotation task and input text. The dimensionality of this space can be computed using the length of the input atom sequence, the dimensionality of the tag set, and the task instructions. We provide examples for how to compute the dimensionality of the output space for a few tasks in §4. Center Introduction In atom selection tasks, the surface form of centers are selected, and in atom classification tasks, the surface form of centers are classified. We assume that for a given annotation task 𝑡𝑖 , if there is a dependency between 𝑡 𝑖−1 and 𝑡 𝑖 , then any number of centers from the output space of 𝑡𝑖−1 can be seeded as an anchor point for the annotation. Seed centers of an instance of task 𝑡 𝑖 , in this context, refers to any center that is passed as a parameter to the instance. We can see an example of mentions and entities passed as seed centers in Figure 1. In this work, we are especially interested in the quantity of centers that are introduced during the annotation task, where these new centers are disjoint from the seed centers, if there are seed centers.

9

3.3

Decomposition

An annotation task 𝑡 can be broken into a sequence of primitives 𝑡 1 , . . . , 𝑡 𝑘 . We model dependencies between primitives in a graphical model with plate notation, where executing the primitives in the order of a topological sort is equivalent to executing the task 𝑡. The graphical model represents the annotation of a single task, and each node in the graph represents the execution of an instance of the task. Edges between nodes 𝑡 𝑖 and 𝑡𝑖+1 represents a dependency, such that the execution of task 𝑡𝑖+1 is dependent on the outcome of executing task 𝑡 𝑖 . For root tasks, the input atoms are from the initial unstructured input document (i.e. the input atoms for 𝑡 0 ). For each child task 𝑡𝑖+1 , the input atom set is a function of the parent task 𝑡 𝑖 output atoms and the initial input document. The map atoms are specified by the task 𝑡𝑖 , and the output atoms of 𝑡 𝑖 is the transformation of the input atom sequence. If there is an edge from 𝑡 𝑖 to 𝑡𝑖+1 , then any subset of centers can be passed as seed centers. The plate notation represents repeating variables in the graphical model. If an edge from 𝑡 𝑖 to 𝑡𝑖+1 crosses a plate, then the set of input atoms of the 𝑗’th repetition of the task 𝑡 𝑖+1 is a function of the 𝑗’th output atom for 𝑡𝑖 . The number of repetitions is specified in the bottom right corner of each plate. Applying any series of edge contractions to the primitive graphical model produces a valid decomposition. We can see examples of two valid decompositions of an NER task in Figure 2. 3.3.1

Constraints on Decomposition

A task should be decomposed if the decomposition represents a lower aggregate inferential load. We consider several dimensions of inferential load of an annotation task, specifically the combinatorial atomic output space, the input space, the map space, and the number of centers realized by the annotation task. For an annotator 𝐴, we assume that each dimension has a unique impact on the annotator’s inferential load, which we represent with weights 𝑤 o , 𝑤 i , 𝑤 m and 𝑤 center . This gives us the following expression to represent inferential load for a decomposition 𝑡1 , . . . , 𝑡 𝑘 .

10

𝑡1

𝑡2

𝑡 1,2

𝑡3 𝑀

𝑡3 𝑀

Figure 2: The top row is a pictoral representation of two candidate decompositions of NER: {𝑡1 , 𝑡2 , 𝑡3 } and {𝑡1,2 , 𝑡3 }. The bottom row is a graphical representation of the decompositions. In the primitive decomposition graph on the left, 𝑡1 is position identification, 𝑡2 is unitization, and 𝑡 3 is typing. Contracting the edge (𝑡1 , 𝑡2 ) produces the graph on the right where 𝑡1,2 represents mention detection. Both graphs are valid decompositions for the NER task 𝑡. In this example 𝑡1 , 𝑡2 , 𝑡1,2 are atom selection tasks, and 𝑡3 is an atom classification task.

Inferential Load({𝑡 1 , . . . , 𝑡 𝑘 }) =

𝑘 ∑︁

Centers Introduced(𝑡 𝑗 ) · 𝑤 center 𝑗=1 + Output Space(𝑡 ) · 𝑤 𝑗 o + Input Space(𝑡 𝑗 ) · 𝑤 i + Map Space(𝑡 𝑗 ) · 𝑤 m

(1) where across all instances of task 𝑡 𝑗 , # Centers Introduced(𝑡 𝑗 ) represents the number of new centers realized during annotation, Output Space(𝑡 𝑗 ) represents the dimension of the combinatorial atomic output space, Input Space(𝑡 𝑗 ) represents the number of atoms in the input spaces, and Map Space(𝑡 𝑗 ) represents the number of atoms in the map spaces. We constrain decompositions based on our notion of inferential load such that a valid decomposition should be used if Equation 2 is satisfied. Inferential Load({𝑡 1 , 𝑡2 , . . . , 𝑡 𝑘 }) ≤ Inferential Load({𝑡})

(2)

These dimensions of inferential load are intended to capture the information processing cost, the complexity of the annotation task, and finally the 11

entities realized by annotation. The input and map space make up the units we consider when performing annotation, while the output space complexity represents the number of ways the annotation could be performed. The number of centers is the number of meaningful new discourse entities in the annotation output. Note on the conditions where Equation 2 is satisfied Decomposition can reduce aggregate inferential load by reducing the output space complexity. As we will observe in §5, the output space dimensionality shrinks when tasks are decomposed such that center introduction is isolated from enrichment tasks. The other dimensions of inferential load may incur a larger cost under decomposition due to redundant information processing. With respect to center identification, if a decomposition does not have a sequence of redundant or unnecessary sub-tasks, the decomposition will not introduce any more centers than the complete task. Task decomposition, does however create multiple instances of the same task, producing redundant information processing. Context will be reused to perform different sub-tasks in a decomposition, so the cost incurred by the aggregate input and map space may outweigh the benefit of reducing the output space complexity for some annotators. In these cases, decomposition would not reduce the inferential load, and the constraint would not be met. We elaborate further on the desiderata of decompositions that meet this constraint (Equation 2) in subsequent sections §4 and §5. 3.3.2

Relationship between Atomic Space and Inferential Load

For machine annotation, decomposing tasks to reduce the aggregate output space is common and in some cases has been shown to be better performing. Zhou et al. (2024) show that extracting mentions for individual types results in better performance than eliciting mentions for all types in a single pass. Liu et al. (2025) decompose event extraction task into sub-tasks that are either performed with distant supervision or using an LLM annotator. When extracting information from tabular data, it is common to decompose tasks into sub-tasks (Zhang et al., 2025) with intermediate representations of tabular data (Wang et al., 2024). To address the large pool of candidates for cross-sentence arguments in implicit event argument detection, Zhang et al. (2020) propose to decompose the search to first detect the argument head word, followed by head-to-span expansion. Zhang et al. (2020) motivate this decomposition in the reduction of the output space dimension: by identifying head words first, the space of candidate spans scales linearly with the document length, not quadratically.

12

Decomposing tasks into sub-tasks with a smaller output space can make smaller models competitive with larger models. Bhattacharyya et al. (2025) finds that for information extraction, processing semantic blocks in visually rich documents independently results in better performance than processing whole documents in a single pass. They further find that the decomposition of processing can make smaller language models (32B) competitive with or outperform larger language models (200B). Similarly, Laurer et al. (2023) develop a BERT-based 0-shot classifier, decomposing a classification task into natural language inference tasks for individual classes, where each class is expressed as a hypothesis. For human annotation, previous empirical and theoretical work suggests that the inferential load is lower for sub-tasks with a smaller atomic space. Wei et al. (2018) investigate factors associated with NER annotation time for clinical text, finding that a larger atomic space is associated with higher annotation time (e.g. number of words, dependency distance, number of entities). Fort et al. (2012) establish several dimensions of annotation complexity, many of which underscore the high inferential load induced by a large atomic space. For example, they put forth the dimension of discrimination of markable units from the complete input document, which is a function of the number of input atoms and output atoms. The tag set dimension is a function of the number of map atoms. The weight of the context for performing the annotation task is a function of the number of input atoms. We note that some dimensions of complexity proposed in Fort et al. (2012) are not captured by the size of the atomic space, namely the expressiveness and the degree of ambiguity of map or input atoms. 3.3.3

Relationship between Center Introduction and Inferential Load

With respect to machine annotation, models tend to excel at center enrichment tasks, where the salient entities are pre-defined. Providing gold triggers enables an unsupervised event extraction model to perform comparably to a fully supervised system trained on thousands of labeled examples (Zhang et al., 2021). In coreference, LLMs can perform mention clustering robustly given gold mentions (Le and Ritter, 2024; Sundar et al., 2024). In crossdocument Abstract Meaning Representation parsing, Ahmed et al. (2024) develop a co-pilot annotation system where given roleset IDs and event triggers, the model suggests arguments. They find that LLMs can accurately recover trigger lemmas and roleset IDs, while struggling with full argument structures. Further, Wei et al. (2018) suggest that center identification may be a more

13

Variable 𝑁 𝐶 𝑀 𝑆 𝑇 𝑑 𝑘 𝑚𝑗

Description Number of tokens in the document Number of characters in the document Number of mentions in thedocument 
Number of possible spans i.e. 𝑁2+1 Number of entity types Maximum span width Number of entities The 𝑗-th mention in a document

Table 1: Variables used in task decomposition examples throughout §4. We express the atomic space and the number of centers introduced as a function of these variables.

important dimension of complexity than atomic space for manual annotation. Wei et al. (2018) find that the number of entities appearing in a document has a larger effect on annotation time than the number of words in the document. Here, the number of entities can represent the minimum number of times that the focus shifts, and the size of the input atom space can be taken as the document length.

4

Example Task Decompositions

We provide examples of task decompositions for two common annotation tasks: NER and coreference resolution to illustrate cases of valid decompositions which do and do not satisfy the constraint that decompositions reduce the aggregate inferential load (Equation 2). We describe for each task the inferential load incurred by the decomposition in contrast with that of the complete task. We show that for the first two examples, applying our model for inferential load suggests that the decomposition would result in a lower aggregate inferential load, assuming a high relative cost for output space complexity. The low inferential load of decomposition in these examples is also supported by our previous empirical work where we validate efficiency gains from decomposition with respect to annotator time. For the last example, we find that decomposition results in a higher aggregate inferential load where there is unnecessary factorization and a high information processing cost (𝑤 i , 𝑤 m ) relative to output complexity (𝑤 o ). NER For NER, the complete task 𝑡 entails identifying named entity spans from a document. We can decompose the task into first selecting typed posi-

14

tions in the text where a span appears 𝑡 1 , then unitizing the span, or identifying the span boundaries for the given typed position 𝑡2 (Figure 3). We can observe a dependence between 𝑡1 and 𝑡 2 , where the position to be typed realized by 𝑡 1 is taken as a seed center in the input space of 𝑡 2 . Each position identified in 𝑡 1 is typed and unified, so there are 𝑀 instances of 𝑡 2 , where 𝑀 denotes the number of mentions in the document. If we let 𝑁 denote the number of
tokens in the document, 𝑆 = 𝑁2+1 is the set of all possible spans. Given 𝑇 entity types, we have (𝑇 + 1) 𝑆 ways to assign types to the set of all possible spans, which is the atomic output space dimension for the complete task 𝑡. For the decomposition, there are (𝑇 + 1) 𝐶 possible typed position selections for a document with 𝐶 characters yielding the output complexity of 𝑡 1 . For a given position, if 𝑑 represents the number of characters in the longest span,
there are at most 𝑑+1 ways to unitize the position. The aggregate output 2
complexity for the decomposition is (𝑇 + 1) 𝐶 + 𝑀 𝑑+1 2 , which is smaller than (𝑇 + 1) 𝑆 as we show in §A. This in combination with no additional centers gives us a lower aggregate inferential load for decomposition. The value of decomposition for NER is supported in Gandhi et al. (2026): We find that decomposing the concept annotation task to typed position identification and unitization results in a more efficient use of manual annotation time, where manual annotation is focused on position identification rather than unitization. The output complexity for typed position annotation is (𝑇 + 1) 𝐶 which is much smaller than the complete task (𝑇 + 1) 𝑆 , suggesting that annotating typed positions instead of complete NER would take less time, which is supported by timed annotation experiments (Andrade et al., 2024). We also found unitization to be a cheap task to develop a machine annotator for, requiring as few as 100-200 example sentences, which is supported by our model of inferential load, as 𝑡3 has a negligible atomic space with a small output complexity, and it introduces no new centers. Coreference Resolution We consider next a typed coreference task 𝑡 in Figure 4, where given a document we extract a set of typed mentions and each pair of mentions is classified based on whether they corefer. We can decompose the complete task into two sub-tasks: mention identification and antecedent linking. For the first sub-task 𝑡1 , we identify a set of typed mentions from the document, giving us a sequence of mentions 𝑚 1 , 𝑚 2 , . . . , 𝑚 𝑀 . In sub-task 𝑡2 , for each mention 𝑚 𝑗 , we identify if there exists an antecedent among the previous mentions 𝑚 1 , 𝑚 2 , . . . , 𝑚 𝑗 −1 . If not, a new entity is introduced, representing 𝑚 𝑗 . We can observe a dependence between 𝑡1 and 𝑡2 , where mentions can be linked only after they are identified. We iterate

15

𝑡1

𝑡2 𝑀

Task

Description

Output

Input

Map

Centers

Instances

Identify named entity spans

(𝑇 + 1) 𝑆

𝑁

𝑇

𝑀

1

𝑡1

Identify typed positions

(𝑇 + 1) 𝐶

𝐶

𝑇

𝑀

1

𝑡2

Unitize position

𝑑+1
2

2𝑑

0

0

𝑀

𝑡

Figure 3: The DAG illustrates a decomposed NER task, and the following table contrasts the dimensions of inferential load for the sub-tasks in the decomposition and
the complete task. Taking the aggregate output space, we have (𝑇 + 1) 𝐶 + 𝑀 · 𝑑+1 ≤ 2 (𝑇 + 1) 𝑆 , which we show in §A. We can also observe that the decomposition does not introduce additional centers. With the exception of the input space, the decomposition represents a lower inferential load than the full task.

through Í each mention pair, so the aggregate output space of the linking task 𝑡2 is 𝑀 𝑗=1 𝑗 = 𝑀 (𝑀 + 1)/2. For sub-task 𝑡 1 , the space of all possible typed spans is (𝑇 + 1) 𝑆 , where 𝑆 is the set of all possible spans and 𝑇 is the number of types. We introduce 𝑀 centers at the mention identification stage, and 𝑘 centers at the linking stage, where there are  𝑘 entities. The output complexity for the complete task, is the product of 𝑀 (i.e. a Sterling number of the 𝑘 second kind denoting the space of possible linking decisions) and (𝑇 + 1) 𝑆 (i.e. the space of possible typed spans). If we assume at least 10 mentions, more than one coreference cluster, and at least one non-singleton coreference cluster, we can observe in §B that the output space of the decomposition is smaller than that of the complete task. This, in combination with no new centers introduced by decomposition gives us a lower inferential load for the decomposition if the additional processing of the input and map space is not dominant (i.e. 𝑤 𝑖 , 𝑤 𝑚 are relatively small). The value of decomposition for annotation efficiency is also empirically supported by Gandhi et al. (2023). Through annotation timing experiments, we found mention annotation to be almost twice as fast to manually annotate than full coreference resolution, which is supported in our model of inferential  load in that the output complexity for 𝑡1 is smaller than 𝑡 by a factor of 𝑀 𝑘 . We also found that decomposing the coreference task into mention detection and linking during annotation results in a more efficient use of annotator time, which also follows from our model. 16

𝑡1

𝑡2 𝑀−1 𝑀

Task

Description Coreference Resolution

𝑡

Output 𝑀 𝑘

(𝑇 + 1) 𝑆

Input

Map

Centers

Instances

𝑁

𝑘

𝑀+𝑘

1

𝑀

1

noref(𝑖, 𝑗 )

𝑀

𝑡1

Mention Identification

(𝑇 + 1) 𝑆

𝑁

0

𝑡2

Link 𝑚 𝑗 to an antecedent

𝑗

1

𝑗−1

Ö

𝑚𝑖 : 𝑖< 𝑗

Figure 4: The DAG illustrates a decomposed coreference task, and the following table contrasts the dimensions of inferential load for the sub-tasks in the decomposition and the complete task. Let the function noref(𝑖, 𝑗) = I(𝑚 𝑗 does not refer to entity(𝑚 𝑖 )), where the product indicates that there does not exist any active entities that 𝑚 𝑖 can link to, and it is necessary to create a new entity. In the decomposition, we first identify all mentions 𝑡1 , and classify each mention pair based on whether the mentions corefer (i.e. whether mention 𝑚 𝑗 corefers with an antecedent 𝑚 𝑖 ) in 𝑡 2 . A new entity type is created when none of the antecedents corefer with 𝑚 𝑗 , giving Í Î us 𝑖 𝑚𝑖 :𝑖< 𝑗 I(𝑚 𝑗 does not refer to entity(𝑚 𝑖 )) = 𝑘. Therefore, the decomposition does not introduce new centers. We further observe that output space  the 𝑡2 aggregate 𝑆 is 𝑀 (𝑀 + 1)/2, giving us 𝑀 (𝑀 + 1)/2 + (𝑇 + 1) 𝑆 ≤ 𝑀 𝑘 (𝑇 + 1) , which we show in §B.

Excessively Decomposed NER For our last example, we consider NER again, but with an alternate decomposition (Figure 5). Given a document with 𝑁 tokens, we first identify untyped positions in the first sub-task 𝑡1 . In the second sub-task 𝑡2 , the 𝑀 positions are unitized. Finally, we perform 𝑇 binary type classification tasks for each of the 𝑀 spans in 𝑡3 . The output complexity of the decomposition is still bounded by the output complexity of the complete task. However, the map space and input space of the decomposition is much higher. We present this example to illustrate the potential cost incurred when the total number of task instances is inflated by decomposition, resulting in a larger aggregate input and map dimension. For a given annotator, if the cost of processing task input and maintaining the set of map atoms (𝑤 𝑖 , 𝑤 𝑚 ) is high relative to the cost of output complexity 𝑤 𝑜 , then the

17

complete task should not be decomposed. While we did not empirically test this setting in our work (Gandhi et al., 2026), it is possible that the additional reading and processing costs incurred from excessive decomposition can outweigh the other dimensions of inferential load, making decomposition a poor use of resources. For example, if a model annotator is equally performant for either the typed position identification task or identifying positions and typing in secession, the incurred cost would be strictly higher for the decomposition. We can also observe more generally that our model of inferential load penalizes the decomposition of enrichment tasks. In this example, typing as a binary classification task doubles the map space and output space, without reducing the inferential load along any other dimension. The potential cognitive relief that would result from meaningful decomposition of classification tasks with respect to a hierarchical schema is not captured by our model of inferential load. It does not take into account the complexity of tag sets (e.g. higher order, ambiguous concepts), in that tags are not treated as divisible, so the potential reduced cognitive load in classifying units with sub-tags is not considered.

𝑡1

𝑡3

𝑡2

𝑇 𝑀

Task

Description

Output

Input

Map

Centers

Instances

𝑡

Identify named entity spans

(𝑇 + 1) 𝑆

𝑁

𝑇

𝑀

1

Identify positions

2𝐶

𝐶

𝑇

𝑀

1

𝑡2

Unitize position

𝑑+1
2

2𝑑

0

0

𝑀

𝑡3

Binary position typing

2

1

2

0

𝑀 ·𝑇

𝑡1

Figure 5: The DAG illustrates an excessively decomposed NER task, and the following table contrasts the dimensions of inferential load for the sub-tasks in the decomposition and the complete task. all possible spans. Let the context window from which we unitize typed positions be 𝑑, which is a heuristic for the maximum span length. Although the output complexity of the decomposition is smaller than of the complete task (§C), the input space and map space of the decomposition are both higher than the complete task. For the input dimensionality, we can see 𝑁 + 2𝑑 · 𝑀 + 𝑀𝑇 ≥ 𝑁, and for the map dimensionality, we can see 2𝑇 · 𝑀 ≥ 𝑇.

18

5

Discussion

5.1

Optimal Task Decomposition

As we observe in the examples from §4, dimensions of inferential load do not uniformly increase or decrease under decomposition. The potential benefit of decomposition is focused in reducing the output space complexity, and this is valuable in cases where annotators are in fact relatively sensitive to this dimension, as we denote with 𝑤 𝑜 . We argue that the majority of annotators experience this sensitivity; In §3.3.2, we highlighted previous work demonstrating that both machine and human annotators are sensitive to output space complexity. In this subsection, we consider features of task decompositions that necessarily reduce the aggregate inferential load, in particular with respect to the dimension of output space complexity. First, we observe that center identification sub-tasks tend to represent source components in the graphical model representation of task decompositions. Enrichment sub-tasks are dependent on center identification sub-tasks, so decompositions that follow the natural flow of annotation would represent center identification as independent sub-tasks. For enrichment tasks, it is necessary to identify an entity apriori to characterize and enrich with additional information. Independent source sub-tasks are naturally executed first in the topological sort of decomposition graphs, such that enrichment subtasks follow, dependent on the centers identified in the first sub-task(s). Second, we can observe that the mechanism for reducing the inferential load for sub-task instances is center identification. Center identification for a sub-task instance will necessarily affect the input space and subsequently the output space complexity for any dependent sub-task instance. For any dependency crossing a plate in a graphical model representation of the decomposition, the seed center is passed to the dependent sub-task instance and the attention of the annotator is restricted. More formally, for any task incorporating an atom selection primitive, the output space is the atomic map dimension scaled exponentially with the atomic input space dimension. For each atom selection sub-task performed to identify centers (e.g. identify positions, identify mentions), the input space shrinks for the following sub-task to the seed centers. Accordingly, we argue that the tasks that can be decomposed with lower aggregate inferential loads contain at least one center identification sub-task. For such tasks, decompositions that reduce the inferential load share the following features: (1) Center identification is isolated from enrichment, and (2) the number of candidate centers is minimized. The dominating term in

19

the aggregate output space complexity is a function of these two features. This term is the size of the map space scaled exponentially with the size of the input space, where the input space is tied to (2) and the map space is tied to (1). Efficient decompositions isolate center identification sub-task In optimal task decompositions, the center identification sub-task is isolated completely from enrichment. Since the center identification sub-task(s) represents the source component of the decomposition graph, the output space of center identification is the atomic map dimension scaled exponentially with the atomic input dimension. We can expect the center identification task to represent the largest input dimensionality of all the sub-tasks, with the largest resultant output dimensionality. In order to reduce the output dimensionality for the source component, the map space can be reduced as much as possible. For example, rather than typing and detecting mentions jointly, our model of inferential load would prefer to detect mentions, and type those mentions in secession, as the aggregate output space would be lower for the decomposition if the mention types are not included in the map space at mention detection time. If we design a decomposition such that center identification sub-tasks are performed first and isolated from other sub-tasks that characterize and enrich the centers, then subsequent sub-tasks operate under a smaller input space and subsequently have a smaller output space. Efficient decompositions minimize the number of candidate centers In order to drive down the output space complexity of center identification tasks, the input space, or the space of possible centers should be as small as possible. The smallest such space of candidate centers is linear with the input space. Consider for example, the position identification sub-task in the decomposition of NER from Figure 3: The space of candidate centers is the set of characters, which is exactly the input space. This is in contrast with standard annotation practices for span-bound center identification, where the space of candidate centers is the space of all possible spans in the document. Considering that center identification tasks contribute the dominant term with respect to output space complexity, representing centers as simply as possible is critical to efficient annotation. Given a rudimentary surface form representation of a center, it is cheap to perform subsequent atom selection sub-tasks over a small, constrained search space in order to arrive at the final span-bound representation of a center (e.g. fixed-width unitization).

20

5.2

When to Abstain from Decomposition

At the same time, even if a task containing center identification is decomposed according to these two conditions, there may be settings where the potential benefits in reduced output complexity are outweighed by processing costs or cascading errors that can plague decomposed settings. High Cost of Information Processing In cases where the cost of repeatedly processing input space is high, decomposition results in a larger inferential load. The developers of GLiNER2, for example, consider such a setting. GLiNER2 is a small multi-task Information Extraction model with fewer than 500 million parameters designed to run on CPU-hardware and trained to perform, in a single pass, compositions of NER, hierarchical structure extraction, and text classification. Zaratiana et al. (2025) inspect an enrichment task, finding that decomposing a 0-shot multi-class classification task to perlabel forward passes with a 435M DeBERTa model is both more computationally expensive and underperforms GLiNER2. This example, in part, highlights the case where training annotators to perform multiple related subtasks jointly may be more efficient than developing specialized annotators for individual sub-tasks. For annotation of complex structure, however, it is unlikely in practice that the cost of information processing in the decomposed setting is less significant than that of the output complexity in the complete task setting. For annotation of any text-bound structure, there will be at least one center identification task. As the number of dependent entities that make up the structure increase, the output space complexity exhibits iterated exponential growth with the number of entities. Under a decomposition, on the other hand, the cost of processing the map and input space grows polynomially for each task subdivision. Considering the strong relationship between atomic space complexity and inferential load cited in prior work (§3.3.2), we argue that the cost of information processing in the decomposed task setting will generally be lower than the potential output complexity in the complete task setting for annotators, especially in cases of complex structure with multiple interdependent salient entities. Cascading Errors under Decomposition Pipelined annotation systems are necessarily vulnerable to cascading errors (He et al., 2013), and the benefit from modeling related tasks in multi-task learning paradigm may outweigh the additional learning overhead for individual annotators. With respect to model annotation and neural architectures in particular, multi-task

21

learning can improve generalization ability (Caruana, 1997). For structured annotation tasks such as semantic role labeling (Peng et al., 2018), coreference resolution (Swayamdipta et al., 2018), and dependency parsing (Peng et al., 2017), multi-task learning with auxiliary objectives have been shown to improve model performance on individual tasks. The superiority of such systems over pipelined approaches is no longer as universal, however, as finetuned pretrained language models can be robust sub-task annotators, minimizing the effect of cascading errors (Gururangan et al., 2020). At minimum, such model annotators can perform sub-tasks that are downstream from high quality pre-annotation of core, ancestral, sub-structures.

5.3

Annotator Allocation: The Strongest Annotator Identifies Centers

In annotation resource constrained settings, concentrating resources on center identification sub-tasks will likely result in higher quality data. For one, center identification has a disproportionately large impact on the overall annotation quality of the complete task. Center identification represents the source component of the decomposition graph, so robust annotation of center identification sub-tasks would evade cascading errors in dependent sub-tasks. Second, we should expect that center identification tasks are cognitively hard and consequently expensive. This follows from our model of inferential load and centering theory (Grosz et al., 1995a). In our model of inferential load, we count the number of new centers introduced for each task, such that center identification tasks have at least one new center while enrichment tasks necessarily have no new centers. Center identification represents a clear shift in focus, where the salient entity is realized during annotation. Such tasks represent a higher cognitive load than enrichment tasks where there is continuity in the center.

5.4

Interaction between Centering Theory and Atomic Space Complexity

Our model of inferential load is consistent with centering theory (Grosz et al., 1995b) in that tasks that introduce centers, or shift the attention of annotators, have a large atomic output space. On the other hand, tasks that enrich or characterize centers (e.g. type position, unitize position, link mention to an existing center) necessarily have a small atomic output space. Such tasks represent a sequence of continuity for annotators, where centers are repeatedly reused. 22

The notion that reducing the atomic space can reduce the inferential load is also grounded in centering theory. For smaller input atomic spaces, there are fewer candidate entities or concepts to process while performing annotation, so we can expect annotators to experience fewer shifts in focus. Similarly, with respect to output space complexity, if there are fewer potential valid outputs to select between, the annotator experiences fewer shifts in focus when weighing candidate annotations.

6

Task Distribution Procedure

In this section, we propose Algorithm 1 for distributing sub-tasks among annotators. We estimate the performance of each annotator for each sub-task on a held-out annotated set, where the cheapest performant model is assigned to each sub-task. In the event that such an assignment exceeds the budget 𝐵, we propose to solve an optimization problem to select a student and teacher annotator for distillation, where we distill a higher quality annotator to a weaker, but cheaper annotator to achieve the best possible performance under the budget constraint. Sets of heterogeneous annotators can include both a range of models with variable size and a range of human annotators such as crowd-workers or costly domain experts. The cost of distillation includes both the cost of development of training materials and the cost of training. If the teacher annotator is a model, the cost of training material development is the cost of inference to produce silver-annotated data. If the teacher annotator is a human, the training material cost includes the time required to annotate examples or develop instructions to correct annotation errors. With respect to the cost of training, if the student annotator is a model, distillation follows the standard practice of training the weaker model with annotations produced by the stronger, teacher annotator (Hinton et al., 2015). For human student annotators, the cost of training includes the time it takes to review annotated examples or instructions. We describe in this section more formally how to select optimal distillation configurations We let 𝑚 annotators 𝐴1 , 𝐴2 , . . . , 𝐴𝑚 and performance metrics for each sub-task 𝑖: 𝑄ˆ 1𝑖 , 𝑄ˆ 2𝑖 , . . . , 𝑄ˆ 𝑖𝑚 . The threshold performance for 𝑗 each sub-task is 𝑄 𝑖 . Assigning annotator 𝑗 to sub-task 𝑖 has cost 𝑐 𝑖 , and the total budget is 𝐵. We denote the performance of annotator 𝑗 on sub-task 𝑖 as 𝑗 𝑄ˆ 𝑖 . The performance threshold for sub-task 𝑖 is denoted by 𝑄 𝑖 . With respect 𝑗 to cost, 𝑐 𝑖 is the cost of performing sub-task 𝑖 with annotator 𝑗, and the dis𝑗,𝑘 tillation cost 𝑑𝑖 is the sum of the costs of annotating a batch of data with 23

Algorithm 1 Assign Annotators to Sub-tasks under Budget Constraint Inputs: Sub-tasks {𝑡1 , . . . , 𝑡ℓ }, annotators { 𝐴1 , . . . , 𝐴𝑚 }, dataset D, quality thresholds {𝑄 1 , . . . , 𝑄 ℓ }, annotation budget 𝐵 Output: Annotation configuration useStrong, useDistilled 1: function assign-annotators({𝑡 1 , . . . , 𝑡 ℓ }, {𝐴1 , . . . , 𝐴𝑚 }, {𝑄 1 , . . . , 𝑄 ℓ }, D, 𝐵) 2: for 𝑖 = 1 to ℓ do ⊲ Initialize annotation assignments 3: for 𝑗 = 1 to 𝑚 do 4: useStrong[𝑖] [ 𝑗] ← False 5: for 𝑘 = 1 to 𝑚 do 6: useDistilled[𝑖] [ 𝑗] [𝑘] ← False 7: 8: 9: 10: 11:

Ddev ← Sample(D) Ddev ← Annotate(Ddev ) for 𝐴 𝑗 ∈ {𝐴1 , . . . , 𝐴𝑚 } do ⊲ Estimate annotator performance for sub-tasks for 𝑡𝑖 ∈ {𝑡1 , . . . , 𝑡ℓ } do 𝑗 𝑄ˆ 𝑖 ← Eval(Ddev , 𝐴 𝑗 , 𝑡𝑖 )

12: 13: 14:

for 𝑡 𝑖 ∈ {𝑡1 , . . . , 𝑡ℓ } do ⊲ Construct initial greedy assignment 𝑗 𝑗 𝑗 ′ ← arg min 𝑗 {𝑐 𝑖 |∀ 𝑗 ∈ [𝑚], 𝑄ˆ 𝑖 ≥ 𝑄 𝑖 } useStrong[𝑖] [ 𝑗] ← true if TotalCost(useStrong) ≤ 𝐵 then ⊲ Assignment satisfies budget constraint return useStrong, useDistilled else ⊲ Optionally select annotator pairs for distillation ˆ 𝑄) useStrong, useDistilled ← Distilled-Assignment(𝑄, return useStrong, useDistilled

15: 16: 17: 18: 19:

teacher annotator 𝑘 and training a weak student annotator 𝑗 on the annotated batch. Objective In Algorithm 1, Distilled-Assignment, refers to the following optimization objective (Equation 3) and constraints. Distilled-Assignment selects, for each sub-task, either a strong annotator or a distilled annotator derived from a stronger annotator, with the goal of maximizing overall annotation quality while satisfying a total budget constraint. The decision variables useStrong𝑖, 𝑗 and useDistilled𝑖, 𝑗,𝑘 indicate whether a strong annotator 𝐴 𝑗 or a distilled annotator derived from 𝐴 𝑗 to 𝐴 𝑘 is used for sub-task 𝑖, respectively. The objective is to maximize the total expected quality of annotations across all sub-tasks, with both direct annotation with strong annotators and distillation-based annotation with weak annotators. Each sub-task con𝑗 tributes according to the estimated performance of the selected annotator: 𝑄ˆ 𝑖 for strong annotators and 𝑄ˆ 𝑘 for distilled configurations, where the latter is 𝑖

24

proxied by the performance of the selected teacher annotator. i ∑︁ h∑︁ ∑︁ 𝑗 max useStrong𝑖, 𝑗 · 𝑄ˆ 𝑖 + useDistilled𝑖, 𝑗,𝑘 · 𝑄ˆ 𝑖𝑘 𝑖

𝑗

(3)

𝑗,𝑘

Constraints The optimization is subject to several constraints. First, exactly one configuration—either a strong annotator or a distillation pair—must be assigned to each sub-task. Second, only annotators whose estimated performance exceeds the sub-task threshold 𝑄 𝑖 may be used, including both strong annotators and teachers used in distillation. Third, the feasible set of teacher and student choices is restricted to Pareto-efficient annotators with respect to estimated performance and cost, ensuring that no dominated annotator is considered in the assignment. Fourth, the total cost of all assignments, including both direct annotation cost and the cost of distillation (i.e. teacher annotation and student training), must not exceed the total budget 𝐵. Finally, all decision variables are binary, reflecting the discrete selection of either a strong annotator or a valid distillation configuration for each sub-task. Constraints to Satisfy for Annotator Distillation Pair Selection (Distilled-Assignment) ∑︁ ∑︁ useStrong𝑖, 𝑗 + useDistilled𝑖, 𝑗,𝑘 = 1 ∀𝑖 ⊲ One annotator per sub-task 𝑗

𝑗,𝑘

useStrong𝑖, 𝑗 = 0 if 𝑄ˆ 𝑖 < 𝑄𝑖 𝑗

⊲ Strong annotators meet threshold

useDistilled𝑖, 𝑗,𝑘 = 0 if 𝑄ˆ 𝑖𝑘 < 𝑄𝑖

⊲ Teacher must meet threshold

𝑘′

𝑘′

useDistilled𝑖, 𝑗,𝑘 = 0 if ∃𝑘 ′ : 𝑄ˆ 𝑖 ≥ 𝑄ˆ 𝑖𝑘 ∧ 𝑐𝑖 ≤ 𝑐𝑖𝑘 ′ ′ ∧( 𝑄ˆ 𝑖𝑘 , 𝑐𝑖𝑘 ) ≠ ( 𝑄ˆ 𝑖𝑘 , 𝑐𝑖𝑘 ) 𝑗′

𝑗′

⊲ Pareto-efficient teachers only

useDistilled𝑖, 𝑗,𝑘 = 0 if ∃ 𝑗 ′ : 𝑄ˆ 𝑖 ≥ 𝑄ˆ 𝑖 ∧ 𝑐𝑖 ≤ 𝑐𝑖 ∑︁ h∑︁ 𝑖

𝑗

𝑗

useStrong𝑖, 𝑗 · 𝑐𝑖 +

∑︁

𝑗

⊲ Pareto-efficient students only 𝑗,𝑘

useDistilled𝑖, 𝑗,𝑘 · (𝑑𝑖

i 𝑗 + 𝑐𝑖 ) ≤ 𝐵

⊲ Satisfy budget

𝑗,𝑘

useStrong𝑖, 𝑗 , useDistilled𝑖, 𝑗,𝑘 ∈ {0, 1}

7

𝑗

⊲ Binary variables

Related Work

Modeling Task Complexity Prior work conceptualizes task complexity from both model and human-centric perspectives. Model-based approaches characterize complexity in terms of the number of reasoning units required (Tang et al., 2025), descriptive capability classes (Wang et al., 2025), or the size of the smallest model capable of performing the task (Bae et al., 2023). In parallel, human annotation studies use proxies such as annotation time 25

(Goel et al., 2023), disagreement between expert and lay annotators (Yang et al., 2019), and multi-dimensional linguistic features correlated with effort (Wei et al., 2018). Many of these dimensions can be mapped into our framework—for example, input length corresponds to the atomic input space, while the number of entities reflects the introduction of centers. Similar to this work, Fort et al. (2012) propose a broad taxonomy of annotation complexity. We differ in explicitly modeling centers as units of annotator focus and in introducing inferential load as a unified measure that can be applied to both human and machine annotators, extending naturally to heterogeneous annotator populations. In contrast with prior work, we propose a simplified model of inferential load to support redesign of annotation tasks for the sake of efficiency in multi-annotator settings. Task Decomposition as a Principle for Resource Efficiency Task decomposition for efficient annotation is closely related to partial annotation, where effort is focused on informative sub-structures rather than full structure annotation. This approach has been effective across tasks such as named entity recognition, dependency parsing, event extraction, and relation extraction (Zhang et al., 2023; Marcheggiani and Artières, 2014). Our work is focused on distributing annotation efforts across sub-tasks, but efficiency could be further improved by granular discrimination of informative sub-task instances with partial annotation. This perspective aligns with output-constrained annotation and interactive correction, where fixing parts of the structure reduces the search space of remaining decisions. For example, Culotta and McCallum (2005) show that partial corrections can constrain model predictions, effectively converting segmentation into simpler classification problems, highlighting that segmentation complexity grows exponentially with sequence length. Similarly, Michael (2023) advocate constructing complex annotations by bootstrapping from simple, narrowly scoped representations. These approaches mirror our notion of centers as core sub-structures that constrain downstream annotation and reduce effective complexity. Patterns in Annotation Task Decomposition Across a range of tasks, decomposition has been adopted to reduce the aggregate inferential load. For example, span extraction is often split into head identification followed by boundary detection, improving robustness in nested NER and mention detection (Lin et al., 2019; Peng et al., 2015). There have been similar decomposition approaches to document-level pseudo-coreference with sub-tasks such as pronoun identification, antecedent resolution, antecedent boundary res26

olution, antecedent head resolution (Jauhar et al., 2015; Liu et al., 2016). Conditioning-based approaches fix part of the structure to simplify the remaining structure, as in QA-SRL, which decomposes semantic role labeling into predicate identification and argument detection, enabling efficient crowd-sourced annotation with minimal expertise (He et al., 2015), further optimized through constrained interfaces such as autocomplete (FitzGerald et al., 2018). Related approaches to enable crowd-sourcing decompose complex questions into atomic operations (Wolfson et al., 2020). More broadly, effective decomposition enables sub-tasks that can be performed by nonexpert annotators or LLMs (Michael, 2023), consistent with early visions of interactive information extraction systems where users specify behavior through simple examples without requiring domain expertise (Cardie and Pierce, 1998).

8

Conclusion

In this work, we propose to decompose structured annotation tasks, isolating center identification as a sub-task for more efficient allocation of annotator effort, specifically in cases where aggregate inferential load resulting from decomposition is lower than performing a complete task with a single annotator. We formalize inferential load as a weighted sum of several dimensions of complexity (§3). This includes the units that are processed and realized over the course of annotation, specifically the atomic input space, map space, and output complexity. The inferential load incorporates the introduction of centers, or salient discourse entities that are realized in performing annotation, which we borrow from centering theory (Grosz et al., 1995a). Through examples in NER and coreference resolution, we illustrate how decomposition of structured annotation tasks, where center identification sub-tasks are isolated and performed first, can reduce the output complexity of subsequent sub-tasks by constraining the focus of annotation (§4). Given a decomposition that does reduce aggregate inferential load, we present an algorithm for selecting annotators for each sub-task, where we either greedily assign annotators to sub-tasks or develop annotators that meet budget constraints using distillation for each sub-task (§6). We highlight several important limitations of this work. Our notion of inferential load does not account for the ambiguity of individual atoms or whether the atoms are implicitly compositional in the input or map space which would likely have an effect in practice. For simplicity, this work also does not distinguish between the space of valid and reasonable annotations.

27

Decomposition necessarily results in context from the input space processed repeatedly for each sub-task instance, and there may be annotators for whom information processing is especially costly. Finally, this work presents a theoretical model of efficient decompositions, and while we have demonstrated the value of two particular decompositions for NER and coreference resolution (Gandhi et al., 2023, 2026), we have not explored alternate decompositions for these two tasks, let alone other structured representations. Despite these limitations, task decomposition is widely applicable for structured annotation under heterogeneous annotator settings, and we present a framework for the efficient design of such decompositions.

Acknowledgments Thanks to David Mortensen and Sireesh Gururaja for valuable feedback on early drafts of this work.

References Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 77–89, Vancouver, Canada. Association for Computational Linguistics. Shafiuddin Rehan Ahmed, Jon Cai, Martha Palmer, and James H. Martin. 2024. X-AMR annotation tool. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 177–186, St. Julians, Malta. Association for Computational Linguistics. Gabriel Herman Bernardim Andrade, Shuntaro Yada, and Eiji Aramaki. 2024. Is boundary annotation necessary? evaluating boundary-free approaches to improve clinical named entity annotation efficiency: Case study. JMIR Medical Informatics, 12(1):e59680. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555– 596. Michael Bada, Miriam Eckert, Donald Evans, Kristin Garcia, Krista Shipley, Dmitry Sitnikov, William A Baumgartner Jr, K Bretonnel Cohen, Karin 28

Verspoor, Judith A Blake, et al. 2012. Concept annotation in the craft corpus. BMC bioinformatics, 13(1):161. Henry Bae, Aghyad Deeb, Alex Fleury, and Kehang Zhu. 2023. Complexitynet: Increasing llm inference efficiency by learning task complexity. arXiv preprint arXiv:2312.11511. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics. Petra Saskia Bayerl and Karsten Ingmar Paul. 2011. What determines intercoder agreement in manual annotations? a meta-analytic investigation. Computational Linguistics, 37(4):699–725. Aniket Bhattacharyya, Anurag Tripathi, Ujjal Das, Archan Karmakar, Amit Pathak, and Maneesh Gupta. 2025. Information extraction from visually rich documents using LLM-based organization of documents into independent textual segments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17241–17256, Vienna, Austria. Association for Computational Linguistics. Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne P Bernard. 2024. NuNER: Entity recognition encoder pretraining via LLM-annotated data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11829– 11841, Miami, Florida, USA. Association for Computational Linguistics. Claire Cardie and David Pierce. 1998. Proposal for an interactive environment for information extraction. Technical report, Cornell University. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. Aron Culotta and Andrew McCallum. 2005. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pages 746–751.

29

Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060, Melbourne, Australia. Association for Computational Linguistics. Karën Fort, Adeline Nazarenko, and Sophie Rosset. 2012. Modeling the complexity of manual annotation tasks: a grid of analysis. In Proceedings of COLING 2012, pages 895–910, Mumbai, India. The COLING 2012 Organizing Committee. Nupoor Gandhi, Michael Bada, and Emma Strubell. 2026. Decomposing unitization and typing for efficient and consistent span-bound concept annotation. In Findings of the Association for Computational Linguistics: ACL 2026, San Diego. Association for Computational Linguistics. Nupoor Gandhi, Anjalie Field, and Emma Strubell. 2023. Annotating mentions alone enables efficient domain adaptation for coreference resolution. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10543–10558, Toronto, Canada. Association for Computational Linguistics. Barney Glaser and Anselm Strauss. 2017. Discovery of Grounded Theory: Strategies for Qualitative Research. Routledge. Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. 2023. Llms accelerate annotation for medical information extraction. In Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceedings of Machine Learning Research, pages 82–100. PMLR. Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, and Alan Akbik. 2025. Familiarity: Better evaluation of zero-shot named entity recognition by quantifying label shifts in synthetic training data. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 820–834, Albuquerque, New Mexico. Association for Computational Linguistics. Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995a. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225. 30

Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995b. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225. Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. Udo Hahn, Katrin Tomanek, Elena Beisswanger, and Erik Faessler. 2010. A proposal for a configurable silver standard. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 235–242, Uppsala, Sweden. Association for Computational Linguistics. He He, Hal Daumé III, and Jason Eisner. 2013. Dynamic feature selection for dependency parsing. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1455–1464, Seattle, Washington, USA. Association for Computational Linguistics. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643–653, Lisbon, Portugal. Association for Computational Linguistics. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Sujay Kumar Jauhar, Raul Guerra, Edgar Gonzàlez Pellicer, and Marta Recasens. 2015. Resolving discourse-deictic pronouns: A two-stage approach to do it. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 299–308, Denver, Colorado. Association for Computational Linguistics. Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting asymmetry for synthetic training data generation: SynthIE and the case of information extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1555–1574, Singapore. Association for Computational Linguistics. 31

J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl_1):i180–i182. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015. The chemdner corpus of chemicals and drugs and its annotation principles. Journal of cheminformatics, 7(Suppl 1):S2. Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2023. Building efficient universal classifiers with natural language inference. arXiv preprint arXiv:2312.17543. Nghia T Le and Alan Ritter. 2024. Are large language models robust coreference resolvers? COLM. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023. CoAnnotating: Uncertainty-guided work allocation between human and large language models for data annotation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1487–1505, Singapore. Association for Computational Linguistics. Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019. Sequence-tonuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5182–5192, Florence, Italy. Association for Computational Linguistics. Wenxuan Liu, Zixuan Li, Long Bai, Yuxin Zuo, Daozhu Xu, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2025. Towards event extraction with massive types: LLM-based collaborative annotation and partitioning extraction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34377–34399, Suzhou, China. Association for Computational Linguistics. Zhengzhong Liu, Edgar Gonzàlez Pellicer, and Daniel Gillick. 2016. Exploring the steps of verb phrase ellipsis. In Proceedings of the Workshop on 32

Coreference Resolution Beyond OntoNotes (CORBON 2016), pages 32– 40, San Diego, California. Association for Computational Linguistics. Junyu Lu, Kai Ma, Kaichun Wang, Kelaiti Xiao, Roy Ka-Wei Lee, Bo Xu, Liang Yang, and Hongfei Lin. 2025. Is LLM an overconfident judge? unveiling the capabilities of LLMs in detecting offensive language with annotation disagreement. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5609–5626, Vienna, Austria. Association for Computational Linguistics. Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10572–10601, Singapore. Association for Computational Linguistics. Diego Marcheggiani and Thierry Artières. 2014. An experimental comparison of active learning strategies for partially labeled sequences. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 898–906, Doha, Qatar. Association for Computational Linguistics. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. Lisa Messeri and Molly J Crockett. 2024. Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002):49–58. Julian Michael. 2023. The case for scalable, data-driven theory: A paradigm for scientific progress in NLP. In Proceedings of the Big Picture Workshop, pages 40–52, Singapore. Association for Computational Linguistics. Yuji Naraki, Ryosuke Yamaki, Yoshikazu Ikeda, Takafumi Horie, Kotaro Yoshida, Ryotaro Shimizu, and Hiroki Naganuma. 2024. Augmenting ner datasets with llms: towards automated and refined annotation. arXiv preprint arXiv:2404.01334. Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 33

Papers), pages 2037–2048, Vancouver, Canada. Association for Computational Linguistics. Hao Peng, Sam Thomson, Swabha Swayamdipta, and Noah A. Smith. 2018. Learning joint semantic parsers from disjoint data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1492–1502, New Orleans, Louisiana. Association for Computational Linguistics. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 12–21, Beijing, China. Association for Computational Linguistics. James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. " O’Reilly Media, Inc.". Dietrich Rebholz-Schuhmann, Antonio José Jimeno Yepes, Erik M. van Mulligen, Ning Kang, Jan Kors, David Milward, Peter Corbett, Ekaterina Buyko, Katrin Tomanek, Elena Beisswanger, and Udo Hahn. 2010. The CALBC silver standard corpus for biomedical named entities — a study in harmonizing the contributions from four independent named entity taggers. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). Basil Cameron Rennie and Annette Jane Dobson. 1969. On stirling numbers of the second kind. Journal of Combinatorial Theory, 7(2):116–121. Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2024. GoLLIE: Annotation guidelines improve zero-shot information-extraction. In The Twelfth International Conference on Learning Representations. Sebastian Sewerin, Lynn H Kaack, Joel Küttel, Fride Sigurdsson, Onerva Martikainen, Alisha Esshaki, and Fabian Hafner. 2023. Towards understanding policy design through text-as-data approaches: The policy design annotations (polianna) dataset. Scientific Data, 10(1):896. Kawshik Manikantan Sundar, Shubham Toshniwal, Makarand Tapaswi, and Vineet Gandhi. 2024. Major entity identification: A generalizable alternative to coreference resolution. In Proceedings of the 2024 Conference on 34

Empirical Methods in Natural Language Processing, pages 11679–11695, Miami, Florida, USA. Association for Computational Linguistics. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic scaffolds for semantic structures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3772–3782, Brussels, Belgium. Association for Computational Linguistics. Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the llmas-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI ’25, page 952–966, New York, NY, USA. Association for Computing Machinery. Bohan Tang, Huidong Liang, Keyue Jiang, and Xiaowen Dong. 2025. On the importance of task complexity in evaluating llm-based multi-agent systems. arXiv preprint arXiv:2510.04311. Elena Volkanovska. 2025. Large language models as annotators of named entities in climate change and biodiversity: A preliminary study. In Proceedings of the 1st Workshop on Ecology, Environment, and Natural Language Processing (NLP4Ecology2025), pages 24–33, Tallinn, Estonia. University of Tartu Library. Zhihu Wang, Shiwan Zhao, Yu Wang, Heyuan Huang, Sitao Xie, Yubo Zhang, Jiaxin Shi, Zhixing Wang, Hongyan Li, and Junchi Yan. 2025. ReTASK: Revisiting LLM tasks from capability, skill, and knowledge perspectives. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4925–4936, Vienna, Austria. Association for Computational Linguistics. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. Chain-of-table: Evolving tables in the reasoning chain for table understanding. ICLR. Qiang Wei, Amy Franklin, Trevor Cohen, and Hua Xu. 2018. Clinical text annotation–what factors are associated with the cost of time? In AMIA Annual Symposium Proceedings, volume 2018, page 1552.

35

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183–198. Kristin Wright-Bettner, Chen Lin, Timothy Miller, Steven Bethard, Dmitriy Dligach, Martha Palmer, James H. Martin, and Guergana Savova. 2020. Defining and learning refined temporal relations in the clinical narrative. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 104–114, Online. Association for Computational Linguistics. Guangzhi Xiong, Eric Xie, Corey Williams, Myles Kim, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. 2025. Toward reliable scientific hypothesis generation: Evaluating truthfulness and hallucination in large language models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 7849–7857. International Joint Conferences on Artificial Intelligence Organization. Main Track. Yinfei Yang, Oshin Agarwal, Chris Tar, Byron C. Wallace, and Ani Nenkova. 2019. Predicting annotation difficulty to improve task routing and model performance for biomedical information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1471–1480, Minneapolis, Minnesota. Association for Computational Linguistics. Urchade Zaratiana, Gil Pasternak, Oliver Boyd, George Hurn-Maloney, and Ash Lewis. 2025. GLiNER2: Schema-driven multi-task learning for structured information extraction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 130–140, Suzhou, China. Association for Computational Linguistics. Hongming Zhang, Haoyu Wang, and Dan Roth. 2021. Zero-shot Label-aware Event Trigger and Argument Classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1331– 1340, Online. Association for Computational Linguistics. Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, and Eduard Dragut. 2024. SciER: An entity and relation extraction dataset

36

for datasets, methods, and tasks in scientific documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13083–13100, Miami, Florida, USA. Association for Computational Linguistics. Xinyu Zhang, Aibo Song, Jingyi Qiu, Jiahui Jin, Tianbo Zhang, and Xiaolin Fang. 2025. Exploring multimodal relation extraction of hierarchical tabular data with multi-task learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26770–26781, Vienna, Austria. Association for Computational Linguistics. Zhisong Zhang, Xiang Kong, Zhengzhong Liu, Xuezhe Ma, and Eduard Hovy. 2020. A two-step approach for implicit event argument detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7479–7485, Online. Association for Computational Linguistics. Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2023. Data-efficient active learning for structured prediction with partial annotation and selftraining. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12991–13008, Singapore. Association for Computational Linguistics. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. Universalner: Targeted distillation from large language models for open named entity recognition. In International conference on learning representations, volume 2024, pages 12276–12294.

A

NER Atomic Output Space

We first show that the aggregate output space of the NER decomposition in Figure 3 is smaller than the output space complexity of the complete task:   𝑑+1 𝑆 𝐶 (𝑇 + 1) ≥ (𝑇 + 1) + 𝑀 · 2

37

Taking sides, we will show that log (𝑇 + 1) 𝑆  the logarithm of
both  𝑑+1 𝐶 log (𝑇 + 1) + 𝑀 · 2 .
   𝑀 · 𝑑+1 𝑑+1  2
𝐶 log (𝑇 + 1) + 𝑀 · = log (𝑇 + 1) 1 + 2 (𝑇 + 1) 𝐶 | {z } 

≥



𝐶

(4)

decompose log product

  𝑀 · 𝑑+1 2 = log((𝑇 + 1) 𝐶 ) + log 1 + (𝑇 + 1) 𝐶
  𝑀 · 𝑑+1 2 = 𝐶 log(𝑇 + 1) + log 1 + (𝑇 + 1) 𝐶 | {z }

(5) (6)

log(1+𝑥 ) ≤ 𝑥

𝑀 · 𝑑+1 2

≤ 𝐶 log(𝑇 + 1) +

(𝑇 + 1) 𝐶 | {z }

(7)

𝑀 · 𝑑+1 ≤ (𝑇 + 1) 𝐶 2

≤ 𝐶 log(𝑇 + 1) +

1 |{z}

(8)

log(𝑇+1) ≥1

= 𝐶 log(𝑇 + 1) + log(𝑇 + 1)

(9)

= (𝐶 + 1) log(𝑇 + 1) | {z }

(10)

𝑇+1≤𝑆

≤ 𝑆 log(𝑇 + 1) = log(𝑇 + 1)

(11)

𝑆

(12) 𝑑+1

In line 7, we assume that the space of 𝑀 spans (𝑀 · 2 ) is smaller than that of all possible typed spans in the document ((𝑇 + 1) 𝐶 ). We assume on line 10 that the set of possible spans for a document is much larger that the number of types (𝑇 + 1 ≤ 𝑆).

B

Coreference Resolution Atomic Output Space

We show that the aggregate output space of the coreference decomposition is smaller than the output space of the complete task. We assume both that the number of mentions is larger than the number of entities (which is necessarily the case if all the coreference clusters are not singleton) and that there is more than one coreference cluster for a given document, giving us 𝑀 >> 𝑘 ≥ 2. We will prove the following claim.   𝑀 𝑀 (𝑀 + 1) (1 + 𝑇) 𝑆 ≥ + (1 + 𝑇) 𝑆 𝑘 2

38

First, we can observe that Stirling numbers of the second kind are lower bounded (Rennie and Dobson, 1969), so it is enough to show   𝑀 (𝑀 + 1) 𝑀 1 + (1 + 𝑇) 𝑆 (13) (1 + 𝑇) 𝑆 ≥ (1 + 𝑇) 𝑆 · (𝑘 2 + 𝑘 + 2)𝑘 𝑀 −𝑘−1 ≥ 2 2 𝑘 Second, we show a supporting claim that 𝑘 𝑀 −𝑘−1 ≥ 𝑀 (𝑀 + 1) for 𝑘 ≥ 2, 𝑀 ≥ 10. 𝑘 𝑀 −𝑘−1 is exponential in 𝑀 as the exponent 𝑀 − 𝑘 − 1 grows linearly with 𝑀. 𝑀 (𝑀 + 1) is quadratic in 𝑀. Exponential growth dominates polynomial growth, giving us the inequality. We show this more formally. 𝑘 𝑀 −𝑘−1 ≥ 2 𝑀 −𝑘−1 | {z } 𝑘 ≥2

≥ 2 𝑀 −3 =

2𝑀 8 |{z} 𝑀 ≥10

≥ 𝑀 (𝑀 + 1) Finally, starting from the inequality we can then show the claim Equation 13: 𝑘 𝑀 −𝑘−1 ≥ 𝑀 (𝑀 + 1) | {z } 2

𝑀 −𝑘−1

(14)

𝑘 ≥2 𝑀 −𝑘−1

(𝑘 + 𝑘 + 2)𝑘 ≥𝑘 ≥ 𝑀 (𝑀 + 1) 1 1 2 𝑀 −𝑘−1 (𝑘 + 𝑘 + 2)𝑘 ≥ 𝑀 (𝑀 + 1) 2 2 1 2 𝑀 −𝑘−1 𝑆 1 ≥ (1 + 𝑇) 𝑆 · 𝑀 (𝑀 + 1) (1 + 𝑇) · (𝑘 + 𝑘 + 2)𝑘 2 2 | {z }

(15) (16) (17)

adding and subtracting 1

𝑀 (𝑀 + 1) 𝑀 (𝑀 + 1) = + ((1 + 𝑇) 𝑆 − 1) (18) 2 2 | {z } 𝑀 ( 𝑀+1) ≥1 2

𝑀 (𝑀 + 1) + ((1 + 𝑇) 𝑆 − 1) 2 𝑀 (𝑀 + 1) ≥ + (1 + 𝑇) 𝑆 2

≥

(19) (20) (21)

39

C

Excessively Decomposed NER Atomic Output Space

To show that the output space of the decomposition is smaller than that of the complete task, it is enough to show the following, in combination with §A.     𝑑+1 𝑑+1 (𝑇 + 1) 𝐶 + 𝑀 · ≥ 2𝐶 + 𝑀 · + 2𝑀𝑇 2 2
Subtracting 𝑀
· 𝑑+1 and taking
the logarithm of both sides, we will show that 2 log (𝑇 + 1) 𝐶 + ≥ log 2𝐶 + 2𝑀𝑇 .      2𝑀𝑇 (22) log 2𝐶 + 2𝑀𝑇 = log 2𝐶 1 + 𝐶 2     2𝑀𝑇 = log 2𝐶 + log 1 + 𝐶 (23) 2 | {z } log(1+𝑥 ) ≤ 𝑥

2𝑀𝑇 ≤ 𝐶 log 2 + 2𝐶 |{z}

(24)

2𝐶 >2𝑀𝑇

≤ 𝐶 log 2 +

1 |{z}

(25)

  3 2

(26)

1 𝐶 ≥6> log(3/2)

≤ 𝐶 log 2 + 𝐶 log = log(3) | {z }

(27)

≤ 𝐶 log(𝑇 + 1)

(28)

𝑇 ≥2

In line 24, we assume that the space of all possible position assignments is larger than the space of mention type assignments, since the number of position assignments grows exponentially with document length. We also assume at least 6 characters (𝐶 ≥ 6) in the document and 2 types (𝑇 ≥ 2).

40