Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline Ronivaldo Ferreira

Federal University of Pará Belém, PA, Brazil ronivaldo.junior@icen.ufpa.br

arXiv:2611.07951v1 [cs.SE] 5 Nov 2026

Igor Steinmacher

Northern Arizona University Flagstaff, AZ, USA igor.steinmacher@nau.edu

Vinicius Costa Girard

Cleidson de Souza

Federal University of Pará Belém, PA, Brazil vinicius.girard@icen.ufpa.br

Federal University of Pará Belém, PA, Brazil cdesouza@ufpa.br

Carla Aguiar

Filipe Roseiro Cogo

University of Brasília (UnB) Brasília, DF, Brazil caguiar@unb.br

Queen’s University Kingston, ON, Canada filipe.cogo@queensu.ca

Gustavo Pinto

Federal University of Pará Belém, PA, Brazil gpinto@ufpa.br

Abstract Governments increasingly use machine learning (ML) to handle data volumes beyond their analytical capacity, yet little is known about how requirements arise and evolve as an ML experiment moves toward operational use. Requirements Engineering (RE) for ML often assumes that requirements can be elicited upfront; in the public sector, however, demands arrive vague, stakeholders speak different professional languages, and what counts as a “good” result is unknown in advance. We report a 15-month case study of a thematic classification pipeline for Brazil’s national digital participation platform, which holds over 60,000 citizen proposals, triangulating 43 project artifacts with interviews of six project participants. Institutional demands arrived as broad goals, endorsed but not sponsored, and only 40% of the 95 reconstructed requirements and criteria were ever formalized. The pipeline’s key requirements emerged largely from experiments, model outputs, and integration work, shaped by technical decisions and institutional constraints; the central one, the classification categories, was settled by a government taxonomy chosen for its institutional legitimacy. Acceptance relied on technical metrics, domain judgment, and political assessment, without shared thresholds or validation of usefulness in practice. In public-sector ML, technical and institutional artifacts thus act as elicitation instruments, and experimentation and integration belong to the requirements process.

Keywords Requirements Engineering, ML-Enabled Systems, Public Sector, Text Classification, Case Study

1

Introduction

Machine learning (ML) is making its way into the public sector. Governments use it to triage citizen requests, detect fraud, and analyze text volumes that would take months to read manually [1]. Digital participation platforms are a prominent case: by opening consultations to any citizen, they collect thousands of free-text contributions that agencies must read, organize, and answer. Brazil’s

national platform has received over 60,000 proposals across 117 participatory processes and two assemblies since 2023, on topics from national planning to climate and cultural policy. At this scale, manual analysis exceeds the agencies’ capacity, making ML a precondition for taking citizen input seriously. Building such solutions is not only a modeling problem but a requirements problem. A request such as “classify the proposals by theme” leaves open what counts as a theme, whose vocabulary the themes should follow, how good the classification must be before managers rely on it, and who judges it. In conventional software, Requirements Engineering (RE) answers these questions by eliciting and specifying stakeholder needs before or alongside implementation. ML-enabled systems challenge this practice: their behavior is learned from data, their quality is expressed in metrics whose thresholds are rarely known in advance, and their requirements depend on data that may not yet exist or be labeled [2, 3]. These differences reshape software engineering practice [4] and create friction between data scientists and their teams [5]. The RE for ML community has characterized how data scientists perceive requirements [2], mapped the challenges of nonfunctional requirements [3], studied RE in safety-critical perception systems [6], and identified gaps in human-centered AI [7]; secondary studies have mapped the field [8]. This literature has two blind spots. First, most studies capture RE as practitioners report it at one point in time [2, 9], rarely following a system long enough to see its requirements emerge, shift, and become operational. Second, the public sector is largely absent. There, demands arrive through administrative instruments, stakeholders come from different professions, and success is defined by institutional accountability. Consider the thematic classifier we studied. The team first clustered proposals with topic modeling, and no requirement named a vocabulary. When the team asked for one, the Controlled Vocabulary of Electronic Government (VCGE), a taxonomy used across the federal administration, was suggested, and the team began seeding the model with its terms. What started as a point of comparison became the criterion for judging topics and helped motivate

Ferreira et al.

2

Study Context

Brazil’s national digital participation platform runs on Decidim, an open-source platform for participatory democracy [10]. Federal agencies use it for participatory processes on public policies such as the Multi-Year Plan (PPA), the Climate Plan, and the National Culture Plan. In each process, the agency opens a consultation, citizens and organizations submit and vote on proposals, managers analyze them, and the agency responds formally to society. By our count of the database, over 60,000 proposal records had accumulated across 117 processes and two assemblies by March 2026, before cleaning. For comparison, a single planning process received 8,254 1 Replication package: doi.org/10.6084/m9.figshare.34867212.

Government side

Requesting body

Work plan and annual backlog

anonymized replica

Product team

AI team

Data team

Delivery team

Moderation team

State-owned IT company

deployment

Ministries

run processes

proposes / endorses

supervised classification, so that categories matched the nomenclature managers use to report on public policy. No requirements document recorded this shift; it surfaced in scripts, presentations, and meeting notes. Such episodes suggest that, in public-sector ML, requirements may be discovered through experimentation and institutional artifacts as much as elicited from stakeholders. We report a qualitative case study of this pipeline over 15 months (January 2025 to March 2026). Rather than assume that requirements existed upfront, we ask how they came into being and what role they played: RQ1: How did institutional demands become requirements for the ML pipeline? We examine where requirements originated, who formulated them, and whether they were elicited upfront or surfaced over time. RQ2: How did requirements and technical decisions shape each other over time? We ask whether requirements drove the main technical decisions, emerged from them, or both. RQ3: How was the acceptability of the pipeline’s results established? We examine which criteria were used, who applied them, and how technical metrics, domain judgment, and institutional concerns combined. We first analyzed all 43 project artifacts, normalizing 95 requirements and criteria and reconstructing a trajectory of 17 episodes. We then interviewed six participants, using the episodes to prompt recall and challenge our reconstruction, coded the transcripts deductively and inductively, and consolidated the episodes. The resulting seven patterns, triangulated with the episodes, answer our RQs.1 Our main findings are: • Requirements existed but were rarely written down: only 40% of the 95 requirements and criteria we reconstructed were ever formalized. • Institutional demands arrived as broad goals. AI was the team’s offer, endorsed but never sponsored, and the team itself wrote down its requirements. • Most requirements emerged from experiments, model outputs, and integration after modeling had started. The central one, the classification categories, had no owner and was settled by a government taxonomy chosen for legitimacy and distant from citizens’ language. • Acceptance relied on technical metrics, domain judgment, and political assessment, with no shared threshold and no evaluation in use.

Figure 1: Actors and demand paths. Solid: formal path from the requesting body through the work plan to the product team. Dashed: AI fronts proposed by the delivery team and endorsed outside the work plan. Dotted: the AI team’s data and deployment dependencies.

proposals [11], and a semantic clustering study analyzed 10,022 proposals from three processes [12].

2.1

Actors and ML Initiatives

On the government side, a national body for social participation requests the work and owns the platform, ministries run the processes and receive the contributions, and a state-owned IT company hosts the infrastructure and data. On the delivery side, a team funded through a public-sector cooperation agreement (a legal instrument by which a government body funds another public entity, such as a university, to carry out a work plan) develops and maintains the platform. It has a product team, the main interface with the requesting body; a data team, which extracts data from an anonymized replica of the production database; an AI team of data scientists, which builds models on these data; and a moderation team of political scientists and lawyers, which moderates content and analyzes proposals thematically. AI work was not in the work plan: the delivery team proposed the AI fronts, which were endorsed outside the agreed deliverables (Section 4.1.1). Fig. 1 shows these actors and how demands and dependencies reached the AI team, the team we studied. The AI team was created to explore how ML could support the participatory cycle, starting from proposals written in citizens’ own words, often far from the administration’s vocabulary. Its initial goals were to flag content that violates the terms of use, classify proposals by theme so managers could systematize them and produce reports and formal responses, and recommend proposals and processes to citizens. We focus on the thematic classifier, which served the last two goals and thus two potential consumers: managers and citizens. Over the study period, the team moved from topic modeling with BERTopic, to seeding it with VCGE terms, to supervised classification on an adaptation of this vocabulary, and finally to a service with inference, training, and model-publishing routines whose deployment depended on the state-owned company’s infrastructure. We draw on other fronts, such as a retrieval-augmented generation (RAG) system for reports and a Climate Plan prototype built on commercial APIs, when they help explain the classifier.

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

2.2

Researcher Positionality

This is an insider study. The first author has been an ML engineer in the AI team since January 2025, working on data preparation, model training, and validation. This position gave us access to participants and artifacts; some presentations and meeting notes came from the first author’s files and were included without selection. It also means that the interviewer was a team member and that the first author analyzed decisions they took part in (Section 5.1). To mitigate this, we kept reflective memos, let participants confirm, correct, or contest the reconstructed episodes, consolidated them under a protocol fixed in advance (Section 3.6), and had the second author review all coding and the candidate patterns (Section 3.5).

3

Methodology

We conducted an exploratory single-case study [13, 14]. The case is the thematic classification pipeline (Section 2); the unit of analysis is the trajectory of its requirements and evaluation criteria from January 2025 to March 2026. The case is revelatory [14]: the first author’s insider position (Section 2.2) gave us access to all project artifacts and to participants on both the delivery and government sides, which is rare for public-sector ML systems. The design is longitudinal in its unit of analysis and retrospective in data collection, with two stages (Fig. 2): we reconstructed a preliminary trajectory from the artifacts, then used it to support interviews and consolidated it with their evidence. Artifacts show what was recorded and when, since they were produced to do the work and dated before outcomes were known; interviews show the motivations, negotiations, and tacit expectations that the artifacts left out [13].

3.1

Operational Definition of Requirement

Lacking a specification, we needed a definition broad enough for requirements never written as such. A requirement is any condition the solution had to satisfy to be accepted, integrated, or used, documented or not. Formalized requirements were recorded as such in some project instrument; tacit ones guided decisions without being recorded. Across both, we mark environmental constraints: conditions imposed by hardware, contracts, or data governance that the team treated as requirements. Following RE for ML terminology [2, 6], we separate functional (FR) and non-functional requirements (NFR) from three kinds of criteria: acceptance criteria (AC), stating when an output or deliverable was accepted; evaluation criteria (EC), stating how the solution was assessed; and success criteria (SC), stating what made the initiative worthwhile.

3.2

Artifact Collection

In June and July 2026, we collected every artifact produced for the pipeline during the study period, without filtering: 10 issues and 15 commits from the project’s GitLab repository, 6 Google Colab notebooks, and 5 presentations and 7 meeting notes from the authors’ files, 43 in total. We cataloged each with an identifier, type, date, description, and phase.

Table 1: Interview participants from the delivery and government sides. Min.: interview length in minutes. ID E01 E02 E03 E04 E05 E06

3.3

Background Political sci. Electrical eng. Political sci. Software eng. Software eng. Data eng.

Role in the project Product and domain ML leadership Director, req. body Product eng., AI team Delivery coordination Data engineering

Side Delivery Delivery Government Delivery Delivery Delivery

Min. 48 73 66 92 69 64

Artifact Analysis and Preliminary Trajectory

We analyzed each artifact with guiding questions for its type: issues and scripts for demands, requirements, and criteria; commits, presentations, and meeting notes for the context, motivations, and results of technical decisions. We cross-referenced artifacts so that one could complete or corroborate another (e.g., an issue announces a task, a script implements it, and a presentation reports its results). The first author conducted this analysis alone. 3.3.1 Normalizing requirements and criteria. The same need often appeared in different words across artifacts and phases. We merged equivalent formulations into canonical items, each with an identifier, a definition, and links to its original formulations and artifacts; for example, “saving”, “publishing”, and “loading” the model became the FR Manage the model lifecycle and availability. We created a new item only when a formulation fit no existing one, yielding 21 FRs, 12 NFRs, 23 ACs, 22 ECs, and 17 SCs. 3.3.2 Preliminary episodes. A preliminary episode (EP) is a significant delivery, turning point, or change of approach or tooling where normalized requirements or criteria first appeared, reappeared, were refined, or were operationalized. For each EP, we recorded the demand, requirements, criteria, technical decisions, results, and supporting artifacts, so the same requirement can be followed across episodes. This produced 17 EPs in six phases: (1) exploration and feasibility; (2) unsupervised modeling and categorization; (3) validation, alignment, and scale; (4) transition to supervised classification; (5) operationalization and deployment; and (6) pipeline governance. For example, EP07, VCGE becomes the institutional reference, links an issue requesting high-level categories, a script seeding BERTopic with VCGE terms, and two presentations of results to five requirements and seven criteria.

3.4

Semi-Structured Interviews

Participants. Through purposive sampling [13], we interviewed the six stakeholders most involved with the pipeline, covering the origin of demands, their translation into requirements, technical decisions, and evaluation (Table 1): two political scientists, one of whom directed the requesting body; two software engineers; a data engineer; and an electrical engineer in ML leadership. For a single, bounded pipeline, we favored depth of involvement over the number of participants. Protocol. The first author conducted all interviews remotely, in Brazilian Portuguese, between August 21 and September 18, 2026 (mean: 68 minutes). Three pilot interviews, excluded from the analysis, refined the guide, which covered five themes: (1) origin of the

Ferreira et al.

Figure 2: Research method: (1) artifact analysis and construction of 17 preliminary episodes; (2) six semi-structured interviews covering five themes, each supported by preliminary episodes; (3) deductive and inductive coding; and (4) consolidation of the episodes and synthesis of patterns. demand and the participant’s role; (2) from demands to requirements; (3) from requirements to technical decisions; (4) acceptance, evaluation, and success; and (5) evolution, negotiation, and priorities. After the open questions, we showed three to five EPs chosen by the participant’s role and asked them to confirm, complete, or correct each one (episode-supported elicitation). We transcribed the interviews with WhisperX (Whisper large-v3), and the first author wrote a reflective memo after each.

Table 2: Deductive codebook: 23 codes in six dimensions, with the works each dimension draws on.

3.5

D3. Data scientists’ challenges [2, 6, 7] 3.1 Communication gap · 3.2 Paradigm shift · 3.3 Training vs. operational performance

Interview Analysis

We coded the interviews in two cycles [15]: deductive and inductive coding, then pattern coding that combined both. Interviews were always coded in the same order, starting with those richest in project context. 3.5.1 Deductive codebook. We derived 23 codes in six dimensions (Table 2) from four RE for ML works chosen for their complementary coverage: data scientists’ practices [2], safety-critical systems [6], human-centered AI [7], and NFRs [3]. Each code has a definition, application criteria, and supporting quotes from its source. 3.5.2

First coding cycle.

Deductive coding. The first author assigned each excerpt one primary code (the dominant interpretation), optional secondary codes, a confidence level, and an analytic memo. Of the 23 codes, 18 were applied as primary codes in 177 entries, three only as secondary codes, and two (discrimination/bias and feedback/control) never. Inductive coding. The first author also coded inductively, without a starting codebook. After the fourth interview, we consolidated the codes, refined their definitions, anchored each in a representative quote, and applied later codes retroactively to earlier interviews. A final LLM-assisted check added six excerpts to existing codes (Section 3.7). This yielded 20 inductive codes with 78 applications across 77 excerpts. By the number of interviews in which each appeared, nine codes were consolidated (three or more), seven moderate (two),

Dimension and codes

Sources

D1. Requirement types 1.1 Quantitative targets · 1.2 Explainability · 1.3 Discrimination/bias · 1.4 Legal/regulatory · 1.5 Data/annotations

[2, 3, 6, 7]

D2. RE process 2.1 Elicitation · 2.2 Analysis · 2.3 Specification · 2.4 Continuous V&V

[2, 6]

D4. RE topics for ML 4.1 Scenarios · 4.2 Requirements breakdown · 4.3 Traceability

[6]

D5. Human-centered AI [7] 5.1 User needs · 5.2 Model needs · 5.3 Feedback/control · 5.4 Graceful failure · 5.5 Tool gap · 5.6 Ethics D6. NFRs for ML 6.1 NFR trade-offs · 6.2 NFR redefinition

[3]

and four provisional (one). Two codes appeared only in the last two interviews (one in each), so we do not claim saturation. Second-author review. The second author reviewed the annotated transcripts of all six interviews, assessing the deductive and inductive code assignments and searching for missed applications and new codes. The review raised 41 items (10 additions, nine removals, and 22 modifications), discussed in two reconciliation meetings, after the third and the sixth interviews. By consensus, 19 items changed the final coding, sometimes differently from the original proposal. 3.5.3 Second coding cycle. Using pattern coding [15], the first author grouped inductive codes that jointly explained a phenomenon into nine overlapping groups, checking each relation against the excerpts themselves. We crossed each group with the primary deductive codes on the same excerpts, classifying each relation as complements, adds a dimension, contrasts, or no sustained relation;

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

all 18 primary codes related to at least one group. Merging groups with the same interpretation produced six candidate patterns, each a claim traceable to its excerpts. In a joint session (Google Meet and a Miro board), the second author reviewed the nine groups and then the six patterns, discussing their interpretations, overlaps, and boundaries, and the authors consolidated them by consensus. While writing the manuscript, the first author identified a seventh pattern, on requirements emerging in response to model outputs, from previously coded excerpts. Each of the seven patterns (A–G) documents its supporting participants and divergent evidence.

Table 3: Requirements and criteria reconstructed from the artifacts.

3.6

Form.: stated explicitly in an issue or meeting note; Tacit: all others. Env.: environmental constraints, marked across formalized and tacit items (Section 3.1). One NFR (RNF12) and one SC (CS05) have no consolidated episode.

Episode Consolidation

We consolidated the 17 EPs with a three-step protocol fixed on October 27, 2026. Each step used a separate LLM session, and the first author reviewed all outputs. First, the LLM located 94 discussion segments and extracted 106 candidate utterances with their preceding questions and suggested effects (confirms, complements, corrects, contests, or does not remember). Only participants’ statements counted as evidence; agreeing with a leading question did not. Twenty-one utterances informed only the gap analysis, and the first author revised 13 effects and one coverage classification. Second, we compared this evidence with five documentary analyses and the normalized items. The LLM proposed mergers, reformulations, and 59 candidate gaps, of which the first author kept 49 as complements or associations. Mergers required evidence of a shared transformation, not just thematic similarity. Third, the LLM drafted consolidated episodes and proposed confirmation levels: high (direct, specific support without unresolved contradiction), medium (partial or qualified support), low (weak, indirect, or contradictory support), and not assessable (insufficient interview coverage). Documentary evidence could not raise these levels. The first author revised 10 of 13 drafts and raised C5, C8, and C10 from medium to high after restricting their claims to direct, uncontradicted evidence, although C10 relies mainly on one participant. The result is 12 sequential episodes (C1–C12) in five phases plus one cross-cutting episode (T1). Three mergers combined EP03+EP05, EP06+EP07, and EP12+EP14+EP15, and the first two phases became Phase I; all 17 EPs remain traceable.

3.7

Use of Large Language Models

We used Claude Opus 5.5 (Anthropic), with maximum reasoning effort, for coding review (October 28, 2026) and episode consolidation (October 29, 2026). After inductive coding, it suggested six additional excerpts for existing codes, all accepted after review. During consolidation, it located segments, extracted utterances, suggested effects, cross-referenced documents, identified gaps, proposed mergers, and drafted episodes and confirmation levels. It also helped produce the trajectory figure (Fig. 3) and the consolidation memo from the final spreadsheet. The first author reviewed all outputs and made all final decisions. A script checked quotations against the transcripts. Names were removed before the coding review, and all reported quotations are anonymized.

3.8

Ethical Considerations and Data Availability

All participants consented to recording and to the use of anonymized quotations. Names were removed before the LLM-assisted review,

FR NFR AC EC SC Total

𝑛 21 12 23 22 17 95

Form. 13 6 13 6 0 38

Tacit 8 6 10 16 17 57

Env. 1 2 2 1 1 7

Phase of first appearance I II III IV V 7 4 2 7 1 5 3 1 2 0 9 3 2 8 1 9 4 3 6 0 6 2 1 6 1 36 16 9 29 3

although context may still permit identification. Project artifacts and full transcripts cannot be released for confidentiality reasons. The replication package includes the interview guide, codebooks, normalized items, preliminary and consolidated episodes, the dated protocol, LLM prompts, and shareable analysis records.

4

Results

The pipeline’s requirements existed but were never written down: “We never formalized this at any point” (E02). Artifacts recorded features, criteria, and constraints, but no integrated specification, so we reconstructed the requirements from the documented demands, decisions, and results (Section 3.1). Fig. 3 shows the consolidated trajectory: 12 sequential episodes (C1–C12) in five phases, from unsupervised exploration through institutional references and supervised classification to integration and governance, plus a cross-cutting episode (T1) on the recurring need to validate LLM-produced labels, whose systematic execution we could not confirm. Interview support was high for C5, C8, and C10; medium for C2, C11, and C12; low for C1, C3, C6, and T1; and not assessable for C4, C7, and C9. These levels measure neither documentary evidence nor implementation success. We report the seven patterns (A–G) by RQ (Table 4); they describe related aspects of one process, not a causal sequence. Patterns A and B rest mainly on inductive codes, whereas G matches the deductive codes from the RE for ML literature most closely. We flag evidence from other AI fronts or from a single interview, and mark with † excerpts in which the participant addresses the first author (Section 2.2). The replication package traces patterns to codes and evidence. Table 3 profiles the 95 requirements and criteria reconstructed from the artifacts. Only 38 (40%) were formalized in an issue or meeting note; the rest were tacit. Two thirds first appear in Phases I and IV, when issues framed the work; Phases II and III left few records, and the requirements tied to the institutional reference were never formalized (Pattern C). Six of the seven environmental constraints come from one Phase IV issue on the production database; hardware and contracts appear only in passing (Pattern F). No success criterion was formalized (Pattern G), and the interviews add 14 requirements, four of them environmental, and 7 criteria beyond the 95.

Ferreira et al.

PHASE I

Onboarding and unsupervised exploration

PHASE II

PHASE III

Institutional reference and semi-supervision

PHASE IV

Supervised classification

PHASE V

Deployment and integration

Pipeline governance

C1

C5

C8

C10

C12

Context, data and infrastructure

Human reference and institutional VCGE

Shift to supervised classification

Training, inference, documentation and handoff

Orchestration and pipeline traceability

PE12 + PE14 + PE15

PE17

PE01

LOW

PE06 + PE07

HIGH

PE10

HIGH

HIGH

C2

C6

C9

C11

Topic extraction for two intended uses

LLM-assisted labels and topic naming

Class imbalance and data strategies

Historical proposals backfill

PE02

PE08

PE11

PE13

MEDIUM

LOW

C3

C7

Semantic mapping and first integrated pipeline

Comparative evaluation protocol

PE03 + PE05

PE09

LOW

NOT ASSESSABLE

MEDIUM

MEDIUM

NOT ASSESSABLE

C4 User groups and profiles (continuity uncertain) PE04

NOT ASSESSABLE

T1

Human validation of LLM labels as a condition for trust • cross-cutting concern, systematic execution not confirmed

INTERVIEW SUPPORT

High

Medium

Low

Not assessable

PE16 | LOW

High / Medium / Low = interview support Not assessable = not discussed in an assessable way PE = preliminary episode; + = merge

Figure 3: Consolidated trajectory of requirements evolution in the ML project, reconstructed from documentary artifacts and interviews. Table 4: Patterns by research question and the consolidated episodes used for triangulation. Pattern RQ1: from demands to requirements A AI entered as the team’s offer, before its use was defined B Endorsement did not become sponsorship C The classification reference had no convergent owner and reoriented the task RQ2: requirements and technical decisions D Values and resources acted as tacit requirements, met unevenly E Classification requirements surfaced in reaction to concrete outputs F Operational requirements surfaced in development and integration RQ3: acceptability G Acceptance rested on judgment, and evaluation in use was missing

4.1

Episodes C1–C3 C10–C12 C3, C5, C8

C1, C5, C8, C10 C2, C3, C5, C7–C9 C10–C12

C5, C7, C10, T1

RQ1: How Institutional Demands Became Requirements

4.1.1 Pattern A: AI entered as the team’s offer, before its use was defined. The need to analyze citizen contributions at scale was recognized but did not translate into an institutional demand for AI. According to the requesting body’s director, AI “does not enter at the level of political definition” and was absent from the first work

plan; it was a team initiative that the body welcomed (E03). The product team called AI a “self-contained” demand, unlike features frequently validated with the requesting body (E04). Work advanced from opportunities the team identified: facing the broad goal of increasing engagement, E02 proposed to “break it into parts” and explore solutions, and the coordination linked proposal classification to other Decidim deployments (E05). Moderation is an exception: the need was acknowledged, although the team translated it into an ML solution (E04, E05). Endorsement did not settle who would consume each output. E03 reported an external expectation that the platform have AI, “almost a certificate that you are riding the wave,” and said the technical team chose the tools and the body “just gave the OK.” Intended uses included informing managers and recommending proposals to citizens, but the feedback to citizens was never designed, the topics came to be seen as “maybe” useful to managers (E02), and the classification was evaluated before anyone defined how a service would display it (E01). The trajectory shows the same precedence: the onboarding (C1) mapped data, tools, and environment constraints while the model’s goal and integration remained underspecified; topic extraction was assessed with technical metrics before any service or acceptance threshold existed (C2); and extraction and categorization were integrated into a pipeline with no evidence that the categories helped managers (C3). C1 and C3 have low interview support, so this part rests mainly on technical records. For the coordination, classification and RAG-based reporting became clearer products only at a workshop “almost two years later” (E05).

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

The artifacts point the same way. Tracing each FR and NFR to the earliest artifact that states it, all were first recorded by delivery-side technical roles: ML leadership dominated Phase I (8 of 12 entries), the AI team led Phases II and III, and data and infrastructure roles joined only in Phases IV and V, with deployment. None was first recorded by the requesting body, the product team, or moderation, although this partly reflects a corpus without government-side artifacts (Section 5.1). Authorship shows who wrote a requirement down, not who conceived it, but it matches the interviews: the team formulated requirements for a use others had only endorsed. The finding is not an absence of purpose but an inversion between component and service: exploring an ML capability preceded a precise definition of its use and recipients. 4.1.2 Pattern B: Endorsement did not become sponsorship. We distinguish endorsement, approving an initiative, from sponsorship, the continued action of someone who can prioritize deliveries and remove institutional obstacles. AI was endorsed but stayed peripheral. “Machine learning was not a political priority” next to keeping the platform running and migrating it (E04); the product “doesn’t need AI to work” (E02); and even as an official front, AI “never came in as a delivery priority” (E05). For the former director, the main concerns were other features and security, while AI was still “very much running in the lab” (E03). Priority also varied with politics. E04 associated stronger delivery pressure with an administration that emphasized innovation, but E03, part of that administration, said AI was not central to it; the accounts do not support stable sponsorship even then. Later, the platform’s turn to participatory budgeting reduced AI’s space (E02), and the electoral calendar blocked the deployment of a solution the team considered ready (E01), although this readiness was never measured. The infrastructure dependency shows the consequences. The coordination reported a long negotiation to obtain GPUs from the state-owned company that hosts the platform, contrasting it with the WhatsApp integration, which reportedly secured a contract quickly once it became a new administration’s “top priority” (E05). For E05, AI lacked the political interest needed to unlock it; E02 likewise called for someone to bridge AI and the product. This link comes mainly from an interested party and does not show that the obstacles were only political. Still, the accounts converge on a mechanism: ML delivery depended on decisions, resources, and teams over which the technical team had little authority, and approval did not ensure that these dependencies would be resolved. Patterns F and G show how they constrained operation and evaluation in use. 4.1.3 Pattern C: The classification reference had no convergent owner and reoriented the task. Defining the categories was a constitutive requirement: without deciding what each class represented, there was no shared reference for labeling proposals, training the model, or interpreting its outputs. According to E05, the category dictionary “went through several hands” (the requesting body, the team, and moderation) and, when it seemed close to done, was rejected: “that’s not at all what I want, from a business point of view.” The reference requested from the domain side also arrived slowly (E05). The problem was not only insufficient labels but disagreement about what the categories were for and who had the authority to

define them. In C3, the semantic-mapping and integration experiments show that the technical coverage of the associations did not establish whether the categories fit the domain. The VCGE offered an institutional way out. E03 stated that the requesting body suggested it when the team asked for a vocabulary, without considering it good, only “the best parameterized one available at that moment.” E01 also claimed to have suggested it, since a reference valid “for any agency, on any topic” was missing. The interviews do not establish who suggested it first, but they reveal distinct justifications: availability (E03), generality across the platform (E01, E05), legitimacy to explain where the themes came from (E01), and managers’ familiarity with the nomenclature (E04). The team did not create the VCGE but appropriated and adapted it. C5 documents the comparison with human classifications and the later VCGE guidance as distinct functions, joined by the pursuit of alignment with public policy. E04 gathered classifications from several people, harmonized the codes, and added public-policy elements the VCGE lacked. In a different front, hate-speech detection, E02 described relabeling “100%” of a dataset because annotators disagreed; this is not the thematic dataset, but it shows a similar mechanism of technically constructing the reference. The institutional reference then helped reorient the method. E04 linked the move to supervised learning to tying results to a nomenclature managers knew; E02 stressed the difficulty of objectively evaluating unsupervised clusters and the possibility of computing metrics against reference labels. These justifications are complementary, not a single trigger, and C8 documents the transition to supervised classifiers and predictive metrics without an exclusive cause or evidence of multi-label output. E01 added that the VCGE “doesn’t change the way of evaluating” on the domain side; it mainly helped to “unlock the project.” The taxonomy thus provisionally settled legitimacy and output specification without showing that its categories suited the language of those who wrote the proposals. The difficulty of establishing shared classes echoes specification problems in RE for ML [6]; here, the reference also had to confer institutional legitimacy.

4.2

RQ2: How Requirements and Technical Decisions Shaped Each Other

The relationship ran in both directions. Values and resources constrained decisions from the outset (D), while classification criteria surfaced from model outputs (E) and operational requirements from development and integration (F). 4.2.1 Pattern D: Values and resources acted as tacit requirements, met unevenly. The political guideline on technology, stated by E03, was to avoid coupling a “ready-made market solution” to the platform. Grounded in open knowledge, data control, and technological autonomy, it favored an in-house, “small-scale” model trained on project data (E03). This choice bounded the solution and required infrastructure able to run an open stack, tying the pipeline to the dependency in Pattern B. C1 records these environment limitations early, though never as formalized NFRs; Pattern F shows how they became a CPU-inference requirement. The principle was applied unevenly: in another front, a prototype built on commercial APIs set the benchmark for the in-house models (Pattern E).

Ferreira et al.

Other conditions came from the purposes of participation. Outputs had to be intelligible to managers and society: E01 called for “less bureaucratic language” and “more citizen language,” and E03 criticized the VCGE as “the language of the administration, not the language of the citizen.” C5 records the adoption of the institutional reference with the caveat that its fit to citizens’ language remained contested. The classification also had to cut across processes and agencies, since Decidim offered no global categorization (E05), and to represent proposals spanning several policies: “almost every proposal involves several policies” (E03). Yet the team limited the output to two categories for simplicity, although a proposal could belong to “three, four categories” (E04), and C8 shows no multilabel output. These conditions competed: autonomy and generality drove effective choices, citizens’ language stayed in tension with the taxonomy, and transversality was only partly met. No interview reports validation with citizens.

4.2.2 Pattern E: Classification requirements surfaced in reaction to concrete outputs. Participants could not always anticipate what would make the classification useful; some expectations became explicit only when they examined concrete results. The rejected category dictionary (Pattern C) exposed disagreement about what made a category useful. In the unsupervised approach, the team realized that the generated themes could impose “a very large cognitive load” on managers (E04), which strengthened the need for understandable, stable categories and contributed to the move to supervised learning. In C2 and C3, experiments already exposed trade-offs among the coherence, diversity, and number of topics, and between coverage and over-association, with no institutional criterion to resolve them. When domain experts examined a sample, they judged the classification “still very far off” (E04), a verdict that identified inadequacies but set no threshold. C7 and C9, not discussed in the interviews, show the same: C7 compared internal and external metrics with no institutional threshold (Pattern G), and in C9 class reorganizations improved metrics without confirmed usefulness to the domain, while synthetic data augmentation brought no relevant gain. Reactions could also define positive references. In the Climate Plan systematization, reported only by E01, the ministry’s agreement with the themes of a prototype built on commercial APIs turned that output into a target for the in-house models: “our models have to at least handle this.” This suggests specification by example, in tension with the autonomy principle (Pattern D), though it does not confirm the mechanism for the thematic classification. One interview showed both the role of examples and a methodological limit. Facing a table shown by the interviewer, E03 questioned the single label (“But do they only accept one tag?”), examined how “domestic violence” related to “gender violence,” and lacked “clarity on how it would be used.” Since E03 said multi-label output had been discussed before, the reaction shows a concrete output letting an existing expectation be discussed and tested, possibly influenced by the interviewer. Overall, outputs judged inadequate prompted revisions, and outputs judged “plausible” could become benchmarks. The episodes document successive adjustments of criteria and approaches, without formal validation cycles. This pattern concerns classification

criteria; Pattern C covers their reference and Pattern F operational requirements. 4.2.3 Pattern F: Operational requirements surfaced in development and integration. Operational requirements surfaced when the component met data, teams, and environments. According to E06, requests reached development without enough technical analysis, and refinement “happened at the development level” whenever a problem required “going back to the definition.” In data preparation, business rules “unknown to us and even to the product team” had been applied inconsistently and required extra cleaning (E06). The model was trained on CSV files delivered before automated ingestion matured, and E06 stated that, “the way it was done,” the training data was not traceable; later pipelines may differ. In the integration with engineering, discussions focused on inputs, outputs, and formats: “The inner logic of how you did it wasn’t that necessary” (E06† ). In one mismatch, the product expected three categories with their scores but received one category with three scores (E06). In the RAG front, an architecture designed “at the algorithmic level” had to be reworked to run as a service (E05). Dependencies among teams also shaped what became deliverable: to avoid another bottleneck, the ML team began to expose the model as an API and to handle part of its integration (E04). C10 documents hand-off, documentation, training, publishing, and inference as requirements of a service for other teams. The environment added further requirements. Inference had been developed on lab GPUs, but the external production infrastructure had none, so deployment required CPU inference (E06), which the coordination also saw as a way to avoid amending the contract (E05); this is a hardware and governance constraint, with no evidence of degraded accuracy. Classifying proposals at creation time also meant classifying existing ones, which required writing results to an auditable production table for which the team lacked credentials (E06); C11 makes this need explicit, with no evidence that the update took place. Since the feedback service to citizens was never designed, it remained unclear how participants could be contacted under data-protection law and whether one or three suggested categories would serve better (E02). Later, C12 documents the adoption of Airflow and MLflow for orchestration, versioning, and traceability, which still did not reach the production database governed by another organization. Part of the requirements thus became visible only when the solution met real data, interfaces, and resources managed by other actors. Some changes during development, however, corrected defects, inconsistencies, or insufficiently refined decisions and were not emergent requirements.

4.3

RQ3: How Acceptability Was Established

4.3.1 Pattern G: Acceptance rested on judgment, and evaluation in use was missing. We found no shared quantitative threshold for accepting the classification. E01 expected “a high hit rate” but admitted that “we hadn’t defined what that would be,” and E04 observed that presenting “the F1” did not tell stakeholders what the result meant. Domain-side assessment examined examples, asking “does this proposal really belong to this theme? Does it make sense?” (E01); it could involve samples and several people, but never became a measurable readiness criterion (E04). For E02, the difficulty

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

(a) Artifact criteria by phase of first appearance (61 of 62)

(b) Interviewonly criteria

Number of criteria

24 20

20

10

9

7

6 2

0

I II Exploration Reference

III IV V Supervised Deployment Governance

Functional/operational check Technical metric Evaluation procedure

Human/domain judgment Use/value Institutional

Figure 4: Nature of the acceptance, evaluation, and success criteria: (a) artifact criteria by phase of first appearance; (b) criteria found only in interviews. SC05 has no consolidated episode and is omitted.

of objectively evaluating clusters was one more reason to adopt supervised learning, where metrics against reference labels became possible. C7, not discussed in the interviews, shows that a comparative protocol of internal and external metrics did not by itself yield an institutional acceptance criterion. The artifacts show the same imbalance (Fig. 4). Most of their 61 criteria are functional or operational checks and technical metrics; only one records a domain judgment, none an institutional condition, and all criteria first appearing in Phases IV and V are operational checks. Judgment-based criteria surface mainly in the interviews, and none of the 17 success criteria was formalized. What decided acceptance thus went largely unrecorded. Judgment took different forms. The technical team checked coherence and compared results with human classifications; managers assessed whether the categories made sense for the domain; and the political assessment weighed whether the solution was “mature enough” to enter production without compromising the platform’s credibility (E05). These expectations could diverge: E03 wanted to review proposals individually, while E04 described a classification that still needed revision after expert testing. Verification also covered input data, as people familiar with the business rules spotted inconsistencies (E06). T1, with low interview support, records human validation of automatic labels as a recurring concern but shows no systematic review of LLM-produced labels. Evaluation in final use remained limited. In moderation, the team expected to compare predictions with moderators’ decisions, but “we never got feedback from them” (E02); for the classification, E01 stated that the feature’s usefulness was never evaluated. “Production” also meant different things: running in the lab differed from being incorporated into the platform maintained by the stateowned company, with access to its database (E02, E06). Neither C10 nor C12 shows that the supervised classifier was deployed in production. Our claim is therefore limited: neither the interviews nor the episodes show the classification’s usefulness being validated by its end recipients, in part because of the unresolved dependencies

of Pattern B. The difficulty of turning metrics into criteria that customers understand resembles the problem described by Vogelsang and Borg [2]; here, political endorsement added an acceptance condition beyond predictive performance. 4.3.2 What did not appear: bias and citizens. Two absences bound these results. First, we found no substantive discussion of performance differences across groups of citizens or ways of writing proposals. The bias and discrimination code (1.3) was never applied, and ethics (5.6) appeared only as a secondary code. In E04, the mention of “bias” concerned possible bias in the human judgment of outputs, not discrimination by the system. Because the interview guide did not specifically address algorithmic bias or discrimination, this absence does not show that such risks were ignored in the project. Second, citizens were not interviewed and, in the coded excerpts, do not help define or evaluate the categories. Their perspective appears only indirectly, through concerns in understandable language (E01, E03), multiple categories (E03), and feedback to society (E01, E03); E05 also mentioned using the classification to guide managers and recommend activities to citizens. The accounts do not show that the categories or the results were validated by the people who submitted the proposals. This absence is limited to the evidence analyzed and cannot be extended to other project activities.

5

Discussion

Requirements were not missing; they were distributed. RE for ML has documented that data scientists rarely receive explicit requirements [2]. Our case suggests a different reading of that absence: the pipeline’s requirements existed, but they were spread across issues, notebooks, presentations, and people, and only 40% were ever formalized. They took shape less through elicitation than through approval, legitimacy, values, choices of reference, and the capacity to prioritize. Experiments and model outputs acted as elicitation instruments: stakeholders who could not state what a useful category was could reject one when they saw it (Pattern E), and a plausible prototype became a target. This suggests treating experimentation as part of the requirements process, as agile RE does with prototyping and as requirements and architecture co-evolve [16, 17]. In practice, this means recording requirements where they surface, for instance as an explicit “requirement revealed” field in experiment reports, so that the 57 tacit items we had to reconstruct are captured as they appear. It also suggests that artifact mining, as we did here, can recover requirements that no one wrote down. Public-sector ML adds requirements that RE for ML catalogs do not name. The central requirement, the classification categories, had no owner, and the taxonomy that filled the gap was chosen for its availability and institutional legitimacy, and it never reflected citizens’ language (Pattern C). Legitimacy thus behaved as a quality requirement of its own, absent from current NFR agendas for ML [3], and it competed with intelligibility for citizens, the people who write the input but do not consume the output. Endorsement without sponsorship (Pattern B) left this and other trade-offs unresolved and every dependency on third-party infrastructure unowned. Human-centered RE for AI calls for eliciting user needs [7]; in this setting, a prior question is who the users are and who has the authority to decide for them. Public-sector

Ferreira et al.

ML projects could make these roles explicit before modeling starts: an owner for the reference vocabulary, a sponsor for deployment dependencies, and a channel for those who produce the input. Acceptance needs to be engineered. As in industrial settings [2, 5, 18], metrics did not translate into stakeholder judgment, and moving the model into the product exposed requirements that modeling had not anticipated. Our case adds two layers: a political judgment of the risk to the platform’s credibility, and an ambiguous notion of “production” tied to infrastructure and contracts outside the team’s control (Patterns F and G). With no shared threshold and no evaluation in use, the pipeline could be technically ready and institutionally unaccepted at the same time. A lightweight remedy is to agree early on a sample-based review protocol with domain experts, possibly expressed as behavioral tests [19], with an explicit acceptance threshold and a plan for evaluation in use. For researchers, the gap between single-point accounts and what we observed over 15 months argues for more longitudinal studies of ML requirements, and for including the managers and citizens our study could not reach.

5.1

Limitations

Our study has several limitations. First, the requirements, criteria, and episodes were not recorded as such; we reconstructed them, and others might have segmented them differently, although each item links to its original artifacts. The deductive codebook may also have drawn our attention toward what its four sources describe, which inductive coding offsets. Second, showing episodes to participants may have steered their answers, so an utterance confirmed an episode only if it added the participant’s own elements (Section 3.6); only one reaction in Pattern E occurred in an interview. Third, the study is retrospective and prone to recall bias, and seven of the 13 consolidated episodes rest mainly on artifacts, in which the government side is underrepresented; participants may also have softened criticism of the government or colleagues before an interviewer from the team. Fourth, the first author was a team member who conducted the interviews and most of the analysis; the second author reviewed all coding (19 of 41 items led to changes; we report no agreement coefficient, as the review aimed to refine the interpretation) and the six candidate patterns, but not the seventh, and we did no member checking. LLM outputs may not be reproducible, although the first author reviewed every suggestion. Fifth, we did not interview citizens or ministry managers, and the guide did not ask about algorithmic bias, so both absences (Section 4.3.2) may partly reflect our design. Finally, this is a single case: we aim at analytical generalization [14] to public-sector ML projects with similar conditions, namely no product owner, input producers distinct from output consumers, and third-party infrastructure. Quotes were translated from Portuguese, and artifacts and transcripts cannot be released, which limits independent verification.

6

Related Work

RE for ML. Data scientists rarely receive explicit requirements, and customers struggle to state performance targets [2]; practitioners of ML-based perception systems cannot specify requirements upfront and rely on operational design domains instead [6, 20]. A

survey of 188 practitioners found that RE for ML is done mostly by project leaders and data scientists, that notebooks are the main documentation format, and that unclear goals and low customer engagement are common problems [9]. Research agendas cover non-functional [3] and human-centered requirements [7], and mapping studies rank specification, explainability, and the gap between ML engineers and end users as the most prevalent challenges [8, 21]. These studies capture RE at one point in time, through interviews and surveys, or treat it as a specification step before modeling. Our findings agree with theirs, notebooks included [9], but we follow one pipeline for 15 months and reconstruct from dated artifacts when and from what each requirement surfaced: most emerged from experiments and integration after modeling had started, which neither single-point accounts nor upfront specification can observe. SE challenges of ML systems. Industrial studies and reviews catalog challenges across the ML workflow, from data management to deployment and maintenance [4, 18, 22, 23], including hidden technical debt such as glue code and undeclared consumers [24], friction between data scientists and software engineers when models enter products [5], and data cascades caused by undervaluing data work in high-stakes domains [25]. Our operational requirements (Pattern F), from mismatched output formats to untraceable training data, resemble these challenges. These works, however, treat them as engineering problems that follow requirements; we show them as a source of requirements and add institutional causes, such as contracts, third-party infrastructure, and missing sponsorship, that industrial settings rarely face. AI in the public sector. Public administration research has mapped AI applications and challenges for governments [1, 26], called for a prior debate with society on the use of AI [26], and shown that stakeholders frame the challenges of adopting AI differently [27]. These studies take a policy view and treat the AI system as given. We look inside the engineering of one system and show how such divergent framings become concrete requirements: a taxonomy chosen for legitimacy, a political judgment of credibility as an acceptance criterion, and citizens absent from defining the categories.

7

Conclusion

RE for ML often assumes that requirements are elicited upfront and then drive design and evaluation. Following a public-sector ML pipeline for 15 months, we found that its requirements existed but nobody wrote them down. AI entered as the team’s offer, endorsed but never sponsored; the classification categories had no owner and were settled by a government taxonomy; and most requirements surfaced from model outputs and from integration with data, teams, and third-party infrastructure. Acceptance rested on layers of judgment, with no shared threshold or evaluation in use. Experiments, model outputs, and institutional artifacts were how these requirements came to exist.

References [1] B. W. Wirtz, J. C. Weyerer, and C. Geyer, “Artificial intelligence and the public sector—applications and challenges,” International Journal of Public Administration, vol. 42, no. 7, pp. 596–615, 2019. [2] A. Vogelsang and M. Borg, “Requirements engineering for machine learning: Perspectives from data scientists,” in Proc. IEEE 27th Int. Requirements Engineering Conf. Workshops (REW), 2019, pp. 245–251.

Requirements Nobody Wrote Down: Emergence in a Public-Sector ML Pipeline

[3] J. Horkoff, “Non-functional requirements for machine learning: Challenges and new directions,” in Proc. IEEE 27th Int. Requirements Engineering Conf. (RE), 2019, pp. 386–391. [4] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and Zimmermann, “Software engineering for machine learning: A case study,” in Proc. IEEE/ACM 41st Int. Conf. Software engineering in Practice (ICSE-SEIP), 2019, pp. 291–300. [5] N. Nahar, S. Zhou, G. Lewis, and C. Kästner, “Collaboration challenges in building ml-enabled systems: communication, documentation, engineering, and process,” in Proc. 44th Int. Conf. Software Engineering (ICSE), 2022, pp. 413–425. [6] K. M. Habibullah, H.-M. Heyn, G. Gay, J. Horkoff, E. Knauss, M. Borg, A. Knauss, H. Sivencrona, and J. Li, “Requirements engineering for automotive perception systems: An interview study,” in Requirements Engineering: Foundation for Software Quality (REFSQ), ser. LNCS, vol. 13975. Springer, 2023, pp. 189–205. [7] K. Ahmad, M. Abdelrazek, C. Arora, M. Bano, and J. Grundy, “Requirements practices and gaps when enrtificial intelligence systems,” Applied Soft Computing, vol. 143, p. 110421, 2023. [8] H. Villamizar, T. Escovedo, and M. Kalinowski, “Requirements engineering for machine learning: A systematic mapping study,” in Proc. 47th Euromicro Conf. Software Eplications (SEAA), 2021, pp. 29–36. [9] A. P. S. Alves, M. Kalinowski, G. Giray, D. Mendez, N. Lavesson, K. Azevedo, H. Villamizar, T. Escovedo, H. Lopes, S. Biffl, J. Musil, M. Felderer, S. Wagner, T. Baldassarre, and T. Gorschek, “Status quo and problems of requirements engineering for machine learning: Results from an international survey,” in Product-Focused Software Process Improvement (PROFES), ser. LNCS, vol. 14483. Springer, 2023, pp. 159–174. [10] X. E. Barandiaran, A. Calleja-López, A. Monterde, and C. Romero, Decidim, a Technopolitical Network for Participatory Democracy: Philosophy, Practice and Autonomy of a Collective Platform in the Age of Digital Intelligence, ser. SpringerBriefs in Political Science. Springer, 2024. [11] L. Jardim, “O ppa federal 2024-2027: análise da plataforma brasil participativo,” 2024. [12] R. Ferreira, G. da Silva, C. Rocha, and G. Pinto, “Semantic clustering of civic proposals: A case study on brazil’s national participation platform,” arXiv preprint arXiv:2509.21292, 2025. [13] P. Runeson and M. Höst, “Guidelines for conducting and reporting case study research in software engineering,” Empirical Software Engineering, vol. 14, no. 2, pp. 131–164, 2009. [14] R. K. Yin, Case Study Research and Applications: Design and Methods, 6th ed. Thousand Oaks, CA: SAGE Publications, 2018. [15] J. Saldaña, The coding manual for qualitative researchers. sage, 2009. [16] L. Cao and B. Ramesh, “Agile requirements engineering practices: An empirical study,” IEEE Software, vol. 25, no. 1, pp. 60–67, 2008. [17] B. Nuseibeh, “Weaving together requirements and architectures,” Computer, vol. 34, no. 3, pp. 115–119, 2001. [18] A. Paleyes, R.-G. Urma, and N. D. Lawrence, “Challenges in deploying machine learning: A survey of case studies,” ACM Computing Surveys, vol. 55, no. 6, pp. 1–29, 2022. [19] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Proc. 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4902–4912. [20] K. M. Habibullah, H.-M. Heyn, G. Gay, J. Horkoff, E. Knauss, M. Borg, A. Knauss, H. Sivencrona, and P. J. Li, “Requirements and software engineering for automotive perception systems: An interview study,” Requirements Engineering, vol. 29, no. 1, pp. 25–48, 2024. [21] U.-e. Habiba, M. Haug, J. Bogner, and S. Wagner, “How mature is requirements engineering for AI-based systems? A systematic mapping study on practices, challenges, and future research directions,” Requirements Engineering, vol. 29, no. 4, pp. 567–600, 2024. [22] L. E. Lwakatare, A. Raj, J. Bosch, H. H. Olsson, and I. Crnkovic, “A taxonomy of software engineering challenges for machine learning systems: An empirical investigation,” in Agile Processes in Software Engineering and Extreme Programming (XP), ser. LNBIP, vol. 355. Springer, 2019, pp. 227–243. [23] L. E. Lwakatare, A. Raj, I. Crnkovic, J. Bosch, and H. H. Olsson, “Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions,” Information and Software Technology, vol. 127, p. 106368, 2020. [24] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 28, 2015, pp. 2503–2511. [25] N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, and L. M. Aroyo, ““everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI,” in Proc. CHI Conf. on Human Factors in Computing Systems. ACM, 2021, pp. 1–15. [26] W. G. de Sousa, E. R. P. de Melo, P. H. D. S. Bermejo, R. A. S. Farias, and A. O. Gomes, “How and where is artificial intelligence in the public sector going? A literature review and research agenda,” Government Information Quarterly, vol. 36, no. 4, p. 101392, 2019.

[27] T. Q. Sun and R. Medaglia, “Mapping the challenges of artificial intelligence in the public sector: Evidence from public healthcare,” Government Information Quarterly, vol. 36, no. 2, pp. 368–383, 2019.

Related documents

• BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks#1467153
• Survey of Cross-Cloud ML Pipelines and Multi-Cluster Governance#695924
• Machine Learning Lineage for Trustworthy Machine Learning Systems: Information Framework for MLOps Pipelines.#489498
• LLMOps, AgentOps, and MLOps for Generative AI: A Comprehensive Review#75685
• On the Impact of Requirement Smells in LLM-Based Code Generation#1077137