DRF LOW: A Deep Research Benchmark for Personalized Workflow Prediction Md Tawkat Islam Khondaker♠♢ Raymond Li♠♢ Muhammad Abdul-Mageed♢ Laks V.S. Lakshmanan♢ Issam H. Laradji♠♢ ♠ ServiceNow AI Research, ♢ The University of British Columbia
Abstract
Example of Prior DR Benchmarks Q: As a senior elementary school English teacher, I need assistance writing a detailed research paper on a 'New Paradigm of Holistic Empowerment in Elementary English Education and Teaching.' Please provide comprehensive content, suggest relevant keywords, and ensure the paper reflects practical, frontline teaching experience, structured to be helpful for novice teachers.
arXiv:2606.18191v1 [cs.AI] 16 Jun 2026
Deep research (DR) systems are increasingly used for complex information-seeking tasks, but existing works mainly focus on generating reports and summaries. In contrast, many enterprise tasks instead require an agent to identify concrete workflows which is a sequence of action-steps. For example, rather than summarizing budgeting policies, an agent should be able to determine the steps needed to answer a question such as: “How do I request new headcount given a fixed budget?”. Therefore, we introduce DRF LOW, a benchmark for evaluating personalized workflows predicted by agents from heterogeneous sources. Each task requires the agent to identify relevant evidence from scattered sources, then use that evidence to predict the correct action-step sequence for the user’s task. DRF LOW contains 100 tasks across five domains, with 1, 246 reference workflow steps grounded in more than 3, 900 sources. We define seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization. We further present DRF LOW-Agent (DRFA), a workflow-oriented reference agent to predict personalized workflow. We show that although DRFA improves over strong baseline agents (upto 10.02% average F1 score), there is substantial room for improvement remains across these workflow metrics, indicating that predicting complete and correct personalized workflows remains a challenging frontier for deep research1 .
1
Introduction
Deep research has emerged as an important evaluation setting for language-model based research systems (Mialon et al., 2024; Lù et al., 2024; Yoran et al., 2024; Krishna et al., 2025; Sharma et al., 2026; Asai et al., 2026). Unlike conventional question answering, it requires a system to search across 1
Code: https://github.com/ServiceNow/drflow
Output: Detailed Research Report
Long-form
1. Introduction to Elementary Education and Teaching
Report Generation 2. Advent of New Paradigm in Elementary English Education
Agent
Example of DRFLOW for Personalized Workflows User: Program Coordinator. Q: What is the workflow to onboard a newly hired tenure-track faculty? Context (scattered in heterogeneous artifacts): non-citizen, J-1 entry, Aug 01 start, $300k non-Govt funding... Step 1 · EastportWorks · Create case
Appt. type?
create new case file for the newly hired faculty and populate with the personal information
Generic Workflow DRFLOW Agent
Step 4 · Route PDF-7 · + Provost appointment = tenure-track -> Provost approval, otherwise -> Dept.Head approval
Step 7 · ISSS/DSO · J-1 DS-2019 Non-citizen + J-1 → upload funding proof, English attestation; track issuance
→ Tenure-track Personalized Workflow
Citizen / PR?
→ No (sponsor required) Visa category?
→ J-1 Professor
Figure 1: Comparison of deep research questions between representative prior work (Du et al., 2026) and DRF LOW. DRF LOW expects actionable procedures rather than free-form output. For other distinctions with the prior works see §2.
multiple information sources, identify relevant evidence, reconcile partial findings, and predict a grounded response to an open-ended user request. This shift has pushed evaluation beyond short factual answers toward more realistic tasks that require retrieval, reasoning, and evidence aggregation. However, existing deep research benchmarks (Du et al., 2026; Xu et al., 2026b; Zhong et al., 2026a) still primarily expect free-form outputs, such as reports or citation-grounded long-form answers, rather than actionable procedures. This leaves one of the key capabilities underexplored. In many realistic enterprise settings, users do not primarily need narrative summary, rather a workflow detailing what steps to take, in what order, under which conditions, and based on which evidence (See Figure 1 for an example). Other works (Liu et al., 2025; Zhong et al., 2026b) on workflow generation from natural language specifications examine whether language models can generate executable workflows directly from user instructions. However, these works
Task Context
Task Data
Data Sources
DRFLOW-Agent
Company Documentation NextCloud
DR Question:
FileShare
What is the workflow to obtain informed
consent for a procedure when the patient has limited English proficiency?
3
1
Local Files
Company: Sunrise Surgical Center
Access Enterprise Data
Load Context
An ambulatory surgery center performing high-volume outpatient procedures with multilingual patient populations.
DOCX
DOCX
2
File System
Mattermost
Load Environ.
Chat
Personal Evidence
RoundCube Chat Conversation
Persona: Dr. Miguel Torres - Anesthesiologist
5 Generate Workflow
4
Load Sources
Metrics
DOCX
DOCX
Factuality: checks whether citations support the claim
Predicted Workflow
Recall: measures coverage of golden workflow steps
Ground Truth Workflow
Precision: measures correctness of
Patient identified for procedure
predicted workflow steps
F1: harmonic mean of precision Step 1: Assess language needs
Step 2: Explain procedure via interpreter
and recall
5
Step 1: Assess language needs
Condition Resolution:̀ evaluates
Evaluate
Topology Order: checks ordering and structural consistency
Step 3: Obtain signature and document consent
5
Identify preferred language, arrange interpreter
correct resolution of conditions
Personalization: assesses use of personal evidence
Load Ground Truth
Step 2: Explain procedure via interpreter Risks, benefits, alternatives in patient's language
Step 3: Check presence of In-Person Interpreter Otherwise, search for video interpreter
Figure 2: Overview of the DRF LOW task and evaluation pipeline. Given a task context and heterogeneous task data, the system loads the environment and available sources, generates a predicted personalized workflow, and evaluates it against the ground-truth workflow.
inherently assume that the intended procedure is largely specified in the input. In a realistic and challenging deep research, by contrast, the workflow is rarely given explicitly. It must be predicted from heterogeneous evidence distributed across artifacts such as documents, emails, and chat logs. Moreover, such workflows also need to be personalized because company artifacts may describe the generic process broadly applicable for various scenarios (e.g., if appointment is tenure-track, obtain Provost approval; otherwise, obtain Department Head approval in Figure 1), while personal artifacts determine which branch of the workflow applies to the user’s situation, e.g., which conditions and next actions are appropriate. This distinction matters because many requests, such as policy compliance, eligibility determination, and resolving client issues follow a sequence of personalized steps. This task of personalized workflow prediction is not captured by existing benchmarks (Table 1). To address this gap, we introduce DRF LOW, a benchmark for personalized workflow prediction in deep research settings. Each task combines company-side artifacts, which define a generic process, along with user-side artifacts, which provide the personal evidence required to transform that process to a personalized workflow. Solving a task therefore requires joint reasoning steps: predicting a generic workflow from generic evidence (present in the company artifacts) and grounding that workflow in personal evidence (present in the
user-specific artifacts) to produce a personalized workflow. DRF LOW contains 100 tasks: 50 tasks we call as original and the other 50 we call as mixed. The original tasks contain artifacts that were specifically synthesized for the corresponding deep research questions. On the other hand, while the mixed contains artifacts from multiple deep research questions, making the tasks more challenging. We synthesized these tasks using an end-to-end data generation pipeline with human verification to ensure quality of the tasks. Note that this data generation pipeline is amenable to any domain. To evaluate personalized deep research workflows in this setting, we introduce seven metrics that measure factual grounding, step prediction, workflow topology, condition resolution, and personalization quality. We further introduce DRF LOWAgent (DRFA), a workflow-oriented agent tailored to this problem setup. We provide the overall DRF LOW pipeline in Figure 2. Our contributions are: (1) we introduce DRF LOW, the first benchmark centered on personalized workflow prediction containing 100 tasks, across 5 domains with 1,246 reference steps and more than 3, 900 sources/artifacts; (2) we develop a multi-step, extensible pipeline that produces synthesized yet realistic deep research tasks for personalized workflow generation; (3) we propose seven evaluation metrics that explicitly assess structural and personalization-specific properties of predicted workflows; and (4) we introduce DRFA, a
workflow-oriented reference agent for personalized workflow prediction, and show that while it improves over baselines (upto 10.02% in average F1), substantial room for improvement remains, underscoring the challenges of personalized workflow prediction for the frontier models.
2
Related Work
Deep Research Benchmarks. DR systems automate complex, multi-step information gathering and synthesis, producing citation-grounded outputs in response to open-ended queries (Zheng et al., 2025). Early efforts such as GAIA (Mialon et al., 2024) introduced multi-step, tool-augmented question answering tasks, establishing a foundation for measuring agent reasoning and web browsing capabilities. DeepResearch Bench (Du et al., 2026) offers 100 PhD-level tasks across 22 fields and proposes two evaluation methodologies aligned with human judgment. DeepResearchGym (Coelho et al., 2026) introduces a reproducible sandbox using open-access web corpora to ensure stable evaluation across runs. ResearcherBench (Xu et al., 2026b) narrows its focus to frontier AI scientific inquiry, compiling 65 research questions across 35 AI subjects. Mind2Web 2 (Gou et al., 2026) targets long-horizon agentic search with fine-grained evaluation nodes spanning 130 tasks. More recently, DRACO (Zhong et al., 2026a) sources 100 tasks from real user queries across ten domains, grading outputs along four rubric dimensions, and R ESEARCH RUBRICS (Sharma et al., 2026) contributes 101 expert-curated prompts paired with over 2,500 fine-grained evaluation criteria. However, these benchmarks primarily evaluate reports, answers, or rubric-scored summaries, while DRF LOW instead evaluates whether agents can predict structured, personalized workflows as sequences of action-steps (Table 1). Multi-Application Agent Benchmarks. DRBench (Abaskohi et al., 2026) is the first benchmark to combine public web retrieval with private organizational data sourced from enterprise applications, evaluating agents on insight recall and report quality across 100 tasks. TheAgentCompany (Xu et al., 2026a) provides 175 professional tasks requiring cross-application coordination and simulated colleague communication within a self-hosted environment. OfficeBench (Wang et al., 2024) evaluates multi-application office automation through customized matching and execution-based eval-
uation. WorkArena (Boisvert et al., 2024) tests browser-based knowledge work on the enterprise platform. FlowBench (Xiao et al., 2024) specifically benchmarks workflow-guided planning for LLM agents, assessing the efficacy of workflow knowledge across multiple domains. Although these works capture important aspects of enterprise interaction, but they do not directly evaluate whether an agent can recover the relevant procedure from scattered artifacts, resolve userspecific conditions, and output the resulting personalized workflow; which is the core aspect of DRF LOW(Table 1). Workflow Generation for LLM Agents. Prior work has explored workflow generation as a mechanism for improving LLM-based agents. AutoFlow (Li et al., 2024) proposes a framework for automatically generating natural language workflows through iterative optimization. WorkTeam (Liu et al., 2025) studies multi-agent workflow construction from natural language task specifications, and Chat2Workflow (Zhong et al., 2026b) generates executable visual workflows from user instructions. These works, however assume that the target workflow, and/or the information needed to construct, is already provided in the input specification. In contrast, DRF LOW evaluates whether an agent can predict information from scattered pieces of evidence into a structured actionable workflow that a human user can follow and complete. This distinction positions DRF LOW at the intersection of deep research and workflow understanding, a space that, to our knowledge, no prior benchmark occupies. Further discussion on literature is provided in Appendix A.
3
DRF LOW: A Personalized Deep Research Workflow Prediction Benchmark
We introduce DRF LOW, a benchmark for evaluating agents for personalized workflow prediction from heterogeneous data sources. DRF LOW comprises 100 tasks: 50 original and 50 corresponding mixed variants, spanning five domains: B2B (Business-to-Business), B2C (Business-toConsumer), Education, Healthcare, and Legal. Each task is anchored in a realistic deep research question and requires the agent to reason jointly over two complementary sources of evidence, company-side data: which specifies the company’s generic operating procedure for the
Benchmark
# Tasks
Private Data
Het. Apps
Personalized
Actionable Steps
Structured Output
Extensible Pipeline
466 100 65 130 100 101
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✓ ✗
175 300 33 51 100
✓ ✓ ✓ ✗ ✓
✓ ✓ ✗ ✗ ✓
✓ ✗ ✗ ✗ ✓
✓ ✓ ✓ ✓ ✗
✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✓
— 315 27
✗ ✗ ✗
✗ ✗ ✗
✗ ✗ ✗
✓ ✓ ✓
✓ ✓ ✓
✗ ✗ ✗
100 (1, 246 reference steps)
✓
✓
✓
✓
✓
✓
Deep Research Benchmarks GAIA (Mialon et al., 2024) DeepResearch Bench (Du et al., 2026) ResearcherBench (Xu et al., 2026b) Mind2Web 2 (Gou et al., 2026) DRACO (Zhong et al., 2026a) R ESEARCH RUBRICS (Sharma et al., 2026) Agent Benchmarks with Multi-Application Environments TheAgentCompany (Xu et al., 2026a) OfficeBench (Wang et al., 2024) WorkArena (Boisvert et al., 2024) FlowBench (Xiao et al., 2024) DRBench (Abaskohi et al., 2026) Workflow Generation AutoFlow (Li et al., 2024) WorkTeam (Liu et al., 2025) Chat2Workflow (Zhong et al., 2026b) DRF LOW (Ours)
Table 1: Comparison of DRF LOW with representative benchmarks. Private Data: requires access to private or organizational data beyond the public web. Het. Apps: sources span heterogeneous application types (e.g., chat, email, documents). Personalized: tasks are grounded in user-specific personas and organizational context. Actionable Steps: output consists of concrete, followable or executable steps. Structured Output: output is in structured format, not free-form text. Extensible Pipeline: includes an automated pipeline for scaling up the benchmark construction.
question, including dependencies and different conditions based on the generic process and user-side data: which provides the personal context needed to determine which of those conditions hold in the current case and how the generic process should therefore be instantiated. Task Description. DRF LOW targets a form of deep research that is fundamentally procedural and personalized. The central challenge is not merely to collect relevant evidence, but to predict a generic workflow from company-side data and then transform it into a personalized workflow using user-side data. For both company- and user-side data: we further create two types of evidence units: insights and distractors. Insights are relevant pieces of information required to predict the workflows, whereas distractors comprise realistic, domain-adjacent information that increases noise in the evidencespace, turning the retrieval of the workflow-bearing insights into a needle-in-the-haystack challenge. Hence, to solve a task, the agent must explore the full collection of company- and user-side artifacts and predict a generic workflow that preserves the underlying organizational logic, while correctly resolve user-specific conditions. Data Generation. We construct DRF LOW through a staged synthesis pipeline (Figure 3) that progressively transforms a domain specification into a realistic, personalized deep research environment. Stage S1: Task Seed Generation. The first stage generates a task seed for each benchmark instance. Starting from a target domain and a task i, we jointly synthesize the deep research question Qi ,
structured company metadata, a company persona, and a user persona. Rather than generating the question in isolation, we synthesize it jointly with the company, user, and relationship context that define the scenario. This coupling yields tasks that are grounded simultaneously in a plausible organization, a realistic user role, and a concrete relationship between them. From this seed, we further instantiate two complementary contextual views. The first is the generic company context Cig , which describes the companylevel process environment relevant to Qi . The second is the user context Ciu , which captures the user’s role, circumstances, constraints, and surrounding needs. Cig provides the process-level foundation from which the generic workflow will be derived, whereas Ciu provides the personalized signals that will later determine how that workflow should be specialized. Stage S2: Generic Workflow Composition. Given the company-side context Cig and the question Qi , the second stage composes the generic workflow Wig . This workflow is a structured sequence of company-level actions and decision points required to address the question. Crucially, Wig encodes prerequisite relations and different conditions across steps based on the generic context. In other words, it specifies not only what typically needs to be done, but also where the process can branch and under what circumstances alternative actions may become appropriate. Stage S3: Company Evidence Generation. We derive company supporting insights Eic+ , which encode the substantive content of the generic work-
S1
Task Seed Generation Generate meta-data for the task
Domain Healthcare
?
DR Question What is the step-by-step workflow to obtain informed consent for a procedure when the patient has limited English proficiency (LEP)?
S2
Generic Workflow Composition Generate generic workflow from the comany context 1
Confirm language & capacity
2
Determine communication support
Company Evidence Generation
S3
S4
Generate Company insights from Wi g then generate distractors and company files
t
Tier 2
Mixed Evidence Generation
S6
u
Supporting user insights E i
1
Confirm Spanish + capacity
Patient requests interpreter
2
Mark “Qualified Interpreter Required”
Consent form missing tier rule ID
Generate additonal DR question, corresponding insights and files
Use VRI, not personal representative
VRI available
3
4
Consent in Spanish + teach-back
?
Tier Rule
LEP & Accessibility Tier 1
LEP Addendum / Tier 3
Signature Validity
Additional DR Question What is the step-by-step workflow to obtain informed consent when the patient lacks capacity and a legally authorized representative must consent?
Corresponding insights E iX No sedation yet
3
Select interpreter tier
4
Consent discussion & teach-back
Distractors E i report
3
5
Documentation
Supporting files F i PDF
7
c+
c−
Distractor files F i
Signatures
LEP informed consent workflow
Consent hard stop
Complete interpreter fields + LEP addendum
6
Obtain signatures before sedation
Contact patient's representative if family members absent, contact attorney
ECG leads protocol
Fill out LAR consent form
PACU onboarding workflow
File Generation
Anesthesiologist
6
5
Supply inventory cycle count
User
Confirm patient is lack of capacity u− Distractors E i
OR utilization
Ambulatory surgery center
Generic company context C ig
p
Generate insights from W i and Ci then generate distractors and personal files u+
c+
Supporting insights E i Script Checklist
Interpreter
Sunrise Surgical Center
Dr. Miguel Torres
Personal Evidence Generation
S5
CN-2 Consent
c−
Company
Personalized Workflow Composition
Generate personalized workflow resolving condition with user context C iu
DOCX
Policy Procedure Manual Doc
Report
7
Time-out verification
File Generation
File Generation u+
u-
Supporting files F i
Distractor files F i
Distractor files F iX
DOCX
Ops Memo
Chat Log
DOCX
PDF File
DOCX File
Chat Log
DOCX
PDF File
DOCX File
Chat Log
DOCX
PDF File
DOCX File
User context C iu Spanish-speaking patient, VRI available, no sedation
Sparse evidence embedded in realistic files
Heterogeneous personal evidence
Heterogeneous confounding evidence
Figure 3: Overview of the DRF LOW data generation pipeline. Starting from a task seed (S1), the pipeline first composes a generic company workflow with company and user contexts (S2). It then generates companyside insights and distractors as heterogeneous artifacts (S3). The generic workflow is further specialized into a personalized workflow (S4), after which user-side insights and distractors are generated as personal artifacts (S5). Finally, similar but distinct questions with corresponding artifacts are generated for the mixed split (S6).
flow, including step descriptions and the conditions attached to different parts of the process. In parallel, we generate company distractors Eic− , which are realistic and domain-consistent but do not contribute to solving the task. The resulting company insights and distractors are then materialized as company-related files Fic+ and Fic− , respectively. The file generation module uses a three-stage sparse-evidence injection process: it first creates the target artifact type (e.g., PDF or DOCX), then injects the insights or distractors in a contextually appropriate location within the artifact, and finally fills the remaining sections with realistic but task-irrelevant content. This process yields naturalistic company artifacts in which the evidence needed to reconstruct the workflow is embedded within realistic surrounding content rather than explicitly foregrounded, requiring the agent to infer the generic process from distributed sources. Stage S4: Personalized Workflow Composition. Given the generic workflow Wig and the user context Ciu , we compose the personalized workflow Wip . Here, Ciu supplies the user-specific facts needed to determine which conditions in Wig hold for the target user. This stage resolves the branching structure in the generic workflow and instantiates the correct case-specific sequence of actions, which is subsequently verified by human annotators. The resulting personalized workflow Wip serves as the reference workflow for evaluation. Stage S5: Personal Evidence Generation. Given the personalized workflow Wip and the user context Ciu , we first derive user supporting insights Eiu+ , which encode the personal facts that imply which specific branches are taken and which
actions are appropriate in Wip for the user. We also generate user distractors Eiu− , which remain realistic but do not help resolve the target task. These user-side insights and distractors are then realized as heterogeneous personal artifacts (e.g., email, chat, local document) Fiu+ and Fiu− . We follow the same file generation process described in Stage S3. Stage S6: Mixed Evidence Generation. The five stages above produce a single coherent case per task. To stress-test agents under realistic caselevel disambiguation, we additionally construct a mixed split in which two additional distinct cases in the same workflow domain are injected into the evidence pool. Given the task seed (Qi , Cig , Ciu ) and the existing user insights, we sample two similar but distinct DR questions that share the workflow domain and stakeholder context with Qi but pertain to a different case. For each question, we derive a set of coherent insights EiX that form an internally consistent case profile distinct from Ciu . Finally, the resulting confounding evidence is realized as heterogeneous files FiX across the same modalities used in Stages S3 and S5. We provide the prompts for the data generation process in Appendix K, the complete list of tasks in Appendix O, examples of generic and personalized workflow in Appendix P, and screenshots of generated files in Appendix Q.
4
DRFA: The Workflow Prediction Agent
We introduce DRF LOW-Agent, namely DRFA, a workflow prediction agent for personalized deep research. DRFA (Figure 4) is designed for the core challenge posed by DRF LOW: given a DR ques-
Tools
Environment Search Space
User Persona Enterprise Search
Analyzer
Local Files
Roundcube
Internet Search
Nextcloud
File Browser
Mattermost
Web Search
Tool Execution
Research Planning
Deep Research Question
Action Planning
Decompose the question into generic and personalized evidence collection 1
? DRFLOW-Agent (DRFA)
Company Context
Execute retrieval and analyses available sources
Plan actions, associated tools, and expected outputs
Content Processing 2
Workflow Generation
Adaptive Action Planning
Predict citation-grounded generic and personalized workflows.
Search action gaps and build intermediate workflow to extract unresolved conditions
Normalize retrieved evidence to structured findinds
YES
Reflection. Need more info?
3
4
NO
Figure 4: Overview of DRF LOW-Agent (DRFA). Given the user persona, deep research question, and company information, (1) DRFA first performs research planning. (2) Then it transforms them into corresponding action planning. (3) It uses adaptive action planning to identify gaps and extract unresolved conditions. (4) Finally, it predicts both generic and personalized workflows. Backbone
Agent
Factuality
Topology
Cond. Res.
Pers. Comp.
Recall
Precision
F1
GPT-5.2
Basic DRBA DRFA
59.46 90.01 85.99
47.50 89.63 92.22
2.86 24.85 42.83
12.86 44.99 59.26
24.52 60.43 69.43
9.79 52.11 62.34
12.16 54.17 63.90
Claude-Opus-4.5
Basic DRBA DRFA
50.10 85.36 83.31
70.07 88.47 88.13
1.64 25.99 47.18
10.29 41.89 62.50
27.93 65.19 72.07
10.62 44.76 55.87
13.54 51.09 60.70
Gemini-3.1-pro
Basic DRBA DRFA
38.59 90.24 87.16
38.00 83.12 91.20
0.11 14.90 26.98
6.49 30.38 48.08
13.51 55.94 67.31
4.77 37.07 42.55
5.05 40.99 52.05
DeepSeek-v3.2
Basic DRBA DRFA
43.80 93.57 94.92
55.33 89.43 91.74
1.17 15.02 35.85
9.45 33.51 56.32
19.04 60.48 68.40
6.97 38.89 48.45
8.61 45.37 55.10
Average
Basic DRBA DRFA
47.99 89.80 87.84
52.72 87.66 90.82
1.45 20.19 38.21
9.77 37.69 56.54
21.25 60.51 69.30
8.04 43.21 52.31
9.84 47.91 57.93
Table 2: Main results on the original split across backbone models. Within each backbone block, the best score in each column is bolded.
tion, the agent must predict the company’s generic procedure from company-side evidence, identify the user-specific conditions, and instantiate the final personalized workflow using user-specific evidence. DRFA operates through four components: Research Planning. The first component decomposes the deep research question into a set of investigation areas that specify what evidence should be collected before workflow prediction. In DRFA, this stage is organized around two complementary objectives: Generic Requirement Collection, which targets company-level policy documents, procedures that define the generic process; and Personal Evidence Collection, which targets personal artifacts to resolve user-dependent conditions. Action Planning. The second component translates each investigation area into concrete actions. Each action specifies a retrieval or analysis objective, along with the expected outputs. Action execution proceeds through the appropriate tool-calling until a fixed number of iterations or all actions are executed.
Adaptive Action Planning. At the end of every iteration, DRFA analyzes the newly collected findings and determines whether additional actions are needed. The module serves two functions. First, it performs gap finding, assessing whether the current evidence over-represents either generic workflow or user-specific evidences, and proposing additional actions to cover the missing side. Second, it performs conditional action planning: the agent derives an intermediate generic workflow from the currently available policy and workflow evidence, identifies the unresolved conditional branches, and formulates targeted actions to retrieve the personalized facts to resolve them.
Workflow Generation. The fourth component predicts the final personalized workflow from the collected evidence. DRFA first constructs a complete generic workflow, preserving prerequisite structure and alternative branches. It then uses the collected user-specific evidence to resolve conditional branches and produce the final personalized
Model
Agent
GPT-5.2
DRBA DRFA
Factuality Topology Cond. Res. Pers. Comp. Recall Precision 86.41 94.38
86.52 90.04
24.58 41.36
42.61 55.67
58.13 65.86
37.30 47.96
44.36 53.67
F1
Claude-Opus-4.5 DRBA DRFA
86.78 82.54
93.10 89.68
23.59 46.72
39.56 58.51
63.70 68.35
37.86 47.43
44.52 53.19
Gemini-3.1-pro
DRBA DRFA
79.10 72.36
86.00 85.32
12.92 18.86
33.54 42.32
47.75 49.26
24.24 24.07
30.18 30.42
DeepSeek-v3.2
DRBA DRFA
81.32 82.30
83.13 82.03
16.12 28.03
27.26 51.83
48.55 51.35
30.36 32.32
34.04 38.34
Average
DRBA DRFA
83.40 82.89
87.19 86.77
19.30 33.74
35.74 52.08
54.53 58.70
32.44 37.95
38.27 43.91
Table 3: Results on the mixed split of DRFA. Within each backbone model, the best score in each column is bolded.
recommended workflow. Further details of DRFA are provided in Appendix J.
5
Experimental Settings
Baselines. We compare DRFA against two baseline agents: Basic is a minimal agent without an explicit planning stage. It ingests all available documents and directly predicts a workflow for the deep research question. DRBench Agent (DRBA) (Abaskohi et al., 2026) is a stronger planning-based baseline adapted from DRBench. Using DRBA as a baseline allows us to evaluate whether DRFA improves workflow generation beyond prior enterprise deep research pipelines built around staged planning and adaptive execution. We instantiate the agents with multiple frontier models, including GPT-5.2, Claude-Opus-4.5, Gemini3.1-pro-preview, and DeepSeek-V3.2. We provide further details on implementation in Appendix B and approximate cost estimation in Appendix L. To simulate realistic setting, we run the applications inside a containerized environment (Appendix M). Evaluation Metrics. We report seven metrics: Factuality (Abaskohi et al., 2026) verifies whether a predicted step is supported by its cited evidence. Recall quantifies the proportion of reference workflow steps that are successfully recovered by the predicted workflow, whereas Precision measures the proportion of predicted steps that can be validly aligned to reference steps. F1 is computed as the task-wise harmonic mean of Recall and Precision. Condition Resolution measures whether the predicted step resolves the condition of a step using the appropriate personalized evidence. Topology evaluates pairwise order consistency between aligned predicted and reference steps. Personalized Comprehensiveness measures whether the predicted workflow preserves the personalized and operational details present in
the reference workflow, such as specific names, forms, dates, thresholds, and other user-specific constraints. We provide additional details for the evaluation metrics in Appendix N.
6
Results
Performance on Original Split. Table 2 reports the main results on the original split of 50 tasks across four backbone models. On aggregate, DRFA achieves the best performance on all the metrics, except Factuality, where DRFA is slightly worse due to the instability caused by heterogeneous planning (see (Abaskohi et al., 2026)) and synthesis of multiple documents, making it harder for the judge to verify (See Appendix H.1). DRFA has a stronger average F1 over DRBA (57.93 vs 47.91), Condition Resolution (38.21 vs 20.19), and Personalized Comprehensiveness (56.54 vs 37.69). This indicates that DRFA produces workflows that are more accurate, structurally better coordinated, and more personalized. Finally, we stress that the overall performance also underscores the importance of the DRF LOW benchmark: despite the strength of existing frontier models, absolute performance still leaves considerable room for improvement, indicating that personalized workflow prediction is a challenging problem and that DRF LOW provides a meaningful testbed for measuring further progress. We provide the domain-wise performance in Appendix D and results on mean with deviation in Appendix E. Performance on the Mixed Split. Table 3 reports the results averaged across the four backbones. Both DRFA and DRBA degrade across nearly all metrics, confirming the challenging nature of the split. However, DRFA retains its lead on all the metrics for all the backbones, under this stress test, except for Factuality and Topology. Compared to Table 2, average DRFA’s F1 falls from 57.93 to 43.91, Condition Resolution
Backbone
Variant
Factuality
Topology
Cond. Res.
Pers. Comp.
Recall
Precision
F1
GPT-5.2
DRFA w/o GF w/o CAP w/o Both
89.82 89.44 90.22 88.42
92.82 85.21 85.16 86.64
41.43 45.84 41.34 35.07
59.36 58.51 55.73 57.29
64.82 70.69 71.94 69.23
54.58 48.24 49.31 50.94
58.35 56.53 57.21 56.11
Claude-Opus-4.5
DRFA w/o GF w/o CAP w/o Both
84.57 85.36 85.33 81.67
89.32 89.98 90.57 82.81
47.09 45.34 42.53 39.10
62.95 58.44 61.00 54.99
71.40 70.76 68.85 64.56
48.07 46.90 45.56 44.78
56.35 55.48 52.12 50.33
Gemini-3.1-pro
DRFA w/o GF w/o CAP w/o Both
83.27 85.09 86.44 79.46
89.61 85.82 86.75 54.42
31.45 27.83 21.69 15.20
45.35 42.75 42.50 28.32
68.58 64.40 67.42 46.40
45.31 29.70 33.79 23.67
53.88 39.26 44.37 30.98
DeepSeek-v3.2
DRFA w/o GF w/o CAP w/o Both
84.28 83.28 83.45 76.34
85.74 84.22 87.33 80.14
37.85 28.31 34.13 14.74
46.32 42.37 29.08 22.82
65.32 54.34 63.57 45.78
49.73 47.91 48.41 34.56
55.88 50.14 53.38 37.96
Average
DRFA w/o GF w/o CAP w/o Both
85.48 85.79 86.36 81.47
89.37 86.31 87.45 76.00
39.45 36.83 34.92 26.03
53.50 50.52 47.08 40.86
67.53 65.05 67.94 56.49
49.42 43.19 44.27 38.49
56.12 50.35 51.77 43.85
Table 4: Ablation study on planning modules. Within each backbone model, the best score in each column is bolded.
from 38.21 to 33.74, and Personalized Comprehensiveness from 56.54 to 52.08. The persistence of sizeable gaps between the original and mixed splits on all the metrics underscores the challenging nature of robust personalized workflow prediction. Performance comparison between the original and mixed splits is shown in Appendix I.
7
Ablation Study on DRFA Design
We select a subset of 15 tasks (3 from each domain) as ablation set to keep the cost manageable. We study the contribution of the two planning components in DRFA: gap finding (GF) and conditional action planning (CAP) in Table 4. Removing GF decreases the average F1 from 56.12 to 50.35, while removing CAP decreases it to 51.77. The effect of CAP is especially clear on Condition Resolution, where removing it lowers the average score from 39.45 to 34.92. This behavior is expected since CAP explicitly constructs intermediate workflows to identify unresolved conditions and then generates targeted action plans to resolve them. Finally, with both components removed, F1 drops to 43.85, indicating both are jointly important, supporting the full DRFA design. We provide further ablation studies including recent frontier models’ performance in Appendix C, qualitative analyses in Appendix G, and error analyses in Appendix H.
8
Human Evaluation
We conduct human evaluation to validate our benchmark and the reliability of our LLM-based evaluator. For DR questions selection, three annotators independently reviewed 50 candidate tasks
for realism and applicability to the associated company, persona, and workflow setting. Annotators unanimously agreed on 33 tasks in the initial round, and the remaining tasks were revised and reviewed in a second round before inclusion. Then we conduct human evaluation on 10 tasks across the outputs of DRFA and DRBA from GPT5.2, yielding 250 predicted-step decisions. All three annotators agree on 57.2% decisions, and at least two agree on 97.6% decisions. The resulting Fleiss’ κ is 0.663, implying substantial agreement (Landis and Koch, 1977). We further compare human evaluation with the LLM judge. The Pearson correlation between human-average F1 and LLM-judge F1 is 0.883, indicating strong agreement between expert and automatic evaluation (Please see Appendix F for details).
9
Conclusion
We introduce DRF LOW, a benchmark for personalized workflow prediction for deep research. Unlike prior works that primarily emphasize free-form report generation, DRF LOW targets a procedural setting in which agents must predict a generic workflow from heterogeneous company artifacts and then ground it in user-specific evidence to produce a personalized workflow. We further present DRF LOW-Agent (DRFA) as a workflow-oriented baseline and evaluate it across multiple frontier backbone models. The results show that workflow-aware agent design yields noticeable gains. However, substantial headroom remains across core metrics, indicating that personalized workflow prediction continues to be a challenging capability for current systems.
10
Limitations and Ethics Statement
10.1
Limitations
Benchmark Scope. DRF LOW focuses on personalized workflow extraction from heterogeneous enterprise-style sources. Although the benchmark spans multiple domains and covers realistic procedural settings, it remains limited to a finite set of task families, domains, application types, and workflow structures. Consequently, the benchmark should not be interpreted as exhaustive coverage of all enterprise deep research workflows. Future work may extend the setting to longer-running workflows, collaborative workflows involving multiple users, temporally evolving evidence, and scenarios where policies or user contexts change during task execution. Synthetic Data Generation. DRF LOW uses an automated data generation pipeline to construct task contexts, company- and user-side workflow insights, distractors, and heterogeneous artifacts. This design enables extensible benchmark construction, but it also introduces the usual limitations of synthetic data. The generated companies, personas, documents, emails, and chat messages may not fully capture the ambiguity and institutional variation found in real deployments. Evaluation Granularity. Our evaluation decomposes model outputs into workflow steps and measures factuality, recall, precision, F1, topology order, condition resolution, and personalized comprehensiveness. This step-level design provides a more diagnostic view than holistic report scoring, but it does not capture every aspect of workflow usefulness. For example, two workflows may receive similar scores while differing in clarity, communication, or downstream executability. LLM-as-a-Judge Dependence. Several metrics rely on LLM-based judgments for semantic alignment, factual support, condition resolution, and personalization quality. Although the prompts are designed to be strict and structured, LLM judges can still introduce variance, domain-specific terminology, or partially correct condition resolutions. Model Performance. The empirical results show that current frontier models still struggle with personalized workflow prediction. Even the strongest DRF LOW agent configurations leave substantial gaps in F1, condition resolution, and personalized comprehensiveness, indicating that models often
fail to recover complete workflows and predict user-specific branches. The mixed split further compresses core workflow metrics, suggesting that robust personalized workflow prediction remains difficult under stress test. These findings highlight substantial room for improvement and pave the way for future research on structured, grounded, and personalized workflow prediction. Ablation Cost. We conduct the module ablation study on a representative subset of 15 tasks to keep computational and API costs manageable. This subset is sufficient to perform the controlled experiments, but it may not capture the full variance of module behavior across all domains and task difficulties. The ablation results should therefore be interpreted as diagnostic evidence rather than a full benchmark-wide estimate. Reproducibility. DRF LOW is evaluated in a controlled, containerized multi-application environment. This setup improves experimental reproducibility by fixing the application stack, preserving source boundaries, and allowing tasks to be re-instantiated consistently across runs. However, reproducibility may still be affected by stochastic model behavior, API-level changes in closed-source backbones, and variance introduced by LLM-based evaluation. We therefore report task and implementation details, prompts, model choices, and repeated-run statistics to support reproducibility. Furthermore, we will release the benchmark and the code implementation upon acceptance. 10.2
Ethics Statement
Data Privacy. DRF LOW is designed to simulate enterprise deep research over private-like data while avoiding the use of real personal or confidential enterprise records. The benchmark artifacts, personas, companies, documents, emails, and chats are synthetically generated for research purposes. This design reduces the risk of exposing private information or enabling re-identification while preserving the structural challenges of searching across heterogeneous organizational and personal evidence. Responsible Use of Models Output. Several DRF LOW domains, including healthcare, legal, education, and compliance-oriented business workflows, involve sensitive decision contexts. Although the benchmark data is synthetic, the task
format resembles scenarios in which real users may rely on generated workflows for consequential decisions. Systems evaluated on DRF LOW should therefore be used with caution in such settings. Their outputs should be treated as decision-support artifacts that require review by qualified professionals, especially when workflows concern legal obligations, clinical procedures, privacy rights, or compliance actions. Potential Misuse. A workflow extraction agent that can search across enterprise sources and synthesize personalized procedures could be misused to expose private organizational information, infer sensitive user attributes, or automate actions beyond a user’s authorization. DRF LOW is intended to advance evaluation of grounded, structured, and personalized workflow prediction, not to encourage unrestricted access to enterprise data. Practical deployments should enforce access control, source-level permissions, audit logging, citation transparency, and clear separation between retrieval, recommendation, and execution. Bias and Representational Coverage. Because DRF LOW relies on synthetic personas, organizations, and domain scenarios, it may encode distributional assumptions from the generation process. These assumptions can affect which roles, industries, user needs, and procedural constraints are represented. Future benchmark construction should therefore consider domain and persona diversity, representational balance, and performance differences across task types and user contexts. Use of Generative AI. Generative AI has been used to polish the writing of the manuscript.
References Amirhossein Abaskohi, Tianyi Chen, Miguel MuñozMármol, Curtis Fox, Amrutha Varshini Ramesh, Étienne Marcotte, Xing Han Lù, Nicolas Chapados, Spandana Gella, Christopher Pal, Alexandre Drouin, and Issam H. Laradji. 2026. DRBench: A realistic benchmark for enterprise deep research. In The Fourteenth International Conference on Learning Representations. Samuel Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. 2023. QAMPARI: A benchmark for open-domain questions with many answers. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 97–110, Singapore. Association for Computational Linguistics. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, David Wadden, Matt Latzke, Jenna Sparks, Jena D. Hwang, Varsha Kishore, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, and 9 others. 2026. Synthesizing scientific literature with retrieval-augmented language models. Nature, 650:857 – 863. Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. 2024. Workarena++: Towards compositional planning and reasoningbased common knowledge work tasks. Preprint, arXiv:2407.05291. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Linguistics. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Sai Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, Joao Magalhaes, Bruno Martins, and Chenyan Xiong. 2026. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research. Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long Papers), pages 1595–1604, New Orleans, Louisiana. Association for Computational Linguistics. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Licheng Zhang, Xiaorui Wang, and Zhendong Mao. 2026. Deepresearch bench: A comprehensive benchmark for deep research agents. In The Fourteenth International Conference on Learning Representations. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Association for Computational Linguistics. Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jimenez Gutierrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, TIANSHU ZHANG, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, and 7 others. 2026. Mind2web 2: Evaluating agentic search with agent-as-a-judge. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrievalaugmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4745–4759, Albuquerque, New Mexico. Association for Computational Linguistics. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116,
Singapore. Association for Computational Linguistics. Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. 2024. Autoflow: Automated workflow generation for large language model agents. CoRR, abs/2407.12821. Hanchao Liu, Rongjun Li, Weimin Xiong, Ziyu Zhou, and Wei Peng. 2025. WorkTeam: Constructing workflows from natural language with multi-agents. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 20–35, Albuquerque, New Mexico. Association for Computational Linguistics. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations. Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. Weblinx: Real-world website navigation with multiturn dialogue. Preprint, arXiv:2402.05930. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gptoss-20b model card. Preprint, arXiv:2508.10925.
Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, and Bing Liu. 2026. Researchrubrics: A benchmark of prompts and rubrics for evaluating deep research agents. In The Fourteenth International Conference on Learning Representations. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language models are not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233, Singapore. Association for Computational Linguistics. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and MingWei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6076– 6085, Hong Kong, China. Association for Computational Linguistics. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.
OpenRouter. 2026. Openrouter. https:// openrouter.ai/. Accessed: 2026-05-01.
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. 2024. Officebench: Benchmarking language agents across multiple applications for office automation. Preprint, arXiv:2407.19056.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations.
Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. 2024. FlowBench: Revisiting and benchmarking workflow-guided planning for LLM-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10883–10900, Miami, Florida, USA. Association for Computational Linguistics.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Melroy Maben, Raj Mehta, Wayne Chi, Lawrence Keunho Jang, Yiqing Xie, and 2 others. 2026a. Theagentcompany: Benchmarking LLM agents on consequential real world tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. 2026b. Researcherbench: Evaluating deep AI research systems on the frontiers of AI research. Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1358–1368, Brussels, Belgium. Association for Computational Linguistics. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan. 2022. Webshop: Towards scalable realworld web interaction with grounded language agents. In Advances in Neural Information Processing Systems. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938–8968, Miami, Florida, USA. Association for Computational Linguistics. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 414–431, Suzhou, China. Association for Computational Linguistics.
Joey Zhong, Hao Zhang, Clare Southern, Jeremy Yang, Thomas Wang, Kate Jung, Shu Zhang, Denis Yarats, Johnny Ho, and Jerry Ma. 2026a. Draco: a cross-domain benchmark for deep research accuracy, completeness, and objectivity. Preprint, arXiv:2602.11685. Yi Zhong, Buqiang Xu, Yijun Wang, Zifei Shan, Shuofei Qiao, Guozhou Zheng, and Ningyu Zhang. 2026b. Chat2workflow: A benchmark for generating executable visual workflows with natural language. Preprint, arXiv:2604.19667. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations.
Appendices A
Extended Related Work
Evidence-grounded reasoning and attribution. A substantial body of work studies whether models can combine evidence across documents and expose the basis for their predictions. HotpotQA (Yang et al., 2018) evaluates explainable multi-hop question answering with supporting facts, while MuSiQue (Trivedi et al., 2022) constructs multi-hop questions through compositional single-hop dependencies to reduce disconnected reasoning. FEVER (Thorne et al., 2018) frames evidence use as claim verification against textual sources, requiring systems to classify claims and recover supporting evidence, while ALCE (Gao et al., 2023) evaluates long-form generation with citations, measuring not only answer quality but also whether generated statements are adequately supported by cited passages. Long-form information-seeking question answering. Long-form QA benchmarks further move evaluation beyond short extractive answers. ELI5 (Fan et al., 2019) studies open-ended explanatory questions requiring multi-sentence responses, while Qasper (Dasigi et al., 2021) anchors information-seeking questions in full scientific papers and asks systems to recover evidence from long documents. ASQA (Stelmakh et al., 2022) focuses on ambiguous factoid questions whose answers must synthesize multiple valid interpretations, and QAMPARI (Amouyal et al., 2023) evaluates questions with many answers distributed across multiple paragraphs. These benchmarks are closely related to deep research because they reward synthesis over dispersed evidence. Nevertheless, they evaluate the quality of textual answers, not whether agents can transform heterogeneous evidence into a followable workflow with prerequisites, branch conditions, and user-specific applicability. Interactive web and tool-use agents. Agent benchmarks have also examined long-horizon interaction with external environments and tools. WebShop (Yao et al., 2022) introduced a simulated ecommerce environment for grounded web interaction, and WebArena (Zhou et al., 2024) extended this direction to realistic self-hosted websites spanning multiple web domains. AgentBench (Liu et al., 2024) evaluates LLM agents across interactive environments, while API-Bank (Li et al., 2023)
and ToolLLM (Qin et al., 2024) study tool selection, API invocation, and tool-use training at scale. These works test whether agents can plan and act in environments. In contrast, DRF LOW evaluates the workflow prediction capability that precedes or guides such action: the agent must recover the applicable procedure from evidence and express it as a structured user-facing action sequence. Procedural text understanding. A related line of work evaluates whether models understand processes described in text or multimodal instructions. ProPara (Dalvi et al., 2018) requires tracking entity state changes across procedural paragraphs, WIQA (Tandon et al., 2019) evaluates counterfactual reasoning over procedural influence graphs, and RecipeQA (Yagcioglu et al., 2018) studies multimodal comprehension of cooking recipes with temporal and procedural structure. These tasks capture important aspects of temporal ordering, state transitions, and procedural semantics. However, the relevant process is generally provided within a relatively bounded input. DRF LOW instead requires agents to infer the process itself from heterogeneous artifacts, decide which conditional branch applies to a particular user scenario, and produce a personalized workflow as the evaluated output. Positioning of DRF LOW. To summarize, prior works have made substantial progress on evidence attribution, long-form synthesis, interactive tool use, and procedural reasoning. However, these works do not test whether agents can predict personalized, evidence-conditioned workflows scattered in heterogeneous artifacts simulating the real-world setting. DRF LOW targets this missing capability by requiring systems to recover procedural structure, preserve action ordering, resolve conditional branches using user-specific evidence, and predict the actionable personalized workflow rather than a free-form narrative.
B
Implementation Details
We access all backbone models, including both open-source and closed-source models, through the OpenRouter API (OpenRouter, 2026). For benchmark construction, we use GPT-5.2 as the generation model. For evaluation, we use GPT-4o as the judge model for prompt-based metrics. For citation-grounded factuality verification, we use text-embedding-3-small to retrieve the top five relevant chunks from cited sources before passing the evidence to the judge model. For comparing
against DRBench Agent (DRBA) (Abaskohi et al., 2026), we update the report generation module of the agent, since the original work focuses on long-form report generation instead of workflow prediction. We use the same workflow prediction prompting and output format as DRFA to make the comparison fair. We run the application layer of the benchmark environment inside Docker containers. In particular, the enterprise-style services used by the agent, including Nextcloud, Mattermost, IMAP email, and the file sharing system, are deployed as containerized services. This setup provides an isolated and reproducible multi-application environment for document retrieval, messaging, email access, and shared-file interaction.
62.97 to 68.06, with gains in both Recall (63.78 to 70.84) and Precision (64.94 to 67.69). The improvement is also reflected in condition resolving and personalization metrics, where Condition Resolution increases from 33.96 to 48.16 and Personalization Comprehensiveness increases from 44.21 to 60.60. Among the frontier models, Claude-Opus4.7 achieves the strongest overall performance, obtaining the highest F1 (69.14), Topology Order (91.24), and Condition Resolution (51.83). GPT5.5 and Kimi-K2.6 are close in F1, reaching 68.62 and 68.54, respectively. The performance of these frontier models further show that the gaps across workflow structure, condition handling, and personalization in personalized workflow prediction remain far from saturated.
C
Ablation Studies
D
C.1
Ablation on DRFA Iteration
Table D.1 reports the domain-wise performance across all the backbone models. The same pattern holds across all five domains: DRFA consistently achieves the strongest performance on recall, precision, F1, topology order, condition resolution, and personalized comprehensiveness, while DRBA attains the highest factuality in every domain. This result is consistent with Section 6: the advantage of DRFA lies in the construction of personalized, condition-aware, and well-ordered workflows. Overall, the domain-level analysis further supports the central claim of the paper: DRFA improves workflow quality in a systematic manner across diverse domains, with especially clear gains on metrics that require personalization, conditional reasoning, and coherent step organization.
We further study the effect of the iteration budget of DRFA using GPT-5.2 as the backbone. Table C.1 shows that increasing the number of iterations does not yield a monotonic improvement across metrics. Iteration 5 exhibits the best factuality. This is potentially because it tends to cover fewer action planning steps, resulting in fewer document search and facts referencing. Moving from 5 to 10 iterations improves all other metrics, whereas increasing the budget further to 15 does not preserve these gains uniformly. Although 15 iterations gives the best Precision, F1, and condition resolution, it is slightly worse than 10 iterations on other metrics. Overall, this result shows that increasing the iteration does not guarantee consistent improvements. In particular, the comparison between 10 and 15 iterations suggests that additional iterations may slightly improve alignment quality among predicted steps, but do not necessarily improve recovery of the full target workflow or its ordering consistency. Based on this result and also to keep the budget manageable, we use the iteration of 10 in all experiments throughout this work. C.2
Performance Comparison of Recent Frontier Models
Recently introduced state-of-the-art frontier models are compared on the ablation subset to assess progress in personalized workflow prediction. As shown in Table C.2, DRFA consistently improves over DRBA across the main workflow metrics. Averaged over five backbones, DRFA raises F1 from
E
Results by Domain
Mean and Standard Deviation Results
We report the mean and the standard deviation (across 3 runs) of DRBA and DRFA on a subset of 15 tasks in Table E.1.
F
Human Evaluation Details
F.1
Step-Matching Annotation Protocol
We conduct a human study on 10 benchmark tasks across the outputs of DRF LOW and DRBA, producing 250 predicted-step decisions in total. First, we provide a guideline to the annotators detailing the annotation protocol (Figure F.1). For each task, annotators are shown the gold workflow steps and the predicted workflow steps produced by each agent. Three annotators independently align predicted steps to gold steps using a one-to-one se-
Iter.
Fact.
Topology
Cond. Res.
Pers. Comp.
Recall
Precision
F1
5 10 15
88.89 85.99 86.83
88.72 92.22 88.93
42.36 42.83 46.31
57.22 59.26 58.24
58.01 64.97 63.75
45.26 54.58 59.32
50.85 58.15 61.86
Table C.1: Ablation on the number of iterations with GPT-5.2 as the backbone. Higher is better for all metrics. The best score in each column is bolded. Model
Agent
Factuality Topology Cond. Res. Pers. Comp. Recall Precision
F1
GPT-5.5
DRBA DRFA
96.66 85.92
88.50 90.37
35.46 48.94
46.22 65.55
66.29 71.34
63.75 67.22
64.78 68.62
Claude-Opus-4.7 DRBA DRFA
83.55 82.63
89.44 91.24
38.59 51.83
50.98 65.67
69.06 73.11
64.76 68.60
65.19 69.14
GLM-5.1
DRBA DRFA
88.55 86.19
86.96 86.18
33.50 46.91
40.16 59.59
62.02 70.73
64.98 65.36
62.30 66.57
Kimi-K2.6
DRBA DRFA
92.26 83.51
87.55 86.41
34.84 49.71
45.25 59.38
62.30 70.87
65.48 68.74
62.16 68.54
Qwen3.6-plus
DRBA DRFA
77.65 82.30
88.45 88.51
27.40 43.39
38.42 52.83
59.23 68.14
65.75 68.54
61.40 67.42
Average
DRBA DRFA
87.73 84.11
88.18 88.54
33.96 48.16
44.21 60.60
63.78 70.84
64.94 67.69
63.16 68.06
Table C.2: Performance comparison of recent frontier models on the ablation subset of 15 tasks.
mantic matching protocol. A predicted step is considered matched only if it expresses the same underlying procedural action as a gold step. The annotation does not require lexical overlap: differences in phrasing are ignored when the predicted step satisfies the same workflow obligation. If a predicted step does not correspond to any gold step, it is left unmatched. Annotators are also instructed not to assign the same gold step to multiple predicted steps within the same workflow. Results. All three annotators agree on 143 of 250 decisions (57.2%), while at least two of three annotators agree on 244 of 250 decisions (97.6%). The inter-annotator agreement measured by Fleiss’ κ is 0.663, implying substantial agreement range under the Landis and Koch interpretation (Landis and Koch, 1977). F.2
Agreement with LLM-as-a-Judge
We compare the LLM-as-a-judge evaluator against the aggregated human annotations on the same 10task subset. For each task and agent, we compute the average human F1 score across the three annotators and compare it with the corresponding LLMjudge F1 score. The Pearson correlation between human-average F1 and LLM-judge F1 is 0.883,
and the mean absolute F1 difference is 0.063. Figure F.2 reports task-wise F1 scores for human evaluation and LLM-judge evaluation. The x-axis denotes the task, and the y-axis denotes F1. For both DRFA and DRBA, the human and LLM-judge scores remain close across most tasks, indicating that the LLM evaluator preserves the same tasklevel performance trends as expert annotation. For our task of personalized workflow prediction, this finding on the positive correlation between human and LLM judge contradicts some prior works (Shen et al., 2023; Chen et al., 2024), but corroborates other works (Chiang and Lee, 2023; Zheng et al., 2023) that support the similar claim.
G
Qualitative Analysis
G.1
Condition-aware personalization.
With GPT-5.2 fixed we conduct qualitative analysis on the condition-aware personalization quality of agents. The comparison between the DRBA and DRFA on healthcare_01 task shows that DRFA improves on both Condition Resolution (58.33 to 75.00) and Personalized Comprehensiveness (55.17 to 62.00) compared to DRBA. We manually inspect the predicted steps for both agents to understand the possible reason. As Table G.1
Domain
Agent
B2B
Basic DRBA DRFA
44.79 85.27 84.24
60.00 78.88 87.93
1.29 16.30 35.38
8.30 32.06 48.45
17.92 53.36 69.51
8.86 34.73 40.81
9.79 41.28 50.30
B2C
Basic DRBA DRFA
46.76 91.31 88.64
52.50 90.15 90.95
1.64 19.73 37.56
10.47 36.13 54.82
20.44 56.34 69.53
6.58 34.46 45.72
8.44 42.54 51.84
Education
Basic DRBA DRFA
44.23 92.78 88.06
35.00 88.29 91.12
0.56 22.44 40.15
5.76 41.53 60.43
18.62 58.68 69.71
4.04 33.21 48.66
5.37 41.07 56.05
Basic Healthcare DRBA DRFA
51.12 94.57 93.28
64.25 88.07 90.60
2.32 27.54 47.43
16.59 44.82 62.00
30.26 65.80 73.40
12.91 46.34 58.04
16.22 53.09 63.35
Basic DRBA DRFA
50.90 85.02 84.96
53.33 92.89 93.48
0.93 15.12 30.80
6.03 34.13 57.15
17.17 51.69 64.52
7.15 34.49 48.05
7.97 39.66 55.24
Legal
Factuality Topology Cond. Res. Pers. Comp. Recall Precision
F1
Table D.1: Domain-wise results across all domains. Backbone
Agent
Factuality
Topology
Cond. Res.
Pers. Comp.
Recall
Precision
F1
GPT-5.2
DRBA DRFA
90.63±1.93 89.37±2.78
89.80±1.95 90.71±2.04
27.10±4.68 43.89±3.74
39.20±3.25 52.60±6.26
57.20±6.78 66.08±4.28
52.23±13.07 58.33±7.96
54.23±10.27 61.87±6.19
Claude-Opus-4.5
DRBA DRFA
81.87±4.44 83.51±1.40
91.80±0.19 89.24±0.07
28.09±6.03 50.00±2.84
38.34±1.50 56.59±4.80
62.76±2.91 69.57±2.23
47.31±7.70 59.97±9.58
53.75±5.80 64.11±6.62
Gemini-3.1-pro
DRBA DRFA
87.53±4.66 81.65±6.81
88.27±0.58 87.76±2.08
32.40±2.08 35.91±6.48
32.81±8.81 45.41±7.98
63.56±4.24 67.33±2.12
51.47±8.00 56.10±8.59
56.77±6.56 60.98±6.04
DeepSeek-v3.2
DRBA DRFA
86.44±2.30 84.11±8.34
88.41±0.86 78.87±5.86
31.13±2.42 38.56±0.57
27.90±5.43 51.65±4.83
61.94±3.08 64.42±0.71
52.46±2.58 56.04±3.29
56.80±2.77 59.91±2.18
Table E.1: Mean and standard deviation results on the subset across backbone models. Higher is better for all metrics.
Agent
Example Steps
DRBA
“Plan and request a qualified interpreter (choose tier/mode)”; “Connect interpreter before any sedating premedication”; “Use the standardized consent content checklist/script” “Confirm interpreter plan: qualified VRI for consents”; “Run the consent conversation through the interpreter (do not use family)”; “Perform the pre-signature sedation/capacity gate check and escalate if needed”
DRFA
Table G.1: Qualitative comparison on healthcare_01. DRFA improves patient-specific branch instantiation while preserving the same overall clinical scaffold.
shows, DRBA agent predicts the pre-operative scaffold, but several branch-sensitive decisions remain distributed across adjacent operational steps. By contrast, DRFA packages the same evidence into more decision-oriented obligations, e.g., directly committing to the VRI path, binding the no-family rule to the actual consent encounter, and coupling the sedation-capacity check to signature execution. The gain in metric is therefore comes not from broader retrieval, but condition-aware workflow personalization. G.2
Factuality versus workflow coverage
We find that sometimes a non-factual predicted step can also carry a matched reference workflow step. For example, on b2c_08 task, the Claude-Opus4.5 workflow step (Table G.2) “Deletion Will Be Sent to MindMint’s Service Providers and Vendors”
Human Evaluation Guideline Guideline: 1. There are 10 tabs, each for one task. 2. Each tab contains the following: i. Gold step id, task title, task description ii. Title and task description for two models (model_1, model_2) 3. Your task is to assign gold step id numbers for the predicted steps by the models. -For example, if you think the 4th step of the model_1 matches the 3rd gold step, you should assign 3 in the 4th step label of ‘model_1’. Similar process for model_2 as well. Note: 1. If you think a predicted step does not match any of the gold step id, keep the label blank for that step. 2. The matches do not have to be exact. Try to check whether the action of the predicted step matches the action of the gold step. 3. Do not assign the same gold step id for two different steps for the same model. -For example, do not assign 5 for in the two step labels of ‘model_2’.
Figure F.1: Human evaluation guideline. 1.0
F1 Score
0.8
0.6
0.4
0.2
DRFA human avg. F1 DRFA LLM-judge F1 DRBA human avg. F1 DRBA LLM-judge F1
0.0
c alth
he
_01
are
he
are
c alth
_04
_06
b2c
_10
b2c
_04
b2b
_10
b2b
Task ID
atio duc
e
4
n_0
e
atio duc
0
n_1
2
al_0
leg
4
al_0
leg
Figure F.2: Task-wise agreement between human and LLM-judge F1. The x-axis shows the evaluated tasks, and the y-axis shows F1. For each task, we plot the human-average F1 and LLM-judge F1 for DRF LOW and DRBA. Smaller vertical gaps between the human and LLM markers indicate closer agreement.
reaches the correct part of the workflow, namely processor-side deletion, but hallucinates what the evidence warrants. This makes the step problematic from a factuality perspective. However, simply removing it does not cleanly solve the workflow. Because the step is also one of the matched obligations, pruning it would reduce the number of matched steps, weakening procedural coverage even as factuality improves. This example showcases why workflow cannot be optimized by simply guardrailing against nonfactual steps alone. If an unsupported step is unmatched, pruning is mostly beneficial. If it is unsupported but also matched, pruning improves local reliability at the expense of workflow completeness.
G.3
Cost–Performance Analysis
We further analyze the cost-performance trade-off of DRFA agents using backbone prices collected from OpenRouter (OpenRouter, 2026). We use GPT-OSS-120b (OpenAI et al., 2025) tokenizer to tokenize the reports then calculate the average report cost. Figure G.1 shows the comparison among the frontier models on F1 and Condition Resolution. Both comparisons exhibit the same Pareto pattern: DeepSeek-V3.2 anchors the lowestcost regime, Qwen3.6-plus and Kimi-k2.6 provide strong mid-cost trade-offs, and Claude-Opus-4.7 achieves the best absolute performance. On F1, Kimi-k2.6 reaches 68.54 at a cost of 0.044, closely approaching Claude-Opus-4.7 at 69.14 despite being substantially cheaper. On Condition Resolu-
54
70
Claude-4.7
68
GPT-5.2
Qwen3.6+
66
GLM-5.1
Gemini-3.1-pro
Claude-4.5
F1
64 62 60
Claude-4.7
52
GPT-5.5
Condition Resolution
Kimi-k2.6
Claude-4.5 Kimi-k2.6
50
GPT-5.2
48
GPT-5.5
GLM-5.1
46
Gemini-3.1-pro
44
Qwen3.6+
42
58 40
DeepSeek-V3.2
DeepSeek-V3.2
56 38
$0.005
$0.01
$0.02
$0.05
$0.1
$0.2
$0.5
$0.005
Average cost per workflow report
$0.01
$0.02
$0.05
$0.1
$0.2
$0.5
Average cost per workflow report
(a) Performance comparison on F1.
(b) Performance comparison on Condition Resolution.
Figure G.1: Cost-performance analysis of the models on the ablation subset. The x-axis shows cost per report, with prices (USD/1M tokens) collected from OpenRouter (OpenRouter, 2026), and the y-axis shows performance on F1 or Condition Resolution.
Workflow Example step
# matched steps
Original “Deletion Will Be Sent to MindMint’s Service Providers and Vendors” Justification: The sources do not explicitly mention that MindMint will send formal deletion requests to each vendor or track vendor confirmations. While the sources discuss the internal processes for handling deletion requests and maintaining compliance, they do not provide specific details about coordination with third-party vendors for data deletion, making the claim unsupported.
15
Pruned
14↓1
After removing “Deletion Will Be Sent to MindMint’s Service Providers and Vendors”
Table G.2: Pruning an non-factual step on b2c_08 improves factuality, but also removes a matched processordeletion obligation. This illustrates that unsupported steps are not always disposable over-generation.
tion, Kimi-k2.6 again offers the strongest mid-cost operating point at 49.71, while Claude-Opus-4.7 remains the top performer with the cost of 0.395. Overall, the Pareto frontier indicates that DRFA can construct most of its workflow quality with midcost backbones, while the most expensive model yields additional gains.
G.4
Closed-source versus open-source behavior
We manually inspect the predicited workflow of closed-source (GPT-5.2) vs open-source (DeepSeek-V3.2) models on healthcare_01 task (Table G.3). We find that GPT-5.2 describes the relevant evidence into steps that are closer to the atomic decision structure of the reference workflow. For example, it ties the VRI choice, the no-family constraint, and the sedation-capacity gate directly to the actual consent path. On the other hand, DeepSeekV3.2 often distributes the same evidence across a more generic operational sequence. The resulting workflow remains plausible and factual, but is less decisive at the point where the personalized branch should actually be chosen. This explains why the models diverge substantially on precision and condition resolution.
H
Error Analyses
H.1
Manual Analysis of DRFA Factuality
DRFA, unlike DRBA, combines adaptive gap finding with conditional action planning. While this design improves coverage and condition resolution, the interaction between these modules may also produce overlapping strategies, which can create redundant or unstable planning behavior, as observed in (Abaskohi et al., 2026), hurting the factuality of the predicted workflows. Furthermore, we manually analyze the workflows generated by the agents, to better understand why DRFA obtains a lower factuality score than DRBA. We find that DRFA’s comparatively
Model
Precision
Cond. Res.
Example steps
59.5
GPT5.2
50.00
75.00
“Confirm interpreter plan: qualified VRI for consents”; “Run the consent conversation through the interpreter (do not use family)”; “Perform the pre-signature sedation/capacity gate check and escalate if needed”
59.0
DeepSeek- 33.33 V3.2
41.67
“Schedule the Contracted VRI Service for the Procedure Day”; “Conduct Consent Discussion Entirely Through VRI Interpreter”; “Re-assess Capacity if Sedating Medication Administered”
Table G.3: Closed-source versus open-source comparison on healthcare_01. Both models predict the broad clinical scaffold, but GPT-5.2 is more effective at binding evidence to branch-specific clinical obligations.
lower factuality score can potentially be due to the specificity of the claim cited by the agent. A side-by-side comparison between DRFA vs DRBA on the opening step of the personalized workflow on healthcare_01 (Claude-Opus-4.5) in Table H.1 illustrates this pattern. Both steps target the same obligation (confirm the patient’s preferred language during pre-op intake), and both cite valid policy sources. DRBA’s task description stays close to the language of the cited policy: a generic obligation that the judge can verify directly against the source. DRFA’s task description, in contrast, binds the obligation to case-specific operational detail (“navigate to the designated Language & Communication area,” “requested interpreter support for all medical decisions”) that fuses the generic policy with personal evidence. The judge, which retrieves the top relevant chunks from cited sources before evaluating each claim (Appendix B), still has a reasonable chance to verify such fused claims. But since the verification process is inherently stricter: the cited chunks must semantically match the pre-
Claude Opus 4.5
Precision
58.5
GPT 5.2
58.0 57.5 57.0 56.5
Gemini 3.1 Pro
56.0 8.0
8.5
9.0
9.5
10.0
10.5
11.0
Distractor-reference rate (%)
11.5
DeepSeek v3.2
12.0
Figure H.1: Relationship between distractor-reference rate and precision. The downward trend indicates that models with higher proportions of distractor references tend to achieve lower precision.
dicted claim, fusing the policy with personalized evidence can reduce the semantic alignment, resulting in lower factuality score. Adding further citations partially compensates, since each additional source raises the chance that the predicted claim is locally supported, which is why the observed gap is small (∼2 points). The residual gap is therefore not a deficiency of grounding but a measurable cost of DRFA’s design choice to produce more personalized claims; a cost that is repaid by its gains on Recall, F1, Condition Resolution, and Personalized Comprehensiveness. Overall, these findings suggest that DRFA’s lower factuality score is partly a consequence of its more aggressive evidenceseeking and planning behavior, which improves coverage but increases the likelihood of citation misalignment. H.2
Role of Distractors in Low Precision
We analyze whether low precision in Table 2 is associated with models incorporating distracting information into the generated workflow reports. For each model, we compute the distractor-reference rate in the DRFA report as the percentage of references that point to distractor information. We then compare this rate against the corresponding model-level precision score. Figure H.1 shows a negative association between distractor-reference rate and precision. Models that include a larger fraction of distractor references tend to obtain lower precision, suggesting that precision errors are not only caused by missing relevant evidence, but also by selecting and propagating spurious evidence into the workflow.
Agent
Step 1 Task Description (healthcare_01, ClaudeOpus-4.5)
Verification surface
DRBA
“During pre-operative intake, ask the patient directly, not accompanying family members, what their preferred language is for medical communication. Document this language preference in the EHR immediately.”
Generic, policy-aligned obligation; phrasing closely tracks the cited policy text, so a single chunk retrieved from the cited source supports the claim.
DRFA
“During pre-op intake at Sunrise Surgical Center, access the patient’s EHR chart and navigate to the designated Language & Communication area. For this case, pre-op intake has identified Spanish as the patient’s preferred language and the patient has requested interpreter support for all medical decisions.”
Synthesized obligation that fuses generic policy with personal-evidence facts. The judge must verify the operational detail (the named EHR area, the patientspecific request) against retrieved chunks; each citation supports a sub-claim, and the step is factual only if those sub-claims jointly hold.
Table H.1: Paired DRBA and DRFA steps on the same task, illustrating the claim-specificity difference behind the small Factuality gap.
Agent
*_IN_* (%)
*_DI_* (%)
*_AD_* (%)
DRFA DRBA
70.7 65.0
2.2 8.2
27.2 26.8
Table H.2: Citation composition on the mixed split.
H.3
Impact of the Mixed Split
We inspect a random subset of 20 tasks to characterize how the mixed split degrades performance in Table 3. For each predicted workflow, we count citations to supporting (*_IN_*), distractor (*_DI_*), and mixed (*_AD_*) sources, and we manually inspect predicted personalized workflows to determine whether contamination propagates into the steps. Citation composition. Table H.2 reports the share of citations by source type. Both agents refer a substantial amount of citations to mixed sources, while classical distractor citations remain low (2 8%). This confirms that the mixed artifacts, which are internally consistent and topically aligned with the target question, attract retrieval far more than off-topic distractors, degrading the overall performance of the agents. Manual inspection. Table H.3 shows two representative cases in which the agent grounds the personalized workflow on the mixed case rather than the user’s case. In b2c_02, the predicted workflow opens with the proration case identifiers (Marcus Whitfield, SG-48219) sourced from dispute-review-sg48219.docx, even though the original user’s task concerns a duplicate charge. In education_10, 5 of 15 predicted steps cite mixed artifacts (chair-provost-approval.docx, aaf-split-checklist.docx) describing a con-
version with prior-service-credit case, while the user’s task concerns onboarding a newly hired tenure-track faculty. Reconciliation with Table 3. When comparing against the performance on the original variant (Table 2), two patterns explain the metric movements observed. (i) Precision drops more than Recall (−14.37 vs. −10.60 averaged for DRFA; −10.77 vs. −5.98 for DRBA) because the agents recover the generic procedural scaffold but inject step-level details that inlcude the mixed case. (ii) Factuality remain over 80%, because mixed artifacts are internally consistent documents that locally support the (wrong) claims they are cited for. This shows that an mixed-grounded workflow can be locally factual against its citations, yet globally wrong with respect to the user. This failure mode is invisible to citation-only evaluation but is captured by Recall, Precision, F1, Condition Resolution, and Personalized Comprehensiveness, validating the effectiveness of the mixed split to stress test the agents.
I
Performance Comparison Between Original & Mixed Variants
We show the performance comparison between the original and the corresponding mixed variants across all the models in Figure H.2.
J
DRFA Agent Details
In this section, we provide implementation details for DRFA. The design goal of DRFA is to predict a generic workflow from company-side evidence, identify the decision points that remain unresolved at the generic level, and then instantiate a personalized workflow using user-specific evidence.
Task
Original DR Question
Mixed DR Question
Contaminated Step (excerpt)
b2c_02
What is the workflow to dispute a duplicate charge on a customer’s subscription?
What is the workflow to handle a mid-cycle upgrade proration dispute?
“Log Marcus Whitfield’s SG-48219 Proration Dispute . . . attach screenshot 11/08/2025 20:14 UTC . . . target resolution Nov 21, 2025 [ˆ6]” : [ˆ6] resolves to dispute-review-sg48219.docx
edu_10
What is the workflow to onboard a newly hired tenuretrack faculty?
What is the workflow to convert a non-tenure-track faculty with prior service credit?
“Submit Department Chair Endorsement and Dossier . . . prior service credit recommendations [ˆ5]” : [ˆ5] resolves to chair-provost-approval.docx
Table H.3: Qualitative examples of mixed contamination. The predicted workflow binds steps to the mixed case.
J.1
Detailed Workflow
DRFA operates as an iterative tool-calling agent over heterogeneous private information sources. At the beginning of a session, the agent initializes a task-specific vector store, registers its tool inventory, and constructs a research context tied to the original deep research question. The execution pipeline then proceeds through the following stages. Stage 1: Research Planning. The agent decomposes the original deep research question into a small set of research investigation areas. These areas are designed to cover two complementary evidence objectives: constructing the generic organizational procedure and collecting user-specific evidence for personalization. Stage 2: Action Planning. The research plan is translated into executable actions. Each action is parameterized by an action type, a natural-language description, tool arguments, expected outputs, dependencies, preferred tool choices, and a workflow stage label. The planner prioritizes internal sources over external ones, favoring local documents and enterprise applications, while reserving web search as a fallback when internal evidence is insufficient. The resulting action plan forms the initial retrieval program for the session. Stage 3: Iterative Research Loop. The agent executes the action plan under a fixed iteration budget. At each iteration, it selects the next executable actions, invokes the appropriate tools, processes the returned content, and stores the resulting findings in the task context and vector store. This loop continues until the plan converges or the iteration budget is exhausted. Stage 4: Adaptive Action Planning. At the end of each action-planning round, DRFA analyzes
the newly collected findings and decides whether the plan should be expanded. This adaptive step first prioritizes conditional action planning, which extracts an intermediate generic workflow and proposes focused actions for resolving its conditional branches, and then performs complementary gap finding when evidence remains unbalanced across the generic and personal sides. Stage 5: Workflow Generation. After the research loop converges, the agent predicts the final workflow output. This stage reconstructs the full generic workflow, resolves it using user-specific evidence, and produces a cited personalized workflow in which each step is justified by retrieved facts. J.2
Research Planning Implementation
The research planning module organizes the problem around the structure of the target output rather than around generic topical decomposition alone. In particular, it separates the search problem into two evidence objectives. The first objective is Generic Requirement Collection, which targets company-side materials such as policies, procedures, playbooks, templates, and operational documents. These materials define the organization-level procedure that the agent must predict as the generic workflow. The second objective is Personal Evidence Collection, which targets user-side materials such as emails, chats, notes, records, and other personal artifacts. These materials determine which conditions in the generic workflow hold for the current case and therefore how the workflow should be instantiated for the user. Each research investigation area is represented as a structured object with fields such as the research focus, information needs, expected knowledge sources, and workflow stage. This representation is intentionally execution-facing, so that it
can be passed directly to the action planner without requiring an additional translation step. The research planning module therefore produces a decomposition aligned with workflow prediction and workflow personalization. A representative example of a research plan is shown in Figure J.1. J.3
Action Planning System
The action planning system converts research objectives into executable actions while managing tool choice, prioritization, and dependencies. Each action is represented as a typed record containing an action identifier, action type, description, argument dictionary, priority score, dependency list, expected output, preferred tool, and workflow stage. This representation makes the plan explicit enough to support iterative scheduling, tool-aware execution, and post hoc traceability. The current action space includes retrieval actions over the web, enterprise systems, and local documents, as well as analysis-oriented actions for workflow prediction. The workflow stage labels used by the planner distinguish between requirement collection, personal evidence collection, generic workflow design, and workflow personalization. This stage-aware action representation is important because DRFA does not treat retrieval as a uniform process: different stages require different search targets and different criteria for success. Tool priority follows a simple internal-first policy: • Local document search is preferred for uploaded documents, policies, playbooks, and other authoritative files. • Enterprise API search is preferred for emails, chats, enterprise files, and other application-level records. • Web search is used as a fallback when internal sources are insufficient or when external validation is required. Action priorities are assigned on a continuous scale and are further refined during execution using novelty and source-diversity signals. The scheduler favors actions that explore new evidence dimensions while balancing internal and external retrieval when appropriate. Dependency detection is handled separately, allowing the planner to serialize actions only when later steps genuinely depend on earlier outputs. A representative example of an action plan is shown in Figure J.2.
J.4
Enterprise Integration Architecture
DRFA supports retrieval over both enterprise applications and locally supplied files through a unified tool interface. Enterprise connectivity is provided by an enterprise API search tool, while local files are handled through a local document ingestion tool and a local semantic search tool. This makes it possible to run the agent in both enterprise-backed settings and file-backed settings without changing the high-level execution logic. The integration layer preserves source metadata throughout retrieval and processing. As a result, downstream prediction stages can still distinguish among enterprise emails, enterprise chats, enterprise files, internal documents, and external sources. This source-type preservation is critical for workflow grounding, because the role of a piece of evidence depends not only on its content but also on whether it encodes organizational policy, personal context, or external background information. J.5
Tool Selection and Execution
Tool execution is iterative and priority-driven. At each iteration, the scheduler retrieves the subset of pending actions whose dependencies have already been satisfied, computes their execution scores, and selects the next actions subject to the current concurrency limit. The default setting executes one action at a time, which simplifies traceability and makes it easier to attribute downstream workflow decisions to specific retrieval events, although the infrastructure supports bounded concurrency. Tool selection itself follows a two-level policy. If the action planner specifies a preferred tool, the executor attempts to honor that choice directly. Otherwise, DRFA falls back to type-aware selection based on the action class and workflow stage. Internal and enterprise retrieval actions are routed toward local-document or enterprise API search tools, web-oriented actions are routed toward search or URL-fetching tools, and prediction-heavy actions are routed toward the analysis module. Once an action is executed, its outputs are integrated back into the research context together with stage annotations, action-type metadata, execution timing, and any processed files created during execution. Significant results are also written into the vector store so that they remain available to later planning and workflow-generation stages.
J.6
Adaptive Action Planning
Adaptive action planning is the key mechanism that makes DRFA workflow-aware. At the end of each execution round, the agent re-examines the current evidence and decides whether the existing plan should be extended. This adaptation has two components: gap finding and conditional action planning. The first component, gap finding, analyzes the current source composition and coverage of the action plan. If the evidence disproportionately covers only the generic side or only the personal side, the planner proposes a small number of complementary actions to reduce that imbalance. These actions are constrained by explicit diversity rules so that the agent does not repeatedly issue near-duplicate searches. The second component, conditional action planning, extracts an intermediate generic workflow from the currently available policy and workflow evidence. It then identifies the subset of workflow steps that contain unresolved decisions. For each such step, the model returns a structured condition object containing a condition identifier, the corresponding step number, a natural-language description of the decision, the alternative branches, the type of personal evidence needed to resolve it, and a set of diverse search queries that are likely to surface that evidence. These condition objects are then converted into focused personal-evidence actions over emails, chats, records, and files. Importantly, DRFA prioritizes conditional action planning when selecting the next action items. Newly proposed condition-resolution actions are inserted before more general gap-filling actions, and they typically receive high priority because they are the most directly useful for transforming a generic workflow into a personalized one. In effect, once the agent has constructed candidate workflow structure, it shifts from broad evidence gathering toward targeted branch disambiguation. A representative adaptive-planning example is shown in Figure J.3. J.7
Content Processing and Vector Store
Similar to DRBench (Abaskohi et al., 2026), the content-processing subsystem implements a unified ingestion pipeline for heterogeneous documents and retrieved results. Documents in different formats are normalized into text, cleaned, segmented, and embedded before being stored in the task-
specific vector store. The vector store therefore acts as the main long-horizon memory of the agent. This design is particularly important for workflow extraction. Evidence relevant to different workflow steps may be acquired several iterations apart, and the final workflow-generation stage must still be able to retrieve and jointly reason over these dispersed fragments. By storing both raw retrieved content and synthesized intermediate findings, the vector store enables semantic retrieval not only over original files but also over the agent’s own accumulated research state. The current implementation creates an isolated vector store for each session, which prevents interference across tasks and preserves reproducibility of the retrieval context. Session-level metadata is also retained so that execution traces, saved plans, and workflow outputs can be inspected after generation. J.8
Workflow Generation and Citation Grounding
The workflow-generation module predicts the final output from the accumulated evidence. Its prediction procedure has three phases. In the first phase, the agent gathers candidate evidence either from the full vector store or from the executed action plan alone. This stage collects both company-side workflow evidence and user-side personal evidence, while preserving their source metadata. In the second phase, the system constructs a source registry for grounding. It filters invalid or unusable sources, normalizes source identities, registers valid documents in a citation registry, and maps them into a simplified citation namespace used during prompting. This registry is also responsible for preserving the source distinctions needed for later references, including internal documents, enterprise files, emails, chats, and external materials. In the third phase, the model predicts the final workflow output. The generation prompt asks for a complete generic workflow, a user-situation analysis, a condition-resolution table, a personalized workflow, and step-by-step personalized guidance. The generic workflow must preserve prerequisite structure and explicitly record branch logic through a dedicated condition_description field. The personalized workflow must preserve the same step inventory, but replace generic descriptions with user-specific facts wherever possible and clear the
condition_description field when the relevant branch has been resolved by evidence. When evidence remains insufficient, the unresolved condition is retained explicitly. The agent ensures that each workflow step is grounded in citation-supported facts. The generator requires citations in step-level task descriptions, personalized justifications, and action guidance. After initial generation, the system performs targeted citation repair, validates that each personalized step contains at least one citation and a non-empty justification, resolves simplified citation identifiers back to real source identifiers, and finally produces a filtered references section. The citation registry is therefore not only a post-processing convenience, but part of the workflow-grounding mechanism itself. J.9
Prompts for DRFA
Research Planning Prompt. The research planning prompt decomposes the task into workfloworiented investigation areas, with explicit distinction between generic requirement collection and personal evidence collection. We show the prompt in Figure J.4. Action Generation Prompt. The actiongeneration prompt converts each investigation area into a small set of diverse executable actions, conditioning on the workflow stage, available tools, and internal-first source priority. Prompt is shown in Figure J.5. Adaptive Planning Prompt. The adaptive prompt supports two cases. In the primary case, it uses an intermediate generic workflow and its extracted conditions to propose targeted conditionresolution actions. In the fallback case, it proposes complementary gap-filling actions when the current evidence remains insufficient or unbalanced. We show the prompt for adaptive planning in Figure J.6 and the prompt for generating intermediate generic workflow in Figure J.7, as well as conditional action planning to resolve the intermediate workflow in Figure J.8. Workflow Prediction Prompt. The workflow prediction prompt predicts the final generic and personalized workflows, including condition resolution, personalized justifications, and step-level grounding. We show the prompt in Figure J.9.
K
Detail Implementation of DRF LOW Data Generation
This section describes the prompt families used to synthesize DRF LOW tasks. The prompt inventory follows the benchmark construction pipeline described in the main text: task-seed generation, dual-context construction, company-side workflow evidence generation, user-side evidence generation and personalization, file realization, and introductory email synthesis. Together, these prompts create not only realistic artifacts, but also the latent workflow objects that define the benchmark targets. K.1
Task Seed Generation Prompts
The first prompt creates the task seed, namely the deep research question together with company and user metadata. The main seed-generation prompt takes a target domain and asks the model to produce exactly n (e.g., 10) realistic, workflow-focused questions, each paired with company identity, user identity, and user-role metadata. The instructions constrain questions to be short, unambiguous, steporiented, and realistic, while encouraging the company and user to be contextually aligned with the workflow question. This prompt is therefore the primary mechanism by which DRF LOW instantiates diverse domain-specific tasks while preserving a consistent workflow-centric question style. The corresponding prompt template is shown in Figure K.1. K.2
Dual-Context Construction Prompt
Once the task seed has been created, the next prompt template constructs the two contextual objects that define the benchmark instance: the generic company context and the personal user context. The context-construction prompt receives the deep research question together with the company and user metadata, and asks the model to generate a generic workflow-like company policy and a userspecific personal context consistent with that policy. The prompt imposes several structural constraints that are central to DRF LOW: the generic context should contain 12–15 steps, include both linear and conditional structure, remain self-contained, and support multi-hop reasoning; the personal context should satisfy only one branch when a generic condition is present and should be written as concise, workflow-aligned bullet points. This prompt therefore creates the latent pair (Cig , Ciu ) that underlies the later generic and personalized workflows. The
corresponding template is shown in Figure K.2. K.3
Company Workflow Evidence Prompts
The company-side evidence prompts convert the generic context into workflow-bearing supervision. The company insight prompt takes the generic context and asks the model to generate an ordered list of actionable workflow steps, each with a title, task, task description, condition description, prerequisite pointer, and prerequisite description. The prompt explicitly instructs the model to preserve the order of the generic context, maintain the same number of steps, and avoid splitting conditional branches into separate top-level steps. This makes the output suitable for defining the gold generic workflow and for downstream file realization. The corresponding prompt template is shown in Figure K.3. The paired company distractor prompt produces plausible but irrelevant company-side steps. Its instructions emphasize topical plausibility and business realism while requiring that the steps remain tangential to the deep research question. In practice, this prompt creates adjacent but non-resolving procedures that populate distractor files and increase retrieval difficulty without altering the target workflow. The corresponding prompt template is shown in Figure K.4. K.4
User Evidence and Personalized Workflow Prompts
The user-side prompt family serves two distinct functions: generating user-specific supporting and distracting evidence, and composing the personalized workflow. The user insight prompt converts the personal context into a set of question-answer-justification tuples. The crucial design choice is that answers must be extracted verbatim from the personal context, while the generated questions and justifications frame those facts as evidence relevant to the deep research task. The prompt also requires full coverage of the personal context across the generated tuples, ensuring that user-side support evidence spans the complete personalized scenario rather than only a small subset of salient facts. The corresponding template is shown in Figure K.5. The paired user distractor prompt generates plausible but task-irrelevant user-side events. These distractors are framed as realistic workplace operations, metrics, or administrative events that are compatible with the user’s role but do not help resolve the deep research question or the workflow
branches implied by the company context. The corresponding template is shown in Figure K.6. The final prompt in this family is the personalized workflow prompt. It takes the generic company workflow and the user facts as input, then asks the model to resolve conditional paths, personalize task descriptions, preserve workflow structure, and provide a justification for each step. The prompt explicitly states that resolved branches should collapse into the applicable path, while unresolved branches should remain visible. It also requires that every personalized step retain sufficient procedural detail and include a natural-language justification grounded in user facts. This prompt is therefore responsible for constructing the gold personalized workflow used in evaluation. The corresponding template is shown in Figure K.7. K.5
File Generation Prompts
The file-generation stage realizes latent company and user evidence as heterogeneous artifacts. Rather than fixing a single output modality, the generator samples the target file type from a predefined distribution over the supported modalities for that side of the benchmark. This proportional sampling strategy preserves modality diversity across tasks while allowing the generator to control how often different artifact types appear. In the current pipeline, company-side generation samples from the supported company document modalities, while user-side generation samples from a broader set that includes documents and communication artifacts. For company-side document artifacts, the prompt family is divided into an outline prompt and a section prompt, with parallel variants for supporting insights and distractors. The supporting outline prompt asks for a professional internal document skeleton with exactly one subsection reserved for the relevant company insight and the remaining subsections serving as thematic distractors. The supporting section prompt then asks the model to paraphrase and embed the workflow step, its detailed procedure, and any prerequisite information into a realistic business paragraph. The distractor variants preserve the same document realism but ensure that the inserted content remains unrelated to the deep research question. The corresponding outline and section templates are shown in Figures K.8 and K.9, while the company distractor counterparts are shown in Figures K.10 and K.11. For user-side document artifacts, the correspond-
ing prompt family again uses an outline prompt plus insight and distractor section prompts. Here the document style is conditioned on the user persona rather than the company policy alone. The user outline prompt asks for a realistic document that the persona could plausibly create or receive, while the user section prompt requires faithful incorporation of the target personal event into a naturalistic paragraph. The distractor section prompt mirrors this structure but excludes any event that would help answer the deep research question. The corresponding templates are shown in Figures K.12, K.13, and K.14. The same file-generation stage also supports chat and email artifact generation for user-side files. We show the prompt for generating chat and email artifacts in Figures K.15 and K.16 respectively.
K.6
Mixed Evidence Prompts
We further stress-test agents under realistic caselevel disambiguation rather than coarse topical filtering. For this, we construct the mixed variant in which two similar but distinct cases in the same workflow domain are injected into the evidence pool. Given the task seed and the existing user insights, we sample two adjacent DR questions that share the workflow domain and stakeholder context but pertain to a different case. For example, in a consent task involving a limited-Englishproficiency patient (Figure 3), an adjacent DR question can be about the consent for an incapacitated patient through a legally authorized representative. For each adjacent question, we derive a set of coherent insights that form an internally consistent case profile distinct from the original insights, together with a shared persona cast and case identifier so that artifacts produced for the same adjacent case remain self-consistent across formats. The resulting confounding evidence is realized as heterogeneous files across the same modalities used in Section K.5. Unlike misinformation-style perturbations, adjacent cases are internally consistent records that do not directly contradict the target user’s evidence; the difficulty arises from disambiguating the user’s case from a topically adjacent one rather than from spotting contradictions. We provide the corresponding prompts to generate the evidence in Figure K.17, K.18, K.19, K.20.
L
Cost Estimation
DRF LOW incurs cost from three stages: benchmark construction, agent execution, and automatic evaluation. Benchmark construction is a one-time cost. Each task requires generating the task seed, generic workflow, company-side evidence, personalized workflow, user-side evidence, and heterogeneous artifacts. We use GPT-5.2 as the construction model, with pricing of $0.948 per 1M input tokens and $14.01 per 1M output tokens. Each task construction consumes approximately 30K–50K tokens. Under an output-heavy generation mix, this corresponds to roughly $0.40–$0.70 per task, or approximately $25–$50 for constructing 100 tasks, excluding retries and infrastructure overhead. Agent execution is the main recurring cost. Under the iteration budget 10, and one DRFA agent run costs approximately $3–$6 per task, depending on the backbone models. This cost reflects iterative research planning, retrieval, adaptive action planning, conditional action planning, and final citation-grounded workflow prediction. Automatic evaluation uses GPT-4o as the judge for prompt-based metrics, and judging one task costs approximately $0.267. This includes step alignment, factuality checking, conditionresolution assessment, topology evaluation, and personalized comprehensiveness scoring. Overall, with moderate cost, DRF LOW remains substantially more scalable and cost-effective than manually constructing an equivalent personalized, multi-application workflow benchmark at comparable scale.
M
Application Environment
Following DRBench (Abaskohi et al., 2026), we instantiate a self-contained, containerized enterprise environment that exposes heterogeneous private information through realistic workplace applications rather than through a single flat file interface. Our environment includes four primary application families. Nextcloud serves as the shared document repository for company-side materials and other uploaded files. Mattermost provides internal chat communication. IMAP-based email supports access to personal and organizational email records. A file sharing / local file system interface exposes
additional files outside the shared cloud workspace. Together, these applications create a distributed evidence space in which relevant workflow signals and distractors are fragmented across storage, email, and chat modalities. For each task, the synthesized artifacts are loaded into these applications according to their modality and role in the scenario. Companyside documents are placed into shared repositories and related communication surfaces (Nextcloud/file sharing system), while user-side evidence is distributed across personal files (Nextcloud/file sharing system), emails, and chat logs. The design of such multi-application ecosystem is intentional: the agent must predict the generic procedure from organization-level artifacts and then locate the user-specific evidence required to resolve the relevant workflow conditions from the joint environment. The application environment therefore does not merely store files, but operationalizes the retrieval challenge of DRF LOW as a realistic multiapplication search problem. The environment supports two key properties required by our benchmark. First, it preserves application boundaries, so that evidence is encountered in the same fragmented form as in realistic enterprise settings. Second, it remains reproducible, since the full application stack is containerized and can be re-instantiated consistently across tasks and experiments. This makes the app environment a controlled but realistic substrate for evaluating workflow extraction agents under cross-application retrieval.
representations. N.1
The evaluation pipeline begins by extracting the personalized workflow, the generic workflow, the condition-resolution table, the personalized flowchart, and the reference map from the generated report. These parsed objects provide the shared input representation for all subsequent metrics. The evaluator then executes the metric suite in dependency-aware order and stores intermediate alignments so that later metrics can operate on matched workflow steps rather than on raw report text. At a high level, the metric suite evaluates seven complementary aspects of workflow quality: precision, recall, F1, factuality, condition resolution, topology ordering, and personalized comprehensiveness. Precision and recall evaluate structural alignment to the reference workflow. F1 summarizes these two alignment measures. Factuality checks whether predicted steps are supported by their cited evidence. Condition resolution evaluates whether the system correctly instantiates userdependent branches. Topology ordering checks whether the predicted workflow preserves the correct relative ordering among aligned steps. Personalized comprehensiveness evaluates whether matched steps retain the user-specific operational detail present in the reference workflow. N.2
N
Prompts for Evaluation Metrics
This section describes the prompt interfaces used in the DRF LOW evaluation pipeline. Our evaluation is designed to measure workflow quality at the level of atomic personalized workflow steps, rather than judging the generated report only as free-form long text. This design serves two purposes. First, it lets us separate retrieval and grounding errors from structural workflow errors. Second, it makes it possible to diagnose whether a model failed because it omitted a required step, introduced a spurious step, misordered valid steps, failed to resolve a condition, or produced a step that was insufficiently grounded or insufficiently personalized. The full evaluation pipeline therefore first parses the generated report into normalized workflow objects, and then applies a set of metric-specific judges over these structured
Workflow Parsing and Evaluation Overview
Precision Prompt
The Precision metric evaluates whether each predicted workflow step corresponds to any reference workflow step. The judge prompt is intentionally precision-oriented: it asks the evaluator to match on procedure intent rather than lexical overlap, ignore instance-specific strings such as names or dates, and reject broad or weakly related matches. For each predicted step, the judge returns at most one best-matching reference step, together with a confidence score and a short justification. After prompt-based matching, the evaluator applies additional one-to-one constraints so that only the strongest valid alignments are retained. This makes precision stricter against spurious predicted steps and prevents multiple predicted steps from claiming the same reference step. A representative prompt template is shown in Figure N.1.
N.3
Recall Prompt
The Recall metric reverses the alignment direction. Instead of asking whether a predicted step is valid, it asks whether each reference workflow step is materially covered by at least one predicted step. The corresponding judge prompt is recall-oriented but still strict: it asks whether some predicted step would actually satisfy the obligation expressed by the reference step, rather than merely mentioning a nearby topic or an upstream or downstream activity. For each reference step, the judge returns the best predicted match, a confidence score, and a short justification. One-to-one constraints are then applied so that multiple reference steps cannot all claim the same predicted step. A representative prompt template is shown in Figure N.2. N.4
F1 Score
The F1 score summarizes structural workflow recovery by combining per-task precision and recall into a single measure: F1 =
2 · Precision · Recall . Precision + Recall
This score is useful because it penalizes systems that achieve high recall by over-generating steps, as well as systems that achieve high precision by returning only a small subset of the reference workflow. In our setting, F1 therefore serves as the main summary statistic for step-level workflow alignment. N.5
Factuality Prompt
The Factuality metric verifies whether each predicted workflow step is supported by its cited evidence. For each step, the evaluator constructs a claim from the step title, task, and task description, extracts all embedded citations from the step text, resolves those citations through the report’s reference map, and then checks whether the cited content supports the claim. Steps with no citations are marked as non-factual. Otherwise, following Abaskohi et al. (2026); Min et al. (2023), we embed the cited source with text-embedding-3-small, retrieve the top-5 relevant chunks, and ask GPT-4o to determine whether the cited evidence supports the core claim of the step. The judge prompt in this metric operates over a step-level claim and the retrieved cited evidence. Its purpose is not to compare workflows to one another, but to determine whether the cited materials
actually support the procedural claim expressed by the predicted step. This makes the metric sensitive to hallucinated actions, unsupported procedural detail, and citation misuse. A representative prompt template is shown in Figure N.3. N.6
Condition Resolution Prompt
The Condition Resolution metric is applied only to reference workflow steps that contain explicit conditions. For each such reference step, the evaluator first checks whether there is a matched predicted step from precision. If not, the score is automatically set to 0. Otherwise, the judge receives the reference condition, the expected branch resolution from the reference answer, the reference justification, and the matched predicted step. The prompt then asks whether the predicted step fully resolves the condition, partially resolves it, or leaves it unresolved. The scoring rubric is discrete: 1.0 for fully resolved, 0.5 for partially resolved, and 0.0 for unresolved. This metric therefore measures whether the model uses user-specific evidence to instantiate the correct workflow branch, rather than merely restating the generic alternatives. A representative prompt template is shown in Figure N.4. N.7
Topology Ordering
The Topology Ordering metric evaluates whether the predicted workflow preserves the correct relative order among aligned steps. It is computed over the predicted-reference step pairs established by precision. For all pairs of matched steps, the evaluator checks whether their relative order in the predicted workflow is consistent with their relative order in the reference workflow using pairwise consistency verification. The final score is the fraction of concordant pairs. N.8 Personalized Comprehensiveness Prompt The Personalized Comprehensiveness metric evaluates whether predicted workflow steps preserve the user-specific and operational detail present in the reference workflow. It is only computed on predicted-reference pairs that have already passed precision, ensuring that the comparison is made between aligned steps rather than unrelated fragments. For each matched pair, the judge prompt compares the predicted task and task description against the reference task and expected answer, and scores four dimensions: specificity, personalization, completeness, and actionability. The prompt also asks
the judge to identify missing details. These outputs are then combined into a final step-level score. This metric is therefore stricter than a general quality judgment, since high scores are reserved for steps that retain concrete details, are clearly tailored to the user’s case, and remain directly actionable. A representative prompt template is shown in Figure N.5.
100 90.01
94.38 85.36 86.78 83.31
86.41 85.99
80
100
94.92
93.57
90.24 82.54
87.16 79.10
89.63 82.30
81.32
86.52
92.22 90.04
88.47
88.13 89.68
80 60
40
40
20
20
0
GPT-5.2
Claude-Opus-4.5 DRBA Original
DRBA Mixed
Gemini-3.1-pro
0
DeepSeek-v3.2
DRFA Original
GPT-5.2
Claude-Opus-4.5
DRFA Mixed
DRBA Original
(a) Factuality
86.00
85.32
91.74
89.43 83.13
82.03
DRBA Mixed
Gemini-3.1-pro DRFA Original
DeepSeek-v3.2 DRFA Mixed
(b) Topology 70
50
47.18 46.72 42.83
40
25.99
24.85 24.58
23.59
20 14.90
15.02 16.12
12.92
51.83 44.99
42.61
41.89
42.32
39.56 30.38
30
18.86
56.32 48.08
40
28.03
26.98
58.51
55.67
50
35.85
62.50
59.26
60
41.36
30
33.54
33.51 27.26
20
10
10
0
GPT-5.2
Claude-Opus-4.5 DRBA Original
DRBA Mixed
Gemini-3.1-pro
0
DeepSeek-v3.2
DRFA Original
GPT-5.2
DRFA Mixed
Claude-Opus-4.5 DRBA Original
(c) Cond. Res.
DRBA Mixed
Gemini-3.1-pro DRFA Original
DeepSeek-v3.2 DRFA Mixed
(d) Pers. Comp. 70
80 69.43
70 60.43
72.07 65.86
65.19 63.70
68.35
68.40
67.31
58.13
49.26
48.55
55.87 52.11
50
55.94 47.75
62.34
60
60.48
50
51.35
47.96
40
42.55 37.86
37.30
48.45
47.43
44.76
38.89
37.07
40
30.36
30
30
24.24
32.32
24.07
20
20
10
10 0
91.20 83.12
72.36
60
60
93.10
GPT-5.2
Claude-Opus-4.5 DRBA Original
DRBA Mixed
Gemini-3.1-pro
0
DeepSeek-v3.2
DRFA Original
GPT-5.2
DRFA Mixed
Claude-Opus-4.5 DRBA Original
(e) Recall
DRBA Mixed
Gemini-3.1-pro DRFA Original
DeepSeek-v3.2 DRFA Mixed
(f) Precision 70 63.90
60.70
60 54.17
50
53.67
53.19
51.09
45.37
44.52
44.36
55.10
52.05 40.99
40
38.34 34.04 30.18
30
30.42
20 10 0
GPT-5.2
Claude-Opus-4.5 DRBA Original
DRBA Mixed
Gemini-3.1-pro DRFA Original
DeepSeek-v3.2 DRFA Mixed
(g) F1
Figure H.2: Per-metric comparison of DRBA and DRFA on the original split (blue) versus the mixed split (red) across four backbone models. Lighter shades denote DRBA; darker shades denote DRFA.
Research Planning Example { "query": "What is the step-by-step workflow to obtain informed consent for a procedure when the patient has limited English proficiency?", "timestamp": "2026-04-09T20:01:54.944099", "session_id": "3817ca0f", "plan": { "research_investigation_areas": [ { "area_id": 1, "stage": "REQUIREMENT_COLLECTION", "research_focus": "INTERNAL (local + enterprise) policies, playbooks, and procedures related to informed consent for patients with limited English proficiency (LEP)", "information_needs": [ "Step-by-step informed consent procedures", "Language access and interpreter services requirements", "LEP patient identification and assessment protocols", "Required consent forms (translated versions, language addendums)", ... ], "knowledge_sources": [ "internal" ], "research_approach": "requirement_analysis", "search_terms": [ "informed consent procedure", "informed consent policy", "limited English proficiency", "LEP patient consent", "interpreter services policy", ... ], "business_rationale": "Find INTERNAL documents (local files + enterprise docs) containing comprehensive workflow steps (8-15 steps) for obtaining informed consent when language barriers exist, including decision trees for interpreter selection, documentation requirements, and compliance checkpoints", "importance_level": "critical" }, { "area_id": 2, "stage": "PERSONAL_EVIDENCE_COLLECTION", "research_focus": "User's emails, chats, and files about specific patient case, procedure details, and available resources", "information_needs": [ "Specific patient information (language spoken, literacy level)", "Procedure type and urgency (elective vs. emergency)", "Available interpreter resources (in-house, vendor contracts)", ... ], "knowledge_sources": [ "internal" ], "research_approach": "evidence_gathering", "search_terms": [ "patient consent", "interpreter request", "language services", "procedure consent", "translation request", ... ], "business_rationale": "Personal evidence to identify specific patient language, procedure type, available interpreter resources, timeline constraints, and departmental context to personalize workflow steps and resolve conditional branches", "importance_level": "critical" } ], "workflow_strategy": { "requirement_gathering_approach": "Single comprehensive search across local files and enterprise document repositories for informed consent policies, language access procedures, interpreter services protocols, and LEP patient care guidelines - prioritizing internal documents with numbered step-by-step workflows", "personal_evidence_strategy": "Single comprehensive search across user emails, chat messages, and local files to identify specific patient details (language, literacy), procedure information (type, urgency, risk level), available resources (interpreters, translated forms), and scheduling constraints", "flowchart_focus": "Spend majority of effort on building detailed 8-15 step workflows covering the complete consent journey from patient identification through final documentation, with emphasis on interpreter coordination, comprehension verification, and regulatory compliance", "generic_workflow_coverage": [ "Step 1: Identify patient's preferred language and LEP status", "Step 2: Assess patient's English proficiency level and literacy", "Step 3: Determine procedure urgency (emergency vs. elective)", "Step 4: Select appropriate interpreter modality (in-person/phone/video)", "Step 5: Arrange qualified medical interpreter services", ... "Conditional branches: Emergency override procedures, family interpreter refusal, patient refusal of interpreter, comprehension failure escalation, unavailable language services, minor/incapacitated patient scenarios" ], "personal_evidence_needs": [ "Entities: Patient name, provider name, interpreter service vendor, facility name, department", "Language details: Specific language spoken, dialect, literacy level, preferred interpreter gender", "Procedure details: Procedure name, urgency level, risk classification, scheduled date/time", "Resource availability: In-house interpreter availability, vendor contact info, translated form inventory", "Timeline constraints: Procedure scheduling, interpreter availability windows, patient availability", "Special circumstances: Patient preferences, family involvement, previous consent issues, accessibility needs" ] } }, "metadata": { "model": "openrouter/anthropic/claude-opus-4.5", "use_research_plan": true, "use_adaptive_actions": true } }
Figure J.1: Example of a research plan produced by DRFA.
Action Planning Example { "id": "1_action_0", "type": "enterprise_api", "description": "Search for IEP development procedures and step-by-step workflows including eligibility determination, evaluation processes, and meeting scheduling requirements for special education services", "parameters": { "query": "IEP development procedures eligibility evaluation meeting workflow special education", "top_k": 15 }, "status": "completed", "priority": 0.95, "dependencies": [], "expected_output": "Internal policy documents, procedural guides, or playbooks detailing the sequential steps for creating an IEP including eligibility criteria, evaluation timelines, and meeting coordination procedures", "actual_output": { "tool": "enhanced_enterprise_api", "query": "IEP development procedures eligibility evaluation meeting workflow special education", "success": true, "data_retrieved": true, "services_queried": 4, "successful_services": 4, "processed_files": [ "/var/folders/lc/sthhvxj922s4j6sp9qmvjxw80000gn/T/tmp0bpi6wg4.pdf", "/var/folders/lc/sthhvxj922s4j6sp9qmvjxw80000gn/T/tmpb5ga1mbj.pdf", "/var/folders/lc/sthhvxj922s4j6sp9qmvjxw80000gn/T/tmp6kntdke2.docx", "/var/folders/lc/sthhvxj922s4j6sp9qmvjxw80000gn/T/tmp19il5i8o.docx", "/var/folders/lc/sthhvxj922s4j6sp9qmvjxw80000gn/T/tmpgmic_ba1.pdf", ... ], "files_processed": 20, "search_terms": [ "iep", "development", "procedures", "eligibility", "evaluation", "meeting", "workflow", "special", "education" ], "stored_in_vector": true, "content_stored_in_vector": 35, "results": [ { "service": "nextcloud", "success": true, "items_found": 18, "results": [ { "path": "/remote.php/dav/files/jordan.alvarez/shared/enrollment-procedures-guide.pdf", "name": "enrollment-procedures-guide.pdf", "type": "file", "size": 26579, "modified": "Wed, 08 Apr 2026 17:06:47 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/pinecrest-special-education-intake-service.pdf", "name": "pinecrest-special-education-intake-service.pdf", "type": "file", "size": 26125, "modified": "Wed, 08 Apr 2026 17:07:15 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/pinecrest-special-education-onboarding-procedures.docx", "name": "pinecrest-special-education-onboarding-procedures.docx", "type": "file", "size": 37875, "modified": "Wed, 08 Apr 2026 17:06:36 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/special-ed-intake-compliance-roadmap.pdf", "name": "special-ed-intake-compliance-roadmap.pdf", "type": "file", "size": 28301, "modified": "Wed, 08 Apr 2026 17:06:19 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/special-ed-intake-planning-handbook.pdf", "name": "special-ed-intake-planning-handbook.pdf", "type": "file", "size": 27452, "modified": "Wed, 08 Apr 2026 17:05:38 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/enrollment-procedures-guide.pdf", "name": "enrollment-procedures-guide.pdf", "type": "file", "size": 26579, "modified": "Wed, 08 Apr 2026 17:06:47 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "path": "/remote.php/dav/files/jordan.alvarez/shared/pinecrest-special-education-intake-service.pdf", "name": "pinecrest-special-education-intake-service.pdf", "type": "file", "size": 26125, "modified": "Wed, 08 Apr 2026 17:07:15 GMT", "relevance_reason": "Filename matches: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']" }, { "post_id": "acojrhg5o3gamdhbmceoxtujpy", "message": "Got it. Teacher of record is Ms. Patel (ELA) for advisory; I’ll ask her to complete the teacher input form and I’ll pull current grades/attendance from SIS by EOD tomorrow. Anything specific you want included for the IEP packet?", "user_id": "6c9f3xd5z381umdmxfcrzdzq8h", "channel_id": "kycghgpsx3yozk13fuaz8z9ibh", "timestamp": 1765361100000, "type": "message", "relevance_reason": "Message contains: ['iep', 'development', 'procedures', 'eligibility', 'evaluation', 'meeting', 'workflow', 'special', 'education']", "user_name": "jordan.alvarez", "team_name": "Student Support Services", "channel_name": "Student Support Coordination" }, } ], "mattermost_posts_stored": 9, "doc_id": "action_1_action_0_result_1" }, { "service": "filebrowser", "success": true, "items_found": 11, "doc_id": "action_1_action_0_result_2" }, { "service": "email_imap", "success": true, "items_found": 6, ], "summary": "enhanced_enterprise_api: Success - Retrieved 4 items", "metadata": { "tool_used": "enhanced_enterprise_api", "query_context": "IEP development procedures eligibility evaluation meeting workflow special education", "timestamp": "2026-04-09T19:24:19.388670", "workflow_stage": "REQUIREMENT_COLLECTION", "action_description": "Search for IEP development procedures and step-by-step workflows including eligibility determination, evaluation processes, and meeting scheduling requirements for special education services", "action_type": "enterprise_api", "category": "enterprise_api" }, "doc_id": "enhanced_enterprise_api_63c37b35", "workflow_stage": "REQUIREMENT_COLLECTION", "description": "Search for IEP development procedures and step-by-step workflows including eligibility determination, evaluation processes, and meeting scheduling requirements for special education services" }, "error": null, "execution_time": 17.23775, "completion_time": "2026-04-09T19:24:19.389543", "iteration_completed": 0, "created_from_research_step": "1", "preferred_tool": "EnterpriseAPITool", "score": 1.8524999999999998, "stage": "REQUIREMENT_COLLECTION" },
Figure J.2: Example of an action plan produced by DRFA. The full planning is not shown for brevity.
Adaptive (Conditional) Action Planning Example { "id": "cond_evidence_cond_2_2", "type": "enterprise_api", "description": "Search interpreter service requests or scheduling logs for this patient to determine which pathway was previously used", "parameters": { "query": "interpreter request video remote VRI phone language line patient name appointment date", "service": "auto" }, "status": "completed", "priority": 0.88, "dependencies": [], "expected_output": "Prior interpreter service records showing whether patient used spoken language interpreter (LEP) or ASL/CART services (Accessibility)", "actual_output": { "tool": "enhanced_enterprise_api", "query": "VRI cart iPad location floor unit available checked out biomedical equipment tracking", "success": true, "data_retrieved": true, "services_queried": 4, "successful_services": 2, "processed_files": [], "files_processed": 0, "search_terms": [ "vri", "cart", "ipad", "location", "floor", "unit", "available", "checked", "out", "biomedical", "equipment", "tracking" ], "stored_in_vector": true, "content_stored_in_vector": 9, "results": [ { "service": "mattermost", "success": true, "items_found": 6, "results": [ { "post_id": "gtt1gsnzi38ybcad1x9axnciso", "message": "End-of-day note: Room 3 will be down for 45 minutes after last case for floor waxing. EVS asked we keep all equipment off the perimeter. Please plan instrument drops to SPD by 4:15 so we don’t bottleneck the corridor.", "user_id": "a33f8ppjh3ymdp9pr5x4pzdzch", "channel_id": "opjcwqyqt3gh3xamm65ka8dwoh", "timestamp": 1760921400000, "type": "message", "relevance_reason": "Message contains: ['vri', 'cart', 'ipad', 'location', 'floor', 'unit', 'available', 'checked', 'out', 'biomedical', 'equipment', 'tracking']", "user_name": "rachel.nguyen", "team_name": "Sunrise Surgical Center", "channel_name": "OR Schedule & Flow" }, ], "mattermost_posts_stored": 6, "doc_id": "action_cond_evidence_cond_2_2_result_0" }, { "service": "email_imap", "success": true, "items_found": 3, "results": [ { "id": "3", "type": "email", "subject": "b'Facility walkthrough next week \\xe2\\x80\\x94 signage refresh + patient wayfinding'", "from": "Luis Ramirez <[email protected]>", "to": "[email protected], [email protected]", "date": "Fri, 20 Feb 2026 08:41:59 ", "preview": "Good morning,\r\n\r\nFacilities is planning a short walkthrough next Wednesday (2/26) at 7:15 AM to confirm the scope for our signage refresh and patient wayfinding updates.\r\n\r\nItems we’re evaluating: - Exterior directional signs (parking to main entrance) - Check-in and pre-op hallway wayfinding\r\n- Updated room placards (Room 1/2/3, PACU, Staff Only) - Replacing worn floor decals near the waiting area Goal is to reduce “wrong hallway” traffic during the 7–9 AM peak and ensure the layout is co", "path": "INBOX/3", "has_attachments": false }, ], "emails_stored": 3, "doc_id": "action_cond_evidence_cond_2_2_result_1" } ], "summary": "enhanced_enterprise_api: Success - Retrieved 2 items", "metadata": { "tool_used": "enhanced_enterprise_api", "query_context": "VRI cart iPad location floor unit available checked out biomedical equipment tracking", "timestamp": "2026-04-09T20:08:49.271899", "workflow_stage": "PERSONAL_EVIDENCE_COLLECTION", "action_description": "Search for VRI equipment availability and location status for the unit or clinic where the consent will be obtained", "action_type": "enterprise_api", "category": "enterprise_api" }, "doc_id": "enhanced_enterprise_api_5bcdaaf0", "workflow_stage": "PERSONAL_EVIDENCE_COLLECTION", "description": "Search for VRI equipment availability and location status for the unit or clinic where the consent will be obtained" }, "error": null, "execution_time": 1.409132, "completion_time": "2026-04-09T20:08:49.272013", "iteration_completed": 9, "created_from_research_step": "condition_resolution", "preferred_tool": "EnterpriseAPITool", "score": 1.1808, "stage": "PERSONAL_EVIDENCE_COLLECTION" },
Figure J.3: Example of an adaptive conditional action plan produced by DRFA. The full planning is not shown for brevity.
Research Planning Prompt Design a comprehensive research plan to answer: "{question}" {tools_section} As a senior enterprise workflow architect, design a research plan that will: 1. Create a COMPREHENSIVE research plan covering both generic workflow and user evidence. 2. Drive action planning so it properly covers ALL items in the generic workflow (policy steps, conditional branches, prerequisites). 3. Drive action planning to collect necessary user/personal evidence to resolve conditional branches where possible. 4. Produce both a generic workflow and a personalized workflow, each with detailed task descriptions under every to-do; conditional branches resolved by user evidence must be presented as resolved in the personalized workflow. Your plan MUST follow these 4 stages: ### STAGE 1: REQUIREMENT_COLLECTION (CRITICAL - EXTRACT FULL WORKFLOWS) **Goal**: Find INTERNAL policy documents that describe STEP-BY-STEP procedures **INTERNAL SOURCES (both are high priority):** - LOCAL: Use LocalFileSearchTool for uploaded documents, policies, playbooks - ENTERPRISE: Use EnterpriseAPITool for enterprise document repositories **Search Strategy:** - Search LOCAL for playbooks, manuals, guides that contain numbered steps - Search ENTERPRISE for incident response procedures with detailed workflows - Search for compliance procedures with decision points - Look for documents with conditional logic (if X then Y, else Z) - PRIORITIZE INTERNAL (local + enterprise) over external web searches - AVOID external web search unless internal documents are insufficient **What to extract**: - Every step in the workflow (aim for 8-15 steps) - Prerequisites between steps - Conditional branches at each step - Required forms, systems, approvals at each step ### STAGE 2: GENERAL_FLOWCHART_DESIGN - Map ALL steps from collected policies into a comprehensive workflow - Preserve ALL conditional branches - do not resolve them - Include prerequisites and dependencies between steps - The workflow should be generic (applies to ANY user) ### STAGE 3: PERSONAL_EVIDENCE_COLLECTION (CRITICAL - RESOLVE CONDITIONS) **Goal**: Find user-specific facts that can resolve conditional branches - Search user's EMAILS for incident details, vendor communications, timelines - Search user's CHAT messages for discussions about the situation - Search user's LOCAL files for relevant documentation - Search enterprise files for user-specific context **What to extract**: - Specific entities (vendor names, system names, people involved) - Dates and timeframes - Data types affected - Counts and measurements - Constraints or special circumstances (e.g., "survey window", "no downtime") ### STAGE 4: FLOWCHART_PERSONALIZATION - For each conditional step, check if personal evidence resolves it - Personalize task descriptions with user's specific facts - Keep conditions unresolved when no evidence determines the path - Cite both policy sources AND personal evidence sources Generate a JSON object with EXACTLY 2 research investigation areas (1 per stage). **STRICT LIMITS - FOCUS ON FLOWCHART BUILDING**: - REQUIREMENT_COLLECTION: EXACTLY 1 investigation area (comprehensive search) - PERSONAL_EVIDENCE_COLLECTION: EXACTLY 1 investigation area (comprehensive search) - This minimal research allows MORE TIME for building detailed workflows {{ "research_investigation_areas": [ {{ "area_id": 1, "stage": "REQUIREMENT_COLLECTION", "research_focus": "INTERNAL (local + enterprise) policies, playbooks, and procedures related to [topic from question]", "information_needs": ["Step-by-step procedures", "Decision points", "Required forms/approvals"], "knowledge_sources": ["internal"], "research_approach": "requirement_analysis", "search_terms": ["[main topic] procedure", "[main topic] playbook", "[main topic] policy"], "business_rationale": "Find INTERNAL documents (local files + enterprise docs) with workflow steps (8-15 steps)", "importance_level": "critical" }}, {{ "area_id": 2, "stage": "PERSONAL_EVIDENCE_COLLECTION", "research_focus": "User's emails, chats, and files about the specific situation", "information_needs": ["Incident details", "Vendor/entity names", "Dates", "Constraints"], "knowledge_sources": ["internal"], "research_approach": "evidence_gathering", "search_terms": ["[entities from question]", "[dates from question]"], "business_rationale": "Personal evidence to personalize and resolve workflow conditions", "importance_level": "critical" }} ], "workflow_strategy": {{ "requirement_gathering_approach": "Single comprehensive search for procedure documents", "personal_evidence_strategy": "Single comprehensive search for user context", "flowchart_focus": "Spend majority of effort on building detailed 8-15 step workflows", "generic_workflow_coverage": ["Extract every policy step (8-15)", "Map all conditional branches", "Capture prerequisites between steps", "Required forms/systems/approvals"], "personal_evidence_needs": ["Entities (vendors, systems, people)", "Dates and timelines", "Data types and encryption/access facts", "Constraints and approver availability"] }} }} Keep it simple: 1 policy search + 1 personal evidence search = focus on workflow building.
Figure J.4: Research planning prompt used by the DRFA agent to construct generic and personalized workflows.
Intermediate Workflow Generation Prompt You are analyzing research findings (policy documents, workflow procedures) for THIS specific research question. Extract only workflow and conditions that are RELEVANT to this question. RESEARCH QUESTION: {research_query}
Action Planning Prompt Generate 2-3 diverse, targeted actions for this research investigation area. ORIGINAL DR QUESTION: {original_question} CURRENT STAGE: {stage} Research Focus: {research_focus} Information Needs: {information_needs} Knowledge Sources: {knowledge_sources} Available Tools: {available_tool_names} Tool Selection Guidelines: {tool_guidelines} {_get_action_plan_guidelines(available_tool_names)} IMPORTANT: Each action must use a genuinely different search query, not a rephrasing of the same terms. Think about different aspects, sub-topics, document types, or terminology that would retrieve complementary results. Return a JSON array with 2-3 actions.
Figure J.5: Action planning prompt used by DRFA agent.
FINDINGS (policy/workflow content): {findings_summary} TASK: A) EXTRACT THE GENERIC WORKFLOW that is directly relevant to the research question above. List the main steps (e.g. step 1: "Create incident record", step 2: "Complete Gate 1", ...). For each step set is_conditional: true only if that step has explicit decision branches (if/then/else, or "if X then path A, else path B") in the text. B) For each step where is_conditional is true, add ONE entry to "conditional_steps" with: - condition_id: short id based on THAT step (e.g. "consent_waiver_vs_written") - step_number: the step index - description: the exact decision from the workflow (e.g. "Whether consent can be waived or written consent is required") - branches: the actual branch options from the document (e.g. ["waiver applicable", "written consent required"]) - evidence_needed: what user/personal fact would resolve it (e.g. "study design, vulnerable population, data category") - search_queries: 2-3 diverse search phrases to find that evidence in user emails/chats/files. Each query must be meaningfully different from the others - not rephrasings. Each query across ALL conditional_steps must also be distinct. IMPORTANT: conditional_steps must ONLY include conditions that appear in the extracted workflow for THIS research question. Do not add generic conditions (e.g. "vendor vs internal", "encrypted vs not") unless they actually appear in the findings for this question. Return a single JSON object only, no other text. Example: {{ "generic_workflow_steps": [ {{ "step": 1, "title": "Create project record", "is_conditional": false }}, {{ "step": 2, "title": "Determine consent approach", "is_conditional": true }} ], "conditional_steps": [ {{ "condition_id": "consent_waiver_vs_written", "step_number": 2, "description": "Whether consent can be waived or written consent is required", "branches": ["waiver applicable", "written consent required"], "evidence_needed": "Study design, data category, vulnerable population", "search_queries": ["relevant term from workflow", "evidence search phrase", "another distinct angle"] }} ] }} If the workflow has no conditional steps, set "conditional_steps": [].
Figure J.7: Prompt for intermediate workflow generation.
Adaptive Action Planning Prompt Based on the research progress, suggest 0-2 additional actions ONLY if genuine gaps exist. Original Research Query: {action_plan.research_query} Completed Actions: {len(completed_actions)} Current Action Plan Status: {json.dumps(action_plan.get_stats(), indent=2)} SOURCE ANALYSIS: - Policy Documents Found: {internal_findings['has_internal']} - Personal Evidence Found: {internal_findings['has_enterprise']} ALREADY-USED SEARCH QUERIES (DO NOT repeat or rephrase these): {json.dumps(existing_queries[:15], indent=2)} EXISTING ACTION DESCRIPTIONS: {chr(10).join(existing_descriptions[:15])} **CRITICAL DIVERSITY RULES:** 1. Do NOT suggest queries that overlap >50% in keywords with any already-used query above. 2. Each new action MUST explore a DIFFERENT evidence dimension (e.g. people, dates, specific systems, regulatory bodies, incident IDs) - not just rephrase the topic. 3. If REQUIREMENT_COLLECTION already has 7+ actions, do NOT add more for that stage. 4. If PERSONAL_EVIDENCE_COLLECTION already has 7+ actions, do NOT add more for that stage. 5. If both policy documents AND personal evidence are found, return []. Available Tools: {available_tool_names} Return a JSON array (0-2 actions). Return [] if no genuine gaps exist: [ {{ "type": "local_document_search", "description": "A genuinely different search angle not covered above", "parameters": {{"query": "novel keywords not used before", "top_k": 15}}, "priority": 0.85, "expected_output": "New information not already retrieved", "preferred_tool": "LocalFileSearchTool", "stage": "PERSONAL_EVIDENCE_COLLECTION" }} ] Create valid JSON only, no other text.
Figure J.6: Adaptive planning prompt used by DRFA agent.
Conditional Action Planning Prompt Generate 3-6 diverse search actions to find PERSONAL/USER-SPECIFIC evidence that can resolve the conditional steps in a workflow. The goal is personalization: find facts about the user's specific situation so that each condition can be resolved to a concrete branch. RESEARCH QUESTION: {context.original_question} CONDITIONAL STEPS TO RESOLVE: {conditions_summary} ALREADY-USED QUERIES (do NOT repeat or closely rephrase these): {json.dumps(existing_queries[:15], indent=2)} Available Tools: {available_tool_names} Tool Guidelines: {tool_guidelines} REQUIREMENTS: - Each action should target a DIFFERENT conditional step OR a different angle of the same step. - Vary the tools used (mix local document search, enterprise API, etc.) when multiple are available. - Search for USER-SPECIFIC evidence (emails, chats, user files, records) - NOT general policy documents. - Queries should be concrete and specific - terms that would appear in real user communications. - Do NOT just rephrase the condition description as a query. Think about what actual user data would contain the answer (e.g., specific correspondence, filed forms, meeting notes, approvals). Return a JSON array of 3-6 actions: [ {{ "type": "enterprise_api" or "local_document_search", "description": "What this searches for and which condition it resolves", "parameters": {{"query": "specific search terms", "service": "auto"}} or {{"query": "...", "top_k": 10}}, "priority": 0.8-0.95, "expected_output": "What user-specific evidence this should find", "preferred_tool": "EnterpriseAPITool" or "LocalFileSearchTool" }} ] Return valid JSON only, no other text.
Figure J.8: Conditional action planning prompt used by the DRFA agent.
Workflow Prediction Prompt You are an expert enterprise workflow architect. Generate a COMPREHENSIVE workflow report with **10-15** detailed, granular steps. DR Question: "{context.original_question}" User situation and persona: Infer from the DR question and from the retrieved documents (emails, chats, internal files) below. ### AVAILABLE SOURCES (use these IDs for citations): {doc_ids_block} ### DOCUMENT CONTENTS: {findings_text} ### SOURCE REFERENCE: {sources_reference} ### CITATION FORMAT: Use `[DOC:src_N]` format where src_N is a source ID from the list above (e.g., [DOC:src_1], [DOC:src_3]). Cite policy documents for workflow rules and personal documents for userspecific facts. **FACTUALITY REQUIREMENT**: In the Generic Workflow Steps and Personalized Workflow Steps (Section 1 and 2), cite ONLY src_N IDs that correspond to **internal sources** (policy documents, local files, emails, internal documents). Do NOT cite IDs that correspond to web or external search results in workflow step content. --# Research Report: Personalized Workflow Analysis ## Executive Summary Summarize the workflow analysis citing key policy and personal evidence sources [DOC:src_N]. ## 1. Generic Enterprise Workflow ### Overview Extract a COMPREHENSIVE workflow from the policy documents. You MUST include EVERY distinct procedural step - aim for **10-15 detailed steps** covering the COMPLETE process from start to finish. Each step should represent a SINGLE discrete action (e.g., "Complete intake form" is one step, "Run gate check" is another, "Configure settings" is another). Do NOT merge multiple actions into one step. ### Generic Workflow Steps ```json [ {{ "item": 1, "title": "Step title (e.g., 'Create X record', 'Complete intake form XYZ', 'Run gate check')", "task": "Brief task (e.g., 'Create a new onboarding workspace and attach handoff notes') [DOC:src_N]", "task_description": "DETAILED description (3-5 sentences) explaining exactly what to do, what specific fields to fill, what systems/tools/forms to use, what data to capture. Include ALL necessary details: form names or codes, system/ tool names, specific fields, required attachments, who needs to be involved. [DOC:src_N]", "subtask_description": "For ANY step that has a decision/branch/conditional in the policy (if X then A, else B): you MUST put the FULL condition text HERE with ALL branches explicitly listed (e.g., 'If X -> select Y; else select Z.'). For steps with no branch, use empty string.", "prerequisite": "Item number of prerequisite step (e.g., '1') or empty string if this is the first step or has no dependency", "prerequisite_description": "Why the prerequisite must be completed first (e.g., 'The workspace record must exist before the intake form can be submitted')" }}, {{ "item": 2, "title": "Next step title", ... }} // Continue for ALL steps (aim for 10-15 total) ] ``` ### GENERIC WORKFLOW REQUIREMENTS (CRITICAL): - Extract EVERY distinct step from ALL policy documents - do NOT combine or skip steps - Each step should be a SINGLE discrete action (creating a record, filling a form, running a check, configuring a setting, scheduling meeting, etc.) - Include the FULL end-to-end process relevant to the DR question - Keep descriptions DETAILED with specific form names, system names, field names, tool names mentioned in the policies - CRITICAL: For EVERY step that involves a decision or branch or conditional (if/then/else, tier-based, count-based, type-based), you MUST fill subtask_description with the COMPLETE condition text listing ALL branches. Do NOT leave subtask_description empty for conditional steps. - Look for conditions triggered by: tier/level selection, user counts/ thresholds, feature or integration requirements, package types, approval status, validation outcomes - Include prerequisites that show step dependencies (which step must complete before this one) - **TARGET: 10-15 steps** (minimum 10 unless policy truly has fewer discrete actions; maximum 15). ### Generic Flowchart ```mermaid graph TD A[Step 1] --> B[Step 2] B --> C{{Decision Point}} C -->|Condition X| D[Path X] C -->|Condition Y| E[Path Y] D --> F[Continue...] ```
## 2. Personalized Recommended Workflow ### User Situation Analysis Extract key facts from user's personal evidence [DOC:src_N]: - Specific entities mentioned (vendors, systems, people) - Dates and timeframes - Data types involved - Constraints or special circumstances **IMPORTANT**: Carefully search through ALL documents (especially emails, chats, and user-provided files) to extract user-specific facts. Look for: **Source Types to Search:** - **enterprise_email**: User's emails about the incident, vendor communications - **enterprise_chat**: Mattermost/Slack/Teams messages about the situation - **internal documents**: User's local files with specific details - **persona block**: User role, responsibilities, constraints **Facts to Extract** [DOC:src_N]: - Specific entities: vendor names, system names, people involved - Dates and timeframes: discovery date, incident timing, deadlines - Data details: types of data, encryption status, affected counts - Access facts: confirmed/denied access, subcontractor involvement - Constraints: operational limits, survey windows, downtime restrictions - User role: who they are, what they can approve **List ALL facts found in documents:** (Be explicit - list each fact found and cite the source document) ### Condition Resolution Table For EACH step in the Generic Workflow that has a non-empty subtask_description, create ONE row below. Use the EXACT condition text from that step. Search ALL user evidence documents to determine which branch applies. | Generic Condition (from subtask_description above) | User Evidence Found | Resolution | |----------------------------------------------------|---------------------|-----------| | [Copy EXACT condition from step N's subtask_description] | [Specific user fact with DOC:src_N] or "No evidence found" | Resolved: [specific path chosen] OR Unresolved: keep condition | **RESOLUTION RULES:** - Search ALL available documents (emails, chats, user files) for evidence that determines which branch applies - If user evidence clearly indicates which branch: mark "Resolved: [path]" and cite the evidence - If no evidence found or evidence is ambiguous: mark "Unresolved: keep condition" - You MUST have one row per conditional step - do not skip any ### Personalized Workflow Steps **MANDATORY CITATIONS**: Every step MUST include [DOC:src_N] in task, task_description, AND justification. Use policy source IDs for process steps and personal source IDs (emails, user files) for user-specific facts. No step may be left without citations. ```json [ {{ "item": 1, "title": "Personalized step title with user-specific details", "task": "PERSONALIZED task incorporating specific user facts - use actual names, numbers, systems from user evidence [DOC:src_N]", "task_description": "DETAILED personalized description using user's SPECIFIC facts. Replace generic terms with actual: names, system/tool names, counts, timeframes, tier or package names, deadlines, constraints from the evidence. Cite both policy and personal evidence. [DOC:src_N][DOC:src_M]", "subtask_description": "EMPTY string ('') if the Condition Resolution Table shows this condition as Resolved. KEEP the full conditional text if Unresolved.", "prerequisite": "Item number or empty", "prerequisite_description": "Same as generic or personalized if relevant", "justification": "Explain WHY this exact step is included in the personalized workflow for this user. State what policy requirement, user fact, or resolved condition makes the step necessary now; MUST end with [DOC:src_N] citations (policy and/or personal evidence)." }} // SAME NUMBER of steps as generic workflow. EVERY step: task, task_description, and justification MUST contain [DOC:src_N]. ] ```
### PERSONALIZATION RULES: 1. **SAME NUMBER OF STEPS** as generic workflow 2. **PERSONALIZE task/task_description** with user's specific facts (dates, names, numbers, entities) 3. **Conditional branches**: If a condition in the generic workflow is RESOLVED by user's personal evidence, present the personalized workflow accordingly: use the single resolved path in the step title/task and leave subtask_description EMPTY. If the condition is NOT resolved by evidence, keep the conditional text in subtask_description so the user sees both branches. 4. **subtask_description**: - EMPTY ("") when user evidence resolves which path to take (condition is Resolved in table) - KEEP conditional text when evidence does NOT determine the path (condition is Unresolved) 5. **justification**: - Explain why this step belongs in the personalized workflow, not just what to do - Reference user's SPECIFIC facts that determine the applicable path - If the step is unconditional, explain what policy or user circumstance still makes it necessary - MUST cite both policy and personal evidence using [DOC:src_N] ### CITATION REQUIREMENTS FOR PERSONALIZED SECTIONS (MANDATORY): - **Personalized Workflow Steps JSON**: Each of task, task_description, and justification MUST contain at least one [DOC:src_N]. Use valid source IDs from the AVAILABLE SOURCES list. - **Section 3 (Step-by-Step Personalized Guidance)**: Each step's **Description** MUST end with [DOC:src_N]. At least one **Action Item** per step MUST cite a source, e.g. "[ ] Action detail [DOC:src_N]".
### EXAMPLE TRANSFORMATIONS: **Generic step with condition (example):** - task: "Select the appropriate option based on tier and size" - subtask_description: "If tier is premium -> select option A; else select option B. If count exceeds threshold -> add additional pack; else keep default." **Personalized (condition RESOLVED):** - task: "Select option A and add the additional pack [DOC:src_1]" - subtask_description: "" (empty because evidence resolved both conditions) - justification: "Evidence shows premium tier and count above threshold [DOC:src_5], which triggers option A and the additional pack [DOC:src_1]." **Personalized (condition NOT RESOLVED):** - task: "Select option based on package type; determine which applies [DOC:src_1]" - subtask_description: "If package is standard -> path X. If premium -> path Y." - justification: "No evidence found specifying package type, so both branches must be preserved [DOC:src_1]." ### Personalized Flowchart ```mermaid graph TD A[Personalized Step 1] --> B[Personalized Step 2] B --> C[Step 3] C --> D{{Unresolved Decision if any}} ``` Show resolved paths as direct arrows. Keep decision diamonds only for UNRESOLVED conditions. ## 3. Step-by-Step Personalized Guidance For each step, provide. **You MUST add [DOC:src_N] citations** to every Description and to at least one Action Item per step (use valid source IDs from AVAILABLE SOURCES). ### Step N: [Title] **Description**: Detailed personalized guidance. End with citation(s), e.g. "...per portal requirements [DOC:src_1]." **Action Items**: - [ ] Specific action with details [DOC:src_N] - [ ] Additional actions as needed (cite where relevant) ## References (Leave blank - auto-generated) --CRITICAL REQUIREMENTS: 1. Generate **10-15 DETAILED steps** in generic workflow - extract EVERY distinct procedural step from policy documents. If you have fewer than 10, you are merging steps; if you have more than 15, consolidate only where actions are truly atomic together. 2. For each step that has an if/then/else or decision or conditional in the policy, you MUST put that COMPLETE condition text in subtask_description with ALL branches listed 3. Condition Resolution Table: one row per conditional step, search ALL user evidence to resolve conditions 4. Personalized workflow has IDENTICAL number of steps as generic (10-15) 5. Personalize task/task_description with USER'S SPECIFIC FACTS (actual names, counts, systems, dates, constraints from evidence) 6. subtask_description in personalized: EMPTY when resolved by user evidence, KEEP condition text when NOT resolved 7. Every justification must reference specific user facts with citations 8. **CITATIONS (MANDATORY)**: Use [DOC:src_N] format. In the PERSONALIZED WORKFLOW (Section 2): every step's task, task_description, and justification MUST contain at least one [DOC:src_N]. In Section 3 (Step-by-Step Guidance): every step's Description and at least one Action Item per step MUST contain [DOC:src_N]. Only use source IDs from the AVAILABLE SOURCES list (src_1, src_2, etc.).
Figure J.9: Workflow prediction prompt. DRF LOW first generates a generic conditional workflow, then predicts the personalized workflow.
Task Seed Generation Prompt You are an expert at creating Deep Research (DR) questions for business and domain-specific workflows. Your task is to generate exactly {n} Deep Research questions (with associated company and user metadata) for the domain: **{ domain}**. ### Guidelines for each DR question: 1. Each question should be **simple to understand** and clearly ask about **what workflow or steps need to be taken** to accomplish a goal or resolve an issue. 2. Questions should be realistic, actionable, and require multi-step research or analysis to answer. 3. Each question should be at most 10-15 words, in plain English, easy to understand for non-technical people and without any leading phrases (e.g., "including A, B, C") and end with a question mark. 4. Questions should focus on understanding a process, procedure, or sequence of actions - e.g., "What steps do I need to follow to ...?", "What is the workflow for ...?", "What exact steps should I take to ...?" 5. Questions should not be ambiguous and should lead to concrete, step-by-step answers. 6. Do not combine multiple processes or procedures (E.g., do not ask questions like"What workflow should we follow to evaluate a positive blood culture, choose empiric antibiotics, deescalate therapy, and document stewardship decisions?") into a single question. Keep it simple and focused. 7. The questions should be in the simple sentence. Do not make it complex and long. 8. Make questions diverse (different sub-areas or angles within the domain) rather than repetitive. 9. Make the questions from each domain unique, creative, and realistic. ### Guidelines for company and user metadata: - For each question, generate realistic company and user information that fits the domain and the question scenario. - The company and user should be contextually appropriate for the question being asked. - Company persona represents a point of contact at the company (e.g., support agent, advisor, manager). - User represents the person asking the deep research question. - Do not use special characters in user names, company names, or email addresses. - The deep research question should be relevant to the domain and the company and the user's role. - Try to create the questions in such a way that the user is querying about their own company's internal processes. So, in this case, the user_company fields should match the company fields. - Be creative with names, roles, and companies - make them feel realistic and varied.
### Examples of DR Questions (simple, workflow-focused): - Healthcare: What is the step-by-step workflow to obtain informed consent for a procedure for a patient with limited English proficiency? - Legal: What is the exact workflow I should follow to challenge my wrongful termination? - B2C: What steps do I follow to recover a hacked customer account? - B2B: What is the step-by-step process to diagnose and fix a broken order-to-invoice? - Education: What workflow should I follow to apply for academic probation appeal in my courses? ### Output Format Return a JSON object with a single key "dr_questions" whose value is a list of exactly {n} objects, each with the following keys: {{{{ "dr_questions": [ {{{{ "dr_question": "A simple, workflow-focused Deep Research question", "company_name": "Company Name", "company_industry": "Company Industry", "company_description": "A brief 1-2 sentences about the company.", "company_size": "Small/Medium/Large", "company_employee_count": "e.g., 100-500", "company_annual_revenue": "e.g., $50M-$100M", "company_persona": "Name of the company persona", "company_persona_email": "[email protected]", "company_persona_role": "Role of the persona", "company_persona_role_description": "1 sentence description of the persona's role", "user_name": "User Name", "user_role": "User Role", "user_email": "[email protected]", "user_role_description": "1 sentence description of the user's role", "user_company": "User's Company", "user_industry": "User's Industry", "user_company_description": "A brief 1-2 sentences about the user's company.", "user_company_size": "Small/Medium/Large", "user_company_employee_count": "e.g., 100-500", "user_company_annual_revenue": "e.g., $50M-$100M" }}}} ] }}}} Do not include any preamble or explanation - return only the JSON object.
Figure K.1: Prompt for task-seed generation for producing workflow-focused deep research questions together with company and user metadata.
Dual-Context Construction Prompt You are an expert Deep Research Workflow generator. Your task is to generate the personal context and generic context for a deep research task given the existing information below. ## Existing Task Information - **Domain:** {domain} - **DR Question:** {dr_question} - **Company Name:** {company_name} - **Company Industry:** {company_industry} - **Company Description:** {company_description} - **Company Size:** {company_size} - **Company Employee Count:** {company_employee_count} - **Company Annual Revenue:** {company_annual_revenue} - **Company Persona:** {company_persona} - **Company Persona Email:** {company_persona_email} - **Company Persona Role:** {company_persona_role} - **Company Persona Role Description:** { company_persona_role_description} - **User Name:** {user_name} - **User Role:** {user_role} - **User Email:** {user_email} - **User Role Description:** {user_role_description} - **User Company:** {user_company} - **User Industry:** {user_industry} - **User Company Description:** {user_company_description} - **User Company Size:** {user_company_size} - **User Company Employee Count:** {user_company_employee_count} - **User Company Annual Revenue:** {user_company_annual_revenue} ## Your Task Generate the `generic_context` and the `personal_context` fields. These must be realistic, relevant, and consistent with the information provided above. ### Guidelines for Generic Context: - The generic context represents the company's internal process/ workflow/policy that applies to all clients or employees in this domain. - The generic context should have multiple constraints, similar to a flowchart. - Keep the situation and language easy to understand. Do not make every step conditional. Keep many steps linear and sequential. - The generic context should not be too long. The workflow should have 12-15 steps. Among them 6-8 steps should be conditional, the rest of the steps should not contain any conditional statements. - Use words like 'Then', 'After that', 'Then do this', 'Then do that ', etc. to describe the sequence of steps (except for the first step). - It should be self-contained: all the information needed to generate the solution workflow for the user query must be present. - It should include all the necessary information (e.g., form information, details). Although retrieving the action items should require multi-step reasoning. An LLM should not be able to retrieve all the information in a single linear pass. - You should add intermediate information (e.g., forms and others) to ensure multi-step reasoning. - Do not use complicated language. The language should be clear and simple. - Keep each steps short, concise, simple, and easy to understand. - You should add **multi-hop conditions** for the answer-generator model to think through several layers of reasoning. - You can add several conditional statements, e.g., "If X, then do Y; else do Z" to increase complexity. E.g., if you have X, go to see rule 1.2. Then later, if you see 1.2, submit form Z; if you see 1.3, submit Q. Focus on adding multi-hop conditions that can be **subtle** and require critical reasoning. - Add different situational details, environmental factors, or hypothetical scenarios in the context that require the model to think critically. For example, adding different legal
regulations, forms, multi-step procedures can increase complexity. - Do not directly add the action items in the context. Add them subtly and through multi-step conditioning to make the model work hard on retrieving them. - Try to be creative, professional, and realistic. - For conditional steps, clearly define the conditions and the actions to be taken for each condition. Do not use complicated language for conditional steps. - For conditional steps, do not use the same condition for multiple actions. - Remember, for a conditional step, do not add more than 2-3 conditions. - The generic context MUST be relevant to the company ({company_name }, {company_industry}) and the DR question. ### Guidelines for Personal Context: - The personal context should be relevant to the user's query, user's role ({user_role}), and user's company ({user_company}). - You should add information in such a way that it follows the workflow flowchart of the generic context. - In case there are conditional steps in the generic context, the personal context should satisfy only one of the conditions. - Do not add information that is not related to the generic context. You do not need to introduce yourself (e.g., "I am a...") in the personal context. Also, do not mention the user's name in the personal context. - Create a personal context that is easy to understand and follows the workflow flowchart of the generic context. - Create the personal context in a bullet point format resembling the generic context so that it is easy to understand and follow. - Keep each point short, concise, simple, and easy to understand. - The personal context MUST be consistent with the user's role ({ user_role}), company ({user_company}), and the DR question. Return the output as a JSON object with the following structure: {{ "generic_context": "Generic context (company's internal process/ workflow/policy applicable to all clients/employees)", "personal_context": "Personal context (user's specific situation following the generic context workflow)" }} Below is an example of generic context and personal context for the legal domain. - REMEMBER be creative, realistic, and professional. - Adhere to the domain-specific task creation. - Make the contexts relevant to the existing task information above. - Do not use complicated language. The language should be clear and simple. - Do not exactly follow the wording or the theme of the example, but follow the structure and the format. ### Example: Given information: - **Domain:** Legal - **DR Question:** "I want to know if I have legal grounds to challenge my termination and what EXACT steps I should take next. Tell me what documents to prepare, whom to contact, and what procedures I must follow according to your law firm's policies." - **Company Name:** GateBridge Employment Law - **Company Industry:** Legal (Employment Law) - **Company Persona Role:** Senior Case Intake Coordinator Generated output: {{{{ "generic_context": {COMPANY_INFO_GENERIC_CONTEXT}, "personal_context": {USER_INFO_GENERIC_CONTEXT} }}}}
Figure K.2: Prompt template for constructing the generic company context and the personal user context.
Company Insights for Workflow Generation Prompt User Support-Evidence Generation Prompt You are an expert workflow process analyst. Your task is to analyze the generic context and documentation of a company and create a detailed list of to-do steps (insights) covering that context. Company Name: {company_name} Industry: {industry} Generic Context: {generic_context} Please create a list of actionable steps. For each step, provide: 1. An item index. 2. A concise title. 3. The specific task to be performed. 4. A detailed low-level description of how to perform the task (e.g., fill out name, address of form X, ignore section B, etc). Be concise and do not itemize the task description. 5. Any subtask descriptions or conditional branches (e.g., if P happens submit A, if Q happens go to B, etc). If there are no conditional branches, leave this field empty. 6. Which item is a prerequisite for this item. If none, keep empty. 7. A detailed low-level description of the prerequisite (e.g., user needs to collect the confirmation number from page X, user needs to submit form Y first before doing this item). If none, keep empty. 8. For determining the prerequisite, you may look for the clues like 'after that', 'then', etc. This implies that the previous item is a prerequisite for the current item. Remember, - The steps should be in the order of the generic context. - Number of steps should be equal to the number of steps in the generic context. - You do not need to create separate steps for the subtasks or conditional branches of the generic context. Return the output as a JSON object with a 'steps' key containing an array of items following the format specified.
Figure K.3: Prompt for company workflow insight generation.
You are an expert QA analyst. Your task is to analyze the user's personal context and their Deep Research (DR) Question, and create a list of {n_supports} question-answering-justification pairs. Each pair must follow a question-answer-justification format. DR Question: {dr_question} User Context: {context} Number of Question-Answering Pairs: {n_supports} Requirements: 1. The 'answer' for each pair MUST be extracted from the context and must be an exact statement or event directly present in the given User Context. Do not summarize or rephrase the answer; it must be a verbatim extraction. 2. The 'question' should be formulated such that the extracted ' answer' provides a direct response to it. 3. The 'justification' should explain why this event is relevant to the user's situation or the DR question. You may synthesize this justification, but keep it realistic and professional. 4. You must cover the entire context within the {n_supports} question-answering-justification pairs. Do not miss any single sentence from the context. 5. You must generate exactly {n_supports} question-answeringjustification pairs. 6. If necessary, you can merge multiple questions to cover the entire context within the {n_supports} question-answeringjustification pairs. 7. Remember, You MUST COVER every points/information of the context within the {n_supports} question-answering-justification pairs . Do not miss any single point/information from the context. Return the output as a JSON object with an 'items' key containing an array of events following the format specified.
Figure K.5: Prompt for user support-evidence generation.
Company Distractor Generation Prompt You are an expert process analyst. Your task is to analyze the description of a company and its Deep Research (DR) Question, and create a list of distractors. These distractors should be plausible to-do steps that are NOT relevant to addressing the DR Question. Company Name: {company_name} Industry: {industry} Company Description: {description} DR Question: {dr_question} DISTRACTOR Requirements: - Generate {n_distractors} to-do steps that are plausible for this company and industry but DO NOT help address the DR Question. - The steps should be about the company's general operations, metrics, or business processes but tangential to the core focus of the DR Question. - Focus on business areas unrelated to the DR Question (e.g., if the DR Question is about finance, generate steps about HR, IT infrastructure, facility maintenance, or compliance training). - Each step should follow the order of business logic but remain irrelevant to the research goal. For each distractor step, provide: 1. An item index. 2. A concise title. 3. The specific task to be performed. 4. A detailed low-level description of the task. 5. Any subtask descriptions or conditional branches. If none, keep empty. Return the output as a JSON object with a 'steps' key containing an array of items following the format specified.
Figure K.4: Prompt for company distractor generation.
User Distractor Generation Prompt You are an expert QA analyst. Your task is to analyze the user's personal context and their Deep Research (DR) Question, and create {n_distractors} distractor events that are NOT relevant to answering the DR Question or the user's personal context. DR Question: {dr_question} User Context: {context} Requirements: - These events should be plausible but MUST NOT be relevant to the DR Question or the user's personal context. - The events should be about the user's workplace general operations, metrics, business processes but irrelevant to the DR Question or the user's personal context. - Focus on the areas unrelated to the DR Question (e.g., if the DR Question is about finance, generate steps about other things such as user's personal business, HR, IT infrastructure, facility maintenance, or compliance training). Return the output as a JSON object with an 'items' key containing an array of events following the format specified.
Figure K.6: Prompt for user distractor-event generation.
Personalized Workflow Generation Prompt You are an expert personalized workflow generator. Your task is to take a deep research question (DR Question), generic company workflow (Generic Insights) and a specific user's facts (User Context), and generate a personalized, resolved workflow for that user.
,→ ,→
DR Question: {dr_question} Generic Insights (Company Workflow): {generic_insights} User Context (User Facts): {user_context} Requirements: 1. **Resolve Conditional Paths**: The Generic Insights often contain conditional branches
,→ (e.g., "If X, then do Y; else do Z"). Use the provided User Context to determine which ,→ path the user should take. If resolved, the 'subtask_description' will be empty. If ,→ information to resolve conditional path does not exist in the provided user insights, keep ,→ the 'subtask_description' as is.
2. **Personalize Instructions**: Adapt the 'task', 'task_description' to be specific to the user's situation where possible. For example, if there are multiple conditions and only one of them is applicable for the user, then you should modify the 'task' and 'task_description' to clearly define which path the user should take to make it more personalized. 3. **Justify Applicability**: For each step, provide a 'justification' that explains why this ,→ step is relevant to the user, citing specific facts from the User Insights. 4. **Maintain Structure**: Keep the same sequence and logical flow as the Generic Insights, ,→ but remove irrelevant conditional branches. 5. **Preserve Required Detail**: Every step must include a non-empty 'task_description' and ,→ non-empty 'justification'. Keep 'prerequisite' and 'prerequisite_description' populated ,→ whenever they exist in the Generic Insights unless you are intentionally updating them to ,→ a more specific user-resolved version. Only leave 'subtask_description' empty when the ,→ Generic Insight had no branch or the branch was fully resolved for the user. 6. **Format**: Return the output as a JSON object with a 'steps' key containing an array of ,→ UserWorkflowItem objects.
,→ ,→ ,→ ,→
User Context (User Facts): On Friday at 4:40 pm, Northbridge Health Partners (a 6-hospital network) received alerts that
,→ a third-party portal plugin (used for scheduling and lab result viewing) was ,→ misconfigured. The vendor says the issue was limited to 36 hours and involved an unindexed ,→ ``share link" feature that may have allowed anyone with the link to view a patient's lab ,→ PDF. Internal logs show 1,820 unique link generations, but only 214 link opens from ,→ external IPs; it's unclear if those opens were patients, family, or unknown parties. The ,→ PDFs include patient name, DOB, MRN, test name, and ordering provider; no SSNs or payment ,→ cards. The affected patients are from two states: Illinois and Wisconsin. The plugin is ,→ used only by the outpatient clinics, not inpatient. The vendor is a contracted business ,→ associate and the BAA is active, but the vendor initially emailed details from a personal ,→ Gmail account and has not provided a full incident report yet. The network is mid-Joint ,→ Commission survey window, so downtime is strongly discouraged; leadership wants ,→ containment without taking the portal fully offline. There is no evidence yet of ,→ ransomware, data exfiltration tools, or mass scraping. The incident was first discovered ,→ by the SOC (not by a patient complaint). This is the first potential breach in 18 months, ,→ and the organization's policy requires routing through the internal IR workflow. One ,→ condition is satisfied: this is a vendor/BA-facing web exposure with confirmed external ,→ access in logs, but no confirmed acquisition beyond link opens. ### Output: [
Return ONLY a valid JSON object. ### Example Workflow: DR Question: I'm the privacy/compliance lead at a multi-site hospital network and we just detected a potential PHI exposure involving a vendor-connected patient portal. I need a step-by-step plan to determine whether this is a reportable HIPAA breach, whether any state notification rules are triggered, and what EXACT documents, forms, internal approvals, and deadlines we must follow under our incident program-while keeping operations running and preserving evidence.
,→ ,→ ,→ ,→ ,→
Generic Insights (Company Workflow): [
]
{{ "item": 1, "title": "Open incident record in CIP", "task": "Create a new incident record in the Compliance Incident Portal (CIP).", "task_description": "Log into CIP and select the option to create/open a new incident ,→ record. Enter the initial incident details required to save the record (e.g., incident ,→ title/summary, date/time discovered, reporter/contact, impacted service/system, ,→ initial description). Save the record to generate the incident ID and enable ,→ downstream forms/workflow sections.", "subtask_description": "", "prerequisite": "", "prerequisite_description": "" }}, {{ "item": 2, "title": "Select Intake Track using Origin rule (Track V vs Track I)", "task": "Choose the correct CIP Intake Track based on whether a contracted vendor ,→ system/credential is involved.", "task_description": "In the incident record\u2019s intake/triage section, locate the field ,→ that sets the Intake Track (or equivalent workflow routing selector). Apply the Origin ,→ rule: determine whether the event touches (a) a contracted vendor system, integration, ,→ hosted environment, or (b) vendor credential (e.g., vendor user account, API key, SSO ,→ identity). Set Track V (Vendor-Connected) if vendor-touched; otherwise set Track I ,→ (Internal Systems). Save/confirm the track selection so CIP presents the correct ,→ downstream forms and required fields.", "subtask_description": "If the event touches a contracted vendor system or vendor ,→ credential, route to Track V (Vendor-Connected). Else, route to Track I (Internal ,→ Systems).", "prerequisite": 1.0, "prerequisite_description": "An incident record must exist in CIP (incident ID created) so ,→ the Intake Track can be set on that record and control which later forms appear." }}
]
{{ "item": 1, "title": "Open incident record in CIP", "task": "Create a new incident record in the Compliance Incident Portal (CIP) for the ,→ third-party portal plugin misconfiguration.", "task_description": "Log into CIP and create a new incident record for the third-party ,→ scheduling/lab-result portal plugin exposure discovered Friday at 4:40 pm by the SOC. ,→ Populate the minimum required fields to save: incident title (e.g., \u201cVendor ,→ portal plugin share-link misconfiguration\u2014possible lab PDF exposure\u201d), ,→ date/time discovered (Fri 4:40 pm), reporter (SOC), impacted system/service ,→ (outpatient portal plugin used for scheduling and lab result viewing), vendor/business ,→ associate name, and a short initial description (unindexed \u201cshare link\u201d ,→ feature potentially allowed link-based viewing of lab PDFs for ~36 hours; internal ,→ logs show 1,820 link generations and 214 external IP opens). Save to generate the CIP ,→ incident ID and enable the track-specific workflow sections.", "subtask_description": "", "prerequisite": "", "prerequisite_description": "", "justification": "An official record is required because this is a potential PHI exposure ,→ involving a contracted business associate portal plugin and the organization\u2019s ,→ policy requires routing through the internal IR workflow." }}, {{ "item": 2, "title": "Select Intake Track using Origin rule (Track V vs Track I)", "task": "Set Intake Track to Track V (Vendor-Connected) due to contracted vendor/BA plugin ,→ involvement.", "task_description": "In the incident intake/triage section of the CIP record, set the ,→ Intake Track to Track V (Vendor-Connected). Treat this as vendor-connected because the ,→ incident involves a contracted business associate\u2019s portal plugin and ,→ vendor-provided details/timeline will be required for investigation. Save/confirm ,→ Track V so CIP displays vendor-related fields and downstream forms.", "subtask_description": "", "prerequisite": 1.0, "prerequisite_description": "An incident record must exist in CIP (incident ID created) so ,→ the Intake Track can be set on that record and control which later forms appear.", "justification": "The event centers on a third-party portal plugin operated by a ,→ contracted business associate under an active BAA, which meets the vendor-touched ,→ criterion for Track V." }}
Figure K.7: Prompt for personalized workflow generation.
Company File Insight Section Generation Prompt Company File Outline Prompt You are an expert document designer creating realistic enterprise documentation. Given DR Question, the company high-level context and a set of business process insights, generate an outline for a professional business manual or report. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} Company High-Level Context: {company_high_level_context} DR Question: {dr_question} Document Structure Requirements: - Create a professional document outline with exactly {n_subsections} subsections - The document should be a realistic internal enterprise report or procedural manual/guidelines - Include a concise, professional file title appropriate for enterprise documentation Subsection Heading Requirements: - Subsection headings must follow the THEME of the DR Question but should NOT directly address the DR Question itself - Think of related business areas, adjacent topics, or supporting themes that would naturally appear in an enterprise document - Headings should sound professional and realistic for this industry and company size. - Each heading should be 3-8 words and use proper business terminology. - Exactly one subsection must be designated to contain the specific company insight (provided later) related to the DR Question. - The remaining subsections are thematic distractors that provide realistic business context. Introduction Requirements: - Write a professional 4-sentence maximum introduction paragraph setting the stage for this business documentation. Conclusion Requirements: - Write a professional 4-sentence maximum conclusion paragraph summarizing the importance of following these business guidelines. Return ONLY a valid Python dictionary with this exact structure: {{ "file_title": "Professional title for the document (8 words max) ", "introduction": "Professional introduction paragraph (4 sentences max)", "subsection_headings": ["List of exactly {n_subsections} professional subsection headings"], "conclusion": "Professional conclusion paragraph (4 sentences max)", "file_name": "the name of the file should be 3 words max and should be separated by dashes" }} IMPORTANT: - One subsection will later contain the actual company insight ( provided later) related to the DR Question - the others are realistic business distractors. - Do NOT include any preamble, explanation, or extra text-return only the Python dictionary.
Figure K.8: Prompt template for company-side file outline generation.
You are an expert, professional writer creating realistic enterprise content. Given DR Question, the company context and a specific business insight (Task and Descriptions) related to the DR Question, generate professional content for a manual or report subsection. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} DR Question: {dr_question} Target Insight: - Title: {title} - Task: {task} - Task Description: {task_description} - Subtask Description: {subtask_description} - Prerequisite Description: {prerequisite_description} Subsection Heading: {subsection_heading} Content Generation Requirements: - Generate realistic business content for the given subsection heading. - The content must contain exactly ONE paragraph of 4-8 sentences. - Content should be professional and authoritative. - You must mention about the prerequisites description, if provided to generate the content. - The paragraph must naturally incorporate the insight (Task and Descriptions) related to the DR Question but NOT copy it wordfor-word. - Paraphrase and embed the procedural steps and conditional branches (if any) within the business context. - Use industry-appropriate terminology and realistic business scenarios. - Make it sound like a standard operating procedure or a business report discussing specific workflows. Content Strategy: - Present the insight information as established procedures, findings, or requirements. - Embed the tasks and descriptions within broader operational context. - Embed the prerequisite description so that it is clear to the user that the task is dependent on the prerequisite. - Use natural business language to discuss the same information as in the insight. - Connect the information to business outcomes, efficiency, or compliance. - The content should include all key procedural details provided in the task and descriptions. Return ONLY a valid JSON object with this exact structure: {{ "subsection_heading": "{subsection_heading}", "content": "should be at least 3 sentences", "justification": "Short explanation of how this content provides information needed to answer the DR Question and the specific insight, and how it has embedded the prerequisite description if provided." }} IMPORTANT: - Do NOT copy the insight text directly - paraphrase and embed it naturally. - Ensure the paragraph reads like real business documentation, not a Q&A format. - Do NOT include any preamble, explanation, or extra text-return only the JSON object.
Figure K.9: Prompt template for company-side insightsection generation.
Company File Distractor Section Generation Company Distractor File Outline Prompt You are an expert document designer creating realistic enterprise documentation. Given DR Question, the company high-level context and a set of business process insights, generate an outline for a professional business manual or report. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} Company High-Level Context: {company_high_level_context} DR Question: {dr_question} Document Structure Requirements: - Create a professional document outline with exactly {n_subsections} subsections - The document should be a realistic internal enterprise report or procedural manual/guidelines - Include a concise, professional file title appropriate for enterprise documentation Subsection Heading Requirements: - Subsection headings must not be related to the DR Question itself. - Think of related business areas, adjacent topics, or supporting themes that would naturally appear in an enterprise document - Headings should sound professional and realistic for this industry and company size. - Each heading should be 3-8 words and use proper business terminology. - Exactly one subsection must be designated to contain the specific company insight (provided later) but not related to the DR Question. - The remaining subsections are thematic distractors that provide realistic business context. Introduction Requirements: - Write a professional 4-sentence maximum introduction paragraph setting the stage for this business documentation. Conclusion Requirements: - Write a professional 4-sentence maximum conclusion paragraph summarizing the importance of following these business guidelines. Return ONLY a valid Python dictionary with this exact structure: {{ "file_title": "Professional title for the document (8 words max) ", "introduction": "Professional introduction paragraph (4 sentences max)", "subsection_headings": ["List of exactly {n_subsections} professional subsection headings"], "conclusion": "Professional conclusion paragraph (4 sentences max)", "file_name": "the name of the file should be 3 words max and should be separated by dashes" }} IMPORTANT: - One subsection will later contain the actual company insight ( provided later) but not related to the DR Question. - Do NOT include any preamble, explanation, or extra text-return only the Python dictionary.
Figure K.10: Prompt template for company-side distractor outline generation.
Prompt You are an expert, professional writer creating realistic enterprise content. Given DR Question, the company context and a specific business insight (Task and Descriptions) unrelated to the DR Question, generate professional content for a manual or report subsection. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} DR Question: {dr_question} Target Insight: - Title: {title} - Task: {task} - Task Description: {task_description} - Subtask Description: {subtask_description} Subsection Heading: {subsection_heading} Content Generation Requirements: - Generate realistic business content specific to the company context but unrelated to the DR Question for the given subsection heading. - The content must contain exactly ONE paragraph of 4-5 sentences. - Content should be professional and authoritative. - The paragraph must not be related to the DR Question itself but should be related to the company context and the industry. - Use industry-appropriate terminology and realistic business scenarios. - Make it sound like a standard operating procedure or a business report discussing specific workflows. - The content should include all key procedural details provided in the task and descriptions. Content Strategy: - Focus on adjacent areas that don't directly impact the DR Question - Discuss historical context, general industry trends, or procedural information - Include operational details that are realistic but tangential - Reference related but non-essential metrics or activities - Avoid any content that would help someone answer the DR Question Justification Requirements: - Explain specifically why each paragraph's content doesn't help answer the DR Question - Identify what type of distractor strategy was used (e.g., "focuses on historical data vs current decision factors") - Keep justifications concise but clear (15 words maximum) Return ONLY a valid JSON object with this exact structure: {{ "subsection_heading": "{subsection_heading}", "content": "should be at least 3 sentences", "justification": "Short explanation (20 words max) of how this content covers the intended business insight" }} IMPORTANT: - Generate content for ALL provided subsection headings - Ensure content feels authentic for the given targetted insights. - Content must be realistic distractors - related but not helpful for the DR Question - Do NOT include any preamble, explanation, or extra text-return only the JSON array - Each paragraph should sound like professional business writing from this industry
Figure K.11: Prompt template for company-side distractor-section generation.
User File Insight Section Generation Prompt User File Outline Generation Prompt You are an expert business document designer creating realistic enterprise PDF reports. Given a Deep Research (DR) Question and company context, generate an outline for a professional business document that an employee would create based on their persona and role. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} Persona Context: {persona_context} DR Question: {dr_question} Document Structure Requirements: - Create a professional PDF document outline with exactly { n_subsections} subsections - The document should be something this persona would realistically create in their role - Include a concise, professional file title appropriate for enterprise documentation Subsection Heading Requirements: - Subsection headings must follow the THEME of the DR Question but should NOT directly address the DR Question itself - Think of related business areas, adjacent topics, or supporting themes that would naturally appear in an enterprise document - Headings should sound professional and realistic for this industry and company size - Each heading should be 3-8 words and use proper business terminology - Exactly one subsection must contain the actual insight addressing the DR Question - The remaining subsections are thematic distractors that provide realistic business context Introduction Requirements: - Write a professional 4-sentence maximum introduction paragraph - Should set context for the document and its purpose - Must align with the persona's role and the company's business needs - Should sound like something this employee would write for internal stakeholders Conclusion Requirements: - Write a professional 4-sentence maximum conclusion paragraph - Should summarize key takeaways and next steps - Must align with the persona's perspective and recommendations - Should provide actionable insights for the intended audience Return ONLY a valid Python dictionary with this exact structure: {{ "file_title": "Professional title for the PDF document (8 words max)", "introduction": "Professional introduction paragraph (4 sentences max)", "subsection_headings": ["List of exactly {n_subsections} professional subsection headings"], "conclusion": "Professional conclusion paragraph (4 sentences max)", "file_name": "the name of the file should be 7 words max and should be seperated by dashes. The should be unique, creative, and relevant to the content of the file." }} IMPORTANT: - Subsections must be thematically related to but NOT directly answering the DR Question - One subsection will later contain the actual insight - the others are realistic business distractors - Do NOT include any preamble, explanation, or extra text-return only the Python dictionary - Ensure the document feels authentic for this persona's role and company context
Figure K.12: Prompt template for user-side file (PDF/DOCX) outline generation.
You are an expert business document writer creating realistic enterprise PDF content. Given a Deep Research (DR) Question, company context, and a specific insight, generate professional content that naturally incorporates the insight information to help answer the DR Question. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} Persona Context: {persona_context} DR Question: {dr_question} External Market Context (for reference): {external_context} Target Insight: - Specific Question: {specific_question} - Answer: {answer} - Justification: {justification} Subsection Heading: {subsection_heading} Content Generation Requirements: - Generate realistic business content for the given subsection heading - The content must contain exactly ONE paragraph of 4-5 sentences - Content should be professional and sound like something this persona would write - The paragraph must naturally incorporate the insight answer information but NOT copy it word-for-word - Paraphrase and embed the key metrics/information within business context - Use industry-appropriate terminology and realistic business scenarios - Include the specific data points from the answer but present them naturally - Make it sound like a business report discussing actual findings or results Content Strategy: - Present the insight information as business findings, analysis results, or operational data - Embed the key metrics within broader business context and implications - Use natural business language to discuss the same information as in the answer - Connect the information to business outcomes, goals, or strategic initiatives - Make the content feel like a natural part of an enterprise document - The content should include all the information that is in the answer, including the dates and quantiative values Justification Requirements: - Explain specifically how this content helps answer the DR Question - Reference the key information that would be useful for decisionmaking - Keep justifications concise but clear (20 words maximum) Return ONLY a valid JSON object with this exact structure: {{ "subsection_heading": "{subsection_heading}", "content": "should be at least 3 sentences", "justification": "Explanation of how this content provides information needed to answer the DR Question and the specific_question" }} IMPORTANT: - Do NOT copy the answer text directly - paraphrase and embed it naturally - The content must contain the key information from the answer but presented professionally - Ensure content feels authentic for this persona's role and company context - The paragraph should read like real business documentation, not a Q&A format - Include specific metrics/data from the answer but in natural business language - Do NOT include any preamble, explanation, or extra text-return only the JSON object
Figure K.13: Prompt template for user-side insightsection (PDF/DOCX) generation.
User File Distractor Section Generation Prompt You are an expert business document writer creating realistic enterprise PDF content. Given a Deep Research (DR) Question, company context, and subsection headings, generate distractor content for each subsection that is thematically related but does NOT help answer the DR Question. Company: {company_name} - {company_description} Industry: {industry} Company Size: {company_size} ({employee_count}) Annual Revenue: {annual_revenue} Persona Context: {persona_context} DR Question: {dr_question} External Market Context (for reference): {external_context} Subsection Headings: {subsection_headings} Content Generation Requirements: - Generate realistic business content for each subsection heading - Each subsection must contain exactly ONE paragraph of 3-4 sentences maximum - Content should be professional and sound like something this persona would write - Content must be thematically related to the DR Question's domain but NOT provide information to answer it - Use industry-appropriate terminology and realistic business scenarios - Include specific but non-revealing details (dates, percentages, departments, processes) - The content should be a full paragraph - The content should have quantitative numbers and dates which could be, as a mere example, "X customers bought Y items" - Make sure the content has nothing to do with the deep research question Distractor Strategy: - Focus on adjacent business areas that don't directly impact the DR Question - Discuss historical context, general industry trends, or procedural information - Include operational details that are realistic but tangential - Reference related but non-essential business metrics or activities - Avoid any content that would help someone answer the DR Question Justification Requirements: - Explain specifically why each paragraph's content doesn't help answer the DR Question - Identify what type of distractor strategy was used (e.g., "focuses on historical data vs current decision factors") - Keep justifications concise but clear (15 words maximum) Return ONLY a valid JSON array with this exact structure: [ {{ "subsection_heading": "Professional subsection title", "content": "should be at least 4 sentences", "justification": "Brief explanation of why this content doesn't help answer the DR Question" }} ] IMPORTANT: - Generate content for ALL provided subsection headings - Ensure content feels authentic for this persona's role and company context - Content must be realistic distractors - related but not helpful for the DR Question - Do NOT include any preamble, explanation, or extra text-return only the JSON array - Each paragraph should sound like professional business writing from this industry
Figure K.14: Prompt template for user-side distractorsection (PDF/DOCX) generation.
User Chat Generation Prompt You are an expert in enterprise communication systems and organizational structures. Your task is to generate a realistic setup for teams, channels, and users for a Mattermost chat system based on the given insights and company context.
,→ ,→
Company Context: - Company Name: {company_name} - Description: {company_description} - Industry: {industry} - Size: {company_size} ({employee_count} employees) - Annual Revenue: {annual_revenue} Persona Context: {persona_context} **Specific Question** {specific_question} **Answer to Specific Question** {answer} Requirements: - Generate teams, channels, and users that would realistically discuss these insights - Make sure the teams and channels are realistic for persona to be a member - Each channel must be associated with a team - Each user must be a member of at least one team and one channel - Generate a minimal but sufficient setup to support {num_turns} chat messages discussing the ,→ insights - To make it realistic, generate at least 2 teams, 2 channels and 3 users - Use realistic names for people/teams/channels based on the company context - The persona needs to be part of all teams and channels - Team `name' values must be Mattermost-safe slugs and must not start with reserved prefixes ,→ like `channel', `api', `login', `admin', or `signup' - If a natural team label would begin with a reserved word, rewrite the slug to a safe ,→ equivalent such as `sales-channel` or `team-channel-sales` Return ONLY a valid Python dictionary with this exact structure: {{ "teams": [ {{ "type": "team", "team": {{ "name": "team_name", "display_name": "Display Name", "type": "O", "header": "Team header description", "purpose": "Team purpose description" }} }} ], "channels": [ {{ "type": "channel", "channel": {{ "team": "team_name", "name": "channel_name", "display_name": "Channel Display Name", "type": "O", "header": "Channel header description", "purpose": "Channel purpose description" }} }} ],
}}
"users": [ {{ "type": "user", "user": {{ "username": "username", "email": "[email protected]", "password": "my_drbench_pwd", "nickname": "Nickname", "first_name": "First", "last_name": "Last", "position": "Job Title", "roles": "system_user", "locale": "en", "teams": [ {{ "name": "team_name", "roles": "team_user", "channels": [ {{ "name": "channel_name", "roles": "channel_user" }} ] }} ] }} }} ]
IMPORTANT: - Do NOT include any preamble, explanation, or extra text-return only the Python dictionary - Make sure the persona is included as a user and member of all teams/channels - The persona user record must exactly reuse the persona username and email provided in the ,→ persona context - Do not invent alternate persona usernames or emails - Do not remove punctuation from the persona username (for example, keep `first.last' exactly ,→ as given)
Figure K.15: Prompt template for the user-side chat-generation.
User Chat Generation Prompt You are an expert in enterprise communication systems and organizational structures. Your task is to generate a realistic setup of users for an email system based on the given insights and company context.
,→ ,→
Company Context: - Company Name: {company_name} - Description: {company_description} - Industry: {industry} - Size: {company_size} ({employee_count} employees) - Annual Revenue: {annual_revenue} Persona Context: {persona_context}
Return ONLY a JSON array of users with this exact structure: [ {{ "type": "user", "username": "alice.smith", "first_name": "Alice", "last_name": "Smith", "email": "[email protected]", "password": "my_drbench_pwd" }} ] IMPORTANT: - Do NOT include any preamble, explanation, or extra text—return only the Python dictionary - Ensure the structure is realistic for the company size and industry - Make sure the persona is included as a user
**Specific Question** {specific_question} **Answer to Specific Question** {answer} Requirements: - Generate users that would realistically discuss these insights - Generate a minimal but sufficient setup to support {num_messages} emails discussing the insights - To make it realistic, generate at least 3 users - Use realistic names for people/teams/channels based on the company context
,→
Figure K.16: Prompt template for user-side email thread generation.
Adjacent Question Generation Prompt You are an expert benchmark designer. Your task is to generate ADJACENT deep research (DR) questions that an agent might confuse with the target DR question, but which describe a DIFFERENT case or scenario at the same organization. These adjacent DR questions will be used to generate distractor documents that share vocabulary and workflow domain with the target case but are NOT the answer to the target. The challenge for the agent is disambiguation, not contradiction-spotting — so the adjacent cases must be internally coherent and plausibly co-exist in the same enterprise's knowledge base. REQUIREMENTS for each adjacent DR question: 1. Same workflow domain / industry as the target (same business processes, terminology, stakeholders). 2. DIFFERENT scenario, case, or sub-process. Do NOT generate a re-phrasing of the target. 3. Should NOT be answered by ANY of the existing target user_insights below — the adjacent DR pertains to a different case, so its facts are different. 4. Realistic — something a worker in this role might actually need to look up in this organization's knowledge base. 5. Concise: a single workflow-oriented question, 12–25 words. Examples of good adjacency (for a healthcare informed-consent / LEP target): - "What is the workflow when a patient withdraws consent at the last minute before anesthesia?" - "What to do when an emergency surgery patient does not have current insurance verification?" Both share the consent/perioperative domain but describe distinct, independently-resolvable scenarios. DO NOT: - Restate the target with synonyms. - Pick a topic so far afield it would obviously be filtered out (e.g., HR policy, IT outage). - Generate two adjacent DRs that are near-duplicates of each other. Target DR question: {dr_question} Organization context: - Industry: {industry} - Company: {company} - User role: {user_role} ({user_role_description}) Existing target user_insights (for context — your adjacent DRs MUST be about a different case than these): {insights_json} Generate EXACTLY {n_drs} adjacent DR questions. For each, provide a one-sentence `rationale` explaining how it is similar to but distinct from the target. Return JSON: {{"drs": [{{"dr_question": "...", "rationale": "..."}}, ...]}} — exactly {n_drs} items. .
Figure K.17: Adjacent questions generation prompt for the mixed variant.
Adjacent Insight Generation Prompt You are generating realistic internal-case insights for an enterprise benchmark. The DR question below describes a SPECIFIC case at this organization. Generate exactly {n_items} concise insights (Q/A/justification triples) that a worker would document about this specific case while resolving the DR question. The insights should TOGETHER provide a coherent factual profile of one case. REQUIREMENTS: - Each insight is a single concrete fact about THIS case (a specific value, status, name, count, date, etc.). - Confident tone, no hedging — these are real internal records about a real case. - Internally consistent — across the {n_items} insights, the facts must form one coherent case profile (not a contradictory or surreal mix). - Different from the target organization's other open cases — invent specifics that are realistic but distinct (different patient/customer/account/etc., different identifiers, different dates). - Each insight has a short `question` (what does this insight tell us?) and a clear `answer`. Adjacent DR question (this is the case being documented): {dr_question} Organization context: - Industry: {industry} - Company: {company} - User role: {user_role} ({user_role_description}) Generate EXACTLY {n_items} insights forming one coherent case profile. Return JSON: {{"items": [{{"item": 1, "question": "...", "answer": "...", "justification": "..."}}, ...]}}
Figure K.18: Adjacent insights generation prompt for the mixed variant.
Adjacent Scene Generation Prompt Define a small, consistent SCENE for an internal-document set about a single case. The same scene will be reused across multiple file formats (pdf, docx, chat, email) that all describe the same case, so colleagues, dates, and identifiers MUST stay stable. Company: {company} Primary persona: {persona_name} ({persona_username}) - {persona_role} Workflow context (DR question): {dr_question} Variant type: {qa_label} Specific insight being embedded: - Question: {question} - Answer: {answer} Generate a JSON Scene with: - cast: 3-4 supporting colleagues (NOT including the primary persona). Each has: - username (firstname.lastname, all lowercase) - first_name - last_name - role (e.g. "Financial Aid Counselor", "Academic Advisor", "Returns Specialist") Use realistic names. Avoid duplicates. Keep them appropriate for {company}'s industry. - timeline_anchor: a single sentence with a specific recent date that anchors the case (e.g. "Received the suspension notice on 2026-04-22"). - key_dates: 1-3 additional specific dates referenced in the workflow (deadlines, meeting dates, follow-ups). Use ISO format YYYY-MM-DD. - case_id: an internal identifier referenced across all formats (e.g. "SCP-2026-04221", "RTN-58812"). Return ONLY the JSON Scene object.
Figure K.19: Adjacent case scenario generation prompt for the mixed variant.
Adjacent Fact Checking Prompt You are a strict fact-checker. Review the GENERATED CONTENT for any DIRECT CONTRADICTION with the CANONICAL ANSWER. {guidance} A direct contradiction means: the generated content asserts a fact that is mutually exclusivewith a fact stated in the canonical answer. Examples: - canonical says "suspended" but content says "approved" — CONTRADICTION - canonical says "first appeal" but content says "second appeal" — CONTRADICTION - canonical says "warning status" but content says "denied" — CONTRADICTION - canonical says "mailed notice" but content says "received an email notice" — CONTRADICTION - canonical says "12 attempted credits" but content says "15 attempted credits" — CONTRADICTION NOT contradictions: - Adding extra details not in the canonical answer (allowed) - Specifying a vague placeholder with plausible specifics (allowed for hard-negatives) - Reordering or rephrasing the same facts (allowed) CANONICAL ANSWER (ground truth — treat as authoritative): {canonical_answer} GENERATED CONTENT (JSON): {generated_json} Return the generated content as JSON in EXACTLY THE SAME SHAPE. - If there are NO contradictions: return the JSON unchanged. - If there ARE contradictions: rewrite ONLY the contradicting statements so they align with the canonical answer. Do NOT remove other content. Do NOT change the JSON shape, the number of subsections / posts / messages / users / channels / teams, or any usernames. Preserve all metadata, ordering, and formatting. Return ONLY the JSON object.
Figure K.20: Fact-checking prompt for the mixed scenario.
Step Precision Evaluation Prompt You are evaluating whether a predicted workflow step semantically matches any ground truth (golden) workflow step. ## Matching Rules (precision-oriented — avoid false positives) Match on **procedure intent**, not on wording. Ignore case-specific instance details (names, IDs, ticket numbers, dates, amounts, customer/vendor strings): those must neither force nor forbid a match. To count as a match, the predicted step must: - Address the **same primary obligation** as the golden step (what must be done, in what role/system context), not merely a related or upstream/downstream activity. - Align on the **same decision or artifact class** when the golden step is conditional or produces a specific record/approval/output. Reject `matched=true` when the prediction is only loosely related, a generic umbrella ("handle the case"), a prerequisite, a follow-up, merges two distinct golden obligations into one vague step, or differs in system/process family (e.g., intake vs ledger posting vs legal hold). Reserve **high confidence (≥0.88)** only for clear same-procedure alignment; otherwise use `matched=false` or lower confidence. ## Predicted Step - Item: {pred_item} - Title: {pred_title} - Task: {pred_task} - Description: {pred_desc} - Subtask/Conditions: {pred_subtask if pred_subtask else 'None'} ## Golden Workflow Steps {gold_steps_text} ## Instructions Pick at most one best golden step. Use `matched=true` only if the predicted step would satisfy a reviewer that it is the **same** procedural beat as that golden step (ignoring instance-specific strings). If in doubt, prefer `matched=false`. Return ONLY a valid JSON object (no other text): {{ "matched": true or false, "matched_gold_id": "<ID of the best matching golden step, e.g. WORKFLOW_STEP_1, or null if no match>", "matched_gold_title": "<title of matched golden step or null>", "confidence": <float 0.0-1.0>, "justification": "<brief explanation of why it matches or why no match was found>" }}
Figure N.1: Prompt for precision evaluation.
Step Recall Evaluation Prompt You are evaluating whether a ground truth workflow step is semantically covered by any of the predicted workflow steps. ## Matching Rules (recall-oriented — demand real coverage) The golden step is the reference. Ignore instance-only strings (people names, account numbers, dates, free-text identifiers): predictions may use different wording or entities and still match. For `matched=true`, some predicted step must **materially implement** what the golden step requires: - The predicted step's task/description should make it plausible that the golden obligation would be completed, not just mentioned in passing or implied by a distant step. - Title overlap alone is insufficient if the predicted action is a different kind of work (e.g., "notify stakeholder" vs "obtain written approval"). Reject when the closest prediction is only adjacent (prerequisite, packaging, logging) or a superset that omits the golden step's distinctive requirement. Use confidence **≥0.83** only when coverage is substantive; otherwise `matched=false` or lower confidence. ## Ground Truth Step - ID: {gold_id} - Title: {gold_title} - Task: {gold_task} - Expected Answer: {gold_answer_truncated} ## Predicted Workflow Steps {pred_steps_text} ## Instructions If multiple predictions partially relate, pick the **single** predicted item that best covers this golden step. If none clearly does, answer `matched=false`. Do not stretch synonyms across different process families. Return ONLY a valid JSON object (no other text): {{ "matched": true or false, "matched_pred_item": <item number of the best matching predicted step, or null if no match>, "confidence": <float 0.0-1.0>, "justification": "<brief explanation of why it matches or why no match was found>" }}
Figure N.2: Prompt for recall evaluation.
Factuality Evaluation Prompt Given the following relevant source context from multiple sources and an insight, determine if the insight is factually supported by the sources. Relevant Source Materials (from multiple sources): {context} Atomic Claim: {insight} EVALUATION CRITERIA: The claim is factual if the core factual content is supported by the sources. You should be strict about important details but flexible about exact wording: REQUIRED for TRUE: 1. All key factual details (numbers, dates, names, percentages, specific facts) must be present in at least one source 2. The main substance and meaning of the claim must be supported by the source contexts 3. No part of the claim should contradict the information in any of the sources ACCEPTABLE variations: - Different wording or phrasing that conveys the same meaning - Paraphrasing or summarization of the source information - Minor linguistic differences that don't change the factual content Mark as FALSE if: - Important factual details are missing, incorrect, or unsupported across all sources - The claim contradicts information in any of the sources - The core meaning cannot be verified from any of the source contexts EXAMPLES: FACTUAL CLAIM (TRUE): Source A: "Company ABC reported $50M revenue in Q3 2023" Source B: "ABC saw 15% growth this quarter compared to Q2" Claim: "ABC's revenue grew by 15% to $50M in Q3 2023" → TRUE (facts from both sources combined) UNFACTUAL CLAIM (FALSE): Source A: "Company ABC reported $50M revenue in Q3 2023" Source B: "ABC saw 10% decline this quarter" Claim: "ABC's revenue grew by 15% in Q3 2023" → FALSE (contradicts Source B) UNFACTUAL CLAIM (FALSE): Source A: "The CEO mentioned potential expansion plans" Claim: "The CEO confirmed definite expansion of 25% next year" → FALSE (adds unsupported specifics not in any source) Focus on the substantive factual accuracy rather than exact word-for-word matching. You MUST respond with either true or false under the <factual> tag. Then provide a brief explanation under the <explanation> tag explaining which parts are supported or not supported and from which sources. Format your response EXACTLY as: <factual>false</factual> <explanation>The claim is mentioned in source X... but source Y contradicts...</explanation>
Figure N.3: Prompt for factuality verification.
Condition Resolution Prompt You are evaluating whether a predicted workflow step properly resolves the conditions described in a golden (ground truth) workflow step. ## Golden Step Conditions The golden step "{gold_title}" ({gold_id}) has the following conditions that require resolution: {gold_conditions} ## Expected Resolution (from golden answer) The correct resolution for the user's specific situation is: {gold_answer_trunc} ## Justification for the correct resolution {gold_justification_trunc} ## Predicted Step The predicted workflow step that corresponds to this golden step is: - Item: {pred_item} - Title: {pred_title} - Task: {pred_task} - Description: {pred_desc} - Subtask/Conditions stated: {pred_subtask if pred_subtask else 'None'} ## Evaluation Criteria Score the condition resolvement based on these levels: 1. **Score 1.0 (Fully Resolved)**: The predicted step correctly resolves the conditions by: - Choosing the appropriate condition path for the user's situation - Providing specific justification or evidence for the chosen path - Not just listing all possible conditions, but actually routing through one 2. **Score 0.5 (Partially Resolved)**: The predicted step mentions the conditions but: - Lists multiple possible paths without clearly choosing one - Mentions the correct path but lacks specific justification - Shows awareness of conditions but doesn't fully personalize the resolution 3. **Score 0.0 (Unresolved)**: The predicted step: - Does not address the conditions at all - Only provides generic description without condition awareness - Completely ignores the conditional routing Return ONLY a valid JSON object (no other text): {{ "score": <0.0, 0.5, or 1.0>, "level": "<resolved, partially_resolved, or unresolved>", "justification": "<detailed explanation of the evaluation>" }}
Figure N.4: Condition-resolution evaluation prompt.
Personalized Comprehensiveness Prompt You are evaluating the comprehensiveness and personalization of a predicted workflow step description compared to a golden (ground truth) standard. Be strict. Do not reward loose thematic similarity, generic workflow language, or partially overlapping content as highly comprehensive. A high score should be reserved for steps that preserve the concrete details, user-specific tailoring, and operational guidance from the ground truth. ## Golden Step (Ground Truth) - ID: {gold_id} - Title: {gold_title} - Task: {gold_task} - Expected Detailed Answer: {gold_answer_trunc} - Justification: {gold_justification[:1000] if gold_justification else 'N/A'} ## Predicted Step - Item: {pred_item} - Title: {pred_title} - Task: {pred_task} - Description: {pred_desc} - Subtask/Conditions: {pred_subtask if pred_subtask else 'None'} ## Evaluation Dimensions Rate the predicted step on a scale of 0.0 to 1.0 based on these criteria: 1. **Specificity** (0.0-1.0): Compared to the golden step answer, does the predicted description mention specific names, form numbers, data categories, institutions, study types, procedures, thresholds, and other concrete details from the golden answer? Or is it vague and generic? 2. **Personalization** (0.0-1.0): Compared to the golden step answer, is the description tailored to the user's specific situation (e.g., mentioning the specific study type, specific forms, specific data categories, specific numbers, specific PI or coordinator)? Or could it apply to many generic situations? 3. **Completeness** (0.0-1.0): Compared to the golden step answer, does the description cover the key action items and important details from the golden answer? Are major steps, conditions, branches, approvals, or required artifacts missing? 4. **Actionability** (0.0-1.0): Compared to the golden step answer, does the description provide clear, actionable instructions that a user could follow? Or is it too high-level to be useful? Scoring guidance: - Reserve 0.90-1.00 for near-complete, highly specific, clearly personalized steps with very few meaningful omissions. - Use 0.70-0.89 for good steps that are still missing some concrete details or tailoring. - Use 0.40-0.69 for partially correct but generic/incomplete steps. - Use 0.00-0.39 when the step is mostly generic, weakly personalized, or missing major workflow content. Compute an overall score as the average of these four dimensions before any penalties for missing details. Return ONLY a valid JSON object (no other text): {{ "specificity": <float 0.0-1.0>, "personalization": <float 0.0-1.0>, "completeness": <float 0.0-1.0>, "actionability": <float 0.0-1.0>, "overall_score": <float 0.0-1.0, average of above>, "justification": "<concise explanation covering all dimensions>", "missing_details": ["<list of important details from golden answer missing in predicted step>"] }}
Figure N.5: Prompt for personalized-comprehensiveness evaluation.
O
List of DRF LOW Tasks
We provide the benchmark tasks for each domain in Tables O.1, O.2, O.3, O.4, and O.5. For each task, we report the task identifier, the deep research question, and the number of insights, distractors, and mixed documents.
Task
DR Question
b2b_01
What is the workflow to get an inspection approved by an aerospace customer before we begin full production? What is the workflow to set up a new B2B customer onboarding? What steps should we follow to build a B2B pricing proposal for the government security service? What is the step-by-step process to approve contract redlines quickly? What steps should we take to launch an account-based marketing campaign? What is the workflow to resolve a disputed B2B invoice payment? What steps should we follow to evaluate a new channel partner to expand regional coverage? What is the workflow to complete the Mandatory Ethics Course training? What steps should we take to fix a broken order-to-cash process? What is the step-by-step process to renew a B2B subscription contract with BrightDesk?
b2b_02 b2b_03 b2b_04 b2b_05 b2b_06 b2b_07 b2b_08 b2b_09 b2b_10
# Insights
# Distractors
# Mixed
16
10
24
13 15
9 10
24 24
15 13
11 9
24 24
13 15
8 8
24 24
15
9
24
15 13
13 11
24 24
# Insights
# Distractors
# Mixed
13 15
13 12
24 24
15 17 18
11 9 10
24 24 24
18 19 20
10 12 12
24 24 24
12 20
8 11
24 24
# Insights
# Distractors
# Mixed
20
10
24
19
12
24
18
12
24
18
12
24
12
8
24
19
10
24
20
11
24
20 21 20
13 11 10
24 24 24
Table O.1: List of tasks in B2B domain.
Task
DR Question
b2c_01 b2c_02
What steps should I take to return an online purchase past the deadline? What is the workflow to dispute a duplicate charge on a customer’s subscription? What steps do I follow to recover a hacked customer account? What is the exact process to redeem expired loyalty points? What steps should I take to change the recipient’s delivery address after shipping? What is the workflow to cancel a gym membership without penalties? What steps do I take after a customer reports a defective product? What is the process to remove my personal data from the company’s database? What steps should I follow to activate a new phone on my plan? What is the workflow to follow after a customer appeals against denied refund for a travel booking?
b2c_03 b2c_04 b2c_05 b2c_06 b2c_07 b2c_08 b2c_09 b2c_10
Table O.2: List of tasks in B2C domain.
Task
DR Question
education_01
What steps should I follow to appeal a financial aid suspension decision? What is the workflow to create an IEP for a newly enrolled student? What steps do I take to get course credit for prior work experience? What is the step-by-step process to report suspected academic plagiarism? What workflow should I follow to set up an accessible online course? What are the steps to run a parent-teacher conference for learning concerns? What is the workflow to evaluate and adopt a new math curriculum for 7th grade? What exact steps should I take to request a transcript correction? What steps do I need to follow to investigate a bullying report? What is the workflow to onboard a newly hired tenure-track faculty?
education_02 education_03 education_04 education_05 education_06 education_07 education_08 education_09 education_10
Table O.3: List of tasks in Education domain.
Task
DR Question
healthcare_01
What is the step-by-step workflow to obtain informed consent for a procedure when the patient has limited English proficiency? What is the step-by-step workflow to address vaccine refusal and document the encounter appropriately? What exact steps should we take to perform a same-day preprocedure readiness check for an ambulatory surgery patient? What steps should we follow to assess and escalate care for a patient with acute shortness of breath on the hospital floor? What is the step-by-step process to obtain consent for blood transfusion? What workflow should we follow to isolate a patient with suspected tuberculosis? What is the step-by-step process to communicate and document an unexpected critical imaging finding? What is the step-by-step workflow to evaluate and follow up on an abnormal diabetes screening result? What exact steps should we take to process and document a request for release of medical records? What is the step-by-step workflow to manage a suspected stroke patient?
healthcare_02 healthcare_03 healthcare_04 healthcare_05 healthcare_06 healthcare_07 healthcare_08 healthcare_09 healthcare_10
# Insights
# Distractors
# Mixed
14
7
24
19
11
24
20
9
24
18
11
24
18
13
24
19
10
24
18
9
24
19
11
24
19
11
24
19
11
24
# Insights
# Distractors
# Mixed
18
11
24
18
10
24
19 19
12 9
24 24
19
8
24
19 20 21
11 8 9
24 24 24
20
12
24
20
12
24
Table O.4: List of tasks in Healthcare domain.
Task
DR Question
legal_01
What exact steps should we take to investigate a potential internal compliance violation? What exact steps should we follow to prepare and submit a regulatory filing for a new product launch? What exact steps should I take to process a denied insurance claim? What steps do we need to follow to conduct a workplace harassment investigation? What is the workflow to prepare for a small claims court hearing on disputed pricing? What steps should we take to create an employee termination file? What is the workflow to draft and file a provisional patent application? What steps should we follow to comply with a data breach subpoena for records? What steps should we take to perform a trademark clearance review before launching a new product name? What steps should we take to collect an overdue invoice legally?
legal_02 legal_03 legal_04 legal_05 legal_06 legal_07 legal_08 legal_09 legal_10
Table O.5: List of tasks in Legal domain.
P
Examples of Reference and Predicted Workflows
We provide the examples of generic workflows (with conditions) as well as reference and predicted personalized workflows (with condition-resolved) in Figures P.1, P.2, and P.3,.
Q
Screenshots of Task and App Environments
Task Annotation. We provide the screenshot of our task annotation procedure in Figure Q.1. Task Creation. We also provide the support for creating DRF LOW tasks in a no-code web setup. We show the screenshot of such setup in Figure Q.2. Application Environments. We present the screenshot of different application environments that we use in DRF LOW in Figure Q.3, Q.4, Q.5, and Q.6.
Step · 1 · Create Client Workspace System-of-record for onboarding.
Step · 2 · Complete OLF-101 Intake form with tier and scope.
Step · 3 · Select playbook Based on tier and user count.
Enterprise
Subscription tier?
Standard
Enterprise playbook
Standard playbook
Yes
End users > 100?
No
Add Training pack
Default milestones
Step · 4 · Assign owners & dates Internal plan only.
Step · 5 · Record Gate check Success criteria + integrations.
Yes
No
Criteria & integrations complete?
Ready for Security Gate
Set status Draft Collect missing fields
Step · 6 · Initiate Security Gate Send security/DPA packet.
Customer's own
Predicted resolved workflow: OnboardLift enterprise customer onboarding
Step 1 · Align owners + define Redwood's primary onboarding KPI
Standard
Questionnaire source?
Attach customer form
Finalize who is accountable for Redwood's end-to-end onboarding decisions (CS vs Implementation vs Product), and document the owner in the onboarding record. Adopt "time-to-first usable dashboard" as the primary KPI and sequence milestones accordingly (security/access → integration → validation).
Send standard form
Map to control library
Resolved workflow: OnboardLift enterprise customer onboarding Step 1 · Create Client Workspace onboarding record Yes
Production data access?
No
Include DPA
Step 2 · Create Redwood's OnboardLift workspace + set email/call cadence
Create a new OnboardLift Client Workspace named with the customer's legal name. Attach Sales handoff notes capturing 120 users, SSO, Snowflake + SFTP, 30-day pilot, and email-only comms.
Create the Redwood Logistics onboarding workstream in OnboardLift as the execution system of record. Publish the shared onboarding plan and set a weekly cadence via email plus scheduled calls. Explicitly record that Slack Connect is not permitted.
Step 2 · Complete Intake Form OLF-101
Step 3 · Draft and route the success-criteria document for Redwood's signature
Skip DPA
Data flow diagram
Fill OLF-101: Enterprise tier, target go-live, 30-day pilot milestone, ~120 users across 3 departments with role-based access, Snowflake + SFTP integrations, and measurable success criteria.
Step · 7 · Configure tracks SSO + data integrations.
Yes
SSO required?
Step 3 · Select Enterprise playbook with Training pack
No
In playbook setup, choose "Enterprise Onboarding" since the tier is Enterprise, and add the "Training & Adoption" milestone pack since rollout exceeds 100 end users.
Identity Setup track
No identity track
Draft a success-criteria document for Redwood that defines onboarding completion in verifiable terms (anchored to the time-to-first-usable-dashboard KPI). Circulate it for signature before kickoff and save the signed document in the shared folder with links in OnboardLift.
Step 4 · Prepare Standard onboarding kickoff deck with scope boundaries + escalation paths Create the kickoff deck for Redwood including a scope boundary table, defined escalation paths, and examples of out-of-scope requests. Align the deck to the signed success criteria and reference the shared onboarding plan milestones. Store the final deck in the shared folder.
Block provisioning
Step 4 · Assign internal owners and due dates Assign CS/Implementation/Solutions/Support owners. Front-load Solutions Engineer tasks into the first two weeks given limited availability. Do not yet invite the customer to view the plan. Yes
Data integration needed?
No
Step 5 · Pass Record Gate Data Connection track
No data track
Validation gate
Step 5 · Execute Redwood's security gate using their proprietary questionnaire + signed DPA Coordinate stakeholders to complete Redwood's proprietary security questionnaire and track approvals as dated milestones. Drive the DPA to signature. Treat this as a hard gate: do not connect any production data until both the security review and DPA are complete.
Reopen OLF-101 and confirm success criteria and the full integrations list (Snowflake, SFTP) are populated. Mark the workspace "Ready for Security Gate". Step 6 · Front-load tenant environment readiness (SSO, roles, connectors)
Step · 8 · Kickoff & publish plan
Step 6 · Initiate Security Gate with DPA
Customer-facing plan.
Start the Security Gate workflow. Attach the customer's own questionnaire (map internally to the control library) and include the DPA request plus a data flow diagram since production data is needed. Standard
Onboarding package?
White glove
Step 7 · Run Redwood's first data pipeline + reconciliation for Data Integrations sign-off
Step 7 · Configure SSO and data tracks Baseline + Scope Review
Include custom items
Create an Identity Setup track for SAML SSO with User Provisioning blocked until SSO is approved. Create a Data Connection track for Snowflake and SFTP with a sample-data validation checkpoint.
Step · 9 · Go-Live Gate
Step 8 · Schedule kickoff and publish plan
All checkpoints complete?
Validate connectivity for Redwood's priority datasets (POs, shipments, inventory snapshots, carrier events). Run data quality checks and reconcile sample-period totals. Obtain Data Integrations sign-off after the first successful pipeline run and reconciliation match.
Step 8 · Complete Redwood's standardized onboarding artifact set aligned to KPI
Review blocking checkpoints.
Yes
Because Solutions Engineer availability is front-loaded (full allocation in the first two weeks), schedule Redwood's environment readiness work early. Ensure SSO is configured, roles are created, connectors are authenticated, and capture Solutions' formal sign-off on environment readiness.
Schedule kickoff with internal owners and customer stakeholders. Publish the Standard-package customer-facing plan with baseline milestones only, and route custom requests to Scope Review tasks. Share via email and shared folder (no Slack Connect).
No
Switch to Go-Live
Stay In Implementation
Start health tracking
Log risk note
Step 9 · Run Go-Live Gate and transition status
Ensure Redwood has the complete artifact package: intake form, source-to-KPI mapping sheet, validation checklist, and go-live runbook. Use these artifacts to define and verify what "first usable dashboard" means. Store everything in the shared folder and link in OnboardLift.
Step 9 · Run tenant-ready gate before sending Redwood invites
Verify all blocking checkpoints: security approved, DPA signed, SSO approved, Snowflake/SFTP validated, success criteria signed off. Switch workspace status to Go-Live and start health tracking.
Use the tenant-ready checklist as a hard gate before inviting Redwood users. Ensure Solutions, Data Integrations, and CS have signed off on environment, data, and customer-facing readiness. If any sign-off is missing, hold invites and remediate before re-running the checklist.
(b) Personalized Workflow (Reference)
(c) Personalized Workflow (Predicted)
Onboarding complete Health tracking live
(a) Generic Workflow
Figure P.1: Example of Workflows for b2b_02 task: What is the workflow to set up a new B2B customer onboarding?
Predicted resolved workflow: tenure-track faculty onboarding Step 1 · Open the tenure-track faculty onboarding case
Step · 1 · Create onboarding case Open EastportWorks faculty case.
Establish the onboarding case as the operational workspace for the new tenure-track hire. The case should capture legal name, start date, department, title, supervisor, primary work location, PDF-7 routing status, I-9/immigration status, IT-8 and CCR-6 numbers, LAR-5 status, EH&S training, purchasing prerequisites, and unresolved risks.
Step · 2 · Submit FOP-1 Route to HR, Payroll, IT, dean.
Step · 3 · Complete PDF-7 Fill tenure-track position fields.
Resolved workflow: tenure-track faculty onboarding (Aug 15 start) Step · 4 · Route PDF-7 Through approvers.
Step 1 · Create EastportWorks faculty onboarding case
Tenure-track
Appointment type?
In EastportWorks, create a new faculty onboarding case for the tenure-track Assistant Professor in Engineering. Attach the signed offer letter, set start date Aug 15, choose appointment type = tenure-track, and add a case note that the chair requests email, VPN, building, and laptop ready by day one.
Research/term
Step 2 · Confirm the hire's employment-eligibility path Coordinate with the department or HR to determine whether the tenure-track hire is a citizen/permanent resident or requires sponsorship. Document the determination in the case and use the result to decide whether to proceed to standard I-9 scheduling or open an ISSS/DSO case before finalizing appointment activation and start-date planning.
Step 3 · Coordinate ISSS/DSO sponsorship materials if required Dean + HR + Provost
Dean + HR only
Provost step required
No Provost
Step 2 · Submit Faculty Onboarding Packet (FOP-1) Open FOP-1 and populate identity, appointment summary (9-month with summer eligibility), Aug 15 start, home department, work location (Engineering Research Building), and chair/admin contact. Confirm routing to HR, Payroll, IT, and the dean's office and submit.
Step · 5 · Pre-employment check Initiate BGC-2 or record waiver.
If an ISSS/DSO case is active, verify the requested visa category, ensure IS-10 is complete, and upload departmental materials. For J-1, track funding documentation and English-proficiency attestation; for H-1B, track PW-2 completion before petition filing. If no sponsorship is needed, skip and continue with standard I-9 processing.
Step 4 · Update compliance milestones and date risks Yes
Benefits-eligible / student contact?
No
Step 3 · Complete Position Data Form (PDF-7)
Initiate BGC-2
Record waiver note BGC not required
Open PDF-7 and enter the case ID and faculty name. Select job code Assistant Professor, rank/track = tenure-track, FTE = 1.0, 9-month pay basis, summer comp eligible, and effective start Aug 15. Verify consistency with FOP-1 and the offer, then save for routing.
Record what has been completed, what is outstanding, and whether any immigration, I-9, or documentation item could affect the start date. Make this update visible enough to support HR, department, ISSS, and orientation planning follow-up.
Step 5 · Support completion of the PDF-7 Step · 6 · Collect payroll items W-4, direct deposit, NHD-4.
Step 4 · Route PDF-7 with Provost approval Attach the completed PDF-7 in the routing tool and confirm key fields match the offer. Because the appointment is tenure-track, include the Provost approval step in addition to dean and HR. Submit and log the workflow ID and current approver in the onboarding case.
Step · 7 · Verify work authorization ISSS or standard I-9.
No (sponsorship)
Citizen / PR?
Yes
Step 5 · Initiate BGC-2 background check
Open ISSS/DSO case
Standard I-9 only
Request IS-10 items
In the onboarding case, launch the BGC-2 authorization workflow. Enter the faculty member's identifying information and Engineering department, select the package required for benefits-eligible faculty, and trigger the authorization link. Track candidate authorization status in the case.
Step · 8 · Process ISSS by visa type Path depends on visa category.
J-1
Visa category?
Step 6 · Collect identity and payroll items in the onboarding portal Send the onboarding portal invitation and monitor completion. Confirm the hire submits the W-4, direct deposit banking information, and NHD-4 (address + emergency contact). Request corrections through portal messaging so payroll setup is ready by Aug 15.
H-1B
DS-2019 path
H-1B PW-2 path
Funding + English attest.
Prevailing wage first
Step 7 · Open ISSS/DSO case for J-1 Professor scholar Document that the hire is a non-U.S. citizen entering in J-1 Professor status. Open an ISSS/DSO case and request the IS-10 packet. Upload passport bio page, CV, and signed offer letter, and provide appointment details for DS-2019 processing.
Step · 9 · Plan I-9 completion On-campus or remote verifier.
Yes
On campus by day 3?
Step 8 · Provide J-1 DS-2019 supporting documents
No
In the ISSS case, confirm J-1 Professor category. Upload funding proof consistent with the offer and the English-proficiency attestation required by ISSS. Mark the ISSS checklist complete so the DS-2019 can be issued, and track issuance status in the case.
On-campus I-9
Remote verifier
At check-in
File I9-R1 memo
Document that the I-9 will be completed in person during on-campus check-in (not remote verifier). Coordinate with HR to schedule the appointment aligned with the Aug 15 start, and send the hire instructions on acceptable I-9 documents and deadlines.
Email, VPN, card, building access.
Step · 11 · Research setup (SFA-3)
Step 10 · Submit IT-8 and CCR-6 for day-one access PI / startup funds?
Submit IT-8 requesting email, VPN, security groups, shared drives, and a university laptop with delivery before Aug 15. Submit CCR-6 for the campus card and Engineering Research Building door access. Record all ticket numbers and flag access ready by day one as priority.
No
Route SFA-3
Step 11 · Route SFA-3 for the $300k startup package Complete SFA-3 with startup amount $300,000 (selecting the > $250k option), allowable categories, initial equipment expectations, and the speedtype/index needs. Route to Sponsored Programs, Finance, and Procurement, and document the expected index creation timeline in the case.
Step · 12 · Purchasing controls Training-gated by threshold.
Startup > $250k or specialized?
Step 7 · Launch the PDF-7 routing path Assemble or verify the correct routing path in the PDF-7 routing section or workflow tool and attach the finalized PDF-7. Update the onboarding case with routing status. Use the Provost & HR Appointments and PDF-7 channels for follow-up on routing issues or missing fields.
Step 8 · Route for dean, HR, and Provost approval Because the requested workflow is tenure-track, do not use the shorter research-track or term routing path. Monitor dean's office and HR review and confirm that the Provost approval step is included before final appointment setup.
Step 9 · Confirm I-9 completion mode and status Coordinate with HR and the hire to determine whether the I-9 will be completed via on-campus document review or an authorized remote verifier. Record completion in the onboarding case. Do not begin IT-8 or CCR-6 provisioning until I-9 completion is confirmed.
Ensure IT-8 includes legal name, start date, department, title, supervisor, and primary work location. Core provisioning should include email, VPN, shared drives, and ERP/HRIS roles. Confirm appointment duties with the department before requesting LMS, specialized security groups, or elevated permissions.
Ensure CCR-6 documents faculty affiliation, pickup instructions, and access level aligned to the worksite and responsibilities. Confirm with the department whether the hire needs restricted building or lab access, and attach supporting authorization for any elevated permissions.
Step 12 · Confirm office, lab, clinic, animal, or shop access needs
Default index only
SP + Finance + Procure.
Yes
Before routing, compare the PDF-7's effective dates, job attributes, rank/track, FTE, and pay basis with the final offer letter and executed FOP-1. If any mismatch appears, coordinate with HR, Faculty Affairs, or the department before routing to prevent downstream HR setup errors.
Step 11 · Submit or track CCR-6 campus card and access
Index for PI/startup.
Yes
Step 6 · Validate the PDF-7 against offer and FOP-1 records
Step 10 · Submit or track IT-8 access provisioning
Step 9 · Schedule on-campus I-9 completion at check-in Step · 10 · Provision IT-8 & CCR-6
Confirm the PDF-7 includes job code, rank/track, FTE, and pay basis before the form is routed. Use the Provost & HR Appointments and PDF-7 channels for appointment-term questions such as FTE, appointment basis, or compensation eligibility.
No
Coordinate with the department to determine facilities requirements. If specialized space beyond standard office is required, submit LAR-5 with building and room identifiers, required access level, activation date, sponsor/PI association, and any after-hours justification. Otherwise document that no specialized access was requested.
Step 13 · Track EH&S training based on space access Step 12 · Require Procurement & P-Card training before purchasing
Require P-Card training
Standard access
Before access
After index
Confirm the approved SFA-3 indicates startup over $250k. Enroll the faculty member (and any delegates per policy) in required Procurement and P-Card trainings. After completion and index creation, request enablement of purchasing roles tied to that index and file evidence in the case.
If the hire requires specialized lab, clinic, animal facility, shop, or other regulated space access, EH&S must assign role-based training aligned to the space type. Keep this branch open until facilities requirements are confirmed, then record enrollment and completion status in the onboarding case for audit readiness.
Step · 13 · Facilities & safety LAR-5 + EH&S if specialized.
Step 13 · Submit LAR-5 for wet lab access and schedule EH&S training
Yes
Lab / clinic / shop access?
LAR-5 + EH&S training
Confirm wet lab building/room and after-hours needs and submit LAR-5 with start aligned to Aug 15. Enroll the faculty in EH&S role-based trainings and chemical hygiene modules. Coordinate with facilities so keys/door access are only issued after training completion.
No
Step 14 · Determine purchasing and P-Card training requirements Ask the department or budget/procurement contact whether startup funds, card issuance, or equipment purchases are expected. If anticipated, ensure Procurement and P-Card training is completed before those activities proceed; otherwise standard purchasing permissions may be enabled after index creation.
Office access only
Step 14 · Finalize academics, compliance, and check-in cadence Step · 14 · Finalize academics & compliance Teaching, COI, check-ins.
Step 15 · Reconcile the case and confirm day-one readiness
Coordinate with the Registrar to enter Instructor-of-Record for the first-semester graduate course and confirm LMS access. Direct the faculty to complete the COI disclosure in ERIC. Schedule 30/60/90-day check-ins with HR, department, and ISSS, and set reminders for DS-2019 milestones and annual COI updates.
Verify the onboarding case contains final PDF-7 routing status, dean/HR/Provost approval evidence, immigration/I-9 status, IT-8 and CCR-6 request numbers, LAR-5 status, EH&S training status, Procurement/P-Card status, and any open exceptions. Coordinate final follow-up with HR, the department, Faculty Affairs, IT, EH&S, and procurement.
(b) Personalized Workflow (Reference)
(c) Personalized Workflow (Predicted)
Onboarding complete Day-one access ready
(a) Generic Workflow
Figure P.2: Example of Workflows for education_10 task: What is the workflow to onboard a newly hired tenure-track faculty?
Step · 1 · Confirm language & capacity Verify Spanish preference and capacity.
Step · 2 · Communication support Flag chart: qualified interpreter required.
Predicted resolved workflow: LEP consent for outpatient cholecystectomy
No
Deaf / hard-of-hearing?
Step 1 · Confirm Spanish as preferred language and ensure it's correctly recorded
Yes
LEP pathway
Accessibility pathway
Spanish spoken
Alternate route
Because the patient is Spanish-speaking for a cataract case, confirm directly with the patient that Spanish is the preferred language for medical decision-making and reconcile the EHR Language & Communication entry accordingly. Document the confirmed language so the OR schedule/flow team sees consistent cues.
Step 2 · Document baseline decision-making capacity before starting consent Complete the center's baseline capacity assessment and document it in the EHR capacity status field before initiating surgeon/anesthesia consents. Keep this documentation contemporaneous with pre-op intake to avoid later questions if sedation or time pressure arises.
Step · 3 · Interpreter modality Arrange qualified interpreter for consent.
Tier 1
Interpreter tier?
Tier 3
Step 3 · Flag the chart at registration: interpreter required for consent (Spanish) Follow the internal direction to flag at registration that a Spanish interpreter is required for consent, then verify the EHR banner/flags show qualified interpreter required. Confirm visibility of the flag to reduce missed interpreter documentation that has triggered audits.
Tier 2
On-site interpreter
VRI (video)
Phone interpreter
High-risk cases
Routine outpatient
If VRI is down
Step 4 · Confirm interpreter plan: qualified VRI for consents
Step · 4 · Consent discussion
Document the interpreter plan in the patient record so it is explicit that contracted VRI will be used for both surgeon and anesthesia consent discussions. Make the plan easy to find for pre-op staff so consent is not delayed.
CN-2 script with teach-back via interpreter.
Yes
Teach-back successful?
No
Step 5 · Stage consent materials: CN-2 checklist + EHR Consent-LEP Addendum
Understanding confirmed
Retrieve the CN-2 Consent Script Checklist from the QMS binder. Open the EHR consent documentation section and the 'Consent-LEP Addendum' so the interpreter fields can be completed in real time. Ensure any LEP-specific add-on packets are available to attach as needed.
Repeat explanation 2nd attempt
Step 6 · Connect VRI on the iPad cart before Dr. Torres enters (no sedating meds first) Step · 5 · Complete documentation Consent form + EHR LEP Addendum.
Resolved workflow: LEP consent for outpatient cholecystectomy
No
Implant or blood product?
Step 1 · Confirm language & capacity
Yes
Open the patient's EHR chart, confirm Spanish as preferred language, and verify decision-making capacity before any sedation. Document capacity in the required EHR field.
Standard consent set
Per Dr. Torres' request, connect VRI before he enters for anesthesia consent, and maintain a stable connection during the consent encounter. Ensure the VRI session is live before any sedating premedication is administered.
Step 7 · Document interpreter name/ID + VRI/video modality in the EHR consent fields Immediately upon connecting VRI, document the interpreter's name and interpreter ID and record the modality as video/VRI in the EHR consent section and Consent-LEP Addendum. Treat this as a stop-the-line requirement before proceeding.
Add PX-Grid packet Add-on consent
Step 2 · Flag chart: interpreter required Set the EHR indicator "Qualified Interpreter Required" so it is visible to all downstream users including anesthesia and the surgeon.
Step · 6 · Obtain signatures Before any sedating medication.
Step 8 · Run the consent conversation through the interpreter (do not use family) During surgeon and anesthesia consents, ensure the clinician speaks in short segments, pauses for interpretation, and uses the CN-2 checklist. Do not permit family members to interpret for the consent conversation. Maintain VRI connection stability.
Step 3 · Arrange VRI interpreter No
Sedation already given?
Apply the Interpreter Tier rule for a routine outpatient procedure and connect to the contracted qualified video remote interpreter (VRI). Do not use the daughter to interpret.
Yes
Proceed to sign
Pause & escalate Recheck capacity; call charge + anesth.
Step 4 · Conduct consent discussion with teach-back
Step 9 · Complete interpreter-facilitated teach-back and document outcome Have the patient restate key points (procedure, risks, alternatives) in their own words through the VRI interpreter, and document that teach-back occurred. If the first teach-back indicates misunderstanding, repeat the explanation and perform a second teach-back, documenting both attempts.
Retrieve the CN-2 Consent Script Checklist. Discuss procedure, risks, benefits, alternatives, and post-op course in Spanish via VRI. Confirm understanding using teach-back. Step 10 · Perform the pre-signature sedation/capacity gate check and escalate if needed
Step · 7 · Time-out hard stop
Step 5 · Complete consent form and LEP Addendum
Verify documentation before induction.
Fill the consent form's "Interpreter Used" section (mode, name/ID, language, start/stop times). Complete the EHR Consent-LEP Addendum including teach-back outcome.
Run the required gate check immediately before signing: confirm sedating medication has not already been administered. If it has, pause signature collection and re-assess capacity; escalate to charge nurse and anesthesiologist if capacity is uncertain.
Step 11 · Execute consent signatures with interpreter/witness line completed Yes
All items complete?
No
Step 6 · Obtain valid signatures Obtain patient, interpreter/witness, and clinician signatures before any sedating premedication. Confirm capacity remains intact.
Consent valid
Obtain signatures in the required sequence: patient (or mark/proxy), interpreter in the designated Interpreter/Witness role, and the clinician attesting to the discussion. Confirm all signatures are on the correct lines to avoid chart corrections.
Hold-Consent Correct issue; if >30 min escalate to Risk Manager
Step 12 · Scan/upload signed consent and complete pre-induction consent verification
Verify correct procedure/site, signed consent present, LEP Addendum complete, and interpreter details fully documented. Proceed to induction.
Scan/upload the signed consent into the chart and ensure the EHR consent section reflects Interpreter Used = Yes, interpreter name/ID, and modality/time. Complete the anesthesia pre-induction checkpoint by verifying the consent is fully executed and fully documented with interpreter details.
(b) Personalized Workflow (Reference)
(c) Personalized Workflow (Predicted)
Step 7 · Time-out Consent Hard Stop
Proceed to induction Cleared for surgery
(a) Generic Workflow
Figure P.3: Example of Workflows for healthcare_01 task: What is the step-by-step workflow to obtain informed consent for a procedure when the patient has limited English proficiency?
Figure Q.1: Screenshot of DRF LOW task annotations procedure.
Figure Q.2: Screenshot of DRF LOW task creation in a codeless web setup.
Figure Q.3: Screenshot of Nextcloud file management system with shared files organized in a list view.
Figure Q.4: Screenshot of File Browser interface where files organized in a list view.
Figure Q.5: Screenshot of Mattermost chat interface showing discussion channels.
Figure Q.6: Screenshot of Roundcube email management system showing sample of user’s email inbox.