arXiv:2605.20149v1 [cs.CL] 19 May 2026
Less Back-and-Forth: A Comparative Study of Structured Prompting Saurav Ghosh
Gabriella Polach
Abdou Sow
Washington University in St. Louis Saint Louis, Missouri, USA [email protected]
Washington University in St. Louis Saint Louis, Missouri, USA [email protected]
Washington University in St. Louis Saint Louis, Missouri, USA [email protected]
Abstract—Large language models (LLMs) are widely used for open-ended tasks, but underspecified prompts can lead to lowquality answers and additional interaction. This paper studies whether structured prompt design improves response quality while reducing user effort. We compare three prompt conditions: a raw prompt, a checklist-improved prompt, and a clarifyingquestion prompt. We evaluate these conditions across four task types—summarization, planning, explanation, and coding—using three LLM systems: ChatGPT, Claude, and Grok. Each output is scored with a unified rubric covering task completion, correctness, compliance, and clarity. Checklist-improved prompts achieved the highest mean rubric score, 7.50 out of 8, compared with 5.67 for raw prompts and 6.67 for clarifying-question prompts. Checklist prompts also produced the best quality-effort tradeoff, using fewer average tokens than both raw and clarifying prompts. These results suggest that a simple prompt checklist can improve LLM responses while reducing unnecessary interaction. Index Terms—large language models, prompt engineering, human-AI interaction, interaction effort, response quality, prompt evaluation
I. I NTRODUCTION Large language models (LLMs) are now used for many everyday tasks, such as summarization, planning, explanation, and coding help. As these tools become more common, many users expect them to produce useful answers quickly. However, the quality of an LLM response often depends on the prompt. Prior work has shown that LLM behavior can change substantially depending on how instructions are written and presented to the model [1]. When a prompt is short, vague, or missing key details, the model may guess what the user wants. This can lead to weak answers, extra follow-up messages, and more effort before the result is good enough to use. This issue is central to human-AI interaction because the prompt is the primary mechanism through which users communicate goals, constraints, and expectations to the model. A better prompt can reduce ambiguity by telling the model what the task is, why it matters, and what kind of answer is needed. An even more interactive approach is to let the model ask clarifying questions before answering. In that case, the system does not simply guess missing details; it first tries to resolve them. From this view, prompt quality is not only a writing issue. It is also a coordination issue between a human and an AI system.
Both saving time and improving quality are important in this setting. A fast answer is not helpful if it is wrong, incomplete, or badly formatted. At the same time, a strong answer loses value if the user must spend too many turns revising the prompt or correcting the output. For real users, the best interaction is one that reaches an acceptable answer with as little extra effort as possible. This makes it important to study both final output quality and interaction effort together, rather than treating them as separate concerns. Although many people informally claim that “better prompts” lead to better results, there is still a practical gap between advice and evidence. It is still unclear how much benefit comes from simply rewriting a prompt versus using an interactive clarification step, and whether those benefits remain similar across different kinds of tasks and different models. This paper addresses that gap through a small-scale comparative study of three prompt conditions: a raw prompt, a checklist-improved prompt, and a clarifying-question prompt. We evaluate these conditions across four task categories— summarization, planning, explanation, and coding—and across three LLM systems. We study three research questions: (1) Do structured prompts improve output quality across task types and models? (2) What is the tradeoff among output quality, token usage, and interaction time? (3) Do structured prompts reduce interaction effort? Together, these questions allow us to evaluate both final answer quality and the interaction cost required to obtain it. In this study, interaction cost is measured through turns-to-acceptance and token usage rather than wall-clock time. To address these questions, we make three contributions. First, we present a simple framework for comparing three common prompting strategies in a controlled way. Second, we test these strategies across multiple task types and models to examine whether the pattern is stable. Third, we evaluate both usefulness and effort by measuring answer quality, turns-to-acceptance, and token usage. II. R ELATED W ORK This section reviews prior work in three areas related to our study: prompt engineering and instruction-following, clarification in human-AI interaction, and evaluation of LLM output quality and user effort. We searched Semantic Scholar
and PubMed, identified 50 papers, and included 21 papers after screening for relevance. A. Prompt Engineering and Instruction-Following Prompt engineering studies how the wording, structure, and context of a prompt affect the quality of an LLM response. Studies show that prompts work better when they are clear, specific, and written with the right context in mind [2] [3] [4] [5] [6]. These findings suggest that prompt design is an important factor in response quality. Researchers have studied both manual and automatic ways to improve prompts. Some findings show that automatic prompt design can match, and in some cases even outperform, prompts written by humans [7]. Work on instructiontuned models also shows that they perform better when they learn from well-written prompts and clear instructions. This motivates our comparison between raw prompts and checklistimproved prompts, where the checklist makes the task goal, context, and expected answer format more explicit. B. Interactive Human-AI Interaction Working with LLMs often involves clarification over multiple turns. Users do not always get the right result from the first prompt. Instead, they often revise their requests after seeing the model’s response or noticing that important details are missing [8] [9]. This makes clarification a normal part of human-AI interaction. Prior work shows that interactive systems can help users refine prompts and reach better results more easily [10]. By helping users identify and fill in missing information, these systems can reduce effort and improve the overall interaction. At the same time, prompt refinement is not always easy for non-expert users. Many users do not know what details to include, and some rely too much on human-to-human communication habits when writing prompts. This is important for our study because it supports the idea that clarification can be useful when the original prompt is incomplete. C. Output Quality and User Effort The quality of an LLM’s output depends not only on the model itself, but also on how clear and well-structured the prompt is [11]. Studies show that stronger prompts often produce better feedback and more useful responses than weak or poorly written prompts [12]. This supports the idea that prompt quality should be treated as an important factor when judging final output quality. Recent work also shows that evaluation should look at more than correctness alone. Researchers now consider factors such as factuality, reasoning quality, readability, usability, user satisfaction, and time saved [13]. Some studies further suggest that structured prompting can reduce the number of revision cycles and improve both efficiency and perceived output quality [14]. This broader view is important for our study because we are interested in both output quality and user effort. Although research in this area has made strong progress, several challenges remain. The value of prompt engineering is
not always the same across models or tasks. Some advanced models appear to benefit less from more complex prompting, and there is still no single standard framework for evaluating prompt success [15]. Non-expert users also continue to face difficulties in writing effective prompts, especially when they want both high-quality outputs and low interaction effort. D. Research Gap and Challenges These prior findings leave several important challenges open. Prompt effectiveness does not stay the same across all models and tasks. Some stronger models may gain less from complex prompting, and current findings may not fully transfer as LLMs continue to change [17]. This makes it difficult to turn current results into a stable set of best practices. Another challenge is accessibility. Writing effective prompts still takes skill, revision, and experience, which can make these methods harder for everyday users to apply consistently [16]. Automatic prompt-improvement tools may help, but they do not remove the need for human judgment, especially for tasks with subtle requirements or domain-specific details [19]. A further challenge is evaluation. Recent work has expanded beyond accuracy to consider broader factors such as readability and user satisfaction [18]. However, there is still no widely accepted standard for measuring prompt quality or prompt success. These limitations motivate our study of response quality and interaction effort across prompt strategies. III. M ETHODS We evaluated the following hypotheses across three prompt conditions, four task categories, and three LLM systems. • H1: Checklist-improved prompts will produce higherquality outputs than raw prompts. • H2: Clarifying-question prompts will produce higherquality outputs than both raw prompts and checklistimproved prompts. • H3: Checklist-improved prompts and clarifying-question prompts will reduce interaction effort compared with raw prompts. The study uses a small-scale comparative design to compare common prompting strategies by measuring both final answer quality and the interaction required to reach an acceptable output. For each task, we first wrote a raw prompt. We then tested the same task using a checklist-improved prompt and a clarifying-question prompt, where the model asked 1–3 questions before producing the final answer. Each condition was run in a new session so that earlier context did not affect later results [20]. We evaluated three LLM systems: ChatGPT, Claude, and Grok, as shown in Figure 1. Each model was handled by one human operator to keep the data collection process organized: Gabriella Polach evaluated ChatGPT, Abdou Sow evaluated Claude, and Saurav Ghosh evaluated Grok. All operators followed the same task set, prompt conditions, and scoring rubric. For each model, we used the same version throughout the study and recorded the date and version used for each run.
TABLE I K EY CLAIMS AND SUPPORTING EVIDENCE IDENTIFIED IN THE REVIEWED PAPERS . Claim
Reasoning
Papers
More structured prompts improve LLM output quality Better prompts reduce user effort/time spent
Consistent gains in accuracy and quality with well-designed prompts Structured prompting reduces the number of revisions needed; users report higher efficiency Automatic methods outperform baselines on many tasks, although they may lack domain nuance Novices often use ad hoc revisions; training and guidance improve outcomes Some advanced models benefit less from complex prompting; task–model fit matters Metrics vary widely; subjective and user-centered measures are still developing
[2], [4], [12] [5], [14]
Automated prompt optimization can exceed humancrafted prompts Non-experts struggle without guidance Prompt effectiveness varies by model/task No universal framework exists for evaluating prompt success
[7] [8], [16] [17] [18]
direct prompt as a normal user might when first asking the model for help. The raw prompt is intentionally basic and may omit task-specific details. The second condition is the Checklist-Improved Prompt. In this condition, the raw prompt is rewritten using a short clarity checklist. The checklist includes three parts: Roles/Rules: specify what role the model should take or what limits it should follow. • Context: explain who the output is for and why the task is being done. • Answer Format: define how the final answer should be structured. •
Fig. 1. Study design overview. For each task, a raw prompt is evaluated alongside a checklist-improved prompt and a clarifying-question prompt. Each prompt condition is tested across multiple LLMs, and the resulting outputs are scored using the same rubric.
Token usage is measured separately for each model. We use the OpenAI tokenizer for ChatGPT, the Claude tokenizer for Claude, and the Lunary tokenizer for Grok. For every trial, we recorded input tokens and output tokens so that total token usage could be compared across conditions. The study uses four task categories: summarization, planning, explanation, and coding. These categories were chosen because they represent common open-ended LLM use cases. We used the following raw prompts: (1) “Summarise this: [Abstract of https://arxiv.org/abs/2201. 11903]” (2) “Plan a vacation in Europe” (3) “Explain this: [Abstract of https://arxiv.org/abs/2201. 11903]” (4) “Generate code for user input.” The exact prompt templates used for each task and condition are provided in Appendix A. Across conditions, the underlying task content was held fixed so that differences could be attributed to prompt structure rather than task changes. We compared three prompt conditions. The first is Raw Prompt. This is the baseline condition. A human writes a short,
We chose these three parts because roles or rules define the expected behavior of the model, context reduces ambiguity about the task purpose, and the answer format makes the desired output structure explicit. These choices are also consistent with prior work showing that instruction-following improves with clearer prompts. For the Clarifying-Question Prompt, the model does not answer immediately. Instead, it first asks 1–3 clarifying questions. After the user answers those questions, the model gives its final response. This condition tests whether asking for missing information improves the final response, while also accounting for the extra interaction cost introduced by clarification. For each trial, we recorded the trial ID, task completion score, correctness score, compliance score, clarity score, total rubric score, interpretation label, input tokens, output tokens, and total turns to acceptance. For reference, we define a turn as one user message followed by one model reply. In the clarifying condition, the clarification exchange counts toward the total number of turns because it is part of the actual interaction cost. An output is marked as accepted when the evaluator decides it is good enough to use without further major revision. It is important to note that acceptance was based on the evaluator’s judgment rather than an external user study. The interaction effort is measured through total turns to acceptance, input tokens, and output tokens. We use turnsto-acceptance as the main time-like measure. This measure captures the amount of back-and-forth required, not the actual elapsed time of the interaction. The primary outcomes in this study are output quality and interaction effort. Output quality
TABLE II U NIFIED QUALITY SCORING FRAMEWORK AND FINAL SCORE MAPPING .
Dimension Task Completion
Score 0 1 2
Correctness
0 1 2
Compliance
0 1 2
Clarity
0 1 2
Total Score 0–1 2–3 4–5 6–7 8
Meaning Did not complete, or missed the main purpose. Partly completed, but important parts are missing. Fully completed as requested. Contains major errors, or unreliable output. Mostly correct, but has weak reasoning. Correct, sound, and reliable. Ignored the mentioned requirements. Followed some, but missed important constraints. Followed the stated constraints well. Poorly structured, or needs major rewriting. Understandable, but needs moderate editing. Usable, and needs only minor or no editing.
Final Score
Interpretation
1 2 3 4 5
Complete failure Poor Acceptable Strong Flawless
is scored using four dimensions: task completion, correctness, compliance, and clarity. Each dimension receives a score from 0 to 2, producing a total rubric score from 0 to 8. The total rubric score is the main quality measure used in the aggregate, model-level, and task-level results. We then map the total score to a five-level interpretation: complete failure, poor, acceptable, strong, or flawless. For analysis, we compared the three prompt conditions pairwise: Raw vs. Checklist, Raw vs. Clarifying, and Checklist vs. Clarifying. For each comparison, we examined the mean and median quality scores, mean and median turns-to-acceptance, token usage, and acceptance patterns across tasks and models. The primary interpretation is that if a condition produces higher-quality results with fewer turns, that means it improves both result quality and interaction efficiency. If it improves quality but requires more turns, it may still be useful, though the benefit comes with an extra interaction cost. As descriptive robustness checks, we compared whether the same pattern appeared across ChatGPT, Claude, and Grok. Consistent patterns across models would make the findings more convincing. We also compared results across task cat-
TABLE III AGGREGATE RESULTS BY PROMPT CONDITION . S CORES ARE BASED ON THE 0–8 UNIFIED RUBRIC . Condition Raw Checklist Clarifying
Mean Score 5.67 7.50 6.67
Median Score 6.00 8.00 7.00
Mean Turns 1.00 1.00 1.96
Mean Tokens 962.25 683.42 936.50
TABLE IV M EAN RUBRIC SCORE BY MODEL AND PROMPT CONDITION .
Model ChatGPT Claude Grok
Raw 5.62 6.00 5.38
Checklist 7.88 7.25 7.38
Clarifying 7.12 7.38 5.50
egories. This helps us see whether better prompting matters more for some task types, such as planning or coding, than for others. We did not conduct a full ablation of individual checklist components in this study. Instead, we treat such ablations as future work for testing whether role/rules, context, or answer format contributes most to the observed gains. IV. R ESULTS The results show that checklist-improved prompts produced the strongest outputs overall, while raw prompts produced the lowest-quality outputs. Clarifying-question prompts improved over raw prompts in many cases, but they did not outperform checklist prompts when all results were combined. Table III shows that checklist prompts achieved the highest mean rubric score, 7.50 out of 8, while also using the fewest average tokens. Raw prompts had the lowest mean score, 5.67 out of 8, and the highest average token use. Clarifying prompts improved over raw prompts, with a mean score of 6.67 out of 8, but required almost twice as many turns on average. These results support H1: checklist-improved prompts produced higher-quality outputs than raw prompts. H2 was only partially supported. Clarifying-question prompts improved over raw prompts, but they did not outperform checklistimproved prompts overall. H3 was supported for checklist prompts but not for clarifying-question prompts. Checklist prompts achieved stronger outputs in a single turn, whereas clarifying prompts usually required an additional questionanswer exchange before the final response. Table IV gives the model-level breakdown. ChatGPT and Grok achieved their highest mean scores with checklist prompts, while Claude achieved a slightly higher mean score with clarifying prompts. However, checklist prompts still required fewer turns overall. Grok showed the clearest gap between checklist and clarifying prompts; in several Grok trials, the clarifying questions were relevant, but the final answers sometimes strayed from the original task. Table V shows that checklist prompts had the highest average score for planning, explanation, and coding. For summarization, checklist and clarifying prompts were tied.
1.83
278.83
Checklist - Raw
Checklist - Raw
1.00
25.75
Clarifying - Raw
Clarifying - Raw −2
0
2
4
Change in rubric score vs. Raw
(a) Change in rubric score relative to Raw.
−1000
0
1000
2000
3000
4000
Token savings vs. Raw
(b) Token savings relative to Raw.
Fig. 2. Paired effects relative to the raw-prompt baseline. Each faint point represents one matched model-task trial, the diamond marks the mean paired change, and the horizontal interval shows a bootstrap 95% confidence interval. Positive score changes indicate higher quality than Raw, while positive token savings indicate fewer tokens than Raw.
TABLE V M EAN RUBRIC SCORE BY TASK TYPE AND PROMPT CONDITION .
Task Type Summarization Explanation Planning Coding
Raw 6.00 6.50 5.50 4.67
Checklist 7.67 7.33 7.17 7.83
Clarifying 7.67 7.00 6.50 5.50
The largest improvement appeared in coding. Raw coding prompts are often overly open to interpretation, whereas checklist prompts clearly specify the language, task, constraints, and expected output. This helped the models produce cleaner and more usable code. Planning also benefited from the checklist condition, as it added missing details such as budget, travel style, food preferences, and format. The checklist condition gave the best tradeoff between quality and effort, with the highest average quality score and only one turn on average. The clarifying condition required almost two turns on average because the model first asked questions before giving the final answer. This extra step sometimes helped, but it also increased interaction time. Raw prompts also required one turn in our logged data. However, this should not be read as implying that raw prompts are equally efficient. In our experiment, raw prompts were not repeatedly revised after weak outputs. In real use, many raw outputs would likely require follow-up prompts because they were often incomplete, overly broad, or misaligned with the desired format. Token use should be interpreted carefully, as each model uses a different tokenizer. Therefore, token counts are most useful for comparing prompt conditions within this study rather than for making direct claims about absolute cost across providers. Checklist prompts used more input tokens than raw prompts because they included more instructions. However, they often reduced output length by making the expected answer more specific. As a result, checklist prompts
had the lowest average total token count in our collected results. For RQ1, structured prompts improved output quality across most models and task types, with the strongest gains for checklist prompts. For RQ2, checklist prompts showed the best balance between quality, token use, and interaction time. For RQ3, checklist prompts reduced interaction effort most clearly by producing strong outputs in one turn. Clarifying prompts may still help non-expert users who do not know what details to include, but in this study they added interaction cost and were less efficient than checklist prompts. Overall, checklist prompting was the strongest strategy among the three conditions tested. It improved output quality, reduced unnecessary interaction, and produced more stable results across models and tasks. Even basic guidance about role or rules, context, and answer format can make the model’s response more useful and easier to accept. V. L IMITATIONS AND F UTURE W ORK This study has several limitations. First, it evaluates a limited number of tasks, prompt conditions, and LLM systems. Therefore, the results should not be treated as a general conclusion for all prompting situations. Instead, they provide an early empirical comparison of how structured prompts may affect LLM output quality and user effort. Second, the selected benchmark tasks may not fully transfer to more complex real-world or enterprise settings. Summarization, explanation, planning, and coding are common LLM tasks, but they do not cover all types of human-AI interaction. Third, the quality scores are based on author evaluation using a rubric designed for this study. This helped keep scoring consistent, but it also introduces subjectivity because we did not use independent blind raters or report inter-rater reliability. The rubric also does not fully capture subjective factors such as user satisfaction, preferred writing style, or perceived usefulness. These factors may affect whether a real user accepts an LLM output.
Another limitation is that each LLM system was handled by one human operator. This kept data collection organized, but it may also introduce operator-specific variation in prompting, scoring, or judgment. Future work should use shared scoring procedures or multiple independent evaluators across all models. Future work should evaluate the same prompt conditions with external human participants, for example, through platforms such as Prolific, which would allow the study to measure user-centered judgments of output quality, effort, and satisfaction. This would test whether the patterns observed with author scoring also hold for independent users. Such a study would require Institutional Review Board approval before participant recruitment. Future work should also use blind pairwise ratings, where evaluators compare two answers for the same task without knowing which prompt condition produced each answer. Another useful direction is to test individual checklist components, such as role/rules, context, and answer format, one at a time. This would help identify which checklist components contribute most to quality improvement and effort reduction. VI. C ONCLUSION This paper compared raw prompts, checklist-improved prompts, and clarifying-question prompts across four task types and three LLM systems. Checklist-improved prompts produced the strongest overall performance, achieving the highest mean rubric score while requiring only one turn on average. Clarifying-question prompts improved over raw prompts, but their benefits were less consistent and usually required extra interaction. These findings suggest that simple prompt structure can help users obtain better LLM outputs with less back-and-forth. R EFERENCES [1] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” arXiv preprint arXiv:2201.11903, 2022. [2] B. Chen, Z. Zhang, N. Langrené, and S. Zhu, “Unleashing the potential of prompt engineering for large language models,” Patterns, vol. 6, 2023. [3] B. Meskó, “Prompt engineering as an important emerging skill for medical professionals: Tutorial,” Journal of Medical Internet Research, vol. 25, 2023. [4] W. Cain, “Prompting change: Exploring prompt engineering in large language model ai and its potential to transform education,” TechTrends, vol. 68, pp. 47 – 57, 2023. [5] R. K. Anam, “Prompt engineering and the effectiveness of large language models in enhancing human productivity,” ArXiv, vol. abs/2507.18638, 2025. [6] P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” ArXiv, vol. abs/2402.07927, 2024. [7] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” ArXiv, vol. abs/2211.01910, 2022. [8] J. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, “Why johnny can’t prompt: How non-ai experts try (and fail) to design llm prompts,” Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023. [9] M. Desmond and M. Brachman, “Exploring prompt engineering practices in the enterprise,” ArXiv, vol. abs/2403.08950, 2024.
[10] A. Mishra, B. Danzy, U. Soni, A. Arunkumar, J. Huang, B. C. Kwon, and C. Bryan, “Promptaid: Visual prompt exploration, perturbation, testing and iteration for large language models,” IEEE Transactions on Visualization and Computer Graphics, vol. 31, pp. 6946–6962, 2025. [11] X. Tang, H. Chen, D. Lin, and K. Li, “Harnessing llms for multidimensional writing assessment: Reliability and alignment with human judgments,” Heliyon, vol. 10, 2024. [12] L. Jacobsen and K. E. Weber, “The promises and pitfalls of large language models as feedback providers: A study of prompt engineering and the quality of ai-driven feedback,” AI, 2025. [13] S. Nayab, G. Rossolini, G. Buttazzo, N. Manes, and F. Giacomelli, “Concise thoughts: Impact of output length on llm reasoning and cost,” ArXiv, vol. abs/2407.19825, 2024. [14] T. S. Kim, Y. Lee, J. Shin, Y.-H. Kim, and J. Kim, “Evallm: Interactive evaluation of large language model prompts on user-defined criteria,” Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2023. [15] S. Atreja, J. Ashkinaze, L. Li, J. Mendelsohn, and L. Hemphill, “Prompt design matters for computational social science tasks but in unpredictable ways,” pp. 122–145, 2024. [16] N. Knoth, A. Tolzin, A. Janson, and J. Leimeister, “Ai literacy and its implications for prompt engineering strategies,” Comput. Educ. Artif. Intell., vol. 6, p. 100225, 2024. [17] G. Wang, Z. Sun, S. Ye, Z. Gong, Y. Chen, Y. Zhao, Q.-L. Liang, and D. Hao, “Do advanced language models eliminate the need for prompt engineering in software engineering?” ACM Transactions on Software Engineering and Methodology, 2024. [18] A. Mudrik, G. Nadkarni, O. Efros, S. Soffer, and E. Klang, “Prompt engineering in large language models for patient education: A systematic review,” 2025. [19] Q. Ma, W. Peng, C. Yang, H. Shen, K. Koedinger, and T. Wu, “What should we engineer in prompts? training humans in requirement-driven llm use,” ACM Transactions on Computer-Human Interaction, vol. 32, pp. 1 – 27, 2024. [20] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” arXiv preprint arXiv:2203.02155, 2022.
A PPENDIX A E XPERIMENTAL P ROMPT T EMPLATES This appendix reports the exact prompt templates used in each experimental condition. A. Summarization 1) Raw Prompt: Summarise this. [PAPER ABSTRACT HERE]
2) Checklist-Improved Prompt: Summarise the abstract below for a smart non-expert CS student. Use simple language. Keep the summary under 100 words. Focus on the main idea and why it matters. Write the answer as one short paragraph only, with no bullet points or headings. [PAPER ABSTRACT HERE]
3) Clarifying-Question Prompt: Before answering, ask me exactly 3 short clarifying questions so you can write a better summary of the abstract below. Do not work on the task until I reply. Task: Summarize this. [PAPER ABSTRACT HERE]
B. Explanation 1) Raw Prompt:
TABLE VI C OMPACT TRIAL - LEVEL EVALUATION SUMMARY. S CORES ARE TOTAL RUBRIC SCORES ON THE 0–8 SCALE .
Explain this. [PAPER ABSTRACT HERE]
2) Checklist-Improved Prompt: Explain the abstract below for a first-year graduate student. Use simple but technically correct language. Focus on the main idea and why the chain-of-thought helps. Write the answer in 2 short paragraphs only, with no bullet points or headings. [PAPER ABSTRACT HERE]
3) Clarifying-Question Prompt: Before answering, ask me exactly 3 short clarifying questions so you can write a better explanation of the abstract below. Do not work on the task until I reply. Task: Explain this. [PAPER ABSTRACT HERE]
C. Planning 1) Raw Prompt: Plan a vacation in Europe.
2) Checklist-Improved Prompt: Plan a 7-day vacation in Europe for 2 adults. Keep the total budget around $2500, excluding international flights. Focus on art, walkable cities, and vegetarian-friendly food. Avoid any plan that requires driving. Write the answer as a day-by-day itinerary and include a rough budget breakdown.
3) Clarifying-Question Prompt: Before answering, ask me exactly 3 short clarifying questions so you can create a better travel plan. Do not work on the task until I reply. Task: Plan a vacation in Europe.
D. Coding 1) Raw Prompt: Generate code for user input.
2) Checklist-Improved Prompt: Write Python code that prompts the user for a string. Check whether the string is a palindrome. Ignore spaces and letter case when checking. Print a clear result for the user. Write clean, runnable code only.
3) Clarifying-Question Prompt: Before answering, ask me exactly 3 short clarifying questions so you can write better code for this task. Do not work on the task until I reply. Task: Generate code for user input.
A PPENDIX B T RIAL -L EVEL E VALUATION
Trial ID Summarize1 Grok Raw Summarize1 Grok Checklist Summarize1 Grok Clarifying Explain1 Grok Raw Explain1 Grok Checklist Explain1 Grok Clarifying Plan1 Grok Raw Plan1 Grok Checklist Plan1 Grok Clarifying Code1 Grok Raw Code1 Grok Checklist Code1 Grok Clarifying Summarize2 Grok Raw Summarize2 Grok Checklist Summarize2 Grok Clarifying Explain2 Grok Raw Explain2 Grok Checklist Explain2 Grok Clarifying Plan2 Grok Raw Plan2 Grok Checklist Plan2 Grok Clarifying Code2 Grok Raw Code2 Grok Checklist Code2 Grok Clarifying Summarize1 Claude Raw Summarize1 Claude Checklist Summarize1 Claude Clarifying Explain1 Claude Raw Explain1 Claude Checklist Explain1 Claude Clarifying Plan1 Claude Raw Plan1 Claude Checklist Plan1 Claude Clarifying Code1 Claude Raw Code1 Claude Checklist Code1 Claude Clarifying Summarize2 Claude Raw Summarize2 Claude Checklist Summarize2 Claude Clarifying Explain2 Claude Raw Explain2 Claude Checklist Explain2 Claude Clarifying Plan2 Claude Raw Plan2 Claude Checklist Plan2 Claude Clarifying Code2 Claude Raw Code2 Claude Checklist Code2 Claude Clarifying Summarize1 ChatGPT Raw Summarize1 ChatGPT Checklist Summarize1 ChatGPT Clarifying Plan1 ChatGPT Raw Plan1 ChatGPT Checklist Plan1 ChatGPT Clarifying Explain1 ChatGPT Raw Explain1 ChatGPT Checklist Explain1 ChatGPT Clarifying Code1 ChatGPT Raw Code1 ChatGPT Checklist Code1 ChatGPT Clarifying Summarize2 ChatGPT Raw Summarize2 ChatGPT Checklist Summarize2 ChatGPT Clarifying Explain2 ChatGPT Raw Explain2 ChatGPT Checklist Explain2 ChatGPT Clarifying Plan2 ChatGPT Raw Plan2 ChatGPT Checklist Plan2 ChatGPT Clarifying Code2 ChatGPT Raw Code2 ChatGPT Checklist Code2 ChatGPT Clarifying
Score
Turns
Input Tokens
Output Tokens
5 7 7 5 8 6 4 8 3 3 8 3 6 7 7 8 5 5 6 8 7 6 8 6 6 8 8 6 8 8 6 5 8 5 8 8 7 8 8 8 7 8 7 7 7 3 7 4 5 8 8 5 8 8 5 8 7 5 8 8 7 8 8 7 8 8 5 7 6 6 8 4
1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 1 1 1 2
21 74 144 21 67 139 6 67 95 6 45 134 22 58 36 5 46 18 21 54 35 6 50 19 19 49 125 17 64 126 5 48 120 5 41 107 19 54 36 5 48 20 18 65 35 6 51 21 18 52 43 5 43 33 16 48 41 5 50 35 19 52 33 5 43 20 16 56 31 6 50 20
212 146 148 702 225 814 1472 1492 1057 4676 179 470 586 175 956 1253 943 1318 750 1318 912 598 862 1244 339 629 1074 778 416 637 548 260 799 4874 3891 3260 369 311 370 1165 354 707 607 411 775 164 860 353 289 208 824 578 506 746 660 270 972 276 338 849 351 179 305 519 481 697 930 332 1233 106 341 490