From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models

From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models | medRxiv Skip to main content Home About Submit ALERTS / RSS Search for this keyword Advanced Search From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models View ORCID Profile Hanjae Kim , View ORCID Profile Minseong Kim , View ORCID Profile Seonji Kim , View ORCID Profile Seng Chan You doi: https://doi.org/10.64898/2026.03.13.26348306 Hanjae Kim 1 Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul, Republic of Korea 2 Yonsei Institute for Digital Health, Yonsei University , Seoul, Republic of Korea 3 Institute for AI and Social Innovation, Yonsei University , Seoul, Republic of Korea B.S. Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Hanjae Kim Minseong Kim 1 Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul, Republic of Korea B.S. Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Minseong Kim Seonji Kim 1 Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul, Republic of Korea 2 Yonsei Institute for Digital Health, Yonsei University , Seoul, Republic of Korea Ph.D. Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Seonji Kim Seng Chan You 1 Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul, Republic of Korea 2 Yonsei Institute for Digital Health, Yonsei University , Seoul, Republic of Korea 3 Institute for AI and Social Innovation, Yonsei University , Seoul, Republic of Korea M.D., Ph.D. Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Seng Chan You For correspondence: chandryou{at}yuhs.ac Abstract Full Text Info/History Metrics Preview PDF Abstract Introduction Implementing target trial emulation (TTE) study methods as end-to-end executable analytic code is technically demanding, and producing standardized, reproducible scripts consistently across research teams remains a persistent challenge. We aimed to develop a framework that translates free-text study descriptions into standardized analytic specifications and executable Strategus R scripts for the Observational Health Data Sciences and Informatics (OHDSI) ecosystem. Methods We developed THESEUS (Text-guided Health-study Estimation and Specification Engine Using Strategus), which operates through two sequential steps. Large language models (LLMs) first map descriptions of the study into a constrained JavaScript Object Notation (JSON) schema (standardization step), after which the structured specifications are converted into R scripts with a self-auditing loop for error correction (code generation step). We evaluated eight proprietary LLMs using texts extracted from the methods section of 15 OHDSI-based TTE studies, and externally validated the framework on texts from 5 non-OHDSI studies, across three input settings: primary analysis text only, full analyses text, and full methods sections. Standardization was evaluated at the study-level (whether all parameters in a study were correctly extracted) and at the field-level (sensitivity and false positive rate per individual parameter) with field-level evaluation applied to the full analyses text and full methods sections input settings. Code generation was assessed by executability of the produced R scripts before and after self-auditing. Results In the standardization step, study-level accuracy across models ranged from 0.91 to 0.98 for primary analysis, 0.67 to 0.87 for full analyses, and 0.67 to 0.85 for full methods sections in OHDSI studies, whereas the corresponding ranges were 0.73 to 0.93, 0.60 to 0.87, and 0.27 to 0.47 in non-OHDSI studies. At the field-level, sensitivity across models under the full analyses text input setting ranged from 0.73 to 0.90 with 0.27 to 0.67 false positives per study in OHDSI studies, and from 0.71 to 0.90 with 0.20 to 1.00 false positives per study in non-OHDSI studies, depending on input setting. For code generation, first-run executability ranged from 0.80 to 1.00 for OHDSI studies and improved to 0.93 to 1.00 after self-auditing. In non-OHDSI studies, first-run executability ranged from 0.60 to 1.00, improving to 1.00 after self-auditing. Discussion THESEUS demonstrates that pairing a standardized data model with a structured analysis framework enables reliable LLM-powered automation of the coding step in observational research. THESEUS supports the reliable translation of natural-language study descriptions into executable, shareable code in standardized observational research settings. This approach has the potential to lower the technical barriers to participation in observational research for a broader range of investigators. Competing Interest Statement SCY reports grants from Daiichi Sankyo. He is a coinventor of granted Korea Patent DP-2023‐1223 and DP-2023‐0920, and pending Patent Applications DP-2024‐0909, DP-2024‐0908, DP-2022‐1658, DP-2022‐1478, and DP-2022‐1365 unrelated to current work. SCY is a chief executive officer of PHI Digital Healthcare. Other authors have no potential conflicts of interest to disclose. Funding Statement This study did not receive any funding Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes Figure 1, Figure 2, Figure 3, Figure 4 revised Copyright The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license . Back to top Previous Next Posted March 19, 2026. Download PDF Email Thank you for your interest in spreading the word about medRxiv. NOTE: Your email address is requested solely to identify you as the sender of this article. Your Email * Your Name * Send To * Enter multiple addresses on separate lines or separate them with commas. You are going to email the following From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models Message Subject (Your Name) has forwarded a page to you from medRxiv Message Body (Your Name) thought you would like to see this page from the medRxiv website. Your Personal Message CAPTCHA This question is for testing whether or not you are a human visitor and to prevent automated spam submissions. Share From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models Hanjae Kim , Minseong Kim , Seonji Kim , Seng Chan You medRxiv 2026.03.13.26348306; doi: https://doi.org/10.64898/2026.03.13.26348306 Share This Article: Copy Citation Tools From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models Hanjae Kim , Minseong Kim , Seonji Kim , Seng Chan You medRxiv 2026.03.13.26348306; doi: https://doi.org/10.64898/2026.03.13.26348306 Citation Manager Formats BibTeX Bookends EasyBib EndNote (tagged) EndNote 8 (xml) Medlars Mendeley Papers RefWorks Tagged Ref Manager RIS Zotero Tweet Widget Facebook Like Google Plus One Subject Area Health Informatics Subject Areas All Articles Addiction Medicine (638) Allergy and Immunology (917) Anesthesia (348) Cardiovascular Medicine (4983) Dentistry and Oral Medicine (493) Dermatology (426) Emergency Medicine (675) Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1685) Epidemiology (16226) Forensic Medicine (36) Gastroenterology (1240) Genetic and Genomic Medicine (7261) Geriatric Medicine (751) Health Economics (1093) Health Informatics (5273) Health Policy (1464) Health Systems and Quality Improvement (1826) Hematology (599) HIV/AIDS (1378) Infectious Diseases (except HIV/AIDS) (16472) Intensive Care and Critical Care Medicine (1199) Medical Education (698) Medical Ethics (156) Nephrology (749) Neurology (7563) Nursing (377) Nutrition (1122) Obstetrics and Gynecology (1295) Occupational and Environmental Health (1038) Oncology (3736) Ophthalmology (1089) Orthopedics (407) Otolaryngology (465) Pain Medicine (499) Palliative Medicine (143) Pathology (720) Pediatrics (1854) Pharmacology and Therapeutics (754) Primary Care Research (785) Psychiatry and Clinical Psychology (6110) Public and Global Health (9968) Radiology and Imaging (2517) Rehabilitation Medicine and Physical Therapy (1511) Respiratory Medicine (1267) Rheumatology (657) Sexual and Reproductive Health (801) Sports Medicine (595) Surgery (800) Toxicology (109) Transplantation (318) Urology (307)

Related documents

• From study design to executable code: automating target trial emulation with large language models.#358269
• A Comprehensive Survey of Prompt Engineering Techniques in Large Language Models#455566
• EXPLORING HUMAN FACTORS IN LARGE LANGUAGE MODEL-ASSISTED SOFTWARE REQUIREMENTS SPECIFICATION#191278
• Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics#120589
• Intent-Driven Mobile GUI Testing with Autonomous Large Language Model Agents#433632