Reclaiming informed consent to train mental health AI with patient data.

Reclaiming informed consent to train mental health AI with patient data - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice letter NPJ Digit Med . 2026 Jun 2;9:423. doi: 10.1038/s41746-026-02843-8 Reclaiming informed consent to train mental health AI with patient data Carlos A Larrauri 1, ✉ , Nicole Martinez-Martin 2 , John Torous 3 Author information Article notes Copyright and License information 1 Executive and Continuing Education, Harvard T.H. Chan School of Public Health, Boston, MA USA 2 Department of Pediatrics, Stanford Center for Biomedical Ethics, Stanford, CA USA 3 Digital Psychiatry Division, Beth Israel Deaconess Medical Center, Boston, MA USA ✉ Corresponding author. Received 2025 Nov 13; Accepted 2026 May 25; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . PMC Copyright notice PMCID: PMC13230920  PMID: 42230776 Abstract Companies increasingly train AI models on therapy transcripts, often relying on terms of service that obscure data use to obtain assent. Because AI systems generate inferences from data and enable unforeseen secondary uses, existing consent models do not adequately protect autonomy and confidentiality. We argue for (1) separate and explicit opt-in consent to train on patient data and (2) patient-led governance to prevent the recurrence of exploitation in medical innovation. Subject terms: Complex networks; Complex networks; Health care; Science, technology and society; Scientific community; Social sciences When researchers obtained Henrietta Lacks’ cells without her consent in 1951, they transformed medicine while also creating a paradigmatic example of biomedical progress built on exploitation 1 . Today, we risk repeating a similar mistake, not with tissue, but with patients’ digital data. Companies developing artificial intelligence (AI) for mental health are using digital therapy transcripts of conversations with both human and AI therapists to train their models, often relying on lengthy terms of service to obtain assent. For instance, recent reporting from STAT News suggested that Lyra Health uses deidentified therapy transcripts to develop its new AI chatbot, which will be marketed to employers and health systems 2 . Lyra is not alone; other digital mental health platforms, including Headspace Health, Spring Health, and Talkspace, are similarly integrating generative AI into their services 3 . Details regarding the sources of their training data remain unclear. Just as HeLa cells became indefinitely reproducible, generating scientific value across contexts in ways Henrietta Lacks or researchers could never have anticipated, patient data achieves a similar “immortality” once incorporated into AI systems, yet with even greater reach. Unlike a discrete tissue sample, patient data can be combined with other datasets to generate new inferences and create reidentification risks (linking data back to specific individuals), making downstream uses difficult to foresee or control. Henrietta Lacks’ story ultimately catalyzed significant reforms in research ethics and patient rights, and the current moment demands a comparable reckoning with the human costs of innovation, given that millions of people have likely already lost control over their most personal data. Reclaiming informed consent, therefore, requires an AI governance framework that ensures separate and explicit opt-in consent and patient-led governance to prevent the recurrence of historical patterns of exploitation in medical innovation. Why contractual assent is not informed consent The therapeutic relationship is traditionally governed by professional ethics and legal duties 4 that protect patient autonomy and confidentiality. Most patients in brick-and-mortar settings would not expect their private therapy sessions to be recorded; however, audio or video recordings of individual therapy sessions are occasionally made for the training or evaluation of human therapists, usually less than one session per year. Such recording requires explicit, direct, and prospective informed consent, with additional consent forms specifically for that purpose, and participants may rescind their consent at any time 5 . Unlike traditional therapy, digital mental health platforms can—and often do—record all sessions to train and improve their AI models, under ethical and legal safeguards far weaker than those governing conventional therapeutic settings. For instance, on January 6, 2026, the FDA issued guidance documents on low-risk devices 6 and clinical decision support software 7 , suggesting that it will consider classifying more AI applications as “wellness” devices or software and thus outside the scope of formal regulation. Understanding what this means for patients requires distinguishing among three frameworks: (1) informed consent for clinical research, governed by principles emphasizing disclosure, voluntariness, and comprehension 8 ; (2) authorization regimes for secondary use of data, which dispense with individual consent by repurposing data beyond its original purpose through de-identification or institutional review board waivers in the United States 9 , or other legal grounds in Europe 10 ; and (3) contractual assent embedded in terms of service. While informed consent in clinical research is the most rigorous of these frameworks, it only documents that information was disclosed, not that it was necessarily understood. Secondary health data sharing outside of research contexts typically operates under far less stringent requirements. Digital mental health platforms largely rely on the third and weakest model, which requires users to click “I agree” to terms that few read or understand 11 , and often reduces complex data permissions into either a blank check or refusal 12 . Those relying on employer-sponsored platforms such as Lyra Health face an even starker choice: accept the terms or forgo care. But by opting in to get help, what is being forfeited? Informed consent requires information about both benefits and risks, yet AI complicates this calculus because the use and implications of the data collected can extend far beyond the immediate encounter 13 . Large language models (the AI systems underlying these modern chatbots) enable secondary uses of data, making it difficult to anticipate how information may later be applied 14 . Moreover, the inferences drawn from the data and the user and group profiles that can be derived from those inferences pose additional risks to individuals and groups, even when the underlying data is not directly identifiable 15 . For instance, employers or insurers could use collected data and resulting profiles to identify debilitating or costly conditions, thereby facilitating discrimination in employment or coverage decisions that patients did not anticipate when seeking care 16 . This risk may be further heightened where companies developing AI chatbots are contracted directly by the very employers whose workforces they serve. Chatbot therapy faces regulatory gaps and conflicted incentives AI holds significant promise for improving psychiatric care, including enhancing access, augmenting clinical decision-making, and improving early risk detection 17 . Yet, current legal regimes inadequately protect patient data. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) protects personal health information (PHI) collected in clinical encounters 18 , but many digital mental health platforms do not qualify as “covered entities” under HIPAA. As a result, regulators and private litigants must rely on a patchwork of authorities to address misleading or undisclosed data practices and privacy breaches, including Section 5 of the Federal Trade Commission Act, the Health Breach Notification Rule under the Health Information Technology for Economic and Clinical Health Act, state consumer protection and privacy laws, and federal and state wiretapping statutes. While the European Union’s General Data Protection Regulation (GDPR) provides broader baseline protections for personal data than the health sector-specific scope of HIPAA 19 , the EU has moved toward policies that harmonize secondary uses of health data without explicit consent. The GDPR already authorizes such processing for certain public interest, statistical, and scientific or historical research purposes, but member states implement these exceptions inconsistently. The European Health Data Space—an EU regulatory framework governing the cross-border sharing and reuse of personal health data—was initially proposed to permit standardized data sharing without consent, and only through late-stage advocacy were opt-out provisions secured 20 . The question of deidentification is central to this debate, yet current regimes treat it as a bright-line rule that eliminates privacy concerns. Once data is deidentified under HIPAA’s standards, it is no longer legally considered PHI 21 . The GDPR’s “right to erasure” offers stronger data protections in principle, extending to pseudonymized and inferred data if linked to an individual (but not de-identified data). However, commentators note that it loses practical force once personal data has been integrated into complex, periodically updated systems, where erasure may be technically ineffective and inferred data or trained models persist despite withdrawal of consent 22 . Moreover, deidentification is not necessarily permanent: reidentification risk increases as public and commercial datasets proliferate and become linked to other sources 23 . Given these regulatory shortcomings, companies are best positioned to prevent harm, yet market dynamics that reward rapid scaling and data accumulation also create structural incentives to collect user data 24 , thereby risking the repetition of historical patterns in which marginalized populations contributed to medical progress without sharing its benefits or protections 25 . Reclaiming informed consent with separate and explicit opt-in consent and patient-led governance Digital mental health platforms do not deviate radically from prevailing data practices; however, patient data in the mental health context is often generated during episodes of vulnerability and distress and is highly stigmatized. These features justify heightened consent standards, even if similar practices in other sectors remain subject to debate. Moreover, AI enables the collection and processing of far more data than people are aware of, a trend likely to continue as newer models integrate facial expressions, wearables, fitness data, voice sampling, and other potential mental health biomarkers. Informed consent must therefore transition from a procedural checkbox to an ongoing process, anchored in accountability and transparency, giving users genuine control over their data. Dynamic consent achieves this shift through a digital interface that enables participants to modify consent preferences over time while maintaining communication about data use and outcomes 26 . Related consent models, including meta consent—where individuals choose their preferred consent method for future data uses—offer an additional model for balancing autonomy with practical feasibility 27 . We propose two complementary policies: one procedural and one structural. First, entities with authority to shape digital mental health practices, such as state and federal regulators and enforcement agencies, payers, and health systems, should mandate separate and explicit opt-in consent for the use of patient data in AI model training, clearly distinguished from consent to receive treatment. This consent documentation must explain, in accessible language, what data companies will use and derive, for what developmental purposes (e.g., developing a new chatbot), and through what commercial arrangements. While companies and health systems often argue that routine clinical care occurs under general consent—a broad, blanket authorization for treatment and associated operational data uses—and that consent for downstream data uses is infeasible and would hamper innovation, patient data derived from therapy transcripts is distinct from routine clinical care. Second, communities whose data fuel these systems should be included in efforts to define their ethical boundaries and consent standards. AI governance models must redistribute authority and accountability to ensure sustained trust and legitimacy, with leadership roles establishing patient and community oversight. Patient advocacy organizations, such as the National Alliance on Mental Illness (NAMI), are well positioned to assist in embedding patient representation on governance boards 28 , algorithmic audit panels 29 , and data ethics review boards 30 , thereby transforming consent from a one-time transaction into an ongoing co-design process 31 . In doing so, mental health AI could evolve from its current extractive data relations to a justice-oriented framework for the governance and use of patient data 32 . Conclusion Today, the patients who can most benefit from additional mental health support are also the least likely to read or understand complex terms of service—and as with HeLa cells, data may already be extracted from communities that will be the last to access the innovations their data helped create. Although separate and explicit opt-in consent may create barriers for some users, and patient-led governance may slow innovation or face industry opposition, the digital mental health sector risks repeating a shameful chapter of progress built on exploitation without such policies. The challenge, therefore, is not to halt innovation or abandon informed consent, but to ensure that progress advances with both individual choice and accountability to the communities most affected. Protecting autonomy and confidentiality must remain a foundation for ethical innovation, not an externality to be managed after the fact. Author contributions Conceptualization: C.A.L., N.M.-M., and J.T.; Writing & editing: C.A.L., N.M.-M., and J.T. All authors have read and approved the manuscript. Data availability No datasets were generated or analysed during the current study. Competing interests C.A.L. has received payments for consulting services from Klick Health, Alkermes plc, K Health, Inc., Janssen Pharmaceuticals, Inc. (a Johnson & Johnson company), Discern Health (now part of Avalere Health), Neurocrine Biosciences, Inc., and Boehringer Ingelheim International GmbH; presentation and speaker fees from Merck & Co., Inc., Biogen Inc., Teva Pharmaceutical Industries Ltd., Intra-Cellular Therapies, Inc., Karuna Therapeutics, Inc. (a Bristol Myers Squibb Company), Cerevel Therapeutics Holdings, Inc., and Center for Patient Advocacy Leaders (Pfizer Inc.); travel support from Boehringer Ingelheim International GmbH; and medical writing support from Boehringer Ingelheim International GmbH and Teva Pharmaceutical Industries Ltd. C.A.L. is a member on a data safety monitoring board (DSMB) and clinical trial steering committee (CTSC) for Zucker Hillside Hospital and a Steering Committee co-chair at Accelerating Medicines Partnership Schizophrenia (AMP SCZ). N.M.-M. reports no financial or non-financial competing interests. J.T. serves on the editorial board of npj Digital Medicine . He is an adviser to Boehringer Ingelheim. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Beskow, L. M. Lessons from HeLa cells: the ethics and policy of biospecimens. Annu. Rev. Genom. Hum. Genet. 17 , 395–417 (2016). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Aguilar, M. Lyra launches ‘clinical-grade’ chatbot amid growing concern about mental health and AI. STAT News (2025). https://www.statnews.com/2025/10/14/lyra-health-ai-chatbot-mental-health/ . 3. Becker, R. First-generation mental health apps look to add next-gen AI chatbots. STAT News (2025). https://www.statnews.com/2025/12/01/ai-chatbot-headspace-talkspace-lyra-sondermind-digital-mental-health/ . 4. Fisher, M. A. Laws affecting confidentiality. In The Ethics of Conditional Confidentiality: A Practice Model for Mental Health Professionals (Oxford Univ. Press, 2013). 5. Barnett, J. E. The role of recording in psychotherapy. Society for the Advancement of Psychotherapy (2009). https://societyforpsychotherapy.org/ask-the-ethicist-the-role-of-recording-in-psychotherapy/ . 6. Food and Drug Administration. General Wellness: Policy for Low Risk Devices . https://www.fda.gov/media/90652/download (2026). 7. Food and Drug Administration. Clinical Decision Support Software . https://www.fda.gov/media/109618/download (2026). 8. US Department of Health and Human Services, Office for Human Research Protections. Informed consent FAQs . https://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/informed-consent/index.html (2026). 9. Chevrier, R., Foufi, V., Gaudet-Blavignac, C., Robert, A. & Lovis, C. Use and understanding of anonymization and de-identification in the biomedical literature: scoping review. J. Med. Internet Res. 21 , e13484 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Becker, R. et al. Purpose definition as a crucial step for determining the legal basis under the GDPR: implications for scientific research. J. Law Biosci. 11 , lsae001 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 11. Su, W. et al. Assessing the readability of app descriptions and investigating its role in the choice of mHealth apps: retrospective and prospective analyses. AMIA Annu. Symp. Proc. 2021 , 1139–1148 (2022). [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Froomkin, A. M. Big data: destroyer of informed consent. Yale J. Law Technol 21 , 27–52 (2019). [ Google Scholar ] 13. Pham, T. Ethical and legal considerations in healthcare AI: innovation and policy for safe and fair use. R. Soc. Open Sci. 12 , 241873 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Moulaei, K., Akhlaghpour, S. & Fatehi, F. Patient consent for the secondary use of health data in artificial intelligence (AI) models: A scoping review. Int. J. Med. Inform . 198 , 105872 (2025). [ DOI ] [ PubMed ] 15. Wachter, S. & Mittelstadt, B. A right to reasonable inferences: rethinking data protection law in the age of big data and AI. Colum. Bus. Law Rev. 2019 , 494–620 (2019). [ Google Scholar ] 16. Price, W. N. & Cohen, I. G. Privacy in the age of medical big data. Nat. Med. 25 , 37–43 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Hua, Y. et al. A scoping review of large language models for generative tasks in mental health care. npj Digit. Med 8 , 230 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 18. Rezaeikhonakdar, D. AI chatbots and challenges of HIPAA compliance for AI developers and vendors. J. Law Med. Ethics 51 , 988–995 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Pendse, S. R. et al. Advancing a consent-forward paradigm for digital mental health data. Nat. Mental Health 2 , 1298–1307 (2024). [ Google Scholar ] 20. Cervera de la Cruz, P., Lalova-Spinks, T. & Shabani, M. The European Health Data Space: an opportunity to strengthen citizen rights and engage citizens in health data governance. Front. Med. 12 , 1699941 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 21. U.S. Department of Health & Human Services. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule . https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html (2012). 22. Custers, B. & Vrabec, H. Tell me something new: data subject rights applied to inferred data and profiles. Comput. Law Secur. Rev. 52 , 105956 (2024). [ Google Scholar ] 23. Sweeney, L. et al. Re-identification risks in HIPAA Safe Harbor Data: a study of data from one environmental health study. Technol. Sci. 2017 , 2017082801 (2017). [ PMC free article ] [ PubMed ] [ Google Scholar ] 24. Carriere-Swallow, Y. & Haksar, V. The economics and implications of data: an integrated perspective. IMF Dept. Pap 2019 , 013 (2019). [ Google Scholar ] 25. Williams, M., Karim, W., Gelman, J. & Raza, M. Ethical data acquisition for LLMs and AI algorithms in healthcare. npj Digit. Med. 7 , 377 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 26. Muller, S. H., van Thiel, G. J., Mostert, M. & van Delden, J. J. Dynamic consent, communication and return of results in large-scale health data reuse: Survey of public preferences. Digit. Health 9 , 1–14 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 27. Ploug, T. & Holm, S. Meta consent: a flexible and autonomous way of obtaining informed consent for secondary research. BMJ 350 , h2146 (2015). [ DOI ] [ PubMed ] [ Google Scholar ] 28. DeCamp, M. et al. Patients’ views about patient engagement and representation in healthcare governance. J. Healthc. Manag 64 , 332–346 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 29. Liu, X. et al. The medical algorithmic audit. Lancet Digit. Health 4 , e384–e397 (2022). [ DOI ] [ PubMed ] [ Google Scholar ] 30. Azad, T. D., Warman, A., McGraw, D. & Saria, S. Lessons from Henrietta Lacks inform a transparency framework to catalyze generative artificial intelligence in medicine. npj Digit. Med 8 , 280 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 31. Quyoum, A. & Wong, M. Valuing lived experience and co-design solutions to counter racial inequality in data and algorithmic systems in UK’s digital services. Inf. Commun. Soc. 27 , 1793–1809 (2024). [ Google Scholar ] 32. Shaw, J. & Sekalala, S. Health data justice: Building new norms for health data governance. npj Digit. Med 6 , 30 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement No datasets were generated or analysed during the current study. Articles from NPJ Digital Medicine are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (346.3 KB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Reclaiming informed consent to train mental health AI with patient data#252115
• AI for Mental Health: A Design Science Approach#261624
• Mental Health and AI Confidentiality#691870
• Towards Mechanism‐Informed Treatments for Mental Health#213143
• Towards Mechanism-Informed Treatments for Mental Health.#209312