Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice J Biomed Inform . Author manuscript; available in PMC: 2026 Apr 11. Published in final edited form as: J Biomed Inform. 2026 Mar 27;178:105034. doi: 10.1016/j.jbi.2026.105034 Search in PMC Search in PubMed View in NLM Catalog Add to search A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning Cheng Peng Cheng Peng , PhD 1 Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA Find articles by Cheng Peng 1, * , Xinyu Dong Xinyu Dong , PhD 3 Selfii Co. Find articles by Xinyu Dong 3, * , Mengxian Lyu Mengxian Lyu , MS 1 Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA Find articles by Mengxian Lyu 1 , Daniel Paredes Daniel Paredes , MS 1 Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA Find articles by Daniel Paredes 1 , Yaoyun Zhang Yaoyun Zhang , PhD 3 Selfii Co. Find articles by Yaoyun Zhang 3, † , Yonghui Wu Yonghui Wu , PhD 1 Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA 2 Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, Florida, USA Find articles by Yonghui Wu 1, 2, † Author information Article notes Copyright and License information 1 Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA 2 Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, Florida, USA 3 Selfii Co. * Equal contribution CONTRIBUTORSHIP STATEMENT Cheng Peng: Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis, Data curation, Conceptualization Xinyu Dong: Writing – review & editing, Investigation, Formal analysis, Data curation, Conceptualization Mengxian Lyu: Writing – review & editing, Investigation, Data curation, Conceptualization, Validation Daniel Paredes: Data curation, Conceptualization, Validation, Software Yaoyun Zhang: Writing – review & editing, Investigation, Formal analysis, Data curation, Project administration, Funding acquisition, Supervision Yonghui Wu: Writing – review & editing, Investigation, Methodology, Project administration, Funding acquisition, Supervision † Co-corresponding author ✉ Corresponding author: Yaoyun Zhang, PhD, [email protected] , Yonghui Wu, PhD, 1889 Museum Road, 7 th Floor, Gainesville, FL, USA, 32611, [email protected] Issue date 2026 Jun. PMC Copyright notice PMCID: PMC13067140 NIHMSID: NIHMS2160505 PMID: 41905520 The publisher's version of this article is available at J Biomed Inform Abstract Background Natural language processing (NLP) is a key technology to extract important patient information from clinical narratives to support healthcare applications. The rapid development of large language models (LLMs) has revolutionized patient information extraction in the clinical domain, yet critical strategies for effectively adopting LLMs for optimal performance need further exploration. This study examines LLMs’ effectiveness in patient information extraction, focusing on LLM architectures, fine-tuning strategies, and multi-task instruction tuning techniques for developing robust and generalizable patient information extraction systems. Methods This study aims to explore key strategies of adopting LLMs for clinical concept and relation extraction tasks, including: (1) encoder-only or decoder-only LLMs, (2) prompt-based parameter-efficient fine-tuning (PEFT) algorithms, and (3) multi-task instruction tuning on few-shot learning performance. We benchmarked a suite of LLMs, including encoder-only LLMs (BERT, GatorTron) and decoder-only LLMs (GatorTronGPT, Llama 3.1, GatorTronLlama), across five widely used benchmarking datasets. We compared traditional full-size fine-tuning and prompt-based PEFT. We explored a multi-task instruction tuning framework that combines both tasks across four datasets to evaluate the zero-shot and few-shot learning performance using the leave-one-dataset-out strategy. Results For single-task clinical CE, the two decoder-only LLMs (Llama 3.1 and GatorTronLlama) achieved the best performance, with average F1 scores of 0.8964 and 0.8981, respectively, across the five datasets, outperforming other LLMs with average F1 improvement of 0.7~3.3%. Encoder-only LLMs with prompt-based learning outperformed those implemented using classification. For RE, the prompt-based PEFT strategy demonstrated remarkable performance, with an F1 improvement of up to 15.9% over traditional fine-tuning on all datasets. All three decoder-only LLMs outperformed encoder-only LLMs, increasing average F1 score by 1.8 to 6.6%, with GatorTronLlama achieving the best performance with an average F1 score of 0.8978. Multi-task instruction tuning showed remarkable improvements, boosting zero-shot and few-shot F1 scores by 1.1~37.8% compared to those without multi-task fine-tuning. Notably, generative LLMs with multitask instruction tuning using only 20% of the full dataset achieved similar performance comparable to the full-size fine-tuning (<0.005 in terms of F1 score). Conclusions Our findings support generative LLMs with PEFT as a cost-effective solution for patient information extraction. In addition, multi-task instruction tuning significantly improves the zero-shot and few-shot performance, contributing to better generalizability. This study provides practical guidelines to develop LLM-based scalable, adaptable, and high-performing patient information extraction systems. Keywords: Patient information extraction, Large language model, Clinical concept extraction, Clinical relation extraction, Instruction tuning 1. INTRODUCTION Clinical notes are a valuable component of electronic health records (EHRs), capturing critical clinical care information such as laboratory tests, diagnoses, treatments, and outcomes [ 1 , 2 ]. Information extraction (IE) is the key technology to extract critical patient information to support downstream healthcare applications, such as decision support and clinical trial matching[ 3 – 5 ]. IE comprises two fundamental subtasks, including clinical concept extraction (CCE), which identifies clinical concepts such as diseases, treatments, and symptoms[ 6 ], and clinical relation extraction (CRE), which identifies relationships between clinical concepts, such as drug and corresponding adverse-event[ 7 , 8 ]. IE has been extensively explored in previous studies and open challenges, such as the i2b2/n2c2 shared tasks between 2006 and 2012 [ 9 – 17 ]. Many rule-based, machine learning-based, and hybrid methods were developed. Previous studies have identified critical challenges of IE, including the complexity of clinical language, domain-specific vocabulary, and high annotation costs. More robust and scalable NLP methodologies are needed[ 18 ]. Recent breakthroughs in transformer-based large language models (LLMs) have revolutionized many clinical NLP tasks. Early stage models, including rule-based [ 19 ]) and traditional machine learning models [ 20 ] have limitations in generalizability; recent advancements in LLMs provide promising solutions. The original breakthrough of LLMs came from the encoder-only transformer models, such as BERT[ 21 ], which were later customized for biomedical and clinical applications through pre-training using biomedical and clinical corpora, contributing to domain-specific transformer models such as BioBERT[ 22 ], ClinicalBERT[ 23 ], and GatorTron[ 24 ]. While the encoder-only transformer models improved clinical IE through bidirectional, embedding-based text representation, the extraction task was still approached using a classification-based extraction, which still suffers from the lack of generalizability inherited from the traditional machine learning models [ 25 , 26 ]. After 2020, decoder-only transformer models with massive numbers of parameters became the mainstream, known as generative LLMs, such as ChatGPT, LLAMA [ 27 – 29 ], and GatorTronGPT [ 30 ]. Generative LLMs adopted prompt-based learning algorithms instead of classification, where human instructions were integrated to the input as additional information, i.e., prompts, to instruct generative LLMs generating correspondence answers following a text-to-text generation procedure. This offers a more flexible and efficient way to instruct machines to identify required information based on human instructions, thus to formulate multiple NLP tasks in a unified text-to-text generation framework. Another very unique advantage of generative LLMs is the few-shot and zero-shot learning capabilities[ 31 , 32 ], enabling them to achieve human-level language processing with a few, or even no labeled data, and greatly reducing the annotation cost. Fine-tuning is a critical technology to adopt LLMs for clinical IE [ 8 ]. Traditional fine-tuning of encoder-only LLMs requires the training of dataset-specific classification layers that require substantial labeled datasets [ 33 ], and the trained classification layer can be used for one task-specific type of IE. As LLMs typically have massive amounts of parameters, typically over billions, it is very expensive to update all parameters during fine-tuning. Parameter-efficient fine-tuning (PEFT), such as P-tuning[ 34 , 35 ] and Low-Rank Adaptation (LoRA)[ 36 ], were proposed to reduce the training cost by updating only a small proportion of parameters. Instruction tuning using multi-task data has also demonstrated better few-shot learning capabilities [ 32 , 37 ], compared with fine-tuning using a single task. Both encoder-only and decoder-only LLMs have been applied for clinical IE. However, most recent studies of generative LLMs mainly focused on free-text question answers (QA). Several critical gaps exist for clinical IE using LLMs. First, there is a lack of comprehensive comparison between encoder-only LLMs and decoder-only generative LLMs across diverse clinical IE tasks; the strengths and weaknesses of using both LLMs for IE are still not clear. Second, the practical trade-offs between traditional full-size fine-tuning and PEFT among LLM architectures have not been systematically evaluated, leaving practitioners without clear guidance on the selection of fine-tuning strategies. Third, the transfer learning ability of the LLMs to generalize to new datasets and annotation schemas with minimal data—a crucial capability for real-world deployment—remains a significant challenge. The potential of multi-task instruction tuning to specifically enhance the transfer learning and generalizability of LLMs has not been explored for clinical IE. To the best of our knowledge, this is the first study to compare both encoder-only and decoder-only LLMs for patient information extraction, focusing on model architectures, fine-tuning strategies, generalizability, and multi-task instruction tuning using extensive clinical data benchmark datasets. Specifically, this study presents a comprehensive empirical study evaluating both encoder-only and decoder-only LLMs for patient information extraction. Our contributions are: (1) provide a benchmark comparison of mainstream encoder-only LLMs (BERT, GatorTron) and decoder-only LLMs (GatorTronGPT, Llama 3.1, GatorTronLlama) for patient information extraction; (2) systematically evaluate the performance and efficiency of traditional fine-tuning versus prompt-based PEFT strategies; (3) examine multi-task instruction tuning in generative LLMs using a leave-one-dataset-out approach to assess the improvement of zero-shot and few-shot transfer learning to unseen datasets; (4) provide empirical guidance for selecting LLM architectures and fine-tuning strategies to develop scalable, adaptable, and high-performing patient information extraction systems. 2. METHODS 2.1. Overall framework This study evaluates two encoder-only LLMs and three decoder-only LLMs for patient information extraction from clinical narratives. We systematically explored the effect of model architecture, fine-tuning strategies, and the generalizability through multi-task instruction tuning. Figure 1 shows an overview of the study design. Figure 1. Open in a new tab An overview of the study design. PEFT: parameter-efficient fine-tuning. 2.2. Clinical NLP tasks This study focuses on two fundamental NLP tasks for patient information extraction, including clinical CE and clinical RE. Clinical CE , also known as named entity recognition (NER), is the task of identifying and categorizing spans of text that refer to predefined medical concepts, such as diseases, treatments, and medications. Clinical RE focuses on identifying and classifying semantic relationships (e.g., treatment for disease, drug causes adverse event) between the clinical concepts. We formulated this task as a relation classification problem conditioned on gold standard entity pairs. For each relation candidate, the model was provided with the clinical text and the marked spans of the subject and object entities. 2.3. Datasets This study utilized five widely used clinical benchmark datasets, including the 2010 i2b2[ 9 ], 2018 n2c2[ 15 ], 2022 n2c2[ 17 ], RadGraph[ 38 ], and an internal dataset from UF Health[ 39 ]. These corpora span diverse clinical domains, from general clinical notes to specialized areas like radiology and social determinants of health (SDoH), and originate from different institutions with diverse documentation styles and annotation schemas. 2010 i2b2 dataset The 2010 i2b2/VA challenge corpus contains de-identified discharge summaries and progress reports from multiple healthcare institutions. It was annotated for three core medical concept categories, including Problem , Test , and Treatment . 2018 n2c2 dataset This dataset was developed in Track 2 of the 2018 n2c2 shared tasks, focused on the extraction of medications and adverse drug events (ADEs) from clinical narratives. It was annotated for 9 categories of clinical concepts (drug, drug attributes, ADEs) and 8 categories of relations among drugs, drug-associated attributes, and ADEs. 2022 n2c2 dataset This dataset was developed in Track 2 of the 2022 n2c2 challenge, which focused on the extraction of Social Determinants of Health (SDoH) in social history sections from clinical notes. The 2022 n2c2 dataset consists of 5 categories of SDoH concepts (e.g., employment status, living status, and substance use) and 9 categories of SDoH-associated attribute concepts (e.g., strength, frequency and dosage), and 28 types of relations among SDoH concepts and SDoH-associated attributes (e.g., frequency of substance use). Radgraph dataset This dataset includes entities and relations in full-text chest X-ray radiology reports from two large-scale datasets, MIMIC-CXR and CheXpert. It is annotated with 4 entity types defined by the anatomical location and clinical observation status, and 3 types of relations between entities. UF Health dataset This dataset was developed from our previous study. It collected two disease cohorts from the University of Florida (UF) Health Integrated Data Repository (IDR), including a cancer cohort and an opioid use cohort, which identified 19 categories of SDoH concepts and 26 categories of relations between SDoH concepts from the clinical notes. Table 1 summarizes the five datasets, including their sources, domains, sizes, concept and relation types. By leveraging these diverse datasets, this study aims to provide a comprehensive evaluation of LLMs for patient information extraction, addressing the challenges of generalizability and robustness across different clinical domains, documentation styles, and annotation schemas. Table 1. Overview of five clinical datasets Dataset Domain Notes Concept types/count Relation types/ count Document length Source 2010 i2b2 Discharge summaries& progress notes 871 3/45,009 - ~ 2100 tokens 2010 i2b2/VA challenge 2018 n2c2 Discharge summaries 505 9/83,869 8/59,810 ~3000 tokens 2018 n2c2 shared task, medication-ADE track 2022 n2c2 Social History Annotated Corpus (SHAC) 1877 14/21,114 28/7,307 ~200 tokens 2022 n2c2 shared task RadGraph Radiology reports 600 4/17,345 3/12,898 ~300 tokens PhysioNet UFHealth Cancer&Opioid use patient cohorts 829 19/17,535 26/6,374 ~3500 tokens UF Health IDR Open in a new tab 2.4. LLM architectures This study investigates two mainstream LLM architectures, including encoder-only LLMs and decoder-only LLMs. 2.4.1. Encoder-only LLMs Encoder-only LLMs are trained using the encoder component of the transformer architecture, such as BERT. Encoder-only LLMs process the input text bidirectionally, attending to all tokens simultaneously to learn contextual representations. These models are typically pre-trained on a self-supervised objective like Masked Language Modeling (MLM), where the model learns to predict randomly masked tokens within a sentence. For clinical CE, researchers typically add a task-specific classification head (e.g., a linear layer) on top of the final hidden state. In this study, we evaluated four widely used encoder-only models, including BERT, ClinicalBERT, SpanBERT, and GatorTron. BERT (Sequence Length: 512): The first successful bidirectional transformer model that set up the foundation to train for pre-training of transformer models for many NLP tasks. ClinicalBERT (Sequence Length: 512): An encoder-only clinical model pre-trained based on BERT on clinical notes from the MIMIC-III database, serving as a widely adopted baseline for clinical NLP tasks. SpanBERT (Sequence Length: 512)[ 40 ]: An encoder-only model that extends BERT by masking contiguous text spans during pretraining and introducing a span-boundary objective to learn stronger span-level representations. GatorTron (Sequence Length: 512): An encoder-only clinical LLM adopted the BERT architecture that has been pre-trained on a large-scale corpus of over 90 billion words of clinical and biomedical text, including de-identified notes from the UFHealth system, making it a powerful clinical foundational LLM. 2.4.2. Decoder-only LLMs Decoder-only LLMs, also known as generative LLMs, are typically trained using the encoder component of the transformer architecture, also known as generative LLMs, such as the GPT. Decoder-only LLMs utilize a unidirectional (or autoregressive) transformer architecture. They are pre-trained on a Causal Language Modeling (CLM) objective, where the model learns to predict the next token in a sequence given all preceding tokens. A very unique characteristic of the decoder-only LLMs is that they can solve multiple NLP tasks using a unified text-to-text learning framework, which is not feasible for encoder-only LLMs. Instead of adding task-specific layers (i.e., encoder-only LLMs), generative LLMs are guided by human instructions to generate the desired answers. This study evaluated four decoder-only models, including GatorTronGPT, Meditron, Llama 3.1, and GatorTronLLAMA. GatorTronGPT (Sequence Length: 2,048): A generative clinical LLM developed in our previous work, which is pre-trained using 277 billion words of text comprising 82 billion words of clinical text and 195 billion words of diverse general English text. Meditron-7b (Sequence Length: 4,096)[ 41 ]: A clinical generative model adapted from Llama 2 through continued pre-training on a comprehensive medical corpus, serving as a decoder-only clinical LLM baseline. Llama 3.1 (Sequence Length: 8,192): A general-domain foundation model from Meta AI, was pretrained on a massive, diverse corpus, offering scalability and robust performance across generative tasks. GatorTronLlama (Sequence Length: 8,192): A specialized model that leverages the Llama 3.1–8B architecture and adapts it with deep clinical domain knowledge through continued pre-training using over 100 billion words of clinical text collected from UF Health. 2.5. LLM fine-tuning strategies Fine-tuning is a supervised learning algorithm to adopt pretrained LLMs for specific NLP tasks, such as patient information extraction, using a small set of annotated corpora. This study examines two fine-tuning strategies, including traditional full-size fine-tuning, where all parameters were optimized, and PEFT – only a very small proportion of parameters were updated. The choice of strategy was aligned with the model’s architecture, allowing for a robust comparison between established and modern adaptation techniques. 2.5.1. Traditional Full-size Fine-Tuning This approach was widely used to fine-tune encoder-only models (e.g., BERT and GatorTron). For clinical CE, a dataset-specific classification layer, which is a linear classification layer applied to each token’s final hidden state, was added to predict its concept label (e.g., B-Problem, I-Problem, O). For clinical RE, we first generate candidate concept pairs and use a binary classification head to determine whether the candidate pair has a relation or not. During training, the entire model, including all transformer layers and the new classification layers, is updated. This method is computationally expensive for LLMs with billions of parameters and requires large datasets to prevent overfitting. 2.5.2. Parameter-Efficient Fine-Tuning (PEFT) We adopted a prompt-based Machine Reading Comprehension (MRC) framework developed in our previous work to fine-tune encoder-only LLMs, where clinical concepts and relations were extracted by using human instructions as prompts. For example, we used the following question, “find the drug events including names, brand names and collective names” as a prompt for MRC models to identify the drug mentions, which were then used to generate relation-related questions as new prompts to find other concepts (e.g., the “Strength” of drugs). To identify the answer, we used two binary classifiers to predict the span (start and end indexes) of the entities. For the decoder-only generative LLMs (GatorTronGPT, Llama 3.1, and GatorTronLlama), we reformulated the two extraction tasks into a unified, text-to-text format, allowing the model to process them using a natural language instruction. For example, we added a prompt “Extract all medical problems from the following clinical note” for the 2010 i2b2 CCE task and a question “Does the relationship between drug event ‘Metformin’ and frequency ‘daily’ exist?” for the 2018 n2c2 CRE task. Instead of classifying tokens, the model is prompted to generate the structured information directly as a text string. Which provides more flexibility, allowing both CE and RE to be handled by the same LLM. We utilized LoRA, a parameter-efficient fine-tuning framework, to reduce the computational cost of fully fine-tuning LLMs with multi-billion parameters. LoRA injects small, trainable low-rank matrices into the attention layers of the Transformer architecture. During fine-tuning, only the low-rank matrices are updated. 2.6. Multi-task instruction tuning Multi-task instruction tuning is a technology to enhance the generalizability of LLMs by training them using a mixed dataset containing multiple tasks, with the assumption that the knowledge learned from one task could help other tasks. In this study, we applied multi-task instruction tuning for decoder-only LLMs (GatorTronGPT, Llama 3.1, and GatorTronLlama). Multitask instruction tuning can only be applied to LLMs that can model different tasks using a unified architecture, such as generative LLMs. For each task, multiple human-designed prompt templates, which vary in phrasing and structure, are employed to specify the desired output format. These prompts, examples of which are shown in supplementary Table S1 , are integrated with our LoRA-based PEFT framework, which updates a small subset of model parameters while adapting to both tasks. To evaluate the effectiveness of multi-task instruction tuning, we adopt a leave-one-dataset-out approach for few-shot and zero-shot learning experiments. In this setup, the model is trained on three of the four datasets (2018 n2c2, 2022 n2c2, RadGraph, and UFHealth datasets), with the remaining dataset held out for evaluation of the few-shot and zero-shot performance. 2.7. Experimental design We conducted single-task benchmarking experiments to compare the selected LLMs for single-task settings and conducted multitask instruction experiments to compare the performance for few-shot learning. 2.7.1. Single-task Benchmarking We finetuned each selected LLM using a single dataset setting to test the performance. Seven LLMs with different sizes were compared, including BERT-large (340 million), GatorTron-base (345 million), GatorTron-large (9 billion), GatorTronGPT-base (5 billion), GatorTronGPT-large (20 billion), Llama 3.1–8B (8 billion), and GatorTronLlama (8 billion). Each model is trained and evaluated on each of the five datasets independently. This experiment allows for a controlled comparison of: (1) encoder-only vs. decoder-only architectures, (2) traditional vs. PEFT fine-tuning, and (3) the impact of model sizes on task performance. 2.7.2. Multi-Task Instruction Tuning Three generative LLMs, including GatoTronGPT-base, Llama 3.1–8B, and GatorTronLlama were explored. We used a leave-one-dataset-out protocol using the 2018 n2c2, 2022 n2c2, RadGraph, and UF Health datasets. This setup involves four folds, using the four datasets. In each fold, one dataset is held out for evaluation under zero-shot and few-shot settings. Zero-shot is assessed by applying the multi-task trained model directly to the held-out dataset without further fine-tuning, while few-shot performance is evaluated using the model fine-tuning on small subsets (5, 10, 20, 50 samples). The performance of these multi-task models is compared with single-dataset fine-tuned models. 2.7.3. Evaluation Metrics The micro-averaged F1-score based on strict match was used for the evaluation of CE, which requires an exact match for both the entity type and the start and end boundaries of the entity span. Similarly, the micro-averaged F1-score was used to evaluate RE for classifying the relation type between pairs of concepts. 2.7.4. Implementation Details For data splits, we used the official train/validation/test partitions provided in the benckmarking datasets, including the 2010 i2b2, 2018 n2c2, 2022 n2c2, and RadGraph datasets. For the internal UF Health dataset, we performed a stratified patient-level split of 70% training, 10% validation, and 20% testing. To address the disparity between document lengths and model context windows, particularly for the 2010 i2b2 and 2018 n2c2 datasets, where notes often exceed the 512-token limit of encoder models, we implemented a consistent sentence-level segmentation strategy. For the shorter datasets (2022 n2c2, and RadGraph), the full input text fit within the context windows of all models. For the longer 2010 i2b2, 2018 n2c2, and UFHealth datasets, we pre-segmented all clinical notes into sentences using the NLTK tokenizer. For Clinical CE, each sentence was fed to the model as an independent input instance. For Clinical RE, we focused on intra-sentence entity pairs, classifying relationships only between entities that appeared within the same sentence boundary. To ensure accurate evaluation of free-form text generated by decoder-only models, we implemented a post-processing pipeline to parse the structured strings. Generations that failed to be parsed into the required format were classified as prediction errors (i.e., False Negatives). Hyperparameter tuning was performed on the validation set using grid search. For encoder-only full fine-tuning, we searched learning rates in {1e −5 , 2e −5 , 5e −5 }, batch sizes in {8, 16}, and dropout in {0.1, 0.2}. The optimal configuration used a learning rate of 2e −5 with the AdamW optimizer and a training batch size of 8. For prompt-based PEFT, we tuned LoRA rank in {64, 128, 256}, learning rate in {1e −4 , 2e −4 }. The best configuration was selected based on validation micro-F1. The final configuration utilized a learning rate of 1e −4 with AdamW and a LoRA rank of 256 with a dropout of 0.2 applied to the adapter layers. All experiments were conducted using four NVIDIA A100 80GB GPUs. Encoder-only models (BERT, GatorTron) were implemented using the Hugging Face Transformers library and PyTorch Lightning, while decoder-only generative models (GatorTronGPT, Llama 3.1, GatorTronLlama) were trained using the NVIDIA NeMo framework. 3. RESULTS 3.1. Results on different LLM architectures and fine-tuning strategies Table 2 compares 12 LLMs with different architectures and fine-tuning strategies for clinical CE using single task finetuning. Three LLMs, including GatorTron-large-MRC, Llama 3.1–8B and GatorTronLlama achieved top performance with average Micro-F1 score over 0.89 and Macro-F1 score over 0.87 across five datasets. GatorTronLlama achieved the highest average Micro-F1score of 0.8981 and Macro-F1 score of 0.8754, narrowly outperforming the general-domain Llama 3.1–8B (0.8964 and 0.8730) and the prompt-based encoder GatorTron-large-MRC (0.8946 and 0.8704), while the smaller GatorTron-base-MRC models had the lowest average Micro-F1 score of 0.8653, with the lowest score observed from the 2022 n2c2 dataset (0.8186) compared with other models. The 95% CI for these top models largely overlapped on the 2018 n2c2 and UF Health datasets, suggesting comparable effectiveness in these domains; however, GatorTronLlama demonstrated a distinct advantage on the RadGraph dataset (0.9179 [95% CI: 0.9152–0.9198]) compared to Llama 3.1 (0.9136 [95% CI: 0.9109–0.9155]). Among fine-tuning strategies for encoder-only models, smaller GatorTron-base-MRC with LoRA-based PEFT (0.8653) slightly underperformed its full fine-tuned counterpart (0.8711) by 0.58%, while larger GatorTron-large-MRC achieved an average F1 score of 0.8946, significantly outperforming the smaller GatorTron-base-MRC by 2.9%, and its fully fine-tuned counterpart (0.8807) by 1.39%. Among prompt-based models, decoder-only LLMs GatorTronGPT-base and GatorTronGPT-large achieved F1 scores of 0.8882 and 0.8899, respectively, slightly underperforming encoder-only GatorTron-large-MRC, while the other two decoder-only LLMs, Llama 3.1–8B (0.8964) and GatorTronLlama-8B (0.8981), outperformed encoder-only models, surpassing GatorTron-base-MRC by 3.11% and 3.28%. Notably, all models consistently scored lowest on the 2022 n2c2 dataset, while GatorTronLlama showed its best performance on the RadGraph and UF Health datasets, with F1-scores of 0.9179 and 0.9199, respectively. Table 2. Performance comparison of LLMs with different architectures and fine-tuning strategies for clinical concept extraction Model Architecture Params Fine-tuning strategy Metric Dataset 2010 i2b2 2018 n2c2 2022 n2c2 RadGraph UF Health Average BERT-large Encoder-only 340M Full fine-tuning Micro F1 (95% CI) 0.8694 (0.8671–0.8709) 0.8807 (0.8782–0.8824) 0.8318 (0.8289–0.8338) 0.8852 (0.8828–0.8869) 0.9048 (0.9021–0.9066) 0.8744 Macro F1 (95% CI) 0.8450 (0.8422–0.8471) 0.8580 (0.8552–0.8601) 0.8050 (0.8018–0.8074) 0.8600 (0.8571–0.8622) 0.8800 (0.8772–0.8821) 0.8496 ClinicalBERT 340M Micro F1 (95% CI) 0.8699 (0.8682–0.8708) 0.8811 (0.8795–0.8815) 0.8328 (0.8314–0.8335) 0.8842 (0.8831–0.8850) 0.9058 (0.9014–0.9067) 0.8748 Macro F1 (95% CI) 0.8324 (0.8301–0.8345) 0.8546 (0.8532–0.8559) 0.8082 (0.8045–0.8097) 0.8621 (0.8601–0.8635) 0.8815 (0.8788–0.8827) 0.8478 SpanBERT 340M Micro F1 (95% CI) 0.8710 (0.8684–0.8724) 0.8804 (0.8758–0.8825) 0.8334 (0.8305–0.8360) 0.8838 (0.8801–0.8852) 0.9025 (0.8997–0.9042) 0.8742 Macro F1 (95% CI) 0.8268 (0.8249–0.8287) 0.8575 (0.8542–0.8590) 0.8025 (0.8005–0.8042) 0.8625 (0.8598–0.8632) 0.8824 (0.8802–0.8846) 0.8463 GatorTron-base 345M Micro F1 (95% CI) 0.8715 (0.8689–0.8732) 0.8817 (0.8791–0.8835) 0.8341 (0.8311–0.8362) 0.8869 (0.8842–0.8887) 0.9114 (0.9088–0.9131) 0.8771 Macro F1 (95% CI) 0.8480 (0.8452–0.8501) 0.8600 (0.8571–0.8622) 0.8080 (0.8049–0.8103) 0.8620 (0.8591–0.8642) 0.8880 (0.8851–0.8902) 0.8532 GatorTron-large 9B Micro F1 (95% CI) 0.8782 (0.8755–0.8801) 0.8825 (0.8798–0.8843) 0.8388 (0.8358–0.8410) 0.8915 (0.8888–0.8934) 0.9125 (0.9097–0.9143) 0.8807 Macro F1 (95% CI) 0.8550 (0.8521–0.8572) 0.8610 (0.8581–0.8631) 0.8120 (0.8088–0.8144) 0.8680 (0.8651–0.8702) 0.8900 (0.8871–0.8922) 0.8572 GatorTron-base-MRC 345M Prompt-based parameter efficient fine-tuning Micro F1 (95% CI) 0.8705 (0.8678–0.8724) 0.8612 (0.8582–0.8633) 0.8186 (0.8155–0.8208) 0.8818 (0.8791–0.8837) 0.8945 (0.8918–0.8964) 0.8653 Macro F1 (95% CI) 0.8460 (0.8431–0.8482) 0.8350 (0.8319–0.8373) 0.7900 (0.7868–0.7924) 0.8550 (0.8521–0.8572) 0.8680 (0.8651–0.8702) 0.8388 GatorTron-large-MRC 9B Micro F1 (95% CI) 0.8953 (0.8928–0.8971) 0.8993 (0.8968–0.9011) 0.8548 (0.8518–0.8570) 0.9081 (0.9056–0.9099) 0.9153 (0.9127–0.9171) 0.8946 Macro F1 (95% CI) 0.8720 (0.8691–0.8742) 0.8750 (0.8721–0.8772) 0.8280 (0.8248–0.8304) 0.8850 (0.8821–0.8872) 0.8920 (0.8891–0.8942) 0.8704 GatorTronGPT-base Decoder-only 5B Micro F1 (95% CI) 0.8784 (0.8756–0.8804) 0.9021 (0.8993–0.9041) 0.8462 (0.8431–0.8484) 0.9021 (0.8993–0.9041) 0.9122 (0.9094–0.9141) 0.8882 Macro F1 (95% CI) 0.8540 (0.8511–0.8562) 0.8780 (0.8751–0.8802) 0.8190 (0.8158–0.8214) 0.8790 (0.8761–0.8812) 0.8880 (0.8851–0.8902) 0.8636 GatorTronGPT-large 20B Micro F1 (95% CI) 0.8806 (0.8778–0.8826) 0.9024 (0.8996–0.9043) 0.8511 (0.8480–0.8533) 0.9025 (0.8997–0.9045) 0.9129 (0.9101–0.9148) 0.8899 Macro F1 (95% CI) 0.8560 (0.8531–0.8582) 0.8790 (0.8761–0.8812) 0.8240 (0.8208–0.8264) 0.8800 (0.8771–0.8822) 0.8890 (0.8861–0.8912) 0.8656 Meditron-7B 7B Micro F1 (95% CI) 0.8815 (0.8788–0.8834) 0.8985 (0.8958–0.9004) 0.8492 (0.8461–0.8514) 0.9015 (0.8988–0.9034) 0.9105 (0.9077–0.9124) 0.8882 Macro F1 (95% CI) 0.8580 (0.8551–0.8602) 0.8740 (0.8711–0.8762) 0.8220 (0.8188–0.8244) 0.8780 (0.8751–0.8802) 0.8860 (0.8831–0.8882) 0.8636 Llama 3.1–8B 8B Micro F1 (95% CI) 0.8903 (0.8876–0.8922) 0.9035 (0.9007–0.9054) 0.8561 (0.8530–0.8583) 0.9136 (0.9109–0.9155) 0.9187 (0.9160–0.9206) 0.8964 Macro F1 (95% CI) 0.8680 (0.8651–0.8702) 0.8800 (0.8771–0.8822) 0.8300 (0.8268–0.8324) 0.8910 (0.8882–0.8931) 0.8960 (0.8931–0.8981) 0.8730 GatorTronLlama 8B Micro F1 (95% CI) 0.8862 (0.8835–0.8881) 0.9054 (0.9026–0.9073) 0.8612 (0.8581–0.8634) 0.9179 (0.9152–0.9198) 0.9199 (0.9172–0.9218) 0.8981 Macro F1 (95% CI) 0.8640 (0.8611–0.8662) 0.8830 (0.8801–0.8851) 0.8360 (0.8328–0.8384) 0.8960 (0.8931–0.8981) 0.8980 (0.8951–0.9002) 0.8754 Open in a new tab Table 3 compares 12 LLMs with different architectures and fine-tuning strategies for clinical RE. Llama 3.1–8B and GatorTronLlama-8B with prompt-based PEFT are the best models achieved comparable average Micro-F1scores of 0.8942 and 0.8978, respectively, with exceptional performance on the 2018 n2c2 dataset (0.9612 and 0.9674) and UF Health dataset (0.9085 and 0.9124). ClinicalBERT and SpanBERT with traditional fine-tuning contributed the lowest average Micro-F1scores of 0.8306 and 0.8071, particularly struggling on the RadGraph dataset (0.6989 and 0.6359). For encoder-only models, two GatorTron models with prompt-based fine-tuning (GatorTron-base-MRC) achieved average Micro-F1 scores of 0.8506 and 0.8661, outperforming their classification-based fine-tuned counterparts (0.8322 and 0.8375) by 1.84% and 2.84%, respectively. Decoder-only models with prompt-based PEFT significantly outperformed encoder-only models, with GatorTronGPT-large (0.8862), Llama 3.1–8B (0.8942), and GatorTronLlama-8B (0.8978) achieving 3.36%~4.72% higher Micro-F1scores than the best encoder-only model GatorTron-large-MRC (0.8661). The performance on the 2018 n2c2 dataset was remarkably high across all models, over 0.95, while RadGraph consistently presented the lowest Micro-F1 scores of 0.6985~0.8514. Table 3. Performance comparison of LLMs with different architectures and fine-tuning strategies for clinical relation extraction Model Architecture Params Fine-tuning strategy Metric Dataset 2018 n2c2 2022 n2c2 RadGraph UFHealth Average BERT-large Encoder-only 340M Full fine-tuning Micro F1 (95% CI) 0.9561 (0.9534–0.9580) 0.7807 (0.7774–0.7831) 0.6985 (0.6953–0.7008) 0.8952 (0.8920–0.8975) 0.8326 Macro F1 (95% CI) 0.9320 (0.9288–0.9344) 0.7450 (0.7414–0.7477) 0.6580 (0.6544–0.6607) 0.8680 (0.8645–0.8706) 0.8008 ClinicalBERT 340 M Micro F1 (95% CI) 0.9572 (0.9540–0.9589) 0.7764 (0.7735–0.7780) 0.6989 (0.6978–0.7009) 0.8898 (0.8872–0.8910) 0.8306 Macro F1 (95% CI) 0.9288 (0.9268–0.9297) 0.7384 (0.7362–0.7389) 0.6502 (0.6482–0.6514) 0.8674 (0.8659–0.8699) 0.7962 SpanBERT 340M Micro F1 (95% CI) 0.9524 (0.9502–0.9538) 0.7752 (0.7722–0.7763) 0.6359 (0.6342–0.6397) 0.8649 (0.8628–0.8662) 0.8071 Macro F1 (95% CI) 0.9275 (0.9264–0.9289) 0.7346 (0.7322–0.7369) 0.6187 (0.6165–0.6199) 0.8458 (0.8426–0.8472) 0.7817 GatorTron-base 345M Micro F1 (95% CI) 0.9575 (0.9548–0.9593) 0.7798 (0.7766–0.7821) 0.6925 (0.6893–0.6948) 0.8988 (0.8956–0.9011) 0.8322 Macro F1 (95% CI) 0.9340 (0.9309–0.9363) 0.7430 (0.7394–0.7457) 0.6510 (0.6474–0.6537) 0.8720 (0.8685–0.8746) 0.8000 GatorTron-large 9B Micro F1 (95% CI) 0.9580 (0.9553–0.9599) 0.7838 (0.7806–0.7861) 0.7058 (0.7026–0.7081) 0.9025 (0.8994–0.9047) 0.8375 Macro F1 (95% CI) 0.9350 (0.9319–0.9373) 0.7480 (0.7444–0.7507) 0.6650 (0.6614–0.6677) 0.8760 (0.8725–0.8786) 0.8060 GatorTron-base-MRC 345M Prompt-based parameter efficient fine-tuning Micro F1 (95% CI) 0.9358 (0.9329–0.9379) 0.8452 (0.8419–0.8476) 0.7358 (0.7326–0.7381) 0.8855 (0.8823–0.8878) 0.8506 Macro F1 (95% CI) 0.9100 (0.9067–0.9125) 0.8120 (0.8084–0.8147) 0.6950 (0.6914–0.6977) 0.8560 (0.8525–0.8586) 0.8182 GatorTron-large-MRC 9B Micro F1 (95% CI) 0.9605 (0.9578–0.9624) 0.8489 (0.8457–0.8512) 0.7552 (0.7520–0.7575) 0.8997 (0.8966–0.9019) 0.8661 Macro F1 (95% CI) 0.9380 (0.9348–0.9404) 0.8180 (0.8144–0.8207) 0.7150 (0.7114–0.7177) 0.8720 (0.8685–0.8746) 0.8358 GatorTronGPT-base Decoder-only 5B Micro F1 (95% CI) 0.9635 (0.9609–0.9654) 0.8520 (0.8488–0.8543) 0.8182 (0.8150–0.8205) 0.9024 (0.8993–0.9046) 0.8840 Macro F1 (95% CI) 0.9400 (0.9369–0.9423) 0.8200 (0.8164–0.8227) 0.7820 (0.7784–0.7847) 0.8760 (0.8725–0.8786) 0.8545 GatorTronGPT-large 20B Micro F1 (95% CI) 0.9657 (0.9630–0.9676) 0.8529 (0.8497–0.8552) 0.8205 (0.8173–0.8228) 0.9056 (0.9025–0.9078) 0.8862 Macro F1 (95% CI) 0.9420 (0.9389–0.9443) 0.8220 (0.8184–0.8247) 0.7850 (0.7814–0.7877) 0.8800 (0.8765–0.8826) 0.8572 Meditron-7B 7B Micro F1 (95% CI) 0.9592 (0.9565–0.9611) 0.8495 (0.8463–0.8518) 0.8050 (0.8018–0.8073) 0.9010 (0.8978–0.9033) 0.8787 Macro F1 (95% CI) 0.9360 (0.9329–0.9383) 0.8160 (0.8124–0.8187) 0.7680 (0.7644–0.7707) 0.8750 (0.8714–0.8777) 0.8488 Llama 3.1–8B 8B Micro F1 (95% CI) 0.9612 (0.9585–0.9631) 0.8615 (0.8584–0.8637) 0.8454 (0.8423–0.8476) 0.9085 (0.9054–0.9107) 0.8942 Macro F1 (95% CI) 0.9380 (0.9349–0.9403) 0.8320 (0.8284–0.8347) 0.8100 (0.8064–0.8127) 0.8820 (0.8785–0.8846) 0.8655 GatorTronLlama 8B Micro F1 (95% CI) 0.9674 (0.9647–0.9693) 0.8601 (0.8570–0.8623) 0.8514 (0.8482–0.8537) 0.9124 (0.9093–0.9146) 0.8978 Macro F1 (95% CI) 0.9450 (0.9419–0.9473) 0.8300 (0.8264–0.8327) 0.8180 (0.8144–0.8207) 0.8880 (0.8845–0.8906) 0.8702 Open in a new tab 3.2. Statistical analysis of LLM architecture and fine-tuning strategies We conducted repeated-measures experiments using 5 random seeds for each configuration across all five datasets for both tasks. We compared the best-performing encoder (GatorTron-large with PEFT) against the best-performing decoder (GatorTronLlama with PEFT), and GatorTron-large with full fine-tuning against PEFT. The results, including mean F1 scores and standard deviations, are reported in Table 4 , with statistical significance determined by paired t-tests (p < 0.05). For architecture, the decoder-only GatorTronLlama demonstrated a consistent, statistically significant advantage over the encoder-only model across all four CRE datasets (p < 0.05) and four of five CCE datasets (p < 0.05), with performance statistically comparable on the 2010 i2b2 CCE task (p > 0.05). Regarding fine-tuning strategies, prompt-based PEFT was statistically better than full fine-tuning on three of four CRE datasets (p < 0.05); the gap was particularly large on the 2022 n2c2 and RadGraph datasets. For CCE, PEFT significantly outperformed full fine-tuning on three of five datasets (2010 i2b2, 2018 n2c2, and 2022 n2c2) and achieved statistically equivalent performance on RadGraph and UF Health (p > 0.05). Table 4. Statistical analysis of model architectures and fine-tuning strategies across five clinical datasets. Experiment Model Architecture Finetuning strategy Concept extraction Relation extraction 2010i2b2 2018n2c2 2022n2c2 RadGraph UF Health 2018n2c2 2022n2c2 RadGraph UF Health Significance of model architecture GatorTron-large Encoder-only PEFT 0.8951 (± 0.008) 0.8990 (± 0.005) 0.8548 (± 0.005) 0.9080 (± 0.004) 0.9154 (± 0.007) 0.9605 (± 0.008) 0.8489 (± 0.004) 0.7554 (± 0.006) 0.8998 (± 0.010) GatorTronLlama Decoder-only 0.8861 (± 0.010) 0.9055 (± 0.005) 0.8612 (± 0.002) 0.9179 (± 0.006) 0.9198 (± 0.008) 0.9675 (± 0.009) 0.8603 (± 0.007) 0.8515 (± 0.008) 0.9127 (± 0.007) Significance of finetuning strategy GatorTron-large Encoder-only Full FT 0.8788 (± 0.006) 0.8824 (± 0.004) 0.8387 (± 0.007) 0.8918 (± 0.010) 0.9126 (± 0.007) 0.9581 (± 0.002) 0.7838 (± 0.005) 0.7055 (± 0.007) 0.9028 (± 0.007) GatorTron-large PEFT 0.8955 (± 0.004) 0.8996 (± 0.003) 0.8547 (± 0.005) 0.9082 (± 0.008) 0.9154 (± 0.006) 0.9608 (± 0.009) 0.8489 (± 0.008) 0.7552 (± 0.009) 0.8999 (± 0.008) Open in a new tab Note: CCE: Clinical Concept Extraction; CRE: Clinical Relation Extraction; Full FT: Full Fine-Tuning; PEFT: Parameter-Efficient Fine-Tuning. Bold number indicate statistically significant improvement (p < 0.05) compared to the baseline in pairing. 3.3. Results of multi-task instruction tuning on zero-shot and few-shot learning The zero-shot and few-shot learning experiments evaluate the generalization capabilities of three decoder-only large language models (LLMs)—GatorTronGPT, Llama 3.1–8B, and GatorTronLlama-8B—using a leave-one-dataset-out approach across four datasets: 2018 n2c2, 2022 n2c2, RadGraph, and UFHealth datasets. Figure 2 compares zero-shot and few-shot learning performance of three decoder-only LLMs, including GatorTronGPT, Llama 3.1–8B, and GatorTronLlama-8B, using four datasets, including 2018 n2c2, 2022 n2c2, RadGraph, and UFHealth datasets. The average F1-scores across the four datasets was used for evaluation. For zero-shot performance, multi-task instruction-tuned GatorTronLlama-8B achieved the highest average F1-scores of 0.3596 for CE and 0.3154 for RE, significantly outperforming its counterpart without instruction tuning (0.0155 for CE, 0.0189 for RE) by 34.41% and 29.65%, respectively. All multi-task tuned models, including GatorTronGPT-base, Llama 3.1, and GatorTronLlama, achieved significant improvement with average F1 scores of 0.2658~0.3955, while all baseline models without instruction tuning performed poorly across the two tasks, with average F1 scores near random chance or close to zero. In few-shot settings (5, 10, 20 samples), multi-task tuned GatorTronLlama-8B achieved the highest F1-scores of 0.7350 (CE) and 0.6815 (RE) at 5 shots, 0.7570 (CE) and 0.7115 (RE) at 10 shots, and 0.7811 (CE) and 0.7685 (RE) at 20 shots, outperforming baselines without instruction tuning by 9.7%~12.6% for CE and 5.9%~8.6% for RE. The three multi-task models consistently outperform their single-task baselines on both CE and RE tasks, achieving performance improvement of 0.8~9.7% for CE, and 1.8~6.0% for RE. While the performance gap between multi-task tuned and baseline models narrows as more examples are provided, the multi-task tuned models consistently take the lead. The Llama 3.1–8B and GatorTronLlama models, when enhanced with multi-task tuning, show comparable and robust few-shot learning performance. Notably, performance on CE is consistently higher than RE across all models. For example, in the 20-shot setting, the multi-task tuned GatorTronLlama achieves an F1-score of 0.7811 for CE, compared to 0.7685 for RE. Figure 2. Open in a new tab Zero-shot and few-shot performance between multi-task instruction-tuned LLMs and baseline models without instruction tuning across CCE and CRE tasks. 4. DISCUSSION This study provided a comprehensive evaluation of LLMs for patient information extraction, including CE and RE tasks. We systematically compared two LLM architectures, two fine-tuning strategies, and the impact of multi-task instruction tuning on generalizability. This study provides key insights into the development of robust and adaptive clinical NLP systems for patient information extraction. The statistical test shows that the choice of model architecture significantly impacts performance. While encoder-only LLMs fine-tuned using the advanced MRC framework remain statistically competitive for easy concept extraction tasks (2010 i2b2 dataset showing no significant difference, p > 0.05), decoder-only generative LLMs tuned with PEFT achieved statistically significant improvements (p < 0.05) across the other four concept extraction datasets and all relation extraction datasets. Our findings suggest that decoder-only models outperformed encoder-only models for patient information extraction. Regarding the training strategy, our statistical analysis shows that LoRA-based PEFT is a cost-efficient and high-performance strategy that frequently outperforms traditional full-size fine-tuning, achieving statistically significant improvements (p < 0.05) on both concept and relation extraction tasks. This study provides guidance on the selection of fine-tuning strategies and their associated computational costs. As shown in Table 5 , LoRA-based PEFT presents a better trade-off between performance and efficiency. Compared to traditional full-size fine-tuning, which requires updating all parameters, LoRA trains less than 1% of the total parameters, reducing the training time and GPU memory requirements, making the adaptation of multi-billion parameter models computationally affordable. For example, fully fine-tuning the 9-billion parameter GatorTron-large model required approximately 48 GPU hours, 78 GB of peak GPU memory, and 84.5 kWh of energy, whereas fine-tuning the comparable GatorTronLlama with LoRA took only 8 GPU hours, 48 GB of memory, and 14.1 kWh of energy, representing reductions of ~83% in time and energy consumption and ~38% in memory footprint. This makes the adaptation of multi-billion parameter models both computationally affordable and environmentally sustainable. In addition, the achieved efficiency does not affect the inference speed; because the LoRA adapter weights can be merged into the base model’s weights, there is few additional latencies during inference. Notably, the GatorTron-base with full fine-tuning performed better than GatorTron-base-MRC, indicating that for smaller models, updating all parameters can still be the best solution if sufficient in-domain data are available. However, for generative LLMs with much larger sizes, prompt-based PEFT is a better choice, with higher performance and efficiency. By enabling high-performance deployment on single-GPU workstations rather than energy-intensive clusters, PEFT significantly lowers the barrier for adopting advanced generative NLP in resource-constrained clinical settings. Table 5. Comparison of computational efficiency for different LLMs Model Params Fine-Tuning Strategy Trainable Params Average Training Time (GPU Hours) Average peak GPU memory (GB) Average estimated energy (kWh) Average Inference Time (ms/note) GatorTron-base 345 million Full fine-tuning 100% ~8 24 14.1 ~15 GatorTron-large 9 billion ~48 78 84.5 ~22 GatorTron-base-MRC 345 million Prompt-based LoRA ~0.5% ~2 20 3.5 ~16 GatorTron-large-MRC 9 billion ~6 60 10.6 ~25 GaotTronLlma 8 billion ~0.8% ~8 48 14.1 ~32 Open in a new tab A significant contribution of this work is that multi-task instruction tuning greatly improves few-shot learning for better generalizability. Our leave-one-dataset-out experiments show that LLMs fine-tuned using a single dataset have limited transfer learning ability when applied to a new, unseen domain (zero-shot). Multitask instruction tuning remarkably improved the performance. Our findings support the hypothesis that through multitask instruction tuning, generative LLMs can learn knowledge with better generalizability to other tasks. To further illustrate, Figure 3 shows the performance trajectory as we fine-tune the Llama 3.1 and GatorTronLlama models on an increasing percentage of the held-out training data. The models exhibit a steep learning curve, rapidly approaching the performance of a fully fine-tuned baseline. As shown in Figure 3 , by using only around 20% of the available training data, the multi-task models achieve F1-scores comparable to models trained using the full datasets. Figure 3. Open in a new tab Performance trend with increasing training samples between multi-task instruction tuned Llama 3.1–8B and GatorTronLlama across CCE and CRE tasks. We conducted an error analysis on the 2018 n2c2 and 2022 n2c2 datasets, comparing the encoder-only GatorTron-large, the decoder-only GatorTronLlama, and the multi-task instruction-tuned GatorTronLlama ( Supplementary Table S3 ). For clinical CE, we observed model-specific failure patterns. The encoder-only model using token-level classification (BIO schema) struggled with nested and overlapping entities, which constituted 35% of its errors on the 2022 n2c2 dataset. In contrast, the single-task decoder-only GatorTronLlama effectively resolved these complex structures (only 7% nested errors) but suffered from hallucinations, where spurious entities (e.g., generating synonyms) accounted for 48–58% of its total errors. The multi-task GatorTronLlama, operating in a zero-shot setting, showed a third pattern: while it maintained low error rate in complex, nested clinical concepts (5–10%), the primary type of error is from Type Misclassification (30–35%). This reflects the special challenge of multi-task instruction tuning, where the model could learn useful knowledge from external tasks, but it inevitably learned concept categories that were not aligned well with the targeted application domain. For RE, the encoder-only models show lower recalls (false negatives comprised 55–60% of errors), which were improved using the decoder-only models (false negatives reduced to 20–25%). Researchers should choose models according to the real-world needs: encoder-only LLMs are good for tasks that require high precision, such as diagnosis, while decoder-only LLMs are better for tasks that require high recall, such as screening and clinical trial screening. This study has limitations. We explored commonly used LLMs and fine-tuning strategies; new algorithms recently proposed need further investigation. The quality of prompts heavily impacts the prompt-based solutions. We identified the best prompts for each task based on prompt engineering to the best of our enumeration capacity; future studies need to explore advanced prompt engineering techniques. 5. CONCLUSION This study presents a comprehensive study of using encoder-only and decoder-only LLMs for patient information extraction, focusing on model architectures, fine-tuning strategies, and multi-task instruction tuning of generative LLMs. Our findings provide a practical guideline in selecting LLM architectures and fine-tuning strategies to facilitate the development of LLM-based solutions for patient information extraction from clinical narratives. Supplementary Material supplement NIHMS2160505-supplement-supplement.docx (21.7KB, docx) Attached in a separate document. Statement of Significance. Problem Large language models (LLMs) for patient information from clinical narratives remains a challenge due to their optimal architectures, fine-tuning strategies, and generalizability for clinical concept and relation extraction are not well understood. What is Already Known Encoder-only and decoder-only LLMs have shown promise in clinical information extraction. Parameter-efficient fine-tuning (PEFT) and multi-task instruction tuning are advanced finetuning strategies to reduce computational costs and improve generalization. What this Paper Adds This study examines LLMs’ effectiveness in patient information extraction, focusing on LLM architectures, including encoder-only and decoder-only transformers, fine-tuning strategies, and multi-task instruction tuning techniques, providing a cost-effective and generalizable approach for developing scalable clinical NLP systems. Who would benefit from the new knowledge in this paper Researchers developing clinical NLP systems, and health informatics professionals deploying LLM-based information extraction pipelines will benefit from the methodological insights and practical guidance on selecting architectures and fine-tuning strategies for scalable patient information extraction. Open in a new tab ACKNOWLEDGEMENTS We acknowledge the contributions of Xinyu Dong and Yaoyun Zhang, employees of Selfii Company. We gratefully acknowledge the support of NVIDIA Corporation and the NVIDIA AI Technology Center (NVAITC) UF program. FUNDING STATEMENT This study was partially supported by grants from Patient-Centered Outcomes Research Institute ® (PCORI ® ) Award (ME-2018C3-14754, ME-2023C3-35934), the PARADIGM program awarded by the Advanced Research Projects Agency for Health (ARPA-H), National Institute on Aging, NIA R56AG069880, National Institute of Allergy and Infectious Diseases, NIAID R01AI172875, National Heart, Lung, and Blood Institute, R01HL169277, National Institute on Drug Abuse, NIDA R01DA050676, R01DA057886, National Cancer Institute, NCI R37CA272473, National Library of Medicine, NLM R01LM011934, and the UF Clinical and Translational Science Institute. Footnotes COMPETING INTERESTS STATEMENT Cheng Peng, Xinyu Dong, Yaoyun Zhang, Mengxian Lyu, Daniel Paredes, and Yonghui Wu have no conflicts of interest that are directly relevant to the content of this study. ETHICS STATEMENT This study utilized de-identified clinical text data from publicly available benchmark datasets (2010 i2b2, 2018 n2c2, 2022 n2c2, and RadGraph) and an internal dataset derived from the University of Florida Health Integrated Data Repository (UF Health IDR). All data used in this research were fully de-identified in accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule, and no protected health information (PHI) was accessed or analyzed. Because the study involved only retrospective analysis of de-identified data and posed no more than minimal risk to individuals, it was determined to be exempt from human subject’s research oversight by the Institutional Review Board (IRB) of the University of Florida. Consequently, informed consent was not required. All procedures were conducted in compliance with relevant U.S. federal regulations and institutional guidelines, and the privacy rights of human subjects were rigorously protected. DATA AND CODE AVAILABILITY Four of the five datasets used in this study are publicly available. The 2010 i2b2, 2018 n2c2, and 2022 n2c2 challenge datasets can be accessed through the n2c2 website ( https://n2c2.dbmi.hms.harvard.edu ). The RadGraph dataset is available for download from PhysioNet ( https://physionet.org/content/radgraph/1.0.0/ ). The UF Health dataset was created internally using clinical notes from the University of Florida Health Integrated Data Repository and is not publicly available due to UF policies. The source code for the experimental framework, including model fine-tuning, data processing, and evaluation scripts, is available at: https://github.com/cpenguf/LLM_for_Patient_Information_Extraction . REFERENCES 1. Bhavnani SP, Parakh K, Atreja A, et al. 2017 roadmap for innovation—ACC health policy statement on healthcare transformation in the era of digital health, big data, and precision health. J Am Coll Cardiol. 2017;70:2696–718. [ DOI ] [ PubMed ] [ Google Scholar ] 2. Gunter TD, Terry NP. The emergence of national electronic health record architectures in the United States and Australia: models, costs, and questions. J Med Internet Res. 2005;7:e3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 3. Jin Q, Wang Z, Floudas CS, et al. Matching patients to clinical trials with large language models. Nat Commun. 2024;15:9074. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Landolsi MY, Hlaoua L, Ben Romdhane L. Information extraction from electronic medical documents: state of the art and future research directions. Knowl Inf Syst. 2023;65:463–516. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 5. Zhou S, Wang N, Wang L, et al. CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records. J Am Med Inform Assoc. 2022;29:1208–16. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 6. Yang X, Bian J, Hogan WR, et al. Clinical concept extraction using transformers. J Am Med Inform Assoc. 2020;27:1935–42. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 7. Yanagisawa Y, Watabe S, Yokoyama S, et al. Identifying adverse events in outpatients with prostate cancer using pharmaceutical care records in community pharmacies: Application of named entity recognition. JMIR Cancer. 2025;11:e69663. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Peng C, Yang X, Smith KE, et al. Model tuning or prompt tuning? A study of large language models for clinical concept and relation extraction. arXiv [cs.CL]. 2023. [ Google Scholar ] 9. Uzuner Ö, South BR, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. 2011;18:552–6. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Uzuner O, Bodnari A, Shen S, et al. Evaluating the state of the art in coreference resolution for electronic medical records. J Am Med Inform Assoc. 2012;19:786–91. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 11. Uzuner O, Solti I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc. 2010;17:514–8. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Uzuner O, Goldstein I, Luo Y, et al. Identifying patient smoking status from medical discharge records. J Am Med Inform Assoc. 2008;15:14–24. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Uzuner O Recognizing obesity and comorbidities in sparse data. J Am Med Inform Assoc. 2009;16:561–70. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. J Am Med Inform Assoc. 2013;20:806–13. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 15. Henry S, Buchan K, Filannino M, et al. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2020;27:3–12. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 16. Wang Y, Fu S, Shen F, et al. The 2019 n2c2/OHNLP track on clinical Semantic Textual Similarity: Overview. JMIR Med Inform. 2020;8:e23375. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Lybarger K, Yetisgen M, Uzuner Ö. The 2022 n2c2/UW shared task on extracting social determinants of health. J Am Med Inform Assoc. Published Online First: 16 February 2023. doi: 10.1093/jamia/ocad012 [ DOI ] [ Google Scholar ] 18. Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: A systematic review. JAMA. 2025;333:319–28. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Yao L, Mao C, Luo Y. Clinical text classification with rule-based features and knowledge-guided convolutional neural networks. BMC Med Inform Decis Mak. 2019;19:71. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 20. Si Y, Wang J, Xu H, et al. Enhancing clinical concept extraction with contextual embeddings. J Am Med Inform Assoc. 2019;26:1297–304. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 21. Devlin J, Chang M-W, Lee K, et al. BERT: Pre-training of deep bidirectional Transformers for language understanding. arXiv [cs.CL]. 2018. [ Google Scholar ] 22. Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36:1234–40. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 23. Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv [cs.CL]. 2019. [ Google Scholar ] 24. Yang X, Chen A, PourNejatian N, et al. A large language model for electronic health records. NPJ Digit Med. 2022;5:194. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Peng C, Yang X, Yu Z, et al. Clinical concept and relation extraction using prompt-based machine reading comprehension. J Am Med Inform Assoc. Published Online First: 14 June 2023. doi: 10.1093/jamia/ocad107 [ DOI ] [ Google Scholar ] 26. Fraile Navarro D, Ijaz K, Rezazadegan D, et al. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. Int J Med Inform. 2023;177:105122. [ DOI ] [ PubMed ] [ Google Scholar ] 27. Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models. arXiv [cs.CL]. 2023. [ Google Scholar ] 28. Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv [cs.CL]. 2023. [ Google Scholar ] 29. Grattafiori A, Dubey A, Jauhri A, et al. The Llama 3 herd of models. arXiv [cs.AI]. 2024. [ Google Scholar ] 30. Peng C, Yang X, Chen A, et al. A study of generative large language model for medical research and healthcare. NPJ Digit Med. 2023;6:210. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 31. Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. arXiv [cs.CL]. 2020. [ Google Scholar ] 32. Wei J, Bosma M, Zhao VY, et al. Finetuned language models are zero-shot learners. arXiv [cs.CL]. 2021. [ Google Scholar ] 33. Chen A, Yu Z, Yang X, et al. Contextualized medication information extraction using Transformer-based deep learning architectures. J Biomed Inform. 2023;142:104370. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 34. Liu X, Zheng Y, Du Z, et al. GPT Understands, Too. arXiv [cs.CL]. 2021. [ Google Scholar ] 35. Liu X, Ji K, Fu Y, et al. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv [cs.CL]. 2021. [ Google Scholar ] 36. Hu EJ, Shen Y, Wallis P, et al. LoRA: Low-Rank Adaptation of large language models. arXiv [cs.CL]. 2021. [ Google Scholar ] 37. Wang L, Li R, Yan Y, et al. InstructionNER: A multi-task instruction-based generative framework for few-shot NER. arXiv [cs.CL]. 2022. [ Google Scholar ] 38. Jain S, Agrawal A, Saporta A, et al. RadGraph: Extracting clinical entities and relations from radiology reports. arXiv [cs.CL]. 2021. [ Google Scholar ] 39. Yu Z, Peng C, Yang X, et al. Identifying social determinants of health from clinical narratives: A study of performance, documentation ratio, and potential bias. J Biomed Inform. 2024;153:104642. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 40. Joshi M, Chen D, Liu Y, et al. SpanBERT: Improving pre-training by representing and predicting spans. Trans Assoc Comput Linguist. 2020;8:64–77. [ Google Scholar ] 41. Chen Z, Cano AH, Romanou A, et al. MEDITRON-70B: Scaling medical pretraining for large language models. arXiv [cs.CL]. 2023. [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials supplement NIHMS2160505-supplement-supplement.docx (21.7KB, docx) Data Availability Statement Four of the five datasets used in this study are publicly available. The 2010 i2b2, 2018 n2c2, and 2022 n2c2 challenge datasets can be accessed through the n2c2 website ( https://n2c2.dbmi.hms.harvard.edu ). The RadGraph dataset is available for download from PhysioNet ( https://physionet.org/content/radgraph/1.0.0/ ). The UF Health dataset was created internally using clinical notes from the University of Florida Health Integrated Data Repository and is not publicly available due to UF policies. The source code for the experimental framework, including model fine-tuning, data processing, and evaluation scripts, is available at: https://github.com/cpenguf/LLM_for_Patient_Information_Extraction . ACTIONS View on publisher site PDF (654.3 KB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top