A hybrid large language model framework for structured data entry from code-switched persian clinical speech.

A hybrid large language model framework for structured data entry from code-switched persian clinical speech - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Sci Rep . 2026 Apr 10;16:16958. doi: 10.1038/s41598-026-46229-y A hybrid large language model framework for structured data entry from code-switched persian clinical speech Mohammad Tanhaei 1, ✉ Author information Article notes Copyright and License information 1 Department of Engineering, Ilam University, Ilam, Iran ✉ Corresponding author. Received 2025 Nov 16; Accepted 2026 Mar 24; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13230711  PMID: 41963402 Abstract Converting spoken clinical dictation into structured electronic health record (EHR) entries poses significant challenges in low-resource languages. These difficulties are primarily driven by phonetic complexity and the prevalent phenomenon of code-switching with English medical terminology, both of which severely degrade the performance of automatic speech recognition (ASR) models. This paper presents a practical hybrid framework that combines ASR, Persian-specific normalization of code-switched and orthographically ambiguous terms, metadata-constrained field mapping, and uncertainty-triggered human review for structured EHR entry. Leveraging large language model (LLM) prompting and specialized bilingual glossaries, this module normalizes code-switched terms and disambiguates Persian-specific homophones. The proposed method employs embedding-based field mapping guided by form metadata and existing clinical annotations, supplemented by selective human-in-the-loop validation for high-stakes fields. Evaluation within the BioArc EHR system demonstrates substantial improvements in transcription quality and structured data accuracy. Experimental results and granular ablation studies confirm that explicitly addressing linguistic challenges such as code-switching and homophone variability is critical for improving field detection rates and supporting data integrity. Our findings underscore that while the framework serves as a robust assistive tool–achieving high field detection accuracy in controlled evaluations–a hybrid workflow remains essential to address residual error rates (approximately 5–10% in complex forms) and support the precision required for medical documentation. Keywords: Electronic Health Records, Voice-to-Form, LLM, Data Extraction, Automated Data Entry, Persian Clinical Speech, Low-Resource Languages, Code-Switching Subject terms: Computational biology and bioinformatics, Engineering, Health care, Mathematics and computing, Medical research Introduction The increasing adoption of voice assistants and advanced natural language processing technologies has facilitated the transformation of spoken input into structured data formats. Structured forms remain essential for data collection across numerous domains, including healthcare, legal, and administrative services, where precision and standardized handling are critical. Existing voice-to-form systems, however, frequently suffer from transcription inaccuracies, loss of contextual information, and misalignment with target fields–challenges that are particularly pronounced in low-resource languages characterized by complex phonetics, diverse dialects, and limited training datasets. In English-speaking clinical environments, ambient documentation systems and speech-to-text integrations in electronic health records (EHRs) have demonstrated substantial reductions in administrative burden, allowing providers to devote more time to patient care 1 , 2 . These advances rely on robust automatic speech recognition (ASR) models trained on extensive English corpora. In contrast, low-resource languages such as Persian face significant barriers: higher word error rates due to phonetic variability, informal speech patterns, scarcity of domain-specific medical audio data, and frequent code-switching with English medical terminology. Few documented voice-driven EHR systems exist for such contexts, leaving a gap in scalable, accurate documentation tools for non-English clinical settings. This paper presents a practical hybrid framework designed to address these challenges through the integration of ASR, large language model (LLM)-driven text refinement, metadata- and annotation-guided field mapping, and targeted human-in-the-loop validation. Applied to the BioArc EHR system–a comprehensive Persian-language platform supporting over 100 medical registries and used daily by more than 3,000 physicians–the framework converts spoken medical dictation into structured form entries while leveraging existing system annotations to constrain extraction and enhance reliability. Unlike standard ASR-to-Mapping pipelines that often fail in non-English clinical settings, our technical novelty lies in the dual-layer normalization architecture. This includes a specialized module for code-switched medical term recovery and a context-aware homophone disambiguation layer, which together bridge the gap between noisy Persian acoustic output and structured EHR metadata requirements. Table 1 provides a comparison of prominent voice-to-text conversion models in terms of accuracy, language support, non-English robustness, and latency. Similarly, Table 2 compares key text refinement models with respect to contextual understanding, coherence enhancement, and grammar accuracy. Table 1. Comparison of voice-to-text conversion models. Model Accuracy Language support Non-English robustness Latency Whisper High Multiple languages Moderate to High Low Google Speech-to-Text Very High Extensive High (primarily English) Low DeepSpeech Medium Moderate Low Medium Open in a new tab Table 2. Comparison of text refinement models. Model Contextual understanding Coherence enhancement Grammar accuracy BERT Very High High Very High GPT-3 Very High Very High High Gemma 3 27b Very High Very High Very High Open in a new tab Contributions This work introduces a practical hybrid framework for voice-driven structured data entry in low-resource language clinical contexts, using Persian dictation in the BioArc EHR system as a case study. The main contributions are: Development and deployment of a metadata- and annotation-guided hybrid pipeline combining ASR, LLM-based refinement, and embedding-based mapping with selective human validation for Persian medical speech in a real EHR setting. Introduction of novel normalization modules addressing code-switching and Persian orthographic ambiguities, challenges previously unaddressed in Persian EHR voice-to-form systems. Empirical demonstration of substantial performance gains from leveraging existing BioArc annotations to constrain LLM extraction and field mapping, particularly for complex forms in low-resource settings. Detailed insights into Persian-specific ASR challenges and corresponding adaptation strategies, contributing to the broader understanding of voice-to-EHR feasibility in non-English environments. A scalable hybrid workflow that supports high data quality in clinical documentation while reducing manual entry time, offering a model extensible to other low-resource languages. These contributions advance applied healthcare informatics by bridging gaps in underrepresented linguistic contexts. Objectives The primary objectives of this work are to develop a metadata-driven system capable of autonomously converting speech input to structured text using ASR; refining raw transcripts with LLMs to correct errors and enrich content, including normalization of code-switched medical terms; mapping refined text to predefined form fields guided by schema metadata and existing annotations; incorporating feedback mechanisms for error handling and quality control; and enabling cross-domain adaptability, with particular emphasis on low-resource language challenges in healthcare. Significance Efficient voice-to-form conversion has the potential to streamline data collection across diverse linguistic contexts, particularly in medical and administrative fields where rapid and accurate entry is essential. In low-resource language settings with prevalent code-switching, such systems can significantly enhance data quality, reduce administrative workload, and minimize human transcription errors. Research on hybrid human–AI systems indicates amplified benefits in high-stakes domains like hospitals, where precision is paramount 3 . By addressing gaps in non-English clinical documentation, this framework supports more equitable deployment of voice-enabled EHR tools globally. Related work Voice-to-text conversion Voice-to-text systems form the foundation of spoken data processing. Models such as OpenAI’s Whisper (2022) provide robust multilingual support 4 , while Google Speech-to-Text and older frameworks like DeepSpeech offer varying degrees of accuracy and language coverage 5 , 6 . Deep learning advancements enable adaptation to noisy environments and accents, though challenges persist in healthcare settings with background noise and domain-specific speech 7 . Fine-tuning on specialized data improves performance, but low-resource languages remain underserved. Text refinement Post-transcription refinement enhances clarity and usability. Pretrained models like BERT (2019) excel in contextual understanding 8 , while generative approaches such as GPT-3 (2020) and newer models like Gemma support coherence and grammar correction 9 , 10 . Iterative techniques, including self-instruction loops, further improve output quality 11 . Data extraction and mapping to forms Extracting structured information from unstructured text relies on named entity recognition and information extraction techniques 12 , 13 . Multilingual models like XLM-R facilitate cross-lingual tasks 14 , and generative AI streamlines complex extractions 15 . Table 3 summarizes the comparative strengths of prominent data extraction models in terms of precision, recall, and flexibility. Table 3. Comparison of data extraction models. Model Precision Recall Flexibility NER High High Medium IE Very High High High XLM-R High Very High Very High Open in a new tab Challenges and system robustness Noisy inputs and contextual ambiguities remain key obstacles in voice-to-structured data pipelines. Research addresses error detection and categorical mapping 16 , 17 . Hybrid approaches in high-stakes domains Hybrid human–AI workflows gain traction in precision-critical fields like healthcare, where automated transcription benefits from expert oversight 2 , 18 – 20 . Multilingual and low-resource language challenges Most ambient clinical documentation systems remain English-centric, with limited support for low-resource languages 4 . Multilingual ASR models like Whisper offer broad coverage but exhibit degraded performance in domain-specific low-resource settings due to data scarcity, phonetic complexity, and insufficient fine-tuning on non-English medical speech. Code-switching challenges in low-resource clinical speech Code-switching–the intrasentential mixing of languages–is common in multilingual clinical environments, where local language discourse frequently incorporates English/Latin medical terminology. This phenomenon significantly degrades ASR performance in low-resource languages, as models trained primarily on monolingual data misrecognize or mistransliterate switched terms 21 . Recent works have addressed code-switching in low-resource ASR, including fine-tuned models for Bengali-English medical transcription 22 , Hindi-English code-mixed speech 23 , and clinical nursing documentation 24 . However, to our knowledge, systematic handling of these challenges in Persian clinical speech remains largely unaddressed in the current literature, particularly regarding the integration of LLM-guided normalization within an operational EHR pipeline. To our knowledge, prior studies have not reported a Persian clinical dictation pipeline that explicitly combines LLM-guided normalization of code-switched medical terms with glossary-based standardization. Persian-specific orthographic and homophone challenges A distinct challenge in Persian ASR is the presence of homophonous letters that are acoustically indistinguishable in spoken form, leading to inconsistent orthography in transcriptions. This issue compounds code-switching errors and contributes to elevated WER in Persian clinical speech (30–40% for Whisper in real-world settings 25 , 26 ). While Gemma 3 models demonstrate strong contextual performance for Persian text correction, we did not identify prior Persian clinical voice-to-EHR studies that jointly addressed homophone disambiguation and code-switch normalization within the same operational pipeline. Methodology The overall processing pipeline of the proposed framework is illustrated in Fig. 1 . Fig. 1. Open in a new tab Overall processing pipeline of the proposed framework. Voice-to-text conversion with ASR The proposed framework initiates the processing pipeline by capturing audio input from the clinician. This audio is subsequently transcribed using an automatic speech recognition (ASR) system, primarily leveraging the Whisper model, which offers robust multilingual capabilities and reasonable performance across diverse accents. Despite these strengths, residual transcription errors persist due to factors such as background noise, rapid speech, and domain-specific challenges. Recent advancements in noise-robust ASR techniques are incorporated to mitigate these issues 27 . On the evaluation dataset, the Whisper model achieved a Word Error Rate (WER) of 18.4% on standard Persian clinical speech, increasing to 28.7% in segments exhibiting heavy code-switching. In comparison, Google Speech-to-Text produced a higher WER of 32.1%, primarily attributable to suboptimal handling of Persian phonetic characteristics. Audio preprocessing Prior to transcription, audio signals undergo preprocessing steps including noise reduction and signal enhancement to improve overall speech quality. These procedures enhance ASR robustness in noisy clinical environments and accommodate variations in speaker characteristics. Segmentation into shorter utterances further facilitates parallel processing while maintaining transcription accuracy. Challenges in non-English language transcription Transcription of non-English languages, particularly Persian, presents distinct challenges owing to phonetic complexity, regional dialectal variations, and limited domain-specific training data. Although state-of-the-art ASR systems perform adequately on English clinical speech, their accuracy on Persian remains lower due to these linguistic factors and the prevalence of informal speech patterns in clinical settings. Improving ASR for non-English languages Performance on non-English languages can be enhanced through targeted fine-tuning on domain-specific corpora that encompass regional accents, idiomatic expressions, and medical terminology. Such adaptations yield measurable improvements in transcription accuracy for specialized applications. Persian orthographic and homophone normalization An additional innovation addresses Persian-specific orthographic ambiguities arising from acoustically indistinguishable homophonous letters. These ambiguities contribute substantially to ASR errors in Persian, where spoken forms provide no disambiguation cues. Gemma 3 27b’s superior performance on Persian text (due to extensive multilingual training) makes it particularly effective for contextual resolution. The module sequentially processes the code-switched normalized transcript: A homophone mapping glossary defines preferred orthography in medical contexts. The LLM applies contextual rules to standardize spelling based on surrounding words and medical domain knowledge. The dedicated prompt is: The proposed framework addresses the phonetic variability inherent in Persian clinical ASR through a dedicated Persian Orthographic and Homophone Normalization process, as illustrated in Fig. 2 . This module leverages LLM-driven contextual refinement to disambiguate homophones–letters that are phonetically identical but orthographically distinct–supporting the semantic integrity of the medical documentation while acknowledging potential residual ambiguities in highly complex cases. Fig. 2. Open in a new tab Persian orthographic and homophone normalization process in a clinical context. Handling code-switching in persian medical terminology A key innovation in this framework is a dedicated post-ASR module for normalizing code-switched medical terms, addressing the common practice among Persian-speaking clinicians of embedding English/Latin terminology within Persian sentences. Raw transcripts from ASR often contain misspelled transliterations or omissions due to model confusion. The module operates as follows: A bilingual medical glossary is compiled from standard sources (e.g., ICD-10 equivalents in Persian, Iranian Ministry of Health terminology databases, containing approximately 5000–10,000 common terms). The raw transcript is processed by the LLM (Gemma 3 27b) using a specialized prompt to identify, correct, and standardize English medical terms while preserving the Persian structure. Optional embedding-based matching (using bilingual sentence embeddings) refines glossary lookups for ambiguous cases. The LLM prompt used for normalization is shown below: This normalized transcript then proceeds to the general LLM refinement and embedding-based mapping stages, significantly improving accuracy for medical entity extraction in BioArc forms. As demonstrated in Fig. 3 , the normalization module effectively resolves ambiguities in code-switched dictations by standardizing transliterated medical entities within the Persian syntactic structure. Fig. 3. Open in a new tab Example of code-switching normalization. Text Refinement with LLM and embedding-based techniques Following normalization, the transcript undergoes refinement using a large language model (Gemma 3 27b) to improve grammatical accuracy, logical flow, and overall readability while preserving clinical meaning. The refinement process encompasses: Grammar and Spelling Correction: Systematic correction of transcription-induced errors. Potential Contextual Enrichment: Clarification of ambiguous phrases using standard medical terminology where contextually appropriate, without introducing unsubstantiated information. Potential Coherence Enhancement: Improvement of logical flow and narrative structure based on contextual inference. Domain-Specific Adaptation: Alignment with clinical documentation conventions. Embedding-based Validation: Utilization of multilingual sentence embeddings to verify semantic consistency. A multi-step iterative prompting strategy is employed, with up to three refinement iterations to progressively improve output quality. To empirically assess qualitative improvements in coherence and contextual richness, a human evaluation was conducted on a subsample of 100 transcripts (10% of the dataset) by the same three senior medical residents who performed gold-standard annotations. Raters scored refined versus raw transcripts on a 5-point Likert scale for narrative coherence and clinical contextual completeness. The refinement stage yielded mean scores of 4.3 0.4 for coherence and 4.1 0.5 for contextual richness, compared to 3.2 0.6 and 2.9 0.7 for raw ASR outputs, respectively (paired t-test, ). These results provide direct evidence of qualitative benefits complementing the quantitative field detection gains observed in ablation studies. The general refinement uses a multi-step prompting strategy with up to three iterations. The primary prompt is: As depicted in Fig. 4 , the refinement module enhances the sentence structure and clinical context of the ASR output, correcting grammatical inconsistencies while preserving medical terminology. Fig. 4. Open in a new tab Example before/after contextual and coherence enhancement. Mapping Text to Structured Forms with Embedding-based Matching The refined transcript is subsequently mapped to the target structured form, which is defined by schema metadata specifying field types (e.g., free text, date, numeric), validation rules (e.g., required fields, value ranges), and inter-field dependencies. The mapping process proceeds in a segmented manner as follows: Field Identification: Individual segments of the refined text are matched to candidate form fields using cosine similarity computed over multilingual sentence embeddings derived from a pretrained SBERT model. Metadata Utilization: Candidate mappings are validated against field-specific constraints to ensure format compliance, such as date validity or numeric range adherence. A cosine similarity threshold of 0.75 is applied to determine viable mappings. Disambiguation with Embeddings: In cases of multiple high-similarity candidates, rankings are determined by similarity scores, with the top candidate selected unless it falls below the threshold, in which case the segment is flagged for human review. Conflict resolution further prioritizes compliance with metadata constraints, such as data type compatibility. Large language models have demonstrated considerable efficacy in extracting structured attributes from unstructured narratives, exhibiting strong performance in similar mapping tasks due to their scalability and semantic understanding. For reproducibility, the embedding model employed is paraphrase-multilingual-mpnet-base-v2. The cosine-similarity threshold ( ) was tuned exclusively on a held-out validation split and then fixed for evaluation on a disjoint test split. The metadata-guided mapping algorithm is formalized below to ensure robust and constraint-aware data entry. Algorithm 1. Open in a new tab Metadata-Guided Mapping Algorithm Error handling and quality control with embedding feedback The framework incorporates a closed-loop feedback mechanism that enables clinicians to review and correct automatically populated form fields. Each mapped field is assigned a confidence score derived from embedding similarity and metadata compliance. Fields with scores below 0.75 or those violating schema constraints are automatically flagged for manual review or supplementary validation. An embedding-based error detection component identifies potential logical or semantic inconsistencies in the extracted values, facilitating rapid correction. Within the BioArc ecosystem, existing functionality allows annotations and comments to be associated with specific fields, thereby reducing ambiguity in subsequent processing and enabling continual improvement of model performance through clinician feedback. Model evaluation and improvement The system’s efficacy is assessed using real-world clinical datasets comprising diverse medical forms and corresponding structured extractions. Performance is benchmarked against established baselines, with primary evaluation metrics including Field Detection Accuracy, Missed Fields Percentage, and Incorrectly Filled Fields Percentage. Particular emphasis is placed on hybrid configurations that integrate automated population with targeted human validation, which is especially critical in healthcare domains requiring exceptional reliability. Empirical results demonstrate superior accuracy when leveraging structured annotations and human oversight. Opportunities for further enhancement are identified, particularly in handling large, complex forms with numerous interdependent fields. Prior research on human-AI collaboration in critical domains similarly underscores the value of such hybrid approaches in optimizing data workflow efficiency 28 . Integration with platforms like BioArc enhances practical utility, facilitating smoother adoption in clinical settings, reducing data entry time, and supporting high accuracy standards while acknowledging residual challenges in complex scenarios. All methods were carried out in accordance with relevant guidelines and regulations. All experimental protocols were approved by Ilam University ethics committee. Illustrative case study: the lifecycle of a clinical dictation To provide a concrete demonstration of the framework’s modular operations, Fig. 5 illustrates the step-by-step transformation of a typical Persian ophthalmic examination dictation. Fig. 5. Open in a new tab The evolutionary pipeline of a clinical transcription: From raw ASR output with phonetic and orthographic errors to structured EHR field population in the BioArc system. The process begins with a high-WER ASR output containing phonetic misspellings and transliterated English terms. As the dictation progresses through the pipeline, the Dual-Layer Normalization module identifies and restores standard medical terminology while resolving Persian orthographic ambiguities. Following LLM refinement –which enhances narrative coherence and clinical context –the final stage employs embedding-based similarity mapping to accurately populate specific BioArc fields. This end-to-end walkthrough underscores how the integrated pipeline mitigates linguistic noise to ensure the semantic integrity of structured clinical data. Evaluation Metrics definition To quantitatively evaluate the performance of the proposed hybrid framework, we define four primary metrics. Let be the total number of target fields in a given EHR form, and represent the number of fields explicitly mentioned by the clinician in the audio input. Field Detection Accuracy (FDA): This metric measures the system’s overall success in identifying and correctly populating target fields. It is defined as the ratio of correctly filled fields ( ) to the total number of target fields in the form: 1 where a field is considered correctly filled only if both the field identity and its extracted value match the ground truth. Missed Fields Percentage (MFP): This metric quantifies the information loss during the extraction process. It represents the proportion of clinical entities mentioned in the dictation that the system failed to capture ( ): 2 Incorrectly Filled Fields (IFF): This metric measures the precision of the extraction by calculating the percentage of fields populated with erroneous or hallucinated values ( ) relative to the total number of fields the system attempted to fill ( ): 3 Correctness of Suggestions: Beyond automated filling, this metric evaluates the reliability of the system’s top-ranked mapping candidates presented to the clinician during the hybrid review phase, calculated as the frequency with which the intended field appears in the top-3 suggestions. All reported improvements were tested for statistical significance using paired t-tests ( for key comparisons). Experimental design and test setup To rigorously evaluate the proposed hybrid framework, we conducted a comprehensive experimental study using 1,000 distinct speech inputs collected within the BioArc ecosystem. This section details the data characteristics, participant demographics, and the technical environment. Dataset and participant profile The evaluation dataset comprises 1000 audio recordings (totaling approximately 15 hours of clinical dictation). These recordings were obtained from 20 unique clinicians (12 male, 8 female) aged between 30 and 55 years, ensuring a diverse range of vocal pitches and speaking rates. All participants are native Persian speakers who routinely utilize English/Latin medical terminology in their professional discourse. Unlike simulated datasets, these inputs were recorded in actual clinical environments–including outpatient clinics and emergency departments–to capture realistic acoustic characteristics such as background medical equipment noise, ambient conversations, and varying microphone distances. The speech patterns exhibit high variability, ranging from formal dictation to rapid, informal clinical notes with significant code-switching. To ensure unbiased evaluation, a strict data split was maintained: 200 recordings were reserved exclusively for threshold tuning and calibration, while the remaining 800 recordings were dedicated to final evaluation; no recording or clinician-specific audio appeared in both splits. Gold-standard annotation process To establish a ground-truth for evaluation, the 1000 speech inputs were manually transcribed and annotated by three senior medical residents with over five years of experience in clinical documentation. Consensus Mechanism: Each input was independently annotated. Discrepancies in field mapping or term normalization were resolved through a consensus-based review. Inter-Annotator Agreement: We measured the reliability of the annotations using Cohen’s Kappa ( ). The resulting score of 0.88 indicates a high level of agreement, confirming the validity of the gold-standard labels. Hardware and system configuration The computational experiments were performed on a high-performance server designed for large-scale LLM inference. The technical specifications are summarized in Table 4 . Table 4. System hardware and software specifications. Component Specification GPUs 2 NVIDIA RTX A6000 (48 GB GDDR6 each) CPU Intel Xeon Gold 6252R (24 Cores, 48 Threads, 2.30 GHz) RAM 128 GB DDR4 ECC Storage 2 TB NVMe Gen4 SSD Software Stack PyTorch 2.1, CUDA 12.8, Transformers Library, Gemma 3 27b Embedding Model paraphrase-multilingual-mpnet-base-v2 Similarity Metric Cosine similarity (threshold: 0.75) Open in a new tab Experimental scenarios We evaluated the framework across four distinct configurations to quantify the impact of clinical annotations and human intervention: Scenario 1 (Autonomous - No Metadata): The system operates purely on ASR and LLM refinement without access to BioArc-specific field annotations or historical metadata. Scenario 2 (Autonomous - Metadata-Guided): The system utilizes BioArc field metadata and existing clinical annotations to constrain the mapping process. Scenario 3 (Hybrid - Metadata-Guided): Combines the automated pipeline from Scenario 2 with selective human validation by clinical interns for fields flagged with low confidence scores. Scenario 4 (Hybrid - No Metadata): A combination of the baseline automated system and human review, intended to isolate the value of structural metadata. Human-in-the-loop review process In hybrid scenarios, flagged fields (confidence < 0.75 or constraint violation) were reviewed by three clinical interns (medical residents with 3–5 years experience). Reviewers were blinded to system configuration and received training on the interface (30-minute session). Disagreements were resolved by majority vote; inter-reviewer agreement (Fleiss’ kappa) was 0.82. Average review time per flagged form was 45 seconds, demonstrating scalability (only 12% of fields typically flagged). Total processing time for hybrid scenarios was calculated as the sum of automated system latency (ASR, normalization, and mapping) and the active time spent by clinicians on the review interface. To provide a granular view, these components are reported separately as and in our results. Baseline comparisons For external validation, we compared the proposed framework against two baselines: (i) a rule-based pipeline using keyword matching and regular expressions on raw ASR transcripts, and (ii) a simulated commercial voice-EHR baseline intended to approximate an off-the-shelf multilingual deployment. The Dragon Medical baseline was simulated by utilizing the state-of-the-art Whisper Large-v3 engine as the ASR backbone, coupled with standard GPT-4o prompting for extraction. Critically, this baseline lacked our specialized Persian medical glossaries, code-switch normalization modules, and BioArc-specific metadata constraints, representing a typical ’off-the-shelf’ deployment scenario. The proposed hybrid with metadata and normalization substantially outperformed both baselines, as detailed in Tables 6 and 7 . Table 6. Comparison of Evaluation metrics for medium-sized forms, including baselines (mean 95% CI). Model/Scenario FDA (%) MFP (%) IFF (%) Processing Time (min) Rule-based Baseline 76% 1.8 15% 1.2 18% 1.5 2.1 0.2 Dragon Medical (Simulated) 83% 1.5 11% 1.0 13% 1.2 1.9 0.2 Proposed (Auto, No Annot.) 90% 1.2 7% 0.8 10% 1.1 1.8 0.2 Proposed (Auto, With Annot.) 94% 0.9 4% 0.5 5% 0.7 1.7 0.1 Proposed (Hybrid, No Annot.) 96% 0.8 2% 0.3 3% 0.4 (Total: 1.6 0.2) Proposed (Hybrid, With Annot.) 98% 0.6** 1% 0.2** 1% 0.2** (Total: 1.6 0.2) Open in a new tab ** vs. both baselines (paired t-test). Table 7. Comparison of evaluation metrics for large forms, including baselines (mean 95% CI). Model/Scenario FDA (%) MFP (%) IFF (%) Processing Time (min) Rule-based Baseline 68% 2.2 22% 1.8 25% 2.0 4.0 0.4 Dragon Medical (Simulated) 75% 1.9 16% 1.4 19% 1.6 3.8 0.3 Proposed (Auto, No Annot.) 85% 1.8 12% 1.3 15% 1.7 3.5 0.3 Proposed (Auto, With Annot.) 92% 1.2** 6% 0.8** 7% 1.0 3.5 0.2 Proposed (Hybrid, No Annot.) 86% 1.6 8% 0.9 10% 1.2 (Total: 3.7 0.3) Proposed (Hybrid, With Annot.) 92% 1.1** 5% 0.6** 5% 0.7** (Total: 3.7 0.3) Open in a new tab ** vs. both baselines (paired t-test). Error analysis and failure modes To identify areas for future improvement, a granular error analysis was performed on the failure cases within the evaluation dataset. Errors were categorized into three primary stages: ASR & Code-Switching Errors (42%): These primarily involved phonetic mistransliterations of English medical entities embedded in Persian dictation, often caused by high ambient noise or rapid speech. While the code-switching module reduced ASR-related errors by 58%, failures occurred when phonetic noise was excessive. Refinement Errors (18%): These instances involved the LLM over-correcting telegraphic dictation or introducing unsupported modifiers, which occasionally altered the original clinical nuance without changing core facts. Mapping Errors (40%): Errors were concentrated in semantically adjacent fields, such as confusing symptom onset with duration or drug dose with frequency, particularly when metadata cues were sparse. Table 5 provides qualitative examples of these recurrent failure modes to illustrate the current limitations of the framework. Table 5. Qualitative examples of primary failure modes. Category Input/audio fragment System error/failure mode ASR “Patient has hypertension” Misrecognized “Hypertension” as a phonetically similar but nonsensical Persian word. Refinement “Pain... severe... 3 days” Over-corrected by introducing stylistic modifiers and function words not present in the original audio. Mapping “Onset 2 days ago” Incorrectly mapped “2 days” to the Duration field instead of the Onset field due to high semantic similarity. Open in a new tab BioArc form evaluation Forms within the BioArc system are categorized by complexity into medium-sized (10–20 fields) and large (exceeding 30 fields) variants. This classification is based on field count, data type diversity, and structural complexity. Representative examples of medium- and large-sized forms are provided in Figs. 6 and 7 , respectively. Fig. 6. Open in a new tab Example of a medium-sized form in BioArc 29 . Fig. 7. Open in a new tab Example of a large form in BioArc 29 . Medium-sized forms Medium-sized forms typically comprise between 10 and 20 fields, predominantly featuring straightforward data types such as free text entries, date fields, and basic numerical inputs. Consequently, the computational and linguistic processing demands imposed on the system remain relatively modest, allowing for more efficient automated handling. Large forms In contrast, large forms contain more than 30 fields and incorporate a greater variety of complex data types, including interconnected categorical selections, nested sections, and conditional logic. The increased density of information and heightened variability in spoken dictation introduce additional challenges, placing greater demands on the system’s robustness in managing field density and structural intricacy. Results for medium-sized forms Evaluation was conducted on one hundred voice inputs corresponding to medium-sized forms. The results, presented in Table 6 , demonstrate substantial performance superiority of the proposed framework over both baselines, with the fully configured hybrid approach achieving the highest metrics. All improvements of the proposed configurations over baselines were statistically significant (paired t-test, ). Results for large forms A parallel evaluation involving one hundred voice inputs targeted at large forms yielded the results summarized in Table 7 . Although performance gains were somewhat attenuated due to increased complexity, the proposed hybrid configuration still substantially outperformed the baselines across key metrics. These improvements achieved statistical significance relative to baselines ( ). Human evaluation of qualitative refinement attributes A dedicated human evaluation was performed to quantify improvements in narrative coherence and contextual richness introduced by the LLM refinement module. Both attributes were rated on a 5-point Likert scale, where 1 indicated severely degraded or clinically unusable output and 5 indicated fluent, coherent, and clinically usable documentation. Raters were instructed to score coherence based on grammatical continuity and logical flow, and contextual richness based on retention of clinically relevant details without unsupported additions. As shown in Table 8 , the LLM refinement module improved both qualitative attributes. The mean coherence score increased from 2.15 ± 0.62 to 4.68 ± 0.45, and contextual richness increased from 2.38 ± 0.58 to 4.52 ± 0.51. These results indicate that refinement improved fluency and preserved clinically relevant content more consistently than raw ASR output. The inter-rater agreement (Fleiss’ kappa) for these qualitative scores was 0.79, indicating substantial consistency among the medical experts. Table 8. Human evaluation scores for qualitative refinement attributes (N = 100, Mean SD). Attribute Raw ASR Output LLM-Refined Output p-value Coherence 2.15 0.62 4.68 0.45 Contextual Richness 2.38 0.58 4.52 0.51 Open in a new tab Granular ablation study: modular component contribution To address the requirement for quantifying the isolated contribution of each pipeline stage, we expanded the ablation study into a four-stage incremental analysis. We evaluated 200 speech inputs (100 medium and 100 large forms) across the following configurations: ASR Only: Raw output from the Whisper Large-v3 engine without any post-processing. +Refinement: Addition of the LLM-based grammar and coherence enhancement iteration. +Normalization: Integration of the dual-layer Code-Switching and Persian Orthographic normalization modules. Full System (+Metadata): The complete pipeline utilizing BioArc field metadata and historical annotations to constrain the final mapping. The results, illustrated in Fig. 8 , reveal that while ASR provides a foundational transcript, its performance in complex clinical scenarios is insufficient (FDA of 64.1% for large forms). The most significant leap in accuracy and Medical Entity F1 occurs with the introduction of the Normalization module, which resolves the phonetic ambiguities of code-switched terms. Furthermore, the final integration of Form Metadata reduces the Incorrectly Filled Fields (IFF) to near-zero levels in medium forms ( ). Fig. 8. Open in a new tab Granular ablation study results demonstrating the incremental performance gains of each modular component across medium and large EHR forms. Error bars indicate standard deviation over 10 independent runs; significance markers (***) denote . Statistical significance was verified using paired t-tests between each incremental stage. The transition from ASR-only to the Full System yielded a highly significant improvement ( ) across all metrics, as indicated by the significance markers in Fig. 8 . Discussion The empirical results indicate that the highest performance across nearly all metrics is achieved by the hybrid approach incorporating form annotations. Benefits are particularly pronounced in more complex scenarios. For medium-sized forms, field detection accuracy increases by approximately 2 percentage points when moving from annotated autonomous processing to the full hybrid configuration, with larger forms exhibiting a more modest 1 percentage point gain. Reductions in missed fields are comparable across form sizes, averaging approximately 3 percentage points in the hybrid setting. The hybrid configuration consistently improves suggestion correctness, while the most substantial error mitigation is observed in the Incorrectly Filled Fields Percentage, which declines markedly–from 10% to 3% in medium forms and by 5 percentage points in large forms. To precisely quantify the human-in-the-loop time savings, we analyzed the workload distribution between the AI and clinicians. While manual entry for a large BioArc form requires minutes, our hybrid framework reduces the active human involvement to an average of only 0.2 minutes (approx. 12 seconds) per form across the dataset. This efficiency is achieved because human intervention is triggered only for the 12% of fields that fall below the 0.75 confidence threshold. Consequently, even with the 45-second overhead required for each flagged review session, the clinician achieves a total time saving of approximately 55–65% compared to traditional manual documentation, without compromising the safety and precision of the EHR record. These performance trends across experimental scenarios and form sizes are further visualized in Figs. 9 (medium forms), 10 (large forms), and 11 (aggregated heatmap). Fig. 9. Open in a new tab Field Detection Accuracy, Missed Fields Percentage, Incorrectly Filled Fields Percentage, and Suggestion Correctness for different scenarios of medium-sized forms. Fig. 10. Open in a new tab Comparing evaluation metrics across scenarios for large forms, illustrating the impact of form complexity on performance gains. Fig. 11. Open in a new tab Heatmap of evaluation metrics across all scenarios and form sizes, with higher values in lighter shades indicating better performance for positive metrics like accuracy. The sample size (1000 inputs, 200 for ablation) provides sufficient statistical power for the observed effects, although broader generalizability to additional low-resource languages warrants future investigation. A key aspect of the framework’s reproducibility is its modular design, but its transferability is partial rather than universal. While the metadata-mapping layer is tailored to the BioArc schema, the core architectural components–including the dual-layer normalization, the iterative LLM refinement prompts, and the confidence-based human-in-the-loop escalation policy–are domain-agnostic. These modules can be directly integrated into other EHR systems or low-resource language frameworks with minimal adaptation of the underlying medical glossaries. Safety, governance, and ethical considerations To ensure responsible deployment in clinical environments, the framework incorporates a multi-layered Safety and Governance Funnel (illustrated in Fig. 12 ). This architecture is designed to enforce clinical data integrity through a combination of deterministic schema constraints and probabilistic uncertainty filters, strictly mitigating the risk of LLM-generated hallucinations. Fig. 12. Open in a new tab The multi-layered Safety and Governance Funnel: A sequence of deterministic schema constraints and probabilistic uncertainty thresholds (e.g., cosine similarity> 0.75) used to safeguard clinical data integrity in the BioArc system. The safety funnel subjects every potential field entry to three rigorous validation layers: Layer 1: Schema-Constrained Extraction: The framework retrieves field-specific metadata directly from the BioArc EHR schema (e.g., data types such as numeric, date, or categorical). During the mapping process, these constraints are injected into the LLM prompts to narrow the generation space. A post-extraction validation script ensures that a numeric field, for instance, does not contain textual noise. Layer 2: Clinical Range and Enum Validation: To ensure medical plausibility, the system validates extracted values against predefined physiological ranges (e.g., blood pressure restricted to 40–250 mmHg) and standard enumerations (e.g., FHIR-compatible status codes). Any out-of-range value is flagged immediately for human intervention. Layer 3: Uncertainty Calibration: Each mapping is assigned a confidence score based on the cosine similarity between the refined text segment and the target field embedding. Entries with a score below the 0.75 threshold are diverted to the clinician for manual verification. Audit Logging and Clinician Override: All pipeline stages are logged with timestamps and versioned transcripts for full traceability. The user interface features a Clinician Override UX with visual indicators (e.g., colored borders and tooltips showing violation details), allowing physicians to modify or reject suggestions without disrupting the clinical workflow. This feedback loop ensures that automated efficiency never compromises the stringent safety standards required for EHR documentation. Ethical considerations and fairness The design and implementation of the framework are guided by rigorous principles of safety, equity, and responsible innovation. Data availability Data utilized in this study were sourced from the BioArc platform 29 . Access is restricted in accordance with institutional licensing agreements and privacy regulations. The dataset is not publicly available due to patient-related sensitivities; however, de-identified subsets may be shared upon reasonable request and with explicit approval from BioArc authorities 29 . All sharing is governed by formal data-use agreements between providers and researchers. Data privacy and security Stringent data governance protocols were adhered to throughout the study. Personally identifiable information was systematically anonymized, and all data were stored and processed using end-to-end encryption. Due to the retrospective nature of the study and the use of anonymized records from the BioArc data warehouse, the requirement for individual informed consent was waived by the Ilam University Ethics Committee. Conclusion The proposed hybrid framework demonstrates substantial improvements in voice-driven structured data entry within the evaluated Persian clinical context using the BioArc EHR system. Although residual error rates of approximately 5–10% persist in complex forms, the approach significantly reduces documentation burden and supports data integrity when augmented with selective human oversight. These results suggest that modular hybrid AI systems can improve structured clinical documentation in low-resource settings, although broader validation across additional institutions, EHR schemas, and languages is still needed. Future work Future research directions focus on refining the hybrid methodology and extending its applicability across diverse domains. Priority areas include the following: Improved Error Detection and Contextual Understanding: Advanced techniques for identifying and resolving errors in challenging acoustic conditions are needed. Enhanced handling of ambiguous or degraded inputs would further elevate performance in both autonomous and hybrid configurations. Targeted fine-tuning of automatic speech recognition models on domain-specific corpora–encompassing regional accents, colloquial variants, background noise, and code-switched medical terminology in languages such as Persian–represents a promising avenue. Optimizing Human Review for Large Forms: Although the hybrid approach excels with medium-sized forms, large forms require more streamlined review interfaces to minimize clinician fatigue while sustaining error reduction. Contextual ranking and prioritized flagging mechanisms could improve efficiency in high-field-density scenarios. Domain-Specific Fine-Tuning: Further specialization of underlying models through continued training on clinical corpora would strengthen recognition and normalization of medical entities, particularly code-switched terminology. Cross-Domain Expansion: Extending the framework beyond healthcare to domains such as inventory management, legal documentation, and administrative reporting would validate its generalizability, with particular attention to additional low-resource linguistic contexts. Ongoing advancements in automatic speech recognition, large language models, and human-computer interaction are expected to drive continued progress in hybrid systems. By synergistically combining automated scalability with human expertise, such frameworks can effectively manage complex structured data entry tasks across both routine and high-stakes applications. Author contributions Mohammad Tanhaei: Conceptualization, Methodology, Software Development, Data Analysis, Writing - Original Draft, Writing - Review & Editing. Funding No funding was received for this work. Data availability The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Hodgson, T. & Coiera, E. Risks and benefits of speech recognition for clinical documentation: A systematic review. J. Am. Med. Inform. Assoc. 23 , e169–e179. 10.1093/jamia/ocv152 (2016). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Ng, J. J. W. et al. Evaluating the performance of artificial intelligence-based speech recognition for clinical documentation: A systematic review. BMC Med. Inform. Decision Making 25 , 236. 10.1186/s12911-025-03061-0 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 3. Jiang, F. et al. Artificial intelligence in healthcare: Past, present and future. Stroke Vasc. Neurol. 2 , 230–243. 10.1136/svn-2017-000101 (2017). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Radford, A. et al. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 10.48550/arXiv.2212.04356 (2022). 5. Cloud, G. Google speech-to-text (2023). Accessed: 2025-09-29. 6. Hannun, A. et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 10.48550/arXiv.1412.5567 (2014). 7. Goss, F. R., Zhou, L. & Weiner, S. G. Incidence of speech recognition errors in the emergency department. Int. J. Med. Inform. 82 , 799–805. 10.1016/j.ijmedinf.2016.05.005 (2013). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) 10.18653/v1/N19-1423 (2019). 9. Brown, T. et al. Language models are few-shot learners. Adv. Neural Inform. Process. Syst. 33 , 1877–1901 (2020). [ Google Scholar ] 10. Team, G. Gemma: Open models based on gemini research and technology. Tech. Rep., Google DeepMind (2024). 10.48550/arXiv.2403.08295. 11. Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. The art of llm refinement: Ask, refine, and trust. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) 10.18653/v1/2024.naacl-long.327 (2024). 12. Li, J., Chen, W., Sun, Y. & Zeng, G. Deep learning for named entity recognition: A survey. Neural Comput. Appl. 10.1007/s00521-024-09646-6 (2024). [ Google Scholar ] 13. Nasar, Z., Jaffry, S. W. & Malik, M. K. Information extraction from scientific articles: A survey. Scientometrics 117 , 1931–1990. 10.1007/s11192-018-2921-5 (2018). [ Google Scholar ] 14. Conneau, A. et al. Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 10.18653/v1/2020.acl-main.747 (2020). 15. Lin, Z. et al. Generative artificial intelligence for automated data extraction from medical images: a comprehensive review. J. Am. Med. Inform. Assoc. Open . 10.1093/jamiaopen/ooaf097 (2025). 16. Sirmokadam, S. Speech to text for data entry–opportunities and challenges. Proceedings of the International Conference on Data Management, Analytics & Innovation 355–362, 10.1007/978-981-19-2600-6_25 (2022). 17. Shaik, T. et al. A machine learning approach for automated filling of categorical fields in data entry forms. ACM Trans. Softw. Eng. Methodol. 31 , 1–40. 10.1145/3533021 (2022). [ Google Scholar ] 18. Alkhammash, E. H., Hadjouni, M. & Elshewey, A. M. A hybrid ensemble stacking model for gender voice recognition. Electronics 11 , 1750. 10.3390/electronics11111750 (2022). [ Google Scholar ] 19. Harihar, D., Shrivastava, V., Talele, P. & Jahagirdar, A. Voice-based user interface for hands-free data entry and automation at workplaces. MethodsX . 10.1016/j.mex.2025.103596 (2025). [ DOI ] [ PMC free article ] [ PubMed ] 20. Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. Human-ai collectives most accurately diagnose clinical vignettes. Proc. Natl. Acad. Sci . 10.1073/pnas.2426153122 (2024). [ DOI ] [ PMC free article ] [ PubMed ] 21. Liu, H. et al. Aligning speech to languages to enhance code-switching speech recognition. arXiv preprint arXiv:2403.05887 10.48550/arXiv.2403.05887 (2024). 22. Ghosh, P. & Talukder, S. Medibeng whisper tiny: A fine-tuned code-switched bengali-english model for medical transcription in healthcare settings. medRxiv preprint 10.1101/2025.04.25.25326406 (2025). 23. Biswas, A., Shevelev, O., Abdaoui, A., Tyagi, V. & Boumadane, A. Adapting whisper for low-resource hindi-english code-mix speech with on-the-fly augmentation & llm-synthesised data. Proc. Interspeech 2025 , 4293–4297 (2025). [ Google Scholar ] 24. Hou, B.-H. & Wu, C.-H. Code-switching automatic speech recognition for nursing record documentation: System development and evaluation. JMIR Nursing 5 , e37562. 10.2196/37562 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Team, S. Soniox vs openai persian speech-to-text comparison (2025). 26. Authors, V. A comprehensive benchmark for evaluating persian asr systems. arXiv preprint arXiv:2505.21230 10.48550/arXiv.2505.21230 (2025). 27. Black, A. W. et al. Spoken dialog challenge 2010: Comparison of live and control test results. In Proceedings of the SIGDIAL 2011 Conference , 2–7 (2011). 28. Chen, L., Zaharia, M. & Zou, J. Human-ai teaming in critical care: A comparative analysis of data scientists and clinicians. JMIR Human Factors 11 , 10.2196/50130 (2024). [ DOI ] [ PMC free article ] [ PubMed ] 29. BioArc. Smart hospital information system, a platform for creating electronic health records for patients, smart assistant for physicians (2024). Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement Data utilized in this study were sourced from the BioArc platform 29 . Access is restricted in accordance with institutional licensing agreements and privacy regulations. The dataset is not publicly available due to patient-related sensitivities; however, de-identified subsets may be shared upon reasonable request and with explicit approval from BioArc authorities 29 . All sharing is governed by formal data-use agreements between providers and researchers. The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. Articles from Scientific Reports are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (3.2 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Augmenting structured diagnoses through effective use of pre-trained large language models on clinical notes.#685643
• Dictionary-Augmented Large Language Model Postprocessing for Bilingual Code-Switched Medical Speech Recognition: Development and Evaluation Study.#354669
• Language-Based Digital Twins for Elderly Cognitive Assistance#310842
• ChatGPT for good? On opportunities and challenges of large language models for education#359595
• A Large Language Model-Based Simulation Protocol Using ChatGPT for Training Novice Critical Care Nurses in Family Crisis Communication and De-escalation.#453015