When LLMs Analyze Scars: From Images to Clinically-Meaningful Features Ruman Wang1 , Hangting Ye2 Liaoning University of Traditional Chinese Medicine1 School of Artificial Intelligence, Jilin University2 [email protected], [email protected]
arXiv:2606.18063v1 [cs.CV] 16 Jun 2026
Abstract Medical image classification faces a fundamental dilemma: while deep learning models achieve remarkable performance at scale, real-world clinical scenarios often suffer from severe data scarcity due to annotation costs, privacy constraints, and disease rarity. This challenge is particularly pronounced in pathological scar classification, where differentiating keloids from hypertrophic scars requires subtle expert knowledge and labeled images are extremely limited. We propose a novel paradigm that repositions large language models (LLMs) as knowledgedriven feature engineers rather than end-to-end classifiers. We call this framework ScaFE (Scar Feature Engineering). Our key insight is that LLMs encode rich medical knowledge that can be externalized as executable feature extraction code, enabling the transformation of high-dimensional images into lowdimensional, clinically interpretable representations. Specifically, we prompt an LLM with established scar assessment criteria to generate deterministic Python code that extracts features aligned with clinical scoring systems such as the Vancouver Scar Scale. Our approach offers three key advantages: (1) data efficiency, achieving robust performance with limited training samples by decoupling knowledge acquisition from statistical learning; (2) privacy preservation, as raw images are processed locally without exposure to external LLMs; and (3) interpretability, through explicit features grounded in clinical reasoning. Extensive experiments on scar classification demonstrate that our method consistently outperforms end-to-end deep learning baselines or using LLMs as black-box classifiers under limited data conditions, establishing a promising direction for integrating LLMs into data-efficient and clinically transparent medical AI systems.
1
Introduction
Pathological scars, including keloids and hypertrophic scars, represent a significant clinical challenge affecting millions of patients worldwide [Bayat et al., 2003; Sidgwick and Bayat, 2012]. Although these two conditions share similar visual
appearances, their underlying pathophysiology, prognosis, and treatment strategies differ substantially: keloids extend beyond the original wound boundaries and exhibit high recurrence rates, while hypertrophic scars remain confined to the wound site and often regress spontaneously [Berman et al., 2017]. Accurate differentiation is therefore essential for appropriate treatment planning; however, diagnosis requires substantial expert experience and remains subject to considerable interobserver variability. From a machine learning perspective, automated scar classification exemplifies a broader challenge in medical AI: the tension between model complexity and data availability. Stateof-the-art deep neural networks demand large-scale labeled datasets to learn robust representations, yet such data are scarce in specialized medical domains. Acquiring high-quality scar annotations requires board-certified dermatologists or plastic surgeons, making large-scale labeling prohibitively expensive. Privacy regulations further restrict data sharing across institutions, resulting in small, heterogeneous datasets that violate the assumptions underlying data-hungry models. Consequently, end-to-end neural networks trained directly on raw images often overfit to spurious correlations and fail to generalize in clinical deployment [Litjens et al., 2017]. Recent advances in large language models (LLMs) offer a promising alternative pathway. Modern LLMs encode vast amounts of medical knowledge distilled from clinical literature, textbooks, and practice guidelines [Singhal et al., 2023; Nori et al., 2023]. Unlike conventional neural networks that learn features purely from data, LLMs can articulate expertlevel clinical reasoning and operationalize established diagnostic criteria. However, directly applying multimodal LLMs to medical images introduces practical concerns: sensitive patient data must be transmitted to external servers, model outputs lack reproducibility due to inherent stochasticity, and the black-box nature of such systems undermines clinical trust. These observations motivate our central question: How can we leverage the rich clinical knowledge embedded in LLMs for medical image understanding without directly exposing raw images or relying on large-scale end-to-end training? Inspired by the diagnostic reasoning process of human clinicians, we observe that expert physicians rarely make diagnoses based solely on holistic visual impressions. Instead, they systematically translate visual patterns into structured, clinically meaningful concepts, such as pigmentation abnor-
mality, surface texture, vascular prominence, and boundary characteristics, before synthesizing these observations into a diagnostic conclusion. This cognitive process naturally suggests a decomposition: first extract interpretable features that capture clinical semantics, then perform classification in this structured feature space. Building on this insight, we propose to use LLMs as knowledge-based feature engineers that bridge the gap between raw medical images and structured clinical reasoning. Rather than training models to learn features from data, we prompt a medical LLM to generate executable Python code that extracts clinically meaningful features according to established scar assessment criteria such as the Vancouver Scar Scale (VSS) and the Patient and Observer Scar Assessment Scale (POSAS) [Busche et al., 2018]. The generated code defines a deterministic mapping from images to interpretable feature vectors, which are then classified using lightweight machine learning models. This design resolves the fundamental tension between model capacity and data availability. By externalizing domain knowledge into the feature construction stage, our approach enables robust learning even with limited training samples. Since feature extraction is performed locally through generated code, patient images never leave the clinical environment. Moreover, the extracted features are directly interpretable and aligned with clinical vocabulary, facilitating transparent decision-making and expert validation. Our contributions can be summarized as follows: • We propose ScaFE (Scar Feature Engineering), a framework that leverages LLMs as knowledge-driven feature engineers, transforming medical images into clinically meaningful structured representations via generated executable code. • We demonstrate that ScaFE with lightweight classifiers achieves superior performance in low-data scar classification settings, outperforming end-to-end deep learning baselines. • We provide an interpretable and privacy-preserving alternative to conventional medical image analysis, with features explicitly grounded in established clinical assessment scales.
2
Related Work
2.1
Automated Scar Assessment
Computational approaches to scar assessment have evolved from handcrafted feature engineering to end-to-end deep learning. Early methods relied on features derived from clinical scoring systems such as the Vancouver Scar Scale [Baryza and Baryza, 1995] and the Patient and Observer Scar Assessment Scale [Draaijers et al., 2004], extracting color, texture, and morphological descriptors that mirror human assessment criteria. While interpretable, these approaches required extensive manual design and exhibited limited robustness across diverse imaging conditions. Recent work has applied convolutional neural networks to scar classification and severity prediction [Zhu et al., 2016], achieving promising results under controlled experimental settings. However, these data-driven approaches assume access to large, well-annotated datasets—an assumption that
rarely holds in clinical practice where labeled scar images are scarce and expensive to obtain. Furthermore, the end-to-end nature of deep learning models renders their decision-making opaque, hindering clinical adoption where interpretability is paramount.
2.2
Large Language Models in Medical Imaging
The emergence of large language models has opened new possibilities for medical image analysis. Multimodal LLMs such as GPT-4V [Achiam et al., 2023] and Med-PaLM [Singhal et al., 2023] have demonstrated impressive capabilities in interpreting medical images and generating clinically relevant explanations. Several studies have explored using visionlanguage models for dermatological diagnosis [Liu et al., 2020; Zhou et al., 2024], radiology report generation [Chen et al., 2020], and general medical visual question answering [Li et al., 2023]. Despite their strong reasoning capabilities, directly applying multimodal LLMs to clinical diagnosis presents significant challenges. First, transmitting sensitive patient images to external API servers raises privacy concerns under regulations such as HIPAA [Gostin et al., 2009]. Second, the stochastic nature of LLM outputs compromises reproducibility, a critical requirement in medical applications. Third, these models often function as black boxes, providing explanations that may not align with established clinical frameworks. Our work addresses these limitations by using LLMs as code generators rather than direct classifiers, preserving privacy while ensuring deterministic and interpretable feature extraction.
2.3
Learning from Structured Representations
Learning from structured, low-dimensional representations has long been recognized as effective in data-limited settings [Bengio et al., 2013]. Classical approaches combine domain knowledge with statistical learning through feature engineering, rulebased systems, or hybrid architectures [Domingos, 2012]. Recent work has revisited this paradigm by integrating symbolic representations with neural networks, demonstrating improved sample efficiency and interpretability [Garcez and Lamb, 2023; Ye et al., 2024]. In medical AI, structured representations derived from clinical knowledge have shown particular promise. Concept bottleneck models [Koh et al., 2020] learn intermediate clinical concepts before final prediction, enabling human intervention and interpretation. Knowledge-guided neural networks incorporate medical ontologies and clinical guidelines into the learning process [Xie et al., 2019]. Our approach extends this line of research by using LLMs as automated generators of clinically grounded feature extractors, eliminating the need for manual feature design while ensuring alignment with established medical criteria.
2.4
Positioning of Our Work
Our method occupies a unique position in the landscape of medical image analysis. Unlike end-to-end deep learning, we do not require large-scale training data, as domain knowledge is externalized through LLM-generated code. Unlike direct LLM-based classification, we preserve privacy by processing images locally and ensure reproducibility through
Table 1: Comparison of our approach with existing paradigms. Method End-to-End DL Multimodal LLM Manual Features Ours
Data Eff.
Privacy
Interpret.
Auto.
✗ ✓ ✓ ✓
✓ ✗ ✓ ✓
✗ ✗ ✓ ✓
✓ ✓ ✗ ✓
deterministic feature extraction. Unlike manual feature engineering, our approach automatically translates clinical criteria into executable code, reducing human effort while maintaining interpretability. Table 1 summarizes these distinctions.
3
Problem Formulation
Let I ⊂ RH×W ×3 denote the space of RGB clinical scar images, and let Y = {1, . . . , C} represent the set of C scar categories (e.g., keloid and hypertrophic scar). Given a labeled training dataset Dtrain = {(Ii , yi )}N i=1 ,
Ii ∈ I, yi ∈ Y,
(1)
where N is typically small due to annotation costs and privacy constraints, our goal is to learn a classifier f : I → Y that accurately predicts scar categories for unseen test images. Challenges of End-to-End Learning. The conventional approach trains a high-capacity neural network fθ : I → Y by minimizing empirical risk over Dtrain . However, when N is small relative to the model capacity, this leads to overfitting: the learned features capture dataset-specific artifacts rather than clinically meaningful patterns. Moreover, the black-box nature of such models hinders interpretability and clinical trust. Proposed Decomposition. To address these challenges, we decompose the classification task into two stages: 1. Feature Extraction: A mapping ϕ : I → RK that transforms raw images into a K-dimensional feature space capturing clinically meaningful attributes. 2. Lightweight Classification: A simple classifier h : RK → Y trained on the structured feature representations. The composite classifier is then f = h ◦ ϕ, with prediction given by ŷ = h(ϕ(I)). The key insight is that designing ϕ to encode clinical knowledge substantially reduces the effective hypothesis space, enabling robust learning from limited data. Rather than learning features from scratch, we leverage prior medical expertise to construct representations that align with established diagnostic criteria. Design Requirements for ϕ. An effective feature extractor ϕ for our setting should satisfy: • Clinical Meaningfulness: Features should correspond to attributes used by clinicians (e.g., erythema, texture, boundary characteristics). • Determinism: Given the same input image, ϕ should produce identical outputs, ensuring reproducibility. • Privacy Preservation: Feature extraction should be performed locally without transmitting raw images externally.
In the following section, we describe how we leverage large language models to automatically construct ϕ by generating executable feature extraction code grounded in established scar assessment criteria.
4
Methodology
4.1
Overview
Our framework explicitly decouples domain knowledge acquisition from statistical learning. While end-to-end neural networks entangle feature learning and classification within a single optimization process, we separate these stages by introducing an LLM-guided feature construction module that injects expert knowledge prior to learning. As illustrated in Figure 1, the framework consists of three components: (i) LLM-based code generation, which externalizes medical knowledge into executable Python code; (ii) structured feature extraction, which deterministically maps images to interpretable clinical features; and (iii) lightweight classification, which performs statistical learning in the lowdimensional feature space.
4.2
LLM-Guided Feature Code Generation
Prompt Design. We design a structured prompt P to elicit clinically grounded feature extraction code from an LLM. The prompt comprises three components: P = Prole ⊕ Pmedical ⊕ Pcode , (2) where ⊕ denotes concatenation and: • Prole establishes the expert persona (e.g., “You are a boardcertified plastic surgeon specializing in scar assessment...”). • Pmedical is accomplished by the assistance of clinicians. It encodes domain knowledge by referencing established clinical scales including the Vancouver Scar Scale (VSS) and Patient and Observer Scar Assessment Scale (POSAS). It also specifies the expected feature categories, which is detailed in Section 4.4. • Pcode specifies the output format, including the Python function signature, required libraries (OpenCV, scikit-image). Prompt Template. We use a structured prompt to instruct the LLM to generate deterministic feature extraction code. An example template is in Template 4.2: Prompt Template Prole : You are a board-certified plastic surgeon specializing in scar assessment. Pmedical : Use the Vancouver Scar Scale (VSS) and POSAS criteria to design interpretable image features. Include features in four groups: color (LAB statistics), texture (LBP, gradients, entropy), morphology (solidity, circularity, elongation), and clinical composite scores. Pcode : Output a Python function extract_features(image) that returns a fixed-length vector. Use OpenCV and scikit-image only. The function must be deterministic and return a NumPy array of floats.
Code Generation. Given an LLM with parameters Θ, we obtain the feature extraction code: G = LLMΘ (P) ∈ CPython , (3) where CPython denotes the space of valid Python programs. The generated code G defines a deterministic function that maps each image I to feature vectors F ∈ RK : G : I → RK ,
I 7→ F = [F1 , F2 , . . . , FK ]⊤ ,
(4)
context You are a plastic surgeon in scar assessment ... ... Based on VSS and POSAS criteria ... Features in four groups … … Use OpenCV and scikit-image only … … Function must be deterministic and return a NumPy array of floats. question
user
Generate interpretable feature code for scar assessment.
Code
LLM
Data（Image）
prompt hypertrop hic scar
Color
Texture
Morphological
Clinical Composite
…
…
…
…
keloid
or Prediction
Features
Machine learning algorithm
Figure 1: Overview of the proposed LLM-guided feature engineering framework. An LLM generates feature extraction code based on clinical criteria, which transforms scar images into structured representations for classification.
where each element of F is a clinical-meaningful feature. Crucially, the code generation process via LLM (Eq. 3) is data-agnostic, and it preserves the patient privacy information without exposing raw scar images to the LLM, where LLM reasons over its own encoded clinical knowledge. In addition, this process yields a reusable code without requiring any LLM fine-tuning, which can then be used to process all the patient samples locally (Eq. 4). Code Validation and Refinement. To ensure robustness, we validate the generated code through: (i) syntax checking, (ii) execution on sample images, and (iii) verification of output dimensionality. If validation fails, we provide error feedback to the LLM for iterative refinement until valid code is obtained. In practice, we find that well-designed prompts typically yield valid code within one or two iterations.
4.3
Clinical Assurance and Case Study
Clinical Assurance. We co-designed the prompt with a board-certified plastic surgeon specializing in scar assessment. The expert reviewed the clinical criteria (VSS/POSAS), verified the terminology, and confirmed that the requested features reflect routine clinical practice. This ensures the generated code encodes clinically meaningful measurements rather than arbitrary image statistics.
4.4
Clinical Assurance and Case Study
Clinical Assurance. We co-designed the prompt with a board-certified plastic surgeon specializing in scar assessment. The expert confirmed that the four feature groups correspond to clinical criteria used in VSS/POSAS: vascularity and pigmentation (color), surface roughness and pliability (texture), boundary spread and elevation (morphology), and holistic severity (composites). This ensures the extractor operationalizes clinically meaningful cues rather than arbitrary image statistics.
Case Study: From Image I to Features. To illustrate how the feature extractor behaves on a representative scar image, we now present a case study for the generated code G. Given an RGB scar image I ∈ RH×W ×3 , the code G first isolates a scar region via a mask: Ω ⊂ {1, . . . , H} × {1, . . . , W } (by simple thresholding and morphological cleanup on color contrast) and then computes features on Ω. The extracted features can be summarized into four clinically interpretable groups: F = [ϕcolor , ϕtexture , ϕmorph , ϕclin ]⊤ ∈ RK ,
(5)
with each group defined below. Color Features ϕcolor . Code G transforms I to CIELAB, where each pixel has three channels: L∗ (lightness), a∗ (red–green axis), and b∗ (yellow–blue axis). Each channel is a scalar field in RH×W . Over Ω, code G computes: µa − µa,ref µL − µL,ref color ∗ ∗ ϕ = , , Var(a ), Skew(b ) . σa,ref σL,ref (6) Here µa and µL are the mean a∗ and L∗ values within Ω, while (µa,ref , σa,ref ) and (µL,ref , σL,ref ) are computed from a nearby normal-skin region. The normalized terms measure how much the scar’s redness/brightness deviates from healthy skin. Var(a∗ ) is the variance of a∗ over Ω, and Skew(b∗ ) is the third standardized moment (skewness) of b∗ , capturing asymmetry in pigmentation distribution. Texture Features ϕtexture . Let Ig ∈ RH×W be the grayscale image. Code G computes: X 1 ϕtexture = LBPuniform , |∇Ig (p)|, Entropy(Ig ) . 8,1 |Ω| p∈Ω
(7) LBPuniform is a uniform local binary pattern histogram statistic 8,1 (radius 1, 8 neighbors). ∇Ig (p) is the gradient at pixel p, and
Algorithm 1 LLM-Guided Feature Engineering Pipeline Require: Training data Dtrain = {(Ii , yi )}N i=1 , test image Itest , LLM with parameters Θ Ensure: Predicted label ŷtest // Stage 1: Feature Code Generation (One-time) 1: Construct prompt P = Prole ⊕ Pmedical ⊕ Pcode 2: Generate code: G ← LLMΘ (P) 3: Validate and refine G if necessary // Stage 2: Feature Extraction 4: for i = 1 to N do 5: Fi ← G(Ii ) ▷ Apply generated code 6: end for ∗ 7: Construct feature dataset: Dtrain = {(Fi , yi )}N i=1 // Stage 3: Classifier Training ∗ 8: Train lightweight classifier h on Dtrain // Stage 4: Inference 9: Extract test features: Ftest ← G(Itest ) 10: Predict: ŷtest ← h(Ftest ) 11: return ŷtest the mean gradient magnitude quantifies surface irregularity. Entropy(Ig ) is the Shannon entropy of grayscale intensities in Ω, measuring texture complexity. Clinically, these reflect scar roughness and pliability. Morphological Features ϕmorph . From Ω, code G computes area A (pixel count), perimeter P (boundary length), convex hull area Aconvex , and ellipse axes (λmajor , λminor ) fitted to Ω: A P 2 λmajor ϕmorph = , , . (8) Aconvex 4πA λminor A/Aconvex (solidity) decreases when scars extend irregularly; P 2 /(4πA) (circularity) captures boundary irregularity; and axis ratio quantifies elongation. These correspond to scar spread beyond the wound boundary. Clinical Composite Features ϕclin . Code G aggregates clinically relevant cues into composite scores: h i d , ϕclin = σ(β ⊤ [ϕcolor , ϕtexture ]), VSS (9) where σ is the sigmoid function, β ∈ Rd is a weight vector that controls the contribution of each color/texture feature to d is a VSS-like severity the composite severity score, and VSS proxy computed from the extracted cues. Clinical Agreement. The clinician reviewed the feature definitions and confirmed that each group corresponds to routine assessment criteria in VSS/POSAS, and that the normalization against nearby healthy skin makes the measurements interpretable and clinically meaningful.
4.5
Classification Pipeline
Algorithm 1 summarizes the complete training and inference procedure. Training Phase. Applying the generated code G to train∗ ing images yields a structured dataset Dtrain = {(Fi , yi )}N i=1 . K We train a lightweight classifier h : R → Y, such as Support Vector Machine [Suthaharan, 2016] and Random Forest [Breiman, 2001], on this low-dimensional representation.
Hyperparameters are selected via cross-validation on the training set. Inference Phase. For a test image Itest , we first extract features Ftest = G(Itest ) using the same generated code, then obtain the prediction ŷtest = h(Ftest ).
4.6
Discussion: Why This Works
Our approach succeeds by addressing the core challenge of medical image classification: the mismatch between model complexity and data availability. We highlight three key mechanisms: Knowledge Externalization. Rather than learning clinical concepts from limited data, we leverage the LLM’s preexisting medical knowledge to construct features that already encode relevant diagnostic criteria. This dramatically reduces the effective hypothesis space that the downstream classifier must search. Dimensionality Reduction. The feature extractor maps high-dimensional images (H × W × 3 ≈ 105 –106 dimensions) to compact representations (K ≈ 10–20 dimensions). This massive compression, guided by clinical relevance, enables effective learning even with small N . Deterministic Processing. Unlike direct LLM-based classification, our generated code produces identical outputs for identical inputs, ensuring reproducibility. The code can be inspected, validated, and modified by domain experts, enhancing trust and enabling human-in-the-loop refinement.
5
Experiments
We conduct comprehensive experiments to evaluate our proposed method, addressing the following research questions: • RQ1: Does our method outperform baselines in low-data scar classification? • RQ2: How does performance scale with varying training set sizes? • RQ3: What is the contribution of each feature category? • RQ4: Is the framework stable across multiple LLM generation runs and different LLMs? • RQ5: Are the generated features interpretable?
5.1
Experimental Setup
Dataset. We focus on binary classification (keloid (KD) vs. hypertrophic scar (HS)) only. We collected patient images from our partner hospitals with informed consent, and all data were approved for research use. Data collection and expert labeling took 10 months, so the cost was very high and only 40 high quality images are available (20 KD, 20 HS). Evaluation Protocol. Given the limited dataset size, we adopt stratified 5-fold cross-validation and report mean and standard deviation across folds. All results are averaged over three runs with different random seeds. We report Accuracy, Sensitivity (Recall), Specificity, and F1-score, which are standard metrics in medical image analysis.
Table 2: Performance comparison on binary scar classification (keloid vs. hypertrophic). ScaFE (SVM) indicates that the downstream classifier h used in our proposed ScaFE is SVM. Best results in bold, second-best underlined. Results are mean ± std over 5 folds. Method
Acc (%)
Sens.
Spec.
F1
CNN-ResNet18 CNN-EfficientNet ViT-Base MLLM-Direct Handcrafted+RF
0.61 ± 0.13 0.58 ± 0.14 0.51 ± 0.04 0.60 ± 0.24 0.66 ± 0.20
0.62 ± 0.13 0.58 ± 0.14 0.50 ± 0.00 0.60 ± 0.00 0.65 ± 0.20
0.62 ± 0.13 0.58 ± 0.14 0.50 ± 0.00 0.60 ± 0.00 0.65 ± 0.20
0.55 ± 0.17 0.55 ± 0.15 0.34 ± 0.02 0.56 ± 0.00 0.64 ± 0.21
ScaFE (SVM) ScaFE (DT) ScaFE (RF)
0.68 ± 0.17 0.69 ± 0.17 0.73 ± 0.19
0.68 ± 0.16 0.69 ± 0.16 0.72 ± 0.18
0.68 ± 0.16 0.69 ± 0.16 0.72 ± 0.18
0.67 ± 0.17 0.68 ± 0.18 0.72 ± 0.19
Implementation Details. The LLM-guided feature extractor is generated once using GPT-4 with temperature 0 to maximize determinism. The generated code uses OpenCV and scikitimage for image processing. For downstream classification, we evaluate SVM (RBF kernel), Random Forest and Decision Tree classifier with inner cross-validation on the training set. Deep baselines are trained with PyTorch on two NVIDIA RTX 4090 GPUs using Adam (lr=1 × 10−4 ), random horizontal flip and color jitter. ResNet18 and EfficientNet-B0 start from ImageNet weights; ViT-Base is trained from scratch.
5.2
Baselines
We compare against representative methods spanning different paradigms: • CNN-ResNet18: ResNet-18 [He et al., 2016] pretrained on ImageNet, fine-tuned on scar images. • CNN-EfficientNet: EfficientNet-B0 [Tan and Le, 2019] pretrained on ImageNet, fine-tuned on scar images. • ViT-Base: Vision Transformer [Dosovitskiy, 2020] trained from scratch on scar images. • Handcrafted+RF: Classical features (color histograms, GLCM texture, shape descriptors) with Random Forest classifier. • MLLM-Direct: GPT-4V prompted to directly classify scar images [Shiraishi et al., 2024] without code generation. MLLM-Direct uses GPT-4V for zero-shot classification on anonymized images with numeric labels; we report singlerun results when available to avoid label leakage. For deep learning baselines, we apply standard data augmentation (random crop, flip, color jitter) and early stopping to mitigate overfitting.
5.3
Main Results (RQ1)
Table 2 presents classification performance between ScaFE (Scar Feature Engineering) and baselines on the binary scar classification task. Our method consistently outperforms handcrafted features and remains competitive with transfer-learned deep models under the low-data regime. Several observations emerge. First, our LLM-guided features substantially outperform handcrafted baselines, confirming that clinically grounded feature generation captures more relevant cues than manual design. Second, while ResNet18
Figure 2: Classification accuracy under different training data sizes. Our method maintains robust performance even with severely limited data. Table 3: Ablation study on feature construction strategies. Feature Type Random Features Handcrafted (No LLM) LLM w/o Medical Prompt LLM-Generated (Ours)
Acc (%)
F1
0.69 ± 0.16 0.66 ± 0.20 0.68 ± 0.16 0.73 ± 0.19
0.69 ± 0.18 0.64 ± 0.21 0.66 ± 0.17 0.72 ± 0.19
benefits from ImageNet pretraining, its performance comes at the cost of heavier models and reduced interpretability. Third, all three of our variants outperform the baselines. Since RF classifier achieves the best performance among our variants, we adopt RF for subsequent analyses (RQ2–RQ4).
5.4
Data Efficiency Analysis (RQ2)
To investigate robustness under varying data availability, we subsample the training set to fixed per-class counts of 2, 4, 6, 8, 10, 12, 14, 16 images. Figure 2 compares ScaFE (ours) with handcrafted features and deep baselines under the same splits. As shown in Figure 2, ScaFE achieves the best overall performance, and is robust across all shots. When the training set size drops significantly to 2 samples per class, the performance does not drop significantly, highlighting robust few-shot behavior.
5.5
Ablation Studies (RQ3)
Feature Construction Strategy. We isolate the contribution of LLM-guided feature engineering by comparing against alternative feature construction strategies. Results in Table 3 show that LLM-guided features outperform handcrafted baselines, and removing medical context from the prompt reduces performance. Feature Group Contribution. We analyze the contribution of each feature category by selectively removing one group at a time. Table 4 shows that removing morphological features causes the largest drop, while clinical composite has a moderate effect, indicating morphology carries the most discriminative signal for keloid vs. hypertrophic scars. This is aligned with clinical experience.
Table 4: Ablation on feature groups. All features combined achieves the best performance. Configuration All Features w/o Color Features w/o Texture Features w/o Morph Features w/o Clin Features
Acc (%)
F1
0.73 ± 0.19 0.70 ± 0.16 0.71 ± 0.16 0.64 ± 0.17 0.71 ± 0.14
0.72 ± 0.19 0.68 ± 0.17 0.71 ± 0.16 0.61 ± 0.20 0.70 ± 0.14
Table 5: Stability analysis (Accuracy) across different LLMs (GPT-4 and Gemini-2.5) with multiple generation runs.
5.6
Run
GPT-4
Gemini-2.5
Run 1 Run 2 Run 3 Run 4 Run 5
0.73 ± 0.19 0.73 ± 0.21 0.72 ± 0.16 0.73 ± 0.18 0.74 ± 0.21
0.72 ± 0.16 0.74 ± 0.19 0.73 ± 0.21 0.72 ± 0.17 0.73 ± 0.19
Stability Analysis (RQ4)
Although LLMs exhibit inherent stochasticity, our framework mitigates this by: (i) using low temperature during generation, and (ii) producing deterministic executable code rather than free-form text outputs. We assess stability by running the feature generation process five times with identical prompts across GPT-4 and Gemini-2.5. Table 5 demonstrates that our approach yields stable and reproducible results. While minor variations in generated code may occur, the extracted features maintain consistent discriminative power.
5.7
Qualitative Analysis (RQ5)
To illustrate interpretability, Figure 3 visualizes representative scar images alongside their extracted features. Since our data comes from hospital collections, patient privacy is a factor that prevents visualization. To clarify this, we searched for publicly available papers on PubMed and collected images specifically for interpretability verification. The feature values align with clinical expectations: keloid samples exhibit higher erythema scores (increased redness) and lower solidity values (irregular boundaries extending beyond the wound), while hypertrophic scars show more confined morphology. This transparency enables clinicians to validate model predictions against their domain expertise, facilitating trust and adoption in clinical workflows.
6
Conclusion
We presented a novel framework that repositions large language models as knowledge-driven feature engineers for medical image classification. Rather than training data-hungry neural networks end-to-end or using LLMs as black-box classifiers, our approach leverages LLMs to generate executable feature extraction code grounded in established clinical criteria. This design explicitly decouples domain knowledge acquisition from statistical learning, enabling effective classification in data-scarce medical domains. Extensive experiments on pathological scar classification demonstrate that our method consistently outperforms both
Figure 3: Visualization of clinically meaningful features. (a) Hypertrophic scar samples with confined morphology and (b) Keloid samples with high erythema and irregular boundaries. Feature values align with established clinical criteria.
end-to-end deep learning models and direct multimodal LLM classification under limited data conditions. The extracted features are inherently interpretable, aligning with clinical assessment scales such as the Vancouver Scar Scale, which facilitates expert validation and clinical trust. Furthermore, by processing images through locally executed code, our approach preserves patient privacy without transmitting sensitive data to external servers. Limitations and Future Work. Our current framework relies on a single LLM generation step; future work could explore iterative refinement based on classification feedback or ensemble multiple generated feature extractors. The approach also assumes access to a capable LLM, which may introduce dependency on proprietary models. Investigating open-source alternatives or fine-tuned domain-specific LLMs would enhance accessibility. Additionally, while we focused on scar classification, the framework is general and could be extended to other medical imaging tasks such as skin lesion analysis, histopathology, and radiology, which we leave for future investigation. Broader Impact. This work contributes to the growing effort of developing trustworthy AI for healthcare. By providing interpretable, privacy-preserving, and data-efficient alternatives to black-box deep learning, we hope to lower barriers for AI adoption in clinical practice, particularly in specialized domains where expert data is scarce. We encourage the research community to further explore the paradigm of using LLMs as knowledge externalizers rather than direct decision-makers in high-stakes applications. Usage of LLMs. Large Language Models were used as assistive tools in the preparation of this manuscript. We employed LLMs for grammar checking, LaTeX formatting, and improving the clarity of technical descriptions. The core scientific contributions and conclusions presented in this paper originate from the authors’ work. Ethics Statement All patient images were collected with informed consent under hospital-approved protocols and used solely for research purposes. Data handling followed clinical privacy guidelines.
References [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [Baryza and Baryza, 1995] Margaret J. Baryza and George A. Baryza. The Vancouver Scar Scale: an administration tool and its interrater reliability. Journal of Burn Care & Rehabilitation, 16(5):535–538, 1995. [Bayat et al., 2003] Ardeshir Bayat, D. Angus McGrouther, and Mark W. J. Ferguson. Skin scarring. BMJ, 326(7380):88–92, 2003. [Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013. [Berman et al., 2017] Brian Berman, Andrea Maderal, and Briana Raphael. Keloids and hypertrophic scars: pathophysiology, classification, and treatment. Dermatologic Surgery, 43:S3–S18, 2017. [Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. [Busche et al., 2018] Marc Nicolai Busche, AliceCaroline Johanna Thraen, Andreas Gohritz, Hans-Oliver Rennekampff, and Peter Maria Vogt. Burn scar evaluation using the cutometer® mpa 580 in comparison to “patient and observer scar assessment scale” and “vancouver scar scale”. Journal of Burn Care & Research, 39(4):516–526, 2018. [Chen et al., 2020] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056, 2020. [Domingos, 2012] Pedro Domingos. A few useful things to know about machine learning. Communications of the ACM, 55(10):78–87, 2012. [Dosovitskiy, 2020] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [Draaijers et al., 2004] L. J. Draaijers, F. R. H. Tempelman, Y. A. M. Botman, W. E. Tuinebreijer, E. Middelkoop, R. W. Kreis, and P. P. M. van Zuijlen. The Patient and Observer Scar Assessment Scale: a reliable and feasible tool for scar evaluation. Plastic and Reconstructive Surgery, 113(7):1960–1965, 2004. [Garcez and Lamb, 2023] Artur d’Avila Garcez and Luis C. Lamb. Neurosymbolic AI: The 3rd wave. Artificial Intelligence Review, 56:12387–12406, 2023. [Gostin et al., 2009] Lawrence O Gostin, Laura A Levit, and Sharyl J Nass. Beyond the hipaa privacy rule: enhancing privacy, improving health through research. 2009. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. [Koh et al., 2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338–5348, 2020. [Li et al., 2023] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023. [Litjens et al., 2017] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60–88, 2017. [Liu et al., 2020] Yun Liu, Ayush Jain, Clara Eng, David H. Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriber, et al. A deep learning system for differential diagnosis of skin diseases. Nature Medicine, 26(6):900–908, 2020. [Nori et al., 2023] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. [Shiraishi et al., 2024] Makoto Shiraishi, Shimpei Miyamoto, Hakuba Takeishi, Daichi Kurita, Kiichi Furuse, Jun Ohba, Yuta Moriwaki, Kou Fujisawa, and Mutsumi Okazaki. The potential of chat-based artificial intelligence models in differentiating between keloid and hypertrophic scars: a pilot study. Aesthetic Plastic Surgery, 48(24):5367–5372, 2024. [Sidgwick and Bayat, 2012] G. P. Sidgwick and A. Bayat. Extracellular matrix molecules implicated in hypertrophic and keloid scarring. Journal of the European Academy of Dermatology and Venereology, 26(2):141–152, 2012. [Singhal et al., 2023] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. [Suthaharan, 2016] Shan Suthaharan. Support vector machine. In Machine learning models and algorithms for big data classification: thinking with examples for effective learning, pages 207–235. Springer, 2016. [Tan and Le, 2019] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114, 2019. [Xie et al., 2019] Yutong Xie, Yong Xia, Jianpeng Zhang, Yang Song, Dagan Feng, Michael Fulham, and Weidong Cai. Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT.
IEEE Transactions on Medical Imaging, 38(4):991–1004, 2019. [Ye et al., 2024] Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dan dan Guo, and Yi Chang. PTaRL: Prototype-based tabular representation learning via space calibration. In The Twelfth International Conference on Learning Representations, 2024. [Zhou et al., 2024] Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, et al. Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4. Nature Communications, 15(1):5649, 2024. [Zhu et al., 2016] Xinliang Zhu, Jiawen Yao, and Junzhou Huang. Deep convolutional neural network for survival analysis with pathological images. In 2016 IEEE international conference on bioinformatics and biomedicine (BIBM), pages 544–547. IEEE, 2016.