KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement

KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement Gusseppe Bravo-Rocca∗§ , Jordi Guitart∗† , Ajay Dholakia‡ , David Ellison‡ , Puneet Jain§ ∗ Barcelona Supercomputing Center, Barcelona, Spain † Universitat Politècnica de Catalunya, Barcelona, Spain ‡ Lenovo Infrastructure Solutions Group, Morrisville, NC, United States § Cotiviti, South Jordan, UT, United States

arXiv:2608.02351v1 [cs.AI] 3 Aug 2026

Email: {gusseppe.bravo, jordi.guitart}@bsc.es, puneet.jain@cotiviti.com, {adholakia, dellison}@lenovo.com

Abstract—Data drift poses significant challenges for machine learning systems in production, requiring continuous model updates to maintain performance. We present KC-Agent, a dual-process cognitive architecture for automated ML model improvement that combines fast pattern recognition (System 1) with deliberate incremental updates (System 2). Our approach implements structured memory systems enabling System 1 to leverage successful solutions previously discovered by System 2, achieving efficient pattern-based responses without costly recomputation. KC-Agent incorporates atomic change principles and rollback capabilities to ensure reliable, verifiable updates in production environments. We evaluate our method on five datasets including real-world NASA turbofan data with authentic temporal degradation and synthetic datasets with controlled drift scenarios. KC-Agent achieves state-of-the-art performance (76.8% accuracy) while maintaining optimal efficiency (13.2s execution time), outperforming established cognitive architectures: CodeAct (+2.4%), Tree of Thoughts (+3.6%), ReAct (+8.0%), and Reflexion (+8.9%). Consensus evaluation by a panel of stateof-the-art LLMs confirms superior strategic efficacy (8.33/10 Smartness score), significantly outperforming baseline agents. The knowledge consolidation mechanism delivers 91% speedup over the slow variant while maintaining higher accuracy. Our approach demonstrates both theoretical foundations and practical viability for cognitive-inspired automated ML improvement systems capable of handling complex real-world data drift scenarios. Index Terms—LLM agents, data drift, model maintenance, dual-process architecture

I. I NTRODUCTION Machine Learning (ML) models in production environments face continuous challenges from evolving data distributions, changing operational requirements, and performance degradation over time [1]. Traditional approaches to model maintenance rely heavily on manual intervention by ML engineers, creating bottlenecks that limit the responsiveness and scalability of deployed AI systems [2]. The complexity of modern ML pipelines, combined with the subtle nature of gradual data drift, makes manual monitoring and updating increasingly impractical for large-scale deployments [3]. The emergence of Large Language Models (LLMs) has created new opportunities for automating complex software engineering tasks, including code generation, debugging, and

system optimization [4]. LLM-based autonomous agents show particular promise for ML model improvement due to their ability to reason about code, understand ML concepts, and generate adaptive solutions. However, developing reliable agents for automated model improvement presents fundamental challenges that current approaches have not adequately addressed [5], [6]. The core challenge lies in balancing two competing requirements: rapid response to critical performance drops versus careful, systematic improvement that avoids unintended consequences. Current LLM-based approaches typically employ single-strategy reasoning [7], forcing a trade-off between speed and thoroughness. Fast approaches risk making hasty changes that degrade performance, while deliberative approaches may respond too slowly to critical issues [8]. Most existing methods lack memory mechanisms to learn from previous improvement attempts, leading to inefficient repeated exploration of unsuccessful strategies [9]. To address these limitations, we introduce KC-Agent, short for Kahneman-Clear Agent, a dual-process architecture that combines fast pattern recognition with deliberate incremental improvement. Our approach draws inspiration from Kahneman’s dual-process theory [10], which describes human cognition through two complementary systems: fast, intuitive responses (System 1) and slow, deliberate reasoning (System 2). We integrate this cognitive framework with atomic improvement principles [11] to create an agent architecture that can both respond quickly to critical issues and systematically enhance model performance over time. Our key insight is that effective automated model improvement requires cognitive flexibility: the ability to switch between rapid pattern-based responses and methodical systematic improvement depending on the situation. By incorporating structured memory systems and learning mechanisms, KC-Agent accumulates expertise over time, becoming more efficient as it encounters similar improvement scenarios. The atomic nature of changes ensures that improvements are verifiable and reversible, providing reliability guarantees essential for production environments. The contribution of this paper is threefold: 1) we intro-

KC-Agent Memories Semantic Memory

A. LLM-based Autonomous Agents Slow Graph (S2)

Episodic Memory

Distill memories

4

Working Memory

Incremental Improvement

Input: old training code Output: new training code

5

New code Guide reasoning Runtime error

Fast Graph (S1)

1 Naive update

Tiny Change

9 Slow graph's knowledge

Improvement History

8

6

Failure/ Success

Apply Change

No errors

Proxy

2

3 No

Evaluate Change

Yes

Target met

End

End

7

9

Fig. 1. KC-Agent dual-process architecture showing the numbered workflow from fast naive updates (1-2) through proxy evaluation to slow deliberate improvement (4-8). System 1 handles immediate responses using pattern recognition (reusing previously successful solutions), while System 2 performs incremental improvements through memory distillation and atomic change cycles. Knowledge transfer (9) enables System 1 to leverage System 2’s solutions for efficient future pattern matching (identifying similar scenarios).

duce a dual-process architecture that combines fast pattern recognition with deliberate incremental improvement for automated ML model maintenance, 2) we formalize mathematical foundations and establish theoretical guarantees for monotonic improvement under data drift, and 3) we demonstrate superior performance across five datasets while achieving 91% computational speedup through knowledge consolidation mechanisms. The rest of this paper is organized as follows. Section II reviews related work on LLM-based agents, cognitive architectures, and automated ML maintenance. Section III presents the KC-Agent architecture and its theoretical foundations. Section IV describes our experimental setup. Section V reports our results, followed by discussion in Section VI. Section VII addresses limitations and future work, and Section VIII concludes the paper. II. R ELATED W ORK Our work builds upon three intersecting research areas: LLM-based autonomous agents, cognitive architectures for AI systems, and automated ML model maintenance.

Recent work in LLM-based autonomous agents has produced various architectures for complex reasoning and action [5]. ReAct [12] introduces reasoning-action interleaving for dynamic problem-solving. Tree of Thoughts [13] explores multiple reasoning paths simultaneously, while Reflexion [7] incorporates self-reflection mechanisms to learn from failures. More sophisticated approaches include Self-Discover [14], which develops task-specific reasoning structures, and Planand-Solve [15], which implements explicit planning phases. CodeAct [16] represents a recent paradigm shift by using executable Python code as a unified action space rather than JSON or text-based formats. Integrated with a Python interpreter, CodeAct enables dynamic action revision through multi-turn interactions and self-debugging, achieving up to 20% higher success rates on complex multi-tool tasks. This code-centric approach is particularly relevant to ML model improvement where generating and executing training code is fundamental. While these approaches demonstrate structured reasoning capabilities, they typically focus on general problem-solving rather than specialized domains like ML model maintenance. Most existing architectures lack systematic approaches for learning from improvement history or transferring successful strategies across scenarios [9]. Additionally, few frameworks provide theoretical guarantees for reliability or monotonic improvement, critical for automated systems operating on production ML models. B. Cognitive Architectures for AI Systems Cognitive science principles have increasingly influenced AI system design, particularly hybrid architectures balancing different reasoning modes. SwiftSage [17] operationalizes dual-process thinking through separate modules for fast pattern recognition and deliberate analysis. The Talker-Reasoner architecture [18] similarly separates intuitive responses from systematic reasoning. These approaches draw inspiration from Kahneman’s dualprocess theory [10], describing human cognition through fast, intuitive responses (System 1) and slow, deliberate reasoning (System 2). While effective for general reasoning tasks, application to specialized domains like ML model improvement remains underexplored. Our work extends this foundation by integrating atomic improvement principles [11] for incremental model enhancement. C. Automated ML Model Maintenance Traditional ML model maintenance relies primarily on continual learning and AutoML techniques. Continual learning approaches address catastrophic forgetting through parameter preservation [19] and memory replay [20], [21], but focus on knowledge retention rather than proactive improvement under distribution shift [22]. AutoML frameworks automate model refinement through hyperparameter optimization and neural architecture search [23], yet remain constrained by predefined search spaces.

Recent frameworks for automated AI maintenance [24] address robustness challenges but primarily handle predefined risk categories without adaptive capabilities. Monitoring systems [1], [2] detect performance degradation and distribution shifts [3] but typically require manual intervention for remediation. Self-adaptive systems [25] provide autonomous response mechanisms but lack sophisticated reasoning capabilities for complex model improvement tasks. Recent empirical work on scaling agent systems [26] provides evidence that multi-agent architectures do not universally outperform single agents. Their analysis reveals that coordination overhead can degrade performance once single-agent capability exceeds certain thresholds, with some task configurations showing performance degradation from multi-agent coordination. These findings support single-agent designs that achieve coordination benefits through internal mechanisms rather than external agent communication. Current approaches face key limitations: single-strategy reasoning forcing speed-thoroughness trade-offs, limited learning mechanisms preventing effective strategy transfer, lack of reliability guarantees restricting production applicability, and domain-general focus missing ML-specific challenges, like distribution shifts and incremental changes. KC-Agent addresses these gaps through a dual-process architecture combining fast pattern recognition with deliberate improvement, structured memory enabling strategy learning, and formal reliability mechanisms ensuring verifiable updates. III. M ETHOD We present KC-Agent, a dual-process architecture for automated ML model improvement that combines fast pattern recognition with deliberate incremental enhancement. Our approach integrates cognitive science principles with practical constraints to ensure both responsive intervention and systematic improvement. A. System Overview KC-Agent operates through two complementary processing systems that address different aspects of model improvement. The fast graph (System 1) provides immediate pattern-based responses using accumulated knowledge from previous improvement attempts. When quick fixes prove insufficient, the slow graph (System 2) engages in systematic incremental improvement through structured reasoning and atomic changes. Algorithm 1 presents the complete dual-process improvement procedure. The architecture incorporates three memory components enabling learning and knowledge transfer between systems. Semantic memory (Ms ) stores general knowledge about model architectures and improvement strategies. Episodic memory (Me ) records specific improvement scenarios and their outcomes. Working memory (Mw ) maintains the current context and immediate reasoning state. Figure 2 illustrates these memory components and their interactions during the improvement process.

Memory Architecture Semantic Memory (SM)

Episodic Memory (EM)

[Training code] [Metadata] Working Memory (WM)

: Improvement history

[EM] [SM] [LLM steps]

Fast Graph

Slow Graph

Decision Procedure Fig. 2. Memory architecture with three components: Semantic memory (SM) storing training code and metadata, Episodic memory (EM) recording specific scenarios (Dnew , H), and Working memory (WM) maintaining current LLM processing steps.

B. Fast Graph: Pattern-based Improvement The fast graph implements immediate response capabilities for straightforward improvement scenarios. Given a model f and datasets D = {Dold , Dnew }, System 1 generates an updated model f ′ through pattern matching (identifying similar scenarios in Me and reusing previously successful solutions): S1 : (Ms , Me , f, D) 7→ f ′ The system evaluates improvement quality using a performance delta: δ(update) = score(f ′ , Dnew ) − score(f, Dnew ) A proxy decision mechanism determines whether the improvement is sufficient: ( accept if δ(update) > µ ′ π(f, f , D) = slow graph otherwise where µ represents the minimum acceptable improvement threshold. C. Slow Graph: Incremental Systematic Improvement When fast pattern matching proves insufficient, the slow graph engages in systematic improvement through atomic changes (small, targeted modifications such as hyperparameter

Algorithm 1: KC-Agent dual-process improvement Input: Model f , datasets D, memories Ms , Me Parameter: Threshold µ, max iterations T Output: Improved model f ∗

1

Old training code

Fast Graph Learning

1: f ′ ← S1 (Ms , Me , f, D) {Fast improvement} 2: δ ← score(f ′ , Dnew ) − score(f, Dnew ) 3: if δ > µ then 4: return f ′ {Accept fast solution} 5: else

f∗ ← f, t ← 0 while t < T and not converged do ∆t ← generate atomic change using S2 so that ∥∆t ∥ ≤ η 9: ftemp ← apply ∆t to f ∗ 10: rt ← evaluate change 11: if rt > 0 then 12: f ∗ ← ftemp 13: Update Me with successful change 14: end if 15: t←t+1 16: end while 17: return f ∗ 18: end if 6: 7: 8:

adjustments, model selection, or ensemble techniques that can be independently evaluated and reverted if unsuccessful). System 2 implements a structured approach: S2 : (Ms , Me , H, f, D) 7→ f ∗ where H represents the improvement history. The slow graph operates through iterative small modifications, where each change ∆t satisfies ∥∆t ∥ ≤ η for small η > 0. In practice, this constraint limits each iteration to a single targeted modification, such as adjusting one hyperparameter, substituting one model class, or adding one preprocessing step, rather than attempting multiple simultaneous changes. This atomic change principle, inspired by Clear’s work on incremental improvement [11], ensures that improvements compound reliably while remaining individually evaluable and reversible. The improvement process follows a structured cycle: memory distillation analyzes current model performance, strategy selection chooses appropriate improvement approaches, code generation creates targeted modifications, and evaluation assesses each change. Figure 3 demonstrates this process through a concrete example. D. Memory Systems and Knowledge Transfer The memory architecture enables bidirectional knowledge transfer between systems. When System 2 discovers successful improvement strategies, these solutions are incorporated into semantic memory for future pattern matching by System 1. This transfer mechanism allows the fast graph to handle increasingly complex scenarios as the agent accumulates experience. For example, when System 2 successfully improves

Old data

New data

Re-training code

2

3

Slow Graph

"I found that random forest works fine for this data"

Dataset description

4

New training code

Dataset title Features Numerical features Categorical features Label Column types Dataset description Label description

Hyperparameters

Model selection

Ensemble

KC-Agent

Metadata

Fig. 3. Example improvement flow showing KC-Agent processing a drift scenario. System 1 applies fast retraining based on stored patterns. When insufficient, System 2 performs systematic improvements through atomic changes including hyperparameter optimization, model selection, and ensemble methods.

a financial model by switching from RandomForest to GradientBoosting model, this strategy is stored in semantic memory, enabling System 1 to quickly apply the same approach when encountering similar financial drift scenarios in the future. Episodic memory stores specific improvement scenarios, including dataset characteristics, model states, and applied solutions. This contextual information helps both systems recognize similar situations and adapt previous solutions to new contexts. The knowledge transfer process operates continuously throughout improvement iterations, significantly improving efficiency over time. We refer to this learning process as knowledge consolidation. Specifically, when System 2 successfully improves a model, three elements are stored: the drift signature (statistical characteristics of the distribution shift), the improvement strategy (the sequence of atomic changes applied), and the performance outcome. During subsequent encounters, System 1 computes scenario similarity by calculating the cosine similarity between the incoming dataset’s metafeatures (including the statistical drift signature) and the stored episodic vectors. To avoid negative transfer, retrieved strategies are only applied when the cosine similarity score exceeds a strict calibration threshold (µ = 0.05 performance proxy); otherwise, the system conservatively defaults to System 2 deliberation. E. Reliability and Theoretical Guarantees KC-Agent incorporates several reliability mechanisms. The incremental nature of System 2 changes limits the potential impact of any single modification. Each atomic change undergoes immediate evaluation, and unsuccessful modifications are reverted before affecting model performance. The proxy

TABLE I DATASET CHARACTERISTICS AND DRIFT PATTERNS Dataset NASA FD001 NASA FD002 Financial Healthcare Eligibility

Features

Samples

Drift type

14 7 10 10 5

10,000 16,000 1,000 1,000 1,000

Temporal Temporal Market noise Population noise Policy noise

decision mechanism ensures that only high-confidence improvements bypass detailed analysis. We establish formal guarantees for KC-Agent’s improvement behavior: Theorem 1 (Monotonic improvement property). Under the atomic change constraint ∥∆t ∥ ≤ η and immediate reversion of unsuccessful modifications, KC-Agent maintains: score(fT , D) ≥ score(f0 , D) where unsuccessful changes are those with negative improvement rt ≤ 0. Proof. By construction, KC-Agent only accepts changes ∆t where rt = score(ft+1 , D) − score(ft , D) > 0. Changes with rt ≤ 0 are immediately reverted, ensuring ft+1 = ft . Therefore, score(ft+1 , D) ≥ score(ft , D) for all t. By induction, score(fT , D) ≥ score(f0 , D). The atomic change constraint ∥∆t ∥ ≤ η ensures that each modification is small enough to be evaluated independently, preventing cascading failures that could compromise the rollback mechanism. IV. E XPERIMENTAL S ETUP We designed our experimental evaluation to assess KCAgent’s effectiveness across diverse scenarios with varying complexity levels and drift patterns. Our evaluation encompasses both real-world industrial datasets and controlled synthetic scenarios to provide comprehensive validation of the dual-process architecture. A. Datasets and Drift Scenarios Our evaluation uses five datasets representing different aspects of data drift challenges in production ML systems. Table I summarizes the key characteristics of each dataset. Real-world datasets. The NASA turbofan datasets (FD001 and FD002) derive from the Commercial Modular AeroPropulsion System Simulation (C-MAPSS)1 and represent authentic industrial complexity with temporal degradation patterns. FD001 contains 14 features from 100 engines operating under single conditions. FD002 includes 7 features from 260 engines across six operational conditions. These datasets exhibit authentic temporal drift through engine lifecycle progression, where early operational phases (healthy engines) 1 https://www.kaggle.com/datasets/behrad3d/nasa-cmaps

constitute the old distribution and late phases (degraded engines) form the new distribution. Synthetic datasets. Three synthetic datasets provide controlled evaluation scenarios with increasing complexity levels. The financial dataset models loan default prediction, the healthcare dataset captures chronic condition prediction, and the eligibility dataset simulates administrative decisionmaking. Each synthetic dataset was initially generated using GPT-42 and refined to ensure realistic feature distributions and coherent drift patterns while eliminating nonsensical relationships. Distribution shift severity was rigorously validated using Kullback-Leibler (KL) divergence analysis, confirming significant and realistic drift across all synthetic environments. Specifically, the Financial dataset exhibited drift (KL > 0.1) in all 10 features (mean KL = 2.001, with the maximum divergence of 4.497 observed in the ’Loan Amount’ feature). Similarly, the Healthcare dataset showed significant drift in 8 of 10 features (max KL = 2.662 for ’Income’), and the Eligibility dataset demonstrated drift across all 5 features (max KL = 2.837 for ’Employment Status’). All datasets include paired old and new distributions that simulate drift scenarios without explicit drift indicators. This design requires agents to independently discover distribution changes and develop appropriate adaptation strategies. B. Experimental Protocol The evaluation protocol simulates realistic model maintenance scenarios where an initial model performs well on historical data but experiences degradation when new data patterns emerge. For each dataset, a RandomForest classifier serves as the baseline model trained on the old distribution. KC-Agent and baseline approaches then attempt to improve this model to accommodate the new distribution while preserving performance on original data. We measure performance across four complementary metrics: accuracy on old data (preservation of historical performance), accuracy on new data (adaptation effectiveness on shifted distributions), execution time (computational efficiency for production deployment), and token consumption (resource utilization and computational cost). The proxy decision threshold µ was set to 0.05 across all experiments, requiring at least a 5% accuracy improvement for System 1 solutions to be accepted without further deliberation. C. Baseline Comparisons Our evaluation compares KC-Agent against eight established approaches representing different paradigms in LLMbased reasoning: ReAct [12], Reflexion [7], Tree of Thoughts [13], Self-Discovery [14], Plan-and-Execute [15], CodeAct [16], and a Standard baseline (simple LLM call). CodeAct is particularly relevant as it represents state-of-the-art in codeas-action paradigms, using executable Python with interpreter integration for dynamic revision and self-debugging capabilities. 2 OpenAI GPT-4, accessed via API

We also evaluate two simplified variants of our agent to assess each component contribution: KC-Fast (only the fast graph for pattern-based improvements) and KC-Slow (only the slow graph for deliberate reasoning). These ablations help isolate the benefits of the dual-process architecture compared to single-strategy approaches. All baselines receive identical inputs including model code, dataset descriptions, and performance metrics. Temperature settings remain consistent at 1.0 across all experiments, with fixed random seeds ensuring reproducible results. Each experiment was conducted multiple times to ensure statistical reliability. D. Implementation Details KC-Agent leverages Llama-3.1-8b3 as the core language model, providing 8 billion parameters with an 8,192 token context window. This model size balances capability with resource requirements, making the approach suitable for deployment in resource-constrained environments. Smaller models (Llama3.2-1b and Llama-3.2-3b) proved insufficient for completing the full improvement loop. The memory system implements three distinct components with specialized storage and retrieval mechanisms. Semantic memory maintains general knowledge using embedding-based indexing for efficient pattern matching. Episodic memory records specific improvement scenarios with contextual information. Working memory maintains current reasoning state during improvement sessions. The dual-process architecture uses LangGraph4 for workflow orchestration, enabling flexible coordination between fast and slow reasoning systems. Experimental infrastructure utilizes Intel® optimized computing environments with dual Intel® Xeon Platinum processors. The implementation leverages Intel® AI Analytics Toolkit5 and Intel® Extension for PyTorch for hardware optimization. Reproducibility support includes comprehensive logging of experimental configurations, random seed management, and version control of all code and data components. Each experiment receives unique identifiers enabling precise reproduction of results. V. R ESULTS Our experimental evaluation demonstrates KC-Agent’s effectiveness across diverse scenarios, from controlled synthetic datasets to challenging real-world industrial data. We present comprehensive analysis including quantitative performance metrics and qualitative expert assessment. A. Overall Performance and Efficiency KC-Agent consistently outperforms all baseline approaches across the five datasets. Table II summarizes the aggregated performance, showing KC-Agent’s superior accuracy while maintaining optimal efficiency. 3 https://openrouter.ai/ 4 https://github.com/langchain-ai/langgraph 5 https://hub.docker.com/r/intel/oneapi-aikit

TABLE II AGENT PERFORMANCE COMPARISON . VALUES REPORTED AS M EAN ± S TANDARD D EVIATION ACROSS THE FIVE EVALUATED DATASETS , DEMONSTRATING CROSS - DOMAIN STABILITY. B OLD VALUES INDICATE BEST MEAN PERFORMANCE . Agent

Avg acc. (%) ↑

Time (s) ↓

Tokens ↓

KC-agent KC-slow KC-fast CodeAct ToT ReAct Reflexion Self-Discovery Plan-Execute Standard

0.768 ± 0.116 0.762 ± 0.108 0.754 ± 0.102 0.744 ± 0.108 0.741 ± 0.103 0.711 ± 0.086 0.705 ± 0.122 0.679 ± 0.139 0.650 ± 0.175 0.630 ± 0.194

13.2 ± 7.3 141.2 ± 41.6 13.9 ± 2.1 113.0 ± 196.7 57.7 ± 62.3 101.7 ± 121.2 104.7 ± 51.3 59.1 ± 41.2 91.2 ± 23.8 16.8 ± 5.5

847 ± 54 37781 ± 18100 643 ± 48 6722 ± 2760 2827 ± 325 12420 ± 12796 20436 ± 18579 4638 ± 676 13256 ± 2730 1569 ± 278

As shown in Table II, KC-Agent achieves the highest average accuracy (0.768) while maintaining robust performance stability across all diverse datasets (± 0.116). The dual-process architecture delivers exceptional computational efficiency, utilizing 18.7× fewer tokens than Reflexion. Furthermore, it demonstrates highly consistent execution times (13.2s ± 7.3) and predictable token consumption. This structural stability stands in stark contrast to state-of-the-art baselines like CodeAct, which, despite achieving competitive accuracy, exhibits extreme variance across domains in both execution time (± 196.7s) and computational cost (± 2760 tokens). B. Qualitative Evaluation To complement our quantitative performance metrics, we conducted a comprehensive qualitative evaluation using a multi-model LLM-as-a-judge framework. To ensure robustness and mitigate individual model bias, we employed a panel of three state-of-the-art Large Language Models as independent evaluators: Google Gemini 3 Pro, OpenAI GPT-5.2, and Anthropic Claude Opus 4.5. This consensus-based approach provides a standardized assessment of the code improvements generated by each agent. For every experimental run across all five datasets, each evaluator was presented with the initial baseline code, the agent’s final improved code, and the corresponding change in performance metrics. The judges independently scored the agents on a scale of 1-10 across three distinct rubrics: • Stability: Measures the minimalism of changes and preservation of the original code structure. A high score (10) indicates surgical, low-risk changes, while a low score implies a complete, high-risk rewrite. • Readability: Assesses code cleanliness, comprehensibility, and organization. High scores reflect productionready, Pythonic code with proper error handling and no redundancy. • Smartness: Evaluates the efficacy and appropriateness of the improvement strategy. A high score indicates a clever, well-tuned algorithmic approach that successfully addresses the distribution shift, whereas a low score indicates suboptimal or irrelevant strategies.

TABLE III LLM EVALUATION COMPARISON AVERAGED ACROSS ALL DATASETS AND EVALUATORS (G EMINI 3 P RO , GPT-5.2, O PUS 4.5). S CORES RANGE FROM 1-10. S TABILITY MEASURES CODE PRESERVATION , R EADABILITY MEASURES CODE CLEANLINESS , AND S MARTNESS MEASURES STRATEGY EFFICACY. B OLD VALUES INDICATE BEST PERFORMANCE . VALUES SHOWN AS MEAN ( STD ). Agent

Stability (↑)

Readability (↑)

Smartness (↑)

6.53(0.9) 6.53(1.0) 7.67(1.2) 7.67(2.1) 6.20(1.0) 6.67(1.7) 3.00(1.5) 5.20(1.3) 3.07(1.0) 2.60(1.0)

7.07(0.9) 6.60(0.7) 6.00(1.0) 7.67(0.9) 6.60(1.4) 6.60(1.4) 5.67(2.1) 4.73(2.0) 3.87(2.1) 2.60(1.3)

8.33(1.0) 7.80(1.1) 6.47(1.7) 5.93(2.0) 5.87(2.1) 5.33(2.7) 3.33(2.3) 2.13(0.8) 1.67(0.7) 1.40(0.6)

KC-agent KC-slow KC-fast CodeAct ToT ReAct Reflexion Standard Self-Discovery Plan-Execute

Table III summarizes the assessment scores averaged across all three evaluators and all datasets. The low standard deviations across evaluators (ranging from 0.6 to 2.7) indicate strong inter-rater agreement, validating the LLM-as-judge methodology as a reliable alternative to human expert evaluation. This high consensus among three state-of-the-art models suggests that the evaluation criteria are sufficiently well-defined and that the assessments capture genuine quality differences rather than model-specific biases. The consensus results reveal a critical distinction between code preservation and problem-solving efficacy. CodeAct and KC-fast achieved the highest scores in Stability (7.67), reflecting their tendency to perform safe, minimal changes: primarily data concatenation or simple retraining. However, while stable, these conservative strategies resulted in moderate Smartness scores (5.93 and 6.47 respectively), limiting their ability to handle complex distribution shifts effectively. In contrast, KC-Agent achieved the highest Smartness score (8.33), significantly outperforming all baselines. While its Stability score (6.53) is lower than the pure concatenation approaches, it remains robust, reflecting the agent’s ability to balance necessary structural refactoring, such as hyperparameter optimization and pipeline adjustments, with code preservation. This qualitative finding strongly corroborates our quantitative results (Table III), confirming that KC-Agent’s superior accuracy is driven by more effective strategic decision-making rather than simple code heuristics. Conversely, baselines such as Plan-Execute and Self-Discovery received the lowest scores across all dimensions, frequently failing to produce executable or logically sound improvements. C. Ablation Study To validate the dual-process architecture’s contribution, we compared KC-Agent with its individual components. Table IV demonstrates that neither System 1 nor System 2 alone matches the integrated architecture’s performance. The integrated architecture successfully combines the strengths of both systems: achieving the highest accuracy while maintaining computational efficiency close to the fast

TABLE IV A BLATION STUDY OF KC-AGENT VARIANTS SHOWING COMPLEMENTARY BENEFITS OF DUAL - PROCESS ARCHITECTURE . Variant

Avg acc. (↑)

Time (↓)

Tokens (↓)

KC-agent (Full) KC-slow only KC-fast only

0.768 0.762 0.754

13.2 141.2 13.8

847 37781 643

vs. KC-slow vs. KC-fast

+0.8% +1.8%

-90.7% -4.3%

-97.8% +31.7%

approach. KC-Agent provides 1.8% accuracy improvement over KC-Fast while using 97% fewer tokens than KC-Slow. The superior performance of the full KC-Agent demonstrates the effectiveness of the learning mechanism where System 1 leverages successful solutions previously discovered by System 2. D. Dataset-specific Analysis and Knowledge Transfer NASA turbofan datasets. The real-world NASA datasets present the most challenging scenarios due to authentic temporal degradation. On NASA-FD001, KC-Agent achieves 61.3% accuracy compared to the 49.3% baseline (+24% relative improvement). NASA-FD002 proves even more challenging with extreme distribution shift, yet KC-Agent maintains consistent improvements. Synthetic datasets. On controlled datasets, KC-Agent achieves 88.6% accuracy on financial data (vs. 81.5% baseline), 86.3% on healthcare data (vs. 79.7% baseline), and 74.4% on eligibility data (vs. 67.7% baseline). LLM evaluation confirms methodological rigor across different domain characteristics. Knowledge consolidation effectiveness. A critical insight from our evaluation is how KC-Agent’s knowledge consolidation mechanism enables progressive learning. Analysis of improvement patterns reveals that System 1 increasingly handles complex scenarios as it accumulates successful strategies from System 2. For instance, on financial datasets, System 1 successfully resolved 73% of improvement scenarios by the third iteration, compared to 45% in the first iteration. This learning effect demonstrates the practical value of the dualprocess architecture beyond simple computational efficiency. E. Reliability Analysis KC-Agent’s reliability mechanisms prove crucial for production deployment. The atomic change principle ensures that each modification undergoes evaluation before acceptance, with unsuccessful changes immediately reverted. Across all datasets, KC-Agent maintains a 100% success rate in terms of execution completion, contrasting sharply with baseline approaches where 34% of attempts resulted in execution failures. The proxy decision mechanism (δ(update) > µ) effectively balances speed and reliability. Our analysis shows that 68% of scenarios are resolved by System 1, with only complex cases

Mean performance across datasets

KC-fast 0.74 0.72

Consistent + High performance Inconsistent + High performance Consistent + Low performance Inconsistent + Low performance Larger bubble = More tokens

KC-slow

0.76

CodeAct

ToT ReAct

Performance distribution by dataset across all agents

Agent adaptability analysis

Reflexion

0.70

Self-Discovery

0.68 0.66

Plan-Execute

0.64

Standard 0.125

0.150

0.175

0.200

0.225

0.9

0.829 Combined performance score

KC-agent

0.250

Coefficient of variation (lower = more consistent)

0.8

0.707 0.7

0.629 0.567

0.6

0.5

0.4

0.275

Fig. 4. Agent adaptability analysis showing performance consistency vs. mean performance. KC-* agents cluster in the high-performance, low-variation region, demonstrating superior cross-domain generalization.

requiring System 2’s deliberate analysis. This filtering mechanism prevents unnecessary computational overhead while ensuring that critical performance drops receive appropriate attention. F. Agent Adaptability and Performance Distribution Analysis Figure 4 examines the relationship between mean performance and performance variability. KC-* agents demonstrate superior adaptability, clustering in the optimal region with both high mean performance (>0.75) and low coefficient of variation (<0.13), indicating robust generalization capabilities across diverse domains. To understand how agent performance varies across different domains, we analyzed the performance distribution of all agents within each dataset. Figure 5 presents violin plots showing the distribution of combined performance scores across all agents for each dataset. The results reveal significant variation in dataset difficulty: the financial dataset demonstrates the highest mean performance (0.824) with relatively tight distribution, while NASA datasets present greater challenges, with NASA-FD001 exhibiting the widest performance distribution (0.629 mean) and NASA-FD002 showing the lowest overall performance (0.567). These findings highlight the importance of domain-specific evaluation and suggest that certain datasets like NASA-FD001 serve as effective discriminators for agent capability. The clustering of KC-* variants in the high-adaptability region demonstrates that our knowledge consolidation approach successfully addresses cross-domain generalization challenges, making it more suitable for real-world deployment scenarios where domain shifts are common. VI. D ISCUSSION Our experimental results reveal several key insights about automated ML model improvement that extend beyond the specific performance metrics.

0.824

healthcare

eligibility

financial

nasa-FD001

nasa-FD002

Fig. 5. Performance distribution by dataset showing violin plots of combined performance scores. Financial and healthcare datasets show higher mean performance, while NASA datasets present greater challenges with NASAFD002 being most difficult. Dataset difficulty ranking and characteristics

healthcare

Most difficult: nasa-FD002 Easiest: healthcare Avg difficulty: 0.218

#5

0.874

0.832

financial

#4

0.861 0.827

eligibility

#3

0.787 0.703

nasa-FD001

#2

0.705 0.639

nasa-FD002

#1

0.684 0.572

0.0

0.2

0.4

Score (0-1 scale)

0.6

Average performance Ease score (1 - difficulty) Performance variability 0.8

Fig. 6. Dataset difficulty ranking showing average agent performance, ease score (1 - difficulty), and performance variability. NASA-FD002 is the most challenging dataset (#1), while Healthcare is the easiest (#5). Realworld NASA datasets exhibit higher performance variability, indicating less predictable agent behavior compared to synthetic datasets.

The value of dual-process architecture. Neither fast pattern matching (KC-fast) nor deliberate reasoning (KC-slow) alone achieves optimal performance. The critical insight is that simple drift scenarios, constituting approximately 68% of cases in our evaluation, can be efficiently handled by System 1 using previously learned strategies. Only complex cases require System 2’s resource-intensive deliberation. This selective engagement mirrors human cognitive efficiency and explains the 91% speedup over KC-slow while maintaining superior accuracy. The stability-smartness trade-off. Our LLM evaluation reveals that the highest-performing agents are not necessarily the most conservative. CodeAct and KC-fast achieved maximal Stability scores (7.67) through minimal code changes, but this conservatism limited their Smartness scores (5.93 and

6.47). KC-Agent’s willingness to make strategic structural changes while maintaining acceptable stability (6.53) enabled the highest Smartness score (8.33) and superior overall performance. This suggests that effective model improvement requires accepting calculated risks rather than prioritizing code preservation. Why KC-Agent outperforms CodeAct. CodeAct represents the state-of-the-art in code-as-action agents, using executable Python as its action space with interpreter integration for dynamic self-debugging. Despite this sophisticated general-purpose design, KC-Agent outperforms it by 2.4% in accuracy while using 7.9× fewer tokens. This suggests that for ML model improvement specifically, a domain-specialized dual-process architecture with knowledge transfer provides advantages over general-purpose code execution frameworks. The memory systems enable KC-Agent to accumulate domain expertise that CodeAct must rediscover each session. Real-world versus synthetic datasets. The performance gap between synthetic and NASA datasets (Figure 5) highlights the importance of real-world evaluation. While all agents perform reasonably on synthetic data, the NASA datasets with authentic temporal degradation patterns expose fundamental limitations in baseline approaches. KC-Agent’s consistent performance across both categories demonstrates robust generalization. Practical deployment implications. The atomic change principle with immediate rollback ensures that failed improvements never degrade model performance, providing the reliability guarantees necessary for automated MLOps pipelines. Combined with the 13.2-second average execution time, KCAgent is suitable for real-time model maintenance in production environments. VII. L IMITATIONS AND F UTURE W ORK While KC-Agent demonstrates strong performance across our evaluation scenarios, several limitations present opportunities for future research. Our current implementation focuses on tabular data and common Python libraries, requiring extension to complex data types and deep learning frameworks for broader applicability. The approach emphasizes hyperparameter tuning and model selection while avoiding substantial architectural changes. Key limitations include requiring models above 8B parameters for complete improvement loops, limiting deployment in resource-constrained environments, and manual tuning of proxy decision thresholds across domains. Future work will prioritize extending capabilities to deep learning frameworks, developing adaptive threshold mechanisms, and investigating sophisticated memory architectures for MLOps integration. Additionally, the current experimental setup utilizes a Random Forest baseline to provide a highly controlled, easily interpretable environment for isolating the cognitive architecture’s individual components. While this simplifies the task compared to modern ML pipelines, extending KC-Agent to natively interface with complex deep learning architectures represents a primary focus for future work.

VIII. C ONCLUSION We present KC-Agent, a dual-process cognitive architecture that addresses fundamental challenges of automated ML model improvement under data drift. Integrating Kahneman’s dual-process theory with atomic improvement principles, our approach balances rapid response and systematic enhancement in real-world scenarios. KC-Agent achieves superior performance (76.8% accuracy) while maintaining exceptional efficiency (13.2 seconds, 847 tokens). Qualitative evaluation using a consensus-based LLMas-judge framework with three state-of-the-art models confirms the strategic superiority of our approach, with high inter-rater agreement (low standard deviations) validating the methodology. KC-Agent achieved the highest Smartness score (8.33/10) among all agents, significantly outperforming baselines including CodeAct (5.93/10), ReAct (5.33/10), and PlanExecute (1.40/10). Ablation studies demonstrate that the dualprocess architecture is essential, as neither system alone matches the integrated approach. Our work provides theoretical foundations and empirical validation for cognitive-inspired automated ML improvement systems, demonstrating readiness for practical deployment across diverse domains. ACKNOWLEDGMENT We thank Lenovo for providing the technical infrastructure to run the experiments in this paper, and we thank Cotiviti for their support of this research. This work was partially supported by Lenovo and Intel® as part of the Lenovo AI Innovators University Research program, by Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under contract PID2024-160996OB-I00, MICIU/AEI/10.13039/501100011033, and by the Generalitat de Catalunya (AGAUR) under contract 2021-SGR-00478. R EFERENCES [1] F. Heinrichs, “Monitoring machine learning models: Online detection of relevant deviations,” 2023. [Online]. Available: https://arxiv.org/abs/2309.15187 [2] B. Eck, D. Kabakci-Zorlu, Y. Chen, F. Savard, and X. Bao, “A monitoring framework for deployed machine learning models with supply chain examples,” in Proceedings of the 2022 IEEE International Conference on Big Data, Osaka, Japan, 2022, pp. 2231–2238. [3] Z. C. Lipton, Y. Wang, and A. J. Smola, “Detecting and correcting for label shift with black box predictors,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 3128–3136. [4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

[5] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang, “AgentBench: Evaluating LLMs as Agents,” 2023. [Online]. Available: https://arxiv.org/abs/2308.03688 [6] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su, “MIND2WEB: towards a generalist agent for the web,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS’23. Curran Associates Inc., 2023. [7] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS’23. Curran Associates Inc., 2023. [8] C. Gao, X. Lan, N. Li, Y. Yuan, J. Ding, Z. Zhou, F. Xu, and Y. Li, “Large language models empowered agent-based modeling and simulation: A survey and perspectives,” Humanities and Social Sciences Communications, vol. 11, 2024. [9] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative Agents: Interactive Simulacra of Human Behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, ser. UIST’23. Association for Computing Machinery, 2023. [10] D. Kahneman, Thinking, fast & slow. Farrar, Straus & Giroux, 2011. [11] J. Clear, Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones. Penguin Publishing Group, 2018. [Online]. Available: https://jamesclear.com/atomic-habits [12] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” in Proceedings of the Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [13] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: deliberate problem solving with large language models,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS’23. Curran Associates Inc., 2023. [14] P. Zhou, J. Pujara, X. Ren, X. Chen, H.-T. Cheng, Q. V. Le, E. H. Chi, D. Zhou, S. Mishra, and H. S. Zheng, “Self-discover: large language models self-compose reasoning structures,” in Proceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS’24. Curran Associates Inc., 2024. [15] L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 2609–2634. [16] X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024. [17] B. Y. Lin, Y. Fu, K. Yang, F. Brahman, S. Huang, C. Bhagavatula, P. Ammanabrolu, Y. Choi, and X. Ren, “SWIFTSAGE: a generative agent with fast and slow thinking for complex interactive tasks,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS’23. Curran Associates Inc., 2023. [18] K. Christakopoulou, S. Mourad, and M. Matarić, “Agents thinking fast and slow: A talker-reasoner architecture,” 2024. [Online]. Available: https://arxiv.org/abs/2410.08328 [19] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, vol. 114, no. 13, p. 3521–3526, Mar. 2017. [20] M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,” 2019. [Online]. Available: https://arxiv.org/abs/1810.11910 [21] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS’20. Curran Associates Inc., 2020. [22] N. Dı́az-Rodrı́guez, V. Lomonaco, D. Filliat, and D. Maltoni, “Don’t forget, there is more than forgetting: new metrics for continual learning,” 2018. [Online]. Available: https://arxiv.org/abs/1810.13166

[23] M.-A. Zöller and M. F. Huber, “Benchmark and survey of automated machine learning frameworks,” Journal of Artificial Intelligence Research (JAIR), vol. 70, pp. 409–472, 2021. [24] P.-Y. Chen and P. Das, “AI Maintenance: A Robustness Perspective,” Computer, vol. 56, no. 2, pp. 48–56, 2023. [25] O. Gheibi, D. Weyns, and F. Quin, “Applying machine learning in selfadaptive systems: A systematic literature review,” ACM Trans. Auton. Adapt. Syst., vol. 15, no. 3, Aug. 2021. [26] Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, M. Malhotra, P. P. Liang, H. W. Park, Y. Yang, X. Xu, Y. Du, S. Patel, T. Althoff, D. McDuff, and X. Liu, “Towards a science of scaling agent systems,” 2025. [Online]. Available: https://arxiv.org/abs/2512.08296

Related documents

• Amplification to Synthesis: A Comparative Analysis of Cognitive Operations Before and After Generative AI#180686
• CM2: Multimodal Cultural Reasoning via an Integrated Multi-Agent Framework#799694
• OrchSLM: Probing the Dynamics of Small Language Model Orchestration#1071256
• From Automated to Autonomous: Hierarchical Agent-native Network Architecture (HANA)#216774
• The Chinese Room Instantiated: Large Language Models Are Not Cognitive Agents but Highly Efficient Solving Machines#696234