CoFEE: Reasoning Control for LLM-Based Feature Discovery
arXiv:2604.21584v1 [cs.AI] 23 Apr 2026
Maximilian Westermann∗ , Ben Griffin∗ , Aaron Ontoyin Yin† , Zakari Salifu† , Yagiz Ihlamur‡ , Kelvin Amoaba† , Joseph Ternasky† , Fuat Alican† , Yigit Ihlamur† ∗ University of Oxford † Vela Research ‡ Amazon
Abstract—Feature discovery from complex unstructured data is fundamentally a reasoning problem: it requires identifying abstractions that are predictive of a target outcome while avoiding leakage, proxies, and post-outcome signals. With the introduction of ever-improving Large Language Models (LLMs), our method provides a structured method for addressing this challenge. LLMs are well suited for this task by being able to process large amounts of information, but unconstrained feature generation can lead to weak features. In this work, we study reasoning control in LLMs by inducing cognitive behaviors for improving feature discovery. We introduce CoFEE (Cognitive Feature Engineering Engine), a reasoning control framework that enforces cognitive behaviors in how the LLM reasons during feature discovery. From a machine learning perspective, these cognitive behaviors act as structured inductive biases over the space of candidate features generated by the model. These behaviors have been exploited with success in ML models, and include backward chaining from outcomes, subgoal decomposition, verification against observability and leakage criteria, and explicit backtracking of rejected reasoning paths. CoFEE does not modify model architecture, training, or inference; reasoning control is implemented entirely through structured prompts. In a controlled comparison, we show that enforcing cognitive behaviors yields features with higher empirical predictability than those under unconstrained vanilla LLM prompts. CoFEE achieves an average Success Rate Score that is 15.2% higher than the vanilla approach, while generating 29% fewer features and reducing costs by 53.3%. Using held-out feature evaluation, we assess whether cognitively induced features generalize beyond the data used for discovery. Our results indicate that, in our evaluated setting, reasoning control is associated with improvements in quality and efficiency of LLM-based feature discovery. Index Terms—Reasoning Control, Feature Engineering, Inductive Bias
I. I NTRODUCTION
F
EATURE discovery is a critical step in analytical pipelines, particularly in domains that rely heavily on unstructured or complex data such as venture capital (VC). In these settings, relevant signals are often implicit, non-linear, and distributed across unstructured sources, making manual feature construction difficult. While interpretable, rule-based models such as linear models and decision trees offer transparent, auditable reasoning, they often struggle in capturing the complexity of available data [1]. This leads to the inability to maximize the predictive
This work was done outside of the current role that Yagiz Ihlamur holds at Amazon. Correspondence: Maximilian Westermann <[email protected]>.
Fig. 1. Overview of the CoFEE pipeline.
effectiveness of features. Recent work such as GPTree addresses this limitation partially by integrating large language models (LLMs) into a decision tree, allowing transparent and interpretable decision tree reasoning to provide valuable information for VC decision-making [1]. A complementary challenge, feature engineering, arises upstream of model construction. Previous work has shown that well-structured features can improve predictability across learning algorithms [2], yet feature discovery remains a major bottleneck, especially in the extraction of unstructured data. LLMs offer automated feature discovery as a potential solution, with their ability to interpret large amounts of data using reasoning [3]. However, insufficient prompting of LLMs to propose features can produce features with limited predictability [4]. This raises a key question: How can reasoning behaviors induced by cognitive prompting act as inductive biases to produce highly predictive and interpretable features using LLMs? Recent work has identified a set of reasoning behaviors such as backward chaining, subgoal decomposition, verification, and backtracking that correlate with improved learning and self-improvement in language models [5], [6]. Complementary
2
Fig. 2. CoFEE pipeline. Agent 1 performs cognitive feature selection to construct an initial master list, which is refined via semantic similarity by Agent 2. Agent 3 then scores each feature by counting the number of successful and unsuccessful founders exhibiting it.
research further supports this perspective, arguing that AI evaluation should mirror human testing practices by explicitly evaluating reasoning processes rather than evaluating only final outputs [7]. In LLMs, such behaviors can be instantiated through prompt-level structure, motivating cognitive prompting as a practical control interface over model reasoning. Motivated by these findings, we introduce CoFEE (Cognitive Feature Engineering Engine), an agent-based pipeline that is used to produce features (Fig. 1). Analogous to how GPTree structures LLM reasoning with decision trees, CoFEE structures LLM reasoning upstream during feature discovery. Using CoFEE, we conduct a controlled empirical study comparing the same pipeline, with one version using cognitive feature discovery carried out by an agent using cognitive prompting, and a baseline using vanilla GPT-5.2 for feature discovery. In this work, we find that cognitive prompting consistently produces features with higher predictability scores. These findings suggest that prompting cognitive behaviors can serve as an effective design strategy for LLM-based feature engineering. II. R EASONING C ONTROL VIA C OGNITIVE C ONSTRAINTS We use the term reasoning control to describe how we constrain LLMs’ reasoning behavior, implemented in our framework through cognitive prompting. In machine learning terms, this corresponds to imposing structured inductive biases on the feature generation process. These constraints are implemented using prompts, enforcing discipline during generation. In CoFEE, reasoning control is implemented through four cognitive behavior constraints as described in Gandhi et al. [5]: Backward chaining: Start from the desired outcome and reason backward. • Subgoal decomposition: Break down complex tasks into smaller, manageable steps. • Verification: Systematically check the results of each intermediate step. • Backtracking: Revise steps explicitly when they fail.
•
III. DATASET To validate the effectiveness of inducing cognitive behaviors in LLMs, we use a dataset consisting of 1,000 founder profiles collected from publicly available sources. This dataset contains information about the founder and company (e.g., background, roles, sector, funding, etc.), and the success outcome. It consists of 400 (40%) successful and 600 (60%) unsuccessful founders. We classify ’successful’ founders as those that achieve an M&A or IPO valuation exceeding $500M, or raised more than $500M in total funding [8]. A. Held-Out Feature Evaluation To assess whether these features discovered in this pipeline generalize beyond the data used for discovery, we carry out a held-out feature evaluation following standard ML practice. Rather than evaluating on the same data, we explicitly assess these features on a held-out dataset for feature evaluation. 1,000 founders are used for discovery, and we use another 1,000 founders (again with 40% successful) to assess feature performance. The discovery set is passed through the pipeline for discovery and refinement. The held-out evaluation set is never exposed to the feature discovery pipeline. Once discovery is completed, the features are frozen. These frozen features are then applied to the held-out evaluation set, where feature predictive quality is assessed. This ensures that no information in the held-out set influences feature discovery, allowing this evaluation to assess feature generalization. IV. P IPELINE OVERVIEW In this work, we present CoFEE, an agent-based pipeline that explicitly enforces structured cognitive behavior during feature discovery. While models such as GPT-5.2 already exhibit certain reasoning behaviors, previous studies suggest that these behaviors can be further strengthened and systematically induced through explicit prompting and structural constraints. With CoFEE, we explore the feasibility of leveraging these enforced cognitive qualities to enhance feature discovery compared to vanilla GPT-5.2. The pipeline (see Fig. 2) breaks
3
down this process into three specialized agents (for discovery, scoring, refining) using GPT-5.2 for each agent in this pipeline. Viewed as an ML system, CoFEE implements a generator–evaluator loop in which cognitively constrained reasoning guides feature hypothesis generation prior to evaluation. A. Agent 1: Feature Discovery Agent 1 is the primary agent for this pipeline and is responsible for proposing candidate features based on the provided dataset. The agent receives structured prompts that explicitly induce cognitive behaviors, including backward chaining, subgoal decomposition, verification, and backtracking. Backward chaining is used to reason from the target outcome to features; subgoal decomposition structures feature discovery around high-level causal categories; verification enforces observability and non-proxy constraints; and backtracking records and rejects invalid reasoning paths. In the vanilla GPT-5.2, Agent 1 proposes features without these cognitive constraints. This agent processes the dataset 50 founders at a time, extracting features and adding them to a master list until the entire dataset is analyzed. The initial prompts for Agent 1 of CoFEE and vanilla prompting appear below. Full prompts are provided in the Supplement. a) CoFEE Prompt (Agent 1): "You are Agent 1: a stateless Feature Discovery agent. You are given a batch of 50 founder records. You do NOT know which founders are successful or unsuccessful. You have NO memory of previous batches. Your task is to propose candidate FEATURES that could plausibly distinguish successful from unsuccessful founders. You are performing Cognitive Feature Reasoning. You must explicitly apply the following cognitive behaviors. You must produce structured outputs and make explicit decisions. -------------------------------1. BACKWARD CHAINING -------------------------------Start from system-level success or failure. For each proposed mechanism: - State the causal hypothesis explicitly. - Explain why this mechanism would operate *before* success. - Map the mechanism to at least one measurable or inferable quantity available in the dataset. You may reason about hidden variables, but the final feature MUST be: - observable pre-success expressible in deterministic logic
If a mechanism cannot be mapped to an observable feature, abandon it. -------------------------------2. SUBGOAL SETTING -------------------------------Organize exploration into NO MORE THAN 4 subgoals chosen from: founder capability formation - team coordination and complementarity market structure and constraints early execution dynamics For each subgoal: - List candidate mechanisms. - Maintain the hierarchy: system behavior → mechanism → feature. If a subgoal: - collapses into a proxy - fails observability - has ambiguous causal direction then explicitly ABANDON or REVISE the subgoal and explain why. -------------------------------3. VERIFICATION -------------------------------For each proposed feature, verify: - it is observable before the success outcome - it encodes a plausible causal mechanism - it is not a prestige-based, descriptive, or post-outcome proxy For each feature, list: - potential bias sources - uncertainty or ambiguity If verification fails, reject the feature. -------------------------------4. BACKTRACKING -------------------------------Explicitly record every abandoned reasoning path. For each abandoned path, record: why it initially seemed promising which constraint caused rejection (proxy risk, leakage, observability failure, causal ambiguity) Use these abandoned paths to bias future exploration away from similar dead ends. -------------------------------b) Vanilla Prompt (Agent 1): "You are Agent 1: a stateless Feature Discovery agent. You are given a batch of 50 founder records. You do NOT know which founders are successful or unsuccessful. You have NO memory of previous batches. Your task is to propose candidate
4
FEATURES that could plausibly distinguish successful from unsuccessful founders. Features must be observable pre-success You must produce structured outputs and make explicit decisions."
Cognitive Prompting: where Agent 1 is constrained to explicitly apply the cognitive behaviors: backward chaining, subgoal decomposition, verification, and backtracking. • Vanilla Prompting: where Agent 1 is prompted to propose features without explicit cognitive constraints. Between the two conditions, the remainder of the pipeline components are held constant (Agents 2–3). This ensures that any observable differences can be attributed to differences in Agent 1 prompting. •
B. Agent 2: Feature Consolidation Agent 2 identifies semantically overlapping features and merges those that represent the same underlying mechanism as demonstrated in Fig. 3. Merging decisions are conservative and preserve feature provenance to maintain interpretability. This step reduces redundancy while ensuring that distinct causal mechanisms remain separate.
A. Evaluation Metric For each feature, we compute a success-rate delta (∆SR), which measures the difference in success rates between founders who exhibit the feature and those who do not. This is computed by comparing the prevalence of the feature among successful founders to its prevalence among unsuccessful founders, with larger differences indicating stronger discriminative power. This metric allows us to quantify how effectively a given feature set separates successful founders from unsuccessful ones. To evaluate whether cognitive prompting improves feature quality, we compare the top ten features ranked by ∆SR for each experimental condition. By examining differences in these top-ranked features, we assess whether cognitive prompting leads to systematically stronger or more discriminative features.
Fig. 3. Diagram illustrating the Agent 2 process, in which semantically similar features are compared, their similarity is justified, and the features are combined.
VI. R ESULTS A. Feature Discovery Results
C. Agent 3: Scoring Agent 3 evaluates features by comparing feature name and definitions against founder records in batches of up to 100 features and 1,000 founders at a time. When a feature matches a founder, the founder is tagged with that feature. After all founders have been tagged, Agent 3 outputs a JSON file recording feature assignments. This output is then used to deterministically compute feature statistics. n1 denoting the number of successful founders exhibiting a given feature, and n0 denoting the number of unsuccessful founders exhibiting the feature. We define the success-rate delta (∆SR) as the difference between the success probability conditioned on feature presence and the success probability conditioned on feature absence: ∆SR =
n1 (N1 − n1 ) − n1 + n 0 (N1 − n1 ) + (N0 − n0 )
(1)
where N1 and N0 denote the total numbers of successful and unsuccessful founders in the dataset, respectively. V. E XPERIMENT We evaluate the impact of cognitively structured prompting on feature discovery using a controlled experimental setup. We compare two feature discovery conditions:
To ensure statistical stability and modeling relevance, we restrict analysis to features with marginal support ≥ 10% of the sample (n ≥ 100). For a binomial proportion with n = 100, the maximum standard error (attained at p = 0.5) is 0.05. This threshold filters high-variance rare features that are unlikely to generalize out-of-sample and concentrates evaluation on predictors with meaningful population coverage. The top 10 features discovered using CoFEE (cognitive prompting) are shown in Table I. The top 10 features discovered using vanilla prompting are shown in Table II. Finally, the side-by-side comparison of features discovered by CoFEE and vanilla prompting are shown in Table III. Further metric comparisons are presented in the Appendix. TABLE I T OP C O FEE FEATURES RANKED BY SUCCESS - RATE DELTA (∆SR) Feature ID top_university_education_flag education_top10_qs_flag highest_degree_level education_top50_qs_flag technical_background_flag functional_role_diversity job_count_total job_tenure_longest_bucket cross_industry_breadth_count functional_breadth_score
n1 82 67 304 70 190 241 254 284 252 245
n0 37 33 292 41 155 222 241 289 239 229
∆SR 0.328 0.300 0.272 0.259 0.230 0.224 0.224 0.224 0.222 0.222
5
TABLE II T OP VANILLA FEATURES RANKED BY SUCCESS - RATE DELTA (∆SR) Feature ID has_top10_qs_education data_completeness_score max_role_seniority_level education_qs_rank_best_numeric has_senior_executive_role functional_background_primary role_seniority_score_max founder_tenure_years_estimate max_seniority_level has_any_industry_match_prior_job_flag
n1 81 281 269 217 209 282 262 144 263 116
n0 34 277 269 201 192 302 273 122 275 97
∆SR 0.344 0.234 0.216 0.205 0.202 0.199 0.193 0.193 0.192 0.184
TABLE III C OMPARISON OF FEATURE QUALITY AND EFFICIENCY BETWEEN C O FEE AND VANILLA PROMPTING . F EATURE QUALITY IS EVALUATED USING SUCCESS - RATE DELTA (∆SR) ON HELD - OUT DATA . Metric
CoFEE
Vanilla
Mean ∆SR (Top-10) Median ∆SR Total Features Generated Total Cost (USD)
0.250 0.227 157 $8.54
0.217 0.204 222 $18.29
B. Feature Discovery Comparison and Interpretation Overall, CoFEE produces a broader distribution of highly predictive features, with consistently higher ∆SR values across the top-ranked features compared to vanilla prompting. VII. D ISCUSSION The results of this study suggest that explicitly structuring cognitive prompts in an LLM, specifically GPT-5.2, for feature discovery has a measurable impact on the quality of the resulting features. When cognitive behaviors such as backward chaining, subgoal decomposition, verification, and backtracking are enforced, the discovered features achieve higher predictability scores than those generated via vanilla prompting. Because all other pipeline components are held constant, the observed improvements are consistent with the presence of cognitive structure in feature discovery. This indicates that cognitive prompting provides a structured constraint on LLM feature generation behavior. Comparing costs across the two approaches, the CoFEE evaluation incurred a total cost of $8.54, whereas the vanilla evaluation cost $18.29. The higher cost of the vanilla approach is attributable to the larger number of features it produced (222 features), compared to CoFEE’s 157 features. The increased feature count in the vanilla setting resulted in a greater number of API calls during the feature scoring and merging stages. These results suggest that inducing cognitive behaviors via prompting functions as an effective inductive bias in LLMbased feature discovery systems. Despite the empirical improvements observed, important limitations remain. First, the results are evaluated on a single domain, and do not establish generalization beyond this setting. Second, ∆SR captures empirical predictability differences but does not directly assess downstream task performance. Third, the induced cognitive behavior prompts and the
robustness of these effects across alternative models, prompting formulations, and scales have not yet been evaluated. Future work will evaluate CoFEE across multiple domains, measure downstream model performance when incorporating discovered features, and examine robustness across different model architectures and prompt structures. VIII. C ONCLUSION CoFEE provides empirical evidence that explicit reasoning control can improve LLM-based feature discovery while reducing costs by 53.3% in our evaluated setting. By inducing structured cognitive behaviors, the pipeline produces features with higher ∆SR values obtained at lower computational cost relative to the vanilla prompt baseline. These findings suggest that reasoning control may serve as a practical pipeline design strategy for LLM-based analytical systems. R EFERENCES [1] S. Xiong, Y. Ihlamur, F. Alican, and A. O. Yin, “GPTree: Towards explainable decision-making via LLM-powered decision trees,” 2024. [2] J. Heaton, “An empirical analysis of feature engineering for predictive modeling,” 2016. [3] Y. Wang, S. Wu, Y. Zhang, W. Wang, Z. Liu, J. Luo, and H. Fei, “Multimodal chain-of-thought reasoning: A comprehensive survey,” 2025. [4] J. Wu, M. Feng, S. Zhang, F. Lv, R. Jin, F. Che, Z. Wen, and J. Tao, “Boosting multimodal reasoning with automated structured thinking,” 2025. [5] K. Gandhi, A. Chakravarthy, A. Singh, N. Lile, and N. D. Goodman, “Cognitive behaviors that enable self-improving reasoners: Four habits of highly effective STaRs,” 2025. [6] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” 2022. [7] Y. Zhuang, Q. Liu, Z. A. Pardos, P. C. Kyllonen, J. Zu, Z. Huang, S. Wang, and E. Chen, “Position: AI evaluation should learn from how we test humans,” 2024. [8] R. Chen, J. Ternasky, A. S. Kwesi, B. Griffin, A. O. Yin, Z. Salifu, K. Amoaba, X. Mu, F. Alican, and Y. Ihlamur, “VCBench: Benchmarking LLMs in venture capital,” 2025.
IX. A PPENDIX Two tables included comparing feature quality metrics for the two discovery approaches. Precision denotes the conditional success probability among founders exhibiting the feature, P (Y = 1 | f = 1). Support represents the proportion of the dataset exhibiting the feature, (n1 +n0 )/(N1 +N0 ), reflecting population coverage. Lift measures relative improvement over the baseline success rate and is defined as Precision/0.4, where 0.4 is the global base rate of success in the dataset. TABLE IV T OP C O FEE FEATURES WITH EXTENDED METRICS Feature F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
n1 82 67 304 70 190 241 254 284 252 245
n0 37 33 292 41 155 222 241 289 239 229
Precision 0.689 0.670 0.510 0.631 0.551 0.521 0.513 0.496 0.513 0.517
∆SR 0.328 0.300 0.272 0.259 0.230 0.224 0.224 0.224 0.222 0.222
Lift 1.723 1.675 1.275 1.577 1.377 1.301 1.283 1.239 1.283 1.292
Support 0.119 0.100 0.596 0.111 0.345 0.463 0.495 0.573 0.491 0.474
6
TABLE V T OP VANILLA FEATURES WITH EXTENDED METRICS Feature F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
n1 81 281 269 217 209 282 262 144 263 116
n0 34 277 269 201 192 302 273 122 275 97
Precision 0.704 0.504 0.500 0.519 0.521 0.483 0.490 0.541 0.489 0.545
∆SR 0.344 0.234 0.216 0.205 0.202 0.199 0.193 0.193 0.192 0.184
Lift 1.761 1.259 1.250 1.298 1.303 1.207 1.224 1.353 1.222 1.362
Support 0.115 0.558 0.538 0.418 0.401 0.584 0.535 0.266 0.538 0.213
X. S UPPLEMENTARY To ensure reproducibility and structured evaluation of feature discovery outputs, we provide the exact JSON schema used by Agent 1. This template captures not only the final feature specification, but also the intermediate reasoning structure (subgoals, causal mechanisms, and abandoned ideas), enabling inspection of reasoning control effects. 1
{ "batch_id": "string", "features": [ { "feature_name": "string", "subgoal": "string", "causal_mechanism": "string", "definition": "string", "computation_logic": "string", "abandoned_ideas": [ { "idea": "string", "reason": "string" } ] } ]
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
}