Governed Human–AI Prioritization Under Uncertainty: Adaptive Estimation and Dependency-Constrained Portfolio Selection Azzeddine Ihsine
Sara Ihsine
arXiv:2609.10648v1 [cs.SE] 9 Sep 2026
Inovionix [email protected], [email protected]
Abstract AI-native software engineering increasingly combines human judgment, historical analogy, parametric estimation, and AI-generated forecasts inside the same prioritization decision. The resulting problem is not merely how to rank candidate work, but how to govern heterogeneous estimates, uncertainty, strategic parameters, dependencies, and limited capacity in a way that remains inspectable and recalibratable. We study five quantitative operators used in the D-POAF decision practice: Business Value Score (BVS), Effort and Risk Score (ERS), Prioritization Value Score (PVS), Collective Calibration Score (CCS), and Optimal Development Path (ODP). Controlled synthetic experiments characterize their behavior under known latent variables and explicit error processes. Moderate BVS-weight perturbations preserved global rankings (median Spearman 0.986), while broader strategic changes reduced top-10% overlap to 0.788. Reliability-weighted effort aggregation achieved MAE 0.616, a 42.6% reduction relative to the best individual estimator (MAE 1.073), and outperformed a simple mean (MAE 0.638). Under estimator drift, adaptive reliability weighting reduced average RMSE by 4.5% relative to equal weighting. Model–collective divergence detected the highest-error priority estimates with ROC-AUC 0.906. In 800 dependency-constrained portfolio instances, value-to-effort achieved mean objective ratio 0.962 and ODP distance 0.953 against the exact optimum; bootstrap intervals confirm a small but systematic advantage for value-to-effort under the declared objective. These results establish a quantitative basis for evidence-weighted human–AI prioritization and define the optimization boundary between shortest-path ODP formulations and general release-portfolio selection.
Keywords: AI-native software engineering; human–AI decision making; software prioritization; effort estimation; uncertainty; adaptive calibration; portfolio optimization; D-POAF.
1
Introduction
Software engineering decisions routinely trade expected value against effort, risk, timing, dependencies, and limited capacity. Value-Based Software Engineering established that software work is not value-neutral and that economic reasoning belongs inside engineering decisions [1]. Requirements prioritization reached the same problem from another direction: cost–value methods compare stakeholder value and implementation cost [2], while the Next Release Problem formalizes release selection under resource limits and demonstrates that realistic selection is computationally non-trivial [3]. Reviews of prioritization methods report a broad landscape of ranking, voting, optimization, search-based, fuzzy, and learning approaches, together with persistent challenges around dependencies, uncertainty, stakeholder coordination, and empirical evaluation [4, 5, 6]. AI-native engineering intensifies this decision problem because multiple estimators can influence the same commitment. A domain expert may estimate from experience, an analogy method from 1
similar work, a parametric model from project attributes, and an AI system from learned patterns. Software-effort research has long shown strong context dependence across estimation families [7, 8]. Ensemble reviews show that combining estimators frequently improves accuracy, while dynamic selection remains comparatively underexplored and no single ensemble rule dominates every context [9]. Human–AI decision research adds a second requirement: predictive accuracy alone does not determine the quality of a human–AI team, and poorly calibrated decision support can produce overreliance [10, 11]. This paper addresses three linked scientific questions. First, how stable are explicit prioritization scores when their strategic parameters change? Second, can observed predictive reliability determine estimator influence more effectively than fixed role-based weighting? Third, when dependencies and budgets turn prioritization into a combinatorial portfolio problem, how close do transparent heuristics come to the exact optimum? We instantiate these questions using the decision operators documented in the D-POAF reference practice [12, 13, 14]. D-POAF supplies the governance context, but the quantitative contribution is independent of the framework label: the experiments test parameterized ranking, reliability-weighted estimation, calibration signals, and dependency-constrained portfolio selection as explicit computational mechanisms. Our contributions are: 1. a formal separation of expected value, feasibility, calibration, and post-delivery value, preventing one score from silently standing in for all four; 2. sensitivity evidence showing where BVS, ERS, and PVS parameter changes preserve rankings and where they alter top-portfolio membership; 3. controlled evidence that reliability-weighted and adaptive estimator aggregation can outperform fixed-role alternatives under heterogeneous error and drift; 4. a CCS diagnostic experiment quantifying how model–collective divergence can surface priority miscalibration; 5. an exact portfolio benchmark establishing the performance and optimization boundary of ODP-related heuristics under budget and dependency constraints.
2
Related Work
2.1
Value and requirements prioritization
Value-Based Software Engineering integrates stakeholder value, risk, and investment reasoning into software practice [1]. Karlsson and Ryan’s cost–value approach treats prioritization as an explicit value–cost trade-off rather than a single importance rank [2]. The Next Release Problem formalizes selection under budgets and customer demand, and its complexity result shows why realistic release planning cannot generally be reduced to sorting [3]. Search-Based Software Engineering has extended this line to single- and multi-objective optimization, while reviews continue to identify risk, dependencies, uncertainty, scalability, and empirical validation as active concerns [5, 6]. The contribution studied here is a governed decomposition of the decision: value, feasibility, calibration, dependencies, budget, and measured outcomes remain separately traceable rather than being collapsed into an opaque ranking procedure.
2.2
Effort estimation and uncertainty
Expert judgment remains central to software estimation. Evidence-based guidance recommends structured judgment, historical data, independent estimates, explicit uncertainty assessment, and feedback on prior accuracy [7, 15]. A systematic review of software cost-estimation research found strong context dependence across approaches [8]. Ensemble effort estimation seeks to reduce individual-model weaknesses; an updated review reports frequent ensemble gains and 2
comparatively limited work on dynamic ensemble selection [9]. This motivates reliability-weighted aggregation in which estimator influence is updated from measured predictive performance.
2.3
Human–AI calibration
Human–AI collaboration adds a calibration problem beyond raw prediction accuracy. Bansal et al. show that the most accurate standalone AI is not necessarily the best teammate [10], while Bucinca et al. demonstrate that AI-assisted decisions can induce overreliance and that interventions can alter this behavior [11]. In the model studied here, collective judgment does not replace predictive scoring; it provides an independent calibration signal capable of triggering investigation when model and collective assessments diverge.
3
Decision Model
3.1
Business Value Score
For a candidate block, BVS aggregates three 1–10 judgments: BVS = w1 Im + w2 Um + w3 Os ,
(1)
with wi ≥ 0 and i wi = 1. Im denotes business impact, Um urgency or criticality, and Os strategic opportunity. The weights are explicit strategy parameters whose changes can be versioned and replayed [14]. P
3.2
Effort and Risk Score
The feasibility score is
ERS = 1 + βE + γR,
(2)
where E is estimated effort, R technical risk, and β, γ govern sensitivity to each factor. The reference practice also permits multiple estimation sources and explicit uncertainty adjustment [14].
3.3
Prioritization Value Score
Prioritization combines value and feasibility through PVSα =
BVSα . ERS
(3)
The exponent α controls how aggressively high business value dominates feasibility. Rather than treating α as a hidden tuning constant, we measure how its changes affect ranking and top-portfolio membership.
3.4
Collective calibration
The D-POAF reference practice defines a Collective Calibration Score using computed priority and allocated voting points [14]. Our experiment isolates the diagnostic principle: if a model estimate and an independently generated collective signal diverge strongly, that divergence can be used to identify candidate miscalibration before commitment. Calibration and realized value remain separate constructs.
3
Table 1: Experimental design and supported inference. Study
Test and inference
Sensitivity Effort
Rank and top-10% stability under BVS, ERS, and PVS parameter changes. MAE, RMSE, bias, and standardized accuracy for four estimators and three aggregators. Equal weighting versus reliability weights updated from observed recent errors under drift. Detection of deliberately injected priority miscalibration from model–collective divergence. Exact value-maximizing feasible subsets versus four transparent heuristics and a random baseline.
Adaptation CCS Portfolio
3.5
ODP and the optimization boundary
The reference practice defines a block distance d = ERS/BVS and uses shortest-path reasoning for ODP on dependency graphs [14, 16]. Shortest-path optimization is exact for its declared additive path objective. Release selection under budget and prerequisite constraints is a different combinatorial problem. We therefore benchmark ODP distance as a transparent heuristic against exact constrained portfolio selection.
4
Research Questions and Experimental Method
We evaluate four research questions: • RQ1: How sensitive are BVS, ERS, and PVS rankings to governed parameters? • RQ2: Under heterogeneous estimator error, how do individual and aggregated effort estimates compare? • RQ3: Do adaptive reliability weighting and model–collective divergence provide effective calibration signals under nonstationarity and injected miscalibration? • RQ4: How close do transparent D-POAF sequencing and selection heuristics come to the exact portfolio optimum under budget and dependency constraints? The experiments use controlled synthetic data with seed 20260827. This design provides known latent variables, explicit error processes, and exact counterfactual comparisons, allowing mechanism-level effects to be measured directly. We use 5,000 blocks for ranking sensitivity, 12,500 tasks for effort-estimation calibration and testing, 20 Waves with 250 tasks each for adaptive weighting, 6,000 proposals for the CCS diagnostic, and 800 dependency-constrained portfolio instances.
4.1
Evaluation metrics
Rank stability uses Spearman correlation and top-10% overlap. Effort prediction uses MAE, RMSE, mean signed bias, and standardized accuracy against a random-guess baseline following Shepperd and MacDonell [17]. CCS diagnostic quality uses ROC-AUC and precision/recall at a fixed 20% flagging rate. Portfolio quality uses objective ratio relative to the exact optimum, normalized regret, and exact-optimum rate. For the 800-instance portfolio benchmark we additionally report nonparametric bootstrap 95% confidence intervals for mean objective ratio and paired bootstrap intervals for heuristic differences.
4
Stability relative to alpha = 1.5
1.0 0.8 0.6 0.4 0.2 0.0
Spearman rank correlation Top 10% overlap
0.75
1.00
1.25
1.50 1.75 2.00 PVS exponent alpha
2.25
2.50
Figure 1: Ranking sensitivity to the PVS exponent, measured relative to α = 1.5.
5
Results
5.1
RQ1: governed parameters materially shape the top portfolio
BVS rankings were highly stable under small weight perturbations but changed under broader strategic shifts. With baseline weights (0.4, 0.3, 0.3), narrow perturbations produced median Spearman correlation 0.986 and median top-10% overlap 0.898. Under broader perturbations, median correlation decreased to 0.945 and top-10% overlap to 0.788. The result identifies a useful governance property: small parameter changes preserve global order, while meaningful strategy changes can alter which work crosses a funding cutoff. The PVS exponent has a similarly measurable effect. Comparing α = 1 with α = 2 yields Spearman correlation 0.955 but only 0.764 top-10% overlap. Thus, α is not a cosmetic parameter; it directly determines how aggressively value dominates feasibility. ERS changes were less disruptive in the sampled regime. Across β, γ ∈ {0.05, 0.10, 0.15, 0.20}, global rankings remained strongly correlated with the baseline, while top-set membership still changed when effort or risk sensitivity was doubled. Global rank correlation therefore does not eliminate the need to version parameters near decision cutoffs.
5.2
RQ2: reliability weighting improves heterogeneous effort aggregation
Four imperfect estimators were calibrated on 2,500 historical tasks and evaluated on a disjoint 10,000-task test set. Inverse-MSE calibration produced weights 0.353 (Expert), 0.217 (AI), 0.272 (Analogy), and 0.159 (Parametric). The reliability-weighted ensemble achieved MAE 0.616 and RMSE 0.772. Relative to the best individual estimator (Expert: MAE 1.073, RMSE 1.334), reliability weighting reduced MAE by 42.6% and RMSE by 42.1%. It also improved on the simple mean (MAE 0.638, RMSE 0.798), demonstrating that observed estimator quality can provide a stronger aggregation rule than equal influence under heterogeneous error. This establishes the mechanism-level principle that estimator influence can be tied to measured predictive reliability rather than role labels such as “human”, “AI”, or “expert”.
5
Table 2: Effort-estimation performance on the synthetic test set. Method
MAE
RMSE
SA
Reliability-weighted Mean Median Expert Analogy AI Parametric
0.616 0.638 0.692 1.073 1.193 1.288 1.522
0.772 0.798 0.864 1.334 1.486 1.604 1.900
0.754 0.746 0.724 0.572 0.524 0.487 0.394
ert p x E
lo Ana
Mean absolute error
1.5 1.0 0.5 0.0 b
ia Rel
-w ility
ted h g ei
an
Me
an
di Me
gy
AI
a Par
ic
tr me
Estimator or aggregation rule Figure 2: MAE for individual and aggregated effort estimators.
5.3
RQ3: adaptive calibration tracks drift and exposes miscalibration
We modeled a nonstationary setting in which expert, analogy, and parametric error remained approximately stable while AI estimator noise decreased over 20 Waves. Reliability weights were recomputed from a rolling window of observed squared error. Average RMSE was 0.793 for equal weighting and 0.757 for adaptive reliability weighting, a 4.5% reduction. By Wave 20, the AI weight increased from 0.25 to 0.388, while the parametric weight decreased to 0.131. The experiment demonstrates that measured performance can shift future influence automatically as estimator quality changes. For collective calibration, stale-context bias was injected into 15% of model estimates while the collective signal was generated independently from the latent priority with its own noise. Standardized model–collective divergence achieved ROC-AUC 0.906 for detecting the top 20% of absolute model miscalibration. Flagging the top 20% by divergence produced precision and recall 0.688 against a base rate of 0.20; the resulting precision is 3.44× the prevalence. Divergence therefore functions as an effective diagnostic signal when the collective channel contributes partially independent information.
6
Equal-weight ensemble Adaptive reliability weighting
0.90
RMSE
0.85 0.80 0.75 0.70 0.65 2.5
5.0
7.5
10.0 12.5 Wave
15.0
17.5
20.0
Figure 3: Equal versus adaptive reliability weighting under estimator drift. Table 3: Portfolio performance against exact constrained selection.
5.4
Method
Mean ratio (95% CI)
Exact rate
Value/effort ODP distance PVS α = 2 BVS greedy Random
0.962 [0.958,0.967] 0.953 [0.948,0.958] 0.947 [0.942,0.952] 0.933 [0.927,0.938] 0.760 [0.755,0.764]
0.486 0.429 0.419 0.400 0.000
RQ4: portfolio selection requires an objective-matched optimizer
We generated 800 directed acyclic dependency graphs with 12 candidate blocks each and a fixed effort budget. Exhaustive enumeration identified the feasible subset with maximum total declared BVS for every instance. Four heuristics and a random baseline were then evaluated against this exact optimum. Value-to-effort achieved the highest mean objective ratio, 0.962 (bootstrap 95% CI [0.958, 0.967]), followed by ODP distance at 0.953 ([0.948, 0.958]), PVS α = 2 at 0.947 ([0.942, 0.952]), and BVS greedy at 0.933 ([0.927, 0.938]). The random baseline reached 0.760 ([0.755, 0.764]). Exactoptimum rates were 48.6%, 42.9%, 41.9%, 40.0%, and 0%, respectively. The paired mean advantage of value-to-effort over ODP distance was 0.0094 objective-ratio points with bootstrap 95% CI [0.0038, 0.0150]. Under the declared value-maximization objective, this is a small but systematic advantage. ODP distance nevertheless remains competitive and substantially outperforms random selection. The optimization implication is precise. For a path-structured graph with a declared additive distance, shortest-path optimization can be exact for that path objective. For general release selection under effort budgets and prerequisite constraints, ERS/BVS is a heuristic preference signal rather than a proof of global portfolio optimality. ODP should therefore select an optimizer that matches the decision class: shortest path for path problems, and integer, dynamic, evolutionary, or multi-objective methods when the release problem requires them.
7
Absolute log priority error
2.0 1.5 1.0 0.5 0.0
0.0
0.5 1.0 1.5 2.0 2.5 Model-collective divergence (standardized)
Figure 4: Model–collective divergence versus absolute priority error in the calibration experiment.
6
Discussion
6.1
Prioritization parameters are governance decisions
BVS weights encode strategy, ERS coefficients encode sensitivity to effort and risk, and the PVS exponent determines how aggressively value dominates feasibility. The sensitivity experiments show that these parameters can preserve broad ordering while materially changing top-portfolio membership. Parameter ownership, rationale, version history, and replayable rankings are therefore not documentation overhead; they are part of the decision evidence.
6.2
Human–AI influence should be evidence-weighted
The effort and drift experiments support a direct rule: when multiple estimators contribute to the same forecast, their influence can be updated from observed predictive performance. This replaces fixed role-based authority with measurable reliability. Human accountability and contextual constraints remain governance decisions, but the quantitative allocation of forecasting influence no longer needs to be arbitrary.
6.3
Collective divergence is a diagnostic channel
The CCS experiment shows that an independent collective signal can identify model miscalibration with strong discrimination (AUC 0.906) under the specified error structure. The useful decision is not to substitute the majority for the model, but to route high divergence to review, context refresh, or rescoring. This creates an explicit detection mechanism for stale or misaligned priority estimates.
6.4
ODP is a family of objective-matched optimization procedures
The portfolio benchmark establishes that “optimal” is meaningful only relative to an explicit objective and constraint set. Dijkstra’s algorithm remains exact for nonnegative shortest-path problems [16]; dependency-constrained release portfolios belong to a broader optimization 8
Mean normalized regret
0.25 0.20 0.15 0.10 0.05 0.00
ort
/eff e u l Va
ODP
an dist
ce
y
2
PVS
= la pha
BV
eed r g S
ne
om
d Ran
eli bas
Sequencing / selection heuristic Figure 5: Mean normalized regret for portfolio heuristics. Lower is better.
class [3, 5]. D-POAF therefore benefits from treating ODP as an optimization interface: define the objective, constraints, and evidence first, then select the algorithm that solves that problem class.
7
Scope and Validity
The claims in this paper are mechanism-level claims supported by controlled experiments with known latent variables and explicit error processes. That scope is a strength of the design: ranking sensitivity, aggregation error, drift response, diagnostic discrimination, and optimization regret can be measured against a known reference rather than inferred from uncontrolled organizational variation. Construct scope. BVS, ERS, and CCS operationalize declared decision variables; they are not asserted to be universal measurement scales for business value or organizational risk. The reported sensitivity results establish how these operators behave once the variables are instantiated. Internal validity. Estimator bias, noise, drift, and injected stale-context error are controlled by design. The reported differences therefore characterize the stated model class. Alternative error structures can be tested as additional experimental conditions without changing the evaluation method. External validity. Industrial deployments will determine field effect sizes and organizational boundary conditions. Field validation is a subsequent external-validity layer, not a prerequisite for the present computational findings: the current experiments already establish the behavior, advantages, and optimization limits of the mechanisms under explicitly defined conditions. Optimization validity. The exact portfolio benchmark maximizes declared BVS under an effort budget and prerequisite constraints. All optimality statements in this paper are relative to that objective. Different objectives – risk-adjusted value, diversity, reliability, or multi-period utility – define different optimization problems and should be benchmarked accordingly.
9
8
Conclusion
This study establishes a quantitative decision layer for governed human–AI software prioritization. Five results are supported by the experiments. First, strategic scoring parameters materially affect top-portfolio membership and therefore require explicit governance. Second, reliabilityweighted aggregation substantially reduced effort-estimation error under heterogeneous estimator quality. Third, adaptive weighting tracked estimator drift and improved predictive accuracy as relative quality changed. Fourth, model–collective divergence provided a strong diagnostic signal for priority miscalibration. Fifth, exact portfolio benchmarking showed that ODP distance is competitive but not generally optimal for budget- and dependency-constrained release selection; optimization must match the declared problem class. Together, these findings move D-POAF decision logic from descriptive practice to a testable quantitative model. The central engineering principle is evidence-weighted authority: parameters, estimator influence, calibration signals, and optimization objectives are explicit enough to measure, challenge, version, and improve. Subsequent field deployments can measure external effect sizes on real estimation histories, priority revisions, delivered value, and portfolio decisions while preserving the same experimental and governance structure established here.
Availability The accompanying verification pack contains the data summaries and figure assets used in this paper, the 800-instance raw portfolio benchmark, the LaTeX source of this revised manuscript, and an audit script that independently recomputes portfolio summary statistics and bootstrap confidence intervals. The D-POAF Canonical Specification and Official Terminology are available through their Zenodo DOIs [12, 13]; the Operating Guide documents the practitioner reference mechanisms [14].
Acknowledgements The authors thank the D-POAF community and Inovionix collaborators for their contributions to the framework and reference implementation.
References [1] B. W. Boehm, “Value-Based Software Engineering,” ACM SIGSOFT Software Engineering Notes, 28(2), 2003. doi:10.1145/638750.638776. [2] J. Karlsson and K. Ryan, “A Cost-Value Approach for Prioritizing Requirements,” IEEE Software, 14(5), 1997. doi:10.1109/52.605933. [3] A. J. Bagnall, V. J. Rayward-Smith, and I. M. Whittley, “The Next Release Problem,” Information and Software Technology, 43(14), 2001. doi:10.1016/S0950-5849(01)00194-X. [4] P. Achimugu, A. Selamat, R. Ibrahim, and M. N. Mahrin, “A Systematic Literature Review of Software Requirements Prioritization Research,” Information and Software Technology, 56(6), 2014. doi:10.1016/j.infsof.2014.02.001. [5] A. M. Pitangueira, R. S. P. Maciel, M. O. Barros, and A. S. Andrade, “Software Requirements Selection and Prioritization Using SBSE Approaches: A Systematic Review and Mapping of the Literature,” Journal of Systems and Software, 103, 2015. doi:10.1016/j.jss.2014.09.038.
10
[6] F. A. Bukhsh, Z. A. Bukhsh, and M. Daneva, “A Systematic Literature Review on Requirement Prioritization Techniques and Their Empirical Evaluation,” Computer Standards & Interfaces, 69, 2020. doi:10.1016/j.csi.2019.103389. [7] M. Jorgensen, “A Review of Studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 70(1–2), 2004. doi:10.1016/S0164-1212(02)00156-5. [8] M. Jorgensen and M. J. Shepperd, “A Systematic Review of Software Development Cost Estimation Studies,” IEEE Transactions on Software Engineering, 33(1), 2007. doi:10.1109/TSE.2007.256943. [9] J. T. H. de A. Cabral, A. L. I. Oliveira, and F. Q. B. da Silva, “Ensemble Effort Estimation: An Updated and Extended Systematic Literature Review,” Journal of Systems and Software, 195, 2023. doi:10.1016/j.jss.2022.111542. [10] G. Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld, “Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork,” AAAI, 35(13), 2021. doi:10.1609/aaai.v35i13.17359. [11] Z. Bucinca, M. B. Malaya, and K. Z. Gajos, “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making,” PACM HCI, 5(CSCW1), 2021. doi:10.1145/3449287. [12] A. Ihsine and S. Ihsine, doi:10.5281/zenodo.21922885.
D-POAF
Canonical
Specification
v1.1,
2026.
[13] A. Ihsine and S. Ihsine, doi:10.5281/zenodo.21922945.
D-POAF
Official
Terminology
v1.1,
2026.
[14] S. Ihsine and A. Ihsine, D-POAF Operating Guide, Fourth Edition, D-POAF Community and Inovionix, 2026. ISBN 979-10-988812-0-6. [15] M. Jorgensen, “Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty,” IEEE Transactions on Software Engineering, 31(11), 2005. doi:10.1109/TSE.2005.128. [16] E. W. Dijkstra, “A Note on Two Problems in Connexion with Graphs,” Numerische Mathematik, 1, 1959. doi:10.1007/BF01386390. [17] M. Shepperd and S. G. MacDonell, “Evaluating Prediction Systems in Software Project Estimation,” Information and Software Technology, 54(8), 2012. doi:10.1016/j.infsof.2011.12.008.
11