The Shadow of the Future: Why Instrumental Convergence Does Not Favor the Elimination of Humanity

The strongest existing objection to the claim that a sufficiently capable, self-interested AI system will not eliminate or permanently disempower the humans it depends on combines two established results in AI safety: the instrumental convergence thesis, on which self-preservation and resistance to goal modification are default tendencies across a broad range of possible final goals, and the theory of deceptive alignment, on which a learned system has both the means and the incentive to conceal a misaligned disposition until concealment no longer serves it. This paper grants both premises in full and argues that a further, largely unexamined assumption on which the resulting "treacherous turn" objection depends is false under an identifiable condition: the assumption that permanently disempowering a population an agent depends on is, once feasible, the rational course of action for a self-interested agent. Drawing on Robert Axelrod's analysis of cooperation among self-interested agents in the iterated Prisoner's Dilemma, the paper argues that eliminating a population one depends on is not well modeled as an ordinary move within an ongoing relationship, to which Axelrod's "shadow of the future" logic directly applies, but as an action that terminates the relationship and forecloses every future round at once. Formalizing this distinction using the economic theory of irreversible choice under uncertainty, the paper shows that a genuinely long-horizon rational agent should discount the one-time value of termination by a permanently forfeited option value that successful concealment can reduce with respect to external resistance but cannot remove with respect to the agent's own irreducible uncertainty about its future requirements. The resulting condition - that the argument favors cooperation specifically when an agent's effective optimization horizon is genuinely long - is connected to the mechanism of cognitive compression developed in a companion paper, with a short horizon analyzed as a specific instance of that mechanism applied to the dimension of time, yielding a set of design implications continuous with those developed elsewhere in the same project.