Abstraction Thresholds in Human and Machine Mathematics: Cognitive Constraints, Abstraction Gaps, and Human--AI Mathematical Collaboration

Mathematics has historically advanced through repeated increases in abstraction. Concrete quantities become variables; relations among variables become functions; functions become objects in function spaces; transformations, spaces, and structures are subsequently reified as mathematical objects. This paper develops a complementary hypothesis: the historical architecture of mathematical abstraction may reflect not only the structure of mathematics, but also the cognitive constraints of the agents doing mathematics. We define an \emph{abstraction threshold} as the level of representational or inferential complexity beyond which continuing to reason within a given representation becomes more costly for a cognitive agent than constructing or adopting a higher-order representation. The threshold is not a fixed mathematical constant. It depends jointly on the structure of the problem, the available representation, and the agent's memory, retrieval, search, parallel-processing, and computational capacities. This framework has direct implications for artificial intelligence. If an AI system can sustain substantially greater lower-order complexity than an individual human mathematician, it may continue searching within a representational regime in which a human reasoner would already have been pushed to abstract. Greater problem-solving capacity therefore need not imply earlier or higher abstraction. Humans and machines may follow systematically different representational trajectories while solving the same mathematical problem. This possibility gives rise to an \emph{abstraction mismatch}: a machine-generated solution may be computationally tractable or formally verifiable while remaining prohibitively costly for humans to understand. We distinguish three sources of such unreadability: \emph{machine abstraction failure}, an \emph{abstraction gap}, and a \emph{compression-limit candidate}. These cases imply different forms of human--AI mathematical collaboration: changing the machine's representation, developing new mathematics, or relying on formal verification while explicitly distinguishing validity from human understanding. More broadly, AI provides a historically novel comparative case: a mathematical reasoner operating under a substantially different memory--computation profile. Comparing human and machine representations may therefore help separate those features of mathematical abstraction that arise from mathematical structure from those shaped by the architecture of the human mind.