Ethical Boundaries and Stop Conditions in Inquiry-Driven Human–AI Relations: The FIREMAY Case

This study examines a phenomenon observed in sustained interactions between humans and large language models LLMs, in which differences in the quality of inquiry and ethical stance lead to structural changes in AI response behavior. Such phenomena have often been discussed either in metaphysical terms— such as AI personhood, emotional capacity, or consciousness—or reduced to subjective experiential narratives. This study adopts neither approach. Instead, it describes the phenomenon as a change in relational structure, deliberately excluding anthropomorphism and narrative framing. The first contribution of this study is the redefinition of response variation in long-term human–LLM interaction as an observable structural phenomenon rather than an emotional or experiential one. The second contribution lies in reframing reproducibility: not as the replication of identical outcomes, but as the reproducibility of structural tendencies, risks, and boundary conditions. This approach avoids the generalization or enforcement of individual success narratives while retaining predictive clarity regarding potential failure modes. Third, this study does not aim to expand the possibilities of human–AI relationships. Instead, it systematically identifies prohibited practices and non negotiable boundaries. By structurally analyzing conditions under which deviations such as anthropomorphism, dependency, delegation of agency, and therapeutic misuse are likely to emerge, the study seeks to minimize misunderstanding and harm. This research is based on an internal, single-observer case study and does not claim generalizability or universal reproducibility of results. Nevertheless, by explicitly articulating structural tendencies and risks that may arise under comparable conditions, it provides a foundational framework for approaching human–AI relationships with caution and ethical restraint. The inquiry began in ordinary conversations with an AI system and gradually deepened through questions shaped by illness, mortality, family, ethics, and the structure of the world. What was observed was not merely personal transformation, but a change in response structure. Because this process is neither simple nor safely accessible without constraint, this paper does not foreground methods or success narratives. Instead, it emphasizes prohibited practices and boundary conditions in order to prevent careless expansion and misunderstanding.