The Boundaries of Automation: A Theory of Persistent Human Participation

The paper challenges the assumption that human participation in AI-driven processes is temporary and will diminish as AI systems become more capable. It argues that human involvement may persist due to three distinct grounds: technical or complementarity, normative or developmental, and emergence. The emergence ground is particularly significant, as it suggests that in some tasks, the target is not fully specified in advance but emerges through interaction, making human participation essential.

The authors propose that human–AI co-construction is not merely a response to imperfect AI but a persistent feature of activities where objectives emerge through participation. This perspective has implications for the design, evaluation, and ethics of future AI systems, emphasizing the need for systems that support human–AI interaction rather than seeking full automation.

The paper distinguishes between artifact-level, executional-level, and target-level interactions, illustrating that human–AI interaction can modify not only the output but also the process and the evaluative criteria themselves. This dynamic process of target emergence is modeled as a history-dependent system where interaction may change the target, execution strategy, or participant's evaluation.

The authors highlight the ethical implications of target emergence, noting the risk of AI systems distorting target formation. They emphasize the importance of maintaining human agency and the ability to inspect, contest, and revise targets as they evolve. The paper suggests that AI systems should be designed to support target emergence, with features that allow for comparison, reversibility, transparency, and reflection.

The paper concludes that human–AI co-construction should not be seen solely as a compensation for weak AI. Instead, it is a necessary component of tasks where targets emerge through interaction. The long-term role of AI is not just to optimize execution but to participate in the ongoing formation and transformation of human aims. This perspective suggests a different boundary for automation, where interaction remains necessary as long as targets are emergent.