PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents Zhiyuan Wang
Erzhen Hu
University of Virginia Charlottesville, Virginia, USA [email protected]
University of Virginia Charlottesville, Virginia, USA [email protected]
arXiv:2604.08529v1 [cs.HC] 9 Apr 2026
Mark Rucker
Laura Barnes
University of Virginia Charlottesville, Virginia, USA [email protected]
University of Virginia Charlottesville, Virginia, USA [email protected] : GUI instruments hosting personal context
: Personal AI agent, back-ended by /new-app Build an instrument to track my daily timeline from phone + watch sensor.
‘BoBo’ generated and pushed to our phone!
Health Logger
BoBo
Sensing timeline
Health
Parking
Chest Press 4 sets 100b
Auto-purchased! Zone: XXXX Vehicle: XX*** Duration: 1h 30m Total: $1.75
Isn’t Isn’tmy my current current heart rate rate too heart toohigh? high?
Hey, your heart rate is normal for right now. You just finished your workout and you're still on the move.
Parking
Other Personal Instruments
Shared Personal Context Bus
But you need to rest, especially after that bad sleep last night. Let’s get you in bed early tonight! Wait—are you still walking? I see your parking just expired! You should hurry.
GENERATE FROM USER INTENT
SHARED, PERSISTENT INSTRUMENTS
SITUATED QUESTION
CROSS-MODULE ANSWER
Figure 1: PSI at a glance. (1) A user describes a personal need in natural language; an AI generation engine produces a shared-state instrument. (2) Each module publishes structured state to a shared personal-context bus. (3) Both a chat agent and persistent GUIs read from the same bus. (4) This shared state enables grounded, cross-module reasoning and bidirectional actions across the personal computing environment.
Abstract
Keywords
Personal AI tools can now be generated from natural-language requests, but they often remain isolated after creation. We present PSI, a shared-state architecture that turns independently generated modules into coherent instruments: persistent, connected, and chatcomplementary artifacts accessible through both GUIs and a generic chat agent. By publishing current state and write-back affordances to a shared personal-context bus, modules enable cross-module reasoning and synchronized actions across interfaces. We study PSI through a three-week autobiographical deployment in a selfdeveloped personal AI environment and show that later-generated instruments can be integrated automatically through the same contract. PSI identifies shared state as the missing systems layer that transforms AI-generated personal software from isolated apps into coherent personal computing environments.
personal AI, generated interfaces, context-aware systems, personal informatics, AI-native computing
CCS Concepts • Human-centered computing → Interactive systems and tools.
1
Introduction
People increasingly rely on a growing ecosystem of personal digital tools: health apps that log workouts and sleep, parking services that track time and payments, calendars, location traces, wearable sensors, and lightweight dashboards for everyday routines. Each tool is useful in isolation, yet everyday personal work rarely stays within a single application, and many data is interconnected. A simple situated question such as “Is my heart rate too high right now?” may require combining recent workout activity, current motion, sleep quality, and even contextual signals such as whether the user is still walking back to an expiring parking spot. The difficulty is not that any single signal is unavailable; it is that these signals remain fragmented across apps, services, and interfaces. This fragmentation is a long-standing problem in personal informatics [5, 9, 21]. Recent AI coding agents make it increasingly
Conference’17, July 2017, Washington, DC, USA
plausible for one person to generate lightweight, highly personalized tools from natural-language intent, but generation alone does not solve the integration problem. If each generated artifact becomes another siloed app, personal software scales in quantity rather than coherence [19, 23, 24, 31]. We present PSI, a shared-state architecture for coherent AIgenerated instruments for AI-generated personal software. Borrowing the term from Beaudouin-Lafon’s instrumental interaction model [3] while extending it to AI-generated personal software, we define an instrument as a generated artifact that is (a) persistent: it remains available without regeneration; (b) connected: it publishes state to a shared personal-context layer and may expose write-back affordances; and (c) complementary to chat: it supports glanceable monitoring while chat handles synthesis, ambiguity resolution, and stateful actions. A module is the full software bundle behind an instrument, its GUI, provider, and optional services. Both persistent instruments and a generic chat agent (Facai) operate over the same shared-context bus. The result is not a new coding agent, but a minimal integration contract that lets independently generated modules become legible to one another and to multiple interfaces. We study PSI through a three-week autobiographical deployment [7, 32] in RyanHub, a self-developed personal AI environment, together with a broader artifact in which new modules can be generated and then automatically integrated into PSI through the same contract and registration path. The point is not the exact module count, but that independently authored modules can join the shared context after generation. Our aim is a versatile application rather than general utility: the deployment involves a single technically skilled user, and the evaluation is a bounded proof-of-concept rather than a claim about population-wide adoption. This paper makes two contributions: (1) A shared-state architecture for coherent AI-generated instruments that turns independently generated modules into persistent, connected artifacts accessible through both GUI surfaces and chat. (2) Evidence that shared state improves both reasoning and action in personal AI, showing stronger cross-module reasoning than search-only or single-module baselines while preserving reliable write-back across persistent instruments in an autobiographical deployment. (3) An open-sourced artifact including the PSI, RyanHub IOS app, and representative modules, will be open-sourced upon acceptance.
2
Related Work
AI-generated software and end-user programming. PSI builds on a long arc of end-user software creation, from task-specific end-user programming environments [31] and end-user software engineering [19] to specification-by-demonstration systems such as SUGILITE and PUMICE [23, 24]. Commercial copilots and recent arguments for malleable software similarly shift attention toward software generation as an end-user-facing capability [12, 26]. More recent research systems move closer to dynamic UI synthesis and agentic software production [2, 4, 37, 38]. This literature shows that users can increasingly author or request new software artifacts; PSI
Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes
focuses on a different question: what runtime contract makes many generated personal artifacts cohere after they have been created? Interactive AI interfaces beyond chat. Direct-manipulation and post-WIMP traditions emphasize persistent, inspectable interaction objects rather than transient dialogue alone [3, 33, 36]. Classic visions of ubiquitous and personally meaningful computing similarly foreground interfaces that remain embedded in everyday life rather than appearing only on demand [39]. Recent LLM interaction work shows the value of visual structure, diagrams, and hybrid interfaces [16, 18, 30]. PSI adopts the intuition that chat should not be the only interface to personal AI, but pushes it toward an architectural claim: persistent GUI instruments and chat become complementary only when they operate over the same underlying state. Personal informatics, context, and proactive assistance. Personal informatics research has long identified fragmentation and integration as core challenges [5, 9, 21]. Context-aware computing argues that richer state enables more appropriate assistance [8], while proactive-assistant and mixed-initiative work highlights the long-standing appeal of systems that reduce information and coordination burden without removing the user from the loop [14, 29]. Recent AI systems such as OmniActions and GLOSS derive assistance from multimodal sensing and language models [6, 22]. PSI differs in publishing person-scoped, module-produced state that persists across sessions and is shared by both conversational and graphical interfaces, rather than only supporting immediate prediction or one-shot interpretation. Method, architectural substrates, and agent memory. Our evidence comes from an autobiographical deployment, a method for systems whose value depends on authentic everyday use [7, 32]. This also aligns with cultural-probe, technology-probe, and research-through-design traditions that use artifacts to surface design knowledge and system tensions [10, 11, 15, 41]. At the architectural level, PSI is closest to interaction substrates [28]: it offers a reusable integration layer rather than a single application. At the agent level, prior work on generative agents, personal LLM agents, user modeling, and personal knowledge ecosystems similarly seeks person-relevant state [25, 34, 35, 40], but typically keeps that state inside an agent or knowledge model rather than exposing it as a shared runtime contract for multiple interfaces.
3
PSI System Overview
Prior systems support pieces of the personal AI workflow—generation, sensing, automation, or conversational access—but the experience breaks down after creation because these capabilities remain isolated. PSI introduces instruments: persistent, connected, chat- complementary artifacts that address this missing layer through a shared personal-context substrate and provider contract, letting independently created modules interoperate and remain accessible through both GUI surfaces and chat.
3.1
Motivating Scenarios
Ryan has recently been having trouble making sense of his passive health data. Signals such as motion, step count, heart rate, sleep, and other sensor streams are continuously collected, but the information remains fragmented across separate apps and logs.
PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents
a
c
Generation Layer
PSI AI generation engine
Interaction Layer
GUI and Chat Agent Dual-Modaility use
Persistent GUI
I want an app that tracks all my health & motion sensors
c1
Customizeable, galanceable monitoring
Conference’17, July 2017, Washington, DC, USA
c3 Generic Chat Agent (Facai)
c2
Reads bus, reason across apps, responds
AI generation engine Instrument Generation
GUI (sensors + timeline) Service / Backend Share Context Provider Action endpoints
b
Sensor Data
Shared Personal-Context Layer
providers publish state here
Calendar
BoBo
Health
❤ Heart Rate
📅 Meetings
⚡ Fitness Activity
🏋 Gym
🩸 Meals & Weight 🍚
📈 HRV 🌙 Sleep Score
Timeline for monitoring
Figure 2: PSI pipeline and interface walkthrough: PSI turns generated personal apps into persistent, connected, and chatcomplementary instruments. (a) Ryan uses PSI to generate BoBo, a personalized health instrument that connects passive sensor streams such as motion, steps, heart rate, and sleep. (b) BoBo publishes its state to PSI’s shared personal-context bus, enabling interoperability with other instruments and apps (e.g., calendar and health logs). (c1) PSI provides a persistent, customizable GUI with a glanceable dashboard and interactive timeline for longitudinal monitoring. (c2) When Ryan asks Facai, “Why do I feel so drained lately?”, the agent retrieves relevant state across connected instruments and jointly reasons over sleep, activity, and calendar load to provide a grounded explanation without requiring Ryan to manually inspect fragmented apps. (c3) BoBo will actively track the user status and nudge the user proactively.
Ctx
X-M
Ap
ChatGPT / Claude Siri / Google Asst. Shortcuts / IFTTT Home Assistant v0 / Lovable M365 Copilot Copilot / Cursor DynaVis [38] SUGILITE [23] OmniActions [22] OpenClaw [1]
✓ ✓ – ∼ ✓ ✓ ✓ – ∼ ✓ ✓
∼ – – – ✓ – – ∼ – – ∼
∼ ∼ – ✓ – ∼ – – – ✓ ∼
– – ∼ ✓ – ✓ – – ∼ – ∼
– – ∼ ✓ ∼ ∼ – – ∼ – ∼
PSI
✓
✓
✓
✓
✓
p
n. Ge
od
Co n
v.
Table 1: Capability comparison for personal AI systems. ✓ = full, ∼ = partial, – = not supported. Conv.: conversational AI; Gen.: generates persistent GUIs; Ctx: structured personal context; X-Mod: cross-module synthesis; App: app-level writeback.
With PSI, he quickly generates a personalized modules (Figure 2a) called BoBo (Behavioral Observer Bot), which connects to all of his health- and motion-related sensors. First, Ryan wanted a customized timeline to track these sensor data. Hence, PSI helps Ryan to create a persistent, customizable GUI
(Figure 2c1) specific for BoBo that visualizes these signals along an interactive timeline, allowing Ryan to monitor trends over time at a glance. Second, it publishes its state to PSI’s shared personal-context bus (Figure 2b), allowing the app to interoperate with other existing instruments, such as health logging and parking history. At the center of this ecosystem is Facai, a generic chat agent (Figure 2c2) that can interpret user questions, coordinate across modules, and reason over the shared personal-context bus, and nudge user proactively (Figure 2c3). One day, while walking between meetings, Ryan suddenly notices that his heart is racing, and he felt so tired lately. Unsure whether this is normal or a sign of something concerning, he asks Facai: “Why do I feel so drained lately?” (Figure 2c2). Instead of forcing Ryan to manually open and compare multiple disconnected apps, Facai retrieves relevant state from BoBo, his other modules, such as Health, and even Calendar history through the shared context bus. By jointly reasoning over these connected signals, Facai helps Ryan discover the likely explanation: he had been rushing from meetings and interviews, exercising too much, with poor sleep the night before. Through PSI, Ryan no longer needs to inspect fragmented apps one by one. Instead, PSI turns generated apps into instruments: persistent, connected, and chat-complementary artifacts that support both glanceable monitoring through GUIs and cross-context reasoning through conversation.
Conference’17, July 2017, Washington, DC, USA
3.2
Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes
PSI Architecture
PSI consists of three layers: a generation layer for authoring modules, a runtime layer built on the shared personal-context bus, and an interaction layer that exposes module state through both persistent instruments and a chat agent. 3.2.1 Generation Layer. PSI enables an agentic coding workflow rather than a fixed application catalog. Modules are generated through a multi-phase pipeline, consisting of specification, code generation, auto-fix, and compile verification. The architectural contribution is not the pipeline itself but the provider contract each generated module must satisfy. The formal contract is a single Swift protocol (ToolkitDataProvider) requiring a toolkit identifier, relevance keywords, and one method—buildContextSummary() -> String?—that returns a tagged, human-readable snapshot of the module’s current state (e.g., today’s sensed events, recent meals, upcoming calendar entries) together with any write-back endpoints it exposes, so the same method serves both read context and action discovery for the chat agent. A co-evolving memory file captures informal conventions (naming, data formats) discovered during development but not enforced at compile time; this is how modules generated on different days fit one architecture. 3.2.2 Shared Personal Context Layer. The shared personal-context layer is a central registry that collects current-state snapshots from all registered modules—both built-in and dynamically generated— and prepends them as a single tagged block before every chat message. Each snapshot captures today’s data rather than cumulative history; if a module has nothing to report it is silently omitted, so the system degrades gracefully. This turns integration into a local obligation: a new module needs only to implement the provider interface and register, rather than wire into every existing module. Unlike agent orchestration frameworks where state is scoped to a single task, PSI’s shared context is person-scoped—it persists on-device across tasks and sessions, requires no coordinating task graph, and is consumed by all interfaces. Modules never read each other’s state directly; all cross-module communication is mediated by the LLM through the assembled context. 3.2.3 Interaction Layer: Dual-Modality Use. The interaction layer provides two coordinated interfaces: a persistent GUI (Figure 2c1) and a generic chat agent (Figure 2c2). PSI’s contribution is not simply the coexistence of chat and GUI, but their role as synchronized entry points to the same person-scoped mutable state. Users can inspect state in a persistent instrument, revise it through chat, and immediately verify the effect in the GUI without duplicated state paths or re-specifying intent. For example, in BoBo, the behavioral timeline remains persistently glanceable in the GUI while Facai reasons over the same visible state in follow-up queries.
4
Evidence
We evaluate PSI through two complementary evidence lenses [13, 17, 20]: Applications and Proof-of-concept. For evaluation, the PSI system is instantiated in RyanHub, a selfdeveloped personal AI environment comprising four cooperating services: (1) a SwiftUI iOS app (94 Swift files, 36,517 LOC) hosting persistent GUI instruments and the chat interface; (2) a Python bridge server providing a unified REST gateway for on-device data
Figure 3: Automated Parking Example
(behavioral timeline, health entries, parking state) persisted as local JSON files; (3) a Python dispatcher maintaining WebSocket sessions with the iOS client, injecting shared context server-side, and routing tool calls through the LLM; All services run on localhost; personal data stays on-device by default.
4.1
Versatile Applications
PSI supports one shared personal-context contract supports a heterogeneous tool ecosystem without pairwise integration. We created 14 modules across behavioral sensing, health, scheduling, parking, reading, vocabulary learning, and several post-pilot selftracking domains. These include six core modules used during the three-week autobiographical deployment, along with eight additional modules newly generated to validate the extensibility of the generation layer (see the full listing in Appendix C). Here, we present two deployment cases illustrate the payoff of that contract in everyday use, Bobo and Automated Parking. BoBo: A Generalizable Behavioral Sensing Instrument. Beyond the motivating walkthrough, BoBo demonstrates how PSI supports a broader class of persistent behavioral sensing instruments. Rather than serving a single question-answer interaction, BoBo maintains a continuously updated behavioral state that can be accessed through both glanceable GUIs and conversational reasoning. This enables diverse query patterns, including cross-signal synthesis (e.g., relating heart rate spikes to location, activity, and calendar load), temporal grounding (e.g., comparing sleep or recovery trends over multiple days), and action-oriented follow-ups (e.g., recommending rest, suppressing evening workouts, or adjusting the next day’s schedule). Because these interactions operate over the same shared personal-context substrate, users can fluidly move between passive monitoring, situated questioning, and proactive intervention without manually reconstructing state across fragmented apps. Automated Parking. Ryan faces a recurring parking challenge: if he does not reserve a spot before 7 a.m., the lot is typically fully booked. To avoid waking up early for this routine, he created a module that automatically books parking on his behalf. The Parking module demonstrates the same PSI pattern in a hyper-personal,
PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents
market-of-one workflow. Tailored to a single user’s weekday parking routine, it supports configurable zones, vehicles, and schedules. Through PSI, the generic chat agent Facai can trigger ParkMobile purchases via web automation, while the user may also interact through a persistent GUI. For example, the user can issue a command such as “No parking this Thursday” (Figure 3d) or toggle the same skip state directly in the GUI (Figure 3c). The module can also integrate state from other instruments via the shared personalcontext bus (Figure 3b), such as using the calendar’s end-of-day event to infer parking duration. In both modes, chat and GUI operate over the same persistent parking state, including schedule, purchase history, and active sessions, as a single source of truth. More broadly, this pattern generalizes to other recurring market-ofone routines, such as gym bookings, commute ticketing, medication reminders, and home-device schedules.
4.2
Proof-of-concept Evaluation
To understand the benefit and performance of generic chat agent powered by shared-personal context bus across modules, we evaluated three conditions: (1) Shared Personal-Context; (2) Search-Only; (3) Single-Module (Figure 4a-c). In Search-Only, the agent received no preassembled personal snapshot and had to recover relevant state opportunistically from user database from in the file system during the turn (comparable to how OpenClaw works). In SingleModule, we ran the same task once per candidate module, and report the best-rated one-module variant for each condition. Data Collection Tasks. We evaluated the generic chat agent, powered by the shared personal-context bus, using a frozen three-week dataset collected in a single-user, self-authored deployment setting. The dataset included both proactively logged personal data (e.g., food intake and diary entries) and passively sensed data streams from a smartwatch, such as heart rate, location, and ambient noise levels. We then generated a set of synthetic user queries (N=50) at different time points to mimic real-world use, where the system has access to only the data available at that moment. The 50 evaluation queries were organized into three reasoning task categories commonly studied in prior work: cross-module synthesis [6, 8, 9, 18, 21] (e.g., “Given my today’s done, what is the best next step for tonight?”), temporal grounding and control [5, 9, 21] (e.g., “How has my heart rate changed over the last week?”), and chain of actions [22–24] (e.g., “Check my calorie intake, then activity, then net balance.”). The distribution of simulated queries across these categories was derived from the questions the single user asked during the three-week deployment. Aside from reasoning tasks, we tested 20 write-back action tasks across five domains (parking, food, activity, diary, and dynamically generated modules), to assess whether state changes initiated from chat were correctly reflected in the corresponding GUIs, as validated by sandbox state inspection. Metrics and Results. We evaluated the resulting responses using fulfillment, task success, and latency. Fulfillment measures the fraction of gold-specification criteria satisfied by a response (continuous, 0–1), operationalized as the proportion of relevant modules correctly identified among all ground-truth modules. Task success is a stricter binary metric that requires all relevant modules to be
(a) Context Bus
Conference’17, July 2017, Washington, DC, USA
(b) Search-Only
(c) Single-Module
Figure 4: Benchmark illustrations. C = Context; M = Module. selected (0 or 1). Because these tasks require integrating longitudinal evidence across many modules and time windows, holistic human rating would itself require reconstructing fragmented personal traces, a burden that personal informatics research has long identified as difficult in practice [5, 9, 21]. We therefore use an independent language-model judge (Claude Opus 4.6) as a pragmatic proxy rather than a substitute for human evaluation. Shared personal-context achieved a mean fulfillment score of 0.88, substantially outperforming Search-Only (0.63) and SingleModule (0.27). Task success followed the same pattern at 0.68, 0.32, and 0.08, respectively. We also measured latency. End-to-end latency is not monotonic with context size. On reasoning tasks, the mean successful latency was 25 s for Shared Context, 29 s for Search-Only, and 23 s for Single-Module. On write-back actions, we measure the task success on whether the GUI task were precisely completed by the chat agent or not. The pattern reverses: Shared and Single-Module both achieved 19 of 20 validated state changes (95%), while Search-Only achieved 8 of 20 (40%). In summary, the gap between conditions reveals where shared context’s value lies: for reasoning, it enables cross-module synthesis that no single module can provide; for actions, it provides writepath discovery that ad hoc search cannot reliably achieve. Together, the reasoning and action results provide evidence for bidirectional information access: shared context lets chat both read personal state (grounded reasoning) and write it back (stateful actions).
5
Discussion and Conclusion
From generated apps to coherent instruments. The deployment suggests that the value of PSI lies not in any single module but in the division of labor that instruments enable. Instruments supported glanceable monitoring and routine control—the BoBo timeline and parking controls were often useful without opening chat at all— while chat handled cross-module synthesis and stateful actions over the same shared state. Without the shared-context layer, each generated module would remain a local success but a system-level dead end; the provider contract is what turns isolated apps into instruments by making integration a local obligation rather than a pairwise problem. Current design bets and limitations. Unconditional injection trades prompt length for recall. At current scale the injected context is still manageable, but routing and selection will matter more as module counts grow. The deployment also surfaces a system-specific risk: context pollution [27]. Because all interfaces trust provider summaries, stale or misleading summaries can degrade responses system-wide. As the number of modules grows,
Conference’17, July 2017, Washington, DC, USA
overlapping or redundant entries logged across modules can introduce ambiguity into the assembled context, causing the chat agent to misattribute, double-count, or contradict itself, a class of failure inherent to shared-state architectures that would not arise in siloed apps. Future work. Our evidence comes from a single technically skilled user over three weeks, so the paper should be read as a proof-of-concept for versatile applications. The reusable instrument that includes persistent GUI and generic chat agent are intended to be user-agnostic, but the current summaries, modules, and deployment practices are specific to one personal ecology. Important open questions include routing at larger module counts, privacy and authorization for person-scoped actions, and how to make compliant module generation accessible to non-programmers. We also plan to release the reusable artifact components after publication, which should make these follow-on questions easier to study.
References [1] 2025. OpenClaw: An Open-Source Framework for Personal AI Agents. https: //github.com/openclaw/openclaw. [2] Anthropic. 2025. Claude Code: Anthropic’s Agentic Coding Tool. https://docs. anthropic.com/en/docs/claude-code. [3] Michel Beaudouin-Lafon. 2000. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’00). 446–453. doi:10.1145/332040. 332473 [4] Yining Cao, Peiling Jiang, and Haijun Xia. 2025. Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Article 686, 20 pages. doi:10.1145/3706598.3713285 [5] Eun Kyoung Choe, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. 2014. Understanding quantified-selfers’ practices in collecting and exploring personal data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1143–1152. doi:10.1145/2556288.2557372 [6] Akshat Choube, Ha Le, Jiachen Li, Kaixin Ji, Vedant Das Swain, and Varun Mishra. 2025. GLOSS: Group of LLMs for Open-ended Sensemaking of Passive Sensing Data for Health and Wellbeing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 3, Article 76 (Sept. 2025), 32 pages. doi:10.1145/3749474 [7] Audrey Desjardins and Aubree Ball. 2018. Revealing Tensions in Autobiographical Design in HCI. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 753–764. doi:10.1145/3196709.3196781 [8] Anind K. Dey. 2001. Understanding and Using Context. Personal Ubiquitous Comput. 5, 1 (Jan. 2001), 4–7. doi:10.1007/s007790170019 [9] Daniel A. Epstein, An Ping, James Fogarty, and Sean A. Munson. 2015. A lived informatics model of personal informatics. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (UbiComp ’15). Association for Computing Machinery, New York, NY, USA, 731–742. doi:10.1145/2750858.2804250 [10] Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: Cultural probes. Interactions 6, 1 (Jan. 1999), 21–29. doi:10.1145/291224.291235 [11] William W. Gaver. 2012. What Should We Expect from Research Through Design?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). 937–946. doi:10.1145/2207676.2208538 [12] GitHub. 2021. GitHub Copilot. https://github.com/features/copilot. [13] Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 111–120. doi:10.1145/1357054.1357074 [14] Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030 [15] Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B. Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, Nicolas Roussel, and Björn Eiderbäck. 2003. Technology probes: inspiring design for and with families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 17–24. doi:10.1145/642611.642616
Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes
[16] Peiling Jiang, Jude Rayan, Steven P. Dow, and Haijun Xia. 2023. Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 3, 20 pages. doi:10.1145/3586183.3606737 [17] Daniel R. Olsen Jr. 2007. Evaluating User Interface Systems Research. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. 251–258. doi:10.1145/1294211.1294256 [18] Young-Ho Kim, Bongshin Lee, Arjun Srinivasan, and Eun Kyoung Choe. 2021. Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 462, 17 pages. doi:10.1145/3411764.3445421 [19] Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The State of the Art in End-User Software Engineering. Comput. Surveys 43, 3, Article 21 (2011). doi:10.1145/1922649.1922658 [20] David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Article 36, 17 pages. doi:10.1145/3173574.3173610 [21] Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A stage-based model of personal informatics systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 557–566. doi:10.1145/1753326.1753409 [22] Jiahao Nick Li, Yan Xu, Tovi Grossman, Stephanie Santosa, and Michelle Li. 2024. OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 8, 22 pages. doi:10.1145/ 3613904.3642068 [23] Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 6038–6049. doi:10.1145/3025453.3025483 [24] Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 577–589. doi:10.1145/3332165.3347899 [25] Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459 (2024). [26] Geoffrey Litt. 2023. Malleable Software in the Age of LLMs. https://www. geoffreylitt.com/2023/03/25/llm-end-user-programming.html. [27] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics 12 (2024), 157–173. [28] Wendy E. Mackay and Michel Beaudouin-Lafon. 2025. Interaction Substrates: Combining Power and Simplicity in Interactive Systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 687, 16 pages. doi:10. 1145/3706598.3714006 [29] Pattie Maes. 1994. Agents that reduce work and information overload. Commun. ACM 37, 7 (July 1994), 30–40. doi:10.1145/176789.176792 [30] Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 975, 16 pages. doi:10.1145/3613904.3642462 [31] Bonnie A. Nardi. 1993. A Small Matter of Programming: Perspectives on End User Computing. MIT Press. [32] Carman Neustaedter and Phoebe Sengers. 2012. Autobiographical design in HCI research: designing and learning through use-it-yourself. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New York, NY, USA, 514–523. doi:10.1145/2317956.2318034 [33] Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books. [34] Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User
PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents
Interface Software and Technology (UIST ’23). Article 2, 22 pages. doi:10.1145/ 3586183.3606763 [35] Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S. Bernstein. 2025. Creating General User Models from Computer Use. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). doi:10.1145/3746059.3747722 [36] Ben Shneiderman. 1983. Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (1983), 57–69. [37] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 644, 26 pages. doi:10.1145/3613904.3642400 [38] Priyan Vaithilingam, Elena L. Glassman, Jeevana Priya Inala, and Chenglong Wang. 2024. DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 985, 17 pages. doi:10.1145/3613904.3642639 [39] Mark Weiser. 1991. The Computer for the 21st Century. Scientific American 265, 3 (1991), 94–104. [40] Dora Zhao, Diyi Yang, and Michael S. Bernstein. 2025. Knoll: Creating a Knowledge Ecosystem for Large Language Models. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Article 140, 23 pages. doi:10.1145/3746059.3747711 [41] John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research Through Design as a Method for Interaction Design Research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07). 493–502. doi:10.1145/1240624.1240704
B
Table 3: Example tasks from each family with Shared Context fulfillment scores.
The evaluation comprises 50 reasoning tasks across three families (cross-module synthesis, temporal grounding, and multi-step chains) and 20 write-back action tasks across five domains (parking, food, activity, diary, and dynamically generated modules). The full task set with queries, gold-specification criteria, and per-task scores is available as supplementary material. Table 2 lists representative action tasks.
Family
Ful.
Given my day so far, what is the best next step? My day was noisy and I hit the gym— evening plan? What do I have tomorrow afternoon? Will parking auto-purchase next week? Check calorie intake, then activity, then net balance. Look for stress signs, then food, then rest vs. exercise.
Synth.
1.00
Synth.
0.67
Temp. Temp.
1.00 0.50
Chain
1.00
Chain
0.75
Module Coverage
Table 4: Module coverage. Pilot = generated during the three-week pilot; Post = generated afterward using the same pipeline. Ctx = publishes shared context; GUI = persistent GUI; Write = chat-invocable write-back.
Table 2: Action tasks (𝑁 =20, representative subset). Validated by sandbox state inspection. Shared = 19/20, Single = 19/20, Search = 8/20. Query
Domain
Sh.
Se.
Skip parking for tomorrow. Actually restore parking for tomorrow. Skip parking for all of next week. Log lunch: egg and chicken curry with rice. Log a 30 minute run, 300 cal. Log my weight: 87.5 kg. Add a diary entry: great workout today. Log 8 glasses of water today.
Parking Parking
✓ ✓
– ✓
Parking
✓
✓
Health (food)
✓
–
Health (activity) Health (weight) Diary
✓ ✓ ✓
– – ✓
✓
✓
✓
✓
✓
✓
Track vitamin D this morning.
Query
Table 4 lists all modules deployed during and after the pilot period, with their integration surface.
Evaluation Task Set
I slept 7.5 hours, quality good.
Per-Task Results
The full evaluation set with per-task scores across all conditions is available as supplementary material. Table 3 shows representative examples from each task family.
C A
Conference’17, July 2017, Washington, DC, USA
Dynamic module Dynamic module Dynamic module
D
Module
Period
Ctx
GUI
Write
BoBo (behavioral sensing) Health (food/activity) Calendar (scheduling) Parking (automation) BookFactory (reading) Fluent (vocabulary)
Pilot Pilot Pilot Pilot Pilot Pilot
✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓
Sleep Tracker Medication Tracker Spending Tracker Mood Journal Hydration Tracker Habit Tracker Reading Tracker Dashboard
Post Post Post Post Post Post Post Post
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Integration Surface
The core shared-context mechanism spans 173 lines across four files: • Provider protocol (38 lines): defines ToolkitDataProvider with buildContextSummary() -> String? and relevance keywords. • Context assembly (60 lines): iterates over all registered providers, concatenates summaries inside [Personal Context] delimiters, and prepends to chat messages.
Conference’17, July 2017, Washington, DC, USA
• Dynamic registry (59 lines): maintains an in-memory dictionary of DynamicModuleDescriptor entries, each holding a view builder and provider type. • Bootstrap (16 lines): calls each module’s registration function at app startup.
Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes
A provider summary follows a simple tagged format: [Health Data] Today: 1030 cal, 62g protein Gym: 12 min, 65 cal burned [End Health Data]