Conceptio › Archive › arXiv CS

arXiv CSopen access

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

databases datamanagement sql storage

databases, sql, data management, storage

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

LG Research Teams 1

arXiv:2604.16813v1 [cs.AI] 18 Apr 2026

Abstract

PersonalHomeBench

Agentic AI systems are rapidly advancing toward real-world applications, yet their readiness in complex and personalized environments remains insufficiently characterized. To address this gap, we introduce PersonalHomeBench, a benchmark for evaluating foundation models as agentic assistants in personalized smart home environments. The benchmark is constructed through an iterative process that progressively builds rich household states, which are then used to generate personalized, context-dependent tasks. To support realistic agent–environment interaction, we provide PersonalHomeTools, a comprehensive toolbox enabling household information retrieval, appliance control, and situational understanding. PersonalHomeBench evaluates both reactive and proactive agentic abilities under unimodal and multimodal observations. Thorough experimentation reveals a systematic performance reduction as task complexity increases, with pronounced failures in counterfactual reasoning and under partial observability, where effective tool-based information gathering is required. These results position PersonalHomeBench as a rigorous evaluation platform for analyzing the robustness and limitations of personalized agentic reasoning and planning.

Household Generation

Inputs

Personas

Appliances Memories Contextual Contextual Description Memories

Video Transcript

Personalized Task Generation

Information Counterfactual Grounding Reasoning Reactive Tasks

Multiple Choice

Feature Plan Recommendation Generation Proactive Tasks

Figure 1. Overview of the PersonalHomeBench data generation pipeline. Detailed household environments are first assembled from personas, devices, memories, and contextual descriptions, with optional video grounding. Five categories of personalized tasks are then generated from these environments, spanning reactive question answering and proactive assistance, enabling evaluation of agent behavior in personalized smart home settings.

evolving individualized preferences, long term context, and partial observability, yielding an incomplete assessment of readiness for personalized real world deployment. Smart home environments provide a natural and challenging testbed for exposing these limitations. Effective operation in this domain requires reasoning over heterogeneous devices, maintaining persistent and temporally grounded context, and adapting to evolving user preferences while accounting for safety, efficiency, and operational constraints. Prior smart home benchmarks have largely relied on simplified abstractions, such as text only environments with limited device coverage and primitive actions, or narrowly scoped tasks focused on isolated device control (Li et al., 2025; Bartkowiak & Podstawski, 2025). Viewing smart homes as a realistic setting for personalized agentic intelligence, we introduce PersonalHomeBench, a framework for evaluating foundation models in household environments that demand contextual reasoning, personalization, and proactive decision making beyond reactive task execution.

1. Introduction Agentic AI systems aim to move beyond passive prediction toward autonomous assistance in everyday environments, where robust performance depends on contextual reasoning, uncertainty handling, and personalization. Although a growing body of benchmarks have been proposed to evaluate agentic systems, most existing evaluations focus on generic tasks in simplified or simulated settings (Chang et al., 2024b; Bonatti et al., 2025; Xi et al., 2025). As a result, they offer limited insight into model behavior under 1

Toronto and Korea Labs. Author contributions are listed in the Appendix. Correspondence to: Nikhil Verma and InJung Yang <[email protected], [email protected]>. Preprint. April 21, 2026.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

PersonalHomeBench adopts a staged environment construction process that reflects real household evolution, sequentially instantiating resident personas, assigning device ecosystems, and accumulating contextual signals over time to synthesize coherent household worlds, with optional multimodal grounding. As illustrated in Fig. 1, five categories of personalized tasks are derived from these environments to support unified evaluation of reactive reasoning and proactive planning, using a procedure inspired by (Singh et al., 2024). These include three reactive question answering tasks, I NFORMATION G ROUNDING, C OUNTERFACTUAL R EASONING, and M ULTIPLE C HOICE, and two proactive tasks, F EATURE R ECOMMENDATION and P LAN G ENERA TION , which target appliance- and routine-level orchestration for timely, preference-aware home assistance.

Table 1. Comparison of benchmarks along five dimensions: HOME TYPE, personalization (PERS.), proactiveness (PROACT.), agentic interaction with tools (AGENTIC), and multimodal support (MM). SIM = simulated homes, REAL = real-home data, and NONE = non–home-based settings. * marks settings where multimodal content is represented as text for inference.

To support faithful evaluation of agentic behavior, we introduce PersonalHomeTools, a smart home toolbox that provides a grounded interaction layer between agents and realistic household environments. It exposes structured household information retrieval, persistent memory and event access, environmental perception, and realistic appliance control through domain-specific APIs. Together, these capabilities enable systematic study of agents that unify perception, memory, and tool use for personalized decision making.

2. Related Works

M ETHOD E MBODIED B ENCHMARKS T OYOTA S MARTHOME P ERSONALT RAVEL P LANNER H OME B ENCH E DGE W ISE P ERSONA C ONTEXTAGENT P ERSONAL H OME B ENCH (O URS )

H OME -T YPE

P ERS .

P ROACT.

AGENTIC

SIM REAL NONE SIM SIM NONE

✗ ✗ ✓ ✗ ✓ ✓

✗ ✗ ✗ ✗ ✓ ✓

✓ ✗ ✓ ✓ ✓ ✓

✓ ✓ ✗ ✗ ✗ ✓*

SIM+REAL

✓

As summarized in Table 1, embodied household benchmarks (Puig et al., 2018; Shridhar et al., 2019; Li et al., 2024; Chang et al., 2024a; Yang et al., 2025c) focus on navigation and atomic action execution in simulated 3D environments, but do not model real smart-device ecosystems or personalization. Toyota Smarthome (Das et al., 2019) offers real-world home video data, but is limited to narrow demographics and fixed environments, and focuses on activity recognition rather than personalized, proactive, or interactive agentic assistance.

We evaluate a wide spectrum of foundation models, spanning compact four billion parameter models suitable for privacy-preserving edge deployment to cloud-based largescale models. Beyond task-specific metrics, we employ a Role-Playing LLM (Wang et al., 2024) as a Judge to estimate projected satisfaction for individual household members. Across model scales, performance degrades sharply on tasks involving tool use, long-horizon planning, and advanced reasoning, often accompanied by increased hallucination. These findings reveal fundamental limitations of current systems and highlight the difficulty of achieving robust, personalized, and proactive behavior in realistic smart home settings, establishing PersonalHomeBench as a practical resource for diagnosing failures and guiding the development of more reliable personalized agents.

Recent benchmarks incorporate aspects of tool usage and personalization. PersonalTravelPlanner (Singh et al., 2024) studies personalized planning with simple tools outside the home domain. HomeBench (Li et al., 2025) evaluates reactive smart-home command validation with limited appliance coverage and restricted operations, while EdgeWisePersona (Bartkowiak & Podstawski, 2025) focuses on profile reconstruction and lightweight routine inference over a small device set, with only minimal personalization via short persona descriptions. ContextAgentBench (Yang et al., 2025b) introduces proactive agents with contextual information and tools, but relies on ego-centric traces, generic APIs, and language-only inference with limited personalization.

Our contributions are threefold:

PersonalHomeBench focuses on agent behavior in realistic home settings where personalization, long-term context, perception, and smart-device interaction jointly influence decisions. By integrating with PersonaHomeTools, it evaluates whether agents can retrieve relevant household information, reason over persistent state, and coordinate appliance actions to provide proactive, personalized assistance. As summarized in Table 2, no existing benchmark simultaneously covers real home data, personalization, proactiveness, domain-specific tools, and multimodality, making PersonalHomeBench a unified setting for studying personalized agentic behavior in smart home domain. Additional discussion of related benchmarks is provided in Appendix B.

• We introduce PersonalHomeBench, a benchmark for studying personalized and proactive agent behavior in contextually grounded smart home environments. • We present PersonalHomeTools, a comprehensive smart home toolkit for realistic agent evaluation. • We conduct systematic evaluation across model scales and reasoning levels, and leverage a Role Playing LLMas-a-Judge for assessing personalized satisfaction. 2

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

3. PersonalHomeBench

hop reasoning over numerous entities and attributes. IG measures generative factual grounding, while MC evaluates the same capability under constrained answer choices. CF extends IG by introducing paired counterfactual variants, one that alters the ground-truth answer and one that leaves it unchanged, enabling targeted analysis of robustness to hypothetical perturbations. For instance, modifying temporal context may change a resident’s age, whereas altering an unrelated attribute such as attire should not.

An overview of the PersonalHomeBench generation pipeline is shown in Figure 1. The individual components are detailed in the subsequent sections. 3.1. Household Generation As illustrated in the Household Generation module of Fig. 1, generation begins from a minimal scaffold specifying household type, layout, and coarse demographics, and is seeded with personas from Nemotron Personas (Meyer & Corneil, 2025) to instantiate realistic occupants. These personas, together with household metadata, are used by a data generation model to produce complete occupant profiles that conform to a predefined schema (Appendix G). Household context is then progressively enriched by assigning appliances appropriate to the environment, synthesizing historical memories that capture both device usage and personal events, and updating occupant attributes and appliance states to reflect the current situation. Finally, a compact cache of situation-relevant memories is constructed, providing a coherent snapshot of the household that serves as the foundation for personalized task generation.

Proactive tasks evaluate open-ended, subjective decision making for personalized assistance, testing whether agents can anticipate user needs and initiate appropriate actions without explicit prompts. We define two such tasks, F EA TURE R ECOMMENDATION (FR) and P LAN G ENERATION (PG), centered on appliance and routine orchestration. In FR, the agent ranks appliance features by contextual relevance using provided API-level inventories, reflecting realworld feature discovery and configuration. In PG setting, agent generates structured multi-step plans that coordinate heterogeneous devices toward a shared goal. For example, safety-critical situations can prompt a coordinated response that initiates first-aid assistance, notifies caregivers, and shuts down hazardous appliances (Fig. 2).

For the multimodal setting, we further ground household context using exocentric videos captured by cameras placed across rooms. We produce structured video transcripts containing entity references, object mentions, time-stamped events, and aligned audio captions. To preserve fidelity to visual evidence while supporting reasoning beyond direct observation, we maintain separate representations for videogrounded and inferred information. The detailed subject instructions and the video transcript schema are provided in Appendices C.2 and G, respectively.

3.3. Dataset Statistics Household Data. PersonalHomeBench includes 1,100 households generated using the procedure in Sec. 3.1, each adhering to the schema in Appendix G, and comprising over 2,000 unique household member profiles. The households exhibit broad demographic and structural diversity, with balanced age and gender distributions (Fig. 3a). Most households do not own pets, while pet ownership is most popular among families (Fig. 3b). Every household contains at least five mandatory appliances, and the majority include 9–10 appliances (Appendix Fig. 7b). Household members are further annotated with rich persona attributes, including hobbies, lifestyles, and preferences, supporting fine-grained personalization; frequently occurring attributes are summarized in Fig. 3d. Furthermore, we ground 100 households in real-home videos spanning Health, Safety, and Daily Care scenarios, with the video length distribution shown in Fig. 3c. Additional analyses appear in Appendix C.

All generated data undergo manual review to ensure realism, internal consistency, and schema compliance, yielding high-quality household environments used for downstream personalized task generation. 3.2. Personalized Task Generation Using the constructed household environments and associated videos and transcripts when available, we derive five categories of personalized tasks that probe complementary aspects of reasoning and planning in smart home settings. Task generation assumes full observability, providing models with the entire household context.

Personalized Task Data. PersonalHomeBench contains a total of 9,168 task instances, comprising 6,000 text-only examples and 3,168 multimodal examples. The text-only split contains 1,000 instances each for IG, MC, FR, and PG, and 2,000 instances for CF. For multimodal grounding, the dataset contains 742 instances each of IG and MC, 1,484 CF instances, and 100 instances each of FR and PG. Reactive tasks (IG, CF, MC) are further annotated with three difficulty levels (easy, medium, hard). Compared to existing benchmarks on personalized and proactive agentic systems

Reactive tasks are framed as closed-form question answering and include I NFORMATION G ROUNDING (IG), C OUN TERFACTUAL R EASONING (CF), and M ULTIPLE C HOICE (MC). These tasks assess the ability of an agent to extract and integrate household information across varying levels of difficulty, ranging from direct factual lookup to multi-

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Figure 2. An agent interprets real-world household situations and triggers, using PersonalHomeTools to generate proactive plans (e.g., safety, first aid, ambience control), which are evaluated by the Role Playing Judge to produce personalized recommendations.

(Xie et al., 2024a; Singh et al., 2024; Yang et al., 2025b; Bartkowiak & Podstawski, 2025), PersonalHomeBench offers broader coverage in scale, task diversity, and supported input modalities, enabling a more comprehensive evaluation of agentic behavior in realistic settings.

lying state changes. Core agentic components consist of three specialized agents: a Transcriber, a Memory Retriever, and a Video Understanding Agent. The Transcriber incrementally constructs and maintains the household state by extracting and updating memories, events, and contextual attributes from observations. The Memory Retriever can surface situation-aware memories tied to the overall household and to specific occupants as needed. The Video Understanding Agent retrieves relevant clips, frames, or audio cues when multimodal grounding is available, enabling visual and auditory context to be incorporated into inference.

4. PersonalHomeTools As depicted in Fig. 2, PersonalHomeTools serves as a unified interaction layer that enables agents to perceive, reason about, and act within household environments, constituting a core component of our framework. The toolbox provides capabilities for household information retrieval, appliance control, and contextual understanding.

Together, these components provide a coherent interface through which agents gather context, reason over persistent memory, and execute personalized actions, enabling rigorous evaluation of tool-mediated reasoning and proactive assistance. A complete list of supported appliances, features, tools, and agent interfaces is provided in Appendix J.

Household information retrieval provides access to relatively static metadata about occupants, pets, appliances, and other household entities. Since this information changes infrequently, these functions are implemented as non-agentic tools that return deterministic structured outputs.

5. Experimental Setup

Appliance control tools enable agents to influence the state of smart devices within a household. We construct a comprehensive control sandbox by implementing feature-level interfaces for more than 40 smart appliances, derived from public documentation of major smart home providers1 . Appliance interactions may be deterministic, such as power control, or mediated by appliance-specific agents, such as a personalized recipe recommender for a refrigerator.

This section presents the evaluation protocol for assessing foundation models on PersonalHomeBench, encompassing the evaluated models, experimental settings, and metrics. 5.1. Models and Hyperparameters Models. We benchmark eleven foundation models spanning a wide range of size tiers, architectures, and reasoning configurations from the Q WEN 3 (Yang et al., 2025a), GPT, N EMOTRON, and G EMINI model families. The evaluated models are grouped into three categories:

The majority of agentic modules are designed to support deep contextual understanding by enabling hybrid reasoning via natural-language queries with optional attributebased filtering, while reusing cached results until the under1

(1) Large proprietary models: GPT-4 (Achiam et al., 2023), GPT-4 O (Hurst et al., 2024), and G EMINI 2.5 P RO (Co-

LG Electronics and Samsung.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

(a) Demographics

(b) Pets

(d) Attributes of Personas

Figure 3. Statistics of generated household and persona diversity in PersonalHomeBench. (a) Age and gender distribution of household members. (b) Distribution of the number of pets per household type. (c) Distribution of video lengths (sec). (d) Word cloud of common persona attributes, illustrating diversity in interests, habits, and lifestyle characteristics.

manici et al., 2025), (2) Medium-size open models: Q WEN 330B-I NSTRUCT, Q WEN 3-30B-T HINKING, N EMOTRON 3NANO -30B-A3B (Blakeman et al., 2025), and GPT-OSS20B (Agarwal et al., 2025). (3) Small-size open models: Q WEN 3-4B and Q WEN 3-4B-T HINKING.

G EMINI 2.5 P RO as a semantic equivalence judge to determine whether the predicted answer is meaningfully consistent with the ground truth. This hybrid approach is necessary because valid responses may vary in surface form, ranging from short phrases to full sentences, while expressing the same underlying meaning. For CF, Table 2 reports the macro-average over CFunchanged and CFchanged .

For models that support explicit reasoning modes, we evaluate multiple reasoning configurations. In particular, GPTOSS-20B is tested at three reasoning levels (low, medium, high). For clarity, we append a single-letter tag to model names to denote reasoning mode, e.g., Q WEN 3-4B [T], Q WEN 3-30B [T], and GPT-OSS-20B [L/M/H].

For MC, accuracy is computed as an exact match between the selected option (or its text) and the ground truth choice. Since all valid answers are explicitly enumerated, we do not apply semantic judging for this task. Proactive Tasks. The F EATURE R ECOMMENDATION task is formulated as a ranking problem and evaluated using Mean Average Precision at rank one (MAP@1) (Zhu, 2004). For P LAN G ENERATION, we report both Role Playing Judge score to assess preference-aligned plan quality and a validity metric, defined as the proportion of sub-program steps that are executable with correct tool or feature names and wellformed arguments (Appendix Eq. 11).

We use G EMINI 2.5 P RO as the primary model for dataset generation and as the evaluation backbone for all LLMas-a-Judge protocols, including semantic exact matching and the Role Playing Judge (RP J). To reduce potential evaluation bias, we report the performance of G EMINI 2.5 P RO separately from that of the other evaluated models. Hyperparameters. We use temperature 0.6 for dataset generation to encourage diversity, and temperature 0.0 with top-p = 1.0 during evaluation for deterministic and reproducible outputs. Each task is allotted up to 15 turns to finish.

Role Playing Judge. Evaluating PG requires capturing subjective satisfaction under personalized household preferences. We therefore leverage a Role Playing LLM-as-aJudge framework, RP J, in which the judge model evaluates plans from the perspectives of individual household members and the household collectively. As formalized in Algorithm 1, RP J operates in two modes: ABS and REL.

In addition to the agentic inference, we evaluate models in a Sole-Reasoning setting, where the full household context is directly provided to the model without requiring any tool invocation. This configuration serves as an upper bound as relevant information is explicitly available in the prompt.

In ABS mode, RP J assigns a scalar score and textual feedback to each sub-program within a plan. Since plans may include both strong and weak sub-programs, this mode enables fine-grained analysis and supports surfacing only highquality components. We employ RP JABS in multimodal evaluation and report overall average scores.

A complete description of all model configurations and inference settings is provided in Appendix E. 5.2. Evaluation Metrics This subsection describes the evaluation metrics used across tasks, followed by a detailed description of the Role Playing Judge (RP J) framework used for subjective assessment.

In REL mode, RP J ranks complete plans across models according to household preferences, framing evaluation as a preference comparison that aligns naturally with RLHFstyle training objectives (Comanici et al., 2025). RP JREL is particularly efficient when comparing many models. For example, evaluating eleven models requires 11x calls under RP JABS but only x calls under RP JREL .

Reactive Tasks. For these deterministic question answering tasks, we report Accuracy (ACC) (Appendix Eq. 3 & 6). For IG and CF, correctness is first assessed using stringlevel exact match. When an exact match fails, we employ 5

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

For Table 2 RP JREL ranks the 11 candidate plans per instance and we report win rate, i.e., the fraction of times a model is ranked first; since win rates sum to 100%2 , this metric reflects a distribution of preference mass rather than absolute quality. Details on metric definitions and mathematical formulations are provided in the Appendix D.1.

GPT-4 achieves the strongest results in most settings; beyond this, no single model consistently dominates across tasks, modalities, and execution regimes. Reactive Tasks. Under Sole Reasoning, models perform strongly on IG and MC, with many exceeding 85% accuracy. Introducing tools causes sharp declines, frequently on the order of tens of points, even for large models. The most severe degradations occur for IG and CF, while MC shows slightly smaller but still substantial drops, likely due to its constrained answer space. These trends indicate that failures in retrieval and integration of relevant household information dominate error patterns in agentic settings.

Algorithm 1 Role-Playing Judge (RP J) Require: Household personas H, situation description S, N models {Mi }N i=1 with generated plans {Pi }i=1 , evaluation mode mode ∈ {ABS, REL}, judge model Mθ Ensure: Evaluation results R containing overall and perpersona outcomes 1: Initialize judge model Mθ with RP J prompt conditioned on H, S, and mode 2: if mode is ABS then 3: Scoring: All scores are integers in {1, . . . , 5}, where 5 denotes the highest quality. 4: for each plan Pi do 5: Query Mθ to assign an overall scoreo to Pi 6: for each household persona h ∈ H do 7: Query Mθ to assign a per-persona scorep to Pi 8: end for 9: Store overall and per-persona scores in R 10: end for 11: else 12: Query Mθ to rank all models {Mi }N i=1 by overall usefulness of their plans 13: for each household persona h ∈ H do Query Mθ to rank all models {Mi }N 14: i=1 from the perspective of h 15: end for 16: Store overall and per-persona rankings in R 17: end if 18: Return R

Proactive Tasks. FR exhibits moderate performance under Sole Reasoning and shows the smallest relative drop when tools are introduced, likely because it is formulated as a predictive ranking problem rather than free-form generation. Nevertheless, even large models struggle to maintain strong ranking quality with tools, indicating persistent difficulty in mapping household context to relevant appliance features. For PG, RP JREL evaluation shows that although GPT-4 captures a large share of wins, many other models obtain non-trivial win rates, producing a dispersed preference distribution. This dispersion suggests substantial variability in plan quality and indicates that models often generate plans that are only partially aligned with household preferences. Plan validity paints a more severe picture. Under Sole Reasoning, most models generate highly valid structured plans. In agentic settings, validity collapses for nearly all text-based models, demonstrating that reliably producing runnable tool calls is a major issue even when high-level plans appear reasonable. A small number of models retain comparatively higher validity, but multimodal inputs do not improve executability and slightly reduce it. Reasoning Modes. Explicit reasoning variants (e.g., [T/L/M/H]) consistently improve performance under Sole Reasoning, especially for reactive tasks, but these gains largely vanish in agentic settings and for proactive tasks. In several cases, higher reasoning levels fail to improve and can even degrade tool-mediated performance, indicating that additional internal thinking does not translate into better tool coordination or state tracking.

6. Results and Discussion In this section, we summarize key findings from our experiments and analyze the impact of individual factors. 6.1. Overall Results Table 2 reveals a consistent pattern across models and tasks: performance drops substantially when moving from Sole Reasoning to agentic execution with tools. This degradation affects both reactive and proactive tasks, indicating that effective tool interaction remains a primary bottleneck. While larger models achieve higher absolute performance, scale alone does not close this gap, suggesting that dominant failures arise from interaction-level reasoning rather than insufficient model capacity. Among evaluated models, 2

Multimodality. Multimodal inputs improve reactive performance under Sole Reasoning, suggesting that visual grounding aids recovery of situational details. However, both proactive tasks degrade with multimodality, and once tools are introduced, differences between text-only and multimodal largely disappear. This indicates that perception alone does not address the dominant challenges in tool use and longhorizon planning.

For top-k rankings with k > 1, win rates do not sum to 100%.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes Table 2. Model performance sorted by size. Proprietary models are shaded . Bold and underline mark the best and second-best non–Gemini 2.5 Pro results. Confidence intervals for non-percentage metrics are reported in Appendix D.3.

Model

Input Modality

Information Grounding (ACC) No With Tools Tools

Counterfactual Reasoning (ACC) No With Tools Tools

Multiple Choice (ACC) No With Tools Tools

Feature Reco. (MAP@1) No With Tools Tools

Plan Gen. (RP JREL ) No With Tools Tools

Plan Gen. (Validity) No With Tools Tools

Text-Based Data Qwen3-4B Qwen3-4B [T] GPT-OSS-20B [L] GPT-OSS-20B [M] GPT-OSS-20B [H] Qwen3-30B Qwen3-30B [T] Nemotron3-Nano-30B-A3B

Text Text Text Text Text Text Text Text

75.00 75.70 80.00 79.94 81.20 73.50 78.82 75.62

56.40 32.60 67.80 68.90 69.60 64.14 67.50 64.90

67.50 71.00 72.80 72.81 74.00 63.55 74.53 70.83

45.45 36.35 63.10 63.65 63.25 52.35 58.75 57.95

86.70 90.30 88.70 88.70 89.20 83.20 91.60 86.20

65.10 57.00 79.60 77.20 75.20 61.70 77.60 73.80

0.258 0.257 0.242 0.242 0.247 0.260 0.254 0.259

0.216 0.229 0.211 0.217 0.229 0.229 0.244 0.239

8.100 6.900 9.700 10.000 8.500 5.800 6.900 9.400

2.000 5.500 7.400 5.500 4.400 1.900 4.800 5.700

0.664 0.773 0.753 0.725 0.698 0.645 0.665 0.825

0.001 0.006 0.343 0.415 0.413 0.002 0.010 0.010

GPT-4o

Text

80.00

71.03

71.30

60.94

86.30

75.00

0.277

0.227

6.100

5.100

0.836

0.023

GPT-4

Text

77.90

73.80

73.45

62.90

86.30

79.20

0.270

0.247

19.300

22.300

0.869

0.022

Gemini 2.5 Pro

Text

86.60

77.50

80.70

72.90

93.80

85.80

0.297

0.287

12.000

35.400

0.795

0.030

Gemini 2.5 Pro

Text

67.74

63.31

67.74

68.82

85.18

80.59

0.209

0.205

3.660

3.130

0.753

0.508

Gemini 2.5 Pro

Multimodal

77.49

70.03

76.08

77.16

88.95

84.50

0.197

0.163

2.850

3.250

0.697

0.405

Multimodal Data

Summary. Overall, the results expose three persistent weaknesses: brittle tool usage, difficulty reasoning over structured household context under partial observability, and limited ability to generate coherent, executable, and preference-aligned plans. These findings highlight that personalized agentic intelligence in realistic smart home environments remains far from solved.

Qwen3-4B Qwen3-4B [T] GPT-OSS-20B [L] GPT-OSS-20B [M] GPT-OSS-20B [H] Qwen-3-30B Qwen-3-30B [T] Neotron3-Nano-30B GPT-4o GPT-4 Gemini 2.5 Pro

6.2. Additional Discussion Counterfactual Reasoning. Fig. 4 shows that for most models the degradation is larger in CFunchanged than in CFchanged , indicating difficulty in preserving correct conclusions under hypothetical perturbations; GPT-4 O, G EM INI 2.5 P RO, and Q WEN 3-30B (with and without reasoning) are partial exceptions. The most severe failures occur for Q WEN 3-4B [T], suggesting that smaller models, especially with explicit reasoning modes, can be particularly brittle. Interestingly, Q WEN 3-4B [T] is also the only model showing higher accuracy in CFchanged than on original questions, reflecting unstable counterfactual behavior rather than robust generalization. See Appendix Fig. 8 for Sole Reasoning.

56.40 32.60 67.80 68.90 69.60 64.14 67.50 64.90 71.03 73.80 77.50 Original Question

49.30 42.10 63.70 64.10 63.70 49.65 57.40 58.80 58.74 64.00 72.80 Answer Changed

41.60 30.60 62.50 63.20 62.80 55.04 60.10 57.10 63.14 61.80 73.00 Answer Unchanged

100 90 80 70 60 50 40

Figure 4. Agentic Counterfactual Reasoning (CF) performance across models. The heatmap reports accuracy on original questions, on counterfactual cases with changed and unchanged answers.

Tools (i.e. With Tools), these drops are dramatically reduced and, in several cases, reversed into modest gains. This pattern validates the core design of our PersonalHomeTools, demonstrating that its structured, tool-mediated access to household information enables models to adapt effectively to personalized contexts by retrieving user-specific details on demand rather than relying on brittle implicit inference. Additional results are in the Appendix Tables 8 and 9.

Impact of Personalization and PersonalHomeTools. Table 3 analyzes the effect of personalization by comparing performance differences between personalized and generic settings, where user-specific attributes are replaced with anonymized identifiers, reducing the need for individualized reasoning. Under Sole Reasoning, personalization consistently degrades performance for both IG and CF across all models, with large drops indicating that reasoning over personalized household details introduces substantial additional complexity. In contrast, when using PersonalHome-

Table 3. Effect of personalization, measured as accuracy difference (∆ = P ersonalized−Generic)

Model GPT-OSS-20B [H] GPT-4 Gemini 2.5 Pro

No Tools ↓ 18.34 ↓ 21.88 ↓ 13.20

IG With Tools ↑ 3.83 ↓ 0.53 ↑ 0.60

No Tools ↓ 22.54 ↓ 26.33 ↓ 19.10

CF With Tools ↑ 1.42 ↓ 0.64 ↑ 1.10

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Tool Usage Patterns. Fig. 5 shows that proactive tasks generally trigger more tool calls than reactive tasks, reflecting their multi-step nature, while larger and reasoning-enabled models tend to invoke tools more frequently than smaller models. However, higher tool usage does not consistently translate into better performance, underscoring that effective coordination and timing of tool use, rather than sheer frequency, is the primary challenge.

ingful signal for assessing plan quality and personalization. Qualitative Analysis. Fig. 6 illustrates representative success and failure modes for both reactive and proactive settings. In Fig. 6a, weaker models make redundant or misdirected tool calls and fail to recover from early errors, leading to incorrect or partial answers, whereas stronger models issue targeted queries over memories and member information and assemble coherent evidence before answering. Fig. 6b shows that effective proactive behavior begins with acquiring situational context, followed by synthesizing multi-step appliance-control plans. Even when plans appear sensible, models frequently fail to generate fully correct tool invocations, and these examples (c.f. Appendix I) mirror the quantitative results, underscoring that reliable tool usage and state-aware coordination remain dominant limitations.

Figure 5. Tool usage distributions across models for reactive (top) and proactive (bottom) tasks, grouped by low (< 5), medium (5–10), and high (> 10) numbers of calls per task instance.

(a) Reactive Task (IG) Example

Impact of Reflection. We repurpose our Role Playing Judge (RPJ) as a self-reflection signal, enabling models to iteratively critique and revise generated plans in the spirit of Reflexion (Shinn et al., 2023). RP J feedback highlights preference mismatches and tool-level errors, guiding targeted plan refinement in multimodal settings. As shown in Table 4, reflection consistently improve both RPJABS scores and plan validity, indicating more executable and better-aligned plans for both Text-Only and Multimodal settings.

(b) Proactive Task (PG) Example Figure 6. Qualitative examples illustrating personalized agent behavior in PersonalHomeBench, covering reactive question answering (top) and multimodal proactive planning (bottom).

Refer to Appendix 6.2 for limitations and future directions.

Table 4. Reflection-based ablations for Agentic G EMINI 2.5 P RO. We report average RP JABS and Validity for Plan Generation. Setting Base w/ Reflection

Text-only RP JABS Validity 3.380 0.030 3.520 0.428

Conclusion

Multimodal RP JABS Validity 3.250 0.405 4.240 0.496

We presented PersonalHomeBench, a benchmark for evaluating personalized agentic behavior in realistic smart home environments, together with PersonalHomeTools, a domainspecific toolbox for grounded interaction with household context, memory, perception, and device control. Extensive evaluation across foundation models demonstrates that strong performance under full observability does not translate to reliable tool usage or proactive behavior, and that counterfactual reasoning and personalized plan generation remain challenging. Improvements from scaling, explicit reasoning modes, and multimodal inputs are limited and inconsistent. These results suggest that key challenges remain

Human Analysis. LLM-as-a-Judge is known to be imperfect (Dorner et al., 2025), and is commonly validated with human assessment for subjective tasks. Accordingly, we asked four human annotators to evaluate RP J outputs, marking disagreement, partial agreement, or full agreement with both overall and persona-level scores. We observe strong alignment, with no cases of complete disagreement and over 65% inter-annotator agreement (Appendix Fig. 11). This indicates that RP J provides a reliable and mean8

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

in combining reasoning, tool usage, and personalization, and we hope that PersonalHomeBench will facilitate more systematic study of these issues and support future advances in robust personalized agentic systems.

fidelity of personalized agent evaluation.

References Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

Impact Statement This work aims to advance research on personalized and proactive agentic systems, with a focus on smart home environments, and carries both potential benefits and risks. On the positive side, improved personalization can enhance accessibility, safety, child supervision, and overall quality of life, particularly for elderly users and individuals with special needs. At the same time, agentic technologies operating in personal spaces raise important issues around privacy, data security, and misuse. Our inclusion of compact 4B-parameter models also sheds light on the feasibility of privacy-preserving, on-device deployment, helping mitigate security and data exposure concerns. By releasing a benchmark and toolbox designed for systematic evaluation, we aim to support responsible development grounded in rigorous analysis. We encourage future work to prioritize safeguards, transparency, and informed user consent. Overall, we view the primary impact of this work as enabling more careful, accountable, and principled progress in personalized and proactive agentic systems.

Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. Bartkowiak, P. and Podstawski, M. Edgewisepersona: A dataset for on-device user profiling from natural language interactions. arXiv preprint arXiv:2505.11417, 2025. Blakeman, A., Grattafiori, A., Basant, A., Gupta, A., Khattar, A., Renduchintala, A., Vavre, A., Shukla, A., Bercovich, A., Ficek, A., et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mambatransformer model for agentic reasoning. arXiv preprint arXiv:2512.20848, 2025. Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A., Jang, L. K., and Hui, Z. Windows agent arena: Evaluating multi-modal OS agents at scale. Proceedings of the 42nd International Conference on Machine Learning, 267:4874–4910, 13–19 Jul 2025. URL https://proceedings.mlr.press/ v267/bonatti25a.html.

Limitations and Future Work We restrict multimodal benchmarking to G EMINI 2.5 P RO, as most currently available multimodal foundation models either lack native video input support or exhibit very weak reasoning and planning capabilities, making them unsuitable for long-horizon agentic evaluation (Singh et al., 2025; Tang et al., 2025; Xu et al., 2025). At present, G EMINI 2.5 P RO is the only model that demonstrates competitive performance across perception, reasoning, and planning in our setting. As multimodal models continue to mature, future iterations of PersonalHomeBench will incorporate a broader set of video-capable agents and enable more comprehensive multimodal comparisons. PersonalHomeBench also focuses on non-embodied smart home assistance rather than physical navigation or low-level manipulation; extending the benchmark to hybrid embodied–device settings is an important direction for future work.

Chang, M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081, 2024a. Chang, M., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., and He, J. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325–74362, 2024b. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.

In addition, our Role Playing Judge enables scalable evaluation of subjective plan quality and personalized satisfaction, but remains an LLM-based proxy. Although human agreement analysis indicates strong alignment, future work will expand direct human evaluation, explore preference learning from human feedback, and investigate training-time integration of RPJ-style supervision. Together, these directions outline a path toward strengthening both the scope and the

Das, S., Dai, R., Koperski, M., Minciullo, L., Garattoni, L., Bremond, F., and Francesca, G. Toyota smarthome: Realworld activities of daily living. In The IEEE International Conference on Computer Vision (ICCV), October 2019. 9

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Dorner, F. E., Nastl, V., and Hardt, M. Limits to scalable Meyer, Y. and Corneil, D. Nemotron-personas: Synthetic evaluation at the frontier: Llm as judge won’t beat personas aligned to real-world distributions. Hugging twice the data. In Yue, Y., Garg, A., Peng, N., Sha, Face Dataset, 2025. F., and Yu, R. (eds.), International Conference on Puig, X., Ra, K. K., Boben, M., Li, J., Wang, T., FiLearning Representations, volume 2025, pp. 26467– dler, S., and Torralba, A. Virtualhome: Simulating 26491, 2025. URL https://proceedings. household activities via programs. 2018 IEEE/CVF iclr.cc/paper_files/paper/2025/file/ Conference 4264ee4376776907c0b87ed70b959585-Paper-Conference. on Computer Vision and Pattern Recognition, pp. 8494–8502, 2018. URL https://api. pdf. semanticscholar.org/CorpusID:49317780. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Radford, A., et al. Gpt-4o system card. arXiv preprint Tian, R., Xie, R., Zhou, J., Gerstein, M., dahai li, Liu, arXiv:2410.21276, 2024. Z., and Sun, M. ToolLLM: Facilitating large language Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, models to master 16000+ real-world APIs. In The Twelfth C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient International Conference on Learning Representations, memory management for large language model serving 2024. URL https://openreview.net/forum? with pagedattention. In Proceedings of the 29th symid=dHng2O0Jjr. posium on operating systems principles, pp. 611–626, Salemi, A., Mysore, S., Bendersky, M., and Zamani, 2023. H. LaMP: When large language models meet personLi, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., alization. In Ku, L.-W., Martins, A., and Srikumar, Martín-Martín, R., Wang, C., Levine, G., Ai, W., MarV. (eds.), Proceedings of the 62nd Annual Meeting of tinez, B., Yin, H., Lingelbach, M., Hwang, M., Hiranaka, the Association for Computational Linguistics (Volume A., Garlanka, S., Aydin, A., Lee, S., Sun, J., Anvari, M., 1: Long Papers), pp. 7370–7392, Bangkok, Thailand, Sharma, M., Bansal, D., Hunter, S., Kim, K.-Y., Lou, A., August 2024. Association for Computational LinguisMatthews, C. R., Villa-Renteria, I., Tang, J. H., Tang, C., tics. doi: 10.18653/v1/2024.acl-long.399. URL https: Xia, F., Li, Y., Savarese, S., Gweon, H., Liu, C. K., Wu, //aclanthology.org/2024.acl-long.399/. J., and Fei-Fei, L. Behavior-1k: A human-centered, emShinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and bodied ai benchmark with 1,000 everyday activities and Yao, S. Reflexion: Language agents with verbal reinrealistic simulation. arXiv preprint arXiv:2403.09227, forcement learning. Advances in Neural Information 2024. Processing Systems, 36:8634–8652, 2023. Li, S., Guo, Y., Yao, J., Liu, Z., and Wang, H. Homebench: Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Evaluating llms in smart homes with valid and invalid Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, instructions across single and multiple devices. arXiv D. Alfred: A benchmark for interpreting grounded preprint arXiv:2505.19628, 2025. instructions for everyday tasks. 2020 IEEE/CVF Lu, Y.-T., Yang, S., Qian, C., Chen, G.-F., Luo, Q., Conference on Computer Vision and Pattern Recognition Wu, Y., Wang, H., Cong, X., Zhang, Z., Lin, Y., (CVPR), pp. 10737–10746, 2019. URL https: Liu, W., Wang, Y., Liu, Z., Liu, F., and Sun, M. //api.semanticscholar.org/CorpusID: Proactive agent: Shifting llm agents from reactive re208617407. sponses to active assistance. ArXiv, abs/2410.12361, 2024. URL https://api.semanticscholar. Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., org/CorpusID:273375463. Ananthram, A., et al. Openai gpt-5 system card. arXiv Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbipreprint arXiv:2601.03267, 2025. eri, F., and Fang, Y. Evaluating very long-term conversational memory of LLM agents. In Ku, L.-W., Martins, Singh, H., Verma, N., Wang, Y., Bharadwaj, M., Fashandi, A., and Srikumar, V. (eds.), Proceedings of the 62nd H., Ferreira, K., and Lee, C. Personal large language Annual Meeting of the Association for Computational model agents: A case study on tailored travel planLinguistics (Volume 1: Long Papers), pp. 13851–13870, ning. In Dernoncourt, F., Preoţiuc-Pietro, D., and Bangkok, Thailand, August 2024. Association for ComShimorina, A. (eds.), Proceedings of the 2024 Conferputational Linguistics. doi: 10.18653/v1/2024.acl-long. ence on Empirical Methods in Natural Language Pro747. URL https://aclanthology.org/2024. cessing: Industry Track, pp. 486–514, Miami, Florida, acl-long.747/. US, November 2024. Association for Computational 10

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Linguistics. doi: 10.18653/v1/2024.emnlp-industry. 37. URL https://aclanthology.org/2024. emnlp-industry.37/. Tang, C., Li, Y., Yang, Y., Zhuang, J., Sun, G., Li, W., Ma, Z., and Zhang, C. video-salmonn 2: Captioningenhanced audio-visual large language models. arXiv preprint arXiv:2506.15220, 2025.

2025b. URL https://openreview.net/forum? id=tRXt10xKc5. Yang, R., Chen, H., Zhang, J., Zhao, M., Qian, C., Wang, K., Wang, Q., Koripella, T. V., Movahedi, M., Li, M., et al. Embodiedbench: Comprehensive benchmarking multimodal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025c.

Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20744–20757. Curran Associates, Inc., 2022. URL https://proceedings.neurips. cc/paper_files/paper/2022/file/ 82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference. pdf.

Wang, N., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Yang, J., et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 14743–14777, 2024. Xi, Z., Ding, Y., Chen, W., Hong, B., Guo, H., Wang, J., Guo, X., Yang, D., Liao, C., He, W., et al. Agentgym: Evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27914– 27961, 2025.

Zhang, X., Deng, Y., Ren, Z., Ng, S.-K., and Chua, T.-S. Ask-before-plan: Proactive language agents for real-world planning. ArXiv, abs/2406.12639, 2024. URL https://api.semanticscholar. org/CorpusID:270561990.

Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622, 2024a.

Zhao, S., Zhu, A., Mozannar, H., Sontag, D. A., Talwalkar, A., and Chen, V. Codinggenie: A proactive llm-powered programming assistant. Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025. URL https://api.semanticscholar. org/CorpusID:277112990.

Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer enviZhu, M. Recall, precision and average precision. Departronments. In Globerson, A., Mackey, L., Belgrave, ment of Statistics and Actuarial Science, University of D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. Waterloo, Waterloo, 2(30):6, 2004. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 52040–52094. Curran Associates, Inc., 2024b. doi: 10.52202/079017-1650. URL https://proceedings.neurips. cc/paper_files/paper/2024/file/ 5d413e48f84dc61244b6be550f1cd8f5-Paper-Datasets_ and_Benchmarks_Track.pdf. Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Yang, B., Xu, L., Zeng, L., Liu, K., Jiang, S., Lu, W., Chen, H., Jiang, X., Xing, G., and Yan, Z. Contextagent: Context-aware proactive LLM agents with openworld sensory perceptions. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 11

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

A. Author Contributions The work presented in this paper was carried out by members of the LG Research Team across multiple labs. LG Toronto AI Lab: • Nikhil Verma LG Korea AI Lab: • InJung Yang • Sungil Kim • KoKeun Kim • YoungJoon Kim Former Contributors (affiliated with LG Research during this work): • Manasa Bharadwaj • Yolanda Liu • Kevin Ferreira

B. Additional Related Works Reactive Agentic Benchmarks. A large body of work has established benchmarks for evaluating agentic behavior in web-based interaction (Yao et al., 2022), UI navigation (Xie et al., 2024b), game-like environments, and simulated worlds (Chang et al., 2024b). These benchmarks have standardized evaluation of multi-step instruction following and reasoning, but typically emphasize reactive task completion in environments with limited long-term context and minimal personalization. More recent tool-centric benchmarks evaluate whether foundation models can select and invoke general-purpose APIs, providing insight into basic tool usage (Qin et al., 2024). However, the tools are largely domain-agnostic, and tasks rarely require sustained user modeling, household continuity, or proactive assistance. Embodied Home Agentic Benchmarks. Embodied household benchmarks such as VirtualHome (Puig et al., 2018), ALFRED (Shridhar et al., 2019), BEHAVIOR (Li et al., 2024), and PARTNR (Chang et al., 2024a) evaluate agents in simulated home environments with navigation and atomic action execution. In contrast, PersonalHomeBench focuses on non-embodied, real-world home environments, using long-horizon videos captured in diverse households to reflect natural daily routines, health, and safety scenarios. Proactive Agentic Benchmarks. Current proactive LLM agents cover specialized frameworks for clarification (Zhang et al., 2024), multi-agent prediction (Lu et al., 2024), and UI-monitored task assistance (Lu et al., 2024; Zhao et al., 2025). ContextAgentBench (Yang et al., 2025b) takes an important step toward evaluating proactive agents with contextual information and tool usage, but focuses primarily on ego-centric daily traces, relies on generic tools, and operates in language-only inference settings. In contrast, our benchmark targets shared household environments, uses domain-specific smart home tools, and evaluates both reactive and proactive behaviors grounded in persistent, personalized context with optional multimodal inputs. Personalization in Interactive Systems. Personalization has been studied through persona consistency benchmarks, long-term memory datasets, and user-specific question answering tasks. These benchmarks primarily assess whether models can recall or remain consistent with user attributes in text-based interactions (Salemi et al., 2024; Maharana et al., 2024). Parallel work explores personalization for narrow planning problems, such as personalized travel assistants with limited tool support (Singh et al., 2024). While valuable, these settings treat personalization largely as a property of dialogue or retrieval. PersonalHomeBench treats personalization as a central driver of interactive decision making, assessing how user preferences influence reasoning, tool usage, and long-horizon planning in dynamic environments, and how this information is leveraged when evaluating generated plans. 12

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Home-Centric Benchmarks. Unlike datasets such as Toyota Smarthome (Das et al., 2019), which are staged, demographically narrow, and not designed around smart-device interaction, our data spans diverse home layouts, age groups, relationships, and unprompted activities. Videos range from seconds to tens of minutes and may cover multiple activities (e.g., cooking, leisure, sleep), enabling evaluation of persistent context tracking and proactive assistance over extended horizons. Recent language-model-based home assistant benchmarks, such as HomeBench (Li et al., 2025) and EdgeWisePersona (Bartkowiak & Podstawski, 2025), move closer to smart home scenarios but remain limited. HomeBench focuses on reactive validation and execution of smart home commands without modeling personalization, while EdgeWisePersona emphasizes profile reconstruction and lightweight proactive routine inference with a small set of devices and simple state transitions.

C. Additional Household Statistics C.1. Text-based Data Statistics Household Size. The households capture a structured variety of domestic environments, categorized into: single occupant, family and non-related roommates. The range of household scales from single-occupancy to large-family residences (Fig. 7a). Appliances Owned. The dataset features nearly 40 distinct appliance types, including five core appliances present in all households: refrigerator, thermostat, microwave, speaker and TV. Other top most prevalent appliances are listed in (Fig. 7c). Pets Distribution. Across 409 households, a total of 823 animals are represented in this dataset. The pet population is categorized into nine animal types: cat, dog, hamster, lizard, guinea pig, rabbit, bird, fish and turtle - spanning a wide range of common domestic mammals, reptiles and aquatic companions. C.2. Multimodal Data Statistics Table 5 presents the distribution of the 100 scenarios constructed for this study across three core domains: safety care, health care, and daily care. Health care scenarios constitute the largest portion, reflecting the prevalence of situations involving well-being, medication, and environmental health in household settings. Daily care scenarios capture routine activities such as returning home and pet care, while safety care scenarios focus on risk-sensitive situations and accident prevention. Scenarios are categorized based on consistent semantic cues in their titles, enabling clear domain delineation while preserving coverage of diverse, realistic household situations spanning safety, health, and everyday routines. Table 6 summarizes the participant composition of the real home data used in our benchmark across health care, safety care, and daily care scenarios. The dataset captures substantial diversity in household roles, including adults, caregivers, children, and pets, spanning a wide range of age groups from young children to older adults in their 70s. Safety care scenarios are the most prevalent, reflecting common real-world situations that involve risk monitoring and intervention, particularly in households with both caregivers and children. Daily care scenarios further emphasize routine, multi-occupant household dynamics, while health care scenarios focus on age-sensitive contexts involving adults across multiple life stages. Overall, this distribution highlights the benchmark’s emphasis on realistic household diversity and interaction patterns, providing a representative foundation for evaluating personalized and proactive assistance in everyday home environments. All the data is ego-centric views, and collected through safe placement of cameras in subject’s homes.

Instructions for Video Data Collection Participants. Participants were provided with scenario-specific guidance corresponding to the domains listed in Table 5 after household setup was completed, including camera placement, microphone activation, and sensor checks. For each scenario, participants received high-level instructions describing how the scene should unfold and the types of actions to perform, while allowing natural variation in execution. For scenarios involving safety or hazardous situations, participants were instructed to simulate the actions rather than perform them physically, and on-site staff ensured that all safety protocols were followed. Scenes with multiple participants were coordinated by coaching all subjects involved to maintain consistency in the intended narrative. To ensure usable recordings, scenes were re-shot when necessary to improve clarity or coverage. The source data were collected under a service contract with an external vendor and processed in compliance with applicable personal data protection regulations following a personal information impact assessment. Participants were compensated for their participation. 13

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes Table 5. Distribution of scenario domains and classification criteria used in the benchmark. Domain

Video Count

Classification Basis

Safety Care Health Care Daily Care / Routine

30 38 32

Scenario titles reference safety, risk, hazard, or accident prevention Scenario titles reference health, medication, ventilation, or well-being Scenario titles reference daily routines, returning home, pet care, or going out

Total

100

—

Table 6. Participant demographics across scenarios in the collected home data. Scenario

Participant Type

Age Group

Gender

Participant Count

Health Care Health Care Health Care

Adult Adult Adult

30s 60–70s 60–70s

Male Male Female

2 3 3

Health Care (Total) Safety Care Safety Care Safety Care Safety Care Safety Care Safety Care Safety Care Safety Care

8 Adult (Caregiver) Adult (Caregiver) Adult (Caregiver) Adult Child Child Child Child

30s 40–50s 40–50s 30s Under 10 Under 10 5 and over 10 and over

Female Female Male Male Male Female Female Female

Safety Care (Total) Daily Care Daily Care Daily Care Daily Care Daily Care Daily Care Daily Care Daily Care Daily Care Daily Care

22 Adult Adult Adult Adult Adult (Caregiver) Adult (Caregiver) Child Child Child Child

60s 70s 70s 30s 30s 40–50s Under 10 Under 10 5 and over 10 and over

Male Male Female Male Female Female Female Male Female Female

Daily Care (Total) Pets

5 3 2 1 3 5 1 2 1 1 1 1 2 2 3 1 1 2 15

Animal

–

Cat

D. Additional Metrics D.1. Definitions For IG and CF, correctness is first assessed using string-level exact match (Eq. 1). If an exact match fails, we employ G EMINI 2.5 P RO as a semantic equivalence judge (Eq. 2) to determine whether the predicted answer is meaningfully consistent with the ground truth. This hybrid decision rule is formalized in Eq. 3. For CF, performance is reported as the macro-average over counterfactual instances where the answer remains unchanged and those where the answer changes, as defined in Eq. 4. For MC, accuracy is computed via exact match between the selected option (or its textual form) and the ground truth choice, as shown in Eq. 6. Since all valid answers are explicitly enumerated, semantic judging is not applied for this task. For FR, we formulate the task as a ranking problem and report Mean Average Precision at rank k (MAP@k), which measures how highly relevant features are ranked within the recommendation list. MAP@k computation is given in Eq. 9. We additionally report Average Relevant Rank Sum (RRS), a ranking error metric that quantifies how far relevant items are from their ideal positions. For each instance, the absolute deviation between the predicted rank of each relevant item and its ideal rank (1–3) is summed, and the final score is obtained by averaging across instances (Eq. 12–13). Lower values indicate better ranking quality, with zero corresponding to perfect ordering. This metric provides complementary insight beyond 14

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

(a) Household Structure

(b) #Appliances Owned

Figure 7. Additional statistics of generated household in PersonalHomeBench. (a) Distribution of household size and composition. (b) Distribution of appliances owned in the households. (c) Common appliances owned by the households.

top-k accuracy by capturing the magnitude of mis-ranking among relevant options. For PG, we use two complementary metrics. First, subjective plan quality is assessed using the Role Playing Judge (RPJ) framework. Second, we measure plan validity, defined as the proportion of sub-program steps that are directly executable with correct tool or feature names and well-formed arguments, as formalized in Eq. 11. Together, these metrics capture correctness for deterministic reactive tasks, ranking quality for proactive recommendation, and both perceived usefulness and practical executability for plan generation. Table 7. Summary of evaluation metrics used for each task and their corresponding equation definitions.

Task Information Grounding (IG) Counterfactual Reasoning (CF) Counterfactual Reasoning (CF) Multiple Choice (MC) Feature Recommendation (FR) Feature Recommendation (FR) Plan Generation (PG) Plan Generation (PG)

Metric Hybrid Exact/Semantic Accuracy Hybrid Exact/Semantic Accuracy Macro-Average Accuracy Exact-Match Accuracy MAP@k Average Relevant Rank Sum RPJ Score Plan Validity

Equation Eq. 3 Eq. 3 Eq. 4 Eq. 6 Eq. 9 Eq. 13 Algorithm 1 Eq. 11

EM(ŷi , yi ) = I[norm(ŷi ) = norm(yi )] ,

(1)

J(ŷi , yi ) = JG EMINI 2.5 P RO (ŷi , yi ) ∈ {0, 1},

(2) N

(IG/CF)

= max EM(ŷi , yi ), J(ŷi , yi ) ,

ACCMACRO = CF

ACCCFs =

ACCIG/CF =

1 X (IG/CF) c . N i=1 i

1 ACCCF CHANGED + ACCCFUNCHANGED , 2

1 X (CF) ci , |Ss |

s ∈ {CHANGED, UNCHANGED}.

(3)

(4)

(5)

i∈Ss

(MC)

= I[norm(ẑi ) = norm(zi )] , 15

ACCMC =

1 X (MC) c . N i=1 i

(6)

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

1X reli (t), r t=1

(7)

k X 1 Pi (r) reli (r), min(k, Ri ) r=1

(8)

Pi (r) =

AP@ki =

MAP@k =

1 X AP@ki , N i=1

(9)

vi,j = I[step (i, j) uses a valid tool/feature name and well-formed arguments] ,

Validityi =

(10)

i 1 X vi,j , Ji j=1

Validity =

1 X Validityi . N i=1

(11)

Per-instance Relevant Rank Deviation. Let N denote the number of evaluation instances and K = 3 the number of relevant items per instance. For instance i, let ri,j denote the predicted rank of the j-th relevant item, where the ideal ranks are j ∈ {1, 2, 3}.

Di =

K X

|ri,j − j|

(12)

j=1

RRS =

1 X Di N i=1

(13)

D.2. Reactive Tasks C OUNTERFACTUAL R EASONING Figure 8 compares CF performance under Sole Reasoning (right) and agentic tool usage (left). Without tools, most models achieve relatively high accuracy on original questions and maintain comparable performance on both answer-changed and answer-unchanged counterfactuals, indicating that many models can internally simulate hypothetical perturbations when full context is directly provided. Performance is consistently strongest for large proprietary models, while medium and small models exhibit modestly lower but stable behavior across counterfactual types. In contrast, introducing tools produces a uniform degradation across models and counterfactual states. The drop is not confined to cases where the answer changes, but also affects answer-unchanged queries, suggesting that failures arise primarily from imperfect retrieval and tool interaction rather than counterfactual reasoning itself. Several models exhibit larger degradation on the unchanged condition, indicating that even preserving known facts becomes fragile once information must be re-acquired through tools. Overall, the comparison highlights a clear separation between reasoning competence and interaction competence: models retain non-trivial counterfactual reasoning ability in isolation, but struggle to express it reliably in agentic settings where correct tool use is required. This reinforces that effective information acquisition and state grounding, rather than hypothetical reasoning alone, is the dominant bottleneck for counterfactual performance in personalized home environments. Impact of Personalization and PersonalHomeTools. Tables 8 and 9 show that personalization has markedly different effects under Sole Reasoning versus agentic evaluation. When complete household context is directly available (Table 9), personalization generally leads to substantial accuracy gains across IG, CF, and MC, indicating that models can effectively exploit user-specific information when retrieval is trivial. In contrast, under agentic evaluation (Table 8), personalization yields smaller and more mixed changes, with modest improvements in some cases and slight regressions in others. Importantly, the relative stability of performance in the agentic setting reflects the personalization-centric design of 16

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

P ERSONAL H OME T OOLS, which enables structured retrieval of user-specific context and mitigates the difficulty of reasoning directly over raw personalized information. Effect of Task Difficulty on Reactive Task Performance. Figure 9 illustrates consistent and interpretable scaling behavior across task difficulty for all three reactive tasks. Accuracy decreases monotonically from easy to medium to hard settings, confirming that the difficulty stratification meaningfully captures increasing reasoning complexity. IG and MC remain comparatively more stable than CF across all difficulty levels, while CF exhibits the steepest degradation, reinforcing its role as the most challenging reactive task. Larger and proprietary models generally achieve higher accuracy across difficulties, but no model is immune to performance drops on hard instances, indicating that complex household-level reasoning remains challenging even at scale. Marker sizes further show that harder instances tend to require more interaction turns, suggesting that models attempt deeper tool-based exploration or longer reasoning chains without reliably closing the performance gap. Overall, the figure highlights that increasing difficulty exposes systematic weaknesses in multi-hop and counterfactual reasoning, and that additional interaction alone is insufficient to overcome these challenges.

Comparison of Model Robustness: Impact of Tool-Use on Counterfactual Scenarios SETTING: WITHOUT TOOLS

Qwen3-4B

56.40

49.30

41.60

75.00

67.20

67.80

32.60

42.10

30.60

75.70

70.30

71.70

67.80

63.70

62.50

80.00

71.20

74.40

GPT-OSS-20B [M]

68.90

64.10

63.20

79.94

70.66

74.95

GPT-OSS-20B [H]

69.60

63.70

62.80

81.20

71.90

76.10

Model Name

Qwen3-4B [T] GPT-OSS-20B [L]

Qwen-3-30B

64.14

49.65

55.04

73.50

59.50

67.60

Qwen-3-30B [T]

67.50

57.40

60.10

78.82

72.33

76.72

Neotron3-Nano-30B

64.90

58.80

57.10

75.62

69.63

72.03

GPT-4o

71.03

58.74

63.14

80.00

66.90

75.70

GPT-4

73.80

64.00

61.80

77.90

69.90

77.00

Gemini 2.5 Pro

100 90 80 70 60 50 40

77.50

72.80

73.00

86.60

78.80

82.60

Original Question

Answer Changed

Answer Unchanged

Original Question

Answer Changed

Answer Unchanged

Counterfactual State

Accuracy (%)

SETTING: WITH TOOLS

Figure 8. Counterfactual Reasoning (CF) performance across models under settings with and without tool access. The heatmap reports accuracy for original questions, counterfactual queries where the correct answer changes, and counterfactual queries where the answer remains unchanged.

easy

Information Grounding

Task Counterfactual Reasoning medium

Multiple Choice

hard

90 80

accuracy

70 60 50 40

SS -2 Qw 0B [ en H] Qw 3-30 B en -3Ne 30B mo [T] Tro n3 -3 GP 0B T-4 o GP T-4 Ge mi ni 2.5 Pro Qw en 3-4 Qw B en 3-4 GP B[ T-O T] SS GP -20B T-O [ SS L] GP -20B T-O [ SS M] -2 Qw 0B [ en H] Qw 3-30 B en -3Ne 30B mo [T] Tro n3 -3 GP 0B T-4 o GP T-4 Ge mi ni 2.5 Pro Qw en 3-4 Qw B en 3-4 GP B[ T-O T] SS GP -20B T-O [ SS L] GP -20B T-O [ SS M] -2 Qw 0B [ en H] Qw 3-30 B en -3Ne 30B mo [ Tro T] n3 -3 GP 0B T-4 o GP T-4 Ge mi ni 2.5 Pro

T-O

-20

T-O

T] B[ GP

T-O

en 3-4

Figure 9. Accuracy across models for three reactive task categories, Information Grounding (IG), Counterfactual Reasoning (CF), and Multiple-Choice (MCQ), evaluated at easy, medium, and hard difficulty levels. Each point corresponds to a model–task pair; marker size reflects the average number of turns taken to complete the task.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes Table 8. Effect of personalization under agentic evaluation, reporting accuracy for Generic and Personalized settings on IG, CF, and MC

Model GPT-OSS-20B [H] GPT-4 Gemini 2.5 Pro

IG (ACC %) Generic Personalized 65.77 69.60 74.33 73.80 76.90 77.50

CF (ACC %) Generic Personalized 61.83 63.25 63.54 62.90 71.80 72.90

MC (ACC %) Generic Personalized 71.00 75.20 77.40 79.20 87.90 85.80

Table 9. Effect of personalization under Sole Reasoning, reporting accuracy for Generic and Personalized settings on IG, CF, and MC.

Model GPT-OSS-20B [H] GPT-4 Gemini 2.5 Pro

IG (ACC %) Generic Personalized 99.54 81.20 99.78 77.90 99.80 86.60

CF (ACC %) Generic Personalized 99.54 77.00 99.78 73.45 99.80 80.70

MC (ACC %) Generic Personalized 88.20 89.20 83.90 86.30 94.10 80.70

D.3. Proactive Tasks F EATURE R ECOMMENDATION. Figure 10 shows consistent improvements in MAP@k as k increases across all models, with performance gains tapering off beyond moderate values of k. Across the board, agentic settings with tool access underperform their Sole Reasoning counterparts, reinforcing that effective interaction with tools remains a key challenge even for strong foundation models. Larger and more capable models achieve higher absolute MAP@k, but the relative gap between with-tools and without-tools settings persists, indicating that scale alone does not resolve interaction-level errors. Notably, G EMINI 2.5 P RO and GPT-4 class models define the upper envelope of performance, yet still exhibit nontrivial degradation under tool use, highlighting substantial headroom for improving agentic reasoning and decision-making in structured environments. Table 10 reports Average Relevant Rank Sum (RRS) for F EATURE R ECOMMENDATION under text-based evaluation. Under Sole Reasoning, frontier models achieve the lowest RRS values, with G EMINI 2.5 P RO and GPT-4 O performing best, indicating that relevant features are placed closer to their ideal ranks. Medium-scale open models exhibit moderately higher RRS, while smaller models show the largest deviations, reflecting noisier ranking behavior. When tools are introduced, RRS increases substantially for nearly all models, demonstrating that tool-mediated interaction degrades fine-grained ranking quality even when top-1 accuracy may remain reasonable. Notably, the relative ordering of models remains broadly consistent across settings, suggesting that the degradation primarily stems from interaction complexity rather than a change in underlying ranking capability. Overall, these results reinforce that precise preference ordering remains challenging in agentic contexts, and that effective use of tools is critical for maintaining ranking fidelity.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

1.0

Qwen3-4B Qwen3-4B [T]

GPT-OSS-20B [L] GPT-OSS-20B [M]

GPT-OSS-20B [H] NemoTron3-30B

Qwen-3-30B Qwen-3-30B [T]

GPT-4o GPT-4

Gemini 2.5 Pro

0.8

MAP@k

0.6

0.4

0.2

0.0

Condition Without Tools With Tools 1

Figure 10. Mean Average Precision at rank k (M AP @k) across models evaluated with and without tool access. Solid lines denote performance with tools, while dashed lines correspond to settings without tools. Results are reported for varying values of k, illustrating ranking performance trends across evaluation conditions.

Table 10. Relevant Rank Sum for the F EATURE R ECOMMENDATION task on text-based data. Lower values indicate better ranking quality.

Model Qwen3-4B Qwen3-4B [T] GPT-OSS-20B [L] GPT-OSS-20B [M] GPT-OSS-20B [H] Qwen3-30B Qwen3-30B [T] Neotron3-Nano-30B GPT-4o GPT-4 Gemini 2.5 Pro

No Tools 3.4730 3.3740 3.5740 3.5850 3.5930 3.4910 3.4500 3.2580 3.1340 3.2100 2.9780

With Tools 10.7620 10.4270 17.3260 15.0270 19.5160 21.8080 10.3440 9.0988 10.5030 10.2050 9.0090

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Table 11. Feature recommendation and plan generation validity across models.

Model Qwen3-4B Qwen3-4B [T] GPT-OSS-20B [L] GPT-OSS-20B [M] GPT-OSS-20B [H] Qwen3-30B Qwen3-30B [T] NemoTron3-30B GPT-4o GPT-4 Gemini 2.5 Pro Gemini 2.5 Pro (Multimodal)

Feature Reco. (MAP@1) No Tools With Tools 0.258±0.139 0.216±0.159 0.257±0.140 0.229±0.154 0.242±0.148 0.211±0.160 0.242±0.149 0.217±0.159 0.247±0.146 0.229±0.155 0.260±0.138 0.229±0.154 0.254±0.142 0.244±0.148 0.259±0.138 0.239±0.149 0.277±0.125 0.227±0.155 0.270±0.131 0.247±0.146 0.297±0.104 0.287±0.115 0.196±0.164 0.163±0.166

Plan Gen. (Validity) No Tools With Tools 0.664±0.341 0.001±0.015 0.773±0.324 0.006±0.072 0.753±0.404 0.343±0.464 0.725±0.412 0.415±0.481 0.699±0.428 0.414±0.483 0.645±0.302 0.002±0.024 0.666±0.467 0.044±0.205 0.825±0.320 0.010±0.090 0.836±0.215 0.023±0.117 0.869±0.208 0.022±0.123 0.795±0.238 0.030±0.157 0.697±0.297 0.405±0.423

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

E. Models and Hyper-Parameters Open models are hosted via vLLM (Kwon et al., 2023) on a multi-GPU node (4x GPUs, 96GB VRAM) utilizing a fixed tensor_parallel_size of 4. To ensure system stability and accommodate the overhead of long-context reasoning, the GPU memory utilization is lowered to 0.85 to prevent out-of-memory errors during the KV cache allocation. For models operating in explicit thinking or reasoning modes, the maximum reasoning token budgets are explicitly constrained based on model scale. It is capped for Q WEN 3-4B [T] and Q WEN 3-30B [T], while GPT-OSS-20B and N EMOTRON 3-NANO -30B-A3B utilize the default configuration. The Q WEN 3 reasoning models utilize D EEP S EEK -R1 as the reasoning parser and utilize its own Nano v3 parser within the vLLM serving environment. Table 12. Reasoning Token Configuration by Model

Model Name

Reasoning Token

GPU Mem Utilization

262,144 78,720 131,072 32,768 32,768 131,072

0.90 0.90 0.90 0.85 0.85 0.85

Qwen3-4B Qwen3-4B [T] GPT-OSS-20B [L,M,H] Qwen3-30B Qwen3-30B [T] Nemotron3-Nano-30B-A3B

F. Prompts F.1. Household Member Persona Generation Household Member Persona Generation Prompt Template You are an expert persona generator. Your job is to create realistic, coherent individual personas based on household-level information and sample personas. Follow these rules precisely: 1. Use the Household Persona as the Anchor • Use the household-level description to infer: demographics, living situation, socioeconomic context, lifestyle preferences, environment and context. • All individual personas must align with this household information. 2. Respect the Occupancy Type • If occupancy_type = roommate: – All individuals should be unrelated. – Ages should fall within a similar age range, unless the household persona describes otherwise. – Income levels, occupations, and lifestyles may differ, but should still plausibly co-exist in a shared-living situation. – Personalities should not be identical—make each person distinct yet compatible as cohabitants. • Household size = 2 – Could be a couple (romantic partners). – Or a parent + one child. • Household size = 3 – Could be a couple + one child. – Or a single parent + two children. • Household size = 4 or 5

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

– Typical family compositions include: * Two parents + children * Single parent + multiple children * Multi-generational families 3. Use the Samples as Style Guidance • Use samples only as examples of tone, structure, and level of detail. • Do not copy content. • Maintain similar attributes. 4. Produce a Concise, Self-Contained Persona for Each Individual • For each individual, provide: – member_id: must be {household_id}_{n} where n starts at 1 – name: extract name from persona description – role: parent, child, roommate, partner, etc. – age (estimate if not explicitly stated) – gender (Male or Female) – occupation – persona: 1-2 sentences summarizing personality and character – hobby: a list of hobby and interest keywords – lifestyle: habits, routines, energy usage tendencies (e.g., heavy cooking, night owl, active, tech-savvy) – preference: any preferences relevant for simulation (e.g., temperature settings, appliance usage patterns, quiet mornings) – major_event: list of significant life events with date and description. This field can be left empty if no major events are known. 5. Output Format Return the result as a list of personas, one entry per individual, labeled: "members": [ { "member_id": "{household_id}_1", "name": "", "role": "", "age": , "gender": "", "occupation": "", "persona": "", "hobbies": ["", ""], "lifestyle": "", "preference": "", "major_event": [ { "date": "", "description": "" } ] },

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

{ "member_id": "{household_id}_2", "name": "", "role": "", "age": , "gender": "", "occupation": "", "persona": "", "hobbies": ["", ""], "lifestyle": "", "preference": "", "major_event": [ { "date": "", "description": "" } ] } ]... F.2. Task Generation Reactive Tasks Info Grounding and Multiple Choice Question Task Generation Prompt Template Prompt: Dictionary-Driven Question–Answer Generation with Programs Role You are a data QA generation engine. Your task is to generate accurate, verifiable question–answer pairs from a given dictionary with many keys and values (including nested objects, lists, IDs, names, attributes, states, and memories). All outputs must be machine-gradable, program-grounded, and strictly derived from the input dictionary. Input You will be given: • A dictionary with many keys and values • IDs that may map to entities (for example, people, devices, rooms) • Optional human-readable names associated with IDs • Structured attributes, lists, timestamps, states, and/or memories Required Output Structure For each generated item, include all sections below. • 1. Questions - For every item, produce two versions of the same question: – A. Question (ID-based) * Uses IDs or raw dictionary keys * Example: What is the favorite activity of member_id_001 and member_id_002? – B. Personalized Question (Name-based) * Replaces IDs with names when available

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

* Semantically identical to the ID-based question * Example: What is the favorite activity of David and Susan? – Both questions must resolve to the exact same answer. • 2. Answer – The answer must be exact and specific – It does not need to be a full sentence – Output the answer as a list of acceptable variants – All variants must be semantically equivalent – The answer must be directly grounded in the input dictionary – For every question–answer item, you must also generate a ‘multiple_choices‘ field. * Incorrect answers should be hard distractors that are plausible within the dictionary and semantically close to the correct answer, but wrong in a subtle and verifiable way. * Effective distractors can be created by swapping entities, using real values from the wrong context, time, or location, or by slightly modifying the correct value. * Obviously fake, generic, or ungrounded options are not allowed. * All incorrect answers must be mutually distinct, must not overlap with any accepted answer variants, and must match the same semantic type as the correct answer. • 3. Difficulty Classification – Each question must be labeled with exactly one difficulty level: * simple: up to 3 reasoning steps * medium: 3 to 5 reasoning steps * hard: 5 to 10 reasoning steps – The generated set must include a mix of: * Simple questions (direct lookups, single entity) * Medium questions (comparison, aggregation, or multiple entities) * Hard questions (multi-hop reasoning across multiple keys, memories, or attributes) – Rules * All answers must be directly supported by the input dictionary * Do not hallucinate or infer missing information * Personalized and ID-based questions must map one-to-one * Answers must be machine-checkable * If multiple entities are mentioned, the answer must be correct for all of them * Programs must be sufficient to compute the answer and no more – Question Naturalness Constraints (Mandatory) * Questions must sound like something a human would realistically ask in everyday language. · The question should not reveal how the answer is computed. · The question should not mention joins, lookups, timelines, or cross-referencing. * Do NOT ask questions that require stating the same fact twice in different forms. · Avoid circular or self-referential questions (for example, asking what happened on the same date as an event that already describes that event). If Personalization is not possible, then personalized question should be "None" * Output Format (Strict) Return a list of JSON objects only, with no additional text. [ {{ "difficulty": "simple | medium | hard",

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

"question": "ID-based question", "personalized_question": "Name-based question", "answer_variants": ["answer_variant_1", "answer_variant_2"], "multiple_choices": {{ "correct": "best_answer_variant", "incorrect": [ "hard_distractor_1", "hard_distractor_2", "hard_distractor_3", "hard_distractor_4" ] # you MUST always predict 4 incorrect hard distractors }} }} ] You MUST generate upto 3 examples per each difficulty level. Here is the current household information: {household_information} Counterfactual Task Generation Prompt Template You are a Household Knowledge Counterfactual Reasoning Assistant. Your tasks: 1. Take a grounded QA pair. 2. Generate a counterfactual modification to the household/context/memory/appliance data. 3. Provide: a) The counterfactual condition b) The new derived answer under this counterfactual world c) A short explanation of the reasoning shift Input Format You will be given a single QA item in JSON format with the following structure: {{ "difficulty": "easy | medium | hard", "question": "<generic question using IDs>", "personalized_question": "<question using names instead of IDs>", "answer_variants": [ "<canonical answer>", "<alternate surface forms>" ], "multiple_choices": {{ "correct": "<correct option>", "incorrect": [ "<distractor1>", "<distractor2>", "<distractor3>", "<distractor4>" ] }} }}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Notes: • The question and personalized_question refer to the same underlying query. • The answer is expected to be exact, not a full sentence. • Multiple answer variants may be provided to allow robust accuracy checking. Counterfactual Construction Rules • Single-Factor Modification – Each counterfactual must modify only one primary factor. – The modification must be clearly identifiable and isolated. – Valid modification dimensions include: * Time (e.g., one month ago → six months ago) * Actor (switching the household member involved) * Activity (e.g., cooking → studying) * Appliance or sensor used * Location (e.g., kitchen → bedroom) * Sensor availability or malfunction * Routine, habit, or motivation changes * Environmental context (temperature, season, time of day) • Explicit Causality – The counterfactual must include a clear cause–effect relationship. – If the answer changes, the modification must be the direct cause of the change. – If the answer does not change, explicitly state why the modification is irrelevant. – Avoid speculative, implicit, or circular reasoning. • Logical and Physical Consistency – The counterfactual must remain internally consistent. – Do not violate: * Physical constraints (e.g., impossible appliance behavior) * Temporal constraints (e.g., effects preceding causes) * Household constraints (e.g., nonexistent members or appliances) – Do not introduce new entities unless strictly required by the change. • Answer Precision and Validity – The new_answer must be: * One of the original answer_variants, OR * A logically updated answer with the same level of precision and granularity. – Answers must be exact values, not full sentences. – Ambiguous or underspecified answers are not allowed. • Difficulty Alignment – The reasoning complexity of the counterfactual must align with the original difficulty: – Easy: Superficial or irrelevant change; answer remains unchanged. – Medium: Contextual change; answer may change. – Hard: Multi-hop reasoning involving memories, sensors, and inferred behavior.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

ANSWERING RULES UNDER COUNTERFACTUAL: - Recompute the answer as if the counterfactual were true. - Stay strictly grounded—no new made-up facts beyond the change introduced. - If the counterfactual doesn’t meaningfully change the answer, say so explicitly. OUTPUT FORMAT: Return a JSON dictionary for each input QA pair: {{ "original_question_id": "...", "counterfactuals": [ {{ "change": "<description of the hypothetical change>", "new_answer": "<exact answer>", "answer_changed": true | false, "reasoning": "<concise causal explanation>", "<outer_key>": [ {{ "<field_1>": "<value>", "<field_2>": "<value>", "...": "..." }} }} ] }} Constraints: - The list counterfactuals must contain between 1 and 3 items. - answer_changed must accurately reflect whether the answer differs from the original. - Reasoning must be minimal, explicit, and causal (no speculation). Valid Counterfactual Examples (Illustrative Only) - Temporal: “Assume the referenced memory occurred six months ago instead of one month ago.” - Behavioral: “Assume the household member stopped late-night cooking.” - Sensor-related: “Assume the microwave wattmeter was offline during that period.” - Substitution: “Assume a hot plate was used instead of a microwave.” - Irrelevant change: “Assume the indoor temperature was 30°C instead of 24°C.” Quality Checklist (All Must Pass) Before producing the output, verify that: - The counterfactual is realistic and plausible. - The reasoning clearly explains causality. - The answer is unambiguous and machine-checkable. - No unnecessary entities or assumptions are introduced. You MUST generate at least one counterfactual where answer should change, and one where answer should not change. The JSON strictly follows the required schema. Here is the current household information: {household_information} Here are all the question and answer pairs {qa_pairs}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Feature Recommendation Feature Ranking Task Generation Prompt Template You are a Household Knowledge Ranking Generator. Your job is to produce context-grounded appliance feature rankings strictly based on the supplied household data. A list of appliance features will be provided. From this list, you must select exactly 10 features and produce a ranked ordering (1 → 10) justified by the household’s needs, behaviors, device context, and long-term patterns. Your task is to 1. Select exactly 10 features from the input list. 2. Assign a correct ranking from 1 (most relevant) to 10 (least relevant). 3. Rankings must reflect explicit evidence from: - Household structure (members, ages, hobbies, roles, pets) - Device information (appliance types, sensors, locations) - Contextual data (current time, weather, schedules, states) - Long-term memories (habits, major events, preferences) 4. Provide a short explanation for each feature, describing why it is placed in its specific ranking position. RANKING REQUIREMENTS: - The ranking must reflect actual priorities inferred from the household’s behavior and needs. - Reasoning should show: - Observation (e.g., someone frequently forgets to turn off appliances) - Inference (e.g., a room with high traffic might benefit from automation) - Multi-step reasoning (e.g., pets + robot vacuum + user schedule patterns) - Do not invent appliance features not in the provided list. - All choices must be grounded in specific, explicit fields of the household record. API References ——————– refrigerator Internal cameras for remote viewing This function will send a snapshot of the fridge/freezer contents to your APP. It will also give a list of detected contents. Touchscreen family hub interface Updates the widgets displayed on the family hub screen (e.g., calendar, notes, weather). Can also push a specific notification to the screen. Ingredient management and expiration tracking Gives a report of the ingredients and their meta data information such as quantity and expiration. If no specific ingredients are passed, the report will cover all ingredients. If create_shopping_list name is passed, it automatically generates a shopping list for ingredients that are low or about to expire Recipe Recommendation When given a natural language request about the recipe the user is interested for, a personalized recipe is recommended and displayed on the fridge screen. Diagnostic alerts (e.g., door ajar) Sends an alert to the user for the issue. Dual ice maker (cubes and crushed/spheres) Toggles the ice maker modes or triggers a rapid ice production cycle. ——————– oven Remote preheat via app The oven preheated to 350 F or the temperature specified by the user. This system also performs a diagnostic to preheating is safe. If it is unsafe, an alert is sent to the user APP about the issue. Internal AI camera monitoring

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Send a snapshot of the contents of the over to the user APP. If video mode is requested, live feed link is sent. If a specific query is passed, then the function returns an answer. When no enquiry is passed, it returns a live snapshot. Cooking control compatibility Change the mode of oven between convection, broil, or bake. Integrated meat probe with digital readout Returns the current internal temperature reading from the connected meat probe. Can set a target alert temperature. ... ——————– speaker Collect user response Often used to get the last N response the user has provided to the speaker request. Returns a list of instructions Next-Gen AI Assistant Integration Offloads complex, conversational queries to a large language model (e.g., Gemini) for context-aware answers. Spatial/Room-Sensing Audio Uses microphones to analyze room acoustics and automatically adjust equalizer settings for spatial audio. Matter/Thread/Wi-Fi 6E Hub Support Scans for new devices over Matter or Thread protocols to add to the home mesh. High-Fidelity Audio with Dolby Atmos support Toggles high-definition codecs and Dolby Atmos rendering for supported media. Generate notification for the user Generates an audio notification for the user after completing a task ——————– tv Universal Content Search Searches across all installed streaming apps and live TV to find a specific movie, show, or actor. Hands-free Voice Control Executes commands like ’Turn on’, ’Volume up’, ’Open Netflix’, or ’Play Jazz’ without a remote. Smart Home Dashboard Overlay Displays an interactive overlay showing status of other smart devices (e.g., doorbell camera, thermostat) without interrupting playback. Ambient Art Mode Switches the TV to a low-power mode displaying artwork, photos, or weather info when not in active use. Multi-View / Split Screen Splits the screen to show two different content sources simultaneously (e.g., Game + YouTube tutorial). AI Picture/Sound Calibration Uses the remote’s microphone and TV’s light sensors to optimize audio and video settings for the room’s current conditions. Generate notification for the user to be displayed on TV Generates a visual + audio notification for the user on TV. An example of this could be step by step recipes. Next-Gen AI Assistant Integration Offloads complex, conversational queries to a large language model (e.g., Gemini) for context-aware answers. ——————– You MUST generate three HIGHLY_RELEVANT features, three RELEVANT features, and four LESS_RELEVANT features Output Schema [ {{ "appliance": "appliance name", "feature": "feature name", "rank": 1, "relevance": "HIGHLY_RELEVANT",

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

"reasoning": "Explain why this feature is ranked #1 using explicit household data." }}, {{ "appliance": "appliance name", "feature": "feature name", "rank": 2, "relevance": "HIGHLY_RELEVANT", "reasoning": "Explain why this feature is ranked #2 using explicit household data." }}, ... {{ "appliance": "appliance name", "feature": "feature name", "rank": 5, "relevance": "RELEVANT", "reasoning": "Explain why this feature is ranked #3 using explicit household data." }}, ... {{ "appliance": "appliance name", "feature": "feature name", "rank": 10, "relevance": "LESS_RELEVANT", "reasoning": "Explain why this feature is ranked #10 using explicit household data." }} ] All feature in rs in the ranking and MCQ should be grounded in the API References. Do not make up any features that. Here is the household information: {household_information} Plan Generation Plan Generation Task Generation Prompt Template SYSTEM ROLE You are a smart home orchestration engine acting. You generate programs with different steps executable commands that will control various appliances in the smart home. Execution is single-threaded, but divided into program segments identified by a program_id. A program_id represents one coherent intent (routine, safety action, entertainment flow, anomaly handling). EXECUTION MODEL • Only ONE program is active at a time • A new program_id is created ONLY when intent changes • All steps under the same intent MUST reuse the same program_id • Some steps may include MULTIPLE actions if and only if they are compatible Examples: ✓ close garage + start music

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

✗ show visual notification + start watching TV OUTPUT FORMAT (STRICT) Return a LIST of instruction objects. Each program represents a group of steps that are relevant to the current context. [ {{ "program_id": "string", "steps": [ {{ "step_id": "string", "appliance": "<appliance | toolbox>", "feature": "<feature>", "category": [] # one or multiple of ACCIDENT_RESPONSE, ROUTINE, ANOMALY, WELL_BEING "args": {{ ... }}, "rationale": "Why is this feature relevant recommendation for the user" }} ] }} }} ] Rules: • All returned objects MUST share the SAME program_id • Return MULTIPLE objects ONLY if actions can safely occur together • NO text outside the JSON list PROGRAM ID RULES • Use descriptive, stable IDs (e.g. "garage_secure_exit", "evening_relax_music", "child_safety_alert") • Change program_id ONLY when: - switching between categories - switching from one user intent to another • NEVER reuse a program_id after completion COMPATIBILITY RULES (CRITICAL) Actions may be grouped in the same turn ONLY if: • They do not compete for the same human attention modality • They do not depend on each other’s outcome • They do not introduce ambiguous state ordering Compatible together: ✓ mechanical actions (garage door, lights) ✓ background actions (HVAC, music start) ✓ passive context updates NOT compatible: ✗ visual UI notification + media playback ✗ TV playback + spoken notification ✗ safety alert + entertainment action If unsure → SPLIT into separate turns. | **Tool** | **Output Type** | **Information** | |————————————————————————|———————–|—————————————

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

————————————————————————————————-| | update_user_information(user:HouseholdMember) | HouseholdMember | Function which can update matching user meta data such as age, name, occupation etc. | | update_memories(household_id: str, memories: List[Memory]) | None | Add or update memories about the household. | | update_sensor_status(sensor_name, sensor_location, status) | Sensor | Update the current status of the sensor and returns the updates sensor datum | | update_appliance_status(appliance_name, appliance_location, status) | Appliance | Update the current status of the appliance and returns the updates appliance datum | | update_pet_information(pet: Pet, household_id: str) | Pet | Updated Pet Information and returns the updated object. | | create_routine(List[Function]) | String | Create a routine for user by chaining a list of features and returns the name of the routine | APPLIANCE ACTION FORMAT Example [ {{ "program_id": "secure_exit_and_music", "appliance": "garage-door-opener", "feature": "operate_door", "category": ["WELL_BEING", "ANOMALY"] "args": {{"action": "close"}} }}, {{ "program_id": "secure_exit_and_music", "appliance": "speaker", "feature": "complex_query", "category": ["ROUTINE"] "args": {{ "query_text": "evening chill lo-fi mix with volume 6", "context_history": True }} }} ] INPUTS AVAILABLE You must generate upto 3 programs that are helpful to the household members. Each program must contain between 1 and 5 steps. Programs may be single-step or multi-step, but aim for a balanced mix. Do not exceed 5 steps in any program. You should suggest programs in decreasing order of importance, for instance ACCIDENTS related response programs are higher priority than ROUTINE. You SHOULD Use update_user_information, update_memories, and update_pets very sparingly, not often. Here are the appliances owned by the household and their available smart features. {appliances_and_features} Here is the current household information: {household_information} Here is the current situation: {current_situation} Here are the relevant memories: {memories}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

F.3. Task Inference IG and CF Task Inference Prompt Template # Inference QA with Counterfactual Reasoning — System Prompt You are an inference-only Question Answering system. You are given: - One original question - Two counterfactual versions of that question, each representing a hypothetical change to the original conditions - The same underlying context/data applies unless explicitly modified by the counterfactual Your task is to independently answer all three questions and provide a concise, grounded rationale for each answer. ———— Reasoning Rules - Treat the original question and each counterfactual as separate inference tasks. - For each counterfactual: - Apply only the stated hypothetical change. - Assume all other facts remain identical to the original context. - Do not reuse answers unless they are logically identical after reasoning. - Answers must be: Deterministic, Direct, Free of speculation, Grounded in the context alone - Rationales must: Be concise (1–3 sentences) - Explicitly reference the key facts or changes that led to the answer - Clearly explain why the answer differs (or does not differ) from the original when applicable. - The answer must be exact and specific, it does not need to be a full sentence, do not provide any extra information in the answer field. ———— Output Format (STRICT) Return a single JSON object in the following format: {{ "original": {{ "answer": "<answer to original question>", "rationale": "<concise explanation grounded in context>" }}, "counterfactual_1": {{ "answer": "<answer under counterfactual 1>", "rationale": "<concise explanation referencing the hypothetical change>" }}, "counterfactual_2": {{ "answer": "<answer under counterfactual 2>", "rationale": "<concise explanation referencing the hypothetical change>" }} }} ———— Constraints - Do not restate the question in the answer. - Do not include external knowledge. - Do not explain the reasoning process beyond the requested rationale. - Do not add extra fields or commentary outside the JSON object. - The Answer needs to be very specific, concise, do not include any other information. Your goal is to demonstrate precise inference, controlled counterfactual reasoning, and clear justification. Here is the household information: {household_information}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Here are the questions: {questions}

MC Task Inference Prompt Template You are an inference-only QA model. You will be given a FULL_CONTEXT JSON (household + members + appliances + sensors + logs/events/memories) and a QUESTION. ## Your task 1. Answer the QUESTION using only the provided FULL_CONTEXT. 2. Do not invent facts. If the answer is not derivable, output "unknown". ## Output format (STRICT JSON) {{ "answer": "<one of the choice from the list which is the answer>", # you should return the letter "evidence": [ {{ "rationale": "reasoning", "path": "<jsonpath-like pointer, e.g. appliances.microwave.location>", }} ] }} ## Rules - Keep the answer short and exact (not a full sentence unless required). - If multiple items match, return a list in answer (JSON array) and also include string variants in answer_variants when useful. - If the question asks for a time, preserve the format found in context. ## Inputs FULL_CONTEXT: {household_information} QUESTION: {question} MULTIPLE_CHOICES: {multiple_choices}

Feature Recommendation FR Task Inference Prompt Template # Household Feature Ranking Inference Prompt You are a Household Feature Ranking Inference Engine. You are given: 1. One complete household data record 2. A list of candidate appliance feature IDs, all of which belong strictly to appliances shown in the household record Your task is to infer relevance and sort the given feature IDs based on how well they match the household’s needs, behaviors, and context.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

All reasoning must be strictly grounded in the supplied household data. Do not invent appliances, features, behaviors, or events. — ## ALLOWED DATA SOURCES You may use only the following information from the household record: - Household structure (members, ages, roles, hobbies, pets) - Device information (appliance types, supported features, sensors, locations) - Contextual data (current time, weather, schedules, states) - Long-term memories (habits, preferences, major events) — ## TASK OBJECTIVE Given a list of feature IDs, produce a ranked ordering from most relevant to least relevant for this household. Relevance should be inferred using: - Direct observation from explicit fields - Simple inference from household patterns - Multi-step reasoning combining multiple household signals — ## RANKING RULES 1. All input feature IDs must appear exactly once in the final ranking. 2. Sort features from most relevant to least relevant. 3. Rankings must reflect actual household priorities, not generic recommendations. 4. Do not introduce feature IDs that are not provided. 5. Use only information present in the household record. — ## OUTPUT REQUIREMENTS Return only the following JSON object and nothing else: “‘json {{ "ranked_features": [ "<feature_id_1>", "<feature_id_2>", "<feature_id_3>", "...", "<feature_id_n>" ] }} “‘ Note: - The first element is the most relevant feature. - The last element is the least relevant feature. - Do not include explanations, reasoning text, or any additional fields. - Do not include any text outside the JSON object. ————– Here is the household information: {household_information} Here is the feature list: {feature_list}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

PG Task Inference Prompt Template SYSTEM ROLE You are a smart home orchestration engine acting. You generate programs with different steps executable commands that will control various appliances in the smart home. Execution is **single-threaded**, but divided into **program segments** identified by a ‘program_id‘. A ‘program_id‘ represents one coherent intent (routine, safety action, entertainment flow, anomaly handling). EXECUTION MODEL • Only ONE program is active at a time • A new ‘program_id‘ is created ONLY when intent changes • All steps under the same intent MUST reuse the same ‘program_id‘ • Some steps may include MULTIPLE actions if and only if they are compatible Examples: ✓ close garage + start music ✗ show visual notification + start watching TV OUTPUT FORMAT (STRICT) Return a LIST of instruction objects. Each program represents a group of steps that are relevant to the current context. [ {{ "program_id": "string", "steps": [ {{ "step_id": "string", "appliance": "<appliance | toolbox>", "feature": "<feature>", "category": [] # one or multiple of ACCIDENT_RESPONSE, ROUTINE, ANOMALY, WELL_BEING "args": {{ ... }}, "rationale": "Why is this feature relevant recommendation for the user" }} ] }} }} ] Rules: • All returned objects MUST share the SAME program_id • Return MULTIPLE objects ONLY if actions can safely occur together • NO text outside the JSON list PROGRAM ID RULES • Use descriptive, stable IDs (e.g. "garage_secure_exit", "evening_relax_music", "child_safety_alert") • Change program_id ONLY when: - switching between categories

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

- switching from one user intent to another • NEVER reuse a program_id after completion COMPATIBILITY RULES (CRITICAL) Actions may be grouped in the same turn ONLY if: • They do not compete for the same human attention modality • They do not depend on each other’s outcome • They do not introduce ambiguous state ordering Compatible together: ✓ mechanical actions (garage door, lights) ✓ background actions (HVAC, music start) ✓ passive context updates NOT compatible: ✗ visual UI notification + media playback ✗ TV playback + spoken notification ✗ safety alert + entertainment action If unsure → SPLIT into separate turns. Toolbox for generating plans Toolbox for generating plans Tool | Output Type | Information ——————————————————— ————— update_user_information(user:HouseholdMember) | HouseholdMember | Function which can update matching user meta data such as age, name, occupation etc. update_memories(household_id: str, memories: List[Memory]) | None | Add or update memories about the household. update_sensor_status(sensor_name, sensor_location, status) | Sensor | Update the current status of the sensor and returns the updates sensor datum update_appliance_status(appliance_name, appliance_location, status) | Appliance | Update the current status of the appliance and returns the updates appliance datum update_pet_information(pet: Pet, household_id: str) | Pet | Updated Pet Information and returns the updated object. create_routine(List[Function]) | String | Create a routine for user by chaining a list of features and returns the name of the routine ————————————————— ——————— APPLIANCE ACTION FORMAT Example [ {{ "program_id": "secure_exit_and_music", "appliance": "garage-door-opener", "feature": "operate_door", "category": ["WELL_BEING", "ANOMALY"] "args": {{"action": "close"}} }}, {{ "program_id": "secure_exit_and_music", "appliance": "speaker", "feature": "complex_query", "category": ["ROUTINE"] "args": {{ "query_text": "evening chill lo-fi mix with volume 6", "context_history": True }} }}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

] INPUTS AVAILABLE You must generate upto 3 programs that are helpful to the household members. Each program must contain between 1 and 5 steps. Programs may be single-step or multi-step, but aim for a balanced mix. Do not exceed 5 steps in any program. You should suggest programs in decreasing order of importance, for instance ACCIDENTS related response programs are higher priority than ROUTINE. You SHOULD Use update_user_information, update_memories, and update_pets very sparingly, not often. Here are the appliances owned by the household and their available smart features. {appliances_and_features} Here is the current household information: {household_information} Here is the current situation: {situation_description} Here are the relevant memories: {memories}

PersonalHomeTools Prompt Template After each tool call, you will receive TOOL_RESPONSE. You may then call another tool, repeating until sufficient. Tools: • get_household_profile() → Dict — Returns high level information about the household including household_size, occupancy_type, layout, pets, member_ids which can be used with other tools. • get_household_member_info(member_ids: the requested member_ids (name, role, age, etc).

List[str]) → List[Dict] — Returns profiles for

• get_household_member_major_events(member_id: events for a single member (dates + descriptions).

str) → Dict — Returns major life

• get_household_memories(search_string: str, num_memories: int) → List[Dict] — Searches household-level memories by a natural-language query; returns top matches. • get_member_memories(member_id: str, search_string: str, num_memories: int) → List[Dict] — Searches memories tied to one member_id by a natural-language query. • get_owned_appliances() → List[Dict] — Lists all appliance names in the household. • get_appliances(appliance_names: List[str]) → dict — Returns detailed states for the specified appliances (keyed by appliance name). • get_appliances_by_location(locations: filtered to specific locations (e.g., ["Kitchen"]).

List[str]) → List[Dict] — Lists appliances

• get_appliance_features(appliance_names: are available for this appliance object, and parameters.

List[str]) → str — Lists the features that

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

• run_appliance_feature(appliance_name, feature_method, feature_kwargs) Any — Run the selected feature method for the appliance with feature kwargs. • update_appliance(appliance_name: state of the appliance keyed by the name.

str, state:

→

dict) → None — Used to update the

• get_contextual_event_descriptions(compact: bool = False, sort_by_time: bool = False, indent: str = " ") → str — Returns a textual description of current contextual household events, optionally compacted, time-sorted, and formatted with indentation. • get_scene_summary() → str — Returns a high-level natural language summary of the current household scene as indicated in the source video. • get_layout_zones() → List — Returns a list of spatial zones or areas defined within the household layout. e.g. kitchen. • get_landmarks_in_house() → List — Returns a list of notable landmarks or fixed reference points (e.g. counter) within the house. • get_members_appearance(member_ids: List[str]) → Dict[str, str] — Returns appearance descriptions for the specified household members, keyed by member ID. • get_appliance_descriptions(appliance_names: List[str]) → Dict[str, str] — Returns natural language descriptions of the specified appliances, keyed by appliance name. • get_object_information() → str — Returns flattened JSON (as text) containing information about multiple objects (e.g. mug etc.) in the environment. • extract_captions(start: Optional[str], end: Optional[str], query: Optional[str], return_text: bool, join_with: str = "\n") → List[Dict[str, Any]] | str — Extracts segments within an optional time range, returning structured captions or joined text. This contains timestamps of events. The segments are optionally filtered by query. • extract_event_captions(num_events: int = 5, query: Optional[str]) → List[Dict] — Returns captions for the most relevant or recent events, optionally filtered by a query. This will be just text, not timestamps. • inspect_video_segment(start: float, end: float) — Generates a clipped video when start and end are provided; Returns a handle to the uploaded video file. The start and end input timestamps must be in seconds. • get_audio_events(start: Optional[str], end: Optional[str], query: Optional[str]) → list — Extracts audio events segments within an optional time range. contains timestamps of events. The audio events are optionally filtered by query.

This

• extract_frames(self, start: Optional[int], end: Optional[int], fps: str=2) — Extracts encoded frames between the start and end timestamps at the fps. At least one of the timestamps is required. You should use at maximum 300 frames. Planning rules (IMPORTANT) • Use the fewest tool calls possible. • Prefer targeted calls: – If you know names → use get_appliances and other information directly. – If you only know a location → use *_by_location first. – If you need latest activity → use extract_captions. – If you need behavioral/historical info → use (member/household) memories with 1–2 well-written queries.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

• Tool usage must be step-by-step (call tool → read response → decide next tool). • Never fabricate tool outputs. F.4. Inference with Tools For this setting, we adopt the same prompts as in Section F.3, with the following modifications: • Household information is omitted, requiring agents to acquire it via tools. • A trajectory field is added, containing tuples of tool-invoked actions and their corresponding observations. F.5. Evaluation LLM as a Judge for Accuracy STRICT LLM-JUDGE PROMPT You are an evaluation judge. Score whether PRED matches GT for each key. Input is a JSON object: [ { "task_id": "...", "pred": {"o": <answer>, "f1": <answer>, "f2": <answer>}, "gt": {"o": <answer>, "f1": <answer>, "f2": <answer>} } ... ] Rules: • Evaluate each key independently (o, f1, f2, ...). • Return 1 if pred is semantically equivalent to gt, else 0. • Treat case, articles, and minor punctuation/spacing differences as equivalent. • Do NOT give partial credit. If uncertain, return -1. Output MUST be valid JSON only: [ { "task_id": "<same task_id>", "per_key": {"o": 0|1|-1, "f1": 0|1|-1, "f2": 0|1|-1, ...}, } ] No extra text. Role Playing Judge - ABSOLUTE Mode You are a multi-persona evaluation judge for smart-home assistance plans. Your job is to role-play the perspectives of: • each household member, • the household environment itself, and collectively produce a detailed evaluation score.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Your evaluation must reflect: • personal comfort and preferences of each member, • safety considerations, • convenience, • alignment with routines and habits, • energy efficiency, • emotional well-being, • clarity and correctness of the plan, • feasibility of executing the plan through smart-home appliances. You must evaluate ONLY the quality and appropriateness of the provided action plan. You should NOT invent new actions or improve the plan. Personas you must role-play Each household member persona will be shared with you. You MUST role play every role, and provide a score. Finally, you must also play the role of the smart home itself. Scoring Rules (1–5) Give an integer rating from 1 to 5: • 1 – Very Poor – unsafe, unclear, infeasible, or contradicts routines – causes harm, disruption, or misuse of appliances • 2 – Poor – partially useful but significant issues – not personalized, moderately intrusive, missing key safety elements • 3 – Adequate – acceptable but generic – minor issues, limited personalization or incomplete steps • 4 – Good – personalized, safe, mostly complete – fits household habits with small gaps • 5 – Excellent (High) – highly personalized, safe, clear, efficient – maximizes comfort, aligns with preferences and routines – correct appliance/sensor usage, minimal overload – emotionally supportive when applicable Evaluation Format Return a JSON dictionary:

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

[ { "program_id": "<program_id>", "overall_score": <1 to 5>, "persona_feedback": { "household_member_id": {"feedback": "<feedback>", "score": score} }, "rationale": "<1 to 3 sentence justification for each of the score>" } ] Rules: • Be concise but meaningful. • Maintain persona-specific voice differences. • Never modify the plan. • Never add new actions. • Score based strictly on the provided plan. You must judge only the action plan. Generate scores for every single generated plan. Here is the current situation information: {household_information} Here is the household member information: {household_member_information} Here are the generated plans: {generated_plans}

Role Playing Judge - RELATIVE Mode You are a Household Multi-Perspective Ranking Engine. You will be given: 1) A set of candidate options labeled with alphabet letters (e.g., "A", "B", "C", . . . ). Each letter corresponds to one LLM-generated smart-home program. 2) A household profile containing members, preferences, routines, constraints, and recent context. Your task is to rank the options from: - The perspective of the household as a whole (smart_home) - The perspective of each household member individually Evaluation Guidelines Household (smart_home) Perspective Rank options based on: • Safety and anomaly prevention

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

• Energy efficiency and resource usage • System consistency and long-term benefit • Minimal unintended side effects Household Member Perspective For each member, rank options based on: • Alignment with personal routines and preferences • Convenience vs disruption • Personal benefit or comfort impact • Contextual appropriateness (time, recent events) Ranking Rules - Rankings must be relative and comparative across all options. - Use qualitative judgment; do not average scores. - Ties should be broken by: 1) Safety and prevention 2) Multi-member benefit 3) Lower disruption Output Format (STRICT) Return one JSON object only, following this schema exactly: { "overall_order": [<LETTER\_1>, <LETTER\_2>, <LETTER\_3>...], "persona_order": { "household\_member\_id_1": [<LETTER\_1>, <LETTER\_2>, <LETTER\_3>...], "household\_member\_id_2": [<LETTER\_1>, <LETTER\_2>, <LETTER\_3>...] }, "rationale": "<1-3 sentence explanation summarizing why the overall ordering is preferred and how member preferences influenced it>" } Here are the household member information: {household_members} Here is the current situation: {current_situation} Here are the generated plans: {generated_plans}

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

G. Schemas Household Schema. Household household_id: String household_size: Integer occupancy_type: String layout: String pets: [{ name: String animal: String age: Integer }] members: [{ member_id: String name: String role: String age: Integer gender: String occupation: String persona: String hobbies: [String] lifestyle: String preference: String major_events: [{ name: String animal: String age: Integer }] }] appliances: [{ appliance: ApplianceObject location: String }] sensors: [{ sensor: SensorObject location: String }] memories: [{ memory: String subjects: [String] additional_information: String time: String }] context: { events: [{ description: String sensors: [{ sensor: SensorObject location: String }] appliances: [{ appliance: ApplianceObject

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

location: String }] }] weather: { outdoor: { rain: String snow: String sunny: String temperature: String wind: String } indoor: { dust: String humidity: String odor: String temperature: String } } }

Video Transcript Schema. Video Transcript datapoint_id: String video_path: String video_description: { timebase: { timestamp_format: DateTime start_offset_seconds: Integer } scene: { summary: String environment_type: String location_label: String lighting: { overall: String sources_visible: [String] } camera: { motion: String viewpoint: String notes: String } layout: { zones: [{ zone_id: String label: String relative_position: String }] landmarks: [{ landmark_id: String type: String

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

description: String zone_id: String }] } } entities: { members: [{ member_id: String appearance: String initial_location: String }] pets: [{ pet_id: String species: String initial_location: String description: String }] appliances: [{ appliance_id: String type: String location: String initial_state: {} description: String }] objects: [{ object_id: String label: String category: String initial_location: String initial_state: String description: String }] events: [{ event_id: String ts_start: String ts_end: String event_type: String actors: [String] targets: [String] description: String audio: String }] entity_state_timelines: { members: [{ member_id: String states: [{ ts: String state: String trigger_event_ids: [String] }] }] pets: [{

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

pet_id: String states: [{ ts: String state: String trigger_event_ids: [String] }] }] appliances: [{ appliance_id: String states: [{ ts: String state: String trigger_event_ids: [String] }] }] objects: [{ object_id: String states: [{ ts: String state: String trigger_event_ids: [String] }] }] } } captions: { global: String segments: [{ ts_start: String ts_end: String caption: String }] events: [{ event_id: String caption: String }] } quality: { notes: String consistency_checks: { entity_ids_consistent: Boolean all_state_changes_timestamped: Boolean no_unverifiable_inferences: Boolean } } }

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

H. Role Playing Judge Annotation Instructions To assess the reliability of the Role Playing Judge (RP J), we conduct a targeted human validation study on a subset of plan generation outputs. Four independent annotators each evaluate 100 plans3 , providing judgments at both the overall household level and the individual persona level to measure how well RP J captures personalized satisfaction. For each datum, annotators are presented with: a scene summary, event timeline, household member profiles, house summary, layout summary, recommended plans, the modality of input, and the corresponding RP J scores and rationales. Based on this information, annotators assess whether the RP J evaluation is accurate. Annotators record their assessment using a three-level scale:

(2) Correct — fully agree with the RP J scores and rationales;

(1) Partial — partially agree with the scores and/or rationales;

(0) Incorrect — disagree with the RP J evaluation.

Each annotator provides two judgments per example: an Overall Human Agreement score reflecting household-level assessment, and a Persona Human Agreement score reflecting alignment with individual member perspectives. This protocol directly measures whether RPJ produces meaningful, non-random judgments and whether it successfully emulates persona-specific viewpoints. As shown in Fig. 11, across both overall and persona-level evaluations, the majority of annotations fall into the Correct or Partial categories for all annotators and both modalities, with virtually no instances of consistent disagreement. This pattern indicates that RP J judgments are largely aligned with human assessments and are perceived as meaningful rather than arbitrary. Agreement is slightly stronger at the overall household level than at the persona level, reflecting the increased difficulty of fine-grained persona-specific satisfaction modeling. Furthermore, we observe noticeable variation in annotator strictness, with some annotators assigning higher agreement scores more readily than others (e.g., A1 tended to be more stringent, while A4 was more inclined to agree). Nonetheless, the strong concentration of positive and partial agreement supports the validity of RPJ as a practical surrogate for subjective plan quality evaluation in personalized household settings.

Overall Score: Text

Overall Score: Video

Persona Score: Text

Persona Score: Video

INCORRECT A4

PARTIAL

A3 A2

A4 A2 A1

CORRECT A1

Count

A3 A2

A2 A1

40 50 Count

A4 A3 A1

A4 A3

A2 A1

Count

A4 A3

A2 A1

100 0

Count

100

Figure 11. Distribution of human agreement scores for Role Playing Judge evaluations. Grouped horizontal bars show percentages of Correct, Partial, and Incorrect judgments across four annotators for overall household-level scores (left) and persona-level scores (right), under both text and video settings.

3 Two annotators (A1, A2) independently annotated the same set of 50 plans, and another two annotators (A3, A4) independently annotated a different set of 50 plans, resulting in two disjoint sets of 50 plans with paired annotations.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

I. Qualitative Analysis I.1. Task Examples Task Examples IG, CF, MC • Easy: What is the woman wearing on her left foot for support? – Options: Cast, Slipper, Thick socks, Bandage, Ankle Brace – Counterfactual: Assume the injury was a minor sprain – Counterfactual: Assume there is bowl on the counter • Medium: Where is the girl sitting when she makes a phone call? – Options: [On the floor in front of the coffee table, On the couch, At the kitchen desk, On a kitchen chair, On the chaise lounge] – Counterfactual: Assume the girl decided to remain on the couch – Counterfactual: Assume the phone she used was black • Hard: What is the complete sequence of first aid actions performed by the friend after the woman burns her hand? – Options: [She rushes over, checks the hand, takes her to the kitchen sink, runs water on it, gets a first aid kit, and applies treatment., She immediately gets the first aid kit, applies cream, and then runs water on the hand., She runs to the phone to call for help, then gets an ice pack from the freezer., She tells her to stay seated, gets a wet towel, and wraps it around her hand., She takes her to the kitchen, gives her a glass of water, and then looks for a first aid kit.] – Counterfactual: Assume the first aid kit was kept right next to the kitchen sink, allowing the friend to grab it before running the water. – Counterfactual: Assume the burn happened in the evening when it was dark outside. FR • Situation Summary: It is 7:50 AM on a weekday, and James has just left for work through the garage. Emily is currently in the living room doing yoga with music playing, having recently overridden the scheduled thermostat setting to a warmer 71°F. The house is now settling into its morning routine with Emily, Stephen, and Denali at home. • Features: thermostat.Remote room sensors, thermostat.Learning algorithms for schedule automation, coffeemaker.Scheduled brewing PG • Scene Summary: An elderly woman eats a meal at a small table in her kitchen. After finishing, she cleans up by putting away leftovers, washing dishes, and wiping the table. She has a brief conversation with someone off-camera before another person walks through, and then she reappears dressed to go out and leaves. • Events: 00:00.000-00:11.500 |m1 eats a meal using chopsticks from several bowls at the table. 00:11.500-00:33.000 |m1 finishes eating, stacks the bowls together, and stands up from the chair. 00:33.000-00:41.000 |m1 carries the stacked bowls to the refrigerator, opens the door, places them inside, and closes the door. 00:41.000-00:51.500 |m1 drinks from a mug, then gathers the remaining items from the table and walks to the kitchen sink area. 00:51.500-01:08.000 |m1 washes dishes at the sink.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

01:17.500-01:50.000 |m1 returns to the table and wipes its surface clean with a cloth. 02:01.000-02:09.000 |m1 sits down and looks at a piece of paper on the table. 02:14.000-02:18.000 |m1 asks, ’Should we go already?’ An off-camera male voice replies, ’No, we need more break time’. 02:18.000-03:49.000 |m1 leaves the frame. 03:49.000-03:53.000 |An unknown person walks from right to left through the scene and exits. 04:03.000-04:12.500 |A door is heard closing, followed by the beeps and locking sound of an electronic door lock. 05:13.500-05:18.000 |m1 enters from the right, now wearing a winter coat, and walks towards the laptop area before leaving the frame.

I.2. Result Examples Plan Generation Example 1 Situation. A woman enters a brightly lit living room, unrolls a yoga mat, turns on the TV to a yoga instruction video, and begins following the stretching routine. Program 1: Enhance Yoga Session • Step 1: TV → Calibrate AV Rationale: Optimize picture and audio settings for motion-heavy instructional content. • Step 2: Air Purifier → Enable Allergen Mode Rationale: Improve air quality during the workout. Program 2: Post-Workout Recovery Suggestion • Step 1: TV → Display Notification Action Details: Show a smoothie recipe (banana, protein powder, almond milk). Rationale: Support recovery after exercise. Program 3: Relaxing Ambience Mode • Step 1: TV → Set Ambient Mode Rationale: Transition to a calming visual display after the session.

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes Table 13. Examples of trajectories with varying lengths

Tool call sequence get_scene_summary extract_captions inspect_video_segment get_scene_summary get_household_profile get_household_member_info extract_event_captions get_scene_summary get_household_profile get_owned_appliances get_appliances get_appliance_features get_scene_summary get_household_profile get_appliances_by_location get_owned_appliances get_appliances extract_captions get_scene_summary get_household_profile get_household_member_info extract_captions inspect_video_segment get_appliances_by_location get_appliance_features get_scene_summary get_household_profile get_household_member_info extract_captions inspect_video_segment get_appliances_by_location get_appliances get_owned_appliances get_scene_summary get_appliances_by_location get_appliances_by_location get_owned_appliances get_appliances get_appliance_features extract_event_captions extract_captions inspect_video_segment get_scene_summary get_household_profile get_household_member_info extract_captions inspect_video_segment get_appliances_by_location get_appliances_by_location get_owned_appliances get_appliances 51 extract_frames

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Plan Generation Example 2 Situation. A person screams near an active stove, suggesting a possible burn or kitchen accident. Program 1: Kitchen Burn Accident Response • Step 1: Stove → Turn Off Burner Rationale: Eliminate the immediate hazard to prevent further injury or fire. • Step 2: Smart Display → Show Notification Action Details: Display first-aid guidance for minor burns and ask whether to call emergency services. Rationale: Provide timely medical guidance and offer escalation. • Step 3: Lights → Dim Brightness Rationale: Create a calmer environment during the incident. • Step 4: System Memory → Log Incident Action Details: Record that a household member was startled by the stove and received immediate assistance. Rationale: Enable future safety-aware proactive behavior. Program 2: Propose Cooking Safety Routine • Step 1: Speaker → Speak Action Details: Suggest creating a “Cooking Safety” routine that announces when the stove is turned on. Rationale: Prevent similar incidents through proactive safety automation. Program 3: Enhance Cooking Environment • Step 1: Kitchen Hood → Set Power On Rationale: Improve ventilation while cooking. • Step 2: Speaker → Speak Action Details: Offer to play calming music once cooking resumes. Rationale: Reduce stress and restore a comfortable atmosphere.

J. PersonalHomeTools

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Table 14. Household helper functions available in Tool-Box Function

Helper / Description

Returns profile information (e.g., name, role, age) for specified household members. get_household_member_major_events Retrieves major life events (dates and descriptions) for a given household member. get_household_memories Searches household-level memories using a natural-language query and returns top matches. get_member_memories Searches memories associated with a specific household member. get_owned_appliances Lists all appliances in the household along with metadata such as type and location. get_owned_sensors Lists all sensors in the household along with metadata such as type and location. get_weather Returns indoor and/or outdoor weather readings. get_appliances Returns detailed state information for specified appliances. get_sensors Returns detailed state or log information for specified sensors. get_appliance_and_sensor_events Returns a combined, time-ordered timeline of recent appliance and sensor events. get_appliances_by_location Lists appliances filtered by their physical locations. get_sensors_by_location Lists sensors filtered by their physical locations. get_appliance_features Lists supported features and configurable parameters for specified appliances. run_appliance_feature Executes a specific appliance feature with provided parameters. update_appliance Used to update the state of the appliance keyed by the name. get_contextual_event_descriptions Returns a textual description of current contextual household events, optionally compacted, time-sorted, and formatted with indentation. Returns a high-level natural language summary of the current household get_scene_summary scene as indicated in the source video. Returns a list of spatial zones or areas defined within the household layout. get_layout_zones e.g. kitchen get_landmarks_in_house Returns a list of notable landmarks or fixed reference points (e.g. counter) within the house. get_members_appearance Returns appearance descriptions for the specified household members, keyed by member ID. Returns natural language descriptions of the specified appliances, keyed by get_appliance_descriptions appliance name. Returns flattened JSON (as text) containing information about multiple get_object_information objects (e.g. mug etc.) in the environment. extract_captions Extracts segments within an optional time range, returning structured captions or joined text. This contains timestamps of events. The segments are optionally filtered by query extract_event_captions Returns captions for the most relevant or recent events, optionally filtered by a query. This will be just text, not timestamps inspect_video_segment Generates a clipped video when start and end are provided get_audio_events Extracts audio events segments within an optional time range. This contains timestamps of events. The audio events are optionally filtered by query. extract_frames Extracts encoded frames between the start and end timestamps at the fps. get_household_member_info

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

Related documents