Blog · JARVIS-Bench

01

Real days, kept as they were logged

Personalization benchmarks rarely follow real people over long periods, and they often reduce personalization to recommendation or fact retrieval. JARVIS-Bench starts from participants who logged their own days through a mobile web app for 28 to 44 days: waking up, meals, work, exercise, entertainment, shopping and bedtime, often with a photo and a mood. Every real entry stays in the trajectory verbatim, and the same people supplied the answers: a 100-question preference survey, a flag whenever they wanted help, and their own mood keywords.

Figure 1 of the paper: daily logs of several users across days, the lifestyle domains and apps they cover, the mobile app, and example entries with moods, photos and a moment where the agent should help. — **Figure 1.** Real-user traces and ground-truth feedback. Daily logs organise multimodal life events across time, users and domains, and carry implicit-preference, emotion and proactive-help signals.

02

Four questions about one person

All tasks read the same trajectory through one event iterator. Implicit Preference Reasoning asks the agent to answer the person's own survey from behaviour alone. Proactive Help Prediction asks whether this is a moment to step in, and why. Emotion-Aware Modeling asks how the person feels. Cross-Domain Transfer asks whether evidence from one part of life carries over to another.

Figure 2 of the paper: four panels, Implicit Preference, Cross-Domain Transfer, Proactive Help and Emotion-Aware, each with a scenario, past episodes and Jarvis's reply, around a lifelong personal intelligence agent. — **Figure 2.** The four task surfaces, each shown with a scenario, the past episodes Jarvis draws on, and its reply.

03

Memory helps, but no model is close

We evaluate 54 LLMs from 11 families on the 100-question survey under five memory levels, from No Memory (C0) to Lifelong Memory (C4). Averaged over the models, accuracy rises from 34.8% with no memory to 47.2% with the long trajectory, while the persona profile adds only 3.5 more points: the behavioural trace, not the profile, drives most of the gain. Even with lifelong memory, the best model, GPT-6-Astra, reaches 73.2%. Random guessing scores 25%.

Stacked bars for 54 models grouped by family (Claude, GPT, Grok, Gemini, Qwen, Gemma, DeepSeek, GLM, Kimi, MiMo, MiniMax): accuracy at C0 to C4, with dashed lines at the C2 mean 47.2 and the C4 mean 61.7. — **Figure 3.** Implicit Preference Reasoning accuracy for all 54 models, grouped by family. Each bar stacks No Memory (C0) up to Lifelong Memory (C4); dashed lines mark the C2 and C4 means. The full table is on the leaderboard.

04

Where a preference lives

Cross-Domain Transfer removes one lifestyle domain from the trajectory (drop-one) or keeps only that domain (keep-only), then asks that domain's questions again. Leisure is both the most necessary and the most self-sufficient domain: without it, same-domain accuracy falls by 12.8 points on average over the three GPT-5 sizes, and kept on its own it lifts accuracy by 20.8 points over No Memory.

Heat maps for gpt-5, gpt-5-mini and gpt-5-nano: accuracy change per question domain when one lifestyle domain is dropped (top) or kept alone (bottom); the diagonal is outlined. — **Figure 4.** Same-domain accuracy change for each GPT-5 size. Top: drop one domain, against the long trajectory. Bottom: keep only one domain, against No Memory. The outlined diagonal is the same-domain effect.