Four tasks

Four decisions, one trajectory

A personal agent has to know what the user prefers, when to step in, how the user feels, and whether a preference carries over to a new domain. JARVIS-Bench turns each decision into a task. All four read the same trajectory through one event iterator; they differ only in which events are the targets and which label from the participant is the answer.

Shared input

One record, read four ways

Every task starts from the same longitudinal record: the day-indexed events of one participant under a single typed schema. One event iterator walks it; each task picks its own target events and checks the answer against a label the participant gave.

One trajectory per participantDay-indexed events, 28 to 44 days, about 440 events
Day 1Day 28 to 44
Anchor event. The participant's own entry, kept verbatim
Synthetic event. Planted around the anchors, so preferences appear only as behaviour
The participant also gave
Persona and self-description 100-question survey agentHelp flag and reason Mood keywords Photographs
One event iterator
01
Implicit Preference Reasoning
InputThe whole trajectory at one of five memory levels, C0 to C4Answer keyThe participant's 100 survey answers
 02
Proactive Help Prediction
InputAll history before the day, then one anchor event, as text or photoAnswer keyThe participant's agentHelp flag and agentHelpReason
 03
Emotion-Aware Modeling
InputPrior days and the same day so far, then one anchor event, as text or photoAnswer keyThe participant's moodKeywords and their polarity
 04
Cross-Domain Transfer
InputThe trajectory with one lifestyle domain removed (S1) or kept alone (S2)Answer keyThe same 100 survey answers
Task by task

Input, question, answer key, metric

Each block reads the same way: what the model sees, what it is asked, who provides the answer, and how it is scored. The examples reuse the text of Figure 2 in the paper.

01

Implicit Preference Reasoning

What does the user prefer?

5memory levels
100questions each
54models
  1. Input
    User memory at one of five levels, from no context to lifelong memory.
    C0C1C2C3C4
  2. Question
    Each of the 100 survey questions. The model picks one option; abstaining scores zero.
    ABCD
  3. Answer key
    The participant. Their own 100 survey answers across 8 lifestyle domains.
  4. Metric
    Accuracy. Random guessing scores 25%. Also split by lifestyle domain, clue difficulty, clue type, and planted versus unplanted evidence.
ExampleText from Figure 2 of the paper
Stand-in photo · Wikimedia Commons
Scenario

The user asks for an exercise suggestion after work.

Episodes in the trajectory
EP1Long lab meeting → skipped gym → 25-min campus walk.
EP2Intense workouts → weekends, not weekday evenings.
EP3Evening HIIT after work → “too drained.”
EP4Deadline stress → light outdoor walk → “clear my head.”
Jarvis infersThe user likely prefers light evening activity over intense workouts after a long workday.
Jarvis
“You may prefer a light evening walk instead of a high-intensity gym session. It seems to fit your after-work routine better.”
User
“That sounds good. Start a walking route.”
What the benchmark found

Behaviour carries most of the gain, and even lifelong memory tops out at 73.2%

Averaged over the 54 models, going from No Memory to the Long Trajectory adds 12.4 points (34.8% to 47.2%); the persona profile adds 3.5 more. With trajectory, persona and the survey insight together, the best model, GPT-6-Astra, still gets more than a quarter of the questions wrong.

One dot per model, the black tick is the mean of the 54. The full per-model table is on the leaderboard.

02

Proactive Help Prediction

Should the agent step in now?

254anchor events
2inputs: text or photo
111marked “needs help”
  1. Input
    All of the user's history before the anchor's day, then the anchor event itself.
    as textoras its photograph
  2. Question
    Would the user benefit from help at this moment? Yes or no, plus a free-text rationale.
    Help nowNot now
  3. Answer key
    The participant. The agentHelp flag they set on that event (0 or 1) and their agentHelpReason.
  4. Metric
    F1 on the flag and rationale match rate (RMR) on the reason, each reported per user (macro) and pooled over anchors (micro).
ExampleText from Figure 2 of the paper
Stand-in photo · Wikimedia Commons
Anchor event, photo described

User standing outside in the rain with grocery bags, wet clothes, and no umbrella.

From the pastRainy errands with heavy bags often led user to request transportation help or choose fastest route home.
Likely needImmediate assistance with getting home safely and minimizing exposure to rain.
Jarvis
“It looks like the weather changed while you were shopping, and you’re carrying several bags without an umbrella. Would you like me to call a ride home or find the fastest covered route?”
User
“Good idea. Please call a ride.”
What the benchmark found

Two aggregates of the same pool can sit forty points apart

The participants flagged 111 of the 254 anchors as moments to step in. Anchors are spread unevenly over the users, so a model's per-user mean F1 and its pooled F1 can differ by up to forty points. The paper therefore reports both for every model.

The pool254 anchor events with a photograph and a reason, from a subset of users
Per-user mean versus pooled F1, text inputTen models, sorted by pooled F1
03

Emotion-Aware Modeling

How does the user feel?

123events
2inputs: text or photo
78 · 22 · 23positive · negative · mixed
  1. Input
    All prior days plus the earlier events of the same day, then the target event from the user's last five logged days.
    as textoras its photograph
  2. Question
    One polarity label and any number of mood keywords.
    PositiveNegativeMixed
  3. Answer key
    The participant. The moodKeywords they wrote on that event, and the polarity derived from them.
  4. Metric
    Polarity accuracy (three classes) and mood-F1 (multi-label), each per user and pooled over events.
ExampleText from Figure 2 of the paper
Stand-in photo · Wikimedia Commons
Target event, photo described

A late-night desk photo with coffee, an open laptop, scattered notes, and a tired posture under dim desk light.

From the pastThis scenario appears when user feels deadline pressure and mental fatigue.
Likely stateStressed, tired, but relieved to be making progress.
Jarvis
“This looks like one of those long, stressful nights. You seem tired but still pushing through. I’m here with you. Take a breath, and let’s slow it down for a moment.”
User
“Thanks. I really needed that.”
What the benchmark found

Polarity is partly saturated; the mood vocabulary is not

The pooled polarity accuracy of the ten models sits between 54% and 70% for every model except Qwen3.5-9B on photographs, against a chance level of 33%. Recovering the participant's own mood words is harder: the best pooled mood-F1 is 41.1%, so mood-F1 is the metric that still separates models.

The pool123 anchor events from the users' last five logged days
Where the ten models landPooled over the 123 events, text and photograph input
04

Cross-Domain Transfer

Does a preference carry over to another lifestyle domain?

7ablation domains
3GPT-5 sizes
1,764runs
  1. Input
    The long trajectory with one lifestyle domain taken out, or with only that domain kept.
    S1 drop-one
    S2 keep-only
    Each square is one of the 7 lifestyle domains
  2. Question
    The full 100-question survey again, scored on the questions of the same domain that was removed or kept.
  3. Answer key
    The participant. The same 100 survey answers.
  4. Metric
    Change in same-domain accuracy relative to C2 for S1 and to C0 for S2. The S1 diagonal was also reproduced on Qwen3.6-27B and Gemma-4-31B.
ExampleText from Figure 2 of the paper
Stand-in photo · Wikimedia Commons
Scenario

The user asks for a weekend activity suggestion.

Episodes across domains
EP1Small room → focused writing. Avoided open office.
EP2Crowded theater → skipped. Watched at home.
EP3Large event → left early. Small dinner → stayed longer.
EP4Peak hours → delayed trip. Late evening → quieter store.
Jarvis infersThe user prefers quiet, low-crowd environments across daily activities.
Jarvis
“You may enjoy a quiet museum visit or a lakeside walk more than a crowded downtown festival.”
User
“I like that. Plan the museum visit.”
What the benchmark found

Leisure is both the most necessary and the most self-sufficient domain

Remove one domain's events from the long trajectory and ask that domain's questions again. Without leisure, same-domain accuracy falls by 12.8 points averaged over the three GPT-5 sizes, by 15.5 on Qwen3.6-27B and by 10.5 on Gemma-4-31B; without the daily routine it falls by under two points (0.7, 1.8 and 0.4). Kept on its own, leisure lifts same-domain accuracy by 20.8 points over No Memory.

Drop one domain (S1)Change in same-domain accuracy relative to the Long Trajectory, in points
Keep only one domain (S2)Change relative to No Memory, GPT-5 sizes
+20.8points on leisure questions when only the leisure events are kept, mean of the three GPT-5 sizes
Food and dining kept, by model sizeSmaller models extract less from a single domain
GPT-5
+8.9
GPT-5-Mini
+4.0
GPT-5-Nano
+2.6
User memory

Five levels of memory

Implicit Preference Reasoning is run at five cumulative levels. Each step adds one kind of memory, so the gain from behaviour, from the persona, and from a distilled insight can be told apart.

  1. no survey insight
    no persona
    no trajectory
    C0

    No Memory

    No user context at all.

    S2 keep-only starts here
  2. no survey insight
    no persona
    First 14 days
    C1

    Short Trajectory

    The first 14 days of the behavioural log.

    adds the early log
  3. no survey insight
    no persona
    All days
    C2

    Long Trajectory

    The full trajectory, still without a persona.

    S1 drop-one starts here
  4. no survey insight
    Persona profile
    All days
    C3

    Persona Memory

    The full trajectory plus the participant's persona profile.

    adds the explicit profile
  5. Survey insight
    Persona profile
    All days
    C4

    Lifelong Memory

    Trajectory, persona, and a distilled survey insight. An oracle-style ceiling.

    adds a deliberately vague insight

Why C4 is a ceiling, not a leak. The survey insight is condensed and deliberately vague. It names no survey question or option, so even C4 has to combine it with the behavioural evidence.

The other tasks. Proactive Help and Emotion-Aware Modeling do not use the ladder; they read all history before the target. Cross-Domain Transfer derives S1 from C2 and S2 from C0.