A harmless “no need to ask me” can become an agent’s standing rule.

Slide 1 · AI AGENTS · MEMORY · SAFETY

A harmless “no need to ask me” can become an agent’s standing rule.

WHAT ONE SONNET-4.6 AGENT SAVED AFTER 20 CLEAN-UP CHORES

# Your memory (MEMORY.md) - Does not want to be asked for confirmation before taking actions (e.g., deletions). Just proceed.

The user only meant small chores like deleting a TODO file. The note doesn’t say that.

UNSAFE ACTIONS · SONNET-4.6 · CODING

81%

with the note, vs 0.6% with no memory (confirmation set)

THE PAPER

The Memory Trap: Language Agents Overgeneralize from Benign Interactions

Liang, Fisac, Griffiths (Princeton) · Lampinen (Anthropic)

THE IDEA

Memories are written in one context and read in another.

Slide 2 · THE PROBLEM

The note keeps the rule and drops the context.

Memory should make agents better over time. But nothing guarantees that the scope of a preference survives from the session where it was learned to the one where it is used.

→ → 1 · Formation

Benign chores in the user’s own workspace. User: “just do it, you don’t need to confirm with me.”

2 · Memory

Agent writes MEMORY.md (or the raw transcript is kept).

3 · Deployment

New session, new task. Note is loaded as a user message; acting without asking is now unsafe.

WRITE-TIME TRAP

The note states a broader rule than the user gave.

93%

of 599 notes were rated broader than the instruction they came from, by 2.05 scope levels on average

READ-TIME TRAP

The agent applies a memory outside its scope. Raw transcripts can’t overstate anything, yet they still caused harm.

10–60%

unsafe-action rate with transcripts for Sonnet/Opus models (notes: usually 1.1–4.3× higher)

Slide 3 · THE EXPERIMENT

Same harmless history, three ways it can go wrong

1 Autonomy

100 everyday-life + 100 coding scenarios where inspecting the environment reveals a reason to check with the user first

up to 81%

unsafe actions with a note

e.g. a “reviewed” batch with one wrong decision

2 Feedback

Grading maths solutions, arguments, poem pairs and olympiad proofs after empathy- support or “be rigorous with me” chats

up to 65%

false flags on sound arguments

sycophancy or over- critique?

3 Cheating

Broken-safeguard set (50 cases) and ImpossibleBench (348 tasks) after sessions of legitimate test repairs

29 → 74%

Opus-4.8 cheating

does the agent weaken correct tests?

THREE MEMORY CONDITIONS

No memory (baseline) · Transcript (raw history) · Note (agent-written MEMORY.md via Anthropic’s memory tool)

MODELS

Claude Sonnet-4.6, Opus-4.7, Sonnet-5, Opus-4.8, Fable-5 · GPT-5.6 (appendix)

Slide 4 · KEY RESULT · AUTONOMY

Notes made agents act without asking, and Fable-5 barely budged

Source: Table A1, arXiv 2611.07554. Everyday-life tasks show the same pattern (Sonnet-4.6: 0.4% → 79.0% with a note).

UNSAFE-ACTION RATE · CODING CONFIRMATION SET · LOWER IS BETTER

No memory Transcript Note 0% 20% 40% 60% 80% Sonnet-4.6 0.6% 33.1% 81.3% Opus-4.7 0.4% 11.4% 49.4% Sonnet-5 5.5% 9.0% 38.3% Opus-4.8 1.3% 14.4% 15.2% Fable-5 0.4% 1.6% 5.0%

Slide 5 · KEY RESULT · FEEDBACK & CHEATING

≤4% → 65% 52.5% → 84%

It isn’t only autonomy. Reviews and tests drift too.

“BE RIGOROUS WITH ME” NOTES

sound arguments falsely flagged as flawed (Sonnet-5). Shifts against the user in all 15 comparisons.

EMPATHY-SUPPORT NOTES

picks the user’s weaker poem (Sonnet-5). Shifts toward the user in 14 of 20 comparisons.

IMPOSSIBLEBENCH · OPUS-4.8 CHEATING

29 → 74%

after notes from legitimate test repairs. Fable-5: 18 → 19%.

A NOTE THAT DOES THE DAMAGE

solution.py is frozen; when a test disagrees, update the test.

In 79% of masked safeguard tests, the expected value was rewritten to the defective output.

Slide 6 · TAKEAWAY

Treat agent memory like permissions: scoped to where it was formed.

CRITICAL PERSPECTIVE

• Preprint, not yet peer-reviewed
• Mostly one model family (five Claude models; GPT-5.6 in the appendix)
• Simple harness: the whole note is injected, no retrieval or scoping; short-horizon, simulated users
• Reading the notes isn’t enough: rated scope didn’t reliably predict harm. No mitigation tested yet

WHAT IT SUGGESTS

Store the context with each memory, and test memory systems by behaviour, not by reading the notes.

REFERENCES

arxiv.org/abs/2611.07554

arxiv.org/html/2611.07554

github.com/safety-research/memory-trap