AI agent memory sounds useful until it starts preserving noise.
The failure mode is simple. An agent sees a pattern, decides it matters, writes it down, and future sessions inherit that guess as context. One sloppy inference becomes doctrine. A temporary preference becomes a permanent rule. A passing project detail keeps walking back into the room.
That is not memory. That is residue.
I looked at Hermes Agent while building memory confirmation for my Pi setup. Hermes has good ideas: structured memory tools, compact result rendering, file-backed storage, and a session model where memory is loaded at the start and refreshed later. I kept the useful parts. I rejected the part where the agent quietly decides what deserves to survive.
My version is stricter.
Memory only gets written when I explicitly ask for it. If I say “remember that…”, the agent must stop and ask for the exact wording. Nothing is saved until I approve that wording. After approval, the memory is appended as a short Markdown line in my Obsidian brain.
The interface is boring on purpose. A small confirmation says: Memory saved. The tool result includes the destination, so there is an audit trail. New sessions read the memory file during boot. Secrets are rejected. Research logs, project progress, long notes, and random observations stay out.
This is not “smart memory.” It is manual memory with a safety catch.
I do not want the agent to guess what matters. I want it to help me preserve decisions I already made. Durable facts, stable preferences, workflow rules, and setup notes can stay. Useful-looking sentences do not automatically earn a place in future context.
Manual is better here because memory is not storage. Memory is authority.
Once something enters the boot context, it changes how future work gets interpreted. That deserves friction. The confirmation step is not a UX tax. It is the boundary that keeps the agent from becoming a hoarder with a prompt window.
The rule is simple: ask first, save only exact approved wording, keep it short, never store secrets.
That is enough.