The hard part of a personal AI agent isn't the agent — it's the memory

I wanted a specific, unglamorous thing: a daily chat companion that knows who I am, remembers across sessions, and can wear a different hat depending on what I'm doing that day. Not a task agent, not a content factory — a companion I'd actually use. Let me be upfront before you read on: this is a design sketch, not a shipped system. Nothing here is built yet. But the design is where all the interesting decisions live, and where I nearly talked myself in circles.

The conversation is easy. Remembering is the problem.

A single turn — you say something, the model replies — is a solved problem. The hard part of a personal agent is everything between the turns: what does it carry from today into next Tuesday, and how does it do that cheaply enough to run every day without a growing bill or a context window that eventually explodes? "Memory" sounds like one feature. It isn't. It's at least four, and they behave differently.

Four tiers, and why they're different jobs

The design splits memory into four tiers, each answering a different question:

• Working — the last few turns, held in the model's context window. No storage; it evaporates when the chat ends.
• Session — the full transcript of the current conversation, one row per message. The verbatim record.
• Summary — a short, embedded recap written after a session ends, searchable later. One row per conversation.
• Semantic — the distilled, durable facts: "prefers terse replies," "is learning React," "works on a consulting brand." Atomic notes, each typed as a preference, fact, or decision, tagged and embedded.

A small honesty flag lives in that list. I first called the third tier "episodic" — until I realised that in the established agent-memory vocabulary, "episodic" means something specific (past action sequences replayed as examples). I was reading the wrong map and using a word that would mislead anyone fluent in it, so I renamed it "Summary." Minor, but it's the kind of terminology drift that quietly makes your design illegible to everyone else in the field.

Modelling memory like an Obsidian vault

The part I liked most: the semantic tier is modelled like a notes vault. Atomic notes, explicitly linked to each other — the same instinct as an Obsidian graph, where the connections carry as much meaning as the notes. Except it isn't markdown files; it's real Postgres, with a note_links table for the graph and pgvector for similarity search. A memory-writer — one cheap background model call at the end of each session — reads the transcript and returns structured JSON: a summary to embed, any new facts to file (with their type), and links to notes that already exist.

Retrieval at each turn is hybrid: ordinary SQL filters (which role, which tags, how recent) to narrow the field, then vector similarity to rank what's left, stitched into the prompt. The role itself is session-scoped — picked when the conversation starts and fixed for its duration. New hat, new session. That one rule sidesteps a whole category of bugs around swapping tools and stale system prompts mid-conversation. The whole thing is designed API-first, on Cloud Run, with the model doing turns and a cheaper model doing the background summarising.

The uncomfortable part: I might be over-building it

Now the honest twist, and it's the reason I'm writing this as a sketch and not a launch. While researching, three different tools each independently told me I was hand-rolling a worse version of something that already exists:

• One minimalist agent's whole philosophy is "keep a small file of high-signal facts and prune it" — suggesting my four tiers might collapse into a single well-curated list.
• A coding agent I use every day stores its memory as markdown files on disk, not a vector database — a strong hint that my "notes vault" instinct was right but my Postgres-and-pgvector machinery was overkill.
• And a purpose-built memory library made the point bluntly: I'd essentially be writing a worse version of it, by hand.

Underneath sits a contradiction I couldn't resolve on paper. My pull toward "simple, linked, plain-text notes" points at files. My pull toward "serverless and SQL" points the exact opposite way. Both can't be the priority. When I reviewed my own notes, the diagnosis was embarrassing and familiar: analysis paralysis — six coherent, mutually exclusive architectures, no decision criterion to choose between them, and a goal that had quietly drifted from "a companion I actually use" to "understand agent memory deeply." Those are different projects.

What's built, and what's next

Nothing's built — and that's the honest headline. The real next move isn't more architecture; it's to stop researching, pick the single goal ("a companion I use"), ship the smallest thing that runs, and let a week of actually talking to it tell me whether I need four tiers or one good list. My bet, writing this, is that reality deletes at least two of the four. That's not a failure of the design — it's the design's job to be pruned by use.

If you're building an agent that has to remember its users — and you want the version that ships and gets pruned by real usage, not the one that's perfect on a whiteboard — that's the kind of work I do at twentytwotensors. Get in touch.