Shared topology¶
The recommended deployment is one shared local data directory per trusted
machine or trust domain, with multiple coding-agent and AI-scientist
sessions using the same memd CLI binary and tenant/project conventions.
Each machine keeps its own store. Complete experience cases can move between
stores through explicit experience export and experience import commands.
These preserve canonical IDs and provenance, reject conflicting content, and
treat exact replay as a no-op. They do not provide continuous replication or
automatic merging of divergent case histories. A case can also travel as a
small case file through an external exchange that serves many machines and
sandboxes: experience note --emit validates the file, and
experience note --shared-id keeps a local copy tied to the exchange's ID. See
Experience memory.
flowchart LR
subgraph machine["Shared local machine"]
coding_agent["Coding agent"]
ai_scientist["AI scientist"]
cli["memd CLI"]
data[("metadata, WAL, and segments")]
coding_agent --> cli
ai_scientist --> cli
cli --> data
end
subgraph files["Workspace files"]
context_file[".memd/context.md"]
search_logs[".memd/search-logs"]
end
cli --> context_file
cli --> search_logs
Single writer, many readers¶
The warm worker is the normal writer. It opens <data_dir>/.writer.lock with
an exclusive flock and records its pid, memd version, and start time in the
lock file for diagnostics. The flock releases when the process dies, and an
idle worker exits on its own after MEMD_WARM_IDLE_TIMEOUT_SECS (default
30 min; 0 disables) so an orphaned worker cannot hold the lock forever. The
socket path is stable across binary upgrades, so a newer CLI reaches — and
replaces — a version-skewed worker on first contact.
Read and write commands use --warm <auto|off|required> where supported
(add, search, agent-context, delete, import-omf, purge,
consolidate, batch, report, and call). --warm auto is the default:
use the local worker, starting it if needed, and fall back to the cold CLI path
if startup or connection fails. --warm off always runs in the current CLI
process. --warm required requires a local worker and fails if one cannot be
started or reached.
Warm-routable commands are search without --include-superseded,
agent-context, report, call, add, delete, import-omf, purge,
consolidate, and batch without --stream. Cold-only variants such as
search --include-superseded and batch --stream silently run locally with
--warm auto, but hard-error with --warm required:
<variant> always runs on the cold path and cannot be routed through the warm worker; re-run with --warm auto for silent local fallback or --warm off.
Direct writes (--warm off, or --warm auto after worker startup fails) take
the same exclusive flock with a bounded retry budget. The default total retry
budget is 10 s and can be changed with MEMD_WRITER_LOCK_TIMEOUT_MS. On
contention, the typed error is:
writer lock held by another process ({holder}) at {lock_path}; if a memd warm worker is running, route this write through it (--warm auto, the default) or stop it with `memd warm stop`; otherwise stop the other memd process or retry later (MEMD_WRITER_LOCK_TIMEOUT_MS)
Reads are lock-free and change no stored data. Read commands, including
search, agent-context, and experience find|get|export, open the store in
ReadOnly mode. A ReadOnly open takes no writer lock and does not create or
change directories, modes, or metadata.db. In a writable directory SQLite
can still create its -shm file and Tantivy its .tantivy-meta.lock; in a
read-only directory they create nothing. It opens SQLite read-only, so it sets no
journal mode and runs no schema creation. It never appends WAL records, writes
segments, or persists indexes or caches. It also records no retrieval episode
(retrieval_episode_id is null), no retrieval_exposures.jsonl line, no
usage event, and no digest. A ReadOnly open of a missing data directory or
database reads as an empty store; mutating operations on a ReadOnly store
return a typed error.
A ReadOnly store works on a read-only mount, such as the data directory inside
a Codex sandbox. A WAL database is read through its -wal and -shm files
when they exist, so a live writer's recent commits stay visible. When both are
gone and the directory cannot be written, memd opens the main file with
SQLite's immutable=1 on the real file (symlinks are resolved first), which is
exact because no WAL frames exist. immutable=1 takes no locks: a writer that
starts during the read can make the reader see a stale snapshot or fail with a
read error. It never writes. A -wal with frames and no -shm fails with an
error instead. The Tantivy sparse index
opens without its lock file when the lock cannot be created.
--warm auto does not try to start a worker when the data directory is not
writable. It uses a worker that already answers, and otherwise runs the cold
read path without a warning. A store written by an older memd needs one writer
command (memd add, a warm worker start) before ReadOnly reads, because a
ReadOnly open does not create newer tables.
The worker probes SQLite data_version before each request. If an external
direct-fallback mutation happened, it schedules a single-flight background HNSW
repair and serves the request without blocking on it (waiting only a short
bounded budget rather than the full backfill). SQLite and sparse reads reflect
the external write immediately; dense/hybrid coverage of externally-added chunks
catches up once the background repair lands. The worker always indexes its own
warm-routed writes synchronously, so same-worker read-your-writes is unaffected.
Measured on the dev machine (2026-06, hardening validation run): an 8-writer × 3-round
write storm leaves 24/24 concurrent writes readable (7 of 16 were lost in the
2026-06-09 audit before the writer lock); warm-routed memd add p50 is 31 ms (vs ~1.6 s cold); a write is
searchable within p95 <70 ms.
memd maintenance is not warm-routable. It takes the data-dir writer lock
directly, so stop the worker first with memd warm stop or it fails after the
writer-lock timeout. memd purge is worker-routed by default, including
--rewrite-segments and --vacuum-metadata; memd purge --warm off takes
the lock directly and also requires the worker to be stopped.
Boundary conditions¶
- Same-machine shared sessions through the CLI are the primary supported path.
memddoes not provide built-in multi-user authentication or account isolation.tenant_idis caller-supplied logical partitioning, not an authentication boundary. Keep separate trust domains in separate data directories or under explicit tenant conventions.- Prefer one stable shared
tenant_idper trust domain; useproject_id,thread_id, andtask_idfor narrower retrieval scopes. - Cross-tenant project aliasing is off by default. Enable it only when consolidating mis-routed history; every widened hit produces a warning log.
memd search/agent-contextpayloads carry ascope_statusblock (retrieval mode, tenant-existence warnings, and awider_scope_hitshint when a project-scoped search misses content that exists tenant-wide).- Do not share a live data directory over NFS or other network filesystems;
flockis unreliable there. Move memory across machines with OMF export/import instead.
Scopes in normal use¶
| Scope | Set via | Use it for |
|---|---|---|
tenant_id |
--tenant-id, .memd/tenant_scope.json |
trust domain (one per machine usually) |
project_id |
--project-id, .memd/project_scope.json |
repository or workflow boundary |
thread_id |
tag thread:<id> or per-call argument |
conversation or PR scope |
task_id |
tag task:<id> |
one unit of work |
The SessionStart hook can auto-create a minimal .memd/project_scope.json
from the default tenant and repo basename. Use memd init when you want the
full .memd/tenant_scope.json, .memd/project_scope.json, and local
guardrail files so subsequent commands can omit the IDs.