Data layout¶
Persistent mode writes to:
~/.memd/data/
├── .writer.lock # Exclusive writer flock
├── embedding_revision # Embedder revision of the dense vectors
├── metadata.db # SQLite metadata (WAL mode, pooled)
├── sparse_index/ # tantivy BM25 index (open_or_create)
├── warm/<hash>/ # 0700 runtime dir for warm worker socket
│ # socket file chmod 0600
└── tenants/
└── <tenant_id>/
├── wal.log # Append-only WAL; fsync before commit
├── segments/ # Immutable chunk segments + payload
└── warm_index/ # HNSW state
├── embeddings.bin # Source of truth for vectors
├── mapping.bin # bincode (legacy: mapping.json)
├── config.json # HnswConfig snapshot
└── graph.hnsw.{graph,data} # Optional fast-load dump
# (skipped when persist_graph_dump=false)
Default data dir: ~/.memd/data. Override with --data-dir.
embedding_revision records which revision of the embedder produced the
vectors under warm_index/. When a release changes how vectors are computed,
the first writable open discards the dense indexes and re-embeds every chunk
from its segments in the background; the file reads <revision> rebuilding
until that pass has finished and the rebuilt indexes are saved. A command that
exits during the rebuild waits up to 10 seconds for it, saves what has been
indexed, and the next writer continues from there. Until then, chunks that are
not yet re-embedded are found through the sparse index only.
Retrieval/list scans are tolerant of stale metadata rows with unreadable segment
payloads: unreadable chunks are logged and skipped. Direct
memory.get remains strict so point lookups still surface storage corruption
instead of silently returning the wrong record.
Disk hygiene¶
Run memd maintenance to sweep orphan HNSW snapshots and report the result.
Use --aggressive to force-merge the global Tantivy sparse index into one
searchable segment:
memd maintenance --dry-run # report what would change
memd maintenance --aggressive # run the full pass
memd maintenance --tenant-id <id> # restrict the HNSW sweep only
The orphan sweep targets graph-NNNN.hnsw.{graph,data} files. Aggressive
output includes sparse_segments_before, sparse_segments_after, and
segments_merged. Output uses key:value lines for shell parsing. The command
takes the data-directory writer lock and does not run through the warm worker.
Why bincode for the mapping?¶
Older builds wrote mapping.json (~5× larger). v0.50.0 packs the same
chunk-id → HNSW-index mapping as bincode mapping.bin. The reader still
accepts the legacy JSON format and auto-migrates on next save.