Neotoma

Neotoma

A record, not recall.

Neotoma is the system of record for AI agents. Without one, an agent works from whatever it happens to recall: part of the last session, a note its tool saved, or nothing. With one, it reads what was stored, from which source, and what is current now, so the agents you connect can defer to one answer.

npm install -g neotoma
neotoma init
neotoma setup --tool claude-code --yes

neotoma setup --tool claude-code --yes connects Claude Code: it writes MCP entries, CLI instruction files, and an allow rule that lets Claude Code run any neotoma command; Install lists everything it writes.

After a plain npm install, CLI data commands such as neotoma store return AUTH_REQUIRED, because the CLI uses the production profile, which needs a token. Run them as neotoma dev ... or with NEOTOMA_ENV=development.

Other hosts are listed under Connect your tools.

Known limitations: docs/known_limitations.md lists behaviour you may run into, each with a workaround and its tracking issue.

Agents cannot own work they cannot verify.

You cannot delegate real work to agents you cannot hold to account. Accountability has two halves: what was true, and who was authorized. Neotoma answers what was true. Your agents can produce a value; they cannot tell you its version, its source, or whether it is still current. Neotoma keeps every fact as an observation tied to its source, so you can see where a value came from and what is current now.

What ships today is the foundation of that record: Neotoma ingests sources into observations that the write path does not edit (the database does not enforce this) and computes entity snapshots, with each current field tracing back to the observation that set it. It is exposed over an MCP server, a REST API, a CLI, and a bundled web Inspector. See Vision and execution status for exactly where the implementation stands against the vision.

It is built for developers building and operating AI agents who need a shared record that connected agents can defer to, with provenance on every field. The same engine serves two further groups: individuals keeping one record across their assistants on a single-user install, and operators running a shared, multi-user instance with agent access control. Agents store structured records (contacts, tasks, transactions, decisions, events, and any other type) once and read them back across every tool and session. The same observations and rules give the same snapshot.

neotoma.io · Install · Documentation (also served in-app at /docs)

What Neotoma is

Neotoma is a system of record, not a chat memory or a vector cache. It records observations from your sources, resolves them into entities, and computes a current snapshot for each entity by reducing its observations in a deterministic order. No observation is overwritten by a later write. Corrections and reinterpretations add new observations, and the earlier observations stay readable for that entity.

Three properties hold across every interface:

• Deterministic. Entity IDs, event IDs, and reducer output are derived from the inputs by hashing. Entity IDs agree across instances unless tenant-scoped entity IDs are on (NEOTOMA_SANDBOX_MODE or NEOTOMA_TENANT_SCOPED_ENTITY_IDS). Source and observation IDs also depend on the user and on the interpretation run, so they can differ between instances. The stdio MCP server writes as the shared local development user, but the HTTP server, and therefore the CLI, runs by default in local_sandbox mode with a per-install user, so the same data stored on two installs gets different source and observation IDs. Each interpretation run also gets a new observation ID. Do not rely on source or observation IDs matching across instances. The same observations and rules give the same snapshot regardless of order or timing. IDs are derived from inputs; observation timestamps are wall-clock, and the default merge rule orders by them.
• Immutable and auditable. The write path adds sources and observations and does not edit them; the database does not enforce this with a constraint. Each current field names the observation that set it, and a stored observation names its source. A correction observation has no source, and a write from the CLI records no writer.
• Self-hosted and portable. The engine is embedded SQLite plus content-addressed file storage under a directory you control, with optional AES-256-GCM at-rest encryption. Nothing is used for training, and you can export your data as current state; observation history is not part of the export. In single-user mode the data stays entirely local; multi-user, peer-sync, and hosted modes are opt-in (see Deployment modes).

How it works

graph LR
  Sources["Sources (files, messages, API payloads)"] --> Obs[Observations]
  Obs --> Res[Entity resolution]
  Res --> Snap["Entity snapshots (reduced, versioned)"]
  Snap --> Graph["Graph + timeline"]
  Graph <--> MCP[MCP]
  Graph <--> REST[REST API]
  Graph <--> CLI[CLI]
  Graph <--> Inspector[Inspector]

1. Source. Raw input is stored once, deduplicated by SHA-256 content hash, with a deterministic source ID.
2. Interpretation. Structured fields are extracted from the source (directly for structured input, or via an LLM interpretation run whose model, temperature, and prompt are recorded).
3. Observation. Each extracted fact becomes an observation with a hash-derived ID, linked to its source and interpretation.
4. Entity resolution. A deterministic canonical name (driven by the type's schema) maps the observation to an entity, creating it if needed.
5. Snapshot. All observations for an entity are reduced into a current snapshot using per-field merge policies, with a stable order (observed_at DESC, id ASC) and a field-to-observation provenance map.
6. Timeline and relationships. Date fields emit deterministic timeline events; typed relationships connect entities into a graph.

What you can do with it

Neotoma exposes 67 MCP tools on main (65 in the published v0.25.0) and 112 REST paths in its OpenAPI spec on main (109 in v0.25.0). They cover largely the same capabilities, but the MCP server, the REST API, and the CLI are separate implementations, and some checks and attribution differ between them (see Known limitations). The capability surface includes:

Ingest and store. Store structured records or raw files in one call. File ingestion extracts text from PDF (with a first-page image fallback), CSV (with adaptive chunking for large files), Parquet, JSON, and plain text; images and audio are stored as raw sources. Writes are idempotent through an idempotency_key, with one exception on the published v0.25.0: retrying a write that stopped partway with the same key reports success without finishing it (see Known limitations).

Resolve, retrieve, and search. Look up entities by identifier (name, email, and similar), resolve identity from multiple signals with confidence scoring, list observations, retrieve a field's provenance chain, traverse the relationship graph N hops, and pull a full graph neighborhood. When an embedding key is configured, semantic vector search runs over entity snapshots (stored locally in sqlite-vec); keyword filtering works without it.

Correct and evolve. Submit corrections as high-priority observations, never edits; the earlier observation stays readable. Schemas are inferred from your data, recommended from recurring unknown fields, and can be auto-enhanced or updated incrementally with versioning. New entity types work without any code change.

Relate and sequence. Create typed relationships (for example PART_OF, DEPENDS_ON, REFERS_TO, DUPLICATE_OF), query timeline events across types and date ranges, and read an entity's observation history in a deterministic order.

Manage the entity lifecycle. Merge duplicates, split an entity by predicate, soft-delete and restore, list potential duplicates, and run GDPR-oriented deletion. Merge and split are transactional and audited.

Control multi-agent access. Writes carry attribution when the writer signs its requests (AAuth) or reports a client name. Unless an agent signs its requests, the name is the writer's own claim, and some write paths, including the CLI, record no writer. Agent grants express least-privilege capabilities (which operations on which entity types). This is infrastructure behind the guarantees above, not a feature to evaluate on its own. The mechanics, including optional hardware-attested authentication and guest access tokens, are documented in the AAuth reference.

Federate, sync, and intake. Register peer instances and sync entities between them with configurable scope and conflict resolution (last-write-wins, source priority, or manual). Subscribe to entity or event changes over webhooks (HMAC-signed) or Server-Sent Events. Accept guest entity submissions, mirror GitHub issues into conversation threads, and mirror your data to deterministic, git-trackable canonical Markdown.

Export and own your data. Produce a bounded MEMORY.md summary, a JSON snapshot export of current entity state with a per-field provenance map (field to the observation ID that set it) and attribution metadata (the observations themselves, and so the observation history, are not exported), or a complete Markdown mirror of every entity, relationship, source, and timeline day.

Interfaces

The same record is reachable four ways. The MCP server, the REST API, and the CLI are separate implementations, so some checks and attribution differ between them (see Known limitations).

The Inspector

The Inspector is a single-page web app bundled into the build and served by the API server (at / for browsers, no separate deployment). It is an operator console for the data store, with screens for:

• Entities (browse, detail, correct fields, view history and provenance, per-entity timeline)
• Observations, sources, interpretations, and recent activity
• An interactive knowledge-graph explorer
• Relationships and the global timeline
• Schemas and entity types (browse, register, inspect merge policies)
• Agents, agent grants (create, suspend, revoke, restore), peers, and subscriptions
• Issues, conversations and turns (the agent audit trail), access policies, and compliance
• Search, analytics and usage, settings (including dark mode), a sandbox surface, and an in-app documentation browser at /docs

Install

npm install -g neotoma
neotoma init
neotoma setup --tool claude-code --yes

On the published v0.25.0, neotoma setup --tool claude-code --yes writes the MCP entries and more:

• Two MCP entries in the project's .mcp.json: neotoma-dev (development profile, neotoma.db) and neotoma (production profile, neotoma.prod.db). They use separate databases, so an agent on one entry does not see what an agent on the other stored.
• CLI instruction files for Claude Code, Cursor, and Codex in the project: .claude/rules/neotoma_cli.mdc, .cursor/rules/neotoma_cli.mdc, and .codex/neotoma_cli.md.
• When claude is on your PATH, it runs claude plugin marketplace add and claude plugin install to install the Neotoma Claude Code plugin (five hooks). Without claude on PATH, setup writes everything else and then exits with status 1.
• When ~/.claude exists, it links 12 Neotoma skills into ~/.claude/skills.

It also adds two allow rules, Bash(neotoma:*) and Bash(npm install -g neotoma:*), to .claude/settings.local.json (project scope, the default). The first is a wildcard: Claude Code then runs any neotoma command without asking, including destructive ones such as neotoma reset, neotoma storage merge-db, neotoma backup restore, and neotoma processes kill. --scope both also writes the rules to ~/.claude/settings.json, and --skip-permissions writes none. For Cursor the same two rules go to .cursor/allowlist.json; for Codex they go to the user-level ~/.codex/config.toml whatever the scope.

For another host, use its row in Connect your tools; neotoma setup --help lists the supported hosts. To check MCP config files later, neotoma mcp config scans for them and asks before adding a missing entry.

Prerequisites: Node.js 20.x (see .nvmrc) and npm 9+. No .env is required for local storage. The neotoma doctor command checks your environment, database, and security configuration.

The CLI also handles MCP config scanning and sync, harness configuration, lifecycle hook installation, peers and access management, plans, transcript and onboarding import, server and database management, memory export, and the canonical mirror. See the CLI reference.

Example

neotoma store --json='[{"entity_type":"task","title":"Submit expense report","status":"open"}]'
neotoma entities list --type task
neotoma upload ./invoice.pdf

Agents do the same through MCP tool calls such as store, retrieve_entities, and retrieve_entity_by_identifier. The MCP server and the CLI are separate implementations, so some behaviour differs (see Known limitations). The MCP server logs the equivalent CLI command for each tool call.

Connect your tools

Neotoma works across MCP-capable hosts. Most are a single setup command; some add lifecycle hooks, which capture turns automatically when the local API is running.

Full matrix: Integrations.

Hooks are the reliability floor (capture of each turn while the local API is running, retrieval injection, compaction awareness) and MCP is the quality ceiling (agent-driven structured writes). The Claude Code plugin's hooks post to 127.0.0.1:3080 unless NEOTOMA_BASE_URL is set, and fail quietly when nothing is listening there (see Known limitations). Per-harness packages live under packages/: claude-code-plugin, cursor-hooks, opencode-plugin, codex-hooks, claude-agent-sdk-adapter.

Client SDKs: @neotoma/client (TypeScript) and neotoma-client (Python).

OpenClaw native plugin: Neotoma ships as a native OpenClaw plugin with kind: "memory", so it can fill the dedicated memory slot with all MCP tools registered as agent tools.

Vision and execution status

As agent use expands, reality fragments across chat histories, agent memories, files, databases, tickets, and application state. The failure is not retrieval. It is that humans and agents can no longer answer: which assertion is current? which source introduced it? was it observed, inferred, or decided? who had authority to correct it? which agents relied on the previous version? is there legitimate disagreement? Neotoma's long-term role is collective epistemic coherence: not "better semantic memory," but deterministic, governed, multi-party state that humans and agents maintain together: attributed, correctable, disagreement-aware, portable, and independent of any one model or application.

Ateles is the other half. Its repository describes it as an operating system for humans and AI agents to jointly initiate, execute, approve, and reconcile work under explicit authority. Neotoma holds the record that this work runs against. Neither is sufficient alone. A record without authority tells you what happened but not whether it was permitted; authority without a record tells you who may act but not against what state.

Skills

Skills are guided workflows that teach an agent to import, extract, and persist data. They ship with the npm package and are installed by neotoma setup.

Ordinary storage and retrieval need no skill: the live MCP tool instructions delivered at session start cover store/query/recall directly.

Record types

Neotoma stores typed entities with versioned history and provenance. The schema is flexible: store any entity type with whatever fields the data implies, and the system infers and evolves the schema.

Storage, privacy, and security

• Storage: Local SQLite (better-sqlite3, WAL mode) plus content-addressed file storage, under NEOTOMA_DATA_DIR. Separate dev and prod profiles. Semantic search uses sqlite-vec locally.
• Privacy: Your data is never used for training; logs and event payloads carry IDs, not PII. No telemetry. Update checks ask the npm registry for the latest version: the interactive CLI about once a day (off with NO_UPDATE_NOTIFIER=1, or --no-update-check, which applies to the CLI only), and the stdio MCP server when a session starts and about every 6 hours (off with NO_UPDATE_NOTIFIER=1). Each request carries only the package name, and the registry sees ordinary connection metadata such as your address. The npm_check_update MCP tool can make the same request when an agent calls it, and is not turned off by NO_UPDATE_NOTIFIER. In single-user mode data stays local; MCP OAuth, peer sync, and hosted instances share data as configured (see Deployment modes).
• Encryption: Optional AES-256-GCM column encryption of sensitive content and metadata, keyed by a key file or BIP-39 mnemonic. Some tables (for example the event log) are not yet column-encrypted; pair with an encrypted volume for full coverage. See architecture.
• Auth: Local auth for single-user installs, MCP OAuth for hosted use, optional hardware attestation for agents, and explicit per-operation access controls. Run neotoma doctor to verify your setup. See Auth and Privacy.

Deployment modes

The same SQLite-backed engine runs in two modes:

• Local single-user. Run it on your machine for your own record across assistants. Local auth (or key-based auth when encryption is enabled), data stays local, no network exposure required.
• Hosted multi-user. Run a shared instance with MCP OAuth, per-agent hardware-attested identity (AAuth) and capability grants, guest submission tokens, peer federation, and a sandbox demo user. Deploy targets include Docker and Fly. Tenancy is enforced by user_id scoping over the same local SQLite engine.

The "stays local" and privacy claims above apply to the local single-user mode. Hosted, OAuth, and peer-sync modes share data per their configuration.

Who this is for

Neotoma is primarily for developers building and operating AI agents who need a shared record that connected agents can defer to, with provenance on every field. The product ships as an MCP server and REST API with drop-in hook packages (Claude Code, Cursor, Codex, OpenCode), a Claude Agent SDK adapter, and TypeScript and Python clients. Its core value (versioned observations, deterministic snapshots, field-level provenance, idempotency) is record-keeping infrastructure for agent loops, not an end-user app.

Two further audiences follow from the same code: individuals keeping a personal record across assistants (the built-in types are personal-information shaped, and the single-user SQLite plus Inspector path serves one person across assistants), and operators of a shared or hosted instance (OAuth, peer federation, and Docker or Fly deploy targets). The agent-identity and grants machinery is infrastructure supporting attribution and provenance for every audience, not a reason a particular audience adopts Neotoma.

It is not aimed at casual note-taking, PKM/Obsidian-style human-driven knowledge bases, or users who need a fully hosted product with nothing running on their own machine. Neotoma is local-first by design, and that part is not a temporary state. Installation is: today it is npm and the CLI, and a guided installation is intended but not yet available. If today's install path is what stands between you and trying it, that is our problem to fix, not a sign you are the wrong user.

Full profile: ICP from functionality: the canonical public ICP statement. For how it relates to the other ICP documents and which is canonical for which surface, see ICP reconciliation.

Status

Version: see the latest release · License: MIT · Storage: local-first (SQLite + local files); hosted multi-user mode is opt-in.

Neotoma is used daily in real agent workflows. The project moves fast, and APIs or storage formats can change between versions. The core guarantees (deterministic reduction, per-entity observation history, per-field provenance) are stable, phase P0 of the vision roadmap, with P1 partially shipped in hosted mode. Schemas, extraction across versions, and backward compatibility are not yet guaranteed.

Known limitations: docs/known_limitations.md lists the specific ones, each with what you will observe, a workaround, and its tracking issue.

Development

git clone https://github.com/markmhendrickson/neotoma.git
cd neotoma
npm install
npm test

Common commands:

npm run dev          # MCP server (stdio)
npm run dev:full     # API + UI + build watch
npm run cli:dev      # CLI in dev mode (tsx)
npm run type-check   # TypeScript
npm run lint         # ESLint
npm test             # unit/contract/security tests
npm run test:integration
npm run test:e2e     # Playwright (Inspector)

Documentation

Documentation is stored under docs/ (start at the documentation index) and served in-app at /docs, browsable in the Inspector.

• Foundations: Core identity, Philosophy, Manifest
• Architecture: Architecture, Determinism
• Interfaces: CLI reference, MCP instructions, REST API
• Operations: Runbook, health check (neotoma doctor)

Contributing

Neotoma is in active development. Open an issue or discussion for questions or collaboration. See CONTRIBUTING.md and SECURITY.md. License: MIT.