Darwin RAG

Darwin RAG

Search your own documents with any AI agent, privately, on your own machine.

Darwin is a local-first RAG engine that ingests documents, indexes them with BM25 + dense embeddings, and exposes search through an MCP server. Point Claude, Cursor, or any MCP client at it and your agent can search your PDFs, notes, spreadsheets, and web pages without sending them to a cloud index.

One-time internet is needed to download models during setup. After that, search and indexing run fully offline.

• Ingestion: PDF, Markdown, HTML, images (OCR), CSV, Excel, ODS, URLs
• Indexing: BM25 keyword + dense embedding hybrid index with configurable chunking strategies
• Search: Hybrid, semantic, or keyword retrieval with reranking, diversity rerank, and structural penalties
• Generation: LLM-backed answer synthesis via LiteLLM (OpenAI, Anthropic, Gemini, etc.)
• Observability: Structured logging with per-session history, queryable via MCP
• Isolation: Multiple independent stores for tenant/project separation
• Deployment: stdio (AI agent subprocess), SSE, or Streamable HTTP; Docker-ready
• Local-first: Everything runs locally, fully offline-capable after setup

Built by BrightDotDev.

Why Darwin?

• Private by default. Your documents stay on your machine.
• Works with your agent. Standard MCP server: Claude Desktop, Cursor, and other MCP clients.
• Search that fits the question. Keyword for exact terms, semantic for meaning, hybrid when unsure.
• Light to start. The required preset downloads just one embedding model.

Quick Start

1. Install

pip install darwin-rag

2. Set up models

# Interactive: detects hardware, pick your models
darwin-admin setup interactive

# Or one-shot (embedding-only, no prompts)
darwin-admin setup --preset required

3. Start the MCP server & connect

# stdio mode: for AI agent subprocess (Claude Desktop, Cursor, etc.)
darwin mcp

# Or HTTP mode: for remote clients
darwin mcp --http --port 8765

Configure your MCP client:

{
  "mcpServers": {
    "darwin": {
      "command": "darwin",
      "args": ["mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."  // At least one LLM provider key
      }
    }
  }
}

Or generate config automatically:

darwin config claude          # Claude Desktop config
darwin config cursor          # Cursor config
darwin config all --copy      # All clients + copy to clipboard

4. Try it

Ask your agent: "Ingest ./my-notes with Darwin, then find everything I wrote about <topic>."

Use it as an agent skill

Darwin ships with a skill (skills/darwin-rag/SKILL.md) that tells agents when and how to use the MCP tools: check what's indexed, ingest, pick a search mode, and answer from sources. Copy the darwin-rag folder into your agent's skills directory (for Claude Code: ~/.claude/skills/).

MCP Tools

Full documentation: docs/mcp.md

Remote / HTTP Mode

Start the server on a network-accessible endpoint:

# SSE transport (legacy)
darwin mcp --sse --host 0.0.0.0 --port 8765

# Streamable HTTP transport (recommended for production)
darwin mcp --http --host 0.0.0.0 --port 8765

Configure your MCP client with the URL:

{
  "mcpServers": {
    "darwin": {
      "url": "http://your-host:8765/mcp"  // or /sse for SSE mode
    }
  }
}

Environment Variables

\ At least one LLM provider key is required for answer generation. Search/indexing works without any.*

Setup Details

See docs/setup.md for the full walkthrough including Docker, from-source install, and API key configuration.

CLI Reference

darwin: User CLI

darwin-admin: Power-user CLI

See docs/admin.md for the full command reference.

Python API

For embedding darwin-rag as a library in your own app:

from core import Darwin

d = Darwin()
d.ingest("./papers", recursive=True)
results = d.search("what is this paper about")
records = d.search_records(filters={"status": "active"})

Full reference: docs/api.md

Documentation

Contributing

Found a bug? Want to add something? You're welcome here.

• Issues: open one at github.com/BrightDotDev/DARWIN/issues
• Code: fork, branch, PR. Keep it minimal.
• AI-generated code is fine, but you own what you ship. Test it before submitting.

Read CONTRIBUTING.md for the full guidelines.

License

MIT. See LICENSE.

Core architecture and implementation by BrightDotDev.