io.github.cyanheads/evals-mcp-server

@cyanheads/evals-mcp-server

Author verifiable eval records through a draft → review → revise → submit loop with server-enforced graders; compile to JSONL/CSV/Inspect/lm-eval via MCP. STDIO or Streamable HTTP.

9 Tools • 1 Resource

Overview

Build verifiable eval records with server-enforced graders. Draft a record with its own executable grader, revise it field by field, and submit through a gate that requires the gold answer to pass, a negative case to fail, and an independent check to agree. Export to JSONL, CSV, Inspect AI, or lm-evaluation-harness.

Tools

Resources

All record data is also reachable through the tool surface — evals_get_record for a single record, evals_list_records to browse. The resource is a convenience mirror for clients that support resources, not the access path.

Capability reference

evals_describe_schema tool

• Accepts task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, or free_response.
• Returns the gold shape, grader kinds, required/optional fields, and authoring notes from static schemas; mcq requires choices, and free_response requires llm_rubric.

evals_create_draft tool

• Accepts a record with its grader and discrimination cases, plus optional verification and captures (EvalsIDs); mcq requires choices, and free_response requires llm_rubric.
• Saves the record as a draft once its grader runs, and returns the normalized record, a review protocol, a verification-subagent prompt, and what submit still needs. A draft whose gold fails its grader is still saved.
• Typed errors: grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

evals_get_record tool

• Reads a draft or submitted record by id, which stays stable across submit.
• Returns the full grader, discrimination cases, and verification evidence; not_found points to evals_list_records.

evals_revise_draft tool

• Applies set, append, and unset by dotted path to a draft; task_type and server-owned fields cannot be changed, and arrays are replaced whole or appended, not edited by index.
• Re-validates the record and grader before writing, then returns the record plus changed entries (op, path, before, after). An unknown path rejects the whole patch.
• Typed errors: not_found, record_frozen for a submitted id, invalid_patch_path, grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

evals_discard_draft tool

• Deletes a draft by draft_id and returns discarded: true.
• A submitted record reports record_frozen; a missing or already discarded id reports not_found.

evals_run_check tool

• Runs a grader against one or more candidates without a saved record; supply gold for exact_match, while target-embedding kinds such as numeric and mcq ignore it.
• Returns PASS/REJECT, detail, and the resolved reference per candidate. llm_rubric can't be run here.
• Numeric grading accepts only finite real values; unit-bearing, complex, nonfinite, and non-scalar candidates are rejected. units is documentation only.
• Typed errors: grader_unexecutable, mcq_choice_mismatch.

evals_submit_draft tool

• Finalizes by draft_id: gold must PASS, ≥1 negative must be REJECTED, and decorrelated verification must agree. Also checks captures against EVALS_CAPTURE_DIR, rejects duplicate content_hash, and can ask for confirmation via confirm or EVALS_REQUIRE_CONFIRMATION.
• On success, freezes the record with submitted_at and checksum and returns status: submitted; a refusal leaves the draft intact. free_response records rely on verification alone and carry server_verified: false.
• Typed errors: not_found, record_frozen, verification_incomplete, grader_failed_on_gold, verification_disagrees_with_gold, missing_negative_case, negative_case_passed, duplicate, decorrelation_violation, capture_unresolved, submit_declined.

evals_list_records tool

• Filters by status (draft/submitted), domain, task_type, or tag; limit is a positive integer and defaults to 50.
• Returns summaries (id, status, task_type, domain, tags, timestamps), newest-first, with shown, cap, totalCount, and truncated.

evals_export_records tool

• Exports submitted records, optionally filtered by domain, task_type, or tag, as jsonl (lossless), csv (lossy summary), inspect (Inspect AI), or lm-eval (lm-evaluation-harness).
• Writes under exports/ and returns the path, record count, byte size, and preview.

eval://record/{id} resource

• Reads id from evals_list_records or a draft/submit response.
• Returns the evals_get_record payload as application/json, or not_found when no record matches.

Features

Built on @cyanheads/mcp-ts-core: stdio and Streamable HTTP transports, pluggable auth (none / jwt / oauth), swappable storage (in-memory, filesystem, Supabase, Cloudflare KV/R2/D1), structured logging with optional OpenTelemetry tracing.

Eval authoring:

• A draft → review → surgical-revise → submit loop, with the server both storing records and running each record's own grader to reject what doesn't hold up
• Seven task types: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
• A grader serialized with each record; deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side, while llm_rubric relies on recorded independent verification
• A submit gate: the gold must pass its own grader, ≥1 negative must be rejected, and a recorded decorrelated verification must agree with the gold
• Plain JSON files under EVALS_DATA_DIR, with drafts, submitted records, and exports kept separate

Agent-friendly output:

• evals_create_draft and evals_revise_draft return the parsed record, a per-field review protocol, and a ready-to-paste verification-subagent prompt
• Draft and revise responses report pass/reject results for each positive and negative case
• evals_list_records reports shown / cap / total count when the limit is hit
• The submit gate fails with a typed reason plus a recovery hint

Getting started

Add the following to your MCP client configuration file. Set EVALS_DATA_DIR to a writable folder — the server manages drafts/, submitted/, and exports/ under it.

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "bunx",
      "args": ["@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with npx (no Bun required):

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with Docker:

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "MCP_TRANSPORT_TYPE=stdio",
        "-v", "evals-data:/usr/src/app/evals-data",
        "ghcr.io/cyanheads/evals-mcp-server:latest"
      ]
    }
  }
}

For Streamable HTTP, set the transport and start the server:

MCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 EVALS_DATA_DIR=./evals-data bun run start:http
# Server listens at http://localhost:3010/mcp

Prerequisites

• Bun v1.4.0 or higher (or Node.js v24+).
• A writable directory for EVALS_DATA_DIR. No external API key is required.

Installation

1. Clone the repository:

git clone https://github.com/cyanheads/evals-mcp-server.git

1. Navigate into the directory:

cd evals-mcp-server

1. Install dependencies:

bun install

1. Configure environment:

cp .env.example .env
# edit .env and set EVALS_DATA_DIR

Configuration

All server configuration is validated at startup via Zod schemas in src/config/server-config.ts.

See .env.example for the full list of optional overrides.

Submit confirmation expires after 10 minutes and must be repeated if the draft changes. When running more than one instance, use shared storage (STORAGE_PROVIDER_TYPE=filesystem with a shared STORAGE_FILESYSTEM_PATH, supabase, or cloudflare-d1; not cloudflare-kv) and set the same MCP_REQUEST_STATE_KEY (at least 32 bytes) on each.

Running the server

Local development

• Build and run:

  # One-time build
  bun run rebuild
  
  # Run the built server
  bun run start:stdio
  # or
  bun run start:http

• Run checks and tests:

  bun run devcheck   # Lint, format, typecheck, security
  bun run test       # Vitest test suite
  bun run lint:mcp   # Validate MCP definitions against spec

Docker

docker build -t evals-mcp-server .
docker run --rm -e MCP_TRANSPORT_TYPE=stdio -v evals-data:/usr/src/app/evals-data evals-mcp-server

The Dockerfile defaults to HTTP transport, stateful session mode, and logs to /var/log/evals-mcp-server. Mount record storage at /usr/src/app/evals-data, the default directory owned by the runtime user. OpenTelemetry peer dependencies are installed by default — build with --build-arg OTEL_ENABLED=false to omit them.

Project structure

Development guide

See CLAUDE.md/AGENTS.md for development guidelines and architectural rules. The short version:

• Handlers throw, framework catches — no try/catch in tool logic
• Use ctx.log for request-scoped logging; records persist to disk via the record-store service, not ctx.state
• Register new tools and resources in the createApp() arrays in src/index.ts
• The server is the source of truth — validate inputs, run the grader as a hard gate, and never admit a record on assertion alone

Contributing

Issues are welcome. Run checks and tests before submitting:

bun run devcheck
bun run test

License

Apache-2.0 — see LICENSE for details.