Spoor

Spoor

Spoor tells an AI agent what changed in your services, and where it started, in a few hundred tokens.

Website: telespoor.com

Spoor is an open-source MCP server and CLI for your observability backends: the VictoriaMetrics stack, Prometheus, Loki, Tempo and New Relic. When something breaks, an agent asks Spoor "what changed in the last 30 minutes?" and gets back a short, ranked answer across metrics, logs and traces, with the service where the failure started at the top and its echoes in other services labelled as such.

Spoor (n.): the track or trail left by an animal. Spoor follows the trail a failure leaves in your telemetry back to where it started.

$ spoor what-changed --since 30m

{
  "summary": "11 change(s) in 3 services. Strongest: payment — Failures start here, in \"POST /charge\" (100% of 100 failed traces). Also: payment — Span \"POST /charge\" errors 0.2% → 43.7% (229 spans). … Errors not significantly up (not new) in: product-catalog, recommendation, ad.",
  "findings": [
    {"service": "payment",  "signal": "traces",  "kind": "error_origin",    "score": 0.99, "title": "Failures start here, in \"POST /charge\" (100% of 100 failed traces)"},
    {"service": "payment",  "signal": "logs",    "kind": "new_log_pattern", "score": 0.94, "title": "New error log pattern: \"charge failed: card processor returned <N> for order <N>\""},
    {"service": "payment",  "signal": "metrics", "kind": "error_rate_up",   "score": 0.92, "title": "Error rate up: 0.2% → 40.5%", "onset": "2026-10-16T09:50:00Z"},
    {"service": "checkout", "signal": "logs",    "kind": "spiking_log_pattern", "score": 0.74, "title": "Error log pattern spiking: \"call to payment.Charge failed with status <N>\" — originates in payment"}
  ],
  "coverage": {"metrics": "ok", "logs": "ok", "traces": "ok"}
}

Findings carry evidence links to the query behind them (when the token budget allows), so a human can check the agent's work in one click.

Why use it

In a microservice failure, the broken service and every caller of it all light up. An agent with only raw query tools has to decide what to query, compare against "normal" itself, page through results and work out which service is the cause and which are echoes. It burns tool calls and tokens, and often blames the loudest symptom or an error that was always there.

Spoor does that work in plain, deterministic Go and hands the agent the conclusion's evidence:

• Cross-signal: request rate, error rate, latency, CPU, memory and consumer lag; new and spiking errors and error volume from logs; per-span errors and latency, all for the same window against the same baseline.
• Built for real logs: messages that differ only by IDs, values, stack traces or log prefixes count as one error; JSON errors are read by their message and code fields; services can use different level fields or put the level in the text; spoor doctor finds your fields and writes the config.
• Cause vs. echo: trace trees show where failures start and where latency is added. Callers that merely suffer are demoted and labelled — originates in payment.
• Steady errors called out: errors that were already there are listed as steady_errors, "not a new problem", so agents stop blaming them.
• Bounded output: every tool takes budget_tokens (default 800).
• Honest about gaps: coverage says when a backend failed or had no data, the baseline was empty, the error filter matched nothing, or lines lack a service or level field. "No change" never stands in for "can't see".
• No LLM inside, read-only, redacted: same input, same output; only query endpoints are ever called; every text field is scrubbed of emails, tokens, IPs and generated IDs before it leaves.

Measured

The benchmark runs Claude Code headless on six incidents (five faults and a no-fault control), five times each, once with the official VictoriaMetrics MCP servers alone and once with Spoor added. On Claude Haiku, 60 isolated sessions:

Synthetic incidents and one small model, so treat it as a first data point.

On real production logs (about 25 services, mixed formats, read-only replay): the digest was 99.8% smaller than the raw evidence it replaces, with zero candidates against four from v0.1.1, and its baseline counts matched independent queries exactly. It was not faster than four hand-written queries (28 s vs 26 s), and accuracy on real incidents is not yet measured.

Method, per-scenario results, the production replay and caveats: docs/benchmark.md.

Quickstart

Requires Go 1.25+, Docker and jq. About five minutes.

git clone https://github.com/asinha07/spoor && cd spoor
make it-up          # empty VictoriaMetrics, VictoriaLogs, VictoriaTraces in Docker
make demo-data      # six synthetic incidents; ground truth in scenarios/out/
make build

T=scenarios/out/payment-failure.truth.json
./bin/spoor what-changed --since $(jq -r .since $T) --until $(jq -r .until $T) --pretty

Connect it to Claude Code:

claude mcp add spoor \
  -e SPOOR_METRICS_URL=http://localhost:8428 \
  -e SPOOR_LOGS_URL=http://localhost:9428 \
  -e SPOOR_TRACES_URL=http://localhost:10428 \
  -- "$PWD/bin/spoor" serve

Run it from the spoor directory (or add -s user to register it everywhere), then ask Claude: "Something is failing in the shop. What changed, and where did it start?"

On your own backends, start with spoor doctor --since 15m: it checks what Spoor can see and suggests config. Other clients, Grafana-only access and the rest: Getting started.

Backends

Each signal picks its backend with type:, and you can mix them:

metrics: {type: prometheus, url: http://prometheus:9090}
logs:    {type: loki, url: http://loki:3100}
traces:  {type: tempo, url: http://tempo:3200}

Setup and what differs per backend (Tempo statistics are sampled; New Relic span changes need the service's metrics to agree): Backends.

Tools

Inputs: service, since, until, baseline (previous, 1d, 7d), budget_tokens. Full reference: docs/tools.md.

How it works

flowchart LR
    Agent[AI agent / CLI] -->|MCP| S[Spoor]
    S -->|"PromQL / NRQL"| M[(VictoriaMetrics, Prometheus, New Relic)]
    S -->|"LogsQL / LogQL / NRQL"| L[(VictoriaLogs, Loki, New Relic)]
    S -->|"LogsQL / TraceQL / NRQL"| T[(VictoriaTraces, Tempo, New Relic)]
    S -->|"ranked digest ≤ budget"| Agent

For a window and a baseline (by default, the six windows just before it), Spoor scores every metric with a robust z-score, compares error-log patterns with a Poisson test, compares span error rates and p95s, walks failed traces to the span where each failure started, boosts changes confirmed by more than one signal, and trims the result to the token budget. Details: architecture · scoring and attribution.

Run it for a team

spoor serve --http :8080 --auth-token … serves MCP over streamable HTTP with bearer-token auth, /healthz and Prometheus /metrics. Ready-made, hardened manifests:

• Kubernetes (Kustomize): deploy/kubernetes/
• Docker Compose: deploy/docker/
• systemd: deploy/systemd/

Security, sizing, monitoring and upgrades: Production guide.

Verified on the OpenTelemetry Demo

On the OpenTelemetry Demo v3.1.0 (about 20 real services), a 50% payment failure came back as "payment: failures start here, in charge", with checkout and frontend errors labelled "originates in payment" (732 tokens); the adHighCpu fault came back as the single finding "ad: CPU up 0.07 → 3.97 cores" (192 tokens). Reproduce it: deploy/demo/.

Documentation

Status and limitations

Early (pre-1.0). Tool output fields may still change between minor versions; see the changelog.

• The scoring was developed and benchmarked on the VictoriaMetrics stack. Prometheus, Loki and Tempo pass the same scenario tests against real containers; New Relic was checked on one real account. Tempo statistics are sampled, and New Relic span changes are reported only when service metrics agree; see backends.
• Cause/echo attribution needs traces; without them Spoor still ranks metric and log changes.
• Attribution reads at most 50,000 span rows per read; on very busy systems a trace can be cut.
• The benchmark uses synthetic incidents written by the author and one model; treat it as a first data point, and replay your own past incidents with bench/replay.py before relying on it.
• Metrics and traces are queried per service; on large estates pass service when you know it.

More in how-it-works.md.

Roadmap

Next: accuracy on labelled real incidents, "since the last deploy" windows, Datadog and Elasticsearch backends, exact Tempo statistics through TraceQL metrics. The full list, and what is deliberately out of scope: docs/roadmap.md.

Contributing

Issues and pull requests are welcome. make test needs nothing but Go; see CONTRIBUTING.md. Security issues: SECURITY.md.

License

Apache-2.0. See LICENSE.