brief bar_

Brief Bar — scoring service (briefs.welance.com)

→ briefs.welance.com — the public pages, live

Eight languages, no build step, no tracking. Same origin serves the pages and the API.

Scores a digital product brief against an open, versioned ruleset. The score says how good a brief is; a separate gate says whether it may publish. An LLM only ever judges (server-side, key never leaves the box); deterministic code owns every number, the gate, and the decision.

The product target is a brief that makes expectations explicit and open to discussion, so client and supplier can understand whether and how to work together before committing. See the product decision and audit of current rule coverage. The audit identifies proposed improvements; it does not change current scoring.

This repo is the service. The ruleset + engine live inside it as an installable package (brief_bar/) so they can later be split into their own OSS repo and consumed here as a pinned dependency — the seam is already drawn.

brief_bar/     the open ruleset + deterministic engine (the OSS core)
  rules/*.yaml       14 rules, weights sum to 100, ★ = gate requirement
  scoring.yaml       the €10k floor, the 4-requirement gate (anonymised is directory-context, deactivatable), the bands
  score.py           weighted average + gate + decision (no model here)
  judge.py           MockJudge (keyword, offline) + LLMJudge protocol
  llm.py             batched-judge & suggestion prompts (versioned with rules)
  fixtures/*.yaml    labelled briefs = the regression corpus / CI immune system
app/                 the FastAPI service
  main.py            routes, rate limit, CORS; mounts site/ at /
  scorer.py          mock/LLM orchestration + Redis verdict cache
site/                THE public pages: landing, console, rules — served both
                     by GitHub Pages and by the service itself (one surface)
CLAUDE.md · PLAN.md  bootstrap + phased mission for an agentic coding session
GOVERNANCE.md · CONTRIBUTING.md · CODEOWNERS · LICENSE   the open-bar machinery (MIT)
docker-compose.yml   api + redis (the only stateful dependency: ephemeral cache)

Quick start

cp .env.example .env          # add a spend-capped PB_OPENROUTER_API_KEY
make up                       # docker compose up --build -d  (api + redis)
make health                   # GET /v1/healthz
open http://localhost:8000    # the console (mock judge works with no key)

Without an API key the service runs mock-only (instant keyword engine, fully deterministic) — enough to develop against and to run the whole test suite. Add the key to unlock the llm judge and /v1/suggest.

API

The console can import PDF, UTF-8 TXT and Markdown documents up to 10 MB (10,000,000 bytes). Text extraction runs locally using the vendored PDF.js display API and worker; the original file is not uploaded. Review the extracted text before explicitly scoring with AI. Imported-document scoring sets no_cache: true.

The API's separate text limit defaults to 1,000,000 characters (PB_REQUEST_MAX_CHARS). Existing environment overrides still apply. Provider context limits may be lower; a successful import does not guarantee that every model can judge the document. Text is never silently truncated or summarised. PDF extraction is limited to 1,000 pages and 60 seconds; pages without extractable text, password-protected PDFs and malformed files are refused. Run OCR or remove blank pages before import when needed. There is no multipart/file storage API.

POST /v1/score

{ "brief": "…", "locale": "it", "judge": "mock" }   // judge: "mock" | "llm"

{ "score": 92.0, "band": "Directory-ready", "decision": "blocked",
  "decision_label": "Blocked — hard requirements unmet",
  "gate": { "passed": false, "missing": ["anonymised"] },
  "verdicts": [ { "rule_id": "anonymised", "status": "fail", "confidence": 0.8,
                  "quote": "Stripe … mara@acme.it", "note": "identifying info present",
                  "weight": 8, "severity": "high", "gate": "pass" }, … ],
  "review_required": false, "low_confidence": [],
  "ruleset_version": "1.0.0+83107baec655", "engine": "brief-bar@1.0.0+83107baec655",
  "judge": "mock", "cached": false }

The ruleset_version is your audit trail: it pins exactly which bar judged a brief, and every verdict carries a verbatim quote from the brief as evidence — inspectable by a human long after any particular model is gone. (The mock judge is reproducible forever; the LLM judge runs at temperature 0 and is cached by (ruleset_version, model, brief), but temperature 0 is not a determinism guarantee across provider updates — the quotes are the part of the trail that never expires.)

Welance uses OpenRouter (PB_OPENROUTER_API_KEY). Publish scoring runs on deepseek/deepseek-v4-pro; interactive suggestion generation runs on google/gemini-3.1-flash-lite. Self-hosters may still configure the optional direct-Anthropic adapter, but it is not part of the Welance deployment. With OpenRouter, requests may pick a model from the server's allowlist (PB_OPENROUTER_MODELS, exact vendor-prefixed slugs, comma-separated); anything else is rejected with 422. The resolved model is returned in every score for a reproducible audit trail. Suggestion generation may use a lower-latency model selected by PB_SUGGEST_MODEL. Single-rule suggestions are editable human choices and default to one generation call; set PB_SUGGEST_VERIFY=true to restore a second LLM review. The authoritative score and the multi-gap repair loop keep their verification behavior. Suggestion responses name their verifier model when one ran. The service reuses its OpenRouter connection pool and applies separate bounded output ceilings (PB_SUGGEST_MAX_TOKENS, PB_SUGGEST_ALL_MAX_TOKENS, and PB_VERIFIER_MAX_TOKENS) because short suggestion JSON does not need the judge's larger allowance. Successful service-funded suggestions are cached; BYOK calls always bypass that shared cache.

The judge normally asks for all fourteen verdicts in one provider call. PB_JUDGE_BATCH_SIZE=5 instead runs deterministic 5/5/4 rule batches with at most PB_JUDGE_CONCURRENCY=3 calls in flight, then merges them before the same deterministic aggregation. It is an operational latency option, not a scoring mode: any incomplete or failed batch refuses the entire score. Keep it at its default 0 until the live make test-llm-batching comparison has passed for the deployed model.

Bring your own key: send X-LLM-Key: <your OpenRouter key> and the call runs on your key — any valid OpenRouter model slug is allowed unless this deployment sets PB_BYOK_MODELS (you pay), used per request, never stored or logged. BYOK scoring deliberately bypasses the shared verdict cache, so the supplied key always funds a fresh call. Use a spend-capped key. The console exposes this as an optional field in live mode.

Sending a key to somebody else's server deserves more than that paragraph. Every line that touches it, where it provably is not, and the honest limits are at briefs.welance.com/security.html; the claim is enforced by tests/test_byok_leak.py, which fails if the key ever reaches a log, a cache write, or a response.

Other endpoints:

• POST /v1/suggest {brief, rule_id, locale, model?} → tailored fixes for one gap (LLM).
• POST /v1/suggest/all {brief, rule_ids?, locale, model?} → one fix per failing gap; omit rule_ids to auto-detect (LLM).
• GET /v1/rules → the full catalogue (id, title, weight, gate, criteria, references) for the directory UI.
• GET /v1/models → the enabled judge models (default + available) for a model picker.
• GET /v1/healthz → service release, ruleset version, engine + LLM availability.
• GET / → the interactive console.

Suggestion responses include X-PB-Generation-Ms, X-PB-Verification-Ms, X-PB-Cached, X-PB-Cache-Ms, X-PB-Model, and the standard Server-Timing header, so callers can distinguish model latency from network/UI latency without exposing prompts or document contents. /v1/healthz also reports whether the optional Redis cache/rate-limit store is currently connected.

MCP: free, bring your own model

https://briefs.welance.com/mcp is a free remote MCP server (Streamable HTTP, stateless JSON). No model of ours runs. Your assistant's own model judges the brief with the service judge's own prompt; the bar's code validates those verdicts and does the arithmetic with the same engine as /v1/score.

Also as prompts (score_with_your_model — the whole workflow, gate first — judge_brief, improve_brief, brief_template) and resources (perfect-brief://rules, …/template, …/judge-prompt, …/verdict-schema). The same contract is plain HTTP at /v1/byo/* (see /docs).

Advisory, never authority. An aggregated result is tagged evaluation_source: "client", authoritative: false, with the ruleset version, evidence errors and pattern conflicts, and a Find a team on welance/directory → link (PB_DIRECTORY_URL). It cannot authorise publication: the welance/directory scores every brief itself on /v1/score.

Install

Claude (claude.ai, Desktop): Settings → Connectors → Add custom connector → https://briefs.welance.com/mcp. Claude Code:

claude mcp add --transport http perfect-brief https://briefs.welance.com/mcp

ChatGPT: Settings → Apps & Connectors → Advanced → enable developer mode → Create → URL https://briefs.welance.com/mcp, no authentication (availability depends on the plan).

Cursor (~/.cursor/mcp.json or .cursor/mcp.json):

{ "mcpServers": { "perfect-brief": { "url": "https://briefs.welance.com/mcp" } } }

VS Code (.vscode/mcp.json, or MCP: Add Server… → HTTP):

{ "servers": { "perfect-brief": { "type": "http", "url": "https://briefs.welance.com/mcp" } } }

Local (stdio) — same tools, for any client that runs commands:

{ "mcpServers": { "perfect-brief": { "command": "uvx",
  "args": ["--from", "git+https://github.com/welance/perfect-brief#subdirectory=mcp-server", "perfect-brief-mcp"] } } }

Protocol: revisions 2024-11-05 → 2025-11-25 negotiate through initialize (an unknown one is answered with 2025-11-25); the stateless 2026-07-28 envelope (server/discover) is served too. JSON-RPC batches are refused with 400. No session, no event stream.

The older connector whose score_brief/suggest_fixes spend the service judge lives at /mcp-hosted, off unless PB_MCP_HOSTED_ENABLED=true, and then under the same per-address limits. A draft Official MCP Registry entry is in registry/.

How welance.com/directory consumes it

The directory is a pure consumer. It renders the verdict; it holds no scoring logic:

const res = await fetch("https://briefs.welance.com/v1/score", {
  method: "POST",
  headers: { "content-type": "application/json", "x-api-key": KEY },
  body: JSON.stringify({ brief, locale, judge: "llm" }),
}).then(r => r.json());
// res.decision drives publish/blocked/with-reservation; res.gate.missing lists why.
// persist res together with res.ruleset_version for a reproducible audit trail.

Score as the user types (debounced) and on submit. otto.welance.com calls the same endpoint server-to-server when it needs to persist a verdict.

Design notes

• Stateless service. The score is a pure function of (brief, ruleset_version). There is no database. Redis holds only an ephemeral verdict cache + rate-limit counters, and the service degrades gracefully if Redis is down.
• Ruleset as a dependency. brief_bar/ is an installable package with its own version and CI corpus. Today it's vendored here; extracting it to welance/brief-bar and pinning a tag is a lift-and-shift, no code change.
• The gate replaces severity caps. A critical gap is an explicit publish requirement (clear-title, problem-defined, budget-floor, anonymised), not a quiet penalty on the number.
• The Operator Covenant. The rules judge briefs; OPERATOR-COVENANT.md judges us — no lead-mining, no training on briefs, the same bar for welance's own briefs (welance pitches on the Directory as a team), with each commitment labelled by how it is enforced, CI-tested where code can reach.
• Governance. Anyone proposes a rule change via PR; the fixture corpus in CI is the immune system — a change that moves the numbers must move the fixtures too, in the open. Rule-change PRs run on a 7-day community discussion window before anyone merges — the PR template walks you through what a complete proposal contains, and GOVERNANCE.md spells out the clock. Disagreeing with the bar is a first-class use of this repo.

Decisions

Architecture decisions with real stakes are recorded as ADRs in docs/decisions/:

• 0001 — cross-model verification: why a second model reviews every AI suggestion (self-preference bias), and why the reviewer currently shares a lab with the generator (cost; one env line restores cross-lab review).
• 0002 — default model selection: the criteria, the date-stamped price snapshot behind the DeepSeek V4 pro/flash pairing, and when to revisit.

Develop & test

pip install -e ".[dev]"
make test        # fixture corpus + API tests, all on the deterministic mock judge
make lint        # ruff
make typecheck   # mypy
make dev         # uvicorn --reload on :8000

Deploy (OVH Kubernetes)

The image is a stateless HTTP service on :8000 with a /v1/healthz liveness probe — a standard Deployment + Service + Ingress, plus a small Redis (a cache, so a single replica or a managed instance is fine). Set PB_* via a Secret/ConfigMap; lock PB_CORS_ORIGINS to the directory's origin. Scale the API horizontally; the cache is shared through Redis.

Text and document boundary

The API accepts text, not uploads: brief is one bounded string. The Directory may read supported PDF, DOCX and text-family documents locally in the browser and include their extracted text as untrusted context. Original bytes, images and arbitrary binaries are not sent to this service, stored by it, or published as attachments. Multipart publication to Directus is a separate Directory flow.

The bundled console defaults to the mock judge (offline, instant). Its live mode calls this service's /v1/score and /v1/suggest; provider credentials stay server-side unless a caller explicitly supplies a per-request BYOK key.