TestChimp

@testchimp/cli

TestChimp CLI and MCP server for calling TestChimp /api/mcp/* endpoints with TESTCHIMP-API-KEY.

This repository folder may still be named testchimp-mcp-client locally; the published npm package is @testchimp/cli.

Install

npm install -D @testchimp/cli@latest

MCP (agents in Cursor, Claude Code, VS Code, etc.)

Register the server so the host runs:

npx -y @testchimp/cli@latest mcp

Example mcpServers.testchimp:

{
  "command": "npx",
  "args": ["-y", "@testchimp/cli@latest", "mcp"],
  "env": {
    "TESTCHIMP_API_KEY": "your-project-key",
    "TESTCHIMP_BACKEND_URL": ""
  }
}

The config file path depends on the host (e.g. Cursor often uses <repo>/.cursor/mcp.json). Tool names use kebab-case (e.g. get-requirement-coverage, create-user-story).

CLI

export TESTCHIMP_API_KEY=...   # required unless TESTCHIMP_OAUTH_TOKEN is set (often read from project MCP env; never commit keys)
testchimp --help
testchimp get-requirement-coverage --branch-name main --help
testchimp create-user-story --platform-file-path plans/stories/foo.md --title "Checkout"
testchimp list-screen-states --json-input '{}'
testchimp upsert-screen-states --json-input '{"screenStates":[{"screen":"Checkout","states":["empty","filled"]}]}'
testchimp list-perf-runs --testchimp-id TC-123 --kind JOURNEY --limit 20
testchimp list-perf-runs --run-id 01ABC --include-raw
testchimp promote-perf-baseline --run-id 01ABC --env-class CI
testchimp compare-perf-to-baseline --run-id 01ABC --max-p95-regression-percent 10
testchimp list-related-perf-tests --scenario-titles "Checkout,Refund"
testchimp list-api-operation-interactions --operation-id 01XYZ --interaction-type REAL --limit 100
testchimp list-meetings --include-filter-options
testchimp list-meetings --from 2026-09-01 --to 2026-09-30 --domain customer.com --label Sales --search "pricing"
testchimp get-meeting-transcript --meeting-id <meeting-id> --summary-only
testchimp ephemeral-environment --action PROVISION --branch-name feat --no-wait
testchimp report-scan-findings --id scan-1 --scanner DAST --report-file zap.json
testchimp upsert-issue --title "Broken checkout"
testchimp list-issues --status ACTIVE --limit 10
testchimp get-release --release 1.2.0
testchimp upsert-release --release 1.2.0
testchimp manage-test-run --release 1.2.0 --title "Sprint 12"

• stdout: API response JSON.
• stderr: progress for ephemeral-environment --action PROVISION when waiting (“still waiting…” polls). Pass --no-wait for create/deploy trigger only.
• performance gate: compare-perf-to-baseline still prints its JSON response but exits nonzero when regressed is true (top-level or under comparison).
• Meetings: list-meetings covers team-wide Meeting Bots meetings only (same filters as the Meetings page). --include-filter-options returns label / participant / domain catalogs. --from / --to take YYYY-MM-DD (inclusive local days), ISO datetimes, or epoch millis; --label, --participant, --domain are repeatable or comma-separated.
• Flags: default for each subcommand; --json-input '<json>' or --json-input @file.json merges over flags (JSON wins on conflicts). Use JSON for nested bodies (e.g. TrueCoverage scopes).
• Release labels: use --release <label> (alias --release-version) on get-release, upsert-release, manage-test-run, and other release-filter commands. --version / -V print the CLI version only.

Authentication and environment

Workspace folder mapping

Map a local repository folder to a TestChimp project for the current user. The mapping lives in ~/.testchimp/projects.json (or $TESTCHIMP_HOME/projects.json), the same file TestChimp Studio uses, so a folder mapped from the CLI shows up in Studio and in the headless AgentWatch daemon.

testchimp workspace map --project-id <id> --folder ~/code/shop [--project-name "Shop"] [--reassign] [--skip-repo-check]
testchimp workspace get --project-id <id>        # prints the mapping JSON; exit 1 when unmapped

workspace map follows the same rules as Studio:

• The folder must exist and be a git work tree. Paths are stored as canonical real paths.
• When TESTCHIMP_API_KEY / TESTCHIMP_OAUTH_TOKEN is set, the CLI loads the project's connected repository (get-git-folder-mapping). If one is connected, the folder must be the repository root and one of its git remotes must match (owner/repo, case-insensitive). If no credential is set, no repo is connected, or the lookup fails, the check is skipped with a note on stderr. The credential's own project is used for this lookup, so run it with the same project's key or bot token. --skip-repo-check skips the lookup entirely.
• A folder belongs to one project. Mapping it to a second project fails unless you pass --reassign, which moves it (and drops the old project's entry if that leaves it with no folders).
• The file is written atomically (temp file, then rename) with mode 0600. Other projects are left alone, and unknown fields written by newer Studio versions are kept. A file that is not valid JSON or does not match schema v2 is moved to projects.invalid.<millis>.json and replaced with an empty registry. Schema v1 files are migrated to v2.

The byte-level format is pinned by fixtures/projects-registry/, with an identical copy in the Studio repo. Both test suites replay the same upsert sequences and compare the output bytes.

Project key for local runners (workspace save-creds)

Local runners (the Playwright reporter, this CLI, k6) need the project API key in their environment. TestChimp Studio keeps it in <repo>/.testchimp/mcp.json; remote MCP (OAuth) users write the same file once per repo:

# CLI: copy TESTCHIMP_API_KEY from the environment (or fetch when TESTCHIMP_OAUTH_TOKEN is set)
testchimp workspace save-creds --folder . --project-id <id> \
  [--project-name "Shop"] [--backend-url <backendUrl>] [--ingress-url <ingressUrl>] [--reassign]

# Local stdio MCP: tool save-workspace-creds { folder, projectId? } — never returns the key
# Hosted / plugin: get-project-credentials, then write .testchimp/mcp.json with a file-write tool
# Legacy: pipe the key on stdin (never as a command-line argument)
printf '%s' "$KEY" | testchimp workspace save-creds --folder . --project-id <id>

• Never put the key in a command. Stdin and TESTCHIMP_API_KEY still work. The key is never printed.
• The file uses Studio's exact format (pinned by fixtures/workspace-mcp-json/, mirrored in the Studio repo): mode 0600 in a 0700 .testchimp/ directory, other servers in the file kept, .testchimp/ added to .gitignore. The folder is also mapped in projects.json like workspace map.
• An existing entry for the same project is kept as is (including Studio's managed binary); only a changed key is refreshed. Another project's entry is refused unless --reassign.

Fallback: when neither TESTCHIMP_API_KEY nor TESTCHIMP_OAUTH_TOKEN is set, the CLI reads TESTCHIMP_API_KEY, TESTCHIMP_PROJECT_ID, TESTCHIMP_BACKEND_URL and TESTCHIMP_INGRESS_URL from the nearest .testchimp/mcp.json at or above the current directory. Exported variables always win, and --bot bindings are unaffected. The file must be private to you (mode 0600, as Studio and save-creds write it); a group- or world-readable copy, such as one committed to a cloned repo, is ignored with a warning. Other runners (Playwright, k6) still need the variables exported in their own shell.

QA bots

testchimp get-my-tasks --user-id <user-id>      # OAuth tokens imply the user
testchimp list-tests-awaiting-verification --limit 20
testchimp get-qa-posture
testchimp bot get-profile
testchimp bot register-profile --role QA_ENGINEER --responsibilities "Checkout + payments" \
  --capability E2E_AUTHORING --capability TEST_BATCH_FIX \
  --subscriptions-json '[{"eventType":"git-push","filters":[{"field":"author","op":"eq","value":"me"}]},{"eventType":"e2e-batch-completed"}]'
testchimp bot ack <eventId> [<eventId>...] --ack-url <delivery ackUrl>
testchimp bot compat --skill-version 1.0.53

bot ack prints one eventId<TAB>status line per id and exits non-zero when any status is BOT_ACK_UNKNOWN_EVENT, BOT_ACK_NOT_A_TARGET, or BOT_ACK_MISSING_BOT_ID. An explicit --ack-url must be https (http only on localhost) and point at a TestChimp host or the TESTCHIMP_INGRESS_URL host. bot compat exits 0 and prints the deployment minimums plus cliUpgradeRequired / skillUpgradeRequired.

AgentWatch without TestChimp Studio

testchimp bot connect --project-id <id>       # browser approval (OAuth, "agentwatch" scope)
testchimp bot connect --pair --project-id <id> # or: no browser, your QA bot approves (see below)
npx -y @testchimp/agentwatch query --project-id <id>
testchimp bot disconnect --project-id <id>    # forget the stored keys

Headless AgentWatch acts as the user, so it needs their user id, personal access key and the project API key. bot connect runs an OAuth 2.1 PKCE login with a loopback redirect, asks for the opt-in agentwatch scope (the consent page warns that keys will be stored locally), fetches the keys from /bots/get_agentwatch_credentials, and writes them to ~/.testchimp/agentwatch/credentials.json (mode 0600, keyed by project) with the backend and ingress URLs. It revokes the OAuth refresh token straight away and never prints the keys. Approving also opts the project in to AgentWatch.

Pairing (no second browser consent; what QA bots use). Every connection approved with Use this connection as my QA bot carries the agentwatch_pair scope, so the keys can reach the user's computer without another browser login, and without passing through the bot:

1. On the user's computer: testchimp bot connect --pair [--project-id <id>] keeps a random verifier in ~/.testchimp/agentwatch/pairing.json (0600) and prints {pairingCode, expiresAtMillis}; the code is the verifier's SHA-256 (base64url).
2. The bot approves it with its own token: MCP tool approve-agentwatch-pairing (or testchimp bot approve-pairing <code>).
3. On the user's computer: testchimp bot connect --finish-pair redeems with the verifier (polls up to --timeout-ms, default 60 s), stores the keys like above and deletes the pending file.

Pairings are single use and expire after 10 minutes. The bot only ever sees the pairing code, which is useless without the verifier.

Remote MCP (Streamable HTTP)

testchimp mcp --http [--port 8080] [--host 0.0.0.0]

Stateless Streamable HTTP at POST /mcp. Every request must carry Authorization: Bearer <OAuth access token>; the caller's token is forwarded to TestChimp for each tool call, and the server's own TESTCHIMP_API_KEY / TESTCHIMP_OAUTH_TOKEN / TESTCHIMP_BOT_ID are never used. Requests without a bearer get 401 with WWW-Authenticate: Bearer resource_metadata="…". Other endpoints: GET /.well-known/oauth-protected-resource (RFC 9728 metadata), GET /healthz. Request bodies are capped at 1 MB.

Per-project URLs. POST /mcp?projectId=<id> scopes every tool call to that project: it is sent as TestChimp-Project-Id, and TestChimp checks that the token's user is a member (an explicit projectApiKey argument still wins). Commit one URL per repo (e.g. {"url": "https://mcp.testchimp.io/mcp?projectId=<id>"}); the same sign-in works for every project. The 401 challenge and the protected-resource metadata echo the query, so the advertised resource matches the configured URL. Plain /mcp keeps using the project picked on the consent page. Hosted / plugin agents persist the runner key by calling get-project-credentials and writing <repo>/.testchimp/mcp.json with a file-write tool (see Workspace folder mapping). Local stdio MCP can call save-workspace-creds.

The repository Dockerfile builds an image that runs testchimp mcp --http as a non-root user (Cloud Run ready):

docker build -t testchimp-mcp .
docker run -p 8080:8080 -e TESTCHIMP_MCP_PUBLIC_URL=https://mcp.example.com testchimp-mcp

MCP API 2.0.0

@testchimp/cli 2.0.2 matches TestChimp MCP API 2.0.0 (88 tools). 2.0.2 adds ask-user-question-async. 2.0.0 shipped 87 tools, including all 70 /api/mcp/* paths plus bot/workspace helpers. Several 1.x tools were consolidated:

get-eaas-config is kept as a standalone connectivity-check tool. destroy-ephemeral-environment is unchanged and stays destructive. Full mapping: CHANGELOG.md and MIGRATION.md.

Migration from testchimp-mcp-client

The npm package testchimp-mcp-client is superseded by @testchimp/cli. Update MCP args to ["-y", "@testchimp/cli@latest", "mcp"] and rename tool references to kebab-case. See MIGRATION.md.

License

MIT