English Français العربية

io.github.bch1212/modelwatch

Official registry AI and agents

Continuous behavioral drift monitoring for LLM apps — catches silent provider model updates.

Source code Website Documentation

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

NPM package modelwatch-mcp
Claude Code 
claude mcp add modelwatch -e 'MODELWATCH_API_KEY=<your modelwatch_api_key>' -e 'MODELWATCH_API_BASE=https://api.modelwatch.app' -- npx -y modelwatch-mcp@0.1.3
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "modelwatch": {
            "command": "npx",
            "args": [
                "-y",
                "modelwatch-mcp@0.1.3"
            ],
            "env": {
                "MODELWATCH_API_KEY": "<your modelwatch_api_key>",
                "MODELWATCH_API_BASE": "https://api.modelwatch.app"
            }
        }
    }
}

Environment variables

  • MODELWATCH_API_KEY required secret API key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).
  • MODELWATCH_API_BASE Override the API base URL for self-hosted deployments.

Documentation

From the project's README on GitHub, refreshed weekly. Links open on GitHub.

ModelWatch

Continuous behavioral drift monitoring for LLM-powered applications. Define a spec (input and expected behavior), ModelWatch replays it on a schedule against your model endpoint, and alerts when the output drifts from the baseline.

Customers bring their own OpenAI or Anthropic key. Auth is a workspace key minted at POST /api/auth/signup (Authorization: Bearer mw_...).

  • API: https://api.modelwatch.app
  • App: https://modelwatch.app
  • HTTP reference: frontend/docs.html
pip install -r requirements.txt
pytest tests/ -q

Tool monitoring

ModelWatch also watches the tools an agent calls. A decorator in your process records latency, success, and a fingerprint of the response shape. The API compares that fingerprint with the dominant shape from the last 24 hours and alerts on schema drift. Response values are not uploaded.

pip install -e sdk
export MODELWATCH_API_KEY=mw_...
from modelwatch import monitor

@monitor(tool_name="search_web", agent_id="researcher")
async def search_web(query: str) -> dict:
    ...

Calls go to https://api.modelwatch.app/api/tools/ingest. Read recent calls at GET /api/tools and schema changes at GET /api/tools/schema-drift. The same mw_ key and the same email/Slack settings are used. Full SDK notes, including the TypeScript wrapper, are in sdk/README.md.

Quotas for tool calls are separate from spec runs: Free 5,000/month, Pro 100,000, Team 1,000,000, Enterprise unlimited.

This feature replaces a standalone ToolPulse deployment for the same buyer. Point clients at ModelWatch (MODELWATCH_API_KEY, https://api.modelwatch.app). The decorator does not call api.toolpulse.io.

Frequently asked questions

What is io.github.bch1212/modelwatch?

Continuous behavioral drift monitoring for LLM apps — catches silent provider model updates.

How do I install io.github.bch1212/modelwatch?

It runs on your machine. In Claude Code: claude mcp add modelwatch -e 'MODELWATCH_API_KEY=<your modelwatch_api_key>' -e 'MODELWATCH_API_BASE=https://api.modelwatch.app' -- npx -y modelwatch-mcp@0.1.3. For other clients, paste the JSON configuration from this page into their MCP settings.

Does io.github.bch1212/modelwatch need an API key?

Yes. It expects MODELWATCH_API_KEY, which you set in its configuration.

Is io.github.bch1212/modelwatch open source?

Its code is public on GitHub, but the repository does not declare a standard license.

Who maintains io.github.bch1212/modelwatch?

It is published by bch1212. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.