English Français العربية

Crawlgraph

Registre officiel Recherche et web

CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph

Code source Site web Documentation

Installation

Ajoutez ceci à la configuration MCP de votre client (Claude Desktop : claude_desktop_config.json ; Cursor : .cursor/mcp.json ; VS Code et les autres utilisent la même forme « mcpServers »).

Remote (streamable-http)
Claude Code 
claude mcp add --transport http crawlgraph 'https://gateway.pipeworx.io/crawlgraph/mcp'
Configuration JSON (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "crawlgraph": {
            "type": "http",
            "url": "https://gateway.pipeworx.io/crawlgraph/mcp"
        }
    }
}

Documentation

Tirée du README du projet sur GitHub, actualisée chaque semaine. Les liens ouvrent GitHub.

Sommaire
  1. mcp-crawlgraph
  2. Tools
  3. Quick Start
  4. No MCP client? Call it over HTTP
  5. Standalone (no gateway account)
  6. Using with ask_pipeworx
  7. More
  8. License

mcp-crawlgraph

CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph

Part of Pipeworx — an MCP gateway connecting AI agents to 1679+ live data sources.

Tools

ToolDescription
crawlgraph_backlinksReferring domains for a target domain from the Common Crawl webgraph (4.4B edges, 120M domains). Returns linking domains ranked by CrawlGraph authority — useful for SEO audits, competitor backlink profiles, link-building gap research, and verifying who is citing a brand. One call = one quota credit.
crawlgraph_gap_analysisBacklink gap analysis: find domains linking to your competitors but NOT to you. Submit your domain + 1–5 competitor domains and CrawlGraph runs an async job (typically completes in seconds; longer for large profiles). This tool polls up to ~10s and either returns the full gap list, or — if still running — returns the job_id so you can resume via crawlgraph_gap_status. Counts against the gap-analysis quota when the job is submitted.
crawlgraph_gap_outreach_targetsThe warm-outreach play: gap_analysis filtered to ONLY the linking domains that link to EVERY competitor you listed but not to you. These are the highest-conviction outreach candidates — every competitor in the set has the link, so the topic/audience is verified. Same async pattern as crawlgraph_gap_analysis (polls up to ~10s; returns job_id if still running). Counts as one gap-analysis quota call.
crawlgraph_gap_statusPoll a previously-submitted gap analysis job by ID. Use this when crawlgraph_gap_analysis or crawlgraph_gap_outreach_targets returned status=pending with a job_id — call this with the same job_id to retrieve the result once status=completed. Read-only; no quota cost.
crawlgraph_releasesList the Common Crawl releases CrawlGraph has indexed (used as release_id in backlinks). Free; not counted against quota.
crawlgraph_domain_changesQuarter-over-quarter diff of referring domains for a domain — what was added or lost between two Common Crawl releases. Useful for catching link rot, sudden spikes (PR coverage), and outreach campaign attribution. Counts as one call against the backlinks quota.

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "crawlgraph": {
      "url": "https://gateway.pipeworx.io/crawlgraph/mcp"
    }
  }
}
What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/crawlgraph/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — ask_pipeworx, discover_tools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

This is deliberate, not multiplexing by accident. The meta-tools are what let a scoped connection answer a question this pack does not cover — via ask_pipeworx, which routes across the whole catalog — without you adding a second MCP server. There is currently no way to mount a pack endpoint without them; if the extra schemas cost you more context than the routing is worth, connect to the full gateway once rather than to several pack endpoints.

Or connect to the full Pipeworx gateway to get every pack's tools listed directly, instead of just this one's:

{
  "mcpServers": {
    "pipeworx": {
      "url": "https://gateway.pipeworx.io/mcp"
    }
  }
}

Both URLs reach the same gateway and the same 1679+ data sources. The only difference is which pack's tools are listed directly; ask_pipeworx reaches all of them from either one.

No MCP client? Call it over HTTP

This pack takes your own API key (_apiKey) — we don't front one for it, so there's no curl here that would run without it. Inspect any tool: GET https://gateway.pipeworx.io/v1/tools/crawlgraph_backlinks. Find one: POST https://gateway.pipeworx.io/v1/tools/search_packs with {"query":"..."}.

Standalone (no gateway account)

This package also runs as a local stdio MCP server — no Pipeworx account, no gateway round-trip:

{
  "mcpServers": {
    "crawlgraph": {
      "command": "npx",
      "args": ["-y", "@pipeworx/mcp-crawlgraph"]
    }
  }
}

Or run it directly to confirm it starts:

npx -y @pipeworx/mcp-crawlgraph

It speaks MCP over stdin/stdout and answers initialize/tools/list/tools/call for only this pack's tools — none of the shared meta-tools the gateway connection above adds. Same source, same tools, no ask_pipeworx routing.

Using with ask_pipeworx

Instead of calling tools directly, you can ask questions in plain English — this works on the pack endpoint above as well as on the full gateway:

ask_pipeworx({ question: "your question about Crawlgraph data" })

The gateway picks the right tool and fills the arguments automatically.

More

License

MIT

Questions fréquentes

Qu'est-ce que Crawlgraph ?

CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph

Comment installer Crawlgraph ?

Il fonctionne comme serveur distant. Dans Claude Code : claude mcp add --transport http crawlgraph 'https://gateway.pipeworx.io/crawlgraph/mcp'. Pour les autres clients, collez la configuration JSON de cette page dans leurs réglages MCP.

Faut-il une clé API pour Crawlgraph ?

Son entrée au registre n'indique aucune clé ni secret obligatoire. Certaines fonctions peuvent tout de même demander un compte auprès du service auquel il se connecte.

Crawlgraph est-il open source ?

Oui. Son code est sur GitHub sous licence MIT.

Qui maintient Crawlgraph ?

Il est publié par pipeworx-io. AI.tn le référence sans le maintenir ni le recommander.

Issu du registre MCP officiel. Être listé n'est pas une recommandation : vérifiez le code et les permissions d'un serveur avant de lui donner accès à vos données.