English Français العربية

TheCrawler

Official registry Search and web

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Source code

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

NPM package thecrawler
Claude Code 
claude mcp add thecrawler -- npx -y thecrawler@0.5.8
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "thecrawler": {
            "command": "npx",
            "args": [
                "-y",
                "thecrawler@0.5.8"
            ]
        }
    }
}

Frequently asked questions

What is TheCrawler?

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

How do I install TheCrawler?

It runs on your machine. In Claude Code: claude mcp add thecrawler -- npx -y thecrawler@0.5.8. For other clients, paste the JSON configuration from this page into their MCP settings.

Does TheCrawler need an API key?

Its registry entry lists no required key or secret. Some features may still need an account with the service it connects to.

Is TheCrawler open source?

Its code is public on GitHub, but the repository does not declare a standard license.

Who maintains TheCrawler?

It is published by manchittlab. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.