RAG Documentation Scraper

RAG Documentation Scraper

Turn any documentation site into a vector-database-ready knowledge base in one click.

Crawls documentation sites with Playwright, strips HTML navigation boilerplate, splits content on heading boundaries, and outputs clean Markdown or JSON chunks with embedded source metadata — ready to drop into Pinecone, Chroma, pgvector, or any LangChain / LlamaIndex / Haystack pipeline.

Why AI engineers choose this over generic scrapers

Benchmarks

Tested against the Crawlee.dev documentation (production run, 2026-10-18):

• 25 pages crawled → 94 chunks produced, 0 errors
• All chunks under 512 tokens (compatible with text-embedding-3-small, nomic-embed-text, bge-m3)
• Per-page time: ~1.8 s average (Playwright, images/fonts blocked)
• Noise stripped: nav, sidebar, footer, cookie banners, pagination, breadcrumbs, "edit this page" widgets

3-line Python quickstart

from apify_client import ApifyClient
chunks = ApifyClient("YOUR_APIFY_TOKEN").actor("YOUR_USERNAME/apify-rag-docs-scraper").call(
    run_input={"startUrls": [{"url": "https://YOUR_DOCS_SITE.com/docs/"}], "chunkSize": 500, "outputFormat": "json-chunks"}
)["defaultDatasetId"]
items = list(ApifyClient("YOUR_APIFY_TOKEN").dataset(chunks).iterate_items())
# items[0] → {"text": "...", "metadata": {"source": "...", "title": "...", "section": "...", "token_count": 487}}

See developer_quickstart/colab_rag_example.ipynb for a full end-to-end walkthrough: actor run → dataset pull → ChromaDB insert → semantic search query.

Key features

• 3 RAG-native output formats — markdown (default), json-chunks (Pinecone/Chroma/pgvector ready), text (bare text)
• 4 framework-specific formats — LangChain Document, LlamaIndex TextNode, Haystack Document, raw JSONL
• Flexible chunk size — Set chunkSize (tokens) to control granularity; works with any embedding model
• 4 chunking strategies — heading boundary (recommended), fixed token window with overlap, paragraph, whole-page
• PDF support — basic pdfjs extraction (free) or AWS Textract OCR for scanned/image PDFs
• Incremental crawling — DynamoDB delta tracking; re-runs skip unchanged pages entirely
• Version-aware URL filtering — restrict to a specific version path (e.g. /v2/) with one input field
• MCP-compatible — includes an MCP server wrapper so Claude Desktop and Cursor can call this actor as a live tool

Output formats

markdown (default) — generic RAG, vector-DB agnostic

{
  "markdown": "## Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "source": "https://crawlee.dev/docs/introduction",
  "title": "Introduction | Crawlee for JavaScript",
  "section": "Introduction > Quick Start",
  "chunk_index": 0,
  "chunk_total": 12,
  "token_count": 312,
  "content_hash": "c4a7e891f23b5d10",
  "scraped_at": "2026-10-18T14:45:22.341Z"
}

json-chunks — Pinecone / Chroma / pgvector

{
  "text": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "title": "Introduction | Crawlee for JavaScript",
    "section": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "last_modified": null,
    "content_hash": "c4a7e891f23b5d10",
    "scraped_at": "2026-10-18T14:45:22.341Z"
  }
}

langchain — drop-in LangChain Document

{
  "page_content": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "heading_path": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "content_hash": "c4a7e891f23b5d10"
  }
}

Input parameters

Pricing

This actor uses Pay-Per-Result pricing — you pay per chunk pushed to the dataset.

Cut costs by up to 80%: Enable useDeltaTracking: true so re-runs only process pages that changed since the last run. Stable documentation sites rarely change more than 10–20% of pages per day.

Pre-configured Published Tasks

Five ready-to-run tasks are available in the Apify Store for the most popular documentation niches:

1. LangChain Documentation Scraper — Full LangChain Python docs, Markdown + RAG chunks
2. Stripe API Docs Ingestion Tool — Stripe API reference in json-chunks format for payment bots
3. Docusaurus / ReadTheDocs Crawler — Universal adapter for Docusaurus and ReadTheDocs sites
4. Cursor / Claude MCP Documentation Extractor — MCP spec + Cursor AI docs into structured chunks
5. GitHub Wiki & Markdown Repository Scraper — GitHub Wikis and Markdown repos

Run python scripts/create_published_tasks.py (after apify push) to create these tasks programmatically.

AWS Integration (optional)

AWS features — Textract OCR for PDFs and DynamoDB delta tracking — are fully optional. Set these environment variables in Apify Console → Actor Settings → Environment Variables:

The DynamoDB table is created automatically on first use (PAY_PER_REQUEST billing, 90-day TTL).

MCP / Agent integration

The repository ships a standalone MCP server (mcp-server/) that wraps the Apify actor as a live tool. Once configured, Claude Desktop, Cursor, or any MCP-compatible AI client can call it directly during a conversation — no API calls or copy-pasting required.

Tool exposed

Prerequisites

# Node 18+ required
git clone https://github.com/LernowaAdmin/rag-docs-scraper.git
cd rag-docs-scraper/mcp-server
npm install
npm run build          # compiles TypeScript → dist/index.js

Claude Desktop (local stdio)

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

Restart Claude Desktop. The tool appears automatically in the tool palette.

Cursor (local stdio)

Edit ~/.cursor/mcp.json:

{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

Reload Cursor (Cmd/Ctrl + Shift + P → "Reload Window").

Remote / Docker (HTTP + SSE transport)

For hosted deployments, the server supports SSE over HTTP:

cd mcp-server
MCP_TRANSPORT=http MCP_PORT=3001 APIFY_TOKEN=apify_api_... node dist/index.js

Then point your MCP client at http://your-host:3001/sse.

The mcp-server/Dockerfile builds a production-ready image:

cd mcp-server
docker build -t rag-docs-mcp .
docker run -p 3001:3001 -e APIFY_TOKEN=apify_api_... -e MCP_TRANSPORT=http -e MCP_PORT=3001 rag-docs-mcp

Apify-hosted MCP gateway (no local server needed)

Apify exposes every actor's API as an MCP-compatible tool via their MCP gateway. Add this to your client config for a zero-install option:

{
  "mcpServers": {
    "apify": {
      "type": "sse",
      "url": "https://mcp.apify.com/sse",
      "headers": {
        "Authorization": "Bearer apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

See developer_quickstart/mcp_client_config_example.json for all three variants side by side.

FAQ

The actor crawls only one page — why aren't more pages enqueued?

A redirect often changes the URL structure after navigation. Check the actual URL in the logs (it may be /en/docs/ instead of /docs/) and update startUrls and urlPatterns to match the post-redirect path.

Chunks are empty or very short on a JavaScript-rendered site.

The page content may not be rendered when Playwright first loads it. Add the main content selector to excludeSelectors's inverse — or simply let the actor wait by setting navigationTimeoutSecs to a higher value (default: 30 s).

I'm seeing nav menu text in every chunk.

Add the offending element's CSS selector to excludeSelectors (e.g. [".custom-nav", "#site-sidebar"]). The actor already strips the most common patterns automatically.

Versioned docs produce duplicate chunks.

Use versionFilter to restrict crawling to one version (e.g. /v3/), or leave deduplicateContent: true to drop identical chunks regardless of which version URL they came from.