io.github.CloudCorpRecords/crawlport

crawlport

crawlport is a doorway between an agent and the readable web, backed by Firecrawl. Give it one page and it hands back clean markdown; give it a whole site and it will map the pages, crawl a section, or scrape a list in batch; give it a question and it can search, pull structured fields out of pages to a schema you describe, research across many sources, or draft the llms.txt a site would publish about itself. It runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

Setup

Get a key from Firecrawl and export it:

export FIRECRAWL_API_KEY=<redacted>

Without the variable set, every command and tool call fails with an error that names FIRECRAWL_API_KEY — nothing is sent anywhere.

Install

npx @reneturcios/crawlport scrape https://example.com/some/page

Or run the container image:

docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport

The container defaults to the MCP server (see below). Pass CLI arguments to use it as a CLI instead.

CLI

# One page, as markdown (add --formats html,links etc. for more)
crawlport scrape https://example.com/article --only-main-content

# Discover a site's URLs, optionally filtered
crawlport map https://example.com --search pricing --limit 100

# Web search, with locale controls
crawlport search "standing desk reviews" --limit 5 --lang en --country US

# Start a crawl, then poll it with the returned job id
crawlport crawl https://example.com/docs --limit 50 --max-depth 2
crawlport crawl-status <job-id>

# Structured extraction and batch scraping
crawlport extract https://example.com/a https://example.com/b --prompt "pull the price"
crawlport batch-scrape https://example.com/a https://example.com/b
crawlport batch-status <job-id>

# Research and llms.txt
crawlport deep-research "home battery storage costs in 2026" --max-urls 40
crawlport llmstxt https://example.com --max-urls 25

crawlport --version

Successful calls print the API's JSON response. Crawl and batch jobs are asynchronous upstream: the start call returns a job id, and the matching status command reads its progress and results. Exit codes: 0 on success, 1 when the API call fails (including a missing key), 2 for usage errors.

MCP server

One bin serves both faces. The mcp subcommand starts a stdio MCP server:

npx -y @reneturcios/crawlport mcp

MCP client configuration:

{
  "mcpServers": {
    "crawlport": {
      "command": "npx",
      "args": ["-y", "@reneturcios/crawlport", "mcp"],
      "env": { "FIRECRAWL_API_KEY": "your-key-here" }
    }
  }
}

Tools

Tool results are the upstream JSON, pretty-printed as text. Optional arguments are forwarded only when provided, so the API's defaults apply otherwise. Upstream failures come back as tool errors carrying the HTTP status and the API's own message.

Development

npm install
npm run build   # tsc -> dist/
npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)
npm run dev     # run the CLI from source via tsx

License

MIT — see LICENSE.