1. Conduid
  2. Scraping
  3. Crawl
MCP server · Scraping

Crawl

· github.com/vibheksoni/crawl

AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.

Unclaimed crawlerweb-scraperfirecrawlmcpaicoding-agentclaudeclidashboardplaywrightextractione2bsandboxwebcrawl
InstallView source
37Low

Scored 3 months ago · breakdown

About Crawl

Crawl is an MCP server in the Scraping category: aI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.

Install

Claude Code
claude mcp add crawl -- npx -y @hasna/crawl
npx
npx -y @hasna/crawl

This server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.

Ask AI

✦

Ask AI about Crawl

Powered by Claude · Grounded in docs

I know everything about Crawl. Ask me about installation, configuration, usage, or troubleshooting.

Security checks

  • ✓README presentThe repository ships documentation.

5 checks have not been run yet.

README

What It Is

crawl is organized into three layers:

  • sdk: the full Python capability surface
  • cli: direct command-line access to the SDK
  • mcp: a smaller workflow-oriented server for AI agents

The SDK is the source of truth. The CLI wraps most of it. The MCP layer intentionally exposes fewer tools so agents do not get flooded with schemas.

Install

Editable install:

python -m pip install -e .

Pinned dependencies:

python -m pip install -r requirements.txt

Entry Points

Repo-root entrypoints:

python cli.py --help
python server.py

Installed scripts:

crawl-cli --help
crawl-mcp

Documentation

Highlights

  • Search providers: google, searxng, auto, hybrid
  • Browser-capable SDK and MCP paths support headless
  • Consent handling and resource blocking are built into browser workflows
  • Structured extraction, article extraction, forms, feeds, contacts, and technology fingerprinting are all in the SDK
  • The MCP server exposes a compact tool surface:
    • search_web
    • inspect_url
    • discover_site
    • extract_structured
    • capture_screenshot

Quick Examples

SDK:

import asyncio

from crawl.sdk import fetch_page, websearch


async def main() -> None:
    search_payload = await websearch("python async browser automation", provider="auto")
    page_payload = await fetch_page("https://example.com", mode="browser", headless=True)
    print(search_payload["count"])
    print(page_payload["final_url"])


asyncio.run(main())

CLI:

python cli.py websearch "python async browser automation" --provider auto --max-results 5 --pages 1
python cli.py fetch-page https://example.com --mode browser --include-html

MCP:

  • run python server.py
  • connect your MCP client to the stdio server
  • use the compact workflow tools documented in docs/mcp.md

README copy captured when this server was indexed; it may lag the repository. The original is authoritative.

Questions

About Crawl

How do I install Crawl?

Run claude mcp add crawl -- npx -y @hasna/crawl, then add the server to your MCP client's configuration.

Is Crawl safe to use with an AI agent?

Its trust score is 37 out of 100 (low). It passes 1 of the 1 static security checks run so far. It has no ConduID identity yet, so agent calls to it are not receipted.

Is Crawl still maintained?

Conduid hasn't recorded a commit date for this repository yet. Check the repository directly for recent activity.