searxNcrawl

searxNcrawl is a minimal MCP server and CLI toolkit for search and crawling, built on top of Crawl4AI and SearXNG.

This project is published as searxNcrawl at https://github.com/DasDigitaleMomentum/searxNcrawl and is maintained by DDM – Das Digitale Momentum GmbH & Co KG. It is the successor to searxng-mcp https://github.com/tisDDM/searxng-mcp (which should be marked deprecated).

Compared to plain Crawl4AI usage, searxNcrawl provides a proven, production-tested crawl configuration for documentation-heavy sites, optimized for clean, model-ready Markdown with less noise and better token efficiency.

It also includes built-in markdown deduplication and early support for authenticated crawling (WIP) via Playwright storage state — including a practical CDP export flow for real Chrome/Chromium login sessions.

Features

Crawling (Crawl4AI + proven defaults)

• Single page crawling - Crawl one URL and return clean markdown
• Multiple pages - Batch crawl a list of URLs with concurrency control
• Site crawling - BFS strategy with max depth and page limits
• Proven extraction config - Production-tested selectors/exclusions and markdown tuning for docs-style websites

Content Quality

• Markdown deduplication - exact (default) removes repeated blocks, off disables dedup
• Dedup guardrails - Non-destructive metadata signals when removal looks unusually aggressive
• Link removal - Strip all links from output for cleaner LLM context (--remove-links)
• Reference extraction - Captures links from crawled pages

Authenticated Crawling (WIP)

• Storage-state based auth - Crawl logged-in pages using Playwright storage_state
• Session capture tool - crawl-capture for manual login capture and CDP session export
• CDP session discovery - List active browser sessions and select one directly in CLI
• Current status: WIP - Auth flow works, but UX/flow is not final yet

Web Search

• SearXNG integration - Privacy-respecting metasearch engine
• Configurable search - Language, time range, categories, engines
• Safe search - Adjustable content filtering

CLI Tools

• crawl - Crawl pages from the command line
• crawl-capture - Manual login capture + CDP session list/select/export
• search - Search the web via SearXNG
• Global installation - Available system-wide after pip install -e .

MCP Server

• STDIO transport - For MCP harnesses (Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.)
• HTTP transport - For remote access and web integrations
• CORS support - Configurable cross-origin resource sharing for browser-based MCP clients (--cors-origins)

Installation

cd searxNcrawl
python -m venv .venv
source .venv/bin/activate
pip install -e .

# Install playwright browsers (required!)
playwright install chromium

MCP Server

The crawler is available as an MCP (Model Context Protocol) server, compatible with common MCP harnesses (Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.).

Running the MCP Server

# STDIO transport (for MCP harnesses such as Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.)
python -m crawler.mcp_server

# HTTP transport (for remote access)
python -m crawler.mcp_server --transport http --port 8000

# Custom host binding
python -m crawler.mcp_server --transport http --host 0.0.0.0 --port 9000

# Enable CORS for specific origins (required for browser-based MCP clients)
python -m crawler.mcp_server --transport http --cors-origins "http://localhost:3000,https://myapp.com"

# Enable CORS for all origins (use with caution)
python -m crawler.mcp_server --transport http --cors-origins "*"

# Or via installed script
crawl-mcp --transport http --port 8000

# With custom SearXNG instance
SEARXNG_URL=https://search.example.com python -m crawler.mcp_server

Environment Variables

Variable	Default	Description
SEARXNG_URL	http://localhost:8888	SearXNG instance URL
SEARXNG_USERNAME	(none)	Optional basic auth username
SEARXNG_PASSWORD	(none)	Optional basic auth password

SearXNG Instance Requirements

SearXNG is a privacy-respecting metasearch engine that aggregates results from multiple search engines without tracking users. To use the search functionality of searxNcrawl, you need access to a SearXNG instance with:

• JSON output format enabled – The instance must have JSON format enabled in its configuration (this is typically set in settings.yml under search.formats).
• Network accessibility – The instance must be reachable from where you run searxNcrawl.

You can either self-host a SearXNG instance or use a public one. For reliable results, self-hosting is recommended as public instances may have rate limits or restricted API access.

Configuration File Search Order

The CLI tools (crawl, search) look for .env files in this order:

1. Current directory - ./.env
2. User config - ~/.config/searxncrawl/.env

If no .env is found and .env.example exists in the package, it will be automatically copied to ~/.config/searxncrawl/.env as a starting point.

Quick setup for global CLI usage:

# Option 1: Copy example to user config
mkdir -p ~/.config/searxncrawl
cp .env.example ~/.config/searxncrawl/.env
# Edit with your SEARXNG_URL

# Option 2: Export environment variable
export SEARXNG_URL=http://your-searxng:8888

CORS Configuration (HTTP Transport)

When using HTTP transport, browser-based MCP clients may need CORS (Cross-Origin Resource Sharing) headers. Use --cors-origins to enable them:

# Allow specific origins
crawl-mcp --transport http --cors-origins "http://localhost:3000,https://myapp.com"

# Allow all origins (convenient for local dev, but use with caution in production)
crawl-mcp --transport http --cors-origins "*"

Flag	Default	Description
--cors-origins	(none)	Comma-separated allowed origins. If not set, no CORS headers are sent. Use * to allow all origins.

Security notes:

• Without --cors-origins, the server sends no CORS headers (browsers will block cross-origin requests).
• Using * allows any website to call your MCP server — only appropriate for local/trusted networks.
• For production, whitelist specific origins instead.

MCP Harness Configuration

Add to your MCP client configuration (examples include Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.):

{
  "mcpServers": {
    "crawler": {
      "command": "python",
      "args": ["-m", "crawler.mcp_server"],
      "cwd": "/path/to/searxNcrawl",
      "env": {
        "SEARXNG_URL": "http://your-searxng-instance:8888"
      }
    }
  }
}

Or with uv:

{
  "mcpServers": {
    "crawler": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
      "env": {
        "SEARXNG_URL": "http://your-searxng-instance:8888"
      }
    }
  }
}

OpenClaw Configuration

OpenClaw is a popular autonomous AI agent (150k+ GitHub stars) that supports MCP natively. To integrate searxNcrawl with OpenClaw, add the following to your OpenClaw MCP config file (~/.clawdbot/mcp.json or openclaw.json):

Python with venv:

{
  "searxNcrawl": {
    "command": "python",
    "args": ["-m", "crawler.mcp_server"],
    "cwd": "/path/to/searxNcrawl",
    "env": {
      "SEARXNG_URL": "http://your-searxng-instance:8888"
    }
  }
}

With uv (no manual venv needed):

{
  "searxNcrawl": {
    "command": "uv",
    "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
    "env": {
      "SEARXNG_URL": "http://your-searxng-instance:8888"
    }
  }
}

Docker HTTP endpoint:

If you prefer running searxNcrawl via Docker, start the server with:

docker compose up --build

Then configure OpenClaw to connect to the HTTP endpoint at http://localhost:9555/mcp.

Once configured, OpenClaw will have access to the crawl, crawl_site, and search tools.

Running with Docker Compose

Create a .env file (see .env.example) and run:

docker compose up --build

The MCP HTTP port is configurable via MCP_PORT in .env. Default is 9555, so the server is available at http://localhost:9555/mcp.

To run real‑world checks against the Docker setup (crawl, crawl_site, search), use:

scripts/test-realworld.sh

For extended tests including new features (remove_links, Unicode handling, schema validation):

scripts/test-extended.sh

MCP Tools

crawl

Crawl one or more web pages and extract their content as markdown.

Parameters:

Parameter	Type	Default	Description
urls	List[str]	required	URLs to crawl
output_format	str	"markdown"	Output format: "markdown" or "json"
concurrency	int	3	Max concurrent crawls
remove_links	bool	false	Remove all links from markdown output
dedup_mode	str	"exact"	Markdown dedup mode: "exact" or "off"
storage_state	str	null	Path to Playwright storage state JSON for authenticated crawling

Output Formats:

• markdown: Clean concatenated markdown with URL headers and timestamps
• json: Full JSON with metadata, references, and statistics

Examples:

# Single page
crawl(urls=["https://docs.example.com"])

# Multiple pages with JSON output
crawl(urls=["https://example.com/page1", "https://example.com/page2"], output_format="json")

# Clean output without links
crawl(urls=["https://example.com"], remove_links=True)

# Disable markdown dedup
crawl(urls=["https://example.com"], dedup_mode="off")

# Crawl with authenticated storage state
crawl(urls=["https://example.com"], storage_state="/path/to/state.json")

crawl_site

Crawl an entire website starting from a seed URL using BFS strategy.

Parameters:

Parameter	Type	Default	Description
url	str	required	Seed URL to start from
max_depth	int	2	Maximum crawl depth (0 = seed only)
max_pages	int	25	Maximum pages to crawl
include_subdomains	bool	false	Include subdomains
output_format	str	"markdown"	Output format: "markdown" or "json"
remove_links	bool	false	Remove all links from markdown output
dedup_mode	str	"exact"	Markdown dedup mode: "exact" or "off"
storage_state	str	null	Path to Playwright storage state JSON for authenticated crawling

Examples:

# Basic site crawl
crawl_site(url="https://docs.example.com")

# Deep crawl with more pages
crawl_site(url="https://docs.example.com", max_depth=3, max_pages=50)

# JSON output with full stats
crawl_site(url="https://docs.example.com", output_format="json")

# Clean output without links
crawl_site(url="https://docs.example.com", remove_links=True)

# Disable markdown dedup for site crawl
crawl_site(url="https://docs.example.com", dedup_mode="off")

# Site crawl with authenticated storage state
crawl_site(url="https://docs.example.com", storage_state="/path/to/state.json")

search

Search the web using SearXNG metasearch engine.

Parameters:

Parameter	Type	Default	Description
query	str	required	Search query string
language	str	"en"	Language code (e.g., 'en', 'de', 'fr')
time_range	str	null	Time filter: 'day', 'week', 'month', 'year'
categories	List[str]	null	Categories: 'general', 'images', 'news', etc.
engines	List[str]	null	Specific engines to use
safesearch	int	1	0 (off), 1 (moderate), 2 (strict)
pageno	int	1	Page number (minimum 1)
max_results	int	10	Maximum results (1-50)

Examples:

# Basic search
search(query="python tutorials")

# Search with time filter
search(query="latest AI news", time_range="week")

# Search specific category
search(query="cute cats", categories=["images"])

# Search in German
search(query="Rezepte", language="de")

# Strict safe search
search(query="programming", safesearch=2)

Response Format (JSON):

{
  "query": "python tutorials",
  "number_of_results": 10,
  "results": [
    {
      "title": "Python Tutorial - W3Schools",
      "url": "https://www.w3schools.com/python/",
      "content": "Well organized tutorials...",
      "engine": "google",
      "category": "general"
    }
  ],
  "answers": [],
  "suggestions": ["python for beginners", "python course"],
  "corrections": []
}

Markdown Output Format

When using output_format="markdown", the output includes:

# https://example.com/page1
_Crawled: 2025-01-09 12:00:00 UTC_

[Page content as markdown...]

---

# https://example.com/page2
_Crawled: 2025-01-09 12:00:01 UTC_

[Page content as markdown...]

JSON Output Format

When using output_format="json", the output includes:

{
  "crawled_at": "2025-01-09 12:00:00 UTC",
  "documents": [
    {
      "request_url": "https://example.com",
      "final_url": "https://example.com/",
      "status": "success",
      "markdown": "...",
      "error_message": null,
      "metadata": {
        "title": "Example",
        "status_code": 200,
        "dedup_mode": "exact",
        "dedup_sections_total": 12,
        "dedup_sections_removed": 3,
        "dedup_chars_removed": 542,
        "dedup_applied": true,
        "dedup_guardrail_checked": true,
        "dedup_guardrail_triggered": false,
        "dedup_guardrail_reason": "within-threshold",
        "dedup_guardrail_section_removal_rate": 0.25,
        "dedup_guardrail_section_rate_threshold": 0.6
      },
      "references": [
        {"index": 1, "href": "https://example.com/about", "label": "About"}
      ]
    }
  ],
  "summary": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "stats": {
    "total_pages": 1,
    "successful_pages": 1,
    "failed_pages": 0
  }
}

Python API

Single Page

from crawler import crawl_page, crawl_page_async

# Sync
doc = crawl_page("https://docs.example.com/intro", dedup_mode="exact")
print(doc.markdown)
print(doc.final_url)
print(doc.references)  # List of Reference(index, href, label)

# Async
doc = await crawl_page_async("https://docs.example.com/intro", dedup_mode="off")

# Async with authenticated state
doc = await crawl_page_async(
    "https://docs.example.com/intro",
    auth={"storage_state": "/path/to/state.json"},
)

Multiple Pages

from crawler import crawl_pages, crawl_pages_async

urls = [
    "https://docs.example.com/page1",
    "https://docs.example.com/page2",
    "https://docs.example.com/page3",
]

# Sync (with concurrency limit)
docs = crawl_pages(urls, concurrency=3, dedup_mode="exact")

for doc in docs:
    if doc.status == "success":
        print(f"--- {doc.final_url} ---")
        print(doc.markdown[:500])
    else:
        print(f"FAILED: {doc.request_url} - {doc.error_message}")

# Async
docs = await crawl_pages_async(urls, concurrency=5, dedup_mode="off")

# Async with authenticated state
docs = await crawl_pages_async(
    urls,
    auth={"storage_state": "/path/to/state.json"},
)

Site Crawl (BFS)

from crawler import crawl_site, crawl_site_async

# Crawl entire site with limits
result = crawl_site(
    "https://docs.example.com",
    max_depth=2,           # How deep to follow links
    max_pages=10,          # Stop after N pages
    include_subdomains=False,
    dedup_mode="exact",   # "exact" (default) or "off"
    auth={"storage_state": "/path/to/state.json"},
)

print(f"Crawled {result.stats['total_pages']} pages")
print(f"Successful: {result.stats['successful_pages']}")
print(f"Failed: {result.stats['failed_pages']}")

for doc in result.documents:
    print(f"{doc.status}: {doc.final_url}")

CLI Usage

After installation (pip install -e .), the crawl and search commands are available globally.

crawl

# Single page to stdout
crawl https://example.com

# Single page to file
crawl https://example.com -o page.md

# Multiple pages to directory
crawl https://example.com/page1 https://example.com/page2 -o output/

# Site crawl
crawl https://docs.example.com --site --max-depth 2 --max-pages 10 -o docs/

# Output as JSON (includes metadata and references)
crawl https://example.com --json

# Clean output without links (better for LLM context)
crawl https://example.com --remove-links

# Disable markdown dedup
crawl https://example.com --dedup-mode off

# Crawl with authenticated storage state
crawl https://example.com --storage-state /path/to/state.json

# JSON output for site crawl
crawl https://docs.example.com --site --max-pages 5 --json -o result.json

# Verbose logging
crawl https://example.com -v

Dedup Mode (crawl)

• --dedup-mode exact (default): removes exact repeated markdown blocks in a single document.
• --dedup-mode off: disables markdown dedup and keeps extracted blocks unchanged.

Dedup metrics are written into each document's metadata (e.g. dedup_sections_removed, dedup_chars_removed). Guardrails are non-destructive and annotate metadata via fields like dedup_guardrail_checked, dedup_guardrail_triggered, and dedup_guardrail_reason when removal ratios are unusually high.

Auth MVP (storage_state)

• API, CLI, and MCP share one auth resolver path and support storage_state only in MVP scope.
• CLI: pass --storage-state /path/to/state.json.
• Python API: pass auth={"storage_state": "/path/to/state.json"}.
• MCP tools (crawl, crawl_site): pass storage_state.
• Session capture is isolated via crawl-capture and does not change normal crawl runtime behavior.
• No-drift guarantee: auth integration does not change crawl config defaults (wait/selectors/SPA/persistent-session defaults).

Session Capture (crawl-capture)

crawl-capture supports two user-friendly ways to produce a Playwright storage_state file.

Option A — Manual login flow (built-in browser)

Use this when your identity provider allows login in Playwright-launched browsers.

# Capture after login redirect matches completion URL regex
crawl-capture \
  --start-url https://example.com/login \
  --completion-url 'https://example.com/dashboard.*' \
  --output ./state.json

# Overwrite existing output only when explicitly allowed
crawl-capture \
  --start-url https://example.com/login \
  --completion-url 'https://example.com/app.*' \
  --output ./state.json \
  --overwrite

Option B — Export from running Chrome/Chromium via CDP

Use this when providers (e.g. Google) reject automated-login browsers.

1. Start your real browser with remote debugging enabled:

# Linux example
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-cdp-searxncrawl"

2. Log in manually to your target app in that browser.

3. List selectable sessions:

crawl-capture --cdp-url http://127.0.0.1:9222 --list-sessions

4. Export by explicit session index:

crawl-capture \
  --cdp-url http://127.0.0.1:9222 \
  --cdp-session 2 \
  --output ./state.json

Or let CLI selection guide you interactively:

crawl-capture \
  --cdp-url http://127.0.0.1:9222 \
  --list-sessions \
  --select \
  --output ./state.json

After capture/export, use the file for authenticated crawling:

crawl https://example.com/private --storage-state ./state.json

Explicit outcomes:

• success (exit 0): storage state written.
• timeout (exit 2): completion condition not reached in time (manual flow only).
• abort (exit 130): browser/session closed before completion (manual flow only).

Safety notes:

• Keep storage_state files out of version control.
• Capture/export is intentionally isolated from standard crawl / crawl_site execution paths.
• If multiple tabs share one browser context/profile, they share the same exported session state.

search

Requires SEARXNG_URL environment variable (or .env file).

# Basic search (markdown output)
search "python tutorials"

# Search in German
search "Rezepte" --language de

# Search with time filter
search "latest AI news" --time-range week

# JSON output
search "python" --json

# Save JSON results to file
search "python asyncio" --json -o results.json

# Limit results
search "docker compose" --max-results 5

CrawledDocument Structure

@dataclass
class CrawledDocument:
    request_url: str          # Original URL requested
    final_url: str            # Final URL after redirects
    status: str               # "success", "failed", or "redirected"
    markdown: str             # Extracted markdown content
    html: Optional[str]       # Raw HTML (if available)
    headers: Dict[str, Any]   # HTTP response headers
    references: List[Reference]  # Extracted links
    metadata: Dict[str, Any]  # Title, status code, dedup metrics, guardrail info
    raw_markdown: Optional[str]  # Unprocessed markdown
    error_message: Optional[str]  # Error details if failed

@dataclass
class Reference:
    index: int
    href: str
    label: str

Configuration

The default configuration is optimized for documentation sites. For advanced customization:

from crawler import crawl_page_async, build_markdown_run_config, RunConfigOverrides

# Custom configuration
config = build_markdown_run_config(
    RunConfigOverrides(
        delay_before_return_html=1.0,  # Wait longer for JS
        mean_delay=1.0,                # Delay between requests
        scan_full_page=True,
    )
)

doc = await crawl_page_async("https://example.com", config=config)

Dependencies

Minimal dependencies:

• crawl4ai>=0.7.4 - The underlying crawler engine
• tldextract>=5.1.2 - Domain parsing for site crawls
• playwright>=1.40.0 - Browser automation
• fastmcp>=2.0.0 - MCP server framework
• httpx>=0.27.0 - HTTP client for SearXNG

License

MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG