- Conduid
- Marketplace
- #web-crawler
MCP servers tagged web-crawler
45 live MCP servers tagged "web-crawler", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with web-crawler.
Webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search…
Tools
First CH Tools の計算ロジック（WCAGコントラスト比・文字数/Xウェイト・WebP変換・JSON-LD生成・llms.txt生成・文字コード/改行コード変換・Marp Markdown→スライド レンダリング・テストデータ生成・テキスト/コード差分・Cron式の解説・Base64/Data URI変換・URLパラメータの分解/再構築・HTM…
Linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits a…
Scrapfly MCP
🚀 Official Scrapfly MCP Server - brings enterprise-grade web scraping, extraction, and anti-bot–aware data access to Cursor, Claude, and any MCP-compatible LLM client.
0xMassi/webclaw
configures Claude, Cursor, Windsurf, Codex, OpenCode.
Crawl4ai MCP
Web crawling tool that integrates with AI assistants via the MCP
Sitewalk
Sitewalk: Technical SEO Site Audit. Open-source technical SEO crawler and MCP server for Claude, Cursor and Codex. 50+ checks, PDF and CSV reports, self-hosted. Built on LibreCraw…
server
Open-source engine for turning documentation sites into searchable, MCP-accessible context for humans and coding agents.
Omnifeed
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
web-content-pick
A tool for extracting structured content from web pages with customizable selectors and crawling options
Web Scrape
🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.t…
Okf Kit
Turn any website into a portable, agent-ready Open Knowledge Format (OKF) bundle — no LLM required. Crawl, sync, chat offline, and serve docs to coding agents over MCP.
Crawlforge MCP Server
CrawlForge MCP Server - Professional Model Context Protocol server with 19 comprehensive web scraping, crawling, and content processing tools.
Docs To MCP Server
MCP server that exposes documentation search tools for AI assistants
Turbo Surf
Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headle…
Atlas Web Compiler & Scraper
Official Model Context Protocol (MCP) server for Atlas - Deterministic Zero-LLM Web Compiler and Scraper for AI Agents
Rednote Assistant
MCP server which collects data from www.xiaohongshu.com social media
Pyfetcher
Agentic web infrastructure: fetch, scrape, extract, download with MCP server, browser fingerprinting, and crawl-scrape-download pipeline
Dsh Mediacrawler
Installable DeepSeek Harness profile bundle and bounded MCP adapter for MediaCrawler.
Data Forge
Turn any website into a quality-scored LLM fine-tuning dataset: crawl, chunk, generate Q&A, score, export to JSONL/Parquet/HF Hub. CLI plus MCP server.
Phantombot
Open-source web crawler and knowledge extraction engine for the AI agentic era. Turns the web into structured, agent-queryable knowledge.
Lookaberry
Autonomous, headless AI Go-to-Market (GTM) outbound engine & MCP Server powered by PostgreSQL 16 + pgvector.
Free Opensource SEO Screaming Toad Not Frog Tool With 100million Url Crawl Potential
DJAI Academy community-built, free open-source technical SEO crawler and Screaming Frog alternative with JavaScript audits, reports, a local dashboard, and an MCP server for AI ag…
Cockroach Crawler
Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.
WebScraperToolkit
AI-first web scraping engine with stealth bypass, MCP server, and multimodal output (Markdown, JSON, PDF) for agents and automation.
Scrapo
AI-native, agent-first web scraping for Python. Cost-aware tiered fetching (HTTP, browser, stealth, agent), model-agnostic LLM extraction with self-healing selector caches, API-fi…
Lookacrawler
Free, open-source, token-efficient local alternative to Firecrawl with native MCP Server for LLMs (>73% token reduction).
Librecrawl MCP
Audit websites with this self-hosted SEO crawler. Connect LibreCrawl to Claude and perform unlimited technical audits without per-crawl fees.
Shiny Fishstick
A semantic web compiler for AI agents. Crawls any website and compiles it into a typed, reliable SDK and Model Context Protocol (MCP) server so browser agents don't have to parse…
Smesh Crawler
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
Zerocrawl MCP
100% Free & Open-Source Web Crawling MCP Server. Zero-budget, no-API-key web scraping and content extraction for Claude Desktop, Cursor, and AI Agents.
Icp Scout
Evidence-first company discovery from the public web. Search, crawl, score & export qualified companies with source URLs behind every fact. CLI + dashboard + MCP server. Zero depe…
Wasl AI
Scores whether a business is legible to AI agents on a 100-point index, then generates the MCP server that makes it legible. Deterministic scoring, cited evidence, published refus…
MachineRead Preview
Audit any website for LLM discoverability and agent readiness. Checks AI crawler access, llms.txt, JSON-LD, MCP/A2A discovery, scrapability, and the SEO fundamentals AI retrieval…
Spider Cloud MCP Server
MCP server for the Spider Cloud web crawling and scraping API
Krawlify App
Self-hosted SEO crawler and audit platform with a REST API and an MCP server.
Vasco
Web research toolkit for LLM agents: search, fetch, extract, answer, map. MCP server + CLI + daemon, with anti-bot escalation and caching.
WebExtratorMCP
MCP server for web extraction and rendering via Ace Data Cloud.
Webdocs MCP
Crawl, index, and hybrid-search documentation sites; served to LLM agents as an MCP server. FastAPI + DuckDB + BM25/embeddings, fully offline-testable.
Crawl4tools
Web crawler for Open WebUI and MCP clients, plus a download CLI: pages as Markdown, HTML, PDF or screenshots, built on crawl4ai
CrawlIt
A web crawling and conversion system that uses FastAPI and MCP to crawl websites and convert them into markdown.
Langgraph Multi Agent Crawler
Multi-agent web harvesting on LangGraph's Supervisor pattern: 9 specialist agents, deterministic-first pipeline where the LLM is an enhancer, never a dependency — run-level circui…
Local Crawl
Run the fastCRW web crawler natively on Windows: tray on/off switch, JS rendering via Chrome, MCP for AI agents, optional sharing over Tailscale.
Inqwise MCP Site2ts
Convert existing websites into TypeScript Next.js apps via an MCP server (Rust) with a Node/Playwright helper. ARM-first, Tailwind-first, sandboxed under `.site2ts/`