- Conduid
- Marketplace
- #crawler
MCP servers tagged crawler
75 live MCP servers tagged "crawler", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with crawler.
Lookacrawler
Free, open-source, token-efficient local alternative to Firecrawl with native MCP Server for LLMs (>73% token reduction).
Advanced Research Agent
Advanced Research Agent built with Python, MCP, Firecrawl, and LangGraph to autonomously crawl, extract, and structure web research data using LLM-driven workflows.
Smesh Crawler
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
Ez Web Search MCP
The repo's name -_- Also this was written entirely by gemini 2.5 pro
Search It
A MCP Server of Search and Crawler
Firescrapling
Self-hosted cost-control layer for web scraping. Bring your own Scrape.do/Scrapfly key, escalate fetch tiers only when needed, get LLM-ready Markdown. MCP server included.
Screamingfrog Audit MCP
Drive the Screaming Frog SEO Spider from Claude, Cursor, or any MCP client: headless crawls, a ranked issue register, derived analysis and a shareable report. Works on the free ti…
Craw MCP
Self-hostable MCP server that lets Claude crawl pages blocked by anti-bot walls, JS challenges, or IP bans, without paid scraping APIs. Bring your own proxies. RabbitMQ queue, Rus…
Prerenderbuddy Plugins
Portable Agent Plugins package for auditing crawler-visible HTML with Prerender Buddy MCP tools.
Marketingovo
Local-first SEO operations system that turns crawl, Search Console, GA4, performance and SERP data into a verified action queue. Apache-2.0, no telemetry, no paid tier.
Octopus Scout
Octoryn Web Ingestion Engine: governed, auditable, AI-native web/PDF ingestion
Sitepulse
Rust CLI and MCP server for technical SEO, sitemap audits, broken link detection, and AI agent readiness.
Aj Agent Crawl Optimizer
The agent-readiness scanner and fixer for WordPress — run the 21-check readiness scan (Level 0-5) with evidence, fix failures in one click, and publish Markdown negotiation, llms.…
Freecrawl
MCP-сервер: 29 инструментов работы с вебом для AI-агента — чтение защищённых страниц, поиск по многим движкам, обратный поиск картинок, извлечение данных, наблюдение за изменениям…
Musu Crawl AI
Agent-native knowledge harvesting and local wiki building.
Scrapling 921
Extract data from websites using a fast, intelligent web scraping library designed for modern HTML structures and dynamic content.
HatFetch
Give any LLM agent the power to read any website — even the ones that block bots. An MCP server that returns clean Markdown and handles blocks automatically via residential proxie…
JobSniper
Intelligence-driven job search decision system.
Zwep
A small, self-hosted search engine. Crawl a curated set of sources, search them in milliseconds, and expose them to AI agents over MCP. Nothing leaves your machine.
Voltcrawl
Verified web evidence and headless browser rendering Model Context Protocol (MCP) server for coding agents
Supacrawlx MCP
Official SupaCrawlX MCP server — add web scraping, crawling, and video/transcript extraction to Claude, Cursor, and any other LLM client.
Site Crawler MCP
Site-wide crawler MCP server: BFS crawl of all internal links with time/title/URL filters, page operations (summarize/click/download), SQLite persistence. Built on crawl4ai.
Ardvark
Crawler & indexer for ARD (Agentic Resource Discovery) ai-catalog.json documents
AUDISTO MCP
An Audisto MCP for Technical SEO analysis, for Gemini CLI and Claude CLI
Apple Rag Collector
Crawls and indexes Apple developer documentation (370K+ docs, 1.4K+ WWDC transcripts) into a vector store. Powers the apple-rag-mcp server.