- Conduid
- Marketplace
- #robots-txt
MCP servers tagged robots-txt
18 live MCP servers tagged "robots-txt", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with robots-txt.
Tools
First CH Tools の計算ロジック（WCAGコントラスト比・文字数/Xウェイト・WebP変換・JSON-LD生成・llms.txt生成・文字コード/改行コード変換・Marp Markdown→スライド レンダリング・テストデータ生成・テキスト/コード差分・Cron式の解説・Base64/Data URI変換・URLパラメータの分解/再構築・HTM…
Llms Txt Hub
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
Agentready
Audit and fix how ready a website is for AI agents: 0–5 readiness score across 30 checks, plus deployable templates
pagespeed-server
MCP server for PageSpeed & SEO audits. Zero config required - works out of the box.
Advisorsai Check
Lists which machine-readable basics a public page declares, and which it does not. One command, no API key, no account; also a read-only MCP tool.
SEO Workflow
Evidence-bound SEO and agent-readiness workflow for scanning live sites and producing validated remediation reports.
hanselhansel/aeo-cli
100 AEO scoring.
AutomateLab-tech/ai-seo-mcp
mcp`.
Cockroach Crawler
Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.
Geo Tool Check
Is this page readable for AI search? CLI + MCP server scoring crawler access, structure and citability for ChatGPT, Perplexity, Claude and Google AI. Runs locally, zero dependenci…
Smesh Crawler
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
Aipush Agent Skills
Free, open, security-verified agent skills for AEO (Answer Engine Optimization) — make any site ready for AI agents. Live: aipush.app
Sitepulse
Rust CLI and MCP server for technical SEO, sitemap audits, broken link detection, and AI agent readiness.
Fearch
Honest web search and page reading for AI agents — an MCP server and CLI that identify themselves truthfully, respect robots.txt, and hand bot-checks to a human in a visible brows…
Agent Adoption Spec
Open specification for agent-readiness — 25 checks across HTTP-protocol and content-shape practices, organized into a 3-tier ladder (L1–L3). CC-BY 4.0.
Agents Txt
agents.txt — an action-policy format letting sites declare what autonomous agents may do, with verifiable receipts
Agent Ready Action
Score your site's AI Agent Readiness (the Cloudflare check) and open a PR that fixes what's failing — generated in a disposable Tenki sandbox and verified on a live preview URL.
Scrape Le Probe
Scrapeability Checker for Smart Web Scraping 2026