1. Conduid
  2. Scraping
  3. polite-crawler
MCP server · Scraping

polite-crawler

by · github.com/mark1russell7/crawler

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Unclaimed scraping
InstallView source
37Low

Scored 3 days ago · breakdown

About polite-crawler

polite-crawler is an MCP server published by mark1russell7 in the Scraping category: a polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Install

Claude Code
claude mcp add polite-crawler -- npx -y polite-crawler
npx
npx -y polite-crawler

This server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.

Ask AI

✦

Ask AI about polite-crawler

Powered by Claude · Grounded in docs

I know everything about polite-crawler. Ask me about installation, configuration, usage, or troubleshooting.

Security checks

6 checks have not been run yet.

Questions

About polite-crawler

How do I install polite-crawler?

Run claude mcp add polite-crawler -- npx -y polite-crawler, then add the server to your MCP client's configuration.

Is polite-crawler safe to use with an AI agent?

Its trust score is 37 out of 100 (low). Conduid hasn't run static security checks on this repository yet, so review the source yourself before granting it credentials. It has no ConduID identity yet, so agent calls to it are not receipted.

Is polite-crawler still maintained?

Conduid hasn't recorded a commit date for this repository yet. Check the repository directly for recent activity.