webpeel/webpeel
URL benchmark.
Ask AI about webpeel/webpeel
Powered by Claude ยท Grounded in docs
I know everything about webpeel/webpeel. Ask me about installation, configuration, usage, or troubleshooting.
0/500
Reviews
Documentation
The web data layer for AI agents.
Fetch, search, crawl, extract, screenshot โ one call, zero boilerplate.
Quick Start ยท Agent Integrations ยท Docs ยท Playground ยท Get API Key
The Problem
Every AI agent that touches the web rebuilds the same brittle stack: HTTP fetch โ headless browser โ anti-bot bypass โ HTML cleanup โ markdown conversion โ token budgeting. Each layer fails differently. Sites change. Cloudflare rotates challenges. Your agent gets empty strings at 2 AM and your pipeline breaks.
WebPeel replaces that entire stack with one function call. It handles engine selection, anti-bot escalation, domain-specific extraction, and token optimization so your agent gets clean, structured data every time โ without managing browsers, proxies, or parsing logic.
Quick Start
# Zero-install โ just run it
npx webpeel "https://example.com"
# Search the web
npx webpeel search "latest AI agent frameworks"
# Crawl an entire site
npx webpeel crawl docs.example.com --max-pages 50
# Screenshot any page
npx webpeel screenshot "https://stripe.com/pricing" --full-page
# Ask a question about any page
npx webpeel ask "https://arxiv.org/abs/2401.00001" "What is the main contribution?"
Or install globally:
npm install -g webpeel
Use as a library:
import { peel } from 'webpeel';
const result = await peel('https://news.ycombinator.com');
console.log(result.markdown); // Clean markdown, ready for your LLM
console.log(result.metadata); // Title, tokens saved, timing, etc.
Use via API:
curl "https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing" \
-H "Authorization: Bearer $WEBPEEL_API_KEY"
{
"url": "https://stripe.com/pricing",
"markdown": "# Stripe Pricing\n\n**Integrated per-transaction fees**...",
"metadata": {
"title": "Pricing & Fees | Stripe",
"tokens": 420,
"tokensOriginal": 8200,
"savingsPct": 94.9
}
}
Get your free API key โ ยท No credit card required ยท 500 requests/week free
Why WebPeel
๐ง 55+ Domain Extractors โ Not Just HTML-to-Markdown
Generic scrapers convert raw HTML to markdown and call it a day. WebPeel has purpose-built extractors for 55+ domains โ Reddit, GitHub, YouTube, Amazon, ArXiv, Hacker News, Wikipedia, StackOverflow, Zillow, Polymarket, ESPN, and more. Each extractor understands the site's structure and returns clean, structured data without browser rendering.
โก 65โ98% Token Savings
Domain extractors strip navigation, ads, sidebars, and boilerplate before content reaches your agent. Less context consumed = lower costs, faster inference, and longer agent chains.
| Site | Raw HTML tokens | WebPeel tokens | Savings |
|---|---|---|---|
| News article | 18,000 | 640 | 96% |
| Reddit thread | 24,000 | 890 | 96% |
| Wikipedia page | 31,000 | 2,100 | 93% |
| GitHub README | 5,200 | 1,800 | 65% |
| E-commerce product | 14,000 | 310 | 98% |
๐ 6-Layer Engine Escalation
WebPeel doesn't just try one method โ it automatically escalates through 6 engines until it gets a good result:
Simple HTTP โ Domain API โ Browser render โ Stealth browser โ Cloaked browser โ Search cache fallback
No manual --render flags for most sites. WebPeel knows which sites need JavaScript, which need stealth, and which have anti-bot protection โ and picks the right engine automatically.
๐ Firecrawl-Compatible Migration Path
Already using Firecrawl-style workflows? WebPeel supports compatible /v1/scrape, /v2/scrape, /v1/crawl, /v1/search, and /v1/map endpoints, which makes migration dramatically easier than rebuilding your pipeline from scratch.
Agent-Native Integrations
MCP Server (Claude, Cursor, Windsurf, VS Code)
Give any MCP-compatible AI the ability to browse, search, and extract from the web.
{
"mcpServers": {
"webpeel": {
"command": "npx",
"args": ["-y", "webpeel", "mcp"],
"env": { "WEBPEEL_API_KEY": "wp_your_key_here" }
}
}
}
7 MCP tools exposed: webpeel_read ยท webpeel_find ยท webpeel_see ยท webpeel_extract ยท webpeel_monitor ยท webpeel_act ยท webpeel_crawl
LangChain
import { WebPeelLoader } from 'webpeel/integrations/langchain';
const loader = new WebPeelLoader({ url: 'https://example.com', render: true });
const docs = await loader.load();
LlamaIndex
import { WebPeelReader } from 'webpeel/integrations/llamaindex';
const reader = new WebPeelReader();
const docs = await reader.loadData('https://example.com');
Python SDK
pip install webpeel
from webpeel import WebPeel
wp = WebPeel(api_key="wp_...")
result = wp.fetch("https://example.com")
print(result.markdown)
Full Feature Set
| Capability | CLI | API | Details |
|---|---|---|---|
| Fetch & extract | webpeel "url" | GET /v1/fetch | Clean markdown from any URL |
| Web search | webpeel search "query" | GET /v1/search | DuckDuckGo (free) or Brave (BYOK) |
| Smart search | โ | POST /v1/search/smart | AI-powered structured results |
| Crawl sites | webpeel crawl "url" | POST /v1/crawl | Depth/page limits, rate control |
| Screenshots | webpeel screenshot "url" | POST /v1/screenshot | Full-page, multi-viewport, visual diff, filmstrip |
| Structured extraction | --extract-schema | POST /v1/extract | JSON schema โ structured data |
| Q&A | webpeel ask "url" "q" | POST /v1/answer | Answer questions about any page |
| Deep research | โ | POST /v1/deep-research | Multi-query autonomous research |
| Content monitoring | webpeel monitor "url" | POST /v1/watch | Change detection with webhooks |
| Browser sessions | โ | POST /v1/session | Persistent sessions for login flows |
| Browser actions | --action 'click:.btn' | actions field | Click, type, scroll, wait |
| Batch scrape | webpeel batch file | POST /v1/batch/scrape | Parallel multi-URL processing |
| URL discovery | webpeel map "url" | POST /v1/map | Sitemap and link discovery |
| YouTube transcripts | auto-detected | auto-detected | Multiple export formats |
| PDF extraction | auto-detected | auto-detected | Text, tables, structure |
| Research agent | โ | POST /v1/agent | Autonomous multi-step research |
Use Cases for Agent Builders
RAG pipelines โ Fetch docs, articles, or entire sites as clean markdown ready for chunking, embedding, and retrieval.
Price monitoring โ Track product pages across major commerce sites with structured extraction and change detection.
Competitive intel โ Monitor competitor pages, pricing tables, and job boards. Visual diff screenshots catch layout changes CSS selectors would miss.
Research agents โ Give Claude, Codex, Cursor, or your own agent grounded web access through the API or MCP server.
Lead enrichment โ Pull company details, public links, and page structure from business sites without writing per-site parsers.
Content aggregation โ Crawl and extract from communities, docs sites, and publications with domain-native extractors that understand each site's structure.
Architecture
Your Agent
โ
WebPeel (npm / API / MCP)
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Engine Ranker โ
โ HTTP โ Domain API โ Browser โ
โ โ Stealth โ Cloaked โ Cache โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ 55+ Domain Extractors โ
โ reddit ยท github ยท youtube โ
โ amazon ยท arxiv ยท zillow ยท ... โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Content Pipeline โ
โ Readability โ Turndown โ โ
โ Token budgeting โ Chunking โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
Clean markdown / structured JSON
Reliability
WebPeel is built for production agent workflows, not just one-off demos.
- Automated evals in-repo โ smart search and fetch eval suites ship with the codebase
- Post-deploy gate โ critical checks run before calling a deploy healthy
- Engine fallback chain โ when one fetch method fails, WebPeel escalates instead of giving up
- Multiple surfaces, one core โ CLI, API, SDK, and MCP all ride the same extraction pipeline
Security
- SSRF protection โ blocks localhost, private IPs, metadata endpoints,
file://schemes - Helmet.js โ HSTS, X-Frame-Options, nosniff, XSS protection on all responses
- Webhook signing โ HMAC-SHA256 on all outbound webhooks
- API key hashing โ SHA-256 with granular scopes
- Rate limiting โ sliding window, per-tier
- Audit logging โ every API call logged with IP, key, and action
- GDPR compliant โ
DELETE /v1/accountfor full data erasure Security policy โ ยท SLA (99.9% uptime) โ
Why teams choose WebPeel instead of stitching a stack together
| Approach | What it gives you | Where it breaks down |
|---|---|---|
| Raw HTTP + HTML parsing | Cheap, simple fetches | Falls apart on JS-heavy sites, anti-bot pages, and noisy HTML |
| Pure browser automation | Maximum control | Expensive, slow, fragile, and high-maintenance for large-scale use |
| Search-only APIs | Great discovery | Weak page extraction, limited structured output, limited downstream actions |
| Single-purpose scrapers | Fast on one job | You end up composing 4โ6 tools for real agent workflows |
| WebPeel | Fetch + search + crawl + extraction + screenshots + monitoring in one layer | Opinionated toward agent workflows rather than generic scraping |
Links
๐ Documentation ยท ๐ฐ Pricing ยท ๐ฎ Playground ยท ๐ Blog ยท ๐ฌ Discussions ยท ๐ Releases ยท ๐ Status ยท ๐ Security ยท ๐ Changelog
Contributing
Pull requests welcome. Please open an issue first to discuss major changes.
git clone https://github.com/webpeel/webpeel.git
cd webpeel && npm install
npm run build && npm test
License
WebPeel SDK License โ free for personal and commercial use with attribution.
