Md Succ AI

Convert any webpage, document, feed, or video to clean, readable Markdown. Built for AI agents, MCP tools, and RAG pipelines. Powered by succ.

Quick Start

# Markdown output
curl https://md.succ.ai/https://example.com

# JSON output
curl -H "Accept: application/json" https://md.succ.ai/https://example.com

# Documents (PDF, DOCX, XLSX, CSV)
curl https://md.succ.ai/https://example.com/report.pdf

# YouTube transcript
curl https://md.succ.ai/https://youtube.com/watch?v=dQw4w9WgXcQ

# RSS/Atom feed
curl https://md.succ.ai/https://blog.example.com/feed.xml

# LLM-optimized (30-50% fewer tokens)
curl "https://md.succ.ai/https://example.com?mode=fit"

# Batch convert
curl -X POST https://md.succ.ai/batch \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com", "https://httpbin.org/html"]}'

That's it. No API key, no signup, no SDK. Just prepend https://md.succ.ai/ to any URL.

Features

Documents are also detected by URL extension (.pdf, .docx, .xlsx, .csv) when Content-Type is application/octet-stream.

API

Base URL: https://md.succ.ai Docs: /docs (interactive Scalar UI) | /openapi.json (OpenAPI 3.1 spec)

Endpoints

Query Parameters

Response Headers

Rate Limits

{
  "title": "Example Domain",
  "url": "https://example.com",
  "content": "# Example Domain\n\nThis domain is for use in...",
  "fit_markdown": "# Example Domain\n\nThis domain is...",
  "fit_tokens": 20,
  "excerpt": "This domain is for use in documentation examples...",
  "tokens": 33,
  "tier": "fetch",
  "readability": true,
  "method": "readability",
  "quality": { "score": 0.85, "grade": "A" },
  "time_ms": 245
}

curl -X POST https://md.succ.ai/batch \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com",
      "https://httpbin.org/html",
      "https://github.com"
    ],
    "options": {
      "mode": "fit",
      "links": "citations"
    }
  }'

Returns an array of results. Up to 50 URLs, processed with 10-way concurrency. Per-URL 60s timeout.

# Submit async job
curl -X POST https://md.succ.ai/async \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "callback_url": "https://your-server.com/webhook"
  }'
# → {"job_id": "abc12345", "status": "processing", "poll_url": "/job/abc12345"}

# Poll for result
curl https://md.succ.ai/job/abc12345

Webhook delivers JSON POST to callback_url on completion/failure. HTTPS required, 3 retries with exponential backoff. Private/internal addresses blocked (SSRF-safe).

curl -X POST https://md.succ.ai/extract \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://github.com/trending",
    "schema": {
      "type": "object",
      "properties": {
        "repositories": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "author": { "type": "string" },
              "description": { "type": "string" },
              "stars_today": { "type": "number" }
            }
          }
        }
      }
    }
  }'

Returns structured JSON matching the provided schema, extracted by LLM. Automatically retries with headless browser for SPA/JS-heavy sites when initial extraction returns empty data.

# Citation-style links (numbered references)
curl "https://md.succ.ai/?url=https://en.wikipedia.org/wiki/Markdown&links=citations"

# LLM-optimized output (pruned boilerplate)
curl "https://md.succ.ai/?url=https://htmx.org/docs/&mode=fit"

# Token limit
curl "https://md.succ.ai/?url=https://example.com&mode=fit&max_tokens=4000"

# RSS feed as markdown
curl https://md.succ.ai/https://hnrss.org/frontpage

How It Works

4-tier conversion pipeline — each tier only activates if the previous one produced insufficient quality:

URL ──→ Cache hit? ──→ Return cached result (Redis, dynamic TTL)
         │
         ├─ YouTube? ──→ Transcript extraction (innertube API)
         │
         ├─ RSS/Atom feed? ──→ Feed parsing with item metadata
         │
         ├─ Document? (PDF, DOCX, XLSX, CSV)
         │   └─→ Document converter → Markdown
         │
         ├─ Tier 1: HTTP fetch + 9-pass extraction
         │   └─→ Readability → Defuddle → Article Extractor → CSS selectors
         │       → Schema.org → Open Graph → Text density → Body fallback
         │
         ├─ Tier 2: Camoufox headless browser (SPA/JS-heavy)
         │   └─→ Same 9-pass pipeline on rendered DOM
         │
         ├─ Tier 2.5: LLM extraction (quality < B)
         │   └─→ nano-gpt API → content extraction
         │
         └─ Tier 3: BaaS anti-bot bypass (CF Turnstile / quality < D)
             └─→ ScrapFly → ZenRows → ScrapingBee (rotation)
             └─→ Same 9-pass pipeline on returned HTML

Cloudflare challenge pages are detected automatically. When fetch gets a CF challenge, browser is skipped (saves IP), and BaaS providers handle the bypass.

When both LLM and BaaS are needed, they race in parallel — saves 30-45s vs sequential.

Two-layer cache system backed by Redis 7:

Cache keys use SHA-256 hashes to prevent poisoning via long/malicious URLs. Tracking parameters (UTM, fbclid, gclid, etc.) are stripped before hashing. Falls back to in-memory Map when Redis is unavailable.

Self-Hosting

Docker (recommended)

git clone https://github.com/vinaes/md-succ-ai.git
cd md-succ-ai
cp .env.example .env  # edit with your API keys and passwords
docker compose up -d

This starts four containers:

The API is available at http://localhost:3100.

Local (without Docker)

npm install
npx camoufox-js fetch
npm start

Redis is optional for local development. Without Redis, caching and rate limiting fall back to in-memory Map, and async jobs are unavailable.

BaaS providers are optional. When configured, they activate as Tier 3 for Cloudflare-protected sites. Providers are tried in order; if one hits rate limits, the next is used automatically.

An example nginx config is in nginx/md.succ.ai.conf:

• Rate limiting: 10 req/s per IP, burst 20
• Connection limit: 10 concurrent per IP
• Proxy timeouts: 60s read (for browser renders)
• POST endpoints with appropriate body limits
• HSTS, security headers (nosniff, X-Frame-Options, Referrer-Policy)
• /metrics blocked (403)
• /grafana/ proxied to Grafana container with WebSocket support

Monitoring

The project ships with a full Prometheus + Grafana stack:

Prometheus scrapes the /metrics endpoint every 10s (internal Docker network only).

Grafana is pre-provisioned with a 15-panel dashboard:

• Request rate, response time percentiles (p50/p95/p99)
• Conversion tier distribution, cache hit rate
• Quality score distribution, tokens per conversion
• Rate limit rejections, async job status
• Browser pool utilization, webhook deliveries
• Node.js process metrics (CPU, memory, event loop lag)

Access Grafana at https://your-domain/grafana/ (proxied via nginx).

Custom Metrics

Plus Node.js default metrics (CPU, memory, event loop, GC) via prom-client.

Security

• SSRF protection — URL validation, DNS resolution checks (IPv4 + IPv6), redirect validation per hop, Camoufox route blocking, webhook callback DNS validation
• Private IP blocking — 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, CGNAT, cloud metadata hostnames, hex/octal IP formats, IPv6 mapped addresses
• Input limits — 5MB response size, 5 max redirects, content-type validation, body size limits per endpoint
• Output sanitization — Error messages stripped of internal paths/stack traces, URLs sanitized in responses
• Cache security — SHA-256 hashed keys (no URL poisoning), tracking params stripped, Redis LRU eviction (128MB cap)
• Redis authentication — --requirepass with password from .env, authenticated connection URL
• API key safety — BaaS API keys only used in outbound requests, never logged or exposed in responses
• LLM hardening — Prompt injection protection (HTML sanitization, document delimiters, output validation), schema field whitelist, blocked schema keywords ($ref, $defs, etc.)
• Rate limiting — Per-IP via Redis INCR+EXPIRE (atomic pipeline), CF-Connecting-IP support, in-memory fallback
• Security headers — HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy
• CDN integrity — Subresource Integrity (SRI) on third-party scripts
• Container security — Non-root user (mduser), no-new-privileges, pinned image versions
• CF challenge detection — Cloudflare challenge pages detected and handled without wasting browser/BaaS credits

Architecture

                    ┌──────────────────┐
                    │   Cloudflare     │
                    │   (TLS + CDN)    │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │   nginx          │
                    │   (rate limit,   │
                    │    HSTS, proxy)  │
                    └────────┬─────────┘
                             │
         ┌───────────────────┼───────────────────┐
         │                   │                   │
┌────────▼─────────┐ ┌──────▼───────┐ ┌─────────▼────────┐
│  md-succ-ai      │ │  Prometheus  │ │  Grafana         │
│  (Node 22, Hono) │ │  (scrape     │ │  (dashboards,    │
│  Camoufox       │ │   /metrics)  │ │   alerting)      │
│  BaaS clients    │ └──────────────┘ └──────────────────┘
│  Pino logging    │
└────────┬─────────┘
         │
┌────────▼─────────┐
│  Redis 7         │
│  (cache, rate    │
│   limit, jobs)   │
└──────────────────┘

License

FSL-1.1-Apache-2.0 — Free for non-competitive use. Apache 2.0 after 2 years.

Disclaimer: Not affiliated with NanoGPT. LLM features use the NanoGPT API for pay-per-prompt model access.

Part of the succ ecosystem.