Opentrace

---

Your AI coding assistant can read your code — but it's blind to production. When something breaks, you copy-paste stack traces, dig through log dashboards, and manually feed context into chat.

OpenTrace fixes that. It's a self-hosted server that connects your AI agent directly to your production observability data via MCP. No dashboards, no UI — the agent is the interface.

You:    "Why is the checkout endpoint throwing 500s?"

Agent:  Error rate on POST /api/checkout spiked from 0.1% to 4.2%
        starting 8 minutes ago. 23 errors, 12 affected users.

        Root cause: NilPointerError at payments_controller.rb:87
        — customer.address is nil when charging.

        This started after the latest deploy (commit a1b2c3d by
        dev@company.com). The commit modified payments_controller.rb
        lines 82-95.

        Want me to fix it?

---

How It Works

Your App                                Your Server
┌──────────────────────┐               ┌──────────────────────────────────┐
│                      │               │         OpenTrace Server          │
│  SDK (Ruby / Node)   │── POST ──────>│                                  │
│  Thin, async, non-   │  /api/v2/logs │  Ingest Pipeline                 │
│  blocking. Just      │  flat JSON    │    PII scrub → fingerprint →     │
│  serialize & send.   │               │    expand in-request logs         │
│                      │               │                 │                 │
└──────────────────────┘               │                 ▼                 │
                                       │  Segmented Log Store             │
Your Laptop                            │    Binary WAL → hourly seal →    │
┌──────────────────────┐               │    columnar chunks + FTS index   │
│                      │               │    45 columns, 6 encoding types  │
│  Claude Code / Cursor│◄── MCP ──────│    + 64MB query cache (tunable)  │
│                      │  over HTTPS   │                                  │
│  Reads .mcp.json     │               │  SQLite (platform data)          │
│  Auto-connects       │               │    Users, watches, error groups  │
│                      │               │                                  │
└──────────────────────┘               │  Connects to your Postgres       │
                                       │    (read-only)                   │
                                       └──────────────────────────────────┘

The SDK captures logs, request performance, SQL queries, external API calls, emails, file operations, and audit trails — then sends everything as flat JSON. Your app never blocks or crashes due to OpenTrace.

The server ingests logs into a custom columnar storage engine (no Elasticsearch, no ClickHouse — just files on disk), monitors health checks, tracks errors, and runs alert rules. Runs on a $4/month VM.

The agent queries all of this through MCP tools — searching logs, investigating errors, explaining slow queries, assessing deploy risk — without you copy-pasting anything.

---

Storage Engine

OpenTrace uses a custom segmented columnar log store instead of SQLite or Elasticsearch for log data:

• Write path: SDK sends flat JSON → server appends to binary WAL → fsync. No indexes on write. 200-500K entries/sec.
• Seal: Every hour, the WAL is sealed into compressed columnar chunks (45 columns, 6 encoding types: dictionary, sparse, delta, bitpack, varint, zstd). 3-5MB peak memory.
• Query: Parallel column scans across segments + custom inverted index for full-text search. Most queries complete in 5-50ms.
• Pruning: rm -rf old segment directories. Instant — no DELETE queries, no VACUUM.
• Storage: ~76MB/hour at 1M logs/hr (vs ~500GB with SQLite). Fits on a $4 VM.

data/logs/
  2026-04-04T10/
    chunk_000.col    3MB     ← 45 compressed columns
    chunk_000.idx    1MB     ← inverted index for FTS
    meta.json        2KB     ← pre-computed histograms
  2026-04-04T11/
    ...
  2026-04-04T12/
    active.wal       12MB    ← current hour, accumulating

Every entry captured by the SDK flows through the full pipeline — PII scrubbing, error fingerprinting, in-request log expansion — and lands in the store as a single row with 45 searchable columns plus an opaque body blob for deep details (SQL queries, stack traces, timeline, etc.).

---

Get Started

1. Deploy the server

Pick one:

ssh root@your-server
curl -fsSL https://raw.githubusercontent.com/adham90/opentrace/main/scripts/install.sh | bash

The installer:

• Downloads the latest binary
• Initializes the database
• Sets up a systemd service
• Optionally installs Caddy for automatic HTTPS
• Prints the connect command when done

docker run -d --name opentrace \
  -p 8080:8080 \
  -v opentrace-data:/data \
  -e OPENTRACE_LISTEN_ADDR=0.0.0.0:8080 \
  ghcr.io/adham90/opentrace:latest

docker compose -f docker-compose.prod.yml up -d

Platform
Railway
Render
DigitalOcean

2. Connect your project

In your project directory, run the connect command the installer printed:

curl -s https://your-server.com/connect | bash

No client install needed. Just curl and bash. The script creates .mcp.json in your project — Claude Code reads this file and connects to OpenTrace automatically.

3. Set up the SDK

Open Claude Code and ask:

"Set up opentrace for my project"

The agent detects your framework, installs the SDK, configures it with the correct API key, and verifies logs are flowing.

SDK	Platform	Install
opentrace	Ruby / Rails	gem 'opentrace'
@opentrace-sdk/node	Node.js	npm install @opentrace-sdk/node

The SDK captures structured logs, request lifecycle data (SQL queries, external API calls, cache metrics, view rendering, email delivery), error traces with stack traces, and runtime metrics — all sent as flat JSON with async I/O. Your app never blocks.

Any other stack — no adapter needed. The SDKs are convenience, not a requirement: ingestion is one HTTP endpoint taking flat JSON, and the full contract is published by your own server at GET /spec. Ask your agent:

"Read https://your-server.com/spec and write an OpenTrace client for this app"

It writes ~100 lines against the spec, then checks its work in a loop against the server's dry-run endpoint:

curl -X POST 'https://your-server.com/api/v2/logs?validate=1' \
  -H 'Authorization: Bearer <key>' -d '{"level":"info","msg":"hello"}'

{"valid": false, "results": [{
  "errors": ["message: required and must be non-empty"],
  "warnings": ["service: not sent — every query groups by service"],
  "unknown_fields": [{"field": "msg", "did_you_mean": "message"}]
}]}

Nothing is stored, every problem in the payload comes back at once, and unknown fields get a suggested correction — so a client for a stack nobody has written an SDK for is a loop, not a guess. Drop the parameter when it reports "valid": true.

4. Ask your agent anything

You're done. Start asking:

Question	What happens
"What errors are happening in production?"	Agent searches error groups, shows impact and stack traces
"Why is the payments endpoint slow?"	Agent checks request performance — duration, SQL count, external API time, N+1 detection
"Show me logs from the last hour with level ERROR"	Agent searches logs with columnar filters
"Is it safe to deploy this change?"	Agent checks blast radius, code risk scores, recent errors
"Generate tests for the most common production errors"	Agent creates regression tests from real error data
"Set up a watcher for checkout error rate > 1%"	Agent creates a threshold alert
"What happened after the last deploy?"	Agent checks deploy impact, error rate changes

---

What Can the Agent Do?

Search & Debug Logs

Full-text search across all services via custom inverted index. Filter by level, service, trace ID, time range, handler, status code, error class. Assemble distributed traces. Compare error rates between time periods.

Deep Request Capture

Every HTTP request captured by the middleware includes: SQL queries with durations and EXPLAIN plans, external API calls, cache hits/misses, view rendering times, email deliveries, file operations, audit trail, and a waterfall timeline — all in one log entry.

Investigate Errors

Errors are automatically grouped by fingerprint (hash of error class + source file + line). The agent sees occurrence counts, affected users, impact scores, and full stack traces. It can resolve or ignore error groups.

Query Your Database

Connect your Postgres databases (read-only). The agent runs EXPLAIN ANALYZE on slow queries, checks index health, detects lock contention, and identifies N+1 query patterns. All queries are validated SELECT-only via SQL AST parsing.

Monitor Uptime

Create HTTP health checks that run on a schedule. The agent sees uptime percentages, response times, and gets notified when endpoints go down.

Set Up Alerts

Create threshold watches on error rate, response time (mean or p95), log volume, error count, SQL count, cache hit rate, or service heartbeat. The agent can create watches for code it just deployed — self-monitoring its own changes.

Watch and health-check alerts can be delivered to Slack or Telegram — see docs/notifications.md for setup.

Assess Code Risk

Every file and endpoint gets a risk score based on error frequency, investigation history, and change velocity. Before modifying a file, the agent checks its production behavior.

Track Deploys

The SDK sends the git commit hash with every log. OpenTrace records a deploy the first time it sees a new commit for a service and environment, so since: "last_deploy" works as a time window on every tool that takes one — no wall-clock guessing.

Catch Up After Being Away

overview(action: "catchup") returns every new error, alert, and deploy since your last catch-up, then advances your cursor. The cursor is per person, so on a small team one member draining the queue does not hide the incident from everyone else. peek: true reads without advancing.

Answer "Customer X Says It's Broken"

users(action: "timeline", user_id: "...") returns everything one person hit — requests, errors, timings. tenant_id does the same for a B2B account, and impact runs it the other way: who is this error actually hurting.

On-Call While You Sleep

When an alert fires, OpenTrace can run your own agent CLI against it and send you a diagnosis instead of a threshold. It shells out to claude -p (or codex exec, or anything reading stdin), so OpenTrace never holds a model credential and is not tied to one vendor — and on a Claude Pro/Max subscription it costs nothing extra.

Optionally it files each diagnosis as a GitHub issue, one per error fingerprint per environment, reopening and commenting on recurrence rather than filing duplicates. Issues, never pull requests: the observability box has your production data, not your code.

Off by default — turning it on sends log excerpts to a model provider. See .env.example.

Monitoring You Didn't Have To Set Up

The first time a service reports, it gets two watches nobody asked for: an error-rate spike (>5% over 5 minutes, twice running) and a heartbeat (15 minutes of silence). If you already made watches for that service, these stay out of the way.

Every new commit hash gets its own 24h watch plus a check-in 1 hour and 24 hours after the deploy — error rate and response time against the baseline captured when it landed, plus any error group that is new since. Delivered to whichever chat channel you configured.

The Morning Brief

overview.catchup only answers when an agent asks. The same payload is also pushed to Telegram/Slack once a day (OPENTRACE_CATCHUP_PUSH_INTERVAL), so the night you did not open an agent is not a night you missed.

Set OPENTRACE_CRITICAL_PATHS=/checkout,billing and anything matching — a route, a service, an error message, or a watch you tagged critical — sorts to the top of the brief and of the error impact ranking. Ten people who cannot pay outrank a thousand who cannot load an avatar.

Notice When OpenTrace Itself Dies

Set OPENTRACE_HEARTBEAT_URL and the server pings it every minute. If the box dies, the pings stop and your external monitor tells you. Without it, a dead OpenTrace looks exactly like an OpenTrace with nothing to report.

---

MCP Tools

OpenTrace exposes 14 tools with 90+ actions via MCP. Each tool returns suggested_tools with pre-filled arguments so the agent knows what to call next.

Tool	Actions	What it does
logs	search, context, attributes, stats, summary, performance, trace, compare	Full-text log search, distributed trace assembly, N+1 detection
errors	list, detail, investigate, impact, user_errors, ranking, resolve, ignore, reopen, new	Error grouping by fingerprint, user impact scoring, stack traces
database	queries, explain, tables, activity, locks, connections, indexes, schema, storage, kill_query, long_transactions	Postgres introspection, EXPLAIN plans, lock and connection analysis
watches	status, create, delete, alerts, dismiss, acknowledge, investigate	Threshold alerts on error rate, latency, request volume
overview	status, triage, catchup, diagnose, timeline, investigate, changes, settings, notes, delete_note	System health, what you missed since last visit, incident timeline, settings, agent memory
users	timeline, errors, impact	What one customer hit, which errors they saw, who an error affects
code	risk, fragile	Risk scores for a set of files, and the riskiest code entities overall
deep_capture	request_capture, sql_captures, http_captures, email_captures, audit_trail, search_audit, search_sql, file_captures, get_pii_config, update_pii_config, get_retention, update_retention	Per-request detail read from the log body blob: SQL, external HTTP calls, emails, audit trail, file ops, PII config
healthchecks	list, uptime, create, delete	HTTP endpoint monitoring with uptime tracking
servers	list, query, health	Server and process metrics (CPU, memory, GC)
connectors	list, get, create, test, update, delete	Manage database connectors (Postgres, MySQL, etc.)
setup	status, detect, guide, verify	SDK setup assistant — detects framework, provides config with API key
admin	update_retention, users, update_role, toggle_active, delete_user, audit	User management, retention, audit log (admin only)

---

Security

Protection	How
No self-registration	First curl .../connect creates admin. Everyone else needs an invite.
Per-user tokens	Each developer gets a personal MCP token, stored in their local .mcp.json. Revocable independently.
HTTPS via Caddy	The install script sets up Caddy with automatic Let's Encrypt certificates.
PII scrubbing	Credit cards, emails, phone numbers, SSNs, and configurable sensitive fields are scrubbed from request bodies before storage.
Rate limiting	Auth endpoints are rate-limited — 10 attempts per minute per IP.
Read-only DB access	All queries against your Postgres are validated SELECT-only via SQL AST parsing.
API key auth	SDK log ingestion requires a Bearer token.
No telemetry	Fully self-hosted. No external calls. No tracking. Your data stays on your server. The on-call agent is the one exception, and it is off by default.
Prompt-injection boundary	Alert data reaches the on-call agent on stdin, below an explicit untrusted-data marker, and the agent is pointed at read-only tools. Error messages come from your users.

---

Configuration

Server-side environment variables (.env file):

Variable	Default	Description
OPENTRACE_LISTEN_ADDR	127.0.0.1:8080	HTTP listen address
OPENTRACE_DATA_DIR	~/.opentrace	Data directory (SQLite + log segments)
OPENTRACE_API_KEY	(auto-generated)	Bearer token for SDK log ingestion
OPENTRACE_MAX_QUERY_ROWS	500	Max rows returned from SQL queries
OPENTRACE_STATEMENT_TIMEOUT_MS	5000	SQL query timeout in milliseconds
OPENTRACE_TRUSTED_PROXIES	(empty)	Comma-separated proxy IPs for rate limiting
OPENTRACE_ALERT_WEBHOOK_URL	(empty)	Generic JSON webhook for watch + health-check alerts
OPENTRACE_HEARTBEAT_URL	(empty)	Liveness ping sent every 60s, for an external monitor to alarm on
OPENTRACE_ONCALL_ENABLED	false	Run your own agent CLI against alerts and deliver a diagnosis
OPENTRACE_ONCALL_CMD	claude -p --permission-mode dontAsk	The agent command; prompt on stdin, diagnosis on stdout
OPENTRACE_ONCALL_GITHUB_REPO	(empty)	owner/name — file each diagnosis as a deduped GitHub issue
OPENTRACE_CATCHUP_PUSH_INTERVAL	24h	Push the catch-up brief to Telegram/Slack on this cadence; off disables
OPENTRACE_CRITICAL_PATHS	(empty)	Comma-separated money-path patterns (/checkout,billing) — matches sort first

See .env.example for all options.

Alert delivery to Slack and Telegram is configured at runtime, not via environment variables — see docs/notifications.md.

---

How It's Built

• Go — single binary, no runtime dependencies, cross-compiled for Linux and macOS
• Custom columnar storage — 45-column format with 6 encoding types (dictionary, sparse, delta, bitpack, varint, zstd). Binary WAL for writes, hourly seal into compressed chunks, custom inverted index for FTS.
• SQLite — for platform data (users, watches, error groups, health checks). Not used for log storage.
• MCP — native Model Context Protocol with Streamable HTTP and SSE transports
• Pure Go — no CGO, no system dependencies, go build and ship

---

Development

git clone https://github.com/adham90/opentrace.git && cd opentrace
cp .env.example .env
go build -o opentrace ./cmd/opentrace
./opentrace serve

go test -short -race ./...    # unit tests (44 packages)
go vet ./...                  # linting

---

License

MIT — use it however you want.