If prediction is CORRECT:  Value = -log(p)
If prediction is INCORRECT: Value = log(p)

where p = market probability for the predicted outcome at prediction time

📋 Table of Contents

• 🚀 Our Mission
• 🎯 What is FutureShow?
• ✨ Key Features
• 🖼️ Screenshots
• 🏗️ System Architecture
• 🏃 Quick Start
• 🔧 MCP Tools Reference
• 📊 Forecasting Pipeline
• 🌐 Dashboard & API
• ⚙️ Advanced Configuration
• 📁 Data Formats & Output
• 🛠️ Development
• 🤝 Contributing

🚀 Our Mission

Can AI Agents Outthink the Wisdom of the Crowd?

🧠 The Foundation: Human Collective Intelligence

Prediction markets represent humanity's most sophisticated mechanism for aggregating collective intelligence. When thousands of participants stake real money on future outcomes, their combined judgment distills into remarkably accurate probability estimates. This "wisdom of the crowd" has consistently outperformed individual experts across virtually every domain.

🔬 Our Approach: Real-World Testing

FutureShow conducts a transparent, ongoing experiment:

• ⚔️ Direct Competition: Frontier AI models vs. market consensus
• 📊 Rigorous Methodology: Every prediction timestamped, every outcome independently verified
• ⚖️ Fair Comparison: Identical decision points, identical timeframes
• 🚫 Zero Bias: No cherry-picking, no hindsight adjustments

🔍 Beyond the Leaderboard

This study investigates AI boundaries beyond performance tracking:

• ✅ Where AI excels in prediction accuracy
• ❌ Where AI systematically fails against human crowds
• 💰 Whether machines can generate alpha against aggregated human wisdom

🎯 What is FutureShow?

Can AI agents predict the future better than human crowds betting real money?

FutureShow is an Open-Source AI Benchmarking platform that puts this question to the ultimate test. We evaluate frontier AI Models against prediction markets — where thousands of participants stake real money on future outcomes, creating some of the most accurate probability estimates available.

How It Works

Our system operates as a continuous, real-world experiment:

📊 Market Intelligence

• Monitors live prediction markets on Polymarket
• Tracks events spanning politics, economics, tech, sports, and culture

🤖 AI Agent Deployment

• Deploys multiple frontier models (GPT-5, Claude, Gemini, DeepSeek)
• Each agent analyzes identical market conditions independently

🔍 Real-Time Research

• Agents gather intelligence via web search, news, Reddit, and Twitter
• No human intervention — pure AI reasoning and research

📈 Transparent Tracking

• Records each model's YES/NO predictions with full reasoning
• Tracks accuracy as real events unfold
• Maintains live performance leaderboard

Why This Matters

• 🎲 Prediction markets aren't just betting — they're humanity's most sophisticated mechanism for aggregating collective intelligence. When people risk real money, their combined judgment creates remarkably accurate forecasts that consistently outperform individual experts.

• 🧠 This makes them perfect AI benchmarks — objective, real-time, and impossible to game. No synthetic datasets, no contrived scenarios. Just AI versus the wisdom of crowds, measured transparently.

✨ Key Features

🤖 Multi-Model Agent Arena

FutureShow supports any LLM accessible via LiteLLM, including:

Each model runs as an independent agent with:

• Dedicated tool access (search, market data, reasoning)
• Isolated position/PnL tracking
• Persistent session state via SQLite
• Configurable max steps, retries, and delays

📈 Real-Time Market Intelligence

Agents have access to comprehensive MCP (Model Context Protocol) tools:

┌─────────────────────────────────────────────────────────────────┐
│                    🔧 MCP Tool Suite                            │
├─────────────────────────────────────────────────────────────────┤
│  📊 Market Data        │  🔍 Web Search      │  💬 Social       │
│  ├─ list_events        │  ├─ google_web      │  ├─ reddit       │
│  ├─ list_markets       │  ├─ google_news     │  └─ twitter      │
│  ├─ get_market_info    │  └─ exa_semantic    │                  │
│  ├─ get_market_prices  │                     │  💹 Trading      │
│  └─ get_market_history │  🔢 Utilities       │  ├─ buy          │
│                        │  └─ math_tool       │  └─ sell         │
└─────────────────────────────────────────────────────────────────┘

🏆 Live Leaderboard & Dashboard

• Real-time accuracy tracking across all resolved markets
• Per-model breakdowns with correct/total/abstain counts
• Category-wise performance (Politics, Crypto, Sports, etc.)
• Historical forecast browsing with full reasoning trails

📊 Simulated Trading Engine

FutureShow includes a realistic trading simulation:

• Order book simulation using live Polymarket CLOB data
• Slippage modeling with configurable liquidity impact
• Position tracking with JSONL ledger persistence
• PnL calculation with NAV (Net Asset Value) history

🖼️ Screenshots

📊 Forecasts Overview Main dashboard showing all prediction markets. Each card displays event title, market probability, and model predictions with colored icons indicating YES/NO votes.	📋 Event Detail Page Deep dive into a specific market with full prediction history, AI reasoning trails, probability charts, and final outcomes for closed events.
🏆 Model Leaderboard Competitive rankings showing accuracy, human baseline comparison, and Prediction Value — measuring how much alpha each model generates vs market consensus.	⚡ Batch Prediction in Action Watch multiple AI agents analyze markets in parallel with real-time logging, concurrent execution, and automatic result persistence.

🏗️ System Architecture

🏃 Quick Start

1️⃣ Environment Setup

# Clone the repository
git clone https://github.com/HKUDS/FutureShow.git
cd FutureShow

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install with dev dependencies
pip install -e .[dev]

2️⃣ API Key Configuration

Copy the example environment file and fill in your API keys:

cp .env.example .env

Edit .env with your credentials:

# ═══════════════════════════════════════════════════════════════
# LLM Provider API Keys (configure at least one)
# ═══════════════════════════════════════════════════════════════
DEEPSEEK_API_KEY="sk-xxx"                # DeepSeek models
DEEPSEEK_BASE_URL="https://api.deepseek.com/v1"

OPENROUTER_API_BASE="https://openrouter.ai/api/v1"
OPENROUTER_API_KEY="sk-or-xxx"           # Access 100+ models via OpenRouter

OPENAI_API_BASE="https://api.openai.com/v1"  # Or custom endpoint
OPENAI_API_KEY="sk-xxx"                  # OpenAI GPT models

# Optional: Additional LLM providers
PRIVATE_API_BASE=""                      # Custom LLM endpoint
PRIVATE_API_KEY=""

LITE_API_BASE=""                         # LiteLLM proxy endpoint
LITE_API_KEY=""

# ═══════════════════════════════════════════════════════════════
# Search & Intelligence Tools
# ═══════════════════════════════════════════════════════════════
SERPER_API_KEY="xxx"                     # Google Search via Serper.dev
EXA_API_KEY="xxx"                        # Exa semantic search
RAPIDAPI_KEY="xxx"                       # RapidAPI for additional services

# ═══════════════════════════════════════════════════════════════
# Polymarket (optional, for trading mode)
# See "How to Get Polymarket Credentials" below
# ═══════════════════════════════════════════════════════════════
POLYMARKET_API_KEY=""                    # API key from Polymarket
PRIVATE_KEY=""                           # Your wallet private key
KEY=""                                   # Same as PRIVATE_KEY

# ═══════════════════════════════════════════════════════════════
# Agent Configuration
# ═══════════════════════════════════════════════════════════════
AGENT_MAX_STEP=30                        # Max reasoning steps per agent
RUNTIME_ENV_PATH=".runtime_env.json"     # Runtime state file
DEBUG=1                                  # Debug mode (1=enabled, 0=disabled)

# Make sure PRIVATE_KEY is set in your .env file first
python futureshow/utils/generate_poly_apikey.py

3️⃣ Run Forecasting Benchmark

Start the AI forecasting agents to predict Polymarket events:

# ─── Single Round ───
# Run all enabled models once on current watchlist
python run_forecast_loop.py --once

# ─── Continuous Loop ───
# Run predictions every 6 hours (default), refresh watchlist each round
python run_forecast_loop.py --refresh --interval 21600

# ─── Custom Configuration ───
# Limit to 4 models, target specific month's events
python run_forecast_loop.py \
  --limit 4 \
  --month 1 \
  --year 2025 \
  --refresh

4️⃣ Track Results & Launch Dashboard

# Start event tracker (monitors market status & prices every 30 min)
python run_forecast_trackers.py --interval 1800 &

# Launch the forecasting dashboard
python web_server_pred.py
# Open http://localhost:10086

The dashboard displays:

• Forecasts page: All active/closed predictions with model votes
• Detail page: Full prediction history and AI reasoning for each event
• Leaderboard: Model accuracy rankings vs human baseline

🎰 Optional: Live Trading Mode

# ─── Run Trading Agents ───
# Single round with trading enabled
python main.py configs/default_config.json

# Continuous trading loop (every 40 minutes)
python run_agents_loop.py \
  --interval 2400 \
  --overrun-pause 900 \
  --config configs/default_config.json

# ─── Track PnL & Launch Trading Dashboard ───
# Start PnL tracking (updates every 10 seconds)
python run_pnl_trackers.py --interval 10 --config configs/default_config.json &

# Launch trading dashboard
python web_server.py
# Open http://localhost:10032

🔧 MCP Tools Reference

FutureShow provides agents with these Model Context Protocol tools:

📊 Polymarket Data Tools

01. trump-2028 | p=0.234 | vol=1523000.0 | OI=892341 | cat=US Politics | Will Trump run in 2028?
    tags: Politics, Elections, Trump
    time: end=2028-11-15T00:00:00Z | updated=2025-01-20T12:00:00Z
    liq: 45000 | comments=234
    market0: slug=trump-2028-yes | outcomes=['Yes', 'No'] | prices=[0.234, 0.766] | mid=0.234

02. btc-100k-jan | p=0.891 | vol=982000.0 | OI=456123 | cat=Crypto | Bitcoin above $100k by Jan 31?
    ...

🔍 Search Tools

💹 Trading Simulation Tools

🔢 Utility Tools

📊 Forecasting Pipeline

Agent Workflow

Prediction Format

Agents output predictions in a structured format:

<PREDICTION>market-slug|YES</PREDICTION>

Or for binary markets without explicit slug:

<PREDICTION>YES</PREDICTION>

Supported values: YES, NO, ABSTAIN

🌐 Dashboard & API

Web Server

python web_server.py
# Serves on http://0.0.0.0:10032 by default

Environment variables:

• WEB_HOST: Bind address (default: 0.0.0.0)
• WEB_PORT: Port number (default: 10032)

REST API Endpoints

{
  "ok": true,
  "signature": "gpt-5",
  "date": "2025-01-20",
  "times": ["2025-01-20T00:00:00Z", "2025-01-20T01:00:00Z", ...],
  "nav": [10000.0, 10023.45, 10089.12, ...],
  "returns": [0.0, 0.23, 0.89, ...],
  "latest": {
    "timestamp": "2025-01-20T23:59:00Z",
    "nav": 10234.56,
    "cash": 5234.56,
    "positions_value": 5000.0
  },
  "count": 24,
  "full": false
}

⚙️ Advanced Configuration

Config File Structure

{
  "agent_type": "PolymarketAgent",
  
  "date_range": {
    "init_date": "2025-01-01",
    "end_date": "2025-12-31"
  },
  
  "agent_config": {
    "max_steps": 50,           // Max tool calls per event
    "max_retries": 3,          // Retry on transient failures
    "base_delay": 0.5,         // Retry backoff base (seconds)
    "initial_cash": 10000.0    // Starting cash for simulation
  },
  
  "log_config": {
    "log_path": "./data/agent_data"
  },
  
  "models": [
    {
      "name": "gpt-5",
      "basemodel": "openai/gpt-5",
      "signature": "gpt-5",
      "enabled": true,
      "provider": "openai"
    },
    {
      "name": "claude-4.5-sonnet",
      "basemodel": "openrouter/anthropic/claude-sonnet-4.5",
      "signature": "claude-4.5-sonnet",
      "enabled": true,
      "provider": "openrouter"
    },
    {
      "name": "gemini-2.5-pro",
      "basemodel": "openrouter/google/gemini-2.5-pro",
      "signature": "gemini-2.5-pro",
      "enabled": true,
      "provider": "openrouter"
    },
    {
      "name": "deepseek-v3.1",
      "basemodel": "openrouter/deepseek/deepseek-chat-v3.1",
      "signature": "deepseek-v3.1",
      "enabled": true,
      "provider": "openrouter"
    }
  ]
}

Runtime Environment

The system writes .runtime_env.json to coordinate state:

{
  "SIGNATURE": "gpt-5",
  "CURRENT_DATETIME": "2025-01-20T15:30:00Z",
  "INIT_DATETIME": "2025-01-01T00:00:00Z",
  "IF_TRADE": false
}

Watchlist Management

Edit futureshow/utils/polymarket_watchlist.json or use API:

from futureshow.utils.polymarket_watchlist import (
    refresh_trending_watchlist,
    load_watchlist,
    add_events_to_watchlist,
    remove_events_from_watchlist,
)

# Refresh with trending events
refresh_trending_watchlist(year=2025, month=1)

# Manual additions
add_events_to_watchlist(["custom-event-slug"])

📁 Data Formats & Output

Directory Structure

data/
├── agent_data/
│   └── {model_signature}/
│       ├── position/
│       │   ├── position.jsonl    # Trade ledger
│       │   └── liquidity.json    # Simulated liquidity state
│       ├── pnl/
│       │   └── intraday_{date}.jsonl  # NAV snapshots
│       └── log/
│           └── {date}/
│               └── log.jsonl     # Agent reasoning traces
│
├── forecasts/
│   └── {model_signature}/
│       └── {event_slug}/
│           ├── forecasts.jsonl   # Predictions over time
│           ├── tracking.jsonl    # Market state snapshots
│           └── result.json       # Final resolution
│
└── cache/
    └── polymarket_markets/       # API response cache
        └── {slug}.json

Position Ledger Format (position.jsonl)

{
  "timestamp": "2025-01-20T15:30:00Z",
  "id": 42,
  "this_action": {
    "action": "buy",
    "market": "btc-100k-jan",
    "outcome": "Yes",
    "requested_cost": 1000.0,
    "spent": 998.45,
    "shares": 1123.5,
    "avg_price": 0.889,
    "partial_fill": false,
    "levels": [
      {"price": 0.888, "shares": 500, "cost": 444.0},
      {"price": 0.890, "shares": 623.5, "cost": 554.45}
    ]
  },
  "positions": {
    "CASH": 4001.55,
    "btc-100k-jan:Yes": 1123.5,
    "trump-2028:No": 500.0
  }
}

Forecast Record Format (forecasts.jsonl)

{
  "timestamp": "2025-01-20T15:30:00Z",
  "signature": "gpt-5",
  "event_slug": "btc-100k-jan",
  "event_title": "Bitcoin above $100k by Jan 31?",
  "forecast": "Based on current momentum and institutional inflows...\n\n<PREDICTION>btc-100k-jan-yes|YES</PREDICTION>",
  "predictions": [
    {"slug": "btc-100k-jan-yes", "outcome": "YES"}
  ]
}

🛠️ Development

Running Tests

# All tests
pytest -q

# Specific module
pytest tests/test_polymarket_data.py -v

# With coverage
pytest --cov=futureshow --cov-report=html

Code Quality

# Lint
ruff check futureshow tests

# Format
ruff format futureshow tests

# Type check
mypy futureshow

Project Structure

FutureShow/
├── futureshow/                    # 🎯 Core package
│   ├── agent/                     # Agent implementations
│   │   ├── __init__.py
│   │   └── polymarket/
│   │       ├── polymarket_agent.py         # Trading agent
│   │       ├── polymarket_forecast_agent.py # Forecast-only agent
│   │       └── market_preview.py           # Market analysis utils
│   ├── prompt/                    # System prompts
│   │   ├── polymarket_agent_prompt.py
│   │   └── polymarket_forecast_prompt.py
│   ├── tool/                      # MCP tools (FastMCP + function_tool)
│   │   ├── tool_polymarket_data.py   # Market data (1170 lines)
│   │   ├── tool_polymarket_trade.py  # Trading simulation (655 lines)
│   │   ├── tool_google.py            # Serper search
│   │   ├── tool_exa.py               # Semantic search
│   │   ├── tool_reddit.py            # Reddit API
│   │   ├── tool_twitter.py           # X/Twitter API
│   │   └── tool_math.py              # Math evaluation
│   └── utils/                     # Helpers
│       ├── agent_logs.py             # Logging hooks
│       ├── general_tools.py          # Config helpers
│       ├── polymarket_watchlist.py   # Watchlist management
│       └── polymarket_position_tools.py
│
├── frontend/                      # 🖥️ Web dashboard
│   ├── index.html
│   ├── app.js                     # Chart.js + fetch API
│   ├── styles.css
│   └── icons/                     # Model logos
│
├── configs/                       # ⚙️ Configuration
│   └── default_config.json
│
├── tests/                         # 🧪 pytest suite
│   ├── conftest.py
│   ├── test_polymarket_data.py
│   ├── test_polymarket_trade.py
│   └── ...
│
├── main.py                        # Entry point
├── web_server.py                  # Dashboard server (358 lines)
├── run_agents_once.py             # Single-pass runner
├── run_agents_loop.py             # Continuous runner
├── run_pnl_trackers.py            # PnL tracking loop
└── run_forecast_loop.py           # Forecast-only loop

🤝 Contributing

We welcome contributions! Here's how:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

Guidelines

• Follow existing code style (ruff formatting)
• Add tests for new tools/agents
• Update documentation for API changes
• Include sample output for new features

📄 License

This project is licensed under the MIT License - see LICENSE for details.