Taking Stock - Fast Database

Analytics backend and AI-powered query interface for Michael Mandiberg's Taking Stock artwork.

Overview

This project migrates the Taking Stock database from MySQL/MongoDB to ClickHouse for fast analytical queries over tens of millions of stock photos. It provides an AI-queryable interface via MCP (Model Context Protocol) and a web chat UI for natural language data exploration.

The MooseStack service runs ClickHouse and exposes an MCP server. The web app provides a chat interface where users ask questions in natural language, Claude generates SQL queries via MCP tools, and results are returned for display.

Architecture

This is a pnpm monorepo with three packages:

packages/moosestack-service/ — ClickHouse-backed data service with MCP server. Handles data ingestion and exposes query tools via API at http://localhost:4000.

packages/web-app/ — Next.js chat interface for AI-powered data exploration. Connects to MooseStack MCP server and uses Anthropic Claude for query generation.

packages/data-faker/ — Test data generator for performance testing. Generates realistic variant data matching the images_analytical schema with checkpoint/resume support. See data-faker README for details.

Getting Started

Clone and install dependencies:

pnpm install

Copy example environment variables

cp packages/moosestack-service/.env.{example,local}
cp packages/web-app/.env.{example,local}

Create API Key authentication tokens

$ cd packages/moosestack-service
moose generate hash-token # use output for the API Key & Token below

# if you get "command not found: moose" try this version instead:
pnpm exec moose generate hash-token  

Set environment variables

1. Set your API Key in packages/moosestack-service/.env.local to the ENV API KEY generated by moose generate hash-token
2. Set your API Token in packages/web-app/.env.local to the Bearer Token generated by moose generate hash-token
3. Set your Anthropic API key in packages/web-app/.env.local

Start both services:

pnpm dev

Or start services individually:

pnpm dev:moose    # Start MooseStack service only
pnpm dev:web      # Start web app only
pnpm dev:faker    # Run data faker (generates test data)

Data Faker

The data-faker package generates realistic test data for performance testing. It supports checkpoint/resume, graceful shutdown, and generates ~100GB of variant data.

Quick start:

# Quick test (100 rows)
pnpm dev:faker --rows 100

# Check current row count
pnpm dev:faker --count

# Full scale (100GB, ~200M rows)
pnpm dev:faker --rows 200000000

# Resume from checkpoint
pnpm dev:faker

See packages/data-faker/README.md for complete documentation.

Note: The data faker requires MooseStack service to be running (or at least ClickHouse accessible) to insert data. Run pnpm dev:moose first if ClickHouse isn't already running.

MCP Tools Available

• query_clickhouse - Execute read-only SQL queries against ClickHouse with automatic result limiting
• get_data_catalog - Discover available tables and views with their schema information

Learn More

• Data Migration Plan - Strategy for migrating from MySQL/MongoDB to ClickHouse
• Development Guide - Data models and ingestion API
• Security - Security features and production considerations
• MooseStack Documentation
• Model Context Protocol
• MCP SDK (@modelcontextprotocol/sdk)
• WebApp Class Reference

Built with MooseStack by Fiveonefour