AI Jam Sessions

What is this?

A piano and guitar that AI learns to play. Not a synthesizer, not a MIDI library — a teaching instrument.

An LLM can read and write text, but it can't experience music the way we do. No ears, no fingers, no muscle memory. AI Jam Sessions closes that gap by giving the model senses it can actually use:

• Reading — real MIDI sheet music with deep musical annotations. Not hand-written approximations — parsed, analyzed, and explained.
• Hearing — six audio engines (oscillator piano, sample piano, vocal samples, physical vocal tract, additive vocal synth, physically-modeled guitar) that play through your speakers, so the humans in the room become the AI's ears.
• Seeing — a piano roll that renders what was played as SVG the model can read back and verify. An interactive guitar tablature editor. A browser cockpit with a visual keyboard, dual-mode note editor, and tuning lab.
• Remembering — a practice journal that persists across sessions, so learning compounds over time.
• Singing — vocal tract synthesis with 20 voice presets, from operatic soprano to electronic choir. Sing-along mode with solfege, contour, and syllable narration.

Each of the 12 genres has a richly annotated exemplar — a reference piece the AI studies first, with historical context, bar-by-bar structural analysis, key moments, teaching goals, and performance tips. The other 96 songs are raw MIDI, waiting for the AI to absorb the patterns, play the music, and write its own annotations.

The Piano Roll

The piano roll is how the AI sees music. It renders any song as SVG — blue for right hand, coral for left, with beat grids, dynamics, and measure boundaries:

Two color modes: hand (blue/coral) or pitch-class (chromatic rainbow — every C is red, every F# is cyan). The SVG format means the model can both see the image and read the markup to verify pitch, rhythm, and hand independence.

The Cockpit

A browser-based instrument and vocal studio that opens alongside the MCP server. No plugins, no DAW — just a web page with a piano.

• Dual-mode piano roll — switch between Instrument mode (chromatic pitch-class colors) and Vocal mode (notes colored by vowel shape: /a/ /e/ /i/ /o/ /u/)
• Visual keyboard — two octaves from C4, mapped to your QWERTY keyboard. Click or type.
• 20 voice presets — 15 Kokoro-mapped voices (Aoede, Heart, Jessica, Sky, Eric, Fenrir, Liam, Onyx, Alice, Emma, Isabella, George, Lewis, plus choir and synth-vox), 4 tract-mapped voices, and a synthetic choir section
• 10 instrument presets — the 6 server-side piano voices plus synth-pad, organ, bell, and strings
• Note inspector — click any note to edit velocity, vowel, and breathiness
• 7 tuning systems — Equal temperament, just intonation (major/minor), Pythagorean, quarter-comma meantone, Werckmeister III, or custom cent offsets. Adjustable A4 reference (392–494 Hz).
• Tuning audit — frequency table, interval tester with beat-frequency analysis, and tuning export/import
• Score import/export — serialize the entire score as JSON and load it back
• LLM-facing API — window.__cockpit exposes exportScore(), importScore(), addNote(), play(), stop(), panic(), setMode(), and getScore() so an LLM can compose, arrange, and play back programmatically

The Learning Loop

 Read                 Play                See                 Reflect
┌──────────┐     ┌───────────┐     ┌────────────┐     ┌──────────────┐
│ Study the │     │ Play the  │     │ View the   │     │ Write what   │
│ exemplar  │ ──▶ │ song at   │ ──▶ │ piano roll │ ──▶ │ you learned  │
│ analysis  │     │ any speed │     │ to verify  │     │ in journal   │
└──────────┘     └───────────┘     └────────────┘     └──────┬───────┘
                                                             │
                                                             ▼
                                                    ┌──────────────┐
                                                    │ Next session  │
                                                    │ picks up here │
                                                    └──────────────┘

The Song Library

120 songs across 12 genres, built from real MIDI files. Each genre has one deeply annotated exemplar — with historical context, bar-by-bar harmonic analysis, key moments, teaching goals, and performance tips (including vocal guidance). These exemplars serve as templates: the AI studies one, then annotates the rest.

Songs progress from raw (MIDI only) → annotated → ready (fully playable with musical language). The AI promotes songs by studying them and writing annotations with annotate_song.

Sound Engines

Six engines, plus a layered combinator that runs any two simultaneously:

Keyboard Voices

Six tunable piano voices, each adjustable per-parameter (brightness, decay, hammer hardness, detune, stereo width, and more):

Guitar Voices

Four guitar voice presets with physically-modeled string synthesis, each with 17 tunable parameters (brightness, body resonance, pluck position, string damping, and more):

The Practice Journal

After every session, the server captures what happened — which song, what speed, how many measures, how long. The AI adds its own reflections: what it noticed, what patterns it recognized, what to try next.

---
### 14:32 — Autumn Leaves
**jazz** | intermediate | G minor | 69 BPM × 0.7 | 32/32 measures | 45s

The ii-V-I in bars 5-8 (Cm7-F7-BbMaj7) is the same gravity as the V-i
in The Thrill Is Gone, just in major. Blues and jazz share more than the
genre labels suggest.

Next: try at full speed. Compare the Ipanema bridge modulation with this.
---

One markdown file per day, stored in ~/.ai-jam-sessions/journal/. Human-readable, append-only. Next session, the AI reads its journal and picks up where it left off.

Install

npm install -g ai-jam-sessions

Requires Node.js 18+. No MIDI drivers, no virtual ports, no external software.

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ai_jam_sessions": {
      "command": "npx",
      "args": ["-y", "-p", "ai-jam-sessions", "ai-jam-sessions-mcp"]
    }
  }
}

MCP Tools

41 tools and 3 prompt templates across six categories:

Learn

Play

Sing

Guitar

Build

MCP Prompts

Three prompt templates for structured teaching workflows:

CLI

ai-jam-sessions list [--genre <genre>] [--difficulty <level>]
ai-jam-sessions play <song-id> [--speed <mult>] [--mode <mode>] [--engine <piano|vocal|tract|synth|guitar|piano+synth|guitar+synth>]
ai-jam-sessions sing <song-id> [--with-piano] [--engine <engine>]
ai-jam-sessions view <song-id> [--measures <start-end>] [--out <file.svg>]
ai-jam-sessions view-guitar <song-id> [--measures <start-end>] [--tuning <tuning>]
ai-jam-sessions info <song-id>
ai-jam-sessions tune <keyboard-id> [--param value ...] [--reset] [--show]
ai-jam-sessions tune-guitar <voice-id> [--param value ...] [--reset] [--show]
ai-jam-sessions keyboards
ai-jam-sessions guitars
ai-jam-sessions stats
ai-jam-sessions library
ai-jam-sessions ports
ai-jam-sessions help
ai-jam-sessions --version

Status

v1.4.0. Six sound engines, 41 MCP tools, 3 prompt templates, 120 songs across 12 genres with deeply annotated exemplars. Song transposition, section markers, per-hand mute/solo for focused practice. Interactive guitar tablature editor. Browser cockpit with 20 vocal presets, 10 instrument voices, 7 tuning systems, and an LLM-facing score API. Piano roll visualization in two color modes. Practice journal for persistent learning. Session state persistence across server restarts. MIDI play-along scoring, annotation quality assessment, and cross-genre pattern recognition. The MIDI is all there — the library grows as the AI learns.

Security & Privacy

Data touched: song library (JSON + MIDI), user songs directory (~/.ai-jam-sessions/songs/), guitar tuning configs, practice journal entries, local audio output device.

Data NOT touched: no cloud APIs, no user credentials, no browsing data, no system files outside the user song directory. No telemetry is collected or sent.

Permissions: MCP server uses stdio transport only (no HTTP). CLI accesses local filesystem and audio devices. See SECURITY.md for the full policy.

License

MIT