TokenOS

Local-first codebase graph intelligence for AI assistants — powered by SQLite, ts-morph, and optional Ollama.

TokenOS is a Model Context Protocol (MCP) server that statically analyses your TypeScript/TSX codebase, stores it as a structural dependency graph in SQLite, enriches nodes with BM25 full-text search via SQLite FTS5, optionally adds semantic embeddings via Ollama, and exposes high-precision query tools for AI coding assistants like Claude, Cursor, or any MCP-compatible client.

The goal: When you start a new chat, the AI already knows your codebase structure. No more "let me analyze all files first" — it queries the graph and gets exactly what it needs, saving tokens and compute.

Works fully offline. FTS5 full-text search is built into SQLite — no Ollama, no internet, no external services required. Ollama adds optional semantic (concept-level) search on top.

---

Table of Contents

• Quick Start
• Commands
• Database Location
• MCP Client Configuration
• MCP Tools
• Node Types
• Edge Types
• Importance Scoring
• Semantic Meta Enrichment
• Conversation Memory
• Visualization Dashboard
• Changing the Embedding Model
• Architecture
• Configuration Reference
• Prerequisites
• Tech Stack
• Limitations
• License

---

Quick Start

Run TokenOS on any codebase instantly via npx. No installation, setup, or config files required!

# In your project folder
npx -y tokenos .

# Or for a specific path
npx -y tokenos /absolute/path/to/project

That's it. The server will:

1. Parse all .ts/.tsx files via ts-morph AST analysis
2. Extract nodes (functions, classes, components, interfaces, types, enums, routes, variables, imports)
3. Extract edges (CALLS, IMPORTS, EXPORTS, EXTENDS, IMPLEMENTS, DEFINES, RENDERS, CONTAINS, TYPE_OF, PART_OF_TAB)
4. Store everything in a per-project SQLite database
5. Auto-generate summaries for all nodes
6. Back-fill semantic embeddings via Ollama (if running)
7. Compute importance scores for architectural ranking
8. Start a chokidar file watcher for real-time incremental updates
9. Serve 6 MCP tools over stdio transport
10. Optionally launch an interactive graph visualization dashboard

---

Commands

Command	Description
npx -y tokenos .	Index the current folder and start the MCP server
npx -y tokenos /path	Index a specific folder and start the server
npx tokenos --version	Print the installed version
npm run dev	(Cloned repo only) Start in local development mode
npm run reset	(Cloned repo only) Wipe the development database

---

Database Location

Each project gets its own isolated database at:

<your-project>/.tokenos/graph.db

This means:

• ✅ Different projects never mix data
• ✅ You can delete .tokenos/ to reset a specific project
• ✅ Add .tokenos/ to your project's .gitignore

SQLite is configured with WAL mode for better concurrent read performance and foreign key enforcement.

Schema

Three tables and one virtual table are created automatically:

Table	Purpose
nodes	All code entities (functions, classes, components, etc.) with metadata, summaries, embeddings, and importance scores
edges	All relationships between nodes (CALLS, IMPORTS, RENDERS, etc.) with unique constraint on (from_node, to_node, type)
memories	Conversation memory storage for persistent context across sessions
nodes_fts	FTS5 virtual table — BM25 full-text index over name, summary, and meta fields. Kept in sync with nodes inside every index transaction.

---

MCP Client Configuration

You can add multiple entries — one for each project you want AI access to. They do not conflict, as each creates an isolated database inside its respective project folder.

Antigravity IDE

Open ~/.gemini/antigravity/mcp_config.json and add your projects under mcpServers:

{
  "mcpServers": {
    "tokenos-myapp": {
      "command": "npx",
      "args": ["-y", "tokenos", "/absolute/path/to/myapp"],
      "env": {}
    }
  }
}

Claude Code

Open ~/.claude.json and add underneath "mcpServers":

"tokenos-myapp": {
  "type": "stdio",
  "command": "npx",
  "args": ["-y", "tokenos", "/absolute/path/to/myapp"],
  "env": {}
}

Claude Desktop

Open ~/Library/Application Support/Claude/claude_desktop_config.json and add:

{
  "mcpServers": {
    "tokenos-myapp": {
      "command": "npx",
      "args": ["-y", "tokenos", "/absolute/path/to/myapp"]
    }
  }
}

---

MCP Tools

All tools are read-only, idempotent, and communicate over stdio transport. Responses are truncated at 25,000 characters to prevent overwhelming the LLM context window.

search

Smart search that understands intent and returns the most relevant code and context.

Args:
  query           (string)  — Natural language question or search term
  response_format (optional) — 'json' (default) or 'markdown'

How it works:

1. Intent detection — Classifies query into one of four modes:
   • semantic (default) — general concept/code search
   • trace — triggered by "trace", "flow", "how", "why" → deeper BFS traversal (depth 2)
   • explore — triggered by "where", "what", "find" → broad shallow search
   • dependency — triggered by "depend", "import", "export" → import/export edges only
2. Hybrid search — Tries FTS5 BM25 first, then Ollama semantic similarity if available, then multi-term text fallback
3. Graph expansion — BFS-expands top results into a contextualized subgraph
4. Memory retrieval — Appends relevant conversation memories (top 3)

Response includes semantic_available: true/false so you always know which search layer was used.

Without Ollama: FTS5 handles all queries automatically. Searches like "attribute listing display" correctly return attribute-related nodes via BM25 term matching — no stale results.

Returns: Compressed, relevant context (code + relationships + memory)

find_nodes

Find code elements by name, type, or meaning.

Args:
  query           (string)  — Function name, class name, or natural-language description
  type            (optional) — 'function' | 'class' | 'file' | 'import' | 'variable' | 'component' | 'interface' | 'type_alias' | 'enum' | 'route'
  mode            (optional) — 'text' (default) or 'semantic' (requires Ollama)
  limit           (optional) — 1–50, default 10
  offset          (optional) — for pagination
  response_format (optional) — 'json' (default) or 'markdown'

Text mode: Uses FTS5 BM25 ranking for multi-word queries, falling back to LIKE matching for single-word exact searches. Matches across name, summary, and meta fields. When type is provided, results are filtered with AND logic.

Semantic mode: Uses Ollama embeddings for concept-level search. Example: searching "authentication handler" finds loginUser() even if "auth" isn't in the name.

Without Ollama (semantic mode fallback): Automatically falls back to FTS5 → multi-term → single-term chain — no error, no empty results.

Response includes semantic_available: true/false in semantic mode so you know if Ollama was actually used.

Returns: List of matching nodes with relevance ranking

get_node

Get full details of a specific code element.

Args:
  id              (string)  — format: 'filePath::name' (e.g. 'src/utils/cache.ts::LRUCache')
  response_format (optional) — 'json' or 'markdown'

Returns: Complete node data (code, type, file, importance)

get_connections

Get directly related code elements.

Args:
  id              (string)  — Node ID
  include_reverse (optional) — true/false, default false. Include nodes that USE this node.
  limit           (optional) — 1–100, default 50
  response_format (optional) — 'json' or 'markdown'

Returns: Connected nodes and their relationships. When include_reverse is true, also includes used_by array.

explore

Explore surrounding code context from a starting point.

Args:
  id              (string)  — Starting node ID
  depth           (optional) — 1–3, default 2
  include_imports (optional) — true/false, default true. Set false to hide import nodes.

Returns: Local graph (nodes + relationships)

top_nodes

Get the most important parts of the codebase.

Args:
  limit           (optional) — 1–100, default 20
  response_format (optional) — 'json' or 'markdown'

Returns: Ranked list of high-impact nodes

status

Get TokenOS system status, statistics, and available capabilities.

Args: (none)

Returns: System stats (node counts, edge counts, FTS index size) and capability flags (text_search, fts5, semantic_search, graph_traversal, reverse_edges).

---

Node Types

The parser extracts 10 distinct node types from TypeScript/TSX source files:

Type	Description	Detection Method
function	Named functions and arrow functions	FunctionDeclaration and VariableDeclaration → ArrowFunction
component	React/JSX components	PascalCase function that contains JSX elements
class	ES6 class declarations	ClassDeclaration
interface	TypeScript interface declarations	InterfaceDeclaration
type_alias	TypeScript type aliases	TypeAliasDeclaration
enum	TypeScript enum declarations	EnumDeclaration
variable	Exported constants and variables	Exported VariableStatement (excluding arrow functions already captured)
import	Import declarations	ImportDeclaration → keyed as import:<module_specifier>
file	Source file entry	One per file, named after the basename
route	Next.js App Router route	Detected from app/**/page.tsx file paths

---

Edge Types

The parser extracts 10 distinct edge types representing relationships between nodes:

Edge	Description
CALLS	Function A calls function B (via CallExpression)
IMPORTS	File imports a module
EXPORTS	File exports a symbol
EXTENDS	Class or interface extends a base
IMPLEMENTS	Class implements an interface
DEFINES	File defines a symbol (function, class, variable, etc.)
RENDERS	React component renders another component (JSX usage)
CONTAINS	Wrapper component contains a child in JSX tree (direct parent-child)
TYPE_OF	Symbol references a type or interface
PART_OF_TAB	Component belongs to a tab (detected from TabsContent, TabContent, TabPanel wrappers)

---

Importance Scoring

Every node is scored using a batch SQL computation:

Score = (inDegree × 2) + outDegree + typeWeight

Type	Weight
class	3.0
component	2.5
function	2.0
route	1.5
interface	1.5
enum	1.5
file	1.5
type_alias	1.0
variable	1.0
import	0.5

Importance is computed in a single aggregation query (no N+1), then batch-updated in a transaction.

---

Semantic Meta Enrichment

The parser automatically infers rich metadata for each node:

UI Role Detection

Components are assigned a UI role based on name/file pattern matching:

Pattern	Role
*panel*	panel
*tab*	tab
*page*	page
*dialog*, *modal*	dialog
*form*	form
*sidebar*, *nav*	navigation
*header*	header
*footer*	footer
*content*	content
*list*	list
*card*	card
*button*	action
*layout*	layout

Feature Inference

Features are extracted from directory structure:

• Next.js App Router: app/(group)/feature-name/... → feature: "feature-name"
• Component directories: components/feature-name/... → feature: "feature-name"

Route Detection

Next.js App Router routes are auto-detected from app/**/page.tsx paths and stored as route nodes.

Tab System Detection

When the parser encounters <TabsContent value="xxx"> (or TabContent, TabPanel), it:

1. Creates PART_OF_TAB edges from children to the parent component
2. Sets meta.tab = "xxx" on the child nodes

Enriched Meta Search

All meta fields (role, tab, feature) are queryable via the searchNodesExtended function, which combines name LIKE matching with JSON meta extraction in a single SQL query.

---

Conversation Memory

TokenOS includes a persistent memory system for storing conversation context across sessions:

• Storage: SQLite memories table with title, summary, key_points (JSON array), tags (JSON array), and optional embeddings
• Auto-indexing: Markdown files in /memory/ or /memories/ directories within the watched project are automatically parsed and stored
• Extraction: Titles from # headings, tags from tags: [...] patterns, key points from bullet lists
• Search: Text-based search across title, summary, and tags
• Integration: Memories are automatically surfaced in search results

---

Visualization Dashboard

Enable the built-in visualization UI by setting ui.enabled: true in your config or passing the --ui CLI flag.

Routes

Route	Description
/	Dashboard — Glassmorphism-styled overview with stats cards, top nodes grid, and full context explorer table. Animated with GSAP.
/graph	Network Graph — Interactive force-directed graph visualization using vis-network. Click nodes for details.
/api/stats	JSON API: node counts by type + top 50 nodes
/api/graph-data	JSON API: full graph data (auto-limited to top 1500 nodes for browser performance)

---

Changing the Embedding Model

Edit tokenos.config.json:

{
  "ollama": {
    "model": "mxbai-embed-large:latest"
  }
}

Then reset and re-index (different models produce incompatible vectors):

npm run reset
npm run dev

Popular models:

• mxbai-embed-large:latest — high quality, larger context
• nomic-embed-text — fast, good general purpose (default fallback)
• all-minilm — lightweight, fast

Embedding Pipeline Details

• Input enrichment: Embeddings are generated from a structured prompt that includes [NAME], [TYPE], [ROLE], [TAB], [FEATURE], [ROUTE], [SUMMARY], and [CODE] (first 300 chars) tags. This produces richer vectors than embedding raw code alone.
• LRU caching: An in-memory LRU cache (1000 entries, 30min TTL) prevents redundant Ollama calls for unchanged text.
• Backfill strategy: On boot, only nodes without existing embeddings are processed. If Ollama becomes unavailable mid-backfill, the process stops gracefully.
• Health checks: Ollama availability is probed once via /api/tags with a 2-second timeout. Subsequent failures reset the health flag for retry on next request.

---

Architecture

src/
├── main.ts                 # Entry point — validates config, bootstraps all systems, graceful shutdown
├── config.ts               # Config loader (CLI args → config file → env vars → defaults)
├── reset.ts                # Reset script — deletes project DB + WAL/SHM files
├── verify-parser.ts        # Parser verification — validates node types, edge types, and meta
├── types.ts                # Shared TypeScript types (10 NodeTypes, 10 EdgeTypes, ConversationMemory)
│
├── db/
│   ├── connection.ts       # SQLite connection + schema (WAL mode, foreign keys, 3 tables, 7 indexes)
│   ├── queries.ts          # 20+ prepared statements (upsert, search, batch importance, meta queries)
│   ├── memory.ts           # Conversation memory CRUD (upsert, search, get all)
│   └── index.ts            # Re-exports
│
├── indexer/
│   ├── parser.ts           # ts-morph AST parser — extracts 10 node types, 10 edge types, semantic meta
│   ├── indexer.ts          # Orchestrates parse → hash-skip → transaction (delete stale + upsert fresh)
│   ├── watcher.ts          # chokidar file watcher (incremental: add/change/unlink, ignoreInitial)
│   ├── ignore.ts           # .gitignore rule loader + hardcoded ignores (node_modules, .git, dist, etc.)
│   └── cli.ts              # One-shot CLI indexer (npm run index)
│
├── embeddings/
│   ├── client.ts           # Ollama HTTP client + enriched embedding input builder + LRU cache
│   ├── similarity.ts       # Cosine similarity + ranked search (top-K)
│   └── index.ts            # backfillEmbeddings() + re-exports
│
├── server/
│   ├── server.ts           # MCP server — registers 6 tools, BFS subgraph builder, node/edge compression
│   ├── visualize.ts        # Optional visualization dashboard (HTTP server, vis-network + GSAP)
│   └── index.ts            # Re-exports
│
└── utils/
    ├── scoring.ts           # Importance score computation (delegates to batch SQL)
    ├── logger.ts            # Vite-inspired colored logger (picocolors, per-module tags)
    └── cache.ts             # Generic LRU cache with TTL (used by embedding client)

Boot Sequence

1. Validate config (watchPath exists)
2. Optionally start visualization UI server
3. Full directory indexing (recursive walk, .gitignore-aware)
4. Probe Ollama → backfill embeddings for un-embedded nodes
5. Batch-compute importance scores (single SQL aggregation)
6. Start chokidar watcher (incremental updates only)
7. Start MCP stdio server (blocks until client disconnects)
8. Print Vite-like status banner with health checks
9. Register SIGINT/SIGTERM handlers for graceful shutdown

Incremental Update Strategy

• Hash-based skip: Each file's content is SHA-256 hashed (truncated to 16 chars). Files with unchanged hashes are skipped entirely.
• Transactional upsert: On file change, a single SQLite transaction deletes all stale nodes/edges for the file then inserts fresh data, ensuring FK consistency.
• Graceful FK handling: Edges referencing nodes in un-indexed files are silently skipped (they'll be wired up when the target file is indexed).

---

Configuration Reference

tokenos.config.json

Field	Type	Default	Description
watchPath	string	process.cwd()	Absolute path to the project you want to index
ollama.url	string	http://localhost:11434	Ollama server URL
ollama.model	string	nomic-embed-text	Embedding model to use
ui.enabled	boolean	false	Start visualization dashboard on boot
ui.port	number	3333	Dashboard HTTP server port

Environment Variable Overrides

Variable	Overrides
OLLAMA_URL	ollama.url
EMBEDDING_MODEL	ollama.model
GRAPH_UI_PORT	ui.port

CLI Arguments

Argument	Description
First non-flag arg	Overrides watchPath
--ui	Enables visualization dashboard (overrides config)

Precedence: CLI args → config file → environment variables → defaults

---

Prerequisites

Dependency	Purpose
Node.js ≥ 18	Runtime
npm	Package manager
Ollama (optional)	Semantic embedding generation

If Ollama is not running, the server starts normally — semantic search falls back to text-mode and embeddings are skipped.

---

Tech Stack

Core Technologies:

• TypeScript / Node.js — Core language and runtime (ES2022 target, Node16 module resolution)
• SQLite — Local, fast, embedded graph database (WAL mode)
• ts-morph — TypeScript AST parsing tool for static analysis
• Model Context Protocol (MCP) — Standardized AI tool integration protocol
• Ollama — Local semantic vector embeddings (optional)

Dependencies

Package	Version	Role
@modelcontextprotocol/sdk	^1.8.0	MCP server + stdio transport
better-sqlite3	^11.9.1	Synchronous SQLite (Node.js)
ts-morph	^25.0.1	TypeScript AST parsing
chokidar	^4.0.3	File watching
ignore	^7.0.5	.gitignore-pattern matching
zod	^4.3.6	Schema validation for MCP tool inputs
picocolors	^1.1.1	Terminal colors

Dev Dependencies

Package	Version	Role
@types/better-sqlite3	^7.6.12	SQLite type definitions
@types/node	^22.13.13	Node.js type definitions
tsx	^4.19.3	TypeScript dev runner
typescript	^5.8.2	TypeScript compiler

---

Limitations

• Only .ts and .tsx files are indexed (no .js, .jsx, .vue, etc.)
• The graph database is rebuilt on first run per project (or when you run tokenos reset)
• Subgraph BFS and search responses are truncated at 25,000 characters
• Cross-file edges to un-indexed targets are gracefully skipped (resolved when the dependency is indexed later)
• Memory file extraction uses basic regex parsing (headings, bullet points, tag patterns)
• Visualization dashboard loads at most 1,500 nodes to prevent browser rendering issues
• Semantic (concept-level) search requires Ollama — all other search modes (FTS5, text) work fully offline

---

License

MIT