LizardBrain

Turn team conversations into structured knowledge.

Your team's conversations have years of knowledge buried in thousands of messages -- chats, meetings, support threads. Who knows what, what was decided, what tasks were assigned, what questions were answered -- it's all there, but impossible to find.

LizardBrain reads your chat messages, extracts the important stuff using any LLM, and stores it in a searchable database. Run it on a cron, and your AI agent always knows what your group has been talking about.

What problems does it solve?

"Who on our team knows about Kubernetes?" -- Instead of asking around, search your memory database. LizardBrain tracks what each member talks about and builds expertise profiles automatically.

"What did we decide about the database migration?" -- Decisions made in chat are captured with context and participants. No more scrolling through months of messages.

"What's the status of the API rewrite?" -- Tasks, assignments, and deadlines mentioned in chat are extracted and searchable.

"Summarize what our community knows about RAG pipelines" -- Facts, opinions, and resources shared by members are stored with confidence scores and attribution.

How it works

Point LizardBrain at your chat messages (SQLite, JSONL, or any custom source). It sends batches to a cheap LLM, which extracts structured knowledge. Everything goes into SQLite with full-text search, and optionally vector search for semantic matching.

Messages  -->  LLM extracts knowledge  -->  SQLite + search indexes

You choose a profile that matches your group type, and LizardBrain adjusts what it looks for:

Quick start

git clone https://github.com/pandore/lizardbrain && cd lizardbrain

cp examples/lizardbrain.json lizardbrain.json
# Edit lizardbrain.json -- set your chat DB path and LLM provider

node src/cli.js init              # Asks which profile fits your group
LIZARDBRAIN_LLM_API_KEY=your-key node src/cli.js extract

# Now query your group's knowledge
node src/cli.js search "RAG pipeline"
node src/cli.js who "python"
node src/cli.js stats

Set it up on a cron and forget about it:

0 */2 * * * cd /path/to/project && LIZARDBRAIN_LLM_API_KEY=key node src/cli.js extract >> /tmp/lizardbrain.log 2>&1

What gets extracted

Depending on your profile, LizardBrain pulls out up to 7 types of structured knowledge:

Everything is deduplicated automatically -- first via FTS keyword matching, then via embedding similarity (kNN). Contradictions are detected and old facts superseded. Expired facts drop out of search. Even rephrased duplicates are caught.

Entity updates

Decisions, tasks, and questions evolve over time. A decision starts as "proposed" and becomes "agreed." A task goes from "open" to "done." A question gets answered.

LizardBrain handles this automatically when context injection is enabled. The LLM sees existing open entities and can update their status instead of creating duplicates:

Run 1: Decision extracted — "Use PostgreSQL" (proposed)
Run 2: LLM sees the decision in context, messages confirm it → status updated to "agreed"

No manual intervention needed. The LLM references entity IDs from context and outputs updates alongside new extractions.

Why LizardBrain?

Use any LLM. OpenAI, Anthropic, Gemini, Groq, Ollama, Mistral -- uses the Vercel AI SDK with Zod-validated structured output, so you get typed extraction results or a clear error, never silently broken JSON. Any OpenAI-compatible endpoint works, plus native Anthropic support.

Zero dependencies to start. The core tier uses Node.js and the sqlite3 CLI that's already on your machine. No npm install needed.

Scales up when you want. Add better-sqlite3 + sqlite-vec for hybrid vector search that combines keyword matching with semantic similarity via Reciprocal Rank Fusion.

Runs incrementally. Tracks where it left off. Each run only processes new messages, so it's cheap and fast even on large chats.

Context-aware. The LLM sees existing knowledge from previous runs. Decisions get confirmed, tasks get closed, questions get answered -- entities evolve naturally across extraction runs.

Contradiction-safe. When new information contradicts existing knowledge ("Team uses Postgres" → "Team migrated to MySQL"), the old fact is automatically superseded. Your agent never sees outdated info.

Temporally aware. Sprint goals expire after 30 days, standup notes after 7 — each fact gets a durability tier and drops out of search when it's no longer valid.

Cross-referenced. Entities link to each other — a task implements a decision, a fact supports another fact. The LLM creates typed relationships during extraction, and agents can query the graph.

Links get context. When someone shares a GitHub repo or web page, LizardBrain fetches metadata (stars, descriptions, titles) before sending to the LLM, so extracted facts are richer.

Built for agents. Generate a compact member roster (~30 tokens per person) designed to fit in an agent's system prompt. At 100 members, that's ~3,000 tokens.

MCP server built in. Run lizardbrain serve and any MCP-compatible client (Claude Desktop, Cursor, custom agents) can read, search, and write knowledge directly. 9 tools including entity link management. No shell-outs, no parsing CLI output — agents talk to LizardBrain over the Model Context Protocol.

v1.0.4 fixes

• Search relevance: natural-language queries like "what is SciGPT?" now actually find SciGPT instead of drowning in OR-soup matches. sanitizeFtsQuery strips an English stopword list (articles, auxiliaries, question words, pronouns) and joins remaining terms with whitespace (FTS5 implicit AND), so all substantive terms must appear. Empty queries (or queries that collapse to all stopwords) short-circuit to [] instead of triggering an FTS5 syntax error

v1.0.3 fixes

• FTS5 query crash: search queries containing backticks, +, ~, @, #, or other special characters caused fts5: syntax error crashes. Replaced character blocklist with Unicode-aware allowlist — only letters, numbers, and whitespace pass through to FTS5 MATCH

v1.0.2 fixes

• sqlite-vec BigInt compatibility: cast row.id to BigInt before inserting into vec0 virtual tables — fixes compatibility with sqlite-vec v0.1.7+ which requires INTEGER (not REAL) primary keys

v1.0.1 fixes

• Silent migration: migrate() now creates lizardbrain_meta if missing (fixes "no such table" errors on pre-v0.4 databases) and checks column existence via PRAGMA table_info before ALTER TABLE (eliminates 26 "duplicate column" warnings per invocation)
• --quiet flag: lizardbrain stats --quiet suppresses all non-error output — useful for cron jobs and plugin integrations
• OpenClaw plugin manifest: replaced non-standard config keys with OpenClaw-compatible ones; custom settings now use env vars (LIZARDBRAIN_DB_PATH, LIZARDBRAIN_ENABLE_DMS)

v1.0 features

{
  "contradiction": { "enabled": true }
}

"Sprint 14 goals are X, Y, Z"  →  durability: short (30 days)
"PostgreSQL supports JSONB"     →  durability: durable (permanent)

# Agent can query the relationship graph
get_links({ entity_type: "decision", entity_id: 5 })
→ task #12 implements decision #5
→ fact #47 supports decision #5

v0.10 features

node src/cli.js dedup --dry-run          # preview what would be removed
node src/cli.js dedup                     # remove semantic duplicates
node src/cli.js dedup --threshold 0.20    # adjust similarity threshold

{
  "dedup": { "semantic": true }
}

v0.7 features

v0.6 features (included in v0.7)

LIZARDBRAIN_SOURCE_AGENT=discord-bot node src/cli.js extract

Configuration

Create lizardbrain.json in your working directory:

{
  "memoryDbPath": "./lizardbrain.db",
  "profile": "knowledge",

  "llm": {
    "baseUrl": "https://api.openai.com/v1",
    "model": "gpt-5-nano"
  },

  "source": {
    "type": "sqlite",
    "path": "./chat.db",
    "table": "messages",
    "columns": {
      "id": "id",
      "content": "content",
      "sender": "sender_name",
      "timestamp": "created_at"
    }
  }
}

API key via .env file, environment variable (LIZARDBRAIN_LLM_API_KEY), or directly in config.

{
  "batchOverlap": 5
}

{
  "context": {
    "enabled": true,
    "tokenBudget": 1000,
    "recencyDays": 30,
    "maxItems": {
      "decisions": 5,
      "tasks": 10,
      "questions": 5,
      "facts": 5,
      "topics": 3
    }
  }
}

LLM providers

OpenAI-compatible or native Anthropic Messages API:

{
  "embedding": {
    "enabled": true,
    "baseUrl": "https://api.openai.com/v1",
    "model": "text-embedding-3-small"
  }
}

npm install better-sqlite3 sqlite-vec
LIZARDBRAIN_EMBEDDING_API_KEY=your-key node src/cli.js embed --backfill

Chat sources

SQLite (default)

Point at any SQLite database with a messages table. Works with Telegram exports, custom bots, or any app that stores messages in SQLite.

{
  "source": {
    "type": "sqlite",
    "path": "./chat.db",
    "table": "messages",
    "columns": { "id": "message_id", "content": "text", "sender": "author", "timestamp": "created_at" },
    "filter": "channel = 'general'"
  }
}

{
  "source": {
    "type": "sqlite",
    "path": "./chat.db",
    "conversationFilter": {
      "column": "conversation_id",
      "detectGroup": { "contentColumn": "content", "marker": "is_group_chat" }
    }
  }
}

JSONL

{
  "source": {
    "type": "jsonl",
    "path": "./messages.jsonl",
    "fields": { "id": "id", "content": "text", "sender": "from", "timestamp": "date" }
  }
}

stdin

Pipe messages directly from any source:

cat messages.jsonl | node src/cli.js extract --source stdin
curl https://api.example.com/messages | node src/cli.js extract --source stdin

Custom adapter

Write your own in ~10 lines:

// my-adapter.js
module.exports = {
  name: 'my-source',
  validate() { return { ok: true }; },
  getMessages(afterId) {
    return [{ id: '1', content: 'hello', sender: 'alice', timestamp: '2026-01-01' }];
  }
};

{ "source": { "type": "custom", "adapterPath": "./my-adapter.js" } }

Profiles in depth

Profiles adapt what the LLM looks for based on the type of group:

• Knowledge profile asks the LLM to extract skills, tools, and project involvement. Roster shows "expertise" and "builds".
• Team profile asks for roles, responsibilities, and current work. Roster shows "role" and "works on". Also captures decisions and tasks.
• Project profile asks for company, role, and scope of work. Roster shows "role" and "scope". Captures decisions, tasks, and questions -- but not topics (project chats tend to be focused, not multi-topic).

The same database columns are reused across profiles -- only the LLM prompt changes. You can switch profiles without migrating data.

{
  "profile": "team",
  "entities": ["members", "facts", "decisions", "tasks"]
}

Programmatic API

const lizardbrain = require('lizardbrain');

// Initialize with a profile
lizardbrain.init('./memory.db', { profile: 'team' });

// Connect to your chat source
const adapter = lizardbrain.adapters.sqlite.create({
  path: './chat.db',
  table: 'messages',
  columns: { id: 'id', content: 'text', sender: 'author', timestamp: 'created_at' },
});
const driver = lizardbrain.createDriver('./memory.db');

// Extract knowledge
await lizardbrain.extract(adapter, driver, {
  llm: { baseUrl: 'https://api.openai.com/v1', apiKey: process.env.OPENAI_API_KEY, model: 'gpt-5-nano' },
});

// Search (hybrid FTS5 + vector, or FTS5-only fallback)
const { mode, results } = await lizardbrain.search(driver, 'kubernetes', { limit: 5 });

// Query helpers
const facts = lizardbrain.query.searchFacts(driver, 'kubernetes');
const experts = lizardbrain.query.whoKnows(driver, 'python');
const decisions = lizardbrain.query.searchDecisions(driver, 'database');
const tasks = lizardbrain.query.searchTasks(driver, 'migration');

// Generate a compact roster for agent context windows
const roster = lizardbrain.query.generateRoster(driver);
// roster.content is ~30 tokens per member, ready for a system prompt

// Assemble context for an agent (used by MCP server internally)
const context = lizardbrain.context.assembleContext(driver, {
  participants: ['Alice', 'Bob'],
  topics: ['kubernetes'],
  recencyDays: 30,
  tokenBudget: 2000,
});

// Start an MCP server programmatically
const server = lizardbrain.createServer({ driver, config });
await server.listen(); // connects stdio transport

driver.close();

CLI reference

lizardbrain init [--force] [--profile <name>]              Create memory database
lizardbrain extract [--dry-run] [--reprocess]              Run extraction pipeline
    [--limit N] [--from <id>]
lizardbrain embed [--stats] [--rebuild]                    Manage vector embeddings
lizardbrain stats                                          Show database statistics
lizardbrain health [--json]                                Check system health
lizardbrain search <query> [--json] [--fts-only]              Search knowledge
    [--limit N] [--conversation <id>]
lizardbrain who <keyword>                                  Find members by expertise
lizardbrain roster [--output path]                         Generate member roster
lizardbrain reset-cursor [--to <id>]                       Reset extraction cursor
lizardbrain dedup [--dry-run] [--threshold N]                 Remove semantic duplicates
lizardbrain prune-embeddings [--orphaned] [--stale] [--model <name>]  Clean up embeddings
lizardbrain serve                                         Start MCP server (stdio)

MCP Server

Run LizardBrain as an MCP server so any compatible agent can read, search, and write knowledge directly:

node src/cli.js serve

Requires better-sqlite3 and @modelcontextprotocol/sdk (installed via npm install). The server communicates over stdio using JSON-RPC.

Tools

The MCP server exposes 9 tools:

Client configuration

{
  "mcpServers": {
    "lizardbrain": {
      "command": "node",
      "args": ["/path/to/lizardbrain/src/cli.js", "serve"],
      "env": {
        "LIZARDBRAIN_LLM_API_KEY": "your-key"
      }
    }
  }
}

{
  "mcpServers": {
    "lizardbrain": {
      "command": "node",
      "args": ["/path/to/lizardbrain/src/cli.js", "serve"],
      "env": {
        "LIZARDBRAIN_LLM_API_KEY": "your-key"
      }
    }
  }
}

{
  "mcpServers": {
    "lizardbrain": {
      "command": "node",
      "args": ["/path/to/lizardbrain/src/cli.js", "serve"],
      "env": {
        "LIZARDBRAIN_LLM_API_KEY": "your-key"
      }
    }
  }
}

Example usage

Once connected, an agent can:

# Load context about specific people and topics
get_context({ participants: ["Alice", "Bob"], topics: ["kubernetes"], tokenBudget: 2000 })

# Search for specific knowledge
search({ query: "database migration", types: ["decisions", "tasks"], limit: 5 })

# Write structured knowledge directly
add_knowledge({ facts: [{ category: "technique", content: "Use pg_dump for migrations", confidence: 0.9 }] })

# Ingest raw text through LLM extraction
ingest({ text: "Alice said we should switch to PostgreSQL. Bob agreed.", profile: "team" })

# Update entity status
update_entity({ type: "task", id: 42, status: "done" })

# Link entities together
add_link({ source_type: "task", source_id: 12, target_type: "decision", target_id: 5, link_type: "implements" })

# Query entity relationships
get_links({ entity_type: "decision", entity_id: 5 })

Notes

• The ingest tool requires LLM configuration (same as extract). All other tools work without an LLM.
• All logging goes to stderr — stdout is reserved for JSON-RPC.
• The server uses better-sqlite3 exclusively (no CLI driver fallback) to prevent SQL injection.
• sqlite-vec remains optional — hybrid search when available, FTS-only fallback otherwise.

Integration Patterns

Roster as system prompt injection

The roster command generates compact member listings (~30 tokens per person) suitable for injection into agent system prompts. Recommended two-layer approach:

1. Static layer (refreshed daily via cron): inject lizardbrain roster output as a system prompt section -- gives agents awareness of team members and their expertise
2. Dynamic layer (per-turn): inject lizardbrain search results for contextual facts relevant to the current conversation

See examples/openclaw-plugin.ts for a full working example.

CLI driver limitations

The CLI driver (used when better-sqlite3 is not installed) passes SQL via stdin to the sqlite3 binary. It uses string escaping rather than parameterized queries. Do not use the CLI driver with untrusted input. Install better-sqlite3 for production deployments with user-facing data.

Log rotation

LizardBrain uses console.log and does not manage log files directly. When running extraction via cron, redirect stdout to a file and use external log rotation:

# Cron entry
0 3 * * * cd /path/to/project && node src/cli.js extract >> /var/log/lizardbrain-extract.log 2>&1

# /etc/logrotate.d/lizardbrain
/var/log/lizardbrain-extract.log {
  weekly
  rotate 4
  compress
  missingok
  notifempty
}

Tuning Guide

LizardBrain ships with defaults tuned for small groups (<50 active members, <5K messages). Below are all parameters you should review when setting up for your specific use case. Ask these questions before configuring:

1. How many active members? Affects knownMembersLimit and roster token budget.
2. How many messages per day? Affects batchSize, minMessages, and cron frequency.
3. What kind of group? Determines the profile — community vs. team vs. project.
4. Do decisions/tasks need tracking? Determines whether to enable context injection.
5. Is cost a concern? Affects LLM model choice, batchSize, and embedding config.

Extraction pipeline

These control how messages are batched and sent to the LLM.

Context injection

Controls what existing knowledge the LLM sees when extracting new batches. Without context, decisions/tasks can never be updated — only created.

Known members prompt

Controls how many existing members are shown to the LLM to prevent redundant re-extraction.

LLM settings

These are hardcoded in src/llm.js but may need adjustment for specific models.

Embedding & search

URL enrichment

Recommended configurations

{
  "profile": "team",
  "batchSize": 40,
  "batchOverlap": 5,
  "context": {
    "enabled": true,
    "tokenBudget": 1000,
    "recencyDays": 60
  }
}

{
  "profile": "knowledge",
  "batchSize": 60,
  "minMessages": 10,
  "batchOverlap": 0,
  "context": {
    "enabled": false
  }
}

{
  "profile": "project",
  "batchSize": 40,
  "batchOverlap": 5,
  "context": {
    "enabled": true,
    "tokenBudget": 1500,
    "recencyDays": 90,
    "maxItems": {
      "decisions": 15,
      "tasks": 25,
      "questions": 10,
      "facts": 5
    }
  }
}

Requirements

License

MIT