Long Tail

Long Tail turns your PostgreSQL database into a workflow engine where human and AI work share the same durable execution path. Hand off, pull back, escalate, automate — without rewriting systems or losing continuity.

npm install @hotmeshio/long-tail

Use Long Tail for

• Durable execution — Your functions run as workflows and checkpoint to Postgres. If the process crashes, execution resumes from the last completed step.
• Identity everywhere — Workflows know who started them, whose credentials govern their execution, and what permissions are in play. IAM is not bolted on — it's woven into every activity call.
• Human-in-the-loop — When confidence is low, the workflow escalates. RBAC-scoped escalation chains route work to the right reviewer. Approval workflows, content review, document verification — the pattern is the same.
• AI triage — When human-in-the-loop teams can't resolve a request, AI takes over. Its tool calls are checkpointed. And when the fix works, it compiles into a deterministic pipeline for next time.
• MCP tool orchestration — Describe what you need. If you've registered the tools, the Pipeline Designer builds the workflow. Every compiled pipeline deploys as a reusable MCP tool.

A dashboard, REST API, and live event stream ship with the package. Use what you need.

Start

Point at Postgres. Everything else is optional.

import { start } from '@hotmeshio/long-tail';
import * as myWorkflow from './workflows/my-workflow';

const lt = await start({
  database: { host: 'localhost', port: 5432, user: 'postgres', password: 'password', database: 'mydb' },
  workers: [{ taskQueue: 'default', workflow: myWorkflow.reviewContent }],
  auth: { secret: process.env.JWT_SECRET },
});

Dashboard at http://localhost:3000. The boilerplate has a working project with custom MCP servers, MinIO, and example workflows.

Write a Durable Workflow

A workflow receives an envelope and returns a result. Each activity call checkpoints — no work is lost, no step runs twice.

import { Durable } from '@hotmeshio/hotmesh';
import type { LTEnvelope } from '@hotmeshio/long-tail';
import * as activities from './activities';

const { analyzeContent } = Durable.workflow.proxyActivities<typeof activities>({ activities });

export async function reviewContent(envelope: LTEnvelope) {
  const analysis = await analyzeContent(envelope.data.content);

  if (analysis.confidence >= 0.85) {
    return { type: 'return' as const, data: { approved: true, analysis } };
  }

  return {
    type: 'escalation' as const,
    role: 'reviewer',
    message: `Review needed (confidence: ${analysis.confidence})`,
    data: { content: envelope.data.content, analysis },
  };
}

Activities are plain functions with side effects — API calls, LLM invocations, database queries. The proxyActivities call wraps them so the engine can checkpoint each result.

// activities.ts
export async function analyzeContent(content: string) {
  const result = await llm.classify(content);
  return { confidence: result.confidence, flags: result.flags };
}

Certify a Workflow

Any durable workflow can be promoted to certified through the dashboard or API. A certified workflow gains interceptor guarantees: failures escalate instead of throwing, escalation chains route through roles, and every error is either handled or surfaced. It cannot silently fail.

curl -X PUT http://localhost:3000/api/workflows/reviewContent/config \
  -H "Authorization: Bearer $TOKEN" \
  -d '{ "invocable": true, "task_queue": "default", "default_role": "reviewer" }'

De-certifying removes the interceptor. The workflow continues as a standard durable workflow — same code, different guarantees.

Register MCP Servers

Long Tail connects to any MCP server — an npm package, a remote service, or one you write yourself. Registered tools become durable activities and are available to the Pipeline Designer.

Use an existing package — no code, just register:

curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "filesystem",
    "transport_type": "stdio",
    "transport_config": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/data"] },
    "tags": ["files", "storage"],
    "auto_connect": true
  }'

Connect a remote server — point at a URL:

curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my-python-server",
    "transport_type": "sse",
    "transport_config": { "url": "http://python-service:8000/mcp" },
    "tags": ["ml", "classification"],
    "compile_hints": "Returns confidence scores. Use threshold 0.85 for auto-approve."
  }'

Write your own and register it in-process:

import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import { registerMcpTool } from '@hotmeshio/long-tail';

export function createImageToolsServer(): McpServer {
  const server = new McpServer({ name: 'image-tools', version: '1.0.0' });

  registerMcpTool(server, 'resize_image', 'Resize an image.', {
    path: z.string().describe('Path to the image'),
    width: z.number().optional().describe('Target width'),
    height: z.number().optional().describe('Target height'),
  }, async (args: any) => ({
    content: [{ type: 'text', text: JSON.stringify(await resize(args)) }],
  }));

  return server;
}

const lt = await start({
  // ...
  mcp: {
    serverFactories: { 'image-tools': createImageToolsServer },
  },
});

All three paths produce the same outcome: tools callable as durable activities. Tags enable discovery. Compile hints guide the compiler when tools are compiled into deterministic pipelines. See the MCP guide for the full registration lifecycle.

Ask It Anything

Once your tools are registered, the Pipeline Designer orchestrates them. Describe what you need in plain language:

"Log into localhost:3000 as superadmin, navigate to every page in the sidebar, and save a screenshot of each."

The system discovers the right MCP servers, calls the tools, chains the results. If it works, the compilation wizard converts the execution into a deterministic pipeline — parameterized inputs, typed schema, no LLM at runtime. It deploys as a new MCP tool that any workflow, agent, or API call can invoke.

The inventory of compiled tools grows over time. The need for LLM reasoning shrinks. Problems that once required a human, then required an AI, eventually require neither.

Full Configuration

const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [{ taskQueue: 'default', workflow: myWorkflow.reviewContent }],

  // Everything below is optional
  mcp: {
    server: { enabled: true },
    serverFactories: { 'my-tools': createMyToolsServer },
  },
  escalation: { strategy: 'mcp' },
  auth: { secret: process.env.JWT_SECRET },
  telemetry: { honeycomb: { apiKey: process.env.HNY } },
  logging: { pino: { level: 'info' } },
  maintenance: true,
});

Use as a Package (No HTTP Server)

Long Tail can run as an embedded package inside your existing application — no Express server, no socket.io, no extra ports. The same API surface is available as direct function calls.

import { start, createClient } from '@hotmeshio/long-tail';

await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  workers: [{ taskQueue: 'default', workflow: reviewContent.reviewContent }],
});

const lt = createClient({ auth: { userId: 'system' } });

// Same operations as the REST API — no HTTP overhead
const tasks = await lt.tasks.list({ status: 'completed', limit: 10 });
const result = await lt.escalations.claim({ id: 'esc_123', durationMinutes: 30 });

Subscribe to events with callbacks instead of socket.io:

lt.events.on('task.completed', (event) => {
  console.log('done:', event.workflowId);
});

lt.events.on('escalation.*', (event) => {
  notifyTeam(event);
});

Every SDK call returns an LTApiResult — same status codes, same validation, same RBAC. The transport is the only thing that changes. See the SDK guide for the full API reference.

Deployment

Three modes from the same codebase:

// 1. Standalone — dashboard + REST API + workers
await start({ database: { connectionString: process.env.DATABASE_URL } });

// 2. Worker-only — workflow execution, no HTTP server
await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  workers: [{ taskQueue: 'default', workflow: reviewContent.reviewContent }],
});

// 3. Embedded — inside your NestJS/Next.js/Express app, SDK calls only
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false } });
const lt = createClient({ auth: { userId: 'service' } });

All modes share PostgreSQL and scale independently. See Cloud Deployment.

Docs

Guide	What it covers
The Long Tail Story	Why this exists, what accumulates over time, what you own
Workflows	Activities, interceptor, escalation lifecycle, composition
IAM	Identity propagation, service accounts, credential exchange
Dashboard	Navigation, key pages, event feed
MCP	Server registration, tool calls, human queue
Compilation	Dynamic → deterministic pipeline wizard
Escalation Strategies	Default, MCP triage, custom handlers
SDK	Embedded usage, createClient, event subscriptions
Architecture	Project structure, conventions, discovery
Cloud	AWS ECS, GCP Cloud Run, Docker
Data Model	Database schema

Adapters: Auth · Events · Telemetry · Logging · Maintenance · OAuth

HTTP API: Workflows · Tasks · Escalations · YAML Workflows · Users · Roles · Service Accounts · MCP Servers · MCP Runs · Exports

SDK: Overview · Workflows · Tasks · Escalations · YAML Workflows · MCP · Events

Contributing

git clone https://github.com/hotmeshio/long-tail.git
cd long-tail
docker compose up -d --build

Open http://localhost:3000. Example workflows seed the dashboard.

User	Password	Role
superadmin	l0ngt@1l	superadmin
admin	l0ngt@1l	admin
engineer	l0ngt@1l	engineer
reviewer	l0ngt@1l	reviewer

See Contributing.

License

See LICENSE.