Llmrix Router

LLMRix exposes multiple model providers through stable provider-neutral ModelClient and modality-specific model interfaces. It selects an eligible model using declared operations, features, input modalities, quality, cost, latency, quota, and health signals, then applies bounded retries and cooldown without leaking routing complexity into application code.

Use it as an embedded Java SDK, a Spring Boot starter, or an OpenAI-compatible routing service.

Project status: General Availability (1.0.3). The Java API and configuration model are production-ready, with Semantic Versioning strictly enforced. Published to Maven Central.

Why LLMRix

• Three first-class providers: OpenAI, DeepSeek, and OpenRouter over one validated OpenAI protocol transport.
• Policy separated from execution: strategies rank model targets; the executor owns timeout, retry, quota, and cooldown correctness.
• Streaming-safe candidate switching: the router can try another configured model before output begins and never replays after output begins.
• Local or distributed state: zero-infrastructure local mode and Redis-backed health, leases, RPM, and TPM for multi-instance deployments.
• OpenAI-compatible edge: Chat, Responses, Embeddings, Rerank, Audio, Images, Videos, Models, and SSE endpoints.
• Framework-neutral client: Orion provides a small Java client plus optional Spring Boot auto-configuration.
• Observable by design: lifecycle events, Micrometer metrics, Spring Observations, request IDs, and health indicators.
• Composable advanced routing: semantic routing, contextual bandits, online shadow traffic, evaluation, and Fugu-style iterative orchestration.

Architecture

Clients enter through embedded Java, Spring Boot, or OpenAI-compatible HTTP APIs. The provider-neutral core filters and ranks model targets, executes calls with reliability controls, shares runtime state, and emits telemetry. The framework owns routing semantics and request correctness. Infrastructure remains responsible for TLS, WAF, load balancing, Redis HA, secret management, telemetry storage, and container orchestration.

Modules

Requirements

• Java 17 or later; Java 21 is recommended.
• Spring Boot 3.x when using either starter.
• Redis is optional and required only for shared multi-instance runtime state.

Quick Start

Maven

<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-core</artifactId>
  <version>1.0.3</version>
</dependency>
<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-integrations</artifactId>
  <version>1.0.3</version>
</dependency>

Programmatic configuration

The same router can be built without Spring or YAML. The runtime Builder lives in Core; the integrations artifact registers the built-in OpenAI-compatible providers through the Core SPI. Integrations own provider credentials and may define multiple models:

try (LlmRouter router = LlmRouter.builder()
        .integration("openai", integration -> integration
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .model("gpt-4.1-mini", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS)))
        .integration("deepseek", integration -> integration
            .apiKey(System.getenv("DEEPSEEK_API_KEY"))
            .model("deepseek-chat", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.CODE)))
        .route("general", route -> route
            .strategy("balanced")
            .quota(600L, 100_000L) // shared route RPM and TPM
            .models("openai/gpt-4.1-mini", "deepseek/deepseek-chat"))
        .build()) {
    ChatResponse response = router.chat("Review this Java code");
}

Route quotas are optional and apply to all targets in the route. The two-argument form is quota(requestsPerMinute, tokensPerMinute). When an authenticated request contains RoutingHints.AUTH_QUOTA_KEY, each key receives an independent quota partition; otherwise the route uses a shared partition. Target-level limits(...) remain independent provider-model limits.

Policy-based routing

RoutedChatModel model = RoutedChatModel.builder()
    .target("reasoning", reasoningModel, target -> target
        .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.REASONING)
        .inputCostPerMillion(1.25)
        .outputCostPerMillion(10.00))
    .target("fast", fastModel, target -> target
        .operations(ModelOperation.CHAT).traits(ModelTrait.CODE)
        .inputCostPerMillion(0.27)
        .outputCostPerMillion(1.10))
    .strategy(Strategies.balanced())
    .timeout(Duration.ofSeconds(30))
    .maxRetries(1)
    .build();

ChatResponse response = model.chat(ChatRequest.builder()
    .userMessage("Find the race condition")
    .routingHints(RoutingHints.builder()
        .require(ModelTrait.CODE)
        .maxCostUsd(0.05)
        .build())
    .build());

Applications can call synchronously, asynchronously, or as a Flow.Publisher<ChatChunk>. Text, images, input audio, tools, structured response formats, usage, finish reasons, and common generation options are represented by provider-neutral Core types.

Routing Model

Every request follows one deterministic execution pipeline:

1. Validate the request and normalize routing hints.
2. Remove targets that violate capability, model, context, cost, quota, concurrency, or health constraints.
3. Rank eligible targets with the configured strategy.
4. Acquire runtime quota and concurrency leases.
5. Execute with a bounded per-attempt and total timeout.
6. Retry only retryable failures and only within the configured budget.
7. Mark failures, apply cooldown, and move to the next eligible model in the route pool.
8. Settle token usage, release leases, and publish lifecycle observations.

Built-in strategies include priority, round-robin, weighted random, balanced scoring, semantic scoring, and contextual bandit selection. Custom policies implement RoutingStrategy; custom runtime persistence implements RouterStateStore or BanditStateStore.

HTTP Protocol

The Spring Boot starter exposes an OpenAI-compatible HTTP API. The model field in every request identifies a configured Router route name (such as general, vision, or multimodal) rather than an upstream provider model ID.

Enable the HTTP API

llmrix:
  model:
    router:
      http:
        enabled: true
        auth:
          mode: api-key
          bootstrap-key: ${LLMRIX_MODEL_ROUTER_API_KEY}

export BASE_URL=http://127.0.0.1:8080
export API_KEY=your-llmrix-http-key

Endpoint Catalog

The server example includes embeddings and rerank routes backed by free OpenRouter models. The request model is the Router route name, not the upstream model ID:

curl --location "${BASE_URL}/v1/embeddings" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"embeddings","input":"Text to embed"}'

curl --location "${BASE_URL}/v1/rerank" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"rerank","query":"refund policy","documents":["Refunds are available within 30 days.","Contact support by email."],"top_n":1}'

Chat Completions

Synchronous

curl --location "${BASE_URL}/v1/chat/completions" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "general",
    "messages": [
      {"role": "user", "content": "Introduce LLMRix Router in three sentences."}
    ],
    "temperature": 0
  }'

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "general",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "This is the model response."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 18, "completion_tokens": 12, "total_tokens": 30}
}

Streaming (SSE)

curl --no-buffer --location "${BASE_URL}/v1/chat/completions" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "general",
    "messages": [{"role": "user", "content": "Explain model routing."}],
    "stream": true
  }'

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{"content":"Model"},"finish_reason":""}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{"content":" routing"},"finish_reason":""}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Protocol Conventions

Success envelope. Chat-style responses return id, object, model (the route name), choices, and a nullable usage block. Provider names and upstream response bodies are never exposed.

Error envelope. Controller-level errors use an OpenAI-shaped structure with error.message, error.type, error.code, and error.param. Authentication failures rejected before the controller may use a compact form without code or param.

HTTP status codes.

Clients should branch on the HTTP status and error.type / error.code, not on complete message text. Third-party names, credentials, and raw upstream response bodies are never included in the public error message.

Multimodal Content

Chat Completions accepts text, image_url, video_url, input_audio, and file content parts. The selected model must declare the corresponding input-modalities value (vision, video, audio, or file). Speech and video content endpoints return binary data instead of a JSON wrapper.

Server Deployment

Server deployment, Spring Boot configuration, provider integrations, HTTP authentication, request ID propagation, startup commands, and additional curl examples are documented in docs/server.md. The full protocol reference with all endpoints, response shapes, and error formats is in docs/api.md.

Client Usage

The Orion Java client, typed model operations, multimodal requests, asynchronous and streaming calls, request options, and Spring Boot client starter are documented in docs/client.md.

Reliability and Streaming

• Retry applies only to failures classified as retryable; after a failed attempt, the router may continue through the configured model pool.
• A streaming request may switch targets before its first chunk, never after data is visible to the caller.
• Cancellation propagates to the active target and releases runtime state.
• Tool-bearing requests must not be blindly replayed because tools may have side effects.
• Upstream HTTP status is retained on provider-domain exceptions; non-HTTP failures use -1.
• Online shadow execution is isolated by sampling, timeout, and concurrency limits and skips tool requests by default.

Observability

Router and Fugu lifecycle listeners are the stable Core observability boundary. Optional Spring integration provides Micrometer counters/timers, first-token latency, Observation context, Actuator health, and request-ID correlation. Orion exposes its own dependency-free listener SPI and adapts it to Micrometer when used through Spring.

The framework emits telemetry but does not deploy Prometheus, Grafana, an OpenTelemetry Collector, or log storage.

Fugu Orchestration

FuguOrchestrator implements iterative Candidate/Role selection for solver-reviewer and refinement workflows. It supports rule-based or ONNX policies, bounded turns, retry/fallback, optional shared cooldown state, and a lifecycle Flow.Publisher. The lifecycle stream represents orchestration events, not model token chunks.

Training, reward modeling, and policy rollout stay offline. Runtime policy manifests are versioned and validated before inference.

Extension Points

In Spring Boot, declare the SPI implementations as beans. The starter discovers them and contributes them to the same LlmRouterBuilder assembly pipeline:

@Bean
ModelProvider acmeProvider() {
    return new AcmeModelProvider();
}

@Bean
ProviderAuthenticator signedRequestAuthenticator() {
    return new AcmeSignedRequestAuthenticator();
}

@Bean
ModelPricingResolver catalogPricingResolver() {
    return context -> pricingCatalog.find(context.providerId(), context.modelName());
}

A custom component with the same ID replaces the built-in implementation, which supports enterprise proxies, proprietary authentication, and internal pricing catalogs.

Build and Test

mvn clean test
mvn package -DskipTests

llmrix-model-examples is a Maven aggregator; each Router or client production module has a corresponding *-examples child. Tests and executable examples live in those children and follow the production module boundaries. Redis integration tests require a real Redis instance; HTTP/SSE tests require permission to bind a local port. CI is expected to run both on Java 17 and Java 21.

Router modules use Lombok to generate ordinary Java-class boilerplate. Lombok is a provided compile-time annotation processor, is not propagated as a consumer runtime dependency, and is configured explicitly in Maven.

Scope

LLMRix is a routing framework, not an API gateway, model host, training platform, secret manager, or infrastructure control plane. Deploy it behind Higress, APISIX, Nginx, or a cloud load balancer when gateway features are required. Provider keys should be supplied through the deployment environment or a managed secret store.

Compatibility

The project follows Semantic Versioning. After 1.0, public Java APIs, Maven coordinates, protocol behavior, and llmrix.model.* configuration remain backward compatible within a major line. Experimental APIs are explicitly documented and excluded from that guarantee.

Contributing

Bug reports, feature requests, and pull requests are welcome — see CONTRIBUTING.md and GIT_CONVENTIONS.md. Please read the Code of Conduct before participating.

Security

To report a vulnerability, follow the Security Policy. Do not open a public issue.

Changelog

Release history is maintained in CHANGELOG.md.

License

MIT License — see LICENSE.