Skip to content

Inferstack

Your agent shouldn’t pay for the same tokens twice.

Inferstack is the session layer under every Lumenar agent: an inference gateway for open models. It keeps each session on the engine that already holds its cache, sends each step to the cheapest model that fits, and records what every request really cost.

What the prototype measures

Measured on a laptop with no GPU.

Two Qwen2.5 models served by llama.cpp behind one gateway. The same two-turn session was run cold, and then with the session cache.

  • 87%less served cost on the second turn, on the small model. 79.6% on the larger one.
  • 98.3%of the prompt came from cache on the second turn.
  • 0.24 sfor a repeated 4,298-token prompt from cache, against 73.4 s cold.

8 CPU cores, 6.9 GiB RAM, Qwen2.5 0.5B and 1.5B Q4_K_M, llama.cpp b12186. Served cost is measured engine time times an hourly rate. On a GPU the absolute times fall; the cache gain is the part that does not depend on hardware.

What Inferstack does

It sits between your agent and your engines.

Your agent keeps using the OpenAI chat API. It adds one header for the session.

  • Keeps each session on its cache

    A session is pinned to the engine it started on, so the engine already holds the prefix and old context is not computed again.

  • Picks a model for each step

    Three policies: cost, balanced, quality. Under balanced, chat and extraction go to the small model; code and reasoning escalate to the large one and stay there.

  • Meters what each request cost

    Served cost is busy engine time times your hourly rate. Reference cost is the list price for the same tokens. Both are stored per request in a ledger.

  • Fails over

    If the chosen engine is unhealthy or returns a retryable error, the request goes to the alternate engine and the response says so.

  • Publishes eval checksums

    A fixed eval suite runs against each model. The score and the hash of the suite ship with the model metadata, so a model swap cannot go unnoticed.

  • Speaks the API you already use

    Streaming and non-streaming chat completions, bearer-token auth, Prometheus metrics, and an inferstack block on every response with engine, cache hit, latency and cost.

  • Runs agents and tasks for you

    Save an agent: instructions, policy, limits, an optional budget. Hand it tasks. The gateway runs them in the background and keeps the agent’s instructions in the cache between tasks.

  • Caps what an agent may spend

    Each agent has its own ledger: tasks, runs, served cost, reference cost. Give it a budget in dollars and runs stop when the measured cost reaches it.

One extra header
from openai import OpenAI
client = OpenAI(base_url="https://<your-gateway>/v1", api_key="sk-lmn-...")
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"x-inferstack-session": "my-agent-1"},
)
Every response says what it cost
[engine=cheap route=session_sticky:cheap cache=97.1% ttft=0.217s served=$0.000008 ref=$0.000002]

Engine, route, cache hit, time to first token, served cost and the list-price reference, on every request. The same numbers go into the ledger.

Inferstack Console

The control panel for a gateway.

Create an agent from a template, give it a task, press Run, and watch the result stream in with the engine, cache hit and cost it took. It runs in your browser and talks to your own gateway; your prompts never pass through us.

LiveOpen the Console
  1. 1Connect

    Gateway URL and API key, stored on your device.

  2. 2Create an agent

    Pick Summarizer, JSON extractor, Code reviewer, Translator, Support reply, Step planner, or start blank.

  3. 3Run a task

    Paste the input, press Run. Re-run it, or run it under cost and quality side by side.

One codebase for Linux, Android and the browser. Windows and macOS builds are next.

Why this instead of a frontier API

A gateway you run changes what you can promise.

Hosted frontier models are the right tool for many jobs. For long-running agents on your own data, you want to measure the cost and own the cache.

  • Your hardware, your models

    Open models served by engines you run, on a laptop today and on your GPUs next. Swap a model by editing one config block; the eval checksum tells you if quality moved.

  • Cost you can audit

    Served cost is measured engine time, not a price list. Every request, session, agent and task carries it, with the list-price reference next to it.

  • Instructions paid once

    An agent’s instructions are the cached prefix of all its tasks. Hosted caches expire in minutes; here the cache is yours and the ledger shows the hit rate.

  • Budgets per agent, not per account

    A runaway agent stops at the dollar amount you set for it. The others keep working.

  • Nothing leaves your network

    The gateway, the engines, the ledger and the console all run where you put them. The console in your browser talks to your gateway, not to us.

  • The API your code already speaks

    OpenAI-compatible chat completions with one extra header for the session. Existing SDKs, agent frameworks and CLI tools keep working.

Next

Switch models without losing the session.

A model’s cache is computed from its own weights, and each model has its own tokenizer, chat template and tool format. So the next layer is a session log that no single model owns.

  1. 1Freeze

    Stop at a clean turn boundary.

  2. 2Compact

    Fold old turns into the session log, keeping the plan, decisions and open files.

  3. 3Render

    Write the log in the new model’s chat template and tool format.

  4. 4Warm up

    One prefill fills the new model’s cache.

  5. 5Check

    A short probe confirms the new model knows the plan.

  6. 6Go on

    The agent continues from the same turn.

Put one agent on one real workflow.

A pilot is one agent, set up with us, on your own work. You keep the receipts.