Inferstack
Your agent shouldn’t pay for the same tokens twice.
Inferstack is the session layer under every Lumenar agent: an inference gateway for open models. It keeps each session on the engine that already holds its cache, sends each step to the cheapest model that fits, and records what every request really cost.
What the prototype measures
Measured on a laptop with no GPU.
Two Qwen2.5 models served by llama.cpp behind one gateway. The same two-turn session was run cold, and then with the session cache.
- 87%less served cost on the second turn, on the small model. 79.6% on the larger one.
- 98.3%of the prompt came from cache on the second turn.
- 0.24 sfor a repeated 4,298-token prompt from cache, against 73.4 s cold.
8 CPU cores, 6.9 GiB RAM, Qwen2.5 0.5B and 1.5B Q4_K_M, llama.cpp b12186. Served cost is measured engine time times an hourly rate. On a GPU the absolute times fall; the cache gain is the part that does not depend on hardware.
What Inferstack does
It sits between your agent and your engines.
Your agent keeps using the OpenAI chat API. It adds one header for the session.
Keeps each session on its cache
A session is pinned to the engine it started on, so the engine already holds the prefix and old context is not computed again.
Picks a model for each step
Three policies: cost, balanced, quality. Under balanced, chat and extraction go to the small model; code and reasoning escalate to the large one and stay there.
Meters what each request cost
Served cost is busy engine time times your hourly rate. Reference cost is the list price for the same tokens. Both are stored per request in a ledger.
Fails over
If the chosen engine is unhealthy or returns a retryable error, the request goes to the alternate engine and the response says so.
Publishes eval checksums
A fixed eval suite runs against each model. The score and the hash of the suite ship with the model metadata, so a model swap cannot go unnoticed.
Speaks the API you already use
Streaming and non-streaming chat completions, bearer-token auth, Prometheus metrics, and an inferstack block on every response with engine, cache hit, latency and cost.
Runs agents and tasks for you
Save an agent: instructions, policy, limits, an optional budget. Hand it tasks. The gateway runs them in the background and keeps the agent’s instructions in the cache between tasks.
Caps what an agent may spend
Each agent has its own ledger: tasks, runs, served cost, reference cost. Give it a budget in dollars and runs stop when the measured cost reaches it.
from openai import OpenAI
client = OpenAI(base_url="https://<your-gateway>/v1", api_key="sk-lmn-...")
reply = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inferstack-session": "my-agent-1"},
)[engine=cheap route=session_sticky:cheap cache=97.1% ttft=0.217s served=$0.000008 ref=$0.000002]Engine, route, cache hit, time to first token, served cost and the list-price reference, on every request. The same numbers go into the ledger.
Inferstack Console
The control panel for a gateway.
Create an agent from a template, give it a task, press Run, and watch the result stream in with the engine, cache hit and cost it took. It runs in your browser and talks to your own gateway; your prompts never pass through us.
- 1Connect
Gateway URL and API key, stored on your device.
- 2Create an agent
Pick Summarizer, JSON extractor, Code reviewer, Translator, Support reply, Step planner, or start blank.
- 3Run a task
Paste the input, press Run. Re-run it, or run it under cost and quality side by side.
One codebase for Linux, Android and the browser. Windows and macOS builds are next.
Why this instead of a frontier API
A gateway you run changes what you can promise.
Hosted frontier models are the right tool for many jobs. For long-running agents on your own data, you want to measure the cost and own the cache.
Your hardware, your models
Open models served by engines you run, on a laptop today and on your GPUs next. Swap a model by editing one config block; the eval checksum tells you if quality moved.
Cost you can audit
Served cost is measured engine time, not a price list. Every request, session, agent and task carries it, with the list-price reference next to it.
Instructions paid once
An agent’s instructions are the cached prefix of all its tasks. Hosted caches expire in minutes; here the cache is yours and the ledger shows the hit rate.
Budgets per agent, not per account
A runaway agent stops at the dollar amount you set for it. The others keep working.
Nothing leaves your network
The gateway, the engines, the ledger and the console all run where you put them. The console in your browser talks to your gateway, not to us.
The API your code already speaks
OpenAI-compatible chat completions with one extra header for the session. Existing SDKs, agent frameworks and CLI tools keep working.
Next
Switch models without losing the session.
A model’s cache is computed from its own weights, and each model has its own tokenizer, chat template and tool format. So the next layer is a session log that no single model owns.
- 1Freeze
Stop at a clean turn boundary.
- 2Compact
Fold old turns into the session log, keeping the plan, decisions and open files.
- 3Render
Write the log in the new model’s chat template and tool format.
- 4Warm up
One prefill fills the new model’s cache.
- 5Check
A short probe confirms the new model knows the plan.
- 6Go on
The agent continues from the same turn.
Put one agent on one real workflow.
A pilot is one agent, set up with us, on your own work. You keep the receipts.