LLM gateway

One endpoint.Every model.

Lattice sits between your app and every model provider. It routes, falls back, caches and meters each request — behind one OpenAI-compatible URL.

Get an API key Read the docs

Everything falling behind this text is traffic: model names, latencies, request ids.

Five SDKs.Four rate limits.One outage at 3 a.m.

Every provider fails differently. Your app finds out the way your users do: a spinner that never stops.

03:02:11  provider A   429  rate limit exceeded
03:02:11  provider B   529  overloaded
03:02:14  provider C   503  quota exhausted
03:02:19  your app     retry(retry(retry(…)))
03:02:49  user         closed the tab

Automatic fallback

Take a provider down.Mid-flight.

Time stops here; your request doesn't. Pull any provider below and watch Lattice re-route the call inside the same request, before your user sees an error.

Pull a provider (keys 1–4)

routed to openai · +0 ms · 0 failed requests

Now you canread the rain.

Every request, every model, every millisecond and cent — one log, one trace, one bill. Demo data.

1204331 requests routed today

timerequestmodelroutetokensmsusd

The control plane

Out here, you write the rules.

Lattice is a single binary or a hosted endpoint. Your app keeps its OpenAI client; everything below is configuration.

Routing

Send each request to the model that should answer it — by priority, by cost, by latency, or by a header your app sets. Change the rule without shipping your app.

# lattice.yaml
routes:
  - match: { header: x-tier, equals: free }
    models: [mistral-small, llama-3.1-8b]
  - match: { default: true }
    models: [gpt-4o, claude-sonnet, gemini-pro]
    strategy: lowest-latency

Fallbacks

When a provider returns 429, 5xx or times out, the same request moves to the next model in the route — streamed responses included. Your app sees one response.

fallback:
  on: [429, 500, 502, 503, 529, timeout]
  timeout_ms: 8000
  max_hops: 2

Cache

Exact and semantic caching at the gateway. Repeated prompts return in single-digit milliseconds and cost nothing. Per-route TTLs; off by default for anything with a user id.

Keys & budgets

Issue virtual keys per team, service or customer. Set a monthly budget and a tokens-per-minute ceiling on each; provider keys never leave the gateway.

Traces

Every request is a trace: route chosen, hops taken, tokens in and out, latency to first token, cost. Export over OpenTelemetry to the tools you already run.

req_7f3a91                         412 ms
├─ auth  key_team-search              2 ms
├─ cache miss                         1 ms
├─ openai     gpt-4o         503     31 ms
└─ anthropic  claude-sonnet  200    376 ms

Where it runs

Hosted in three regions, or self-hosted as one container in your own cloud. Same config either way.

Pricing

Free to 1M requests a month. Then $0.40 per million routed.

Tokens: Billed by your providers, on your keys. Lattice never marks them up.
Self-hosted: Free under the source-available licence. Support contracts on request.
Limits: No seat fees, no per-model fees, no minimum.

Change one line.

Keep your SDK. Point it at Lattice.

- client = OpenAI(base_url="https://api.openai.com/v1")
+ client = OpenAI(base_url="https://gw.lattice.dev/v1")

$ python app.py
200 OK  req_7f3a91  gpt-4o  412 ms  $0.0021

Get an API key Read the docs