An LLM gateway
Lattice carries each request from your app to the model that should answer it — routing, falling back, caching and metering on the way — behind one OpenAI-compatible URL.
Every provider fails in its own weather. Without a gateway, your app finds out the way your users do: a spinner that never stops.
Undelivered — 03:02
Automatic fallback
Take a provider down (keys 1–4)
delivered to openai · +0 ms · 0 lost requests
Most days, nothing goes wrong.
You just don't see the weather we flew through.
Each request becomes a line: which model answered, by which route, in how many milliseconds, for how much. Demo data.
1204331 requests delivered today
How it works
Send each request to the model that should answer it — by priority, cost, latency, or a header your app sets. Change the rule without shipping your app.
# lattice.yaml
routes:
- match: { header: x-tier, equals: free }
models: [mistral-small, llama-3.1-8b]
- match: { default: true }
models: [gpt-4o, claude-sonnet, gemini-pro]
strategy: lowest-latencyOn a 429, a 5xx or a timeout, the same request moves to the next model in the route — streamed responses included. Your app sees one answer.
fallback:
on: [429, 500, 502, 503, 529, timeout]
timeout_ms: 8000
max_hops: 2Exact and semantic caching at the gateway. Repeated prompts come back in single-digit milliseconds and cost nothing. Off by default for anything with a user id.
Virtual keys per team, service or customer, each with a monthly budget and a tokens-per-minute ceiling. Provider keys never leave the gateway.
Every request is a trace — route, hops, tokens, time to first token, cost — exported over OpenTelemetry.
Hosted in three regions, or one container in your own cloud. Same config either way.
Pricing
Keep your SDK. Point it at Lattice.
- client = OpenAI(base_url="https://api.openai.com/v1")
+ client = OpenAI(base_url="https://gw.lattice.dev/v1")
$ python app.py
200 OK req_7f3a91 gpt-4o 381 ms $0.0021Delivered