LLM 429 Rate Limit & Worker Concurrency Calculator
Compute required RPM, Input TPM (ITPM), and Output TPM (OTPM) across 60 live 2026 models, spot quota bottlenecks before launch, and copy full-jitter backoff configs.
Quota Utilization & Bottleneck Diagnosis
// Full-jitter exponential backoff + Retry-After guard (google/gemini-4-hydrogen)
export async function callWithRateLimitGuard<T>(
fn: () => Promise<Response>,
maxRetries = 5,
baseDelayMs = 500,
maxDelayMs = 30_000
): Promise<T> {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
const res = await fn();
if (res.status !== 429 && res.status < 500) {
return (await res.json()) as T;
}
if (attempt === maxRetries) {
throw new Error(`LLM quota exhausted (HTTP ${res.status}) after ${maxRetries} retries`);
}
const retryAfterSec = Number(res.headers.get('retry-after'));
const ceiling = Math.min(maxDelayMs, baseDelayMs * 2 ** attempt);
const sleepMs = Number.isFinite(retryAfterSec) && retryAfterSec > 0
? retryAfterSec * 1000
: Math.floor(Math.random() * ceiling);
await new Promise((r) => setTimeout(r, sleepMs));
}
throw new Error('Unreachable');
}How LLM 429 Rate-Limit Bottlenecks Happen
Enterprise LLM API gateways enforce three simultaneous token-bucket ceilings: RPM (requests per minute), ITPM (input tokens per minute, including RAG chunks), and OTPM (output completion tokens per minute). Breaching any single bucket returns HTTP 429 even when the other two buckets sit below 20% utilization. Pair quota sizing with 30-Day API Spend and verify model deprecation timelines on LLMEOL.
Production Queueing & Retry Guardrails
- Cap in-flight worker concurrency: Bound worker pools using Little’s Law (
Concurrency = Safe_RPM × Avg_Latency_Sec ÷ 60) with a 15% safety headroom. - Honor
Retry-Afterheaders first: Fall back to capped exponential backoff with full random jitter so synchronized agent retries never stampede the quota window. - Offload non-interactive jobs to Batch API: Moving nightly evaluations and document extraction to 24-hour asynchronous queues bypasses real-time RPM limits while cutting LLM token costs by 50%.
Frequently Asked Questions
Why do I hit HTTP 429 errors when my request volume is far below my RPM limit?
Long RAG prompts and multi-turn agent histories exhaust Input TPM (ITPM) or Output TPM (OTPM) token buckets long before reaching your Requests per Minute (RPM) ceiling.
Does Prompt Caching increase my Input TPM rate limit?
Prompt Caching lowers input token billing by 90% to 95%, and several providers exclude cached prefix hits from uncached ITPM quotas—model exact cost savings in the LLM Cost Calculator.
Why use full-jitter exponential backoff instead of fixed retry delays?
Fixed retry intervals cause concurrent workers throttled in the same second to retry in lockstep waves; full random jitter spreads retries evenly across the recovery window in TypeScript and Python SDKs.