Janady13/crawler-signal-reference / framework-http-rate-control-headers.md

1# framework-http-rate-control-headers.md 2 3Comprehensive reference for the HTTP response headers that communicate rate limit state and backoff guidance to clients: the standardized `Retry-After` (RFC 9110) and the rate limit family covering both the widely deployed de facto convention `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` and the IETF standardization track `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `RateLimit-Policy`. Includes the status code companions `429 Too Many Requests` and `503 Service Unavailable` that activate Retry-After, plus the full nginx `limit_req` and `limit_conn` configuration that produces them. Built for Bubbles (Debian, Nginx 1.26+, self hosted origin at 169.155.162.118, no Cloudflare or third party CDN in front). Companion to framework-http-caching-headers.md, framework-http-content-headers.md, framework-http-seo-headers.md, framework-http-security-headers.md, framework-http-performance-headers.md, framework-http-cors-headers.md, UNIVERSAL-RANKING-FRAMEWORK.md, and SEO BUILD REFERENCE v2.4. 4 5Audience: humans configuring nginx rate limiting, AI assistants generating or repairing rate limit config, API engineers shaping client backoff behavior, SEO operators ensuring Googlebot and AI crawlers are not accidentally throttled, and anyone troubleshooting "429s on legitimate users", "Googlebot dropped my pages", "client retry storm after maintenance", or "rate limit headers missing on API responses" anomalies on a self hosted stack. 6 7--- 8 9## TABLE OF CONTENTS 10 111. Definition 122. Why It Matters 133. What This Covers 144. The Rate Control Mental Model (read this first) 155. Retry-After (the standardized backoff signal) 166. X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (the de facto convention) 177. The IETF RateLimit-* Standardization (the future) 188. Status Code Pairing (429 vs 503 vs 3xx) 199. Nginx Rate Limiting Configuration (limit_req and limit_conn) 2010. The Crawler Protection Rule (never rate limit Googlebot, ClaudeBot, GPTBot) 2111. Client Backoff Patterns (exponential, jitter, honoring Retry-After) 2212. How These Headers Interact 2313. Asset Class And Use Case Recipes 2414. Bubbles Nginx Reference Block (paste ready) 2515. Audit Checklist (50+ items) 2616. Common Pitfalls 2717. Diagnostic Commands 2818. Cross-References 29 30--- 31 32## 1. DEFINITION 33 34Rate control headers communicate two things to clients: (a) how much quota remains right now, so well behaved clients can self throttle, and (b) when to come back after being rejected, so retry storms do not amplify the problem they were meant to solve. The headers split into three families: 35 36* **The backoff signal**: `Retry-After`. Standardized in RFC 9110. Sent with `429 Too Many Requests`, `503 Service Unavailable`, or `3xx` redirect responses. Tells the client (including search engine crawlers) exactly when it is safe to try again. 37* **The rate limit announcement (de facto)**: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`. Widely deployed (GitHub, Twitter, Stripe, most public APIs) but never standardized. Sent on every response to let clients see their current quota state. 38* **The rate limit announcement (standards track)**: `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `RateLimit-Policy`. IETF draft (draft-ietf-httpapi-ratelimit-headers, in late stages). Replaces the X- convention with un prefixed names. Modern APIs (GitLab, CircleCI, OKX, others) already emit both forms during the transition. 39 40`Retry-After` is the only one with a hard standard. The X- and RateLimit-* families are conventions; clients have to know what your specific API uses. The standardization track aims to fix this so any RFC compliant client can read any server's rate limit state without API specific code. 41 42--- 43 44## 2. WHY IT MATTERS 45 46Six independent pressures push correct rate control headers from "operational hygiene" to "required infrastructure" in 2025 and forward. 47 48**Search engines drop pages after 2 days of 429 or 503.** Google's official documentation states that returning 503 or 429 for more than 2 days will cause Google to drop those URLs from the index. The Retry-After header is what tells Googlebot "this is temporary, come back when X". Without it, Googlebot guesses, and after 48 hours of guessing wrong it stops trying. A Bubbles client site that accidentally rate limits Googlebot for a long weekend can lose its rankings. 49 50**AI crawlers honor Retry-After.** ClaudeBot, GPTBot, PerplexityBot, and other LLM crawlers respect Retry-After (when they respect anything). For sites Joseph has optimized for AI discoverability, throttling these crawlers without proper Retry-After defeats the optimization investment. The same 2-day rule applies in spirit: persistent throttling teaches crawlers to skip the site. 51 52**Retry storms amplify the problem.** Without Retry-After, clients retry on their own schedule. After a server briefly returns 503 during maintenance, every client retries within seconds, the next batch of 503s triggers the next round of retries, and the server stays down because it cannot drain the queue faster than retries arrive. Retry-After with a sensible value lets the storm spread out over minutes instead of seconds. 53 54**Rate limit headers prevent self imposed denial.** Without X-RateLimit-* or RateLimit-* headers, well behaved clients have no way to throttle themselves. They must blindly send requests and react to 429s. A client that emits Server-Sent Events or polls frequently will hit limits constantly. With headers exposing remaining quota, the same client smooths its rate and never gets throttled. 55 56**Nginx defaults are wrong for APIs.** nginx's `limit_req` and `limit_conn` modules return `503 Service Unavailable` by default. For a REST API, this is misleading: 503 implies the whole service is down, when really only this client exceeded its quota. The correct status is `429 Too Many Requests`. Fixing this is a one line config change (`limit_req_status 429`) that most deployments miss. 57 58**Per IP rate limiting breaks legitimate clients sharing IPs.** Office networks, corporate VPNs, mobile carriers, and IPv6 with carrier grade NAT all put many users behind a single IP. Per IP rate limits configured for "individual user" levels block entire organizations. The fix is either a generous burst, per token rate limits when authenticated, or both. 59 60**Cost of getting it wrong.** Misconfigured rate control produces silent ranking damage, real user pain, and runaway retry storms. Real examples: 61 62* nginx rate limiting set to 10r/s with no burst, no Googlebot whitelist. Googlebot crawls the site, hits the limit, gets 503 for two days. Google drops 5,800 URLs from the index. Ranking recovery takes weeks. 63* API returns 429 without Retry-After. Client library backs off exponentially starting at 1 second, doubling each retry. Hits the API 6 times in 90 seconds, all rate limited. With Retry-After: 60, would have hit once and succeeded. 64* Office of 200 users behind one NAT IP. Rate limit at 30 requests per minute per IP. The first 30 users to load the morning dashboard succeed; the rest see 429s. Support tickets flood in. 65* nginx returns 503 for rate limited API responses. Frontend monitoring alerts on "service unavailable" rate. On call engineer paged at 2 AM for what is actually expected client behavior. 66* Maintenance window planned 30 minutes. Site returns 503 with no Retry-After. Browsers and CDN caches cache the 503. Even after maintenance ends, repeat visitors get cached 503s for hours. 67 68All preventable with the rules below. 69 70--- 71 72## 3. WHAT THIS COVERS 73 74Each of the primary headers gets the same six part treatment: 75 761. **What it does**: the canonical RFC or IETF draft definition plus the practical implication. 772. **Syntax and values**: every legal format, what it means, when it is wrong. 783. **How to build it on Bubbles**: paste ready nginx config plus FastAPI sidecar code where the upstream is the natural emitter. 794. **How to verify it**: curl commands that trigger rate limits and read responses. 805. **How to troubleshoot**: the four or five failure modes seen in the field and how to recognize each. 816. **How to fix common breakage**: ordered repair steps. 82 83The status code companions (429 and 503) get a dedicated section because they activate Retry-After. The nginx `limit_req` / `limit_conn` configuration gets its own dedicated section because it produces these headers and is where most operators spend their debugging time. The crawler protection rule gets its own section because it is the single most ranking impactful decision in this framework. 84 85--- 86 87## 4. THE RATE CONTROL MENTAL MODEL (READ THIS FIRST) 88 89Every cross origin or API request runs through the rate control decision tree. Internalize it and every header decision becomes obvious. 90 91``` 92Request arrives at nginx 93 | 94 v 95==================== IDENTIFICATION ==================== 96 | 97 v 98Who is the client? 99 * IP based: $binary_remote_addr 100 * Token based: $http_authorization (or extracted) 101 * Session based: $cookie_session 102 * Endpoint based: $request_uri 103 | 104 v 105Is the client on the whitelist? 106 * Googlebot (verified by reverse DNS) 107 * ClaudeBot, GPTBot, PerplexityBot 108 * Bubbles admin IPs 109 * Local Tailscale network 110 | 111 YES (whitelisted) 112 | 113 v 114Bypass rate limit, process normally 115 | 116 | 117 NO (subject to rate limits) 118 | 119 v 120==================== RATE LIMIT CHECK ==================== 121 | 122 v 123Has this client exceeded the per second rate? 124 | | 125 NO YES 126 | | 127 | v 128 | Is there burst capacity? 129 | | | 130 | YES NO 131 | | | 132 | v v 133 | Allow (queued) Reject with 429 (or 503) 134 | or process Set Retry-After: <seconds> 135 | immediately Set RateLimit-Reset 136 | (with nodelay) Log at warn level 137 | | 138 v | 139==================== ANNOUNCE RATE LIMIT STATE ================= 140 | | 141 v | 142Decorate response with rate limit headers 143 X-RateLimit-Limit: 100 (current quota) 144 X-RateLimit-Remaining: 47 (what is left) 145 X-RateLimit-Reset: 1742658600 (when it resets) 146 AND modern equivalents: 147 RateLimit-Limit: 100 148 RateLimit-Remaining: 47 149 RateLimit-Reset: 60 150 RateLimit-Policy: 100;w=60 151 | 152 v 153==================== CLIENT REACTION ==================== 154 | 155 v 156Well behaved client: 157 Reads X-RateLimit-Remaining 158 Slows down or pauses before hitting zero 159 On 429, reads Retry-After and waits exactly that long 160 | 161 v 162Misbehaving client (or no headers): 163 Retries immediately or on hardcoded interval 164 May get throttled into a longer cooldown 165 May trigger fail2ban or equivalent 166``` 167 168Five rules govern the system: 169 1701. **Set Retry-After on every 429 and 503.** It tells crawlers it is temporary and tells well behaved clients exactly when to come back. Missing it is the difference between graceful recovery and a retry storm. 1712. **Use 429 for rate limiting, 503 for actual outage.** They are different things. Conflating them confuses both crawlers and monitoring. 1723. **Whitelist verified crawlers.** Googlebot, ClaudeBot, GPTBot are not enemies of your bandwidth budget; they are how your content gets discovered. 1734. **Announce limits proactively.** Sending RateLimit-* headers on every response (not just 429) lets clients self throttle and never hit the limit. 1745. **Migration: send both X-RateLimit-* and RateLimit-*.** The X- convention is what existing clients understand. The un prefixed form is the standardization track. Send both during the transition. 175 176A correctly configured rate control stack throttles abusive clients without harming legitimate ones, communicates exact backoff windows to crawlers (preserving search ranking), and never produces unnecessary retry storms during maintenance. 177 178--- 179 180## 5. RETRY-AFTER (THE STANDARDIZED BACKOFF SIGNAL) 181 182### 5.1 What It Does 183 184`Retry-After` tells the client how long to wait before retrying the request. Standardized in RFC 9110. Sent with three classes of responses: 185 186* `503 Service Unavailable`: the service is temporarily down. Retry-After indicates expected duration of unavailability. 187* `429 Too Many Requests`: the client exceeded a rate limit. Retry-After indicates when the limit window resets. 188* `3xx` redirect: scheduled redirect with a minimum delay before following. 189 190``` 191Retry-After: 120 192Retry-After: Wed, 21 Oct 2026 07:28:00 GMT 193``` 194 195Crawlers including Googlebot honor this header. Without it, crawlers guess at when to retry; with it, they wait precisely the indicated duration. 196 197### 5.2 The Two Formats 198 199| Format | Example | When to use | 200|---|---|---| 201| `<delta-seconds>` | `Retry-After: 60` | Relative wait. Most common. Use for rate limits, short maintenance | 202| `<HTTP-date>` | `Retry-After: Wed, 21 Oct 2026 07:28:00 GMT` | Absolute time. Use for scheduled maintenance with known end time | 203 204Delta seconds is recommended for most cases. It avoids clock skew issues between client and server. HTTP date is appropriate when the resume time is independent of the request time (a known maintenance window). 205 206### 5.3 The 2 Day Rule For Search Engines 207 208Per Google's official documentation: returning 503 or 429 for more than 2 days will cause Google to drop those URLs from the index. The mechanism: 209 210* Days 1 to 2: Googlebot treats 503/429 as temporary, honors Retry-After, retries. 211* Day 2+: Google starts treating sustained 5xx/429 as a permanent signal that the content is gone. 212* Days later: URLs drop from the index. 213 214Recovery: when the site returns 200, Googlebot recrawls. Recovery time depends on crawl budget; popular sites recrawl in hours, smaller sites can take weeks. 215 216**The implication for Bubbles operators:** never rate limit Googlebot. Any maintenance window longer than 48 hours requires a different strategy than 503 (consider a static maintenance page returning 200 with a notice). 217 218### 5.4 How To Build It On Bubbles 219 220**Static 503 maintenance page with Retry-After:** 221 222```nginx 223server { 224 server_name example.com; 225 226 location / { 227 # During maintenance, return 503 with Retry-After 228 return 503; 229 } 230 231 error_page 503 /maintenance.html; 232 location = /maintenance.html { 233 root /var/www/sites/example.com; 234 internal; 235 add_header Retry-After "3600" always; # 1 hour 236 add_header Cache-Control "no-store" always; # do not cache the 503 237 } 238} 239``` 240 241**Programmatic Retry-After from FastAPI sidecar:** 242 243```python 244from fastapi import FastAPI, Response, status 245from fastapi.responses import JSONResponse 246import time 247 248app = FastAPI() 249 250# Simple in memory rate limiter (production should use Redis or similar) 251client_buckets = {} 252 253@app.middleware("http") 254async def rate_limit(request, call_next): 255 client_ip = request.client.host 256 now = time.time() 257 bucket = client_buckets.setdefault(client_ip, {"count": 0, "reset": now + 60}) 258 259 if now > bucket["reset"]: 260 bucket["count"] = 0 261 bucket["reset"] = now + 60 262 263 if bucket["count"] >= 100: 264 retry_after = int(bucket["reset"] - now) 265 return JSONResponse( 266 status_code=429, 267 content={"error": "rate limit exceeded"}, 268 headers={ 269 "Retry-After": str(retry_after), 270 "X-RateLimit-Limit": "100", 271 "X-RateLimit-Remaining": "0", 272 "X-RateLimit-Reset": str(int(bucket["reset"])), 273 "RateLimit-Limit": "100", 274 "RateLimit-Remaining": "0", 275 "RateLimit-Reset": str(retry_after), 276 "RateLimit-Policy": "100;w=60", 277 } 278 ) 279 280 bucket["count"] += 1 281 response = await call_next(request) 282 response.headers["X-RateLimit-Limit"] = "100" 283 response.headers["X-RateLimit-Remaining"] = str(100 - bucket["count"]) 284 response.headers["X-RateLimit-Reset"] = str(int(bucket["reset"])) 285 response.headers["RateLimit-Limit"] = "100" 286 response.headers["RateLimit-Remaining"] = str(100 - bucket["count"]) 287 response.headers["RateLimit-Reset"] = str(int(bucket["reset"] - now)) 288 response.headers["RateLimit-Policy"] = "100;w=60" 289 return response 290``` 291 292**Scheduled maintenance with absolute time:** 293 294```nginx 295server { 296 location / { 297 return 503; 298 } 299 300 error_page 503 /maintenance.html; 301 location = /maintenance.html { 302 root /var/www/sites/example.com; 303 internal; 304 # Maintenance ends at a specific time 305 add_header Retry-After "Sat, 24 May 2026 22:00:00 GMT" always; 306 add_header Cache-Control "no-store" always; 307 } 308} 309``` 310 311### 5.5 How To Verify 312 313```bash 314# 1. Confirm Retry-After is sent on 429 315curl -sI -X POST https://api.example.com/endpoint \ 316 -H "Authorization: Bearer test" \ 317 --data '...' | grep -iE "^(HTTP|retry-after)" 318# After hitting the rate limit: 319# HTTP/2 429 320# retry-after: 60 321 322# 2. Confirm Retry-After on 503 maintenance 323curl -sI https://example.com/ | grep -iE "^(HTTP|retry-after)" 324# HTTP/2 503 325# retry-after: 3600 326 327# 3. Test that Googlebot sees Retry-After 328curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 329 https://example.com/ | grep -i retry-after 330 331# 4. Trigger rate limit deliberately to test response shape 332for i in $(seq 1 200); do 333 curl -so /dev/null -w "%{http_code}\n" https://api.example.com/endpoint 334done | sort | uniq -c 335# Expect: most 200, then 429s appear once burst is exhausted 336``` 337 338### 5.6 Troubleshooting 339 340**Symptom: Googlebot drops pages despite returning 503 with Retry-After.** 3411. The 503 persisted longer than 2 days. Google treats sustained 5xx as permanent removal. 3422. The Retry-After value was unrealistic (e.g., far future date). Googlebot ignores extreme values. 3433. The 503 was returned to too many URLs simultaneously (server wide rather than scoped). Google interprets as site outage. 344Fix: keep maintenance windows under 48 hours; use Retry-After values that match actual maintenance duration; for longer downtime, serve a 200 maintenance page instead. 345 346**Symptom: 503s being cached by browsers despite Retry-After.** 347The 503 response did not include `Cache-Control: no-store`. Browsers may cache 503s if not told otherwise. 348Fix: always pair 503 with `Cache-Control: no-store, no-cache, must-revalidate`. 349 350**Symptom: Clients retry immediately after 429 instead of waiting.** 3511. Client library does not honor Retry-After. 3522. The header value is invalid (non integer for delta seconds). 3533. The value is 0 or negative. 354Fix: ensure value is a positive integer of seconds, or a valid HTTP date. Update client to honor the header. 355 356**Symptom: Retry-After date format rejected by some clients.** 357Some clients are strict about RFC date format. The format is the same as the Date header: `Wed, 21 Oct 2026 07:28:00 GMT`. Note: no day name extensions, GMT timezone literal, comma after day name. 358 359**Symptom: Browser displays a "service unavailable" error page during expected rate limiting.** 360The client is a browser (not a script) and got a 429. The browser shows its own error page. This is correct behavior; the rate limit should not apply to browsers loading HTML pages, only to API endpoints or to clients clearly exceeding sensible thresholds. 361 362### 5.7 How To Fix Common Breakage 363 364**Case: Need to take site down for maintenance without losing Google rankings.** 365Two options: 366 367Option 1 (maintenance under 24 hours): serve 503 with Retry-After: 368 369```nginx 370return 503; 371error_page 503 /maintenance.html; 372location = /maintenance.html { 373 internal; 374 add_header Retry-After "3600" always; 375 add_header Cache-Control "no-store" always; 376 add_header X-Robots-Tag "noindex" always; 377 root /var/www/sites/example.com; 378} 379``` 380 381Option 2 (maintenance over 24 hours): serve 200 with a static "we are working on it" page that includes the same content URLs in a sitemap. Google keeps the URLs indexed because they return 200. 382 383**Case: API client library does not respect Retry-After.** 384Wrap the client in a retry layer that does: 385 386```python 387import requests 388import time 389 390def request_with_backoff(method, url, **kwargs): 391 for attempt in range(5): 392 response = requests.request(method, url, **kwargs) 393 if response.status_code == 429: 394 retry_after = int(response.headers.get("Retry-After", "60")) 395 time.sleep(retry_after) 396 continue 397 return response 398 return response 399``` 400 401--- 402 403## 6. X-RATELIMIT-LIMIT, X-RATELIMIT-REMAINING, X-RATELIMIT-RESET (THE DE FACTO CONVENTION) 404 405### 6.1 What They Do 406 407The X-RateLimit-* family announces the rate limit state to the client on every response. Not formally standardized but widely deployed since the early 2010s (GitHub, Twitter, Stripe, Shopify, Reddit, etc). 408 409``` 410X-RateLimit-Limit: 100 411X-RateLimit-Remaining: 47 412X-RateLimit-Reset: 1742658600 413``` 414 415A client can read these on a successful response and decide to slow down before hitting the limit. On a 429 response, the same headers tell the client how to plan the next attempt. 416 417### 6.2 The Three Headers 418 419| Header | Value | Meaning | 420|---|---|---| 421| `X-RateLimit-Limit` | Integer | Total quota for the current window | 422| `X-RateLimit-Remaining` | Integer | Requests remaining in the current window | 423| `X-RateLimit-Reset` | Integer | When the window resets. Two common formats: Unix timestamp or seconds until reset | 424 425The format of `X-RateLimit-Reset` is the single most inconsistent thing in this family. Two formats are common: 426 427* **Unix epoch seconds**: `X-RateLimit-Reset: 1742658600` (GitHub, Reddit, Stripe) 428* **Seconds until reset**: `X-RateLimit-Reset: 60` (Twitter formerly, some others) 429 430The IETF standardization track resolves this by mandating "seconds until reset" (a delta). The X- convention is ambiguous; document which form your API uses. 431 432**Bubbles convention:** use Unix epoch seconds for X-RateLimit-Reset (matches the most prominent deployments) and seconds until reset for the un prefixed RateLimit-Reset (matches the standardization track). 433 434### 6.3 How To Build It On Bubbles 435 436X-RateLimit-* headers must be set by the application layer because they depend on application state (how many requests this client has made). nginx can do basic IP based rate limiting but cannot count requests per token; that requires the upstream. 437 438**FastAPI sidecar emitting full rate limit headers:** 439 440```python 441from fastapi import FastAPI, Request, Response 442from fastapi.responses import JSONResponse 443import time 444import asyncio 445import redis.asyncio as redis 446 447app = FastAPI() 448r = redis.from_url("redis://127.0.0.1:6379") 449 450WINDOW_SECONDS = 60 451LIMIT = 100 452 453async def check_rate_limit(client_id: str): 454 """Returns (remaining, reset_epoch, allowed).""" 455 now = int(time.time()) 456 window_start = now - (now % WINDOW_SECONDS) 457 reset_epoch = window_start + WINDOW_SECONDS 458 key = f"rl:{client_id}:{window_start}" 459 460 pipe = r.pipeline() 461 pipe.incr(key) 462 pipe.expire(key, WINDOW_SECONDS + 5) 463 count, _ = await pipe.execute() 464 465 remaining = max(0, LIMIT - count) 466 allowed = count <= LIMIT 467 return remaining, reset_epoch, allowed 468 469@app.middleware("http") 470async def rate_limit(request: Request, call_next): 471 # Identify client (token preferred, fall back to IP) 472 auth = request.headers.get("authorization", "") 473 if auth.startswith("Bearer "): 474 client_id = f"token:{auth[7:32]}" 475 else: 476 client_id = f"ip:{request.client.host}" 477 478 remaining, reset_epoch, allowed = await check_rate_limit(client_id) 479 now = int(time.time()) 480 seconds_until_reset = reset_epoch - now 481 482 if not allowed: 483 return JSONResponse( 484 status_code=429, 485 content={"error": "rate limit exceeded"}, 486 headers={ 487 "Retry-After": str(seconds_until_reset), 488 # X- convention 489 "X-RateLimit-Limit": str(LIMIT), 490 "X-RateLimit-Remaining": "0", 491 "X-RateLimit-Reset": str(reset_epoch), 492 # IETF standardization track 493 "RateLimit-Limit": str(LIMIT), 494 "RateLimit-Remaining": "0", 495 "RateLimit-Reset": str(seconds_until_reset), 496 "RateLimit-Policy": f"{LIMIT};w={WINDOW_SECONDS}", 497 } 498 ) 499 500 response = await call_next(request) 501 # X- convention 502 response.headers["X-RateLimit-Limit"] = str(LIMIT) 503 response.headers["X-RateLimit-Remaining"] = str(remaining) 504 response.headers["X-RateLimit-Reset"] = str(reset_epoch) 505 # IETF standardization track 506 response.headers["RateLimit-Limit"] = str(LIMIT) 507 response.headers["RateLimit-Remaining"] = str(remaining) 508 response.headers["RateLimit-Reset"] = str(seconds_until_reset) 509 response.headers["RateLimit-Policy"] = f"{LIMIT};w={WINDOW_SECONDS}" 510 return response 511``` 512 513**Required CORS exposure** (from framework-http-cors-headers.md): 514 515```nginx 516add_header Access-Control-Expose-Headers "X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, RateLimit-Policy, Retry-After" always; 517``` 518 519Without exposing them, cross origin JavaScript cannot read these headers and the announcement is wasted. 520 521### 6.4 How To Verify 522 523```bash 524# 1. Confirm headers are present on normal responses 525curl -sI https://api.example.com/endpoint \ 526 -H "Authorization: Bearer test" | grep -iE "^(x-ratelimit|ratelimit|retry-after)" 527 528# Expected: 529# x-ratelimit-limit: 100 530# x-ratelimit-remaining: 99 531# x-ratelimit-reset: 1742658660 532# ratelimit-limit: 100 533# ratelimit-remaining: 99 534# ratelimit-reset: 60 535# ratelimit-policy: 100;w=60 536 537# 2. Verify decrement on subsequent requests 538for i in 1 2 3; do 539 REMAINING=$(curl -sI https://api.example.com/endpoint \ 540 -H "Authorization: Bearer test" \ 541 | grep -i "x-ratelimit-remaining:" | grep -oE "[0-9]+") 542 echo "Request $i: $REMAINING remaining" 543done 544 545# 3. Verify CORS exposure works 546# Browser console on a different origin: 547# fetch("https://api.example.com/endpoint", {headers: {"Authorization": "Bearer test"}}) 548# .then(r => console.log("Remaining:", r.headers.get("X-RateLimit-Remaining"))) 549# Should print a number, not null 550``` 551 552### 6.5 Troubleshooting 553 554**Symptom: Headers absent from cross origin requests despite being set.** 555The headers are not in `Access-Control-Expose-Headers`. Browser hides them from JavaScript. 556Fix: add to Expose-Headers list (Section 6.3 nginx config). 557 558**Symptom: X-RateLimit-Reset value confusion (Unix epoch vs delta).** 559Clients written for GitHub style API expect Unix epoch; clients written for Twitter style expect delta. 560Fix: document the format in API docs. For new APIs, send both forms (X-RateLimit-Reset as epoch, RateLimit-Reset as delta) for clarity. 561 562**Symptom: Different clients see different remaining counts.** 563The rate limit is per IP and they share an IP. Or the rate limit is per token and they share a token. Or the counter is incorrect. 564Fix: verify the identification logic (IP vs token vs session) is what you intend. For multi tenant SaaS, per token rate limits are the norm. 565 566**Symptom: Counter resets unexpectedly.** 567The application restarted (counter lived in memory). The Redis backing store evicted the key. The window calculation is off. 568Fix: use persistent storage for counters; configure Redis with appropriate maxmemory and policy. 569 570### 6.6 How To Fix Common Breakage 571 572**Case: Migrating from in memory counter to Redis.** 573 574```python 575# Before: in memory (lost on restart) 576counters = {} 577counters[client_id] = counters.get(client_id, 0) + 1 578 579# After: Redis with TTL (survives restarts) 580import redis.asyncio as redis 581r = redis.from_url("redis://127.0.0.1:6379") 582 583await r.incr(f"rl:{client_id}:{window_start}") 584await r.expire(f"rl:{client_id}:{window_start}", WINDOW_SECONDS + 5) 585``` 586 587**Case: Heterogeneous rate limits per endpoint.** 588Some endpoints are cheap (search), others expensive (export). Configure per endpoint limits: 589 590```python 591LIMITS = { 592 "/api/search": (1000, 60), # 1000 per minute 593 "/api/export": (10, 3600), # 10 per hour 594 "/api/upload": (50, 60), # 50 per minute 595} 596 597def get_limit(path): 598 for prefix, limit in LIMITS.items(): 599 if path.startswith(prefix): 600 return limit 601 return (100, 60) # default 602``` 603 604--- 605 606## 7. THE IETF RATELIMIT-* STANDARDIZATION (THE FUTURE) 607 608### 7.1 Why The Standardization Matters 609 610`X-RateLimit-*` was never standardized. Every major API implements it differently: 611 612* Reset format: Unix epoch (GitHub, Stripe) vs delta seconds (Twitter). 613* Window interpretation: sliding (some) vs fixed (others). 614* What "Limit" means: per second vs per minute vs per hour. 615* What "Remaining" means: current request count vs throughput percentage. 616 617The IETF draft `draft-ietf-httpapi-ratelimit-headers` (in late drafts as of 2026) standardizes the format. The headers drop the `X-` prefix and define exact semantics. Modern APIs (GitLab, CircleCI, OKX) already emit both during the transition. 618 619### 7.2 The Standardized Headers 620 621``` 622RateLimit-Limit: 100 623RateLimit-Remaining: 47 624RateLimit-Reset: 60 625RateLimit-Policy: 100;w=60 626``` 627 628| Header | Format | Definition | 629|---|---|---| 630| `RateLimit-Limit` | Integer | The expiring limit for the current window | 631| `RateLimit-Remaining` | Integer | Quantity of requests remaining in the current window | 632| `RateLimit-Reset` | Integer | Number of seconds until the limit resets (delta only, not epoch) | 633| `RateLimit-Policy` | Quoted policy | The policy in detail: `<limit>;w=<window_seconds>` | 634 635### 7.3 The Policy Header 636 637`RateLimit-Policy` describes the policy being applied. Useful when multiple windows are in effect or for documenting the policy without resorting to API docs. 638 639``` 640RateLimit-Policy: 100;w=60 641RateLimit-Policy: 100;w=60, 1000;w=3600 642RateLimit-Policy: "api-default";q=100;w=60 643``` 644 645The format is `<quota>;w=<window>` with optional policy name in quotes. Multiple policies can be listed. 646 647### 7.4 The Single Combined RateLimit Header (Draft 10+) 648 649The latest IETF draft direction is a single combined `RateLimit` header replacing the three: 650 651``` 652RateLimit: "default";r=999;t=60;pk=:Ym9vbWVy: 653``` 654 655Where: 656* `r=` is remaining. 657* `t=` is reset time (delta seconds). 658* `pk=` is partition key (which quota bucket). 659 660This format is not yet widely deployed. For Bubbles, stick with the three header form (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset) plus RateLimit-Policy as the migration target. The X-RateLimit-* family is the present, the three RateLimit-* headers are the standardization track, the single combined header is the future once the RFC publishes. 661 662### 7.5 How To Build It On Bubbles 663 664See Section 6.3 FastAPI sidecar example which already emits both X- and un prefixed forms. 665 666The key principle: **send both forms during the migration**. Clients written for the X- convention still work; clients written for the standard will work in the future. The header overhead is small. 667 668### 7.6 Migration Glide Path 669 670For new APIs in 2026: 671 6721. **Now**: emit both `X-RateLimit-*` and `RateLimit-*` forms. Document both. Use whichever clients prefer. 6732. **2027 and beyond**: continue emitting both. Most ecosystems still use X-. 6743. **When the RFC publishes**: stop emitting X- in new versions; keep emitting on legacy endpoints. The transition will take years. 675 676The cost of emitting both is one extra round of header lines. The benefit is forward compatibility without breaking existing clients. The cost benefit math says: emit both. 677 678--- 679 680## 8. STATUS CODE PAIRING (429 VS 503 VS 3XX) 681 682The status code that accompanies Retry-After matters because clients react to status code first, header second. 683 684### 8.1 429 Too Many Requests 685 686Use for: a specific client exceeded their rate quota. The service is operating normally; this specific request was rejected. 687 688``` 689HTTP/1.1 429 Too Many Requests 690Content-Type: application/json 691Retry-After: 60 692X-RateLimit-Limit: 100 693X-RateLimit-Remaining: 0 694X-RateLimit-Reset: 1742658660 695 696{"error": "rate_limit_exceeded", "message": "Try again in 60 seconds"} 697``` 698 699Defined in RFC 6585. Specifically intended for rate limiting scenarios. Crawlers including Googlebot treat 429 as "throttle me, not a problem with the site". 700 701**Critical:** nginx default for `limit_req` and `limit_conn` is 503, not 429. Configure with `limit_req_status 429` to return the correct status. 702 703### 8.2 503 Service Unavailable 704 705Use for: the entire service is temporarily unavailable. Maintenance, overload, dependency outage. 706 707``` 708HTTP/1.1 503 Service Unavailable 709Content-Type: text/html 710Retry-After: 3600 711Cache-Control: no-store, no-cache, must-revalidate 712 713<html>...<body>We are doing scheduled maintenance...</body></html> 714``` 715 716Defined in RFC 9110. Indicates a transient service wide problem. Crawlers treat 503 as "site is down, come back later". The 2-day rule applies: persistent 503 leads to index drops. 717 718### 8.3 The Difference In One Sentence 719 720429: "This specific client hit their quota; other clients are fine". 721 722503: "This whole service is broken or down; everyone is affected". 723 724Use the right one for the right scenario. Conflating them confuses monitoring, crawlers, and clients. 725 726### 8.4 3xx Redirects With Retry-After 727 728Less commonly used. A 3xx response with Retry-After tells the client to wait before following the redirect. 729 730``` 731HTTP/1.1 301 Moved Permanently 732Location: https://new.example.com/page 733Retry-After: 60 734``` 735 736Useful for staged redirect rollouts where the destination might be cold. Most clients ignore Retry-After on 3xx; the redirect is followed immediately. 737 738### 8.5 How To Verify Status Code Pairing 739 740```bash 741# Test that rate limit produces 429 (not 503) 742for i in $(seq 1 200); do 743 STATUS=$(curl -so /dev/null -w "%{http_code}" \ 744 -H "Authorization: Bearer test" \ 745 https://api.example.com/endpoint) 746 if [ "$STATUS" != "200" ]; then 747 echo "Request $i: $STATUS" 748 break 749 fi 750done 751# Expected: eventually "Request N: 429" 752# Wrong: "Request N: 503" (means limit_req_status not set) 753 754# Test that maintenance page produces 503 755curl -sI https://example.com/ # during scheduled maintenance 756# Expected: HTTP/2 503 with Retry-After 757``` 758 759--- 760 761## 9. NGINX RATE LIMITING CONFIGURATION (LIMIT_REQ AND LIMIT_CONN) 762 763nginx has two rate limiting modules: 764 765* **`limit_req`**: requests per second per key. The most common. 766* **`limit_conn`**: simultaneous connections per key. 767 768Both produce 503 by default; both should be configured for 429 in API contexts. 769 770### 9.1 The Leaky Bucket Model 771 772`limit_req` implements a leaky bucket. Imagine a bucket that fills with incoming requests. The bucket leaks at the configured rate. If the bucket overflows, additional requests are rejected (or queued). 773 774Configuration: 775 776```nginx 777http { 778 # Zone definition: 10MB shared memory, rate 10 requests per second 779 limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s; 780} 781 782server { 783 location /api/ { 784 # Apply zone, allow burst of 20 above the rate 785 limit_req zone=api_limit burst=20 nodelay; 786 limit_req_status 429; 787 } 788} 789``` 790 791What this configuration does: 792 793* `rate=10r/s`: the bucket leaks 10 requests per second. 794* `burst=20`: the bucket can hold 20 requests above the steady rate before rejecting. 795* `nodelay`: process burst requests immediately (no queuing). If the bucket is full, reject. 796 797Without `nodelay`, burst requests are queued and processed at the leaked rate (spacing them out). With `nodelay`, burst requests are processed immediately as long as they fit in the bucket. 798 799### 9.2 The Two Stage Pattern (nginx 1.15.7+) 800 801The `delay` parameter (introduced 1.15.7) gives a middle ground: some burst is processed immediately, the rest is delayed: 802 803```nginx 804location /api/ { 805 limit_req zone=api_limit burst=20 delay=10; 806} 807``` 808 809This means: 810 811* First 10 burst requests: processed immediately (no delay). 812* Next 10 burst requests: queued and spaced at the rate. 813* Beyond burst 20: rejected. 814 815Useful for accommodating typical web browser request patterns (initial page load with concurrent asset fetches) while still throttling sustained abuse. 816 817### 9.3 The Three Common Patterns 818 819**Pattern 1: API endpoint (strict, return 429):** 820 821```nginx 822http { 823 limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s; 824 limit_req_status 429; 825} 826 827location /api/ { 828 limit_req zone=api burst=60 nodelay; 829} 830``` 831 832**Pattern 2: Login endpoint (very strict, prevent brute force):** 833 834```nginx 835http { 836 limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s; 837 limit_req_status 429; 838} 839 840location /api/login { 841 limit_req zone=login burst=5 nodelay; 842} 843``` 844 845**Pattern 3: General site (lenient, accommodate browsers):** 846 847```nginx 848http { 849 limit_req_zone $binary_remote_addr zone=general:10m rate=100r/s; 850 limit_req_status 429; 851} 852 853location / { 854 limit_req zone=general burst=200 delay=100; 855} 856``` 857 858### 9.4 The Connection Limit Module 859 860`limit_conn` controls simultaneous connections rather than request rate. Useful against download abuse (someone with 100 parallel connections downloading a large file): 861 862```nginx 863http { 864 limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m; 865 limit_conn_status 429; 866} 867 868location /downloads/ { 869 limit_conn conn_per_ip 5; 870} 871``` 872 873Limit: any single IP may have at most 5 simultaneous connections to `/downloads/`. 874 875### 9.5 Whitelisting Trusted IPs And Crawlers 876 877Use a `geo` plus `map` combination to skip rate limiting for trusted sources: 878 879```nginx 880http { 881 geo $rate_limit_skip { 882 default 1; 883 884 # Bubbles operator IPs 885 100.90.0.0/16 0; # Tailscale subnet 886 127.0.0.1 0; 887 ::1 0; 888 889 # Office VPN range 890 # 198.51.100.0/24 0; 891 } 892 893 map $rate_limit_skip $rate_limit_key { 894 0 ""; # whitelisted: empty key skips rate limit 895 1 $binary_remote_addr; # rate limited: use IP as key 896 } 897 898 limit_req_zone $rate_limit_key zone=api:10m rate=30r/s; 899 limit_req_status 429; 900} 901``` 902 903When `$rate_limit_skip` is 0 (whitelisted), `$rate_limit_key` is empty, and nginx skips rate limiting entirely for that request. 904 905### 9.6 Setting Retry-After On limit_req Rejections 906 907nginx does not automatically set `Retry-After` on `limit_req` rejections. Use a custom error page: 908 909```nginx 910http { 911 limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s; 912 limit_req_status 429; 913} 914 915server { 916 location /api/ { 917 limit_req zone=api burst=60 nodelay; 918 919 proxy_pass http://127.0.0.1:9090; 920 } 921 922 error_page 429 = @rate_limited; 923 924 location @rate_limited { 925 internal; 926 add_header Retry-After "60" always; 927 add_header Content-Type "application/json" always; 928 return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}'; 929 } 930} 931``` 932 933The `@rate_limited` named location returns a JSON error body with Retry-After set. For HTML responses, swap the JSON for an HTML body. 934 935### 9.7 Logging Rate Limit Events 936 937Default nginx logs rate limit events at `error` level, which is noisy. Configure `warn` or `notice`: 938 939```nginx 940http { 941 limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s; 942 limit_req_status 429; 943 limit_req_log_level warn; 944} 945``` 946 947And ensure the error log level is at least `warn`: 948 949```nginx 950error_log /var/log/nginx/error.log warn; 951``` 952 953Now rate limit events appear in the error log at warn level but not the noisier info events. 954 955### 9.8 The Dry Run Mode 956 957nginx 1.17.1+ supports `limit_req_dry_run on`: log what would have been rate limited without actually rejecting. Useful for tuning limits before enforcing: 958 959```nginx 960location /api/ { 961 limit_req zone=api burst=60 nodelay; 962 limit_req_dry_run on; # log but do not reject 963} 964``` 965 966Run with dry_run for a week, observe the logs, adjust limits, then disable dry_run. 967 968### 9.9 The Default Zone Sizing 969 970`zone=name:10m` means 10 MB of shared memory. nginx documentation states 1 MB holds approximately 16,000 IP addresses. 10 MB holds about 160,000. For most Bubbles sites this is generous. For very high traffic sites, increase to 50m or 100m. 971 972### 9.10 The 503 To 429 Conversion (One Line Fix) 973 974The single most common configuration mistake is leaving the default 503. The fix: 975 976```nginx 977http { 978 # Change default 503 to 429 for all limit_req and limit_conn rejections 979 limit_req_status 429; 980 limit_conn_status 429; 981} 982``` 983 984This single pair of lines should be in `nginx.conf` for every Bubbles deployment that serves APIs. 985 986--- 987 988## 10. THE CRAWLER PROTECTION RULE (NEVER RATE LIMIT GOOGLEBOT, CLAUDEBOT, GPTBOT) 989 990The single most ranking critical decision in this framework: **do not rate limit verified search and AI crawlers**. The cost of getting it wrong is index drops, ranking loss, and AI invisibility. 991 992### 10.1 The Crawlers To Protect 993 994Bubbles client sites depend on these crawlers for organic and AI visibility: 995 996* **Googlebot** (search.google.com): the dominant search crawler. 997* **Bingbot** (bing.com): the second search engine, also feeds DuckDuckGo and ChatGPT search. 998* **ClaudeBot, Claude-Web** (anthropic.com): Anthropic's crawlers. 999* **GPTBot, OAI-SearchBot** (openai.com): OpenAI's crawlers. 1000* **PerplexityBot, Perplexity-User** (perplexity.ai): Perplexity. 1001* **AmazonBot, Amazonbot** (amazon.com): for Alexa and product search. 1002* **Applebot** (apple.com): for Spotlight and Siri. 1003* **YandexBot, Baiduspider**: for international SEO (less relevant for NWA/SWMO clients). 1004 1005### 10.2 Verifying Crawler Identity (User Agent Is Not Enough) 1006 1007User Agent strings are trivially spoofed. Real verification uses reverse DNS plus forward DNS: 1008 1009```bash 1010# 1. Resolve the request's IP back to a hostname 1011host 66.249.66.1 1012# Expected for Googlebot: crawl-66-249-66-1.googlebot.com 1013 1014# 2. Resolve that hostname forward to an IP 1015host crawl-66-249-66-1.googlebot.com 1016# Expected: 66.249.66.1 (matches original) 1017 1018# If both match AND the hostname is in *.googlebot.com or *.google.com, it is genuine Googlebot 1019``` 1020 1021For ClaudeBot, Anthropic publishes their IP ranges. For GPTBot, OpenAI publishes IP ranges. For Bingbot, the verification is similar to Googlebot (reverse DNS to msn.com or search.msn.com). 1022 1023### 10.3 The nginx Pattern: User Agent Plus IP Whitelist 1024 1025For Bubbles deployments, the practical approach combines user agent matching (for AI crawlers without easy reverse DNS verification) with a known good IP list: 1026 1027```nginx 1028http { 1029 # Detect crawlers by user agent (basic but useful) 1030 map $http_user_agent $is_crawler { 1031 default 0; 1032 ~*googlebot 1; 1033 ~*bingbot 1; 1034 ~*claudebot 1; 1035 ~*claude-web 1; 1036 ~*gptbot 1; 1037 ~*oai-searchbot 1; 1038 ~*chatgpt-user 1; 1039 ~*perplexitybot 1; 1040 ~*perplexity-user 1; 1041 ~*amazonbot 1; 1042 ~*applebot 1; 1043 ~*duckduckbot 1; 1044 ~*facebookexternalhit 1; 1045 ~*twitterbot 1; 1046 ~*linkedinbot 1; 1047 } 1048 1049 # Optionally verify crawler IPs as additional check 1050 geo $is_crawler_ip { 1051 default 0; 1052 66.249.64.0/19 1; # Googlebot range (one of several) 1053 # Add Bing, Claude, OpenAI ranges as needed 1054 } 1055 1056 # Skip rate limiting if either user agent OR IP matches 1057 map "$is_crawler$is_crawler_ip" $rate_limit_skip { 1058 "00" 0; # not a crawler, apply rate limit 1059 default 1; # any crawler signal, skip rate limit 1060 } 1061 1062 map $rate_limit_skip $rate_limit_key { 1063 1 ""; # crawler: empty key skips rate limit 1064 0 $binary_remote_addr; # normal: use IP 1065 } 1066 1067 limit_req_zone $rate_limit_key zone=api:10m rate=30r/s; 1068 limit_req_status 429; 1069} 1070``` 1071 1072This is permissive (allows anyone claiming to be a crawler to bypass rate limits via spoofed UA), but the risk is minimal for non sensitive endpoints. For sensitive endpoints (login, payment, admin), require strict IP verification. 1073 1074### 10.4 The Bubbles Conservative Pattern: Whitelist UA Match Plus Generous Limits 1075 1076For all Bubbles sites, the recommended pattern is permissive UA matching combined with generous overall limits: 1077 1078```nginx 1079# Skip rate limiting for any UA matching a known crawler 1080map $http_user_agent $rate_limit_skip { 1081 default 0; 1082 ~*googlebot 1; 1083 ~*bingbot 1; 1084 ~*claudebot 1; 1085 ~*gptbot 1; 1086 ~*oai-searchbot 1; 1087 ~*perplexitybot 1; 1088 ~*applebot 1; 1089 ~*amazonbot 1; 1090} 1091 1092map $rate_limit_skip $rate_limit_key { 1093 1 ""; 1094 0 $binary_remote_addr; 1095} 1096 1097# Generous limits for everyone else 1098limit_req_zone $rate_limit_key zone=site:10m rate=10r/s; 1099limit_req_status 429; 1100 1101server { 1102 location / { 1103 limit_req zone=site burst=50 delay=20; 1104 } 1105} 1106``` 1107 1108If a malicious client spoofs a crawler UA to bypass rate limiting, the worst case is they get faster access to public site content (no real harm). For locations that DO need to apply to all (login, admin, payment), use a separate zone without UA whitelist. 1109 1110### 10.5 The 503 Trap During Maintenance 1111 1112If maintenance returns 503 for everyone (including Googlebot), the 2-day rule applies. The mitigation: 1113 1114```nginx 1115# During maintenance: serve 503 to humans, 200 to crawlers (with maintenance notice) 1116map $http_user_agent $maintenance_response { 1117 default "503"; 1118 ~*googlebot "200"; # Googlebot sees 200 with notice 1119 ~*bingbot "200"; 1120 ~*claudebot "200"; 1121 ~*gptbot "200"; 1122} 1123 1124server { 1125 if ($maintenance_response = "503") { 1126 return 503; 1127 } 1128 # Crawlers fall through to normal handling (or a maintenance notice page) 1129} 1130``` 1131 1132This is unconventional and some SEO purists object. The alternative is to keep maintenance under 24 hours so the 2-day rule does not trigger, which is the cleaner approach. 1133 1134### 10.6 How To Verify Crawler Bypass 1135 1136```bash 1137# Simulate Googlebot user agent 1138curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 1139 https://example.com/ | head -3 1140 1141# Try to trigger rate limit as Googlebot (should NOT see 429) 1142for i in $(seq 1 200); do 1143 STATUS=$(curl -so /dev/null -w "%{http_code}" \ 1144 -A "Mozilla/5.0 (compatible; Googlebot/2.1)" \ 1145 https://example.com/) 1146 if [ "$STATUS" != "200" ]; then 1147 echo "Googlebot got: $STATUS at request $i (BAD)" 1148 break 1149 fi 1150done 1151echo "Googlebot completed all requests at 200 (GOOD)" 1152 1153# Same test with a regular UA (should eventually 429) 1154for i in $(seq 1 200); do 1155 STATUS=$(curl -so /dev/null -w "%{http_code}" \ 1156 -A "Mozilla/5.0 (BadBot/1.0)" \ 1157 https://example.com/) 1158 if [ "$STATUS" != "200" ]; then 1159 echo "BadBot got: $STATUS at request $i (correctly throttled)" 1160 break 1161 fi 1162done 1163``` 1164 1165--- 1166 1167## 11. CLIENT BACKOFF PATTERNS (EXPONENTIAL, JITTER, HONORING RETRY-AFTER) 1168 1169The server only emits headers; the client must read and honor them. Documenting client side patterns is part of API design. 1170 1171### 11.1 The Basic Retry Pattern 1172 1173```python 1174import requests 1175import time 1176 1177def request_with_retry(method, url, max_attempts=5, **kwargs): 1178 """Honor Retry-After; fall back to exponential backoff.""" 1179 for attempt in range(max_attempts): 1180 response = requests.request(method, url, **kwargs) 1181 1182 if response.status_code in (429, 503): 1183 retry_after = response.headers.get("Retry-After") 1184 if retry_after: 1185 # Honor server's directive 1186 try: 1187 delay = int(retry_after) 1188 except ValueError: 1189 # HTTP date format 1190 from email.utils import parsedate_to_datetime 1191 from datetime import datetime, timezone 1192 target = parsedate_to_datetime(retry_after) 1193 delay = max(0, (target - datetime.now(timezone.utc)).total_seconds()) 1194 else: 1195 # Fall back to exponential backoff 1196 delay = 2 ** attempt 1197 1198 time.sleep(delay) 1199 continue 1200 1201 return response 1202 1203 return response 1204``` 1205 1206### 11.2 Adding Jitter 1207 1208If many clients hit a 429 simultaneously and all read the same Retry-After, they all retry at the same time, triggering another wave of 429s. Adding random jitter spreads them out: 1209 1210```python 1211import random 1212 1213def jittered_delay(retry_after_seconds): 1214 """Add up to 25% random delay to spread retries.""" 1215 jitter = random.uniform(0, 0.25) * retry_after_seconds 1216 return retry_after_seconds + jitter 1217``` 1218 1219### 11.3 Proactive Self Throttling Using RateLimit-Remaining 1220 1221Better than reactive retry on 429: read RateLimit-Remaining and slow down before hitting zero: 1222 1223```python 1224def request_with_self_throttle(method, url, **kwargs): 1225 """Self throttle based on remaining quota.""" 1226 response = requests.request(method, url, **kwargs) 1227 1228 remaining = int(response.headers.get("X-RateLimit-Remaining", "999")) 1229 limit = int(response.headers.get("X-RateLimit-Limit", "1")) 1230 reset = response.headers.get("X-RateLimit-Reset") 1231 1232 # Slow down if running low 1233 if remaining < limit * 0.1: # under 10% 1234 # Calculate seconds until reset 1235 if reset: 1236 try: 1237 # Unix epoch format 1238 seconds_until_reset = int(reset) - int(time.time()) 1239 except (ValueError, OverflowError): 1240 seconds_until_reset = 60 1241 1242 # Spread remaining quota over time to reset 1243 if remaining > 0: 1244 delay = max(1, seconds_until_reset / remaining) 1245 time.sleep(delay) 1246 1247 return response 1248``` 1249 1250### 11.4 Honoring The Standardized Format 1251 1252For clients written for the IETF standardization track: 1253 1254```python 1255def get_retry_delay(response): 1256 """Get retry delay from any of the rate limit header forms.""" 1257 # Try Retry-After first (standardized) 1258 retry_after = response.headers.get("Retry-After") 1259 if retry_after: 1260 try: 1261 return int(retry_after) 1262 except ValueError: 1263 pass # date format, parse separately 1264 1265 # Try RateLimit-Reset (IETF standard, delta seconds) 1266 reset = response.headers.get("RateLimit-Reset") 1267 if reset: 1268 return int(reset) 1269 1270 # Fall back to X-RateLimit-Reset (could be epoch) 1271 reset = response.headers.get("X-RateLimit-Reset") 1272 if reset: 1273 delta = int(reset) - int(time.time()) 1274 if delta > 0: 1275 return delta 1276 return int(reset) # might already be delta 1277 1278 # No header, fall back to default 1279 return 60 1280``` 1281 1282--- 1283 1284## 12. HOW THESE HEADERS INTERACT 1285 1286Several specific interactions matter. 1287 1288### 12.1 Retry-After And RateLimit-Reset Both On The Same Response 1289 1290For a 429 response, both are appropriate: 1291 1292``` 1293HTTP/1.1 429 Too Many Requests 1294Retry-After: 60 1295RateLimit-Limit: 100 1296RateLimit-Remaining: 0 1297RateLimit-Reset: 60 1298RateLimit-Policy: 100;w=60 1299``` 1300 1301Retry-After is the standardized signal; RateLimit-Reset is the standardized rate limit detail. Both communicate "60 seconds until you can retry" in compatible ways. Smart clients prefer Retry-After (it is the RFC). 1302 1303### 12.2 Vary: Authorization For Token Based Limits 1304 1305If the rate limit varies per authorization token (which it usually does in multi tenant APIs), the response varies based on the Authorization header. Add Vary: 1306 1307```nginx 1308add_header Vary "Authorization" always; 1309``` 1310 1311Without this, intermediate caches may serve cached rate limit data approved for token A to a request bearing token B. 1312 1313### 12.3 CORS Expose-Headers Requirement 1314 1315For cross origin requests to read the rate limit headers from JavaScript, they must be in Access-Control-Expose-Headers (see framework-http-cors-headers.md): 1316 1317```nginx 1318add_header Access-Control-Expose-Headers "X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, RateLimit-Policy, Retry-After" always; 1319``` 1320 1321Without exposure, browser JavaScript sees null for these headers even though the network panel shows them. 1322 1323### 12.4 Cache-Control On 429 And 503 Responses 1324 1325Always pair with `Cache-Control: no-store` to prevent intermediate caches from serving stale errors: 1326 1327```nginx 1328location @rate_limited { 1329 add_header Retry-After "60" always; 1330 add_header Cache-Control "no-store, no-cache, must-revalidate" always; 1331 return 429 '...'; 1332} 1333``` 1334 1335A cached 429 served to a different client (one who has not hit their limit) creates a confusing failure. A cached 503 served after maintenance ends extends the outage. 1336 1337### 12.5 X-Robots-Tag On Rate Limited Responses 1338 1339Add `X-Robots-Tag: noindex` to 429 and 503 responses so search engines do not accidentally index the error page: 1340 1341```nginx 1342location @rate_limited { 1343 add_header Retry-After "60" always; 1344 add_header X-Robots-Tag "noindex, nofollow" always; 1345 return 429 '...'; 1346} 1347``` 1348 1349This pairs with framework-http-seo-headers.md. 1350 1351### 12.6 The limit_req And FastAPI Coordination 1352 1353If both nginx and FastAPI implement rate limiting, they must agree on policy or one will mask the other: 1354 1355* **nginx layer**: per IP, blunt instrument, fast (no application traversal). 1356* **FastAPI layer**: per token, fine grained, more expensive (runs in application). 1357 1358Recommendation: nginx for crude DDoS protection (very generous limits, block obvious abuse). FastAPI for per token / per user limits announced via headers. They serve different purposes and do not conflict. 1359 1360--- 1361 1362## 13. ASSET CLASS AND USE CASE RECIPES 1363 1364Paste ready blocks per scenario. 1365 1366### 13.1 Standard Bubbles API rate limit (per IP, return 429) 1367 1368```nginx 1369# In http block of nginx.conf 1370map $http_user_agent $rate_limit_skip { 1371 default 0; 1372 ~*googlebot 1; 1373 ~*bingbot 1; 1374 ~*claudebot 1; 1375 ~*gptbot 1; 1376 ~*oai-searchbot 1; 1377 ~*perplexitybot 1; 1378} 1379 1380map $rate_limit_skip $rate_limit_key { 1381 1 ""; 1382 0 $binary_remote_addr; 1383} 1384 1385limit_req_zone $rate_limit_key zone=api:10m rate=30r/s; 1386limit_req_status 429; 1387limit_req_log_level warn; 1388``` 1389 1390```nginx 1391# In server block 1392location /api/ { 1393 limit_req zone=api burst=60 nodelay; 1394 proxy_pass http://127.0.0.1:9090; 1395} 1396 1397error_page 429 = @rate_limited; 1398 1399location @rate_limited { 1400 internal; 1401 add_header Retry-After "60" always; 1402 add_header Content-Type "application/json" always; 1403 add_header Cache-Control "no-store" always; 1404 add_header X-Robots-Tag "noindex" always; 1405 return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}'; 1406} 1407``` 1408 1409### 13.2 Login endpoint (strict, prevent brute force) 1410 1411```nginx 1412# In http block 1413limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s; 1414limit_req_status 429; 1415``` 1416 1417```nginx 1418# In server block 1419location /api/login { 1420 limit_req zone=login burst=5 nodelay; 1421 proxy_pass http://127.0.0.1:9090; 1422} 1423 1424location /api/password-reset { 1425 limit_req zone=login burst=3 nodelay; 1426 proxy_pass http://127.0.0.1:9090; 1427} 1428``` 1429 1430Plus paired with fail2ban for IPs that persistently hit the login limit: 1431 1432```ini 1433# /etc/fail2ban/jail.d/nginx-rate-limit.conf 1434[nginx-rate-limit] 1435enabled = true 1436filter = nginx-rate-limit 1437action = iptables-multiport[name=NginxRateLimit, port="http,https"] 1438logpath = /var/log/nginx/error.log 1439maxretry = 10 1440findtime = 600 1441bantime = 3600 1442``` 1443 1444### 13.3 Static site general protection (lenient, accommodate browsers) 1445 1446```nginx 1447limit_req_zone $rate_limit_key zone=site:10m rate=20r/s; 1448limit_req_status 429; 1449``` 1450 1451```nginx 1452server { 1453 location / { 1454 limit_req zone=site burst=50 delay=25; 1455 try_files $uri $uri/ $uri.html =404; 1456 } 1457 1458 error_page 429 = @rate_limited_html; 1459 1460 location @rate_limited_html { 1461 internal; 1462 add_header Retry-After "30" always; 1463 add_header Cache-Control "no-store" always; 1464 return 429 '<html><body><h1>Slow down!</h1><p>Too many requests. Try again in 30 seconds.</p></body></html>'; 1465 } 1466} 1467``` 1468 1469### 13.4 Download protection (limit_conn for parallel downloads) 1470 1471```nginx 1472limit_conn_zone $binary_remote_addr zone=download_conn:10m; 1473limit_conn_status 429; 1474``` 1475 1476```nginx 1477location /downloads/ { 1478 limit_conn download_conn 3; 1479 limit_rate 1m; # 1 MB/s per connection cap as a bonus 1480} 1481 1482error_page 429 = @too_many_conns; 1483 1484location @too_many_conns { 1485 internal; 1486 add_header Retry-After "60" always; 1487 return 429 '{"error": "too_many_connections", "max": 3}'; 1488} 1489``` 1490 1491### 13.5 Scheduled maintenance with Retry-After and Googlebot 200 fallback 1492 1493```nginx 1494map $http_user_agent $maintenance_response { 1495 default "503"; 1496 ~*googlebot "200"; 1497 ~*bingbot "200"; 1498 ~*claudebot "200"; 1499 ~*gptbot "200"; 1500 ~*perplexitybot "200";