A continuation of Janady13/crawler-signal-reference / framework-http-rate-control-headers.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓2,287 lines · 82.8 KB
}
server {
location / {
if ($maintenance_response = "503") {
return 503;
}
# Crawlers see the maintenance notice with 200 status
root /var/www/sites/example.com;
try_files /maintenance.html =503;
}
error_page 503 /maintenance.html;
location = /maintenance.html {
internal;
add_header Retry-After "1800" always; # 30 minutes
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
root /var/www/sites/example.com;
}
}
```
### 13.6 FastAPI sidecar with proper rate limit headers using slowapi
```python
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
app = FastAPI()
limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter
@app.exception_handler(RateLimitExceeded)
async def rate_limit_handler(request: Request, exc: RateLimitExceeded):
return JSONResponse(
status_code=429,
content={"error": "rate_limit_exceeded"},
headers={
"Retry-After": "60",
"X-RateLimit-Limit": "100",
"X-RateLimit-Remaining": "0",
"RateLimit-Limit": "100",
"RateLimit-Remaining": "0",
"RateLimit-Reset": "60",
"RateLimit-Policy": "100;w=60",
}
)
@app.get("/api/data")
@limiter.limit("100/minute")
async def data(request: Request):
return {"data": "..."}
```
### 13.7 thatwebhostingguy.com wildcard subdomain protection
```nginx
# Wildcard subdomains share a single rate limit zone
limit_req_zone $rate_limit_key zone=wildcard:50m rate=50r/s;
limit_req_status 429;
server {
server_name *.thatwebhostingguy.com;
location / {
limit_req zone=wildcard burst=100 delay=50;
# ... rest of config ...
}
}
```
50 MB zone holds approximately 800,000 IPs which is generous for the demo platform.
### 13.8 CORS exposed rate limit headers for SPA
```nginx
# For an API consumed by a JS frontend on a different origin
add_header Access-Control-Expose-Headers "X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, RateLimit-Policy, Retry-After" always;
add_header Vary "Origin, Authorization" always;
```
The frontend can now read rate limit state and self throttle:
```javascript
async function apiCall(url, options) {
const response = await fetch(url, options);
const remaining = parseInt(response.headers.get("X-RateLimit-Remaining") || "999");
const limit = parseInt(response.headers.get("X-RateLimit-Limit") || "1");
if (remaining < limit * 0.1) {
console.warn(`Rate limit at ${remaining}/${limit}, slowing down`);
}
if (response.status === 429) {
const retryAfter = parseInt(response.headers.get("Retry-After") || "60");
await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
return apiCall(url, options);
}
return response;
}
```
### 13.9 Search endpoint with permissive crawler access
```nginx
location /search {
# Crawlers indexing search pages should not be throttled
if ($is_crawler) {
proxy_pass http://127.0.0.1:9090;
break;
}
# Regular users: rate limited
limit_req zone=search burst=30 delay=15;
proxy_pass http://127.0.0.1:9090;
}
```
Note: `if ($is_crawler) { proxy_pass; break; }` is one of the unsafe `if` patterns. Better approach: use map plus a wrapper location, or rely on the global skip pattern from Section 13.1.
### 13.10 Dry run mode for tuning a new rate limit
```nginx
limit_req_zone $rate_limit_key zone=new_api:10m rate=10r/s;
limit_req_status 429;
location /api/v2/ {
limit_req zone=new_api burst=20 nodelay;
limit_req_dry_run on; # log but do not enforce
proxy_pass http://127.0.0.1:9090;
}
```
Run for a week. Check error log for `limiting requests, dry_run` entries. Tune the rate and burst values based on observed traffic, then remove `limit_req_dry_run on;` to enforce.
---
## 14. BUBBLES NGINX REFERENCE BLOCK (PASTE READY)
The complete rate control stanza, layered with the previous six frameworks.
```nginx
# /etc/nginx/nginx.conf (http context)
http {
# Crawler detection
map $http_user_agent $rate_limit_skip {
default 0;
~*googlebot 1;
~*bingbot 1;
~*duckduckbot 1;
~*claudebot 1;
~*claude-web 1;
~*gptbot 1;
~*oai-searchbot 1;
~*chatgpt-user 1;
~*perplexitybot 1;
~*perplexity-user 1;
~*amazonbot 1;
~*applebot 1;
~*facebookexternalhit 1;
~*twitterbot 1;
~*linkedinbot 1;
}
map $rate_limit_skip $rate_limit_key {
1 "";
0 $binary_remote_addr;
}
# Rate limit zones
limit_req_zone $rate_limit_key zone=site:10m rate=20r/s;
limit_req_zone $rate_limit_key zone=api:10m rate=30r/s;
limit_req_zone $rate_limit_key zone=login:10m rate=1r/s;
# Connection limit zone
limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;
# Use 429 instead of 503 for all rate limit rejections
limit_req_status 429;
limit_conn_status 429;
limit_req_log_level warn;
limit_conn_log_level warn;
}
```
```nginx
# /etc/nginx/sites-available/example.com
server {
listen 443 ssl;
listen 443 quic;
http2 on;
http3 on;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_early_data on;
add_header Alt-Svc 'h3=":443"; ma=86400' always;
# Security baseline
include snippets/common-security-headers.conf;
root /var/www/sites/example.com;
index index.html;
# ===== Site general (lenient) =====
location / {
limit_req zone=site burst=50 delay=25;
limit_conn conn_per_ip 20;
try_files $uri $uri/ $uri.html =404;
}
# ===== API endpoints (moderate, return 429) =====
location /api/ {
limit_req zone=api burst=60 nodelay;
limit_conn conn_per_ip 20;
# Per CORS framework
if ($cors_origin) {
add_header Access-Control-Allow-Origin $cors_origin always;
add_header Access-Control-Allow-Credentials "true" always;
add_header Access-Control-Expose-Headers "X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, RateLimit-Policy, Retry-After, Server-Timing, ETag, Link, X-Total-Count" always;
add_header Vary "Origin, Authorization" always;
}
add_header X-Robots-Tag "noindex, nofollow" always;
proxy_pass http://127.0.0.1:9090;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# ===== Login (strict) =====
location /api/login {
limit_req zone=login burst=5 nodelay;
# All other API headers inherited or repeated
proxy_pass http://127.0.0.1:9090;
}
location /api/password-reset {
limit_req zone=login burst=3 nodelay;
proxy_pass http://127.0.0.1:9090;
}
# ===== Downloads (connection limited) =====
location /downloads/ {
limit_conn conn_per_ip 3;
limit_rate_after 5m;
limit_rate 2m; # 2 MB/s after first 5 MB
# Other download specific config
}
# ===== Error handlers =====
error_page 429 = @rate_limited;
error_page 503 = @maintenance;
location @rate_limited {
internal;
add_header Retry-After "60" always;
add_header Content-Type "application/json" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}';
}
location @maintenance {
internal;
add_header Retry-After "1800" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
try_files /maintenance.html =503;
}
}
```
After deploying:
```bash
nginx -t && systemctl reload nginx
```
Verify:
```bash
# Crawler bypass works
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://example.com/ | head -3
# Rate limit triggers and returns 429 with Retry-After
for i in $(seq 1 100); do
curl -so /dev/null -w "%{http_code} " https://example.com/api/data
done; echo
```
---
## 15. AUDIT CHECKLIST
Run through these 50 items for any production rate control configuration.
### Retry-After
1. [ ] Retry-After present on every 429 response.
2. [ ] Retry-After present on every 503 response.
3. [ ] Values are positive integers (or valid HTTP dates).
4. [ ] Values reflect actual quota window reset times (not arbitrary or stale).
5. [ ] No Retry-After on success responses (2xx) where it would be confusing.
6. [ ] 503 responses paired with `Cache-Control: no-store` to prevent stale errors.
7. [ ] Maintenance pages return 503 with Retry-After AND complete in under 48 hours.
### X-RateLimit-* family
8. [ ] X-RateLimit-Limit set on every API response.
9. [ ] X-RateLimit-Remaining decrements correctly across requests.
10. [ ] X-RateLimit-Reset format documented (Unix epoch vs delta seconds).
11. [ ] Headers added to Access-Control-Expose-Headers for cross origin access.
12. [ ] Per token rate limits implemented (not just per IP) where authentication exists.
### IETF RateLimit-* family
13. [ ] RateLimit-Limit emitted in parallel with X-RateLimit-Limit.
14. [ ] RateLimit-Remaining emitted in parallel with X-RateLimit-Remaining.
15. [ ] RateLimit-Reset uses delta seconds format (matches standard).
16. [ ] RateLimit-Policy describes the policy (quota + window).
17. [ ] Both X- and un prefixed forms exposed via CORS.
### Status codes
18. [ ] Nginx `limit_req_status 429` set (not the default 503).
19. [ ] Nginx `limit_conn_status 429` set (not the default 503).
20. [ ] 429 used for rate limiting (per client quota).
21. [ ] 503 used for service outage (whole service unavailable).
22. [ ] No 503 served for more than 48 hours (to avoid Google deindex).
### Nginx configuration
23. [ ] limit_req zones defined with reasonable sizes (10m default).
24. [ ] limit_req_zone keys appropriate ($binary_remote_addr, not $remote_addr).
25. [ ] Burst parameter set on every limit_req usage.
26. [ ] Use of nodelay or delay matches the use case (API: nodelay; browsers: delay).
27. [ ] Log level set to `warn` not the default `error`.
28. [ ] error_log level supports warn (`error_log /var/log/nginx/error.log warn`).
29. [ ] Custom 429 error page returns Retry-After.
30. [ ] Custom 503 error page returns Retry-After.
### Crawler protection
31. [ ] User agent whitelist for known crawlers (Googlebot, Bingbot, ClaudeBot, GPTBot, PerplexityBot, etc).
32. [ ] Crawler whitelist applied via `map` to skip rate limiting.
33. [ ] Tested that Googlebot UA bypasses rate limits.
34. [ ] Sensitive endpoints (login, payment) NOT whitelisted by UA (strict on those).
35. [ ] No 429/503 served to Googlebot in normal operation.
### Application layer
36. [ ] FastAPI sidecar emits rate limit headers (when responsible for per token limits).
37. [ ] Rate limit storage is persistent (Redis or similar), not in process memory.
38. [ ] Rate limit keys are sensible (per token preferred over per IP for authenticated APIs).
39. [ ] Different endpoints have different limits where appropriate (cheap vs expensive).
### Client coordination
40. [ ] API documentation describes rate limit headers and format.
41. [ ] API documentation includes example backoff code.
42. [ ] Client libraries (if shipped) honor Retry-After.
43. [ ] Client libraries add jitter to retry delays.
### Cross cutting
44. [ ] Vary: Authorization set when rate limit is per token.
45. [ ] X-Robots-Tag: noindex on 429 and 503 error pages.
46. [ ] No Set-Cookie on 429 responses (avoid setting cookies for failed requests).
47. [ ] No 429 cached by intermediate proxies (Cache-Control: no-store).
48. [ ] Monitoring distinguishes 429 (expected) from 503 (problem).
49. [ ] Alerts on sustained 503 above threshold but not on routine 429.
50. [ ] Quarterly review of rate limit thresholds against actual traffic patterns.
A site that passes all 50 has correctly configured rate control for production.
---
## 16. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: Nginx returning 503 for rate limited requests.**
Symptom: monitoring alerts on "service unavailable" rate but the service is fine.
Why it breaks: nginx defaults to 503 for `limit_req` rejections.
Fix: `limit_req_status 429;` in http block.
**Pitfall 2: Googlebot getting rate limited for two days, pages drop from index.**
Symptom: rankings collapse, search traffic drops, GSC shows crawl errors.
Why it breaks: no crawler whitelist; aggressive limit applied to everyone equally.
Fix: implement user agent whitelist (Section 10.3). Re submit affected URLs via GSC after fix.
**Pitfall 3: 429 with no Retry-After.**
Symptom: clients retry immediately, triggering more 429s, ratchet effect.
Why it breaks: clients have no signal to wait. Default behavior is "retry now".
Fix: always set Retry-After on 429 responses.
**Pitfall 4: Maintenance 503 cached by browser.**
Symptom: site is back up but users still see 503 from cache.
Why it breaks: missing Cache-Control: no-store.
Fix: add `add_header Cache-Control "no-store" always;` to 503 responses.
**Pitfall 5: X-RateLimit headers invisible to JavaScript.**
Symptom: frontend shows null for response.headers.get("X-RateLimit-Remaining").
Why it breaks: missing from Access-Control-Expose-Headers.
Fix: add to Expose-Headers in CORS configuration.
**Pitfall 6: Per IP limits break office networks.**
Symptom: large office reports "rate limit exceeded" while individual users are not heavy.
Why it breaks: many users share one NAT IP; per IP limits are too low for the aggregate.
Fix: increase per IP limits for unauthenticated endpoints; use per token limits for authenticated.
**Pitfall 7: X-RateLimit-Reset format confusion.**
Symptom: client backoff uses wrong duration (treating epoch as seconds, or vice versa).
Why it breaks: no standard for this header; format is API specific.
Fix: document the format. Use Unix epoch for X-, delta for RateLimit-Reset. Both makes intent clear.
**Pitfall 8: Sustained 503 for over 48 hours during scheduled work.**
Symptom: Google deindexes URLs.
Why it breaks: 2-day rule.
Fix: keep maintenance under 48 hours, OR return 200 with maintenance notice for crawlers.
**Pitfall 9: Rate limit zone too small, hash collisions.**
Symptom: legitimate clients occasionally rate limited despite being under threshold.
Why it breaks: 10m zone overflows for very high traffic sites, causing key eviction.
Fix: increase zone size (50m, 100m) for high traffic.
**Pitfall 10: Using `if ($is_crawler) { ... }` with proxy_pass.**
Symptom: unexpected behavior, sometimes works sometimes does not.
Why it breaks: `if` plus `proxy_pass` is the nginx if antipattern.
Fix: use map plus separate locations, or the global skip pattern.
**Pitfall 11: Login endpoint with no special rate limit.**
Symptom: credential stuffing attack succeeds because attacker can try thousands of password combinations per minute.
Why it breaks: general rate limit (10r/s) is too permissive for login.
Fix: separate strict zone for login (1r/s, burst 5, nodelay).
**Pitfall 12: limit_conn defending against parallel download misses HTTP/2.**
Symptom: client uses HTTP/2 multiplexing, all requests come over 1 connection, bypasses limit_conn.
Why it breaks: limit_conn counts connections, HTTP/2 multiplexes many requests per connection.
Fix: combine limit_conn with limit_req for layered defense.
**Pitfall 13: Counter resets on application restart.**
Symptom: rate limits become ineffective after every deploy; clients see Limit reset to maximum.
Why it breaks: in process counters lost on restart.
Fix: persist counters in Redis or similar shared storage.
**Pitfall 14: Rate limit headers absent on error responses.**
Symptom: client got 500 and has no info about quota state.
Why it breaks: error responses bypass middleware that adds the headers.
Fix: add headers in exception handlers as well.
**Pitfall 15: Retry-After value of 0.**
Symptom: client retries immediately, defeating the rate limit.
Why it breaks: integer 0 means no wait.
Fix: minimum value of 1; for very short delays use 1 second.
---
## 17. DIAGNOSTIC COMMANDS
Reference of every command useful for rate control investigation.
### Inspect rate limit headers
```bash
# Single request, all rate control headers
curl -sI -H "Authorization: Bearer test" https://api.example.com/data \
| grep -iE "^(x-ratelimit|ratelimit|retry-after|cache-control|vary)"
# Pretty print
curl -sI -H "Authorization: Bearer test" https://api.example.com/data \
| grep -iE "^(x-ratelimit|ratelimit|retry-after)" \
| sort
```
### Trigger rate limit deliberately
```bash
# Send 200 requests rapidly, count status codes
for i in $(seq 1 200); do
curl -so /dev/null -w "%{http_code}\n" \
-H "Authorization: Bearer test" \
https://api.example.com/data
done | sort | uniq -c
# Expect: most 200, some 429 once limit exceeded
```
### Verify crawler bypass
```bash
# As Googlebot: should not get 429
for i in $(seq 1 500); do
STATUS=$(curl -so /dev/null -w "%{http_code}" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1)" \
https://example.com/)
[ "$STATUS" != "200" ] && { echo "Bad: got $STATUS at $i"; break; }
done
# As regular UA: should eventually 429
for i in $(seq 1 500); do
STATUS=$(curl -so /dev/null -w "%{http_code}" \
https://example.com/)
if [ "$STATUS" = "429" ]; then
echo "Correctly throttled at request $i"
break
fi
done
```
### Check Retry-After value
```bash
# Trigger rate limit then read Retry-After
for i in $(seq 1 100); do curl -so /dev/null https://api.example.com/data; done
curl -sI https://api.example.com/data | grep -iE "retry-after|^HTTP"
# Expected:
# HTTP/2 429
# retry-after: 60
```
### Test exponential backoff with Retry-After
```bash
# Bash retry helper
request_with_retry() {
local url=$1
local attempts=0
while [ $attempts -lt 5 ]; do
local headers=$(curl -sI "$url" 2>/dev/null)
local status=$(echo "$headers" | head -1 | awk '{print $2}')
if [ "$status" = "200" ]; then
echo "Success after $attempts retries"
return 0
fi
if [ "$status" = "429" ] || [ "$status" = "503" ]; then
local retry=$(echo "$headers" | grep -i retry-after | tr -d '\r\n' | awk '{print $2}')
retry=${retry:-60}
echo "Got $status, waiting $retry seconds"
sleep "$retry"
attempts=$((attempts + 1))
continue
fi
echo "Unexpected status: $status"
return 1
done
}
```
### Server side investigation
```bash
# Show rate limit zones
nginx -T 2>/dev/null | grep -E "limit_req_zone|limit_conn_zone"
# Show rate limit usages
nginx -T 2>/dev/null | grep -E "limit_req |limit_conn "
# Show status overrides
nginx -T 2>/dev/null | grep -E "limit_req_status|limit_conn_status"
# Check crawler whitelist exists
nginx -T 2>/dev/null | grep -A 30 "map \$http_user_agent"
# Find rate limit events in log
sudo grep -i "limiting requests" /var/log/nginx/error.log | tail -20
# Count rate limit events by IP
sudo grep -i "limiting requests" /var/log/nginx/error.log \
| grep -oE 'client: [0-9.]+' | sort | uniq -c | sort -rn | head -10
# Per minute rate limit event rate
sudo grep -i "limiting requests" /var/log/nginx/error.log \
| awk '{print $1, $2}' | cut -d: -f1,2 | uniq -c | tail -10
# Apply changes
nginx -t && systemctl reload nginx
```
### Browser DevTools quick reference
In Chrome DevTools Network panel:
1. Make a request that hits rate limit.
2. Click the 429 response in the request list.
3. Headers tab shows Retry-After, X-RateLimit-*, RateLimit-*.
4. Console panel: errors related to rate limiting appear.
Useful console commands:
```javascript
// Check current rate limit state
async function checkRateLimit(url) {
const r = await fetch(url, {credentials: "include"});
console.log({
status: r.status,
limit: r.headers.get("X-RateLimit-Limit"),
remaining: r.headers.get("X-RateLimit-Remaining"),
reset: r.headers.get("X-RateLimit-Reset"),
retryAfter: r.headers.get("Retry-After"),
});
}
// Self throttling client
async function safeApiCall(url, options = {}) {
const r = await fetch(url, options);
if (r.status === 429) {
const retry = parseInt(r.headers.get("Retry-After") || "60");
console.warn(`Rate limited, waiting ${retry}s`);
await new Promise(resolve => setTimeout(resolve, retry * 1000));
return safeApiCall(url, options);
}
return r;
}
```
---
## 18. CROSS-REFERENCES
* [framework-http-caching-headers.md](framework-http-caching-headers.md): Cache-Control no-store pairing with 429/503 responses.
* [framework-http-content-headers.md](framework-http-content-headers.md): Content-Type for JSON error bodies on 429.
* [framework-http-seo-headers.md](framework-http-seo-headers.md): X-Robots-Tag noindex on rate limit error pages. The 2-day Google deindex rule is critical context here.
* [framework-http-security-headers.md](framework-http-security-headers.md): rate limiting is complementary to CSP and HSTS for defense in depth.
* [framework-http-performance-headers.md](framework-http-performance-headers.md): Server-Timing can include rate limit lookup time. Server header suppression also applies to error responses.
* [framework-http-cors-headers.md](framework-http-cors-headers.md): Access-Control-Expose-Headers must list rate limit headers for cross origin reads. Vary: Authorization for per token limits.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. The 2-day rule for crawlers is also covered there.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook.
* [framework-fail2ban-deployment.md](framework-fail2ban-deployment.md): pairing with fail2ban for persistent abusers.
* [framework-bot-management.md](framework-bot-management.md): broader bot management strategy including crawler verification.
* RFC 9110 (HTTP Semantics, Retry-After): https://www.rfc-editor.org/rfc/rfc9110
* RFC 6585 (Additional HTTP Status Codes, 429): https://www.rfc-editor.org/rfc/rfc6585
* IETF draft RateLimit headers: https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/
* nginx limit_req module: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html
* nginx limit_conn module: https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html
* Google crawl rate troubleshooting: https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors
* MDN Retry-After: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After
* MDN 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
* MDN 503 Service Unavailable: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Bubbles minimum viable rate control
```nginx
# http context
http {
map $http_user_agent $rate_limit_skip {
default 0;
~*googlebot 1;
~*bingbot 1;
~*claudebot 1;
~*gptbot 1;
~*oai-searchbot 1;
~*perplexitybot 1;
}
map $rate_limit_skip $rate_limit_key {
1 "";
0 $binary_remote_addr;
}
limit_req_zone $rate_limit_key zone=site:10m rate=20r/s;
limit_req_zone $rate_limit_key zone=api:10m rate=30r/s;
limit_req_zone $rate_limit_key zone=login:10m rate=1r/s;
limit_req_status 429;
limit_conn_status 429;
limit_req_log_level warn;
}
# server context
location / {
limit_req zone=site burst=50 delay=25;
}
location /api/ {
limit_req zone=api burst=60 nodelay;
}
location /api/login {
limit_req zone=login burst=5 nodelay;
}
error_page 429 = @rate_limited;
location @rate_limited {
internal;
add_header Retry-After "60" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}';
}
```
### Header purpose table
| Header | One line purpose |
|---|---|
| Retry-After | When to come back after 429/503 |
| X-RateLimit-Limit | Current quota (de facto convention) |
| X-RateLimit-Remaining | Requests left in window |
| X-RateLimit-Reset | When window resets (Unix epoch or delta) |
| RateLimit-Limit | Standardized version of X-RateLimit-Limit |
| RateLimit-Remaining | Standardized version of X-RateLimit-Remaining |
| RateLimit-Reset | Standardized: delta seconds (not epoch) |
| RateLimit-Policy | Quota and window in detail |
### Status code decision
| Situation | Status |
|---|---|
| Specific client over quota | 429 |
| Entire service down | 503 |
| Service down for over 48 hours | NOT 503, serve 200 with notice |
### The 2-day rule
Google drops URLs from the index after 48 hours of sustained 429 or 503. Always:
1. Whitelist Googlebot from rate limits.
2. Keep maintenance windows under 48 hours.
3. For longer downtime, return 200 with a maintenance page.
### Five commands every operator should know
```bash
# 1. View rate limit state
curl -sI -H "Authorization: Bearer test" https://api.example.com/data | grep -iE "ratelimit|retry-after"
# 2. Trigger rate limit
for i in $(seq 1 200); do curl -so /dev/null -w "%{http_code} " https://api.example.com/data; done; echo
# 3. Verify Googlebot bypass
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://example.com/ | head -3
# 4. Check rate limit events in nginx error log
sudo grep "limiting requests" /var/log/nginx/error.log | tail -20
# 5. Apply changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. Rate limit triggers correctly
for i in $(seq 1 500); do
STATUS=$(curl -so /dev/null -w "%{http_code}" https://example.com/api/data)
[ "$STATUS" = "429" ] && { echo "Rate limited at $i"; break; }
done
# 2. Retry-After is present and reasonable
curl -sI -X GET https://api.example.com/data -H "X-Test-Many: yes"
# Look for: retry-after: <positive integer>
# 3. Googlebot does not get throttled
COUNT_429=0
for i in $(seq 1 200); do
STATUS=$(curl -so /dev/null -w "%{http_code}" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1)" \
https://example.com/)
[ "$STATUS" = "429" ] && COUNT_429=$((COUNT_429 + 1))
done
echo "Googlebot 429 count: $COUNT_429 (should be 0)"
```
If all three pass AND no recent Googlebot 429 events in error log, the rate control stack is correctly wired.
---
End of framework-http-rate-control-headers.md.