A continuation of Janady13/crawler-signal-reference / framework-http-5xx-status-codes.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓2,299 lines · 80.8 KB
error_page 500 502 503 504 /5xx.html;
location = /5xx.html {
root /var/www/sites/example.com;
internal;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
}
```
```html
<!-- /var/www/sites/example.com/5xx.html -->
<!doctype html>
<html lang="en">
<head>
<title>Server Error</title>
<meta name="robots" content="noindex">
</head>
<body>
<h1>Something went wrong on our end</h1>
<p>We're investigating. Please try again in a few minutes, or contact admin@thatdeveloperguy.com if the problem persists.</p>
</body>
</html>
```
### 16.7 FastAPI global exception handler
```python
import logging
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
app = FastAPI()
logger = logging.getLogger(__name__)
@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception):
logger.exception(f"Unhandled exception on {request.method} {request.url.path}")
return JSONResponse(
status_code=500,
content={
"error": "internal_server_error",
"request_id": request.headers.get("x-request-id", "unknown"),
},
headers={
"Cache-Control": "no-store",
"X-Robots-Tag": "noindex",
}
)
```
### 16.8 FastAPI 503 with Retry-After for dependency failure
```python
from fastapi import FastAPI, HTTPException
@app.get("/api/data")
async def get_data():
try:
return await db.fetch_one("SELECT ...")
except DatabaseConnectionError as e:
logger.error(f"Database connection failed: {e}")
raise HTTPException(
status_code=503,
detail="database temporarily unavailable",
headers={"Retry-After": "30"}
)
```
### 16.9 nginx upstream with retry on 5xx
```nginx
upstream fastapi_backend {
server 127.0.0.1:9090 max_fails=3 fail_timeout=30s;
keepalive 32;
}
location /api/ {
proxy_pass http://fastapi_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_next_upstream error timeout http_502 http_504;
proxy_next_upstream_tries 2;
}
```
### 16.10 Health check endpoint
```python
@app.get("/healthz")
async def healthz():
# Simple liveness check
return {"status": "ok"}
@app.get("/readyz")
async def readyz():
# Readiness check (verify dependencies)
try:
await db.fetch_one("SELECT 1")
except Exception:
raise HTTPException(status_code=503, detail="database not ready")
return {"status": "ready"}
```
```nginx
location = /healthz {
access_log off;
proxy_pass http://127.0.0.1:9090;
}
```
### 16.11 The Bubbles standard timeout block (template)
```nginx
# Default timeouts for /api/ endpoints
proxy_connect_timeout 5s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
# For long running endpoints, override per location:
# location /api/long-endpoint {
# proxy_read_timeout 300s;
# }
# For streaming endpoints:
# location /api/stream {
# proxy_buffering off;
# proxy_read_timeout 86400s;
# }
```
### 16.12 Systemd service for FastAPI (production grade)
See Section 14.1 for the complete service file.
---
## 17. BUBBLES NGINX REFERENCE BLOCK (PASTE READY)
The complete 5xx aware configuration for a Bubbles client site.
```nginx
# /etc/nginx/sites-available/example.com
upstream fastapi_backend {
server 127.0.0.1:9090 max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen 443 ssl;
listen 443 quic;
http2 on;
http3 on;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_early_data on;
add_header Alt-Svc 'h3=":443"; ma=86400' always;
include snippets/common-security-headers.conf;
root /var/www/sites/example.com;
index index.html;
# ============= STATIC =============
location ~* \.(css|js|woff2|jpg|jpeg|png|webp|avif|svg)$ {
add_header Cache-Control "public, max-age=31536000, immutable" always;
add_header Accept-Ranges "bytes" always;
}
# ============= HEALTH CHECKS =============
location = /healthz {
access_log off;
proxy_pass http://fastapi_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_read_timeout 2s;
}
# ============= API =============
location /api/ {
# OPTIONS preflight (per CORS framework)
if ($request_method = OPTIONS) {
add_header Access-Control-Allow-Origin $cors_origin always;
return 204;
}
add_header X-Robots-Tag "noindex, nofollow" always;
proxy_pass http://fastapi_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Standard timeouts
proxy_connect_timeout 5s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
# Auto retry on transient upstream failure
proxy_next_upstream error timeout http_502 http_504;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 10s;
}
# ============= LONG RUNNING ENDPOINTS =============
location /api/reports/generate {
proxy_pass http://fastapi_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_read_timeout 300s; # 5 minutes for report generation
}
# ============= ROOT =============
location / {
try_files $uri $uri/ $uri.html =404;
}
# ============= 5xx ERROR PAGES =============
error_page 500 502 504 /5xx.html;
error_page 503 /maintenance.html;
location = /5xx.html {
internal;
root /var/www/sites/example.com;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
}
location = /maintenance.html {
internal;
root /var/www/sites/example.com;
add_header Retry-After "3600" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
}
}
```
After deploying:
```bash
nginx -t && systemctl reload nginx
```
Verify:
```bash
# Health check (should be 200)
curl -sI https://example.com/healthz | head -1
# Expected: HTTP/2 200
# Test that timeouts are configured
nginx -T 2>/dev/null | grep -E "proxy_(connect|send|read)_timeout"
# Expected: proxy_connect_timeout 5s; proxy_send_timeout 30s; proxy_read_timeout 30s;
# Run the diagnostic script
sudo bubbles-health-check.sh
```
---
## 18. AUDIT CHECKLIST
Run through these 50 items for any production deployment.
### Systemd self healing
1. [ ] FastAPI sidecar systemd unit file exists.
2. [ ] `Restart=always` is set.
3. [ ] `RestartSec=5` (or similar) prevents thrashing.
4. [ ] `StartLimitInterval` and `StartLimitBurst` configured.
5. [ ] `MemoryMax` and `CPUQuota` limits set.
6. [ ] Service enabled (`systemctl enable fastapi-sidecar`).
7. [ ] Service starts on boot (verified with reboot test).
### Nginx upstream configuration
8. [ ] `upstream` block defined with `max_fails` and `fail_timeout`.
9. [ ] `keepalive 32` (or similar) for connection pooling.
10. [ ] `proxy_next_upstream` configured for retry.
11. [ ] `proxy_connect_timeout 5s` (or aggressive default).
12. [ ] `proxy_send_timeout 30s` for typical endpoints.
13. [ ] `proxy_read_timeout 30s` for typical endpoints.
14. [ ] Long running endpoints have higher `proxy_read_timeout`.
### 500 Internal Server Error
15. [ ] FastAPI has top level exception handler returning 500 cleanly.
16. [ ] Exception handler logs the traceback.
17. [ ] 500 responses have `Cache-Control: no-store`.
18. [ ] 500 responses have `X-Robots-Tag: noindex`.
19. [ ] 500 rate monitored and alerts on baseline deviation.
### 502 Bad Gateway
20. [ ] Bubbles diagnostic script `/usr/local/bin/bubbles-health-check.sh` deployed.
21. [ ] systemd ensures FastAPI runs even after crashes.
22. [ ] nginx upstream definition includes `max_fails` and `fail_timeout`.
23. [ ] 502 rate alerts on sustained presence.
24. [ ] 502 error page provides useful information.
### 503 Service Unavailable
25. [ ] **Retry-After header on every 503 response.**
26. [ ] 503 has `Cache-Control: no-store`.
27. [ ] 503 has `X-Robots-Tag: noindex`.
28. [ ] Maintenance pattern documented and tested.
29. [ ] Crawler split pattern (Section 7.6) ready for use if maintenance exceeds 24 hours.
30. [ ] **Maintenance windows planned for under 48 hours (Google 2 day rule).**
31. [ ] limit_req returns 429 not 503 (per framework-http-rate-control-headers.md).
### 504 Gateway Timeout
32. [ ] Standard `proxy_read_timeout` matches typical endpoint duration.
33. [ ] Long running endpoints have explicit higher timeout OR use 202 async pattern.
34. [ ] Streaming endpoints have `proxy_buffering off` and very high timeout.
35. [ ] 504 rate monitored; sustained 504 triggers investigation.
### Monitoring and alerting
36. [ ] 5xx rate alert configured (e.g., >1% of requests over 5 minutes).
37. [ ] Disk space alert at 80% (prevents 507).
38. [ ] FastAPI process memory alert (prevents OOM kill leading to 502).
39. [ ] Database connection pool exhaustion alert.
40. [ ] CPU saturation alert (prevents 504 from slow processing).
### Operational hygiene
41. [ ] Maintenance config exists at `/etc/nginx/sites-available/<site>.maintenance`.
42. [ ] Maintenance HTML page substantive (not soft 404 risk).
43. [ ] Health check endpoint (`/healthz`) is fast and dependency free.
44. [ ] Readiness endpoint (`/readyz`) checks dependencies and returns 503 if not ready.
45. [ ] Custom 5xx page provides "contact admin@thatdeveloperguy.com" info.
### Cross cutting
46. [ ] `nginx -t` passes without warnings.
47. [ ] `nginx -T | grep error_page` shows expected error page definitions.
48. [ ] `bubbles-health-check.sh` shows all green during normal operation.
49. [ ] Recovery procedures documented (`/var/bubbles/runbooks/5xx.md`).
50. [ ] Quarterly drill: simulate FastAPI crash, verify automatic recovery.
A site that passes all 50 has correctly configured 5xx handling for SEO preservation, operational resilience, and crawler friendly maintenance.
---
## 19. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: Maintenance with 503 but no Retry-After.**
Symptom: site returns to normal after maintenance but crawlers slow to re crawl.
Why it breaks: without Retry-After, crawlers guess; some treat it as transient, others as permanent.
Fix: always include Retry-After matching the planned duration.
**Pitfall 2: Maintenance window exceeds 48 hours.**
Symptom: URLs drop from Google index during prolonged maintenance.
Why it breaks: Google's 2 day rule.
Fix: keep maintenance under 48 hours, OR use crawler split pattern (Section 7.6), OR serve 200 with substantive maintenance content.
**Pitfall 3: FastAPI service has no `Restart=always`.**
Symptom: single crash takes the site offline for hours until manual intervention.
Why it breaks: systemd does not restart by default.
Fix: add `Restart=always` and `RestartSec=5` to the unit file.
**Pitfall 4: 503 cached by browsers and intermediaries.**
Symptom: maintenance ends but users still see 503 for hours.
Why it breaks: missing `Cache-Control: no-store`.
Fix: always include `Cache-Control: no-store, no-cache, must-revalidate` on 503 responses.
**Pitfall 5: nginx returns 503 for rate limiting (default behavior).**
Symptom: monitoring alerts on "service unavailable" but service is fine.
Why it breaks: nginx `limit_req` default is 503.
Fix: `limit_req_status 429;` (per framework-http-rate-control-headers.md).
**Pitfall 6: Long running endpoint hits 504 timeout.**
Symptom: report generation endpoint returns timeout error after 60 seconds.
Why it breaks: nginx default `proxy_read_timeout` is 60 seconds.
Fix: increase timeout for that specific location OR redesign with 202 Accepted + status URL.
**Pitfall 7: 500 errors silently ignored.**
Symptom: site appears to work; some specific actions intermittently fail.
Why it breaks: 5xx rate not monitored; users not reporting.
Fix: alert on 5xx rate; log every 5xx with request context.
**Pitfall 8: 502 from FastAPI crash during deploy.**
Symptom: brief 502 during deployment as old process stops and new starts.
Why it breaks: no graceful shutdown handling, no overlap during deploy.
Fix: graceful SIGTERM handling in FastAPI; for zero downtime, run two instances behind nginx and roll one at a time.
**Pitfall 9: 504 because of database deadlock.**
Symptom: specific endpoint occasionally takes 60+ seconds.
Why it breaks: long held database locks; concurrent transactions blocking each other.
Fix: query optimization, transaction isolation review, possibly redesign the operation.
**Pitfall 10: 5xx error page returns 200.**
Symptom: 500/502/503/504 errors served, but Google indexes the error page as a real page.
Why it breaks: nginx `error_page` directive may serve the page with the original error code, but some configs convert to 200.
Fix: verify with curl: status line should match the original error, not 200.
**Pitfall 11: nginx error_page directive misconfigured.**
Symptom: error pages return generic nginx defaults instead of custom pages.
Why it breaks: `error_page` not defined or `internal;` missing on the location.
Fix: ensure `error_page` directive in server block AND `internal;` on the location.
**Pitfall 12: FastAPI workers exhausted, no 5xx returned but slow.**
Symptom: site slow but no error code; users hang on requests.
Why it breaks: all uvicorn workers busy with long requests.
Fix: increase worker count (`--workers 4` or higher), or add async patterns to prevent worker exhaustion.
**Pitfall 13: Cloudflare 521 troubleshooting on a non Cloudflare site.**
Symptom: looking for Cloudflare specific 5xx codes that don't apply to Bubbles.
Why it breaks: confusion between Cloudflare codes and origin codes.
Fix: for Bubbles, 521/522/524 do not exist. Look for 502/504 from nginx instead.
**Pitfall 14: Health check endpoint depends on database, returns 503 when DB is slow.**
Symptom: monitoring shows site unhealthy when database is just busy, not down.
Why it breaks: confusing liveness with readiness.
Fix: split `/healthz` (liveness, just returns 200) from `/readyz` (readiness, checks dependencies).
**Pitfall 15: No log rotation, disk fills, all 5xx.**
Symptom: gradual increase in errors over weeks; eventually site unavailable.
Why it breaks: nginx and application logs fill disk; system unable to write.
Fix: ensure logrotate configured for `/var/log/nginx/*` and `/var/log/bubbles/*`. Monitor disk usage.
---
## 20. DIAGNOSTIC COMMANDS
Reference of every command useful for 5xx investigation.
### Quick health check
```bash
# All in one diagnostic
sudo bubbles-health-check.sh
# Or manually:
systemctl is-active nginx
systemctl is-active fastapi-sidecar
sudo ss -tlnp | grep :9090
curl -sI http://127.0.0.1:9090/healthz | head -1
curl -sI https://example.com/healthz | head -1
```
### Status code distribution from logs
```bash
# All 5xx in last hour
sudo awk -v since="$(date -d '1 hour ago' '+%d/%b/%Y:%H')" '
$4 ~ since && $9 ~ /^5/ {count[$9]++}
END {for (s in count) print s, count[s]}
' /var/log/nginx/access.log
# 5xx URLs (which endpoints are failing)
sudo awk '$9 ~ /^5/ {print $9, $7}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -20
# 5xx by status code over time (per minute)
sudo awk '$9 ~ /^5/ {
gsub(/\[/, "", $4)
print substr($4, 1, 17), $9
}' /var/log/nginx/access.log | sort | uniq -c | tail -30
```
### Locate FastAPI errors
```bash
# Recent errors only
sudo journalctl -u fastapi-sidecar --since "10 min ago" -p err
# Recent everything (more verbose)
sudo journalctl -u fastapi-sidecar --since "10 min ago"
# Search for specific exception types
sudo journalctl -u fastapi-sidecar | grep -iE "traceback|exception|error"
# Most recent crash
sudo journalctl -u fastapi-sidecar -p crit -n 10
```
### Investigate 502
```bash
# Is FastAPI running?
systemctl status fastapi-sidecar
# When was it last restarted?
sudo systemctl show fastapi-sidecar --property=ActiveEnterTimestamp
# Port listening?
sudo ss -tlnp | grep :9090
# Can nginx reach it?
sudo -u nginx curl -sI http://127.0.0.1:9090/healthz
# Recent nginx errors mentioning upstream
sudo tail -100 /var/log/nginx/error.log | grep upstream | tail -20
```
### Investigate 504
```bash
# Find slow endpoints
sudo awk '$9 == 504 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
# Recent upstream timeouts in error log
sudo grep "upstream timed out" /var/log/nginx/error.log | tail -20
# Current uvicorn process usage
top -p $(pgrep -f uvicorn) -b -n 1
# Database slow queries (Postgres example)
sudo -u postgres psql -d bubbles -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
```
### Verify 503 maintenance config
```bash
# Confirm 503 with Retry-After
curl -sI https://example.com/ | grep -iE "^(HTTP|Retry-After|Cache-Control|X-Robots-Tag)"
# Expected:
# HTTP/2 503
# retry-after: 3600
# cache-control: no-store
# x-robots-tag: noindex
# Confirm the maintenance page is substantive
curl -s https://example.com/ -o /tmp/maintenance.html
WORDS=$(wc -w < /tmp/maintenance.html)
echo "Maintenance page words: $WORDS"
# Expected: 50+ words (avoid soft 404 risk)
```
### Restart sequence
```bash
# Standard restart sequence
sudo systemctl restart fastapi-sidecar
sleep 3
systemctl is-active fastapi-sidecar
# Reload nginx (if needed)
sudo nginx -t && sudo systemctl reload nginx
# Full restart (if needed)
sudo systemctl restart nginx
# Verify
curl -sI https://example.com/healthz | head -1
```
### Server resource investigation
```bash
# Disk space (prevent 507)
df -h /var/www /var/log
# Memory
free -h
# Load average
uptime
# Process count
ps aux | wc -l
# Open file descriptors (nginx limit)
sudo cat /proc/$(pgrep -f "nginx: master")/limits | grep "open files"
# Active connections
sudo ss -tan state established | wc -l
```
### Apply changes
```bash
# Test and reload nginx
sudo nginx -t && sudo systemctl reload nginx
# Reload systemd if unit file changed
sudo systemctl daemon-reload
sudo systemctl restart fastapi-sidecar
```
---
## 21. CROSS-REFERENCES
* [framework-http-2xx-status-codes.md](framework-http-2xx-status-codes.md): 200 with `X-Robots-Tag: noindex` is the alternative to 503 for long maintenance windows.
* [framework-http-3xx-status-codes.md](framework-http-3xx-status-codes.md): 301 to a temporary mirror is the alternative to 503 for extended maintenance.
* [framework-http-4xx-status-codes.md](framework-http-4xx-status-codes.md): 503 vs 429 distinction (covered there); custom error pages pattern.
* [framework-http-caching-headers.md](framework-http-caching-headers.md): `Cache-Control: no-store` on 5xx responses prevents cached errors.
* [framework-http-seo-headers.md](framework-http-seo-headers.md): `X-Robots-Tag: noindex` on 5xx error pages prevents accidental indexing.
* [framework-http-rate-control-headers.md](framework-http-rate-control-headers.md): the full 503+Retry-After+crawler whitelist pattern. The 2 day rule for crawlers is documented at length here as well.
* [framework-http-request-headers.md](framework-http-request-headers.md): User-Agent based crawler whitelist for the crawler split maintenance pattern.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. The 2 day rule and sustained 5xx index removal are covered there.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook including maintenance window planning.
* Google HTTP status codes documentation: https://developers.google.com/search/docs/crawling-indexing/http-network-errors
* Google maintenance handling: https://developers.google.com/search/docs/crawling-indexing/pause-online-business
* MDN 500: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/500
* MDN 502: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/502
* MDN 503: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503
* MDN 504: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/504
* RFC 9110 (HTTP Semantics): https://www.rfc-editor.org/rfc/rfc9110
* nginx error_page directive: https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page
* nginx proxy_next_upstream: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
* systemd Restart configuration: https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=
---
## 22. THE COMPLETE 12 FILE BUBBLES HTTP WIRE REFERENCE (SERIES SUMMARY)
This framework completes the series. The full reference is now 12 files covering every HTTP header and status code Bubbles can produce or receive.
### 22.1 The Complete File Set
**Track 1: HTTP Headers (response and request side)**
| File | Headers covered |
|---|---|
| framework-http-caching-headers.md | Cache-Control, ETag, Last-Modified, Expires, Vary, Age |
| framework-http-content-headers.md | Content-Type, Content-Language, Content-Encoding, Content-Length, Content-Disposition |
| framework-http-seo-headers.md | X-Robots-Tag, Link, Location |
| framework-http-security-headers.md | HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, CORP, COEP, COOP |
| framework-http-performance-headers.md | Server-Timing, Timing-Allow-Origin, Server, Alt-Svc |
| framework-http-cors-headers.md | Access-Control-Allow-Origin/Methods/Headers/Expose-Headers/Max-Age/Allow-Credentials |
| framework-http-rate-control-headers.md | Retry-After, X-RateLimit-*, RateLimit-* |
| framework-http-request-headers.md | User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset, From, Host, If-Modified-Since, If-None-Match, Referer, Connection |
**Track 2: HTTP Status Codes**
| File | Codes covered |
|---|---|
| framework-http-2xx-status-codes.md | 200, 201, 202, 204, 206 |
| framework-http-3xx-status-codes.md | 301, 302, 303, 304, 307, 308 |
| framework-http-4xx-status-codes.md | 400, 401, 403, 404, 410, 422, 429, 451 (plus 405, 408, 411, 413, 414, 415 briefly) |
| framework-http-5xx-status-codes.md (this) | 500, 502, 503, 504 (plus 505, 507, 508, 511 briefly, plus Cloudflare 520-527 briefly) |
### 22.2 The Cumulative Totals
* **Files**: 12
* **HTTP headers covered**: 46 (response and request side)
* **HTTP status codes covered**: 23 (primary) plus 12 briefly
* **Combined audit checklist items**: approximately 670
* **Common pitfalls documented**: approximately 195
* **Combined size**: approximately 1 MB of markdown reference
### 22.3 The Cross Cutting Patterns
Several patterns appear across multiple frameworks. The framework where each is first defined as the canonical reference:
* **The crawler taxonomy and verification** (Googlebot, ClaudeBot, GPTBot et al with current 2026 UA strings and DNS verification): framework-http-request-headers.md Sections 5 and 6.
* **The 2 day Google deindex rule** for sustained 5xx/429: framework-http-rate-control-headers.md Section 5.3, deeply cross referenced from framework-http-4xx-status-codes.md and this framework.
* **The Bubbles thin content cleanup (Joseph's 5,800 page work)**: framework-http-4xx-status-codes.md Section 17 (the 410 pattern) plus framework-http-2xx-status-codes.md Section 10 (the soft 404 trap context).
* **The 14 paying client subdomain redirects**: framework-http-3xx-status-codes.md Section 16 (the verbatim nginx config from the recent infrastructure overhaul).
* **The default_server returning 444**: framework-http-request-headers.md Section 12 (Host header routing).
* **The FastAPI sidecar architecture**: this framework Section 13 (the canonical diagnostic pattern).
### 22.4 The Operating Conventions
All 12 frameworks consistently follow Joseph's standing rules:
* Zero em dashes, en dashes, smart quotes, or Unicode arrows.
* Nginx focused for Bubbles (Debian, no Cloudflare).
* `/var/www/sites/[domain]/` paths.
* Every nginx change ends with `nginx -t && systemctl reload nginx`.
* FastAPI sidecar on port 9090.
* 50+ item audit checklist per framework.
* 15 common pitfalls per framework.
* Diagnostic commands section per framework.
* Cross references to siblings and authoritative external docs (RFCs, MDN, Google).
### 22.5 The Usage Pattern
For any new Bubbles client deployment, the workflow:
1. Provision the server, install nginx and the FastAPI sidecar.
2. Apply the standard configurations from the relevant framework's "Bubbles Nginx Reference Block" section.
3. Run the audit checklists across all 12 frameworks (the combined ~670 items).
4. Verify each test from the diagnostic commands sections.
5. Document deviations and gaps for the specific client's needs.
For any incident:
1. Identify the affected status code (which framework covers it).
2. Use that framework's diagnostic commands section.
3. Apply the recovery pattern from the relevant recipe section.
4. Confirm via the verification commands.
### 22.6 The Living Document Note
HTTP is a stable protocol. RFCs change rarely. Crawler behaviors change occasionally (Google deprecates Claude-Web in favor of ClaudeBot, OpenAI splits training from search). The frameworks are designed to be updated incrementally as the ecosystem evolves; the underlying patterns (the 2 day rule, the soft 404 trap, the 401 vs 403 distinction) are stable across decades.
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Status code decision tree
```
Server side failure. Which one?
|
|---> Application threw exception ........................ 500 Internal Server Error
|
|---> Application not running (port not bound) ........... 502 Bad Gateway
|
|---> Application returned garbage to nginx ............... 502 Bad Gateway
|
|---> Application accepted request but didn't respond .... 504 Gateway Timeout
|
|---> Operator intentionally signaling unavailability .... 503 Service Unavailable (+ Retry-After)
```
### The 2 day rule
Sustained 503 or 429 for more than 48 hours = URLs drop from Google index.
* Under 24 hours: 503 with Retry-After is safe.
* 24 to 48 hours: 503 with Retry-After is risky; use crawler split pattern.
* Over 48 hours: do NOT use 503. Serve 200 with substantive maintenance content.
### Bubbles 5xx most likely causes
| Symptom | Most likely cause | Fix |
|---|---|---|
| 502 immediately after deploy | FastAPI failed to start | `journalctl -u fastapi-sidecar` |
| 502 intermittent | FastAPI crashes, no restart | systemd Restart=always |
| 504 specific endpoint | Slow query, slow API call | Profile, optimize, or async |
| 500 specific endpoint | Application bug | Fix the bug |
| 503 unexpected | limit_req default | `limit_req_status 429` |
| All 5xx | Disk full, OOM, etc | Resource check |
### Five rules to memorize
1. 503 needs Retry-After. Always.
2. Maintenance windows under 48 hours. Always.
3. systemd Restart=always for FastAPI. Always.
4. Health check (`/healthz`) independent of dependencies.
5. nginx `proxy_next_upstream` for transient failures.
### Five commands every operator should know
```bash
# 1. Run the health check
sudo bubbles-health-check.sh
# 2. Find current 5xx rate
sudo awk '$9 ~ /^5/ {c++} END {print c+0}' /var/log/nginx/access.log
# 3. Restart FastAPI
sudo systemctl restart fastapi-sidecar
# 4. Check upstream timeouts
sudo grep "upstream timed out" /var/log/nginx/error.log | tail -10
# 5. Apply nginx changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. Health check returns 200
[ $(curl -so /dev/null -w "%{http_code}" https://example.com/healthz) = "200" ] && \
echo "OK: healthz" || echo "FAIL: healthz"
# 2. Maintenance 503 has Retry-After when active
# (Only run during planned maintenance)
RA=$(curl -sI https://example.com/ | grep -i retry-after | wc -l)
[ $RA = "1" ] && echo "OK: Retry-After present" || echo "Verify maintenance config"
# 3. systemd self healing works
sudo systemctl stop fastapi-sidecar
sleep 6
[ "$(systemctl is-active fastapi-sidecar)" = "active" ] && \
echo "OK: self healed" || echo "FAIL: did not restart"
```
If all three pass AND no sustained 5xx in the last 24 hours, the 5xx layer is correctly wired.
---
End of framework-http-5xx-status-codes.md.
End of the HTTP status codes series.
End of the 12 file Bubbles HTTP wire reference.