Janady13/crawler-signal-reference / framework-http-request-headers.md

1# framework-http-request-headers.md 2 3Comprehensive reference for the eleven HTTP request headers most relevant to server side decisions about how to respond, route, log, throttle, or filter: `User-Agent` (identity, never to be trusted alone), `Accept` (content type negotiation), `Accept-Language` (language negotiation), `Accept-Encoding` (compression negotiation), `Accept-Charset` (legacy charset negotiation), `From` (legacy contact identifier), `Host` (virtual host selection), `If-Modified-Since` (conditional GET by date), `If-None-Match` (conditional GET by ETag), `Referer` (request origin), and `Connection` (transport disposition). Built for Bubbles (Debian, Nginx 1.26+, self hosted origin at 169.155.162.118, no Cloudflare or third party CDN in front). Companion to the seven response side frameworks (caching, content, SEO, security, performance, CORS, rate control), UNIVERSAL-RANKING-FRAMEWORK.md, and SEO BUILD REFERENCE v2.4. 4 5**This is the first request side framework.** The previous seven frameworks documented what the server SENDS in responses. This one documents what crawlers and clients SEND in requests, and how the server reads, validates, and reacts to those headers. The two sides interlock: the server's `Last-Modified` response is meaningful because clients send `If-Modified-Since` requests; the server's `Vary: Accept-Encoding` response only makes sense because clients send `Accept-Encoding` requests. 6 7Audience: humans configuring nginx and FastAPI sidecars to react correctly to incoming headers, AI assistants generating server logic that handles crawler traffic intelligently, SEO operators verifying that high value crawlers are actually who they claim to be, and anyone troubleshooting "Googlebot is being spoofed", "crawlers not getting 304s", "compression not negotiated", or "wrong language served to Spanish speaking crawlers". 8 9--- 10 11## TABLE OF CONTENTS 12 131. Definition 142. Why It Matters 153. What This Covers 164. The Request Header Mental Model (read this first) 175. User-Agent (identity claim, never trust alone) 186. The Crawler Verification Procedure (reverse plus forward DNS, IP ranges) 197. Accept (content type negotiation) 208. Accept-Language (language and locale negotiation) 219. Accept-Encoding (compression negotiation, the gzip plus brotli plus zstd stack) 2210. Accept-Charset (legacy charset negotiation) 2311. From (legacy contact identifier some bots use) 2412. Host (virtual host selection) 2513. If-Modified-Since (conditional GET by date) 2614. If-None-Match (conditional GET by ETag) 2715. Referer (request origin, often absent for crawlers) 2816. Connection (transport disposition) 2917. How These Headers Interact 3018. Asset Class And Use Case Recipes 3119. Bubbles Nginx Reference Block (paste ready) 3220. Audit Checklist (50+ items) 3321. Common Pitfalls 3422. Diagnostic Commands 3523. Cross-References 36 37--- 38 39## 1. DEFINITION 40 41Request headers are everything the client (browser, crawler, API client) attaches to its HTTP request before the body. They communicate identity, capability, preference, and context. The server reads them to decide what to do: which virtual host to route to, what content type to serve, whether to compress, whether the cached version is still valid, whether the client deserves rate limiting or special handling. 42 43The eleven headers in this framework split into five concerns: 44 45* **Identity and verification**: `User-Agent`, `From`. They answer "who claims to be making this request?" but never authoritatively. Verification requires looking beyond the request body. 46* **Content negotiation**: `Accept`, `Accept-Language`, `Accept-Encoding`, `Accept-Charset`. They answer "what can this client accept in the response?" The server picks the best match. 47* **Routing**: `Host`. It answers "which virtual host is this request for?" Essential for multi tenant nginx serving 181 hostnames from one IP. 48* **Conditional GET**: `If-Modified-Since`, `If-None-Match`. They answer "do I already have a fresh copy?" Pair with response side `Last-Modified` and `ETag` from framework-http-caching-headers.md. 49* **Context**: `Referer`, `Connection`. They answer "where did this request come from?" and "how should this connection behave after the response?" 50 51A correctly configured server reads these headers, validates them, makes content negotiation decisions, returns 304 Not Modified when appropriate, suppresses or amplifies based on identity (with proper verification), and logs them for security and analytics. 52 53--- 54 55## 2. WHY IT MATTERS 56 57Six independent pressures push correct request header handling from "passive" to "actively shaped" infrastructure in 2025 and forward. 58 59**AI crawler traffic is now a measurable percentage of all web traffic.** Cloudflare reported in March 2025 that AI crawlers generated more than 50 billion requests per day across its network. By January 2026, Cloudflare's two month analysis showed Googlebot reached 1.70 times more unique URLs than ClaudeBot, 1.76 times more than GPTBot, 2.99 times more than Meta-ExternalAgent. The fleet of crawlers your server sees is fundamentally different from 2022. Correctly identifying and responding to them is now a first class concern. 60 61**User-Agent spoofing is trivial.** Any scraper, scanner, or attacker can claim to be Googlebot by setting a User-Agent string. Sites that trust the User-Agent without verification get tricked into bypassing rate limits, serving privileged content, or skipping bot management. The verification procedure (reverse DNS plus forward DNS, or IP range matching against published lists) is the actual identity check. 62 63**Conditional GET saves crawl budget and bandwidth.** When a crawler sends `If-Modified-Since` or `If-None-Match` and the content has not changed, the server should return `304 Not Modified` with no body. Google specifically requests this behavior for AdsBot crawls. Every 304 saved is real bandwidth and crawler goodwill. A site that ignores conditional GET headers and always returns 200 with full body is wasting both server resources and crawler time. 64 65**Modern Accept-Encoding has three formats.** In 2026, browsers and most clients negotiate gzip, brotli, and zstd. Servers that handle only gzip leave 20 to 30 percent compression savings on the table for modern browsers. Servers that get the negotiation wrong serve broken or oversized content. 66 67**Accept-Language matters for international SEO.** Bingbot and Baidu reportedly use Accept-Language when crawling to discover language variants. Even when not used by crawlers, browser visitors send it and use it to pick the right localized version of a multilingual site. 68 69**Host header attacks are real.** Sending a request to one IP with a Host header for a different domain (Host header injection) can confuse virtual host routing, password reset emails, and cache poisoning. nginx servers that have no `default_server` block and accept any Host header are vulnerable. 70 71**Cost of getting it wrong.** Misconfigured request handling produces silent SEO damage, security incidents, and crawler frustration. Real examples: 72 73* Site trusts User-Agent for "Allow Googlebot to bypass paywall" feature. Attacker spoofs Googlebot UA, reads all paywalled content. Discovery years later in security audit. 74* Crawler sends If-Modified-Since on every revisit. Server ignores it and always returns 200 with full HTML. Googlebot wastes crawl budget on unchanged pages; new pages crawl slower. 75* Site only supports gzip in Accept-Encoding. Modern browsers send `gzip, deflate, br, zstd` and get gzipped responses 20 percent larger than necessary. LCP regression no one investigates. 76* nginx has no `default_server` and a wildcard SSL certificate. Attacker sends request with Host header `evil.com`, gets routed to whichever server block parses first. Cache key now includes a contaminated Host value. Cache poisoning at scale. 77* Site uses Referer for auth ("only allow from our own pages"). Browsers stop sending Referer due to privacy settings. Auth fails for users who installed privacy extensions or use specific browsers. Support tickets flood in. 78 79All preventable with the rules below. 80 81--- 82 83## 3. WHAT THIS COVERS 84 85Each of the eleven headers gets the same six part treatment: 86 871. **What it does**: the canonical RFC definition plus practical implication. 882. **Syntax and values**: every common format, what it means, when it is wrong. 893. **How to access it in nginx and FastAPI**: variable names, request object attributes, with concrete examples. 904. **How to use it for server side decisions**: routing, content negotiation, logging, identity gates. 915. **How to verify and troubleshoot**: curl commands and diagnostic patterns. 926. **Crawler specific behavior**: what major search and AI crawlers actually send. 93 94The User-Agent header gets the largest treatment because verification is non trivial. Sections 5 and 6 are paired: identity claim plus identity verification. 95 96--- 97 98## 4. THE REQUEST HEADER MENTAL MODEL (READ THIS FIRST) 99 100A request arrives. The server has milliseconds to make several decisions before responding. Each decision is driven by request headers. Internalize the flow. 101 102``` 103TCP connection accepted by nginx 104 | 105 v 106==================== ROUTING ==================== 107 | 108 v 109Read Host header 110 * Match against server_name in each server block 111 * If no match and no default_server: nginx returns 444 (close connection) 112 | 113 v 114Routed to correct server block 115 | 116 v 117==================== IDENTITY ==================== 118 | 119 v 120Read User-Agent header 121 * Is this a known crawler? (UA pattern match) 122 * Is it actually that crawler? (reverse DNS check, optional) 123 * Is it on the rate limit whitelist? (from framework-http-rate-control-headers.md) 124 | 125 v 126==================== CONTENT NEGOTIATION ==================== 127 | 128 v 129Read Accept (which content types client wants) 130 * Pick best match for this resource 131 * Set Vary: Accept on response if we vary 132 133Read Accept-Language (which languages client prefers) 134 * Pick best match for multilingual content 135 * Set Vary: Accept-Language on response if we vary 136 137Read Accept-Encoding (which compression algorithms client supports) 138 * gzip, br, zstd: pick best supported by both client and server 139 * Set Vary: Accept-Encoding on response (mandatory when compressing) 140 141Read Accept-Charset (legacy, almost universally ignored in 2026) 142 * Just use UTF-8 and move on 143 | 144 v 145==================== CONDITIONAL GET ==================== 146 | 147 v 148Read If-Modified-Since (RFC date) 149 * Compare to resource's Last-Modified 150 * If unchanged: return 304 Not Modified, no body 151 152Read If-None-Match (ETag value) 153 * Compare to resource's current ETag 154 * If matches: return 304 Not Modified, no body 155 * If both If-* present: ETag takes precedence 156 | 157 v 158==================== CONTEXT FOR LOGGING ==================== 159 | 160 v 161Read Referer (where the request originated) 162 * Log for analytics (referral tracking) 163 * Be aware: often empty for crawlers, often empty for privacy aware browsers 164 165Read From (legacy contact field, used by some bots) 166 * Log for crawler accountability 167 * Rarely actionable 168 | 169 v 170==================== TRANSPORT DISPOSITION ==================== 171 | 172 v 173Read Connection (close, keep-alive, upgrade) 174 * keep-alive: subsequent requests can reuse the connection 175 * close: close after this response 176 * upgrade: switch protocol (WebSocket negotiation) 177 | 178 v 179==================== APPLICATION ==================== 180 | 181 v 182Process the request, build the response with appropriate Vary headers 183``` 184 185Five rules govern the system: 186 1871. **Never trust identity claims alone.** User-Agent, From, and Host can all be set to anything. Verify before granting privileges. 1882. **Always respond to conditional GET headers.** Return 304 when appropriate. It is free server side bandwidth. 1893. **Always negotiate Accept-Encoding correctly.** Modern clients want brotli or zstd; gzip is the universal fallback. 1904. **Set Vary on every response that varies on a request header.** Without Vary, caches serve the wrong variant to subsequent visitors. 1915. **Always have a default_server.** nginx routes unmatched Host headers to default_server. Without one, it falls back to the first server block (potentially the wrong one). 192 193A correctly configured server reads identity carefully, negotiates content faithfully, returns 304 when warranted, and logs context for analytics. Every header is either a decision point or a logging signal. 194 195--- 196 197## 5. USER-AGENT (IDENTITY CLAIM, NEVER TRUST ALONE) 198 199### 5.1 What It Does 200 201`User-Agent` is the client's claim about its own identity, typically including the software name, version, and operating system. Defined in RFC 9110. Trivially spoofable. Used everywhere despite being unverifiable on its own. 202 203``` 204User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 205User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 206User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com 207User-Agent: GPTBot/1.0 (+https://openai.com/gptbot) 208``` 209 210The User-Agent string is the **identity claim**. The identity verification is separate (Section 6). 211 212### 5.2 The 2026 Crawler Taxonomy 213 214The list of crawlers your Bubbles sites will see, with their current 2026 user agent patterns: 215 216**Search engine crawlers (the historical mainstays):** 217 218| Crawler | User Agent Pattern | Purpose | 219|---|---|---| 220| Googlebot | `Googlebot/2.1` | Primary Google search index | 221| Googlebot-Image | `Googlebot-Image/1.0` | Google Images | 222| Googlebot-News | `Googlebot-News` | Google News | 223| Googlebot-Video | `Googlebot-Video/1.0` | Google Videos | 224| AdsBot-Google | `AdsBot-Google` | Google Ads landing page quality | 225| AdsBot-Google-Mobile | `AdsBot-Google-Mobile` | Google Ads mobile landing pages | 226| Bingbot | `bingbot/2.0` | Bing search + ChatGPT search backend | 227| BingPreview | `BingPreview/1.0b` | Bing snapshot generation | 228| DuckDuckBot | `DuckDuckBot/1.1` | DuckDuckGo | 229| YandexBot | `YandexBot/3.0` | Yandex (Russian search) | 230| Baiduspider | `Baiduspider` | Baidu (Chinese search) | 231| Applebot | `Applebot/0.1` | Apple Spotlight and Siri | 232| AmazonBot | `Amazonbot/0.1` | Amazon products and Alexa | 233 234**AI training crawlers (data harvesting for LLM training):** 235 236| Crawler | User Agent Pattern | Operator | 237|---|---|---| 238| GPTBot | `GPTBot/1.0` | OpenAI (training only) | 239| ClaudeBot | `ClaudeBot/1.0` | Anthropic (training only; replaces deprecated Claude-Web) | 240| Google-Extended | (used in robots.txt only) | Google (Gemini training opt out) | 241| CCBot | `CCBot/2.0` | Common Crawl (open dataset used by many LLMs) | 242| meta-externalagent | `meta-externalagent/1.1` | Meta (Llama training) | 243| Bytespider | `Bytespider` | ByteDance (TikTok parent) | 244 245**AI search and retrieval crawlers (per query fetches for answering users):** 246 247| Crawler | User Agent Pattern | Operator | 248|---|---|---| 249| OAI-SearchBot | `OAI-SearchBot/1.0` | OpenAI ChatGPT search | 250| ChatGPT-User | `ChatGPT-User/1.0` | OpenAI user initiated fetches | 251| Claude-SearchBot | `Claude-SearchBot/1.0` | Anthropic search | 252| Claude-User | `Claude-User/1.0` | Anthropic user initiated fetches | 253| PerplexityBot | `PerplexityBot/1.0` | Perplexity index | 254| Perplexity-User | `Perplexity-User/1.0` | Perplexity user fetches | 255| YouBot | `YouBot/0.1` | You.com | 256| DuckAssistBot | `DuckAssistBot/1.0` | DuckDuckGo Assist | 257| Mistral-User | `Mistral-User/1.0` | Mistral user fetches | 258 259**Social and link preview crawlers:** 260 261| Crawler | User Agent Pattern | Operator | 262|---|---|---| 263| facebookexternalhit | `facebookexternalhit/1.1` | Facebook link previews | 264| Twitterbot | `Twitterbot/1.0` | Twitter/X card previews | 265| LinkedInBot | `LinkedInBot/1.0` | LinkedIn preview | 266| Slackbot | `Slackbot-LinkExpanding 1.0` | Slack link previews | 267| Discordbot | `Discordbot/2.0` | Discord embeds | 268 269**Important taxonomy notes for 2026:** 270 2711. **ClaudeBot replaced Claude-Web.** Sites still using `User-agent: Claude-Web` in robots.txt are not actually blocking Anthropic's current crawler. Update to `ClaudeBot`. 272 2732. **OpenAI splits training from search.** `GPTBot` is training only; `OAI-SearchBot` is for ChatGPT search; `ChatGPT-User` is for user initiated fetches. A site can block GPTBot to protect its content from training while allowing OAI-SearchBot to remain visible in ChatGPT search results. 274 2753. **Google-Extended is a robots.txt only token.** It is used in robots.txt to opt out of Google's generative AI training while still allowing standard Googlebot search indexing. It does not appear in actual User-Agent strings. 276 2774. **The browser like AI fetches (Claude-User, Perplexity-User, ChatGPT-User) are user initiated.** Each represents a real person asking the AI assistant to fetch a URL. They are not crawl traffic in the traditional sense; they are pull through views. Allow these for GEO (generative engine optimization) visibility. 278 279### 5.3 How To Access User-Agent 280 281**In nginx:** 282 283```nginx 284# Variable: $http_user_agent 285log_format main '$remote_addr - $remote_user [$time_local] ' 286 '"$request" $status $body_bytes_sent ' 287 '"$http_referer" "$http_user_agent"'; 288 289# Use in map for routing decisions 290map $http_user_agent $is_crawler { 291 default 0; 292 ~*googlebot 1; 293 ~*bingbot 1; 294 ~*claudebot 1; 295 ~*gptbot 1; 296 ~*oai-searchbot 1; 297 ~*perplexitybot 1; 298} 299``` 300 301**In FastAPI:** 302 303```python 304from fastapi import FastAPI, Request 305 306app = FastAPI() 307 308@app.get("/") 309async def index(request: Request): 310 user_agent = request.headers.get("user-agent", "") 311 # Use it for content decisions, logging, etc 312``` 313 314### 5.4 Using User-Agent For Server Side Decisions (With Caveats) 315 316**Acceptable uses (UA alone is sufficient):** 317 318* Logging and analytics. Track which crawlers visit which pages. 319* Bot detection signals for analytics dashboards. 320* Permissive routing where being wrong is harmless (rate limit whitelist for a public page). 321* Content adaptation hints (mobile vs desktop layout, though not the recommended approach). 322 323**Unacceptable uses (UA alone is insufficient, must verify):** 324 325* Paywall bypass for Googlebot. Verify by reverse DNS first. 326* Premium content access for known crawlers. Same. 327* Skipping security checks (CAPTCHA, fraud detection). Always verify. 328* Any privilege grant where being wrong has cost. 329 330The reverse DNS verification procedure is Section 6. 331 332### 5.5 Common UA Anti-Patterns 333 334**Cloaking (illegal in Google's eyes):** Serving different content to Googlebot than to users. Google penalizes sites that do this when detected. The exception is well disclosed dynamic rendering for JavaScript heavy sites. 335 336**Browser sniffing:** Detecting "Internet Explorer" or "Edge" and serving different code. Modern best practice is feature detection, not UA detection. 337 338**Mobile vs desktop UA routing:** Historically common, increasingly discouraged. Use responsive design instead. Google's mobile first indexing prefers responsive sites. 339 340### 5.6 How To Verify The UA Was Set 341 342```bash 343# 1. Check what UA your test client sends 344curl -sI -A "test-bot/1.0" https://example.com/ -v 2>&1 | grep -i "User-Agent" 345# Expected (in request side): User-Agent: test-bot/1.0 346 347# 2. Verify nginx logs the UA correctly 348curl -sI -A "TestBot/1.0 Research" https://example.com/ 349sudo tail -1 /var/log/nginx/access.log 350# Look for: "TestBot/1.0 Research" 351 352# 3. Simulate major crawlers for testing routing logic 353for ua in "Googlebot/2.1" "bingbot/2.0" "ClaudeBot/1.0" "GPTBot/1.0" "OAI-SearchBot/1.0" "PerplexityBot/1.0"; do 354 echo "=== $ua ===" 355 curl -sI -A "Mozilla/5.0 (compatible; $ua)" https://example.com/ | head -1 356done 357``` 358 359### 5.7 Troubleshooting 360 361**Symptom: log shows requests from unfamiliar UA strings.** 362Likely scenarios: 3631. A new crawler launched. Check against the 2026 taxonomy (Section 5.2). 3642. A scraper or scanner spoofing a known crawler. Verify by reverse DNS (Section 6). 3653. A monitoring tool from a customer or operator. Check if Bubbles or client tools are visiting. 3664. An attacker probing for vulnerabilities. Check if the requests have malicious paths. 367 368**Symptom: a UA pattern matches but verification fails.** 369The client is spoofing the UA. Treat as unverified traffic; apply standard rate limits and bot management. 370 371**Symptom: Googlebot stopped sending traditional crawl UA, sending different format.** 372Google updates their UA strings periodically. Check the official documentation at `https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers` for current strings. 373 374**Symptom: Cloaking penalty from Google.** 375Detected serving different content to Googlebot than to users. Audit any UA based content variation. Move to consistent content delivery; use Hreflang for language variation, structured data for content disambiguation. 376 377### 5.8 The Bubbles UA Logging Pattern 378 379For Joseph's 130+ client sites, the goal is to track AI crawler visibility (which is part of the engine optimization investment). Add a dedicated log field for AI crawlers: 380 381```nginx 382map $http_user_agent $crawler_type { 383 default "other"; 384 ~*googlebot "search-google"; 385 ~*bingbot "search-bing"; 386 ~*duckduckbot "search-ddg"; 387 ~*applebot "search-apple"; 388 ~*claudebot "ai-train-claude"; 389 ~*gptbot "ai-train-openai"; 390 ~*google-extended "ai-train-google"; 391 ~*ccbot "ai-train-cc"; 392 ~*meta-externalagent "ai-train-meta"; 393 ~*oai-searchbot "ai-search-openai"; 394 ~*chatgpt-user "ai-user-openai"; 395 ~*claude-searchbot "ai-search-claude"; 396 ~*claude-user "ai-user-claude"; 397 ~*perplexitybot "ai-search-perplexity"; 398 ~*perplexity-user "ai-user-perplexity"; 399 ~*youbot "ai-search-you"; 400 ~*facebookexternalhit "social-fb"; 401 ~*twitterbot "social-twitter"; 402 ~*linkedinbot "social-linkedin"; 403 ~*slackbot "social-slack"; 404 ~*discordbot "social-discord"; 405} 406 407log_format with_crawler '$remote_addr [$time_local] "$request" $status $body_bytes_sent ' 408 '"$http_referer" "$http_user_agent" crawler=$crawler_type'; 409 410access_log /var/log/nginx/access.log with_crawler; 411``` 412 413Now `awk` can aggregate crawler types per day: 414 415```bash 416awk '/crawler=ai-/ {print $NF}' /var/log/nginx/access.log | sort | uniq -c | sort -rn 417``` 418 419This is the foundation for the Bubbles AI visibility analytics dashboard. 420 421--- 422 423## 6. THE CRAWLER VERIFICATION PROCEDURE (REVERSE PLUS FORWARD DNS, IP RANGES) 424 425User-Agent is a claim. This section is how to verify the claim. The procedure differs by operator: some publish IP ranges; some require reverse DNS. 426 427### 6.1 The Two Verification Methods 428 429**Method A: Reverse DNS plus forward DNS lookup.** 430 4311. Take the IP from the request. 4322. Do a PTR lookup: what hostname does this IP resolve to? 4333. Check the hostname against an allow list (e.g., `*.googlebot.com`, `*.search.msn.com`). 4344. Do a forward A or AAAA lookup on that hostname: what IP does it resolve to? 4355. Confirm the forward lookup IP matches the original request IP. 436 437If all four match and the hostname is in the operator's domain, the request is verified. 438 439**Method B: IP range matching against published lists.** 440 441Most major operators publish their IP ranges as JSON files. Match the request IP against the published list. If it falls within an operator's range, it is verified. 442 443| Operator | IP range URL | Notes | 444|---|---|---| 445| Google (Googlebot, AdsBot, Crawler) | `https://developers.google.com/static/search/apis/ipranges/googlebot.json` | Updated periodically | 446| Google (special crawlers) | `https://developers.google.com/static/search/apis/ipranges/special-crawlers.json` | AdsBot, Vertex AI | 447| Google (user triggered fetchers) | `https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json` | Google-Read-Aloud, etc | 448| Bing | `https://www.bing.com/toolbox/bingbot.json` | All Bing crawlers | 449| OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) | `https://openai.com/gptbot.json`, `https://openai.com/searchbot.json`, `https://openai.com/chatgpt-user.json` | Separate files per crawler | 450| Perplexity (PerplexityBot, Perplexity-User) | `https://www.perplexity.ai/perplexitybot.json`, `https://www.perplexity.ai/perplexity-user.json` | Separate files | 451| Common Crawl (CCBot) | (uses AWS ranges, not published cleanly) | Verify via UA only, IP varies | 452| Apple (Applebot) | (use reverse DNS, `*.applebot.apple.com`) | No published range | 453| Amazon (Amazonbot) | (use reverse DNS, `*.amazonbot.amazon`) | No published range | 454| **Anthropic (ClaudeBot, Claude-SearchBot, Claude-User)** | **Not published** | **Reverse DNS only; check for hostnames in claude.ai or anthropic.com domains** | 455 456The notable gap: Anthropic does not publish IP ranges as of late 2025. Verification of ClaudeBot must use reverse DNS or rely on UA pattern matching combined with reasonable IP heuristics (data center ASNs). 457 458### 6.2 The Reverse Plus Forward DNS Pattern (Manual) 459 460```bash 461# Step 1: get the IP from the request log 462REQUEST_IP="66.249.66.1" 463 464# Step 2: reverse DNS lookup (PTR) 465PTR=$(host "$REQUEST_IP" | awk '{print $NF}' | sed 's/\.$//') 466echo "Reverse: $REQUEST_IP -> $PTR" 467# Expected for Googlebot: crawl-66-249-66-1.googlebot.com 468 469# Step 3: check the hostname against expected domain 470case "$PTR" in 471 *.googlebot.com|*.google.com|*.googleusercontent.com) 472 echo "Claims to be Google. Verifying forward DNS..." 473 ;; 474 *) 475 echo "FAIL: hostname does not match Google patterns" 476 exit 1 477 ;; 478esac 479 480# Step 4: forward DNS lookup 481FORWARD_IP=$(host "$PTR" | awk '/has address/ {print $NF}' | head -1) 482echo "Forward: $PTR -> $FORWARD_IP" 483 484# Step 5: confirm forward IP matches original 485if [ "$FORWARD_IP" = "$REQUEST_IP" ]; then 486 echo "VERIFIED: $REQUEST_IP is genuine Googlebot" 487else 488 echo "FAIL: forward IP $FORWARD_IP does not match original $REQUEST_IP" 489fi 490``` 491 492### 6.3 The Verification Pattern As A FastAPI Helper 493 494```python 495import socket 496import asyncio 497from functools import lru_cache 498 499# Hostnames that indicate verified crawler ownership 500VERIFIED_HOSTS = { 501 "googlebot": ["googlebot.com", "google.com", "googleusercontent.com"], 502 "bingbot": ["search.msn.com"], 503 "applebot": ["applebot.apple.com"], 504 "amazonbot": ["amazonbot.amazon"], 505 "yandexbot": ["yandex.com", "yandex.net", "yandex.ru"], 506} 507 508@lru_cache(maxsize=10000) 509def verify_crawler_by_dns(ip: str, claimed_operator: str) -> bool: 510 """Verify a crawler IP using reverse plus forward DNS.""" 511 expected_suffixes = VERIFIED_HOSTS.get(claimed_operator, []) 512 if not expected_suffixes: 513 return False 514 515 try: 516 # Reverse DNS (PTR) 517 hostname, _, _ = socket.gethostbyaddr(ip) 518 hostname = hostname.lower().rstrip(".") 519 520 # Check hostname matches expected suffix 521 if not any(hostname.endswith(suffix) for suffix in expected_suffixes): 522 return False 523 524 # Forward DNS check 525 try: 526 forward_ips = socket.gethostbyname_ex(hostname)[2] 527 except socket.gaierror: 528 return False 529 530 return ip in forward_ips 531 except (socket.herror, socket.gaierror): 532 return False 533 534 535# Usage in middleware 536@app.middleware("http") 537async def verify_crawler(request, call_next): 538 ua = request.headers.get("user-agent", "").lower() 539 client_ip = request.client.host 540 541 claimed_operator = None 542 if "googlebot" in ua: 543 claimed_operator = "googlebot" 544 elif "bingbot" in ua: 545 claimed_operator = "bingbot" 546 elif "applebot" in ua: 547 claimed_operator = "applebot" 548 549 if claimed_operator: 550 verified = await asyncio.get_event_loop().run_in_executor( 551 None, verify_crawler_by_dns, client_ip, claimed_operator 552 ) 553 request.state.crawler_verified = verified 554 request.state.crawler_operator = claimed_operator 555 else: 556 request.state.crawler_verified = False 557 request.state.crawler_operator = None 558 559 return await call_next(request) 560``` 561 562### 6.4 The IP Range Matching Pattern (For Operators Who Publish) 563 564```python 565import asyncio 566import json 567import ipaddress 568import time 569import httpx 570 571# Cache of IP ranges, refreshed periodically 572ip_range_cache = {} 573 574async def refresh_googlebot_ranges(): 575 """Refresh Google's published IP ranges.""" 576 async with httpx.AsyncClient() as client: 577 response = await client.get( 578 "https://developers.google.com/static/search/apis/ipranges/googlebot.json" 579 ) 580 data = response.json() 581 582 networks = [] 583 for entry in data.get("prefixes", []): 584 if "ipv4Prefix" in entry: 585 networks.append(ipaddress.ip_network(entry["ipv4Prefix"])) 586 if "ipv6Prefix" in entry: 587 networks.append(ipaddress.ip_network(entry["ipv6Prefix"])) 588 589 ip_range_cache["googlebot"] = { 590 "networks": networks, 591 "fetched_at": time.time(), 592 } 593 594def is_in_googlebot_range(ip: str) -> bool: 595 """Check if an IP is in Google's published Googlebot range.""" 596 cache = ip_range_cache.get("googlebot") 597 if not cache: 598 return False 599 600 addr = ipaddress.ip_address(ip) 601 return any(addr in network for network in cache["networks"]) 602 603# Refresh on startup and every 6 hours 604@app.on_event("startup") 605async def startup(): 606 await refresh_googlebot_ranges() 607 608 async def refresher(): 609 while True: 610 await asyncio.sleep(6 * 3600) 611 try: 612 await refresh_googlebot_ranges() 613 except Exception: 614 pass 615 616 asyncio.create_task(refresher()) 617``` 618 619### 6.5 The Bubbles Practical Approach 620 621For most Bubbles client sites, full DNS verification on every request is unnecessary overhead. The practical pattern: 622 623* **Logging level (no privilege grant)**: UA pattern matching is sufficient. Tag crawler type in logs and move on. 624* **Rate limit whitelist**: UA pattern matching is sufficient. The cost of false positives (a scraper bypassing rate limits) is small for public marketing pages. 625* **Sensitive endpoint privilege grant**: full verification required. Examples: paywall bypass, admin access, "give Googlebot a peek behind login". 626 627Document the threat model explicitly: for marketing pages, UA matching is fine; for premium content, verification is mandatory. 628 629### 6.6 How To Test Verification 630 631```bash 632# Test with a real Googlebot IP (use one from logs) 633GOOGLEBOT_IP="66.249.66.1" # known Googlebot range 634host "$GOOGLEBOT_IP" 635# Expected: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. 636 637host crawl-66-249-66-1.googlebot.com 638# Expected: crawl-66-249-66-1.googlebot.com has address 66.249.66.1 639 640# Test with a spoofed UA from a non Google IP 641# (Use your own IP for testing) 642MY_IP=$(curl -s ifconfig.me) 643host "$MY_IP" 644# Should NOT show googlebot.com hostname 645 646# Bash one liner to verify 647verify_googlebot() { 648 local ip=$1 649 local ptr=$(host "$ip" 2>/dev/null | awk '/pointer/ {print $NF}' | sed 's/\.$//') 650 case "$ptr" in 651 *.googlebot.com|*.google.com) 652 local forward=$(host "$ptr" 2>/dev/null | awk '/has address/ {print $NF}' | head -1) 653 if [ "$forward" = "$ip" ]; then 654 echo "VERIFIED: $ip is Googlebot ($ptr)" 655 return 0 656 fi 657 ;; 658 esac 659 echo "NOT VERIFIED: $ip (PTR: ${ptr:-none})" 660 return 1 661} 662 663verify_googlebot 66.249.66.1 664``` 665 666--- 667 668## 7. ACCEPT (CONTENT TYPE NEGOTIATION) 669 670### 7.1 What It Does 671 672`Accept` tells the server which content types the client can process. Defined in RFC 9110. The server uses it to pick the best representation when multiple are available. 673 674``` 675Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 676Accept: application/json 677Accept: */* 678``` 679 680The format is a comma separated list of media types, each optionally followed by a quality factor (`q=`). Higher q is more preferred. Absent q means q=1.0. 681 682### 7.2 Typical Values 683 684| Client | Common Accept value | 685|---|---| 686| Modern browser navigating to a page | `text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8` | 687| Browser fetching JSON API via fetch | `application/json,text/plain,*/*` | 688| curl by default | `*/*` | 689| Googlebot rendering page | `text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8` | 690| Image fetcher | `image/avif,image/webp,image/png,image/svg+xml,image/*,*/*;q=0.8` | 691 692### 7.3 How To Access In Nginx 693 694```nginx 695# Variable: $http_accept 696map $http_accept $client_wants_json { 697 default 0; 698 ~*application/json 1; 699} 700 701map $http_accept $client_wants_webp { 702 default 0; 703 ~*image/webp 1; 704} 705``` 706 707### 7.4 How To Access In FastAPI 708 709```python 710@app.get("/data") 711async def data(request: Request): 712 accept = request.headers.get("accept", "") 713 if "application/json" in accept: 714 return JSONResponse({"data": "..."}) 715 else: 716 return HTMLResponse("<html>...</html>") 717``` 718 719FastAPI's automatic content negotiation handles common cases; for complex negotiation use the `accept-types` library: 720 721```python 722from accept_types import get_best_match 723 724@app.get("/data") 725async def data(request: Request): 726 accept = request.headers.get("accept", "*/*") 727 best = get_best_match(accept, ["application/json", "text/html", "application/xml"]) 728 if best == "application/json": 729 return JSONResponse({...}) 730 elif best == "application/xml": 731 return Response(content="<xml>...</xml>", media_type="application/xml") 732 else: 733 return HTMLResponse("<html>...</html>") 734``` 735 736### 7.5 The Vary: Accept Requirement 737 738If the response varies based on Accept (different representation for JSON vs HTML), set `Vary: Accept`: 739 740```nginx 741add_header Vary "Accept" always; 742``` 743 744Without this, caches may serve the JSON variant to a browser asking for HTML, or vice versa. 745 746### 7.6 The Modern Image Format Negotiation Pattern 747 748For sites serving AVIF or WebP to capable browsers and falling back to JPEG/PNG for others: 749 750```nginx 751map $http_accept $image_format { 752 default "jpg"; 753 ~*image/avif "avif"; 754 ~*image/webp "webp"; 755} 756 757location ~* /images/(.+)\.(jpg|png)$ { 758 set $base $1; 759 set $ext $2; 760 761 # Try the modern format first based on Accept, fall back to original 762 try_files /images/$base.$image_format /images/$base.$ext =404; 763 764 add_header Vary "Accept" always; 765} 766``` 767 768When a browser supports AVIF and requests `/images/hero.jpg`, this serves `/images/hero.avif` if it exists, falling back to `/images/hero.jpg`. 769 770### 7.7 Troubleshooting 771 772**Symptom: cache serves JSON to a browser navigation request.** 773The endpoint varies on Accept but `Vary: Accept` is missing. Add it. 774 775**Symptom: crawler always gets HTML even when requesting JSON.** 776Verify the crawler is actually sending `Accept: application/json`. Some crawlers send `*/*` and rely on URL extension to distinguish. 777 778**Symptom: AVIF served to a browser that does not support it, image broken.** 779The browser is sending an outdated or compatibility Accept header. Or the AVIF file is corrupt. Check both. 780 781--- 782 783## 8. ACCEPT-LANGUAGE (LANGUAGE AND LOCALE NEGOTIATION) 784 785### 8.1 What It Does 786 787`Accept-Language` tells the server which natural languages the client prefers. Defined in RFC 9110. Uses BCP 47 language tags. Used by browsers to pick the right localized version and by some crawlers (Bingbot, Baiduspider) for language signal. 788 789``` 790Accept-Language: en-US,en;q=0.9 791Accept-Language: es-MX,es;q=0.9,en;q=0.8 792Accept-Language: fr-FR,fr;q=0.9,en;q=0.5 793``` 794 795### 8.2 Format 796 797Comma separated list of language tags with optional quality factors: 798 799* `en-US`: US English 800* `en`: any English 801* `es-MX`: Mexican Spanish 802* `q=0.9`: 90% preference (lower than absent which is 1.0) 803 804The client typically lists their browser language first at high q, then fallback languages at lower q. 805 806### 8.3 How To Access In Nginx 807 808```nginx 809# Variable: $http_accept_language 810 811# Simple language detection 812map $http_accept_language $preferred_language { 813 default "en"; 814 ~*^es "es"; 815 ~*^fr "fr"; 816 ~*^de "de"; 817 ~*^zh "zh"; 818} 819 820# Redirect to language specific path 821location = / { 822 return 302 /$preferred_language/; 823} 824``` 825 826### 8.4 How To Access In FastAPI 827 828```python 829@app.get("/") 830async def index(request: Request): 831 accept_lang = request.headers.get("accept-language", "") 832 preferred = parse_preferred_language(accept_lang) 833 return RedirectResponse(f"/{preferred}/") 834 835def parse_preferred_language(header: str) -> str: 836 """Parse Accept-Language and return the top language code.""" 837 if not header: 838 return "en" 839 # Get first entry, strip quality 840 first = header.split(",")[0].split(";")[0].strip().lower() 841 # Get language part (en, es, fr) 842 return first.split("-")[0] if "-" in first else first 843``` 844 845### 8.5 The Vary: Accept-Language Requirement 846 847If the response varies based on Accept-Language, set `Vary: Accept-Language`: 848 849```nginx 850add_header Vary "Accept-Language" always; 851``` 852 853### 8.6 Why Hreflang Is Preferred Over Accept-Language Routing 854 855Google's official recommendation for multilingual sites is `hreflang` (Link header or HTML element), not Accept-Language based redirects. Reasons: 856 857* Googlebot does not send Accept-Language matching a target locale. 858* Accept-Language based redirects can confuse the user (clicking a French link gets you redirected to Spanish based on browser settings). 859* Hreflang provides explicit URLs Google can crawl independently. 860 861The Bubbles convention: use Accept-Language to suggest a language switcher in the UI but do not auto redirect. Use hreflang annotations in HTML and Link header for crawlers. 862 863### 8.7 Troubleshooting 864 865**Symptom: User typed example.com/page but got redirected to example.com/es/page they did not want.** 866Auto redirect based on Accept-Language. Bad UX. Remove and use hreflang plus a language switcher UI instead. 867 868**Symptom: Googlebot only crawls English pages.** 869Hreflang is not set up. Add hreflang annotations. Googlebot will discover and crawl other language variants. 870 871**Symptom: Bingbot crawls Spanish pages from a Spanish IP.** 872Bingbot does send Accept-Language sometimes. This is correct behavior. 873 874--- 875 876## 9. ACCEPT-ENCODING (COMPRESSION NEGOTIATION, THE GZIP PLUS BROTLI PLUS ZSTD STACK) 877 878### 9.1 What It Does 879 880`Accept-Encoding` tells the server which compression algorithms the client can decompress. Defined in RFC 9110. The server picks the best supported algorithm and sets `Content-Encoding` on the response. 881 882``` 883Accept-Encoding: gzip, deflate, br, zstd 884Accept-Encoding: gzip, deflate, br 885Accept-Encoding: gzip 886Accept-Encoding: identity 887``` 888 889### 9.2 The 2026 Compression Landscape 890 891| Algorithm | Identifier | Browser Support | Compression Ratio | Decompression Speed | 892|---|---|---|---|---| 893| **gzip** | `gzip` | Universal (since 1990s) | Baseline (good) | Fast | 894| **brotli** | `br` | Chrome, Firefox, Safari, Edge (since ~2017) | 15-25% better than gzip | Medium | 895| **zstd** | `zstd` | Chrome 123+, Firefox 126+, Edge 123+ | Similar to brotli, faster decompression | Very fast | 896| **deflate** | `deflate` | Universal but discouraged | Baseline | Fast | 897| **identity** | `identity` | Always supported | No compression | Trivial | 898 899Modern browsers in 2026 send `Accept-Encoding: gzip, deflate, br, zstd`. The server should support gzip plus at least one of br or zstd. 900 901Safari notable gap: as of late 2025, Safari does NOT advertise zstd. iOS users still fall back to brotli or gzip. Brotli is the most universal modern format; zstd is the most aggressive choice. 902 903### 9.3 How To Access In Nginx 904 905```nginx 906# Variable: $http_accept_encoding 907 908# Modules required for server side compression: 909# - gzip: built into nginx (always available) 910# - brotli: requires libnginx-mod-brotli (apt install) 911# - zstd: requires nginx-module-zstd (third party) 912 913http { 914 # gzip (universal fallback) 915 gzip on; 916 gzip_vary on; 917 gzip_min_length 256; 918 gzip_comp_level 6; 919 gzip_types 920 text/plain text/css text/xml application/json application/javascript 921 application/xml application/atom+xml application/rss+xml 922 image/svg+xml font/ttf font/otf font/woff font/woff2; 923 924 # brotli (modern browsers) 925 brotli on; 926 brotli_static on; # serve pre compressed .br files when available 927 brotli_comp_level 5; 928 brotli_min_length 256; 929 brotli_types 930 text/plain text/css text/xml application/json application/javascript 931 application/xml application/atom+xml application/rss+xml 932 image/svg+xml font/ttf font/otf font/woff font/woff2; 933 934 # zstd (cutting edge, Chrome 123+) 935 zstd on; 936 zstd_static on; 937 zstd_comp_level 6; 938 zstd_min_length 256; 939 zstd_types 940 text/plain text/css text/xml application/json application/javascript 941 application/xml image/svg+xml; 942} 943``` 944 945Nginx automatically negotiates the best supported algorithm based on the request's Accept-Encoding header. 946 947### 9.4 The Vary: Accept-Encoding Requirement 948 949When compression is in use, `Vary: Accept-Encoding` is mandatory: 950 951```nginx 952gzip_vary on; # nginx adds Vary: Accept-Encoding automatically when gzipping 953``` 954 955Without this, caches may serve a gzipped response to a client that requested no compression, breaking the response. 956 957### 9.5 The Pre Compressed Static Files Pattern 958 959For maximum efficiency, pre compress static files at build time and let nginx serve them directly: 960 961```bash 962# Build time: pre compress with highest levels (no runtime CPU cost) 963cd /var/www/sites/example.com 964for file in *.html *.css *.js; do 965 gzip -k -9 "$file" # creates file.gz 966 brotli -k -q 11 "$file" # creates file.br 967 zstd -k -19 "$file" # creates file.zst 968done 969``` 970 971```nginx 972# nginx serves the pre compressed version when supported 973gzip_static on; 974brotli_static on; 975zstd_static on; 976``` 977 978For a typical HTML page, this means: brotli level 11 (slow at build time, but only done once) yields about 20% smaller files than gzip level 6 (fast but less efficient). The savings compound across many requests. 979 980### 9.6 Troubleshooting 981 982**Symptom: client sends `Accept-Encoding: gzip, br, zstd` but receives uncompressed response.** 983Possible causes: 9841. Response Content-Type not in `gzip_types` / `brotli_types` / `zstd_types`. Add it. 9852. Response shorter than `gzip_min_length`. Lower the threshold or accept. 9863. nginx compression modules not loaded. Verify with `nginx -V 2>&1 | grep -E "gzip|brotli|zstd"`. 9874. Already compressed (image, video). nginx correctly skips these. 988 989**Symptom: Content-Encoding header on response but content is double encoded or broken.** 990Two compression layers are interfering. Check if a proxy or upstream is also compressing. Disable one. 991 992**Symptom: cache serves gzipped response to client that requested no compression.** 993Missing `Vary: Accept-Encoding`. Add `gzip_vary on;`. 994 995**Symptom: Safari user reports site is slow; brotli not negotiated.** 996Safari might be sending only `gzip, deflate` (older Safari versions). Brotli was added later. Verify in Safari's Network panel that `br` is in the Accept-Encoding. 997 998**Symptom: Chrome 123+ user complains about broken downloads.** 999nginx is serving zstd to a client whose handling broke for some reason. Disable zstd temporarily and observe. Common when third party tools (antivirus, corporate proxies) get in the middle and do not understand zstd yet. 1000 1001### 9.7 Crawler Encoding Behavior 1002 1003Googlebot sends `Accept-Encoding: gzip` (only gzip historically). Bingbot sends `gzip, deflate`. ClaudeBot and GPTBot also typically advertise gzip. None of the major crawlers advertise brotli or zstd as of 2026. 1004 1005For crawlers, gzip is sufficient. The brotli and zstd savings benefit human visitors using modern browsers, which is most traffic. 1006 1007--- 1008 1009## 10. ACCEPT-CHARSET (LEGACY CHARSET NEGOTIATION) 1010 1011### 10.1 What It Does 1012 1013`Accept-Charset` was the original character encoding negotiation header. Defined in RFC 9110 but deprecated in practice. Modern HTTP assumes UTF-8 everywhere. 1014 1015``` 1016Accept-Charset: utf-8, iso-8859-1;q=0.5 1017Accept-Charset: utf-8 1018``` 1019 1020### 10.2 The Reality In 2026 1021 1022Modern browsers and major HTTP libraries do NOT send Accept-Charset. The MDN documentation states: "Browsers omit this header, as it is no longer needed for content negotiation; servers ignore it." 1023 1024For Bubbles deployments: ignore Accept-Charset entirely. Always serve UTF-8. The Content-Type header (from framework-http-content-headers.md) declares charset=utf-8 explicitly: 1025 1026``` 1027Content-Type: text/html; charset=utf-8 1028``` 1029 1030### 10.3 Edge Cases Where Accept-Charset Might Matter 1031 1032* Legacy enterprise software still parsing HTTP 1.0 era headers. 1033* Some older Asian language localizations (Shift_JIS, Big5). 1034* Embedded systems with limited UTF-8 support. 1035 1036These are rare. For typical web traffic, treat Accept-Charset as inert. 1037 1038### 10.4 If You Must Handle It 1039 1040```nginx 1041# Default to UTF-8 unless the client explicitly says otherwise 1042map $http_accept_charset $response_charset { 1043 default "utf-8"; 1044 ~*iso-8859-1 "iso-8859-1"; # legacy support if needed 1045 ~*shift_jis "shift_jis"; # Japanese legacy 1046} 1047``` 1048 1049In practice, do not do this. Serve UTF-8 always. Convert content at the application layer if a specific edge case requires another charset. 1050 1051--- 1052 1053## 11. FROM (LEGACY CONTACT IDENTIFIER SOME BOTS USE) 1054 1055### 11.1 What It Does 1056 1057`From` is a request header containing an email address identifying the human controlling the requesting agent. Defined in RFC 9110. Largely unused in 2026 but still appears in some crawler traffic. 1058 1059``` 1060From: googlebot(at)googlebot.com 1061From: webmaster@example.com 1062``` 1063 1064### 11.2 The Reality In 2026 1065 1066Browsers do NOT send this header. Some courteous crawlers and scrapers include it as a "if you have a problem, contact this email" gesture. It is informational, not actionable. 1067 1068Examples of crawlers that may include From: 1069 1070* Custom academic crawlers and research bots. 1071* Older versions of Googlebot (rare in 2026). 1072* Polite scrapers that want to be reachable if blocked. 1073 1074### 11.3 How To Access 1075 1076```nginx 1077# Variable: $http_from 1078 1079# Log it for crawler accountability 1080log_format with_from '$remote_addr - $remote_user [$time_local] ' 1081 '"$request" $status $body_bytes_sent ' 1082 '"$http_referer" "$http_user_agent" ' 1083 'from="$http_from"'; 1084``` 1085 1086```python 1087@app.get("/") 1088async def index(request: Request): 1089 from_email = request.headers.get("from", "") 1090 # Log for accountability if it is a crawler 1091``` 1092 1093### 11.4 What To Do With From 1094 1095* **Log it.** If a crawler causes problems, the From header gives you an email address. 1096* **Do not authenticate by it.** It is freely settable to any value. 1097* **Do not use for access control.** A spoofed From email grants no real identity. 1098 1099### 11.5 Practical Bubbles Use 1100 1101For sites Joseph operates, the From header is essentially never actionable. Log it for completeness; do not invest in tooling around it. 1102 1103--- 1104 1105## 12. HOST (VIRTUAL HOST SELECTION) 1106 1107### 12.1 What It Does 1108 1109`Host` tells the server which virtual host the request is for. Defined in RFC 9110. Mandatory for HTTP/1.1 and later. Allows multiple sites to share a single IP. 1110 1111``` 1112Host: example.com 1113Host: www.example.com 1114Host: api.example.com 1115Host: example.com:8443 1116``` 1117 1118For Bubbles, the Host header is what nginx uses to pick which `server_name` matches and which `server` block to route to. With 181 hostnames sharing 169.155.162.118, the Host header is the critical routing key. 1119 1120### 12.2 How nginx Uses Host 1121 1122```nginx 1123server { 1124 listen 443 ssl; 1125 server_name example.com www.example.com; 1126 # ... 1127} 1128 1129server { 1130 listen 443 ssl; 1131 server_name api.example.com; 1132 # ... 1133} 1134 1135# The default server catches any unmatched Host header 1136server { 1137 listen 443 ssl default_server; 1138 server_name _; 1139 return 444; # close connection without response 1140} 1141``` 1142 1143nginx compares the Host header against `server_name` directives. The first match (with priority: exact match > leading wildcard > trailing wildcard > regex > default_server) wins. 1144 1145### 12.3 The default_server Requirement 1146 1147Without a `default_server`, nginx falls back to the first server block defined for the listen address. This can produce wrong routing for unmatched Host headers (cache poisoning, information disclosure, accidental SSL certificate use). 1148 1149**Mandatory pattern:** every Bubbles deployment has a default_server returning 444 (connection close) for unmatched Hosts: 1150 1151```nginx 1152server { 1153 listen 80 default_server; 1154 listen [::]:80 default_server; 1155 listen 443 ssl default_server; 1156 listen [::]:443 ssl default_server; 1157 listen 443 quic reuseport default_server; 1158 1159 server_name _; 1160 1161 # Use a snake oil certificate for default SSL 1162 ssl_certificate /etc/ssl/snakeoil/cert.pem; 1163 ssl_certificate_key /etc/ssl/snakeoil/key.pem; 1164 1165 # Reject any request that did not match a known server_name 1166 return 444; 1167} 1168``` 1169 1170444 is a non standard nginx code that closes the connection without sending a response. The client gets a connection reset. 1171 1172### 12.4 How To Access In Nginx 1173 1174```nginx 1175# Variable: $http_host (the raw Host header value) 1176# Variable: $host (Host header value, falls back to server_name if not set) 1177 1178log_format with_host '$remote_addr - [$time_local] ' 1179 '"$request" $status host="$host"'; 1180``` 1181 1182`$host` is preferred over `$http_host` for most uses because it handles edge cases (very old HTTP/1.0 clients without Host header) gracefully. 1183 1184### 12.5 How To Access In FastAPI 1185 1186```python 1187@app.get("/") 1188async def index(request: Request): 1189 host = request.headers.get("host", "") 1190 # When behind nginx with proxy_set_header Host $host: 1191 # this is the actual client requested Host 1192``` 1193 1194In FastAPI behind nginx, ensure `proxy_set_header Host $host;` is set so the upstream sees the real Host value, not the upstream's internal hostname. 1195 1196### 12.6 The Host Header Injection Threat 1197 1198Host header injection occurs when an application uses the Host header value to construct URLs (in emails, redirects, etc) without validation: 1199 1200```python 1201# DANGEROUS 1202@app.post("/api/reset-password") 1203async def reset_password(request: Request, email: str): 1204 host = request.headers.get("host") 1205 reset_url = f"https://{host}/reset?token={token}" 1206 send_email(email, reset_url) 1207``` 1208 1209An attacker sends a request to `example.com` with `Host: evil.com`. The application sends `evil.com/reset?token=...` to the user. User clicks, the token goes to attacker. 1210 1211**The fix:** use a configured value, not the request's Host: 1212 1213```python 1214# SAFE 1215ALLOWED_HOSTS = ["example.com", "www.example.com"] 1216CANONICAL_HOST = "example.com" 1217 1218@app.post("/api/reset-password") 1219async def reset_password(request: Request, email: str): 1220 reset_url = f"https://{CANONICAL_HOST}/reset?token={token}" 1221 send_email(email, reset_url) 1222``` 1223 1224Or validate the Host header before using: 1225 1226```python 1227host = request.headers.get("host", "") 1228if host not in ALLOWED_HOSTS: 1229 raise HTTPException(status_code=400, detail="invalid host") 1230``` 1231 1232### 12.7 The Bubbles Pattern: One Server Block Per Tenant 1233 1234Joseph's Bubbles serves 27 custom domains plus 40 dedicated subdomains plus 119 wildcard subdomains. Each gets its own routing: 1235 1236```nginx 1237# Custom domain 1238server { 1239 listen 443 ssl; 1240 server_name client1.com www.client1.com; 1241 # ... 1242} 1243 1244# Wildcard subdomain (catch all *.thatwebhostingguy.com) 1245server { 1246 listen 443 ssl; 1247 server_name *.thatwebhostingguy.com thatwebhostingguy.com; 1248 # Logic to map subdomain to client directory 1249 set $subdomain ""; 1250 if ($host ~ "^([^.]+)\.thatwebhostingguy\.com$") { 1251 set $subdomain $1; 1252 } 1253 root /var/www/sites/$subdomain; 1254} 1255 1256# Default (catches everything else, 444s it) 1257server { 1258 listen 443 ssl default_server; 1259 server_name _; 1260 return 444; 1261} 1262``` 1263 1264The Host header is the decision point. nginx does the routing based on it; the application sees a normalized Host via `proxy_set_header Host $host`. 1265 1266### 12.8 Troubleshooting 1267 1268**Symptom: requests landing on wrong server block.** 12691. server_name patterns conflict. Check priority order (exact > wildcard > regex). 12702. SSL certificate does not match Host. Different cert per server is fine; mismatched certs are not. 12713. No default_server, fallback is wrong. 12724. Host header malformed (port mismatch, IP literal instead of hostname). 1273 1274**Symptom: cache poisoning via Host header.** 1275Application is using Host without validation. Restrict to configured canonical hostnames. 1276 1277**Symptom: SSL handshake error for some clients.** 1278SNI (Server Name Indication) failure. Very old clients without SNI cannot reach name based virtual hosts. They get the default_server's certificate, fail validation. Few clients affected in 2026. 1279 1280--- 1281 1282## 13. IF-MODIFIED-SINCE (CONDITIONAL GET BY DATE) 1283 1284### 13.1 What It Does 1285 1286`If-Modified-Since` is sent by clients (especially crawlers and caches) to say "I have a copy from this date; only send the body if it has changed since". Defined in RFC 9110. Pairs with the `Last-Modified` response header from framework-http-caching-headers.md. 1287 1288``` 1289If-Modified-Since: Wed, 22 May 2026 14:30:00 GMT 1290``` 1291 1292If the resource has not changed since that date, the server returns `304 Not Modified` with no body. The client uses its cached copy. Massive bandwidth savings for both server and client. 1293 1294### 13.2 How It Works End To End 1295 12961. Initial request: client GETs `/page`. Server responds 200 with body and `Last-Modified: Wed, 22 May 2026 14:30:00 GMT`. 12972. Subsequent request: client sends `If-Modified-Since: Wed, 22 May 2026 14:30:00 GMT`. 12983. Server compares to file's mtime or content fingerprint: 1299 * **Unchanged**: returns 304 Not Modified, no body. Client uses cached copy. 1300 * **Changed**: returns 200 with new body and new Last-Modified. 1301 1302Googlebot specifically requests this behavior. From Google's documentation: "Googlebot's crawlers don't send the headers with all crawl attempts; it depends on the use case. AdsBot is more likely to set the If-Modified-Since and If-None-Match HTTP request headers." 1303 1304### 13.3 How nginx Handles It 1305 1306For static files, nginx handles If-Modified-Since automatically. The file's mtime becomes the Last-Modified value, and incoming If-Modified-Since is compared automatically. 1307 1308```nginx 1309# Default behavior: nginx handles conditional GET on static files 1310location /assets/ { 1311 root /var/www/sites/example.com; 1312 # nginx compares mtime against If-Modified-Since 1313 # Returns 304 if unchanged 1314} 1315``` 1316 1317For dynamic content, the upstream must handle it: 1318 1319```nginx 1320location /api/ { 1321 proxy_pass http://127.0.0.1:9090; 1322 proxy_cache_revalidate on; # nginx honors upstream's revalidation 1323} 1324``` 1325 1326### 13.4 How FastAPI Handles It 1327 1328FastAPI does not automatically handle conditional GET; the upstream code must: 1329 1330```python 1331from email.utils import formatdate, parsedate_to_datetime 1332from datetime import datetime, timezone 1333from fastapi import FastAPI, Request, Response 1334 1335app = FastAPI() 1336 1337@app.get("/dynamic-page") 1338async def dynamic_page(request: Request): 1339 # Compute or look up the resource's Last-Modified time 1340 last_modified = datetime(2026, 5, 22, 14, 30, 0, tzinfo=timezone.utc) 1341 1342 # Check If-Modified-Since 1343 ims = request.headers.get("if-modified-since") 1344 if ims: 1345 try: 1346 ims_date = parsedate_to_datetime(ims) 1347 if last_modified <= ims_date: 1348 return Response(status_code=304) 1349 except (TypeError, ValueError): 1350 pass # malformed header, ignore 1351 1352 # Generate full response with Last-Modified 1353 return Response( 1354 content=generate_content(), 1355 headers={ 1356 "Last-Modified": formatdate(last_modified.timestamp(), usegmt=True), 1357 "Cache-Control": "public, max-age=300", 1358 } 1359 ) 1360``` 1361 1362### 13.5 Why It Saves Crawler Budget 1363 1364Googlebot's crawl budget for a site is finite. Each unchanged page that returns 200 with full body wastes: 1365 1366* Server CPU and bandwidth. 1367* Crawler bandwidth and processing time. 1368* Crawl budget that could be spent on new or changed pages. 1369 1370A site that returns 304 for unchanged pages can effectively double or triple its crawl coverage. New pages get indexed faster because the crawler is not wasting time on unchanged old ones. 1371 1372### 13.6 The Date Format Strictness 1373 1374The HTTP date format is strict (RFC 7231 IMF-fixdate): 1375 1376``` 1377Wed, 22 May 2026 14:30:00 GMT 1378``` 1379 1380Common errors: 1381 1382* Missing comma after day name. 1383* Day not zero padded (`22` correct, `2` wrong). 1384* Time zone other than GMT (UTC is wrong even though equivalent). 1385* Year not four digits. 1386* Day name extension like `Wednesday`. 1387 1388The Python `email.utils.formatdate(stamp, usegmt=True)` produces the correct format. 1389 1390### 13.7 Troubleshooting 1391 1392**Symptom: Googlebot never sends If-Modified-Since.** 1393Googlebot does not always send it (depends on use case). Not a misconfiguration on your side. 1394 1395**Symptom: server returns 200 to revisits despite content unchanged.** 13961. Server is not comparing If-Modified-Since against Last-Modified. 13972. Dynamic content with no Last-Modified concept (always considered "fresh"). 13983. Application overrode 304 logic. 1399Fix: in FastAPI, add the conditional GET check (Section 13.4). 1400 1401**Symptom: 304 returned with body (bug).** 1402304 must have no body per RFC. Some buggy frameworks send 304 with content. Verify by: 1403 1404```bash 1405curl -sI -H "If-Modified-Since: Wed, 22 May 2026 14:30:00 GMT" https://example.com/ 1406# Expected: HTTP/2 304 with no body 1407``` 1408 1409**Symptom: Clock skew causes incorrect 304s.** 1410Client and server clocks differ. Use ETag instead (Section 14) which is independent of clock. 1411 1412--- 1413 1414## 14. IF-NONE-MATCH (CONDITIONAL GET BY ETAG) 1415 1416### 14.1 What It Does 1417 1418`If-None-Match` is sent by clients to say "I have content with this ETag value; only send body if the current ETag differs". Defined in RFC 9110. Pairs with the `ETag` response header from framework-http-caching-headers.md. 1419 1420``` 1421If-None-Match: "abc123" 1422If-None-Match: "abc123", "def456" 1423If-None-Match: W/"weak-tag-123" 1424If-None-Match: * 1425``` 1426 1427Unlike If-Modified-Since (date based), ETag based revalidation is unaffected by clock skew and can represent any content fingerprint (hash, version number, opaque ID). 1428 1429### 14.2 How It Works End To End 1430 14311. Initial request: client GETs `/page`. Server responds 200 with body and `ETag: "abc123"`. 14322. Subsequent request: client sends `If-None-Match: "abc123"`. 14333. Server compares to current resource's ETag: 1434 * **Match**: returns 304 Not Modified, no body. 1435 * **No match**: returns 200 with new body and new ETag. 1436 1437### 14.3 Strong vs Weak ETags 1438 1439``` 1440ETag: "abc123" (strong: byte for byte identical) 1441ETag: W/"abc123" (weak: semantically equivalent, possibly different bytes) 1442``` 1443 1444* **Strong ETag**: matches only if the content is byte for byte identical. Used for partial range requests, byte level caching. 1445* **Weak ETag**: matches if content is semantically equivalent. Common for dynamic responses where minor variations exist (timestamps, ad rotations) but the substance is unchanged. 1446 1447The `W/` prefix marks a weak ETag. 1448 1449### 14.4 How nginx Handles It 1450 1451nginx auto generates strong ETags for static files from the file's size and mtime: 1452 1453```nginx 1454# Default behavior on static files 1455# nginx auto generates ETag and compares If-None-Match 1456 1457location /assets/ { 1458 root /var/www/sites/example.com; 1459 etag on; # default 1460} 1461``` 1462 1463For dynamic content, the upstream handles it. 1464 1465### 14.5 How FastAPI Handles It 1466 1467```python 1468import hashlib 1469 1470@app.get("/dynamic-page") 1471async def dynamic_page(request: Request): 1472 content = generate_content() 1473 etag = '"' + hashlib.md5(content.encode()).hexdigest()[:16] + '"' 1474 1475 # Check If-None-Match 1476 inm = request.headers.get("if-none-match", "") 1477 if etag in [tag.strip() for tag in inm.split(",")]: 1478 return Response(status_code=304, headers={"ETag": etag}) 1479 1480 return Response( 1481 content=content, 1482 headers={ 1483 "ETag": etag, 1484 "Cache-Control": "public, max-age=300", 1485 } 1486 ) 1487``` 1488 1489### 14.6 If-Modified-Since And If-None-Match Together 1490 1491A client can send both. The RFC specifies that **ETag takes precedence**: 1492 1493```python 1494def is_not_modified(request, etag, last_modified): 1495 inm = request.headers.get("if-none-match") 1496 if inm is not None: 1497 # ETag check takes precedence 1498 return etag in inm.split(",") 1499 1500 ims = request.headers.get("if-modified-since")