A continuation of Janady13/crawler-signal-reference / framework-http-request-headers.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓2,892 lines · 107.6 KB
if ims:
# Fall back to date check
try:
ims_date = parsedate_to_datetime(ims)
return last_modified <= ims_date
except (TypeError, ValueError):
return False
return False
```
### 14.7 The Wildcard If-None-Match
```
If-None-Match: *
```
The wildcard `*` matches any existing resource. Used for "only respond if the resource does not exist" semantics (e.g., conditional PUT to create only if absent). Rare in GET contexts.
### 14.8 Troubleshooting
**Symptom: 304 returned but the client still seems to make a full request.**
The 304 response should be tiny (headers only). If the client is downloading body, something is wrong:
1. Check Content-Length is 0 on the 304 response.
2. Check that nginx is not adding a body to 304s.
**Symptom: every request gets a new ETag despite content unchanged.**
ETag is being generated from variable data (current timestamp, request ID, random nonce). It must be derived from content. Use a hash of the content body.
**Symptom: 304 served when content has changed.**
ETag is too coarse or based on stale data. Regenerate ETag on every content update.
---
## 15. REFERER (REQUEST ORIGIN, OFTEN ABSENT FOR CRAWLERS)
### 15.1 What It Does
`Referer` (note the historical misspelling preserved in the spec) tells the server which page the request came from. Defined in RFC 9110. Used for analytics, security checks, and content adaptation.
```
Referer: https://example.com/products
Referer: https://www.google.com/
(absent: no Referer sent)
```
The Referer value is the full URL of the referring page (or just the origin, depending on the Referrer-Policy on the source page, see framework-http-security-headers.md).
### 15.2 What Crawlers Send
Most crawlers send Referer when crawling links. The Referer is typically the URL where the link was found:
```
Referer: https://example.com/products
Request: GET /product/widget HTTP/2
```
This tells you the crawler followed a link from `/products` to `/product/widget`. Useful for analytics: see what entry points drive crawl traffic.
Some crawlers do NOT send Referer:
* Direct robots.txt or sitemap fetches.
* Many user initiated AI fetches (Claude-User, ChatGPT-User, Perplexity-User often have empty Referer).
* Privacy aware browsers behind a strict Referrer-Policy.
### 15.3 How To Access
```nginx
# Variable: $http_referer
log_format with_referer '$remote_addr - [$time_local] '
'"$request" $status '
'ref="$http_referer"';
```
```python
@app.get("/page")
async def page(request: Request):
referer = request.headers.get("referer", "")
# Use for analytics, not security
```
### 15.4 Common Server Side Uses
**Acceptable:**
* Analytics: track inbound traffic sources.
* Identifying broken links: if Referer is your own site and the request is a 404, you have a broken internal link.
* Content adaptation hints: tweak content based on entry point (e.g., highlight a specific feature if visitor came from a feature page).
**Unacceptable (Referer cannot be trusted):**
* Security checks ("only allow if Referer is our own page"). Referer can be omitted, spoofed, or set to anything.
* Authentication or authorization.
* CSRF protection (use proper CSRF tokens instead).
### 15.5 The Hotlink Protection Antipattern
Historically, sites used Referer to block hotlinking (other sites linking directly to images):
```nginx
# DANGEROUS: easy to bypass and breaks legitimate uses
location /images/ {
valid_referers none blocked example.com *.example.com;
if ($invalid_referer) {
return 403;
}
}
```
Problems:
1. Trivially bypassed (curl with custom Referer).
2. Breaks legitimate uses: bookmark managers, archive.org, password managers, IDE previews.
3. Privacy aware browsers send empty Referer; this breaks them.
If you genuinely need to prevent hotlinking, use signed URLs or token based access instead.
### 15.6 Logging Referer For Analytics
Most useful pattern: log Referer and analyze in batch:
```nginx
log_format ref '$remote_addr [$time_local] "$request" $status '
'"$http_referer" "$http_user_agent"';
access_log /var/log/nginx/access.log ref;
```
Analyze:
```bash
# Top referring domains
awk -F'"' '{print $4}' /var/log/nginx/access.log | \
grep -oE "https?://[^/]+" | sort | uniq -c | sort -rn | head -20
```
### 15.7 The Referer Field Type
Even though "referrer" is the correct English spelling, the HTTP header is `Referer` (with the misspelling). The CSS property `referrer` (no `Referer`), the JavaScript property `document.referrer`, and the security header `Referrer-Policy` all use the correct spelling. Only the HTTP request header retains the historical typo.
### 15.8 Troubleshooting
**Symptom: Referer often empty for legitimate browser traffic.**
The source page has `Referrer-Policy: no-referrer` or `same-origin`. The visitor's browser is following the policy. Cannot fix from your side.
**Symptom: hotlink protection breaks for legitimate users.**
Remove the Referer check. Use signed URLs or token based protection instead.
**Symptom: analytics shows huge volume of empty Referer traffic.**
Mix of direct visits, privacy browsers, and crawlers. Normal. Treat empty Referer as a category in its own right.
---
## 16. CONNECTION (TRANSPORT DISPOSITION)
### 16.1 What It Does
`Connection` controls what happens to the TCP connection after the response. Defined in RFC 9110. In HTTP/2 and HTTP/3, much of its function is absorbed by the protocol itself.
```
Connection: keep-alive
Connection: close
Connection: upgrade
```
### 16.2 Values
| Value | Meaning |
|---|---|
| `keep-alive` | Reuse this connection for subsequent requests. HTTP/1.1 default. Allows pipelining and avoids TCP setup overhead |
| `close` | Close the connection after this response. Forces a new TCP setup for the next request |
| `upgrade` | Switch protocol (WebSocket: `Connection: Upgrade, Upgrade: websocket`) |
In HTTP/2 and HTTP/3, connections are persistent by default and Connection header is largely irrelevant (or required to be ignored).
### 16.3 How To Access In Nginx
```nginx
# Variable: $http_connection
# Log it (rarely necessary)
log_format with_conn '$remote_addr [$time_local] "$request" $status '
'conn="$http_connection"';
# Required for WebSocket upgrade
location /ws {
proxy_pass http://127.0.0.1:9090;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
```
### 16.4 The WebSocket Upgrade Pattern
When the client wants to upgrade an HTTP connection to WebSocket, it sends:
```
GET /ws HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
```
nginx must explicitly proxy this:
```nginx
map $http_upgrade $connection_upgrade {
default upgrade;
"" close;
}
location /ws {
proxy_pass http://127.0.0.1:9090;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 86400; # 24 hours for long lived WebSocket
}
```
Without this `Connection: upgrade` passthrough, WebSocket negotiation fails.
### 16.5 Crawler Connection Behavior
Most crawlers prefer `keep-alive` to amortize TCP and TLS setup costs across many requests:
* Googlebot uses HTTP/2 (when supported) which is keep-alive by default.
* ClaudeBot and GPTBot use HTTP/1.1 with keep-alive.
* Older crawlers may explicitly send `Connection: close` for each request.
Either way, nginx handles connection management automatically. No action required.
### 16.6 Troubleshooting
**Symptom: WebSocket connection fails to establish.**
nginx is not passing Connection: upgrade to upstream. Add the `proxy_set_header Connection "upgrade"` configuration.
**Symptom: long lived WebSocket disconnects after 60 seconds.**
nginx default `proxy_read_timeout` is 60 seconds. Increase for WebSocket:
```nginx
proxy_read_timeout 86400; # 24 hours
proxy_send_timeout 86400;
```
**Symptom: HTTP/2 client sends Connection header.**
HTTP/2 spec says this is invalid but some clients still send. nginx ignores per spec. No action needed.
---
## 17. HOW THESE HEADERS INTERACT
Several specific interactions matter.
### 17.1 Accept-Encoding And Vary
When compression is applied, `Vary: Accept-Encoding` is mandatory on the response (covered in framework-http-caching-headers.md). Caches that ignore this serve gzipped responses to clients that did not request compression, breaking them.
### 17.2 If-None-Match And ETag (Strong/Weak Mode)
Range requests require strong ETags. Weak ETags cannot be used with `If-Range` for partial content negotiation.
### 17.3 If-Modified-Since And If-None-Match On Same Request
When both are present, ETag (If-None-Match) takes precedence. If-Modified-Since is ignored. Document this clearly when implementing conditional GET logic.
### 17.4 User-Agent And Accept-Encoding (Crawler Compression Behavior)
Different crawlers advertise different Accept-Encoding:
| Crawler | Accept-Encoding |
|---|---|
| Googlebot | `gzip, deflate, br` (recent) or `gzip` (older) |
| Bingbot | `gzip, deflate` |
| ClaudeBot | `gzip` |
| GPTBot | `gzip` |
| Modern browser | `gzip, deflate, br, zstd` |
For Bubbles: brotli and zstd savings are for human visitors. Crawlers get gzip and that is fine.
### 17.5 Host And Routing
The Host header is the routing key. SNI (Server Name Indication) over TLS is a separate concept: the client tells the server which hostname during the TLS handshake. Most clients send the same value for both. Mismatches indicate misconfigured proxies or crafted attack traffic.
### 17.6 Referer And Accept-Language (Inferring Locale)
Sometimes Accept-Language is generic (`en`) but Referer reveals a specific market (e.g., from a Spanish language partner site). Cannot be relied on but useful as a hint.
### 17.7 User-Agent Verification And Rate Limit Whitelist
The rate limit whitelist from framework-http-rate-control-headers.md is UA pattern based for simplicity. For sensitive endpoints, layer DNS verification on top.
---
## 18. ASSET CLASS AND USE CASE RECIPES
Paste ready blocks per scenario.
### 18.1 Standard Bubbles request handling
```nginx
# /etc/nginx/nginx.conf (http context)
http {
# Crawler classification
map $http_user_agent $crawler_type {
default "other";
~*googlebot "search-google";
~*bingbot "search-bing";
~*duckduckbot "search-ddg";
~*applebot "search-apple";
~*claudebot "ai-train-claude";
~*gptbot "ai-train-openai";
~*ccbot "ai-train-cc";
~*oai-searchbot "ai-search-openai";
~*chatgpt-user "ai-user-openai";
~*claude-searchbot "ai-search-claude";
~*claude-user "ai-user-claude";
~*perplexitybot "ai-search-perplexity";
~*perplexity-user "ai-user-perplexity";
~*facebookexternalhit "social-fb";
~*twitterbot "social-twitter";
~*linkedinbot "social-linkedin";
}
# Log with crawler type
log_format request_aware
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'host="$host" '
'"$http_referer" "$http_user_agent" '
'enc="$http_accept_encoding" '
'lang="$http_accept_language" '
'crawler=$crawler_type '
'rt=$request_time';
access_log /var/log/nginx/access.log request_aware;
# Compression stack
gzip on;
gzip_vary on;
gzip_min_length 256;
gzip_comp_level 6;
gzip_types
text/plain text/css text/xml application/json application/javascript
application/xml application/atom+xml application/rss+xml
image/svg+xml font/ttf font/otf font/woff font/woff2;
brotli on;
brotli_static on;
brotli_comp_level 5;
brotli_min_length 256;
brotli_types
text/plain text/css text/xml application/json application/javascript
application/xml application/atom+xml application/rss+xml
image/svg+xml font/ttf font/otf font/woff font/woff2;
# Default server: reject unmatched Host headers
}
```
### 18.2 The mandatory default_server
```nginx
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
listen 443 quic reuseport default_server;
server_name _;
# Snake oil cert (not user facing)
ssl_certificate /etc/ssl/snakeoil/cert.pem;
ssl_certificate_key /etc/ssl/snakeoil/key.pem;
ssl_protocols TLSv1.2 TLSv1.3;
return 444; # close connection without response
}
```
### 18.3 Modern image negotiation (AVIF then WebP then JPG)
```nginx
map $http_accept $image_format {
default "jpg";
~*image/avif "avif";
~*image/webp "webp";
}
location ~* /images/(.+)\.(jpg|jpeg|png)$ {
set $base $1;
set $ext $2;
try_files
/images/$base.$image_format
/images/$base.$ext
=404;
add_header Vary "Accept" always;
add_header Cache-Control "public, max-age=2592000" always;
}
```
### 18.4 Conditional GET on dynamic responses (FastAPI)
```python
from email.utils import formatdate, parsedate_to_datetime
from datetime import datetime, timezone
import hashlib
from fastapi import FastAPI, Request, Response
app = FastAPI()
async def conditional_get_response(request: Request, content: bytes, last_modified: datetime):
"""Return 304 if client has fresh copy, else 200 with content."""
etag = '"' + hashlib.md5(content).hexdigest()[:16] + '"'
# ETag check takes precedence
inm = request.headers.get("if-none-match", "")
if etag in [tag.strip() for tag in inm.split(",") if tag.strip()]:
return Response(
status_code=304,
headers={
"ETag": etag,
"Last-Modified": formatdate(last_modified.timestamp(), usegmt=True),
"Cache-Control": "public, max-age=300",
}
)
# Date check as fallback
ims = request.headers.get("if-modified-since")
if ims:
try:
ims_date = parsedate_to_datetime(ims)
if last_modified <= ims_date:
return Response(
status_code=304,
headers={
"ETag": etag,
"Last-Modified": formatdate(last_modified.timestamp(), usegmt=True),
}
)
except (TypeError, ValueError):
pass
return Response(
content=content,
media_type="text/html",
headers={
"ETag": etag,
"Last-Modified": formatdate(last_modified.timestamp(), usegmt=True),
"Cache-Control": "public, max-age=300",
}
)
```
### 18.5 Crawler aware logging for AI visibility analytics
```nginx
# Separate access log for crawler traffic only
map $crawler_type $is_crawler {
default 1;
"other" 0;
}
access_log /var/log/nginx/crawlers.log request_aware if=$is_crawler;
```
Then aggregate:
```bash
#!/usr/bin/env bash
# /usr/local/bin/crawler-report.sh
LOG=/var/log/nginx/crawlers.log
echo "=== Crawler hits today ==="
grep "$(date '+%d/%b/%Y')" "$LOG" | \
grep -oE "crawler=[a-z-]+" | sort | uniq -c | sort -rn
echo ""
echo "=== AI crawler unique URLs today ==="
grep "$(date '+%d/%b/%Y')" "$LOG" | grep "crawler=ai-" | \
awk '{print $7}' | sort -u | wc -l
echo ""
echo "=== Top URLs hit by AI crawlers ==="
grep "$(date '+%d/%b/%Y')" "$LOG" | grep "crawler=ai-" | \
awk '{print $7}' | sort | uniq -c | sort -rn | head -20
```
This gives Joseph daily visibility into AI crawler traffic per client site.
### 18.6 The hostnames JSON for Bubbles
For Joseph's reverse DNS verification on the Bubbles server, a config file mapping crawler names to expected hostname suffixes:
```python
# /opt/bubbles/crawler_hostnames.json
{
"googlebot": ["googlebot.com", "google.com", "googleusercontent.com"],
"bingbot": ["search.msn.com", "msn.com"],
"applebot": ["applebot.apple.com", "applebot.cdn-apple.com"],
"amazonbot": ["amazonbot.amazon", "compute.amazonaws.com"],
"yandexbot": ["yandex.com", "yandex.net", "yandex.ru"],
"duckduckbot": ["duckduckgo.com"],
"facebookexternalhit": ["facebook.com", "fbsv.net"],
"twitterbot": ["twitter.com", "twttr.com"],
"linkedinbot": ["linkedin.com"]
}
```
Loaded by the FastAPI sidecar to verify any incoming UA claim.
### 18.7 WebSocket upgrade proxy
```nginx
map $http_upgrade $connection_upgrade {
default upgrade;
"" close;
}
location /ws {
proxy_pass http://127.0.0.1:9090;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
}
```
### 18.8 Language switcher hint (do NOT auto redirect)
```python
@app.get("/")
async def home(request: Request):
accept_lang = request.headers.get("accept-language", "")
# Detect preferred language but do NOT redirect automatically
preferred = "en"
if accept_lang.lower().startswith("es"):
preferred = "es"
elif accept_lang.lower().startswith("fr"):
preferred = "fr"
# Inject hint into the rendered HTML for a switcher banner
html = render_template("home.html", preferred_language=preferred)
return HTMLResponse(html, headers={"Vary": "Accept-Language"})
```
Then in the HTML template, if `preferred_language != current_language`, show a switcher banner: "Looks like you might prefer Spanish. Switch to es.example.com?"
### 18.9 Pre compressed assets at build time
```bash
# Build time script (run after each site update)
cd /var/www/sites/example.com
# Find all compressible files
find . -type f \( -name "*.html" -o -name "*.css" -o -name "*.js" -o -name "*.svg" -o -name "*.json" \) | while read -r file; do
# gzip (level 9, maximum)
gzip -k -9 "$file"
# brotli (level 11, maximum)
if command -v brotli >/dev/null; then
brotli -k -q 11 "$file"
fi
# zstd (level 19, maximum)
if command -v zstd >/dev/null; then
zstd -k -19 -q "$file"
fi
done
echo "Pre compression complete. nginx will serve .gz, .br, .zst variants automatically."
```
With `gzip_static on`, `brotli_static on`, `zstd_static on` in nginx, the pre compressed variants are used directly without runtime compression.
### 18.10 Crawler aware rate limiting (combining with framework-http-rate-control-headers.md)
```nginx
# Use the existing crawler_type map for rate limit decisions
map $crawler_type $rate_limit_skip {
default 0; # not a crawler, apply rate limit
"search-google" 1; # search crawlers: skip
"search-bing" 1;
"search-ddg" 1;
"search-apple" 1;
"ai-train-claude" 1;
"ai-train-openai" 1;
"ai-train-cc" 1;
"ai-search-openai" 1;
"ai-user-openai" 1;
"ai-search-claude" 1;
"ai-user-claude" 1;
"ai-search-perplexity" 1;
"ai-user-perplexity" 1;
"social-fb" 1;
"social-twitter" 1;
"social-linkedin" 1;
# "other" stays at 0 (rate limited)
}
map $rate_limit_skip $rate_limit_key {
1 "";
0 $binary_remote_addr;
}
limit_req_zone $rate_limit_key zone=site:10m rate=20r/s;
limit_req_status 429;
```
Whitelist via UA pattern; sensitive endpoints layer DNS verification on top.
---
## 19. BUBBLES NGINX REFERENCE BLOCK (PASTE READY)
The complete request side stanza, layered with the previous seven frameworks.
```nginx
# /etc/nginx/nginx.conf (http context)
http {
# ============= CRAWLER CLASSIFICATION =============
map $http_user_agent $crawler_type {
default "other";
~*googlebot "search-google";
~*bingbot "search-bing";
~*duckduckbot "search-ddg";
~*applebot "search-apple";
~*amazonbot "search-amazon";
~*claudebot "ai-train-claude";
~*gptbot "ai-train-openai";
~*ccbot "ai-train-cc";
~*meta-externalagent "ai-train-meta";
~*oai-searchbot "ai-search-openai";
~*chatgpt-user "ai-user-openai";
~*claude-searchbot "ai-search-claude";
~*claude-user "ai-user-claude";
~*perplexitybot "ai-search-perplexity";
~*perplexity-user "ai-user-perplexity";
~*youbot "ai-search-you";
~*facebookexternalhit "social-fb";
~*twitterbot "social-twitter";
~*linkedinbot "social-linkedin";
~*slackbot "social-slack";
~*discordbot "social-discord";
}
map $crawler_type $rate_limit_skip {
default 0;
"search-google" 1;
"search-bing" 1;
"search-ddg" 1;
"search-apple" 1;
"search-amazon" 1;
"ai-train-claude" 1;
"ai-train-openai" 1;
"ai-train-cc" 1;
"ai-train-meta" 1;
"ai-search-openai" 1;
"ai-user-openai" 1;
"ai-search-claude" 1;
"ai-user-claude" 1;
"ai-search-perplexity" 1;
"ai-user-perplexity" 1;
"ai-search-you" 1;
"social-fb" 1;
"social-twitter" 1;
"social-linkedin" 1;
"social-slack" 1;
"social-discord" 1;
}
# ============= CONTENT NEGOTIATION =============
# Image format negotiation
map $http_accept $image_format {
default "jpg";
~*image/avif "avif";
~*image/webp "webp";
}
# ============= UPGRADE NEGOTIATION =============
map $http_upgrade $connection_upgrade {
default upgrade;
"" close;
}
# ============= COMPRESSION =============
gzip on;
gzip_vary on;
gzip_min_length 256;
gzip_comp_level 6;
gzip_proxied any;
gzip_types
text/plain text/css text/xml application/json application/javascript
application/xml application/atom+xml application/rss+xml
image/svg+xml font/ttf font/otf font/woff font/woff2;
brotli on;
brotli_static on;
brotli_comp_level 5;
brotli_min_length 256;
brotli_types
text/plain text/css text/xml application/json application/javascript
application/xml application/atom+xml application/rss+xml
image/svg+xml font/ttf font/otf font/woff font/woff2;
# zstd if module available
# zstd on;
# zstd_static on;
# zstd_comp_level 6;
# zstd_min_length 256;
# zstd_types text/plain text/css application/json application/javascript;
# ============= LOGGING =============
log_format request_aware
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'host="$host" '
'"$http_referer" "$http_user_agent" '
'enc="$http_accept_encoding" '
'lang="$http_accept_language" '
'crawler=$crawler_type '
'rt=$request_time '
'$server_protocol "$http3"';
access_log /var/log/nginx/access.log request_aware;
# Separate log for crawlers
map $crawler_type $is_crawler {
default 1;
"other" 0;
}
access_log /var/log/nginx/crawlers.log request_aware if=$is_crawler;
}
```
```nginx
# ============= DEFAULT SERVER (mandatory) =============
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
listen 443 quic reuseport default_server;
server_name _;
ssl_certificate /etc/ssl/snakeoil/cert.pem;
ssl_certificate_key /etc/ssl/snakeoil/key.pem;
ssl_protocols TLSv1.2 TLSv1.3;
return 444;
}
# ============= EXAMPLE.COM (canonical client site) =============
server {
listen 443 ssl;
listen 443 quic;
http2 on;
http3 on;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_early_data on;
add_header Alt-Svc 'h3=":443"; ma=86400' always;
# Security headers
include snippets/common-security-headers.conf;
root /var/www/sites/example.com;
index index.html;
# ============= IMAGE NEGOTIATION =============
location ~* /images/(.+)\.(jpg|jpeg|png)$ {
set $base $1;
set $ext $2;
try_files /images/$base.$image_format /images/$base.$ext =404;
add_header Vary "Accept" always;
add_header Cache-Control "public, max-age=2592000" always;
}
# ============= STATIC ASSETS =============
location ~* \.(css|js|woff2)$ {
# nginx auto generates ETag, handles If-None-Match and If-Modified-Since
add_header Cache-Control "public, max-age=31536000, immutable" always;
}
# ============= API ENDPOINTS =============
location /api/ {
# Rate limit applied via framework-http-rate-control-headers.md
proxy_pass http://127.0.0.1:9090;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# ============= WEBSOCKET =============
location /ws {
proxy_pass http://127.0.0.1:9090;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
}
# ============= ROOT =============
location / {
try_files $uri $uri/ $uri.html =404;
}
}
```
After deploying:
```bash
nginx -t && systemctl reload nginx
```
Verify request side handling:
```bash
# 1. Default server rejects unknown Host
curl -sI -H "Host: random.example.com" https://169.155.162.118/ -k 2>&1 | head -3
# Expected: connection reset (444)
# 2. Crawler classification works
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://example.com/ \
&& sudo tail -1 /var/log/nginx/crawlers.log
# 3. Image format negotiation
curl -sI -H "Accept: image/avif" https://example.com/images/hero.jpg
# Expected: served as AVIF if hero.avif exists
# 4. Conditional GET on static
ETAG=$(curl -sI https://example.com/style.css | grep -i etag | awk '{print $2}' | tr -d '\r')
curl -sI -H "If-None-Match: $ETAG" https://example.com/style.css | head -1
# Expected: HTTP/2 304
# 5. Compression negotiation
curl -sI -H "Accept-Encoding: gzip, br, zstd" https://example.com/ | grep -i content-encoding
# Expected: content-encoding: br (or zstd if enabled)
```
---
## 20. AUDIT CHECKLIST
Run through these 50 items for any production request handling configuration.
### User-Agent and crawler classification
1. [ ] UA pattern map in nginx covers all major search crawlers (Googlebot, Bingbot, DuckDuckBot, Applebot, Amazonbot).
2. [ ] UA pattern map covers AI training crawlers (ClaudeBot, GPTBot, CCBot, meta-externalagent).
3. [ ] UA pattern map covers AI search/retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, YouBot).
4. [ ] UA pattern map covers social crawlers (facebookexternalhit, Twitterbot, LinkedInBot).
5. [ ] Deprecated `Claude-Web` pattern removed (replaced by `ClaudeBot`).
6. [ ] UA value logged in access log for analytics.
7. [ ] Crawler verification (reverse DNS) implemented for sensitive endpoints (admin, paywall, premium content).
### Host and routing
8. [ ] `default_server` block exists for every listen address (80, 443 SSL, 443 QUIC).
9. [ ] Default server returns 444 (or 421 Misdirected) for unmatched Host.
10. [ ] FastAPI sidecar receives correct Host via `proxy_set_header Host $host`.
11. [ ] No application code uses raw `request.headers["host"]` for URL construction without validation.
12. [ ] All 27 custom domains have their own server block with proper server_name.
13. [ ] Wildcard server (*.thatwebhostingguy.com) routes to subdomain specific directories.
### Accept and content negotiation
14. [ ] Image format negotiation map (avif/webp/jpg) configured.
15. [ ] Vary: Accept set on responses that vary by Accept.
16. [ ] Content-Type negotiation handles JSON vs HTML correctly for API endpoints.
17. [ ] Default content type is text/html with charset=utf-8.
### Accept-Language
18. [ ] Accept-Language logged for analytics.
19. [ ] No auto redirect based on Accept-Language (use hreflang plus UI switcher).
20. [ ] If multilingual: Vary: Accept-Language set on language variant responses.
### Accept-Encoding and compression
21. [ ] gzip enabled with appropriate gzip_types.
22. [ ] brotli module loaded and brotli enabled (if available).
23. [ ] zstd module loaded if maximum compression desired (optional).
24. [ ] gzip_vary on; (ensures Vary: Accept-Encoding).
25. [ ] gzip_min_length set (256 or similar) to avoid compressing tiny responses.
26. [ ] Static assets pre compressed at build time (.gz, .br, .zst variants exist).
27. [ ] gzip_static on; brotli_static on; for pre compressed serving.
28. [ ] Compression negotiated correctly with curl tests against modern UAs.
### Accept-Charset
29. [ ] All responses use UTF-8 (Content-Type includes charset=utf-8).
30. [ ] No legacy charset negotiation code (Accept-Charset should be ignored).
### From
31. [ ] From header logged where present (low priority, informational only).
32. [ ] No authentication uses From for identity.
### If-Modified-Since and If-None-Match
33. [ ] Static assets return 304 when If-Modified-Since matches mtime.
34. [ ] Static assets return 304 when If-None-Match matches generated ETag.
35. [ ] Dynamic FastAPI endpoints implement conditional GET where appropriate.
36. [ ] ETag takes precedence over If-Modified-Since when both present.
37. [ ] 304 responses have no body.
38. [ ] 304 responses include ETag and/or Last-Modified for next revalidation.
### Referer
39. [ ] Referer logged for analytics.
40. [ ] No security checks rely on Referer alone.
41. [ ] No hotlink protection via Referer (use signed URLs instead).
### Connection
42. [ ] WebSocket location has proper upgrade headers (proxy_set_header Upgrade $http_upgrade; Connection $connection_upgrade).
43. [ ] Long lived WebSocket has appropriate proxy_read_timeout.
### Logging and analytics
44. [ ] log_format includes user agent, referer, host, accept-encoding.
45. [ ] Separate access log for crawlers (or crawler tag in main log).
46. [ ] Crawler analytics script runs periodically (daily aggregation).
47. [ ] AI crawler visibility tracked over time.
### Cross cutting
48. [ ] `nginx -t` passes without warnings.
49. [ ] `nginx -T | grep -E "map|default_server|listen"` shows expected configuration.
50. [ ] Curl tests with major crawler UAs return expected behavior.
A site that passes all 50 has correctly configured request handling.
---
## 21. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: Trusting User-Agent for paywall bypass.**
Symptom: scraper accesses paywalled content by spoofing Googlebot UA.
Why it breaks: UA alone is unverifiable. Reverse DNS verification is the actual identity check.
Fix: implement DNS verification (Section 6) for any UA based privilege grant.
**Pitfall 2: Using deprecated Claude-Web in robots.txt.**
Symptom: site believes it is blocking Anthropic but ClaudeBot still crawls.
Why it breaks: Claude-Web was deprecated; ClaudeBot is the current pattern.
Fix: update robots.txt to use `User-agent: ClaudeBot`.
**Pitfall 3: No default_server block.**
Symptom: unmatched Host headers route to whichever server block parses first.
Why it breaks: nginx falls back to first server block defined.
Fix: add a default_server that returns 444 for all listen addresses.
**Pitfall 4: Host header used in password reset URL without validation.**
Symptom: attacker triggers password reset with spoofed Host, victim's reset link points to attacker.
Why it breaks: Host can be set to anything; using it as authority is unsafe.
Fix: use a configured canonical hostname for outbound URLs.
**Pitfall 5: Ignoring conditional GET headers.**
Symptom: Googlebot crawl budget wasted on unchanged pages.
Why it breaks: server returns 200 with full body even when If-Modified-Since/If-None-Match match.
Fix: implement conditional GET (Section 13.4 and 14.5).
**Pitfall 6: ETag generated from non content data.**
Symptom: every request gets new ETag even when content unchanged.
Why it breaks: ETag derived from timestamp or random nonce, not content hash.
Fix: derive ETag from content hash (MD5 or similar of body).
**Pitfall 7: Compression negotiated for already compressed types.**
Symptom: PNG, JPG, MP4 served gzipped (CPU waste, no size savings).
Why it breaks: gzip_types includes image/video MIME types.
Fix: restrict gzip_types to text and code formats.
**Pitfall 8: Missing Vary: Accept-Encoding when compression in use.**
Symptom: cache serves gzipped response to client that did not request compression.
Why it breaks: missing Vary causes cross variant pollution.
Fix: `gzip_vary on;` (nginx auto adds Vary: Accept-Encoding when compressing).
**Pitfall 9: Auto redirect based on Accept-Language.**
Symptom: users frustrated by being redirected to wrong language version.
Why it breaks: Accept-Language reflects browser settings, not user intent.
Fix: use hreflang plus UI language switcher; do not auto redirect.
**Pitfall 10: Hotlink protection via Referer breaks legitimate uses.**
Symptom: bookmark managers, archive.org, password managers all fail to load images.
Why it breaks: Referer is unreliable; many legitimate clients send empty Referer.
Fix: use signed URLs or token based protection instead.
**Pitfall 11: Compressing very short responses.**
Symptom: response is larger after compression due to overhead.
Why it breaks: gzip_min_length too low (default 20 bytes).
Fix: set gzip_min_length 256 or higher.
**Pitfall 12: WebSocket disconnects after 60 seconds.**
Symptom: WebSocket connection drops despite being active.
Why it breaks: nginx default proxy_read_timeout is 60 seconds.
Fix: set proxy_read_timeout 86400 for WebSocket locations.
**Pitfall 13: 304 response with body.**
Symptom: 304 response includes content (bug in framework).
Why it breaks: 304 per RFC must have no body.
Fix: ensure 304 responses return only headers.
**Pitfall 14: Crawler whitelist applied to sensitive endpoints.**
Symptom: spoofed Googlebot UA accesses admin endpoint.
Why it breaks: UA whitelist is sufficient for public pages but not for sensitive ones.
Fix: layer DNS verification on top for sensitive endpoints (login, admin, payment).
**Pitfall 15: Compressing API responses but not setting Vary.**
Symptom: cross origin API client gets gzipped response but no Vary header, downstream cache poisoning.
Why it breaks: missing gzip_vary on or explicit Vary header.
Fix: ensure Vary: Accept-Encoding is set on compressed responses.
---
## 22. DIAGNOSTIC COMMANDS
Reference of every command useful for request header investigation.
### Inspect what your client sends
```bash
# See exactly what curl sends
curl -v -o /dev/null https://example.com/ 2>&1 | grep "^>"
# As a specific crawler UA
curl -v -A "Mozilla/5.0 (compatible; Googlebot/2.1)" -o /dev/null https://example.com/ 2>&1 | grep "^>"
```
### Simulate crawler requests
```bash
# Cycle through major crawler UAs
for ua in \
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0"
do
echo "=== $ua ==="
curl -sI -A "$ua" https://example.com/ | head -3
echo
done
```
### Test conditional GET
```bash
# Get current ETag and Last-Modified
HEADERS=$(curl -sI https://example.com/style.css)
ETAG=$(echo "$HEADERS" | grep -i "^etag:" | awk '{print $2}' | tr -d '\r')
LM=$(echo "$HEADERS" | grep -i "^last-modified:" | cut -d: -f2- | sed 's/^ //')
# Verify 304 on If-None-Match
echo "Testing If-None-Match with ETag $ETAG"
curl -sI -H "If-None-Match: $ETAG" https://example.com/style.css | head -1
# Expected: HTTP/2 304
# Verify 304 on If-Modified-Since
echo "Testing If-Modified-Since with $LM"
curl -sI -H "If-Modified-Since: $LM" https://example.com/style.css | head -1
# Expected: HTTP/2 304
# Verify 200 on stale validators
curl -sI -H "If-None-Match: \"stale\"" https://example.com/style.css | head -1
# Expected: HTTP/2 200
```
### Test compression negotiation
```bash
# What does the server actually negotiate?
echo "Plain gzip:"
curl -sI -H "Accept-Encoding: gzip" https://example.com/ | grep -i content-encoding
echo "Plus brotli:"
curl -sI -H "Accept-Encoding: gzip, br" https://example.com/ | grep -i content-encoding
echo "Plus zstd:"
curl -sI -H "Accept-Encoding: gzip, br, zstd" https://example.com/ | grep -i content-encoding
# Compare compressed sizes
for enc in gzip br zstd; do
SIZE=$(curl -s -H "Accept-Encoding: $enc" --compressed -w "%{size_download}" -o /dev/null https://example.com/)
echo "$enc: $SIZE bytes"
done
```
### Verify crawler identity by reverse DNS
```bash
verify_crawler() {
local ip=$1
local domain_pattern=$2
local ptr=$(host "$ip" 2>/dev/null | awk '/pointer/ {print $NF}' | sed 's/\.$//')
if [ -z "$ptr" ]; then
echo "FAIL: no reverse DNS for $ip"
return 1
fi
case "$ptr" in
*$domain_pattern)
local forward=$(host "$ptr" 2>/dev/null | awk '/has address/ {print $NF}' | head -1)
if [ "$forward" = "$ip" ]; then
echo "VERIFIED: $ip is $domain_pattern ($ptr)"
return 0
else
echo "FAIL: forward $forward != $ip"
return 1
fi
;;
*)
echo "FAIL: $ptr does not match $domain_pattern"
return 1
;;
esac
}
# Test
verify_crawler 66.249.66.1 ".googlebot.com" # should verify
verify_crawler 8.8.8.8 ".googlebot.com" # should fail
```
### Analyze crawler traffic patterns
```bash
# Daily crawler hits by type
awk '/crawler=/' /var/log/nginx/access.log | \
grep -oE "crawler=[a-z-]+" | sort | uniq -c | sort -rn
# Hourly Googlebot rate
grep "crawler=search-google" /var/log/nginx/access.log | \
awk '{print $4}' | cut -d: -f1-2 | uniq -c
# Top URLs by AI crawler
grep "crawler=ai-" /var/log/nginx/access.log | \
awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# Unique URLs by AI crawler type
for ct in ai-train-claude ai-train-openai ai-search-openai ai-search-claude ai-search-perplexity; do
COUNT=$(grep "crawler=$ct" /var/log/nginx/access.log | awk '{print $7}' | sort -u | wc -l)
echo "$ct: $COUNT unique URLs"
done
```
### Test Host routing
```bash
# Verify default_server rejects unknown Host
curl -sI -H "Host: random.example.invalid" https://169.155.162.118/ -k 2>&1 | head -3
# Expected: connection reset
# Verify Host routing to correct server block
for host in example.com www.example.com api.example.com other.com; do
echo "=== $host ==="
curl -sI -H "Host: $host" https://169.155.162.118/ -k 2>&1 | head -3
echo
done
```
### Server side investigation
```bash
# Show all maps related to request headers
nginx -T 2>/dev/null | grep -A 30 "map \$http_"
# Show all default_server declarations
nginx -T 2>/dev/null | grep "default_server"
# Show compression configuration
nginx -T 2>/dev/null | grep -E "gzip|brotli|zstd"
# Show all log_format definitions
nginx -T 2>/dev/null | grep -A 10 "log_format"
# Apply changes
nginx -t && systemctl reload nginx
```
### Browser DevTools quick reference
In Chrome DevTools Network panel:
1. Click any request.
2. Headers tab: "Request Headers" section shows what the browser sent.
3. Look for: User-Agent, Accept, Accept-Language, Accept-Encoding, Host, Referer, Connection.
Useful for understanding what your real users send vs what curl tests send.
```javascript
// In Console: see what fetch will send
fetch("/test").then(r => console.log(r));
// Then check Network tab > the /test request > Headers tab
```
---
## 23. CROSS-REFERENCES
* [framework-http-caching-headers.md](framework-http-caching-headers.md): Last-Modified and ETag response headers pair with If-Modified-Since and If-None-Match request headers covered here.
* [framework-http-content-headers.md](framework-http-content-headers.md): Content-Type response header is the result of Accept based negotiation. Content-Encoding response header pairs with Accept-Encoding here. Content-Language pairs with Accept-Language.
* [framework-http-seo-headers.md](framework-http-seo-headers.md): X-Robots-Tag and Link headers are responses; the User-Agent header here is how crawlers identify themselves before receiving those responses.
* [framework-http-security-headers.md](framework-http-security-headers.md): Referrer-Policy controls what Referer the browser sends out (response header that affects future requests). Cross origin headers interact with Origin (request) and Host.
* [framework-http-performance-headers.md](framework-http-performance-headers.md): Alt-Svc response header advertises HTTP/3; clients then connect via different transport. Server-Timing depends on the upstream measuring time.
* [framework-http-cors-headers.md](framework-http-cors-headers.md): Origin request header (not in this framework) triggers CORS handling. Access-Control-Request-Method/Headers are preflight request headers.
* [framework-http-rate-control-headers.md](framework-http-rate-control-headers.md): the User-Agent crawler whitelist used here directly drives rate limit bypass decisions.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. AI crawler identification is core to GEO (generative engine optimization).
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook.
* [framework-bot-management.md](framework-bot-management.md): deeper dive on crawler verification, AS number based identification, fail2ban integration.
* [framework-robotstxt-crawlbudget.md](framework-robotstxt-crawlbudget.md): how robots.txt directives interact with the crawler taxonomy here.
* Google crawler documentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
* Google verify Googlebot: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
* OpenAI bot documentation: https://platform.openai.com/docs/bots
* Anthropic crawler documentation: https://docs.claude.com/en/docs/agents-and-tools/web-fetch-tool (and operator portal)
* Bingbot verification: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
* PerplexityBot documentation: https://docs.perplexity.ai/guides/bots
* RFC 9110 (HTTP Semantics): https://www.rfc-editor.org/rfc/rfc9110
* RFC 7234 (HTTP Caching, conditional GET): https://www.rfc-editor.org/rfc/rfc7234
* BCP 47 language tags: https://www.rfc-editor.org/info/bcp47
* MDN HTTP Headers reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Bubbles default request handling
```nginx
http {
# Crawler classification (drives logs, rate limit whitelist, analytics)
map $http_user_agent $crawler_type {
default "other";
~*googlebot "search-google";
~*bingbot "search-bing";
~*claudebot "ai-train-claude";
~*gptbot "ai-train-openai";
~*oai-searchbot "ai-search-openai";
~*claude-searchbot "ai-search-claude";
~*perplexitybot "ai-search-perplexity";
~*claude-user "ai-user-claude";
~*chatgpt-user "ai-user-openai";
~*perplexity-user "ai-user-perplexity";
}
# WebSocket upgrade
map $http_upgrade $connection_upgrade {
default upgrade;
"" close;
}
# Image negotiation
map $http_accept $image_format {
default "jpg";
~*image/avif "avif";
~*image/webp "webp";
}
# Compression
gzip on; gzip_vary on; gzip_min_length 256;
brotli on; brotli_static on;
}
# MANDATORY default server
server {
listen 443 ssl default_server;
server_name _;
ssl_certificate /etc/ssl/snakeoil/cert.pem;
ssl_certificate_key /etc/ssl/snakeoil/key.pem;
return 444;
}
```
### Header purpose table
| Header | Variable | One line purpose |
|---|---|---|
| User-Agent | `$http_user_agent` | Client identity claim (unverified) |
| Accept | `$http_accept` | Content types client supports |
| Accept-Language | `$http_accept_language` | Languages client prefers |
| Accept-Encoding | `$http_accept_encoding` | Compression algorithms supported |
| Accept-Charset | `$http_accept_charset` | Legacy charset (ignore in 2026) |
| From | `$http_from` | Legacy contact email (log only) |
| Host | `$host` or `$http_host` | Which virtual host (routing key) |
| If-Modified-Since | `$http_if_modified_since` | Conditional GET by date |
| If-None-Match | `$http_if_none_match` | Conditional GET by ETag |
| Referer | `$http_referer` | Where request came from |
| Connection | `$http_connection` | Transport disposition (keep-alive, close, upgrade) |
### Crawler quick reference (2026)
| Crawler | UA pattern | Operator |
|---|---|---|
| Googlebot | `googlebot` | Google |
| Bingbot | `bingbot` | Microsoft |
| ClaudeBot | `claudebot` (NOT claude-web) | Anthropic |
| GPTBot | `gptbot` | OpenAI training |
| OAI-SearchBot | `oai-searchbot` | OpenAI search |
| Claude-SearchBot | `claude-searchbot` | Anthropic search |
| Claude-User | `claude-user` | Anthropic user fetches |
| PerplexityBot | `perplexitybot` | Perplexity index |
### Verification quick reference
| Crawler | Method |
|---|---|
| Googlebot | Reverse DNS + IP range JSON |
| Bingbot | Reverse DNS + IP range JSON |
| OpenAI (GPTBot et al) | Published IP range JSON |
| Perplexity | Published IP range JSON |
| Anthropic (ClaudeBot et al) | Reverse DNS only (no IP range published) |
| Applebot, Amazonbot | Reverse DNS |
### Five rules to memorize
1. Never trust User-Agent alone. Verify by reverse DNS for sensitive endpoints.
2. Always have a default_server returning 444 for unmatched Host.
3. Always implement conditional GET (304 on If-Modified-Since or If-None-Match match).
4. Always negotiate Accept-Encoding correctly (gzip plus brotli at minimum).
5. Use hreflang plus UI switcher for languages; never auto redirect on Accept-Language.
### Five commands every operator should know
```bash
# 1. Verify a request IP belongs to claimed crawler
host 66.249.66.1 && host crawl-66-249-66-1.googlebot.com
# 2. Simulate a crawler
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://example.com/
# 3. Test conditional GET
ETAG=$(curl -sI https://example.com/file.css | grep -i etag | awk '{print $2}' | tr -d '\r')
curl -sI -H "If-None-Match: $ETAG" https://example.com/file.css | head -1
# 4. Test compression negotiation
curl -sI -H "Accept-Encoding: gzip, br, zstd" https://example.com/ | grep -i content-encoding
# 5. Apply nginx changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. Default server rejects unknown Host
curl -sI -H "Host: nonexistent.example" https://169.155.162.118/ -k 2>&1 | head -3
# Expected: connection reset
# 2. Major crawlers classified correctly
for ua in Googlebot bingbot ClaudeBot GPTBot OAI-SearchBot PerplexityBot; do
curl -sI -A "Mozilla/5.0 (compatible; $ua/1.0)" https://example.com/ > /dev/null
done
sudo tail -6 /var/log/nginx/crawlers.log | grep -oE "crawler=[a-z-]+"
# Expected: six different crawler= tags
# 3. 304 returned for unchanged content
ETAG=$(curl -sI https://example.com/style.css | grep -i etag | awk '{print $2}' | tr -d '\r')
STATUS=$(curl -so /dev/null -w "%{http_code}" -H "If-None-Match: $ETAG" https://example.com/style.css)
[ "$STATUS" = "304" ] && echo "Conditional GET works" || echo "FAIL: got $STATUS"
```
If all three pass AND the daily crawler analytics show expected AI crawler visibility, the request handling stack is correctly wired.
---
End of framework-http-request-headers.md.