A continuation of Janady13/crawler-signal-reference / framework-http-caching-headers.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓1,705 lines · 76.7 KB
Why it breaks: nginx strips ETag when the body is modified post generation; the on disk content is no longer authoritative.
Fix: pre render SSI at build time (use a static site generator), or set Last-Modified manually from the upstream, or accept the lost validator and rely on a short `max-age`.
**Pitfall 11: Service worker cached forever.**
Symptom: site is stuck on an old version even after deploys; users must manually unregister the service worker.
Why it breaks: `sw.js` was served with a long `max-age` and is being served from browser cache instead of fetched fresh.
Fix: see Section 12.5. Service worker MUST be `no-store, no-cache, must-revalidate`.
**Pitfall 12: Authenticated response cached as public.**
Symptom: user A sees user B's account dashboard or order history.
Why it breaks: response containing user specific data was served `Cache-Control: public, max-age=300` and a shared cache stored it.
Fix: every authenticated endpoint MUST be `Cache-Control: private, no-store`. Audit immediately if this is suspected; the privacy implications are serious.
**Pitfall 13: Mixing `max-age` and `s-maxage` inconsistently.**
Symptom: CDN cache and browser cache disagree on freshness, causing intermittent stale content.
Why it breaks: shared cache uses `s-maxage`, private cache uses `max-age`, and the two values are not coherent.
Fix: not applicable on Bubbles (no CDN). If you ever front Bubbles with a cache, set both values explicitly and verify behavior.
**Pitfall 14: Stale validators after restoring from backup.**
Symptom: after a backup restore, conditional requests start returning 200 instead of 304 for unchanged content.
Why it breaks: the restore set file mtime to "now" instead of preserving original mtime.
Fix: restore with `tar --preserve` or `rsync --times`. Verify with `stat`.
**Pitfall 15: Ignoring the `always` keyword.**
Symptom: caching headers appear on 200 responses but not on 404 or 500.
Why it breaks: `add_header` without `always` only applies to 200, 201, 204, 206, 301, 302, 303, 304, 307, 308.
Fix: append `always` to every `add_header` directive.
---
## 16. DIAGNOSTIC COMMANDS
Reference of every command useful for caching header investigation. Each one has a comment explaining what it tells you.
### Inspect headers
```bash
# Just the headers, no body
curl -sI https://example.com/page.html
# Just the caching family
curl -sI https://example.com/page.html | grep -iE "cache|etag|last-modified|expires|vary|age"
# Headers AND first lines of body
curl -si https://example.com/page.html | head -30
# Headers as Googlebot
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/
# Headers from a specific IP (bypass DNS)
curl -sI --resolve example.com:443:169.155.162.118 https://example.com/
# Headers including the negotiated protocol
curl -sI --http2 https://example.com/page.html
curl -sI --http3 https://example.com/page.html
```
### Test conditional requests
```bash
# Capture ETag and replay
ETAG=$(curl -sI https://example.com/style.css | awk -F': ' '/^[Ee]tag/{print $2}' | tr -d '\r')
echo "ETag is: $ETAG"
curl -sI -H "If-None-Match: $ETAG" https://example.com/style.css | head -1
# Capture Last-Modified and replay
LM=$(curl -sI https://example.com/style.css | awk -F': ' '/^[Ll]ast-[Mm]odified/{print $2}' | tr -d '\r')
echo "Last-Modified is: $LM"
curl -sI -H "If-Modified-Since: $LM" https://example.com/style.css | head -1
# Verify body is empty on 304
curl -s -o /tmp/304-body -H "If-None-Match: $ETAG" https://example.com/style.css
wc -c /tmp/304-body
# Expected: 0 /tmp/304-body
```
### Test compression and Vary
```bash
# Test with and without gzip
curl -s -H "Accept-Encoding: identity" https://example.com/page.html | wc -c
curl -s -H "Accept-Encoding: gzip" https://example.com/page.html | wc -c
# Confirm Content-Encoding is gzip
curl -sI -H "Accept-Encoding: gzip" https://example.com/page.html | grep -i content-encoding
# Confirm Vary header is present
curl -sI -H "Accept-Encoding: gzip" https://example.com/page.html | grep -i vary
# Test brotli
curl -sI -H "Accept-Encoding: br" https://example.com/page.html | grep -iE "content-encoding|vary"
```
### Cache behavior across time
```bash
# Watch Age increase from an intermediate cache
for i in $(seq 1 10); do
echo "=== Request $i at $(date +%T) ==="
curl -sI https://example.com/page.html | grep -iE "age|cache-control"
sleep 30
done
```
### Bubbles server side investigation
```bash
# Check nginx config syntax
nginx -t
# Show active config for a server name
nginx -T 2>/dev/null | grep -A 50 "server_name example.com"
# Find every place a header is set
grep -rn "Cache-Control" /etc/nginx/
# Find every add_header in a specific site config
grep -n "add_header" /etc/nginx/sites-available/example.com
# Watch access logs for cache related status codes
tail -f /var/log/nginx/access.log | grep -E " 304 | 200 "
# Count 304 vs 200 ratio over last 1000 requests
tail -1000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# Reload after changes
nginx -t && systemctl reload nginx
```
### Browser dev tools quick reference
When using Chrome DevTools Network panel:
* Set "Disable cache" off to see real caching behavior.
* Look at the "Size" column: "(memory cache)" or "(disk cache)" means the browser served from cache without contacting the server.
* "304" in the Status column means conditional revalidation succeeded.
* Click a request, then "Headers" tab, then look for "Response Headers" to see all the caching family.
* Hard reload (Ctrl Shift R) sends `Cache-Control: no-cache` from the client, bypassing fresh cached responses but still allowing conditional revalidation.
---
## 17. CROSS-REFERENCES
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Section 24 (Bubbles Nginx config) inherits the caching stanzas defined here.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook. References this framework for any caching configuration decision.
* [framework-corewebvitals.md](framework-corewebvitals.md): LCP, INP, CLS targets. Caching headers directly affect LCP for repeat visitors.
* [framework-crawlbudget.md](framework-crawlbudget.md): how Google allocates crawl resources. Proper 304 responses are the primary lever for increasing crawl efficiency.
* [framework-robotstxt-crawlbudget.md](framework-robotstxt-crawlbudget.md): robots.txt as a caching surface (Section 12.6 here).
* [framework-security-headers.md](framework-security-headers.md): HSTS, CSP, X-Frame-Options. Configured alongside caching headers via the shared `common-security-headers.conf` snippet.
* [framework-compression.md](framework-compression.md): gzip and brotli configuration. Tightly coupled to ETag (gzip weakens ETags) and Vary (Vary: Accept-Encoding is required when compression is active).
* Google official documentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
* Google December 2024 blog post on caching: https://developers.google.com/search/blog/2024/12/crawling-december-caching
* RFC 9111 (HTTP Caching): https://www.rfc-editor.org/rfc/rfc9111
* MDN Cache-Control reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
| Asset type | Cache-Control | ETag | Last-Modified | Expires | Vary |
|---|---|---|---|---|---|
| HTML | `public, max-age=0, must-revalidate` | auto | auto | auto | `Accept-Encoding` |
| CSS/JS fingerprinted | `public, max-age=31536000, immutable` | auto | auto | auto | `Accept-Encoding` |
| CSS/JS unfingerprinted | `public, max-age=3600, must-revalidate` | auto | auto | auto | `Accept-Encoding` |
| Fonts | `public, max-age=31536000, immutable` | auto | auto | auto | none |
| Images | `public, max-age=2592000` | auto | auto | auto | none |
| Service worker | `no-store, no-cache, must-revalidate` | off | off | off | none |
| robots.txt | `public, max-age=3600` | auto | auto | auto | none |
| sitemap.xml | `public, max-age=600` | auto | auto | auto | none |
| Public API JSON | `public, max-age=300, stale-while-revalidate=60` | manual at upstream | manual at upstream | none | `Accept-Encoding` |
| Authenticated API | `private, no-store` | none | none | none | none |
| PDF/zip | `public, max-age=86400` | auto | auto | auto | none |
Three commands every operator should know by heart:
```bash
# Validate the config is syntactically correct
nginx -t
# Apply the config without dropping connections
systemctl reload nginx
# Combined: only reload if syntax check passes
nginx -t && systemctl reload nginx
```
And two curl invocations that prove caching works end to end:
```bash
# First request: should be 200 with body
curl -sI https://example.com/style.css | head -1
# Second request with ETag: should be 304 with no body
ETAG=$(curl -sI https://example.com/style.css | awk -F': ' '/^[Ee]tag/{print $2}' | tr -d '\r')
curl -sI -H "If-None-Match: $ETAG" https://example.com/style.css | head -1
```
If the first returns 200 and the second returns 304, caching is correctly wired. Everything else in this document is detail behind that core test.
---
End of framework-http-caching-headers.md.