Janady13/crawler-signal-reference / framework-http-content-headers.md · Lines 1501–2325

1501## 11. ASSET CLASS RECIPES 1502 1503Each block is a paste ready location stanza for the five content headers. Copy the ones relevant to your build. 1504 1505### 11.1 HTML (UTF-8 always, no Content-Disposition) 1506 1507```nginx 1508location ~* \.html$ { 1509 # Content-Type: text/html; charset=utf-8 (auto from mime.types + charset utf-8) 1510 # Content-Language set at server level: en-US 1511 # Content-Encoding: gzip or br (auto from gzip/brotli on) 1512 # Content-Length: auto for static, chunked when on the fly compression active 1513 # Content-Disposition: none (default inline) 1514 expires 0; 1515 add_header Cache-Control "public, max-age=0, must-revalidate" always; 1516} 1517``` 1518 1519### 11.2 CSS 1520 1521```nginx 1522location ~* \.css$ { 1523 # Content-Type: text/css; charset=utf-8 (auto) 1524 # Content-Encoding: gzip/br/zstd (auto, large savings) 1525 # Content-Length: auto for precompressed, chunked otherwise 1526 expires 1y; 1527 add_header Cache-Control "public, max-age=31536000, immutable" always; 1528} 1529``` 1530 1531### 11.3 JavaScript 1532 1533```nginx 1534location ~* \.(js|mjs)$ { 1535 # Content-Type: application/javascript; charset=utf-8 (auto) 1536 # Content-Encoding: gzip/br/zstd (auto) 1537 expires 1y; 1538 add_header Cache-Control "public, max-age=31536000, immutable" always; 1539} 1540``` 1541 1542### 11.4 JSON 1543 1544```nginx 1545location ~* \.json$ { 1546 # Content-Type: application/json; charset=utf-8 (auto) 1547 # Content-Encoding: gzip/br/zstd (auto) 1548 expires 1h; 1549 add_header Cache-Control "public, max-age=3600, must-revalidate" always; 1550} 1551 1552# For schema and structured data 1553location ~* \.jsonld$ { 1554 types { application/ld+json jsonld; } 1555 expires 1h; 1556 add_header Cache-Control "public, max-age=3600" always; 1557} 1558``` 1559 1560### 11.5 XML (sitemaps, feeds) 1561 1562```nginx 1563location ~* \.xml$ { 1564 # Content-Type: application/xml; charset=utf-8 (auto) 1565 # Content-Encoding: gzip/br/zstd (auto) 1566 expires 10m; 1567 add_header Cache-Control "public, max-age=600" always; 1568} 1569``` 1570 1571### 11.6 SVG 1572 1573```nginx 1574location ~* \.svg$ { 1575 # Content-Type: image/svg+xml; charset=utf-8 (auto) 1576 # Content-Encoding: gzip/br/zstd (auto; SVG is text, compresses ~70 percent) 1577 expires 30d; 1578 add_header Cache-Control "public, max-age=2592000" always; 1579} 1580``` 1581 1582### 11.7 Raster images (PNG, JPEG, WebP, AVIF) 1583 1584```nginx 1585location ~* \.(png|jpe?g|gif|webp|avif|ico)$ { 1586 # Content-Type: image/png, image/jpeg, etc (auto) 1587 # Content-Encoding: none (already compressed internally) 1588 # Make sure these are NOT in gzip_types or brotli_types 1589 expires 30d; 1590 add_header Cache-Control "public, max-age=2592000" always; 1591} 1592``` 1593 1594### 11.8 Fonts (woff2, woff, ttf, otf) 1595 1596```nginx 1597location ~* \.(woff2|woff|ttf|otf|eot)$ { 1598 # Content-Type: font/woff2, font/ttf, etc (auto) 1599 # Content-Encoding: NONE for woff2 (already brotli compressed) 1600 # gzip/br OK for ttf, otf (uncompressed formats) 1601 # CORS: required if loaded from a different origin 1602 add_header Access-Control-Allow-Origin "*" always; 1603 expires 1y; 1604 add_header Cache-Control "public, max-age=31536000, immutable" always; 1605} 1606``` 1607 1608### 11.9 PDF (with explicit attachment) 1609 1610```nginx 1611location ~* \.pdf$ { 1612 # Content-Type: application/pdf (auto) 1613 # Content-Encoding: NONE (PDF already compressed internally) 1614 expires 1d; 1615 add_header Cache-Control "public, max-age=86400" always; 1616 add_header Content-Disposition "attachment" always; 1617 add_header X-Content-Type-Options "nosniff" always; 1618} 1619 1620# Or for inline preview 1621location /preview/ { 1622 add_header Content-Disposition "inline" always; 1623} 1624``` 1625 1626### 11.10 Downloadable archives (zip, tar, gz) 1627 1628```nginx 1629location ~* \.(zip|tar|gz|bz2|7z|rar)$ { 1630 # Content-Type: application/zip, application/x-tar, etc (auto) 1631 # Content-Encoding: NONE (these ARE the archive; do not double compress) 1632 expires 1d; 1633 add_header Cache-Control "public, max-age=86400" always; 1634 add_header Content-Disposition "attachment" always; 1635} 1636``` 1637 1638### 11.11 User uploaded content (security hardened) 1639 1640```nginx 1641location /user-uploads/ { 1642 # Force download, force binary type, prevent XSS via uploaded HTML 1643 add_header Content-Type "application/octet-stream" always; 1644 add_header Content-Disposition "attachment" always; 1645 add_header X-Content-Type-Options "nosniff" always; 1646 add_header Content-Security-Policy "default-src 'none'" always; 1647 expires 1d; 1648} 1649``` 1650 1651### 11.12 robots.txt, ai.txt, llms.txt, security.txt 1652 1653```nginx 1654location ~* /(robots\.txt|ai\.txt|llms\.txt|humans\.txt|\.well-known/security\.txt)$ { 1655 # Content-Type: text/plain; charset=utf-8 (auto from .txt) 1656 # Content-Encoding: gzip (auto; saves bandwidth on long robots files) 1657 expires 1h; 1658 add_header Cache-Control "public, max-age=3600" always; 1659} 1660``` 1661 1662### 11.13 WebAssembly modules 1663 1664```nginx 1665location ~* \.wasm$ { 1666 # Content-Type: application/wasm (must add to types {} if missing from mime.types) 1667 types { application/wasm wasm; } 1668 # Content-Encoding: gzip/br/zstd (auto; .wasm benefits from compression) 1669 expires 1y; 1670 add_header Cache-Control "public, max-age=31536000, immutable" always; 1671 add_header Cross-Origin-Embedder-Policy "require-corp" always; 1672 add_header Cross-Origin-Opener-Policy "same-origin" always; 1673} 1674``` 1675 1676### 11.14 PWA manifest 1677 1678```nginx 1679location ~* \.webmanifest$ { 1680 types { application/manifest+json webmanifest; } 1681 # Charset: utf-8 (auto via charset_types) 1682 expires 1d; 1683 add_header Cache-Control "public, max-age=86400" always; 1684} 1685``` 1686 1687### 11.15 Generated CSV exports (from FastAPI sidecar) 1688 1689```nginx 1690location /export/ { 1691 proxy_pass http://127.0.0.1:9090; 1692 proxy_pass_header Content-Type; 1693 proxy_pass_header Content-Disposition; 1694 # FastAPI must emit: 1695 # Content-Type: text/csv; charset=utf-8 1696 # Content-Disposition: attachment; filename="export-2026-05-24.csv" 1697} 1698``` 1699 1700--- 1701 1702## 12. BUBBLES NGINX REFERENCE BLOCK (PASTE READY) 1703 1704The complete content header stanza, layered with the caching stanza from framework-http-caching-headers.md. 1705 1706```nginx 1707# /etc/nginx/nginx.conf (or http context) 1708http { 1709 # ===== CONTENT TYPE ===== 1710 include /etc/nginx/mime.types; 1711 default_type application/octet-stream; 1712 1713 types { 1714 image/avif avif; 1715 font/woff2 woff2; 1716 application/wasm wasm; 1717 application/manifest+json webmanifest manifest; 1718 application/ld+json jsonld; 1719 model/gltf-binary glb; 1720 model/gltf+json gltf; 1721 text/markdown md; 1722 } 1723 1724 # ===== CHARSET ===== 1725 charset utf-8; 1726 charset_types 1727 text/html 1728 text/css 1729 text/plain 1730 text/xml 1731 text/csv 1732 application/javascript 1733 application/json 1734 application/xml 1735 application/atom+xml 1736 application/rss+xml 1737 application/ld+json 1738 image/svg+xml; 1739 1740 # ===== COMPRESSION: GZIP ===== 1741 gzip on; 1742 gzip_vary on; 1743 gzip_min_length 1100; 1744 gzip_comp_level 6; 1745 gzip_proxied any; 1746 gzip_static on; 1747 gzip_types 1748 text/plain 1749 text/css 1750 text/xml 1751 text/csv 1752 application/javascript 1753 application/json 1754 application/xml 1755 application/atom+xml 1756 application/rss+xml 1757 application/ld+json 1758 application/manifest+json 1759 application/wasm 1760 image/svg+xml 1761 font/ttf 1762 font/otf; 1763 gzip_disable "msie6"; 1764 1765 # ===== COMPRESSION: BROTLI (if module installed) ===== 1766 brotli on; 1767 brotli_static on; 1768 brotli_comp_level 6; 1769 brotli_min_length 1100; 1770 brotli_types 1771 text/plain 1772 text/css 1773 text/xml 1774 text/csv 1775 application/javascript 1776 application/json 1777 application/xml 1778 application/atom+xml 1779 application/rss+xml 1780 application/ld+json 1781 application/manifest+json 1782 application/wasm 1783 image/svg+xml 1784 font/ttf 1785 font/otf; 1786 1787 # ===== COMPRESSION: ZSTD (if module installed) ===== 1788 zstd on; 1789 zstd_static on; 1790 zstd_comp_level 6; 1791 zstd_min_length 1100; 1792 zstd_types 1793 text/plain 1794 text/css 1795 text/xml 1796 application/javascript 1797 application/json 1798 application/xml 1799 application/ld+json 1800 image/svg+xml; 1801} 1802 1803# Per server block 1804server { 1805 listen 443 ssl; 1806 http2 on; 1807 server_name example.com www.example.com; 1808 root /var/www/sites/example.com; 1809 index index.html; 1810 1811 # ===== CONTENT-LANGUAGE ===== 1812 # Default language for the site 1813 # NOTE: snippets pattern is required so location blocks do not drop this 1814 set $site_language "en-US"; 1815 1816 # ===== SECURITY FAMILY (no MIME sniffing, etc) ===== 1817 include snippets/common-security-headers.conf; 1818 add_header Content-Language $site_language always; 1819 1820 # ===== ASSET CLASS BLOCKS ===== 1821 # (See Section 11 for all asset class recipes; combine with caching recipes from framework-http-caching-headers.md) 1822 1823 location ~* \.html$ { 1824 include snippets/common-security-headers.conf; 1825 add_header Content-Language $site_language always; 1826 add_header Cache-Control "public, max-age=0, must-revalidate" always; 1827 } 1828 1829 location ~* \.(css|js|mjs)$ { 1830 include snippets/common-security-headers.conf; 1831 add_header Cache-Control "public, max-age=31536000, immutable" always; 1832 } 1833 1834 location ~* \.(woff2|woff|ttf|otf|eot)$ { 1835 include snippets/common-security-headers.conf; 1836 add_header Access-Control-Allow-Origin "*" always; 1837 add_header Cache-Control "public, max-age=31536000, immutable" always; 1838 } 1839 1840 location ~* \.(png|jpe?g|gif|webp|avif|svg|ico)$ { 1841 include snippets/common-security-headers.conf; 1842 add_header Cache-Control "public, max-age=2592000" always; 1843 } 1844 1845 location ~* \.pdf$ { 1846 include snippets/common-security-headers.conf; 1847 add_header Content-Disposition "attachment" always; 1848 add_header Cache-Control "public, max-age=86400" always; 1849 } 1850 1851 location /user-uploads/ { 1852 include snippets/common-security-headers.conf; 1853 add_header Content-Type "application/octet-stream" always; 1854 add_header Content-Disposition "attachment" always; 1855 add_header Content-Security-Policy "default-src 'none'" always; 1856 add_header Cache-Control "public, max-age=86400" always; 1857 } 1858 1859 location /api/ { 1860 include snippets/common-security-headers.conf; 1861 proxy_pass http://127.0.0.1:9090; 1862 proxy_set_header Host $host; 1863 proxy_set_header X-Real-IP $remote_addr; 1864 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 1865 proxy_set_header X-Forwarded-Proto $scheme; 1866 proxy_pass_header Content-Type; 1867 proxy_pass_header Content-Disposition; 1868 proxy_pass_header Content-Language; 1869 add_header Vary "Accept-Encoding" always; 1870 } 1871 1872 location / { 1873 include snippets/common-security-headers.conf; 1874 add_header Content-Language $site_language always; 1875 add_header Cache-Control "public, max-age=0, must-revalidate" always; 1876 try_files $uri $uri/ $uri.html =404; 1877 } 1878} 1879``` 1880 1881The shared security header snippet at `/etc/nginx/snippets/common-security-headers.conf`: 1882 1883```nginx 1884add_header X-Frame-Options "SAMEORIGIN" always; 1885add_header X-Content-Type-Options "nosniff" always; 1886add_header Referrer-Policy "strict-origin-when-cross-origin" always; 1887add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always; 1888add_header Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=()" always; 1889``` 1890 1891After deploying: 1892 1893```bash 1894nginx -t && systemctl reload nginx 1895``` 1896 1897--- 1898 1899## 13. AUDIT CHECKLIST 1900 1901Run through these 50 items for any production site. Each item is a single curl command or visual inspection. Pass means the listed expected value appears. 1902 1903### Content-Type 1904 19051. [ ] HTML response has `Content-Type: text/html; charset=utf-8`. 19062. [ ] CSS response has `Content-Type: text/css; charset=utf-8`. 19073. [ ] JavaScript response has `Content-Type: application/javascript; charset=utf-8` (or `text/javascript`). 19084. [ ] JSON response has `Content-Type: application/json` (charset optional). 19095. [ ] XML sitemap has `Content-Type: application/xml`. 19106. [ ] SVG response has `Content-Type: image/svg+xml`. 19117. [ ] AVIF image has `Content-Type: image/avif`. 19128. [ ] WebP image has `Content-Type: image/webp`. 19139. [ ] WOFF2 font has `Content-Type: font/woff2` (NOT `application/octet-stream`). 191410. [ ] WASM module has `Content-Type: application/wasm`. 191511. [ ] PDF has `Content-Type: application/pdf`. 191612. [ ] `X-Content-Type-Options: nosniff` is present on every response. 191713. [ ] No response has `Content-Type: application/octet-stream` for known file extensions. 191814. [ ] No response has `Content-Type: text/html` for non HTML files. 191915. [ ] `default_type` in nginx config is `application/octet-stream` (not `text/html`). 1920 1921### Content-Language 1922 192316. [ ] HTML response has `Content-Language: en-US` (or the correct site language). 192417. [ ] Language code uses hyphen, not underscore (`en-US` not `en_US`). 192518. [ ] Region code is uppercase two letter ISO 3166-1 (`US` not `usa`). 192619. [ ] HTML `<html>` element has matching `lang` attribute. 192720. [ ] HTML `<head>` has matching `<meta http-equiv="content-language" content="...">`. 192821. [ ] Multi region sites have hreflang annotations in `<head>` for every language version. 192922. [ ] hreflang includes `x-default` for the global default. 193023. [ ] Localized paths (e.g. `/es/`) emit the correct `Content-Language` for that locale. 1931 1932### Content-Encoding 1933 193424. [ ] HTML response with `Accept-Encoding: gzip` returns `Content-Encoding: gzip`. 193525. [ ] HTML response with `Accept-Encoding: br` returns `Content-Encoding: br` (if brotli installed). 193626. [ ] HTML response with `Accept-Encoding: zstd` returns `Content-Encoding: zstd` (if zstd installed). 193727. [ ] CSS, JS, JSON, XML, SVG all compress. 193828. [ ] Already compressed types (JPEG, PNG, WebP, MP4, WOFF2) do NOT have `Content-Encoding`. 193929. [ ] Compressed responses have `Vary: Accept-Encoding`. 194030. [ ] Request without `Accept-Encoding` returns uncompressed body. 194131. [ ] No use of `Content-Encoding: deflate`. 194232. [ ] Brotli module loaded via `load_module` if used. 194333. [ ] Zstd module loaded via `load_module` if used. 194434. [ ] Precompressed `.gz`, `.br`, `.zst` files exist for static assets in production. 1945 1946### Content-Length 1947 194835. [ ] Static file response has `Content-Length` (when not on the fly compressed). 194936. [ ] On the fly compressed response uses `Transfer-Encoding: chunked`. 195037. [ ] No response has both `Content-Length` and `Transfer-Encoding`. 195138. [ ] Declared `Content-Length` matches actual body byte count. 195239. [ ] Large precompressed files have `Content-Length` showing the compressed size. 1953 1954### Content-Disposition 1955 195640. [ ] PDF download endpoints have `Content-Disposition: attachment`. 195741. [ ] Zip and archive endpoints have `Content-Disposition: attachment`. 195842. [ ] User upload serving endpoints have `Content-Disposition: attachment` AND `Content-Type: application/octet-stream`. 195943. [ ] Non download responses (HTML pages, CSS, JS, images) do NOT have `Content-Disposition: attachment`. 196044. [ ] Filename parameters are wrapped in double quotes. 196145. [ ] Non ASCII filenames include both `filename=` (ASCII fallback) and `filename*=` (RFC 5987 UTF-8). 196246. [ ] User input is never directly concatenated into Content-Disposition values without sanitization. 1963 1964### Cross-cutting 1965 196647. [ ] `nginx -t` passes without warnings. 196748. [ ] `nginx -T` (effective config) shows all expected directives. 196849. [ ] Lighthouse "Properly size images" and "Enable text compression" warnings are clear. 196950. [ ] Bing Webmaster Tools shows correct language detection (if Bing account is connected). 1970 1971A site that passes all 50 is ready for production traffic and crawling. 1972 1973--- 1974 1975## 14. COMMON PITFALLS 1976 1977Fifteen patterns to recognize and avoid. 1978 1979**Pitfall 1: Default type set to `text/html`.** 1980Symptom: unknown file extensions render as broken HTML in the browser. 1981Why it breaks: `default_type text/html` causes nginx to label every unknown extension as HTML, opening an XSS surface for any user controlled path. 1982Fix: `default_type application/octet-stream`. Add specific types via `types {}` block for any extension you actually serve. 1983 1984**Pitfall 2: Charset missing from text Content-Type.** 1985Symptom: pages with non ASCII characters (curly quotes, em dashes, accented characters) display as mojibake. 1986Why it breaks: browser falls back to Windows-1252 or another locale default. 1987Fix: enable `charset utf-8;` and `charset_types` directive (Section 5.5). 1988 1989**Pitfall 3: Mime type missing for new file extensions.** 1990Symptom: AVIF images not loading, WOFF2 fonts falling back to system fonts, WASM modules failing to instantiate. 1991Why it breaks: `/etc/nginx/mime.types` on older Debian releases predates these formats. nginx falls back to `default_type`. 1992Fix: add types in the http block (Section 5.6). 1993 1994**Pitfall 4: Compressing already compressed content.** 1995Symptom: high CPU on nginx, no measurable savings. 1996Why it breaks: JPEGs, PNGs, MP4s, WOFF2 fonts are already compressed. Gzipping them costs CPU and may even increase size by a few bytes. 1997Fix: do not list `image/jpeg`, `image/png`, `image/webp`, `image/avif`, `video/*`, `audio/*`, `font/woff2`, `application/zip` in `gzip_types`, `brotli_types`, or `zstd_types`. 1998 1999**Pitfall 5: Vary: Accept-Encoding missing.** 2000Symptom: random clients receive binary garbage; cache hit rates from CDNs or corporate proxies break. 2001Why it breaks: a cache serves a gzipped body to a client that did not request gzip. 2002Fix: `gzip_vary on;` (Section 7.4 and framework-http-caching-headers.md Section 9). 2003 2004**Pitfall 6: Content-Disposition: attachment on every response.** 2005Symptom: every page tries to download instead of render. 2006Why it breaks: a server wide `add_header Content-Disposition "attachment"` was added somewhere. 2007Fix: remove the global declaration. Only apply `attachment` to specific download locations. 2008 2009**Pitfall 7: Unsanitized filename in Content-Disposition.** 2010Symptom: security scanner reports vulnerability; user uploads can inject CRLF and forge headers. 2011Why it breaks: unsanitized user input flowing directly into a header value. 2012Fix: sanitize per Section 9.4. Never trust client supplied filenames. 2013 2014**Pitfall 8: User uploaded HTML rendered inline.** 2015Symptom: stored XSS in user uploads. 2016Why it breaks: file served with `Content-Type: text/html` and no `Content-Disposition: attachment`. 2017Fix: see Section 9.5. Force `attachment` plus `application/octet-stream` on every user upload endpoint. 2018 2019**Pitfall 9: Content-Language using underscores or wrong case.** 2020Symptom: language detection fails in screen readers, translation tools, Bing indexing. 2021Why it breaks: `en_US` is POSIX locale syntax, not BCP 47. `EN-US` violates the conventional casing. 2022Fix: `en-US` exactly. Lowercase language, uppercase region, hyphen between. 2023 2024**Pitfall 10: Setting Content-Language without matching `<html lang>`.** 2025Symptom: inconsistent language signals; some crawlers ignore the page, some misindex. 2026Why it breaks: the HTTP header says `es-MX`, the HTML says `<html lang="en">`. Crawlers do not know which to trust. 2027Fix: align all three layers: HTTP header, meta tag, html attribute. 2028 2029**Pitfall 11: Both Content-Length and Transfer-Encoding present.** 2030Symptom: HTTP request smuggling vulnerability scanner alert. 2031Why it breaks: per the RFC, this combination must not occur. Different intermediaries may interpret framing differently, enabling smuggling. 2032Fix: find the upstream emitting both. Remove the manual Content-Length and let chunked dominate. 2033 2034**Pitfall 12: Content-Length wrong for an upstream response.** 2035Symptom: clients report "incomplete response" or truncated content. 2036Why it breaks: upstream computed Content-Length before adding (or after removing) some bytes. 2037Fix: find the upstream code path. Either trust the framework to set Content-Length automatically, or compute it after the body is finalized. 2038 2039**Pitfall 13: Double compression (serving a .gz file with Content-Encoding: gzip).** 2040Symptom: browser fails to decompress. 2041Why it breaks: a `.tar.gz` archive is application data; serving it as `Content-Encoding: gzip` tells the browser to decompress the wire bytes, but then the result is still a `.tar` that the browser cannot show. 2042Fix: serve `.gz` files (when intended as downloads) with `Content-Type: application/gzip` and NO `Content-Encoding`. Use `gzip_static on` for precompressed CSS/JS, which sets the headers correctly. 2043 2044**Pitfall 14: Add_header inheritance trap on Content-Type.** 2045Symptom: nosniff or other security headers missing on certain file types after a location block override. 2046Why it breaks: any `add_header` in a location wipes out all parent add_headers (see framework-http-caching-headers.md Section 5.5). 2047Fix: snippet include pattern in every location. 2048 2049**Pitfall 15: Hardcoded Content-Type overriding nginx mime detection.** 2050Symptom: PNG returns `Content-Type: text/html`, JS returns `Content-Type: text/plain`, etc. 2051Why it breaks: someone added `add_header Content-Type "..."` in the wrong scope. 2052Fix: search the config: `grep -rn "Content-Type" /etc/nginx/`. Remove inappropriate manual declarations. Let mime.types handle it. 2053 2054--- 2055 2056## 15. DIAGNOSTIC COMMANDS 2057 2058Reference of every command useful for content header investigation. 2059 2060### Inspect content headers 2061 2062```bash 2063# Full header set 2064curl -sI https://example.com/page.html 2065 2066# Content family only 2067curl -sI https://example.com/page.html | grep -iE "content-(type|language|encoding|length|disposition)" 2068 2069# As Googlebot 2070curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 2071 https://example.com/page.html 2072 2073# As Bingbot (matters for Content-Language) 2074curl -sI -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" \ 2075 https://example.com/page.html 2076 2077# As ClaudeBot 2078curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \ 2079 https://example.com/page.html 2080``` 2081 2082### Verify Content-Type 2083 2084```bash 2085# Confirm declared type 2086curl -sI https://example.com/script.js | grep -i content-type 2087 2088# Verify actual file type 2089curl -s -o /tmp/asset https://example.com/script.js 2090file --mime-type /tmp/asset 2091# Compare to declared 2092 2093# Detect mojibake risk: dump first 200 bytes 2094curl -s https://example.com/ | head -c 200 | xxd 2095 2096# Verify HTML meta charset is in first 1024 bytes 2097curl -s https://example.com/ | head -c 1024 | grep -i charset 2098``` 2099 2100### Verify Content-Language 2101 2102```bash 2103# Header check 2104curl -sI https://example.com/ | grep -i content-language 2105curl -sI https://example.com/es/ | grep -i content-language 2106 2107# HTML lang attribute 2108curl -s https://example.com/ | grep -oE '<html[^>]*lang="[^"]*"' 2109 2110# Meta http-equiv 2111curl -s https://example.com/ | grep -oE '<meta[^>]*http-equiv="content-language"[^>]*>' 2112 2113# All hreflang annotations 2114curl -s https://example.com/ | grep -oE 'hreflang="[^"]*"' | sort -u 2115``` 2116 2117### Verify Content-Encoding 2118 2119```bash 2120# Compression negotiation 2121for enc in gzip br zstd identity; do 2122 SIZE=$(curl -s -H "Accept-Encoding: $enc" --output - https://example.com/page.html | wc -c) 2123 HDR=$(curl -sI -H "Accept-Encoding: $enc" https://example.com/page.html | grep -i content-encoding | awk '{print $2}' | tr -d '\r') 2124 echo "$enc: declared=$HDR size=$SIZE" 2125done 2126 2127# Verify compressed payload actually decompresses to expected content 2128curl -s --compressed https://example.com/page.html > /tmp/decompressed.html 2129curl -s -H "Accept-Encoding: identity" https://example.com/page.html > /tmp/raw.html 2130diff /tmp/decompressed.html /tmp/raw.html && echo "Match" 2131 2132# Check that brotli module is loaded 2133nginx -V 2>&1 | grep -o brotli 2134ls /etc/nginx/modules/ | grep brotli 2135 2136# Check that zstd module is loaded 2137nginx -V 2>&1 | grep -o zstd 2138ls /etc/nginx/modules/ | grep zstd 2139``` 2140 2141### Verify Content-Length 2142 2143```bash 2144# Declared vs actual (identity, no compression) 2145DECLARED=$(curl -sI -H "Accept-Encoding: identity" https://example.com/style.css | grep -i content-length | awk '{print $2}' | tr -d '\r') 2146ACTUAL=$(curl -s -H "Accept-Encoding: identity" https://example.com/style.css | wc -c) 2147echo "Declared: $DECLARED, Actual: $ACTUAL" 2148[ "$DECLARED" = "$ACTUAL" ] && echo "Match" || echo "MISMATCH" 2149 2150# Confirm chunked when compressed 2151curl -sI -H "Accept-Encoding: gzip" https://example.com/page.html | grep -iE "content-length|transfer-encoding" 2152 2153# Look for the dangerous both-headers combo 2154curl -sI https://example.com/api/data | awk 'tolower($0) ~ /^(content-length|transfer-encoding):/ {print}' 2155``` 2156 2157### Verify Content-Disposition 2158 2159```bash 2160# Confirm attachment for downloads 2161curl -sI https://example.com/downloads/report.pdf | grep -i content-disposition 2162 2163# Confirm inline (or absent) for renderable content 2164curl -sI https://example.com/page.html | grep -i content-disposition 2165 2166# Test that user uploads are forced to download 2167curl -sI https://example.com/user-uploads/file.html | grep -iE "content-(type|disposition)" 2168# Both must be present and restrictive 2169 2170# Check filename quoting 2171curl -sI https://example.com/downloads/report.pdf | grep -i content-disposition 2172# Filename should be wrapped in double quotes 2173``` 2174 2175### Server side investigation 2176 2177```bash 2178# Check nginx mime.types 2179cat /etc/nginx/mime.types | grep -i avif 2180cat /etc/nginx/mime.types | grep -i woff2 2181cat /etc/nginx/mime.types | grep -i wasm 2182 2183# Show all add_header in active config 2184nginx -T 2>/dev/null | grep -i add_header 2185 2186# Show all compression config in effect 2187nginx -T 2>/dev/null | grep -iE "gzip|brotli|zstd" 2188 2189# Show all charset directives 2190nginx -T 2>/dev/null | grep -i charset 2191 2192# Confirm modules loaded 2193nginx -V 2>&1 | tr ' ' '\n' | grep -iE "brotli|zstd" 2194 2195# Inspect source file encoding 2196file -bi /var/www/sites/example.com/index.html 2197# Expected: text/html; charset=utf-8 2198 2199# Find files that are not UTF-8 2200find /var/www/sites/example.com -name "*.html" -exec file -bi {} \; | grep -v utf-8 2201 2202# After config changes 2203nginx -t && systemctl reload nginx 2204``` 2205 2206### Browser DevTools quick reference 2207 2208In Chrome DevTools Network panel, click any request, then: 2209 2210* **Headers** tab: shows all response headers including the five in this framework. 2211* **Response** tab: shows the decompressed body. If the body looks like binary garbage here, Content-Encoding is wrong. 2212* **Size column**: shows wire size (compressed). Hover for "encoded size" and "actual size" (decompressed). 2213* The "Type" column shows the browser's interpretation based on Content-Type (Document, Stylesheet, Script, XHR, Image, Font, etc). 2214 2215To force a download when the server returned inline: 2216 2217* Right click on a link, "Save Link As". 2218* Or add `?download=true` and have the upstream check for the query and add `Content-Disposition: attachment`. 2219 2220--- 2221 2222## 16. CROSS-REFERENCES 2223 2224* [framework-http-caching-headers.md](framework-http-caching-headers.md): Cache-Control, ETag, Last-Modified, Expires, Vary, Age. Vary: Accept-Encoding is shared between this framework and that one; gzip weakens ETags is shared knowledge. 2225* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Section 24 (Bubbles Nginx config) inherits content header configuration from here. 2226* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook. References this framework for any content header decision. 2227* [framework-security-headers.md](framework-security-headers.md): HSTS, CSP, X-Frame-Options, X-Content-Type-Options. The nosniff header is critical for the Content-Type discipline described here. 2228* [framework-corewebvitals.md](framework-corewebvitals.md): LCP, INP, CLS targets. Content-Encoding directly affects LCP (smaller payloads transmit faster). 2229* [framework-hreflang.md](framework-hreflang.md): international SEO. Companion to Content-Language Section 6 here. 2230* [framework-compression.md](framework-compression.md): deep dive on gzip, brotli, zstd configuration, performance tuning, and migration. This file references it for advanced compression scenarios. 2231* Google official documentation on Content-Type and indexing: https://developers.google.com/search/docs/crawling-indexing/indexable-content 2232* Google international and multilingual sites: https://developers.google.com/search/docs/specialty/international/overview 2233* MDN Content-Type: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Type 2234* MDN Content-Disposition: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Disposition 2235* RFC 9110 (HTTP Semantics): https://www.rfc-editor.org/rfc/rfc9110 2236* RFC 6266 (Content-Disposition in HTTP): https://www.rfc-editor.org/rfc/rfc6266 2237* RFC 5987 (Character set and language encoding for HTTP header parameters): https://www.rfc-editor.org/rfc/rfc5987 2238 2239--- 2240 2241## APPENDIX A: ONE PAGE QUICK REFERENCE 2242 2243For the person who just wants the answer. 2244 2245### Content-Type table 2246 2247| Asset type | Content-Type | 2248|---|---| 2249| HTML | `text/html; charset=utf-8` | 2250| CSS | `text/css; charset=utf-8` | 2251| JavaScript | `application/javascript; charset=utf-8` | 2252| JSON | `application/json` | 2253| JSON-LD | `application/ld+json` | 2254| XML sitemap | `application/xml` | 2255| SVG | `image/svg+xml` | 2256| PNG | `image/png` | 2257| JPEG | `image/jpeg` | 2258| WebP | `image/webp` | 2259| AVIF | `image/avif` | 2260| WOFF2 | `font/woff2` | 2261| PDF | `application/pdf` | 2262| ZIP | `application/zip` | 2263| WASM | `application/wasm` | 2264| Web manifest | `application/manifest+json` | 2265| Plain text | `text/plain; charset=utf-8` | 2266| Unknown (safe default) | `application/octet-stream` | 2267 2268### Content-Language 2269 2270* Always `en-US` (or correct BCP 47) on every HTML response. 2271* Match `<html lang="en-US">`. 2272* Match `<meta http-equiv="content-language" content="en-US">`. 2273* Add hreflang for multi region sites. 2274 2275### Content-Encoding 2276 2277* Compress: HTML, CSS, JS, JSON, XML, SVG, plain text, fonts (ttf/otf only). 2278* Do NOT compress: JPEG, PNG, WebP, AVIF, MP4, WOFF2 (already compressed). 2279* Always pair with `Vary: Accept-Encoding`. 2280* Prefer precompressed (`gzip_static`, `brotli_static`, `zstd_static`) for static assets. 2281 2282### Content-Length 2283 2284* Auto for static files. 2285* Auto for precompressed files (the compressed size). 2286* Replaced with `Transfer-Encoding: chunked` when on the fly compression is active. 2287* Never appears with `Transfer-Encoding` together. 2288 2289### Content-Disposition 2290 2291* `attachment` for downloads (PDF, ZIP, generated CSVs). 2292* Omitted (default `inline`) for HTML, CSS, JS, images that should render. 2293* User uploads: always `attachment` plus `application/octet-stream`. 2294* Filenames wrapped in double quotes, both `filename` and `filename*` for non ASCII. 2295 2296Three commands every operator should know: 2297 2298```bash 2299# Confirm content type is correct 2300curl -sI https://example.com/script.js | grep -i content-type 2301 2302# Confirm compression is working 2303curl -sI -H "Accept-Encoding: br" https://example.com/page.html | grep -iE "content-encoding|vary" 2304 2305# Apply changes 2306nginx -t && systemctl reload nginx 2307``` 2308 2309And two end to end test invocations: 2310 2311```bash 2312# Verify a JSON API returns the right type and decompresses correctly 2313curl -s --compressed -H "Accept-Encoding: br, gzip" https://example.com/api/data | head -c 200 2314curl -sI --compressed https://example.com/api/data | grep -iE "content-(type|encoding|length)" 2315 2316# Verify a PDF download has the right disposition 2317curl -sI https://example.com/downloads/report.pdf | grep -iE "content-(type|disposition|length)" 2318``` 2319 2320If both produce expected output, the content header stack is correctly wired. 2321 2322--- 2323 2324End of framework-http-content-headers.md. 2325