Janady13/crawler-signal-reference / framework-http-seo-headers.md

1# framework-http-seo-headers.md 2 3Comprehensive reference for the three HTTP response headers that control indexing, resource discovery, and URL forwarding: `X-Robots-Tag` for indexing directives on any file type, `Link` for canonicals/hreflang/resource hints expressed at the HTTP layer, and `Location` for redirects. Built for Bubbles (Debian, Nginx 1.26+, self hosted origin at 169.155.162.118, no Cloudflare or third party CDN in front). Companion to framework-http-caching-headers.md, framework-http-content-headers.md, UNIVERSAL-RANKING-FRAMEWORK.md, and SEO BUILD REFERENCE v2.4. 4 5Audience: humans configuring nginx, AI assistants generating or repairing nginx config, and anyone troubleshooting "PDF appearing in Google when it should not", "preload not picked up", "redirect chain killing crawl budget", or "wrong page is canonical" anomalies on a self hosted stack. 6 7--- 8 9## TABLE OF CONTENTS 10 111. Definition 122. Why It Matters 133. What This Covers 144. The Indexing Authority Mental Model (read this first) 155. X-Robots-Tag (every meta robots directive at the HTTP layer) 166. Link (canonical, alternate, hreflang, preload, prefetch at HTTP) 177. Location (redirect target with status code semantics) 188. How These Headers Interact 199. Asset Class And Use Case Recipes 2010. Bubbles Nginx Reference Block (paste ready) 2111. Audit Checklist (50+ items) 2212. Common Pitfalls 2313. Diagnostic Commands (curl, GSC URL Inspection, browser devtools) 2414. Cross-References 25 26--- 27 28## 1. DEFINITION 29 30SEO and indexing headers tell search engine crawlers what they may or may not do with a resource, where its canonical version lives, what related resources exist, and where a moved URL can now be found. They operate at the HTTP layer, which means they apply to every response regardless of body content type. The three headers split into three concerns: 31 32* **Indexing control**: `X-Robots-Tag`. It answers "may you index this, follow its links, show a snippet, cache it, translate it?" 33* **Resource relationships**: `Link`. It answers "what is the canonical URL, what alternates exist, what resources should the browser preload, what is related to this resource?" 34* **Forwarding**: `Location`. It answers "this URL has moved; here is where to go next." 35 36Together these three headers shape every aspect of how search engines and AI crawlers understand a site's structure, what they index, and how PageRank or its modern equivalents flow between URLs. Getting any one of them wrong has direct ranking consequences: deindexed pages, lost canonical signals, broken hreflang clusters, redirect chains that bleed crawl budget, or worse, redirects that point to attacker controlled domains. 37 38--- 39 40## 2. WHY IT MATTERS 41 42Five independent pressures push correct SEO headers from "nice to have" to "required infrastructure" in 2025 and forward. 43 44**Indexing is selective by default.** Googlebot's crawl budget is finite. ClaudeBot, GPTBot, PerplexityBot, OAI-SearchBot, and other AI crawlers ration their requests even more aggressively. Sites that signal `noindex` or `nofollow` for low value URLs (filtered listings, search result pages, faceted navigation, internal admin) reclaim that budget for pages that actually matter. Sites that fail to signal end up with thousands of useless URLs in the index diluting their topical authority. 45 46**Non HTML files cannot use meta tags.** PDFs, images, videos, CSV exports, ZIP archives, and any other binary file cannot contain `<meta name="robots">`. The only way to control their indexing is `X-Robots-Tag`. A site that publishes PDFs without the header is silently allowing every PDF to be indexed, which causes brand pages and outdated material to outrank current HTML pages for the same query. 47 48**Resource hints are now a Core Web Vitals lever.** Modern browsers pre fetch resources declared in HTTP `Link` headers before parsing the HTML body. This shaves 50 to 300 ms off LCP for the largest contentful paint candidate (typically the hero image or above the fold font). Google's recommended pattern for hero image LCP optimization explicitly uses `Link: <hero.webp>; rel=preload; as=image; fetchpriority=high` as an HTTP header, not just the HTML link element. 49 50**103 Early Hints exists and matters.** Nginx 1.25+ supports HTTP 103 Early Hints, which lets the server send `Link` preload headers before the final response is ready. Chrome and Edge consume Early Hints and begin fetching critical resources during the time the server is still computing the page. On dynamic sites this can shave 100 to 500 ms off LCP at zero application cost. 51 52**Redirect chains are crawl killers and ranking killers.** Each hop in a redirect chain adds latency, consumes one unit of crawl budget, and risks losing equity along the way. Google has stated that all redirect codes (301, 302, 307, 308) are treated equivalently for ranking purposes, but the practical reality is that chains over three hops trigger soft 404 handling and lost consolidation. A single direct redirect is always better than a chain. 53 54**Open redirects are an active attack surface.** A redirect endpoint that accepts an unvalidated target parameter (`/redirect?url=...`) lets attackers craft phishing URLs that appear to come from your domain. Modern phishing kits scan for these endpoints. Every redirect endpoint that accepts user input must validate the target against an allow list. 55 56**Cost of getting it wrong.** Misconfigured SEO headers produce silent ranking failures and loud security failures. Examples from production sites: 57 58* A staging subdomain was indexed because nobody set `X-Robots-Tag: noindex`. Six months later Google merged duplicate content signals from staging and production, halving organic traffic for shared terms. 59* A site's brand PDFs from 2018 ranked above the 2024 HTML refresh because the old PDFs were indexed and Google had no signal that they were superseded. 60* A new product launch shipped with `Link: <previous-product>; rel=canonical` accidentally pointing at the old product page, blocking the new launch from indexing for three weeks. 61* A 301 chain (`/old` → `/new` → `/newer` → `/current`) bled crawl budget for years until someone audited and collapsed it to a single hop. 62* An open redirect on `/r?url=...` was used in a phishing campaign against the client's email list, prompting cease and desist orders and emergency cleanup. 63 64All preventable with the rules below. 65 66--- 67 68## 3. WHAT THIS COVERS 69 70Each of the three headers gets the same six part treatment: 71 721. **What it does**: the canonical RFC and Google specification plus the practical implication. 732. **Syntax and directives**: every legal value, what it means, and when it is wrong. 743. **How to build it on Bubbles**: paste ready nginx config. 754. **How to verify it**: curl commands plus Google Search Console URL Inspection where applicable. 765. **How to troubleshoot**: the four or five failure modes seen in the field and how to recognize each. 776. **How to fix common breakage**: ordered repair steps. 78 79Recipes for the most common asset classes and use cases are collected in Section 9. 80 81--- 82 83## 4. THE INDEXING AUTHORITY MENTAL MODEL (READ THIS FIRST) 84 85Every search engine and AI crawler that fetches a URL runs the same dispatch loop. Internalize it and every header choice becomes obvious. 86 87``` 88Crawler requests URL 89 | 90 v 91Server responds with status code 92 | 93 v 94Status 200, 304? --> proceed to indexing logic 95Status 301, 308? --> follow Location, consolidate signals to target 96Status 302, 307? --> follow Location, do NOT consolidate (temporary) 97Status 4xx? --> mark URL as gone or soft 404 98Status 5xx? --> defer, retry later, may eventually drop from index 99 | 100 v 101Read X-Robots-Tag header 102 | 103 v 104Directives present? 105 | | 106 YES NO 107 | | 108 v v 109Apply directives: Default behavior: 110- noindex: do not add to index - index this URL 111- nofollow: do not follow links - follow all links on it 112- nosnippet: no preview text - show snippet in SERP 113- noarchive: no cached copy - allow cached copy 114- max-snippet:N: cap snippet - default snippet length 115 | 116 v 117Read Link header 118 | 119 v 120- rel="canonical": treat target as primary, consolidate signals 121- rel="alternate" hreflang: register language variants 122- rel="alternate" media: register mobile/print variants 123- rel="prev" / rel="next": pagination (deprecated for SEO) 124- rel="preload" / "prefetch": browser hints, ignored by crawlers 125 | 126 v 127Read HTML body (if Content-Type is text/html and not noindex) 128 | 129 v 130Parse <head> for: 131- <meta name="robots"> (may override or supplement X-Robots-Tag) 132- <link rel="canonical"> (may override or supplement Link header) 133- <link rel="alternate" hreflang="..."> 134- <link rel="amphtml"> 135 | 136 v 137Reconcile signals from HTTP and HTML 138 | 139 v 140Update index entry 141``` 142 143Three rules govern the system: 144 1451. **HTTP signals reach every file type.** Meta tags only work for HTML. PDFs, images, JSON, CSV all rely exclusively on HTTP headers. If you publish any non HTML file you intend to control, you must use the HTTP layer. 1462. **The most restrictive directive wins.** Combining `noindex` and `index` (across HTTP and HTML, or across multiple X-Robots-Tag entries) resolves to `noindex`. Combining `nosnippet` and `max-snippet:200` resolves to `nosnippet`. There is no "negotiation" mechanism; restrictive always beats permissive. 1473. **Redirects consolidate or do not consolidate based on status code semantics.** Permanent codes (301, 308) tell crawlers to merge signals into the target URL. Temporary codes (302, 307) tell crawlers to keep the original URL indexed and treat the redirect as situational. Google has said the difference is "smaller than people think" for ranking, but the difference is real and matters for index hygiene. 148 149A correct header set produces consistent behavior across Googlebot, Bingbot, ClaudeBot, GPTBot, PerplexityBot, Applebot, OAI-SearchBot, and every other crawler. The same response, the same outcome everywhere. 150 151--- 152 153## 5. X-ROBOTS-TAG (EVERY META ROBOTS DIRECTIVE AT THE HTTP LAYER) 154 155### 5.1 What It Does 156 157`X-Robots-Tag` is the HTTP response header that conveys indexing directives to crawlers. Functionally equivalent to `<meta name="robots" content="...">` but applies to any response type, not just HTML. Documented by Google at developers.google.com/search/docs/crawling-indexing/robots-meta-tag. 158 159``` 160X-Robots-Tag: noindex 161X-Robots-Tag: noindex, nofollow 162X-Robots-Tag: googlebot: nosnippet, notranslate 163X-Robots-Tag: max-snippet:160, max-image-preview:large 164X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 GMT 165``` 166 167This is the only way to control indexing for non HTML resources: PDFs, images (when accessed directly by URL), videos, JSON endpoints, CSV exports, ZIP downloads, any binary file. For HTML pages, both `X-Robots-Tag` and `<meta name="robots">` work and can be combined. 168 169### 5.2 Every Directive Available 170 171The full set of directives Google supports as of 2026. Other major crawlers (Bing, Yandex, Baidu, ClaudeBot, GPTBot) support most of these. 172 173**Indexing directives:** 174 175| Directive | Meaning | 176|---|---| 177| `all` | Default. Equivalent to "no restrictions". Same as omitting the header | 178| `noindex` | Do not include this URL in the search index | 179| `nofollow` | Do not follow any links on this page (does not pass equity) | 180| `none` | Shorthand for `noindex, nofollow` | 181| `noarchive` | Do not show a "Cached" link in search results | 182| `nosnippet` | Do not show a text or video snippet in search results. Also blocks use of the content in AI Overviews and similar features | 183| `noimageindex` | Do not index any images on this page or use them as canonical sources | 184| `notranslate` | Do not offer translation of this page in search results | 185| `indexifembedded` | Allow indexing of this content when embedded via iframe in another page, even when `noindex` also applies. Useful for media publishers who want syndication | 186| `unavailable_after: <date>` | After the specified date and time, treat this URL as `noindex`. Date format: `Day, DD Mon YYYY HH:MM:SS GMT` | 187 188**Snippet control directives:** 189 190| Directive | Meaning | 191|---|---| 192| `max-snippet: <number>` | Maximum length in characters for text snippets. Special values: `0` (no snippet, same as `nosnippet`), `-1` (no limit) | 193| `max-image-preview: <setting>` | Largest image preview allowed in snippets. Values: `none`, `standard`, `large` | 194| `max-video-preview: <number>` | Maximum number of seconds for video previews. Special values: `0` (static image only), `-1` (no limit) | 195 196**Removed or deprecated:** 197 198| Directive | Status | 199|---|---| 200| `noodp` | Removed. Open Directory Project is defunct | 201| `noydir` | Removed. Yahoo Directory is defunct | 202| `noyaca` | Yandex specific. Yandex Catalog is defunct | 203 204### 5.3 Bot Specific Syntax 205 206By default, directives apply to every crawler. To target a specific bot, prefix the directives with the bot's user agent token followed by a colon. Multiple `X-Robots-Tag` headers can stack to specify different rules for different bots. 207 208``` 209X-Robots-Tag: googlebot: nosnippet, notranslate 210X-Robots-Tag: bingbot: noarchive 211X-Robots-Tag: noindex 212``` 213 214In this example: 215 216* Googlebot: gets `nosnippet, notranslate` AND inherits the global `noindex` (because the unprefixed line applies to it too). 217* Bingbot: gets `noarchive` AND the global `noindex`. 218* All other crawlers: get only `noindex`. 219 220**Important nuance about Google specifically:** Google only honors bot prefix syntax for `googlebot` and `googlebot-news`. Other Google crawlers (Googlebot-Image, Googlebot-Video, Storebot-Google, AdsBot-Google) do not respond to bot prefixed `X-Robots-Tag` directives. They are controlled via robots.txt user agent blocks instead. 221 222Common bot tokens worth knowing: 223 224| Bot | Token | 225|---|---| 226| Googlebot (web) | `googlebot` | 227| Googlebot News | `googlebot-news` | 228| Bingbot | `bingbot` | 229| Applebot | `applebot` | 230| ClaudeBot | `claudebot` | 231| GPTBot | `gptbot` | 232| PerplexityBot | `perplexitybot` | 233| OAI-SearchBot | `oai-searchbot` | 234| YandexBot | `yandex` | 235| Baiduspider | `baiduspider` | 236 237### 5.4 How To Build It On Bubbles 238 239**Block all indexing for a staging or demo site:** 240 241```nginx 242server { 243 server_name staging.example.com; 244 245 # Block every page from every crawler 246 add_header X-Robots-Tag "noindex, nofollow" always; 247 248 # Also block via robots.txt for belt and suspenders 249 location = /robots.txt { 250 return 200 "User-agent: *\nDisallow: /\n"; 251 } 252} 253``` 254 255**Block indexing for non HTML files only:** 256 257```nginx 258location ~* \.(pdf|doc|docx|xls|xlsx|csv|zip|tar|gz)$ { 259 add_header X-Robots-Tag "noindex" always; 260 expires 1d; 261 add_header Cache-Control "public, max-age=86400" always; 262} 263``` 264 265This is the primary use case for `X-Robots-Tag`: file types where you cannot use meta tags. 266 267**Block image indexing while allowing the HTML page to index:** 268 269```nginx 270# Apply at the image location only, not the parent HTML 271location ~* \.(jpg|jpeg|png|gif|webp|avif|svg)$ { 272 add_header X-Robots-Tag "noindex" always; 273} 274``` 275 276The HTML page itself remains indexable; only direct image URLs are blocked from being indexed as standalone results. 277 278**Set snippet limits site wide:** 279 280```nginx 281server { 282 add_header X-Robots-Tag "max-snippet:200, max-image-preview:large, max-video-preview:-1" always; 283 # Other directives... 284} 285``` 286 287This is the recommended baseline for most sites. It tells Google: allow up to 200 character text snippets, allow large image previews, no limit on video preview length. 288 289**Different rules for different bots:** 290 291```nginx 292# In a server block, stack multiple add_header directives 293add_header X-Robots-Tag "googlebot: max-snippet:200, max-image-preview:large" always; 294add_header X-Robots-Tag "bingbot: max-snippet:300" always; 295add_header X-Robots-Tag "noai" always; 296``` 297 298The order does not matter for parsing. Each header is independent. 299 300**Time limited content (unavailable_after):** 301 302```nginx 303location = /promotion/black-friday-2026.html { 304 add_header X-Robots-Tag "unavailable_after: 1 Dec 2026 00:00:00 GMT" always; 305 expires 1d; 306 add_header Cache-Control "public, max-age=86400" always; 307} 308``` 309 310After December 1, 2026, Google will drop this URL from search results without requiring a manual removal request. The page itself can remain accessible. 311 312**The add_header inheritance trap applies here too.** A nested location's `add_header` wipes parent declarations. Use the snippet include pattern: 313 314```nginx 315# /etc/nginx/snippets/robots-baseline.conf 316add_header X-Robots-Tag "max-snippet:200, max-image-preview:large, max-video-preview:-1" always; 317 318# In every location that needs it 319location /blog/ { 320 include snippets/robots-baseline.conf; 321 add_header Cache-Control "public, max-age=300" always; 322} 323``` 324 325### 5.5 The robots.txt vs X-Robots-Tag Distinction 326 327These two mechanisms control different things. Confusing them is the most common SEO mistake on the planet. 328 329| Mechanism | Controls | 330|---|---| 331| `robots.txt` | Which URLs the crawler is allowed to fetch | 332| `X-Robots-Tag` (or `<meta robots>`) | Which fetched URLs the crawler is allowed to index | 333 334A URL blocked by robots.txt is never fetched, which means the crawler never sees the `X-Robots-Tag` header. If the URL has external links pointing to it, Google may index it anyway as a "URL only" entry (no title, no snippet, just the URL) based on those external signals. 335 336To actually deindex a URL: 337 3381. **Remove the robots.txt block** so the crawler can fetch the URL. 3392. **Add `X-Robots-Tag: noindex`** to the response. 3403. **Wait for the crawler to re fetch and process the directive.** 3414. **Then optionally add the robots.txt block back** to prevent further crawling, but only after Google has processed the noindex. 342 343Get this order wrong and the URL stays indexed forever. Get it right and the URL drops from the index within a crawl cycle (days to weeks). 344 345### 5.6 How To Verify 346 347```bash 348# 1. Confirm header is present 349curl -sI https://example.com/some-page | grep -i x-robots-tag 350 351# 2. Check for a specific file type 352curl -sI https://example.com/downloads/report.pdf | grep -i x-robots-tag 353 354# 3. Check what Googlebot sees 355curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 356 https://example.com/page.html | grep -i x-robots-tag 357 358# 4. Test bot specific directives 359curl -sI -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" \ 360 https://example.com/page.html | grep -i x-robots-tag 361 362# 5. Compare HTTP header to HTML meta tag 363echo "=== HTTP X-Robots-Tag ===" 364curl -sI https://example.com/page.html | grep -i x-robots-tag 365echo "=== HTML meta robots ===" 366curl -s https://example.com/page.html | grep -i 'name="robots"' 367 368# 6. Verify a site wide block is in effect 369for path in / /about /blog /contact /robots.txt; do 370 echo "=== https://staging.example.com$path ===" 371 curl -sI "https://staging.example.com$path" | grep -i x-robots-tag 372done 373``` 374 375**Use Google Search Console URL Inspection** for the authoritative check on whether Google sees and respects the directive: 376 3771. Open Google Search Console for the property. 3782. Paste the URL into the top search bar. 3793. Click "Test live URL". 3804. Under "Page indexing", look for "Indexing allowed?" and the source of any "noindex" detection. 381 382The Live Test fetches the URL with Googlebot's actual user agent and shows the headers it saw. This is the ground truth for SEO purposes. 383 384### 5.7 Troubleshooting 385 386**Symptom: Page is still in Google index despite `X-Robots-Tag: noindex`.** 387Causes ranked by frequency: 3881. The URL is blocked by robots.txt, so Googlebot cannot fetch it and never sees the header. Remove the robots.txt block, wait, then optionally re add. 3892. The page has not been re crawled since the directive was added. Wait or request indexing via Search Console URL Inspection. 3903. The directive has a typo (e.g. `no-index` instead of `noindex`). Verify with curl. 3914. A CDN is stripping the header. Not applicable on Bubbles. Verify with `curl --resolve` directly against the origin. 3925. The page has strong external links and Google retains a URL only entry. This is rare and resolves over time. 393 394**Symptom: PDF is being indexed and outranking the corresponding HTML page.** 395Cause: No `X-Robots-Tag` on the PDF location. 396Fix: see Section 5.4. Add `X-Robots-Tag: noindex` to a location matching `\.pdf$`. If you want the PDF indexable but not outranking the HTML, add `<link rel=canonical>` to the HTML and `X-Robots-Tag: noindex, nofollow` to the PDF. 397 398**Symptom: Two `X-Robots-Tag` headers appear and seem to conflict.** 399Multiple directives across multiple headers are interpreted as a union. The most restrictive wins. `X-Robots-Tag: index` plus `X-Robots-Tag: noindex` resolves to `noindex`. There is no conflict; this is the spec. 400 401**Symptom: `unavailable_after` date passed but URL is still indexed.** 402Causes: 4031. Date format is wrong. Must be RFC 850 or RFC 822 format. The Google docs recommend `Day, DD Mon YYYY HH:MM:SS GMT`. Verify with curl. 4042. Google has not re crawled since the date passed. Crawl frequency depends on the URL's profile. 4053. `unavailable_after` is intentionally not retroactive in a strong sense; it influences future crawls. 406 407**Symptom: AI engines (ChatGPT, Claude, Perplexity) are still using content despite `nosnippet`.** 408The `nosnippet` directive applies to Google search results, including AI Overviews. AI engines that crawl independently (ChatGPT search, Claude search, Perplexity) honor their own published opt out signals which often differ: 409* OpenAI: respect `User-agent: GPTBot` in robots.txt 410* Anthropic: respect `User-agent: ClaudeBot` in robots.txt 411* Perplexity: respect `User-agent: PerplexityBot` in robots.txt 412* Common Crawl: respect `User-agent: CCBot` in robots.txt 413Add explicit robots.txt blocks for each AI agent you want to opt out of, in addition to any `X-Robots-Tag` directives. 414 415**Symptom: Bot prefixed directive ignored.** 416Google only honors `googlebot:` and `googlebot-news:` prefixes. Other Google crawlers (Googlebot-Image, Googlebot-Video) are not controllable via this mechanism. Use robots.txt for those. 417 418### 5.8 How To Fix Common Breakage 419 420**Case: Staging site indexed in Google.** 421Add to the server block immediately: 422 423```nginx 424server { 425 server_name staging.example.com; 426 add_header X-Robots-Tag "noindex, nofollow" always; 427 # ... other config ... 428} 429``` 430 431`nginx -t && systemctl reload nginx`. Then in Search Console, request removal for the entire staging subdomain. The combination of the noindex header (preventing re indexation) and the removal request (immediate effect) resolves the issue within hours. 432 433**Case: All PDFs are appearing in search and bleeding click traffic from HTML pages.** 434Block all PDFs from indexing: 435 436```nginx 437location ~* \.pdf$ { 438 add_header X-Robots-Tag "noindex" always; 439 add_header Cache-Control "public, max-age=86400" always; 440} 441``` 442 443The PDFs remain downloadable; only their index entries disappear. Existing index entries drop on next crawl. 444 445**Case: A campaign page needs to be deindexed after the campaign ends.** 446Add `unavailable_after` with the campaign end date: 447 448```nginx 449location = /campaigns/spring-2026.html { 450 add_header X-Robots-Tag "unavailable_after: 30 Jun 2026 23:59:59 GMT" always; 451} 452``` 453 454No manual intervention required after June 30. 455 456**Case: Need to allow ClaudeBot but block GPTBot.** 457Use robots.txt (not X-Robots-Tag, since both ClaudeBot and GPTBot have limited bot prefix support): 458 459``` 460User-agent: GPTBot 461Disallow: / 462 463User-agent: ClaudeBot 464Allow: / 465 466User-agent: * 467Allow: / 468``` 469 470Serve via: 471 472```nginx 473location = /robots.txt { 474 expires 1h; 475 add_header Cache-Control "public, max-age=3600" always; 476} 477``` 478 479--- 480 481## 6. LINK (CANONICAL, ALTERNATE, HREFLANG, PRELOAD, PREFETCH AT HTTP) 482 483### 6.1 What It Does 484 485The `Link` header expresses relationships between the current response and other resources. Defined in RFC 8288. The same semantic content as the HTML `<link>` element, but in HTTP form. Critical for non HTML responses (PDFs cannot contain `<link>` tags but can have HTTP `Link` headers) and for performance optimization (HTTP `Link` is processed before HTML parsing begins). 486 487``` 488Link: <https://example.com/page>; rel="canonical" 489Link: <https://example.com/es/page>; rel="alternate"; hreflang="es-MX" 490Link: </css/main.css>; rel="preload"; as="style" 491Link: </hero.webp>; rel="preload"; as="image"; fetchpriority="high" 492Link: <https://cdn.example.com>; rel="preconnect" 493``` 494 495Multiple relationships can be comma separated in a single header or split across multiple `Link` headers. Both forms are equivalent. 496 497### 6.2 Syntax 498 499The general form: 500 501``` 502Link: <URI>; param1=value1; param2="value 2" 503Link: <URI1>; rel="..."; param="..." , <URI2>; rel="..." 504``` 505 506Rules: 507 508* URI is wrapped in angle brackets: `<https://example.com/x>`. 509* Parameters are separated from the URI by a semicolon. 510* Parameters are separated from each other by semicolons. 511* Multiple relationships are separated by commas (use one comma per Link header line; do not nest). 512* Quote values with spaces or special characters using double quotes. 513* `rel` is required for almost every meaningful Link header. 514 515### 6.3 Relationship Types 516 517The values for `rel` come from the IANA Link Relations registry. The ones that matter on the web: 518 519**SEO and canonicalization:** 520 521| rel | Purpose | 522|---|---| 523| `canonical` | Declare the canonical URL for this resource. Critical for SEO. The target URL is treated as the authoritative version | 524| `alternate` (with `hreflang`) | Declare a language or region variant of the current resource. Used for international SEO | 525| `alternate` (with `type="application/rss+xml"`) | Declare an RSS or Atom feed for this page | 526| `alternate` (with `media`) | Declare a print stylesheet, mobile variant, or other media variant | 527| `amphtml` | Declare the AMP version of this page. Less relevant in 2026 as AMP has been deprecated by Google | 528| `prev` | Previous page in a paginated series. Google announced in 2019 they ignore this; still useful for browser navigation | 529| `next` | Next page in a paginated series. Same status as prev | 530| `up` | Parent resource in a hierarchical structure | 531 532**Performance:** 533 534| rel | Purpose | `as` required | 535|---|---|---| 536| `preload` | Fetch this resource at high priority for the current page. Discovered before HTML parsing finishes | Yes | 537| `prefetch` | Fetch this resource at low priority for likely future navigation | No, but recommended | 538| `preconnect` | Open a TCP/TLS connection to this origin in advance | No | 539| `dns-prefetch` | Resolve DNS for this origin in advance | No | 540| `modulepreload` | Preload a JavaScript ES module and parse it | No | 541| `prerender` | Render this URL in a background tab. Largely replaced by Speculation Rules | 542 543**Other:** 544 545| rel | Purpose | 546|---|---| 547| `stylesheet` | This resource is a CSS stylesheet. Almost never used as HTTP Link header; use HTML link tag | 548| `icon` | Favicon. Almost never used as HTTP Link; use HTML link tag | 549| `manifest` | PWA manifest URL | 550| `help` | Reference to a help document | 551| `license` | Reference to license information for the content | 552| `author` | Reference to author information | 553 554### 6.4 The `as` Parameter For Preload 555 556`rel="preload"` requires an `as` parameter declaring the resource type. The browser uses this to set the correct request priority, Accept headers, and CSP enforcement. Wrong `as` value causes the browser to discard the preload and re fetch. 557 558| `as` value | For | 559|---|---| 560| `script` | JavaScript files | 561| `style` | CSS files | 562| `font` | Web fonts. Requires `crossorigin` parameter | 563| `image` | Images (JPEG, PNG, WebP, AVIF, SVG) | 564| `fetch` | XHR or fetch() targets (JSON APIs, etc). Requires `crossorigin` if cross origin | 565| `document` | iframe document | 566| `video` | Video files | 567| `audio` | Audio files | 568| `track` | WebVTT track files | 569| `worker` | Web Worker scripts | 570 571### 6.5 fetchpriority 572 573Modern browsers (Chrome 101+, Edge 101+, Safari 17.2+, Firefox 132+) support the `fetchpriority` parameter on preload and other resource hints: 574 575``` 576Link: </hero.webp>; rel=preload; as=image; fetchpriority=high 577``` 578 579Values: `high`, `low`, `auto` (default). Used to bump or demote a specific resource's priority within the browser's fetch queue. Critical for LCP optimization: marking the hero image as `fetchpriority=high` typically moves LCP earlier by 100 to 400 ms. 580 581### 6.6 crossorigin 582 583Required for fonts and for any cross origin preload that uses CORS: 584 585``` 586Link: </fonts/main.woff2>; rel=preload; as=font; crossorigin 587Link: <https://cdn.example.com/api.js>; rel=preload; as=script; crossorigin=anonymous 588``` 589 590Values: `anonymous` (no credentials sent, default for `crossorigin`) or `use-credentials` (send cookies and auth). 591 592### 6.7 How To Build It On Bubbles 593 594**Canonical for a PDF or other non HTML asset:** 595 596```nginx 597location = /downloads/whitepaper.pdf { 598 add_header Link '<https://example.com/whitepapers/q3-2026>; rel="canonical"' always; 599} 600``` 601 602The PDF is the asset; the canonical URL is the HTML landing page about the whitepaper. This consolidates index signals onto the HTML. 603 604**Hreflang via HTTP header (for non HTML):** 605 606```nginx 607location = /downloads/manual.pdf { 608 add_header Link '<https://example.com/downloads/manual.pdf>; rel="alternate"; hreflang="en-US", <https://example.com/es/downloads/manual.pdf>; rel="alternate"; hreflang="es-MX", <https://example.com/downloads/manual.pdf>; rel="alternate"; hreflang="x-default"' always; 609} 610``` 611 612Each PDF in a multi region cluster declares every other language version. The set must include `x-default` and must be reciprocal (every variant lists every other variant including itself). 613 614**Preload critical assets:** 615 616```nginx 617location = / { 618 # Preload the hero image, critical CSS, and primary font 619 add_header Link '</css/critical.css>; rel="preload"; as="style", </fonts/manrope.woff2>; rel="preload"; as="font"; crossorigin, </images/hero.webp>; rel="preload"; as="image"; fetchpriority="high"' always; 620 expires 0; 621 add_header Cache-Control "public, max-age=0, must-revalidate" always; 622} 623``` 624 625These preloads start fetching as soon as the response headers arrive at the browser, before HTML parsing begins. For a hero image LCP candidate, this typically improves LCP by 100 to 300 ms. 626 627**Preconnect to external origins (rare on Bubbles since we self host everything):** 628 629```nginx 630location / { 631 add_header Link '<https://www.googletagmanager.com>; rel="preconnect", <https://www.google-analytics.com>; rel="preconnect"' always; 632} 633``` 634 635Useful when third party origins are unavoidable (analytics, tag manager). Establishes the TCP/TLS connection before the page needs the resource. 636 637**RSS feed declaration via HTTP:** 638 639```nginx 640location / { 641 add_header Link '</rss>; rel="alternate"; type="application/rss+xml"; title="Site Feed"' always; 642} 643``` 644 645Feed readers and browsers that support RSS auto discovery will find the feed through this header. 646 647**Snippet pattern for the add_header inheritance trap:** 648 649```nginx 650# /etc/nginx/snippets/preloads.conf 651add_header Link '</css/critical.css>; rel="preload"; as="style", </fonts/manrope.woff2>; rel="preload"; as="font"; crossorigin' always; 652 653# In every relevant location 654location ~* \.html$ { 655 include snippets/preloads.conf; 656 add_header Cache-Control "public, max-age=0, must-revalidate" always; 657} 658``` 659 660### 6.8 HTTP 103 Early Hints (the killer feature) 661 662Nginx 1.25+ supports HTTP 103 Early Hints, which lets the server send `Link` preload headers before the final response is ready. The flow: 663 6641. Browser requests `/`. 6652. Server returns `103 Early Hints` immediately with `Link` headers. 6663. Browser sees the preload directives and begins fetching critical resources. 6674. Server takes 200 to 800 ms to compute the page (database queries, template rendering). 6685. Server returns `200 OK` with the page body. 6696. By the time the page parses, critical resources are already in cache. 670 671Configuration: 672 673```nginx 674location / { 675 early_hints 103; 676 add_header Link '</css/critical.css>; rel="preload"; as="style"' always; 677 proxy_pass http://127.0.0.1:9090; 678} 679``` 680 681The `early_hints 103` directive (nginx 1.25+) tells nginx to emit a 103 status with the `Link` headers before the final response. Supported in Chrome 103+, Edge 103+, Safari 17+, and Firefox 120+. Older browsers ignore the 103 and process the final response normally. 682 683This is the single highest impact optimization for dynamic sites with slow first byte times. On a FastAPI sidecar that takes 400 ms to render, Early Hints typically reduces LCP by 200 to 400 ms with no application change. 684 685### 6.9 How To Verify 686 687```bash 688# 1. Confirm Link header is present 689curl -sI https://example.com/ | grep -i link 690 691# 2. Pretty print multiple Link entries 692curl -sI https://example.com/ | grep -i '^link:' | tr ',' '\n' | sed 's/^/ /' 693 694# 3. Verify canonical for a PDF 695curl -sI https://example.com/downloads/report.pdf | grep -i link 696 697# 4. Check hreflang variants 698curl -sI https://example.com/ | grep -i link | grep -o 'hreflang="[^"]*"' | sort -u 699 700# 5. Test preload is being emitted 701curl -sI https://example.com/ | grep -i link | grep -o 'rel="preload"' 702 703# 6. Verify Early Hints (requires nginx 1.25+ and a client that supports 103) 704curl -sv --http2 https://example.com/ 2>&1 | grep -i "HTTP/2 103" 705 706# 7. Check that Googlebot sees the same Link header 707curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 708 https://example.com/ | grep -i link 709 710# 8. Verify the URLs in Link headers actually resolve 711for url in $(curl -sI https://example.com/ | grep -i link | grep -oE '<[^>]+>' | tr -d '<>'); do 712 echo "=== $url ===" 713 curl -sI "$url" | head -1 714done 715``` 716 717### 6.10 Troubleshooting 718 719**Symptom: Preload appears in header but browser fetches the resource twice.** 720Causes: 7211. `as` value mismatch. The browser fetched with `as=script` priority but the page actually loaded the resource as a stylesheet. Fix the `as` to match. 7222. Cross origin without `crossorigin` parameter. Fonts especially fail this way. Add `crossorigin` to the Link header. 7233. Different URL between preload and actual use. The preload was for `/css/main.css` but the HTML loaded `/css/main.v2.css`. The URLs must match byte for byte. 724 725**Symptom: Canonical via HTTP Link header is ignored by Google.** 726Causes: 7271. Conflicts with an HTML `<link rel=canonical>` declaring a different URL. Decide which is authoritative and remove the other. 7282. The Link header URL is relative when it should be absolute. Always use absolute URLs in `rel=canonical`. 7293. The canonical target returns 404 or 5xx. Google ignores broken canonicals. 7304. The canonical target itself canonicalizes elsewhere (a chain). Resolve to the final canonical. 731 732**Symptom: Early Hints not being sent.** 733Causes: 7341. Nginx version under 1.25. Check `nginx -v`. 7352. `early_hints 103;` directive missing from the location. 7363. The client does not support 103 (most older HTTP clients do not). Test with `curl -sv --http2`. 7374. The upstream returned the final response too quickly for nginx to emit the 103 first. Acceptable; the browser still uses the final response's Link headers. 738 739**Symptom: Hreflang cluster has missing reciprocity warnings in Search Console.** 740Every variant in the cluster must list every other variant including itself. Audit: 741 742```bash 743for variant in en-US es-MX en-GB; do 744 URL="https://example.com/$variant/page" 745 echo "=== $URL ===" 746 curl -sI "$URL" | grep -i link | grep -oE 'hreflang="[^"]*"' | sort -u 747done 748``` 749 750Each should list the same set including `x-default`. Add any missing. 751 752**Symptom: Browser console warns "The resource was preloaded but not used within a few seconds".** 753The preload is wasted. Either the resource is not actually critical and should be removed from preload, or the page is not loading it correctly. Audit which resources truly contribute to LCP and preload only those. 754 755### 6.11 How To Fix Common Breakage 756 757**Case: PDFs have no canonical, are competing with HTML in search.** 758Add canonical headers to the PDF location: 759 760```nginx 761location ~ ^/downloads/(.*)\.pdf$ { 762 # Compute the canonical HTML URL based on the PDF name 763 # Adjust the pattern to match your site structure 764 add_header Link '<https://example.com/whitepapers/$1>; rel="canonical"' always; 765 add_header X-Robots-Tag "noindex" always; 766} 767``` 768 769The PDF declares the HTML page as canonical and is itself noindex. Index signals consolidate onto the HTML. 770 771**Case: LCP is slow due to hero image discovery latency.** 772Preload the hero image with high fetchpriority: 773 774```nginx 775location = / { 776 add_header Link '</images/hero-2026.webp>; rel="preload"; as="image"; fetchpriority="high"' always; 777} 778``` 779 780If the hero image varies by viewport (responsive images), preload only the largest variant and let the browser pick the right one via srcset. Or, more advanced, use the `media` attribute on the preload: 781 782``` 783Link: </hero-large.webp>; rel=preload; as=image; media="(min-width: 1024px)", </hero-small.webp>; rel=preload; as=image; media="(max-width: 1023px)" 784``` 785 786**Case: Web font loads late and causes FOIT (flash of invisible text).** 787Preload the font with crossorigin: 788 789```nginx 790location ~* \.html$ { 791 add_header Link '</fonts/manrope-400.woff2>; rel="preload"; as="font"; crossorigin' always; 792} 793``` 794 795The font fetch starts immediately. Combined with `font-display: swap` in CSS, FOUT and FOIT are largely eliminated. 796 797**Case: Dynamic page from FastAPI sidecar is slow to TTFB.** 798Enable Early Hints: 799 800```nginx 801location /dashboard { 802 early_hints 103; 803 add_header Link '</css/dashboard.css>; rel="preload"; as="style", </js/dashboard.js>; rel="preload"; as="script"' always; 804 proxy_pass http://127.0.0.1:9090; 805} 806``` 807 808Modern Chrome and Edge users see LCP improve by 200 to 400 ms with no application code change. 809 810--- 811 812## 7. LOCATION (REDIRECT TARGET WITH STATUS CODE SEMANTICS) 813 814### 7.1 What It Does 815 816`Location` declares the URL a client should fetch instead of the current one. Required for any 3xx redirect response. Defined in RFC 9110 Section 10.2.2. 817 818``` 819HTTP/2 301 820Location: https://example.com/new-url 821``` 822 823The status code that accompanies `Location` carries the semantic meaning: is this redirect permanent or temporary, must the HTTP method be preserved, and how should clients cache the redirect. 824 825### 7.2 The Five Redirect Status Codes 826 827| Code | Name | Permanent? | HTTP method preserved on redirect? | Cacheable by browser? | 828|---|---|---|---|---| 829| 301 | Moved Permanently | Yes | Often changed to GET historically; modern clients preserve | Yes, indefinitely | 830| 302 | Found | No (temporary) | Often changed to GET; modern clients vary | Sometimes | 831| 303 | See Other | No | Always changed to GET | No | 832| 307 | Temporary Redirect | No | Strictly preserved | No | 833| 308 | Permanent Redirect | Yes | Strictly preserved | Yes | 834 835**The rules in practice:** 836 837* **301**: use for permanent URL changes where the original URL is being abandoned. Most common redirect code. Browsers cache aggressively, which means changing the destination later requires cache busting. Search engines consolidate signals into the target URL. 838* **302**: use for temporary URL changes that may revert. The original URL remains indexed. Common for A/B tests, geographic redirects, maintenance pages. 839* **303**: use after a form POST to redirect to a confirmation page (Post/Redirect/Get pattern). Forces GET on the redirect to prevent accidental form resubmission on browser refresh. 840* **307**: same semantics as 302 (temporary) but strictly preserves the HTTP method. A POST stays a POST. Use for temporary redirects of API endpoints where method preservation matters. 841* **308**: same semantics as 301 (permanent) but strictly preserves the HTTP method. Use for permanent redirects of API endpoints. 842 843**Google's stance** (John Mueller, 2016 and reaffirmed multiple times): all redirect codes are treated equivalently for SEO ranking purposes. The full PageRank passes through any of them. Choose based on technical semantics, not perceived SEO benefit. 844 845**Practical Bubbles defaults:** 846 847* HTTP to HTTPS migration: 301. 848* www to non www (or vice versa): 301. 849* Old URL to new URL after a site restructure: 301. 850* Marketing campaign redirect: 302 (in case the campaign URL needs to change). 851* Geographic or device redirect: 302. 852* API endpoint version migration: 308 (preserves POST method). 853* Login redirect after form submit: 303. 854 855### 7.3 How To Build It On Bubbles 856 857**Simple permanent redirect:** 858 859```nginx 860location = /old-page { 861 return 301 https://example.com/new-page; 862} 863``` 864 865**Wildcard redirect for an entire path:** 866 867```nginx 868location /old-section/ { 869 return 301 https://example.com/new-section/$request_uri; 870} 871``` 872 873Caution: `$request_uri` includes the leading `/old-section/`. For a clean rewrite, use a different variable: 874 875```nginx 876location ~ ^/old-section/(.*)$ { 877 return 301 https://example.com/new-section/$1; 878} 879``` 880 881**HTTP to HTTPS redirect (every Bubbles site has this):** 882 883```nginx 884server { 885 listen 80; 886 listen [::]:80; 887 server_name example.com www.example.com; 888 return 301 https://$host$request_uri; 889} 890``` 891 892**www to non www canonicalization:** 893 894```nginx 895server { 896 listen 443 ssl; 897 server_name www.example.com; 898 return 301 https://example.com$request_uri; 899} 900 901server { 902 listen 443 ssl; 903 server_name example.com; 904 # ... actual site config ... 905} 906``` 907 908**Subdomain to canonical domain (for the 14+ paying client subdomains on thatwebhostingguy.com that 301 to client custom domains):** 909 910```nginx 911server { 912 listen 443 ssl; 913 server_name arcounselingandwellness.thatwebhostingguy.com; 914 return 301 https://arcounselingandwellness.com$request_uri; 915} 916``` 917 918**Conditional redirect based on URL pattern:** 919 920```nginx 921server { 922 server_name example.com; 923 924 # Redirect old blog URLs to new structure 925 location ~ ^/blog/(\d{4})/(\d{2})/(.+)$ { 926 return 301 https://example.com/articles/$1-$2/$3; 927 } 928 929 # Redirect old product URLs based on a map 930 location /products/ { 931 try_files $uri @product_redirects; 932 } 933 934 location @product_redirects { 935 if ($product_redirect != "") { 936 return 301 $product_redirect; 937 } 938 return 404; 939 } 940} 941 942# Defined at http level 943map $request_uri $product_redirect { 944 default ""; 945 /products/old-widget /products/new-widget; 946 /products/old-gadget /products/new-gadget; 947 /products/discontinued /; 948} 949``` 950 951**Temporary redirect for maintenance page:** 952 953```nginx 954server { 955 server_name example.com; 956 957 set $maintenance 0; 958 # Toggle this to 1 during maintenance 959 # if ($maintenance) { 960 # return 302 /maintenance.html; 961 # } 962 963 # ... rest of config ... 964} 965``` 966 967**API endpoint version migration (308 to preserve POST):** 968 969```nginx 970location /api/v1/ { 971 return 308 https://example.com/api/v2/$request_uri; 972} 973``` 974 975A POST request to `/api/v1/something` becomes a POST to `/api/v2/something`. The body is preserved. 976 977### 7.4 The Open Redirect Vulnerability 978 979A redirect endpoint that takes the target URL from user input without validation is a phishing vector. Attackers craft URLs like: 980 981``` 982https://example.com/redirect?url=https://phishing-site.com/login 983``` 984 985The link appears to come from your domain, lending credibility. The victim clicks, lands on the phishing site, enters credentials. 986 987**Defenses:** 988 989**1. Never accept full URLs from user input.** Only accept path components and prepend a fixed origin: 990 991```nginx 992# Wrong (vulnerable) 993location = /r { 994 return 302 $arg_url; 995} 996 997# Right 998location = /r { 999 if ($arg_to ~ ^/[a-z0-9/_-]+$) { 1000 return 302 https://example.com$arg_to; 1001 } 1002 return 400; 1003} 1004``` 1005 1006**2. Validate against an allow list of full URLs:** 1007 1008```nginx 1009map $arg_url $is_allowed_redirect { 1010 default ""; 1011 https://example.com/promo "1"; 1012 https://example.com/login "1"; 1013 https://example.com/help "1"; 1014} 1015 1016location = /r { 1017 if ($is_allowed_redirect = "") { 1018 return 400; 1019 } 1020 return 302 $arg_url; 1021} 1022``` 1023 1024**3. Require an HMAC signature on the redirect parameter** so attackers cannot forge new targets. Implement at the upstream FastAPI sidecar level for any redirect endpoint that must accept dynamic targets. 1025 1026**4. Never combine user input with a redirect status code without one of the above defenses.** Treat redirect endpoints with the same care as authentication endpoints. 1027 1028### 7.5 Redirect Chains And Crawl Budget 1029 1030A redirect chain is a sequence of redirects: `/a` → `/b` → `/c` → `/d`. Each hop costs: 1031 1032* One crawler request (against finite crawl budget). 1033* One TLS handshake plus one TCP round trip for users (latency adds up). 1034* Potentially lost PageRank, although Google denies this in modern documentation. 1035 1036Google's official guidance is to keep chains under 5 hops. Practical recommendation: always 1 hop. Audit periodically: 1037 1038```bash 1039# Trace any chain 1040curl -sLI https://example.com/old-url | grep -i "^location:" 1041 1042# Count hops 1043curl -sLI https://example.com/old-url | grep -c "^HTTP/" 1044``` 1045 1046A `1` indicates direct response (no redirect or single redirect). A `2` or higher indicates a chain. 1047 1048When a chain is detected: 1049 10501. Identify the final destination. 10512. Update the original redirect to point directly to the final destination. 10523. Leave the intermediate redirects in place for a few months in case other consumers have cached the intermediate URLs. 10534. Remove the intermediate redirects once they show zero traffic in logs. 1054 1055### 7.6 How To Verify 1056 1057```bash 1058# 1. Confirm redirect status and target 1059curl -sI https://example.com/old-page | grep -iE "^(HTTP|Location):" 1060 1061# 2. Follow the chain 1062curl -sLI https://example.com/old-page | grep -iE "^(HTTP|Location):" 1063 1064# 3. Verify final destination resolves 1065curl -sIL -o /dev/null -w "%{url_effective} %{http_code}\n" https://example.com/old-page 1066 1067# 4. Verify the redirect preserves the path (for wildcard redirects) 1068for path in "" "/sub/page" "/file.pdf?query=1"; do 1069 URL="https://example.com/old-section$path" 1070 echo "=== $URL ===" 1071 curl -sI "$URL" | grep -i location 1072done 1073 1074# 5. Test that POST is preserved (for 307 / 308) 1075curl -X POST -sI https://example.com/api/v1/endpoint | grep -iE "^(HTTP|Location):" 1076 1077# 6. Audit: find chains 1078for url in https://example.com/old-1 https://example.com/old-2 https://example.com/old-3; do 1079 HOPS=$(curl -sLI "$url" | grep -c "^HTTP/") 1080 DEST=$(curl -sIL -o /dev/null -w "%{url_effective}" "$url") 1081 echo "$url -> $DEST ($HOPS hops)" 1082done 1083 1084# 7. Verify Googlebot follows correctly 1085curl -sIL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ 1086 https://example.com/old-page | grep -iE "^(HTTP|Location):" 1087``` 1088 1089### 7.7 Troubleshooting 1090 1091**Symptom: Redirect loop (browser shows "ERR_TOO_MANY_REDIRECTS").** 1092Causes: 10931. Two redirects pointing at each other (`/a` → `/b` → `/a`). 10942. Server block redirects HTTP to HTTPS, but the HTTPS server block also has an HTTP to HTTPS redirect, causing infinite loops. 10953. A wildcard redirect catches its own target (`location /new/ { return 301 /new/page; }` redirects `/new/page` to `/new/page`). 1096Fix: trace the chain with curl, identify the loop, rewrite one of the redirects to terminate. 1097 1098**Symptom: Redirect target has trailing slash but original did not (or vice versa) and Google flags duplicates.** 1099Cause: inconsistent trailing slash policy. 1100Fix: pick one policy (typically without trailing slash for non directory URLs) and enforce site wide: 1101 1102```nginx 1103# Force no trailing slash on file like URLs 1104rewrite ^/(.*)/$ /$1 permanent; 1105``` 1106 1107Or the opposite. Document which is the canonical form and stick to it. 1108 1109**Symptom: 301 cached forever in browsers, even after fixing the redirect target.** 1110Cause: browsers cache 301 indefinitely per the HTTP spec. The cached redirect bypasses the server entirely. 1111Fix options: 11121. Change the original URL to a new path that has not been cached as a redirect. 11132. Wait for the browser cache to expire (could be months). 11143. Use a 302 in the future for redirects that might change. 1115 1116**Symptom: POST request becomes GET after redirect.** 1117Cause: the redirect status code is 301, 302, or 303, all of which historically changed the method to GET. 1118Fix: use 307 (temporary) or 308 (permanent) to strictly preserve the method. 1119 1120**Symptom: Redirect works in browser but Googlebot still indexes the original URL.** 1121Causes: 11221. The redirect status is 302 or 307, which signals "do not consolidate". Change to 301 or 308 for permanent moves. 11232. There is a chain. Google may not follow long chains. Collapse to one hop. 11243. The original URL has internal links pointing to it. Update internal links to point directly to the new URL. 11254. Time. Consolidation takes weeks. Be patient. 1126 1127**Symptom: Open redirect security alert.** 1128The redirect endpoint accepts unvalidated user input. 1129Fix: implement an allow list, signature validation, or both. See Section 7.4. 1130 1131### 7.8 How To Fix Common Breakage 1132 1133**Case: Old domain serving 301 to new domain, but new domain still points back to old in some links.** 1134Audit and fix: 1135 1136```bash 1137# Find any link to the old domain in current site 1138grep -r "old-domain.com" /var/www/sites/new-domain.com/ 1139``` 1140 1141Replace every occurrence with the new domain. Then verify no infinite loop exists. 1142 1143**Case: Redirect chain longer than 1 hop discovered in Search Console.** 1144List the chain: 1145 1146```bash 1147curl -sLI https://example.com/old-url | grep -iE "^(HTTP|Location):" 1148``` 1149 1150Update the originating server block to redirect directly to the final destination: 1151 1152```nginx 1153location = /old-url { 1154 # Was: return 301 /intermediate; 1155 return 301 https://example.com/final-destination; 1156} 1157``` 1158 1159`nginx -t && systemctl reload nginx`. 1160 1161**Case: HTTPS redirect causing certificate warning.** 1162The redirect target is `https://example.com` but the SSL certificate covers only `https://www.example.com`. Either: 1163 11641. Get a certificate covering both names (typical with Let's Encrypt: include both in `-d` flags). 11652. Redirect to the name actually covered. 1166 1167**Case: Need to redirect a thousand old URLs to specific new URLs.** 1168Use a map for performance: 1169 1170```nginx 1171# In http context 1172map $request_uri $legacy_redirect { 1173 default ""; 1174 /old/page-1.html /new/page-1; 1175 /old/page-2.html /new/page-2; 1176 /old/page-3.html /new/page-3; 1177 # ... thousands more ... 1178} 1179 1180# In server context 1181if ($legacy_redirect != "") { 1182 return 301 $legacy_redirect; 1183} 1184``` 1185 1186Maps in nginx are O(1) lookups; even thousands of entries add negligible overhead. 1187 1188For maps with thousands of entries, externalize: 1189 1190```nginx 1191map $request_uri $legacy_redirect { 1192 include /etc/nginx/maps/legacy-redirects.map; 1193 default ""; 1194} 1195``` 1196 1197And maintain `legacy-redirects.map` as a simple text file with one mapping per line. 1198 1199**Case: Need to migrate from www to non www, preserving all paths.** 1200 1201```nginx 1202server { 1203 listen 443 ssl; 1204 server_name www.example.com; 1205 return 301 https://example.com$request_uri; 1206} 1207``` 1208 1209Ensure your SSL certificate covers `example.com` (the redirect target). Verify with curl. 1210 1211--- 1212 1213## 8. HOW THESE HEADERS INTERACT 1214 1215The three headers do not act independently. Several combinations matter. 1216 1217### 8.1 X-Robots-Tag On Redirects 1218 1219A 301 or 302 response can carry an `X-Robots-Tag` header, but the directive is largely ignored by Google. The crawler follows the redirect and applies indexing rules to the target URL, not the redirect itself. The exception is `nofollow`, which can prevent the crawler from following the redirect at all, but this is rare and usually unintended. 1220 1221```nginx 1222# Almost always unnecessary 1223location = /old-url { 1224 add_header X-Robots-Tag "noindex" always; 1225 return 301 /new-url; 1226} 1227``` 1228 1229The `X-Robots-Tag: noindex` on a 301 has no effect: the old URL was already being deindexed by the 301 itself. 1230 1231### 8.2 Canonical In HTTP vs HTML 1232 1233When both `Link: <...>; rel=canonical` HTTP header and `<link rel=canonical>` HTML tag are present, Google treats them as duplicate signals. If they agree, fine. If they disagree, Google picks one based on its own canonicalization logic, which may not match either. **Pick one and stick to it.** 1234 1235For HTML pages: use the HTML `<link rel=canonical>` tag because it is easier to manage with templates and is what most tools expect. 1236 1237For non HTML (PDF, image, JSON): use the HTTP `Link` header because there is no HTML to put a tag in. 1238 1239### 8.3 Redirect Plus Canonical Tag On The Target 1240 1241The redirect target should have a self referencing canonical tag: 1242 1243```html 1244<!-- On the new URL: https://example.com/new-page --> 1245<link rel="canonical" href="https://example.com/new-page"> 1246``` 1247 1248This is correct and expected. It reinforces that this is the canonical version, not a duplicate of some other page. 1249 1250### 8.4 hreflang Cluster Cross References 1251 1252Hreflang annotations declared via HTTP `Link` header or HTML `<link rel=alternate hreflang>` must form a reciprocal cluster. Every variant lists every other variant including itself. Missing any reciprocal reference triggers a Search Console warning and may cause hreflang to be ignored entirely. 1253 1254For a PDF that exists in two languages: 1255 1256```nginx 1257# On /en/manual.pdf 1258add_header Link '<https://example.com/en/manual.pdf>; rel="alternate"; hreflang="en-US", <https://example.com/es/manual.pdf>; rel="alternate"; hreflang="es-MX", <https://example.com/en/manual.pdf>; rel="alternate"; hreflang="x-default"' always; 1259 1260# On /es/manual.pdf 1261add_header Link '<https://example.com/en/manual.pdf>; rel="alternate"; hreflang="en-US", <https://example.com/es/manual.pdf>; rel="alternate"; hreflang="es-MX", <https://example.com/en/manual.pdf>; rel="alternate"; hreflang="x-default"' always; 1262``` 1263 1264Both PDFs declare the same cluster. The cluster includes both files and an `x-default`. 1265 1266### 8.5 Preload And Cache-Control 1267 1268Preloaded resources should still have correct `Cache-Control` headers. If the preloaded asset has `Cache-Control: no-store`, the browser fetches it via the preload, but cannot store it for use by the actual page reference. Result: double fetch. 1269 1270Ensure preloaded resources are cacheable. Combine with framework-http-caching-headers.md guidance: fingerprinted assets should be `Cache-Control: public, max-age=31536000, immutable` and are perfect preload candidates. 1271 1272### 8.6 Location Plus Vary 1273 1274A redirect response should typically not have `Vary` headers. The redirect itself does not vary by request headers. However, if the redirect target differs based on `Accept-Language` (geographic or language based redirect), then `Vary: Accept-Language` is required to prevent caches from serving the wrong target to a different audience. 1275 1276```nginx 1277location = / { 1278 # Redirect based on Accept-Language 1279 if ($http_accept_language ~* "^es") { 1280 add_header Vary "Accept-Language" always; 1281 return 302 /es/; 1282 } 1283 add_header Vary "Accept-Language" always; 1284 return 302 /en/; 1285} 1286``` 1287 1288Note: language based redirects are a poor pattern for SEO. They confuse crawlers and prevent users from sharing links between languages. Prefer letting the user select their language explicitly and remembering the choice via cookie or path. 1289 1290--- 1291 1292## 9. ASSET CLASS AND USE CASE RECIPES 1293 1294Each block is paste ready. Copy the ones relevant to your build. 1295 1296### 9.1 Production HTML page (full SEO header set) 1297 1298```nginx 1299location ~* \.html$ { 1300 # X-Robots-Tag: allow indexing, set snippet limits 1301 add_header X-Robots-Tag "max-snippet:200, max-image-preview:large, max-video-preview:-1" always; 1302 1303 # Link: preload critical resources 1304 add_header Link '</css/critical.css>; rel="preload"; as="style", </fonts/manrope.woff2>; rel="preload"; as="font"; crossorigin' always; 1305 1306 # Cache directives from framework-http-caching-headers.md 1307 add_header Cache-Control "public, max-age=0, must-revalidate" always; 1308} 1309``` 1310 1311### 9.2 PDF (block from index, canonical to HTML) 1312 1313```nginx 1314location ~ ^/whitepapers/(.*)\.pdf$ { 1315 add_header X-Robots-Tag "noindex" always; 1316 add_header Link '<https://example.com/whitepapers/$1>; rel="canonical"' always; 1317 add_header Cache-Control "public, max-age=86400" always; 1318 add_header Content-Disposition "attachment" always; 1319} 1320``` 1321 1322### 9.3 Image asset (allow indexing for image search) 1323 1324```nginx 1325location ~* \.(jpg|jpeg|png|webp|avif)$ { 1326 # Default: allow image indexing 1327 # If you want to block: add_header X-Robots-Tag "noindex" always; 1328 add_header Cache-Control "public, max-age=2592000" always; 1329} 1330``` 1331 1332### 9.4 Image asset (block from image search but page itself indexable) 1333 1334```nginx 1335location ~* \.(jpg|jpeg|png|webp|avif)$ { 1336 add_header X-Robots-Tag "noindex" always; 1337 add_header Cache-Control "public, max-age=2592000" always; 1338} 1339``` 1340 1341### 9.5 Staging or development subdomain (full block) 1342 1343```nginx 1344server { 1345 server_name staging.example.com; 1346 1347 # Universal block 1348 add_header X-Robots-Tag "noindex, nofollow" always; 1349 1350 # Plus robots.txt for belt and suspenders 1351 location = /robots.txt { 1352 return 200 "User-agent: *\nDisallow: /\n"; 1353 add_header Content-Type "text/plain"; 1354 } 1355 1356 # Plus HTTP basic auth as a third layer 1357 auth_basic "Staging"; 1358 auth_basic_user_file /etc/nginx/.staging-htpasswd; 1359} 1360``` 1361 1362### 9.6 API endpoint (do not index, do not follow) 1363 1364```nginx 1365location /api/ { 1366 add_header X-Robots-Tag "noindex, nofollow" always; 1367 proxy_pass http://127.0.0.1:9090; 1368} 1369``` 1370 1371### 9.7 Search result page (do not index, do follow) 1372 1373```nginx 1374location /search { 1375 add_header X-Robots-Tag "noindex, follow" always; 1376 # The follow allows crawlers to discover linked content 1377} 1378``` 1379 1380Internal search result pages are duplicate content for each query and should not be indexed. Allow follow so crawlers can discover the linked actual content pages. 1381 1382### 9.8 Paginated section (allow indexing of all pages, no rel prev/next) 1383 1384```nginx 1385location ~ ^/blog/page/([0-9]+)$ { 1386 # Google ignored prev/next as of 2019; do not bother with Link rel=prev/next 1387 add_header X-Robots-Tag "max-snippet:160, max-image-preview:standard" always; 1388} 1389``` 1390 1391### 9.9 Time limited campaign page 1392 1393```nginx 1394location = /promotions/black-friday-2026 { 1395 add_header X-Robots-Tag "unavailable_after: 1 Dec 2026 00:00:00 GMT" always; 1396} 1397``` 1398 1399### 9.10 Permanent www to non www redirect 1400 1401```nginx 1402server { 1403 listen 443 ssl; 1404 listen [::]:443 ssl; 1405 http2 on; 1406 server_name www.example.com; 1407 ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; 1408 ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; 1409 return 301 https://example.com$request_uri; 1410} 1411``` 1412 1413### 9.11 HTTP to HTTPS redirect 1414 1415```nginx 1416server { 1417 listen 80; 1418 listen [::]:80; 1419 server_name example.com www.example.com; 1420 return 301 https://example.com$request_uri; 1421} 1422``` 1423 1424### 9.12 Old blog URL structure to new 1425 1426```nginx 1427# Old: /2024/05/post-title 1428# New: /blog/post-title 1429location ~ ^/(\d{4})/(\d{2})/(.+)$ { 1430 return 301 /blog/$3; 1431} 1432``` 1433 1434### 9.13 Subdomain to client custom domain (Bubbles wildcard pattern) 1435 1436```nginx 1437server { 1438 listen 443 ssl; 1439 server_name handledtax.thatwebhostingguy.com; 1440 ssl_certificate /etc/letsencrypt/live/thatwebhostingguy.com/fullchain.pem; 1441 ssl_certificate_key /etc/letsencrypt/live/thatwebhostingguy.com/privkey.pem; 1442 return 301 https://handledtax.com$request_uri; 1443} 1444``` 1445 1446### 9.14 API version migration (preserve POST method) 1447 1448```nginx 1449location /api/v1/ { 1450 return 308 /api/v2/$request_uri; 1451} 1452``` 1453 1454### 9.15 Maintenance mode toggle 1455 1456```nginx 1457# At server level 1458set $maintenance 0; 1459 1460if (-f /var/www/sites/example.com/.maintenance) { 1461 set $maintenance 1; 1462} 1463 1464location / { 1465 if ($maintenance) { 1466 return 302 /maintenance.html; 1467 } 1468 try_files $uri $uri/ $uri.html =404; 1469} 1470 1471location = /maintenance.html { 1472 # Allow access to this even in maintenance mode 1473 internal; 1474} 1475``` 1476 1477To enable maintenance: `touch /var/www/sites/example.com/.maintenance`. To disable: `rm /var/www/sites/example.com/.maintenance`. No nginx reload required. 1478 1479--- 1480 1481## 10. BUBBLES NGINX REFERENCE BLOCK (PASTE READY) 1482 1483The complete SEO header stanza, layered with caching (framework-http-caching-headers.md) and content (framework-http-content-headers.md). 1484 1485```nginx 1486# /etc/nginx/sites-available/example.com 1487 1488# HTTP to HTTPS redirect 1489server { 1490 listen 80; 1491 listen [::]:80; 1492 server_name example.com www.example.com; 1493 return 301 https://example.com$request_uri; 1494} 1495 1496# www to non www 1497server { 1498 listen 443 ssl; 1499 listen [::]:443 ssl; 1500 http2 on;