A continuation of Janady13/crawler-signal-reference / framework-html-meta-robots.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓1,749 lines · 66.0 KB
# URLs expected to be noindex
EXPECTED_NOINDEX=(
"/search?q=test"
"/login"
"/admin/"
"/account/profile"
)
for path in "${EXPECTED_NOINDEX[@]}"; do
NOINDEX_COUNT=$(curl -s "https://example.com$path" 2>/dev/null | grep -ic 'content="noindex')
if [ "$NOINDEX_COUNT" = "0" ]; then
echo "MISSING NOINDEX: $path"
fi
done
```
### Cross check HTTP header AND meta tag alignment
```bash
URL="https://example.com/admin/"
# HTTP header
HEADER=$(curl -sI "$URL" | grep -i "x-robots-tag" | tr -d '\r')
# HTML meta
META=$(curl -s "$URL" | grep -oE 'meta name="robots"[^>]+' | head -1)
echo "URL: $URL"
echo "HTTP: $HEADER"
echo "META: $META"
# If both should say noindex, both should be present
```
### Diff meta robots between two pages
```bash
URL1="https://example.com/services/cassville/" # expected indexable
URL2="https://example.com/services/detroit/" # expected noindex
diff \
<(curl -s "$URL1" | grep -oE 'meta name="robots"[^>]+') \
<(curl -s "$URL2" | grep -oE 'meta name="robots"[^>]+')
```
### Find pages in GSC with unexpected status
In Google Search Console:
1. Page Indexing report to "Indexed, though blocked by robots.txt".
2. Page Indexing report to "Excluded by 'noindex' tag".
3. Compare counts to expected (matches your noindex_follow tier).
Manual but critical step; cannot be automated via standard CLI.
### Browser DevTools verification
In Chrome DevTools:
1. Open the page.
2. View Source (Cmd/Ctrl + U) to see initial HTML.
3. Find `<meta name="robots">` in the source.
4. Inspect Element (Cmd/Ctrl + Shift + I) to Network tab to check headers for `X-Robots-Tag`.
5. Console: `document.head.querySelector('meta[name="robots"]')?.content` shows the current state.
### Bulk audit script
```bash
#!/bin/bash
# /usr/local/bin/audit-meta-robots.sh
# Run weekly to verify meta robots state across critical URLs
BASE=https://example.com
# Pages that SHOULD be indexed
INDEXABLE=(
"/"
"/about"
"/services"
"/blog"
"/contact"
)
# Pages that SHOULD have noindex
NOINDEX=(
"/login"
"/search?q=test"
"/admin"
)
echo "=== Indexable pages ==="
for path in "${INDEXABLE[@]}"; do
NOINDEX_COUNT=$(curl -s "$BASE$path" 2>/dev/null | grep -ic 'content="noindex')
if [ "$NOINDEX_COUNT" != "0" ]; then
echo "PROBLEM: $path has noindex but should not"
else
echo "OK: $path"
fi
done
echo ""
echo "=== Should-be-noindexed pages ==="
for path in "${NOINDEX[@]}"; do
NOINDEX_COUNT=$(curl -s "$BASE$path" 2>/dev/null | grep -ic 'content="noindex')
if [ "$NOINDEX_COUNT" = "0" ]; then
echo "PROBLEM: $path missing noindex"
else
echo "OK: $path"
fi
done
```
### After template changes
```bash
# Clear any cached templates
systemctl restart fastapi-sidecar
# Verify the change is live
curl -s https://example.com/test-page | grep -oE 'meta name="robots"[^>]+'
# Then GSC URL Inspection for the changed pages to accelerate recrawl
```
---
## 20. CROSS-REFERENCES
* [framework-http-seo-headers.md](framework-http-seo-headers.md): the X-Robots-Tag HTTP header is the equivalent for non HTML resources and path level policy at the nginx layer. The directives are the same; the delivery mechanism differs.
* [framework-http-request-headers.md](framework-http-request-headers.md) Section 5: the AI crawler taxonomy (current UA strings for ClaudeBot, GPTBot, OAI-SearchBot, etc).
* [framework-http-2xx-status-codes.md](framework-http-2xx-status-codes.md) Section 10: the soft 404 trap; pages with `noindex` plus thin content are different from `200 with thin content` (which is the soft 404).
* [framework-http-3xx-status-codes.md](framework-http-3xx-status-codes.md) Section 16: the Bubbles 14 paying client subdomain redirect pattern (the 301 layer; this framework covers the noindex meta tier).
* [framework-http-4xx-status-codes.md](framework-http-4xx-status-codes.md) Section 17: the Bubbles thin content cleanup HTTP side (410 Gone for bucket 3); this framework covers the HTML side (noindex for bucket 2).
* [framework-http-caching-headers.md](framework-http-caching-headers.md): meta robots tags are content; cache headers control how the response containing them is cached.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Meta robots is one of the levers for controlling what gets indexed.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook including meta robots placement in templates.
* Google robots meta tag documentation: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
* Google indexifembedded blog post: https://developers.google.com/search/blog/2022/01/robots-meta-tag-indexifembedded
* Google data-nosnippet documentation: https://developers.google.com/search/docs/appearance/snippet
* Bing robots meta tag documentation: https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240
* MDN meta name="robots": https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Meta robots directive decision matrix
```
What is this page?
|
|---> Substantive editorial content ............ (no tag needed; default is index, follow)
| Optional: max-snippet:-1, max-image-preview:large
|
|---> Search results, filters, pagination 2+ ... noindex, follow
|
|---> Login, account, registration .............. noindex, follow
|
|---> Admin, internal tools ..................... noindex, nofollow
|
|---> Thank you, confirmation ................... noindex, nofollow
|
|---> Staging, demo, development ................ noindex, nofollow + robots.txt block
|
|---> Time limited (event, sale) ................ unavailable_after: DATE
|
|---> Want searchable but NOT in AI Overviews ... nosnippet
|
|---> Podcast or video host page ................ noindex, indexifembedded (Google only)
|
|---> Print version ............................. noindex + canonical to main version
|
|---> Page with proprietary images .............. noimageindex
```
### Tag scope
| Name | Applies to |
|---|---|
| `robots` | All crawlers (default) |
| `googlebot` | Googlebot (overrides robots for Google) |
| `bingbot` | Bingbot (overrides robots for Bing) |
| `googlebot-news` | Google News specifically |
### Five rules to memorize
1. **Default is index, follow.** Do not emit it; it is noise.
2. **noindex requires crawlability.** Do not combine with robots.txt Disallow.
3. **Most restrictive wins.** When directives conflict.
4. **Server side render the tag.** Client side JS modifications may not be seen.
5. **Sitemap should never list noindex URLs.** They should not be there.
### Five commands every operator should know
```bash
# 1. Check meta robots for a URL
curl -s https://example.com/page | grep -oE 'meta name="robots"[^>]+'
# 2. Find unexpected noindex in sitemap
curl -s https://example.com/sitemap.xml | grep -oE "<loc>[^<]+</loc>" | sed 's/<[^>]*>//g' | head -20 | while read url; do
NOINDEX=$(curl -s "$url" | grep -ic 'content="noindex')
[ "$NOINDEX" != "0" ] && echo "PROBLEM: $url"
done
# 3. Verify expected noindex pages
for p in /login /admin/ /search; do
HAS=$(curl -s "https://example.com$p" | grep -ic 'noindex')
[ "$HAS" = "0" ] && echo "MISSING NOINDEX: $p" || echo "OK: $p"
done
# 4. Cross check HTTP header AND meta tag
URL=https://example.com/admin/
curl -sI "$URL" | grep -i x-robots-tag
curl -s "$URL" | grep -oE 'meta name="robots"[^>]+'
# 5. After template change
systemctl restart fastapi-sidecar
curl -s https://example.com/changed-page | grep -oE 'meta name="robots"[^>]+'
```
### Three end to end tests
```bash
# 1. Homepage is indexable
HAS_NOINDEX=$(curl -s https://example.com/ | grep -ic 'content="noindex')
[ "$HAS_NOINDEX" = "0" ] && echo "OK: homepage indexable" || echo "FAIL: homepage has noindex!"
# 2. Admin pages have noindex
HAS_NOINDEX=$(curl -s https://example.com/admin/ | grep -ic 'content="noindex')
[ "$HAS_NOINDEX" != "0" ] && echo "OK: admin noindexed" || echo "FAIL: admin missing noindex"
# 3. Sitemap URLs are all indexable
PROBLEMS=$(curl -s https://example.com/sitemap.xml | grep -oE "<loc>[^<]+</loc>" | sed 's/<[^>]*>//g' | head -10 | while read url; do
HAS=$(curl -s "$url" | grep -ic 'content="noindex')
[ "$HAS" != "0" ] && echo "$url"
done | wc -l)
[ "$PROBLEMS" = "0" ] && echo "OK: no noindex URLs in sitemap" || echo "FAIL: $PROBLEMS sitemap URLs have noindex"
```
If all three pass AND GSC Page Indexing report shows expected "Excluded by noindex" count matching your intent, the meta robots layer is correctly wired.
---
End of framework-html-meta-robots.md.
End of the first framework in the HTML signal track beyond the HTTP wire layer.