A continuation of Janady13/crawler-signal-reference / framework-http-2xx-status-codes.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓1,913 lines · 73.6 KB
6. [ ] No URLs return 200 with empty body for unauthenticated requests (use 401 instead).
7. [ ] Login pages and account pages have X-Robots-Tag: noindex if reachable.
8. [ ] No login walls hidden behind 200 with JavaScript redirect.
### 201 Created
9. [ ] All POST endpoints that create resources return 201 (not 200).
10. [ ] Every 201 response includes a Location header pointing to the new resource.
11. [ ] The Location URL actually exists and returns 200 when fetched.
12. [ ] Response body is the representation of the new resource (or empty if minimal).
13. [ ] API endpoints noindex via X-Robots-Tag (covered in /api/ location).
### 202 Accepted
14. [ ] Long running operations return 202 (not blocking 200).
15. [ ] Every 202 response includes Location pointing to status URL.
16. [ ] Status URL responds with current state (queued, processing, complete, failed).
17. [ ] Status URL also returns final result location when complete.
18. [ ] Job queue is durable (Redis, RQ, Celery, not just BackgroundTasks).
### 204 No Content
19. [ ] DELETE endpoints return 204 (not 200 with empty body).
20. [ ] OPTIONS preflight returns 204.
21. [ ] Beacon endpoints (/beacon, /rum, /csp-report) return 204.
22. [ ] 204 responses have empty body (no JSON, no HTML).
23. [ ] No 204 responses on URLs that crawlers might index.
24. [ ] 204 responses on mutating endpoints include Cache-Control: no-store.
### 206 Partial Content
25. [ ] Video files served from static location (nginx auto Range support).
26. [ ] Audio files served from static location (same).
27. [ ] Large download files have Accept-Ranges: bytes header.
28. [ ] curl -H "Range: bytes=0-1023" returns 206 on streamable assets.
29. [ ] Invalid ranges return 416 Range Not Satisfiable.
30. [ ] Content-Range header present and correct on 206 responses.
31. [ ] HTML5 video player seek works without rebuffering from start.
### Soft 404 prevention
32. [ ] Thin pages identified via GSC Page Indexing > Soft 404 list.
33. [ ] Thin pages classified: improve, noindex, or 410.
34. [ ] Noindex tier configured in nginx for low value but accessible URLs.
35. [ ] 410 tier configured for permanently removed URLs.
36. [ ] Dynamic search results pages noindex by default.
37. [ ] Filter and sort parameter URLs noindex (or canonical to base URL).
38. [ ] Empty category pages return 404 or 410, not 200 with "no items".
### Crawler interaction
39. [ ] Bubbles crawler analytics distinguish 200, 204, etc.
40. [ ] Daily report includes any 204 traffic from crawlers (should be near zero).
41. [ ] Any 5xx in crawler logs investigated (covered in framework-http-5xx-status-codes.md when built).
### Cross cutting
42. [ ] nginx -t passes without warnings.
43. [ ] nginx -T | grep "return " shows expected status codes.
44. [ ] curl tests against major URL patterns return expected status codes.
45. [ ] FastAPI sidecar status_code parameter set explicitly where appropriate (not defaulting to 200).
46. [ ] No accidental fall through to 200 in error paths.
47. [ ] Error handlers in FastAPI return correct status (4xx or 5xx, not 200 with error body).
48. [ ] Healthcheck endpoint returns 200 with body for monitoring tools.
49. [ ] Logging captures status code (already in standard log format).
50. [ ] Monitoring alerts on unexpected 2xx mix (sudden spike in 204 or 206 on crawled URLs).
A site that passes all 50 has correctly configured 2xx status code usage for SEO, API conventions, and streaming.
---
## 16. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: 200 OK for "no results found" page.**
Symptom: empty search results pages flagged as soft 404 in GSC.
Why it breaks: 200 with thin content = soft 404 by Google's classifier.
Fix: noindex empty results, OR show related categories/popular items as substantive content, OR return 404.
**Pitfall 2: 204 returned for normal page URL.**
Symptom: previously indexed pages drop from index after change to 204.
Why it breaks: Google explicitly treats 204 on a URL as "no content to index".
Fix: only use 204 on API endpoints, beacons, OPTIONS preflight. Never on navigable pages.
**Pitfall 3: 201 without Location header.**
Symptom: API client cannot find the resource just created.
Why it breaks: REST convention violation; client expects Location.
Fix: always set `Location: /api/resource/<id>` on 201 responses.
**Pitfall 4: 200 returned with `{"error": "..."}` body.**
Symptom: API client treats failed request as success.
Why it breaks: status code is the canonical signal; body is metadata.
Fix: return 4xx or 5xx for errors, not 200 with error body.
**Pitfall 5: 200 returned for "coming soon" placeholder page.**
Symptom: Google indexes the placeholder, ranks it generically.
Why it breaks: 200 with thin content = soft 404 risk or low quality signal.
Fix: noindex the placeholder, or return 503 with Retry-After (more honest).
**Pitfall 6: DELETE returns 200 with deleted object body.**
Symptom: API client SDK fails to parse the response, expecting void.
Why it breaks: REST convention says DELETE returns 204 (or 200 with operation result, but never the deleted object).
Fix: return 204 (no body) for successful DELETE.
**Pitfall 7: 204 with body sneaked in.**
Symptom: some clients fail to parse subsequent responses; pipeline desync.
Why it breaks: RFC says 204 MUST have empty body.
Fix: ensure server returns no body; use Response(status_code=204) in FastAPI.
**Pitfall 8: video player rebuffers from start every seek.**
Symptom: poor video UX, high bandwidth costs.
Why it breaks: server returning 200 with full file instead of 206 for Range.
Fix: serve video from static nginx location (auto Range support), or implement Range in upstream.
**Pitfall 9: 200 returned for client side rendered SPA shell.**
Symptom: SPAs flagged as soft 404 across Page Indexing report.
Why it breaks: Googlebot sees `<div id="app"></div>` and nothing else.
Fix: server side render or static generate for indexable pages. SPA pattern stays for app only routes.
**Pitfall 10: 202 without status URL.**
Symptom: client has no way to know when async operation completes.
Why it breaks: client must poll something; without Location they have nowhere to look.
Fix: include Location pointing to status endpoint.
**Pitfall 11: API endpoint returning 200 to crawlers (no noindex).**
Symptom: API endpoints (with JSON bodies) showing in search results.
Why it breaks: 200 plus indexable URL plus no noindex = indexable.
Fix: add `X-Robots-Tag: noindex, nofollow` to /api/ location.
**Pitfall 12: 206 returned without Content-Range header.**
Symptom: clients misinterpret the range delivered.
Why it breaks: spec requires Content-Range on 206.
Fix: always include Content-Range with start, end, and total size.
**Pitfall 13: 200 returned for unauthenticated request to private URL.**
Symptom: empty or minimal body indexed as soft 404; users redirected via JS.
Why it breaks: should be 401 (or 403) with appropriate body.
Fix: return 401 with WWW-Authenticate header for unauthenticated requests.
**Pitfall 14: Range request returns 200 with full file.**
Symptom: streaming and resumable downloads do not work; bandwidth wasted.
Why it breaks: server does not parse Range header; returns full content.
Fix: for static files use nginx (handles Range automatically); for upstream implement Range handling.
**Pitfall 15: 200 with empty JSON body returned where 204 should be used.**
Symptom: API client unnecessarily processes empty object.
Why it breaks: 200 implies meaningful body; 204 means body intentionally absent.
Fix: use 204 for operations that have no meaningful return value.
---
## 17. DIAGNOSTIC COMMANDS
Reference of every command useful for 2xx status code investigation.
### Inspect status codes across a site
```bash
# Status code summary for top URLs
SITE=https://example.com
for path in / /about /services /blog /contact /api/health; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "$SITE$path")
SIZE=$(curl -so /dev/null -w "%{size_download}" "$SITE$path")
echo "$path: $STATUS ($SIZE bytes)"
done
```
### Detect thin content (soft 404 risk)
```bash
# Scan sitemap for thin pages
curl -s https://example.com/sitemap.xml | \
grep -oE "<loc>[^<]+</loc>" | sed 's/<[^>]*>//g' | \
while read url; do
SIZE=$(curl -so /dev/null -w "%{size_download}" "$url")
if [ "$SIZE" -lt 5000 ]; then
echo "THIN: $url ($SIZE bytes)"
fi
done
# Count words in a specific page
URL=https://example.com/some-page
WORDS=$(curl -s "$URL" | sed 's/<[^>]*>//g' | wc -w)
echo "$URL: $WORDS words"
# Under 300 words is soft 404 risk
```
### Verify 201 with Location
```bash
RESPONSE=$(curl -sI -X POST https://api.example.com/items \
-H "Content-Type: application/json" \
-d '{"name": "test"}')
STATUS=$(echo "$RESPONSE" | head -1 | awk '{print $2}')
LOCATION=$(echo "$RESPONSE" | grep -i "^location:" | awk '{print $2}' | tr -d '\r')
echo "Status: $STATUS"
echo "Location: $LOCATION"
# Verify Location resource exists
if [ -n "$LOCATION" ]; then
FOLLOWUP=$(curl -so /dev/null -w "%{http_code}" "https://api.example.com$LOCATION")
echo "Location returns: $FOLLOWUP"
fi
```
### Verify 204 has empty body
```bash
# DELETE should return 204 with no body
RESPONSE=$(curl -sI -X DELETE https://api.example.com/users/test)
STATUS=$(echo "$RESPONSE" | head -1 | awk '{print $2}')
CONTENT_LENGTH=$(echo "$RESPONSE" | grep -i "^content-length:" | awk '{print $2}' | tr -d '\r')
echo "Status: $STATUS"
echo "Content-Length: ${CONTENT_LENGTH:-(not set)}"
echo "Expected: 204 with content-length 0 or absent"
# Also test that no body comes back
BODY_SIZE=$(curl -s -X DELETE https://api.example.com/users/test -o /dev/null -w "%{size_download}")
echo "Body size: $BODY_SIZE bytes"
echo "Expected: 0"
```
### Verify 206 Range support
```bash
# Test that Range requests get 206
URL=https://example.com/video.mp4
echo "=== Full request ==="
curl -sI "$URL" | head -3
# Expected: HTTP/2 200, accept-ranges: bytes
echo "=== Range request (first 1KB) ==="
curl -sI -H "Range: bytes=0-1023" "$URL" | head -4
# Expected: HTTP/2 206, content-range: bytes 0-1023/<total>
echo "=== Range request (last 1KB) ==="
curl -sI -H "Range: bytes=-1024" "$URL" | head -4
echo "=== Invalid range ==="
curl -sI -H "Range: bytes=999999999-" "$URL" | head -3
# Expected: HTTP/2 416
# Verify partial bytes downloaded
BYTES=$(curl -s -H "Range: bytes=0-99" "$URL" | wc -c)
echo "Bytes downloaded: $BYTES (expected 100)"
```
### Audit for soft 404 patterns
```bash
# Find URLs returning 200 but containing common soft 404 phrases
URL=https://example.com
SUSPICIOUS_PHRASES=("no results found" "out of stock" "page not found" "coming soon" "under construction")
for phrase in "${SUSPICIOUS_PHRASES[@]}"; do
echo "=== Checking for '$phrase' ==="
# Sample 20 URLs from sitemap
curl -s "$URL/sitemap.xml" | grep -oE "<loc>[^<]+</loc>" | head -20 | \
sed 's/<[^>]*>//g' | while read u; do
if curl -s "$u" | grep -qi "$phrase"; then
STATUS=$(curl -so /dev/null -w "%{http_code}" "$u")
if [ "$STATUS" = "200" ]; then
echo "SOFT 404 RISK: $u contains '$phrase' (status 200)"
fi
fi
done
done
```
### Status code distribution from logs
```bash
# Top status codes in the access log today
sudo awk -v date="$(date '+%d/%b/%Y')" '$0 ~ date {print $9}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -10
# 2xx breakdown
sudo grep -oE '" [0-9]{3} ' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -15
# Soft 404 candidates: 200 with low response size
sudo awk '$9 == 200 && $10 < 5000 {print $7, $10}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -20
```
### Server side investigation
```bash
# Find explicit return statements in nginx config
nginx -T 2>/dev/null | grep -E "return [0-9]+"
# Find status_code in FastAPI source
grep -rn "status_code" /opt/bubbles/api/
# Check what GSC reports for the domain
# (Manual: visit Google Search Console > Page Indexing > Soft 404)
# Apply changes
nginx -t && systemctl reload nginx
```
---
## 18. CROSS-REFERENCES
* [framework-http-caching-headers.md](framework-http-caching-headers.md): `Last-Modified` and `ETag` pair with conditional GET requests; the 304 Not Modified response (in framework-http-3xx-status-codes.md when built) is the conditional success.
* [framework-http-content-headers.md](framework-http-content-headers.md): Content-Type for 200/201/202 responses; Content-Range for 206; Content-Length 0 (or absent) for 204.
* [framework-http-seo-headers.md](framework-http-seo-headers.md): `X-Robots-Tag` controls indexability of 200 responses; 410 Gone (covered in 4xx framework) is the SEO friendly removal.
* [framework-http-security-headers.md](framework-http-security-headers.md): security headers apply to all 2xx responses; CSP report endpoint returns 204.
* [framework-http-performance-headers.md](framework-http-performance-headers.md): Server-Timing on 2xx responses for backend observability.
* [framework-http-cors-headers.md](framework-http-cors-headers.md): OPTIONS preflight returns 204 with CORS headers.
* [framework-http-rate-control-headers.md](framework-http-rate-control-headers.md): 429 and 503 (covered there) are the 4xx and 5xx companions to 2xx success; Retry-After applies to those.
* [framework-http-request-headers.md](framework-http-request-headers.md): `If-Modified-Since` and `If-None-Match` request headers determine when 200 becomes 304; `Range` request header determines when 200 becomes 206.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Soft 404s degrade overall site quality signals.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook including thin content prevention.
* [framework-http-3xx-status-codes.md] (next in series): 301, 302, 303, 304, 307, 308 redirects.
* [framework-http-4xx-status-codes.md] (later in series): 400, 401, 403, 404, 410, 429.
* [framework-http-5xx-status-codes.md] (later in series): 500, 502, 503, 504.
* Google HTTP status codes documentation: https://developers.google.com/search/docs/crawling-indexing/http-network-errors
* MDN 200 OK: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/200
* MDN 201 Created: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/201
* MDN 202 Accepted: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/202
* MDN 204 No Content: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/204
* MDN 206 Partial Content: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/206
* RFC 9110 (HTTP Semantics): https://www.rfc-editor.org/rfc/rfc9110
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Status code decision tree
```
Operation succeeded. What now?
|
|---> Returning a substantive resource (page, JSON data) ........... 200 OK
|
|---> Just created a new resource (POST /api/users) ............... 201 Created (+ Location)
|
|---> Operation queued, will complete later (long running job) ..... 202 Accepted (+ Location for status)
|
|---> Operation succeeded, no body to return (DELETE, OPTIONS) ..... 204 No Content (empty body)
|
|---> Range request fulfilled (video Range, resumable download) .... 206 Partial Content (+ Content-Range)
```
### Indexability quick reference
| Code | Indexable? | Use for crawled URLs? |
|---|---|---|
| 200 | YES (if substantive) | YES |
| 201 | Waits, may soft 404 | NO (creation is POST only) |
| 202 | Waits, may soft 404 | NO (async is POST only) |
| 204 | NO (soft 404) | NEVER |
| 206 | NO (partial) | NEVER (media only) |
### Five rules to memorize
1. 200 needs substantive content. Thin = soft 404.
2. 201 needs Location header. Always.
3. 204 needs empty body. Always.
4. 206 needs Content-Range. Always.
5. API endpoints noindex regardless of status code.
### Five commands every operator should know
```bash
# 1. Check status of a URL
curl -sI https://example.com/page | head -1
# 2. Check response size (thin = soft 404 risk)
curl -s -o /dev/null -w "%{size_download} bytes\n" https://example.com/page
# 3. Test Range support
curl -sI -H "Range: bytes=0-1023" https://example.com/video.mp4 | head -3
# 4. Find soft 404 candidates in logs
sudo awk '$9 == 200 && $10 < 5000 {print $7, $10}' /var/log/nginx/access.log | sort -u | head -20
# 5. Apply nginx changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. Substantive page returns 200 with real content
SIZE=$(curl -s -o /dev/null -w "%{size_download}" https://example.com/)
[ $SIZE -gt 5000 ] && echo "OK: substantive content" || echo "FAIL: thin"
# 2. DELETE returns 204 with empty body
RESPONSE=$(curl -sI -X DELETE https://api.example.com/test-resource/123)
echo "$RESPONSE" | head -1
echo "$RESPONSE" | grep -i content-length
# Expected: HTTP/2 204; content-length: 0 (or absent)
# 3. Video supports Range (206)
curl -sI -H "Range: bytes=0-1023" https://example.com/videos/test.mp4 | head -3
# Expected: HTTP/2 206; content-range: bytes 0-1023/...
```
If all three pass AND GSC Page Indexing > Soft 404 is empty or stable, the 2xx layer is correctly wired.
---
End of framework-http-2xx-status-codes.md.