A continuation of Janady13/crawler-signal-reference / framework-http-4xx-status-codes.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓2,619 lines · 89.1 KB
Credential-bearing and unrelated personal details were removed from this reading copy. The source identity and original checksum remain recorded.
```nginx
location / {
if ($is_gone) {
return 410;
}
try_files $uri $uri/ $uri.html =404;
}
```
**Step 4: Deploy and monitor.**
```bash
nginx -t && systemctl reload nginx
# Verify a few URLs return 410
for url in $(head -5 /tmp/urls-to-remove.txt); do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://example.com$url")
echo "$url: $STATUS"
done
# Monitor GSC Coverage report weekly
# "Not found (410)" count should drop steadily
```
### 15.5 The "Stuck" 404 Problem
If you have URLs that have been 404 for months but are still in the index:
1. **Confirm they actually return 404 (or 410)** via direct curl.
2. **Submit URL removal in GSC** for the most critical ones (URL Removals tool).
3. **Convert to 410** for faster removal:
```nginx
# Was returning 404 by default; convert to explicit 410
location ^~ /old-section/ {
return 410;
}
```
4. **Resubmit the sitemap** without those URLs.
5. **Wait 2 to 4 weeks** for Google to recrawl and process the change.
### 15.6 The Custom 410 Page
For user experience, provide a custom 410 page:
```nginx
error_page 410 /410.html;
location = /410.html {
root /var/www/sites/example.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
```
```html
<!-- /var/www/sites/example.com/410.html -->
<!doctype html>
<html lang="en">
<head>
<title>Content Removed</title>
</head>
<body>
<h1>This content has been removed</h1>
<p>The page you were looking for has been permanently removed and will not return.</p>
<p><a href="/">Return to homepage</a> or <a href="/blog/">browse our latest articles</a>.</p>
</body>
</html>
```
Note: substantive enough that if accidentally indexed, it is not a soft 404.
---
## 16. THE 400 VS 422 DISTINCTION (VALIDATION SEMANTICS)
### 16.1 The Cleanest Split
* **400**: the request is malformed. Fix the request structure (JSON syntax, missing headers, wrong content type).
* **422**: the request is well formed. Fix the data values.
### 16.2 Decision Examples
| Scenario | Code |
|---|---|
| JSON syntax error in body | 400 |
| Missing `Content-Type` header | 400 |
| Required field absent | 422 (FastAPI default) |
| Field present but wrong type | 422 (FastAPI default) |
| Email field present and well formed, but business rejects | 422 |
| Foreign key references non existent resource | 422 |
| Date in past when business requires future | 422 |
| HTTP method wrong | 405 |
| Body too large | 413 |
| Body Content-Type not supported | 415 |
### 16.3 The FastAPI Default
FastAPI's request validation (Pydantic based) defaults to 422 for any validation error including missing fields and type mismatches. This is somewhat more permissive than the strict RFC reading (which might suggest 400 for missing required fields).
For consistency with FastAPI conventions: use 422 for all schema validation failures. Use 400 only when the request format itself is broken (JSON parse errors).
### 16.4 The Error Body Convention
422 responses typically include detailed error breakdown:
```json
{
"detail": [
{
"type": "string_too_short",
"loc": ["body", "password"],
"msg": "String should have at least 8 characters",
"input": "abc"
}
]
}
```
400 responses are often less structured:
```json
{
"error": "invalid_request",
"detail": "JSON parse error: expected ',' or '}'"
}
```
For API clients, the difference matters: 422 detail can be displayed inline next to specific form fields; 400 typically requires a generic error message.
---
## 17. THE BUBBLES THIN CONTENT REMOVAL PATTERN (JOSEPH'S 5,800 PAGE CLEANUP)
### 17.1 The Background
The thatdeveloperguy.com infrastructure overhaul included a 5,800+ thin location page cleanup. The pages were dynamically generated `/services/<city>/` templates with minimal unique content per city. Google was soft 404 flagging them and dragging down the overall site quality signal.
### 17.2 The Cleanup Approach
The pages fell into three buckets:
1. **High value cities** (Northwest Arkansas and Southwest Missouri core market): kept, content beefed up to substantive per page.
2. **Adjacent market cities** (still relevant but not core): X-Robots-Tag: noindex, follow. Users could still reach them; Google would stop indexing.
3. **Far flung cities** (irrelevant to ThatDeveloperGuy's actual service area): 410 Gone. Hard removal.
### 17.3 The Nginx Implementation
```nginx
# /etc/nginx/snippets/services-cleanup.conf
# Bucket 3: cities to 410 Gone
map $request_uri $service_410 {
default 0;
~*^/services/(detroit|chicago|denver|portland|seattle)/ 1;
# ... thousands of city slugs ...
~*^/services/(other-far-flung-cities)/ 1;
}
# Bucket 2: cities to noindex but keep accessible
map $request_uri $service_noindex {
default 0;
~*^/services/(little-rock|fayetteville-edge|tulsa)/ 1;
# ... adjacent market cities ...
}
# Bucket 1: core cities (no map needed, default serve)
```
In the server block:
```nginx
server {
# ...
location ~* ^/services/ {
if ($service_410) {
return 410;
}
if ($service_noindex) {
add_header X-Robots-Tag "noindex, follow" always;
}
try_files $uri $uri/ $uri.html =404;
}
# Generic 410 page
error_page 410 /410.html;
location = /410.html {
root /var/www/sites/thatdeveloperguy.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
}
```
### 17.4 The Verification Pattern
```bash
# Verify 410s are returning correctly
for url in /services/detroit /services/chicago /services/portland; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://thatdeveloperguy.com$url")
echo "$url: $STATUS"
done
# Expected: 410 for each
# Verify noindex tier
curl -sI https://thatdeveloperguy.com/services/little-rock/ | grep -i x-robots-tag
# Expected: x-robots-tag: noindex, follow
# Verify kept pages serve normally
curl -sI https://thatdeveloperguy.com/services/cassville/ | head -1
# Expected: HTTP/2 200
```
### 17.5 The Timeline Of Recovery
The 5,800 page cleanup followed approximately this timeline:
* **Week 1**: deployment of 410s and noindex. Googlebot starts encountering the new responses.
* **Week 2 to 4**: 410 URLs drop from the index rapidly (within days of the second consecutive 410). GSC "Not found (410)" count climbs initially then drops.
* **Month 2**: noindex URLs gradually deindex. The "noindex" count grows as URLs are reprocessed.
* **Month 3**: most URLs in both removal tiers are out of the index. Site quality signal improves.
* **Month 4 to 6**: rankings on the kept (substantive) pages improve as the site quality signal stabilizes.
This pattern is now documented as the canonical Bubbles approach for any client experiencing thin content soft 404 issues.
### 17.6 The Repeatable Audit Workflow
For any new Bubbles client showing soft 404 issues:
1. **Export GSC Coverage report.** Filter to "Soft 404" and "Not indexed".
2. **Classify each URL.** Keep + improve, noindex, or 410?
3. **Build the nginx map config.** Replicate the pattern from Section 17.3.
4. **Deploy and reload.** `nginx -t && systemctl reload nginx`.
5. **Submit URL removal in GSC** for the most critical 410s (URL Removals tool).
6. **Update sitemap** to exclude removed URLs.
7. **Monitor weekly.** GSC Coverage report; "Not found (410)" should rise then fall.
---
## 18. HOW MAJOR CRAWLERS REACT TO EACH 4XX CODE
Quick reference table.
| Code | Googlebot | Bingbot | ClaudeBot | GPTBot | PerplexityBot |
|---|---|---|---|---|---|
| 400 | Reduces crawl frequency over time | Treats as error | Skip | Skip | Skip |
| 401 | Access denied, reduces crawl, may deindex | Similar | Skip | Skip | Skip |
| 403 | Access denied, **backs off**, may deindex | Similar | Skip | Skip | Skip |
| 404 | Retries on slowing schedule (24h, 7d, 30d, 90d), drops slowly | Retries less | Caches negative result | Caches negative | Caches negative |
| 410 | Drops in days, retry frequency drops 92% | Treats like 404 | Caches negative result | Caches negative | Caches negative |
| 422 | Like 400 | Like 400 | Skip | Skip | Skip |
| 429 | Reduces crawl rate, honors Retry-After, **2 day rule** | Reduces crawl rate | Honors Retry-After | Honors Retry-After | Honors Retry-After |
| 451 | Treats as removal, may show "removed for legal complaint" in GSC | Similar | Removes from cache | Removes from cache | Removes from cache |
**Key implications for Bubbles SEO:**
* For bulk removal: **410** is the strongest signal (days vs months).
* For access denial of public URLs: **403** is the back off signal.
* For login walls: **401** (with WWW-Authenticate) tells crawlers "not for you".
* For rate limiting: **429** with Retry-After AND crawler whitelist (per framework-http-rate-control-headers.md).
* For legal removal: **451** is the spec compliant signal.
---
## 19. ASSET CLASS AND USE CASE RECIPES
Paste ready blocks per scenario.
### 19.1 Default 404 with custom page
```nginx
server {
root /var/www/sites/example.com;
location / {
try_files $uri $uri/ $uri.html =404;
}
error_page 404 /404.html;
location = /404.html {
root /var/www/sites/example.com;
internal;
}
}
```
### 19.2 Bulk 410 for content cleanup
```nginx
# In http context
map $request_uri $is_gone {
default 0;
~*^/old-section/ 1;
~*^/removed-blog/ 1;
/specific-removed-page-1 1;
/specific-removed-page-2 1;
}
# In server block
location / {
if ($is_gone) {
return 410;
}
try_files $uri $uri/ $uri.html =404;
}
error_page 410 /410.html;
location = /410.html {
root /var/www/sites/example.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
```
### 19.3 Admin endpoint with 401 then 403
```python
from fastapi import Depends
async def get_current_user(request: Request):
session = request.cookies.get("session")
if not session:
raise HTTPException(
status_code=401,
detail="auth required",
headers={"WWW-Authenticate": 'Cookie realm="bubbles"'}
)
user = await load_session(session)
if not user:
raise HTTPException(
status_code=401,
detail="invalid session",
headers={"WWW-Authenticate": 'Cookie realm="bubbles"'}
)
return user
@app.get("/api/admin/users")
async def admin_users(user = Depends(get_current_user)):
if not user.is_admin:
raise HTTPException(status_code=403, detail="forbidden")
return await list_all_users()
```
### 19.4 IP based access control with 403
```nginx
location /admin/ {
allow 100.90.0.0/16; # Tailscale
allow 127.0.0.1;
allow ::1;
deny all;
proxy_pass http://127.0.0.1:9090;
proxy_set_header X-Real-IP $remote_addr;
}
```
### 19.5 Block scraper bots with 403
```nginx
# In http context
map $http_user_agent $is_unwanted_bot {
default 0;
~*Bytespider 1;
~*PetalBot 1;
~*MJ12bot 1;
~*SemrushBot 1;
~*AhrefsBot 1;
~*DotBot 1;
}
# In server block
if ($is_unwanted_bot) {
return 403;
}
```
### 19.6 Validation with 422 (FastAPI)
```python
from pydantic import BaseModel, EmailStr, Field
from datetime import date
class CreateBooking(BaseModel):
name: str = Field(min_length=1, max_length=100)
email: EmailStr
start_date: date
end_date: date
@app.post("/api/bookings")
async def create_booking(booking: CreateBooking):
if booking.end_date <= booking.start_date:
raise HTTPException(
status_code=422,
detail=[{"loc": ["end_date"], "msg": "must be after start_date"}]
)
if booking.start_date < date.today():
raise HTTPException(
status_code=422,
detail=[{"loc": ["start_date"], "msg": "cannot be in the past"}]
)
# ...
```
### 19.7 JSON parsing error returning 400
```python
from fastapi.exceptions import RequestValidationError
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
# If the body is malformed JSON, return 400 instead of 422
if any("JSONDecodeError" in str(err.get("type", "")) for err in exc.errors()):
return JSONResponse(
status_code=400,
content={"error": "invalid_request", "detail": "malformed JSON body"}
)
return JSONResponse(
status_code=422,
content={"detail": exc.errors()}
)
```
### 19.8 Rate limit with 429 and Retry-After
See framework-http-rate-control-headers.md Section 13.1. Quick reference:
```nginx
limit_req_status 429;
error_page 429 = @rate_limited;
location @rate_limited {
internal;
add_header Retry-After "60" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}';
}
```
### 19.9 Legal removal with 451
```nginx
# Specific URL legally removed
location = /legally-removed-content {
add_header Link '<https://example.com/legal/takedown/12847>; rel="blocked-by"' always;
return 451;
}
error_page 451 /451.html;
location = /451.html {
root /var/www/sites/example.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
```
### 19.10 File upload size limit with 413
```nginx
location /api/upload {
client_max_body_size 100m;
proxy_pass http://127.0.0.1:9090;
}
# Custom 413 error page
error_page 413 /413.json;
location = /413.json {
internal;
add_header Content-Type "application/json" always;
return 413 '{"error": "payload_too_large", "max_size_mb": 100}';
}
```
### 19.11 Content type enforcement with 415
```python
@app.post("/api/upload")
async def upload(request: Request):
content_type = request.headers.get("content-type", "")
if not content_type.startswith("multipart/form-data"):
raise HTTPException(
status_code=415,
detail="content-type must be multipart/form-data"
)
# ...
```
### 19.12 Method enforcement with 405
```nginx
location = /api/items {
limit_except GET POST {
deny all; # returns 403; nginx limit_except is different from method restriction
}
# For proper 405 with Allow header, handle in upstream:
proxy_pass http://127.0.0.1:9090;
}
```
FastAPI returns 405 automatically when no endpoint matches the method:
```python
@app.get("/api/items")
async def get_items():
return []
# PUT /api/items will get 405 Method Not Allowed
# Allow header will list: GET (and any other defined methods on this URL)
```
---
## 20. BUBBLES NGINX REFERENCE BLOCK (PASTE READY)
The complete 4xx aware configuration for a Bubbles client site.
```nginx
# /etc/nginx/snippets/4xx-error-pages.conf
# Custom error pages and 4xx handling
error_page 404 /404.html;
error_page 410 /410.html;
error_page 429 = @rate_limited;
error_page 451 /451.html;
location = /404.html {
root /var/www/sites/example.com;
internal;
}
location = /410.html {
root /var/www/sites/example.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
location = /451.html {
root /var/www/sites/example.com;
internal;
add_header X-Robots-Tag "noindex" always;
}
location @rate_limited {
internal;
add_header Retry-After "60" always;
add_header Cache-Control "no-store" always;
add_header X-Robots-Tag "noindex" always;
return 429 '{"error": "rate_limit_exceeded", "retry_after": 60}';
}
```
```nginx
# /etc/nginx/sites-available/example.com
# In http context, the gone-urls map (built from the cleanup list)
# This would be in nginx.conf or an http-level snippet
http {
map $request_uri $is_gone {
default 0;
~*^/old-section/ 1;
~*^/removed-content/ 1;
# ... add bulk removal URLs here ...
}
# Blocked bot map
map $http_user_agent $is_blocked_bot {
default 0;
~*Bytespider 1;
~*PetalBot 1;
~*MJ12bot 1;
~*SemrushBot 1;
}
}
server {
listen 443 ssl;
listen 443 quic;
http2 on;
http3 on;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_early_data on;
add_header Alt-Svc 'h3=":443"; ma=86400' always;
include snippets/common-security-headers.conf;
include snippets/4xx-error-pages.conf;
root /var/www/sites/example.com;
index index.html;
# Body size limit
client_max_body_size 10m;
# Block unwanted bots
if ($is_blocked_bot) {
return 403;
}
# ============= 410 GONE FOR REMOVED URLs =============
location / {
if ($is_gone) {
return 410;
}
try_files $uri $uri/ $uri.html =404;
}
# ============= ADMIN (403 if not from allowed IPs) =============
location /admin/ {
allow 100.90.0.0/16; # Tailscale
allow 127.0.0.1;
deny all;
proxy_pass http://127.0.0.1:9090;
proxy_set_header X-Real-IP $remote_addr;
}
# ============= API (401/403 from upstream) =============
location /api/ {
if ($request_method = OPTIONS) {
add_header Access-Control-Allow-Origin $cors_origin always;
add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
add_header Access-Control-Allow-Headers "Content-Type, Authorization" always;
return 204;
}
add_header X-Robots-Tag "noindex, nofollow" always;
proxy_pass http://127.0.0.1:9090;
}
}
```
After deploying:
```bash
nginx -t && systemctl reload nginx
```
Verify each pattern:
```bash
# 404 for unknown URL
curl -sI https://example.com/does-not-exist | head -1
# Expected: HTTP/2 404
# 410 for explicitly removed URL
curl -sI https://example.com/old-section/anything | head -1
# Expected: HTTP/2 410
# 403 from blocked bot UA
curl -sI -A "Mozilla/5.0 Bytespider" https://example.com/ | head -1
# Expected: HTTP/2 403
# 403 from disallowed IP for /admin (assuming you're not on Tailscale)
curl -sI https://example.com/admin/ | head -1
# Expected: HTTP/2 403
# 401 from API (no auth)
curl -sI https://example.com/api/me | head -3
# Expected: HTTP/2 401 with WWW-Authenticate
# 413 from oversized upload
dd if=/dev/zero of=/tmp/big.bin bs=1M count=20
curl -sI -X POST --data-binary @/tmp/big.bin https://example.com/api/upload | head -1
# Expected: HTTP/2 413
# 422 from invalid data
curl -sI -X POST -H "Content-Type: application/json" \
-d '{"email":"not-email","password":"abc"}' \
https://example.com/api/users | head -1
# Expected: HTTP/2 422
```
---
## 21. AUDIT CHECKLIST
Run through these 50 items for any production deployment.
### 400 Bad Request
1. [ ] Used for malformed requests (JSON parse errors, missing headers).
2. [ ] Not used for "resource not found" (use 404 instead).
3. [ ] Not used for "permission denied" (use 401/403 instead).
4. [ ] Error body explains what was wrong with the request format.
### 401 Unauthorized
5. [ ] Used for unauthenticated requests to protected endpoints.
6. [ ] **WWW-Authenticate header always present on 401 responses.**
7. [ ] Realm and scheme correctly specified in WWW-Authenticate.
8. [ ] Cookie based auth uses custom Cookie scheme (not Basic, to avoid browser prompts).
9. [ ] Login pages return 401, not 200 with login form (avoids soft 404).
10. [ ] Expired tokens return 401 with `error="invalid_token"` in WWW-Authenticate.
### 403 Forbidden
11. [ ] Used for authenticated requests lacking permission.
12. [ ] Used for IP based access denial.
13. [ ] Used for blocked user agents (scraper bots).
14. [ ] Response body minimal (no information leak).
15. [ ] Admin endpoints return 403 to non admins.
16. [ ] CSRF check failures return 403.
17. [ ] **Not used in place of 401 for unauthenticated requests.**
### 404 Not Found
18. [ ] Used for URLs that do not match any resource.
19. [ ] Custom 404 page is substantive (no soft 404).
20. [ ] Custom 404 page returns 404 status (NOT 200).
21. [ ] **Deleted pages that are gone forever use 410, NOT 404.**
22. [ ] Sitemap does not include 404 URLs.
23. [ ] Internal links do not point to 404 URLs.
24. [ ] GSC "Not found (404)" count monitored.
### 410 Gone
25. [ ] Used for bulk removal of low quality content.
26. [ ] Used for permanently removed pages.
27. [ ] Used for spam/hacked content cleanup.
28. [ ] **Not used for pages that might come back.**
29. [ ] Bulk 410 map config in place if cleanup project active.
30. [ ] Custom 410 page provided.
31. [ ] Custom 410 page has `X-Robots-Tag: noindex`.
32. [ ] GSC URL Removals tool used for critical removals to accelerate.
33. [ ] GSC "Not found (410)" count tracked weekly during cleanup.
### 422 Unprocessable Entity
34. [ ] Used for validation failures on well formed requests.
35. [ ] Response body includes per field error detail.
36. [ ] FastAPI default 422 handler returning useful detail.
### 429 Too Many Requests
37. [ ] **Retry-After header always present on 429 responses.**
38. [ ] nginx `limit_req_status 429` set (not default 503).
39. [ ] Crawler whitelist in place (Googlebot, ClaudeBot, GPTBot not throttled).
40. [ ] X-Robots-Tag: noindex on rate limit error page.
41. [ ] No 429 to crawlers for more than 2 days (Google deindex rule).
### 451 Unavailable For Legal Reasons
42. [ ] Used for GDPR right to erasure compliance.
43. [ ] Used for DMCA takedown responses.
44. [ ] Optional Link header with rel="blocked-by".
### Other 4xx codes
45. [ ] 413 returned for oversized uploads (client_max_body_size configured).
46. [ ] 415 returned for unsupported Content-Type.
47. [ ] 405 returned with Allow header when method wrong.
### Cross cutting
48. [ ] All 4xx responses include `X-Robots-Tag: noindex` if the error page itself might be indexed.
49. [ ] `nginx -t` passes without warnings.
50. [ ] `nginx -T | grep "return 4"` shows expected status codes; no accidental 4xx where 2xx/3xx intended.
A site that passes all 50 has correctly configured 4xx status code usage for SEO, security, API design, and operational hygiene.
---
## 22. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: 404 used for bulk content removal.**
Symptom: removed URLs persist in Google index for 3+ months.
Why it breaks: 404 says "might come back"; Google retries for weeks.
Fix: use 410 for explicitly permanent removal. The 92% retry frequency drop after second consecutive 410 produces deindex in days.
**Pitfall 2: 401 returned without WWW-Authenticate header.**
Symptom: clients cannot complete the auth flow.
Why it breaks: RFC requires WWW-Authenticate on 401.
Fix: always include the header with scheme and realm.
**Pitfall 3: Login wall returns 200 with login form.**
Symptom: login page indexed as soft 404; account pages flagged in GSC.
Why it breaks: 200 with non substantive content is soft 404.
Fix: return 401 with WWW-Authenticate for programmatic clients; render login page directly with `X-Robots-Tag: noindex` for browsers.
**Pitfall 4: 403 used when 401 is correct.**
Symptom: legitimate users hit "forbidden" before they have a chance to authenticate.
Why it breaks: 403 means "I won't serve this regardless of credentials"; 401 means "I will if you authenticate".
Fix: 401 first; 403 only when authentication won't help.
**Pitfall 5: 401 used when 403 is correct.**
Symptom: authenticated user gets challenged again instead of denied.
Why it breaks: 401 prompts auth retry; the user is already authenticated.
Fix: 403 for "you're authenticated but lack permission".
**Pitfall 6: WWW-Authenticate uses Basic scheme on cookie auth site.**
Symptom: browser shows native auth dialog instead of using login form.
Why it breaks: Browsers see `Basic` and prompt with their own dialog.
Fix: use custom `Cookie` scheme or Bearer with appropriate realm.
**Pitfall 7: Custom 404 page returns 200.**
Symptom: thousands of "soft 404" URLs in GSC despite being deleted.
Why it breaks: page exists per the server (200), content is "not found" message.
Fix: ensure the error page handler returns 404 status, not 200.
**Pitfall 8: 410 used for a page that might come back.**
Symptom: when the page comes back, it has lost all SEO equity.
Why it breaks: 410 is a permanent commitment.
Fix: use 404 if uncertain. Reserve 410 for genuine permanent removal.
**Pitfall 9: 429 without Retry-After.**
Symptom: clients retry immediately, exacerbating the rate limit problem.
Why it breaks: without Retry-After, clients have no signal to back off.
Fix: always include Retry-After on 429.
**Pitfall 10: 429 sent to Googlebot persistently.**
Symptom: site URLs drop from Google index over 2-4 days.
Why it breaks: Google's 2 day rule: sustained 429/503 leads to index removal.
Fix: implement the crawler whitelist (framework-http-rate-control-headers.md Section 10).
**Pitfall 11: 400 used for resource not found.**
Symptom: API clients retry with same request, infinite loop.
Why it breaks: 400 suggests "fix the request"; client cannot fix.
Fix: 404 for resource not found.
**Pitfall 12: 403 error body leaks information.**
Symptom: attackers learn role names, user IDs, permission details from error responses.
Why it breaks: detailed errors enable enumeration attacks.
Fix: minimal 403 body: `{"error": "forbidden"}` is sufficient.
**Pitfall 13: 422 used where 400 is correct.**
Symptom: API clients confused about whether to fix request or data.
Why it breaks: 422 is for valid format with invalid data; 400 is for invalid format.
Fix: use 400 for JSON parse errors; 422 for validation.
**Pitfall 14: Deleted pages redirect to homepage with 301.**
Symptom: Google flags as soft 404; SEO equity lost.
Why it breaks: redirect target is irrelevant; Google detects no canonical match.
Fix: 410 for removal, or 301 to a relevant new page (not homepage).
**Pitfall 15: 451 not used for legal removals.**
Symptom: legally removed content shows 404 or 410; legal transparency missing.
Why it breaks: 451 is the spec compliant signal for legal mandates.
Fix: 451 with optional Link rel="blocked-by" for transparency.
---
## 23. DIAGNOSTIC COMMANDS
Reference of every command useful for 4xx investigation.
### Inspect specific status codes
```bash
# Check what status a URL returns
curl -sI https://example.com/some-url | head -1
# Check for 401 with WWW-Authenticate
curl -sI https://example.com/protected | grep -iE "^(HTTP|WWW-Authenticate)"
# Check for 410 vs 404 (the SEO critical pair)
for url in /old-page-1 /old-page-2 /old-page-3; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://example.com$url")
echo "$url: $STATUS"
done
```
### Bulk audit removed URLs
```bash
# Verify list of URLs all return 410
while read url; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://example.com$url")
if [ "$STATUS" != "410" ]; then
echo "WRONG: $url returns $STATUS (expected 410)"
fi
done < /tmp/urls-to-410.txt
```
### Status code distribution from logs
```bash
# Top status codes today
sudo awk -v d="$(date '+%d/%b/%Y')" '$0 ~ d {print $9}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -10
# 4xx specific
sudo grep -oE '" 4[0-9]{2} ' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Most common 404 URLs
sudo awk '$9 == 404 {print $7}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -20
# 403 by user agent (find what's being blocked)
sudo awk '$9 == 403 {for(i=11;i<=NF;i++) printf "%s ", $i; print ""}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head -10
```
### Test auth challenge (401)
```bash
# Get the WWW-Authenticate header for a protected resource
curl -sI https://api.example.com/private | grep -iE "^(HTTP|WWW-Authenticate)"
# Test with invalid token
curl -sI -H "Authorization: Bearer invalid_token" https://api.example.com/private | \
grep -iE "^(HTTP|WWW-Authenticate)"
# Expected: 401 with WWW-Authenticate: Bearer realm="api", error="invalid_token"
# Test with valid token
curl -sI -H "Authorization: Bearer $VALID_TOKEN" https://api.example.com/private | head -1
# Expected: HTTP/2 200
```
### Test 422 validation responses
```bash
# Invalid email
curl -s -X POST -H "Content-Type: application/json" \
-d '{"email":"not-an-email","password":"[credential removed]"}' \
https://api.example.com/users | python3 -m json.tool
# Missing required field
curl -s -X POST -H "Content-Type: application/json" \
-d '{"email":"valid@example.com"}' \
https://api.example.com/users | python3 -m json.tool
```
### Find soft 404 candidates (200 returned for what should be 4xx)
```bash
# URLs returning 200 with very thin body (potential soft 404)
sudo awk '$9 == 200 && $10 < 500 {print $7, $10}' /var/log/nginx/access.log | \
sort -u | head -20
# Cross check with sitemap
curl -s https://example.com/sitemap.xml | \
grep -oE "<loc>[^<]+</loc>" | sed 's/<[^>]*>//g' | \
while read url; do
SIZE=$(curl -so /dev/null -w "%{size_download}" "$url")
if [ "$SIZE" -lt 500 ]; then
echo "THIN: $url ($SIZE bytes)"
fi
done
```
### Server side investigation
```bash
# Find all explicit 4xx returns in nginx config
nginx -T 2>/dev/null | grep -E "return 4[0-9]{2}"
# Find all error_page directives
nginx -T 2>/dev/null | grep "error_page"
# Find the gone URL map
nginx -T 2>/dev/null | grep -A 20 "map.*is_gone"
# Check for limit_req_status
nginx -T 2>/dev/null | grep "limit_req_status"
# Expected: limit_req_status 429;
# Apply changes
nginx -t && systemctl reload nginx
```
### GSC monitoring (manual but critical)
In Google Search Console:
1. **Page Indexing report**: track "Not found (404)", "Not found (410)", "Soft 404" counts week over week.
2. **URL Removals tool**: for critical 410 removals, submit removal request to accelerate.
3. **Index Coverage**: monitor "Valid" count to confirm legitimate pages are not being removed.
---
## 24. CROSS-REFERENCES
* [framework-http-2xx-status-codes.md](framework-http-2xx-status-codes.md): the soft 404 trap (200 with thin content) is the upstream of many 4xx misuses.
* [framework-http-3xx-status-codes.md](framework-http-3xx-status-codes.md): 301 redirect to relevant new page is the alternative to 410 for moved content.
* [framework-http-caching-headers.md](framework-http-caching-headers.md): 4xx responses should generally have `Cache-Control: no-store` to avoid serving stale errors.
* [framework-http-seo-headers.md](framework-http-seo-headers.md): `X-Robots-Tag: noindex` on 4xx error pages prevents accidental indexing.
* [framework-http-security-headers.md](framework-http-security-headers.md): security headers apply to 4xx responses; auth interaction.
* [framework-http-rate-control-headers.md](framework-http-rate-control-headers.md): the full 429 treatment including crawler whitelist and the 2-day Google deindex rule.
* [framework-http-request-headers.md](framework-http-request-headers.md): `Authorization` request header triggers 401/403 decisions; `Host` header missing or mismatched can cause 400.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Soft 404 prevention and bulk cleanup are core SEO hygiene topics.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook including the thin content cleanup pattern.
* [framework-http-5xx-status-codes.md] (next in series): 500, 502, 503, 504.
* Google HTTP status codes documentation: https://developers.google.com/search/docs/crawling-indexing/http-network-errors
* Google URL Removals tool: https://search.google.com/search-console/remove-outdated-content
* MDN 400: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/400
* MDN 401: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/401
* MDN 403: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/403
* MDN 404: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/404
* MDN 410: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/410
* MDN 422: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/422
* MDN 429: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
* MDN 451: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/451
* RFC 9110 (HTTP Semantics): https://www.rfc-editor.org/rfc/rfc9110
* RFC 7725 (451 Unavailable For Legal Reasons): https://www.rfc-editor.org/rfc/rfc7725
* RFC 6585 (Additional HTTP Status Codes, 429): https://www.rfc-editor.org/rfc/rfc6585
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### Status code decision tree
```
Client did something wrong. What?
|
|---> Request malformed (bad JSON, missing header) ........... 400
|
|---> Request well formed but data invalid .................... 422
|
|---> No auth credentials ..................................... 401 (+ WWW-Authenticate)
|
|---> Has auth, lacks permission .............................. 403
|
|---> Resource not here, might return ......................... 404
|
|---> Resource gone forever (bulk removal) .................... 410
|
|---> Client too fast ......................................... 429 (+ Retry-After)
|
|---> Legal mandate to block .................................. 451 (+ optional Link)
|
|---> Method not allowed ...................................... 405 (+ Allow)
|
|---> Body too large .......................................... 413
|
|---> Content-Type not supported .............................. 415
```
### SEO removal speed
| Code | Time to drop from Google index |
|---|---|
| 410 | Days (typically 1-2 weeks) |
| 404 | Weeks to months |
| 403 | Weeks (Googlebot backs off) |
| 451 | Days (treated as legal removal) |
### The 401 vs 403 question
**Can the client gain access by authenticating differently?**
* **YES** to 401 (with WWW-Authenticate)
* **NO** to 403
### The 404 vs 410 question
**Will this URL ever return?**
* **Maybe** to 404
* **No, never** to 410 (faster deindex)
### Five rules to memorize
1. 401 needs WWW-Authenticate. Always.
2. 429 needs Retry-After. Always.
3. 410 over 404 for permanent removal. 92% retry frequency drop after second consecutive 410.
4. 403 backs Googlebot off. 401 tells them "not for you".
5. 451 for legal removal. Optional Link rel="blocked-by".
### Five commands every operator should know
```bash
# 1. Check status of a URL
curl -sI https://example.com/url | head -1
# 2. Verify 401 has WWW-Authenticate
curl -sI https://example.com/protected | grep -iE "^(HTTP|WWW-Authenticate)"
# 3. Find 4xx hot spots in logs
sudo awk '$9 ~ /^4/ {print $9, $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
# 4. Audit bulk 410 deployment
while read url; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://example.com$url")
[ "$STATUS" != "410" ] && echo "WRONG: $url ($STATUS)"
done < /tmp/urls-to-410.txt
# 5. Apply nginx changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. 410 for removed URLs (not 404)
for url in /old-page-1 /old-page-2; do
STATUS=$(curl -so /dev/null -w "%{http_code}" "https://example.com$url")
[ "$STATUS" = "410" ] && echo "OK: $url is 410" || echo "FAIL: $url is $STATUS"
done
# 2. 401 has WWW-Authenticate
HAS_AUTH=$(curl -sI https://example.com/api/private | grep -i "www-authenticate" | wc -l)
[ "$HAS_AUTH" = "1" ] && echo "OK: 401 has WWW-Authenticate" || echo "FAIL: missing header"
# 3. 429 has Retry-After
HAS_RETRY=$(curl -sI -H "X-Force-Rate-Limit: 1" https://example.com/api/data 2>/dev/null | grep -i "retry-after" | wc -l)
[ "$HAS_RETRY" = "1" ] && echo "OK: 429 has Retry-After" || echo "verify rate limiting"
```
If all three pass AND GSC Page Indexing > Not found (410) drops weekly during cleanup campaigns, the 4xx layer is correctly wired.
---
End of framework-http-4xx-status-codes.md.