A continuation of Janady13/crawler-signal-reference / framework-html-meta-charset.md. The complete source identity, every recorded occurrence, and linked equations are on its first page.
Source transcript · continued
Download complete source text ↓1,918 lines · 66.0 KB
fi
# Layer 3: file (if domain matches a known Bubbles site)
SITE_DIR="/var/www/sites/$DOMAIN"
if [ -d "$SITE_DIR" ]; then
echo ""
echo "Layer 3 (file):"
find "$SITE_DIR" -name "*.html" -type f | head -3 | while read f; do
ENC=$(file -i "$f" | grep -oE "charset=[^ ;]+")
echo " $f: $ENC"
done
fi
# Layer 4: database
echo ""
DB_ENC=$(sudo -u postgres psql -At -c "SHOW server_encoding;" 2>/dev/null)
echo "Layer 4 (db): server_encoding = $DB_ENC"
[ "$DB_ENC" = "UTF8" ] && echo " Status: OK" || echo " Status: NOT UTF8"
# Layer 5: forms
echo ""
FORMS=$(curl -s "$URL" | grep -oE 'accept-charset="[^"]+"' | head -3)
if [ -n "$FORMS" ]; then
echo "Layer 5 (forms): $FORMS"
else
echo "Layer 5 (forms): no accept-charset attributes (relies on page encoding)"
fi
echo ""
echo "=== End audit ==="
```
Make executable and use:
```bash
sudo chmod +x /usr/local/bin/charset-audit.sh
charset-audit.sh https://example.com/
```
---
## 20. AUDIT CHECKLIST
Run through these 50 items for any production deployment.
### HTML layer
1. [ ] Every HTML page has `<meta charset="utf-8">` declared.
2. [ ] Declaration is the FIRST tag inside `<head>`.
3. [ ] Declaration is within the first 1024 bytes of the document.
4. [ ] Declaration uses the HTML5 short form (not the legacy http-equiv form).
5. [ ] Declaration uses lowercase: `<meta charset="utf-8">` not `UTF-8`.
### HTTP layer (nginx)
6. [ ] `charset utf-8;` directive set in nginx (http or server context).
7. [ ] `charset_types` includes all text types served by the site.
8. [ ] Content-Type response header includes `charset=utf-8` for HTML.
9. [ ] Verify with `curl -sI` that Content-Type matches expected.
### File system layer
10. [ ] All HTML files saved as UTF-8 (verified via `file -i`).
11. [ ] No BOM in any HTML file.
12. [ ] All CSS files UTF-8 without BOM.
13. [ ] All JavaScript files UTF-8 without BOM.
14. [ ] All JSON files UTF-8 without BOM.
15. [ ] Editor configuration (`.editorconfig` or equivalent) enforces UTF-8.
### Database layer (PostgreSQL)
16. [ ] `server_encoding = UTF8` per `SHOW server_encoding;`.
17. [ ] Every database uses UTF8 encoding (`pg_database` table).
18. [ ] `client_encoding` set to UTF8 in FastAPI connection string.
19. [ ] Sample query returns UTF-8 characters correctly.
20. [ ] LC_COLLATE and LC_CTYPE set to a UTF-8 locale.
### Form layer
21. [ ] Forms have `accept-charset="utf-8"` attribute (defense in depth).
22. [ ] FastAPI Form processing receives UTF-8 correctly.
23. [ ] Form submissions with special characters round trip correctly.
### Content layer
24. [ ] No mojibake visible on any page (smart quotes, em dashes, accented characters).
25. [ ] Page titles in browser tab show correctly.
26. [ ] Page titles in Google search results show correctly.
27. [ ] Meta descriptions with special characters preserved.
28. [ ] Special characters render correctly in social media share previews.
### Marshallese-Voices specific (or any non Latin content)
29. [ ] Macron diacritics (ā, ē, ī, ō, ū) display correctly.
30. [ ] Cedilla diacritics (ļ, ņ, m̧) display correctly.
31. [ ] Combining characters preserved through copy paste.
32. [ ] Search functionality finds Marshallese terms (DB collation supports it).
### Security
33. [ ] `X-Content-Type-Options: nosniff` header set.
34. [ ] No UTF-7 vulnerability surface (charset always declared).
35. [ ] CSP header set (covered in framework-http-security-headers.md).
### Cross cutting
36. [ ] robots.txt file UTF-8 encoded.
37. [ ] sitemap.xml UTF-8 encoded.
38. [ ] All template includes/partials UTF-8 encoded.
39. [ ] All log files UTF-8.
40. [ ] All configuration files UTF-8.
### Tooling and process
41. [ ] Pre commit hook (or CI step) verifies UTF-8 encoding of source files.
42. [ ] Editor settings documented for new contributors.
43. [ ] Database backup/restore preserves UTF-8 through round trip.
44. [ ] `charset-audit.sh` script in `/usr/local/bin/`.
### Verification commands tested
45. [ ] `curl -sI URL | grep -i content-type` returns expected.
46. [ ] `curl -s URL | head -c 1024 | grep charset` finds declaration.
47. [ ] `file -i FILE` shows UTF-8 for all source files.
48. [ ] `psql -c "SHOW server_encoding;"` returns UTF8.
49. [ ] `nginx -t` passes.
50. [ ] `nginx -t && systemctl reload nginx` applied after any config changes.
A site that passes all 50 has correctly configured UTF-8 across all five operational layers and is resilient to surprise content with special characters.
---
## 21. COMMON PITFALLS
Fifteen patterns to recognize and avoid.
**Pitfall 1: `<meta charset>` placed after other tags in head.**
Symptom: characters render as mojibake in some browsers.
Why it breaks: charset declaration pushed past the 1024 byte cutoff.
Fix: place `<meta charset="utf-8">` as the FIRST tag inside `<head>`.
**Pitfall 2: HTTP Content-Type contains charset that differs from HTML.**
Symptom: browser ignores HTML meta tag; uses HTTP charset (which is wrong).
Why it breaks: HTTP wins over HTML in the encoding cascade.
Fix: ensure nginx `charset utf-8;` is set; both layers say UTF-8.
**Pitfall 3: BOM in HTML file.**
Symptom: weird whitespace at top of page, PHP "headers already sent" errors, validator complaints.
Why it breaks: BOM bytes appear in the rendered document.
Fix: strip BOM via `sed -i '1s/^\xEF\xBB\xBF//' file.html`.
**Pitfall 4: Database created with SQL_ASCII encoding.**
Symptom: special characters round trip through some queries but break on others.
Why it breaks: SQL_ASCII stores raw bytes with no encoding awareness.
Fix: dump, drop, recreate database with UTF8 encoding, restore.
**Pitfall 5: File saved as ISO-8859-1 with UTF-8 meta tag.**
Symptom: special characters in the file appear as mojibake.
Why it breaks: file bytes don't match declared encoding.
Fix: convert file with `iconv -f ISO-8859-1 -t UTF-8 file.html > new.html`.
**Pitfall 6: Mojibake fixed by adding more encoding declarations.**
Symptom: mojibake persists despite adding more `<meta charset>` tags.
Why it breaks: the issue is at a different layer; more meta tags do not help.
Fix: identify the actual broken layer (HTTP, file, database) and fix it there.
**Pitfall 7: Form submission corrupts special characters.**
Symptom: form data with é, ā, ñ comes through as ’, Ä?, ñ.
Why it breaks: form encoding inconsistent with page or server.
Fix: add `accept-charset="utf-8"` to form; verify FastAPI receives UTF-8.
**Pitfall 8: nginx default Content-Type without charset.**
Symptom: HTTP response is `Content-Type: text/html` (no charset).
Why it breaks: browser falls back to sniffing.
Fix: nginx `charset utf-8;` directive in http or server context.
**Pitfall 9: Smart quotes from Microsoft Word in editorial content.**
Symptom: copy paste from Word produces “ and †in published content.
Why it breaks: Word's smart quotes are Windows-1252 specific bytes interpreted as UTF-8 (or vice versa).
Fix: ensure entire pipeline is UTF-8; verify Word saved as UTF-8 before paste, or normalize smart quotes in content sanitization.
**Pitfall 10: Marshallese (or other non Latin) text shows as ??? in production.**
Symptom: development works, production shows question marks.
Why it breaks: a layer in the pipeline (often database or file) is not UTF-8.
Fix: audit all five layers; find the mismatch.
**Pitfall 11: Emoji in titles or content stripped or replaced.**
Symptom: 🎉 appears as ?? or is missing.
Why it breaks: a layer is not 4 byte UTF-8 capable (some databases need utf8mb4 in MySQL terms; PostgreSQL UTF8 handles all).
Fix: verify all layers handle 4 byte UTF-8.
**Pitfall 12: HTML entities used for characters that should be raw UTF-8.**
Symptom: source code is bloated with `&#xNNNN;` entities for normal accented characters.
Why it breaks: legacy practice; not wrong but inefficient.
Fix: declare UTF-8, use raw characters. Reserve entities for the five mandatory characters (&, <, >, ", ').
**Pitfall 13: charset declaration in `<meta http-equiv>` form only (no short form).**
Symptom: works but takes more bytes; some validators warn.
Why it breaks: legacy HTML4 syntax; verbose.
Fix: replace with `<meta charset="utf-8">`.
**Pitfall 14: nginx adds charset to binary files (images, PDFs).**
Symptom: image MIME types include unexpected charset parameter.
Why it breaks: `charset utf-8;` applies to types in `charset_types`; if image types are accidentally included.
Fix: ensure `charset_types` lists only text MIME types.
**Pitfall 15: Inconsistent encoding declarations across templates.**
Symptom: some pages have meta charset, others don't.
Why it breaks: inconsistent template hygiene.
Fix: enforce template includes; every layout has `<meta charset="utf-8">` as first head element.
---
## 22. DIAGNOSTIC COMMANDS
Reference of every command useful for charset investigation.
### Inspect a single URL
```bash
# Check HTTP Content-Type
curl -sI https://example.com/ | grep -i content-type
# Check HTML meta charset (first 1024 bytes)
curl -s https://example.com/ | head -c 1024 | grep -i charset
# Check exact byte position of charset declaration
curl -s https://example.com/ | head -c 2000 | grep -bn 'charset' | head -3
# Verify response body is valid UTF-8
curl -s https://example.com/ | iconv -f UTF-8 -t UTF-8 -c > /dev/null && echo "OK" || echo "BROKEN"
```
### Inspect a file
```bash
# Check encoding
file -i /var/www/sites/example.com/index.html
# Check for BOM
hexdump -C /var/www/sites/example.com/index.html | head -1
# A BOM is: ef bb bf at the start
# Check character composition
awk '{ for (i=1; i<=length($0); i++) { c=substr($0,i,1); ord=ord ? ord c : ord c; if (length(c) > 1) print c, "MULTIBYTE" } }' file.html
```
### Bulk file audit
```bash
# Find all non UTF-8 files
find /var/www/sites/ -type f \( -name "*.html" -o -name "*.css" -o -name "*.js" \) | while read f; do
ENC=$(file -i "$f" | grep -oE "charset=[^ ;]+")
if [ "$ENC" != "charset=utf-8" ] && [ "$ENC" != "charset=us-ascii" ]; then
echo "$f: $ENC"
fi
done
# Find files with BOM
find /var/www/sites/ -type f -exec file -i {} \; | grep "byte order mark"
```
### Database checks
```bash
# Server encoding
sudo -u postgres psql -c "SHOW server_encoding;"
# All databases
sudo -u postgres psql -c "SELECT datname, pg_encoding_to_char(encoding), datcollate, datctype FROM pg_database WHERE datistemplate = false;"
# Sample query with special characters
sudo -u postgres psql -d bubbles -c "SELECT title FROM articles WHERE title ~ '[āēīōū]' LIMIT 5;"
# Check byte vs character length to detect multi byte content
sudo -u postgres psql -d bubbles -c "SELECT id, title, octet_length(title) AS bytes, length(title) AS chars FROM articles WHERE octet_length(title) != length(title) LIMIT 10;"
```
### Convert files
```bash
# Convert single file
iconv -f WINDOWS-1252 -t UTF-8 input.html > output.html
# Detect encoding (uchardet must be installed: apt install uchardet)
uchardet input.html
# Strip BOM
sed -i '1s/^\xEF\xBB\xBF//' file.html
```
### Test with special characters
```bash
# Submit form with non ASCII
curl -X POST -d "name=Mojiba ké" -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" \
https://example.com/submit
echo "Status: $?"
# Inspect form submission encoding
curl -s -d "name=test" -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" \
https://example.com/submit | grep -i charset
```
### Apply nginx changes
```bash
nginx -t && systemctl reload nginx
```
### Editor and tooling verification
```bash
# Verify .editorconfig exists and sets charset
cat /var/www/sites/example.com/.editorconfig | grep -i charset
# Verify vim default
vim -c ":set encoding?" -c ":q" 2>&1 | grep encoding
# Verify VS Code settings (if accessible)
cat ~/.config/Code/User/settings.json | grep -i encoding
```
---
## 23. CROSS-REFERENCES
* [framework-http-content-headers.md](framework-http-content-headers.md) Section 5: the Content-Type HTTP header which carries the charset parameter; the precedence rule (HTTP wins over HTML meta).
* [framework-http-security-headers.md](framework-http-security-headers.md): `X-Content-Type-Options: nosniff` prevents charset based XSS attacks.
* [framework-html-meta-robots.md](framework-html-meta-robots.md): the first framework in the HTML signal track; the meta tag family.
* [UNIVERSAL-RANKING-FRAMEWORK.md](UNIVERSAL-RANKING-FRAMEWORK.md): the master ranking reference. Mojibake in titles and snippets hurts CTR which hurts rankings.
* [SEO-BUILD-REFERENCE.md](SEO-BUILD-REFERENCE.md) v2.4: the build playbook including charset declaration in every template.
* WHATWG HTML Living Standard - character encoding: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
* MDN meta charset: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta#charset
* WHATWG Encoding Standard: https://encoding.spec.whatwg.org/
* OWASP XSS Prevention Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html
* PostgreSQL Character Set Support: https://www.postgresql.org/docs/current/multibyte.html
* RFC 8259 (JSON, requires UTF-8): https://www.rfc-editor.org/rfc/rfc8259
---
## APPENDIX A: ONE PAGE QUICK REFERENCE
For the person who just wants the answer.
### The canonical pattern
```html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"> <!-- FIRST inside head -->
<!-- everything else follows -->
</head>
```
### The five layer rule
| Layer | Check | Expected |
|---|---|---|
| HTTP | `curl -sI URL | grep content-type` | `text/html; charset=utf-8` |
| HTML | `curl -s URL | head -c 1024 | grep charset` | `<meta charset="utf-8">` |
| File | `file -i path/file.html` | `charset=utf-8` |
| Database | `psql -c "SHOW server_encoding;"` | `UTF8` |
| Form | inspect HTML `<form>` | `accept-charset="utf-8"` |
### Five rules to memorize
1. **UTF-8 everywhere, no exceptions.**
2. **`<meta charset="utf-8">` is the FIRST tag inside `<head>`.**
3. **Place it within the first 1024 bytes (this is automatic if it's first in head).**
4. **No BOM in any file.**
5. **HTTP header wins over HTML meta tag; align both to UTF-8.**
### Five commands every operator should know
```bash
# 1. Check HTTP charset
curl -sI URL | grep -i content-type
# 2. Check HTML charset
curl -s URL | head -c 1024 | grep -i charset
# 3. Check file encoding
file -i /path/to/file.html
# 4. Check database encoding
sudo -u postgres psql -c "SHOW server_encoding;"
# 5. Apply nginx changes
nginx -t && systemctl reload nginx
```
### Three end to end tests
```bash
# 1. HTTP and HTML both UTF-8
URL=https://example.com/
HTTP_OK=$(curl -sI "$URL" | grep -i "charset=utf-8" | wc -l)
HTML_OK=$(curl -s "$URL" | head -c 1024 | grep -ic "charset=\"utf-8\"\|charset=utf-8")
[ "$HTTP_OK" = "1" ] && [ "$HTML_OK" -ge "1" ] && echo "OK" || echo "FAIL"
# 2. Sample file is UTF-8 no BOM
FILE=/var/www/sites/example.com/index.html
ENC=$(file -i "$FILE")
[[ "$ENC" =~ "charset=utf-8" ]] && ! [[ "$ENC" =~ "byte order mark" ]] && echo "OK" || echo "FAIL"
# 3. Database is UTF-8
DB_ENC=$(sudo -u postgres psql -At -c "SHOW server_encoding;" 2>/dev/null)
[ "$DB_ENC" = "UTF8" ] && echo "OK" || echo "FAIL"
```
If all three pass AND no mojibake visible in browser AND special character forms round trip correctly, the charset layer is correctly wired across the five operational layers.
---
End of framework-html-meta-charset.md.