“Accepted” entries belong to the mathematical catalog. “Rejected” means the extracted span is not a complete mathematical record; it is still preserved here and in its full source. Where an adjacent source passage supplies the complete expression, the recovered replacement is linked.
Download every scan decision ↗
scan-2e68559c303d8121
=== GOSSIP: SEND VERSION + DELTA ===
Decorative section heading surrounded by equality characters.
scan-06d7b73616a33a1a
=== GOSSIP: RECEIVE & APPLY DELTA ===
Decorative section heading surrounded by equality characters.
scan-7112462857357e22
Version = file modification time
Reviewed as descriptive prose, application syntax, control-flow annotation, type/field mapping or an incomplete non-mathematical fragment.
scan-2ec2c9fb3f63ad14
=== USE SYSTEM ZLIB (standard compression library) ===
Decorative section heading surrounded by equality characters.
scan-ac4cfd2fd630ab33
=== RSYNC ROLLING CHECKSUM ===
Decorative section heading surrounded by equality characters.
scan-5d65c0f9c530b359
=== GET MY IP (IPv4 packed) ===
Decorative section heading surrounded by equality characters.
scan-a053d74d20917471
=== FNV-1a HASH FOR VERSIONING ===
Decorative section heading surrounded by equality characters.
scan-65e962862eb4c790
=== LOAD BINARY + SIGNATURE ===
Decorative section heading surrounded by equality characters.
scan-bb06918b9c4400f6
=== UDP HEARTBEAT (8 bytes: IP + version) ===
Decorative section heading surrounded by equality characters.
scan-e0268e27fc01eb31
(uint32_t*)(msg + 4) = htonl(my_version & 0xFFFFFFFF);
Reviewed as an object-pointer assignment, structural diagram, application/protocol prose or qualitative label mapping rather than a mathematical equation.
scan-10f4dd81cbed04f8
=== UDP LISTENER ===
Decorative section heading surrounded by equality characters.
scan-3b835edc434bd9c1
=== TCP SYNC SERVER (compressed full binary) ===
Decorative section heading surrounded by equality characters.
scan-cdc12b34c5b743ee
=== SYNC LOOP ===
Decorative section heading surrounded by equality characters.
scan-2a8ec7302d3cc8ab
greaterNumbers = numbers.reduce((acc, num) => num > 3 ? acc.concat(num) : acc, []);
Reviewed as descriptive prose, application syntax, control-flow annotation, type/field mapping or an incomplete non-mathematical fragment.
scan-eeeae8afa0966b57
LAYER 1: Web App Manifest (the canonical signal)
|
|---> /manifest.webmanifest (linked via <link rel="manifest">)
{
"name": "Full app name",
"short_name": "Short label",
"icons": [...],
"start_url": "/",
"display": "standalone",
"theme_color": "#6B3FA0",
"background_color": "#FFFFFF"
}
LAYER 2: iOS Safari Fallback Meta Tags (because Safari is partial)
|
|---> apple-mobile-web-app-capable (enables standalone + splash screen)
|---> apple-mobile-web-app-title (icon label; manifest short_name preferred but Safari behavior varies)
|---> apple-mobile-web-app-status-bar-style (iOS specific, no manifest equivalent)
|---> apple-touch-icon link elements (icon assets)
|---> apple-touch-startup-image link elements (splash screens, requires apple-mobile-web-app-capable)
LAYER 3: Chrome Android Modern Equivalent (largely redundant with manifest)
|
|---> mobile-web-app-capable (replaces Apple tag for Chrome warnings)
LAYER 4: Cross Platform Identity
|
|---> application-name (still used in some browser bookmark UI)
LAYER 5: Microsoft Windows Tile (largely deprecated)
|
|---> msapplication-TileColor (Windows 10 Live Tile background; vestigial in Windows 11)
|---> msapplication-TileImage (Windows 10 Live Tile image; vestigial in Windows 11)
|---> msapplication-config (pointer to browserconfig.xml; vestigial)HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-6318648f3e1ec80b
For each page P in the cluster: ✓ P has a self referencing <link rel="canonical"> ✓ P has <link rel="alternate" hreflang="X"> for every variant X in the cluster, including P itself ✓ Every other page in the cluster also has <link rel="alternate" hreflang="P"> pointing to P
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-125fe0be02b55e51
THE FIVE FILES
|
|---> /favicon.ico (legacy fallback, multi size, ~10 KB)
|---> /icon.svg (modern primary, scalable, ~2 KB)
|---> /apple-touch-icon.png (iOS home screen, 180x180, ~20 KB)
|---> /icon-192.png (Android home screen, 192x192, referenced in manifest)
|---> /icon-512.png (Android splash and install, 512x512, referenced in manifest)
THE HTML HEAD DECLARATIONS
|
|---> <link rel="icon" href="/favicon.ico" sizes="32x32">
|---> <link rel="icon" type="image/svg+xml" href="/icon.svg">
|---> <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
|---> <link rel="manifest" href="/manifest.webmanifest">
THE MANIFEST
|
|---> {
"icons": [
{ "src": "/icon-192.png", "sizes": "192x192", "type": "image/png" },
{ "src": "/icon-512.png", "sizes": "512x512", "type": "image/png" }
]
}HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-ca6fbb3e79242c51
Favicon history: Initial deploy: 2025-08-15, /favicon.ico, /icon.svg, /apple-touch-icon.png Brand refresh: 2026-03-10, bumped to v=2 (?v=2 query parameters)
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-a77c662dc55dad88
Crawler parses your page and extracts authorship signals: | v ==================== SOURCE PRIORITY ==================== | |---> Visible byline in article content (HIGHEST signal) | e.g., <p class="byline">By Joseph Anady</p> | |---> Schema.org structured data (Article.author or BlogPosting.author) | Strong signal; includes credentials, links, sameAs to other identities | |---> Meta author tag | Supporting signal; backs up byline and schema | |---> Open Graph article:author | Social media specific; ensures share previews show author | |---> Twitter creator | Twitter/X specific | |---> Page footer attribution | Weak signal but useful for site wide authorship (e.g., "Crafted by ThatDeveloperGuy") ==================== AGGREGATION ==================== | v Google extracts an "author entity" from all signals. | v If signals align (same author name consistently): strong author confidence. | v If signals conflict (different names): reduced author confidence. | v If signals missing entirely: page treated as anonymous. ==================== E-E-A-T IMPACT ==================== | v For YMYL topics (medical, financial, legal, safety): | |---> Strong author signal with credentials: high E-E-A-T |---> Weak or anonymous author: low E-E-A-T; ranking penalty | For non YMYL topics: | |---> Author signal helpful but less critical |---> Strong content quality can compensate for weak author signals
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-064e75be4f6203aa
Byte sequence: 0xE2 0x80 0x99
|
|---> Interpreted as UTF-8 ........... right single quotation mark (')
|
|---> Interpreted as ISO-8859-1 ...... three characters: â, €, ™
|
|---> Interpreted as Windows-1252 .... three characters: â, €, ™
================ THE ENCODING JOURNEY ================
Author writes content with special characters
|
v
File saved with some encoding (file metadata or implicit)
|
v
Server reads file, sends to client
|
v
HTTP response: Content-Type: text/html; charset=??
|
v
HTML content includes <meta charset="..."> declaration
|
v
Browser parses encoding declaration, interprets bytes accordingly
|
v
Characters render correctly (or as mojibake)
================ WHERE MISMATCHES HAPPEN ================
* File saved as Windows-1252 but server declares UTF-8.
* HTML declares UTF-8 but HTTP header declares ISO-8859-1.
* Form submits as UTF-8 but server parses as Windows-1252.
* Database stores as SQL_ASCII but expected UTF-8.
* JavaScript reads localStorage as Latin-1 but stored as UTF-8.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-a055533f2688260e
HTTP/2 200 OK Content-Type: text/html; charset=ISO-8859-1 <html> <head> <meta charset="utf-8">
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-3807520ce5eb7174
Browser starts rendering the page.
|
v
==================== EARLY DECISIONS ====================
|
|---> What color scheme should the canvas use?
| (area outside HTML element; overscroll area)
|
|---> What styling for native form controls?
| (date pickers, scrollbars, dropdowns)
|
|---> What default text and background colors?
| (before custom CSS specifies)
|
|---> Should I render light or dark by default?
==================== WITHOUT color-scheme ====================
|
v
Browser assumes LIGHT (for legacy compatibility):
- White canvas.
- Light form controls.
- Black on white default text.
- Light rendering.
|
v
If page CSS specifies dark mode via media query:
- Page background turns dark when CSS loads.
- But canvas stays white (jarring).
- Native form controls stay light (jarring).
- Brief light flash visible.
==================== WITH color-scheme: light dark ====================
|
v
Browser knows page supports both schemes:
- Canvas adapts to system color scheme.
- Native form controls adapt to system color scheme.
- Default text and background colors appropriate to system scheme.
- No flash; rendering matches scheme from first paint.
==================== THE DECLARATION FORMS ====================
|
v
HTML meta tag (recommended for timing):
<meta name="color-scheme" content="light dark">
|
CSS on :root (equivalent functionality, slightly later):
:root { color-scheme: light dark; }
|
Both: works fine; redundant but harmless.
==================== ORDER VARIANTS ====================
|
"light dark" - primary light, fallback dark.
"dark light" - primary dark, fallback light.
"only light" - light only; force light even on dark mode systems.
"only dark" - dark only; force dark even on light mode systems.
"normal" - no preference (browser default).HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-de55e74815f5ffc8
Page contains language signals:
|
v
==================== PER SIGNAL READING ORDER ====================
|
|---> HTTP Content-Language response header (set at server)
| Sent before HTML parsing begins.
| Authoritative for server-declared language.
|
|---> <html lang="..."> attribute (parsed early in HTML)
| Primary signal for browsers and search engines.
| Used for hyphenation, screen reader, search ranking.
|
|---> <meta http-equiv="content-language" content="...">
| LEGACY: ignored by Google since 2010.
| Still parsed by some tools and older crawlers.
| No value in modern web.
|
|---> <lang="..."> on body elements
| For mixed language content.
| Per element override.
|
|---> <link rel="alternate" hreflang="...">
| For multi language site targeting.
| The international SEO signal.
==================== THE DECISION TREE ====================
|
v
What is the primary page language?
|
v
Set <html lang="en-US"> (or appropriate BCP 47)
|
v
Does any element contain different language?
|
|---> YES: add lang="..." to that element
|
|---> NO: continue
|
v
Is this a multi language site (translations exist)?
|
|---> YES: add hreflang link elements for each translation
|
|---> NO: continue
|
v
Should HTTP Content-Language header be set?
|
|---> YES: configure in nginx (covered in framework-http-content-headers.md)
|
|---> NO: html lang is sufficient
|
v
Should <meta http-equiv="content-language"> be included?
|
|---> ALMOST NEVER: deprecated by Google; redundant with html langHTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-f368434e0f8f2d92
Language signaling implementation: Default: English (en-US) Alternate: Spanish (es) HTML attribute: <html lang="en-US"> on English; <html lang="es"> on Spanish. hreflang: reciprocal between en-US and es. HTTP Content-Language: set per route in nginx. No deprecated meta http-equiv="content-language" used. All BCP 47 language tags conform to standard format. Aligns with: federal multi language access requirements (where applicable).
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-e1f61e1ad50b54c4
Language signaling: Primary language: en-US Multi language: no HTML attribute: <html lang="en-US"> on every page HTTP header: Content-Language set in nginx (optional) hreflang: not applicable (single language) http-equiv meta: omitted (deprecated by Google)
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-782617888516c05e
Language signaling: Primary language: English (en-US) Alternate: Marshallese (mh) HTML attribute: <html lang="en-US"> on English; <html lang="mh"> on Marshallese HTTP header: Content-Language set per route in nginx hreflang: reciprocal between en-US and mh; x-default to English http-equiv meta: omitted Mixed content: lang="mh" on Marshallese elements within English pages UTF-8: full chain end to end (HTML, nginx, forms, database)
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-64aad4926dc0d95b
Language signaling:
Primary language: {{ primary }}
Multi language: {{ yes/no }}
Alternates: {{ list }}
HTML attribute: <html lang="{{ primary }}"> on every page
HTTP header: Content-Language set via nginx (optional)
hreflang: {{ implemented/not applicable }}
http-equiv meta: OMITTED (deprecated by Google)
Special considerations: {{ e.g., UTF-8 chain for Marshallese }}HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-7127b9c94f1f69bc
Page is fetched. What reads <meta name="generator">?
|
|---> Google: parses, stores, ignores for ranking
|
|---> Bing: parses, stores, ignores for ranking
|
|---> Wappalyzer/BuiltWith/SimilarWeb: parses for tech stack catalog
|
|---> Security scanners (automated): parses for vulnerability targeting
|
|---> Manual security researchers: parses for fingerprinting
|
|---> Competitors using browser dev tools: visible source code
==================== THE INFORMATION FLOW ====================
|
v
"<meta name='generator' content='WordPress 6.4.2'>"
reveals:
- Site is running WordPress
- Specific version is 6.4.2
- Known vulnerabilities in 6.4.2 are exploitation candidates
- Stack is presumably PHP-based with MySQL
- Likely plugins (other signatures combine with this)
==================== THE EVALUATION ====================
|
v
For each level of disclosure, weigh:
- Security risk: how much does this aid an attacker?
- Competitive intelligence: how much does this reveal strategy?
- Operational benefit: who legitimately needs this info?
==================== THE BUBBLES MATRIX ====================
|
v
Include with version (WordPress 6.4.2): NEVER.
|
Include without version (WordPress): rare. Only if necessary disclosure.
|
Include as brand statement (Hand coded by ThatDeveloperGuy.com): optional.
|
Omit entirely: safe default.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-09b43cc207cbe95f
Is this a new hand coded Bubbles build?
|
|---> YES: Optional brand statement.
| <meta name="generator" content="Hand coded by ThatDeveloperGuy.com">
| OR omit entirely.
|
|---> NO: Is this a migration from CMS?
|
|---> YES: AUDIT for stale generator tags. REMOVE.
|
|---> NO: Existing CMS site we're maintaining?
|
|---> Yes: Configure CMS to suppress generator.
|
|---> No (other): omit by default.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-4f6aef7d4237e08b
Crawler fetches your page, sees <meta name="keywords" content="...">
|
v
The tag is parsed and stored in the crawler's understanding of the page.
|
v
What happens next depends on the search engine:
==================== PER ENGINE BEHAVIOR ====================
Google:
- Reads the tag.
- Stores the value in its index.
- Does NOT use it for ranking.
- May use it for anti spam signal detection (excessive keywords flagged).
Bing:
- Reads the tag.
- Uses as one of many relevance signals.
- Excessive or irrelevant keywords trigger anti spam.
- Conservative use is neutral.
Yandex (Russia):
- Reads the tag.
- Uses as a ranking factor.
- More important than for Google/Bing.
Baidu (China):
- Reads the tag.
- Uses as a ranking factor.
DuckDuckGo:
- Derives from Bing's index.
- Inherits Bing's behavior.
Internal site search:
- Often uses meta keywords as a relevance signal.
- Varies by implementation.
==================== THE PRACTICAL EFFECT ====================
For US/English market sites (the Bubbles client base):
- Omitting meta keywords: no negative effect.
- Including conservative meta keywords (3-5 relevant terms): no positive effect.
- Including aggressive meta keywords (10+ terms, especially irrelevant): negative
effect on Bing.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-fe3f368fe24079fa
Decision: omit meta http-equiv="refresh" entirely. Redirect needs: implement via HTTP 301 at nginx layer. Auto refresh needs: implement via JavaScript with WCAG compliant user controls.
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-cf2e1dafd3e951bc
Redirect handling: Implementation: HTTP 301 via nginx No meta http-equiv="refresh" used Per route redirects: see nginx config Migration cleanup: completed; no legacy meta refresh remains. Auto refresh: Not used. If real time data is needed: JavaScript polling with user pause control.
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-53661cb5237ddfe0
Crawler fetched URL, got 200 with HTML.
|
v
==================== STEP 1: PARSE META ROBOTS ====================
|
v
Find all <meta name="robots|googlebot|bingbot|...">
|
|---> Generic <meta name="robots"> applies to all crawlers
|
|---> Crawler specific (e.g., <meta name="googlebot">) overrides generic FOR THAT CRAWLER
|
|---> Multiple tags merge; most restrictive directive wins
|
v
==================== STEP 2: APPLY DIRECTIVES ====================
|
v
Does noindex apply?
YES to URL excluded from index (but body still parsed if needed for links)
NO to URL considered for indexing
|
v
Does nofollow apply?
YES to Links on the page not followed for crawl discovery
NO to Links followed normally
|
v
Snippet generation rules:
nosnippet to No snippet shown
max-snippet:N to Snippet capped at N characters
max-image-preview:X to Image preview size limited
max-video-preview:N to Video preview capped at N seconds
|
v
Other behaviors:
noarchive to No "cached" link in results
notranslate to No Google Translate offer
noimageindex to Images on page not indexed (but page still indexed)
unavailable_after:DATE to Drop from index after this date
indexifembedded to Index when embedded even if noindex
|
v
==================== STEP 3: RESULT ====================
|
v
URL processed per the combined directives.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-e4987062ab3d161b
HTTP/2 200 OK X-Robots-Tag: index <html> <head> <meta name="robots" content="noindex"> </head>
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-2afaa7d4a41575c6
Browser checks for theme color signal:
|
v
==================== PER BROWSER BEHAVIOR ====================
|
|---> Chrome Android: reads <meta name="theme-color">. Honors it.
| Picks media query variant matching system color scheme.
| Applies to address bar AND status bar.
|
|---> Chrome Desktop: reads <meta name="theme-color">. Limited honor.
| Affects tab color in some configurations.
|
|---> iOS 26 Safari (2025+): IGNORES <meta name="theme-color">.
| Uses body/html background color OR fixed element near top of viewport.
|
|---> iOS pre-26 Safari: read <meta name="theme-color"> with restrictions.
| Apple's prohibited color list applied.
| Only worked if "Show color in compact tab bar" enabled in Safari prefs.
|
|---> Firefox: limited support; treats theme-color informally.
|
|---> Brave, Edge: typically follows Chrome behavior.
|
|---> PWA mode (any OS): manifest.json theme_color preferred.
| Meta tag supplements but manifest is canonical.
==================== THE DECISION TREE ====================
|
v
For each Bubbles client site:
|
|---> 1. Define brand primary color.
|
|---> 2. Add <meta name="theme-color" content="#brand">.
|
|---> 3. Add dark mode variant if site supports dark theme.
|
|---> 4. Set body { background-color: #brand } in CSS (for iOS 26).
|
|---> 5. Add manifest.json theme_color if PWA capable.
|
|---> 6. Add iOS specific tags if PWA install enabled.
|
|---> 7. Verify against Apple's prohibited color list.
|
|---> 8. Test on real Chrome Android AND real iOS 26 Safari.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-51028ad5a486b532
[Notch / Status bar area = theme-color] [URL bar = theme-color] [Page content = whatever the page renders]
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-1b67528ce5175639
Brand colors:
Primary: #6B46C1 (purple)
Secondary: #1a1a1a (near black)
Accent: #88c70a (green for ThatDeveloperGuy)
Theme color implementation:
<meta name="theme-color" content="#6B46C1">
<meta name="theme-color" content="#0a0a0a" media="(prefers-color-scheme: dark)">
body { background-color: #6B46C1; }
@media (prefers-color-scheme: dark) { body { background-color: #0a0a0a; } }
PWA manifest:
theme_color: #6B46C1
background_color: #1a1a1a
Apple prohibited check: PASSED.HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-0058628a3c25bf84
Mobile browser with <meta name="viewport" content="width=device-width, initial-scale=1">: 1. Set viewport width to the device's CSS pixel width (e.g., 390 on iPhone 14). 2. Render the page at that width. 3. Display at 100% (no shrinking). 4. Result: layout is responsive to actual device width; text is full size.
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-46c999af2334dbb7
Is this a mobile responsive site?
|
|---> YES: width=device-width, initial-scale=1 (the canonical default)
|
|---> NO: do not include the viewport tag (let mobile do its default behavior)
|
v
==================== ADDITIONAL OPTIONS ====================
|
v
Does the design extend full bleed under iPhone notch?
|
|---> YES: add viewport-fit=cover
| AND use CSS env(safe-area-inset-*) for interactive elements
|
|---> NO: leave viewport-fit default (auto)
|
v
Does the page have forms with on screen keyboard interaction?
|
|---> YES: consider interactive-widget=resizes-content if layout
| should reflow under keyboard
|
|---> NO: default (resizes-visual) is fine
|
v
==================== ACCESSIBILITY GUARDRAILS ====================
|
v
NEVER include any of these (WCAG 1.4.4 violations):
- user-scalable=no
- maximum-scale=1 (or any value less than 2)
- minimum-scale=1 (locks initial zoom)HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-4db91abda2fdc835
Accessibility audit: ✗ [accessibility-2.21] [user-scalable="no"] is used in the <meta name="viewport"> element, or the [maximum-scale] attribute is less than 5.
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-c9b1f12060b283c6
LAYER 1: Required (4 properties on every page) | |---> og:title (the social preview headline) |---> og:type (website, article, video.*, music.*, profile, book) |---> og:image (the 1200x630 preview image) |---> og:url (the canonical URL of the page) LAYER 2: Recommended (3 properties on every page) | |---> og:description (the social preview body text) |---> og:site_name (the site identity displayed in the card) |---> og:locale (the language and region, e.g. en_US) LAYER 3: Image sub properties (4 to 5 properties on the og:image) | |---> og:image:width (pixel width, typically 1200) |---> og:image:height (pixel height, typically 630) |---> og:image:alt (accessibility description) |---> og:image:type (MIME type, e.g. image/jpeg) |---> og:image:secure_url (HTTPS URL when og:image is HTTP, rarely used in 2026) LAYER 4: Type specific (only when og:type matches) | |---> article:* properties when og:type=article (blog posts) |---> video:* properties when og:type=video.* |---> music:* properties when og:type=music.* |---> profile:* properties when og:type=profile |---> book:* properties when og:type=book LAYER 5: Multi language (when site serves multiple languages) | |---> og:locale (primary locale) |---> og:locale:alternate (each additional locale, repeatable)
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-fab9007165ce164d
Request arrives at server
|
v
==================== PROCESSING ====================
|
v
What just happened?
|
|---> Existing resource was returned in full
| |
| v
| 200 OK
| Body: the resource itself
| Indexable: YES (if content is substantive)
| Crawler: passes to indexing pipeline
|
|---> A new resource was just created
| |
| v
| 201 Created
| Body: representation of new resource (or empty)
| Header: Location pointing to new resource
| Crawler: waits for content, may show soft 404 in Search Console
|
|---> Request accepted, will process asynchronously
| |
| v
| 202 Accepted
| Body: status info (or empty)
| Header: Location pointing to status endpoint
| Crawler: waits for content, may show soft 404
|
|---> Success, no body to return
| |
| v
| 204 No Content
| Body: MUST be empty
| Crawler: signals indexing pipeline that no content was received
| Search Console: typically shows soft 404 for indexed URLs
|
|---> Partial response to Range request
| |
| v
| 206 Partial Content
| Body: requested byte range only
| Header: Content-Range describing the slice
| Crawler: not for indexing; this is mid stream contentHTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-5c11c86827fd0ee8
GET /video.mp4 HTTP/2 Range: bytes=1024-4096 ---- HTTP/2 206 Partial Content Content-Type: video/mp4 Content-Range: bytes 1024-4096/2097152 Content-Length: 3073 Accept-Ranges: bytes
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-ea636e6f48ef33c4
Range: bytes=0-1023 (request first 1024 bytes) Range: bytes=1024- (request from byte 1024 to end) Range: bytes=-1024 (request last 1024 bytes) Range: bytes=0-1023,2048-4095 (request multiple ranges) Content-Range: bytes 0-1023/47022 (delivered bytes 0 to 1023 of 47022 total) Content-Range: bytes */47022 (used in 416 response, file size only) Accept-Ranges: bytes (server supports byte range requests) Accept-Ranges: none (server does NOT support range requests)
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-f60b44ce077ef6e7
GET /video.mp4 HTTP/2 Range: bytes=1024-4096 If-Range: "abc123"
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-67e4c1e32e3a6d39
GET /style.css HTTP/2 If-None-Match: "abc123" ---- HTTP/2 304 Not Modified ETag: "abc123" Cache-Control: public, max-age=86400
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-9426db6d3f6ddf86
The request cannot be fulfilled. Whose fault is it?
|
Server's fault Client's fault
| |
v v
5xx codes (next framework) 4xx codes (this framework)
|
v
==================== WHAT DID THE CLIENT DO WRONG? ====================
|
+-------- The request itself is malformed --------+
| |
| Wire format wrong (bad JSON, missing field) ... 400 Bad Request
| |
| Wire format OK, data semantically invalid ... 422 Unprocessable Entity
| |
| HTTP method not allowed for this URL ........ 405 Method Not Allowed
| |
| Payload too large ........................... 413 Payload Too Large
| |
| Content type not supported .................. 415 Unsupported Media Type
| |
+--------------------------------------------------+
|
+-------- The client is not authorized --------+
| |
| No credentials provided ................. 401 Unauthorized
| |
| Credentials provided but insufficient .... 403 Forbidden
| |
+-----------------------------------------------+
|
+-------- The resource doesn't exist (or never will) --------+
| |
| Not currently here, might return ......................... 404 Not Found
| |
| Permanently gone, will never return ...................... 410 Gone
| |
+-------------------------------------------------------------+
|
+-------- The client is sending too fast --------+
| |
| Rate limit exceeded ........................ 429 Too Many Requests
| |
+-------------------------------------------------+
|
+-------- Legal reasons prevent serving --------+
|
Court order, DMCA, GDPR, etc ............... 451 Unavailable For Legal ReasonsHTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-502bf7aa9765d605
HTTP/2 401 Unauthorized
WWW-Authenticate: Bearer realm="api"
Content-Type: application/json
{"error": "auth_required", "detail": "please authenticate"}HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-e6689bee5ad6f313
WWW-Authenticate: Cookie realm="bubbles-session"
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-739a4a20cbf3edad
WWW-Authenticate: Bearer realm="api"
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-63affa206a3aca5f
A 5xx is being returned. Where does it originate?
|
|---> The upstream application (FastAPI sidecar) generated it
| |
| |---> Application threw an uncaught exception ............... 500 Internal Server Error
| |
| |---> Application is intentionally signaling unavailability . 503 Service Unavailable
| |
| |---> Application returned something nginx couldn't parse ... 502 Bad Gateway (rare from FastAPI; more from misconfigured backends)
|
|---> Nginx generated it because the upstream had a problem
| |
| |---> Connection refused (upstream not listening) ............ 502 Bad Gateway
| |
| |---> Upstream sent invalid HTTP response ..................... 502 Bad Gateway
| |
| |---> Upstream timed out (didn't respond in time) ............. 504 Gateway Timeout
| |
| |---> Upstream returned 503 (signaling unavailability) ........ 503 (passed through)
|
|---> The operator chose to signal maintenance
|
503 with Retry-After (the only "intentional" 5xx) ................. 503 Service Unavailable
==================== CRAWLER RECOVERY STRATEGY ====================
|
|---> Transient 5xx (rare, intermittent) ............... Crawlers retry; URL stays in index
|
|---> Sustained 5xx (hours to 2 days) ................... Crawlers reduce crawl rate
|
|---> 5xx beyond 2 days ................................ URLs drop from indexHTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-8d69ae1dd25b54e5
Cache-Control: public, max-age=31536000, immutable Cache-Control: private, max-age=0, must-revalidate Cache-Control: no-store Cache-Control: s-maxage=3600, max-age=60, stale-while-revalidate=30
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-b317384362ecd214
Cache-Control: max-age=31536000 Expires: Wed, 24 May 2027 14:00:00 GMT
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-2fb3552ec13c9284
Response arrives
|
v
Read Content-Encoding header
|
v
Is the body compressed?
| |
YES NO
| |
v v
Decompress using algorithm Skip decompression
in Content-Encoding
(gzip / br / zstd / deflate)
|
v
Read Content-Type header
|
v
What is this?
|
v
text/html -> parse as HTML, render in document context
text/css -> parse as stylesheet, only if <link rel=stylesheet>
application/javascript -> parse and execute as script
application/json -> parse as JSON
image/* -> decode and render
application/pdf -> render inline (PDF viewer) OR prompt to save
(decided by Content-Disposition)
|
v
Read Content-Disposition
|
v
inline? -> render in page context
attachment? -> prompt download with given filename
|
v
Read Content-Language (if set)
|
v
Pass to language tools, accessibility, search indexersHTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-43dae9a65802c9b9
Content-Type: text/html; charset=utf-8 Content-Type: application/json Content-Type: image/webp Content-Type: font/woff2 Content-Type: application/pdf
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-90cf3a75ccbf2326
Content-Type: text/html; charset=utf-8 Content-Type: application/javascript; charset=utf-8 Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-8cbb6d9c697bee59
Content-Type: text/html; charset=utf-8 Content-Type: application/javascript; charset=utf-8 Content-Type: application/json; charset=utf-8
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-e31f4c8fc8cc3ae7
Content-Disposition: inline Content-Disposition: attachment Content-Disposition: attachment; filename="report.pdf" Content-Disposition: attachment; filename="report.pdf"; filename*=UTF-8''report-%E2%82%AC.pdf
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-490ac36c29a1b0ac
Content-Disposition: <disposition-type> Content-Disposition: <disposition-type>; <parameter>=<value>
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-83f89e83914d7391
Content-Disposition: attachment; filename="resume.pdf"; filename*=UTF-8''r%C3%A9sum%C3%A9.pdf
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-c9420e9a4535e4a6
attachment; filename="My Document.pdf"
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-c1bea6cb88bf09e6
attachment; filename=My Document.pdf
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-74418742ca6fedf3
Content-Type: application/json; charset=utf-8 Content-Encoding: br Content-Length: 1247
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-539030ba392aef6a
Server-Timing: db;dur=12.4 Server-Timing: db;dur=12.4, render;dur=8.2, cache;desc=hit Server-Timing: total;dur=187;desc="full request"
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-f274f4a06571eb56
Server-Timing: <name>[;dur=<duration>][;desc=<description>][, ...]
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-216c016375099020
Server-Timing: db;dur=12 Server-Timing: db;dur=12;desc="user lookup" Server-Timing: db;dur=12, cpu;dur=45 Server-Timing: cache;desc=miss Server-Timing: total;dur=187
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-3aff0a29e34b54aa
Server-Timing: db;dur=12.4;desc="user lookup", render;dur=8.2, cache;desc=miss, total;dur=24.1
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-e183cf2d910e3c0c
Alt-Svc: h3=":443"; ma=86400 Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 Alt-Svc: clear
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-de493e9263adbab2
Alt-Svc: <protocol>=<authority>; ma=<seconds> [; persist=1] Alt-Svc: clear
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.
scan-c4bb89baeb347f51
# Advertise HTTP/3 on the same port for 1 day Alt-Svc: h3=":443"; ma=86400 # Same as above, persistent across network changes Alt-Svc: h3=":443"; ma=86400; persist=1 # Multiple HTTP/3 versions for compatibility Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 # Clear all alternatives (useful when disabling HTTP/3) Alt-Svc: clear
HTTP, HTML or crawler-configuration example; equality signs identify attributes or protocol settings.