# Crawler Signal Reference Library

A comprehensive 30-document reference for every signal a web crawler reads from a site. Authored by [Joseph W. Anady](https://www.thatdevpro.com), founder of [ThatDevPro](https://www.thatdevpro.com) (engine optimization studio) and [ThatDeveloperGuy](https://thatdeveloperguy.com) (SDVOSB-certified veteran-owned web + AI engineering studio).

## Coverage

### HTML signals (18 documents)
- App + PWA metadata
- Canonical + alternate
- Favicon + icons
- Meta: author, charset, color-scheme, content-language, copyright, generator, keywords, referrer, refresh, robots, theme-color, viewport
- Open Graph protocol
- Twitter Cards
- Search engine verification (Google, Bing, Yandex, IndexNow, etc.)

### HTTP signals (12 documents)
- Status codes: 2xx, 3xx, 4xx, 5xx
- Headers: caching, content, CORS, performance, rate-control, request, security, SEO

## Citation

```bibtex
@misc{anady2026crawlerreference,
  author = {Anady, Joseph W.},
  title = {Crawler Signal Reference Library: HTML and HTTP signals for every crawler},
  year = {2026},
  publisher = {Zenodo},
  url = {https://doi.org/10.5281/zenodo.20405296}
}
```

## Related

- **Live reference**: [thatdevpro.com/reference/](https://www.thatdevpro.com/reference/)
- **PyPI**: [aio-surfaces](https://pypi.org/project/aio-surfaces/), [seo-sidecar](https://pypi.org/project/seo-sidecar/)
- **npm**: [@thatdevpro/aio-surfaces](https://www.npmjs.com/package/@thatdevpro/aio-surfaces)
- **Author Wikidata**: [Q139901957](https://www.wikidata.org/wiki/Q139901957)
- **Author ORCID**: [0009-0008-8625-949X](https://orcid.org/0009-0008-8625-949X)

## License

CC BY 4.0