Crawler Lens

Pick a real crawler user-agent from this site's robots.txt and a page from the sitemap. This tool live-fetches robots.txt, the page's raw HTML and llms.txt from this same origin and reports the actual access verdict and extracted signal — title, meta robots, canonical, JSON-LD entities and plain-text content — exactly as that bot would receive it. No mocked responses.

What this actually checks

Access verdicts are computed with the standard longest-matching-rule algorithm (ties resolve to Allow) against whichever User-agent block matches, falling back to User-agent: *. Extracted signal is parsed straight out of the fetched HTML with DOMParser — the same raw markup a non-rendering crawler receives, since this site ships zero client-side rendering. If a fetch fails, it's most likely because this page isn't being served over HTTP (same-origin fetches don't resolve from a local file:// preview).