robots_inspect

Fetch only the origin /robots.txt for 2 sats. Redirect targets and sitemap contents are not fetched. Robots directives are crawler guidance, not access control. Crawl-delay support varies by crawler.

{
  "target": "https://example.com"
}