AI visibility

AI Crawler robots.txt Generator

Generate allow/block robots.txt rules for common AI crawlers with an explainable policy summary.

generated

AI crawler robots.txt policy

Allow policy generated for 8 crawler(s).

Findings

Policy modeallow

Recommended next steps

  1. Only publish policies that match your business intent.
  2. Add a Sitemap directive and keep llms.txt consistent with robots.txt.
  3. Review crawler documentation periodically.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot
Allow: /
Show technical details
{
  "title": "AI crawler robots.txt policy",
  "summary": "Allow policy generated for 8 crawler(s).",
  "status": "generated",
  "findings": [
    {
      "label": "Policy mode",
      "severity": "info",
      "detail": "allow"
    }
  ],
  "recommendations": [
    "Only publish policies that match your business intent.",
    "Add a Sitemap directive and keep llms.txt consistent with robots.txt.",
    "Review crawler documentation periodically."
  ],
  "details": {
    "robotsTxtSnippet": "User-agent: GPTBot\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nUser-agent: CCBot\nAllow: /\n\nUser-agent: Applebot\nAllow: /"
  }
}