Webmaster
robots.txt tester
Enter a URL to see whether crawlers may fetch it, which line of robots.txt decides, and what’s wrong with the file.
Allowed · GPTBot → /help/dns
Allowed by “Allow: /” (line 2) in the group for *.
Search and AI crawlers on this path
| Crawler | Who | Result |
|---|---|---|
| Googlebot | Google Search | allowed |
| Bingbot | Bing | allowed |
| DuckDuckBot | DuckDuckGo | allowed |
| Applebot | Apple | allowed |
| GPTBot | OpenAI training | allowed |
| OAI-SearchBot | ChatGPT search | allowed |
| ClaudeBot | Anthropic | allowed |
| Google-Extended | Google AI training | allowed |
| PerplexityBot | Perplexity | allowed |
| CCBot | Common Crawl | allowed |
| Bytespider | ByteDance | allowed |
| Meta-ExternalAgent | Meta AI | allowed |
Sitemaps
The file https://tools.nexayori.com/robots.txt · 0.1 KB
1User-agent: * 2Allow: / 3Disallow: /admin 4 5Sitemap: https://tools.nexayori.com/sitemap.xml 6