Webmaster
robots.txt tester
Enter a URL to see whether crawlers may fetch it, which line of robots.txt decides, and what’s wrong with the file.
Allowed · GPTBot → /customer-stories/knock
No rule matches in the group for gptbot, oai-searchbot, claudebot, anthropic-ai, perplexitybot: allowed.
Search and AI crawlers on this path
| Crawler | Who | Result |
|---|---|---|
| Googlebot | Google Search | allowed |
| Bingbot | Bing | allowed |
| DuckDuckBot | DuckDuckGo | allowed |
| Applebot | Apple | allowed |
| GPTBot | OpenAI training | allowed |
| OAI-SearchBot | ChatGPT search | allowed |
| ClaudeBot | Anthropic | allowed |
| Google-Extended | Google AI training | allowed |
| PerplexityBot | Perplexity | allowed |
| CCBot | Common Crawl | allowed |
| Bytespider | ByteDance | blocked |
| Meta-ExternalAgent | Meta AI | allowed |
Notes
- iCrawl-delay is set. Bing respects it; Google ignores it.
Sitemaps
The file https://github.com/robots.txt · 14.2 KB
1# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots 2# We also provide an extensive API: https://docs.github.com 3Sitemap: https://github.com/sitemap_marketing.xml 4 5User-agent: GPTBot 6User-agent: OAI-SearchBot 7User-agent: ClaudeBot 8User-agent: anthropic-ai 9User-agent: PerplexityBot 10Crawl-delay: 1 11Allow: /*?tab=achievements&achievement=* 12Allow: /marketplace/*?tab=readme$ 13Allow: /marketplace/*?tab=transparency$ 14Disallow: /account-login 15Disallow: /copilot/ 16Disallow: /copilot/c/ 17Disallow: /ekansa/Open-Context-Data 18Disallow: /ekansa/opencontext-* 19Disallow: /Explodingstuff/ 20Disallow: /gist/ 21Disallow: /search/advanced 22Disallow: /search$ 23Disallow: /*/*/pulse_committer_data.json$ 24Disallow: /*/*/pulse_committer_data.json?* 25Disallow: /*/archive/ 26Disallow: /*/blame/ 27Disallow: /*/cache/ 28Disallow: /*/raw/ 29Disallow: /*/tree/ 30Disallow: */tarball/ 31Disallow: */zipball/ 32Disallow: /.git/ 33Disallow: */.git/ 34Disallow: /*.git$ 35Disallow: /*.atom$ 36Disallow: /*report-abuse?report=* 37Disallow: /*/*/commits/ 38Disallow: /marketplace/*/*?tab=readme$ 39Disallow: /marketplace/*/*?tab=transparency$ 40Disallow: /marketplace/*?*?tab=readme$ 41Disallow: /marketplace/*?*?tab=transparency$ 42Disallow: /marketplace/actions?tab=readme$ 43Disallow: /marketplace/actions?tab=transparency$ 44Disallow: /marketplace/free-trials?tab=readme$ 45Disallow: /marketplace/free-trials?tab=transparency$ 46Disallow: /marketplace/manage?tab=readme$ 47Disallow: /marketplace/manage?tab=transparency$ 48Disallow: /marketplace/models?tab=readme$ 49Disallow: /marketplace/models?tab=transparency$ 50Disallow: /marketplace/new?tab=readme$ 51Disallow: /marketplace/new?tab=transparency$ 52Disallow: /marketplace/publishers?tab=readme$ 53Disallow: /marketplace/publishers?tab=transparency$ 54Disallow: /marketplace/recently-added?tab=readme$ 55Disallow: /marketplace/recently-added?tab=transparency$ 56Disallow: /biztools$ 57Disallow: /biztools?* 58Disallow: /biztools/* 59Disallow: /businesses$ 60Disallow: /businesses?* 61Disallow: /businesses/* 62Disallow: /enterprises$ 63Disallow: /enterprises?* 64Disallow: /enterprises/* 65Disallow: /organizations$ 66Disallow: /organizations?* 67Disallow: /organizations/* 68Disallow: /settings$ 69Disallow: /settings?* 70Disallow: /settings/* 71Disallow: /stafftools$ 72Disallow: /stafftools?* 73Disallow: /stafftools/* 74Disallow: /users/*/billing$ 75Disallow: /users/*/billing?* 76Disallow: /users/*/billing/* 77Disallow: /writing$ 78Disallow: /writing?* 79Disallow: /writing/* 80Disallow: /*/*/branches$ 81Disallow: /*/*/branches?* 82Disallow: /*/*/branches/* 83Disallow: /*/*/comments$ 84Disallow: /*/*/comments?* 85Disallow: /*/*/comments/* 86Disallow: /*/*/compare$ 87Disallow: /*/*/compare?* 88Disallow: /*/*/compare/* 89Disallow: /*/*/contributors$ 90Disallow: /*/*/contributors?* 91Disallow: /*/*/contributors/* 92Disallow: /*/*/contributors_list$ 93Disallow: /*/*/contributors_list?* 94Disallow: /*/*/contributors_list/* 95Disallow: /*/*/contributors-list$ 96Disallow: /*/*/contributors-list?* 97Disallow: /*/*/contributors-list/* 98Disallow: /*/*/download$ 99Disallow: /*/*/download?* 100Disallow: /*/*/download/* 101Disallow: /*/*/forks$ 102Disallow: /*/*/forks?* 103Disallow: /*/*/forks/* 104Disallow: /*/*/graphs$ 105Disallow: /*/*/graphs?* 106Disallow: /*/*/graphs/* 107Disallow: /*/*/issues/new$ 108Disallow: /*/*/issues/new?* 109Disallow: /*/*/issues/new/* 110Disallow: /*/*/issues/search$ 111Disallow: /*/*/issues/search?* 112Disallow: /*/*/issues/search/* 113Disallow: /*/*/milestones/new$ 114Disallow: /*/*/milestones/new?* 115Disallow: /*/*/milestones/new/* 116Disallow: /*/*/network$ 117Disallow: /*/*/network?* 118Disallow: /*/*/network/* 119Disallow: /*/*/projects$ 120Disallow: /*/*/projects?* 121Disallow: /*/*/projects/* 122Disallow: /*/*/pulse$ 123Disallow: /*/*/pulse?* 124Disallow: /*/*/pulse/* 125Disallow: /*/*/pulse-new$ 126Disallow: /*/*/pulse-new?* 127Disallow: /*/*/pulse-new/* 128Disallow: /*/*/pulse_committer_data$ 129Disallow: /*/*/pulse_committer_data?* 130Disallow: /*/*/pulse_committer_data/* 131Disallow: /*/*/pulse_diffstat_summary$ 132Disallow: /*/*/pulse_diffstat_summary?* 133Disallow: /*/*/pulse_diffstat_summary/* 134Disallow: /*/*/revisions$ 135Disallow: /*/*/revisions?* 136Disallow: /*/*/revisions/* 137Disallow: /*/*/stargazers$ 138Disallow: /*/*/stargazers?* 139Disallow: /*/*/stargazers/* 140Disallow: /*/*/tags$ 141Disallow: /*/*/tags?* 142Disallow: /*/*/tags/* 143Disallow: /*/*/watchers$ 144Disallow: /*/*/watchers?* 145Disallow: /*/*/watchers/* 146Disallow: /*?source=* 147Disallow: /*&source=* 148Disallow: /*?ref_cta=* 149Disallow: /*&ref_cta=* 150Disallow: /*?plan=* 151Disallow: /*&plan=* 152Disallow: /*?return_to=* 153Disallow: /*&return_to=* 154Disallow: /*?ref_loc=* 155Disallow: /*&ref_loc=* 156Disallow: /*?setup_organization=* 157Disallow: /*&setup_organization=* 158Disallow: /*?source_repo=* 159Disallow: /*&source_repo=* 160Disallow: /*?ref_page=* 161Disallow: /*&ref_page=* 162Disallow: /*?referrer=* 163Disallow: /*&referrer=* 164Disallow: /*?report=* 165Disallow: /*&report=* 166Disallow: /*?author=* 167Disallow: /*&author=* 168Disallow: /*?since=* 169Disallow: /*&since=* 170Disallow: /*?until=* 171Disallow: /*&until=* 172Disallow: /*?tab=* 173Disallow: /*&tab=* 174Disallow: /*?q=* 175Disallow: /*&q=* 176 177User-agent: Bytespider 178Disallow: / 179 180User-agent: AdsBot-Google 181User-agent: AdsBot-Google-Mobile 182Disallow: /account-login 183Disallow: /copilot/ 184Disallow: /copilot/c/ 185Disallow: /ekansa/Open-Context-Data 186Disallow: /ekansa/opencontext-* 187Disallow: /Explodingstuff/ 188Disallow: /gist/ 189Disallow: /search/advanced 190Disallow: /search$ 191Disallow: /*/*/pulse_committer_data.json$ 192Disallow: /*/*/pulse_committer_data.json?* 193Disallow: /*/archive/ 194Disallow: /*/blame/ 195Disallow: /*/cache/ 196Disallow: /*/raw/ 197Disallow: /*/tree/ 198Disallow: */tarball/ 199Disallow: */zipball/ 200Disallow: /.git/ 201Disallow: */.git/ 202Disallow: /*.git$ 203Disallow: /*.atom$ 204Disallow: /*report-abuse?report=* 205Disallow: /*/*/commits/ 206Disallow: /marketplace/*/*?tab=readme$ 207Disallow: /marketplace/*/*?tab=transparency$ 208Disallow: /marketplace/*?*?tab=readme$ 209Disallow: /marketplace/*?*?tab=transparency$ 210Disallow: /marketplace/actions?tab=readme$ 211Disallow: /marketplace/actions?tab=transparency$ 212Disallow: /marketplace/free-trials?tab=readme$ 213Disallow: /marketplace/free-trials?tab=transparency$ 214Disallow: /marketplace/manage?tab=readme$ 215Disallow: /marketplace/manage?tab=transparency$ 216Disallow: /marketplace/models?tab=readme$ 217Disallow: /marketplace/models?tab=transparency$ 218Disallow: /marketplace/new?tab=readme$ 219Disallow: /marketplace/new?tab=transparency$ 220Disallow: /marketplace/publishers?tab=readme$ 221Disallow: /marketplace/publishers?tab=transparency$ 222Disallow: /marketplace/recently-added?tab=readme$ 223Disallow: /marketplace/recently-added?tab=transparency$ 224Disallow: /biztools$ 225Disallow: /biztools?* 226Disallow: /biztools/* 227Disallow: /businesses$ 228Disallow: /businesses?* 229Disallow: /businesses/* 230Disallow: /enterprises$ 231Disallow: /enterprises?* 232Disallow: /enterprises/* 233Disallow: /organizations$ 234Disallow: /organizations?* 235Disallow: /organizations/* 236Disallow: /settings$ 237Disallow: /settings?* 238Disallow: /settings/* 239Disallow: /stafftools$ 240Disallow: /stafftools?* 241Disallow: /stafftools/* 242Disallow: /users/*/billing$ 243Disallow: /users/*/billing?* 244Disallow: /users/*/billing/* 245Disallow: /writing$ 246Disallow: /writing?* 247Disallow: /writing/* 248Disallow: /*/*/branches$ 249Disallow: /*/*/branches?* 250Disallow: /*/*/branches/* 251Disallow: /*/*/comments$ 252Disallow: /*/*/comments?* 253Disallow: /*/*/comments/* 254Disallow: /*/*/compare$ 255Disallow: /*/*/compare?* 256Disallow: /*/*/compare/* 257Disallow: /*/*/contributors$ 258Disallow: /*/*/contributors?* 259Disallow: /*/*/contributors/* 260Disallow: /*/*/contributors_list$ 261Disallow: /*/*/contributors_list?* 262Disallow: /*/*/contributors_list/* 263Disallow: /*/*/contributors-list$ 264Disallow: /*/*/contributors-list?* 265Disallow: /*/*/contributors-list/* 266Disallow: /*/*/download$ 267Disallow: /*/*/download?* 268Disallow: /*/*/download/* 269Disallow: /*/*/forks$ 270Disallow: /*/*/forks?* 271Disallow: /*/*/forks/* 272Disallow: /*/*/graphs$ 273Disallow: /*/*/graphs?* 274Disallow: /*/*/graphs/* 275Disallow: /*/*/issues/new$ 276Disallow: /*/*/issues/new?* 277Disallow: /*/*/issues/new/* 278Disallow: /*/*/issues/search$ 279Disallow: /*/*/issues/search?* 280Disallow: /*/*/issues/search/* 281Disallow: /*/*/milestones/new$ 282Disallow: /*/*/milestones/new?* 283Disallow: /*/*/milestones/new/* 284Disallow: /*/*/network$ 285Disallow: /*/*/network?* 286Disallow: /*/*/network/* 287Disallow: /*/*/projects$ 288Disallow: /*/*/projects?* 289Disallow: /*/*/projects/* 290Disallow: /*/*/pulse$ 291Disallow: /*/*/pulse?* 292Disallow: /*/*/pulse/* 293Disallow: /*/*/pulse-new$ 294Disallow: /*/*/pulse-new?* 295Disallow: /*/*/pulse-new/* 296Disallow: /*/*/pulse_committer_data$ 297Disallow: /*/*/pulse_committer_data?* 298Disallow: /*/*/pulse_committer_data/* 299Disallow: /*/*/pulse_diffstat_summary$ 300Disallow: /*/*/pulse_diffstat_summary?* 301Disallow: /*/*/pulse_diffstat_summary/* 302Disallow: /*/*/revisions$ 303Disallow: /*/*/revisions?* 304Disallow: /*/*/revisions/* 305Disallow: /*/*/stargazers$ 306Disallow: /*/*/stargazers?* 307Disallow: /*/*/stargazers/* 308Disallow: /*/*/tags$ 309Disallow: /*/*/tags?* 310Disallow: /*/*/tags/* 311Disallow: /*/*/watchers$ 312Disallow: /*/*/watchers?* 313Disallow: /*/*/watchers/* 314Disallow: /*?source=* 315Disallow: /*&source=* 316Disallow: /*?ref_cta=* 317Disallow: /*&ref_cta=* 318Disallow: /*?plan=* 319Disallow: /*&plan=* 320Disallow: /*?return_to=* 321Disallow: /*&return_to=* 322Disallow: /*?ref_loc=* 323Disallow: /*&ref_loc=* 324Disallow: /*?setup_organization=* 325Disallow: /*&setup_organization=* 326Disallow: /*?source_repo=* 327Disallow: /*&source_repo=* 328Disallow: /*?ref_page=* 329Disallow: /*&ref_page=* 330Disallow: /*?referrer=* 331Disallow: /*&referrer=* 332Disallow: /*?report=* 333Disallow: /*&report=* 334Disallow: /*?author=* 335Disallow: /*&author=* 336Disallow: /*?since=* 337Disallow: /*&since=* 338Disallow: /*?until=* 339Disallow: /*&until=* 340Disallow: /*?tab=* 341Disallow: /*&tab=* 342Disallow: /*?q=* 343Disallow: /*&q=* 344 345User-agent: bingbot 346User-agent: adidxbot 347User-agent: BingPreview 348Disallow: /ekansa/Open-Context-Data 349Disallow: /ekansa/opencontext-* 350Disallow: /account-login 351Disallow: */tarball/ 352Disallow: */zipball/ 353Disallow: /Explodingstuff/ 354Disallow: /copilot/ 355Disallow: /copilot/c/ 356 357User-agent: baidu 358crawl-delay: 1 359 360User-agent: * 361Allow: /*?tab=achievements&achievement=* 362Allow: /marketplace/*?tab=readme$ 363Allow: /marketplace/*?tab=transparency$ 364Disallow: /account-login 365Disallow: /copilot/ 366Disallow: /copilot/c/ 367Disallow: /ekansa/Open-Context-Data 368Disallow: /ekansa/opencontext-* 369Disallow: /Explodingstuff/ 370Disallow: /gist/ 371Disallow: /search/advanced 372Disallow: /search$ 373Disallow: /*/*/pulse_committer_data.json$ 374Disallow: /*/*/pulse_committer_data.json?* 375Disallow: /*/archive/ 376Disallow: /*/blame/ 377Disallow: /*/cache/ 378Disallow: /*/raw/ 379Disallow: /*/tree/ 380Disallow: */tarball/ 381Disallow: */zipball/ 382Disallow: /.git/ 383Disallow: */.git/ 384Disallow: /*.git$ 385Disallow: /*.atom$ 386Disallow: /*report-abuse?report=* 387Disallow: /*/*/commits/ 388Disallow: /marketplace/*/*?tab=readme$ 389Disallow: /marketplace/*/*?tab=transparency$ 390Disallow: /marketplace/*?*?tab=readme$ 391Disallow: /marketplace/*?*?tab=transparency$ 392Disallow: /marketplace/actions?tab=readme$ 393Disallow: /marketplace/actions?tab=transparency$ 394Disallow: /marketplace/free-trials?tab=readme$ 395Disallow: /marketplace/free-trials?tab=transparency$ 396Disallow: /marketplace/manage?tab=readme$ 397Disallow: /marketplace/manage?tab=transparency$ 398Disallow: /marketplace/models?tab=readme$ 399Disallow: /marketplace/models?tab=transparency$ 400Disallow: /marketplace/new?tab=readme$