Free tool
Find crawl-policy problems with a robots TXT checker.
Check robots.txt policies for your URL alongside a neutral server request. Review search and training permissions separately before making changes.
This robots TXT checker compares a URL with policies for nine bot names and makes one neutral reachability request. It does not verify requests from real crawlers, test every firewall rule or predict AI recommendations.
Bot purposes differ. See OpenAI crawler documentation and Google crawler controls before changing permissions.
What happens when you start
Turn a diagnostic into a repeatable improvement workflow.
Add your business context and review suggested services, markets and competitors.
Choose relevant questions and engines within your plan. Review scan evidence and the recommended next action.
Review and implement appropriate fixes, then recheck the finding and compare later visibility observations.
Read the policy result before changing access
| Result | What it means | Next action |
|---|---|---|
| Allow and successful HTTP | The parsed policy permits the path and a neutral request responded | Use logs to verify actual crawler access |
| Disallow | The parsed robots policy restricts the path for that bot name | Confirm whether that restriction is intentional |
| HTTP error | The neutral request failed or returned an error | Check the URL, redirects and server/CDN logs |
| Unknown policy | The robots file was not retrieved | Review the file directly; do not treat unknown as allowed |
Check the page path, not only the homepage
Enter the exact public URL you want inspected. A rule can allow the homepage while restricting a directory. Open the linked robots file, review the relevant user-agent group and path directives, and repeat the check after a reviewed change. This is a lightweight policy diagnostic, not a complete robots syntax validator.
Search access and training permission are separate
OAI-SearchBot concerns OpenAI search, while GPTBot concerns training. User-triggered retrieval has different behaviour again. Blocking training does not by itself establish that ChatGPT search cannot find your site. Choose permissions for the purpose you intend; do not blanket-allow every bot to chase a green score.
Google-Extended is a control token, not a separate crawler user-agent. Check the current provider documentation before changing policy. A robots file is not access control for private information.
From a policy check to a useful fix
Review the detailed table, identify unintended restrictions and use the copyable prompt to brief a small, specific change. Preserve private paths and unrelated rules. Publish the reviewed edit, recheck the URL and inspect actual server logs. Then use a page audit for content readiness or visibility monitoring for observed mentions.
Frequently asked questions
Does this checker make requests as each AI crawler?
Does an allowed result guarantee ChatGPT will cite the page?
Is the robots TXT checker free?
Stop guessing. Start measuring.
Track selected buyer questions and inspect where AI names a competitor. Coverage and frequency depend on your plan.
Scan my site freeSee how AI sees your site.
Start with a limited free visibility preview, then choose the coverage and workflow your business needs.