robots.txt Testing & Validation Tool
Uses the Google Robots.txt Parser and Matcher Library to check a list of URLs against the live, or a custom, robots.txt file one to see if they are allowed, or blocked and if so, by what rule.
The tools main aim is to see how your robots.txt works against URLs, not test the robots.txt itself, but it will validate the rules in a robots.txt file, and give you feedback if there's anything in a robots.txt file the parser doesn't understand. But in general, entering a URL of a robots.txt file will not give you a useful result.
If you want to test changes before you publish your robots.txt file, set the Use Live robots.txt? control to 'Custom' and try changes to your robots.txt.
Parsing and matching is one part of the picture, sometimes a search engine or other service might choose to fall back or ignore certain rules, like Googlebot-Image falling back to Googlebot rules if no specific User-agent rule is found. This tool attempts to mimic the behaviour here for Google and Applebot.
If using a live robots.txt to test, you need to enter full, qualified URLs, including http /
https. If using a custom robots.txt file, you don't need to enter full urls, you can use a url
like /foo or https://example.com/foo
New! Get the Chrome Extension
Get the Chrome Extension to quickly access this tool quicker with a simple right-click. Grab it from the Chrome Store
Using: Live robots.txt | User-agent: Googlebot
One per line, should start with http / https, 0 entered
Settings
(will overwrite any custom rules)
Custom robots.txt will apply to all URLs, regardless of origin example:
| URL | Live robots.txt | Custom robots.txt |
|---|---|---|
| https://example.com/test | https://example.com/robots.txt | the custom robots.txt |
| https://www.differentexample.com/some/test/product.html | https://differentexample.com/robots.txt | the custom robots.txt |
Brought to you in conjunction with Jamie Indigo of Not a Robot, find them on LinkedIn.
Find This Useful?
You can help support the cost of these free tools by sponsoring me on github:
Or buy me a coffee at Kofi
