robots.txt Testing & Validation Tool
Uses the Google Robots.txt Parser and Matcher Library to check a list of URLs against the live, or a custom, robots.txt file one to see if they are allowed, or blocked and if so, by what rule.
The tools main aim is to see how your robots.txt works against URLs, not test the robots.txt itself, but it will validate the rules in a robots.txt file, and give you feedback if there's anything in a robots.txt file the parser doesn't understand. But in general, entering a URL of a robots.txt file will not give you a useful result.
If you want to test changes before you publish your robots.txt file, set the Use Live robots.txt? control to 'Custom' and try changes to your robots.txt.
Parsing and matching is one part of the picture, sometimes a search engine or other service might choose to fall back or ignore certain rules, like Googlebot-Image falling back to Googlebot rules if no specific User-agent rule is found. This tool attempts to mimic the behaviour here for Google and Applebot.
If using a live robots.txt to test, you need to enter full, qualified URLs, including http /
https. You can test multiple URLs at once, from different domains / sites, just enter one on
each line. If using a custom robots.txt file, you don't need to enter full urls, you can use a
url like /foo or https://example.com/foo.
New! Get the Chrome Extension
Get the Chrome Extension to quickly access this tool quicker with a simple right-click. Grab it from the Chrome Store
Using: Live robots.txt | User-agent: Googlebot
One per line, should start with http / https, 0 entered
Settings
(will overwrite any custom rules)
Custom robots.txt will apply to all URLs, regardless of origin example:
| URL | Live robots.txt | Custom robots.txt |
|---|---|---|
| https://example.com/test | https://example.com/robots.txt | the custom robots.txt |
| https://www.differentexample.com/some/test/product.html | https://differentexample.com/robots.txt | the custom robots.txt |
Install as MCP Server - NEW!
You can install this tool as a Model Context Protocol (MCP) server. This is currently in Beta, and may go away or change in the future, depending on usage, and costs to run.
To install this tool as an MCP server, follow the instructions in the documentation of your favoured tool. Here's some guides for popular tools.
The MCP server Connection details will look like this:
{
"mcpServers": {
"tamethebots-robotstxt": {
"command": "npx",
"args": [
"mcp-remote",
"https://robotstxt-mcp.tamethebots.com/sse"
]
}
}
}After Installing, you could ask it to check a URL by sending a request to the MCP server, for example:
Is https://example.com/ blocked by robots.txt?Does https://www.example.com/ use robots.txt to block Bingbot?
Example:
Why Test Robots.txt?
robots.txt files can be complex, and it's easy to make mistakes that can lead to URLs being blocked when you don't expect them to be, or allowed when you do expect them to be blocked. Testing your robots.txt file against a list of URLs can help you catch these issues before they become a problem.
Even if you think your robots.txt file is correct, it's worth testing it against a list of URLs to see how it works in practice, and to catch any edge cases or unexpected behaviour, which would prevent search engines and other services, like LLMs, from crawling, indexing or otherwise understanding your content as you expect.
I recommend testing:
- Changes you want to make to your robots.txt file before you publish them, by using the custom robots.txt option.
- Your live robots.txt file after making any changes.
- When you add new sections, or featrures to your site, to make sure your robots.txt file is working as you expect with the new content.
Along with occasional checks to ensure your robots.txt file continues to work as expected, and changes haven't been missed.
Oh no, my robots.txt file was wrong!
Naturally the first step it to publish a corrected version of your robots.txt file, make sure to clear any caches, including CDN caches if applicable. Then test it again.
Also be aware that different services might cache your robots.txt file, so it might take some time for changes to take effect. Google for example can cache your robots.txt file for up to 48 hours. Some consumers of robots.txt files might give you the option to request a recrawl of your robots.txt file, which can help speed up the process, for example, Google offers this in Search Console's robots.txt report.
Brought to you in conjunction with Jamie Indigo of Not a Robot, find them on LinkedIn.
Find This Useful?
You can help support the cost of these free tools by sponsoring me on github:
Or buy me a coffee at Kofi
