Free, privacy-first
Last updated:
Check how a specific bot will read your robots.txt rules for any path before you ship a crawl-control change.
What this tool helps you answer
Use this tool when you need a per-URL crawl decision for a specific bot: before deploying a robots.txt change, after a CMS migration, or when a crawler is behaving unexpectedly on a path you thought was open.
The result for each URL shows three things: the matched user-agent group, the winning rule, and the final allow or disallow verdict. A clean allow result means the URL passes the robots check. A disallow result with a specific winning rule shows exactly which directive is blocking the path. If no rule matched at all, the URL is implicitly allowed.
Next step
Check how a specific bot will read your robots.txt rules for any path before you ship a crawl-control change.
Editorial review
This page combines the live tool, input guidance, worked examples, and operating limits so Robots.txt Tester stays useful even before users interact with the calculator.
Reviewed by Klartext Tools against the current Robots.txt Tester workflow on 2026-03-06.
Last updated:
Use with judgment
Page scope
A common setup that blocks an admin area but still allows a specific help path and a Googlebot preview path.
Useful for seeing how a more specific allow rule can override a broader disallow pattern.
Change the user-agent after loading the example to compare how the verdict changes for other crawlers.
A rule set that only works as expected when query strings are included in the matching logic.
Good for checking whether query-string handling changes a crawl decision for search or checkout URLs.
Switch match mode back to path only after loading the example to see how the verdict changes.
Paste the robots.txt content you want to test: not just the relevant section, but the whole file so that group interactions are resolved correctly.
Paste the full robots.txt content into the input area.
Enter the user-agent name you want to test, such as Googlebot or a custom crawler token.
Add the URL paths to check, one per line: use full URLs or path-only strings.
Choose strict or permissive syntax mode and whether to include query strings in matching.
Run the tester and check the matched group, winning rule, and allow or disallow verdict for each path.
Load a simple crawl-control file or a query-sensitive rule set to test precedence, allow overrides, and user-agent targeting.
A common setup that blocks an admin area but still allows a specific help path and a Googlebot preview path.
Sample inputs
Sample outcome: Useful for seeing how a more specific allow rule can override a broader disallow pattern.
Change the user-agent after loading the example to compare how the verdict changes for other crawlers.
A rule set that only works as expected when query strings are included in the matching logic.
Sample inputs
Sample outcome: Good for checking whether query-string handling changes a crawl decision for search or checkout URLs.
Switch match mode back to path only after loading the example to see how the verdict changes.
A robots.txt rule that looks correct in isolation can be overridden by a more specific directive elsewhere in the file, and the override is silent. User-agent specificity, path length precedence, and the allow-versus-disallow interaction all combine in ways that are not always intuitive. This tester applies the same deterministic precedence logic that major crawlers use, so you can verify the actual access decision for a specific bot and URL before a change goes live.
Paste your robots.txt content, enter the crawler name you want to test, and provide one or more URL paths. The tester evaluates which user-agent group applies, finds the most specific matching directive, and returns an allow or disallow verdict for each URL: along with the winning rule and the group that governed it.
When multiple rules could match a URL, the longest path match wins regardless of rule order. A specific allow directive for /admin/help/ beats a broader disallow for /admin/ because it is longer. If two rules have the same path length, allow takes precedence over disallow. User-agent-specific groups take priority over the wildcard (*) group.
The tester first looks for a group that names the exact user-agent you entered (case-insensitive). If no named group exists, it falls back to the wildcard (*) group. A bot that matches a named group is not also governed by the wildcard group: only the most specific match applies.
The result for each URL shows three things: the matched user-agent group, the winning rule, and the final allow or disallow verdict. A clean allow result means the URL passes the robots check. A disallow result with a specific winning rule shows exactly which directive is blocking the path. If no rule matched at all, the URL is implicitly allowed.
Strict mode flags directives that do not conform precisely to the robots.txt specification: useful when you want to catch edge cases that some crawlers might reject. Permissive mode applies a more lenient parser that accepts common informal patterns. Use strict mode before deploying changes to a production file; use permissive mode when analyzing an existing file you did not write.
By default the tester matches on the URL path only, ignoring query strings: which is how most major crawlers behave. Switching to path-and-query mode includes the full query string in the pattern match, which is relevant if your robots.txt uses directives like Disallow: /search?session=. Test both modes when your rules include query-sensitive patterns.
Most launch robots mistakes are avoidable. The problem is not that robots.txt is hard. The problem is that teams review it too late, test too little, or confuse a few working paths with a safe crawl policy.
Hreflang errors are expensive because they waste localization work after launch. A multilingual release can look structurally complete and still fail on language targeting if reciprocal links, URL mapping, or page availability are not checked before publishing.
These tools overlap, but they answer different launch questions. Robots.txt Auditor is better when you need to inspect the whole file as a policy document. Robots.txt Tester is better when you need a fast yes or no answer for a specific URL and user agent.
Small teams often reach a decision point before launch: are free browser-based tools enough, or does this release justify a paid SEO suite? The honest answer depends less on ideology and more on scale, accountability, and how much risk is packed into the release window.
Solo operators and individual publishers do not need agency-grade QA. They need a short, repeatable pass they will actually run before hitting publish. The most common small-site launch failure is not a wrong method. It is skipping the check entirely because no process forces it. This shortlist covers the browser-based tools that remove the highest-risk mistakes in the least amount of time for sites built and shipped by one or two people.
Technical website utility tools for robots.txt, hreflang, sitemap validation, crawl checks, and performance diagnostics.
Run a full structural audit of the file: conflicts, syntax warnings, and best-practice checks across all groups.
Check that the URLs you want crawled are represented correctly in the sitemap.
Validate international page annotations once crawl access is confirmed.
If the job spills into another category, these tools help with the next step.
Model long-term growth from starting capital, recurring contributions, and inflation before you commit to a savings plan.
Open toolProject how a starting balance and regular monthly contributions grow with compound interest over any time horizon.
Open toolProject your retirement savings balance from a current portfolio, monthly contributions, expected return, and inflation rate.
Open toolSolve the equation and interpret the parabola in one run.
Open tool