DiagramPreview
Basic parserNo AILocal previewDebuggingExport

robots.txt Tester

Paste a target URL and robots.txt rules to preview Allow and Disallow decisions, matched rule groups, and sitemap declarations.

Examples
Robots decisionReady
Your preview will appear here.

Continue with a related tool

Move this preview into a nearby workflow for conversion, debugging, or export.

HTML Preview Sandbox

Preview HTML components, email snippets, and landing sections in a script-blocked sandbox.

Open
CSS Gradient Preview

Preview CSS linear, radial, conic, and repeating gradients with extracted color stops.

Open
Base64 Image Preview

Preview Base64 image strings and data URLs with MIME type and size analysis.

Open

How to use

  1. 1Put the target URL on the first line, then add --- and the robots.txt content.
  2. 2Check whether the URL is allowed or disallowed for the relevant user-agent.
  3. 3Review broad Disallow rules such as /, /api, /en, or wildcard paths before publishing.
  4. 4Confirm production behavior with Search Console or the crawler tools you rely on.

Common use cases

robots.txt QATechnical SEO reviewIndexing issue debuggingAI robots snippet validationLaunch checklist

Related tools

HTML Preview Sandbox

Preview HTML components, email snippets, and landing sections in a script-blocked sandbox.

CSS Gradient Preview

Preview CSS linear, radial, conic, and repeating gradients with extracted color stops.

Base64 Image Preview

Preview Base64 image strings and data URLs with MIME type and size analysis.

FAQ

What robots.txt mistake is most dangerous?

A broad Disallow such as / or a locale prefix can block important pages from crawling even when the sitemap lists them.

Does robots.txt remove indexed pages?

Not directly. It controls crawling. Use noindex or removal tools when you need deindexing behavior.

Why test a single URL?

A specific URL test makes broad rules and wildcard behavior easier to reason about than reading the file abstractly.

robots.txt Tester checks Allow and Disallow rules for a target URL so technical SEO changes can be reviewed before deployment.

It is useful when AI, CMS settings, or manual edits produce robots rules that look simple but accidentally block tool pages, docs, assets, or localized routes.

Use it together with sitemap and Open Graph preview checks before submitting important pages for indexing.

Demo: make sure tool pages are crawlable

Before launching a tool site, test representative URLs from each section. A single broad rule can block pages even when sitemap.xml is correct.

  • Test home, tool, blog, and asset URLs separately.
  • Avoid blocking locale prefixes such as /en or /zh-CN by accident.
  • Disallow /api/ without blocking documentation paths such as /api-error-flow-diagram.
https://diagrampreview.com/en/mermaid-preview
---
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://diagrampreview.com/sitemap.xml

SEO release checklist: test robots rules with real URLs

robots.txt should be reviewed with concrete URLs, not only by reading the file. Locale prefixes, API-looking slugs, and static asset paths are easy to block accidentally on a tool site.

  • Test one page from each major directory before submitting a sitemap.
  • Keep /api/ blocked if needed, but verify similar public slugs are not matched by mistake.
  • Pair robots checks with canonical, hreflang, and sitemap checks for launch review.
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://diagrampreview.com/sitemap.xml

Review checklist for robots.txt Tester

Use robots.txt Tester when you need to inspect source content visually before it becomes documentation, a pull request note, an incident write-up, or a handoff artifact. Test robots.txt Allow and Disallow rules for a target URL.

Before exporting, check that labels are readable, relationships match the source, generated examples do not contain private data, and the preview still makes sense after you edit the input.

Limits and troubleshooting

If the preview fails, reduce the input to the smallest complete example, confirm the format syntax, and then add sections back one at a time. Many rendering failures come from partial files, indentation mistakes, missing diagram headers, or copied snippets that depend on hidden context.

Treat the preview as a review surface rather than a source of truth. Generated diagrams, converted files, dashboards, and rule examples should be checked before they are used in production documentation or operations.

Example inputs to test

  • Blog allowed: URL: https://example.com/blog/har-guide --- User-agent: * Disallow: /admin/ Allow: /blog/ Sitemap: https://example.com/sitemap.xml
  • Private blocked: URL: https://example.com/private/report --- User-agent: * Disallow: /private/ Allow: /private/public/
  • Sitemap rule: URL: /tools/jsonpath-tester --- User-agent: * Disallow: /api/ Allow: /tools/ Sitemap: https://example.com/sitemap.xml

Tool maturity

Basic parser

Basic parser

This tool focuses on fast structure inspection. Use it for review and debugging, then verify critical output before production use.

The maturity label is not a quality score. It tells visitors whether the tool is best for stable export, deeper debugging, quick parsing, or AI-assisted generation.