Plainview Web

SEO & search

Build crawl rules and see which one actually applies.

Generate a clean robots.txt file or inspect pasted rules line by line. Then run a Google-compatible local simulation for one crawler and URL—without requesting the site or inventing a crawlability score.

  • Runs in your browser
  • RFC-grounded parsing
  • No live-site claims

Define crawler rules

Build the file.

Rules apply only to this exact scheme, hostname, and port. No request is made.

Expected file location: https://example.com/robots.txt

Crawler group 1

Use * or product tokens such as Googlebot. Separate multiple tokens with commas.

Use a root-relative path. Wildcards and a final dollar-sign end marker are supported.

Use a root-relative path. Wildcards and a final dollar-sign end marker are supported.

Optional sitemap URLs

Use a fully qualified HTTP or HTTPS sitemap URL.

Runs entirely in your browser. No URL is fetched and no file contents are sent or stored.

Google-compatible local review

Generated file

No blocking issues

Lines
6
UTF-8 bytes
94
Groups
1
Sitemaps
1
Generated robots.txt file
User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

Parsed crawler groups

  • *2 rules · starts line 1

Findings

  • No blocking issuesReady

    The recognized fields parse cleanly in this browser-only review.

Test a crawler and URL

Simulate the supplied text only. This does not contact the site or identify a live crawler.

Enter one exact crawler product token, such as Googlebot.

Enter an absolute HTTP or HTTPS URL on the exact site origin above.

What this tool cannot verify
  • Whether a live file exists at the expected location or what HTTP response it returns.
  • Redirects, caching, content type, crawler identity, or whether a bot follows the request.
  • Whether a URL is indexed, canonicalized, secure, private, or even exists.
  • Whether a declared sitemap URL loads or contains a valid sitemap.

robots.txt controls crawl requests—not authorization, privacy, or guaranteed indexing.

Use it with context

A crawl request is not an access control.

Crawlers are asked to honor Allow and Disallow rules for the exact scheme, hostname, and port that serves the file. A disallowed URL can still appear in search results without a snippet, and a crawler that ignores the protocol can still request it.

Use robots.txt for crawling

Manage crawler requests for URL paths and advertise optional sitemap URLs.

Use page controls for indexing

Use robots meta tags or X-Robots-Tag when supported crawlers must see a noindex instruction.

Do not list secrets.

A robots.txt file is public and can reveal paths. Protect private resources with real authentication and authorization, not Disallow rules.

Primary sources

Rules you can verify.

Reviewed August 22, 2026. The simulator follows RFC 9309 and labels Google-specific compatibility behavior rather than presenting it as universal.