LLight SEO Tools
Free browser helper

Robots.txt Checker & Generator

Fetch and analyze any site's robots.txt file, or build your own crawler rules with a simple generator.

Checker

Inspect a robots.txt

Enter a domain or URL. The tool fetches its robots.txt and summarizes the directives.

Site

Enter a full URL or a bare domain, e.g. https://example.com.

Results

Raw file plus a parsed summary of rules and potential issues.

Fetching note: The checker reads a remote robots.txt from your browser. If a site blocks cross-origin requests, the tool falls back to a public proxy; if that is unavailable the fetch will report an error. For offline or large-scale checks, download your URL list and use a crawler or spreadsheet workflow.
Generator

Build your own robots.txt

Fill in the rules and copy or download the result.

Rules

Leave fields empty to omit them from the output.

Output

Your generated robots.txt file.

Free browser helper

Robots.txt Checker & Generator

Two tools on one page: fetch and analyse an existing robots.txt to see exactly what crawlers are told, then build a clean file of your own and copy it straight into your site root.

Step by step

  1. In the checker, enter a domain or full URL and click Check robots.txt.
  2. Review the parsed summary: user-agents, disallow rules, allow rules, sitemap references and missing-file warnings.
  3. Switch to the generator and add your user-agent rules plus your sitemap location.
  4. Copy or download the result and upload it as /robots.txt in your site root.

Tips for better results

  • A missing robots.txt is not an error — with no file everything is crawlable — but publishing one lets you point crawlers at your sitemap.
  • Never rely on robots.txt to hide private pages. Use authentication or a noindex directive instead, because anyone can read a robots.txt file.
  • Test after every edit: a single stray Disallow: / line can pull an entire site out of the index.

Frequently asked questions

Does robots.txt remove pages from search results?

No. Disallow stops crawling, not indexing. If other pages link to the URL it can still appear in results. Use a noindex directive on the page itself to remove it.

Where does robots.txt have to live?

Always in the root of the domain, for example https://example.com/robots.txt. Rules apply per host, so www and non-www are treated separately.

Can I check robots.txt at scale?

This checker is built for individual domains. For multi-domain audits, use a dedicated crawler or check each domain separately.

Related free SEO tools

Looking for another SEO tool?

Browse the full collection of free browser-based helpers or contact us with a general question.

Browse all tools