TL
Tool Lab
💰Donate
💰Donate

robots.txt Generator

Visually build a robots.txt file to control which pages search engine crawlers can access on your site.

Presets
Rule 1
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

About This Tool

The robots.txt file tells compliant web crawlers which pages or sections of your site they are allowed or not allowed to crawl and index. It must be placed at the root of your domain.

Use presets for common configurations or build custom rules for each crawler. Adding a Sitemap URL lets crawlers discover your XML sitemap automatically.

How to Use

  1. Choose a preset or fill in User-agent, Allow, and Disallow fields manually.
  2. Add multiple rules with the + Add Rule button for different bots.
  3. Enter your Sitemap URL and optionally set a Crawl-delay.
  4. Copy the output or download robots.txt and upload it to your site root.

Use Cases

SEO professionals, web developers, and site administrators use it to optimize crawl budget, protect sensitive areas, and block aggressive scrapers.

FAQ

  • Does robots.txt prevent pages from appearing in search? — It prevents crawling but not indexing. Use a noindex meta tag or X-Robots-Tag header to prevent indexing of specific pages.
  • Do all crawlers respect robots.txt? — Reputable crawlers like Googlebot and Bingbot respect it. Malicious scrapers may ignore it.
  • Where should robots.txt be placed? — Always at the root of your domain. Example: https://example.com/robots.txt