Robots.txt Generator

Robots.txt Generator — Control Search Engine Crawling Behavior

Every time Googlebot or Bingbot lands on your site, the very first thing it looks for is a file called robots.txt at your domain root. That tiny text file can tell the crawler to go wherever it wants or to stay out of certain rooms entirely. It's part of the Robots Exclusion Protocol — a standard adopted by every major search engine, though technically still advisory. Well-behaved bots respect it. Malicious scrapers and spam bots don't. But for the search engines that actually matter, robots.txt is your primary tool for managing crawl behavior.

The file sits at https://example.com/robots.txt and uses a dead-simple syntax: declare which user-agent (crawler) you're talking to, then tell it what it can or can't access. No special software needed — you can write robots.txt in Notepad. This generator handles the formatting for you and ensures the syntax is valid, but understanding the rules behind the output makes you a better site owner.

Robots.txt Syntax Explained

A robots.txt file consists of groups of directives. Each group starts with a User-agent: line identifying which crawler the rules apply to, followed by one or more Disallow: or Allow: directives.

User-agent: * applies to all crawlers. User-agent: Googlebot applies only to Google's crawler. You can have multiple groups — for example, specific rules for Googlebot and a catch-all * rule for everything else.

Disallow: /admin/ blocks all URLs that start with /admin/. Disallow: with an empty path allows everything (it's effectively a no-op). The path is a prefix match — Disallow: /blog blocks /blog, /blog/post-1, and /blogging/tips because they all start with /blog.

Allow: overrides a Disallow: for a more specific path. This is useful when you want to block a directory but allow access to a specific page within it. For example: Disallow: /private/ followed by Allow: /private/public-page blocks everything in /private/ except /private/public-page.

The order of directives matters. When a path matches both a Disallow and an Allow, the most specific (longest) path wins. So Disallow: /blog and Allow: /blog/images means /blog/images is accessible but everything else under /blog is blocked.

Common Configurations for Real Websites

Here are robots.txt setups you'll encounter frequently, along with why they're configured that way:

WordPress (default): Blocks /wp-admin/ for all crawlers except Googlebot, which gets Allow: /wp-admin/admin-ajax.php because that file is needed for page rendering. Blocks /wp-includes/ since it contains library files, not content. Includes a Sitemap: reference.

E-commerce store: Blocks /cart/, /checkout/, /account/, /search?sort= (parameterized URLs that waste crawl budget), and /private/. Allows /products/, /categories/, and /blog/. May include a Crawl-delay: 5 if the server struggles with heavy crawl traffic.

Multi-language site: Blocks language-specific parameterized URLs that create duplicate content (like /page?lang=fr when a clean /fr/page exists). Blocks internal search results. Allows all localized subdirectories.

Minimal (default for new sites): A single group with User-agent: *, an empty Disallow:, and a Sitemap: reference. This tells crawlers they can access everything and points them to your sitemap for efficient discovery.

Mistakes That Can Hurt Your SEO

Robots.txt errors are silent — you won't get a notification when you've accidentally blocked Google from your entire site. Here are the most common mistakes:

Blocking the entire site. A Disallow: / under User-agent: * blocks all crawlers from everything. This happens more often than you'd think, especially during site migrations when someone copies a staging environment's robots.txt to production.

Blocking CSS and JavaScript. If you block Googlebot from accessing your CSS or JS files, Google can't properly render your pages. Modern Googlebot renders JavaScript and needs CSS to understand layout and content. Blocking these resources can cause pages to appear as blank or poorly rendered in search results.

Using robots.txt instead of noindex. Blocking a page with Disallow prevents crawling but doesn't prevent indexing. If other pages link to the blocked URL, Google can still index it using those external signals. For guaranteed removal, use noindex meta tags — but note that Google must crawl the page first to see the tag, so you can't block the page in robots.txt and use noindex simultaneously.

Forgetting the trailing slash inconsistency. Disallow: /blog blocks /blog, /blog/, and everything under /blog/. But Disallow: /blog/ only blocks /blog/ and its subdirectories — it doesn't block /blog (the page itself). Be intentional about whether you include or omit the trailing slash.

Crawl Delay and Sitemap Directives

Crawl-delay: Tells supported crawlers to wait a specified number of seconds between requests. Bing and Yandex honor this directive; Google does not. Google uses its own crawl rate controls in Search Console instead. If you're on shared hosting and Bing's crawling is overwhelming your server, Crawl-delay: 10 gives your server breathing room. For Google, adjust crawl rate settings in Search Console → Settings → Crawl Rate.

Sitemap: Points crawlers to your XML sitemap. Place this directive at the top level (not inside a User-agent block) so it applies to all crawlers. While Google finds sitemaps through Search Console, not all crawlers are registered there. The Sitemap: directive in robots.txt is the universal way to ensure every bot knows where your sitemap lives.

The Host: directive (used by Yandex to specify which version of a multi-domain site to index) and Crawl-delay: values have different scales across engines. Bing interprets Crawl-delay: 1 as one request per second; Yandex interprets it differently. If you're targeting multiple search engines, test your crawl delay values.

Frequently Asked Questions About Robots.txt