/r
Tool

Robots.txt Generator

Configure crawler access rules and generate your robots.txt file

Global Settings

Search Engine Robots

defaultGoogle
defaultGoogle Image
defaultGoogle Mobile
defaultMSN Search
defaultYahoo
defaultYahoo MM
defaultAsk / Teoma
defaultGigaBlast

Custom Agents

One per line — Bingbot: allowed or SomeBot: refused

Restricted Directories

Output

# Click "Generate robots.txt"
# to build your file

What is robots.txt

A robots.txt file instructs web crawlers which parts of your site they may access. It lives at the root of your domain and is the first place any crawler checks before indexing.

How to use

  • 01Choose a default rule for all crawlers
  • 02Set per-engine overrides if needed
  • 03Add restricted paths with Disallow
  • 04Add your sitemap URL
  • 05Save as robots.txt at site root

Example

User-agent: *
Disallow:

User-agent: Googlebot
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Tip: upload the generated file to your server's web root.

Allow all crawlers

Disallow:

Block all crawlers

Disallow: /

Block a path

Disallow: /admin/

Add crawl delay

Crawl-delay: 5