Robots.txt Generator
Set crawler instructions.
Add excluded paths, one per line. These public instructions do not protect private pages.
Your input
Crawler, exceptions and sitemap (optional)
Use * for all crawlers. Specific names use letters, hyphens and underscores, such as Googlebot-Image.
Live preview