robots.txt generator
Generate your robots.txt
Build a robots.txt with per-crawler Allow/Disallow rules, crawl-delay, sitemaps and host — with handy presets, all in your browser.
Presets
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Frequently asked questions
What is a robots.txt file?
A file at your site root that tells search-engine crawlers which paths they may or may not crawl.
Does Disallow keep a page private?
No. It asks crawlers not to crawl a path but does not secure it — use authentication for real protection.
Where do I put the file?
At the root of your domain, for example https://example.com/robots.txt.