🤖 Free Robots.txt Generator
Build a robots.txt file for your website. Control which search engine bots can crawl your pages and where your sitemap is located.
All Bots (User-agent: *)
Allow all pages (default)
Common pages to disallow:
Block Specific Bots
Generated robots.txt
What Is robots.txt?
The robots.txt file lives at the root of your domain (e.g., example.com/robots.txt) and tells search engine crawlers which pages they can and cannot access. It's the first file most bots check when visiting your site.
Robots.txt is a courtesy protocol — it's not a security measure. Malicious bots may ignore it. For truly private pages, use password protection or authentication. However, for search engines like Google, Bing and others, robots.txt is respected and critical for controlling your crawl budget.