Robots.txt Generator
Generate robots.txt files to control web crawler access to your website. Create SEO-optimized directives with templates and best practices.
🔒 100% Private & Secure
No Data Collection • No Tracking • No Servers
🎯 Quick Templates
🤖 User Agents (Crawlers)
📋 Rules for Selected Crawler
Disallow Paths (Block Access)
Allow Paths (Explicitly Allow)
Crawl Delay (seconds)
Time between requests (not all crawlers respect this)🗺️ Sitemaps
🌐 Host Directive (Optional)
Preferred host for Yandex (deprecated by Google)📄 Generated robots.txt
💡 Common Paths to Consider
🔒 Security & Privacy
/admin/- Admin area/wp-admin/- WordPress admin/private/- Private files/config/- Configuration/.git/- Git repository/.env- Environment files
🛒 E-commerce
/cart/- Shopping cart/checkout/- Checkout pages/account/- User accounts/order/- Order pages/search?- Search results/*?sort=- Sorted pages
⚙️ Technical
/api/- API endpoints/cgi-bin/- CGI scripts/tmp/- Temporary files/backup/- Backup files*.sql- Database dumps*.log- Log files
📄 Content
/tag/- Tag pages/category/- Categories/page/- Pagination/feed/- RSS feeds/*?print- Print versions/*?replytocom- Comments
📚 Best Practices
✅ Do's
- Test with Google Search Console
- Include sitemap reference
- Use specific paths over wildcards
- Place robots.txt in root directory
- Keep file under 500KB
- Use UTF-8 encoding
❌ Don'ts
- Don't block CSS/JS files needed for rendering
- Don't use robots.txt for security
- Don't list sensitive URLs
- Don't use multiple wildcards
- Don't forget trailing slashes for directories
- Don't block important pages accidentally
Caiatech Developer Tools | caiatech.com
Robots.txt Generator v1.0.0 | © 2025 Caiatech. All rights reserved.
🔐 100% Private Tool by Caiatech - Your data NEVER leaves your browser! All processing happens locally on your device. Zero data collection, zero tracking, zero servers. This tool works completely offline once loaded. Built with absolute privacy by Caiatech.
Privacy Guarantee: No cookies • No analytics • No third-party scripts • No cloud processing • No data storage