Robots.txt Generator
Create robots.txt files to control how search engines crawl and index your website.
How to Use
Select user-agent
Choose which search engine bots to create rules for, or use '*' for all bots.
Add allow/disallow rules
Specify which paths to allow or block. Use the quick-add buttons for common patterns.
Add sitemap and download
Add your sitemap URL, then download the robots.txt file to upload to your server.
Features
Frequently Asked Questions
Robots.txt is a text file placed in your website's root directory that tells search engine crawlers which pages or sections they can or cannot access. It helps control how search engines crawl and index your site.
The robots.txt file must be placed in your website's root directory. For example, if your domain is example.com, the file should be accessible at example.com/robots.txt.
Create a robots.txt file to control how search engines crawl your website. Allow or block specific bots, protect sensitive directories, and optimize your crawl budget with ease.
Did You Know?
The robots.txt standard was created in 1994 and is one of the oldest web standards still in use. It's based on the Robots Exclusion Protocol, a voluntary agreement between webmasters and search engines - there's no enforcement mechanism!
Pro Tips
- robots.txt is a suggestion, not a security measure - don't use it to hide sensitive data
- Use 'User-agent: *' to apply rules to all bots
- Always include your sitemap URL in robots.txt
Common Use Cases
- Blocking crawlers from admin areas
- Preventing indexing of duplicate content
- Managing crawl budget for large sites
- Blocking bad bots and scrapers
- Directing crawlers to your sitemap
Best Practices
- Test your robots.txt with Google Search Console
- Place robots.txt in your site's root directory
- Be specific with paths to avoid accidentally blocking important content
Explore Other Categories
Discover more useful tools from different categories
Word Counter
Count words, characters, sentences and paragraphs in your text instantly.
JSON Formatter
Format, validate and beautify your JSON data with syntax highlighting.
QR Code Generator
Create QR codes for URLs, text, and more. Download as PNG.
Percentage Calculator
Calculate percentages, percentage change, and percentage of numbers.