How it works
Use the robots.txt generator in three steps
- 01
Enter your sitemap URL and paths that should not be crawled.
- 02
Choose whether to include common AI crawler directives.
- 03
Copy the file to /robots.txt at the root of your domain.
Free · no signup required
Free Robots.txt Generator
Generate a straightforward robots.txt policy that tells compliant crawlers which paths they may request. Review every rule before publishing it.
User-agent: * Allow: / Disallow: /admin/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
What it provides
Practical output you can use
- Produces standards-friendly plain text
- Optional directives for common search and AI crawlers
- Warns that robots.txt controls crawling, not secure access or guaranteed de-indexing
Common questions
Robots.txt Generator FAQ
Does robots.txt remove a page from Google?
Not reliably. It controls crawling, while removal from search generally requires appropriate indexing controls such as noindex on an accessible page.
Can robots.txt protect private information?
No. The file is public and compliant crawlers may honor it, but it is not authentication or access control.
Where should I upload it?
At the root of the relevant host, such as https://example.com/robots.txt.
Need implementation help?