Robots.txt Generator
Generate a valid robots.txt file. Choose a preset or build custom allow and disallow rules, add your sitemap URL, then copy or download.
Place robots.txt at your domain root, e.g. https://example.com/robots.txt. Test it with Google's robots.txt tester.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of a website that instructs search engine crawlers which pages or sections they are permitted to access. Crawlers read it before indexing any other page on the site.
Does blocking a URL in robots.txt remove it from Google?
Not necessarily. Disallowing a URL prevents the crawler from visiting that page, but the URL can still appear in search results if other sites link to it. To fully de-index a URL, use a noindex meta tag or the Google Search Console URL removal tool.
Where should the robots.txt file be placed?
Place robots.txt at the root of your domain, for example https://example.com/robots.txt. It must be publicly accessible without authentication. For WordPress sites, upload it to the same directory as wp-config.php.