Robots.txt Generator
Build a robots.txt file without memorizing the syntax. Choose a preset or add your own disallow paths, include your sitemap, and copy the result straight to your site root.
How to generate a robots.txt file
- Pick a preset: allow all, block all, or custom rules.
- Add the paths you want to disallow and paste your sitemap URL.
- Copy the output or download robots.txt and upload it to your site root.
Examples
Block an admin area and add a sitemap
Disallow /admin, /cart with sitemap https://example.com/sitemap.xml
User-agent: * Disallow: /admin Disallow: /cart Sitemap: https://example.com/sitemap.xml
Frequently asked questions
What does a robots.txt file do?
It tells search engine crawlers which paths they may or may not request on your site. Bots read it from yourdomain.com/robots.txt before crawling.
What is the difference between Allow and Disallow?
Disallow lists paths crawlers should not fetch. Allow re-opens a specific path inside a disallowed folder. An empty Disallow line means everything is allowed.
Does robots.txt stop a page from being indexed?
No. It only controls crawling. A blocked URL can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag instead.
Where do I put the robots.txt file?
Upload it to the root of your domain so it lives at https://yourdomain.com/robots.txt. It will not work in a subfolder, and the filename is always lowercase.
Is my data sent to a server?
No. The file is generated entirely in your browser, so nothing you type is uploaded anywhere.
What does Crawl-delay do?
It asks bots to wait a set number of seconds between requests. Google ignores it, but several other crawlers honor it to reduce server load.
Related tools
Meta Tag Generator
Generate clean HTML meta tags for any page. Set title, description, keywords, author and robots, then copy the head block. Runs in your browser.
Keyword Density Checker
Free keyword density checker. Paste your content to see which words and phrases appear most, with counts and density percentages. Runs in your browser.
Article Schema Generator
Generate Article, NewsArticle or BlogPosting JSON-LD schema. Add headline, author, publisher and dates, then copy the script into your page.
Breadcrumb Schema Generator
Build BreadcrumbList JSON-LD structured data. Add your breadcrumb trail, reorder the steps, then copy or download the script for your page.
Bulk Hreflang Generator
Generate hreflang annotations for many languages at once. Add language and URL rows, pick HTML, XML sitemap, or HTTP header output. Runs in your browser.
Canonical Tag Checker
Paste page HTML to find every rel=canonical link and catch issues: missing, duplicate, relative or wrong-target canonicals. Runs in your browser.