WEBSITE STUDIO · FREE

robots.txt Generator

Make a correct robots.txt: keep private pages out of search, and choose whether AI crawlers may read your site.

Runs in your browser. Nothing you type is stored or sent.

Your websitePages to keep out of search (one per line)

Block AI training crawlers (optional)

Point to my sitemap

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

How it works

1Enter your site and the pages to keep out.
2Choose any AI crawlers to block.
3Download robots.txt and upload it to your site's root.

Questions

Does robots.txt hide a page?

No. It asks search engines not to crawl it. For private pages, use a login or a noindex tag.

Is my data sent anywhere?

No. This runs in your browser, so what you type stays on your device.

Want a person to look at your result? Free, and no newsletter.

Need a website or app built?

We build business websites in 7 days and web apps in weeks, for a fixed price. Then we automate the busywork around them.

Book a free strategy call

AI Prompt Generator and Optimizer

A rough idea in, a prompt that works the first time out.

Free Grammar Checker

Catch typos, repeated words, wrong a/an and long sentences. Private by default.

AI Code Explainer

Any code explained in plain English, line by line.

About the robots.txt

The robots.txt generator makes the file that tells search engines and AI crawlers which parts of your site they may visit. It includes ready rules for GPTBot, ClaudeBot, Google-Extended and other AI crawlers.

When to use it

Block admin, cart and login pages from crawlers.
Choose whether AI tools may read your site.
Point crawlers to your sitemap.

Tips for a better result

robots.txt does not hide a page from Google results. Use a noindex tag for that.
Never block your CSS and JavaScript files.

What a healthy website needs

Search engines and visitors reward the same things: pages that load in under about 2.5 seconds, a clear title and description, working links, HTTPS, and a layout that works on a phone. These checks show where your site stands. If you would rather have it done for you, we build and fix business websites, usually in 7 working days.

More free tools: see every tool.