What is a robots.txt file?

A robots.txt file tells search engines and other bots which pages they can or cannot crawl on your website. It's the first file bots check when visiting your site.

How do I block AI scrapers like GPTBot and ClaudeBot?

Simply select the AI bot presets (GPTBot, ChatGPT-User, CCBot, ClaudeBot, etc.) in the tool and add the paths you want to block. The generator will create the appropriate Disallow rules.

Is Robots.txt Generator free to use?

Yes, Robots.txt Generator is completely free with no hidden fees or usage limits. Pro features are available for power users.

Do I need to install any software?

No installation required. Works directly in your web browser on any device. Your robots.txt is generated locally in your browser.

Where do I upload my robots.txt file?

Upload your robots.txt file to the root directory of your website (e.g., yoursite.com/robots.txt). Most CMS platforms have built-in options to manage this file.

Free Robots.txt Generator

Create and download your robots.txt file in seconds

How It Works

Select a CMS template or start from scratch
Choose which bots can access your site
Add paths to allow or block
Add your sitemap URL(s)
Copy or download your robots.txt file

CMS Templates

Start from template:

Rule Group 1

User-Agent

Disallow Paths

Allow Paths

Crawl-Delay

Sitemaps

AI Bot Controls

AI Crawler Controls

Block AI training and browsing bots from accessing your site. Checked bots will have a Disallow: / rule added to the output.

GPTBot OpenAI

Allowed

ChatGPT-User OpenAI browsing

Allowed

ClaudeBot Anthropic

Allowed

PerplexityBot Perplexity AI

Allowed

CCBot Common Crawl

Allowed

OAI-SearchBot OpenAI search

Allowed

Meta-ExternalAgent Meta AI

Allowed

Bytespider TikTok / ByteDance

Allowed

# Robots.txt # Generated by Atelier Tools User-agent: *

Select Bot

Test a URL Path

Wildcard Pattern

Test Paths (one per line)

Help & Guides

How to Use This Tool FAQ

How It Works

CMS Templates

AI Bot Controls

Import Paths

Template Library

Upgrade to Pro

Help & Guides

Related Tools

Canonical Tag Generator

FAQ Schema Generator

Open Graph Debugger

Heading Structure Checker