Instant SEO Presets
Click a preset to quickly configure production-grade rules:
1
Default Bot Policy (
Default Bot Policy (User-agent: *)
Define the baseline access permission applied to all standard search engines.
2
Search Engine Bots
Configure custom directives for specific major search crawlers.
Googlebot
Google Search Engine
Bingbot
Microsoft Bing & Yahoo
DuckDuckBot
DuckDuckGo Privacy Search
YandexBot
Yandex Search
Baiduspider
Baidu Search Engine
3
AI Crawlers & Data Scrapers Shield
Prevent Large Language Models (LLMs) from scraping your content for model training.
•
GPTBot
OpenAI (ChatGPT / GPT-4)
CCBot
Common Crawl (AI Corpus)
ClaudeBot
Anthropic (Claude AI)
Bytespider
ByteDance / TikTok AI
Google-Extended
Gemini / Vertex AI Training
PerplexityBot
Perplexity AI Indexer
4
Dynamic Path Rules (Disallow & Allow)
Add directories, specific URLs, or wildcard regex patterns to restrict or permit.
5
Sitemaps & Crawl Optimization
Link XML sitemaps and specify server rate limits.
ℹ️ Bing/Yandex only
sec
Live robots.txt Output
RFC 9309 Valid
1
🧪
Robots Exclusion Protocol (RFC 9309) Syntax Validator & Path Tester
Paste your raw robots.txt file below and simulate real bot crawl permissions against specific URL paths in your browser.
Robots.txt Content
Type or paste raw directives for syntax inspection
Syntax & SEO Health Inspector
Type or paste rules to inspect syntax.
URL Path Permission Simulator
Test whether a crawler is ALLOWED or BLOCKED on a specific path
Click "Test Path" to evaluate crawl permissions.