Robots.txt Generator
Generate a robots.txt file for your website with custom rules online for free.
Quick Presets
Rules
Disallow
Allow (overrides Disallow)
Crawl-delay (seconds)
Sitemap URLs
Extra / Comments
Preview
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Place this file at: https://yourdomain.com/robots.txt
このツールについて
Create a valid robots.txt file with custom allow/disallow rules for different crawlers. Add sitemap URLs, set crawl delays, and preview the generated file before downloading.
使い方
Add Rules
Choose a user-agent and add Allow/Disallow paths.
Configure Sitemap
Optionally add your sitemap URL.
Download
Preview and download the robots.txt file.
活用シーン
SEO Control
Control which pages search engine bots are allowed to crawl.
よくある質問
Where do I put the robots.txt file?
Place it at the root of your domain: https://yourdomain.com/robots.txt