Robots.txt Generator
Generate a robots.txt file for your website with custom rules online for free.
Quick Presets
Rules
Disallow
Allow (overrides Disallow)
Crawl-delay (seconds)
Sitemap URLs
Extra / Comments
Preview
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Place this file at: https://yourdomain.com/robots.txt
關於此工具
Create a valid robots.txt file with custom allow/disallow rules for different crawlers. Add sitemap URLs, set crawl delays, and preview the generated file before downloading.
使用方法
Add Rules
Choose a user-agent and add Allow/Disallow paths.
Configure Sitemap
Optionally add your sitemap URL.
Download
Preview and download the robots.txt file.
使用場景
SEO Control
Control which pages search engine bots are allowed to crawl.
常見問題
Where do I put the robots.txt file?
Place it at the root of your domain: https://yourdomain.com/robots.txt