🤖 Web Crawler Generator

Robots.txt & XML Sitemap Generator

Generate production-ready robots.txt and XML sitemap configuration files for search bots.

Domain / Website Base URL

Disallowed Path Rules (One per line)

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://webtools.com/sitemap.xml

sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://webtools.com/</loc>
    <lastmod>2026-08-15</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
</urlset>