← SitemapWarden

SitemapWardenBot

This is the official identity page for the SitemapWarden crawler. We fetch robots.txt and XML/text sitemaps so SaaS products can monitor URL inventory and robots changes. We do not use fetched content to train AI models.

Purpose

• Download and store sitemaps and robots.txt for subscribed domains
• Detect additions, removals, and other changes over time
• Expose results to customer applications through the SitemapWarden API

User-Agent

Mozilla/5.0 SitemapWardenBot (info: www.sitemapwarden.com)

Verification

Requests are cryptographically signed with Web Bot Auth (HTTP Message Signatures). Public keys:

https://www.sitemapwarden.com/.well-known/http-message-signatures-directory

Rate limits & behaviour

• At most one request per host about every 350ms, plus backoff on 429/5xx
• We follow redirects and accept XML/text sitemap formats
• We identify ourselves clearly and do not bypass site security challenges

Opt-out & contact

To request exclusion or ask about our crawler, email [email protected] with the domain and any relevant details.

Content use: search=no, ai-train=no, monitoring/reference only.