Sitemaps & Robots
Control what search engines crawl: XML sitemap, robots.txt and meta robots directives.
Tools in Sitemaps & Robots
- 036Robots.txt GeneratorPick a preset, add your sitemap and rules for specific crawlers. The robots.txt is generated and re-read by an RFC 9309 parser before you download it.Free · runs in your browser
- 037Robots.txt CheckerPaste your file or enter a domain. The tool applies the RFC 9309 rules, flags errors and tells you whether a given crawler may fetch a URL.Free · no account needed
- 039Meta Robots GeneratorTick the directives you need: the tool outputs the meta robots tag, the X-Robots-Tag header and the matching Apache and Nginx snippets.Free · runs in your browser
- 040Meta Robots & X-Robots-Tag CheckerEnter a URL, or paste the HTML and headers: the tool reads every robots directive, spots contradictions and explains what each one does.Free · no account needed
- 042XML Sitemap GeneratorPaste your URLs, one per line: the tool outputs a valid, correctly escaped sitemap.xml and splits it with an index if you go over the limits.Free · runs in your browser
- 043XML Sitemap ValidatorEnter the sitemap address or paste its XML: the tool checks the structure, the limits and every value, and works out how many files you need.Free · no account needed
- 044Sitemap URL ExtractorEnter a sitemap or just a domain: the tool lists every URL and its lastmod, follows a sitemap index one level deep and exports the result.Free · no account needed
- 113Sitemap vs CrawlYour sitemap says one thing, your site shows another. Compare the two to find the gaps.Free account · 3 live checks/day
- 114Robots Blocking AuditEnter a URL: the tool reads the site's robots.txt, applies RFC 9309 for each crawler and also checks the page's noindex directives.Free account · 3 live checks/day