DailyTools
All articles
SEOMay 25, 20268 min read

XML Sitemaps and robots.txt: The Technical SEO Files Every Website Needs

Two small text files at the root of your domain have an outsized impact on how search engines discover, crawl, and index your content. A developer's guide to getting both right.

Search engines discover web pages through two mechanisms: following links from known pages (crawling) and reading a provided list of URLs (sitemaps). The robots.txt file controls which pages crawlers are allowed to access, while the XML sitemap tells them which pages exist and which are most important. Together, these two files form the foundation of technical SEO — and misconfiguring either one can make your pages invisible to search engines or waste your site's crawl budget on irrelevant pages.

robots.txt: Controlling Crawler Access

The robots.txt file lives at the root of your domain (https://example.com/robots.txt) and uses the Robots Exclusion Protocol to tell search engine crawlers which URL paths they may or may not access. It is the first file a well-behaved crawler checks before crawling any page on your site.

text
# Basic robots.txt example
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /staging/
Disallow: /search?
Disallow: /*.pdf$

# Allow Google to crawl everything except admin
User-agent: Googlebot
Allow: /
Disallow: /admin/

# Sitemap location
Sitemap: https://example.com/sitemap.xml

robots.txt Directives Explained

  • User-agent: Specifies which crawler the rules apply to. Use * for all crawlers, or name specific bots: Googlebot, Bingbot, GPTBot, etc.
  • Allow: Explicitly permits crawling of a URL path. Only needed to override a broader Disallow rule.
  • Disallow: Blocks crawling of a URL path. An empty Disallow: means 'allow everything.' Disallow: / blocks the entire site.
  • Crawl-delay: Requests crawlers wait N seconds between requests. Honored by Bing and Yandex but ignored by Google (use Search Console to set Google's crawl rate).
  • Sitemap: Points crawlers to your XML sitemap. Can include multiple Sitemap directives for multiple sitemaps.

Critical distinction: robots.txt blocks crawling, not indexing. If other pages link to a URL you have blocked in robots.txt, Google may still index that URL — it just will not be able to read the page content, so the search result will show a bare URL with no snippet. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead.

Common robots.txt Mistakes

  • Accidentally blocking CSS and JavaScript: If you disallow /assets/ or /static/, Googlebot cannot render your pages and may misinterpret their content. Always allow crawling of CSS, JS, and image files.
  • Blocking your entire site: A misplaced 'Disallow: /' under 'User-agent: *' blocks all crawlers from your entire site. This is sometimes left over from a staging environment configuration.
  • Blocking parameterized URLs too broadly: 'Disallow: /?' blocks all URLs with query parameters, including legitimate paginated content and filtered views that should be crawled.
  • Not including the Sitemap directive: robots.txt is the most reliable way to tell search engines where your sitemap is located. Always include it.
  • Forgetting that rules are case-sensitive: 'Disallow: /Admin/' does not block '/admin/' — the paths must match exactly.

XML Sitemaps: Your Content Inventory for Search Engines

An XML sitemap is a structured file that lists the URLs on your site that you want search engines to know about. While search engines can discover pages through crawling links, a sitemap ensures that pages with few or no incoming links — newly published content, deep pages, or orphaned pages — are still discovered and indexed.

xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-05-25</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/tools/json-formatter</loc>
    <lastmod>2026-05-20</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

Sitemap Elements

  • <loc> (required): The full, canonical URL of the page. Must match the canonical URL exactly — including protocol (https), trailing slashes, and subdomain.
  • <lastmod> (recommended): The date the page was last significantly modified, in ISO 8601 format (YYYY-MM-DD). Google uses this to prioritize crawling recently updated pages. Do not set this to the current date on every build — only update it when the content actually changes.
  • <changefreq> (optional): How frequently the page is expected to change: always, hourly, daily, weekly, monthly, yearly, never. Google largely ignores this in favor of its own crawl patterns.
  • <priority> (optional): A value from 0.0 to 1.0 indicating the page's importance relative to other pages on your site. Only meaningful within your own sitemap — it does not affect ranking against other sites.

Sitemap Best Practices

  • Include only canonical, indexable URLs — no redirects, no noindex pages, no pages blocked by robots.txt
  • Keep sitemaps under 50,000 URLs and 50MB uncompressed. For larger sites, use a sitemap index file that references multiple child sitemaps
  • Update <lastmod> only when content meaningfully changes — not on every deploy. Google trusts lastmod dates that correlate with actual content changes and ignores them when they are always the current date
  • Submit your sitemap to Google Search Console and Bing Webmaster Tools after initial setup and after any major site restructuring
  • For dynamic sites, generate the sitemap programmatically from your CMS, database, or framework's routing system rather than maintaining it manually
  • Validate your sitemap XML against the sitemaps.org schema before submitting — malformed XML causes the entire sitemap to be rejected

Generating Sitemaps Programmatically

Modern web frameworks provide built-in sitemap generation. In Next.js, export a sitemap() function from app/sitemap.ts. In Django, use django.contrib.sitemaps. In Rails, use the sitemap_generator gem. In static site generators (Hugo, Gatsby, Astro), sitemaps are typically generated automatically during the build process.

typescript
// Next.js App Router — app/sitemap.ts
import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  const tools = getToolsFromRegistry(); // Your tool data source

  return [
    { url: 'https://example.com', lastModified: new Date(), priority: 1.0 },
    ...tools.map(tool => ({
      url: `https://example.com/tools/${tool.category}/${tool.slug}`,
      lastModified: new Date(),
      changeFrequency: 'monthly' as const,
      priority: 0.8,
    })),
  ];
}

Monitoring and Debugging

After deploying your robots.txt and sitemap, monitor their effectiveness in Google Search Console:

  • Coverage report: Shows which URLs are indexed, excluded, or erroring. Look for 'Excluded by robots.txt' entries that should not be excluded.
  • Sitemap report: Shows how many URLs from your sitemap have been discovered and indexed. A large gap between submitted and indexed URLs indicates potential crawl or content quality issues.
  • URL Inspection tool: Check individual URLs to see if Google can access them and whether robots.txt is blocking them.
  • robots.txt Tester: Google Search Console includes a tester that lets you check whether specific URLs are allowed or blocked by your rules.

Try the free tool referenced in this article

XML Sitemap Validator
Back to all articles