Next-Sitemap Configuration and SEO Optimization Guide

Introduction

Search engine optimization starts with a proper sitemap. A well-configured sitemap tells search engines which pages to crawl, how often to check for updates, and which pages are most important. For Next.js applications, next-sitemap is the standard tool for generating sitemaps automatically.

After deploying a Next.js blog that had zero organic traffic for three months, I discovered the sitemap was misconfigured, the robots.txt was blocking crawlers, and canonical URLs were missing. Fixing these issues increased organic traffic by 400% within six weeks.

Environment

• OS: Windows 11
• Node.js: v20.10.0
• Next.js: 14.0.4
• next-sitemap: 4.2.8

Problem

The default sitemap generation had several issues:

# Generated sitemap.xml


  
    https://example.com/
    2024-06-05
    daily
    0.7
  

Missing pages, incorrect priorities, and no image sitemaps. Google Search Console showed:

Sitemap could not be read
Warning: Sitemap contains URLs which are blocked by robots.txt

The robots.txt was generated with overly restrictive rules that blocked important pages.

Analysis

The issues were:

1. Default configuration ignores dynamic routes. next-sitemap needs explicit configuration to include dynamically generated pages.

2. Missing image sitemaps. For content-heavy sites, image sitemaps significantly improve image search visibility.

3. Incorrect robots.txt rules. The default robots.txt was too permissive or too restrictive depending on the configuration.

4. No sitemap index. Large sites need a sitemap index that points to multiple sub-sitemaps for different content types.

Solution

Fix 1: Complete next-sitemap configuration

// next-sitemap.config.js
/** @type {import('next-sitemap').IConfig} */
module.exports = {
  siteUrl: process.env.SITE_URL || 'https://example.com',
  generateRobotsTxt: true,
  generateIndexSitemap: true, // Creates sitemap-index.xml
  outDir: './public',
  robotsTxtOptions: {
    additionalSitemaps: [
      'https://example.com/api/sitemap.xml', // Custom API sitemap
    ],
    policies: [
      {
        userAgent: '*',
        allow: '/',
      },
      {
        userAgent: 'GPTBot',
        disallow: '/',
      },
      {
        userAgent: 'CCBot',
        disallow: '/',
      },
    ],
  },
  transform: async (config, path) => {
    // Custom priority and change frequency
    const customPriority = {
      '/': 1.0,
      '/blog': 0.9,
      '/products': 0.8,
      '/about': 0.5,
      '/contact': 0.3,
    }

    // Check if path matches a pattern
    let priority = 0.7
    let changefreq = 'weekly'

    if (path === '/') {
      priority = 1.0
      changefreq = 'daily'
    } else if (path.startsWith('/blog/')) {
      priority = 0.9
      changefreq = 'weekly'
    } else if (path.startsWith('/products/')) {
      priority = 0.8
      changefreq = 'daily'
    }

    return {
      loc: path,
      changefreq,
      priority,
      lastmod: new Date().toISOString(),
      alternateRefs: [
        { href: `https://example.com/ko${path}`, hreflang: 'ko' },
        { href: `https://example.com/pt${path}`, hreflang: 'pt' },
      ],
    }
  },
  additionalPaths: async (config) => {
    // Add dynamic routes that need to be in the sitemap
    const posts = await fetch('https://api.example.com/posts')
      .then((res) => res.json())

    return posts.map((post) => ({
      loc: `/blog/${post.slug}`,
      changefreq: 'weekly',
      priority: 0.9,
      lastmod: post.updatedAt,
    }))
  },
}

Fix 2: Add image sitemap support

// next-sitemap.config.js
module.exports = {
  // ... other config
  additionalPaths: async (config) => {
    const posts = await fetch('https://api.example.com/posts')
      .then((res) => res.json())

    return posts.map((post) => ({
      loc: `/blog/${post.slug}`,
      changefreq: 'weekly',
      priority: 0.9,
      lastmod: post.updatedAt,
      images: post.images.map((img) => ({
        loc: img.url,
        title: img.alt,
        caption: img.description,
      })),
    }))
  },
}

Fix 3: Generate sitemap in a route handler

// app/api/sitemap.xml/route.ts
import { prisma } from '@/lib/prisma'

export async function GET() {
  const posts = await prisma.post.findMany({
    select: {
      slug: true,
      updatedAt: true,
    },
  })

  const xml = `

  ${posts
    .map(
      (post) => `
  
    https://example.com/blog/${post.slug}
    ${post.updatedAt.toISOString()}
    weekly
    0.8
  `
    )
    .join('')}
`

  return new Response(xml, {
    headers: {
      'Content-Type': 'application/xml',
      'Cache-Control': 'public, max-age=3600, s-maxage=3600',
    },
  })
}

Fix 4: Add canonical URLs to pages

// app/layout.tsx
export const metadata = {
  alternates: {
    canonical: 'https://example.com',
    languages: {
      'en': 'https://example.com',
      'ko': 'https://example.com/ko',
      'pt': 'https://example.com/pt',
    },
  },
}

// app/blog/[slug]/page.tsx
export async function generateMetadata({ params }: { params: { slug: string } }) {
  const post = await getPost(params.slug)

  return {
    title: post.title,
    description: post.excerpt,
    openGraph: {
      title: post.title,
      description: post.excerpt,
      url: `https://example.com/blog/${params.slug}`,
      images: [{ url: post.coverImage, width: 1200, height: 630 }],
    },
    alternates: {
      canonical: `https://example.com/blog/${params.slug}`,
    },
  }
}

Lessons Learned

1. Generate sitemaps after build. Use the postbuild script to generate sitemaps automatically:

{
  "scripts": {
    "build": "next build",
    "postbuild": "next-sitemap"
  }
}

2. Block AI crawlers in robots.txt. If you do not want your content used for AI training, explicitly disallow GPTBot, CCBot, and other AI user agents.

3. Submit your sitemap to Google Search Console. Automatic discovery works, but manual submission ensures faster indexing.

4. Include lastmod dates. Search engines use lastmod to prioritize crawling of recently updated pages.

5. Monitor crawl errors regularly. Google Search Console provides detailed reports on which pages are being crawled and which are failing.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.