Auto Sitemap, Schema, and Robots.txt: The 3 Files Most Developers Forget

The three most commonly deferred SEO files in a Next.js project are sitemap.ts, the schema markup components, and robots.ts. They are not hard to build. They just feel like polishing details when you are in the middle of building features.

The problem with deferring them: Google indexes your site starting from day one. Every day you wait on the sitemap is a day of indexing lag. Every page without schema markup is a page the AI search engines categorize with less confidence.

Here is how to automate all three so they require zero maintenance after setup.

Auto Sitemap From Supabase

Next.js 13+ supports a sitemap.ts file in the app/ directory that returns a MetadataRoute.Sitemap array. If you generate this from your database, every new piece of content is automatically included.

// app/sitemap.ts
import { MetadataRoute } from 'next';
import { supabase } from '@/lib/supabase';

const SITE_URL = 'https://yourdomain.com';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const staticRoutes: MetadataRoute.Sitemap = [
    { url: SITE_URL, lastModified: new Date(), priority: 1.0, changeFrequency: 'weekly' },
    { url: `${SITE_URL}/about`, lastModified: new Date(), priority: 0.9 },
    { url: `${SITE_URL}/blog`, lastModified: new Date(), priority: 0.9 },
  ];

  const { data: posts } = await supabase
    .from('blog_posts')
    .select('slug, updated_at')
    .eq('published', true);

  const postRoutes: MetadataRoute.Sitemap = (posts ?? []).map(post => ({
    url: `${SITE_URL}/blog/${post.slug}`,
    lastModified: new Date(post.updated_at),
    priority: 0.8,
    changeFrequency: 'never' as const,
  }));

  return [...staticRoutes, ...postRoutes];
}

Accessible at /sitemap.xml automatically. Submit this URL to Google Search Console once. After that, every new post is in the sitemap on the next build cycle.

Schema Markup Per Content Type

Schema markup tells search engines (and AI systems) what your content is. For a blog, you need at minimum BlogPosting schema on each post page and BreadcrumbList on every page with a hierarchy.

// components/JsonLd.tsx
export function BlogPostingSchema({ title, description, slug, publishedAt, image, readingTime }) {
  const schema = {
    '@context': 'https://schema.org',
    '@type': 'BlogPosting',
    headline: title,
    description,
    url: `${SITE_URL}/blog/${slug}`,
    datePublished: publishedAt,
    dateModified: publishedAt,
    author: {
      '@type': 'Person',
      name: 'Santosh Paudel',
      url: SITE_URL,
    },
    publisher: {
      '@type': 'Person',
      name: 'Santosh Paudel',
    },
    ...(image && { image }),
    ...(readingTime && { timeRequired: `PT${readingTime}M` }),
  };

  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}
    />
  );
}

Add this to the blog post page. The data comes from the database record — no manual schema writing per post.

For FAQ sections, use FAQPage schema:

export function FAQSchema({ questions }) {
  return (
    <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify({
      '@context': 'https://schema.org',
      '@type': 'FAQPage',
      mainEntity: questions.map(q => ({
        '@type': 'Question',
        name: q.question,
        acceptedAnswer: { '@type': 'Answer', text: q.answer },
      })),
    })}} />
  );
}

robots.ts

The robots.ts file in the app/ directory generates /robots.txt automatically:

// app/robots.ts
import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin', '/api/'] },
      { userAgent: 'CCBot', disallow: '/' }, // block training-only crawler
    ],
    sitemap: `${SITE_URL}/sitemap.xml`,
  };
}

Key decisions in this config:

• —Allow all search bots including GPTBot, PerplexityBot, ClaudeBot (they need access to cite you)
• —Block /admin and /api/ from crawlers
• —Block CCBot specifically (Common Crawl, used for training datasets — separate from search bots)
• —Include the sitemap URL so bots find it automatically

The Checklist

Before deploying any site:

• — sitemap.ts generates from the database and is accessible at /sitemap.xml
• — robots.ts allows search bots and blocks /admin
• — BlogPosting schema on all post pages
• — BreadcrumbList schema on all pages with hierarchy
• — generateMetadata in Next.js populates title, description, and OG tags per page
• — Sitemap URL submitted to Google Search Console

All of this takes about 3 hours to build once. After that, it maintains itself.

---

Resources

• —Next.js Metadata API — sitemap.ts — the official reference for generating sitemaps from a function in Next.js 13+
• —schema.org vocabulary — the full list of schema types; BlogPosting, FAQPage, HowTo, and BreadcrumbList are the most useful for a blog
• —Google structured data testing tool — validate your schema markup before submitting to Search Console
• —Google robots.txt documentation — the official spec; particularly important for Disallow rule syntax

Related Posts

• —AI SEO vs. Traditional SEO: What I Am Actually Doing Differently in 2026 — why the AI bot allow-list in robots.txt matters as much as the Google rules
• —Programmatic SEO for a Local News Site: What Actually Moved the Needle — how auto-generated sitemaps handle programmatic pages at scale
• —How I Structure a Supabase + Vercel Admin Panel From Scratch — where to put the CMS that feeds the auto-sitemap

---

Want the full SEO layer built into your Next.js project from day one? Sitemap, schema, robots.txt, and Open Graph are part of every project I build. See my services or get in touch.