How to handle large amounts of URLs

[hodge-jai]

Hi there,

I'm trying to figure out how to generate sitemaps for a few hundred thousand urls using next-sitemap. Each url is of the form https://mywebsite.com/<permalink> where <permalink> is a unique identifier. These pages are served dynamically using SSR. Essentially, I want next-sitemap to generate a set of sitemaps for these URLs, but I'm not entirely sure what's going on under the hood. I've followed some examples online, and this is my set up so far

//next-sitemap.config.js

/** @type {import('next-sitemap').IConfig} */

const siteUrl = "https://mywebsite.com/"

module.exports = {
  siteUrl,
  generateRobotsTxt: true,
  sitemapSize: 40000,
  exclude: ["/404"],
  robotsTxtOptions: {
    policies: [
      {
        userAgent: "*",
        disallow: ["/404"],
      },
      { userAgent: "*", allow: "/" },
    ],
    additionalSitemaps: [
      `${siteUrl}sitemap.xml`,
      `${siteUrl}/permalink-sitemaps.xml`,
    ],
  },
};

//pages/permalink-sitemaps.xml/index.js

export const GetPost = async () => {
  const data = await fetch("http://mywebsite.com/api/permalinks", {
    method: "GET",
  });
  const res = await data.json();
  return res;
};

export const getServerSideProps = async (ctx) => {
  const siteUrl = "https://mywebsite.com/";
  const data = await GetPost();
  const fields = data?.map((data) => ({
    loc: `${siteUrl}/${data}`,
    lastmod: new Date().toISOString(),
  }));

  return getServerSideSitemapLegacy(ctx, fields);
};

export default function Site() {}

At build time, it will generate the sitemap.xml index which contains two locations, https://mywebsite.com/sitemap-0.xml which contains all URLs for my static pages, and https://mywebsite.com/permalink-sitemaps.xml, which when accessed errors out. Ideally I'd like sitemaps generated for my permalinks to be of the form permalink-sitemaps-0.xml,permalink-sitemaps-1.xml, permalink-sitemaps-2.xml, etc. How would I go about doing this?

[vzrenggamani]

Hello @hodge-jai, hope you doing well,
Since Nextjs 13.4.7, its recomended to use app route to render sitemaps using SSR, since ServerSideProps are not "quite" supported since now, here some example to getting start you with:

//app/sitemap.ts
export const GetPost = async () => {
  const data = await fetch("http://mywebsite.com/api/permalinks", {
    method: "GET",
  });
  const res = await data.json();
  return res;
};

export default function sitemap(): MetadataRoute.Sitemap {
  // News Article
  const posts = getPost()
  const routesNews = posts.map((post) => ({
    url: `${URL}/${post.slug}`,
    lastModified: new Date().toISOString(),
  }))

  // Static pages
  const routes = [
    '',
    '/another-static-routes'
  ].map((route) => ({
    url: `${URL}${route}`,
    lastModified: new Date().toISOString(),
  }))

  return [...routes, ...routesNews]
}