So you’ve built your awesome Next.js website, now it’s time to make sure next.js robots.txt is set up correctly. But what if search engines aren’t seeing it the way you want? That’s where the robots.txt file comes in, and it’s more important than most developers realize.

Whether you want to improve your Next.js SEO, reduce unnecessary crawling, or block bots from accessing private routes like /admin or /api, having the correct Next.js robots.txt setup is essential.

In this easy-to-follow guide, we’ll show you exactly how to create and configure a robots.txt file in a Next.js app, step-by-step, with no technical jargon. Let’s help search engines crawl your site the right way.

What is robots.txt?

The robots.txt The file is like a polite bouncer at a club. It tells search engine bots which parts of your website they’re allowed to visit and which areas are off-limits.

Basic Syntax of robots.txt

Here’s a simple example:

User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml

• User-agent: Specifies which bot (Googlebot, Bingbot, etc.)
• Disallow: Blocks access to paths
• Allow: Grants access even in disallowed folders
• Sitemap: Points bots to your sitemap

Why robots.txt Matters for SEO

If Google crawls pages it shouldn’t (like /cart or /dashboard), you waste crawl budget and risk exposing sensitive content. robots.txt is how you take control of what bots see.

Top Reasons You Need One:

• Prevent indexing of non-public pages
• Improve crawl efficiency
• Avoid duplicate content issues
• Help Google focus on your money pages

How Next.js Handles SEO

Next.js is fantastic for performance and user experience. But it doesn’t create a robots.txt file by default.

Why This Matters

Since your site might be dynamic (e.g., server-side rendered or hybrid), you need to be intentional about robots.txt.

Setting Up robots.txt in Next.js

There are two main ways to add a robots.txt file in Next.js:

1. Static robots.txt
2. Dynamic robots.txt via API route

Let’s explore both.

Method 1 – Static robots.txt File

This is the easiest and most common approach.

Steps:

1. Go to your Next.js project root.
2. Navigate to the /public directory.
3. Create a new file named robots.txt.

Example:

User-agent: *
Disallow: /admin/
Sitemap: https://yourdomain.com/sitemap.xml

User-agent: *
Disallow: /admin/
Sitemap: https://yourdomain.com/sitemap.xml

Once deployed, this will be available at:
https://yourdomain.com/robots.txt

When to Use:

• Your rules don’t change by environment
• You don’t need dynamic paths

Method 2 – Dynamic robots.txt with API Route

Need more control? Maybe you want to serve different robots.txt content in development vs production? That’s where API routes shine.

Steps:

1. Create a new file under pages/api/robots.ts or .js

export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain');
  res.send(
`User-agent: *
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xml`
  );
}

export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain');
  res.send(
`User-agent: *
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xml`
  );
}

2. In your next.config.js, redirect /robots.txt to the API route:

module.exports = {
  async rewrites() {
    return [
      {
        source: '/robots.txt',
        destination: '/api/robots'
      }
    ];
  }
}

module.exports = {
  async rewrites() {
    return [
      {
        source: '/robots.txt',
        destination: '/api/robots'
      }
    ];
  }
}

Now you can generate robots.txt dynamically. Super useful for multi-environment setups!

Dynamic vs Static – Which One Should You Use?

Using Next.js with next-sitemap for SEO Automation

Want to automate everything? next-sitemap is your best friend. It can generate sitemap.xml and robots.txt automatically – saving tons of time.

But before you dive into automation, you might want to manually generate and test your robots.txt file to make sure you understand the basics and get the right output.

👉 Use this free online tool: ServerAvatar Robots.txt Generator
It’s a quick and simple way to build a clean, SEO-friendly robots.txt in seconds.

Install next-sitemap:

npm install next-sitemap

npm install next-sitemap

Create a config file: next-sitemap.config.js

module.exports = {
  siteUrl: 'https://yourdomain.com',
  generateRobotsTxt: true,
  robotsTxtOptions: {
    policies: [
      { userAgent: '*', allow: '/' },
    ],
    additionalSitemaps: [
      'https://yourdomain.com/my-custom-sitemap.xml',
    ],
  },
}

module.exports = {
  siteUrl: 'https://yourdomain.com',
  generateRobotsTxt: true,
  robotsTxtOptions: {
    policies: [
      { userAgent: '*', allow: '/' },
    ],
    additionalSitemaps: [
      'https://yourdomain.com/my-custom-sitemap.xml',
    ],
  },
}

Update package.json:

"scripts": {
  "postbuild": "next-sitemap"
}

"scripts": {
  "postbuild": "next-sitemap"
}

After building your app, it will auto-generate robots.txt and sitemap.xml under /public.

Best Practices for robots.txt in Next.js

• Always include your sitemap
• Don’t block essential JS or CSS files
• Use different rules for staging vs production
• Avoid Disallow: / unless under construction

Tips for Handling Crawlers and Bots

• Use User-agent: Googlebot for Google-only rules
• Block known bad bots if needed
• Monitor crawl errors via Google Search Console

Validating and Testing Your robots.txt

Tools You Can Use:

• Google robots.txt Tester
• Use curl:

curl https://yourdomain.com/robots.txt

curl https://yourdomain.com/robots.txt

• Browser testing works too!

Common Mistakes to Avoid

• Blocking /static/ or _next/ – your site might break!
• Leaving Disallow: / in production
• Forgetting Sitemap: directive
• Using uppercase file names (use robots.txt, not Robots.TXT)

Advanced Tips for Large-Scale Projects

• Multi-language support? Serve dynamic robots.txt per locale
• Different environments? Use environment variables
• Subdomains? Each subdomain should have its own robots.txt

Conclusion

Getting your robots.txt right might seem like a small thing – but it can make a big difference in how search engines see (or ignore) your content. With Next.js, you’ve got powerful tools to make this setup easy, whether you go the static route, API-based, or automated with next-sitemap.

Now that you’ve got the full picture, it’s time to take control of your SEO game. Go build, block, and boost!

FAQs

The post How to Optimize Next.js robots.txt for Better SEO first appeared on ServerAvatar.