Coolguyzone/chore/ai rules robots

[coolguyzone]

DESCRIBE YOUR PR

This PR addresses some of the issue found here: https://www.mintlify.com/score/sentry

These updates to robots.txt add AI bot rules and content signals to help agents better navigate our docs.

More on content signals: https://contentsignals.org/

Some details of this change:

• All bots get Allow: / — appropriate for public documentation
• The wildcard User-agent: * block now also has an explicit Allow: /
  (previously it had no rule at all, which is technically ambiguous)
• If we ever wants to block a specific crawler, it's a one-line change to Disallow: /.
• isDeveloperDocs is now evaluated at module load time rather than inside the
  handler, which is fine since it's a build-time env var

IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs to go live.

• Urgent deadline (GA date, etc.):
• Other deadline:
• None: Not urgent, can wait up to 1 week+

SLA

• Teamwork makes the dream work, so please add a reviewer to your PRs.
• Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it.
  Thanks in advance for your help!

PRE-MERGE CHECKLIST

Make sure you've checked the following before merging your changes:

• Checked Vercel preview for correctness, including links
• PR was reviewed and approved by any necessary SMEs (subject matter experts)
• PR was reviewed and approved by a member of the Sentry docs team

LEGAL BOILERPLATE

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

EXTRA RESOURCES

• Sentry Docs contributor guide

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
develop-docs	Ready	Preview, Comment	May 18, 2026 8:49pm
sentry-docs	Ready	Preview, Comment	May 18, 2026 8:49pm

[sergical]

noice