Indexability Signals for Generative Engines: Eliminating Accidental noindex and Header Dropouts

TL;DR

A page can return 200 OK and look perfect in a browser, yet still tell crawlers not to index it. The usual causes are a stray noindex meta tag, an X-Robots-Tag HTTP header set by a server or CDN, or a robots.txt rule that stops crawlers from ever seeing the directive. If a page is not indexed, AI search engines that retrieve from those indexes cannot cite it.

• Check both the HTML <meta name="robots"> tag and the X-Robots-Tag response header on every public template.
• Do not block a page in robots.txt if you want crawlers to read its noindex. A blocked crawler cannot see the directive.
• When directives conflict, crawlers apply the most restrictive one.

Why indexability matters for generative engines

AI answer engines such as ChatGPT search, Perplexity, Copilot and Google AI Overviews ground many answers in pages fetched from a search index or a live retrieval step. If your page is excluded from the index, it is usually excluded from the pool of sources an answer can cite. Crawler access (robots.txt) decides whether a bot may fetch a page. Indexability decides whether the fetched page may be stored and shown.

The three places a noindex can hide

Signal	Where it lives	Typical accidental cause
Robots meta tag	HTML <head>	A staging setting left on after launch, or a CMS "discourage search engines" checkbox
Crawler-specific meta tag	HTML <head>, for example name="googlebot"	A plugin targeting one crawler that nobody remembers adding
X-Robots-Tag header	HTTP response headers	A server, CDN or edge rule applied to a whole path or file type

Header directives are the hardest to spot because they never appear in the page source. Google documents that any directive usable in the robots meta tag can also be sent as an X-Robots-Tag header, which makes it useful for PDFs and images, and easy to apply too broadly.

How conflicts are resolved

When several directives apply, Google states that it uses the most restrictive one. A page carrying index in its meta tag and noindex in its header is treated as noindex.

The most common conflict involves robots.txt. If robots.txt disallows a URL, compliant crawlers never fetch it, so they never see its noindex. The URL can still appear in results without a description when other sites link to it. To remove a page from the index, allow crawling and serve noindex. To keep a page indexed, remove both blocks.

Audit workflow

1. List your public templates. Home, article, product, category and landing pages usually cover most URLs.
2. Check the response headers. Fetch each template and look for X-Robots-Tag.
3. Check the rendered HTML. Search the <head> for robots and crawler-specific meta tags, including tags injected by JavaScript.
4. Cross-check robots.txt. Confirm none of the pages you want indexed are disallowed.
5. Confirm with the search engine. Use the URL Inspection tool in Google Search Console and the equivalent in Bing Webmaster Tools.

# Show only the robots header for a page
curl -sI https://YOUR-DOMAIN.COM/YOUR-PAGE/ | grep -i x-robots-tag

# Fetch as a search crawler to catch user-agent specific rules
curl -sI -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" https://YOUR-DOMAIN.COM/YOUR-PAGE/

# Find robots meta tags in the HTML
curl -s https://YOUR-DOMAIN.COM/YOUR-PAGE/ | grep -io '<meta[^>]*name="[^"]*bot[^"]*"[^>]*>'

Correct configurations

For a public page you want indexed and cited, you do not need a robots meta tag at all. Indexing is the default. If your CMS requires one, use:

<meta name="robots" content="index, follow, max-snippet:-1">

For private areas such as dashboards or account pages, send the header from the server so it covers every response:

# nginx example
location /YOUR-PRIVATE-PATH/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

Note that max-snippet:0 and nosnippet limit how much text a search engine may show. Google documents that these controls also apply to its AI features, so use them only where you intend to restrict quoting.

Common antipatterns

• Shipping a staging-wide X-Robots-Tag: noindex rule to production.
• Blocking a URL in robots.txt and expecting its noindex to work.
• Adding noindex to paginated or filtered pages that hold your only copy of key content.
• Setting noindex in JavaScript after load, so crawlers see different signals depending on whether they render.
• Using nosnippet sitewide, which leaves engines nothing to quote.

References

• Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications." developers.google.com
• Google Search Central, "Block search indexing with noindex." developers.google.com
• Koster, M., Illyes, G., Zeller, H. and Sassman, L. (2022). RFC 9309: Robots Exclusion Protocol. IETF. rfc-editor.org
• Fielding, R., Nottingham, M. and Reschke, J. (2022). RFC 9110: HTTP Semantics. IETF. rfc-editor.org
• Aggarwal, P. et al. (2023). "GEO: Generative Engine Optimization." arXiv:2311.09735. arxiv.org