You added X-Robots-Tag: noindex to a page, but Google still indexes it. Or the page disappears from the index but comes back later. This is a common and confusing problem. X-Robots-Tag is a powerful tool, but it can be ignored or overridden. This guide explains why X-Robots-Tag noindex is ignored and how to fix it.
What X-Robots-Tag does
X-Robots-Tag is an HTTP header that tells search engines how to handle a page. It works like meta robots but is sent in the HTTP response instead of the HTML.
Example:
X-Robots-Tag: noindex, nofollow
It applies to the specific URL and can be used for non-HTML files like PDFs and images. It is often used when you cannot add meta tags to the HTML.
But X-Robots-Tag is not always respected. It can be ignored if other signals conflict or if the header is not delivered correctly.
Reason 1: header is not sent
The most common issue: the header is not actually sent in the response. You may have configured it in the server, but it is not applied.
What to check:
— is the header present in the response;
— is it sent for the correct URL;
— is it sent for all responses;
— is it removed by a proxy or CDN.
Use HTTP headers lookup to see whether X-Robots-Tag is present.
Reason 2: conflict with meta robots
If the page has both X-Robots-Tag and meta robots, they can conflict. Google may choose one over the other.
What to check:
— does the page have meta robots;
— does it conflict with X-Robots-Tag;
— which signal is stronger.
The fix: use only one method. If you need noindex, use X-Robots-Tag or meta robots, not both.
Reason 3: header is cached
If the header is cached by a CDN or proxy, changes may not take effect immediately. The old header may still be served.
What to check:
— is the header cached;
— does the CDN cache headers;
— was the cache cleared after changes;
— is the header updated on all edge nodes.
The fix: clear the CDN cache and verify the header is updated.
Reason 4: canonical or redirect overrides
If the page has a canonical pointing to another URL, Google may ignore the noindex on the current page and index the canonical URL instead.
What to check:
— does the page have a canonical;
— does it point to another URL;
— is the canonical URL also noindexed.
The fix: if the page should be noindexed, remove the canonical or ensure the canonical URL is also noindexed.
Reason 5: Google has not re-crawled the page
If the header was added recently, Google may not have re-crawled the page yet. The page may still be indexed from the previous crawl.
What to check:
— when was the header added;
— when was the last crawl;
— is the page in the index;
— has the page been re-crawled since the change.
The fix: request re-crawl in Search Console. Wait for Google to process the change.
How to diagnose X-Robots-Tag issues
A practical sequence:
— check the response headers;
— check meta robots;
— check canonical;
— check CDN cache;
— check the last crawl date;
— request re-crawl if needed.
If the header is present and no conflicts exist but the page is still indexed, Google may not have re-crawled yet. Wait and monitor. For a broader audit, SEO audit can help verify headers, canonical and indexing in one place.
How to fix X-Robots-Tag issues
Fix steps:
— ensure the header is sent;
— remove conflicting meta robots;
— clear CDN cache;
— fix canonical conflicts;
— request re-crawl;
— monitor indexing.
FAQ
Why is X-Robots-Tag noindex ignored?
Most often because the header is not sent, conflicts with meta robots, is cached, or the page has not been re-crawled.
Is X-Robots-Tag stronger than meta robots?
Both are signals. If they conflict, Google may choose one. Use only one method to avoid confusion.
Can a CDN remove X-Robots-Tag?
Yes. Some CDNs strip or modify headers. Check CDN settings and verify the header is present.
Does X-Robots-Tag work for PDFs?
Yes. X-Robots-Tag is often used for non-HTML files where meta tags cannot be added.
How long does it take for noindex to take effect?
Days or weeks. It depends on crawl frequency. Request re-crawl to speed up the process.
What if the page is still indexed after noindex?
Check if the header is present, if there are conflicts, and if the page has been re-crawled. Wait or request re-crawl.
Can I use noindex and canonical together?
Not recommended. If the page is noindexed, canonical is ignored. Use one or the other.
Does X-Robots-Tag affect ranking?
Indirectly. If the page is noindexed, it cannot rank. If it should be indexed, noindex removes it from search.
What if only some pages ignore noindex?
Check those pages for conflicts, caching, or missing headers. They may have different configuration.
How do I verify X-Robots-Tag is working?
Check response headers, use URL Inspection, and monitor indexing. If the page disappears from the index, noindex is working.
