Over the weekend, discussions shared with Claude unexpectedly appeared in Google search results.
On July 27, I checked the crawl rules on Claude’s sharing pages and found a noindex directive hidden behind a block, preventing Google from accessing it.
The sharing path is blocked by robots.txt on claude.ai. These pages also return an X-Robots-Tag header set to none, which Google Bot Meta Tag Guidelines treat it the same as noindex and nofollow.
For this reason, the rules conflict with each other. According to Google’s advicea noindex tag only works if the crawler is allowed to access and read the page. If a page is blocked by robots.txt, it can still be indexed if other pages link to it, since Googlebot notes the URL without actually opening it.
This pairing is not unique to AI companies. If Google discovers a URL blocked elsewhere, it can list the URL without crawling the noindex.
What happened
404 media reported On Monday, Claude’s sharing pages appeared via the site: operator. TechCrunch reported be able to find medical equipment, internal company documents and files containing the names and telephone numbers of children of primary school age. The artifacts, documents and mini-applications built inside Claude, appeared via a second path.
Much of the coverage, including The decoderreported that sharing pages lacked a noindex tag. Since July 27, a noindex header is served and is behind a robots.txt block. It’s unclear if this header was in place before the weekend, as I was unable to confirm it.
What exploration rules return
Results of my July 27 checks on claude.ai:
- The robots.txt file on claude.ai disallows /share/* under the User-agent: * group, without a separate Googlebot group, so this is the group Google uses.
- A live share URL responds with the x-robots-tag: none header during a GET request.
- The same header appears when the request claims to be Googlebot and the response includes Vary: User-Agent.
- The path /public/artifacts/ is not listed in robots.txt.
I didn’t check if the artifact URLs returned a page-level noindex, so these two paths can’t be compared based on their checks.
Independent IT consultant Daniel J. Glover reported the same conflict on July 26. My check shows that the configuration was still in place on July 27, after it was reported that discussions were no longer displayed in search results, but I have not been able to establish what was served to Google when these URLs were first discovered.
Why blocking a page doesn’t delete it
Robots.txt helps guide how search engines crawl your website, but it doesn’t decide what will be indexed. Google has been clarifying this for years.
For example, John Mueller explains that even pages blocked by robots.txt can appear in search results if other pages link to them. If you want to keep a page out of search results, using the “noindex” tag is the right choice. Martin Splitt also recommends avoiding placing both rules on the same page. Google recently re-emphasized this distinction, as shown their update last week.
Blocking URLs with robots.txt does not remove pages that are already indexed. For more permanent removal, Google’s recommendation is to use a “noindex” tag that its crawler can read.
Anthropic’s response
Anthropic told TechCrunch that share links will only appear in search results if people post them in locations accessible to crawlers, and that links sent via private messages will not be indexed.
Spokeswoman Amie Rotherham added that these links “cannot be guessed or discovered unless people choose to share them themselves”. This addresses one way to discover a public sharing URL, but it doesn’t resolve the conflicting rules I found on July 27.
Google spokesman Ned Adriance said search engines do not determine which pages are made public and that Google provides site owners with control over crawling and indexing and tracks those controls.
Why it matters
Any discussions or artifacts you haven’t shared are still accessible to anyone with the link. In Claude, go to Settings > Privacy > Shared Discussions to see a full list.
Unsharing a discussion disables the direct link, but removing the URL from Google is another thing. As a Claude user, you cannot control what it serves to Googlebot or submit a removal request, as these are managed by Anthropic.
Artifacts are published through their own process, making unsharing a discussion and unpublishing an artifact from that discussion two separate actions.
Beyond changing settings, consider establishing a rule around sharing discussions and AI artifacts. If a discussion contains something you wouldn’t post publicly, don’t share it via a link that might be indexed later.
Looking to the future
The model is not unique to Anthropic. OpenAI removed shared ChatGPT chats of search in August 2025 and Google Bard’s blocked transcripts of indexing in 2023. Each case involved a public sharing URL that traveled further than its owner might have expected.
Three companies have now seen discussions with their customers appear in searches. This is reason enough to treat a public sharing link as a public web page whenever it exists.
Featured Image: Cast of Thousands/Shutterstock
