Ask a chatbot a question and watch what happens to the web behind it. He reads 30 or 40 pages to construct your answer, deletes what it needs, and gives you a tidy paragraph. You never see the pages, never click on them. The site that “won”, whatever “win” means today, receives a citation in light gray text and not a single visitor.
This is most of what is read on the web today, and the companies that operate the pipelines can see this happening. On the Cloudflare network, bots have surpassed people in real web page requests this year, from 57.5% to 42.5%. Its general director marked the crossover in Juneabout 18 months ahead of its own forecast, and attributed the increase to agents scraping pages on people’s behalf. AI is by far the fastest growing segmentup about eight times faster than human visits over the last year. The Web is more read than ever. But not by people.
Here’s the uncomfortable part that no one really wants to say out loud. The rules we made for the web, those about quality, access, and honesty, were written for a human audience. The audience has changed. Rules are repealed to match, quietly, without anyone putting them to a vote. Three of them are already going there. Here’s which one and why.
Who Controls the Web and Why They Care
For about twenty years, the web had a traffic manager, and that person was Google. Insert keywords where no one can see them, buy a thousand links, spin front door pages, and sooner or later something falls on you from a great height. Most people thought it was a hygiene issue. Google keeps the web tidy on behalf of everyone, out of the goodness of their hearts.
It was nothing of the sort. Google controlled the web because it sold advertising based on an index of it, and an index full of garbage is worth less to advertisers. Housekeeping was store front maintenance.
Andy Baio observed the same logic a decade ago, when Google let Books and its news archives rot as they ceased to make a living, and warned against trust a company to do a library’s job. He was generous. The library has never been more than a side effect of the advertising industry, kept alive while it paid and abandoned when it didn’t.
I spent the better part of six years on the search quality and web spam of this operation, so I can tell you that the work was real and the engineers meant it. There was never any doubt as to why it was funded.
Now look at the new readers. A response engine doesn’t sell ads in a tidy index because it doesn’t keep any for you to browse. He reads, weighs what he finds and repeats what he wants. A weak page is not punished. It just doesn’t get picked up, which from the page’s perspective is worse, because at least one penalty came with an email. There is no longer a guard. There is a doorman who never tells you why you didn’t enter.
And Google now has both positions, which is really funny. It still runs the ad-supported index, the a court has just ruled A illegal monopoly in both research and advertising sold against it. And it builds the response engine that makes that index irrelevant, then inserts ads into it as quickly as the format allows: sponsored images mixed with image results, ads in AI summaries, a brand new bolted shopping pipe. A company that defends its old business by building what ends it and selling advertisements for the murder weapon. You don’t need a diagram.
Who pays to enter
The old arrangement was a fair, even generous, trade. Let our robot in for free, Google said, and we’ll send readers back to you. The sites not only tolerated exploration; they fought to be explored faster and indexed more deeply, because exploration was the on-ramp to an audience.
AI exploration makes no such offer. It takes the same content, turns it into a response and returns nothing. No clicks, no readers, no ads to sell. Cloudflare, which represents about a fifth of the web, said the silent party into a microphone in July 2025: the old agreement is brokenSo new sites in its network now block AI crawlers by default, and owners can charge per crawl through a marketplace that gives any crawler unwilling to pay a polite “required payment” and nothing else. Thirty years of begging Google to explore more, the reflex is now to install a toll.
In Britain, 31 publishers have overcome the blockage. They transformed the old robots.txt filethe polite ones please – haven’t the robots learned to ignore, in a binding contract: load the page, reuse an item without paying and you’ve accepted a £500 bill which a county court can enforce like any other debt. No one has collected OpenAI yet, but this move tells you where this is going. The toll now has a price list.
Why now, and not five years ago, when the scratching started? Because writing has become a rare thing. When a model is only as good as the text it learns from, and the open web fills with the model’s own exhausts, authentic human writing ceases to be raw material and becomes the reward. The owners understood that they were sitting on input, and input has a price.
So look at what they actually do, because that tells you the real argument. The New York Times is suing OpenAI for training on its archives and, at the same time, licensing of this same archive to Amazon. OpenAI has signed the Guardian, Atlantic, Washington Post, News Corp and many others. No one on this list is trying to return the machines. They are haggle over the rate. The genie is out, the demand is structural, and the fight was never about whether it would happen. It’s about who gets paid and how much. The road is built one way or another. The debate concerns only the balance sheet.
What counts as cheating now
There is one rule older than all the others: never show the robot something different from what you show the person. It’s camouflage, and camouflage erases you. Every SEO learns this from day one.
But read Google’s own definitionnot folklore. Cloaking offers different content to users and search engines with the intention of manipulating rankings and deceive people. What caused cheating was never that the two versions differed, but the intention to deceive sitting below. Showing the crawler a page about vacations and the human a page about counterfeit pharmaceuticals is camouflage. Serving the same facts in a cleaner, machine-readable form to something that only reads machine-readable forms is not, and never has been. Google’s own advice says it: adapt the presentation to your liking, as long as the background is the same.
So when the reader is an agent who wants structured data instead of your hero image and cookie banner, passing them structured data is not a trick. He answers in the language he asked. The taboo assumed that the crawler replaced a pair of human eyes. Forget this assumption, and most taboos go with it.
This is where I have to be the annoying ex-spam for a while, because the line still exists and it just moved. Lie to the machine to change a ranking, or give it something you would never back up in front of a person, and it will still likely be considered an intent to deceive, and in matters of health, money, and anything else where a wrong answer hurts someone, it remains dangerous. Deception is the line. Formatting your content for a reader that happens to be a machine has never been a bad thing.
What do people have left?
None of this is a prediction. This has already happened to the parts of the web that machines are most interested in, and it’s continuing from there. The direction is not up for debate. The only current questions concern the terms: who gets paid, who gets isolated, and which business model gets the right to decide what a good answer looks like.
This last point is wide open and worth watching closely, because the people building the new gateway can’t agree on how to profit from it. Google embeds advertisements in responses. Perplexity I tried the ads, I killed themand now swears that a user must believe they are getting the best answer rather than the highest paid one. Anthropic keeps Claude publicity-free and says it loudly. OpenAI is test ads while promising that they won’t change the answer, which is precisely the promise Google made regarding search results, and we all remember when that got old. Whoever wins this argument inherits the old gatekeeper position and gets to define “good” for a network that most people now experience second-hand.
Which leaves a smaller, weirder job for the rest of us. The machine-read web doesn’t care how clever your title is or how smooth your page is. He cannot be charmed or flattered. It keeps what is useful to the answer and ignores the rest, so the work that survives is the work beneath the decoration: the original reporting, and the judgment of whether an answer is indeed right before it reaches someone who will act on it. The web has spent 30 years learning how to work for people. Now he must be useful for something without thumbs to raise or hands to clap.
More resources:
This article was originally published on The inference.
Featured image: Natalya Kosarevich/Shutterstock
