Charging an AI bot to crawl your website is a viewability decision, not a revenue decision. Every bot you place behind a paywall is a choice as to which agents can still access read, cite and recommend to youand most website owners are on the verge of making this choice without realizing that it is now up to them to do so. The mechanism is HTTP 402, a status code written on the web in 1997 and lying dormant for almost thirty years. In July 2025, Cloudflare dusted it off to charge AI crawlers. On June 15, 2026, AWS added the same functionality to its firewall. Two of the biggest web infrastructure companies are now selling the same thing: a toll booth for machines. The question they ask you is who you can afford to refuse.
The toll booth that everyone reads as a revenue line
Launch messages present pay-per-crawl as new money, but it’s really about consent. For years, AI companies explored the open web to train models and find answers, and website owners had two straightforward options: leave the door open or block crawlers altogether and hope the rules were followed. Pay per crawl is the first mechanism that allows a website to say something more specific than yes or no. It allows you to set conditions. This is real change where the real price is consent and control, not pennies.
The reason all of this exists in the first place is a broken market. For 30 years, the principle was simple: let the robot in, it indexes you, it sends people away. The AI robots kept the first half and abandoned the second. Cloudflare explains why these bots are crawling places training at almost 80% of AI bot activity, with retrievals for research purposes that can actually return a citation a small portion of what remains. The rest is extraction that takes the content and returns no one. So this toll booth is the web trying to renegotiate a deal that AI crawlers have already stopped honoring.
This is where the celebration of the announcement messages and my reading separate. Once you can set conditions, they become a trade-off for visibility, and that’s the part that launch posts forget.
Who charges AI crawlers and how the mechanism works
Cloudflare launched the first major release on July 1, 2025. A publisher sets a flat rate per request for their domain. When an AI crawler requests a page, it either presents the payment intent in its request headers and gets the content with a 200 response, or it gets a 402 Payment Required response including price. Cloudflare describes itself as the merchant of record and handles checkout, which is important: because Cloudflare sits between crawlers and publishers, it can serve as a clearinghouse for both parties. At launch, the feature was a private beta and Cloudflare presented it as an early experiment, not a finished market.
AWS added the same idea to its Web Application Firewall on June 15, 2026. When an AI crawler requests a protected item, data feed, or licensed archive, AWS WAF may return HTTP 402, and the pricing and payment details pass through the x402 protocol in the form of a machine-readable manifest. Payment is made in stablecoin via a Coinbase facilitator, controls are carried out in WAF Bot Control and prices can be set by content path and bot type without touching the website code. Two of the largest infrastructure providers on the web are now selling a toll booth for machines within a year of each other.
They are not alone, even if the rest of the peloton is structured differently. TollBit operates a paywall and marketplace of bots and agents, charging for access to AI on usage-based terms rather than a flat rate per request. Akamai takes this monetization to its climax by integrating TollBit and Skyfire, announced in September 2025, so that its customers can apply tolling at the network layer without building it. Under the new tolls is x402the open standard from Coinbase for the same HTTP 402 status code. It is installed as a stablecoin and does not require any paying agent account, which would extend the burden from known and identified robots to anonymous robots. To get a complete idea of which crawlers are reaching you, the AI User Agent Landscape is the reference, because you cannot set the price for a bot that you cannot name.
In each of them, the toll is at the edge, in the CDN and the firewall, the same layer that already decides who gets in. Payment and authorization merge into a single point of control.
The open web market is being unbundled
The founding agreement of the open web is dismantled and resold line by line. The free traffic crawling deal that powered search, and the entire practice built on it, assumed that crawling was a cost a website paid to gain distribution. Pay per crawl is the first bill for a crawl that no longer reliably reciprocates.
This unbundling has a specific form. Access control and billing were once separate concerns, managed by separate systems. Now they’re on the same edge, in the same WAF rule, in the same Cloudflare dashboard. The firewall that decides whether a request is allowed becomes the meter that decides the cost of the request. When authorization and payment collapse into one layer, the decision to admit a visitor ceases to be a default choice and becomes a deliberate, priced choice. This is the question of the agent as visitor, the one that lies beneath the Amazon case v. Perplexity over whether an officer is even an authorized visitorgone from a legal abstraction to a setting that you configure.
And the favor that the crawl rendered has not disappeared; it moved. AI surfaces return people: Adobe Data for 2026 Shows 393% YoY Increase in AI-Referred Traffic to U.S. Retailers. This figure explains why tolling is firstly a visibility decision and secondly a billing decision. The crawler you would charge for and the response that refers a customer to you are often the same pipeline.
Whether you should charge depends on who you are
A publisher with an extensive licensed archive, news media, reference database, or proprietary data set has two things that make charging rational: content that’s actually worth paying for, and crawlers that it can afford to turn down because its distribution doesn’t depend on them. For this owner, the price to pay is a lever, and its use is judicious.
A content or commerce website seeking AI visibility is on the opposite side. For this owner, the robot is the distribution. Murder it and you might collect a few pennies, or you might remove yourself from where your customers now ask their questions. Tolling only works when two things are true at once: what’s behind it is worth paying for, and the robot you’re blocking is the one you don’t need to be seen by. The website whose strategy is to be the response given by an agent describes a a robot that he does not have the means to recharge. It’s the same reason why a website is now a source rather than a megaphone: The value is to be read, not to be walled off.
So the solution is to monitor the door first. Find out which AI bots actually reach you and what they take, separate those that provide answers and send referrals from those that extract and return nothing, and only then decide where and if a toll belongs. This is the test worth doing before anyone charges a dime: measure a bot on a path, see what it does on both your crawl volume and response presence, and let the result, not post-launch hype, inform your policy. You shouldn’t put a toll on traffic you’ve never looked at.
The open question is whether anonymous bots will pay
No public study has yet shown how much citation share a website actually loses by relying on a specific bot. The cost logic is sound: a bot that both reads you and provides an AI response can remove you from that response when you decline it. The magnitude of the effect in practice is unknown, and that’s why it remains a decision you make on your own data, not a rule of thumb.
The signal I will be watching is whether anonymous bot payment becomes real. Today, actionable models rely on identified crawlers that exhibit payment intent. The ambition of x402 is payment without prior registration, settled in stablecoin, which would extend the toll to the anonymous majority of automated traffic. If it works at scale, the paid web goes much further than a handful of named bots. When it comes to enterprise plumbing, pay-per-crawl remains a tool for big publishers and a curiosity for everyone else. Which of these comes true will determine whether the toll plaza is a niche or the new shape of the road.
Either way, the stand is built and it’s up to us to operate it. The bill you can see is the one the robot pays. The bill you can’t see is the answer you disappear from. Decide which one you watch.
More resources:
This article was originally published on No hacks.
Featured image: Roman Samborskyi/Shutterstock
