AI bots are increasingly affecting website performance, analytics, infrastructure costs, and content visibility. New research and infrastructure data suggests the challenge is no longer just scraping, but managing how automated traffic interacts with websites and the businesses that rely on them.
Scratching is the least of the problems
Many discussions among SEOs and site owners center around AI bot scraping. There is a legitimate concern that AI systems harvest content for LLM training with virtually zero attribution when the content is remixed into an AI response.
- Site owners are concerned about intellectual property.
- Search marketers are concerned about how AI systems use their content.
But infrastructure teams are increasingly facing different and equally consequential problems.
The banality of robots that get lost and scratch objects
The problem is increasingly that many robots create unnecessary overhead, consume resources, and sometimes get trapped in inefficient loops.
According to the report, a recurring pattern involved Meta’s external meta-agent crawler tracking URL variations for days before mitigation systems were taken into account.
This type of behavior is not malicious. This is automation operating with poor coding practices or insufficient safeguards.
Cloudflare’s David Belson illustrated the banality of lost bots draining resources:
“There’s the person who didn’t know what they were doing yesterday, but Vibe coded a robot today and let it go. They don’t even bother to check the robots.txt file.”
This observation reflects an important reality. Today’s infrastructure problems now come from poorly designed automation operating at scale.
Robots consume resources without creating value
The consequence of this behavior is that websites spend resources serving automated traffic that may provide little or no business value in return.
This is a big problem for e-commerce sites. Unlike static page requests, shopping cart requests typically bypass caching and require the server to use resources. Depending on the site architecture, these requests can trigger the execution of PHP, database queries, session management, and other resource-intensive processes.
Seen in this light, scraping is the least of a website’s problems. A crawler that repeatedly triggers expensive application logic and consumes server resources degrades the performance of legitimate visitors.
The economic impact should not be ignored. According to the report, approximately 80% of AI exploration activities are associated with model training, eclipsing search, or user-driven explorations.
For many businesses, the question is: Is there value returned from this traffic to justify the resources consumed?
Businesses are stuck between visibility and cost
If the solution was to simply block bots, the problem would be solved. Unfortunately, many resource-consuming automated systems are also tied to discoverability and visibility.
Some bots help search engines discover content. Some can contribute to AI citations and visibility in AI-generated answers. Others may simply consume content and resources without producing directly measurable business benefits.
Businesses are asked to absorb the costs of automated traffic while evaluating whether that traffic provides enough visibility to justify those costs.
The question now: which robots are worth paying for?
The report argues that site owners should ask themselves this question:
Which robots, on which parts of my site, under what conditions?
Bot management affects visibility, infrastructure costs, and site performance. The goal is to align automated traffic with business goals.
Traffic numbers may already be affected
Automated traffic also affects website analytics. According to the report, AI bot traffic has increased by 300% over the past year. By the end of 2025, approximately one in 31 visits to the TollBit network came from an AI bot.
As automated traffic increases, traffic volume alone becomes a less reliable indicator of audience growth.
A site may see an increasing number of visits without experiencing a corresponding increase in customers, subscribers, conversions or revenue. In some cases, additional traffic can be automated.
The report claims that the most meaningful signals come from metrics related to real business outcomes, including branded search demand, direct traffic, engagement quality, and revenue.
As automated systems account for a larger share of overall traffic, raw visit counts become less useful as a standalone measure of success.
Solutions and mitigation tactics
The report calls for a deliberate approach to managing robots.
The first step is visibility.
Before making any changes, site owners need to understand what automated traffic actually does. The goal is not to identify each bot individually, but to identify patterns such as repeated queries, loops, and dynamic endpoint-centric activities.
The second step is to protect expensive site functions.
Shopping cart URLs, checkout paths, internal search pages, filtered product pages, and parameter-rich URLs often consume significantly more resources than standard content pages. Restricting unnecessary robot access to these areas can reduce waste without affecting important content.
The report also recommends separating search bots from AI bots.
Not all robots offer the same value. Search bots directly contribute to discoverability and deserve broader access than AI crawlers or unknown scrapers.
A single policy applied to each automated system can no longer be justified as the ecosystem becomes more complex. This is why the report calls for targeted changes rather than blanket restrictions.
The goal is not to eliminate automated traffic. The goal is to manage it in a way that supports business goals while reducing unnecessary costs. One solution is to decide which bots can access specific parts of a site and under what circumstances.
Takeaways
Bot traffic is no longer primarily a scraping problem. Data suggests this is now a business infrastructure, visibility, analytics and management problem.
The biggest challenge is that many bots consume resources, trigger expensive features, inflate traffic metrics, and generate costs that site owners must absorb.
Bot management is not about blocking as many bots as possible. This involves managing bots based on what the site optimizes for by distinguishing between valuable and useless automated traffic.
Read the report based on Kinsta data:
Reality Checking AI and Bot Traffic
Featured image by Shutterstock/DC Studio
