The flood of AI crawlers and automated agents is forcing publishers and content owners to face the challenge of deciding who has access to their content and under what conditions.
To help address this issue, the IAB Tech Lab today released new guidance on bot and crawler management strategies, opening the document for public comment until June 26, 2026.
These tips are designed to help publishers, content owners and ad tech companies think about how they manage non-human traffic, including AI systems that remove or access content. It also complements the organization’s recently released CoMP V1 API, a framework intended to support communication and permissions between AI systems and publishers.
“Giving content owners clear, actionable guidance is essential if we want adoption to meaningfully advance,” said Anthony Katsur, CEO of IAB Tech Lab, in a statement. “This work helps simplify a complex area so that companies can make decisions that fit their business while supporting a more sustainable market.” »
Should you give AI access to your content?
The IAB paper highlights how quickly AI exploration has become a business issue in publishing and advertising. Many organizations do not have a formal policy regarding bots and crawlers, despite growing concerns about content consumption by AI systems, strain on infrastructure, and complexity of monetization.
According to the IAB Tech Lab, these guidelines were created because many publishers and content owners were struggling to make the tradeoffs needed to manage access to AI. According to the group, blanket blocking of bots is no longer practical as AI systems become more integrated into the web ecosystem.
Instead, the paper outlines a range of approaches that businesses can take, as well as the operational, financial and strategic implications of each.
Your customers are searching everywhere. Make sure your brand introduces himself.
The SEO toolkit you know, plus the AI visibility data you need.
Start free trial
Start with
This advice is aimed less at engineers than at business leaders trying to understand the costs and implications of managing AI crawlers.
“Content owners are being asked to make important decisions quickly, often without a clear framework,” said Shailley Singh, executive vice president and chief operating officer at IAB Tech Lab. “These tips help them break down the options so they can choose an approach that fits their goals.”
The IAB Tech Lab said it will continue to work with its members and industry participants to support the implementation of the CoMP API and Bot Management Guidelines. The organization is accepting public comments on the proposal until June 26 before finalizing the framework.
Go here to read and comment on the guidance.
