Although publicly available generative AI platforms have become popular and prolific in recent years, it is important to remember the potential pitfalls that accompany them in B2B IT purchasing decisions. Lionfish offers a solution to traditional analyst search and generic AI search.
IT managers evaluating enterprise software and technology vendors are increasingly turning to general-purpose AI models for quick answers, sometimes in place of the analyst research and market reports that companies like Gartner traditionally provide. Lionfish says shortcuts carry real risk in a category where a bad recommendation could mean months of wasted implementation time and real budget at stake.
Today, many IT buyers rely on general-purpose artificial intelligence (AI) models to research suppliers and evaluate purchasing decisions. No doubt, this is partly due to the ease with which users can get near-immediate answers to questions or prompts. However, the confidence with which these systems operate also plays a vital role in their popularity. Ask such a model about a vendor’s capabilities or market position, and it will answer with immense speed and certainty. However, in high-stakes B2B IT purchasing decisions, is this confidence informed by the latest product and vendor information?
Rob Smith, a former Gartner Analyst and founder of Lionfish Technical Advisorsbelieves that this is often not the case, at least in the area where IT buyers can rely on generic AI models for their supplier selection. Although these systems operate with confidence whether or not they are correct, Smith says that confidence is not a reliable signal of correctness when it comes to vendor-specific claims.
Fortunately, Lionfish has built Periscopea product that relies on a specific claim: For B2B IT purchasing decisions, public AI models and traditional analyst research each miss something that the other has, and bridging the gap between them, rather than choosing one, is what makes a recommendation trustworthy.
This is a more accurate statement than most comparisons between AI and analysts make. Here’s what’s wrong when the AI responds a question of vendor selection in itself, what is outdated in analyst reports when updated on an annual basis and how human verification is supposed to fit in between for the specific job of choosing an IT vendor.
Where public AI models fall short
Smith’s criticism is not that the models are poorly constructed. This is because they are heavily trained in public data, and public data on IT suppliers is often wrong or obsolete. Vendor sites update marketing timelines, not precision timelines: product names change, features are renamed, pricing pages lag behind reality. A trained model that inherits its blind spots. Smith also points out a more subtle bias: Models trained on vendor-published content may lean toward vendors that post the most, rather than those that are best suited for a given buyer.
Where analyst reports fall short
With traditional analyst research, a market report takes hundreds of hours to build. It undergoes a thorough internal and legal review before shipping, both because the findings are rigorous and because they carry financial consequences, since a supplier’s placement can change its valuation. This same cycle explains why reports are only updated periodically, often once a year. . Smith says the underlying data may be out of date within three to six months of publication.
Comments and rating sites don’t reliably fill the gap either. Smith notes that vendors often encourage positive reviews with small gift cards, which boost overall scores regardless of product quality. His rule of thumb: Trustworthy reviews are usually negative ones, because almost no one writes spontaneous praise about a service they pay for, but many people speak up when they feel disappointed.
The role of human verification
It sits between the two failure modes rather than trying to outperform one or the other. The Aquarium, exclusive competitive and commercial intelligence from Lionfish the platform extracts data from AI as would a general model. Yet each data point is verified by one of the company’s offices about 30 former Gartner analysts before it is usable, and the underlying data is refreshed at least daily, more often in rapidly evolving categories like agentic AI. This cadence is the direct response to the annual report lag: instead of an already stale snapshot on release day, the system reflects what’s really true that week for the vendor a buyer is evaluating.
There is already a concrete example of how the verified layer works. Veracode used a survey automation feature inside the Aquarium, built on the same human-verified data, to complete a full Gartner Magic Quadrant vendor survey in a single day. This process usually takes two to three weeks. It’s narrow, designed for providers, but it’s a real, verifiable illustration of AI and human-verified data working together rather than one replacing the other.
The most useful question to ask
The slogan that Michael Disabato, former vice president at Gartner Research and now at Lionfish Tech Advisors, uses is four words: use AI, but verify. This isn’t to say that Lionfish has better AI; this is certainly not the case, and Smith does not pretend otherwise. The statement is more precise and more defensible: a recommendation is only valid to the extent that the data underlying it, and the data verified by a analyst who has spent years covering a market beats data pulled from all vendors that released this quarter. For an IT manager deciding which vendor to trust for a real technology purchase, this beats wondering which model is smarter.
Final Thoughts
Lionfish isn’t betting its AI is smarter. It is a safe bet that the verified data beats abundant data when a real the purchase is online. Public AI brings speed, the analyst brings judgment, and the recommendation is only trusted when both touch it before a buyer takes action.
FAQs
Why is general-purpose AI unreliable for B2B IT vendor selection?
It is trained largely on public data, including outdated or renamed vendor pages, so it can produce a reliable answer based on outdated facts, with no built-in way to report which assertions are current.
Why can an annual analyst report be outdated?
It takes hundreds of hours, as well as internal and legal review, before release, and Smith says the underlying data can be out of date within three to six months of that release, with an annual cadence incapable of reflecting an event that happens the week after it’s delivered.
What is the Aquarium?
The Aquarium is Lionfish’s competitive and commercial intelligence platform. It ingests data from AI, has former Gartner analysts verify each point, and updates at least daily, more often in fast-paced categories like agentic AI.
