\
We face a conflict between economic pressure to produce rapid and cheap research results and the scientific demand for rigor. Hundreds, if not thousands, of realistic characters can be generated in minutes by vendors promising solid results. But these often function as methodological black boxes, producing results that cannot be validated, may contain hidden biases, and can quietly mislead decision-making.
The synthetic data market is growing rapidly, with valuations expected to increase from approximately $267 million in 2023 to more than $4.6 billion by 2032. Driven by the demand for instant information in an always-on economy, 95% of insight leaders are planning to use synthetic data over the next year and the appeal is clear. Speed, scalability, cost-effectiveness and the ability to generate insights to niche audiences are key factors.
To move synthetic testing from a purely experimental approach to a reliable and scalable practice, organizations must address these risks directly. Several approaches can help overcome skepticism and create a more sustainable model. It is important to identify the main problems and solve them directly.
Although cost savings and speed of information are compelling reasons for adoption, several challenges remain. The most successful organizations understand the strengths and weaknesses of different synthetic tools and when to use them.
Your customers are searching everywhere. Make sure your brand introduces himself.
The SEO toolkit you know, plus the AI visibility data you need.
Start free trial
Start with
Common challenges with synthetic search approaches
Why general LLMs fail to meet expectations
Why can’t you just ask your research questions to ChatGPT? A common misconception in synthetic research is that providing an LLM with a detailed history guarantees a representative result. Recent large-scale experiments suggest otherwise.
Initial studies show that incentivizing an LLM such as ChatGPT, Claude, or Gemini to produce more content per person increases bias/homogeneity instead of creating a diverse set of results. For example, the personas used to predict the results of the 2024 US presidential election (with detailed stories provided by an LLM) swept all states in favor of the Democrats and failed to reflect the political diversity of the population.
This phenomenon highlights a problem known as bias laundering, a pervasive problem in AI that affects everything from facial recognition to synthetic research, because LLMs are trained on Internet data that disproportionately reflects a Western, educated, industrialized, wealthy, and democratic (WEIRD) worldview. Asking models to be diverse personalities produces a statistical average filtered through this bias, whitewashing the exclusion as the neutrality of the AI.
Additionally, synthetic respondents may suffer from the Pollyanna Principle, or the tendency of LLMs to be overly pleasant and positive in their responses to user prompts. Most users of generative AI chat interfaces have probably encountered this problem: ideas are met with encouragement such as “good idea” or “good choice” rather than with objective evaluation.
For example, in a usability testing comparing synthetic and human respondentssynthetic users reported taking all courses online. While human users report abandoning most online courses, synthetic users report completing them.
The high dropout rates among real users confirmed that the synthetic respondents were trying to say what they thought the experimenters wanted to hear. This sycophancy can lead to the assertion of poor product concepts by helpful AI agents.
Fine-tuning provides context that synthetic approaches lack
Aren’t LLMs trained on a broad enough set of information to produce realistic use cases in almost any scenario? The most effective way to align synthetic respondents with reality is to refine the use of proprietary data. Although general LLMs provide decent baseline estimates for existing products, they face new issues and underrepresented segments.
In an experiencea team queried a basic GPT model on a fictional pancake-flavored toothpaste and ran head-on into the Pollyanna Principle. Without training data, the model predicted that people would like it – in other words, it predicted a preference for novelty. Once the researchers refined the model using data from previous surveys on toothpaste preferences, the results correctly shifted to the negative.
In another study on the desirability of a projector built into laptops, the base model overestimated willingness to pay by a factor of three. After fine-tuning with survey data on standard laptops, the error was corrected, bringing the synthetic results into line with human benchmarks.
Get the best results with synthetic
The competitive advantage of synthetic research does not lie in the model itself – which becomes a commodity – but in the proprietary context that conditions it. For example, Dollar Shave Club used synthetic panels powered by category data to validate new customer segments in days rather than months, achieving results that reflect human behavior with a fraction of the effort.
A few approaches can help you get the best results from synthetic search.
Train in synthetic, test in real life
To address some of these challenges, the market research industry has proposed an industry-wide validation methodology known as train-synthetic, test-live (TSTR). In this approach, models are trained on synthetic data and tested for their predictive validity against a withheld sample of real-world data. The first results were positive.
In research Led by Stanford University and Google DeepMind, digital agents trained on interview data replicated human survey responses with 85% accuracy and social forces with 98% correlation.
This approach recognizes the disadvantages of relying solely on commercially available LLMs as a starting point, as well as the risks of taking synthetic results at face value without validation. By using synthetic methods from the start and validating them with real data, teams can save time and money while building confidence in the results.
Use governance and transparency
Succeeding with synthetic research means that researchers and readers cannot adopt the synthetic personality fallacy – the belief that LLMs possess the equivalent of human psychology and personality traits.
Instead, a more rigorous validation approach is needed, supported by governance guardrails, well-documented processes and transparency in the methods used.
A people transparency checklist can guide researchers when interacting with synthetic characters:
- Area of application: The specific task the character is supposed to accomplish.
- Target population: The target demographic the character is intended to represent, as opposed to generic descriptions.
- Source of data: Whether existing data sets were reused or modified to build the personas.
- Ecological validity: Whether the experimental interaction reflects real-world usage contexts.
Transparency solves two challenges. It addresses ethical concerns around disclosure and builds trust by showing how synthetic approaches work and where they fail. As synthetic influence grows, the distinction between real and synthetic content will become crucial.
Trust but verify
A realist approach to synthetic research means abandoning the belief that LLMs inherently reflect human psychology and focusing instead on empirical comparative analysis, fine-tuning, and transparency.
Synthetic research works if we respect its limits
Synthetic research shows great potential but constitutes a promise with a catch. The promise is unprecedented speed and scale, and the pitfall is the risk of bias and hallucinations.
Recognizing these challenges and putting governance and safeguards in place to mitigate them will help you succeed. It also transforms internal skepticism into a structured governance approach that balances efficiency and results, creating a win-win situation.
