{ “@context”: “https://schema.org”, “@type”: “AnalysisNewsArticle”, “mainEntityOfPage”: { “@type”: “WebPage”, “@id”: “https://martech.org/how-to-run-a-ctv-incrementality-test-and-why-most-fail/” }, “headline”: “How to run a CTV incrementality test (and why most fail)”, analytical “description”: “A guide exploring the technical execution of Connected TV (CTV) incrementality testing and the architectural pitfalls that cause multi-channel measurement failures. The article details best practices for establishing clean control groups and isolating conversion growth on modern digital ad networks. “CEO”, “worksFor”: { “@type”: “Organization”, “name”: “SpotlightIQ” }, “sameAs”: “https://www.linkedin.com/in/seannowlin/” }, “publisher”: { “@type”: “Organization”, “name”: “MarTech”, “url”: “https://martech.org”, “logo”: { “@type”: “ImageObject”, “url”: “https://martech.org/wp-content/themes/martech/images/martech-logo.png” } }, “backstory”: { “@type”: “CreativeWork”, “text”: “This operational analysis synthesizes B2B Connected TV ad campaign methodologies, attribution modeling data, and platform experiences with IP-level device graphs. It leverages performance benchmarks collected from multi-channel media validation frameworks to address systemic tracking errors in modern digital advertising. }, “speakable”: { “@type”: “SpeakableSpecification”, “cssSelector”: ( “h1”, “.article-content p:first-of-type” ) } }
Most Connected TV (CTV) incrementality tests fail before a single ad is served. They fail not because the media underperformed, but because the test produced a number that looks convincing without proving anything.
This failure is growing bigger every quarter, as those who control the budget want more and more proof that CTV works. As of this year, cross-platform measurement was a top priority for 72% of advertisers, up from 64% in 2025, according to the IAB Outlook 2026 Study. Advertisers must prove that investing in streaming produces results that search and social cannot.
The most honest way to address this challenge is to perform an incrementality test. This involves retaining a control group, advertising to the remaining audience, and measuring the difference in results. Although the concept is simple, the execution can be difficult. Even the largest and most well-resourced advertising teams have struggled to answer the question of incrementality. If the biggest spenders are struggling, the rest of us need a reliable method.
Here’s how to design a CTV incrementality test for B2B that holds up, avoid mistakes which quietly invalidate most incrementality tests and transform the result into a number accepted by your CFO.
10X your SEO with Semrush for business.
The world’s most powerful SEO platform, built specifically for businesses.
@media (maximum width: 768 px) { .headline-responsive { font-size: 30px !important; line-height: 1.3 !important; } }
Incrementality is not an attribution
Attribution focuses on identifying ad exposure that deserves credit for a conversion. On the other hand, incrementality addresses a more complex question: would this conversion have taken place without the advertising? The distinction between these two lines of research highlights the shortcomings of CTV measurement.
The most common incrementality testing method compares people who saw the ad with those who did not, then reports the difference as a measure of improvement. Although this may seem rigorous, it is an imperfect approach. The people who saw your CTV ad are fundamentally different from those who didn’t. They tend to broadcast more, may already be interested in your product, and are deliberately targeted. This metric captures the effects of targeting rather than the impact of the ad itself. It’s like attributing a person’s fitness to their attendance at a gym simply because they chose to go.
True incrementality requires deciding, before a single impression is served, which accounts you will deliberately not advertise to. This retained group is the control. What happens to them is what would have happened without your exposure to your advertising. Anything the exposed group does above this baseline is incremental. If you skip this step, nothing you do later in the analysis will make up for it.
Choose the correct unit to randomize
Unlike many other forms of digital marketing, CTV does not have a persistent cookie. Ads are served to households and IP addresses, and multiple people may be looking at the same screen at the same time, so you can’t neatly organize distribution at the individual level. Instead, go up a level.
B2B still targets individuals across all channels (e.g. named buying committee members, on LinkedIn, via email, and in your ABM program), but that committee makes the buying decision as a group and ABM already reports at the account level. A complex B2B transaction involves a group of decision makers, not just one. Randomize where the decision lands. Count is the correct unit of randomization.
For account-based programs, divide your list of target accounts into two groups. Half of the accounts are eligible for advertising on CTV, while the other half are excluded from all advertising, including streaming. This approach aligns with how B2B reporting is typically structured, focusing on accounts and their progression through the sales pipeline.
Choose the testing method and understand what it proves
Not all testing methods are equal. From strongest to weakest evidence:
- A hard-delete account resistance is the most rigorous test most B2B teams can run. Exclude IP addresses and device IDs from the control accounts of each associated line item, not just the test campaign.
- A geo-matched market test uses a difference-in-differences design, highlighting the evolution of test markets minus the evolution of control markets over the same period.
- Media mix modeling is a fallback, not a test. It’s a correlation between spending and results. This is useful for planning, but don’t present it as causal evidence for a single channel.
Select the metric before you start testing and avoid focusing on preliminary metrics such as impressions, as they indicate delivery rather than impact.
For B2B, the primary metric should be qualified pipeline or opportunities generated from target accounts. You can also use faster signals, such as target account site visits, increased branded searches, improved paid social metrics, and demo requests, to gauge whether the test is working before the pipeline has time to mature.
Make sure your test size is statistically significant
The main reason most incrementality tests fail is not the media. B2B conversions are both rare and slow. If your base opportunity rate is low and you only retain a small number of accounts, you won’t have enough conversions to detect significant improvement. As a result, the test will indicate “no effect” regardless of the performance of the CTV.
Before committing, check whether the test can achieve statistical significance. Three things determine this: how often these accounts convert today, how much of an increase you would need to call the result real, and how many accounts you can place on either side of the split.
Unfortunately, when conversions are rare and the list of accounts is short, no sample will show any effect, even if CTV works. If this is where you land, you have two choices: set a top funnel metric as primary and treat the pipeline as directional, or not run the test yet. A test that is too small to detect an effect is worse than no test at all, because it produces a sure wrong answer.
Protect the control group
Keep two things in mind when performing incrementality testing. First, measure the test and control groups before the campaign begins to ensure they are following parallel trends. If the control group has already performed better, any perceived improvement in the campaign may be misleading.
Second, protect yourself against data leaks. Factors like shared corporate IP addresses, co-viewing, and users exposed to multiple active campaigns within the same account can contaminate your control group. Apply suppression to all line items and monitor it closely during testing.
Also allow enough time for the campaign to take effect. Top-of-funnel signals typically take weeks to emerge, while the sales pipeline requires campaign duration as well as an appropriate analysis period aligned with your sales cycle. If you measure a two-week test against a pipeline, it can look like a failure simply because the pipeline hasn’t had enough time to develop.
Why disciplined teams stick to their budget
A clear incrementality test only matters if the result reaches the person controlling the budget. Your CFO doesn’t think in percentage increases. They think about the pipeline, cost per opportunity, and how long it will take for expenses to pay off. Report incremental pipeline, cost per incremental opportunity, and payback period. These are the numbers you can defend in a budget meeting.
The teams that maintain their CTV budget until 2026 will not be the ones that have the nicest dashboards. They will be the ones to decide – even before the campaign launches – which accounts they were willing to leave alone. None of this is glamorous. It’s the difference between knowing CTV worked and hoping it did. First build the control group. The rest follows.
The position How to Run a CTV Incrementality Test (And Why Most Fail) appeared first on MarTech.
