A good PPC experiment starts with a decision, not a test idea. When competing PPC optimization strategies each have an advocate, a controlled experiment is the honest way to choose between them. Before you touch a platform, you state the causal hypothesis, the treatment and control, the randomization unit, the population, the primary metric, the guardrails, the minimum effect worth detecting, and the stopping rule. Everything else in experiment design, including sample size, duration, and the readout, follows mechanically from those declarations. This is the discipline /ads experiment enforces, and it is worth understanding even if you never run the command.
Start with the decision, not the test
Most failed ad tests were never really tests. They were changes someone made while a dashboard happened to be open. The tell is that nobody can say, before the data arrives, what result would change what decision.
So write the decision down first. "Should we move prospecting creative to UGC-style video?" is a decision. "UGC-style hooks lift purchase conversion rate versus our studio creative" is the causal hypothesis that informs it. From there, the treatment and control define themselves, and you can name the one thing that differs between them.
That last point matters more than any statistics. Change one decision surface per test unless the design explicitly estimates interactions. If the treatment cell has new creative, a new audience, and a different bid strategy, a win tells you nothing about which lever worked, and the test cannot inform the decision it was built for.
Pick the randomization unit deliberately
The randomization unit is what gets assigned to treatment or control. It determines what interference looks like, how much sample you have, and how honest your readout can be.
| Randomization unit | Typical use | Main tradeoff |
|---|---|---|
| User or device | Platform split tests with user-level assignment | Cleanest inference, but only where the platform supports it |
| Ad set or campaign | Creative and structure tests | Auction learning phases and budget dynamics can contaminate cells |
| Geography | Incrementality and budget tests | Fewer units, so you need larger effects or longer runs |
| Time period | Sequential before/after splits | Seasonality and external events confound easily |
There is no universally correct choice. A geo split resists cross-cell contamination but starves you of units. An ad set split gives you volume but exposes you to the platform's own optimization interfering with your assignment. The design step is where you check platform constraints, overlapping experiments, and interference between cells, because these problems cannot be fixed after the data is in.
Size the test from declared assumptions
Sample size and duration are not opinions. They are calculations from inputs you declare: baseline conversion rate, the minimum effect you care about detecting, your tolerance for false positives and false negatives, and the traffic you can actually route into the test.
Two of those inputs deserve extra scrutiny in paid media:
Minimum detectable effect
Pick the smallest lift that would still change your decision, not the lift you hope for. If a 5 percent improvement in CPA would not change your budget allocation, do not size the test to detect it. Smaller effects need dramatically more sample, and an underpowered test is worse than no test, because it produces a number people will act on anyway.
Conversion lag and seasonality
If your conversion window is long, the last days of a test are systematically undercounted, and stopping "when the numbers look done" biases the readout. Duration also has to cover enough of a business cycle that day-of-week and promotional effects hit both cells equally. Sizing math that ignores lag and seasonality is an approximation, and approximations should be disclosed in the design, not discovered in the argument afterward.
Pre-register guardrails and decision rules
A primary metric tells you whether the treatment won. Guardrail metrics tell you what it is not allowed to break while winning. A creative test whose primary metric is conversion rate might carry CPA and spend pacing as guardrails: a variant that lifts conversion rate while blowing past the CPA target is not a winner you can ship.
Pre-registration means fixing, before any data arrives:
- which observations get excluded and why
- what data-quality checks must pass
- the analysis method
- the thresholds that map results to decisions
- the stopping rule
The stopping rule is where most real-world tests die. Repeatedly peeking at results and stopping the moment they look favorable inflates false positives badly, because you are giving noise many chances to look like signal. Set the rule at design time and honor it. /ads experiment treats this as a hard guardrail: the design forbids peek-and-stop, and the readout refuses to call underpowered noise a winner.
The readout: verify integrity before estimating anything
When the test ends, resist the urge to look at the lift first. The readout order that holds up is:
- Verify assignment integrity. Did units stay in their assigned cells? Did an overlapping experiment or a platform automation cross-contaminate them?
- Verify data completeness. Is conversion lag fully resolved? Are tracking gaps hitting both cells equally?
- Only then estimate the effect and its uncertainty against the pre-registered thresholds.
And one final discipline: do not generalize beyond the tested population. A creative win on prospecting audiences in one region says nothing about retargeting or another market until you test there.
What running it looks like
The /ads experiment command walks this exact sequence: state the decision and hypothesis, check the environment, size the test from declared assumptions, pre-register the analysis, then read out with integrity checks first. A design session looks like this:
/ads experiment
Design a test for our Meta prospecting campaigns. Hypothesis:
UGC-style video hooks lift purchase conversion rate versus our
current studio creative. Randomize at the ad set level. Primary
metric is purchase conversion rate, guardrail is CPA. Conversion
window is 7 days and Q4 promos start in six weeks.
The output is a versioned JSON experiment design: sample-size and duration estimates with every assumption and approximation disclosed, pre-registered exclusions and quality checks, and decision thresholds. The same command handles the readout when the test finishes, returning effect and uncertainty estimates plus a plain-language decision, so the artifact that ends the debate is the one that started it.
Experiments also should not float free of strategy. /ads plan builds the broader paid-media plan that experiments belong to, including which PPC optimization strategies are worth testing at all, so your test queue is ordered by decision value rather than by whoever argued loudest last week.
Try it
Claude Ads is free and open source, and it runs inside Claude Code. Install it from the marketplace:
/plugin marketplace add AgriciDaniel/claude-ads
/plugin install claude-ads@ai-marketing-hub-claude-ads
Then run /ads experiment against your next contested test idea. Full setup details are on the install page.