/ads experiment
Design or read out a controlled test
/ads experimentInstalled as a plugin? The same command is namespaced: /claude-ads:ads experiment
What it does
Designs and reads out paid-media experiments with the rigor a statistician would insist on: explicit hypotheses, randomization units, sample-size and duration math from declared assumptions, guardrail metrics, and pre-registered decision rules. Covers both new test design and the readout of a finished one.
Use it to
- Design an A/B or split test with a real hypothesis and stopping rule
- Estimate how much traffic and time a test actually needs
- Check whether platform constraints or overlapping experiments would contaminate a test
- Read out a finished experiment into a plain-language decision
- Settle a creative or bidding debate with a controlled test instead of opinion
How it works
State the decision and hypothesis
Decision, causal hypothesis, treatment, control, randomization unit, population, primary metric, guardrails, minimum effect, and stopping rule.
Check the environment
Platform constraints, overlapping experiments, conversion lag, seasonality, interference, and measurement quality.
Size the test
Sample and duration are calculated from declared assumptions, with every approximation disclosed.
Pre-register the analysis
Exclusions, quality checks, analysis method, and decision thresholds are fixed before data arrives.
Read out with integrity checks
Assignment integrity and data completeness are verified before any effect is estimated.
Deliver the decision
Setup or readout lands in versioned JSON with a plain-language decision attached.
Reads & produces
What it reads
- The decision the test should inform and its causal hypothesis
- Treatment, control, randomization unit, and population
- Primary metric, guardrail metrics, and minimum detectable effect
- Conversion lag, seasonality, and measurement-quality context
- For readouts: assignment and outcome data from the completed test
What it produces
- A versioned JSON experiment design or readout
- Sample-size and duration estimates with assumptions disclosed
- Pre-registered exclusions, quality checks, and decision thresholds
- For readouts: effect and uncertainty estimates plus a plain-language decision
Guardrails
What /ads experiment will and will not do, by contract. These boundaries are enforced in the skill itself, not left to judgment.
- No peeking: the design forbids repeatedly checking results and stopping on a favorable one.
- Underpowered noise is never called a winner. If the test cannot detect the stated effect, the readout says so.
- One decision surface changes at a time unless the design explicitly estimates interactions.
- Results are not generalized beyond the tested population.
- Readouts verify assignment integrity and data completeness before estimating anything.
Questions
Can it tell me if my test result is significant?
Yes, and more: it verifies assignment integrity and data completeness first, then estimates effect and uncertainty against the pre-registered thresholds, and returns a plain-language decision rather than a bare statistic.
Why does it refuse to end my test early when results look good?
Stopping on a favorable peek inflates false positives. The stopping rule is set at design time from declared assumptions, and the readout honors it.
Can I test two changes at once?
Only if the design explicitly estimates interactions. Otherwise it holds to one decision surface per test, because a confounded test cannot inform the decision it was built for.