25 important questions about your AB test
Most of the product teams don't get all useful learning from AB tests and usually, this happens because of a shallow-minded approach to AB tests. Below is the list of good questions that will help you to think about your experiments on a deeper level.
What is the hypothesis? Is it a valid hypothesis? Is this the most valuable hypothesis?
What prior work/data/research influenced the hypothesis? Why was this hypothesis generated?
Does it resonate and contribute to the overall product goals?
What are you hope to learn?
What are you going to do if your experiment will fail or succeed? Do you have a sense of your next steps?
Will this data be informative for developing new hypotheses or further experiments?
How will the data be collected in the new version of the product? Will the logic of events stay the same?
Did you calculate the sample size? How long will this test run?
How effective is the design reflecting the hypothesis?
How does the design you've made help you gather the right kind of data that will provide evidence in favor of or against your hypothesis?
What is the minimum number of test variations that designer need to create in order to get the learning that you are looking for?
How does the new variation can influence user behavior? Can it influence other products?
Have you defined one of the most important metric? Will these really measure the validity of the hypothesis?
Can you improve this metric, but make the product worse? If yes, this metric is bad.
Do you have good secondary metrics so you can make a deeper analysis?
Have you defined metrics that you don't want to reduce in any way?
Results and analysis:
Was the hypothesis proven? Why or why not?
What did the team learn and what can be applied to other work that is going on?
Did the results support any other larger trends that you might have seen before?
How do these results compare to prior work?
What are the next steps?
Did we see anomalies or surprises?
Did we test the right things?
Could or should we have tested other things at the same time?