Questions:
- We’ve heard that A/B tests are awesome and all, but should we use them for everything?
- And if not, then when should we run an A/B test?
Answers:
- We should treat A/B testing just like any other decision making tool – it has costs and benefits and we should use A/B testing when the benefits outweigh the costs
- Like all other tools it might yield incorrect results (false positives or false negatives) and it carries costs (implementation complexity and time, waiting for results).
- When deciding whether or not to run an A/B test we should consider the tradeoffs versus alternative weighs to make a decision.
Decision making tools
Here’s a quick summary of a handful of decision making tools and when to use each. At TaskRabbit, we generally like to use gut calls, qualitative research, and A/B tests to make most business and product decisions.
Couple of observations:
- As a startup, we’re OK with trading off decision-making confidence for speed (i.e., if an A/B test is looking particularly hairy to set up and run, then we’re perfectly fine with making the call based on gut or small-sample qualitative research).
- The folks who work at TaskRabbit generally start to develop pretty good intuition about where problems/opportunities are in our space because we use our own service a lot. So, if making a gut call can instantaneously yield a ~80% confidence decision, then running an A/B test for a week to get an incremental 10% more confidence is a pretty high price to pay.
* = our preferred decision-making methods
Coin Flip | Gut Call* | Majority Vote | Qualitative Research* | Survey | A/B Test* |
---|---|---|---|---|---|
Pros:
|
Pros:
|
Pros:
|
Pros:
|
Pros:
|
Pros:
|
Cons:
|
Cons:
|
Cons:
|
Cons:
|
Cons:
|
Cons:
|
Hope that helps! Please feel free to comment…
Read more at the source