Building Haskell Projects with Halcyon

A field report on using Halcyon for developing, testing, and deploying our first production Haskell application, Carnival.

When to A/B Test

March 11, 2013

Questions:

We’ve heard that A/B tests are awesome and all, but should we use them for everything?
And if not, then when should we run an A/B test?

Answers:

We should treat A/B testing just like any other decision making tool – it has costs and benefits and we should use A/B testing when the benefits outweigh the costs
Like all other tools it might yield incorrect results (false positives or false negatives) and it carries costs (implementation complexity and time, waiting for results).
When deciding whether or not to run an A/B test we should consider the tradeoffs versus alternative weighs to make a decision.

Decision making tools

Here’s a quick summary of a handful of decision making tools and when to use each. At TaskRabbit, we generally like to use gut calls, qualitative research, and A/B tests to make most business and product decisions.

Couple of observations:

As a startup, we’re OK with trading off decision-making confidence for speed (i.e., if an A/B test is looking particularly hairy to set up and run, then we’re perfectly fine with making the call based on gut or small-sample qualitative research).
The folks who work at TaskRabbit generally start to develop pretty good intuition about where problems/opportunities are in our space because we use our own service a lot. So, if making a gut call can instantaneously yield a ~80% confidence decision, then running an A/B test for a week to get an incremental 10% more confidence is a pretty high price to pay.

* = our preferred decision-making methods

Coin Flip	Gut Call*	Majority Vote	Qualitative Research*	Survey	A/B Test*
Pros: Instaneous decisions	Pros: Fast decisions You could be the next Steve Jobs if your gut calls are right 90% of the time	Pros: Team buy-in Access and summarize an entire group’s expertise	Pros: Get to know users and build empathy with their concerns Observing users is a great way to discover and surface unknown problems/opportunities Very difficult to argue with video footage or direct quotes from real users	Pros: Good way to gauge sentiment of a large sample of people Medium-speed decisions; it’s fairly easy/fast to recruit and drive traffic to a survey	Pros: Statistically significant results can be tuned to make them as confident as you want them to be (most teams use a 90%+ confidence interval) Collects very reliable data on real users interacting with a live product Very difficult to argue with statistically significant results
Cons: 50/50 chance of correct decision Only handles binary-choices, not so good with multi-variate permutations	Cons: Difficult to justify decisions to a skeptical audience Incorrect decisions made this way get attributed to you, personally	Cons: Can turn into a slow, political decision making process Groupthink can de-rail rational decision making	Cons: The research subjects you choose may not be representative of the entire userbase Recruiting research subjects and running tests takes time and budget	Cons: What people say they would do in a situation often turns out to be different than what they’ll actually do when faced with a real, working product or a real, live human being A skeptical audience will argue with your survey design	Cons: Takes time, code, and sometimes marketing coordination to set up an A/B test Once a test is running, takes a while (a week or more) to reach a large enough sample size Can only run a limited number of A/B tests at one time It’s a bit like panning for gold – many/most tests will be inconclusive, i.e., will not yield statistically significant differences

Hope that helps! Please feel free to comment…

[…]

The Ruby Rails Aggregator