I love smoke tests. They’re a great way of testing whether or not there is any serious demand for our value proposition.
One common form of smoke test is a landing page test. The landing page states a value proposition with a call to action that asks the user to commit some form of value such as an email address or even money to sign up. Based on the % of users who signup, we get a rough signal from the market if the value proposition is in sufficient demand to build a minimal solution. If not, pivot!
But as much as I like smoke tests, I hate premature surrender. That’s what happens when the signal we get from the market is NO, but we don’t truly grasp why.
Landing Page Test
Let’s say we’re selling a new type of shoe that cures plantar fasciitis. We put up a landing page test with our value prop and a “buy now” button. Then we put $1000 in to Google Adwords for “shoes” to drive traffic and sit back to wait for the money to start rolling in…
…and our conversion rate is 0%
Should we give up? That’s what the landing page test says. There is insufficient demand for this product.
By this point, some of you are probably asking, “What the hell is plantar fasciitis?”
Exactly. All those people coming to our site are asking the same thing.
Is our test failing because customers aren’t interested? Or do they fail because they simply can’t understand the value proposition?
Comprehension Test
Before we ever run a smoke test, we should run a comprehension test.1It’s a simple test we can do that usually takes less than one hour:
- Write out our value proposition in 1-3 sentences.
- Show the value proposition to a participant for a few moments, just enough for them to read it. Then take it away.
- Ask them to explain the value proposition back to us in their own words.
If the participant’s explanation is roughly comparable to our own, we count that as a positive result. If not, then it’s a negative. For this sort of test, we generally want a sample size of about 20 people and a positive conversion of about 80%.
The conversion has to be very high because regardless of what our value proposition is, people should understand it.
Who?
Who we bring to this sort of test is important, but we don’t actually need to use our target customer. We just need someone with the same basic education level and vocabulary. That’s because we are not testing to see if someone wants our value prop, just if they can understand it.
If our target market has a high school level education, then we can just walk out onto the street for participants.
If our target market has a specialized vocabulary such as Chief Marketing Officers, we can probably run a comprehension test with anyone with a marketing background. MBAs seniors might do just fine and they are a lot more accessible than CMOs.
Once we’re certain people understand our value proposition, only then can we run our smoke test and be confident about the results.
Qualitative vs. Quantitative
Don’t get caught up on qualitative vs. quantitative. Comprehension tests are both.
We get a yes/no quantitative response depending on the % of people that can successfully explain our value proposition.
We get the actual explanations as qualitative data. This data can be incredibly useful. If we use the comprehension test on our value prop “shoes that cure plantar fasciitis” we might get responses such as:
- “Don’t know.”
- “No idea.”
- “Don’t know.”
- “Something that cures a scary sounding disease.”
- “Don’t know.”
- “Shoes that help with a common runner’s condition that causes foot pain in the morning.”
Clearly the first five aren’t very useful, but the last one is. The last response is from someone who actually understood, probably a runner. Plantar fasciitis is a very common problem for runners.
Although our original value proposition failed the comprehension test, our respondent has given us a new value proposition to test!
Not only that, they’ve suggested some great keywords we can use for our Adwords campaign…something a lot more specific than “shoes” and with a lot more search volume than “shoes for plantar fasciitis.”
Rules of Thumb
- Before testing commitment, test comprehension
- Qualitative vs. quantitative is a useless distinction, we need both.
________________________________
About the Author
This article was written by Tristan Kromer of Grasshopperherder. Tristan helps product teams go fast. As a lean startup coach, he works with innovation teams to run at least one experiment/research per week to improve their product and business model.
If you’re interested in cutting to the chase and reading the entire Real Book, you can download it here.
