fbpx

How to Evaluate Your Product with Smoke Testing

Smoke tests

Think you’re sitting on the next big idea for a product or a product feature? Before you spend dozens of hours and tens of thousands of dollars on the idea, you need to validate it.

In other words, you need to make sure your audience is just as excited about the idea as you are.

So, how can you validate the ideas sitting in your startup notebook or your product feature backlog without wasting resources? By running a smoke test.

What is smoke testing?

Smoke testing is a testing method that evaluates the critical and most important features and functionalities of a system.

So, using this definition, smoke testing addresses questions like…

  • Does the program run?
  • Does the UI open?
  • Does clicking the key functionality do anything?

Essentially, does everything work as it’s intended to? Smoke tests are specifically designed to be run quickly and frequently during the software development stages.

In the marketing world, smoke test has a similar meaning. However, instead of being focused on “does it work properly” like programmers, marketers want to know “will it make me money”.

Trevor Lohrbeer of Teikametrics and Lab Escape defines a smoke test as…

“Trevor

Trevor Lohrbeer, Teikametrics & Lab Escape:

“A way to test market acceptance before creating a product.” (via SlideShare)

So, if you put the time and money into creating the product, will it actually sell or will it fall flat? A smoke test is designed to help you answer that question.

Dominic Coryell of GIMME GROWTH explains further…

“Dominic

Dominic Coryell, GIMME GROWTH:

“It’s helping to shape your product roadmap by using growth experiments in a pretty fast way. What we’re trying to do basically is just prove that your product backlog is actually worth moving forward.

Validate, learn, iterate.” (via YouTube)

Why do marketers use smoke tests?

Consider these two situations…

  1. You have a new business idea that you’re excited about and want to develop.
  2. You have a new product feature idea that you’re excited about and want to develop.

In both cases, you’re about to spend a lot of time and money. You will also likely need help along the way, whether it comes in the form of a co-founder or a product team.

What if that idea, after months of development and tens of thousands of dollars invested, falls flat on its face? Crickets.

Marketers use smoke tests to avoid this exact scenario. Dan Martell, a 5x founder who now helps entrepreneurs build their tech products, explains…

“Dan

Dan Martell, Entrepreneur:

“People get excited about an idea and then build the product and then find out 6 months later that nobody else was excited.

1. Idea: Problem, Pain, Passion
2. Validated Research
3. Clickable Prototype
4. Pre-Sales

I’ve used this approach to build ALL my companies and it doesn’t matter if it’s for a consumer application, or a solution for big Fortune 500 companies – it can be done.” (via DanMartell.com)

Dan’s four step process might look something like this, when applied…

  1. How do you come up with or choose the right idea? Choose a problem you’ve experienced, choose something that’s a must-have (not a “nice to have”), choose something you’re passionate about.
  2. Find customers using your competitor’s product and conduct validation research. Your friends will tell you your idea is great; people who are already paying to solve the problem you want to solve are the best choice.
  3. Design 3 core features of the idea, not the entire product. What can you build in a 6-week product sprint vs. a 6-month development period?
  4. Get your target audience (see step two) to pre-buy the product. Work with them to develop the full product.

Now, this process doesn’t just apply to startups and entrepreneurs. It’s certainly the way that Dan positions it above, but it can easily be applied to bigger companies looking to test new product ideas, feature ideas, service ideas, etc.

This type of agile thinking is what matters.

Dan’s process proposes something very interesting: that there’s something wrong with the current minimum viable product (MVP) concept. If you’re a marketer who is interested in rapid experimentation and product efficiency, then there is.

Here’s how the product team at Spotify explains it…

The most common objection to this pre-sale (or smoke test) process is that customers will be upset that full functionality is not available. It’s never smart to fall short of customer expectation, right?

Eric Ries, CEO at Long-Term Stock Exchange and author of Lean Startup, explains why this isn’t as big of a deal as it seems for agile teams…

“Eric

Eric Ries, Long-Term Stock Exchange:

“If you’re worried about disappointing some potential customers – don’t be. Most of the time, the experiments you run will have a zero percent conversion rate – meaning no customers were harmed during the making of this experiment.

And if you do get a handful of people taking you up on the offer, you’ll be able to send them a nice personal apology.

And if you get tons of people trying to take you up on your offer – congratulations. You probably have a business.

Hopefully that will take some of the sting out of the fact that you had to engage in a little trickery.” (via Startup Lessons Learned)

Example: Pijon Box

During a presentation on smoke testing, Dominic shared a case study from Pijon Box, a care package service for the parents of college kids. Pijon Box wanted to test add an “Add to Box” feature so that parents could add more products to their care packages, if they wanted to.

Here’s how rolling out that feature would look typically…

Of course, if the Pijon Box team had taken this route, they would’ve spent a month on development and missed the end of April deadline for the last box.

So, instead, they tried this smoke test approach…

They found out which products they had extra of in the inventory, put them into an email and sent the emails out to customers. If the customers clicked the link, the Pijon Box team would have to manually invoice them and notify the warehouse.

The result? 9.55% of active subscribers clicked the link and converted. Monthly gross profit grew by 45% because of the “Add to Box” feature. Marginal gross profit per Pijon Box (on those boxes) grew over 357%.

Not too bad for a day’s work. Now Pijon Box had validated the idea and knew they wouldn’t be wasting a month of resources developing it.

Example: Flowtown

Another example comes from Dan, who used smoke testing frequently to valid Flowtown. Dan calls it a minimum economical viable offer (MEVO)…

“Dan

Dan Martell, Entrepreneur:

“Create a MEVO.

It’s saying, ‘I’m gonna create an offer, a valuable offer, and the minimum amount of features that I feel will get somebody to part with their money and I’m gonna create that landing page and get them to sign up, even if the product doesn’t exist on the backend.’

It’s what I used in my company, Flowtown. And what you do is once they sign up, if you have no product, just say, ‘Hey, we’re still early. We’re releasing people little bit by little. Your credit card wasn’t charged.’ Even though, again, you accept the credit card and you use different tools for that, like Stripe. And say, ‘When we’re ready, we’re gonna let you know.’

That is true validation.” (via DanMartell.com)

Here’s what Flowtown looked like in February 2009…

That’s right, a service company. After three months of working with 15+ customers, they decided to use all of the feedback they had gathered to double down on creating a product. (The plan all along.)

That resulted in…

Which only resulted in one customer over three months. So, they went through many more iterations (smoke tests)…

Finally, they arrived here…

Within the first day, they had their first customer. Then, growth took a turn for Flowtown…

Two years later, they were acquired by Demandforce.

Something that they emphasize is what they call the “bullshit bar”. That is, having people actually pay for the idea…

Here’s another one…

It’s important to consider the transfer of funds when validating an idea for a product or feature. A lot of people will tell you the idea is great. “Oh, yeah, sure, I’d use it.” But would they really spend their money on it? Even something $5 or $9.99 a month can add an extra layer of validation.

What should you smoke test first?

Let’s assume you’re in one of these two scenarios…

  1. You have a dozen business ideas you’d like to smoke test.
  2. You have a dozen product features you’d like to smoke test.

How do you decide which to smoke test first?

Derek Sivers suggests you “test the riskiest assumptions first”.

Otherwise, consider the following…

  1. High impact ideas. If you’re prioritizing your experimentation, you likely have some measure for “anticipated impact”.
  2. Ideas that have been backlogged for too long. Smoke test it and move on.

The 6-Step Smoke Test Process

In his presentation, Dominic lays out the six steps of his smoke test process, which is arguably the most definitive and widely used.

Step 1: Define the necessary proof.

As with most tests, smoke tests begin with a hypothesis. If I do X, I expect Y.

For example…

  • If we improve monthly gross profit by X% with the “Add to Box” upsell email, I expect it’s worth spending the resources to automate the feature.

  • If we capture info from more than X% of visitors to the landing page in the first X weeks, I expect it’s worth building the actual product.

  • If we can pre-sell $X in X weeks, I expect it’s worth building the actual product.

Step 2: Design an experiment without engineers.

The idea of a smoke test is to limit the amount of time and money invested before validation. So, your smoke test should be able to run without the help of an engineer.

For example…

  • Manually invoicing customers and notifying the warehouse vs. having engineers automate the process.

  • Creating a landing page to capture zip code and email for an eventual release vs. having engineers create the entire product.

Step 3: Drive the right traffic to your experiment.

Once you have the smoke test set up, you’ll need to drive some traffic to the test. This shouldn’t be a spray and pray effort. You want to make sure you’re recruiting the right kind of traffic for your test.

Here’s how Dominic knows if he’s recruiting the right audience for his smoke tests…

“Dominic

Dominic Coryell, GIMME GROWTH:

“I think that there is no better way to find the right audience than to see if people share what you’re showing them because if people do share, that means that they’re willing to show their networks about what it is that you’re doing, that they believe in what you’re doing.

Sometimes that can get skewed if you incentivize this.

It helps guide the audience you’re creating. So, on Facebook, if people aren’t sharing after they come in from an ad, then I know it’s probably the wrong audience whereas from a search ad, if they are sharing, maybe the intent is higher.

So it helps guide the decision of where to put money for your experiments.” (via YouTube)

Eric turns to AdWords when he’s looking to generate traffic for his smoke tests…

“Eric

Eric Ries, Long-Term Stock Exchange:

“If you want to build an online service, and you don’t test it with a fake AdWords campaign ahead of time, you’re crazy. That’s the conclusion I’ve come to after watching tons of online products fail for a complete lack of customers.

Just make sure you choose a maximum daily budget that is affordable for you to run the campaign for a few weeks with. I would aim to get no more than 100 clicks per day – over the course of a week or two, you’ll get pretty good conversion data.

Here are the stats you want to pay particular attention to: the overall conversion % of customers from landing page to completed registration, the click-through-rate for your ads on different keywords, and the bounce rate of your landing page for different keywords.” (via Startup Lessons Learned)

Now, if you’re testing a feature, you can simply send existing users to the smoke test. You may want to consider using segmentation, however. For example, perhaps you will only test the feature among active users who have signed in at least 3 times in the last 30 days.

Whatever route you go, be sure of two things…

  1. You have tapped into the right audience. The best product or feature in the world will fail miserably if it’s presented to the wrong audience.
  2. You have generated enough traffic to make a decision. If you’re a large enough company to be A/B testing, use a sample size calculator upfront as you normally would. If you’re a small company (or not yet a company at all), aim for those 100 clicks / day for two weeks that Eric mentions to start.

Step 4: Define your metrics and measure them.

This (almost) goes without saying, but be sure you have tracking in place and are measuring KPIs before starting your smoke test. You should know where people are coming from, how they’re behaving at each stage of the funnel, what device they’re using, etc.

Choose KPIs that relate back to the proof you established in step one.

Step 5: Analyze the results and iterate / optimize.

If you fall into the “testing a product idea from scratch” camp, you likely do not have the traffic (and cannot afford the traffic) to make your test statistically significant.

That doesn’t mean you can’t run a smoke test. It just means that your smoke test will not be perfect and there is a lot of room for error, so take results with a grain of salt. (Think of it the way you think of heuristic analysis, for example.)

Even if you can get a statistically significant result, chances are that result will be “maybe”.

That’s where A/B testing and informal iteration comes into play. Eric explains his iterative process…

“Eric

Eric Ries, Long-Term Stock Exchange:

“Use Google Optimizer to try different landing pages (even radically different landing pages) to see if any particular way of talking about your product makes a difference to your conversion rates.

And if you’re getting conversion rates that you feel good about, try asking for a credit card or other method of payment.

If that takes your conversion rate to zero, that doesn’t necessarily mean you don’t have a business, but I’d give it some serious thought. Products that truly solve a severe pain for early adopters can usually find some visionary customers who will pre-order, just based on the vision that you’re selling.

If you can’t find any, maybe that means you haven’t figured out who your customer is yet.” (via Startup Lessons Learned)

Example: Suprizr

Another example Dominic gave during his presentation was Suprizr, which promised to deliver a surprise meal based on your preferences within an hour.

Here’s what the landing page looked like before…

It had a <3% email capture rate. Here’s where the importance of testing comes into play again. This low conversion rate might, to some, indicate that the idea is not valuable.

Instead of abandoning it altogether, Suprizr changed their approach by first asking for zip code…

And then email, when they were told that Suprizr was not yet available in their area…

The result? 30% of visitors entered their zip code, 25% entered their email and 10% of them shared the site.

Sometimes, your first attempt at a smoke test will not pan out the way you thought. Continue iterating before giving up on the idea completely.

Step 6: Call it.

Some of you will have the luxury of being bigger businesses with statistically significant data to analyze while some of you will not. No matter your situation, eventually you will have to make a decision.

It can be a good idea to set a designated timeframe for step five as analysis and iteration can go on for ages, which would defeat the purpose of running a smoke test in the first place.

Work with the data you have, make a decision and move on to the next.

Insights > Winning

As with regular A/B testing, strive to create an insights > wins smoke test culture. After all, as Dominic explains, even if your idea is proven to be a loser, you still win…

“Dominic

Dominic Coryell, GIMME GROWTH:

“Now, if we hadn’t have won, that would have been fine too because we would have still won.

We would’ve felt good as a team that we didn’t waste time, we didn’t waste money. And we would’ve learned more about our customers.

It’s this type of process and approach that I find really, really valuable because it’s so easy for me, especially on the product side, to get so focused on the perfection and what the product should feel like and make all of these assumptions that people are going to use it without actually testing to see if people are going to use it.” (via YouTube)

Always consider what other insights you can draw out…

  • Why might this idea have failed?
  • Was it an execution error?
  • Did some iterations work better than others?

Conclusion

Before you launch your next product or feature idea, run a smoke test to validate it. You’ll save yourself time, money and (potentially) embarrassment.

Just follow these simple smoke test steps…

  1. Define the necessary proof.
  2. Design an experiment without engineers.
  3. Drive the right traffic to your experiment.
  4. Define your metrics and measure them.
  5. Analyze the results and iterate / optimize.
  6. Call it.

Remember, smoke tests are often thought of strictly in the startup sense, but the process is easily applicable to big companies looking to launch product features or new products / services as well.

Working on something related to this? Post a comment in the CXL community!

Related Posts

Join the conversation Add your comment

  1. Impressive article and gave me directions to our new venture. Start marketing without a product is a super cool idea.

    Niroshan (Last time I met you during conversionxl)

    1. Avatar photo

      Hey Niroshan! Great to hear from you again. Glad I could help and good luck with your new product.

Comments are closed.

Current article:

How to Evaluate Your Product with Smoke Testing

Categories