Is A/B testing dead?

Martech

“Today, A/B testing is thriving — it’s been a huge improvement from non-A/B testing,” said George Khachatryan, CEO of AI company OfferFit, in a recent webinar. “At Fortunately, A/B testing is evolving.

Master the art of A/B testing

Search Engine Land

Learn how you can test before launching, improve your return and understand why your consumers selected the asset they did. The post Master the art of A/B testing appeared first on Search Engine Land.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

A/B testing is dead, long live A/B testing!

Martech

Odds are, you’re A/B testing. But this method is now a thing of the past … and why is that? What’s making A/B testing obsolete? Register today for “A/B Testing is Dead, Long Live A/B Testing!”

Best Practices to Establish a Single Source of Truth (SSOT) for A/B Testing

CXL

Your CX testing lives or dies on the quality of your data. And you can’t trust the outcomes of your tests if you don’t know you’re looking at accurate metrics. That’s why you need to build your testing program around a Single Source of Truth (SSOT) dataset.

Master the art of A/B testing

Martech

Traditional A/B testing only lets you test messaging, creative and other marketing assets after they have gone live in the market – with no warning of how they will perform. The post Master the art of A/B testing appeared first on MarTech.

Bayesian A/B Testing: A More Calculated Approach to an A/B Test

Hubspot

What are some of the reasons you run an A/B test? When I think of the benefits of A/B testing, I think of one of the most popular and concrete ways to experiment with ad designs that are effective for target audiences. Bayesian A/B Testing.

Predicting Winning A/B Tests Using Repeatable Patterns

ConversionXL

If you ever ran a highly trustworthy and positive a/b test, chances are that you’ll remember it with an inclination to try it again in the future – rightfully so. Testing is hard work with many experiments failing or ending up insignificant. In 2017 we systematically started to categorise similar test results as patterns to help us better predict more winning tests. Sometimes we simply take a piece of paper and sketch our ideas.

How To Visualize A/B Test Results

ConversionXL

You may be wondering, “why should I make my own visualization of my A/B test results?” ” Because the A/B testing tools in the market already provide you all the necessary tables and graphs, right? They tell you when an A/B test is significant and what the expected uplift is. The thing is, these tables and graphs are comprehensible when you – a data driven analyst – take a look at them.

PXL: A Better Way to Prioritize Your A/B Tests

ConversionXL

If you’re doing it right, you probably have a large list of A/B testing ideas in your pipeline. Some good ones (data-backed or result of a careful analysis), some mediocre ideas, some that you don’t know how to evaluate. We can’t test everything at once, and we all have a limited amount of traffic. You should have a way to prioritize all these ideas in a way that gets you to test the highest potential ideas first.

How to A/B Test CTAs Like HubSpot Experts

Hubspot

The call to action is a put up or shut up moment that can turn curiosity into a legitimate business opportunity. It's the bridge between visitors and leads — a point of entry that allows you to capitalize on the web traffic you generate. A lot goes into creating an effective CTA.

How to Segment A/B Test Results to Find Gold

ConversionXL

You run an A/B test , and it’s a winner. Does it mean that the treatments that you tested didn’t resonate with anyone? If you target all visitors with the A/B test, it merely reports overall results – and ignores what happens in a portion of your traffic, in segments. Why is conducting post-test segmentation important? Analyzing data in your post-test analysis. Segmenting your data: before or after your test?

How to Minimize A/B Test Validity Threats

ConversionXL

You have an A/B testing tool, a well-researched hypothesis and a winning test with 95% confidence. There are factors threatening the validity of your test, without you even realizing it. Quite simply, validity threats are the factors that threaten the validity of your A/B test results. Type I errors occur when you find a difference or correlation when one doesn’t exist. After a week, the tool declared a 23.8%

Ignorant No More: Crash Course on A/B Testing Statistics

ConversionXL

A major portion of your test results are probably invalid. While testing tools are getting more sophisticated, blogs are brimming with ‘inspiring’ case studies, and experimentation is becoming more and more common for marketers – statistics know-how is still severely lacking. If you don’t know basic statistics, you won’t be able to tell whether your tests suck. Why Do I Need to Know A/B Testing Statistics? During the test?

Why Your A/B Tests Are Failing

ConversionXL

In a 2013 study by eConsultancy & RedEye , surveying almost 1,000 client-side & agency marketers, it was found that 60% found A/B testing to be “quite valuable” to their business. Yet, only just over a quarter (28%) report being satisfied with their conversion rates. What’s interesting is that another study by VWO found that only 1 in every 7 A/B tests is a statistically significant winning test. A/B testing.

How to A/B Test Your Pricing (And Why It Might Be a Bad Idea)

Hubspot

Choosing the right pricing for your product is a little bit like Goldilocks. Too high, and you risk alienating a large majority of your potential customers. Too low, and you likely won't have enough revenue to run a sustainable business. This is where A/B testing comes into play.

A/B testing alternatives for low-traffic websites

ConversionXL

In a world where A/B tests are done by over 70% of online businesses , choosing not to follow a data-driven methodology to make informed decisions on website changes might seem unreasonable. Or if your management refuses to justify the costs of A/B testing?

How to Analyze Your A/B Test Results with Google Analytics (Updated)

ConversionXL

A/B testing tools like Optimizely or VWO make testing easy, and that’s about it. They’re tools to run tests, and not exactly designed for post-test analysis. Most testing tools have gotten better at it over the years, but still lack what you can do with Google Analytics – which is like everything. Looking at a summary screen like this is not enough: Use these at-a-glace views for a quick check to see what the overall status is.

How to A/B Test Your Pinterest Ads: A Step-by-Step Guide

Hubspot

Personally, I’m a huge fan of the platform and use it as a source of inspiration — I make Pinterest boards related to my interests (travel, style, etc.). Pinterest’s 335 million active users are always searching for new ideas and your ads can be a great way to pique their interest.

Confidence Intervals: A Guide for A/B Testing

ConversionXL

Confidence intervals are a standard output of many free and paid A/B testing tools. Most A/B test reports contain one or more interval estimates. Even if you’re simply a consumer of such reports, understanding confidence intervals is helpful. Note: If you want a deeper dive, you’re in luck: I just released a book, Statistical Methods in Online A/B Testing , and I teach CXL’s course on A/B testing statistics.).

A Crash Course on A/B Testing Facebook Ad Campaigns

ConversionXL

Having a well-thought-out plan for A/B testing Facebook ad campaigns is essential if you want to improve your performance reliably and consistently. And the more you test, the better. A study of 37,259 Facebook ads found that “most companies only have 1 ad, but the best had 100’s” A/B testing Facebook ad campaigns can get complicated quickly (and easily produce invalid results). The Structure of a Facebook Ad Campaign.

What Does A/B Testing Have to do with Machine Learning?

Hubspot

To some marketers, A/B tests are a no-brainer. Even though the test is a timeless marketing technique, it can be a little frustrating because of the uncertainty of it all. What is an A/B Test and Machine Learning? A/B testing.

11 Things That Work More Often Than Not in A/B Tests

ConversionXL

This article will give you some of these tactics to test for yourself. Let’s be realistic: websites are highly contextual, and what works for website A, doesn’t work for website B. In addition, the concept of a tactic is one thing, its impact is all about the specific implementation. That being said, there are a few tactics that we’ve seen win time and time again. Sure enough, a lot of the answers were similar. What is a value proposition?

11 Things That Work More Often Than Not in A/B Tests

ConversionXL

This article will give you some of these tactics to test for yourself. Let’s be realistic: websites are highly contextual, and what works for website A, doesn’t work for website B. In addition, the concept of a tactic is one thing, its impact is all about the specific implementation. That being said, there are a few tactics that we’ve seen win time and time again. Sure enough, a lot of the answers were similar. What is a value proposition?

18 Top A/B Testing Tools Reviewed by CRO Experts

ConversionXL

Finding a proper A/B testing tool isn’t the problem anymore. If you work in conversion optimization – whether at an agency, in-house, or as a consultant – you almost certainly run A/B tests. Though you could hire someone full time to sift through and analyze the pros and cons of each tool, it’s easier to learn from the experience of others and make a decision based on that. Some cons: WYSIWYG Editor a bit slow to load.

Don’t Do A/B Testing [Rant]

ConversionXL

A/B testing is highly useful, no question here. But a lot of businesses should not be doing it. A lot of microbusinesses, startups and small businesses just don’t have that transaction volume (yet). You *might be* able to run A/B tests with just 500 transactions per month too (read: how many conversions do I need? ), but you need bigger impacts per successful experiment to improve the validity of those tests.

Beyond “One Size Fits All” A/B Tests

ConversionXL

If you’re invested in improving your A/B testing game, you’ve probably read dozens of articles and discussions on how to plan and run A/B tests. You might also have heard it’s best to come up with many variants to test against the control to improve your chance of finding the best option. No matter what rule is offered, such advice seems to rest on the assumption that there is a one-size-fits-all solution that works in most situations.

Sample Pollution: The A/B Testing Problem You Don’t Know You Have

ConversionXL

Here’s an uncomfortable truth about conversion rate optimization: lots of people are running bad tests without even knowing it. In Bart’s words, “If you don’t know about sample pollution, stop testing.”. The first step to running better tests is to really understand sampling. It’s how many visitors or conversions you need in your test. If you don’t calculate your sample size before you run your test, you’ll run into bad data, likely without even realizing it.

Maximize the Results of your Nurture Campaigns Through A/B Testing

Heinz Marketing

For many years, I worked in B2C marketing for large technology companies where A/B testing was an ongoing process with teams dedicated to testing and refining in all areas of marketing. Testing is not always a part of the equation. Testing is key to making any marketing program a success. That’s why testing is necessary. That’s why companies should always test. Start by testing your subject lines.

5 Uncomfortable A/B Testing Questions

ConversionXL

AB testing is supposed to be straightforward and extremely transparent. You ran a test, and had a solid winner – the sample size was adequate, test ran for 2 business cycles, confidence level was above 95% – and the lift was like 10%. Yes, re-testing it seems to be the easy way out, but in reality it might not be so easy. Testing something is an opportunity cost – means you can’t test something else.

A/B Testing on Facebook: How to Do It Right

Hubspot

To illustrate, do you always have a tab devoted to Facebook Advertising efforts? Further, are you testing those ads before they get published? Facebook offers so many ways to test the performance of your ads before they go live. A/B Testing Facebook Ads.

How to Analyze Your A/B Test Results with Google Analytics

ConversionXL

A/B testing tools like Optimizely or VWO make testing easy, and that’s about it. They’re tools to run tests, and not exactly designed for post-test analysis. Most testing tools have gotten better at it over the years, but still lack what you can do with Google Analytics – which is like everything. Looking at a summary screen like this is not enough: Use these at-a-glace views for a quick check to see what the overall status is.

What Do You Do With Inconclusive A/B Test Results?

ConversionXL

So you ran a test – and you ran it correctly, following A/B testing best practices – and you’ve reached inconclusive results. A surprising amount of tests end up inconclusive. According to Experiment Engine’s data, anywhere from 50% to 80% of test results are inconclusive, depending on the vertical and stage of the testing program. In Massey’s case, tests of video footage on an apparel site came up inconclusive.

How to Estimate a “Net Value” for Your A/B Testing Program

ConversionXL

In my experience, I find that teams and organizations report many winning A/B tests with high uplifts, but somehow they don’t seem to bring those uplifts in reality. Five types of A/B test “wins” can exaggerate discovered uplifts. I use the acronym “de FACTO”: F alse winners; A nti-winners; C hanging winners; T ricked winners; O verestimated winners. So keep increasing the quality, (statistical) trustworthiness, and velocity of your A/B tests.

How 8 Different A/B Testing Tools Affect Site Speed (Original Study)

ConversionXL

But A/B testing tools actually may slow down your site. We researched 8 different testing tools to show how your site performance is affected by each one. Visitors have a patience threshold. Just like in a any shop. For you, that means a slower site is less revenue. How a Testing Tool Cost Us 12 Points On Google Page Speed. Simyo (telco company) had a 99/100 Google Page Speed score for their homepage.

Server-Side Vs. Client-Side A/B Testing Tools: What’s The Difference?

ConversionXL

We nerd out on testing tools. Though no optimization program has ever hinged on which tool you used , there are important distinguishments between A/B test tools, from the statistics they use to the price and more. One thing that is often either overlooked or misunderstood, is the difference between client-side and server-side testing tools. When the visitor lands on your page, a randomly picked version of your test is sent straight from your server.

A/B Testing Mastery: From Beginner To Pro in a Blog Post

ConversionXL

A/B testing – for all the content out there about testing, huge amounts of people still mess it up. From testing the wrong things to running the tests incorrectly, there are lots of ways to get it wrong. What is A/B testing and How Does It Work? What to Test to Improve Our Chances of Winning? How to Prioritize Test Hypotheses? How Long to A/B Test? How to Set up A/B Tests?

Bayesian vs Frequentist A/B Testing – What’s the Difference?

ConversionXL

There’s a philosophical statistics debate in the optimization in the world: Bayesian vs Frequentist. This is not a new debate; Thomas Bayes wrote “ An Essay towards solving a Problem in the Doctrine of Chances ” in 1763, and it’s been an academic argument ever since. So what the hell does Bayesian statistics mean for a/b testing? You’re probably familiar with the Frequentist approach to testing. is a basic building block of this approach.”.

The Key Difference Between Multivariate Testing & A/B Testing

Hubspot

There's seemingly no end to the things you can test in your marketing, and if you've laid a solid framework for your inbound marketing programs, now's a great time to start optimizing and making what works pretty well work amazingly well. And the best way to get started is to conduct some A/B tests ! Or.multivariate tests? What's the difference between A/B tests and multivariate tests? What Is an A/B Test?

10 Statistics Traps in A/B Testing: The Ultimate Guide for Optimizers

ConversionXL

Even A/B tests with well-conceived test concepts can lead to non-significant results and erroneous interpretations. And this can happen in every phase of testing if incorrect statistical approaches are used. Statistical stumbling blocks lurk at every phase of a test. Here’s a quick navigation of what we’ll cover in the post. Statistics trap #4: Fear of multivariate tests. During Testing. After Testing.

Three Hard Truths About A/B Testing

ConversionXL

Sometimes A/B testing is made to seem like some magical tool that will fix all problems at once. Well run a test and increase your conversions by 12433%! Setting up and running tests is indeed easy (if you’re using the right tools), but doing it right requires thought and care. Most A/B tests won’t produce huge gains (and that’s okay). I’ve read the same A/B testing case studies as you have.