My Sixth Course

The sixth ever course I made at CXL was the A/B testing course given by peep Laja. In this course, I learned what is A/B testing, what is a test, what should we test first, and the statistical tools and methodology used in order to conduct an appropriate test. Though it was a 59th minutes course, I took 3 hours to fully understand the whole concept and see how it can help me in my journey as a Digital Marketer.

Why A/B Testing?

A/B Testing is a concept that is used when there is a decision to be taken regarding an optimization problem. An example is a Facebook ad campaign with 2 visuals of the same product. The person conducting this campaign would want to test the 2 visuals to see which visual performs better and in the end, the visual can be used throughout the whole campaign. The objective is to actually optimize his sales with the visuals which perform better in a pre-campaign test.

This simply means that when we conduct an A/B test we want to validate a business impact through provable measurement. Each technological company has its proven methodology and it’s not prescriptive like a ready- to use manual for a particular problem. Google, Facebook, and Apple have a different methodology to conduct a test yet change is inevitable. The alternative to A/B testing is either a harmless A or a risky B so it makes testing safer for consumers and businesses alike.

The Apple Myth

Apple is one of the most valuable companies in the world and it’s well known as the most innovative company with products such as IPods, iPhones, and recently smartwatches and earbuds. Most consumers see Apple as a very intuitive and futuristic company which turn to be setting trends in technology and leading the mobile phone market ever since iPhone was introduced.

The idea that a new product can magically work is cultured from stone age product development practices which are not acceptable in a competitive and ever-changing tech environment.

Well before a test is validated, you need to make a hypothesis and during the procedure, we may or may not have a winning test which is crucial for our storytelling cause. The storytelling part is usually the explanation behind a successful test. What happens when a test fails?

First, what is a test?

What Is A Test?

Can we do an A/B Testing without knowing what is a test? How much does it cost? Can you evaluate the time cost? Testing cost? Or creative cost?. It is important to look at testing as a way to solve a particular problem. You will need to test whether an idea is delivering business impact or not. In General most tests you will execute are inconclusive.

To solve a problem we must first ask ourselves certain questions like

Where is the problem? Is it your landing page, contact page, or product page? What is the problem? Can you identify the root cause of the problem? It’s popularly said that the solution to 80% of our problems can be done by just optimizing 20% of the core features.

How To Solve A Test

-Step one is gathering a team(cross-functional). A team that consists of different departments with different points of view.

-Step two is generating a central hypothesis with many other hypotheses to ensure clarity and be able to do appropriate checkouts.

Note that ideas could be what isn’t yet there, inserting a new feature on an existing product

But if you have multiple hypotheses, you must certainly prioritize testing because remember we have limited resources.

What Should We Test First

After validating which test can be done and which one can’t be done we have to understand that we may have a mastery of the problem but we still ignore the appropriate solution. We may have a lot of right answers because we don’t yet know what will work and what won’t work.

Luckily companies such as Google are currently using frameworks that help them prioritize their test when they have several hypotheses to choose. One such framework is the P.I.E Framework

The P.I.E Framework

It is a framework developed to validate a test idea among many hypotheses. It scores each idea among 3 dimensions which are Potential, Importance, and Ease on a scale of 0–10.

Potential here to the opportunities these tests can bring. Is it an increase in sales? Are we going to have a better mastery of web analytics?

The importance here deals with the cost of the test, Are we going to have an R.O.I of the test? Are we going to increase our traffic by 10x, 20x, or 100x.

Ease stands for the effort that the technical team will have to do so as to realize these

An example could be a website losing money or sales are dropping so we want to use the P.I.E Framework to validate the test idea

From the above score, It’s recommendable to start by running an A/B Test on the homepage because it has a high P.I.E score hence optimizing it

We can conclude that in the P.I.E method we have a lot of subjectivity as we are the ones actually giving the scores on the scale. That is the reason why you may have scores of a common range and this can make the test inconclusive.

I.C.E methods stand for Impact Cost Effort (I.C.E)

This method is aimed to either score a 0(Low) or 1(High) to a pertinent test regarding the impact the test will have on the company, the cost of realizing this test, and the effort that the technical team will have to put to make this solution work while the company is still functional.

The score at the end determines if we can pursue the idea or not

0-Should not be pursue

1-Minor Opportunity

2-Opportunity

3-Strong opportunity

4-Extraordinary opportunity

This method faced some difficulties as it can difficulty integrate multiple users with different parameters and it still not objective.

Peep Laja of CXL Institute developed a more objective framework to validate a test idea in its PXL Framework https://cxl.com/blog/better-way-prioritize-ab-tests/

Overall, the course was very rich and powerful and the instructor had a total mastery of the subject matter with resourceful insights and practical examples. I now have a clear idea of the importance of running the A/B Test, Which framework is more appropriate depending on the hypothesis we currently have, How to deal with inconclusive tests, and how we can optimize these results in our business or that of our client.