How to use Optimizely to create your first A/B Testing experiment

A few months ago we discussed the importance of A/B testing to help your business build a winning digital strategy. In this blog post, I’ll show you how to use Optimizely, a leading A/B testing platform, to create your first experiment to study, analyze and decide what’s the best move for your content and key site pages.

In this post, I’ll give you a general idea of how to use Optimizely to run a simple experiment on an event ticketing page. This use case is pretty basic, but should give you an idea of how powerful tools like Optimizely can be.

What is A/B testing?

A/B testing is a randomized experiment where two or more versions of a webpage or app are compared against each other, typically by studying how a user interacts with one of the versions. So essentially, an A/B test is an experiment that allows you to test the impact of changes you would like to implement on key user behaviors, such as sales conversion. And once those changes are running, the experiment will collect data and tell you which variation performs better. This ensures that every change you make produces positive results.

Some of the most common A/B test subjects are:

• Change a headline
• Update an image
• Change or add a new Call-To-Action (CTA) button
• Update or add a menu option
• Change the position of your newsletter section

But these are just some basic examples. You can A/B test any scale of change, even updating the entire content of a webpage.

What is Optimizely?

Optimizely is a leading customer experience optimization platform that allows you to create a wide range of website experiments. (Similar tools include Google Optimize or Adobe Target.) Optimizely has been around since 2010 and has become one of the leading testing platforms in the market.

To create your first A/B test experiment with Optimizely, you can either use its web platform or download and install the desktop application, which will provide you a better and faster experience. I definitely recommend the desktop app, but it’s completely up to you. Both apps have exactly the same UI. For this post, I'm using the desktop app.

Steps to create your first A/B test experiment with Optimizely

First, log into your Optimizely account, create or choose an existing project, and click on the Experiments option you will find in the left menu sidebar. The main content section will show a blue Create New... button. Click on it and choose A/B Test from the resulting menu.

NOTE: The layout of the Experiments interface changes if you are already running experiments; in this case, the Create New... button will be placed in the right-top corner.

The next step is to provide to your experiment a name, description (optional) and target.

The Targeting field is the most important property here. You can target by URL or by page. In this example, we’re going to focus on the URL method, so just provide the URL of the webpage where you want to run the A/B test, e.g. https://www.this-is-my-site.com/, and create the experiment.

Next, you’ll see the experiment overview page.

By default, when you create a new experiment, Optimizely will automatically create two variations for you – the Original, also known as the control, and variation#1. You can rename a variation or add as many as you want. Most experimentation platforms allow you to define multiple variations. Such multiple-variation tests are also known as A/B/n tests.

Now you’ll see a bunch of new options beneath your experiment name.

The most important are:

- Variations

The place where you can manage your variations. Click on a variation to make the desired changes to an element of your website or mobile app experience, like updating an existing button or adding a new element. You can find how to customize an experiment variation further down.

- Page Targeting

You already set the page targeting while creating the experiment, but this is where you can update this setting. In addition, you can also add conditions, such as  matching a specific query param in your URL.

- Audience Targeting

Here you can set the audience you would like to see this experience. Audiences help you show an experiment or campaign to a specific group of visitors who come to your site, based on conditions that you specify, such as if the visit is coming from a specific AdWords campaign.

By default, everyone will be able to see your experiment.

- Metrics

A metric is a quantitative measurement of a visitor action. With the proper metrics, you will be able to track actions like user clicks, form submissions, purchases and scroll depth.

Note that you need to set at least one metric to start the experiment.

- Traffic allocation

By default, Optimizely sends 100 percent of visitors to experiments and distributes traffic equally among all your variations. If you’d rather that only half of your visitors participate in experiments, for example, you can redistribute your A/B test traffic.

- Schedule

You can also schedule your experiment to start and stop at a certain date and time. Note that you can set only one schedule.

- Settings

Here, you update the experiment name and description.

The scenario

Let’s imagine you have an event ticketing page with three different options. At first you thought your “General Admission + All” ticket was going to be most popular, but the truth is that people are not buying it as you’d like. This hypothetical scenario is a perfect candidate for an A/B/n testing experiment.

One easy change you could make is to highlight the “General Admission + All” display card to capture users’ attention. To do so, I'm going to create two variations of the “General Admission + All” ticket box.

The first variation will show a new floating highlight element, located at the right-top corner relative to its parent box.

The second variation will show an extra wrapper with the "See it all" text at the top.

Customize your variations

Although you can add, update or remove webpage elements by using the Optimizely editor's tools, knowing a little CSS is highly recommended to better understand what’s going on and do more advanced testing. For this exercise, I’m going to assume you already have a good understanding of CSS.

Select the variation you want to update and click the Create button. You will see a new set of options. Select Insert HTML, since we want to add a new HTML element. The next step is to choose where you want to place the new element. You can either click on the live preview page to select the element or do it manually using the HTML Placement field. Once you have targeted your HTML element, you can add the new element within the HTML field. I've added the following:

<div class="see-it-all">SEE IT ALL</div>

At this point, you should be able to see the new element right above the box. Save your changes.

You just created your first HTML element. Now it’s time to apply some CSS styles to make the box look just like you want.

To add your custom CSS, click on the Code editor icon located beside the URL targeting field. Write your custom CSS within the Variation Code Editor section.

The new styling will be updated as you go, but once you finish, don't forget to hit the Save button.

Define your metrics

Once all your variations are implemented, you need to set your metrics. Metrics will generate results you'll use to measure your experiment performance.

To add a metric, click on the Metrics menu option you'll find in the experiment Overview column and select the metrics you want to record. You will see two main sections, Global Metrics and Events.

For our scenario, we are going to add two metrics, one to track the number of unique users that land on our ticketing page and a second to track which of the three ticketing options a user clicks. (Note that if this is your first time using Optimizely, your Events list will look almost empty. As you define more metrics, they are saved to your account for quick re-use.)

The first metric we’ll adjust, Visit Page: Ticketing Landing page, is located in the Page Events list. This metric was automatically added while you were creating the experiment, so just click on it, leave the default values (increase and unique conversions) and click on the Save to Experiment button.

For the second metric, you first have to add a new event to the Click Event list. To do so, navigate to the implementation page (option located within the main sidebar menu), click on the Events tab, and create your click event. For a complete step-by-step guide, please read this how to set up events in Optimizely blog post.

Once you have created the click event, go back to your experiment metrics, select the click event you just created, and publish the changes. Metrics can also be reordered using drag-and-drop. Your first metric will be considered your primary metric.

At this point your first A/B testing experiment should be ready! Click at the Start Experiment button to start your test.

Watch and analyze your experiment results

It's been 10 days since you started the experiment and now you are wondering which one of the variations is performing better.

Go to your experiment and click on the results icon. The results will be presented to you in different sections, including a summary with the total number of visitors per variation and a comparison for each one of your metrics. Each metric will also present the data from the experiment and show you how the different versions of your page (variations) are performing.

If your variation is a winner, congratulations! You can now apply those changes to production knowing that the hypothesis you or your team built will generate positive results. If the experiment didn't work as expected, don't fret. Use the experiment as a learning experience and create new experiments to find your winning digital strategy.

Conclusions

Optimizely is definitely a great and powerful experimentation tool to create and test your digital experiences. What we just covered is just the tip of the iceberg. With Optimizely you can build more complex experiments, analyze multiple variations at once, target several pages, create extensions to reuse in your experiments, embed custom JavaScript, and much more.

Have questions, doubts, or concerns about Optimizely? Need help creating an experiment? Leave us a comment or get in touch anytime. We love to chat!