How to run A/B tests – The Brief

Understanding A/B Testing

This article will review A/B testing and its use in digital marketing.

What is it?

A/B testing in digital marketing compares two versions of a creative to determine which one performs better. The purpose of A/B testing is to optimize conversion rates and improve the effectiveness of digital marketing campaigns by identifying the most effective version of a creative or element. By testing just one element at a time, marketers can determine what specific changes lead to better performance and use that knowledge to improve future campaigns.

How does it work?

To begin, clearly define the goal of your ad, such as increasing clicks, conversions, or impressions. Next, create two ad variations, each differing in only one element (e.g., headline, call to action, or image). Use your chosen Demand-Side Platform (DSP) to display both ad versions to your target audience and track their performance.

You can monitor the performance of each creative, including clicks, impressions, and Click-Through Rate (CTR), both during and after the test, via the side panel view in the listing area. To declare a winner, one creative must achieve an 85% probability of outperforming the other in a repeated scenario. Once this probability is reached, the A/B test concludes, and the winning creative will continue to be served. You also have the option to end the A/B test at any time and manually select a winner, or none at all.

Best Practices

Testing only one variable at a time will help you isolate the impact of each change and get more accurate results.
Evaluate and prioritize ideas with the highest impact based on their potential to have a significant effect or result.
Use the top-performing setup for non-tested elements to help maintain consistency in the campaign and ensure that it is aligned with the brand's messaging and values.
Creating new campaigns for A/B tests is a best practice in digital marketing that can help to ensure accurate and reliable results and lead to more effective and efficient campaigns.
Define the test's metrics and how to define success to set realistic expectations for the A/B test and measure the effectiveness of each version of the campaign consistently and objectively.
Setting the right A/B testing budget can ensure that the test is conducted properly and that the results are relevant and helpful in making informed decisions about the campaign, leading to better performance and more efficient use of resources.
Allow the test to run its course to gather reliable data and draw accurate conclusions about the performance of each version of the campaign.
Applying new findings to existing campaigns can improve performance and more effective digital marketing strategies.
Track and document your A/B tests to provide valuable insights and improve the overall performance of campaigns.
Adopt an "always be testing" mindset to stay ahead of the competition and continuously improve the effectiveness of your digital marketing strategies over time.

Here we will go over the necessary steps to set up an A/B test campaign in The Brief and how to check its progress. We've listed some requirements and limitations you should be aware of.

Requirements & Limitations

When setting up an A/B test campaign, you can select up to 5 creatives.
The selected creatives used in your A/B test must be the same size.
Currently, design sets can't be used in your A/B test campaign.
Once you've finished setting up your A/B test campaign, you have three days to input the ad tags into your chosen DSP.
The duration of an A/B test campaign will range between 7 to 30 days.
Deciding on a winning creative requires an 85% probability, after which the test will end, and that creative becomes the winner, continuing to be served and running your ads.

Setting up an A/B Test

Select the creative(s).
Click on Set up an A/B test from the listing above.
- Alternatively, you can right-click on a creative and select Set up an A/B test.
Once you've chosen the creatives, click Continue.
Give a name to the campaign and set a description (optional).
Click Finish setup.

Set the Ad Serving settings

Select the A/B Test
Go to Ad Serving settings.
Adjust Ad Serving settings as needed, including choosing your DSP.
1. Additionally, you can adjust your ClickTag options if needed.
Copy the ad tag into your DSP of choice¹.

Follow the results

You can access the progress of your A/B testing campaign from the side panel view while the test is ongoing, or you can check it from the project's analytics section.

Side-panel view

Locate and click on the A/B test in the project.
A side-panel view will open with the Test Results.
You can filter through the available metrics (Clicks, Impressions, and CTR).

Project's Analytics

Open the Project.
Click on Analytics.
Click on Add Filter.
Go to A/B test, and select your test.
Filter through Clicks, Impressions, and CTR.

A/B testing with Feed

If you created a design that uses a Feed to display multiple variants, you can use the A/B/n testing to see the top-performing feed row.

In order to do so you will need to:

Create a design attaching a Feed
Open the Ad Serving panel
Click on Feed variations display
Select A/B/n testing from the Dropdown
Choose your network
Copy the code and paste it into your DSP of choice

Important: We do not collect cookie information from the users targeted by the A/B test.

Remember that while the A/B test itself is free of charge, there will be costs associated with the impressions generated by the creatives used in the A/B test. To learn more about impressions, please visit this article.

Note: To end the A/B test, click the End test button in the side panel. When you end the A/B test, you can either select a winner or none. The creative you choose as a winner will continue to run in your ads, and your other creative(s) won't get served any longer.

Note¹: To set the A/B Test live, copy the creative's ad tag code to your ad platform of choice within three days.