Experimentation vs. Testing in PPC Campaigns: Dos, Don’ts & Key Differences

There are certain facets of marketing that have always seemed to capture more attention than others in regards to growing your business online. One of these is testing.

This adherence (and some might conclude, obsession) with data is not confined solely to the marketing world, as it seems most aspects of society have transformed into “data-driven” models.

Data-driven decision-making in marketing is incredibly valuable, but there is a time and place for everything, and marketing itself is a lot more than just numbers.

Image source

The data that drives successful marketing campaigns can’t materialize prior to creating said campaigns. The chicken that lays the egg that makes testing possible is the creativity that drives results fuels the data that informs decisions.

In this post I’ll walk you through what A/B testing is from a paid media perspective, when it’s necessary, and when human abilities like creativity, intuition, and common sense lay the groundwork.

What exactly do we mean by “testing”?

There are many forms of testing within marketing including multivariate testing, usability testing, and content testing, but for the sake of simplicity within paid advertising we will mostly be referring to A/B testing.

With an A/B test, you create two variants of an ad or landing page, and everything is identical except for one element so you can see which variation leads to more conversions. You can test ad copy, button colors, creative elements, the length of the landing page, and more.

The variant that performs better needs to reach statistical significance, which Investopedia defines as “the claim that a result from data generated by testing or experimentation is likely to be attributable to a specific cause. A high degree of statistical significance indicates that an observed relationship is unlikely to be due to chance.”

You generally want a confidence of 95% to consider that a change or variant is statistically significant.

Image source

When is it appropriate to begin testing?

Many folks take an extremely “scientific” approach right out of the gate, analyzing every impression and click with a magnifying glass and conducting micro experiment upon micro experiment with slight alterations in ad copy and creative.

The problem with this is that they are limiting themselves by hyper-focusing on small details too early on. Successful paid media is not always an immediate victory in regards to the quality of feedback or results that you may experience when you get started.

With a channel like Google Ads, for example, patience and observation is key to learning which levers to pull and where to make strategic adjustments. You may create an ad or make and adjustment and wait a week or two to observe the impact. Plus, there’s the algorithmic learning period to consider.

So the question is, when should you begin testing?

Every business is different—their stage of growth, the number of creative resources available, and the industry they are competing in. When it comes to paid media channels, everyone starts with a blank slate at some point. The promotions and strategy that you choose from the start often influence the evolution of your paid advertising accounts for better or worse.

From my experience, testing should begin after you start to get some traction with the results you are looking for. So for example, if you are looking to generate conversions from a lead form submit, then you should begin testing after you begin to see some results. So you might now be thinking, “What if I’m not driving any results and therefore need to test in order to do so?” My answer to that question is to separate testing from experimentation.

Testing vs experimentation

From a paid marketing perspective, testing is comparing a control to an alteration. You are “testing” a new variant against a pre-existing one to see if the changes you have made are statistically significant or not.

Experimentation, on the other hand, is pushing your creative juices to produce a variety of different ads or promotions against your target audience to see what appears to gain the most traction.

Experimentation is not to be confused with Google Ads experiments, a feature within the platform.

Some may call this testing but in my opinion, it is less scientific and restrictive. Once you begin to accumulate meaningful data and feedback you can then identify and zero in on what works and test within that framework.

How to do experimentation the right way

When you are in the experimentation and exploration phase, there is a strategy to apply as well. On paid social networks, I like to create the target persona as best as I can manually, then create 3-4 ads targeting that persona.

Given the objective of the client, these ads will be in alignment from a conversion or web traffic perspective. I will also typically create a retargeting audience to test against visitors who are already familiar with the brand. From there, we may incorporate more variations of ads, play around with copy or creative, and allow the campaign to run and collect data.

This stage allows us to gauge the audience’s receptiveness to different messages and ultimately the data will inform us on a direction to explore further. For one client of mine, we were able to increase lead-to-sales opportunity ratios by over 15% by simply identifying that certain language and creative elements resonated much better with our audiences than others. We arrived at that conclusion, however by not being too stringent on testing early on, rather, letting the ads play out then assessing the findings.

Bottom line? The purpose of the experimentation or exploration phase is to create baselines—which allow you to set expectations as well as goals for improvement. This will move you beyond the experimental phase into the testing phase. From there on you can create a model for testing that allows marginal improvements to performance without all the risk of trying an entirely new promotional set.

How (and when) to move from experimentation to testing

To simplify my process for experimentation and testing I will outline from my experience, the stages of growth within a paid advertising account and what you should be doing:

Early stage: Experimentation only

If you’re starting your Google Ads campaign or account from scratch, you should be focused primarily on setting up proper tracking, pixel implementation, etc. first. The most important part of this early phase is that you know your general goal for advertising and ensuring your website and CRM are tracking these results properly, whether they are website visits or conversions.

From there you should take that overall goal and begin to construct campaigns in which to enter the experimentation phase.

If it is paid search, create campaigns for your branded search terms and through keyword research, some of your most relevant terms. Try to start with something that has the highest probability of producing the result that you want, then expand from there once baselines are established. You can learn how to set up conversion tracking in Google Ads here.

In paid social, you should do the same but with website remarketing and a carefully constructed target persona. In paid social you should establish the goal you are looking for and try numerous promotions that align with that goal.

You can learn how to set up your Facebook Pixel here.

You may also want to set up the Facebook Conversions API for the most complete tracking.

Middle stage: Start testing

What I refer to as the middle stage is where you have experimented with different offers and have figured out what appears to work. You have baseline costs for these promotions and are ready to start testing variations of this offer in a more scientific way. This is where you can begin to A/B test.

Growth stage: Refine your testing

This is where you are certain of what drives business within the account. When an account is in this stage you want to find ways to grow but do so within the framework that has proven to be successful. At this stage in the game, you want to test very stringently with fairly smaller changes to variations to mitigate risk of decreasing performance. Larger riskier experiments should be isolated to their own testing campaigns, separate from the ones that are currently driving business for you.

Patience is key with campaign testing

The main takeaways you should have from this advice is to treat new and early paid advertising much looser than what some others might tell you. Although everyone wants to score a touchdown right out of the gate, it is much more beneficial to your paid accounts and development as a marketer to be patient.

Not relying too heavily on data to make decisions early on allows you to be more creative and push the envelope with your marketing abilities. If you are more open to new ideas early on, it will inevitably reduce the amount of time you otherwise would have been testing micro changes to one idea. For long-term success in advertising, you need to take some risks to set a better baseline for the future.