7 Things We Learned from 31 Days of A/B Headline Testing

It’s all about that click.

 

This is true whether you’re writing headlines for PPC ads or content that’s meant to earn organic search traffic, links, and social media shares.

You can’t honestly call your content “high quality” if it isn’t getting read. And if your headline is boring, people won’t click. They’ll ignore you.

That means lots of bad things for your business and brand. For instance, you won’t:

• Grow your audience.
• Attract links naturally.
• Generate qualified leads.
• Boost your social media engagement and shares.
• Make enough money.

Oh, and if we’re talking paid search? You’ll pay more for your ads, since Quality Score heavily rewards a high click-through rate.

If only there were a way to know in advance whether that headline you’ve been writing will generate tons of traffic or be ignored…

Well, there might be one way – headline analyzer tools. But do they actually work?

What Are Headline Analyzers?

Headline analyzers are free tools that score your headline, giving you some sense ahead of time as to whether people are going to want to click on your headline.

Here are three headline analyzer tools you can test drive:

CoSchedule

 

This headline analyzer is my personal favorite. I’ve been using it for a couple years. It analyzes and scores your headline based on factors like your word balance, length, keywords, and sentiment. Plus, CoSchedule’s tool will show you what your headline will look like as a Google search result and an email subject line.

Advanced Marketing Institute

 

This headline tool analyzes the “emotional marketing value” of your headline and also gives it an “emotion classification.” They say that most headlines should have between 30 and 40 percent EMV words at minimum, but ideally should be in the 50 to 75 percent range.

ShareThrough

 

I’m least familiar with this tool. It uses an algorithm that analyzes your headline based on 300 variables. It provides some feedback on your strengths and also points out a few areas where you can potentially improve (e.g., mention a brand, body part, or celebrity… Um, seriously?)

So if you’re looking to craft that perfect headline, problem solved, right? Surely one of these will do the trick.

Or will they?

Analyzing the Headlines in Google Search Results

We know that Google will sometimes reward posts that generate more clicks with higher search positions. And it makes sense for them to do so.

 

Think about it:

If a page Google has ranking in Position 1 gets 10 percent of clicks while a page ranking in Position 3 gets 30 percent of clicks, that would clearly signal to Google that something may be wrong. Perhaps the page in Position 3 is the better answer for a given query (as long as users aren’t immediately bouncing back to the search results) because it aligns with the user’s intent.

That means a more clickable headline could be the difference between ranking in Position 1, 2, or 3 in the SERPs.

So is there any correlation between the headline score and ranking positions?

Anecdotally, at least with CoSchedule, it sometimes seems so. But this could be an exhaustive study on its own.

For now, let’s just look at one search. Fittingly: [how to write headlines].

Here’s what you get, along with their scores from the three headline analyzers (CoSchedule / AMI / ShareThrough):

1. Headline Writing 101: How to Write Attention Grabbing Headlines That Convert (Quicksprout) (52 / 18.18 / 73)
2. How to Write Better Headlines [Infographic] (HubSpot) (73 / 16.67 / 65)
3. 5 Easy Tricks to Write Catchy Headlines (Goins, Writer) (62 / 42.86 / 65)
4. How to Write Magnetic Headlines (Copyblogger) (63 / 20 / 65)
5. The Step-by-Step Guide to Writing Powerful Headlines (Neil Patel) (63 / 14.29 / 69)
6. 10 questions to help you write better headlines (Poynter) (67 / 25 / 66)
7. 9 Tips for Writing Great Headlines in 2017 (WordStream) (63 / 0 / 61)
8. 17 Easy Tricks How to Write Catchy Titles and Headlines (DreamGrow) (70 / 30 / 67)
9. 55 Easy Ways To Write Headlines That Will Reach Your Readers (CoSchedule) (75 / 36.36 / 63)

I know what you’re thinking. CoSchedule, the creator of the headline analyzer, ranks ninth?

Well, I never said your headline was the only – or even the most important – Google ranking factor. But it definitely is one of your most important content elements.

So what does this one analysis teach us?

There is absolutely no correlation between headline analyzer scores and ranking position – at least on this one query.

Honestly, it’s pretty much impossible to draw any significant comparisons from this one example. After all, it’s just one search out of trillions of searches that happen on Google every year.

All we know is that, according to CoSchedule’s headline analyzer, the CoSchedule post has the best headline; Goins, Writer has the best headline according to AMI’s tool; and QuickSprout has the best headline, based on ShareThrough’s headline analyzer.

So is there any value in these headline scores?

Can Headline Analyzers Actually Predict a Winner?

Which one of these articles will get the most clicks?

That was the name of the game for the last month for me at Search Engine Journal (full disclosure: I am Executive Editor of SEJ).

For this particular test, I relied on CoSchedule’s headline analyzer when writing headlines.

Typically, my goal is to write headlines that score 70 or higher. But sometimes, depending on the topic, it’s surprisingly hard to get a score of 70. A score of 80+ is even rarer. Forget about a score in the 90s let alone a perfect 100 (I’ve yet to achieve either).

At SEJ, we use a custom-made headline A/B testing tool (well, technically A/B/C tool). Why custom?

• We tried using the King Sumo WordPress plugin. However, this causes issues for cached sites (the same headline would end up being shown repeatedly, resulting in flawed tests).
• We also tried the Title Experiments Free WordPress plugin. However, this one was bad news for SEO. Basically, the H1 tag would look empty to Google whenever it crawled and indexed the page because Google still has problems with JavaScript. Plus, if you have a high traffic day, your server could quickly get overloaded because this plugin sends an AJAX request for every page load.

During our headline test, we wrote three headlines for every new post published. Then, for 120 hours, a test ran. During this time the three headlines were shown an equal number of times to our website visitors.

Our headline A/B testing tool has one constraint worth mentioning. Our “A” headline can be any length we want, but our two alternative headlines (“B” and “C”) must be less than 50 characters total.

7 Things I Learned From 31 Days of A/B Headline Testing

OK, let’s talk results.

For the sake of argument, let’s assume the following about CoSchedule’s headline scores:

• 100 = Perfect Headline
• 80-99 = Great Headline
• 70-79 = Good Headline
• 60-69 = Average Headline
• Poor/Bad (59 or lower)

Here’s what I found in the past month.

1. Headline Analyzers Can’t Actually Predict the Winning Headline

In total, 50 articles and 150 headlines were tested.

During these 31 days of headline testing, the CoSchedule score matched the most clicked headline only 40 percent of the time.

The other 60 percent of the time, a headline with a lower score actually got more clicks.

Here’s one example.

One of these three headlines got clicked on 37 percent of time. Which one?

 

You would think the “A” headline should win since it has a score of 80, but actually the “B” headline (with its score of 70) got the most clicks.

Why? My best guess, based on using CoSchedule’s tool for a while now, is that they heavily weight the word “things,” whereas readers of SEJ might be more inclined to click on “tips” than “things.”

So if you think that a great score on a headline analyzer is a reliable indicator of whether your headline will actually be successful, think again.

2. A Great Score Doesn’t Guarantee Lots of Clicks

If you get a great score (between 80 and 99), you’d expect to get lots of clicks, right?

Nope.

The best score I managed to get for the month was 84 for this headline: This Is the Most Important Aspect of SEO.

This was the “B” headline. It got 35 clicks. That was good enough to come in second place, but it was a loser nonetheless.

As for that previously mentioned headline, How to Rank for Featured Snippets: 9 Things You Need to Know? It got a great score as well – 80.

That was good for last place. Third out of three. Worse yet, it was beaten handily by headlines that had scores of 70 and 66.

This is a perfect example of why you must A/B test your headlines.

3. The Most Clicked Headline Had a Good (Not Great) Score

One headline dominated all the others. Your Rankings Have Dropped – 10 Things to Do Now earned 192 clicks during our testing period.

What score did it get from CoSchedule’s Headline Analyzer?

A 71.

That means 33 out of the 150 total headlines we wrote that month had a better score from the headline analyzer.

 

That said, any headline over 70 should do well. But if one headline with a lower score can outperform 33 other headlines, this is yet another reason not to completely trust a headline analyzer.

Surprisingly, the headline that got the second most clicks was actually the “C” headline for that same article (What to Do When Google Rankings Drop Dramatically). It got 181 clicks and had a score of 68 from CoSchedule’s tool. That means it outperformed 51 other headlines with higher scores – and we didn’t even end up using it!

4. A Headline With a Good (or Even Average) Score Can Drive More Clicks Than a Great Score

In looking over these 150 articles, there are numerous examples of great headlines generating less clicks than good or average headline scores.

 

For instance:

• A Simple Guide to Smart Social Media Aggregation scored 73, but got just 36 clicks.
• Misrepresentative Content: What You Need to Know scored 79, but only got 20 clicks.
• How to Filter Referral Spam in Google Analytics scored 72, but got 29 clicks.

On the other end, there are numerous examples of average and good headlines generating more clicks than posts with a great score:

• 3 Ways SEO & Content Work Better Together scored only a 54, but it got 110 clicks.
• 8 SEO Tips to Optimize Your URL Structure scored just 59, yet it got 139 clicks.
• Hiring an SEO? Don’t Ask These 13 Dumb Questions got a score of 64, but managed to get 131 clicks.

Bottom line: Some “how to write headline” posts try to make the process sound easier than it actually is. The truth is you really can never know with 100 percent certainty that your headline will click with readers – and get their clicks.

 

5. Your Content Is Only as Good as Your Headline

Spending hours writing what you consider “high-quality” content is essentially all time wasted if your headline is boring.

Don’t just spend 2-3 hours writing a post and slap some lazy headline on it. You’re sabotaging yourself.

While some people might argue that you should write dozens or even a hundred headlines for every piece of content, that’s nonsense.

If you can’t write at least an average-good headline for an article within 30 minutes, maybe there’s something seriously wrong with the content.

6. Your Headline Is Only as Good as Your Content

You can write the greatest headline in the world. But if it’s on a topic your audience doesn’t care about, it’s simply a headline wasted.

A topic that is of great interest ­– even if it has an average headline – will almost always outperform a brilliant headline on a piece of content that is more niche or simply is off-topic to your core audience.

Sometime we tend to overthink things during the content research and ideation stage. It’s really not rocket science – identify topics your audience is interested in and find new and interesting ways to talk about them.

7. Headline Analyzers Can Help You Write Better Headlines

Despite all their flaws, the process of testing your headlines with an analyzer will help you write better headlines. The secret of any kind of writing is to do it more. That’s the only way you’ll improve.

For instance, I pretty much confirmed one thing that I’ve been suspecting lately – that question headlines are losing their effectiveness. In fact, I believe that question headlines can sabotage your content.

 

We tried out nine question headlines on SEJ. Only one attracted any significant engagement (Hiring an SEO? Don’t Ask These 13 Dumb Questions). However, this headline doesn’t fall into the trap that many question headlines do – you can’t answer it with a yes or no (a.k.a., Betteridge’s Law).

Data vs. Creativity

 

Sometimes you just “know” a headline is going to be awesome. It’s perfect. It’s catchy. It’ll grab all the attention and convert like crazy.

Except, if you don’t test it against any other headlines, how do you truly know you couldn’t have had an even better headline?

If there truly is a “perfect” headline for every piece of content you create, you won’t find it unless you’re testing.

The only limit to your headline is your own creativity – and your ability to admit that you might be incredibly biased about how awesome that headline you wrote is.

Be creative. Run SEO tests. Just let the data be the final decider.

Data always wins.

TL;DR

Can you trust the scores of free headline analyzers to accurately predict which headline will get the most clicks? Nope. You’ll be wrong more than half of the time.

Are headline analyzers worth your time anyway? Yes. Although you’re really writing your headline for one tool’s algorithm, it forces you to re-examine every word you choose carefully.

The more headlines you write and test, the more chances you have to discover the right headline. If you aren’t testing, you’re potentially missing out on traffic, rankings, leads, shares, and revenue.

The more data you collect, the more you’ll know about what YOUR audience responds to. Ultimately, this is why you need to A/B test your headlines.

For even more tips and ideas, head to our impressive collection of 88 headline examples (they’re total click magnets).