You are here

The Change in Quality Score Reporting: What Really Happened?

August 26, 2013
25

A few weeks ago, Google announced that it was changing the way Quality Scores would be reported. Not how they are calculated, they pointed out, just how the scores are reported. This seems like clear confirmation from Google that the 1-10 Quality Score you see in AdWords is just a “front” for whatever your “real” score is.

If nothing changed about your “real” Quality Score, and this is truly just a glossy reporting change, then there should be no effect on your other metrics. BUT, there’s another way to interpret Google’s announcement. Google might be saying that they haven’t changed the factors that go into Quality Score, but they have changed how they normalize the results (which is how they can compare disparate advertisers). If that’s true, then there could be substantial impacts to your account. Quality Score is half the equation to determine your Ad Rank and an important part of determining the price you pay.

Quality Score Changes

Either way, if you carefully track your Quality Scores (or ask your agency to do it for you), you probably noticed a sudden change with no logical explanation. Since the announcement by Google, we’ve been watching our accounts carefully, trying to figure out just exactly what Google was planning to do.

The first challenge was looking for the date of the change. Google hadn’t told us exactly when it was coming, so we had to stay on top of our tracking and keep “slicing and dicing” the data. After all, we didn’t even know exactly what we were looking for. Was it going to affect all keywords? Just branded terms? Just low-quality score keywords? Generally raise scores? Generally lower them?

All in all, it actually took us until a few days after the fact to be sure of what we’d seen. And what we found was ...

Google Took Away Your 10’s

Quality Scores (the ones we can see) have generally fallen into a bell curve. For a long time, when we asked prospective WordStream clients “What do your Quality Scores look like today?” they all said “Mostly 7s.” More recently, they said “Mostly 4s.” But even in that period, most people had a healthy number of 10s that drove a good amount of their traffic – branded terms, by and large. Which makes sense. As Elisa pointed out last week, you are obviously the most relevant advertiser on your brand. Now, that’s simply not true. Some of our brand terms still have Quality Scores of 10. But a lot of them now have 7s and 8s.

Google Quality Score Reporting Changes

Without going into specifics, our clients’ accounts in the past have generally gotten about 20% of their impressions from keywords with a Quality Score of 4 and another 20% from their 10’s, with a  relatively smooth curve between. It ends up looking like a roller coaster. Since the change, however, we see something that looks more like a traditional bell curve, with about 20% of the impressions coming from 5’s and a smooth curving going down in each direction. (Except for that funny little blip tail at Quality Score 1. Isn’t that weird?)

What does all this mean?

A few things. First, please don’t punish your agency for your recent drop in Quality Score 10’s. It’s all Google’s fault. (And Google’s not punishing you, either.)

Second, mark this date in your calendar. If you’ve been measuring the impact of your optimizations on your Quality Scores, August 2 will be a “break” in your data. Similarly, if you are a huge quantitative analysis nut and are making optimizations based on a correlation between CTR, Average Position, and Quality Score, you’ll need to redo all that math.

Only time will tell whether this change in Quality Score is just a reporting change or something more significant. I would expect, over time, for average cost-per-click to go up if Google is no longer giving out the Quality-Score-10 discount as freely.

Caveats

I should close by saying that (hopefully obviously) I’m working with a small sample size. I looked at data from WordStream’s client base, which tends to consist of small and medium-sized businesses rather than huge enterprise accounts. And I tried to control for other changes that might have impacted Quality Score (basically looking for campaigns where “all else is equal”), but our clients are constantly optimizing their campaigns. All this is to say “YMMV” – your mileage may vary. In fact, one other marketer did find something very different (a shift to higher scores). However, Martin Roettgerding saw results closer to ours – fewer 10's, more 7's and 8's.

How about your account? Have you been tracking your Quality Score changes? Did you notice a loss of 10’s? Sound off in the comments.

Andy Stefano is a longtime WordStreamer who has worked on the marketing, sales, product and quality assurance teams. A former teacher and PPC manager, Andy also trains new WordStream employees, does data analysis and makes WordStream fun. Andy lives in the Boston area with his wife and children.

Comments

Victor Pan
Aug 26, 2013

Hey Andy,Were these changes just cosmetic? (e.g. Your 10 became a 7, but you're still paying the same CPC)Thanks for digging in! 

Andy Stefano
Sep 03, 2013

I want to give it a little more time and maybe expand my research to some other accounts before I say for sure, but that's definitely the million-dollar question. If it's all cosmetic, then it hardly makes a difference. But if it's not and our CPC's are changing too...that's a horse of a different color.

Adam
Aug 26, 2013

Hi Andy,Thanks for digging into this data. I'm not surprised that Google made the change to a more "normal" bell curve. Hopeuflly this will make the distribution more intuitive and help people understand how their quality really measures up (in Google's eyes).-Adam  

Andy Stefano
Sep 03, 2013

I think it will make things easier. Having so many 7's and 10's, but practically no 8's or 9's always confused people. (I remember trying to explain it to senior leadership once!) Normal bell curves are so much simpler to talk about. It's too bad that in order to get to "simple" conversations later, first we have to go through the explanation of why we lost all our 10's.

Richard H.
Aug 26, 2013

Thanks for the update, I didn't even hear about this. I wonder what the real reasoning was behind this.  Was it just a massive update on Adwords or something else altogether different?  

Andy Stefano
Sep 03, 2013

Thanks for reading. Here's the link to the Google announcment: http://adwords.blogspot.com/2013/07/improving-quality-score-reporting.html

BSc Computing
Aug 27, 2013

This is what i wanted to learn, found these Google recent quality score changes but was unable to understand. Many thanks to write detailed post.

Andy Stefano
Sep 03, 2013

I'm glad I could help!

Dino Kestic
Aug 27, 2013

Hi Andy,Thanks for presenting this.I work with a base of clients similiar to yourself within a online marketing company in Sweden. I've also noticed this change in QS for my clients and their accounts. Keywords that were on the scale between 4-6 in QS dropped to 1! Some got a bump higher up on the scale though.So I contacted Google Support but of course, they didn't give me a logical explanation. I wonder if this is just "for show", cosmetics like Victor wrote or if they actually changed they way they compare advertisers? 

Andy Stefano
Sep 03, 2013

Keep an eye on your other metrics - average position is where I would look to see if there was a change in how they compare advertisers. Of course there could be confusion with lots of other factors (a new competitor joining or an old one dropping out, for instance). But if average position starts moving in unexpected ways, keep this in mind as a possible factor. I plan to double check with our clients as well!

Tim
Aug 27, 2013

Thank you for this article. I did notice major changes with the two largest accounts I monitor. One had all the quality scores moved up to 4 or higher. The second had a large number of scores bumped down to 2's and 3's. I was wondering what happened!

Elisa Gabbert
Aug 27, 2013

Hi Tim! Hopefully performance won't be affected!

Phil Pearce
Aug 28, 2013

Clearly the mean avg has now changed from a 7/10 to a 5/10 looking at the above graph.My view... is that only the Tripwire penalties (Relevance, Landing page, IsMobileOptimised) which are blended with VisableQS have been updated.Previously I believe these penalties were:

  • Above Average: +5% to VisableQS
  • Average: +2.5% to VisableQS
  • Below Average: 0% no change
e.g. an actual QS of 5/10 + 5%*3 for Above Average: Relevance, Landing page & MobileOptimised = 7/10Now I think they are using:
  • Above Average: +0% no change
  • Average: -2.5% to VisableQS
  • Below Average: -5.0% to VisableQS
e.g. an actual QS of 5/10 + 0%*3 for Above Average: Relevance, Landing page & MobileOptimised = 5/10This would explain why the "top of the bell" no longer exists, and why you see a mini-spike a 1/10 where the 3*-5.0% is triggering.Your thoughts?ThanksPhil.Ref: penalties in Adwords API:https://developers.google.com/adwords/api/docs/reference/v201306/AdGroupCriterionService.QualityInfo#isKeywordAdRelevanceAcceptable

Christine Derrel
Aug 28, 2013

Phil, thanks a lot for that link to AdWords penalties!!! Learned a lot!!!

Randall Magwood
Aug 28, 2013

Was never a big fan of quality score - i guess cause my were always at an acceptable level. But in some niches of mine, they were poor. With the changes, i'm definitely going to have to stay on top of my game. Great reporting.

Andy Stefano
Sep 03, 2013

Thanks for the compliment. I can understand not being a fan - it's a fuzzy metric that (clearly!) can change at Google's whim, rather than the sort of cold-hard metrics that are more in your control. But it's a great bit of insight into where you need to (and can) improve.

Fade Adenile
Aug 29, 2013

Thus is the life of Online marketing. Ever changing and ever adapting. Adapt or die I guess you could say.

Andy Stefano
Sep 03, 2013

That's what I like about it. (Well, not the dying part.) But it does keep you on your toes.

Stoixima
Aug 29, 2013

HiYour post is really interesting.The google recent updates in the quality score was a little bit weird and your info is very helpful.Regards from Greece

Andy Stefano
Sep 03, 2013

I thought it was weird, too! 

Natalia
Oct 16, 2013

Great article! Could you please let me know where I can find the new reporting. I tried to add QS as a column and I still get the 'old' QSThanks!

Sahil
Jan 02, 2014

Really Useful info, Best part is that its very Clearly written which ensures good understanding for the users.

pame stoixima
Mar 04, 2014

Hello Andy !I'm a little bit confused about quality score by definition but i can start to understand by contextual meanings. I wanted to ask if there is any other info apart from http://adwords.blogspot.gr/2013/07/improving-quality-score-reporting.html, for the quality score basics, or other adwords meanings basics. Thx  

Spook SEO
Jun 09, 2014

Hello Andy!You got a great presentation here. I learned something new from your post. Anyway, I tend to agree with you that only time will tell whether these changes in Quality Score is just a reporting change or something more important that we should take into consideration.

karen duff
Jun 11, 2014

Great article! Really Useful info

Leave a Comment