The Ultimate Guide to Keyword Competition: Tips from 35 Experts on Competitive Keyword Analysis
Before targeting a new keyword vertical, it's imperative to evaluate the difficulty of the market. This is done by analyzing keyword competition.
What is keyword competition? Keyword competition is the measure of how difficult it will be to rank for a particular keyword. The competition for a keyword can vary depending on how popular the keyword is and industry competition.
Search marketers estimate how much time and effort it may take to achieve top rankings for particular keywords or search terms.But the question is, how do you judge keyword competitiveness? What are the factors involved in competitive keyword analysis? Is there a specific keyword tool or tools you can use to analyze keyword competition effectively? Look no further for our in-depth SEO guide.
The following feedback for determining keyword competitiveness was provided by our panel of 35 search marketing experts. We asked them each a single question, “What is your best tip or trick for determining keyword competition?” and aggregated their answers into one comprehensive guide for competitive keyword analysis.
When considering entering a new market with a new website: I look at the search results with SEO for Firefox turned on. That gives me lots of data about site age, links to the ranking pages and sites, if people are leveraging domain names, site traffic estimates, and if there is much brand strength in the market. That last bit mostly comes from knowing the web pretty well and understanding the markets you operate in well. And if an area is new and you are uncertain of how strong it is then clicking on some of the background information links can help give you more information and insights.
When considering a new keyword set for an established website: Sometimes it is easy to just publish content and see how well you rank for it. Even better so long as you optimize page titles to capture relevant longer tail keyword variations, then even if you don't rank for the core/root keyword you can still make some good money by rankings for variations of the keyword. And keep in mind the content does not have to be sales-oriented, perfect content just to test the market...look at the crap eHow publishes profitably...you could just make a new blog post and test. Then from there, for areas where you get good results, you could always chose to make higher-quality, sales-oriented content targeting those keywords more from the conversion perspective.
Keyword Competition Tool
We're actually in the process of designing a new version of our Keyword Difficulty Tool. I've attached a screenshot of some wireframes.
The tool can serve as a keyword competition checker and help you analyze keyword competition by running a Google keyword difficulty check. Our process is to get the top ranking pages for a particular query (the top 10 is usually sufficient since any results after that receive very little traffic), then run analysis on the domain and page authority metrics. Since these numbers are directly tied to the ranking models for Google's ordering of search results, we've found that the data is especially accurate for running a Google keyword difficulty check, predicting the relative difficulty of ranking on page 1 for a particular search.
We're also looking to give the keyword competition tool the ability to detect and report vertical search results in the SERPs so we can quantify the impact of image, local, video, business news, blog, real-time, etc. on the rankings.
Historically, our keyword competitiveness tool used data like:
- # of results for a given keyphrase
- # of results in quotes
- # of results using allintitle
- PageRank of the top ranking pages/sites
- # of links pointing to the top ranking pages/sites
- Maximum bid price in the paid search results
- # of ads showing for a given query
However, these were all poor proxies for the actual data of how competitive and difficult to unseat the top results might be. We're pretty bullish on the new process and the new Google keyword competition research tool being a significant upgrade to our previous second-order measurements.
Take the top 5 results, do a whois for the domains and see when the original registration date is for each of the domains. If all or most of the domains have been registered for more than 5 years, you're going to need a trusted domain to rank.
Does domain age mean better results in the SERPS? Domain age really isn't what you're looking for, but the trusted links that have come from being around and publishing that long. If you're on a new domain, you've got a 5 year link building hole to try and overcome.
How to Analyze Keyword Competition
Well, as with most things I do it is a combination of data points. At the end of the day it is part of the art -- being able to analyze the competition for keywords. Getting intimate with a query space is the way to go, and there is nothing like digging in and looking through the top 10-20 listings to see where there may be holes.
It is worth mentioning that it is also a balancing act. Just because a space isn't competitive doesn't mean we want it. So it's not exactly seeking non-competitive spaces, but ones where we can get a foot in the door or with the volume to chase the big dogs.
- So, we can start with the usual suspects (Google keyword competition research tools mentioned already)
- Then cross-reference some PPC data, always a reasonable gage of value/competitiveness
- Juxtapose data from straight search, exact match, allintitle, allinurl
- Just for fun have a peek at Trends/Insights...
Then, dig in, see what the competing sites have working for them and where there are opportunities. What will be the estimated cost/time frame?
Analyze Keyword Competition
For me all the best keyword competition data comes from SEO for Firefox. If I'm looking for a really quick, high level keyword analysis, I'll just run the query and pull the data into a CSV, then sum the following columns:
- Y! Links
- Y! Page Links
- Majestic SEO Link Domain
- Page Rank
- Age (for this I strip the months then just sum the numbers: lower is better for this one :))
Typically I find this to be a much better indicator than number of documents or even allintitle (which is pretty good, and is a great link building query) simply because my intent is to crack that top five/ten, so the strength of those pages is what I'm concerned with (and in most cases if I'm doing this level of depth of analysis on a specific query, it's pretty unlikely the top five will be omitting it from their document/title).
Competitive Keyword Analysis
I’ve never worked in a search vertical that wasn’t super competitive, nor have I ever had the good fortune of inheriting an old, trusted domain. So I’ve always operated under the assumption that every keyword I target is going to be hard and that the competition of keywords will be high. And rather than developing my own formulas for measuring keyword competition, I take a slightly different, iterative approach to competitive keyword research.
For organic search, it looks like this:
- Publish something - It doesn’t have to be perfect. Just something quick to get an read on how difficult it is for your site to rank on a particular term. Who knows? You might get lucky and your content might rank well immediately. Or it may only require minor optimization to rank better.
- If you got lucky, then mission accomplished. Move on to next keyword targets.
- If you can’t find your page in the SERPS, then try moving to an adjacent, longer-tail variation of the word. At WordStream, we invented the Keyword Niche Finder for doing exactly this: finding related, yet less competitive keywords so that you could avoid hypercompetitive niches and uncover less competitive and potentially more profitable keyword niches.
In paid search, it’s more or less the same idea:
- Start by trying out bidding on head terms
- If the ROI meets your target objective, mission accomplished – Move on to next keyword targets.
- If ROI is terrible, then adjust to target long tail keywords, which are likely to be less competitive and better value, particularly if you do a good job at grouping together relevant keywords and being relevant with your ad-text creation and landing page.
So in summary, I guess my tip for determining keyword competitiveness boils down to two key points:
- Don't get hung up in estimating keyword competition
- Perform a quick test to ascertain true keyword competitiveness for your website or paid search account, then iterate on those results
And a finally, a Bonus Tip: Stop thinking of keyword competitiveness as something to apply to individual keywords. A site like WordStream generates millions of visits through search every year through millions of different search queries. Trying to figure out keyword competitiveness for each one is a path to madness. Instead, we’ve organized our keyword taxonomy into around 500 groupings of similar keywords, and look at the competitive landscape on a per-keyword grouping basis.
My quick and dirty trick is to find the most relevant keyword phrases that have decent search counts, then do an Allintitle:"keyword phrase" check in Google on them. If you put them in a spreadsheet with the number of searches and the AIT you get a clear picture of those with high number of searches vs. low Allintitles and your "keyword gems" become clear.
It's usually a combination of tools, but here's a quick rundown of a good process we employ at Audette Media:
- Look at search results, and total returns for intitle:[key phrase] and allintitle:[key phrase] searches. The search volume numbers will show a rough idea of how many are competing for these terms on their pages.
- SEMRush has excellent data (for example, see the attached screenshot).
- AdCenter's Ad Intelligence tool for Excel is excellent, and although looking at a smaller sample of data on MSN's engine, will show a number of revealing competitive insights. I especially like their Monetization segment for keywords. Here's more from Aaron Wall on this.
- AdWords competition data; SEMRush also shows CPC bid estimates for AdWords buys.
- If I could only use one tool, it would be Google's awesome keyword research tool here. It shows a number of interesting data points, including the top terms by category. You can use this with the Google Traffic Estimator tool to find approximate keyword values, best used alongside a tool like SEMRush.
Keyword Competition Analysis
For a bird's eye keyword competitive analysis, I use a few things: two toolbars, two metrics, and gut feel on four variables (which you should obviously back up with some hard data).
- SEOMoz Total unique linking domains
- SEMRush Value from the SEO Book Toolbar
Four variables specific to each site:
- Content volume (do they have 10 pages or 10 million?)
- User data (Alexa, others) and social graph metrics (are they actively participating in social media?)
- Anchor text and title tags (what are they targeting with these?)
- Domain name keywords (do they have an exact match?)
As important as competition is the BENEFIT of ranking for a keyword. Pick your keywords based on benefit to YOUR site, and look for the sweet spots with low competition.
Starting with the top 3 non-news and non-personalized results in the Google's organic SERPs (permanent results), we look at ToolBar Page Rank, SEOmoz's mozRank (mR), mozTrust (mT), domainRank (dR), domainTrust (dT) and inbound anchor text semantics using LinkScape. If any given result is not the site's homepage, we have a look at the Google's toolbar PageRank of the site's homepage as a very general indicator of inbound link strength. Also, it's reasonable to have a gander at Yahoo Site Explorer for an additional view, albeit vague, at the inbound link-strength spectrum.
Look at your own site stats! Find the keywords that generate traffic to your top site pages. Then use WordStream to expand a keyword set around those core traffic generators. You'll build long-tail traffic, fast, and grow quality traffic.
Assuming I need to make a quick review, I look at the advertising associated with the query results. If it's substantial and promoting relevant domains (as opposed to "broad match" advertisers), that's a signal a query is competitive. I also look at the first two pages of organic results. If they all use the query in title tags and page URLs, that's a signal the query is competitive. Finally, if a quick perusal of keyword activity in any major tool shows substantial related queries (in addition to significant traffic for the primary query), that's a signal the query is competitive.
I use the Google query allintitle: “keyphrase” to get a rough estimate on how many people use that keyphrase in their title tag. This will roughly let you know how many people have deliberately or not have minimally optimized their page for that keyphrase. After using the query look at the upper-right corner and see how many results were returned.
For example, simply searching for SEO Firm returns 1,990,000 sites but searching allintitle: “SEO Firm” returns 70,900 sites. This provides a much clearer idea.
I look at keyword competitiveness from an organic SEO perspective. I want to know how hard will it be for me to get my site to rank in the non-paid SERPs.
The main things I look at when determining keyword competitiveness are Google AdWords data (especially search volume), and the quality of the sites that rank well organically for that keyword phrase. I then do a direct comparison with the site I'm working with against the top organically ranking sites to give me an idea of how far I have to go. I also like to look at related long-tail keywords, because the competition and performance can vary greatly.
Ultimately though, it's really about the marketing strategy, not necessarily the keyword competition (which many people can get mired in). If you have sufficient control and flexibility over the website you're trying to rank with – including the ability to frequently publish very high-quality content, create altruistic resources, and improve how the site is coded – you'll be able to start improving your SERPs quickly. And over enough time, if the link building techniques you use aren't too risky, and don't get your site penalized or banned, the site will rank very well organically for most of the keyword phrases you're targeting.
Another thing to keep in mind is that short-tail keywords aren't always the best keywords for a site. Going after highly competitive short-tail keywords will not only take you longer to rank for, they may also be driving the wrong type of traffic. This is especially true if you're trying to sell a niche widget. Instead of focusing on the competition related to the keyword "widget," consider focusing on who your competition is for long-tail keywords that are more closely related to what you're trying to sell. Then make your content, marketing, and link building strategies focus only on those terms. That will improve your overall organic search referrals and conversions much faster than a more competitive, broad, and short-tail term.
Initially, I keep it simple: Look at query volume and the overall number of SERPs for the phrase, placement in title tags and anchor text links in ranking pages. After that, break out the tools.
So, let’s say the term in question is “Green Widgets”:
- Take the term and drop it into the WordStream Keyword Tool (or Google’s AdWords Tool) and pull out the top 50/100/500 results.
- Copy and paste these results into Notepad.
- Do a global delete of all the “spaces between words.”
- Drop all the “words” into your domain registrar’s “bulk search” tool and search the availability of .com, .net and .org domains for each term.
- The lower the available inventory, the more competitive the keyword niche.
With Google keyword competition, always start with what you already know. As I often work on Google.de in many cases I know most of the sites that rank well already. This way it takes sometimes only a few seconds to determine how difficult a keyword is. I see where Wikipedia is, I see where the strongest shopping search engine is, I see where the major newspaper is.
Also I look for the SEO'ed sites. When I see something like "Buy example, examples, cheap examples" at #1, #2 and #3 I know that the competition is fierce. Then I start using the manifold tools we have these days for keyword research.
I check against "similar sized" keywords I already know. Especially in Google Insights for Search you can find out how competitive a keyword is by comparing it to other terms. Other people use a matrix to determine keyword strength or difficulty in numbers, but I'm a very intuitive non-technical person, so I judge based on my gut feeling and the above comparisons.
After I did that with one keyword, all other keyword difficulties for that market are easy to determine as you can compare to the first keyword. Then I use a simple table where I rank the keywords based on their difficulty.
I always look at the number of paid advertisers to get a sense of keyword competitiveness, the number of results in the top 10 that look "optimized" (keywords in the title, etc.), and the number of homepages that rank for the term. Nothing scientific, just a quick way to gut-check a space.
Focus on Keyword Search Frequency
I start at keyword demand in terms of how often it's searched. Once I collect "X" number of keywords and keyword search frequency, I segment the keywords based on those search frequencies. Once I have a set of those keywords, I use Aaron Wall's SEO for Firefox extension to view the domain age for each of the competing results. As a general rule, I find that search results owned by older domains (on average) are the most competitive due to Google's trust algorithms. That said, whenever I find a young domain in a large set of older domains, I want to study that site to see what they're doing to get a leg up on the rest of the competition.
Search Engine Optimization for Highly Competitive Keywords
Determining keyword competitiveness requires a study of a variety of factors, including a understanding of the query space and using one's intuition. Insights are gained by looking at term popularity, analysis of the search results and competing sites, and related trends and conversations.
The tips below show how to determine keyword phrase popularity and a competition utilizing free tools. This is part of a 101 framework for those who are beginner to intermediate in their SEO efforts. The following screenshots display select columns from an Excel worksheet one can create for evaluating two key insights, phrase demand and competition. Ideally, you want to find a balance between competitiveness and popularity of keywords and phrases.
Term Popularity / Phrase Demand
Research keyword popularity across various databases.
- Use Google AdWords Keyword Tool, and display results by "Match Type: Exact" & "columns to display: Show All." Evaluate:
- Exact Match Local search volume count. (Use a formula to divide by 30 for an estimated Daily Estimate.)
- Estimated Average CPC cost for positions 1-3 for PPC.
- Use Wordtracker Free Keyword Suggestion Tool . Evaluate the number of searches for the exact phrase.
- Use WordStream Free Keyword Tool to acquire a CSV. Evaluate the number of searches.
- Evaluate the average count for Google, Wordtracker and WordStream daily estimates.
- Evaluate current CPC costs. Higher cost indicates highly competitive terms.
Evaluate competition by looking at search engine results (SERPs) to determine how many sites are competing for the exact keyword phrase and if these sites are well optimized and have link authority.
- In Google, search for the keyword phrase in quotes to find the number of indexed pages for the exact phrase.
- Use the allintitle: Google search operator to evaluate the number of competing pages with the phrase in the title. (allintitle:"keyword phrase")
- Divide the Competing Pages allintitle: results by the Google AdWords Exact Match Local searches per month to return a competing SERP to Search Ratio.
- Use Yahoo! Site Explorer to evaluate and average the number of incoming links for the top 5 SERPs.
- Keyword phrases that have the highest SERP to Search Ratios and largest number of backlinks indicate most competitive keywords.
- Proceed by evaluating keyword optimization efforts for the top 5 results.
- Evaluate page 1 of the search results and note Google One Box listings that display in universal search.
A keyword phrase is highly competitive if the term is popular, with a high SERP/Search Ratio and if the competition has link authority is optimizing for that term. If the SERPs display more than the standard 10 blue links and are filled with universal listings and numerous PPC ads, then you have a ringer and a lot of work to compete in that query space.
My first act is to view the SERP and see the types of domains that rank for the term. Are the domains established and names I have heard of? Are they spammy looking (.biz, .info, excess of hyphens, misspellings, etc.)? This usually gives me some indication of the competitiveness of the keyword. If this doesn't answer it for me, I check the top 5 results in the mozBar to gauge how many linking root domains these domains have. (This metric is highly correlated to good rankings right now). Lastly, if I really need more data I use Google's AdWords Tool to see how many searches there were for the term. This is not exactly the same as competitiveness of the keywords but it usually correlates.
For keyword competition, I basically have a feel for SERPs based on:
- Yahoo! SE linkdomain numbers (via SEO for Firefox)
- Whether there are exact match domains
- Whether deep pages are ranking (domain authority + a few links) or homepages
- Digging around the top ranking sites' backlinks to get a view to quality
- Any brands in the results
Achieve #1 ranking for it and reflect on how much of a pain in the ass it was to get there. :)
There are a number of ways to determine keyword competitiveness such as how many links the top sites have or how many results there are (though this is less accurate). One good way to determine competitiveness that most people don't look at is how many sites on the first page are homepages, and how many are communities. Generally, search engines follow people so if there are a number of large social sites like forums ranking around your keyphrase, it's going to be hard to rank above them.
On top of that, I find it far harder to outrank homepages with my affiliate sites than article pages. If a lot of the results are homepages, i.e., they end in .com and are not a file name like /blog/keyphrase-here/, then that could be a sign the phrase is going to be tough to rank for.
Well, in the old days I would review the SERP for the obvious and "learn the query space" players, then do G searches using allititle syntax to ascertain overall title strength, then do all in anchor to see the amount of linkage. Another recent addition was using exact match with the terms, which is the most competitive. This basically indicates the degree of "professional grade optimization" in the query space.
Currently, I take that a step further with universal search. IMO, you also have to add a "content" review, i.e., can we use video and other UNI components like news to fill in spots. IMO, all SEO's should be taking care when adding video. I was early into that and found the 300 vids we added often blew out the text position and in that case ... no indented listing just a demotion from above the fold to below the fold of the SERP since that seems to be where vid ends up. So be sure that when optimizing vids you do not knock the higher text-based position out of the SERP.
For AdWords keyword competition, I have often relied on the Google Keyword Tool as a keyword competition analyzer. It serves as a keyword competition analysis tool since it shows competitiveness from a paid search perspective. However, since it doesn't provide exact numbers and generates additional keywords, I find it useful for high-level estimates only. I am a big fan of technology and APIs so I developed a web app in C# which uses Google's AJAX API and the Yahoo API to return the actual number of competitors you'd see on the search engine results page. It has a batch-mode available so running dozens of keywords for competitiveness is not a big issue.
The first thing that comes to mind with keyword competition research is to use the "intitle" search operator. So, if you do a search for: intitle:"craft supplies." The search results will only show pages that have the exact phrase "craft supplies" in the title. That means that those people have either optimized intentionally or probably optimized the page naturally for your target phrase. That search returns over 1.9 million results. So, chances are, it's going to be pretty tricky to break into the "craft supplies" results.
I’m going to tackle this question from the PPC side. First, let’s get one thing straight: the Estimated Average CPC that Google reports in their keyword tool is so fictional that it should be on the New York Times’ Bestseller list. Ignore it.
Instead, it’s more useful to focus on the Google keyword tool competition column of the reports:
Like many things in determining levels of competition, these data are meant to be relative. In fact, Google creates those bar charts on a scale of 0.00 to 1.00. It’s easier to see this if you export the data. Look at the bottom of their keyword list:
Now you can see the (completely useless) Estimated Avg. CPC column and the more useful Advertiser Competition column on a numerical scale, instead of a graph:
Chances are that you’re going to pay more for keywords at the top of the list vs. those lower, though this isn’t always the case. Your bid actually plays an indirect role in determining your CPC and your Quality Score is just as important. Depending on your Quality Score, you could pay a penalty or get a discount that increases or decreases your CPC.
Honestly, if I do a search for that keyword in Google, and see that thousands to millions of URLs are being shown for the same keyword, that pretty much answers my question. However I do the same search in Yahoo and MSN to get the full effect of the keywords competitiveness. I also like to use the SEOMoz Keyword Difficulty Tool which also gives me an idea of how competitive a keyword will be. Point being, you can’t just use one source to come up with an answer. Check multiple sources and get a birds eye view of how competitive a keyword truly is.
Check for the keyword term on Google. NOT broad match because this will show every page or at least most indexed pages mentioning the keyword, but with quotes. Another thing you can do is to check for that keyword with "allintitle:" command as well, such as "allintitle: 'lava lamps'" wich will give you a more accurate landscape of people trying to rank for that specific keyword = to real competing sites.
Other factors to check:
- How many AdWords competitors (Yahoo, Bing also)
- Minimum bid for keyword
- Run your keywords in keyword suggestion tools, such as SEOMoz Keyword Difficulty Tool, SEO Book Keyword Tool, SEMRush, Google Keyword Tool , MSN Adlabs, etc.
For me keyword competitiveness is directly proportional to the competitiveness of website and page which appear top in SERP for that keyword. Total number of result is for my ego top results are for my work.
I will look for following things to measure the competitiveness of a keyword:
- Link profile of website
- Reputation/authority of website and page
- Quality of SEO/content
- What other keywords the page is targeting and its ranking for those
I like to allintitle:"key phrase" to get me started and see keyword competition. Then the link data of the top 5 for the keyword and how optimized their content is. I also like use the AdWords.Google.com/Select/KeywordTool to estimate the traffic levels which also gives some indication to the competitiveness on the phrase, and also see the avg. CPC on the key phrase, if people are willing to bid a lot, chances are they are using money on seo too.
Most of the time I get people that already have a website and therefore some kind of ranking for the key phrase and then I look at what's been done internally to rank that page, and see what could be improved (content, internal links, external links etc.) and how I believe that can change their ranking. And that gives you an indication of the amount of work needed too.
I tend to look more (I'm not saying "only," I say "more") at the site I'm optimizing itself, than to the keyword competition. Well, of course I check keyword competitiveness when I choose keyphrases for the first time (I use a method that looks a lot like this one). But after that, it's all about continuously analyzing the way these keyphrases perform for YOUR site, and adjust accordingly.
Looking to get more out of your keywords? WordStream can help you:
- Discover Profitable Keyword Opportunities
- Eliminate Wasted Time & Wasted Advertising Spend
- Generate Better ROI in a Fraction of the Time