I am a big believer in transparency, and my biggest problem with other tools out there is that they don’t explain EXACTLY how they judge their competition. Perhaps they think this is their secret-sauce, or perhaps they don’t want backlash because it’s impossible to please everyone. But I believe they are doing their customers a disservice.
So on that note, I’d like to share the exact spec that my current version of CMI uses (at the bottom of this post). Go ahead and spec this out to a programmer if you want to just recreate my tool, but if you release it give me some props at least :)
In the coming months I will be doing some linear regressions, testing different scoring and even new metrics. My goal is to turn the tool into a tool that users could input their own weightings as to competition (along with some presets, like the Aaron Wall version or the Rand Fishkin version). Everyone has a different opinion. So if you believe PR matters you can weight it more heavily. If you believe Title tag matches are more important you could weight it just so.
And here’s the basic spec ():
Hit Google with the keyword and parse all the records on the first result page.
Each url in the result: check them in SEOMoz API and get its PA and DA.
For each url calculate APA from PA as follows, in this order:
1.) Links to page Modifier Metrics
If PA=1 , it means 0 external linking root domains, make APA=40% from DA.
Else do nothing, leave APA as is.
2) Page Title Modifier Metrics
If no words or variations from KW match words in the title, then deduct 60% from APA. If 100% match then deduct nothing. Everything in between is calculated according to the formula in the attached spreadsheet.
After that, if any number of words are not in the same order, deduct 5.
If there is any number of variations of words deduct 3.
These conditions are cumulative, that is: if any number of words are not in the same order AND there is any number of variations of words, deduct 8.
3.) Domain Name Modifier Metrics
If the domain exactly matches our keywords, then add 20 points. (For example, kw: shopping centres, domain:shoppingcentres.org, make APA=PA+20)
Else, if kw is under 4 words and domain name includes all the targeted keywords, then add 10 points.
(example: kw= shopping centres, domain = BestShoppingCentresBaltimore.org , add 10. If kw= Good shopping centres Baltimore , that is 4 words, then do not check them and don’t add points)
Else, if kw is under 4 words and domain name includes all the targeted keywords or their variations (like writing vs. writer), then add 7 points. (example: kw= shopping centres , domain = ShopCentresBaltimore.org )
Else, if kw has exactly 3 words and domain name includes 2 of the targeted keywords or their variations (like writing vs. writer), then add 7 points.
Else, if kw has exactly 3 words and domain name includes 1 of the targeted keywords or their variations (like writing vs. writer), then add 5 points. (example: kw= best shopping centres, domain = ShopStores.org )
(New:)
Else if domain name includes all the targeted keywords or their variations (like writing vs. writer), then add 7 points. (example: kw= shopping centres , domain = ShopCentresBaltimore.org )
Else If kw has over 3 words and domain name includes one less the targeted keywords or their variations (like writing vs. writer), then add 8 points. IE if it’s a 4 word keyword and has domain name has 3 words, or if it’s a 6 word keywords and the domain name has 5 words.
Else If kw has over 3 words and domain name includes two less the targeted keywords or their variations (like writing vs. writer), then add 5 points. IE if it’s a 4 word keyword and has domain name has 2 words, or if it’s a 6 word keywords and the domain name has 4 words.
Else If kw has over 3 words and domain name includes three less the targeted keywords or their variations (like writing vs. writer), then add 2 points. IE if it’s a 4 word keyword and has domain name has 1 word, or if it’s a 6 word keywords and the domain name has 3 words.
Update October 17, 2012: Google has made some changes and I have posted an update to my keyword research tactics here: https://www.nohatdigital.com/blog/how-to-choose-keywords-for-your-adsense-domains
Steve says:
Hi Hayden,
First off, this is some great information you are putting out there which I for one appreciate, so thanks!
I just wanted to ask, when you are referring to modifications to the PA based on domain, are you talking literally about the domain, or the full URL? I suspect domain but if this is the case, do you take the full URL into account at all?
Thanks again
Steve
Marc says:
Hi Hayden the attached spreadsheet link doesn’t work.
Marc
Hayden says:
Thanks Marc, the link is fixed. It is just =60-X*0.6, made for debugging the values.
Steve K says:
Hi Hayden,
I was really excited to use your stuff. You seem to have a great system in place, and like I told you in a previous comment, I am trying to get to my goal of $1000 a month. I was very encouraged by your reply.
Problem is that I am not having much luck finding keywords that fit the criteria. Can you tell me about what percentage of keywords you run through the formula end up fitting?
Thanks!
Hayden says:
Hi Steve,
It’s very hard to give a percentage. Like I showed in the previous videos, I have scraped large lists of keywords (about 2 million) and I ran (and continue to run) batch check on them.
Here’s a good way to get started:
Find a niche that has high CPCs in the contextual tool (>$4). Then find the 4+ word searches in those niches that have over 1000 LMS. You should be able to find a lot like that. And once you find one site, pickup EVERYTHING in that niche that you can.
That’s the best I can do short of telling you what niches to target (which will likely immediately make them competitive lol).
Steve K says:
Ha Ha well you have my e-mail so feel free to send those suggestions! Maybe you could sell lists of keywords. I know I know…give a man a fish, teach a man to fish.
Marc says:
Hi Hayden for now since I have a Mac I’ll have to do this manually until you get the online version up which isn’t a big deal.
Question, does the % of match with domain pertain to keyword if it’s a subdomain? Example… http://www.widgets.com/KEYWORD.
michael says:
I don’t know the answer but technically a sub-domain would be http://keyword.widgets.com.
Marc says:
Ok thanks but I’m referring to a page maybe, I might be using wrong term. If the target is large blue widgets as an example…
domain.com/largebluewidgets
Or is above considered keyword in URL and it’s not relevant according to the parameters?
Hayden says:
It doesn’t pertain to anything BUT the root domain.
Don’t worry about the page URL, and subdomains give next to no benefit.
Marc says:
Thanks you sir! I found a couple niches, exact match isn’t avail in com, net or org but available in us and biz. Do you suggest a us or biz or as you said in I think part 1 that EMD’s are not as relevant as long as keyword is in domain?
largebluewidgetshq, largebluewidgetsnow etc
If we can’t find an exact match all isn’t lost.
Marc
Hayden says:
I don’t suggest getting the .us or .biz. I also generally don’t get the hyphenated .org or .net. This is because both are very rarely seen in the top of the SeRPS. Just get the phrase match in that case.
rlander says:
Hayden,
Thanks for the excellent info. I just wanted to add that the PA/DA info can also be found on opensiteexplorer.org.
I’m a CS/machine learning grad and I have great interest in automation tools.
Like you, I also built an expired/abandoned domain/subdomain crawler and it relied heavily on Yahoo Site Explorer’s API. Now that it is no longer available and Semoz’s API is severely rate-limited, it is much less performant/useful than it used to be.
All of my tests with trying to come up with a definitive keyword competition index did not do very well, so now I’m experimenting with keyword profitability prediction based on past Google Analytics data +Adwords data + machine learning algorithms. Have you experimented with prediction/classification (you mentioned linear regression at the beginning of the post, hence the question)?
The greatest impact with the least effort that I’ve seen so far was adsense optimization with genetic algorithms.
We should talk.
rlander
Andre Garde says:
rlander,
Really interesting post regarding KW profitability. How big was your sample size of Analytics/Adwords data prior to your analysis? Or does it develop the data set over time and further refine itself?
Further, was this profitability prediction only, or were you able to extend your tool to do ranking difficulty as well?
rlander says:
Andre,
I have two problems with my dataset:
1) small size: around 600k keywords, total;
2) I only started recording serps rankings a few months ago, so a lot of the data is useless without this info.
Right now, I’m only trying to predict profitability, regardless of difficulty (although it is an important part of profitability). The goal is being able to predict wich keyword/domain/link profile combo would yield the greatest return on investment.
One thing that occurred to me after seeing the videos is modeling seomoz’s keyword difficulty index (which shouldn’t be too difficult) in order to reduce the manual part of checking each keyword.
Hayden says:
Hey Rlander,
We should definitely talk, shoot me an email if you like. Have you ever worked with the Ahrefs API? It is a good option, though can get expensive if you don’t set limits to what you analyze.
To answer your question, no I haven’t really experimented with it thus far. Right now the algorithm is based on my understanding of the SERPs, but I do get 95% of my sites indexed. I do plan on using multiple linear regression to see which versions of CMI best correlate with ranking.
I have also planned on doing automating Adsense optimization with genetic algorithms for a long time but never got around to it. Would love to hear your experiences there.
rlander says:
Excellent, Hayden, thanks for the pointer to Ahref. BTW, just sent you an email.
Andre Garde says:
The reply button is missing above so hopefully rlander sees this:
Do you think it’s possible to account for anomalies in your predictor? Specifically in terms of conversion ratio. What I mean is that you can have the right KW/traffic/link building but can’t convert to save your life. Statistically speaking, if you’ve got enough data you should be able to pull a reasonable standard deviation over time between what the tool thinks you should earn vs. what you really earn.
That feature would be bloody amazing.
william says:
Hi, Hayden
Followed you here from Spencers blog and really love the videos with the matter of fact way of demonstrating exactly what you do and how you do it. I’m fairly new to this game (coming from a mainly publishing/marketing background) And recently discovered a tool called SECockpit by Shane Melaugh although it has the Semoz API built into it. I believe you have a more refined business angle showing actuals rather than scores and as such I will be trying out your processes to see if there is a middle ground in the elusive world of the SEO goldrush post-panda and penguin. Once again really impressed with your skill-set and openness to share your knowledge.
Hayden says:
Thanks William. Ya I reviewed SECockpit based on another reader asking about it. It’s an impressive tool, but has one huge flaw (like most tools), its competition score seems to be keyword agnostic.
I went through a bunch of my domains that I ranked really easily and it was really off on the 4+ word domains, because it wasn’t taking page titles into account. For 2 and 3 word domains it was quite good though (since they generally all have title matches).
Marc says:
Hi Hayden you’re like a mad scientist of SEO. I want to compliment you on how well you explain things and make things as clear as possible.
I’m just starting out in niche sites to sell products not necessarily Adsense at this point in my career. Can your keyword research and competition techniques work in creating niche sites to sell products?
Marc
Hayden says:
Hi Marc,
Yup they work just as well for selling products and identifying niches for products as for Adsense. In fact I have a couple businesses I made strictly from this keyword tool. I identified some huge terms that can get 6 figures in traffic each month, that were uncompetitive. Went to alibaba and ordered a bunch of product, teamed up with a guy who is a master at fulfillment.
I may turn that into a case study – how to identify good niches to sell online products (not affiliates).
Marc says:
Hi again Hayden I am also looking into niche’s where I can import my own product. I’m referring to physical products like Amazon in which I’m starting out as.
Since I’m not able to use your Macro’s currently because our house is all Mac’ed out I have to do this by hand which I think I could do quickly with practice. Since I’m
starting out with Amazon and/or products I’m going to use your research methods. Down the road I am interested in Adsense sites but don’t want to get off the path I’m on.
I have some question/s…
Are we looking at keyword difficulty and page authority or domain authority in SEO moz to get ‘adjusted’ numbers according to your percentages?
1.) Links to page Modifier Metrics
If PA=1 , it means 0 external linking root domains, make APA=40% from DA.
Else do nothing, leave APA as is.
The above you’re referring to DA which is Domain Authority?
Marc
Hayden says:
You are looking at PA. The only exception is the case you just pointed out, which makes the PA 40% of the DA, before being adjusted by the further filters.
Marc says:
Ok so if we see a site with with a PA of 1 and a DA of 98 then the PA is 58.8 and we are looking at PA.
Hayden says:
Yup, and it would then be adjusted by the other steps.
Marc says:
Thanks SEO moz is the best paid took I’ve used. What I like about it is it’s fast and it’s web based. I can do research on the go on my iPhone or iPad. I heard open site explorer is one of the best on the market.
Marc says:
Thanks again Hayden. Let’s say keyword is cheap hybrid golf clubs. The #1 page title is hybrid golf clubs and gloves.
My keyword is 4 words, cheap hybrid golf clubs. The #1 site looks like 4 keywords, hybrid golf clubs and gloves(exclude ‘and’) and missing the word cheap.
How does page title modifier work in this case?
Marc
Hayden says:
It is a 75% match and therefore a 15% deduction in that case. It is all defined in the spreadsheet in this post.
Marc says:
Hi Hayden after we get the APA on all 1st
Marc says:
Hi Hayden is Keyword Difficulty based on average of SEO moz’s PA average of top 10 sites? If so then after we get the APA of the top 10 do we get the average APA?
Marc
Marc says:
I’m still trying to get the knack of doing this manually.
SEO m
Marc says:
Ooops sorry hit reply by accident. As I said I’m trying to get knack of doing this manually. Below are the #’s.
SEO moz Keyword Difficulty: 38%
APA of top 10 sites(using your metrics)
1- 36.05
2- 48
3- 17.86
4- 24.72
5- 10
6- 29
7- 52.2
8- 38
9- 38
10- 18.7
Hayden says:
Looks good. I’m guessing the #7 result got penguined, will likely fall out of the SERPs.
Marc says:
Hi Hayden from looking at #7 site it’s a 75% page title(-15%) and since keyword is 4 words, #7 domain is has 3 words( kw has over 3 words and domain name includes one less the targeted keyword, + 8 points). It originally started out 52 PA and API turned out .20 higher.
Marc says:
Hi Hayden since I don’t have Windows and on Mac’s I might be interested in seeking to get a tool developed that will automate the process of keyword research/competition. Since I’m more interested in the start of my career of selling physical products whether it be Amazon or products within an industry that I was previously involved with offline, this process seems better than the likes of Market Samurai etc as I assume SEO moz is just a premium tool in the industry.
Doing this manually isn’t a big deal as I’m starting to get the knack. About how much funds in your estimation would it entail to get this tool made for Mac? And if I do decide to get it developed for a Mac is all we need to is give the above ‘exactly how’ to a developer?
Marc
Hayden says:
Sent you an email Marc.
Marc says:
Hayden you’re right! That #7 as you said isn’t in SERP’s. Why?
Marc
Marc says:
Oops my bad it’s at #4 now.
Marc says:
Hi Hayden what about a keyword that is under 4 words and domain included 1 keyword?
“Else, if kw is under 4 words and domain name includes all the targeted keywords or their variations (like writing vs. writer), then add 7 points. (example: kw= shopping centres , domain = ShopCentresBaltimore.org )”
Some domains in this niche don’t include ALL targeted keywords which in this case is a 2 word keyword.
Marc
Daniel says:
Hayden I found a couple of great keywords that have only .info EMD free and people on forums are saying a lot of different things about info domains and seo… In your experience are .infos way harder to rank then net’s and org’s ?
Thanks once again :)
Hayden says:
Daniel, I’d recommend a phrase match .com domain over a .info, any day.
Daniel says:
Yeah I know that :) but there are few great keywords that don’t have com/net/org free, only .info
Those are the ones I am not sure about. I am deciding between mykeywordHQ.com and mykeyword.info
Jason says:
@Daniel
I would go with the HQ…
I also have great luck with .cc EMD
Daniel says:
Damn I am confused … some are saying that .info isn’t different from net/org and it still counts as EMD while others are 100% sure that only com/net/org get EMD bonus from google :s
Marc says:
That’s why you’re confused. As Hayden said if you’re going to follow one person don’t read anything else.
Hayden says:
Personally I stick to .com, .net, .org EMD. And then go to hyphenated .com, and if nothing than phrase match .com.
Marc says:
Hyphenated is prob the next best if EMD isn’t available especially of you aren’t building for branding.
So the process is EMD in com, net or org. If exact match isn’t available the hyphenated. If hyphenated isn’t available then phrase match.
Jason says:
Sounds like nobody really knows….. ;)
I use .com, .org, .cc, .co for niche adsense sites.
Danny says:
If your KW never exceeds 5 words, your only possible [% word match values] are:
0
20 (1/5)
25 (1/4)
33 (1/3)
50 (2/4, 1/2)
60 (3/5)
67 (2/3)
75 (3/4)
80 (4/5)
100
The CMI-spreadsheet can be shortened to this list, or further to:
[% APA deduction] = 60 – 0.6*[% word match].
Marc says:
Has anyone tried Hayden’s keyword/competition research using the above metrics? If so share your experiences.
Marc
Marc says:
Hi Hayden got question about page title metrics.
After that, if any number of words are not in the same order, deduct 5.
Same order do you mean exact match order?
Keyword Blah blah blah blah Keyword Keyword.
Above is 3 word KW has all 3 keywords present not exact match, but 2/3 are exact match.
Hayden says:
Yup exact match order. You’d deduct 5 in that example
Danny says:
(duplicate)
If your KW never exceeds 5 words, your only possible [% word match values] are:
0
20 (1/5)
25 (1/4)
33 (1/3)
50 (2/4, 1/2)
60 (3/5)
67 (2/3)
75 (3/4)
80 (4/5)
100
The CMI-spreadsheet can be shortened to this list, or further to:
[% APA deduction] = 60 – 0.6*[% word match].
Marc says:
Hi Danny I happen to be analyzing a 5 word KW.
Marc says:
Question; let’s say there is the words…
is, the in the target keyword. Do we count these?
With the words is, the my target keyword is 7 words. Without them it’s 5.
Bob says:
This is probably a dumb question but how are you guys defining page title. The url page name – xyz.com/pagetitle.html, the meta title tag, or The word press title of the page? They could all be different.
Adrian says:
It is answered somewhere above in the comments. “page title” refers to what you call “wordpress title of the page”. xyz.com/pagetitle.html is called an URL period even if the file name is similar to the page title.
Marc says:
The text that shows up above your browser.
Bob says:
Im coding my own version of this, any suggestions for handling the natural lanaguage equivalents/word variations rules? I have no experience with this. Honestly it seems to me really difficult to do without a lot of fudging that would end up being very inaccurate and swamp out any usefulness.
thanks in advance
Jeremy says:
Bob, Any luck on finding a reliable way to handle the keyword variants variations in an automated way?
I’ve been trying to figure this out as well but nothing seems very reliable/accurate
Hayden says:
We use a PHP stemmer.
Same logic Python NLTK uses in its stemmers, online demo:
http://text-processing.com/demo/stem/
Jeremy says:
Thanks, Hayden!
Interseting stuff.
Bob says:
Hayden had mentioned you can get top 10 serps results from the SEOMoz API but I cant find any reference to how to do this on their site or Googling. Can anyone point me to some info on this?
Thanks
Bob
Bob says:
I found it. Its called Top Pages and only available on the Mozscape API paid accounts starting at $500 a month. Going to have to find a free efficient way to get this from another tool or API.
Bob says:
I have been working on recreating your automated bulk keyword analysis tools. Its been a fantastic learning experience. One thing that has come up though is the near impossibility of extracting THE official Google Top 10 serps for many, maybe most keywords in the US. It just doesn’t seem to exist. Even if you can manage to account for: personalized, local, and all the universal search features, and (btw there is no consensus that I could find of how to reliably do the above) the resulting list of plain vanilla urls varies from tool to tool. This is easy to confirm for yourselves. Given the high level of granularity being applied in your APA filtering this really matters. Also given that a high and increasing number of searches are seeing these personalized, localized, universal results, even getting in the top 3 results is going to give you a significantly lower ctr than seen previously.
Im bringing all this up because im wondering if it might make more sense to analyze only the top 3-5 results or conversely extend it to the top 15 – 20. Im leaning towards 5 because even with the inevitable bouncing around I think this small set will give you a more accurate analysis. Just a gut feeling.
Any thoughts anyone?
Doug says:
Hey Hayden, I’m loving your tutorials, I’ve learn’t more from your site about MNS than any other site. Looking for to some more tutorials.
I was wondering if you’re still planning on releasing a paid service of the above??
If not I’ll look into getting this coded up. Thanks.
david says:
Hayden,
In your videos, you made a few comments about disregarding stop words when calculating the APA. Can you clarify or confirm the following example:
Arbitrary starting keyword phrase:
“information about some nursing training abroad”
The APA rules are applied to the stop-word removed version:
“information nursing training”
However, the exact match domain rule is applied to:
“informationaboutsomenursingtrainingabroad”
Thanks for any clarification.
John says:
Hi Hayden,
When you say add or deduct ‘points’, do you mean percentage, or actual numbers?
So, add 10 points = add 10% or add 10?
If anyone else knows the answer I would really appreciate it.
Thank you,
John
Matt says:
10 points. Not percent
Jeremy says:
What about assessing whether or not the keyword actually appears on the ranking page?
The description Google shows for each search result will show in bold the keywords that match the query.
So you can use this and apply the formula for assessing titles to the description.
In many cases you will see that the ranking page doesn’t even use the exact keyword/phrase anywhere on the page.
Just a thought but I was thinking of applying the rules for evaluating the title but reducing the “points” you add/subtract by half.
Any thoughts?
Jeremy says:
Sorry, one more question.
After we get the APA for all of the results, do we add them all together and then get the average for the final “difficulty score” for that keyword?
This was asked previously but not yet answered.
Jay says:
hi guys anyone nhd team . please answer this question i stumble upon on your page by reading at nichepursuit .
1. does this metrics was the same on the LTP using by spencer haws ?
2. from the thirst method , how do you compute the links needed ? im not so sure if you said how it is done in the video so im hoping you answer this question
Hayden says:
1 – Yes it is based on this with some small changes.
2 – This is all in the thirst video.