Domain Finding Strategy: Public Data Sets

This was originally part of a private course for interns, but I’ve decided to open it up to the world.  It was written by Scott, one of the original interns who headed a group of V2 Interns, and is now my partner and manager of my entire inventory.

Update: Although PBNs still work, they now have a history of being targeted by Google and therefore may not be the safest option. This is why we now focus on creating online businesses that are independent of SEO traffic.

Note that this is part of the expired domain series which has been compiled in order of importance here.

Another related method of finding high value domains involves tracking down existing lists of old URLs and using a variation of the Scrapebox/Xenu method. Ideally, your list should be “old” (2008 or earlier), as this provides a better probability that the domains are available and of high authority. Also, the list should be BIG, as the vast majority of these domains will not be available or will not be high authority. This method is more of a “shotgun” approach instead of a “sniper rifle” approach to finding good domains… volume over finesse.  The diagram below provides an overview of the basic idea. As always, understanding the concepts is more important than repeating the process verbatim… and being creative, organized and persistent goes a long way.

(Click to enlarge)

Descriptions for each step:

1 – Find a list of URLs. Along the same lines as the 2006 AOL Dataset, search for big, public data sets that are available for download. Government, university, and research initiatives often have compiled data available for all sorts of reasons. Seek out data sets that would have lots of URLs… for example, census data from the 1900s would be less interesting than a directory of blogs from 2005.  Here are just a few examples of lists of lists… do a little digging and you’ll find many more:

http://www.reddit.com/r/datasets/
http://delicious.com/pskomoroch/dataset?&page=31
http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html
http://www-personal.umich.edu/~mejn/netdata/
http://www.brbpub.com/free-public-records/
https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv

Finding a good list is the most challenging and time consuming part of the whole process but can be worth the time investment.

2 – Cleanse the list into something more workable. If the list is ~400,000 or less, use Excel to manipulate the data. If greater than 400,000, use a database to remove duplicates and segment the list into chunks that are able to be easily manipulaed in Excel.
Strip the URL to its subdomain using ‘URL Tools For Excel Add-in’: http://connect.icrossing.co.uk/url-tools-addin-excel_5458 (tip of the hat to Danny for this one). From there, use text-to-columns function in Excel (or whatever is easiest for you) to get all URLs down to ‘domain.com’ format.

3 – A cleansed list has only unique values in ‘domain.com’ format (i.e. no http://, www, subdomains, etc.)

4 – Bulk PR checking utility. http://pagerank.my-addr.com (a second hat tip to Danny) can be a good way of checking PR for lots of URLs. It’s $10 to check 1,000,000 domains… faster than Scrapebox and won’t burn out your proxies or draw upon your system resources.

5 – If the PR is NA, then there is a decent chance that it is available. Continue down the path of seeing if it is a high value domain.

6 – Lots of PR6s and PR7s from the big list provide a good data set on which to run Xenu or SB in the hopes of finding PR4s or PR5s

7 – Run Xenu (1 deep) or Scrapebox Link Extractor (External links only) to find pages linking from the PR6s and PR7s.

8 – After running Xenu or SBLE, you’ll have a big list of URLs that receive a link from a PR6 or PR7. Cleanse this list for further processing.

9 – Name.com bulk domain check is (to my knowledge) the easiest and fastest way to check domain availability through a web page. Maximum domains to check at once is 4000. Put in a perfectly cleansed list (‘domain.com’) or else it will freeze up. Sort the results (by ‘Availability’ column) for easy copy/paste.

10 – Domains that are available may or may not be of high value. Run through the FREE DA Checker to find out. Export from the Free DA Checker will have DA, wwwDA (usually always the same as DA but sometimes different–take the higher value), PA and wwwPA.

11 – Evaluate the results of the export from the Free DA Checker.  Any domain with EITHER (DA or wwwDA) > 25 OR (PA or wwwPA) > 30 should be selected for a second run through the PAID DA Checker

12 – Export from the Paid DA checker will have LRD and rdMT metrics. Combine this with the export from the Free DA Checker to arrive at the ‘Domain Value’ metric.

13 – For domains that have good metrics, check for spam. See ‘Spam Checking‘ post for more information.

14 – For anything with PR=NA, there is a decent chance the domain is available and may have a backlink that is PR6 or PR7.

15 – Check for spam (‘Spam Checking‘ post) and also confirm that there is a real PR6 or PR7 backlink to the home page. A PR6 or PR7 linking page that has 60+ outbound links will likely NOT pass it’s TBPR.

16 – What’s left should be domains that are available, not spam and of high value for a PBN or Adsense site. Buy and launch!

17 – What’s left should be domains that are available, not spam and, once launched, be a PR4 or PR5… making them good for a TLA site or PBN site. Buy and launch!

The NoHatDigital Internship.

Learn the Exact Skills and Processes the NoHatDigital Team use to Dominate Online.

Learn More...