Linkscape Index Update Doubles Size! – Moz
Skip to content
Moz logo
Menu open
Menu close
Search
Products
Moz Pro
Moz Pro Home
Moz Local
Moz Local Home
STAT
Mozscape API
Free SEO Tools
Competitive Research
Link Explorer
Keyword Explorer
Domain Analysis
MozBar
More Free SEO Tools
Learn SEO
Beginner’s Guide to SEO
SEO Learning Center
Moz Academy
SEO Q&A
Webinars, Whitepapers, & Guides
Blog
Why Moz
Agency Solutions
Enterprise Solutions
Small Business Solutions
Case Studies
The Moz Story
New Releases
Log in
Log out
Products
Moz Pro
Your All-In-One Suite of SEO Tools
The essential SEO toolset: keyword research, link building, site audits, page optimization, rank tracking, reporting, and more.
Learn more
Try Moz Pro free
Moz Local
Complete Local SEO Management
Raise your local SEO visibility with easy directory distribution, review management, listing updates, and more.
Learn more
Check my presence
STAT
Enterprise Rank Tracking
SERP tracking and analytics for SEO experts, STAT helps you stay competitive and agile with fresh insights.
Learn more
Book a demo
Mozscape API
The Power of Moz Data via API
Power your SEO with the proven, most accurate link metrics in the industry, powered by our index of trillions of links.
Learn more
Get connected
Compare SEO Products
Free SEO Tools
Competitive Research
Competitive Intelligence to Fuel Your SEO Strategy
Gain intel on your top SERP competitors, keyword gaps, and content opportunities.
Find competitors
Link Explorer
Powerful Backlink Data for SEO
Explore our index of over 40 trillion links to find backlinks, anchor text, Domain Authority, spam score, and more.
Get link data
Keyword Explorer
The One Keyword Research Tool for SEO Success
Discover the best traffic-driving keywords for your site from our index of over 500 million real keywords.
Search keywords
Domain Analysis
Free Domain SEO Analysis Tool
Get top competitive SEO metrics like Domain Authority, top pages, ranking keywords, and more.
Analyze domain
MozBar
Free, Instant SEO Metrics As You Surf
Using Google Chrome, see top SEO metrics instantly for any website or search result as you browse the web.
Try MozBar
More Free SEO Tools
Learn SEO
Beginner’s Guide to SEO
The #1 most popular introduction to SEO, trusted by millions.
Read the Beginner’s Guide
How-To Guides
Step-by-step guides to search success from the authority on SEO.
See All SEO Guides
SEO Learning Center
Broaden your knowledge with SEO resources for all skill levels.
Visit the Learning Center
Moz Academy
Upskill and get certified with on-demand courses & certifications.
Explore the Catalog
On-Demand Webinars
Learn modern SEO best practices from industry experts.
View All Webinars
SEO Q&A
Insights & discussions from an SEO community of 500,000+.
Find SEO Answers
August 7-9, 2023
Lock in Super Early Bird savings for MozCon
Snag tickets
Blog
Why Moz
Small Business Solutions
Uncover insights to make smarter marketing decisions in less time.
Grow Your Business
The Moz Story
Moz was the first & remains the most trusted SEO company.
Read Our Story
Agency Solutions
Earn & keep valuable clients with unparalleled data & insights.
Drive Client Success
Case Studies
Explore how Moz drives ROI with a proven track record of success.
See What’s Possible
Enterprise Solutions
Gain a competitive edge in the ever-changing world of search.
Scale Your SEO
New Releases
Get the scoop on the latest and greatest from Moz.
See What’s New
New Feature: Moz Pro
Surface actionable competitive intel
Learn More
Log in
Moz Pro
Moz Local
Moz Local Dashboard
Mozscape API
Mozscape API Dashboard
Moz Academy
Avatar
Moz Home
Notifications
Account & Billing
Manage Users
Community Profile
My Q&A
My Videos
Log Out
December 8, 2008
Linkscape Index Update Doubles Size!
Moz News
The long awaited Linkscape index update is here. We’ve gotten a lot of feedback, we’ve heard about a few success stories and we have a few thoughts from the development side to share with you.
First, we’ve included about 38 billion URLs, from about 230 million sub-domains (e.g., twopieceset.blogspot.com) inside about 48 million second level domains (e.g., *.blogspot.com). As Live’s Nate Buggia recently pointed out, there’s a Netcraft survey which suggests that there are ~75 million “active” domains. So we’re certainly reaching a scale which gives us a comprehensive view of the web. 38 billion URLs is not double the previous number of URLs in the index (nearly 30 billion); however, it reflects that we are doing some deeper crawling of URLs and domains we already had indexed. Really, of more interest is that we have about 450 billion links, which is more than double our previous index of approximately 170 billion links.
We’re also making the top 3000 links available per URL and domain in advanced reports. These links are also filtered so that no more than 10 links from any domain are shown. This dramatically increased volume and diversity of links gives you the opportunity to see many more of the top links along many dimensions (mozRank, mozTrust, etc.). And the anchor text analysis is much more representative of your presence on the web as a whole.
To illustrate the variations in our link counts, consider these sites and pages. You can see, almost across the board we know about substantially more links for any site and page, and have used this broader view of the web to update mozRank. The small general decline in mozRank is an indication that we’ve spread mozRank across more pages. In general we’ve found a higher correlation in our latest data to Google’s toolbar PageRank, when excluding penalized sites.
You should note that because our index has grown substantially, these additional links and changes to mozRanks do not reflect growth in new links, but rather in new links we’ve discovered. It would be unwise to compare link counts from the old index with counts from the new one. Instead comparisons should be confined to metrics for sites and pages drawn from this latest update. This artificial update effect will diminish as we refine our processes and reach the end of the beta period.
How does this benefit SEOs?
A bigger and fresher index means:
Greater accuracy in link counts and domains
Greater representation of what the search engines see and how they might interpret and use the data
More accuracy in mozRank & mozTrust, leading to better data comparisons & analysis
More fresh data that helps understand what’s happened in the recent past
Up to 3000 links per URL in the report means:
Know about more links that point to you, so you can request anchor text changes, conduct better self analysis, or fix links that are broken
Reverse competitive strategies more comprehensively to analyze how they’re winning
Find links that you could possibly acquire from your competitors
Get better anchor text distribution data
URL normalization means:
Link counts aren’t biased by sites and pages that create duplicates
Our data is more like the major search engines who also do this stripping and canonicalization
Limiting to ten links per domain means:
You can see a wider variety of links from different domains
3,000 links will show you at least 300 unique linking domains (often many more)
Here’s a quick list of some of the things people have used Linkscape reports for:
Analyze their link counts, mozRank, mozTrust against those of pages ranking above them in the SERPs
Look at anchor text distribution and numbers to see why a site might be ranking where it is for a given term/phrase
Reverse competitor links to find sources they can get themselves
Look at the relative value of particular links based on the juice they pass and the quality of their domains/pages
Compare mozRank to PageRank to see if there is a large discrepancy (often indicating a penalty if mR is much greater than PR)
Use link counts in conjunction with traffic data from sources like Compete, Hitwise, Quantcast, etc. to see how link numbers, mozRank, mozTrust, etc. map to traffic
Speaking of link counts, there’s a lot of ways to interpret links, especially when you’re adding them up. Here are a few thoughts about how we count links:
We do not double count duplicate links from and to the same page. For instance, we don’t consider the two links to our homepage in our header and footer as separate links.
We do a great deal of URL normalization. We strip common URL parameters (e.g., SESSID, jsessionid, redirect, etc.) and remove any resulting duplicate links.
We do not collapse the source and target of 301s, 302s, meta refreshes, etc. for the purposes of link counts. Of course, we do pass the properties (e.g., mozRank) of the source to the target.
That last point has been a controversial design decision and has led to some confusion. To get a full view of links to a page you should run reports for several versions (e.g. www and non-www). However, one advantage to this approach is that it lets you analyze your link profile at a very fine granularity. For instance, we can see who’s linking to “https://moz.com/web2.0”, “http://web2.0awards.org”, and “http://web2.0awards.com” all separately from each other. This helps us to understand our marketing efforts and quantifies the contribution of each of these different URLs which point to the same content. Also if we wanted to remove the 301 and rebrand one of these domains, we have some idea of where we would be starting out from. We do list 301s as single links in advanced reports for the destination of the redirect.
Unfortunately, this makes some of our link counts look smaller than you might see from some other tools. Because we’re consistent within our tool, you can compare the numbers you see for different pages to get a relative sense of popularity. But you can’t, unfortunately, directly compare our numbers to other tools. I suppose this is the sort of thing you come to discover in any beta 😉
It turns out that most the technological challenges with the back-end revolve not around scaling our data collection, but rather around processing and serving data. So we back-end developers have been very busy re-writing our processing pipeline and completely distributing our API architecture, which is why this update took so long to get out the door. You guys probably care about this work because of our substantially improved performance for our PRO toolbar, which we’re also publicly announcing today! I’ll let Danny tell you more about the toolbar, but both of these back-end changes should support our API product and help us to provide you with much more frequent index updates.
We’ll probably see quite a few other changes to the product both visually and in terms of the data throughout Linkscape’s beta period. Obviously we’d like to continue to improve our coverage of the web while keeping the quality, relevance, and freshness of our data equally impressive. If you have any feedback, feel free to post comments on our feedback thread. We always appreciate it, and I hope some of you can see that some of your feedback has made its way into this update.
Snag your MozCon 2022 video bundle for even more SEO insights.
Buy the video bundle!
Read Next
The MozCon 2022 Video Bundle Is Here (Plus, Our 2020 Videos are FREE!)
Read this post
Announcing the Local SEO Certification from Moz Academy
Read this post
Gather ‘Round the Campfire for the MozCon 2022 Day Three Recap!
Read this post
Comments
Please keep your comments TAGFEE by following the community etiquette
Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.
Moz logo
Contact
Community
Free Trial
Terms & Privacy
Jobs
Help
News & Press
Copyright 2022 © Moz, Inc. All rights reserved.