Blocking Bots Based on User-Agent – Moz
Skip to content
Moz logo
Menu open
Menu close
Search
Products
Moz Pro
Moz Pro Home
Moz Local
Moz Local Home
STAT
Mozscape API
Free SEO Tools
Competitive Research
Link Explorer
Keyword Explorer
Domain Analysis
MozBar
More Free SEO Tools
Learn SEO
Beginner’s Guide to SEO
SEO Learning Center
Moz Academy
SEO Q&A
Webinars, Whitepapers, & Guides
Blog
Why Moz
Agency Solutions
Enterprise Solutions
Small Business Solutions
Case Studies
The Moz Story
New Releases
Log in
Log out
Products
Moz Pro
Your All-In-One Suite of SEO Tools
The essential SEO toolset: keyword research, link building, site audits, page optimization, rank tracking, reporting, and more.
Learn more
Try Moz Pro free
Moz Local
Complete Local SEO Management
Raise your local SEO visibility with easy directory distribution, review management, listing updates, and more.
Learn more
Check my presence
STAT
Enterprise Rank Tracking
SERP tracking and analytics for SEO experts, STAT helps you stay competitive and agile with fresh insights.
Learn more
Book a demo
Mozscape API
The Power of Moz Data via API
Power your SEO with the proven, most accurate link metrics in the industry, powered by our index of trillions of links.
Learn more
Get connected
Compare SEO Products
Free SEO Tools
Competitive Research
Competitive Intelligence to Fuel Your SEO Strategy
Gain intel on your top SERP competitors, keyword gaps, and content opportunities.
Find competitors
Link Explorer
Powerful Backlink Data for SEO
Explore our index of over 40 trillion links to find backlinks, anchor text, Domain Authority, spam score, and more.
Get link data
Keyword Explorer
The One Keyword Research Tool for SEO Success
Discover the best traffic-driving keywords for your site from our index of over 500 million real keywords.
Search keywords
Domain Analysis
Free Domain SEO Analysis Tool
Get top competitive SEO metrics like Domain Authority, top pages, ranking keywords, and more.
Analyze domain
MozBar
Free, Instant SEO Metrics As You Surf
Using Google Chrome, see top SEO metrics instantly for any website or search result as you browse the web.
Try MozBar
More Free SEO Tools
Learn SEO
Beginner’s Guide to SEO
The #1 most popular introduction to SEO, trusted by millions.
Read the Beginner’s Guide
How-To Guides
Step-by-step guides to search success from the authority on SEO.
See All SEO Guides
SEO Learning Center
Broaden your knowledge with SEO resources for all skill levels.
Visit the Learning Center
Moz Academy
Upskill and get certified with on-demand courses & certifications.
Explore the Catalog
On-Demand Webinars
Learn modern SEO best practices from industry experts.
View All Webinars
SEO Q&A
Insights & discussions from an SEO community of 500,000+.
Find SEO Answers
August 7-9, 2023
Lock in Super Early Bird savings for MozCon
Snag tickets
Blog
Why Moz
Small Business Solutions
Uncover insights to make smarter marketing decisions in less time.
Grow Your Business
The Moz Story
Moz was the first & remains the most trusted SEO company.
Read Our Story
Agency Solutions
Earn & keep valuable clients with unparalleled data & insights.
Drive Client Success
Case Studies
Explore how Moz drives ROI with a proven track record of success.
See What’s Possible
Enterprise Solutions
Gain a competitive edge in the ever-changing world of search.
Scale Your SEO
New Releases
Get the scoop on the latest and greatest from Moz.
See What’s New
New Feature: Moz Pro
Surface actionable competitive intel
Learn More
Log in
Moz Pro
Moz Local
Moz Local Dashboard
Mozscape API
Mozscape API Dashboard
Moz Academy
Avatar
Moz Home
Notifications
Account & Billing
Manage Users
Community Profile
My Q&A
My Videos
Log Out
By: Ken Wisnefski
December 12, 2011
Blocking Bots Based on User-Agent
Advanced SEO
This YouMoz entry was submitted by one of our community members. The author’s views are entirely his or her own (excluding an unlikely case of hypnosis) and may not reflect the views of Moz.
Normally as SEOs our focus is on giving Google and other search engines as much access to the sites we are working on as possible. There are times however in which privacy is a good thing. We have found blocking bots based on the user-agent very useful for development servers where you might be hosting multiple sites which you do not want crawled or indexed.
With major SE emphasis on real time content and freshness it is imperative to be certain in our live pages and their elements within. Blocking bots access has certainly saved us the embarrassment and any potential problems with indexation of content in advance of intended release.
Below are examples in accomplishing this on either Apache or IIS.
APACHE
On Apache servers it is very easy to block unwanted bots using the .htaccess file. Simply add the following code to the file to block the engines.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} AdsBot-Google [OR]
RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
RewriteCond %{HTTP_USER_AGENT} AltaVista [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule . – [F,L]
The following code will return a 403 error to Google
If you would prefer to redirect the bots to a specific page then just replace the last line above with the following.
RewriteRule ^.*$ “http\:\/\/www.yoursite\.com\/nobots.html” [R=301,L]
IIS
It is possible to do the same type of filtering on an IIS 7 server but it is more complicated. The easiest way I have found involves first installing the IIS URL Rewrite Module.
After the module is installed open the IIS Server Manager, select the site you would like to block, and open URL Rewrite.
Go to ‘Add Rule(s)…’ in the upper left of the window.
Select ‘Request blocking’ and click OK
Use the following settings:
Block access based on: User-agent Header
Block request that: Matches the Pattern
Pattern (User-agent Header): Googlebot|Ads-Google|msnbot|Altavista|Slurp
Using: Regular Expressions
How to block: Send an HTTP 403 (Forbidden) Response
Then click ‘OK’
You can test your settings using Chris Pederick’s great tool User Agent Switcher. When you first install it the options are a bit limited but a very exhaustive list of user agents can be found here. Using these methods to block user agents, while somewhat technical, will you or your developer the flexibility to work on-site without fear of indexation of pages that are in progress
About Ken Wisnefski —
WebiMax is a recognized leader in SEO (Search Engine Optimization) within the online marketing field. Our vast experience in assisting our clients gain substantial online visibility through our efforts has set us apart from the competition. Our “no-nonsense” approach has been developed through our 10+ years of hands on online marketing experience.
We focus our efforts on truly partnering with our clients to provide them with the strategic partner they require to receive the optimal success from their online marketing. WebiMax manages online marketing campaigns for clients within SEO, PPC, SEM, Social Media and Reputation Management. WebiMax is also a leading web design agency.
Specialties
Online Marketing, Search Engine Optimization (SEO), Pay Per Click Management (PPC), Search Engine Marketing (SEM) Advertising, Display Ad Campaigns, Social Media Marketing, Online Reputation Management, Content Marketing, Inbound Marketing, Blogging, Web Design, Conversion Optimization, Consulting
With Moz Pro, you have the tools you need to get SEO right — all in one place.
Start your free trial!
Read Next
How to Use Chrome to View a Website as Googlebot
Read this post
SEO in Real Life: Harnessing Visual Search for Optimization Opportunities
Read this post
How We Increased Revenue with Speed Optimization [Local SEO Case Study]
Read this post
Comments
Please keep your comments TAGFEE by following the community etiquette
Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.
Moz logo
Contact
Community
Free Trial
Terms & Privacy
Jobs
Help
News & Press
Copyright 2022 © Moz, Inc. All rights reserved.