groupaaa.com

Applied on: Linux (Cpanel) Hosting

There are various types of bots and spiders that transverse sites for different reasons. Among them activities such as email harvesting and other malicious activity. By using a .htaccess file, you can block the majority of well know malicious bots.

Bad bots typically ignore the wishes of your robots.txt file, so you’ll want to ban them using means such as .htaccess. The trick is to identify a bad bot. Below is a useful code block you can insert into .htaccess file for blocking a lot of the known bad bots and site rippers currently out there.

Note: Bots that are listed below will all receive a 403 Forbidden error when trying to view your site. The amount of bandwidth savings and decrease in server resource usage as a result may be significant in many cases.

RewriteEngine On   RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]   RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]   RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]   RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]   RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]   RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]   RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]   RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]   RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]   RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]   RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]   RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]   RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]   RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]   RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]   RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]   RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]   RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]   RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]   RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]   RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]   RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]   RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]   RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]   RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]   RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]   RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]   RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]   RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]   RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]   RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]   RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]   RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]   RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]   RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]   RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]   RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]   RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]   RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]   RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]   RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]   RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]   RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]   RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]   RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]   RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]   RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]   RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]   RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]   RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]   RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]   RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]   RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]   RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]   RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]   RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]   RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]   RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]   RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]   RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]   RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]   RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]   RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]   RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]   RewriteCond %{HTTP_USER_AGENT} ^Zeus   RewriteRule ^.* - [F,L]