如何使用htaccess阻止Backlink Checker Bots

时间:2016-05-16 23:36:54

标签: wordpress .htaccess mod-rewrite

我试图用我的Wordpress网站的htaccess文件阻止Backlink Checker Bots,但面临一个奇怪的问题。

此代码非常适合阻止Ahrefs和Majestic机器人:

RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Majestic-SEO [NC]
RewriteRule ^.* - [F,L]

但是当我上传机器人的完整列表时,代码由于某种原因无法运行,而且AhrefsBot能够访问我的网站。由于我不是开发人员,我无法弄清问题是什么。

如果您能查看完整的机器人列表并告诉我是否存在某种错误,我将非常感谢您的帮助。

请在下面找到完整的机器人列表:

    RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Majestic-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Alexibot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Aqua_Products [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BackDoorBot/1.0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BecomeBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BlekkoBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BlowFish/1.0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BotALot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BuiltBotTough [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Bullseye/1.0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BunnySlippers [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^CheeseBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Copernic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^CopyRightCheck [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Crescent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^DISCo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^DittoSpyder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Dotbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Enterprise_Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EroCrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Exabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Gigabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GrabNet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HMView [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Harvest/1.5 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InfoNaviRobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InterGET [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JennyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JetCar [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Jetbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LNSpiderguy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LexiBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkScan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkextractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MIIxpc [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MSIECrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Navroad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NearSite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetAnts [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Octopus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Openbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Openfind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PerMan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ProPowerBot/2.14 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Python-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^RMA [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ReGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^RealDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^RepoMonkey [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Rogerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Sitebot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SpankBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Stanford [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Surfbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Szukacz/1.4 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Teleport [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^TeleportPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Telesoft [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Teoma [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^TheNomad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^True_Robot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^URL_Spider_Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^VCI [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWW-Collector-E [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebAuto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebBandit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebCopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebEnhancer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebFetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebReaper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebSauger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebStripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebVac [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebZip [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebmasterWorldForumBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Xenu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Zeus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^asterias [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^b2w/0.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^cosmos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^dotbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^dumbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^eCatch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^es [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^exabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^gigabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^grub [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^grub-client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^hloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^httplib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^humanlinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^libWeb/clsHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^looksmart [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial/1.34 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^moget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^naver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pavuk [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^psbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^rogerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^scooter [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^searchpreview [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^sootle [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^spanner [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^suzuran [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^turingos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Blinkx/DFS-Fetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^CheckLinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EC2LinkFinder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^findlinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^freshlinks.exe [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InfoLink [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^link_checker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkAlarm [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^linkbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkChecker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^linkdex.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkLint [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^linklooker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Linkman [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkScan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinksManager.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LinkSweeper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^radian6_linkcheck [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ShowLinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebLinker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^yolinkBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ScrapeBox [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Majestic-SEO [NC]
    RewriteRule ^.* - [F,L]

1 个答案:

答案 0 :(得分:0)

我遇到了同样的问题,所以我开始使用robots.txt阻止代码。 它100%工作。 在robots.txt中使用以下

User-agent: AhrefsBot
Disallow: /