Just add the spider's IP or website's IP to the whitelist to allow them to enter your website and use your content.

Thank you.

Senior Developer.

Hello Senior Developer.

This sadly is not really a great answer. Spiders IP addresses are not often made public and often they do not have specific IP ranges. I already looked into this as a solution, google had a great blog post on it but I unfortunately could not find it to post it here.

You say in the plugin Description that you only need to delete one file to allow spiders to grab content. Since it is not a huge stress on our site to allow spiders I think we shall just allow all of them and then add some of the more nefarious spiders to the IP blacklist to cut down on their requests. 

Thank you for your response. 

I'm going to see what's the best way to allow all the good ones and update this plugin, meanwhile if you want to try something:

You can log in at your ftp server and change the name of this 2 files. "botsblocker/classes/libs/zbblock/signatures/spd.sig".

"botsblocker/classes/libs/zbblock/signatures/hn.sig".

That should allow some spiders to get in your website.

If you can please send me your "botsblocker/classes/libs/zbblock/vault/killed_log.csv" file to take a look at it and see where they are coming from to your server and being blocked.

Thank you.

Senior Developer

That would be an amazing idea if you can update it to allow popular spiders! In the mean time I will get back to you with the requested files and directories changes. 

Thank you very much for your support!

Hello Senior Developer.

Sent you the requested files a while back now. Just wondering if there is any further updates to this issue. After I re-named the requested files Spiders are still blocked. 

This is solved by adding the hostname to the new "host" field in whitelist settings.

Senior Developer.

Hi SD,

Not sure i understand, so the updates allow the following as per Jamie's request ?

One problem I have now encounted is that our site nolonger embeds into chats  like Discord, Facebook and Twitter. I would like to allow spider to crawl my site so that it may be embedded however still like to block Yandex and untrustworthy spiders on the site.