We build. You grow.

Get best community software here

Start a social network, a fan-site, an education project with oxwall - free opensource community software

Get started

Spider Crawler plugin | Forum

Topic location: Forum home » Community Exchange » General Chat

Dubz Sep 20 '13

Not sure if this should be posted somewhere else so I'll just post it here.

I'm just throwing out an idea to whomever wants it, but oxwall needs a plugin that supports search crawlers. More specifically, one that would allow a user to define a specific account that the crawler would use to access the site and crawl data not normally seen by guests, especially if it's set to members only.

Just an idea to throw out there to whatever developer wants something new to make.Keep in mind it would have to check in with a list of some sort to get updates regularly on either user-agents or IP addresses (whichever you use).

Leader

dave Sep 22 '13

I would advise against this for several reasons. First of all you have a robots.txt file (or you should) which helps you manage bots.

Secondly if you for example allow google (which you can) to access and build search queries for member only content, then you are infact IMO (in my opinion) doing bait and switch. To me it is very aggrivating to click on a link only to have it tell me that i have to register to see the information. So if it is member only, then IMO keep it that way for users and bots alike.

The Forum post is edited by dave Sep 22 '13

Austeyr Nov 26 '13

Pretty sure this is against googles T's And C's Showing content to Users but showing another For Google at the same link is not good at all. +1 Dave...

Oxwall Software

Please sign in

We build. You grow.

Start a social network, a fan-site, an education project with oxwall - free opensource community software

Spider Crawler plugin | Forum