You may have noticed that in the basisoft.com webpage there is hardly any notice about ‘bad robots’. The reason is quite intentional since it is hard to place a distinct line between robots which are considered good and those considered bad. And is always the way one man’s good robot is anotherone’s bad robot.
Usually a robot is considered bad if it checks one of the following options:
- It harvests email addresses
- It crawls the site very frequently
- It doesnt follow the robots.txt instructions
- It grabs content from the website for improper use
Although as we said the definition is hard to make Advanced Robots.txt Generator Professional has separated the robots which are generally considered bad at the bottom of the list under the first set of alphabetically listed robots and under the robot ‘Pasiphae’. This list however, should not by any means be considered as complete as a robot may be ‘bad’ specifically for a website so you may wish to ban it.
There are a few ways and techniques you can use to identify bad robots for your website and ban them either using robots.txt or (if it doesn’t follow that) using your .httaccess file. We will talk about these techniques in the future.
Advanced Robots.txt Generator
