Expanding Our Support

Users of Advanced Robots.txt Generator know that we are proud of the support we provide as we tend to respond almost immediately to any support questions. As our client base is expanding we wish to extend the support we offer so we are currently researching more advanced support solutions that can handle webbased support and a ticketing system. This aims to make our service more professional, allow you to have a database of your requests to us and allow us to handle your questions in the most optimal and quick way.

We are currently evaluating a variety of solutions and we will reach a decision soon. Stay tuned for our advanced support line!

Advanced Robots.txt Generator

Updated Robots List

Today we publish a new updated robots and crawlers list that can be used with Advanced Robots.txt Generator Professional.

You can download our complete list from http://www.basisoft.com/botslist.html

In order to use the most recent list, download the XML document, and place it in the installation directory of Advanced Robots.txt Generator.

Let us know of any other additions so they can be used in the next update.

robots.txt

Yahoo! Slurp 3.0

Yahoo! announced the launch of a new crawler version Slurp 3.0. According to the Yahoo! Search blog:

Over the past few weeks, we’ve been preparing for the latest version of the Yahoo! Search crawler with some infrastructure updates, which recently caused a variance in our crawl behavior.

With everything now in place, the rollout has officially begun. The new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for ‘Yahoo! Slurp,’ though it’ll identify itself as Slurp 3.0 in your web logs.

As the new software undergoes a phased rollout to our production crawlers over the next several weeks, you’ll see the following changes:

a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it’ll still be from the crawl.yahoo.net domain. Any reverse DNS checks to identify our crawler will continue to work. Please note that if you’re using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.
b) The crawlers will also publish a new user-agent, ‘Yahoo! Slurp/3.0.’ Existing robots.txt directives for ‘Slurp’ or ‘Yahoo! Slurp’ will continue to work, but if you have directives specific to ‘Slurp/2.0,’ they won’t be recognized by the new crawler (though usage of the ‘Slurp/2.0′ user-agent is very rare on the web, so you won’t likely be affected). We recommend specifying the shorter version of: User-agent: Slurp. Check out “How do I prevent my site or certain subdirectories from being crawled?” on our Help page for more details.

These changes will affect the main Yahoo! Web Search crawlers. Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

Naturally Advanced Robots.txt Generator supports this new version without problems or other issues.

Advanced Robots.txt Generator

Bad Robots

You may have noticed that in the basisoft.com webpage there is hardly any notice about ‘bad robots’. The reason is quite intentional since it is hard to place a distinct line between robots which are considered good and those considered bad. And is always the way one man’s good robot is anotherone’s bad robot.

Usually a robot is considered bad if it checks one of the following options:

• It harvests email addresses
• It crawls the site very frequently
• It doesnt follow the robots.txt instructions
• It grabs content from the website for improper use

Although as we said the definition is hard to make Advanced Robots.txt Generator Professional has separated the robots which are generally considered bad at the bottom of the list under the first set of alphabetically listed robots and under the robot ‘Pasiphae’. This list however, should not by any means be considered as complete as a robot may be ‘bad’ specifically for a website so you may wish to ban it.

There are a few ways and techniques you can use to identify bad robots for your website and ban them either using robots.txt or (if it doesn’t follow that) using your .httaccess file. We will talk about these techniques in the future.

Advanced Robots.txt Generator

Video Tutorials

We launch today the first in a series of video tutorials that aim to help users in their first attempts in creating robots.txt files. The first ‘volume’ in this series is a simple video that shows the steps that are necessary for creating a robots.txt file and uploading it to your web server.

You can access the tutorial in the following link:

Advanced Robots.txt Generator Tutorial Vol 1 [2:24]

or you can watch it on:

YouTube

or

Vimeo

.

Advanced Robots.txt Generator

Updated Robots and Crawlers List

Today we publish our updated robots and crawlers list that can be used with Advanced Robots.txt Generator Professional.

You can download our complete list from http://www.basisoft.com/botslist.html

In order to use the most recent list, download the XML document, and place it in the installation directory of Advanced Robots.txt Generator.

Let us know of any other additions so they can be used in the next update.

robots.txt

A Notice to Existing Customers

Customers of previous versions of Advanced Robots.txt Generator that own a version prior to the present 3.2 are entitled to a free upgrade to the present version (in line with the ‘all upgrades are free‘ policy that we keep). As we did not have a Professional / Standard distinction in the product before, they are entitled to a free upgrade to the Professional version. You can download the new version off our download page.

You will need an activation key for this upgrade so do email us at upgrade [at] basisoft [dot] com with your initial purchase details so we can ship you the new key.

Advanced Robots.txt Generator

We are up!

The new version 3.2 is now available on the website in two flavours Standard and Professional. The website is going up as we speak and the plan is to eliminate all teething issues by Monday. Bare with us until then!

Advanced robots.txt Generator

New Version

The new version is nearly ready and should appear on your screens within a week. The plan is to have two versions of the software, a Standard version and a Professional version. There is also the thought of a free version but this will not appear for some time.

Stay tuned for the formal launch!

Hello Robots.txt world!

Welcome to the blog of BasiSoft Software Solutions. We are a software company based in Athens Greece and this is our supporting blog for our award winning best seller software Advanced Robots.txt Generator. Our aim in this blog is to post information about the world of search engine crawlers on frequent basis and inform our customers of changes in the software.