robots.txt

[Maljonic]

robots.text

I'm sure this is a very old question but what exactly do I do with robots text? Do I need to do anything at all? Can I just not bother with it?

 

[sibanac]

I'm sure this is a very old question but what exactly do I do with robots text? Do I need to do anything at all? Can I just not bother with it?

It is used to tell webcrawlers (indexing from search engines) what they can and cant index from your page.
this can be a good option for /cgi-bin or other dynamic fast changing pages.


look here for more info

 

[Jonty]

Hi Maljonic

Sibanac is spot on in that robots.txt effectively helps search engines decide which pages they should crawl (protecting private directories is also a major use of this technology).

More recently, you can also use a meta tag to save you have to worry about creating robots.txt files; e.g.

<meta content="all" name="robots" />

If you search Google or the likes for this method you'll find all the information you need. Note, however, that not all search engines support this meta robots format (although most major operators do I believe) so the traditional robots.txt method may prove to be the most effective.

Kind Regards

 

[Maljonic]

Thanks Jonty, I don't have any private directories as yet and I am using the meta tag you mentioned. And sibanac, I just read that stuff before I posted this; I'm just wondering if it's okay not to have robots text? Or if it is better to have it, how do I create it and where do I put it?

 

[sibanac]

Its no problem to not have one.
If you create one put it in your webpage root (eg : http://your.domain/robots.txt)

 

[Maljonic]

do I write it as a web page then or is it just a text file, like you might write in notepad or something?

 

[sibanac]

do I write it as a web page then or is it just a text file, like you might write in notepad or something?

just a notepad thing will do

# My robot.txt file
# <- means comment 
# this robot.txt file allows all robots to index the site except /images/* and /cgi-bin/*

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

 

[Mike]

Check out http://www.robotstxt.org for info on robots.txt, how it works, etc. It doesn't look particularly appealing, but all the info you need is there.

Be aware that the file has to be readable, by anyone. So by adding a private dir to it, you are effectively telling everyone the url of that private dir Therefore you should password such dirs, if you add them to robots.txt