Creating the robots.txt file
There is nothing difficult about creating a basic robots.txt file. It can be created using notepad or whatever is your favorite text editor. Each entry has just two lines:
User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]
This line can be repeated for each directory or file you want to exclude, or for each spider or bot you want to exclude.
A few examples will make it clearer.
1. Exclude a file from an individual Search Engine
You have a file, privatefile.htm, in a directory called ‘private’ that you do not wish to be indexed by Google. You know that the spider that Google sends out is called ‘Googlebot’. You would add these lines to your robots.txt file:
User-Agent: Googlebot
Disallow: /secure/secure.php
2. Exclude a section of your site from all spiders and bots
You are building a new section to your site in a directory called ‘newsection’ and do not wish it to be indexed before you are finished. In this case you do not need to specify each robot that you wish to exclude, you can simply use a wildcard character, ‘*’, to exclude them all.
User-Agent: *
Disallow: /data/
Note that there is a forward slash at the beginning and end of the directory name, indicating that you do not want any files in that directory indexed.
3. Allow all spiders to index everything
Once again you can use the wildcard, ‘*’, to let all spiders know they are welcome. The second, disallow, line you just leave empty, that is your disallow from nowhere.
User-agent: *
Disallow:
4. Allow no spiders to index any part of your site
This requires just a tiny change from the command above – be careful!
User-agent: *
Disallow: /
Thanks for your time 
Incoming search terms:
- curl robots txt (4)
- script robot google (3)
- robots txt tricks (2)
- php feed verse java google spiders (2)
- how to exclude yahoo directory listing using robots (2)
- visichat invisible id (2)
- script to allow spiders in robots file (1)
- php curl robots txt disallow (1)
- robots txt hot to disallow java scripts (1)
- script to use google spider (1)
- symfony robot txt (1)
- why use googlerobots txt (1)
- symfony where to put robots txt (1)
- visichat tricks in mac (1)
- visichat tricks 2012 (1)
- visichat invisible tricks (1)
- visichat invis trick (1)
- visichat hacking methods scripts (1)
- vischat tricks (1)
- tricks for visichat (1)
- tips in visichat (1)
- robots txt file tricks (1)
- robots txt disallow application symfony (1)
- robots txt cookie trick (1)
- can i use wildcard characters with google robots (1)
- creare robots txt script (1)
- creare script robots google (1)
- create a robots file for each country (1)
- create robot txt (1)
- firefox googlerobots (1)
- generate simple robots txt (1)
- google robots and php (1)
- google xml application of orkut redirection script (1)
- how to create web spiders using php (1)
- how to find robots txt file using curl (1)
- new tricks send virus to visichat (1)
- robot exclude login page symfony (1)
- ROBOTS AND THERE CREATIONS (1)
- robots file disallow script (1)
- robots txt change symfony (1)
