Robots.txt

masboy · June 25th, 2004, 7:50 PM

I dont understand, I saw some people were trying to open mydomain.com/robots.txt. What is it ?? Robots.txt ?? something like backdoor or somthing ??

saucemaster · June 25th, 2004, 7:58 PM

robots.txt is a file that search engines use to see how you want your site to be documented. it isn't anything to worry about. in fact, if you don't have one, you might find it useful, as it helps you control traffic to your site.

http://www.robotstxt.org/

masboy · June 25th, 2004, 8:01 PM

Ahh okey, but then why are they sooo curious about it ? I don't have robots.txt tho...hmm good tips. Thanks for the reply !

saucemaster · June 25th, 2004, 8:11 PM

sites like google look for the file automatically, whether you have one or not. the file is useful in telling the search engine whether or not you want the page or the links on the page to show up in people's searches. i believe some search engines have their own commands that they look for in the file. check out robotstxt.org for more information that i can ever provide.

masboy · June 25th, 2004, 8:13 PM

cool ! thanks ! Im reading it right now.

**sam** · June 25th, 2004, 10:37 PM

you want the bots to read robots.txt

thats a well behaved bot.

a bad bot just ignores it.

**kall** · June 26th, 2004, 1:56 AM

and some really bad ones will deliberately jump into the restricted directories.

**sam** · June 26th, 2004, 9:04 AM

bad bot! bad bad bad bot!

:shark:

**bassdaddy777** · June 26th, 2004, 11:21 AM

Quote:

Originally Posted by sam

bad bot! bad bad bad bot!

:shark:

Spanks bad bot with a rolled-up user manual. :lol3:

June 25th, 2004, 7:50 PM	#1 (permalink)
masboy Registered User Seasoned Poster Joined in Jul 2003 33 posts Gave thanks: 0 Thanked 0 times	Robots.txt I dont understand, I saw some people were trying to open mydomain.com/robots.txt. What is it ?? Robots.txt ?? something like backdoor or somthing ??

June 25th, 2004, 7:58 PM	#2 (permalink)
saucemaster Registered User Comfy Contributor Joined in Oct 2003 Lives in midtown atlanta Hosted on viva 174 posts Gave thanks: 0 Thanked 1 Time in 1 Post	robots.txt is a file that search engines use to see how you want your site to be documented. it isn't anything to worry about. in fact, if you don't have one, you might find it useful, as it helps you control traffic to your site. http://www.robotstxt.org/ __________________ pass8: www.sisyphean.com saprus: www.postbaroque.com

June 25th, 2004, 8:11 PM	#4 (permalink)
saucemaster Registered User Comfy Contributor Joined in Oct 2003 Lives in midtown atlanta Hosted on viva 174 posts Gave thanks: 0 Thanked 1 Time in 1 Post	sites like google look for the file automatically, whether you have one or not. the file is useful in telling the search engine whether or not you want the page or the links on the page to show up in people's searches. i believe some search engines have their own commands that they look for in the file. check out robotstxt.org for more information that i can ever provide. __________________ pass8: www.sisyphean.com saprus: www.postbaroque.com

June 26th, 2004, 1:56 AM	#7 (permalink)
kall Surpass Fan Comfy Contributor Joined in May 2004 Lives in Auckland, New Zealand Hosted on Sync 166 posts Gave thanks: 0 Thanked 0 times	and some really bad ones will deliberately jump into the restricted directories. __________________ From NZ? Want Free Messageboards and Games? - NZB From NZ? Want Kiwi Web Hosting and Design? - Sparkle Hosting Sync sync sync...sync sync sync..sync your bootie!

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

June 25th, 2004, 8:01 PM	#3 (permalink)
masboy Registered User Seasoned Poster Joined in Jul 2003 33 posts Gave thanks: 0 Thanked 0 times	Ahh okey, but then why are they sooo curious about it ? I don't have robots.txt tho...hmm good tips. Thanks for the reply !

June 25th, 2004, 8:13 PM	#5 (permalink)
masboy Registered User Seasoned Poster Joined in Jul 2003 33 posts Gave thanks: 0 Thanked 0 times	cool ! thanks ! Im reading it right now.

June 25th, 2004, 10:37 PM	#6 (permalink)
sam Surpass Fan Super #1 Joined in Dec 2003 Lives in NJ 5,057 posts Gave thanks: 0 Thanked 4 times	you want the bots to read robots.txt thats a well behaved bot. a bad bot just ignores it.

June 26th, 2004, 9:04 AM	#8 (permalink)
sam Surpass Fan Super #1 Joined in Dec 2003 Lives in NJ 5,057 posts Gave thanks: 0 Thanked 4 times	bad bot! bad bad bad bot! :shark:

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)