robots txt to prevent crawling of freesites

		Greenguy's Board > Possible Cheaters
robots txt to prevent crawling of freesites

Rate Thread

Display Modes

2006-09-16, 04:02 AM

ponygirl

on vacation

Join Date: Sep 2004

Posts: 2,095

Quote:

Originally Posted by Jel

On the freesites themselves, in the head, is the meta:
meta name="robots" content="index, follow"

that is the thing that bugs me...why have that in the meta unless he forgot to take it out from a cut & paste off something else. Kinda tricky imo.

he's a member of the board, maybe we'll hear something.

__________________
Submit Your Porn
I do CSS coding
Hosting - JMB Scripts - Tube Site Resources

« Previous Thread | Next Thread »

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules

All times are GMT -4. The time now is 01:42 AM.

Mark Read