robots txt to prevent crawling of freesites

[virgohippy]

Originally Posted by ponygirl

yes me too. I wouldn't consider it cheating, really - it's not breaking any rules, but it's misleading.

If someone's going to take the extra time and energy to stuff their robots.txt file in an attempt to get more back then they're giving I would consider that following the rules, but still acting with malicious intent.

Malicious intent falls under the unspoken rule of, "I don't like your business practices, therefore, I don't want to do business with you."

In this situation, I probably wouldn't send a rejection email, or even ask what's up. I'd just silently make their sites dissappear with a quick click of the delete button.

Jel, thanks for bringing this issue up. As if I didn't already have enough to check for...