Pulling Free Sites - Might Wanna Check This
 

Just so there's no misuderstandings - I posted this in another thread and there seemed to be a little surprise -

When youre pulling sites and the bots at most of the larger LLs find them 404/302 - the "normal" approach in the past has been to pull everything on that domain - I noticed lately some free site makers that are going 404 on some sites but then submitting before the bot shows up on the same domain - so all of the sites get pulled - its something to keep in mind as I do know some LLs also just pull everything based on submitter (not here but Ive seen others do it) - but more importantly - if you pull some sites on a domain but not all of them - there's a real good chance, unless we knew about specific sites ahead of time, that all sites on that domain got pulled when the bot runs happened last week.

At this time I'm just pulling stuff that redirects or goes no DNS and not all in the domain.

I am thinking that soon we should all go back to business as usually since everyone that pulled stuff do to 2257 concerns should have pulled stuff by then.

What Cleo said

I am giving it a few more days considering I am not completely done myself. I am at this point pulling only what shows as 302/404.

HOWEVER, I have noticed a few people pulling only the galleries and leaving the index / inside pages up. This is complete total nuclear option ban for this. Your misleading me, my bot, and my surfers.

Alex

Yep - thats I guess the point I wanted to make - is that right now its getting back to operations as normal - which in the past has always meant pulling everything on the domain - so if you are still pulling sites - its probably not the best thing to do without at least posting here

Alex - Im getting really frustrated by the number of people doing that - and the worst is the ones that are redirecting to paysites when you click the gallery links - permanent ban here

We, at Debauchery, are sticking to business as usual. We're still accepting submissions (and still behind on our reviews ;) ). We are checking a bit more often right now for redirects, pulled sites, errors and such, but we still give submitters the benefit of the doubt in that we will re-check the next day before we pull the links to sites that are down.

Just a thought - Greenie and others - it would probably be good if we publish a cut-off date after which we will go back to the entire domains/submitter removals - that way everyone can be expecting the same thing - and to get us started - I propose we start this weekend - say Sunday?

Just a more general question about 404s. What happens if my server temporarily goes down for say 24hrs and you bot happens to check at that time. Would you then pull all my sites - or does your bot check back a day later or something?

I do a 3 day cycle on my link checker. You have to be down for 3 consecutive days before I pull your sites.

Sunday works for me.

I'll go along with whatever the rest think is good. :)

neveremail - I think that most pull the sites if they find them 404 - I know I had 30 sites pulled from LOR one time when my server was down for about 8 hours :)

Thats a pain - so you have continuous uptime for a year - drop for a few hours one day and find your sites removed. Thats a bit of a bastard indeed.

No it's not.

When my bot flags a domain down I don't know if it is for a few hours or forever. You can resubmit them and they get on the new list again so it's not all bad.

Starting a new thread on GG&J saying that you are having server problems will usually prevent your stuff from getting pulled.

Thanks for the info

I only check once a week, at most. When I find domains that are down, I wait a day and check'em again before I pull'em. I think that's fair...someone's domain willl have to be down for 2 consecutive days (from when I catch it) before I pull'em. If I know the submitter (or they are a regular submitter), I will try to contact them as well. It'll be end of next week before I check again...

When I pull stuff, the items still stay in the database for a while. It really isn't all that hard to put them back.

Sunday?

Alex

Pretty worthless 'bot, then, I'd say.

Mishi - do you know of a link-check bot that could tell that info? At least without running constantly?

Maybe that would be that bot that also picks the winning lotta numbers each week

I would like a bot like that. It could even predict when links will go down days before the event. If it's predictions were far enough ahead, it could email the link owner and warn them that they are about to have hosting issues.

Is Sparky working in this ESP Bot yet? ;)

It's not terribly difficult. A simple log will tell you whether a link is temporarily or chronically on error. Combine that with a 4x/day cron job and you're golden.

well - thats kinda what I meant by constantly - I think most people - if they saw a bot hitting them on all of their free sites 4x a day - would just add the referrer to their ban list - at least I would :)

My bot runs two times a day and does keep a history but that still doesn't tell me if someone's stuff goes down if it is temporary or forever and I'm still left with surfers that all they know is I have listings that don't work.

If I recognize a name from the boards or something I will often wait a day but many of the people who submit to me I have no idea who they are.

the border line cheater took advantage of this 2257 BS - oh I am removing sites but really I am redirecting to sponsor tours.. BS

plateman - the bigger problem now is going through all of the links manually - seems a few have left up their sites - but changed the inner pages like the galleries - from things like no pics, no recips, to an all out redirect and toolbad downloads - its amazing to see how many people that we trusted as good WMs that have posted here and other places have done this :(