If you could build your own link list, what would you want?

[Greenguy]

Originally Posted by Verbal

Well, why can't this be done? Have the bot spider each page, then add up the total size of the entire freesite and add up the total number of links for the freesite. Store these values to check against when the bot comes back later.

Re-read my entire post my friend

[Verbal]

Originally Posted by Greenguy

Re-read my entire post my friend

Ah, I just re-read it and noticed I've totally overlooked your point (and a good one). So I have to agree with you, the freesite link-checker would certainly be a bitch to code without some sort of crazy limits on where it goes and stores. I would sure hate to code this thing.

Now I'm interested in how Swedguy has implemented this so that it works so well.

[swedguy]

Originally Posted by Verbal

Now I'm interested in how Swedguy has implemented this so that it works so well.

It's not that hard actually.

Go to the index page and fetch all links, keep all links that are 1 dir deep (works on 99% of all sites). I'll fine tune that part when I get some time over. It should be dynamic depending on how many dirs deep the original site is.

Now we have a bunch of links that the script thinks are main page links. Fetch all links on the "main" page and keep the ones that are 1 dir deep. Go to the links and see if it has links to images, if it doesn't, discard it, it's not a gallery. If it does, keep it.

Now we have an index, main, gallery and images trail that we can do whatever we want to do with

There are some other filters. But that's the basics of it.