Quote:
|
Originally Posted by Mr. Stiff
Hi,
Curl is a good program for getting webpages, headers, etc. It's installed on most (good) hosting servers.
Here's how I use it:
- Column 'lastspider' on my gallery table
- Query table, getting URL's not spidered the last xxx days/hours/weeks/whatever
- Use curl extension to connect to URL.
- You can choose only to download headers, which is much faster than downloading the full page
- Check header respons (must be 200). If it's 404 -> page not found, 301 or 302 -> redirect)
- Update your table!
|
You'll want to be a bit more discerning than just ensuring you get a "200 OK" response. If you'd like a script coded up let me know - however most of the better LL scripts already have checkers built into them.