Some people have been asking recently in regards to scripts capable of indexing a website (for use with SEs and the such).
I found this little number today which works really nicely
http://www.hotscripts.com/cgi-bin/dload.cgi?ID=33698
There's no documentation but for the most part there doesn't need to be. More or less just slap the script somewhere on your site and specify your base domain directory and it'll do recursive listing of everything and display it (HTML, images, movies, etc). You can add in extension, directory and filename filters as well if there's certain things you want to keep out of the listings.
It's realtime generated and could be perfect to submit to SEs for so their spiders constantly have an up-to-date list of all HTML on your sites.
If you have any questions about setting it up feel free to ask here and I'll do my best to help you.