me, but not by choice. |gabbo| renamed me and a general vote assigned the name. I'm in denial and protest.

You might try benchmarking the regexp engine & the fallthrough matches sometime. You may be pleasantly surprised. I do agree with you regarding the trailing match, but, I disagree with your assessment that the regexp engine is a high performance solution that cannot be aided.

I do benchmark my code, including my work on mod_rewrite,
before releasing it publically. Perhaps you should benchmark it
before you presume to tell me how my own software works.
Indeed, simply parsing the extra line in the .htaccess uses more
cycles before it even compiles the regex and starts looking for a match.

The directives are read from .htaccess line by line, which means
that each character is checked to see if it's a newline.
It takes an extra 50 character comparisons just to read the line,
even before it's recognized as a condititon at all.
On the other hand the regex engine must make only 2 comparisons
to see whether the next character is a "w" or a "y".
Indeed once your extra regex is compiled it's on the order of 2,000 times slower.

http://jakarta.apache.org/regexp/applet.html

A nice little tool if you want to test your regexps.

|popcorn|
When coders disagree…
Seems that there are a lot of ways to do htaccess and the people that really understand this stuff don't even agree on what works best. Personally I'm just going to keep on doing it my way since it seems to work for me.

Back to fucking Fusker and site rippers…
So besides having hotlinking protection in place using whatever flavor of htaccess you feel best with what else can be done?
One Problem seems to be the no referral line that we have in our htaccess.

Site rippers are another big problem with paysite member's areas, at least it was with the one that I manage.
Besides putting in Strongbox what else can we do to block site rippers?

you are correct, your method is faster.

with 20 concurrent threads, 100k requests, no keepalives, .46 seconds faster and 3.4 tps faster.
oddly, at 20/10k/no keepalives, the margin tilts the opposite direction and is just as slim.

Although, I tested Apache2/mpm-prefork, not Apache1. When I benchmarked Apache1.3 last spring, there was a considerable difference edging towards having separate lines. However, with the config file in the virtual host rather than being read from .htaccess, and turning Override's off, it narrows down to .16 seconds faster and less than 1.1 tps faster. Granted this was just a seat-of-the-pants quick benchmark using ab, with all 5 tests run 5 times and the lowest score taken. Each used a referrer that required every rule to be evaluated.

So, current wisdom would suggest this:

Slightly corrected to remove two typos:

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9]+\.)?yourdomain.com(/.*)?$ [NC]

this disallows RFC compliant hostnames that have a - in them, fixing the regexp to allow that makes it a bit more complex since the leading character cannot be a -, but, lets skip that for now and allow - anywhere in the hostname and allow for the port designation which some edge proxy servers like to insert. We might also consider allowing _ which was not in the original specs, but, Microsoft supported it, and it was later adopted. This rule also doesn't allow for the new International characters, but, if someone is using those, they can change the regexp.

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

This disallows subdomain.subdomain.yourdomain.com, which might be part of the + typo in the earlier post.

RewriteCond %{HTTP_REFERER} !^http://(([a-z0-9-]+\.)+)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

So, Raymor, does the above rule seem sufficient for as many cases as possible?

Offhand, I can think of one really simple way to circumvent this -- Firefox pops a warning, but IE seems to have no problem with it.

I've run that rule against a regression test to make sure that it appears to behave exactly the way I would expect and intend it to work and short of the one possible regexp match which could be very easily implemented on fuskers side, it seems fairly complete.

Now, of course we're back to the age old problem of Google/Yahoo's hotlinking image search. By doing the rules this way, we are only allowing the listed domains to be able to hotlink. If the rule is sufficiently loose to allow google/yahoo, a slight url change on fusker's side will make all of this for naught.

slide rulers at twenty paces? |couch|

One additional atom I included in my original 1999 regex
but I removed along with the port number was http(s)?
in case
some part of the site used SSL either presently or in the future.
That would cover an additional case, whether or not it's worth including
is another matter.
Additionally I unless you're running a web server on some port other
than port 80 you may want to just say 80 for the port number.
Makes it just a tad faster. Of course that analagous to my subdomain
part, where I chose to allow more than just www.
Another judgement call speed / versus flexibility.

I was thinking that we had taken care of user:pass@host URLs,
but I'm not seeing exactly how just at this moment.
Perhaps the tested browsers wouldn't allow blah.com/
in the authentication part, but treated it as a host/path when
it saw the forward slash (/).
Unless I'm missing something, we actually want to gobble up
anything up to and including @ before going any further:
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

(([a-z0-9-]+\.)+)? ? :) Maybe ([a-z0-9-]+\.)* ?

For those who don't catch the difference,
I had a typo where I said "optionally with a subdomain"
rather than "optionally with subdomains".
His fix for my typo was to say:
(([a-z0-9-]+\.)+)?
ergo "one or more subdomains, repeated one or zero times"
Simpler, we say "with or without subdomains":
([a-z0-9-]+\.)*

BTW, regarding the hyphen and the multiple subdomains guess
I should have pasted from my own old page from years ago and
I wouldn't have forgotten those:

http://216.239.63.104/search?q=cache...htaccess&hl=en

Okay, now that the geeks have had their moment in the sun (hope they don't burn), let's get back to the simple deal:

Program owners that run freehosted galleries: You need to lock up your content.

It's good business, and it is respectful of your affiliates.

Alex

True IMO - There must/should be a united and concerted response to stoping these thieves.

Yet another unproven unsupported theory proposed by those that don't know or are too lazy to care..

I notified one prominent program that their stolen images were being used on a message board just 2 days ago.. and their response(ignorant and ill-informed resonse - IMO) was..

"As our URL's are on the images, and it does not appear to be an entire set, I am inclined to let it go on this instance."


In my mind this makes them part of the problem, and encourages the theft of their content. In fact it is essentially giving me(and others) implied permission to use their content without permission.


Stopping these thieves making money is the only answer.. and AVN/Adbrite hold the key to stopping them. Lack of action on their part will cost YOU money.

DD

So, with that, we have:

RewriteEngine on
# optionally allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg)$ - [NC,F,L]

That basic rule should work in 99% of the situations. I've regression tested it with all of the test-set that I created earlier. I'll run with this on some real world testing on some clients.

Thank you Raymor for helping us come up with a fairly well protected drop-in rule that we can perhaps slap in a faq somewhere for webmasters to somewhat protect themselves. I know the .wmv is somewhat useless, but, it will stop some of the siterippers that do send referrers.

Then maybe we can get the submission pages to link back to a FAQ page at greenguysboard.com and educate the freesite/tgp submitters a bit.

can you lay that htaccess out like i did (the whole thing)
so us laymen only have to copy and paste

I read the thread carefully and just looked at my stats. So I´m a victim of fusker, too. I tried out cd34 code and it works. My pics are gone from fusker and they can be seen from my domain.

Thank´s all of you... |thumb

It's looking good. I would add a line for the IP too.

RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$

or

RewriteCond %{HTTP_REFERER} !^http://(.*@)?(([a-z0-9-]+\.)*yourdomain.com|1\.2\.3\.4)(:[0-9]+)?(/.*)?$ [NC]

I just ran a quick test and the latter is slightly faster, but a lot more messy if people should just copy/paste it and replace domain and IP in it.

OK I'm like Tommy, I really don't understand the code of htaccess...
And I've got a growing collection of different htaccess files saved but now I'm not sure which is the best one to use.

I also found this in my collection, labelled: "Htaccess for site strippers"
So I thought I'd throw it in to further confuse the situation LOL

RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Iria.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Stripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Offline.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Copier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Crawler.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snagger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Reaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Sucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Downloader.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Siphon.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Collector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mag-Net.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DA.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snake.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BackWeb.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*gotit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Vacuum.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ninja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JOC.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Memo.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*leech.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mirror.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Recorder.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*attach.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Magnet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Bandit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ants.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Buddy.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Whacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Drip.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Gets.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*IBrowse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Internet\Ninja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JustView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\tool.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mister\PiX.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Offline\Explorer.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Papa\Foto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SpaceBison.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Image\Collector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Sucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*lftp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FileHound.*$

Is someone going to put all of this together for us Htaccess code challenged folks?

This line..for some reason I removed it from my htaccess files. I can not remember why. What does it do?

I share phpadsnew with all my domain ergo I have to list them all to get the ads to show.
I've also added what grandma scrotum has listed so my htaccess is a mile long..
so..how can we shorten it alos getting the most protection possible?

This rule allows empty referrers:

RewriteCond %{HTTP_REFERER} !^$ [NC]

GrandmaScrotum, phew, that ruleset has been floating around the net for a long time, and, its got many common errors. First, there is a space missing before the P, so, it would never match this. The non-anchored match at the front and the floating match to end of string aren't needed either.

RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]

Would be better as:

RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]

and I sort of changed these by putting:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteRule /* - [L]

In front of your list of rules.

However, I really question whether those rules really do much anymore, since many surfers pretend to be Mozilla anyway.

But, if you were going to do it:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteRule /* - [L]

RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Aculinx [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ants [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^Backstreet [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Burner [OR]
RewriteCond %{HTTP_USER_AGENT} ^Caitoo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Commander [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^Curl [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^Devil [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Down2Web [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^DownloadIt [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eater [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eclipt [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Enterprise [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express [OR]
RewriteCond %{HTTP_USER_AGENT} ^Extractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FairAd\ Client [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlipBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Get [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grab [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^Greed [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Hoover [OR]
RewriteCond %{HTTP_USER_AGENT} ^HtGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^IBrowse [OR]
RewriteCond %{HTTP_USER_AGENT} ^iFox98 [OR]
RewriteCond %{HTTP_USER_AGENT} ^IGJpg$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [OR]
RewriteCond %{HTTP_USER_AGENT} ^IPhoto [OR]
RewriteCond %{HTTP_USER_AGENT} ^iwantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Keepoint [OR]
RewriteCond %{HTTP_USER_AGENT} ^leech [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MemoWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetDrag [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^PerManSurfer [OR]
RewriteCond %{HTTP_USER_AGENT} ^PlantyNet_WebRo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Retriever [OR]
RewriteCond %{HTTP_USER_AGENT} ^SilentSurf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snarf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snatcher [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^wantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Weazel [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web [OR]
RewriteCond %{HTTP_USER_AGENT} ^WFARC [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWCopy [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWoffle
# whatever rule you want here.... block every request with the following rule
RewriteRule .* - [F]

By the way, if that huge "site stripper" htaccess is flawed, please feel free to point that out. As I said, I have no real idea what I'm doing here and I just copied that list from somewhere.
Edit: Woops, looks like I should refresh before I post LOL. Thanks for the info :D

slight leak in the rule, adjusted with yourdomain\.com (thanks Swedguy)

RewriteEngine on
# leave this line in allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ - [NC,F,L]

In the last hour, its caught 14683 hotlink requests on one client's machine. Mostly fusker/usefulidiot, but, a few others that were surprising. hotmail, msngroups, a bunch of blogs, a few forums. Overall, it seems to be doing well -- negligable impact on the traffic on the machine though.

I just tried to fusker my site and it failed so I know this code works:

If you all see a problem with it please let me know.

Additionally if you want to protect your directory structure from being viewed pop this little bit of code in there:

Then you will not have to put an index.html in each directory.

-A

For the .htaccess challenged, a summation if you please...
 

Okay, so can someone answer a couple of questions for me, please?

First, I'm working under the assumption that this code...

RewriteEngine on
# leave this line in allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ - [NC,F,L]

...is the recommended .htaccess for anti-fuskering my domains? I can/should repeat this line...

RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]

... for each of my domains that I want included.

Second, the code that grandmascrotum put up there a couple of posts... I heard somewhere that that huge string can actually put a load on your server each time a page, any page, is loaded. Any truth to that rumor? And, assuming the rumor is false, is there any clear evidence that using that "collection" in my .htaccess actually does any good these days?

Thanks |thumb

Wenchy, I wouldn't put the long list except possibly inside a member's area, or an area that has high quantities of content, or possibly on key personal link areas (where many of your own sites are listed). Otherwise it's a ton of work for very little return, IMHO.

As for the "blocking fusker", the code you listed blocks ALL attempts to reach your images by a third party. If you want to allow things like google or yahoo to index your images, you need to let them in.

The alternate route is to specifically block all access from certain domains. You can do this on a more global level (either at the root level of your webserver, or in the webserver config). This could become a long list, so it really is up to you how you decide to handle it.

Either way, making your sites unfriendly to Fusker is an important move.

Alex
Alex

Thanks, Alex; sounds like excellent advice.

I'd prefer to allow google, yahoo, etc., and logic dictates I do that by using...

RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]

... and replacing "yourdomain" with the appropriate SE domain.

Back in the day I was the victim of a hotlinker (japanese, I think) and it cost me a fortune in BW. I'm on a mad quest to prevent a repeat of those events whenever and by what ever means possible. I prefer to only have my nightmares when I'm sleeping |shocking|

Appreciate the assistance! |cool|

That long anti-ripper .htaccess is bad/wrong
in at least a couple ways. All of those rules will
impact performance. Even with all of those rules,
though, it's not nearly complete, so it won't block
more than half of the rippers.
The first rule of security is to disallow everything
that isn't specifically allowed.
That .htaccess violates that rule, leading to
the two problems I mentioned.
Rather, it would be better to list the 3 or 4 user agents
that are allowed and disallow everything else.
You'd allow IE, the Gecko browsers (Mozilla, Firefox and Safari are all Gecko and
thus would probably use just one rule),
Opera and perhaps you'd come up with a couple more.
Anything besides IE, Firefox, Safari, Mozilla, and Opera would be redirected.
Of course you may wish to also allow the main SE spiders.

This also has the inherent flaw that you're
assuming one thing based on another thing,
and in fact based on what the user tells you.
The major rippers will let the user set the User-agent
however they want, so just because it
SAYS it's IE doesn't mean that it is.
In fact several rippers are IE based and will therefore report as IE.
On the other hand some people using IE, Mozilla, or Firefox set their user-agent
to something else, such as "None of Your Business Version 0".
But in fact it's not the software name that you're
concerned with, it's a particular BEHAVIOR of the software.
So why not blocked based on that behavior?
That's what Strongbox does. Strongbox blocks
anyone who goes ripping your site, blindly following every single link.
On the other hand it does not block any browser where
the user actually clicks on the links.
THAT is what you really want to block, so that's
what Strongbox looks at, rather than the reported
name of the software.

I have always denied hotlinking and never added any special rules to allow google or yahoo, but I have images that are indexed in images.google.com and images.yahoo.com. My thumbnails that appear on those sites are not hotlinked -they are hosted on google & yahoo's servers and each thumb links to the page that that it was taken from, much like a thumbnail tgp.

Just checked and found some of my stuff in fusker. Updated htaccess file uploaded and no more fusker! Nice |thumb

That´s what you get when you "fusker" the url "fusker.leww.com":

That URL has been banned for illegal content, on the owners request, or because it is a members only page!

maybe the guy now has realized that he is a thief... :D :D :D

I just wonder would the condition
RewriteCond %{HTTP_REFERER} !^http://([-a-z0-9A-Z]+\.)*yourdomain\.com(/|$|:[0-9])
work faster than
RewriteCond %{HTTP_REFERER} !^http://(([a-z0-9-]+\.)+)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
since it is unlikely subdomains contain uppercases.
I hope it will provide adequate security.
And by the way as I recall Domains are case sensitive (potentially).


And one more clue : I would not place .htaccess with such antihotlinking technics in the root of domain. Instead I would place it in the subdirectory(ies) under which "real heavy" content resides. So leachers still could see your hosted banners linked to sponsors but not actual images/video.

Thanks for your time.

This is some sound advice right here, instead of maintaining a huge blocked list. |thumb

IMHO it is not important to pay attention to user agents at all.
1)Absolutely everyone who uses "teleporting" soft is able and does impersonate their kind of "browser" as "IE" or alike.
2)Surfers from leeching sources are using usual browser soft like IE or Opera and sequently not differ in this way from others.
3)Many new SEs started today, so you either have to include them all in your allowed browsers list or will loose their traffic.
4)Either black list or allowable list is larger is a kind of question.
I think it is waste of time to pay attention on user browser. Though you can exclude "TELEPORT" and very few other "defaults".

Thanks.