Programs: Lock up your servers against Fusker - Page 3

cd34 · 2005-03-28, 04:32 PM

So, with that, we have:

RewriteEngine on
# optionally allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg)$ - [NC,F,L]

That basic rule should work in 99% of the situations. I've regression tested it with all of the test-set that I created earlier. I'll run with this on some real world testing on some clients.

Thank you Raymor for helping us come up with a fairly well protected drop-in rule that we can perhaps slap in a faq somewhere for webmasters to somewhat protect themselves. I know the .wmv is somewhat useless, but, it will stop some of the siterippers that do send referrers.

Then maybe we can get the submission pages to link back to a FAQ page at greenguysboard.com and educate the freesite/tgp submitters a bit.

Tommy · 2005-03-28, 04:37 PM

can you lay that htaccess out like i did (the whole thing)
so us laymen only have to copy and paste

frankthetank · 2005-03-28, 05:05 PM

Quote:

Originally Posted by cd34

So, with that, we have:

RewriteEngine on
# optionally allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg)$ - [NC,F,L]

That basic rule should work in 99% of the situations. I've regression tested it with all of the test-set that I created earlier. I'll run with this on some real world testing on some clients.

Thank you Raymor for helping us come up with a fairly well protected drop-in rule that we can perhaps slap in a faq somewhere for webmasters to somewhat protect themselves. I know the .wmv is somewhat useless, but, it will stop some of the siterippers that do send referrers.

Then maybe we can get the submission pages to link back to a FAQ page at greenguysboard.com and educate the freesite/tgp submitters a bit.

I read the thread carefully and just looked at my stats. So I´m a victim of fusker, too. I tried out cd34 code and it works. My pics are gone from fusker and they can be seen from my domain.

Thank´s all of you...

swedguy · 2005-03-28, 05:18 PM

Quote:

Originally Posted by cd34

RewriteEngine on
# optionally allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg)$ - [NC,F,L]

It's looking good. I would add a line for the IP too.

RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$

or

RewriteCond %{HTTP_REFERER} !^http://(.*@)?(([a-z0-9-]+\.)*yourdomain.com|1\.2\.3\.4)(:[0-9]+)?(/.*)?$ [NC]

I just ran a quick test and the latter is slightly faster, but a lot more messy if people should just copy/paste it and replace domain and IP in it.

Ms Naughty · 2005-03-28, 07:32 PM

OK I'm like Tommy, I really don't understand the code of htaccess...
And I've got a growing collection of different htaccess files saved but now I'm not sure which is the best one to use.

I also found this in my collection, labelled: "Htaccess for site strippers"
So I thought I'd throw it in to further confuse the situation LOL

RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Iria.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Stripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Offline.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Copier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Crawler.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snagger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Reaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Sucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Downloader.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Siphon.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Collector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mag-Net.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DA.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snake.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BackWeb.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*gotit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Vacuum.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ninja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JOC.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Memo.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*leech.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mirror.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Recorder.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*attach.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Magnet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Bandit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ants.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Buddy.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Whacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Drip.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Gets.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*IBrowse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Internet\Ninja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JustView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\tool.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mister\PiX.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Offline\Explorer.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Papa\Foto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SpaceBison.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Image\Collector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Sucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*lftp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FileHound.*$

Chop Smith · 2005-03-28, 08:00 PM

Is someone going to put all of this together for us Htaccess code challenged folks?

amber438 · 2005-03-28, 08:53 PM

Quote:

Originally Posted by cd34

RewriteCond %{HTTP_REFERER} !^$ [NC]

This line..for some reason I removed it from my htaccess files. I can not remember why. What does it do?

I share phpadsnew with all my domain ergo I have to list them all to get the ads to show.
I've also added what grandma scrotum has listed so my htaccess is a mile long..
so..how can we shorten it alos getting the most protection possible?

cd34 · 2005-03-28, 09:48 PM

This rule allows empty referrers:

RewriteCond %{HTTP_REFERER} !^$ [NC]

GrandmaScrotum, phew, that ruleset has been floating around the net for a long time, and, its got many common errors. First, there is a space missing before the P, so, it would never match this. The non-anchored match at the front and the floating match to end of string aren't needed either.

RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]

Would be better as:

RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]

and I sort of changed these by putting:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteRule /* - [L]

In front of your list of rules.

However, I really question whether those rules really do much anymore, since many surfers pretend to be Mozilla anyway.

But, if you were going to do it:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteRule /* - [L]

RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Aculinx [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ants [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^Backstreet [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Burner [OR]
RewriteCond %{HTTP_USER_AGENT} ^Caitoo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Commander [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^Curl [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^Devil [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Down2Web [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^DownloadIt [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eater [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eclipt [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Enterprise [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express [OR]
RewriteCond %{HTTP_USER_AGENT} ^Extractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FairAd\ Client [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlipBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Get [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grab [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^Greed [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Hoover [OR]
RewriteCond %{HTTP_USER_AGENT} ^HtGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^IBrowse [OR]
RewriteCond %{HTTP_USER_AGENT} ^iFox98 [OR]
RewriteCond %{HTTP_USER_AGENT} ^IGJpg$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [OR]
RewriteCond %{HTTP_USER_AGENT} ^IPhoto [OR]
RewriteCond %{HTTP_USER_AGENT} ^iwantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Keepoint [OR]
RewriteCond %{HTTP_USER_AGENT} ^leech [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MemoWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetDrag [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^PerManSurfer [OR]
RewriteCond %{HTTP_USER_AGENT} ^PlantyNet_WebRo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Retriever [OR]
RewriteCond %{HTTP_USER_AGENT} ^SilentSurf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snarf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snatcher [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^wantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Weazel [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web [OR]
RewriteCond %{HTTP_USER_AGENT} ^WFARC [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWCopy [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWoffle
# whatever rule you want here.... block every request with the following rule
RewriteRule .* - [F]

Ms Naughty · 2005-03-28, 10:03 PM

By the way, if that huge "site stripper" htaccess is flawed, please feel free to point that out. As I said, I have no real idea what I'm doing here and I just copied that list from somewhere.
Edit: Woops, looks like I should refresh before I post LOL. Thanks for the info

cd34 · 2005-03-29, 02:10 AM

slight leak in the rule, adjusted with yourdomain\.com (thanks Swedguy)

RewriteEngine on
# leave this line in allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ - [NC,F,L]

In the last hour, its caught 14683 hotlink requests on one client's machine. Mostly fusker/usefulidiot, but, a few others that were surprising. hotmail, msngroups, a bunch of blogs, a few forums. Overall, it seems to be doing well -- negligable impact on the traffic on the machine though.

Altheon · 2005-03-29, 03:11 AM

I just tried to fusker my site and it failed so I know this code works:

Code:

Options +FollowSymlinks
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain.com(/)?.*$     [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp|fla|mov|avi|wmv)$ - [F,NC]

If you all see a problem with it please let me know.

Additionally if you want to protect your directory structure from being viewed pop this little bit of code in there:

Code:

Options -Indexes

Then you will not have to put an index.html in each directory.

-A

Wenchy · 2005-03-29, 05:23 PM

Okay, so can someone answer a couple of questions for me, please?

First, I'm working under the assumption that this code...

RewriteEngine on
# leave this line in allow empty referrers, remove to disallow empty referrers
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(.*@)?1\.2\.3\.4(:[0-9]+)?(/.*)?$
RewriteRule .*\.(asf|mpg|mpeg|wmv|avi|rm|gif|jpeg|jpg|zip)$ - [NC,F,L]

...is the recommended .htaccess for anti-fuskering my domains? I can/should repeat this line...

RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]

... for each of my domains that I want included.

Second, the code that grandmascrotum put up there a couple of posts... I heard somewhere that that huge string can actually put a load on your server each time a page, any page, is loaded. Any truth to that rumor? And, assuming the rumor is false, is there any clear evidence that using that "collection" in my .htaccess actually does any good these days?

Thanks

RawAlex · 2005-03-29, 05:37 PM

Wenchy, I wouldn't put the long list except possibly inside a member's area, or an area that has high quantities of content, or possibly on key personal link areas (where many of your own sites are listed). Otherwise it's a ton of work for very little return, IMHO.

As for the "blocking fusker", the code you listed blocks ALL attempts to reach your images by a third party. If you want to allow things like google or yahoo to index your images, you need to let them in.

The alternate route is to specifically block all access from certain domains. You can do this on a more global level (either at the root level of your webserver, or in the webserver config). This could become a long list, so it really is up to you how you decide to handle it.

Either way, making your sites unfriendly to Fusker is an important move.

Alex
Alex

Wenchy · 2005-03-29, 05:53 PM

Thanks, Alex; sounds like excellent advice.

I'd prefer to allow google, yahoo, etc., and logic dictates I do that by using...

RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain\.com(:[0-9]+)?(/.*)?$ [NC]

... and replacing "yourdomain" with the appropriate SE domain.

Back in the day I was the victim of a hotlinker (japanese, I think) and it cost me a fortune in BW. I'm on a mad quest to prevent a repeat of those events whenever and by what ever means possible. I prefer to only have my nightmares when I'm sleeping |shocking|

Appreciate the assistance! |cool|

raymor · 2005-03-29, 07:24 PM

That long anti-ripper .htaccess is bad/wrong
in at least a couple ways. All of those rules will
impact performance. Even with all of those rules,
though, it's not nearly complete, so it won't block
more than half of the rippers.
The first rule of security is to disallow everything
that isn't specifically allowed.
That .htaccess violates that rule, leading to
the two problems I mentioned.
Rather, it would be better to list the 3 or 4 user agents
that are allowed and disallow everything else.
You'd allow IE, the Gecko browsers (Mozilla, Firefox and Safari are all Gecko and
thus would probably use just one rule),
Opera and perhaps you'd come up with a couple more.
Anything besides IE, Firefox, Safari, Mozilla, and Opera would be redirected.
Of course you may wish to also allow the main SE spiders.

This also has the inherent flaw that you're
assuming one thing based on another thing,
and in fact based on what the user tells you.
The major rippers will let the user set the User-agent
however they want, so just because it
SAYS it's IE doesn't mean that it is.
In fact several rippers are IE based and will therefore report as IE.
On the other hand some people using IE, Mozilla, or Firefox set their user-agent
to something else, such as "None of Your Business Version 0".
But in fact it's not the software name that you're
concerned with, it's a particular BEHAVIOR of the software.
So why not blocked based on that behavior?
That's what Strongbox does. Strongbox blocks
anyone who goes ripping your site, blindly following every single link.
On the other hand it does not block any browser where
the user actually clicks on the links.
THAT is what you really want to block, so that's
what Strongbox looks at, rather than the reported
name of the software.

airdick · 2005-03-29, 10:40 PM

Quote:

Originally Posted by RawAlex

As for the "blocking fusker", the code you listed blocks ALL attempts to reach your images by a third party. If you want to allow things like google or yahoo to index your images, you need to let them in.

Alex

I have always denied hotlinking and never added any special rules to allow google or yahoo, but I have images that are indexed in images.google.com and images.yahoo.com. My thumbnails that appear on those sites are not hotlinked -they are hosted on google & yahoo's servers and each thumb links to the page that that it was taken from, much like a thumbnail tgp.

kalle7 · 2005-03-31, 08:10 AM

Just checked and found some of my stuff in fusker. Updated htaccess file uploaded and no more fusker! Nice

frankthetank · 2005-03-31, 12:46 PM

That´s what you get when you "fusker" the url "fusker.leww.com":

That URL has been banned for illegal content, on the owners request, or because it is a members only page!

maybe the guy now has realized that he is a thief...

jmf000 · 2005-04-01, 10:00 AM

I just wonder would the condition
RewriteCond %{HTTP_REFERER} !^http://([-a-z0-9A-Z]+\.)*yourdomain\.com(/|$|:[0-9])
work faster than
RewriteCond %{HTTP_REFERER} !^http://(([a-z0-9-]+\.)+)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]
since it is unlikely subdomains contain uppercases.
I hope it will provide adequate security.
And by the way as I recall Domains are case sensitive (potentially).

And one more clue : I would not place .htaccess with such antihotlinking technics in the root of domain. Instead I would place it in the subdirectory(ies) under which "real heavy" content resides. So leachers still could see your hosted banners linked to sponsors but not actual images/video.

Thanks for your time.

Verbal · 2005-04-01, 11:41 AM

Quote:

Originally Posted by raymor

Rather, it would be better to list the 3 or 4 user agents that are allowed and disallow everything else.

This is some sound advice right here, instead of maintaining a huge blocked list.

jmf000 · 2005-04-02, 04:08 AM

IMHO it is not important to pay attention to user agents at all.
1)Absolutely everyone who uses "teleporting" soft is able and does impersonate their kind of "browser" as "IE" or alike.
2)Surfers from leeching sources are using usual browser soft like IE or Opera and sequently not differ in this way from others.
3)Many new SEs started today, so you either have to include them all in your allowed browsers list or will loose their traffic.
4)Either black list or allowable list is larger is a kind of question.
I think it is waste of time to pay attention on user browser. Though you can exclude "TELEPORT" and very few other "defaults".

Thanks.

2005-03-28, 04:32 PM	#51
cd34 a.k.a. Sparky Join Date: Sep 2004 Location: West Palm Beach, FL, USA Posts: 2,396	So, with that, we have: RewriteEngine on # optionally allow empty referrers, remove to disallow empty referrers RewriteCond %{HTTP_REFERER} !^$ [NC] RewriteCond %{HTTP_REFERER} !^http://(.@)?([a-z0-9-]+\.)yourdomain.com(:[0-9]+)?(/.)?$ [NC] RewriteRule .\.(asf\|mpg\|mpeg\|wmv\|avi\|rm\|gif\|jpeg\|jpg)$ - [NC,F,L] That basic rule should work in 99% of the situations. I've regression tested it with all of the test-set that I created earlier. I'll run with this on some real world testing on some clients. Thank you Raymor for helping us come up with a fairly well protected drop-in rule that we can perhaps slap in a faq somewhere for webmasters to somewhat protect themselves. I know the .wmv is somewhat useless, but, it will stop some of the siterippers that do send referrers. Then maybe we can get the submission pages to link back to a FAQ page at greenguysboard.com and educate the freesite/tgp submitters a bit. __________________ SnapReplay.com a different way to share photos - iPhone & Android

2005-03-28, 04:37 PM	#52
Tommy NYC Boy That Moved To The Island Join Date: Apr 2003 Posts: 2,940	can you lay that htaccess out like i did (the whole thing) so us laymen only have to copy and paste __________________ Accepting New partners

2005-03-28, 07:32 PM	#55
Ms Naughty old enough to be Grandma Scrotum Join Date: Aug 2003 Location: Australia Posts: 1,408	OK I'm like Tommy, I really don't understand the code of htaccess... And I've got a growing collection of different htaccess files saved but now I'm not sure which is the best one to use. I also found this in my collection, labelled: "Htaccess for site strippers" So I thought I'd throw it in to further confuse the situation LOL RewriteCond %{HTTP_USER_AGENT} ^.WebZIP.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Iria.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Stripper.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Offline.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Copier.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Crawler.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Snagger.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Teleport.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Reaper.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Wget.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Grabber.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Sucker.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Downloader.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Siphon.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Collector.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Mag-Net.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Widow.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Pockey.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.DA.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Snake.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.BackWeb.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.gotit.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Vacuum.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.SmartDownload.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Pump.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.HMView.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Ninja.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.HTTrack.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.JOC.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.likse.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Memo.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.pcBrowser.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.SuperBot.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.leech.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Mirror.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Recorder.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.GrabNet.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Likse.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Navroad.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.attach.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Magnet.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Surfbot.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Bandit.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Ants.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Buddy.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Whacker.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.DISCo\Pump.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Drip.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.EirGrabber.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.ExtractorPro.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.EyeNetIE.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.FlashGet.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.GetRight.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Gets.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Go!Zilla.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Go-Ahead-Got-It.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Grafula.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.IBrowse.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.InterGET.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Internet\Ninja.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.JetCar.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.JustView.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.MIDown\tool.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Mister\PiX.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.NearSite.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.NetSpider.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Offline\Explorer.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.PageGrabber.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Papa\Foto.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Pockey.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.ReGet.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Slurp.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.SpaceBison.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.SuperHTTP.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Teleport.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebAuto.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebCopier.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebFetch.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebReaper.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebSauger.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebStripper.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebWhacker.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.WebZIP.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Web\Image\Collector.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Web\Sucker.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Webster.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.Wget.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.eCatch.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.ia_archiver.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.lftp.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.tAkeOut.$ [OR] RewriteCond %{HTTP_USER_AGENT} ^.FileHound.$ __________________ Promote Bright Desire

2005-03-28, 08:00 PM	#56
Chop Smith Eighteen 'til I Die Join Date: Apr 2003 Location: Mississippi Posts: 2,168	Is someone going to put all of this together for us Htaccess code challenged folks? __________________

2005-03-28, 09:48 PM	#58
cd34 a.k.a. Sparky Join Date: Sep 2004 Location: West Palm Beach, FL, USA Posts: 2,396	This rule allows empty referrers: RewriteCond %{HTTP_REFERER} !^$ [NC] GrandmaScrotum, phew, that ruleset has been floating around the net for a long time, and, its got many common errors. First, there is a space missing before the P, so, it would never match this. The non-anchored match at the front and the floating match to end of string aren't needed either. RewriteCond %{HTTP_USER_AGENT} ^.DISCo\Pump.$ [OR] Would be better as: RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR] and I sort of changed these by putting: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteRule /* - [L] In front of your list of rules. However, I really question whether those rules really do much anymore, since many surfers pretend to be Mozilla anyway. But, if you were going to do it: RewriteEngine on RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteRule /* - [L] RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR] RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR] RewriteCond %{HTTP_USER_AGENT} ^Aculinx [OR] RewriteCond %{HTTP_USER_AGENT} ^Ants [OR] RewriteCond %{HTTP_USER_AGENT} ^attach [OR] RewriteCond %{HTTP_USER_AGENT} ^Backstreet [OR] RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR] RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR] RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR] RewriteCond %{HTTP_USER_AGENT} ^Burner [OR] RewriteCond %{HTTP_USER_AGENT} ^Caitoo [OR] RewriteCond %{HTTP_USER_AGENT} ^Collector [OR] RewriteCond %{HTTP_USER_AGENT} ^Commander [OR] RewriteCond %{HTTP_USER_AGENT} ^Copier [OR] RewriteCond %{HTTP_USER_AGENT} ^Crawler [OR] RewriteCond %{HTTP_USER_AGENT} ^Curl [OR] RewriteCond %{HTTP_USER_AGENT} ^DA [OR] RewriteCond %{HTTP_USER_AGENT} ^Devil [OR] RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR] RewriteCond %{HTTP_USER_AGENT} ^Down2Web [OR] RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR] RewriteCond %{HTTP_USER_AGENT} ^DownloadIt [OR] RewriteCond %{HTTP_USER_AGENT} ^Drip [OR] RewriteCond %{HTTP_USER_AGENT} ^Eater [OR] RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR] RewriteCond %{HTTP_USER_AGENT} ^Eclipt [OR] RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR] RewriteCond %{HTTP_USER_AGENT} ^Enterprise [OR] RewriteCond %{HTTP_USER_AGENT} ^Express [OR] RewriteCond %{HTTP_USER_AGENT} ^Extractor [OR] RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR] RewriteCond %{HTTP_USER_AGENT} ^FairAd\ Client [OR] RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR] RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR] RewriteCond %{HTTP_USER_AGENT} ^FlashSite [OR] RewriteCond %{HTTP_USER_AGENT} ^FlipBrowser [OR] RewriteCond %{HTTP_USER_AGENT} ^Get [OR] RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR] RewriteCond %{HTTP_USER_AGENT} ^gotit [OR] RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR] RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR] RewriteCond %{HTTP_USER_AGENT} ^Grab [OR] RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR] RewriteCond %{HTTP_USER_AGENT} ^Greed [OR] RewriteCond %{HTTP_USER_AGENT} ^HMView [OR] RewriteCond %{HTTP_USER_AGENT} ^Hoover [OR] RewriteCond %{HTTP_USER_AGENT} ^HtGet [OR] RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR] RewriteCond %{HTTP_USER_AGENT} ^IBrowse [OR] RewriteCond %{HTTP_USER_AGENT} ^iFox98 [OR] RewriteCond %{HTTP_USER_AGENT} ^IGJpg$ [OR] RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR] RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [OR] RewriteCond %{HTTP_USER_AGENT} ^IPhoto [OR] RewriteCond %{HTTP_USER_AGENT} ^iwantmy [OR] RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR] RewriteCond %{HTTP_USER_AGENT} ^JOC [OR] RewriteCond %{HTTP_USER_AGENT} ^JustView [OR] RewriteCond %{HTTP_USER_AGENT} ^Keepoint [OR] RewriteCond %{HTTP_USER_AGENT} ^leech [OR] RewriteCond %{HTTP_USER_AGENT} ^lftp [OR] RewriteCond %{HTTP_USER_AGENT} ^likse [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR] RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR] RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR] RewriteCond %{HTTP_USER_AGENT} ^Memo [OR] RewriteCond %{HTTP_USER_AGENT} ^MemoWeb [OR] RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR] RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR] RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR] RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR] RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR] RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR] RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR] RewriteCond %{HTTP_USER_AGENT} ^NetDrag [OR] RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR] RewriteCond %{HTTP_USER_AGENT} ^Offline [OR] RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR] RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR] RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR] RewriteCond %{HTTP_USER_AGENT} ^PerManSurfer [OR] RewriteCond %{HTTP_USER_AGENT} ^PlantyNet_WebRo [OR] RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR] RewriteCond %{HTTP_USER_AGENT} ^Pump [OR] RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR] RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR] RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR] RewriteCond %{HTTP_USER_AGENT} ^Retriever [OR] RewriteCond %{HTTP_USER_AGENT} ^SilentSurf [OR] RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR] RewriteCond %{HTTP_USER_AGENT} ^Snagger [OR] RewriteCond %{HTTP_USER_AGENT} ^Snake [OR] RewriteCond %{HTTP_USER_AGENT} ^Snarf [OR] RewriteCond %{HTTP_USER_AGENT} ^Snatcher [OR] RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR] RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR] RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR] RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR] RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR] RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR] RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR] RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR] RewriteCond %{HTTP_USER_AGENT} ^Vampire [OR] RewriteCond %{HTTP_USER_AGENT} ^wantmy [OR] RewriteCond %{HTTP_USER_AGENT} ^Weazel [OR] RewriteCond %{HTTP_USER_AGENT} ^Web [OR] RewriteCond %{HTTP_USER_AGENT} ^WFARC [OR] RewriteCond %{HTTP_USER_AGENT} ^Wget [OR] RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR] RewriteCond %{HTTP_USER_AGENT} ^Widow [OR] RewriteCond %{HTTP_USER_AGENT} ^WWWCopy [OR] RewriteCond %{HTTP_USER_AGENT} ^WWWoffle # whatever rule you want here.... block every request with the following rule RewriteRule .* - [F] __________________ SnapReplay.com a different way to share photos - iPhone & Android

2005-03-28, 10:03 PM	#59
Ms Naughty old enough to be Grandma Scrotum Join Date: Aug 2003 Location: Australia Posts: 1,408	By the way, if that huge "site stripper" htaccess is flawed, please feel free to point that out. As I said, I have no real idea what I'm doing here and I just copied that list from somewhere. Edit: Woops, looks like I should refresh before I post LOL. Thanks for the info __________________ Promote Bright Desire

2005-03-29, 02:10 AM	#60
cd34 a.k.a. Sparky Join Date: Sep 2004 Location: West Palm Beach, FL, USA Posts: 2,396	slight leak in the rule, adjusted with yourdomain\.com (thanks Swedguy) RewriteEngine on # leave this line in allow empty referrers, remove to disallow empty referrers RewriteCond %{HTTP_REFERER} !^$ [NC] RewriteCond %{HTTP_REFERER} !^http://(.@)?([a-z0-9-]+\.)yourdomain\.com(:[0-9]+)?(/.)?$ [NC] RewriteCond %{HTTP_REFERER} !^http://(.@)?1\.2\.3\.4(:[0-9]+)?(/.)?$ RewriteRule .\.(asf\|mpg\|mpeg\|wmv\|avi\|rm\|gif\|jpeg\|jpg\|zip)$ - [NC,F,L] In the last hour, its caught 14683 hotlink requests on one client's machine. Mostly fusker/usefulidiot, but, a few others that were surprising. hotmail, msngroups, a bunch of blogs, a few forums. Overall, it seems to be doing well -- negligable impact on the traffic on the machine though. __________________ SnapReplay.com a different way to share photos - iPhone & Android

2005-03-29, 03:11 AM	#61
Altheon Heh Heh Heh! Lisa! Vampires are make believe, just like elves and gremlins and eskimos! Join Date: Jan 2004 Posts: 74	I just tried to fusker my site and it failed so I know this code works: Code: Options +FollowSymlinks RewriteEngine on RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain.com(/)?.$ [NC] RewriteRule .\.(gif\|jpg\|jpeg\|bmp\|fla\|mov\|avi\|wmv)$ - [F,NC] If you all see a problem with it please let me know. Additionally if you want to protect your directory structure from being viewed pop this little bit of code in there: Code: Options -Indexes Then you will not have to put an index.html in each directory. -A

2005-03-29, 05:23 PM	#62
Wenchy Trying is the first step towards failure Join Date: Jan 2005 Location: Mile High City Posts: 120	For the .htaccess challenged, a summation if you please... Okay, so can someone answer a couple of questions for me, please? First, I'm working under the assumption that this code... RewriteEngine on # leave this line in allow empty referrers, remove to disallow empty referrers RewriteCond %{HTTP_REFERER} !^$ [NC] RewriteCond %{HTTP_REFERER} !^http://(.@)?([a-z0-9-]+\.)yourdomain\.com(:[0-9]+)?(/.)?$ [NC] RewriteCond %{HTTP_REFERER} !^http://(.@)?1\.2\.3\.4(:[0-9]+)?(/.)?$ RewriteRule .\.(asf\|mpg\|mpeg\|wmv\|avi\|rm\|gif\|jpeg\|jpg\|zip)$ - [NC,F,L] ...is the recommended .htaccess for anti-fuskering my domains? I can/should repeat this line... RewriteCond %{HTTP_REFERER} !^http://(.@)?([a-z0-9-]+\.)yourdomain\.com(:[0-9]+)?(/.)?$ [NC] ... for each of my domains that I want included. Second, the code that grandmascrotum put up there a couple of posts... I heard somewhere that that huge string can actually put a load on your server each time a page, any page, is loaded. Any truth to that rumor? And, assuming the rumor is false, is there any clear evidence that using that "collection" in my .htaccess actually does any good these days? Thanks __________________ I was thinking of the immortal words of Socrates, who said, "I drank what?"*

2005-03-29, 05:37 PM	#63
RawAlex Took the hint. Join Date: Mar 2003 Posts: 5,597	Wenchy, I wouldn't put the long list except possibly inside a member's area, or an area that has high quantities of content, or possibly on key personal link areas (where many of your own sites are listed). Otherwise it's a ton of work for very little return, IMHO. As for the "blocking fusker", the code you listed blocks ALL attempts to reach your images by a third party. If you want to allow things like google or yahoo to index your images, you need to let them in. The alternate route is to specifically block all access from certain domains. You can do this on a more global level (either at the root level of your webserver, or in the webserver config). This could become a long list, so it really is up to you how you decide to handle it. Either way, making your sites unfriendly to Fusker is an important move. Alex Alex

2005-03-29, 05:53 PM	#64
Wenchy Trying is the first step towards failure Join Date: Jan 2005 Location: Mile High City Posts: 120	Thanks, Alex; sounds like excellent advice. I'd prefer to allow google, yahoo, etc., and logic dictates I do that by using... RewriteCond %{HTTP_REFERER} !^http://(.@)?([a-z0-9-]+\.)yourdomain\.com(:[0-9]+)?(/.)?$ [NC] ... and replacing "yourdomain" with the appropriate SE domain. Back in the day I was the victim of a hotlinker (japanese, I think) and it cost me a fortune in BW. I'm on a mad quest to prevent a repeat of those events whenever and by what ever means possible. I prefer to only have my nightmares when I'm sleeping \|shocking\| Appreciate the assistance! \|cool\| __________________ I was thinking of the immortal words of Socrates, who said, "I drank what?"*

2005-03-29, 07:24 PM	#65
raymor The only guys who wear Hawaiian shirts are gay guys and big fat party animals Join Date: Jan 2004 Posts: 178	That long anti-ripper .htaccess is bad/wrong in at least a couple ways. All of those rules will impact performance. Even with all of those rules, though, it's not nearly complete, so it won't block more than half of the rippers. The first rule of security is to disallow everything that isn't specifically allowed. That .htaccess violates that rule, leading to the two problems I mentioned. Rather, it would be better to list the 3 or 4 user agents that are allowed and disallow everything else. You'd allow IE, the Gecko browsers (Mozilla, Firefox and Safari are all Gecko and thus would probably use just one rule), Opera and perhaps you'd come up with a couple more. Anything besides IE, Firefox, Safari, Mozilla, and Opera would be redirected. Of course you may wish to also allow the main SE spiders. This also has the inherent flaw that you're assuming one thing based on another thing, and in fact based on what the user tells you. The major rippers will let the user set the User-agent however they want, so just because it SAYS it's IE doesn't mean that it is. In fact several rippers are IE based and will therefore report as IE. On the other hand some people using IE, Mozilla, or Firefox set their user-agent to something else, such as "None of Your Business Version 0". But in fact it's not the software name that you're concerned with, it's a particular BEHAVIOR of the software. So why not blocked based on that behavior? That's what Strongbox does. Strongbox blocks anyone who goes ripping your site, blindly following every single link. On the other hand it does not block any browser where the user actually clicks on the links. THAT is what you really want to block, so that's what Strongbox looks at, rather than the reported name of the software. __________________ Ray Morris support@bettercgi.com Strongbox/Throttlebox & more TXDPS #A14012

2005-03-31, 08:10 AM	#67
kalle7 Hey, can you take the wheel for a second, I have to scratch my self in two places at once Join Date: Jan 2004 Location: Here and there... Posts: 182	Just checked and found some of my stuff in fusker. Updated htaccess file uploaded and no more fusker! Nice

2005-03-31, 12:46 PM	#68
frankthetank Stupid risks make life worth living Join Date: Jan 2005 Location: Renesse NL Posts: 386	That´s what you get when you "fusker" the url "fusker.leww.com": That URL has been banned for illegal content, on the owners request, or because it is a members only page! maybe the guy now has realized that he is a thief...

2005-04-01, 10:00 AM	#69
jmf000 WHO IS FONZY!?! Don't they teach you anything at school? Join Date: Sep 2003 Posts: 43	I just wonder would the condition *RewriteCond %{HTTP_REFERER} !^http://([-a-z0-9A-Z]+\.)yourdomain\.com(/\|$\|:[0-9]) work faster than RewriteCond %{HTTP_REFERER} !^http://(([a-z0-9-]+\.)+)?yourdomain.com(:[0-9]+)?(/.)?$ [NC]* since it is unlikely subdomains contain uppercases. I hope it will provide adequate security. And by the way as I recall Domains are case sensitive (potentially). And one more clue : I would not place .htaccess with such antihotlinking technics in the root of domain. Instead I would place it in the subdirectory(ies) under which "real heavy" content resides. So leachers still could see your hosted banners linked to sponsors but not actual images/video. Thanks for your time.

2005-04-02, 04:08 AM	#71
jmf000 WHO IS FONZY!?! Don't they teach you anything at school? Join Date: Sep 2003 Posts: 43	IMHO it is not important to pay attention to user agents at all. 1)Absolutely everyone who uses "teleporting" soft is able and does impersonate their kind of "browser" as "IE" or alike. 2)Surfers from leeching sources are using usual browser soft like IE or Opera and sequently not differ in this way from others. 3)Many new SEs started today, so you either have to include them all in your allowed browsers list or will loose their traffic. 4)Either black list or allowable list is larger is a kind of question. I think it is waste of time to pay attention on user browser. Though you can exclude "TELEPORT" and very few other "defaults". Thanks.