Programs: Lock up your servers against Fusker

[Tommy]

my stuff always shows up as broken images on all those fucker sites

I feel very safe

the only annoying thing is that the stats counts it as a reffer so my server stats are filled up with that junk

[swedguy]

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com$
RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com/

That would make it safe.

#1 - http://www.yourdomain.com
#2 - http://www.yourdomain.com/**anything**

[Cleo]

This is what I've found that works.
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com:80/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com:80.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80.*$ [NC]

[airdick]

Has anyone in analyzed their logs to check the % of non-referrer image requests to fuskered objects vs. non-fuskered images?

I suppose that if I see a real impact from fusker I could use the same software I use for protecting my wmv files to insure that only surfers that have hit some html on the my domain in the last X minutes can get at my images, but I haven't tried it yet for fear of causing problems for surfers, reviewers and tgp scripts that might be using multiple proxies. Are any gallery/freesite builders using the antithotlinking.com product to protect images as well as movies?

Regarding paysites that don't mind fusker traffic: I suspect that if sites can make money giving away watermarked images on Usenet that they can also make some money from watermarked images on fusker, although unlike Usenet the paysite would end up footing the bill for the bandwidth.

[cd34]

and would block requests from certain edge proxy servers (ask Cleo)

http://www.yourdomain.com:80/blah

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com(\:[0-9]+)?/

I think.

[RawAlex]

You can also go through the process at a higher level and block off all access by people being refered by fusker and such, which is another powerful way to stop the most blatant site rippers. If you do get fuskered and your listing comes up as broken images on one of their sites, a surfer will click "images are broken" and your site will be removed.

Fusker appears to use XML or RSS to share a feed of all of the fuskered sites amoungst it's site system. Get yourself removed from one, and you will disappear from all of them in a short time.

All programs should look very closely at their servers and assure that they are secure. You know, 90% of the bank card phishing would be gone if the banks did a better job of protecting their sites and checking incoming referals.

Alex

[swedguy]

Sometimes IE doesn't want to add the trailing slash, so the referer will be http://yourdomain.com instead of http://yourdomain.com/

This one will take care of that and everything else....

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com(:[0-9]+)?($|/)+ [NC]
RewriteCond %{HTTP_REFERER} !^http://1.2.3.4(:[0-9]+)?($|/)+
RewriteRule \.(jpg|gif)$ http://www.google.com/ [NC,R,L]

[raymor]

Originally Posted by Cleo

In Tommy's example you can hotlink by putting the page in a directory called "yourdomain.com"

No, you can't. Try it. It allows only that which matches
the regex, which says:
^http:// First, http://

([a-z0-9-]+\.)* then allow letters, numbers, and dots (but not slashes or anything else)

yourdomain.com then your domain name.

http://hacker.com/yourdomain.com would not match the
second atom of the regex, which allows only letters, numbers,
and dots, but not the slash that would have to come between
hacker.com and yoursite.com.

[swedguy]

Raymor, the one sparky posted will go through.

[raymor]

Originally Posted by swedguy

Raymor, the one sparky posted will go through.

Who is sparky? The one bad one I see is what Cleo posted,
which includes a bunch of meaningless BS that does nothing.
For example, look at thr last couple of atoms of this line:

RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]

The last bit says "anything, then the end of the string".
Well that's pointless, if anything and everything is allowed
all the way to the end all that crap should be left off.
Also that ruleset is quite repetitive, making it terribly inefficient.
Instead the rules should be combined. For example, these two:
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]

One allows it with a "www" "subdomain", the other allow it without.
They should be combined into one rule that allows it with or without:
RewriteCond %{HTTP_REFERER} !^http://(www\.)+yourdomain.com/.*$ [NC]

As mentioned before, other subdomain besides www are also possible,
so rather than just "www." we allow letters, numbers, and dots:
RewriteCond %{HTTP_REFERER} !^http://([a-z0-9]\.)+yourdomain.com/.*$ [NC]

As mentioned above, the "allow anything at all after the domain name"
part is pointless, as it matches anywhere in the target string,
so we get rid of that:
RewriteCond %{HTTP_REFERER} !^http://([a-z0-9]\.)+yourdomain.com/ [NC]
The only thing left is that there may or may not be a slash,
and only if there is a slash can you have anything else.
This is to avoid allowing http://yourdomain.com.hacker.com
or:
http://yourdomain.comedyhack.com
"Allow only if" requires a bit that looks a little complex:
RewriteCond %{HTTP_REFERER} !^http://([a-z0-9]\.)+yourdomain.com(/.*)?$ [NC]

BTW, when deciding who to listen to on this stuff, whether to
follow the advice of someone who "found something" which "seems to work"
or of someone ellse who seems to actually know what this stuff means,
take a look at the Contributors file for mod_rewrite and see which
of the people posting in this thread helped write the part of Apache that we're dealing with.

[Cleo]

Sparky= cd34

I don't really understand much of this. I just know what has worked for me over the years through much trial and error.

I removed all wildcards from my stuff after getting hit by hotlinjers either using folders with my domain in them or subdomains.

I stated putting these 3 variations in after I found that it was blocking images sometimes when I didn't want it to.
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com:80.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80.*$ [NC]

As with anything much testing is always good.

[swedguy]

Raymor,

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)*yourdomain.com [NC]

There's no trailing slash or checks if Referer is only http://yourdomain.com = you can make yourdomain.com to a sub-domain (http://yourdomain.com.fusker.com).

[raymor]

Originally Posted by Cleo

Sparky= cd34
I stated putting these 3 variations in after I found that it was blocking images sometimes when I didn't want it to.
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com:80.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com:80.*$ [NC]

I used to include the port number, the :80 in my suggested code
years ago. My business partner thought that it wasn't necesary,
so we logged several hundred thousand requests and didn't find
any with the port number, so I took it out. I would agree, though,
that in theory it should be allowed.

[Cleo]

Back in the nineties I did Mac computer consulting and while I was out using clients computers I would try surfing my stuff mostly on Macs but sometimes Windows machines. The port number got put in there after seeing my images blocked if I remember correctly by Macs behind proxy servers, same thing with the trailing slash.

[RawAlex]

remember also you can do http://www.fuskerdomain.com/yourdoma...nthispage.html and that will get through if you are not looking SPECIFICALLY for the whole http string.

As I said, you can also take steps to block all fusker access from their main sites. If they can't access you from their main sites, they are VERY likely to not list you or have your listing pulled very quickly.

Alex

[cd34]

Originally Posted by raymor

Who is sparky?

me, but not by choice. |gabbo| renamed me and a general vote assigned the name. I'm in denial and protest.

Originally Posted by raymor

Also that ruleset is quite repetitive, making it terribly inefficient.
Instead the rules should be combined. For example, these two:
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]

You might try benchmarking the regexp engine & the fallthrough matches sometime. You may be pleasantly surprised. I do agree with you regarding the trailing match, but, I disagree with your assessment that the regexp engine is a high performance solution that cannot be aided.

[raymor]

Originally Posted by cd34

me, but not by choice. |gabbo| renamed me and a general vote assigned the name. I'm in denial and protest.



You might try benchmarking the regexp engine & the fallthrough matches sometime. You may be pleasantly surprised. I do agree with you regarding the trailing match, but, I disagree with your assessment that the regexp engine is a high performance solution that cannot be aided.

I do benchmark my code, including my work on mod_rewrite,
before releasing it publically. Perhaps you should benchmark it
before you presume to tell me how my own software works.
Indeed, simply parsing the extra line in the .htaccess uses more
cycles before it even compiles the regex and starts looking for a match.

The directives are read from .htaccess line by line, which means
that each character is checked to see if it's a newline.
It takes an extra 50 character comparisons just to read the line,
even before it's recognized as a condititon at all.
On the other hand the regex engine must make only 2 comparisons
to see whether the next character is a "w" or a "y".
Indeed once your extra regex is compiled it's on the order of 2,000 times slower.

[swedguy]

http://jakarta.apache.org/regexp/applet.html

A nice little tool if you want to test your regexps.

[Cleo]

When coders disagree…
Seems that there are a lot of ways to do htaccess and the people that really understand this stuff don't even agree on what works best. Personally I'm just going to keep on doing it my way since it seems to work for me.

Back to fucking Fusker and site rippers…
So besides having hotlinking protection in place using whatever flavor of htaccess you feel best with what else can be done?
One Problem seems to be the no referral line that we have in our htaccess.

Site rippers are another big problem with paysite member's areas, at least it was with the one that I manage.
Besides putting in Strongbox what else can we do to block site rippers?

[cd34]

you are correct, your method is faster.

with 20 concurrent threads, 100k requests, no keepalives, .46 seconds faster and 3.4 tps faster.
oddly, at 20/10k/no keepalives, the margin tilts the opposite direction and is just as slim.

Although, I tested Apache2/mpm-prefork, not Apache1. When I benchmarked Apache1.3 last spring, there was a considerable difference edging towards having separate lines. However, with the config file in the virtual host rather than being read from .htaccess, and turning Override's off, it narrows down to .16 seconds faster and less than 1.1 tps faster. Granted this was just a seat-of-the-pants quick benchmark using ab, with all 5 tests run 5 times and the lowest score taken. Each used a referrer that required every rule to be evaluated.

So, current wisdom would suggest this:

Slightly corrected to remove two typos:

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9]+\.)?yourdomain.com(/.*)?$ [NC]

this disallows RFC compliant hostnames that have a - in them, fixing the regexp to allow that makes it a bit more complex since the leading character cannot be a -, but, lets skip that for now and allow - anywhere in the hostname and allow for the port designation which some edge proxy servers like to insert. We might also consider allowing _ which was not in the original specs, but, Microsoft supported it, and it was later adopted. This rule also doesn't allow for the new International characters, but, if someone is using those, they can change the regexp.

RewriteCond %{HTTP_REFERER} !^http://([a-z0-9-]+\.)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

This disallows subdomain.subdomain.yourdomain.com, which might be part of the + typo in the earlier post.

RewriteCond %{HTTP_REFERER} !^http://(([a-z0-9-]+\.)+)?yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

So, Raymor, does the above rule seem sufficient for as many cases as possible?

Offhand, I can think of one really simple way to circumvent this -- Firefox pops a warning, but IE seems to have no problem with it.

I've run that rule against a regression test to make sure that it appears to behave exactly the way I would expect and intend it to work and short of the one possible regexp match which could be very easily implemented on fuskers side, it seems fairly complete.

Now, of course we're back to the age old problem of Google/Yahoo's hotlinking image search. By doing the rules this way, we are only allowing the listed domains to be able to hotlink. If the rule is sufficiently loose to allow google/yahoo, a slight url change on fusker's side will make all of this for naught.

[Torn Rose]

slide rulers at twenty paces?

[raymor]

Originally Posted by cd34

So, Raymor, does the above rule seem sufficient for as many cases as possible?

Offhand, I can think of one really simple way to circumvent this -- Firefox pops a warning, but IE seems to have no problem with it.

One additional atom I included in my original 1999 regex
but I removed along with the port number was http(s)?
in case
some part of the site used SSL either presently or in the future.
That would cover an additional case, whether or not it's worth including
is another matter.
Additionally I unless you're running a web server on some port other
than port 80 you may want to just say 80 for the port number.
Makes it just a tad faster. Of course that analagous to my subdomain
part, where I chose to allow more than just www.
Another judgement call speed / versus flexibility.

I was thinking that we had taken care of user:pass@host URLs,
but I'm not seeing exactly how just at this moment.
Perhaps the tested browsers wouldn't allow blah.com/
in the authentication part, but treated it as a host/path when
it saw the forward slash (/).
Unless I'm missing something, we actually want to gobble up
anything up to and including @ before going any further:
RewriteCond %{HTTP_REFERER} !^http://(.*@)?([a-z0-9-]+\.)*yourdomain.com(:[0-9]+)?(/.*)?$ [NC]

(([a-z0-9-]+\.)+)? ? Maybe ([a-z0-9-]+\.)* ?

For those who don't catch the difference,
I had a typo where I said "optionally with a subdomain"
rather than "optionally with subdomains".
His fix for my typo was to say:
(([a-z0-9-]+\.)+)?
ergo "one or more subdomains, repeated one or zero times"
Simpler, we say "with or without subdomains":
([a-z0-9-]+\.)*

[raymor]

BTW, regarding the hyphen and the multiple subdomains guess
I should have pasted from my own old page from years ago and
I wouldn't have forgotten those:

http://216.239.63.104/search?q=cache...htaccess&hl=en

[RawAlex]

Okay, now that the geeks have had their moment in the sun (hope they don't burn), let's get back to the simple deal:

Program owners that run freehosted galleries: You need to lock up your content.

It's good business, and it is respectful of your affiliates.

Alex

[DangerDave]

Originally Posted by Tommy

if all webmasters protected their images 99% of these cock suckers would be fliping burgers

True IMO - There must/should be a united and concerted response to stoping these thieves.

Originally Posted by MrBlue

I read on another board that programs don't necessarily mind fusker grabbing their pictures. Still people are seeing the watermark on these photos, still seeing the URL, maybe they might type in the url and the program would get some type in traffic.

Yet another unproven unsupported theory proposed by those that don't know or are too lazy to care..

I notified one prominent program that their stolen images were being used on a message board just 2 days ago.. and their response(ignorant and ill-informed resonse - IMO) was..

"As our URL's are on the images, and it does not appear to be an entire set, I am inclined to let it go on this instance."


In my mind this makes them part of the problem, and encourages the theft of their content. In fact it is essentially giving me(and others) implied permission to use their content without permission.


Stopping these thieves making money is the only answer.. and AVN/Adbrite hold the key to stopping them. Lack of action on their part will cost YOU money.

DD