Are people stealing your bandwidth by hotlinking you Vids?

[nate]

If you are running a blog with embedded vids, it is damn near impossible to stop hot linking goddamn theives from also embedding your vids unless you deny all viewers without a referrer. After some lengthy thought I came up with an idea that may help alleviate the problem. If anyone wants the solution, message me and I'll fill you in. It may not be the best solution available, but its the best I could think of. No, it doesn't cost anything and it isn't some sort of shit I'm promoting. It isn't foolproof, but nothing is.

Also you might note, it will deter hotlinking fuckers from using jscript to hijack your bandwidth by using your images too.

This is a public service announcement with the intent of stopping goddamn thieving hotlinkers. You may now return to your previously scheduled activities.

[nate]

addnedum:

come to think of it, if you deny all viewers without a referrer, nobody will see the vids embedded in your site, unless your player somehow gives a referrer (mine dont). So don't just willy-nilly deny people without referrers!! You will be denying all your legit viewers.

I just thought I'd add that, my solution to hotlinkers isnt related to referrers anyway.

[cd34]

http://redmine.lighttpd.net/projects...BeforeDownload

http://trafficguardian.com/ (anti hotlinking)

TrafficGuardian worked pretty well with apache until his hosting went down or his domain expired. Every morning when logs rotated, apache had to check in with his server for authorization. That might have changed and his site appears to be up now.

Another trick people use is to write a cookie on the page then use mod_rewrite to check for the existence of that cookie to determine whether to serve the media. Some browsers spawn a helper player that doesn't have access to those cookies.

The other thing I have noticed is that a lot of the rewrite rules have gotten away from specifics, allowing specially crafted urls to contain enough of the url pattern to match the referrer.

And there is good hotlinking, so, make sure your system accounts for that.

[Useless]

Originally Posted by nate

This is a public service announcement with the intent of stopping goddamn thieving hotlinkers. You may now return to your previously scheduled activities.

Balls to you.

In order for it to be a public service announcement, you'd have to ANNOUNCE IT PUBLICLY.

If someone wants to PM this guy and post his idea in this thread, I'll give you 5 cents on Epasssporte.

[nate]

Originally Posted by Useless Warrior

Balls to you.

In order for it to be a public service announcement, you'd have to ANNOUNCE IT PUBLICLY.

If someone wants to PM this guy and post his idea in this thread, I'll give you 5 cents on Epasssporte.

I'll post the gist of it. rewrite your .htaccess in your /videos subdirectory everytime a surfer hits your page header to include his IP. if his IP isnt in the htaccess he gets denied when he tries to view whatever the hotlinker tried to steal with his nifty jscript player.

I didnt want to post it on a board because it might aid the goddamn hotlinking thieves in figuring out why they cant steal your shit.

Is that public enough? Or do you also want me to post the 7 lines of PHP code you include in the header as well?

LOL. This is what you get for trying to help people out.

[nate]

Originally Posted by cd34

http://redmine.lighttpd.net/projects...BeforeDownload

I'm running lighttpd on a different IP address (same server) to check out how well it holds up. Its just running a page or two for now, but using apache's utility to check server speed, lighttp is indeed fuckall bunches faster for serving static pages. I installed it on a local linux server, but it was crashy there, but that's probably because of something other than lighttpd. Its an old server that is really mainly just for backing shit up. lighttpd has run flawlessly so far on my remote server. One of these days I'll get around to putting some tgp galleries on it and seeing how well it works out under a load.


I also read something about trip pages under lighttpd, but havent had time to reallly look into it. Is that what you were referring to?

That was one of the ideas I was going to look into further, as well as some weird iptables and proxy stuff, but the idea to dynamically rewrite the .htaccess for allow/deny was a whole lot simpler, at least under apache.

[Cleo]

Originally Posted by cd34

TrafficGuardian worked pretty well with apache until his hosting went down or his domain expired. Every morning when logs rotated, apache had to check in with his server for authorization. That might have changed and his site appears to be up now.

Pretty much took my server down when this happened if I remember correctly.

Good thing I had a great host that got me running again right away.

[cd34]

Originally Posted by nate

I'm running lighttpd on a different IP address (same server) to check out how well it holds up. Its just running a page or two for now, but using apache's utility to check server speed, lighttp is indeed fuckall bunches faster for serving static pages.

lighttpd has some interesting behind-the-scenes limitations that most people probably won't run into. There are reasons that it and alternatives are faster than apache and if you really want to be fair about it, configure apache2 with mpm-worker, strip out a ton of the default modules, turn off overrides and then do your test on a multicore machine. A quadcore or a dual dual-core with apache will handle traffic as quickly as lighttpd.... though.... without flv streaming. Google's infrastructure is core oriented which amazes me that they use lighttpd for youtube. lighttpd does not scale well with multiple cpus/cores. It is a limitation in the core of their code and would require a rewrite from the ground up to fix. It does well for what it is, and when it was written. There are newer alternatives out there that are considerably faster. nginx and varnish are two alternative methods that soundly beat lighttpd's serving. Both solve different problems though.

lighttpd removes quite a few capabilities that apache includes by default and lighttpd starts as a stripped down server.

Apache is by no means a great performer, but, it is consistent, has a huge installed base, and everyone writes software expecting all of the things that would normally be present in an apache environment.

As for your idea, verifying a cookie, trigger before download and trafficguardian, there is a very, very simple way to bypass those protections which makes those methods somewhat fruitless to someone determined to hotlink the sites.

Short of some very specific cases, not using apache will add headaches down the line. If you liked lighttpd, check nginx. If you're looking for something relatively seamless to make apache quicker, check varnish. I'll warn you -- once you step off the path from apache, you either have to change the way you work, or spend a lot of time making them work the way you do. And while they all claim to have somewhat sane defaults, those defaults are nowhere near production quality.

Just to give you an idea:

ab -n 10000 -c 50 http://66.55.44.33/info.php

nginx, Requests per second: 320.79 [#/sec] (mean)
apache mpm-worker, Requests per second: 320.65 [#/sec] (mean)
lighttpd, Requests per second: 278.49 [#/sec] (mean)
varnish, Requests per second: 3391.36 [#/sec] (mean)

(varnish cached the response and answered the request out of the cache. Depending on your site, php isn't cacheable since Wordpress and others stuff expire times in there to require that the page be generated on every pageload. Yes, you can tell varnish to ignore that.)

info.php is just <?php phpinfo();?>

71k .jpg image

nginx, Requests per second: 3505.96 [#/sec]
apache mpm-worker, Requests per second: 814.97 [#/sec]
apache mpm-worker with allowoverride turned off:
Requests per second: 2488.22 [#/sec] (mean)

lighttpd, Requests per second: 3819.69 [#/sec] (had 17 failed to serves)
varnish, Requests per second: 3012.98 [#/sec]

There's another test that I did on throughput -- lighttpd actually serves files slower than each of the others. 100mb file took longer to download on lighttpd than it did on the other three. So, lighttpd dropped 17 requests out of 10000 and serves slower on long files than the others. Granted a benchmark is a crude test, but, it gives you some idea.

So, if you're out for true performance, you've already swapped out apache-mpm-prefork for apache-mpm-worker. Or, if you like to live on the edge (no pun intended), you might be using mpm-event. If you're still using prefork and comparing that to lighttpd and have overrides set and mod_status and the dozen other apache modules still installed, then its not really a fair test.

Apache does extremely well until you start pushing 70-80mb/sec or get close to 1000 requests per second. If you like to use server side includes, varnish has Edge Side Includes which are extremely powerful.

Sort of diverted from the original post.

[nate]

Originally Posted by cd34

lighttpd has some interesting...

Damn, you seem to know this stuff backwards and forwards. Kudos. I just started looking into lighttpd for serving static tgp galleries. If I ever need help with anything server related, your the guy I'll grovel to.

[MadCat]

Originally Posted by cd34

lighttpd has some interesting behind-the-scenes limitations that most people probably won't run into.
....
....

Sort of diverted from the original post.

The only issue with mpm-worker is that you can so seriously wreck mod_perl with it -- one reason I'm still using lighttpd with FastCGI since it does scale up easier than apache does -- what limits exactly are you speaking of for lighty? I know it's pretty core bound but other than that I haven't really seen any issues with the 1.5.x tree.

Waiting for 2.0 really

Anyway back to the regularly scheduled thread...

[cd34]

lighttpd is bound by core due to their spinlocks. Their core is not properly threaded and they hacked together a solution that works most of the time. When it was written, we didn't have that many SMP or Hyperthreaded machines. They also don't use some of the hooks in the kernel that other solutions use. With cpu affinity and pinning interfaces, you can work around that, or, you can use nginx without all that mess. But, nginx isn't without its quirks either.

I've heard, mpm-worker and mod_perl don't get along but I've not done enough mod_perl to really run into it. I really don't understand why perl would have a problem with mpm-worker since worker only uses pthreads and perl has been threadsafe for years. It is possible that cpan modules are not threadsafe which would cause problems.

There are a bunch of potential solutions and its a matter of finding the right tool for the job. Apache isn't all things to all people -- it is about the most flexible, most generic solution and properly tuned will give the performance that 90% of the people out there will be happy with. mpm-prefork doesn't fall too far behind mpm-worker in most situations. Measure the tradeoffs, make the decision as to which way to head.

The task dictates the solution -- or something like that.

[MadCat]

Originally Posted by cd34

lighttpd is bound by core due to their spinlocks. Their core is not properly threaded and they hacked together a solution that works most of the time. When it was written, we didn't have that many SMP or Hyperthreaded machines. They also don't use some of the hooks in the kernel that other solutions use. With cpu affinity and pinning interfaces, you can work around that, or, you can use nginx without all that mess. But, nginx isn't without its quirks either.

Yeah, although if you run OpenSolaris you can sort of kludge your way around it by setting up zones, and there's some trickery where you can lock a zone into a core - at which point you're running 2 or 4 lighty instances. Not my bag of fun really

I've heard, mpm-worker and mod_perl don't get along but I've not done enough mod_perl to really run into it. I really don't understand why perl would have a problem with mpm-worker since worker only uses pthreads and perl has been threadsafe for years. It is possible that cpan modules are not threadsafe which would cause problems.

Bingo. A lot of the CPAN modules that are used with mod_perl based apps aren't thread safe, or even if they claim to be thread safe, they might not be thread safe in a persistent environment. mod_perl has it's own baggage, combined with threading turns it into a very tasty mess of potential pitfalls

There are a bunch of potential solutions and its a matter of finding the right tool for the job. Apache isn't all things to all people -- it is about the most flexible, most generic solution and properly tuned will give the performance that 90% of the people out there will be happy with. mpm-prefork doesn't fall too far behind mpm-worker in most situations. Measure the tradeoffs, make the decision as to which way to head.

The task dictates the solution -- or something like that.

Good point -- for really high performance setups the favorite of mine is to have nginx front things and serve static content, with 1 apache instance being proxied by nginx that has mpm-prefork going for mod_perl, and 1 apache instance with mpm-worker for php and any other jazz that isn't perl.

My regular setup is lighty -> fastcgi which does sort of alleviate the core bound problem since lighty just sits there forwarding things and all the heavy lifting comes off the fastcgi instances