mod_offload

mod_offload is an Apache module and PHP script for redistributing server load from a "base" server to one or more "offload" servers. This was written because my site, icculus.org, was constantly slow due to large demand for many large files. Since the webspace was big enough to make a complete copy to another server infeasible, and its contents changed frequently enough that I didn't want to limit my users, I came up with this system.

mod_offload is inspired by The Coral CDN but I wanted something more lightweight and less complex. The Coral Cache has to solve problems I could avoid, as can most web operators that aren't trying to mirror the whole internet.

The gist of this project is that you install the Apache module on the base server, and the PHP script on offload servers where it will handle every request. The module on the base server will decide if a given web request should be offloaded, and redirects the client to one of the offload servers. The offload server gets a request from the redirected client and sends an HTTP HEAD request to the base server, so it can decide if it needs to (re)cache the file in question. If so, it pulls a copy from the base server via HTTP and serves it to the client on-the-fly. If not, it feeds the client from the cached copy. To the users on the base server adding and changing files and the downloading client, this is all basically transparent, beyond the base server suddenly being less loaded once the caches start owning copies of large and generally-unchanged files.

For those without PHP and those in low-memory situations, there is a cgi-bin version of the PHP script, written in C. It uses significantly less memory than the PHP version, but needs a server that will run cgi-bin programs.

Places mod_offload is known to be used:

• icculus.org
• Let me know if you use this module with your website!

What works:

• Basic streaming of static content.
• Won't stream passworded data for safety's sake.
• Can set a minimum filesize for offloading.
• Can exclude files from offloading by mimetype (w/ wildcards).
• Can exclude files from offloading by User-Agent (w/ wildcards), to keep things sane for, say, Googlebot.
• Can stream by directory, location, .htaccess or virtual host.
• Can stream to multiple, rotating offload servers.
• Streaming content of data from content server to multiple users, even while it is still caching.
• Offload servers only need a PHP script (Apache module is only for base server).
• Module supports both Apache 1.3 and 2.x APIs.

• Process crash/power outage/forceful kill of caching process isn't gracefully handled.
• Untested on Windows. Apache module probably works there, PHP script may or may not.
• Probably other things.

mod_offload is licensed under the zlib license. This means you can statically link it into a closed-source application and otherwise manipulate it however you like, without a warranty. I appreciate credit when you use it, but don't require it. I find this code useful, and hope you will, too.

(coming soon.)

(coming soon.)

Downloading with Mercurial:

mod_offload's source code may be downloaded using Mercurial (aka: "hg"). Hg allows you to get up-to-the-minute fixes and enhancements; as a developer works on a source tree, you can use hg to mirror that source tree instead of waiting for an official release. Please look at the Mercurial website for more information on using hg, where you can also download software for Mac OS X, Windows, and Unix systems.

Here is the web interface to mod_offload's Mercurial repository.

To download mod_offload via hg:

hg clone http://hg.icculus.org/icculus/mod_offload/

Page maintained by Ryan C. Gordon.