Hide
Don't get spied on – We respect your privacy and provide numerous options to protect it. Join Siafoo Now or Learn More

You might have noticed Siafoo was down for a few hours today. It turns out, like back in 2008, it all comes down to a configuration value setting.

In this case, we discovered that while moving the site to a virtual hosting provider, we'd somehow set a setting equivalent to APACHE_TAKES_UP_ALL_OF_YOUR_RAM = 1. In reality, somehow the per-thread RAM limit for Apache had been set to 4GB. (it should be in the hundreds of kilobytes!) In theory, this was causing us to run out of RAM and start swapping to disk, making everything so slow it might as well have been down.

To exacerbate the problem, we have a script that automatically checks that the site is up and that apache forwarding (siafoo.net -> www.siafoo.net, for example) is working correctly. While robots and real users back off when a site is swamped, our script isn't that smart, and so by opening up more connections probably just made things worse.

Anyways, we've tweaked the RAM settings and futzed with some timeout values... hopefully giving you a faster and more responsive site.

The silver lining here though is that even without this swapping occuring though, we were probably using way more RAM than we should have been (it always seemed to us that the site shouldn't need so much RAM). So these new settings might dramatically lower our RAM usage, letting us spend less money on web hosting and therefore (probably) more time improving Siafoo. Moral of the story: if something seems wrong, it probably is.

One More Thing

Oh, and I want to personally apologize for the site being down for so long... I know you might rely on Siafoo for your day-to-day programming needs, and we want to be there when you need us.

Comments

over 6 years ago (10 Feb 2010 at 08:33 PM) by Stou S.
Welp... I am glad you took responsibility for that little problem. You should try to be more careful with apache next time.