Midnight Oil

Hosting a Rails app has always been a bit of a pain. Standard shared hosting was always pretty much out of the question, so most people naturally migrated to some sort of VPS. The obvious upside of being able to do whatever you want is a big draw. However, there is a dark side to every VPS: sysadmin work. Updating with patches, installing software, setting file permissions, and all that plumbing takes a lot of time. Then, there’s the big-daddy of them all: managing your own SMTP server. Sure, turning on Postfix or Exim is pretty easy. I just hope you don’t mind it when all your emails disappear into spam boxes.

For Obsidian Portal, we have been happily hosting with Slicehost for a while. Their uptime has been great, and we appreciate their services like automated backup and DNS management. However, we have been handling all our sysadmin tasks ourselves. So, when Ryan recently met one of the representatives from Blue Box Group, and he said that they’re happy to handle all that for us (at no additional charge!), we were quite interested.

Since we’re currently spinning up DoLeaf, we decided it would be a good time to see what we can get with a more hands-on hosting provider. And, after just a few days, we’re very impressed.

Choice of Distro

Blue Box’s basic installation is some flavor of Red Hat. Ryan and I are Ubuntu guys, so this was a problem. A single email to the support staff: “Hey, can you rebuild our server with Ubuntu server edition?” Answer: “Yep, it’s done.” Awesome.

Firewall

Anyone else think iptables is a bit of a pain? I can use Firestarter or Webmin, but when I try to manage it manually, forget it. Luckily it only has to be done once. Or, in the case of BB, never. Just send an email with the ports you want open. They set it up and replied, plus added a few standard ports that we forgot.

Rails Stack

BB supports several different Rails stacks. We’re used to Apache2 + Ruby Enterprise Edition + Passenger + Rails 2.3.2 + MySQL 5. Even though this kind of setup is pretty straightforward, it always seems to take me at least 1-2 hours just to get things set up in the most basic configuration. Now? You guessed it. Send an email and I’m done.

Functioning SMTP – The Great White Whale

We’ve always had problems with emails disappearing en-route. Getting Postfix set up is a 5 minute job, but your emails randomly get flagged as spam, especially by the big boys like Google and Yahoo. There are many different ways to counter this, like SPF and DKIM, but they are a pain to set up, and it’s always tough to know if you got it correct. This was the tipping point for me with BB. They have an SMTP server preconfigured with SPF and DKIM and you can use it for sending emails from your app. They limit you to 750 emails / hour, which is way more than most apps need, especially in the beginning. Considering that we had evaluated similar solutions that wanted to charge $0.01 / email, having this included for free in our hosting plan was mind blowing. Never again will I host anywhere that doesn’t do this.

And the cost?

The most surprising thing is that I haven’t been charged $0.01 for additional support. I’m still testing the waters to see exactly where that line is, but so far, I haven’t hit it. Plus, their prices are quite competitive. Slicehost is a little bit cheaper, but when you factor in the time saved on setup, I’d say we’re already way ahead.

We’ve only just started with Blue Box, and things may go south at some point, but for now we’re ecstatic with what we’ve received. If you want to spend more time coding and less time on irritating sysadmin tasks, then I highly recommend taking a look!

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

Before we switched to Slicehost, we had our VPS on Rimuhosting. I must say that I was very pleased with Rimuhosting’s service, and we never had any problems with stability. We eventually had to switch to Slicehost anyway because it was simply cheaper and we’re on razor-thin margins here.

Still, switching hosts is not a trivial process, and you have to factor in the switching effort when deciding if the price differential is worth it. If you’ve found a cheaper host and are thinking about jumping ship, the very first thing you should do is contact your current provider and see if they’ll match the new host’s price. That’s what we did initially with Rimuhosting. They didn’t match the deal, but they did give us a little more RAM for free. That tided us over for another month or two before we decided we wanted to give Slicehost a shot.

So, for posterity, and to help other bargain hunters, here is the letter I sent, and Rimuhosting’s response.

I’ve been a Rimuhosting customer for several months now. In general I am happy with the service, although my server seems to be getting more and more sluggish in recent weeks.

Due to the performance issues, I began shopping around for alternatives and I came across Slicehost ( http://www.slicehost.com/ ). Since they’re a competitor, I’m sure you’re already familiar with them.

According to their price list, they will give me 256MB RAM plus more disk space for $20/mo, compared to the 160MB of RAM I’m getting for $29. For me, the RAM is the big deal, and the extra storage is just icing on the cake.

I am seriously considering moving to slicehost, as I am finding that the 160MB I currently have is eaten quickly by Tomcat, mongrel, and MySQL. Plus, as I mentioned earlier, I have been seeing some very serious performance issues lately, with slowdowns on ssh as well as terrible response times from the web server. This could be due to not enough memory on my VPS, but I’m concerned it has to do with too few CPU cycles being devoted to the VPS.

In all honesty, I would rather not switch hosting companies, as I have been pleased with the level of service from Rimuhosting. In addition, moving a VPS installation is no small task and I do not relish the idea of trying to replicate all my settings on a new machine. Still, Slicehost’s pricing plan is very attractive.

Before I make any decisions, I would like to ask what sort of plan you would be willing to offer in order to compete with Slicehost. If you could offer a combination of more RAM and a lower price, it would go a long way towards tipping the scales in Rimuhosting’s favor.

Thanks for your time and I look forward to hearing from you.

Micah Wedemeyer

http://www.aisleten.com

Their response to me:

Hi Micah,

I’ve added some additional memory to your VPS:

[root@obsidianportal ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           300        148        151          0         16         55
-/+ buffers/cache:         76        223
Swap:           95          0         95

Also moved your current pricing from 39.95, to where our current MiroVPS2 plans are, at 29.95 a month.

We feel that we offer a good value for the service we provide. We appreciate your business and hope you will stay with us.