Midnight Oil

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

We have finally gotten to the point where we are ready to start offering subscriptions to Obsidian Portal. We don’t expect there will be a lot of interest, but it’s always a sort of chicken v. egg problem. If you don’t have paying subscribers, then it’s not worth the effort to make the features. Conversely, without the features, no one is going to pay. On second thought, I guess it’s not chicken and egg, it’s pretty clear: you need features or no one will pay.

Asking for payment means you will need to be able to accept it. Currency on the web is passed almost exclusively via credit cards (except for PayPal…), so that’s the direction we need to go in. That requires us to select a credit card processor. For today, we will be looking at TrustCommerce.

I won’t go into the details of how credit card processing works, mainly because I don’t really understand it myself. Suffice it to say, there are a lot of middle-men, and they are all trying to take a cut. Each cut is either a percentage of the total charge or a flat fee or both. So, a typical fee structure might be $0.30 flat fee plus 2.5% of the total transaction.

Note: If you don’t care about the analysis and just want to see a rundown of their prices, then jump to the pricing.

Go easy on me; it’s my first time

When selecting a processing agent, our first priority right now is ease of use. We don’t expect there will be a lot of people signing up for our premium service, so we don’t want to expend a lot of effort on a payment system only to never see it used. Also, we’re willing to pay a higher rate to the processor since 3% of $30/month is a lot different than 3% of $30,000/month. I’ll pay 3% vs 2.5% if the 3% service takes 2 hours to implement and the 2.5% service takes 10. So, for us, ease of use trumps competitive pricing.

Since we’re talking about subscriptions as opposed to purchases, there is a recurring element to the payments. Since we want easy-to-implement solutions, we are scoping our search to only include the payment processors that offer a recurring service. This is a very important thing to note, especially if you’re in the same boat. A 1-time payment processor model (like Google Checkout) just will not work if you want to do subscriptions. The main reason is that you will have to store the users’ credit card info on your server in order to pass it to the payment processor each billing cycle. Do not do this! There are actual laws and regulations detailing what sort of security procedures you have to maintain in order to hold that sort of sensitive data. It’s much easier to simply pay someone else to deal with that crap. If you do choose to store their info in your database, you should be looking for a lawyer right now, not a payment processor.

Just plug in your credit card info

In Rails, ease of use means finding a plugin. I write a lot about plugins on this blog, so why should credit card processing be any different? Doing a quick Google search led me to the TrustCommerce subscription payment plugin.

Finding this bit of code brought a smile to my face, as I thought I had just finished 90% of the work. Sign up for an account, drop in the plugin, and wait for the money to roll in. Too bad there were a few red flags that derailed the money train.

Sitting by the phone

TrustCommerce does not list any pricing on their website. Instead, they say you have to sign up for a test account, and then you’ll be contacted. Not a big deal, I guess. So, I signed up for a test account.

The first red flag went up when I did not get an immediate callback. Sure, I signed up at 11:00pm Eastern Time, but that’s normal business hours in Internet time. In other words, if you’re an Internet company that requires phone contact, you had better have someone manning the phone at all hours. A lot of Web jockeys like me have a regular 9-5 job that precludes us from doing our business dealings during normal business hours. I want to deal with companies that understand this and have staff available during my normal business hours.

Red flags: 1

The ball sits in my court

The second red flag went up at their lackluster eventual response. My cell is in a dead zone at work, so whenever I leave for lunch, I get all my messages. On the day after requesting contact, I had a voicemail message from TrustCommerce. Still no pricing info, just a short message to call them back. Seeing as how I was busy, I couldn’t do it right away. Then I forgot. Dead silence on their end. No e-mails, no more calls, nothing.

Now a lot of people may disagree with me on this, but I think they should have been hitting my inbox and voicemail pretty hard. “Mr. Wedemeyer, we’re still interested in talking to you about blah blah.” or “Send us an e-mail with the best time to call you.” That’s how the mortgage people behaved when I used LendingTree. Sure, it was annoying, but you knew they wanted your business. To me, an anemic response indicates that someone isn’t really serious about recruiting me as a customer.

Red flags: 2

Little fish: prepare to get fried

When I finally did get in touch with someone from TrustCommerce, he was quite happy to answer my pricing questions. I don’t know if I’m allowed to post that info, but since they didn’t expressly forbid it, here you go:

Basic pricing

• $95 1-time fee
• $20 / month
• $0.20 / transaction

Citadel (recurring payments)

• $145 1-time fee
• $10 / month
• $0.10 / month / billing id (ie. subscription)

Holy crap! $240 just to get started, plus an additional $30 per month, just to be allowed to use their service? Seeing as how I expect Obsidian Portal to be making around $10 / month, at least until we can recruit more people, this is insane! I politely said thank you to the salesman, hung up the phone, and started writing this post.

I guess I see these huge front-loaded fees like this: If you’re making enough money that the fees don’t matter, then you already have a lot of subscribers, which means you’re already handling credit cards. Maybe their service is so great compared to the competition that it’s worth it for the big boys. But, if you’re a small time operator like me, forget about it.

Red flags: 240 + 30 / month

The search continues

Although I said pricing was not our top priority, the front loaded fees with TrustCommerce completely invalidate them as a viable option. It would be a very long time before we paid off the initial investment, and with our none-to-clear business prospects with Obsidian Portal, that’s a gamble I’m not willing to take.

In the next exciting chapter we will be looking at Amazon Flexible Payment System (FPS). This new web service from Amazon is meant to rival Google Checkout and PayPal. I’ve been extremely pleased with S3, and maybe they can do one better with FPS. Stay tuned to find out.