Midnight Oil

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

We’re currently evaluating possible payment gateways for our new project, DoLeaf. Given that it’s a marketplace app, your standard one-size-fits-all payment system just won’t work. We need to be able to transfer money from any buyer to any seller, and hopefully take a cut for ourselves in the process. With that in mind, we’re looking at both Amazon FPS and PayPal Express (via ActiveMerchant).

Here is our list of pros and cons for each solution:

PayPal Express

Pros

• Can assume that most sellers will already have a PayPal account.
• Buyers are used to using PayPal.
• Integrates with ActiveMerchant, and PayPal + AM looks fairly simple.

Cons

• High barrier of entry for sellers:
  1. Seller must sign up for PayPal Premier or Business account.
  2. Seller must create API credentials (PayPal provides a walkthrough)
  3. Seller must provide us with their credentials (cut & paste, with the possibility of typos and such)
• Credentials are quite sensitive and must be treated as such, with encryption in the database at the very least.

Amazon FPS

Pros

• CoBrand UI can be used to install recipient tokens, making setup easy for sellers.
• Tokens are transferred automatically, meaning no possibility of typos or transmission errors when seller is etting up.
• Can run all the transactions for a cart quickly and seamlessly, rather than 1-per-seller.
• Probably could set up a 1-click ordering sort of system.
• Overall fees will probably be lower since there’s only 1 transaction per sale (rather than 1 for buyer-to-seller and 1 for seller-to-us)
• Tokens are not nearly as sensitive as PayPal API credentials.

Cons

• FPS and Remit are both still kind of unstable.
• Locks us out of ActiveMerchant, unless we want to support multiple pathways.
• No convenient pre-built seller interface. We’d probably need to create an account panel for handling refunds and such.

Thoughts

Overall, I’m leaning toward PayPal at this point. ActiveMerchant and the ability to possibly integrate future payment gateways with ease is a big plus in PayPal/ActiveMerchant’s favor. In addition, people are used to using PayPal for these sorts of transactions. So, buyers in the market we’re targeting will almost certainly have a PayPal account, but not necessarily an Amazon payments account. If only there were a way to make it easier for sellers to get set up.

Anyone else go down this path and come to a different conclusion?

You’re here because you want to be fully RESTful with Flex, found as3httpclientlib, and are now having some troubles. It could be because of the wonky Flash TCP security policies, problems with maintaining Rails sessions, or any one of a handful of other problems. Well, let me take away your burden, friend: Give up now and descend back into the darkness that is GET and POST.

Adobe, in their infinite wisdom, saw fit to only cover the basics with their built-in HTTP services. GET, POST, 200, 404, 500. Maybe there’s more, but not much. Stray into such strange territory as 201 (Created) at your own risk. Luckily, an enterprising 3rd party came along and figured out a way to get around this, creating the as3httpclientlib. Cleverly using XML sockets and connecting to port 80 allows for building your own web service controls inside of Flash. Brilliant! Unfortunately, it’s useless (or nearly so) for use in a real Flash app with a Rails backend. Give up!

I wrestled with as3httpclientlib for a few days, hoping that I could pair up Rails’s RESTful view of the HTTP with Flex. The first hurdle (as you know doubt know by now) was dealing with the Flash TCP security policy files. This stymied me for a few hours, as I had trouble serving the file from localhost to localhost. Eventually though, I figured it out. Still, requiring access to port 843 is a big red flag, as it immediately knocks out shared hosting and many other situations. So, unless you have control of every single server your Flex app will ever need to connect to, you’re screwed. One red flag…

Following my limited success with the policy files, I decided to push ahead. Immediately, I ran into problem two: session management. Without going into too much detail (since I don’t really understand it myself), the session information is not preserved across subsequent calls using the as3httpclientlib. URLLoader uses the browser’s connection to make the requests, and the browser handles transferring the session information back and forth. So, except in certain specific cases, using the URLLoader means that your Rails sessions just work. Not so with the as3httpclientlib. I didn’t test in detail, but it looks like a total game ender. I don’t think it’s possible (or at least easy) to read the cookies from Flash, and you’re going to have to manage sending that information yourself anyway. Red flag number two goes up…

Take my advice: Give up now and go back to URLRequest and URLLoader. The RESTafarian in me hates to say that, but you necessarily have to give up a lot of ivory-tower purity when you descend into Flex development.

Note: This is not a knock against the as3httpclientlib. From what I can tell, it’s a great library. I lay the blame for my problems at Adobe’s doorstep.

I’ve been playing around with Engines a bit at work, and I ran into an issue where I had to restart the server over and over due to class caching issues.

To make a long story short, I was able to force Rails to reload the particular engine I was using by adding the following to the engine’s init.rb:


%w(controllers helpers models views).each {|path| Dependencies.load_once_paths.delete File.join(File.dirname(__FILE__), 'app', path) }
Dependencies.load_once_paths.delete File.join(File.dirname(__FILE__), 'lib')


This worked with Rails 2.1 and an unknown version of the Engines plugin.

So the time for reflection and has come once again, and I’d like to make a simple resolution: Be less of a jerk on this blog and be more constructive.

Reading back through some old posts, I see that I’ve become prematurely curmudgeonly. I also tend to judge too quickly, and am then forced to recant and eat crow. Since I’m not a fan of the taste of crow, perhaps I should hold my tongue for a bit and try to come up with something useful to say, rather than trying to be snarky and clever.

So, here it is:

* Offer only applies as it pertains to this blog. Author will continue to commit douchebaggery in his real life, especially while riding his bike because that makes him better than you.