Midnight Oil

I’ve been playing with PubSubHubbub (PuSH) and Superfeedr for several hours now, and overall I’m mainly just disappointed. I’m not sure who is to blame, but it’s not the plug-and-play experience I had hoped for.

The Goal

I actually have a purpose, not just play. I wanted to sync up the Obsidian Portal blog (Atom feed available) with the Obsidian Portal homepage. Essentially, we’d like to display a teaser from the latest post on the blog, along with a list of some other recent posts that might be interesting.

The Plan

After putting it in the back of my head for a while, the idea of using PuSH to handle syncing the blog entries over to Obsidian Portal jumped to the forefront. The basic idea was simple: instead of building an RSS poller/parser/whatever, just have someone POST notifications of new blog entries directly to the main Obsidian Portal app. That’s exactly what PuSH is for. Excellent!

Looking around, I found that Superfeedr was a PuSH hub that seemed perfect for this kind of thing.

Poll vs Notify

The first thing I did was start playing around with using Superfeedr to poll the blog feed. This immediately became a frustrating experience. The polling speed was 15 minutes, so every little test/debug/experiment I wanted to try had a 15 minute window. Completely unacceptable.

So, I went out and downloaded a WordPress plugin to support PuSH on the blog side. Now the blog should send notifications to Superfeedr so there would be no need for polling.

Um…nope. I’m not sure if it’s the plugin or Superfeedr, but it’s still polling at a rate of about every 15 minutes. It’s sometimes faster, but on the order of 5-10 minutes instead of 15. That makes it tough to do the sort of playing around that is necessary to figure out the quirks of a new tool.

Post body, not params

Now, this is my own damn fault, so blame lies squarely in my lap on this one. I just want to make sure that other Rails guys reading this know not to make the same mistake.

The Atom entry you receive as part of a notification is in the POST body, not a parameter! This took me a long time to figure out (long time due to waiting 15 minutes between tests, due to polling ). Here’s eventually what I did in the action that receives a notification:

atom = request.body.read

Now you’ve got the Atom text and can parse and go.

Summary vs Content

The next wtf? moment came when dealing with summary vs content. In the Atom protocol, it’s possible to have both the full-text content and a summary. This would be perfect for me, since I could leave the full-text content for any Atom subscribers we have, while using the summary field for the excerpt that appears on the Obsidian Portal homepage. WordPress actually puts a summary/excerpt in by default. Excellent!

Bad news adventurers: For some reason, Superfeedr only displays the summary element if there is not already a content element. So, it’s content XOR summary. There may be a valid technical reason for this, but otherwise it makes absolutely no sense to me. Why truncate possibly valuable data? Why not preserve the underlying Atom feed as closely as possible?

An easy solution for me is to remove the content and only include the summary, which WordPress supports out of the box. Still, I really hate this, as I (as a user) much prefer when a feed includes the full text of the article. None of that “come to the site and see all my ads to read the rest!” crap. It irritates me that I’ve been unnecessarily backed into this corner.

No Updates

The worst thing of all is that Superfeedr doesn’t seem to send notifications for updated entries, only sparkly new ones. This is completely unacceptable for my purposes. Let’s examine a common scenario:

Blog post: “Go their too read more…”

First 20 comments: “You used the wrong ‘there’ and ‘to’, dumbass…”

Ok, so I update the blog post and thank the commenters for their kind words. Still, Superfeedr never sends a notification, so the changes aren’t reflected on the Obsidian Portal homepage which gets seen by way more people than the blog.

This decision really has me scratching my head. The Atom protocol (and WordPress feed) have an updated datetime field that can be used to determine when the last update occurred. It seems simple to check that field and send a notification if it’s changed since the last notification. Again, I don’t want to blame Superfeedr here if it’s a PuSH thing.

Going Forward

At this point, having spent probably 5-6 hours playing with this, I’m reluctant to just chuck it. I’m in that “Surely the answer is just around the corner” mindset, but the lack of updates may be a deal-killer. If I have to manually poll for updates or manually edit entries that require update, then PuSH/Superfeedr becomes pretty much useless to me.

Should I switch to the official Google hub? I assumed that a commercial operation like Superfeedr would be preferable to the reference implementation, and I like their web panel more, but the lack of updates plus stripping the summary field are big no-nos in my book. Since I’ve already invested so much time, I guess I owe it to myself to give the Google hub a try before just throwing my laptop out the window.

Am I crazy? Stupid? I thought PuSH was supposed to be easier than this. If I’m not mistaken, my problem/goal is a textbook example of where to use PuSH/Superfeedr, but so far it’s been nothing but a major hassle.

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

Obsidian Portal was mentioned today on Wired’s Geekdad blog as a great place to do worldbuilding. Well, I would hope so since that’s one of the main purposes of the site.

A big thanks to the author, Michael Harrison, for the bump! I can’t wait for part two!

Today, we received a note from the Earl of Stirling regarding what I assume is his appearance on Obsidian Portal. (Search for “sterling”)

Please do not use my name in your gaming/whatever. I am the Earl of Stirling (short form Lord Stirling).

Thank you,
Stirling

Unfortuntely, I think we’re going to need a little more proof than a yahoo.com email address to verify the identity of a Scottish noble.

Update: This guy might be legit…or at least he thinks so. Check out his blog, where you can buy your very own Scottish Title! Impress your friends and get girls! Only US$90,000! Act within the next 30 minutes and we’ll throw in a free set of castle blueprints and a 25% off coupon for your next order at Arms & Armor!

Update 2: As always, the true story is on the Wikipedia article discussion. Long story short, a blowhard from Indiana claims to be Scottish nobility through some arcane legal loophole, friendly Wikipedians keep reverting his edits to the Earl of Stirling page, he repeatedly threatens legal action, he gets banned for threatening legal action. Ain’t Wikipedia great?

Today, Yax at DungeonMastering put up a good post about energizing your campaign by using a wiki. I thought this was common knowledge by now, but there were apparently lots of people who hadn’t thought of it. I still forget that I’m not a very good representative sample of the technical savvy of your average Joe.

Yax was an early adopter at Obsidian Portal, and was kind enough to give us a nod in his article. In what I’m sure is a total coincidence, we got a ton of traffic today and about twice as many new signups as our previous high.

So, thanks Yax and keep the good press coming!

Wizards of the Coast has announced they are shutting down Gleemax. Although I was never a big fan (being a teeny-tiny competitor…), it’s sad to see an online meeting place for RPG players disappear. In addition, despite all the bumps and stumbles, they somehow garnered the support of some really great people.

Of course, now all those supporters are going to be left out in the cold. I suggested to the powers-that-be that they allow people to retrieve their entire blogs as an RSS feed. It shouldn’t be that hard to do and would vastly simplify the process of moving a blog to another platform. We’ll just have to see if the WizOs have mercy on their loyal followers.

To all you Gleemaxers who need a place to brag about your campaign, Obsidian Portal is still open for business…

I’ve had my first brush with EC2 today, and I’m thoroughly confused and exhausted. It definitely has a much steeper learning curve than the other Web Services. However, since you’re dealing with booting and configuring fully functional virtual machines, I guess that’s to be expected.

Still, I managed to spawn an instance, set all the necessary permissions, and then connect to it via ssh. Not exactly something to brag about, but it is a milestone. I consider it $0.20 well spent. (I had to terminate and re-run the instance because I didn’t store the key-pair RSA key the first time.)

The goal

For Obsidian Portal, the map processing and tiling is very CPU and RAM intensive. If run on our Slicehost VPS, it totally bogs down the whole system, making the main website completely unresponsive.

So, we’ve offloaded the processing to a machine in Ryan’s attic. Although it’s worked so far, it’s not exactly a professional solution. Plus, Ryan’s getting ready to move and his machine will be off for at least a week. That means we need a new place to run our stuff. Since I’ve been meaning to dive into EC2 for a while now, this seems like the perfect opportunity to put in place a permanent and professional solution.

The plan

Whenever we need to tile a map, I plan to spin up an EC2 instance and run the tiler. I’ll keep the instance up for the rest of the hour, in case more maps come in. When the hour runs out, if there are no more maps to process, then we’ll spin down the instance. Computing power on demand…nice!

grempe-amazon-ec2

If you’re a Ruby hacker, skip the EC2 toolset altogether. Just go get the grempe-amazon-ec2 gem and run all your EC2 requests through an irb shell. This allows for opening up the EC2 developer reference and learning the API commands straight from the docs, rather than having to learn the EC2 toolset. You’re probably going to have to spawn/terminate instances dynamically at some point, so you may as well just learn how right from the start.

Besides, the EC2 toolset route involves setting up a JVM, setting JAVA_HOME, and all that Java mumbo-jumbo that always takes way longer than it should. Save yourself a headache and go straight to the source.

ec2onrails

I’m not looking to run a full Rails server (yet), but we do want to run some rake tasks. Rather than hand build an image, I’ve decided to start with the ec2onrails image and see if that gets me most of the way there.

The main downside is that this image will need to install RMagick, ImageMagick, and possibly other gems and packages each time it spins up. ec2onrails has built-in support for this, but it means that the map processing can’t start for several minutes after the image boots up, meaning our users have to wait to see their maps. For now, it should be acceptable, but it’s something to improve on.

Next Steps

I’ve had enough for today, but there’s still a ways to go. I need to complete the following:

1. Setup the cron job on our VPS to spawn the EC2 instance whenever a map needs processing (and there’s not already an instance running)
2. Setup the script (rake task?) for the EC2 image that will do the processing. It may as well just run forever. Maybe it can be responsible for terminating the image? Hmm…that sounds dangerous. If it should crash out, then the instance will keep running and billing us!
3. Surely there’s more…

Oh, and one other next step: I’d better go terminate the instance I left running!

We’ve taken a small leap and added several microformats to Obsidian Portal. It was just so easy, there was no reason not to. For starters, we’re going with hCard and XFN.

Right from the start, we’re going whole-hog on hCard. I made a helper function to output an hCard every time we display a user’s avatar. So, every comment, every recent update, every friend link has an hCard on it. If the microformat-aware spiders cannot figure out who is who on Obsidian Portal, then they’re brain-dead.

Secondly, we’ve added XFN rels on the friend-links in the user’s profile. I don’t know of any (non-trivial) tools that are using XFN yet, but it’s easy enough to add, so we did. We’ve got rel=me on links to user’s other websites, and even self-referential rel=me links on the profile pages. We’re basically screaming “I am me” at the spiders.

Still on the TODO list is rel-tag for the NPC character tags. We’ve got a lot of plans for tags in the future, we just need to free up some time to work.

In the future, I might look at adding hAtom for the Adventure Logs, but honestly, we’ve already got RSS feeds, and I can’t see the value-add here. Am I missing something? Let me know.

Examples

• My profile – hCard and XFN enabled
• The homepage – Notice the hCard for each recent-update user
• An Adventure Log post – Notice the hCards for the comments

Resources

• Microformats.org – The definitive source for information.
• Operator – A microformat parser plugin for Firefox. Adds a nice little toolbar that lights up when microformats are detected.
• Tails Export – Another microformat parser plugin for Firefox. Seems to handle de-duping better than Operator.
• rel-lint – A validator / checker for XFN and rel-tag