Midnight Oil

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

Today, Yax at DungeonMastering put up a good post about energizing your campaign by using a wiki. I thought this was common knowledge by now, but there were apparently lots of people who hadn’t thought of it. I still forget that I’m not a very good representative sample of the technical savvy of your average Joe.

Yax was an early adopter at Obsidian Portal, and was kind enough to give us a nod in his article. In what I’m sure is a total coincidence, we got a ton of traffic today and about twice as many new signups as our previous high.

So, thanks Yax and keep the good press coming!

There are a handful of websites that have sprung up to help gamers find each other. It’s a great idea, and was one of our original motivations for creating Obsidian Portal. At the time, I didn’t know about the other player matching sites, and I was too scared to ask, thinking someone would steal my awesome idea.

Unfortunately, most of these sites just don’t have the critical mass necessary to make them useful. I have signed up on a few, and I have never received any sort of response. Seeing as how I live in Atlanta, that’s a bad sign. If I can’t get matched up with players in a big city, there’s no way people in rural areas are getting any use out of it.

What if there were a way to combine and aggregate all the data from all the sites into a single index of gamers? In this day and age of web services and mashups, there is no reason that this should be impossible. So, for your consideration, I am proposing a communications protocol that will allow all the player matching sites to interoperate for the purpose of providing more results to their users. I am calling this the Lonely Gamer Protocol, or LGP.

Update 2007-11-06 I’m looking at using Google’s OpenSocial as a possible alternative to Atom. OpenSocial’s people element has almost all the necessary fields, and the gd:ExtendedProperty might be sufficient for the rest.

Motivations

Of course, the main motivation is to provide a better experience to our users. By sharing our data with each other, we will beef up the number of results returned to our users, thereby greatly increasing their chances of getting a hit.

However, at least in my case, there is a secondary, more commercial motivation. With Gleemax breaking onto the scene, Wizards of the Coast has become the 800 pound gorilla in the room that none of us small timers can compete with in terms of marketing or exposure. Unfortunately for the users, Gleemax just isn’t that great, at least not right now. They have a lot of promises, but very little in the way of actual features.

If we all try to compete directly with Gleemax, we will probably fail. They have an established userbase and a war chest they can dip into for advertising and paying their employees. However, if we band together and attempt to make interoperable services, we’ve got a shot. Pretty much any of the current gamer-finder sites is already 1000x better than Gleemax, and working together will only strengthen that lead.

Of course, Gleemax is free to support LGP as well, and this would be a great boon to the entire community. However, given its current state and their speed of resolving issues and feature requests, I don’t see this happening for a looong time.

Goals

• Free and open
• Game and system neutral
• Low barrier to adoption / easy to implement
• Simple and lightweight
• Language, culture, and country neutral

Please comment below if there are goals I should add or remove. Please note, however, that projects that are able to condense their goals down to a short list do better than ones that throw in everything and the kitchen sink.

Technical Specifications

LGP consists of a communication protocol along with a simple REST API for retrieving the data from participating providers.

Fields and Schema

The actual data involved in such an exchange should be very simple. Shooting from the hip, here are the fields I could come up with:

• Player name
• Geographic location (lat/lng)
• Games interested in playing
• Looking for game and/or Looking for players
• Link to a profile page (w/ contact info)
• Last update / login

XML is the obvious choice for the data exchange. That’s a no-brainer. Further, in order to further lower the barrier of entry, I suggest that we use a preexisting XML protocol as a base layer. Most of the fields I’ve suggested could be easily stuffed into an RSS or Atom feed.

Looking back on that list, there are a few fields that need to be fleshed out and some formats need to be agreed upon, but the core functionality easily fits. Using Atom as the base for this protocol provides a lot of great benefits. As I mentioned before, most languages will have an Atom parser, making feed consumption that much easier. In addition, many will have tools to help construct feeds. Finally, it will be possible for the data to be made available to individual users, if they wish, just by subscribing with an Atom feed syndication tool. It might strip out some of the additional fields that we add, but if you provide filtering at the upstream level (ie. “Subscribe to a feed of all the gamers within 50 miles”) then it shouldn’t matter.

Query API

I’ll be the first to admit that I’m not really sure what’s required here. There needs to be a way to harvest the data from the individual sources. Further, there will need to be a way to filter the results, both in order to save on bandwidth, but also to make the results useful.

The following parameters allow for filtering based on set size, preffered games, and the last time a player’s profile was updated.

Filtering on geographic location would probably also be incredibly useful, but it’s not nearly as trivial as the previous parameters listed. Some tools make geocoding and “within X miles of Y” type searching very easy. Still, this would raise the barrier of entry. Perhaps it could be an optional filtering scheme that individual sites could implement, but was not required in order to be compliant with the protocol.

Issues

Adoption

Probably the largest hurdle to LGP would be getting the individual sites to share their data. For the larger sites, their userbase is their most valuable asset. Making it freely available might not seem in their best interests. However, it’s obvious that from the users’ perspective, LGP makes a lot of sense. Once the user data becomes a commodity, the sites will be forced to differentiate themselves based on their other features. For Obsidian Portal, that should be easy.

Still, joining up and sharing your data should generate a lot of traffic back to your site, and therefore signups or purchases or whatever else you count as a conversion. This is because the primary way of contacting a user will be through their profile page, which most likely exists on the site that provided the data. You could even require that someone sign up for your site before they can contact anyone, although that might be sort of a jerky thing to do.

Privacy

Everyone implementing LGP as a provider would need to make this very clear to their userbase and give them the ability to opt-out. In fact, perhaps opt-in would be a better policy, forcing users to specifically elect to be include. Still, why would they sign up on a player matching site in the first place unless they wanted to be found? I imagine most users would jump at the chance to be included.

The main issue with privacy that I can see is related to the geographic data that is passed with the feed. Giving out your address on the Internet is generally considered a Bad Thing, and this is no different. To that end, here are a few approaches.

First, you could restrict the lat/lng to a city area only. Therefore, you pass nothing more specific than a user’s city. The obvious downside is that for a city like New York there would be thousands of identically located entries. If you tried to map them, it would end up a mess, with all stacked one on top of the other.

Second, it might be possible to include an intentional error into the geographical data. Instead of feeding out the user’s exact coordinates, the system could give a random point within X miles of the user’s actual location. This would effectively mask their exact location, but would probably be very complicated to implement. Further, users would jump around on a map every time it was refreshed. Not a great solution.

Finally, the solution I propose is to pass the onus of anonymity on to the users. When asking them for address info, make it clear that they are not to enter their exact, home address. Instead, ask for nearby landmarks or street intersections. The Google Maps geocoding API is good enough to translate such strings into lat/lng coordinates. This would allow the users to accurately place their location, while at the same time assuring their anonymity.

Duplicate Entries

I am already signed up on multiple sites, and I imagine there are others as well. So, if all the data were aggregated, I would appear multiple times.

For now, we’ll punt on this issue from the LGP perspective. Individual consumers are free to try whatever de-duping mechanism they think will work best, or just forget about it. I imagine that even very dumb de-duping (ie. throw away all hits with the same name within 1 mile of each other) would be pretty good. Plus, unless people routinely sign up on 50 different sites, it won’t be a huge problem.

Ultimately, it would be nice to have a central registry that assigns unique identifiers, but trying to establish such a registry would probably be more work than it’s worth at this point.

Feedback

I’d love to hear feedback from anyone interested. Protocols like this only succeed if they achieve wide adoption and serve everyone’s needs. So, if I’m unfairly excluding someone, or I’ve missed something from the main list of goals, let me know in the comments. I will update this post over time as we come up with new ideas.

Throwing down the gauntlet

By no means is this document meant to be an academic discussion or pointless pontification. This is a very rough draft of what I hope will be an actual service that users can interact with very soon. So, I will go ahead and make a challenge to my fellow site owners: Let’s finalize a spec for LGP by December 2007, and have it implemented on our sites by Christmas (Dec 25th). If we could help our users find someone to game with over the break, it would be quite a present.

For my end, I solemnly swear to have Obsidian Portal implement at least the provider side within 2 weeks of the spec being finalized. I feel comfortable making this promise because I know just how easy it will be to do. Heck, it will probably take less than 3-4 hours to do the actual programming.

To any actual gamers reading this, you have a job too. Go back to the sites that you’ve registered on, post some links to this article, and demand that they support LGP. Tell them that by supporting and participating in the network, they will greatly enhance the user experience on their site. Believe me, once the users start demanding things, it gets very hard to ignore.

References

• Obsidian Portal
• NearbyGamers
• Pen and Paper Games
• Spielerz.com
• RPG Registry
• Gleemax
• CritOrMiss.com