Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

Argh! A SERP in my SERP!

For the most part, in order for Google to index your results, you need to have it linked from somewhere. For content pages, that makes sense. However, for dynamic search results, it’s doubtful that you have links going there. So, we need to find another way to get our results into the index.

Our secret inroad will be your site’s Google Sitemap. This index of content on your site helps Google find content that might otherwise be hidden or unlinked.

Assuming that your search is GET based (instead of POST), which it really should be, it’s now only a matter of adding search URLs to your sitemap. For example, you could add the following:

http://my.site.com/search/term1

http://my.site.com/search/term2

...

http://my.site.com/search/termN

Google will extract these URLs and crawl them just like any other page.

For bonus points, you can get your users to do the hard work of generating the terms to index. Just track the terms that they are searching on in your site, then stick these in your sitemap. If there are too many, then just use the top 1,000 or 10,000 terms.

Before you know it, you’ll have thousands of search pages in Google’s index, irritating (ahem…”informing”) your fellow Internet denizens.

Note: It’s a solid theory, but I haven’t personally tried this yet, so maybe Google is smart enough to discard these results. Let me know if this works for you.

Yax was an early adopter at Obsidian Portal, and was kind enough to give us a nod in his article. In what I’m sure is a total coincidence, we got a ton of traffic today and about twice as many new signups as our previous high.

So, thanks Yax and keep the good press coming!

Anyways, thanks Yax for the link and knowing that we exist!

Getting onto the syndication services is not that hard. Or, at least that what’s they say. I have had a devil of a time getting the Obsidian Portal adventure logs to show up. By any standard definition they are blogs, and therefore should not be excluded from the syndication services. Still, here I am after a month with very little to show for it. Worst of all, it’s extremely difficult to track down where I’m going wrong.

Under the assumption that someone out there is in the same boat, here are a few tips to help out. I’ve grouped my tips into two categories. “Concrete Advice” covers tips that could definitely make a difference and should be attempted first. “Shots in the Dark” are some ideas I had that may or may not do anything at all. But if you’re desperate…

Concrete Advice

Targets

Rather than searching out all the services, just use pingomatic. You can add any additional ping targets if you wish, but pingomatic has found some pretty good ones.

Testing

Actually testing your pinging is probably the hardest part. You send a ping out into the ether, get a response like “Thanks for the ping!” and then you wait. And wait. And wait.

Sometimes you will have to wait several hours for the blog or post to show up on Technorati. Sometimes it won’t even show up at all. Not knowing if the ping is working is the most frustrating part of the entire experience. That’s where weblogs.com comes in.

Weblogs.com is your testing buddy!

Ever hear of weblogs.com? Neither had I, until I started this journey. Apparently, they’re the wackos who came up with this crazy pinging idea in the first place. To boot, they provide the best way of testing whether or not your ping service is working. They have a list of the most recent pings they’ve received available as an XML file. So, here’s how the testing works:

1. Send a ping to pingomatic. Verify that pingomatic responds correctly (ie. “Forwarding your ping to 16 services”).
2. Get a cup of coffee or something.
3. Download shortChanges.xml (use wget to avoid caching by your browser).
4. grep for the URL of the blog you pinged.

Note: shortChanges.xml seems to be cached on the server side and updated every couple minutes or so, so keep checking if you’re not there right away. After 5-10 minutes, you should either be listed or your ping never made it.

If it’s there, then you can be absolutely sure of 2 very important things:

1. Your ping to pingomatic was successfully received.
2. Pingomatic forwarded your ping to 1 other service successfully.

That may not seem like much, but we can infer (ie. assume) a lot more, namely that pingomatic is forwarding your pings to all the other services. This means that whatever problems you’re having getting registered with the syndication services, it’s not related to your pinging process. So, if you’re still not showing up on Technorati, it’s time to do some more digging.

Are you valid?

Ok, now your ping is working, what’s next? Validate your site and your feed!

The first thing a syndication site will do is pull down your feed and spider your site. You want to be as welcoming as possible when that happens. That means having valid, well-formed XHTML for your site and a valid RSS/Atom feed. Both of these are easy enough to check:

• RSS/Atom Feed Validator
• W3C XHTML Validator

They will tell you what’s wrong with your site. Get it whipped into shape so the syndication spiders find what was promised by your ping.

Extend your best foot forward

Since we’re using pingomatic, we have our choice of a regular ping or an extendedPing. Just go whole-hog and send the extendedPing. It allows you to specify both the site URL and the associated RSS/Atom URL. Send all the info you can to pingomatic, and let them decided what to forward on to the other guys, depending on who can accept it.

“Ping Test” is a crappy post title

When looking for your posts on the syndication sites, make it easier on yourself, and use a test post with easy to search for text. “Ping test” is going to lump you in with all the other people doing the exact same thing. Instead, stick in a string of nonsensical text like “flatly waking Oberon” that doesn’t show up in a Google phrase search. This will make your tests just a little bit easier to find.

Shots in the Dark

Tag, you’re it!

Add a few categories to your post in the RSS feed. A lot of spiders and search engines (as well as blog apps) seem to treat the category field as a place to dump social tags. So, you should too! Even if you have to hard code a few categories in there, go ahead and do it. For the Obsidian Portal adventure logs, every post is tagged with ‘games’, ‘gaming’, ‘rpgs’, and ‘roleplaying.’ Does it help them get picked up? I don’t know. That’s why it’s a shot in the dark.

Check your title tag for your blogs

For a while, I was adding “Obsidian Portal” to the title tag of every adventure log. For the fleeting moments when they were showing up on Technorati, it looked terrible. Suddenly, they all disappeared. It occurred to me that they might have been flagged as duplicates or too similar. Same domain and similar titles. Is that the case? Who knows?

Parting thoughts

I have not had much luck getting listed with Technorati or any of the other services. Sometimes it works, sometimes it doesn’t. After about a month of trying, I’m about ready to throw in the towel. I’ll keep the pings going, but I’m not going to devote much more time to testing and analyzing whether or not they’re working.

If you do manage to find the secret to getting listed, please speak up in the comments or write a blog post of your own!

References

• XML-RPC ping API – This is what you should be sending to pingomatic.
• weblogs.com shortChanges.xml – This lists all the pings that weblogs.com has received in the last 5 minutes.
• RSS 2.0 specification – Explains what a valid RSS feed should look like.

*Author is full of crap and is still unable to get his site’s blogs listed on Technorati. If you know what he’s doing wrong, please post a comment!

Fast forward a few months and now they’re tired of working in the dark. At a certain point, you just have to push it out there and sink or swim. In fact, their business depends on both sinking and swimming. Ha ha, lame. Anyways, if you’re a scuba diver or were ever interested in being one, check them out:

From the guys here at AisleTen, we wish the best of luck to MyNextDive May you find your niche and get your user base!

References

• MyNextDive – The actual site.
• Introducing MyNextDive – A press release for the launch.

Following Ryan’s experiences at Barcamp San Diego, I’ve decided to attend and (hopefully) present at Barcamp Atlanta. If I get the chance, I will be doing a presentation on custom Google Maps using S3. I will center on Ruby on Rails, of course, but most of the idea is language agnostic.

I’m hoping to use BarCamp as an opportunity to network with other Atlanta area entrepreneurs and hackers. I’ve got a lot of ideas and the skills to execute them, but I’m looking for people that are better than me in the marketing department. If you’re the kind of person who can sell water to a fish, then look me up at BarCamp!

Here’s a picture of me, looking as I probably will at the conference (lost and confused). Please don’t be shy, just come up and introduce yourself.

Recently, I found TreasureTables, a blog for tabletop RPG Game Masters (GM). In the forums, there is a section devoted to GM resources. Since Obsidian Portal is definitely a resource for GM’s, I went ahead and put up the pitch statement, along with a call for feedback.

Within a day, there were two new campaigns signed up (Dark Skies and Shadows of Skullport), and several people had posted specific, useful feedback regarding the site. Some had even commented on the possible commercial aspect of the site and how much they would pay to use it. Of course, that has to be taken with a grain of salt because it’s much easier to say you’ll pay than it is to actually drop the money.

From the 20 minutes to create the forum posting, and the probably 1-1.5 hours since then updating it as people ask questions, I have been able to generate 2 campaigns and probably 5-6 signups. That may seem small, but for a small site trying to grow, that’s a big deal.

The moral of the story is to keep hammering away at blog comments, forums, and the like. I have learned that “word of mouth” advertising often means your words, your mouth. It can seem frustrating, constantly searching for new places to regurgitate the same pitch over and over, but it is necessary at this stage. Keep marching on and eventually someone will bite.

Apparently, at some point about two years ago, Google decided to stop refreshing the Google Directory and updated the PageRank algorithm to discount any gains from DMOZ. However, this seems only to affect sites listed after they did this. The end result is that getting listed on DMOZ now seems to have no effect on your score.

Part of the reason my findings were so skewed is that the categories I was looking in were pretty static. They had not changed much in the two years that Google was not updating. So, I never checked the PageRank of a DMOZ listed site of less than two years. In my defense, there seems to be no direct way to see when a site was listed on DMOZ. However, using a more dynamic category might have helped.

Bottom line: DMOZ seems to have absolutely no effect on your PageRank score for newly listed sites. Unless you have a time-machine, it won’t do anything for you. I’d still submit your link, as someone might just stumble upon it by browsing DMOZ, but definitely don’t stress over getting in. There are more important things to worry about.

Original (and flawed) article follows:

While looking for ways to promote our Dungeons and Dragons website, I stumbled upon the Open Directory Project, also known as DMOZ. I had heard from an SEO expert (see my earlier post on Atlanta Web Entrepreneurs) about submitting a link to DMOZ. However, he just sort of mentioned it in passing. Luckily, I happened to jot down the 4 letters. I guess he just assumed we all already knew about it.

While following up on those notes, I went to the DMOZ site and started browsing around their Roleplaying section. Most of the links are good, and some are excellent. On the other hand, there is a sprinkling of sites that are just plain dumb. Not that I’m in any position to judge anyone else, but seriously folks, an entire site devoted to discussion of how to roll a d3? Funny? Yes. An unbeatable resource for RPG players worldwide? Um, no.

On a whim, I decided to run these sites through a PageRank checker. Imagine my surprise when they all turned up a PageRank of 6! This is equal to the score for The Wizards of the Coast homepage. Sit back a moment and process this: a single page with an ugly background and a crappy poll devoted to how to roll a d3 has the same PageRank as the homepage for the publisher of Dungeons and Dragons, the most popular tabletop RPG ever.

Don’t believe me? Then check it out for yourself:
PageRank Checker
How do u D3?
Wizards of the Coast

For fun, I decided to compare the sites in Alexa. Now, Alexa is totally separate from PageRank, and there is no reason to expect any sort of direct correlation. However, it’s reasonable to expect an indirect correlation. After all, high traffic probably implies that there are a lot of links pointing to you, and the more traffic you receive, the more likely you are to receive new links. Reasonable, right?

Anyways, according to Alexa, wizards.com has a reach of 0.03%, meaning that of all the users measured by Alexa, 0.03% of them visited wizards.com. If you think about it, that’s a perfectly respectable number. Predictably, the d3 poll site was not even ranked. Basically, they are orders of magnitude apart in actual traffic. A candle against the sun.

So, what’s the catch? How can they have the same PageRank? The PageRank algorithm is based on the number of incoming links and their respective PageRank scores. If you have highly ranked incoming links, then you are blessed with some of the value of those links. The more you have and the better they are, the higher ranked you are. So, maximizing PageRank is a fairly simple process (in theory): get popular people to link to you.

DMOZ matches up perfectly with this, for one main reason: a single link on DMOZ translates to dozens, perhaps hundreds of links elsewhere. This is due to the fact that DMOZ publishes their directory freely as an RDF feed. Websites are encouraged to pick up the feed and repackage or republish it as desired, with very few restrictions. So, get a link on DMOZ, and you automatically (with patience) get a link on all the sites that pick up their feed.

I checked the d3 website’s incoming links according to Google. To do this, just go to Google and type in link:www.somesite.com. Looking at the results, it’s easy to see that pretty much all of the highly ranked incoming links are repackaged feeds from DMOZ.

While a bunch of free links is already a great thing, there is one massive cherry sitting on top of the pie: Google itself republishes the DMOZ feed as Google Directory! As you can probably imagine, a link from Google to you is a massive PageRank boost, due to Google’s own maximized PageRank value. It’s like getting Batman as a character witness at your trial. His word carries a lot of weight.

Unfortunately (or luckily, depending on which side of the fence you’re on), getting a link in DMOZ is not exactly easy. It’s not hard, either, but it seems to take a while. As of writing this, I have yet to actually get Obsidian Portal listed. Still, if you’re looking for a way to get your site noticed, I can think of no better way to spend the next half hour than by browsing DMOZ, finding the perfect category., and submitting your site. If you’ve got quality content and your site is truly worthwhile, then all you have to do at that point is wait.

Good luck getting listed

Update (2007/05/29): From what I’ve been reading, expect a very long wait to get listed. Apparently it can take months or even years to finally appear in the DMOZ listings, mainly due to a huge backlog of submissions and very few editors. Still, there are worse ways to spend a half hour of your time than taking a long-shot at what could end up being the perfect link for your site.

Update 2 (2007/05/31): I have been accepted as the editor for the Games – Roleplaying – Software category! My first order of business was to list Obsidian Portal along with the other two sites that were included as part of my application. I had to apply twice in order to be accepted, but the entire process took under a week. So, if you have a site that you want listed, and the correct category has no editor, then your first order of business should be to apply for that category. Otherwise, it will probably take several months in order for someone to get around to actually reviewing your site.

Note: I was completely honest about my affiliations, as well as my intentions, and they apparently did not penalize me for it. So, be honest, be sincere, and good luck!

All in all, it was a good talk, and I didn’t feel too sleazy for being there. I was worried that an SEO talk would devolve into discussions of how to fool Googlebot into putting you at the top of the list for keywords like ‘stud’ and ‘teen.’ Luckily, John stressed one main point on how to best climb the charts: provide high quality, original content that has value to your readers.

Beyond that, he gave many good tips that are applicable across a wide variety of sites. I’ll list off the ones that I was quick enough to write down.

John’s Tips

Make sure your links are spider-friendly

When the search engine spiders reach your page, they will rely on you to navigate them around. The only way they can find their way to all your pages is if there are links providing navigation to the each page. Also important, these links should be text links, not flash. The text of the link helps the spider identify the keywords to associate with the material on the page. Further, some links are incomprehensible, such as Flash or Javascript links. The more plain text you have, the friendlier it is to the spider.

Avoid URLs with lots of query string variables

I can’t say much to this one, since I don’t know how the spiders work. John, however, claims that they do not like URLs with lots of query string variables. In any case, these URLs are definitely not friendly to users, and they look ugly in a search result. Instead, allow the URL to express some sort of organization of your site. For example, compare the following two URLs:

http://mysite.com/products/jewelry/gold/diamond-tennis-bracelet

http://mysite.com?cat=44&subcat=92&prodId=2412145

Which one is more descriptive? They could be talking about the same thing, but it’s impossible to tell from the URL.

In the Rails world, URLs like this are fairly easy to achieve, thanks to routing. By implementing a few routes, you are able to clean up your URLs and make them look nice and pretty. Take a quick look at the official manual on routing and you’ll see just how easy it is.

On Obsidian Portal, we are trying to use nice, easy to read URLs for as much as we can. Currently, we’re not doing as well as we could, but that’s mainly because we have so many features we’re trying to implement. Making URLs look pretty is not exactly at the top of the list. However, we have had some success with campaigns and game content. For example, http://www.obsidianportal.com/campaign/kensing takes you directly to my campaign, called Kensing. In another example, http://www.obsidianportal.com/game_contents/show/memory-steel, takes you to an item I created called memory steel. That URL could use some cleanup, but you get the idea.

Unique <title> tag on every page

The search engines place a lot of weight on the words they find inside the title tag. So, make sure every page has keywords in the title. Personally, I am not a big fan of long, jumbled titles that are just a mish-mash of keywords. Instead, give each page a meaningful title that happens to contain 1 or 2 keywords related to the material presented on the page. This will give it a nice appearance in the search engine result pages.

Establish a baseline for your current stats

Before you can get better at anything, you need to know where you currently are. For search engines and page views, you need to establish your current baseline and then track your stats over time. Probably the easiest way to do this is by setting up a Google Analytics account. It provides a nice graphical view of how you are doing over time.

use link: on your competition

This is an excellent strategy for finding out why your competition ranks better than you. Go to Google and enter:

link:your.competitors.site.com

This will return a list of all the links (that Google knows about) to your competitor’s site. You may discover that they have several high-ranking incoming links that could also apply to you. Track these places down and see if you can get yourself listed there as well.

New friends and plans for next month

Besides listening to the SEO presentation, I also met a few Ruby on Rails developers here in Atlanta. Calvin Yu and Neil Green are members of the ATLRUG (Atlanta Ruby Users Group) and they’re working on a new site, but they asked me not to discuss their project just yet, so I’ll keep it under wraps.

Next month’s meeting will be a round-table discussion of various web technologies (such as RoR). I cornered the organizer after the meeting and offered to sit in as a RoR “expert.” Just don’t tell him how little I know