Midnight Oil

I’m a big fan of Behavior Driven Development (BDD). It really illustrates how one little change in your code can have significant impact on the rest of you application, which you would never be able to catch without testing. Not to to mention the benefit of being able to write code and make sure it works even if you don’t have production data available.

So, upon completing a clean upgrade to Leopard I noticed that after installing Growl and growlnotify, that growlnotify would not work even through Growl itself was working.

1) First let’s set up the .autotest file. For this you’ll need the gems rspec (1.1.3), ZenTest (3.9.1), and redgreen since ZenTest change how it handled exceptions in 3.9.

sudo gem install rspec
sudo gem install ZenTest
sudo gem install redgreen

Now open up your ~/.autotest file in Textmate

mate ~/.autotest

Paste in the following code. Notice the exceptions at the bottom of the file that really helps speed up autotests as well as keeps your cpu usage low. My Macbook would get really hot really fast prior to using this execptions.

require ‘autotest/redgreen’

module Autotest::Growl
def self.growl title, msg, img, pri=0, stick=”"
system “growlnotify -n autotest –image #{img} -p #{pri} -m #{ msg.inspect} #{title} #{stick}”
end

Autotest.add_hook :ran_command do |autotest|
filtered = autotest.results.grep(/\d+\s.*examples?/)
output = filtered.empty? ? ” : filtered.last.slice(/(\d+)\s.*examples?,\s(\d+)\s.*failures?(?:,\s(\d+)\s.*pending)?/)
if output =~ /[1-9]\sfailures?/
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_fail.png”
elsif output =~ /pending/
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_pending.png”
else
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_ok.png”
end
end

end

Autotest.add_hook :initialize do |at|
%w{.svn .hg .git vendor}.each {|exception| at.add_exception(exception)}
end

Next, you’ll want to download the Pass, Fail, and Pending images below and place them in ~/Library/autotest:

First, the prerequisite to growlnotify is Growl. You can download it from http://growl.info/. It’s a graphical installer, so you shouldn’t have an issues.

Once you’ve installed Growl, pop open the terminal and enter the following commands.

cd /Volumes/Growl 1.1.2/Extras/growlnotify
sudo sh install.sh

Give it a test run:

growlnotify -m “Testing growlnotify” Test

So if that doesn’t work, it’s probably because of an issue with the default permissions and Mac OS X 10.5 Leopard. Here’s what my permssions were. (I have no idea why the @ symbol is there)

cd /usr/local/bin
ls -l
-rwxr-xr-x@ 1 ryan 501 130288 Jan 22 17:38 growlnotify

To fix the permissions, I ran the following:

sudo chown root:admin growlnotify

Resulting in the following permssions:

cd /usr/local/bin
ls -l
-rwxr-xr-x@ 1 root admin 130288 Jan 22 17:38 growlnotify

Give it a test run again:

growlnotify -m “Testing growlnotify” Test

Now you can change directories to your rails app and run autotest and growlnotify should be working now..

cd ~/myrailsapp
autotest

References:
http://blog.codefront.net/2007/04/01/get-your-testing-results-via-growl-notifications/
http://www.danielfischer.com/2007/05/14/ruby-on-rails-bdd-with-autotest-growl-rspec/
http://railsontherun.com/2008/1/30/misc-tips-and-tricks/

Update: 2008-02-21 We’re looking into using ActiveMessaging and Amazon SQS to help with the workflow for background processing. Stay tuned for an updated post.

With before_save and after_save filters being so easy to use, it’s tempting to add more and more pre and post-processing to saving an ActiveRecord model. For Obsidian Portal, we update permissions, set timestamps of associated objects, and do all sorts of stuff. Unfortunately, all this extra work takes time, and can significantly slow down your application. The more work you do on the main execution thread, the more time Mongrel is tied up doing stuff unrelated to servicing requests. If something goes wrong in any of the filters, Rails will rollback the database transaction, and *poof* it’s all gone!

A while back, we started seeing ‘rbuf_fill’ timeout errors in the server logs. From what we could see, calls to acts_as_solr indexing were timing out, interrupting the save. For us, this was really bad. People would spend lots of time painstakingly crafting their perfect blog posting or wiki page, only to have it evaporate into nothing. All they saw was our default “Internal Server Error” page. Sure, it looks nice, but no one wants to see that

Tracing the timeout back to Solr was not hard, and the solution was clear: take the indexing out of the main execution thread and move it to a background process. Luckily, acts_as_solr made this a fairly easy refactoring process. Here’s what we did:

Add an :if clause to your acts_as_solr macro call

acts_as_solr supports an :if clause that will be used to determine whether or not the record will be indexed when save is called. We want this to always evaluate to false, except when we explicitly set it to true during off-line processing. Below is an example from one of our models:

acts_as_solr :fields => [:name, :body, :post_title, :post_tagline, :slug],
:if => :solr_index?

def solr_index?
@solr_index
end
attr_writer :solr_index

Use rake/cron to do the indexing in the background.

Now that indexing does not happen on save, we need to make sure it happens at some point. Our solution was to move it to a rake task that gets executed by a periodic cron job. Rake + cron has worked well for us in the past, so we’ll stick with it.

The task itself is very simple. Find all the objects that have been updated since the last indexing, and push them to Solr.

Below is the rake task that I wrote. If I were more clever, I would probably come up with a neat trick for automatically finding all the models that support Solr indexing. Now that I’m an official committer on acts_as_solr, maybe I’ll try to figure something out and get it into the trunk. Still…I’m lazy

namespace :solr do
namespace :index do
desc “Indexes campaigns”
task :campaigns => :environment do
index_class(Campaign)
end

desc “Indexes wiki pages”
task :wiki_pages => :environment do
index_class(WikiPage)
end

desc “Indexes game contents”
task :game_contents => :environment do
index_class(GameContent)
end

desc “Indexes users”
task :users => :environment do
index_class(User)
end

desc “Indexes everything that we’re storing in solr”
task :all => [:campaigns, :wiki_pages, :game_contents, :users]

def index_class(klass)
# If REBUILD is set to “true” then we rebuild the entire index
rebuild = ENV["REBUILD"] ? ENV["REBUILD"] == “true” : false

interval = rebuild ? 100.years : 30.minutes

objects = klass.find(:all,
:conditions => ["updated_at > ?", Time.now - interval],
:page => {:size => 20, :auto => true}
)

objects.each do |o|
puts(“Indexing #{klass.to_s}: #{o.id}”)
o.solr_index = true
o.solr_save
end
klass.solr_optimize

end
end
end

Set up a cron job to run this every thirty minutes or so. For most sites, a half hour will be a good balance between keeping the load down and making sure the searching is fairly up to date.

By moving the indexing off the main thread, we’ve noticed a significant reduction in the number of Solr related exceptions. That means our users have seen a significant reduction in the number of “Sorry, we lost all your data” errors, and that is exactly what we were hoping for.

References

• Solr homepage
• acts_as_solr Trac instance
• ActiveMessaging in Rails
• Obsidian Portal – Create your own D&D campaign website!

Moving to Rails 2.0 or above will break Comatose 0.8.1 (the latest as of this writing), but luckily it’s very easy to fix. In order to get running again, all you need to do is install two plugins that used to be part of Rails core:

script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_list/
script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_tree/

Once again, a great big thanks to Matt McCray for putting out such a useful plugin.

The built in Rails tools for selecting dates are adequate, but just barely. Most people are used to ticking dates off on calendars rather than choosing from a bunch of pulldown menus for year, month, day, and so on. Still, I’m lazy so that’s still what I’m using in most places.

Luckily, today I found a decent list of several date pickers. I haven’t tried any of them out yet, but any one of them would surely give Obsidian Portal a bit of a usability boost.

My podcasts / vidcasts at SD Ruby have been posted to the podcast section.

Simple Sidebar Plugin

How to use Simple Sidebar plugin to DRY up sidebar content in applications.

Related Blog Posts:

http://blog.aisleten.com/2007/06/03/simplesidebar-if-you-have-sidebars-you-need-this-plugin/

Ajax CSS Star Rating with ActsAsRateable

How to build an Ajax-powered, CSS star rater using the ActsAsRateable plugin and Komodo Media’s CSS Star Rating Redux technique.

Related Blog Posts:

http://blog.aisleten.com/2007/05/03/ajax-css-star-rating-with-acts_as_rateable/
http://blog.aisleten.com/2007/05/17/find-the-top-5-highest-rated-objects-with-acts_as_rateable/

At this month’s meeting we’re going to be having our first Rails Roundtable so come and check it out.
SD.rb December Meeting Schedule

The Atlanta Ruby Users Group (ATLRUG) recently moved their meetings to a much friendlier location for me. Rather than heading up into the forbidden north of Atlanta, it is now being held in the convenient location of Tech Square in Midtown. Of course this probably means nothing to anyone reading this from outside Atlanta.

In any case, it was a really great group, and they were very welcoming to new people, of which there were very many. Following the free pizza and pre-meeting chit-chat, we saw two presentations.

Metaprogramming

First up was Stephen Touset with a talk on metaprogramming. This is what is going on behind the scenes with all the method_missing stuff. Probably the most well known example would be the dynamic finders (ie. “find_by_email_and_username”). Stephen delved deep into metaclasses, and it was fairly confusing. I’ve dealt with class_eval and such before, but mainly from a “copy someone else’s stuff and modify to suit” sort of way. As I said during the meetup, “It’s possible to do metaprogramming without knowing what you’re doing.”

Nginx

Mark Percival followed Stephen with a presentation on Nginx, a fast little web server. I’ve always been a fan of Apache, but mainly because I felt safe with it. Even though it was a pain to configure, I knew that if I ever had to ask “How do I do X on the web?” then the answer would be in Apache language.

However, it never occurred to me that it might be overkill. As Mark explained, if all your Apache server is doing is proxying to mongrel, then it’s a big hammer for a tiny problem. The memory footprint is fairly high, and the configuration is a pain. Face it, Apache is not very friendly.

Mark claimed that Nginx was friendlier. To be fair, the config file was shorter. However, it still looked fairly arcane to me. I guess easy is in the eyes of the beholder.

Still, the memory issues are something I can’t ignore. Obsidian Portal runs on a VPS from Rimuhosting, so memory is a scarce resource. If I can reclaim enough memory to run another mongrel instance, that would be quite a coup. However, the real offender on my VPS is Tomcat. What!?! Tomcat!?! Well, I needed something to run Solr, and Tomcat is the only servlet container I’m familiar with. So, even though it’s a memory hog, it’ll have to stay for now.

Overall thoughts

All in all, the meetup was great. As I said, everyone was very nice, and all seemed fairly down to earth. There were no pissing contests over who was the better hacker, and newbie questions were answered honestly and humbly. I can’t wait until next month’s meeting. Maybe I’ll even try to scrape something together to present.

So you want on Technorati, huh? You’ve got a website with blogs or at least something that could be loosely considered a blog, and you want more exposure? Well, you’ve come to the right place! Step on in and I’ll tell you all my secrets of getting XML-RPC pinging to work for you, driving the hordes of the Internet to your doorstep! *

Getting onto the syndication services is not that hard. Or, at least that what’s they say. I have had a devil of a time getting the Obsidian Portal adventure logs to show up. By any standard definition they are blogs, and therefore should not be excluded from the syndication services. Still, here I am after a month with very little to show for it. Worst of all, it’s extremely difficult to track down where I’m going wrong.

Under the assumption that someone out there is in the same boat, here are a few tips to help out. I’ve grouped my tips into two categories. “Concrete Advice” covers tips that could definitely make a difference and should be attempted first. “Shots in the Dark” are some ideas I had that may or may not do anything at all. But if you’re desperate…

Concrete Advice

Targets

Rather than searching out all the services, just use pingomatic. You can add any additional ping targets if you wish, but pingomatic has found some pretty good ones.

Testing

Actually testing your pinging is probably the hardest part. You send a ping out into the ether, get a response like “Thanks for the ping!” and then you wait. And wait. And wait.

Sometimes you will have to wait several hours for the blog or post to show up on Technorati. Sometimes it won’t even show up at all. Not knowing if the ping is working is the most frustrating part of the entire experience. That’s where weblogs.com comes in.

Weblogs.com is your testing buddy!

Ever hear of weblogs.com? Neither had I, until I started this journey. Apparently, they’re the wackos who came up with this crazy pinging idea in the first place. To boot, they provide the best way of testing whether or not your ping service is working. They have a list of the most recent pings they’ve received available as an XML file. So, here’s how the testing works:

1. Send a ping to pingomatic. Verify that pingomatic responds correctly (ie. “Forwarding your ping to 16 services”).
2. Get a cup of coffee or something.
3. Download shortChanges.xml (use wget to avoid caching by your browser).
4. grep for the URL of the blog you pinged.

Note: shortChanges.xml seems to be cached on the server side and updated every couple minutes or so, so keep checking if you’re not there right away. After 5-10 minutes, you should either be listed or your ping never made it.

If it’s there, then you can be absolutely sure of 2 very important things:

1. Your ping to pingomatic was successfully received.
2. Pingomatic forwarded your ping to 1 other service successfully.

That may not seem like much, but we can infer (ie. assume) a lot more, namely that pingomatic is forwarding your pings to all the other services. This means that whatever problems you’re having getting registered with the syndication services, it’s not related to your pinging process. So, if you’re still not showing up on Technorati, it’s time to do some more digging.

Are you valid?

Ok, now your ping is working, what’s next? Validate your site and your feed!

The first thing a syndication site will do is pull down your feed and spider your site. You want to be as welcoming as possible when that happens. That means having valid, well-formed XHTML for your site and a valid RSS/Atom feed. Both of these are easy enough to check:

• RSS/Atom Feed Validator
• W3C XHTML Validator

They will tell you what’s wrong with your site. Get it whipped into shape so the syndication spiders find what was promised by your ping.

Extend your best foot forward

Since we’re using pingomatic, we have our choice of a regular ping or an extendedPing. Just go whole-hog and send the extendedPing. It allows you to specify both the site URL and the associated RSS/Atom URL. Send all the info you can to pingomatic, and let them decided what to forward on to the other guys, depending on who can accept it.

“Ping Test” is a crappy post title

When looking for your posts on the syndication sites, make it easier on yourself, and use a test post with easy to search for text. “Ping test” is going to lump you in with all the other people doing the exact same thing. Instead, stick in a string of nonsensical text like “flatly waking Oberon” that doesn’t show up in a Google phrase search. This will make your tests just a little bit easier to find.

Shots in the Dark

Tag, you’re it!

Add a few categories to your post in the RSS feed. A lot of spiders and search engines (as well as blog apps) seem to treat the category field as a place to dump social tags. So, you should too! Even if you have to hard code a few categories in there, go ahead and do it. For the Obsidian Portal adventure logs, every post is tagged with ‘games’, ‘gaming’, ‘rpgs’, and ‘roleplaying.’ Does it help them get picked up? I don’t know. That’s why it’s a shot in the dark.

Check your title tag for your blogs

For a while, I was adding “Obsidian Portal” to the title tag of every adventure log. For the fleeting moments when they were showing up on Technorati, it looked terrible. Suddenly, they all disappeared. It occurred to me that they might have been flagged as duplicates or too similar. Same domain and similar titles. Is that the case? Who knows?

Parting thoughts

I have not had much luck getting listed with Technorati or any of the other services. Sometimes it works, sometimes it doesn’t. After about a month of trying, I’m about ready to throw in the towel. I’ll keep the pings going, but I’m not going to devote much more time to testing and analyzing whether or not they’re working.

If you do manage to find the secret to getting listed, please speak up in the comments or write a blog post of your own!

References

• XML-RPC ping API – This is what you should be sending to pingomatic.
• weblogs.com shortChanges.xml – This lists all the pings that weblogs.com has received in the last 5 minutes.
• RSS 2.0 specification – Explains what a valid RSS feed should look like.

*Author is full of crap and is still unable to get his site’s blogs listed on Technorati. If you know what he’s doing wrong, please post a comment!

In an earlier post on the Atlanta Web Entrepreneurs meetup group, I mentioned that I had met some fellow Ruby on Rails hackers from here in Atlanta. They were working on a secret project and didn’t want to publicize it. I can definitely understand the feeling.

Fast forward a few months and now they’re tired of working in the dark. At a certain point, you just have to push it out there and sink or swim. In fact, their business depends on both sinking and swimming. Ha ha, lame. Anyways, if you’re a scuba diver or were ever interested in being one, check them out:

From the guys here at AisleTen, we wish the best of luck to MyNextDive May you find your niche and get your user base!

References

• MyNextDive – The actual site.
• Introducing MyNextDive – A press release for the launch.

We have finally gotten to the point where we are ready to start offering subscriptions to Obsidian Portal. We don’t expect there will be a lot of interest, but it’s always a sort of chicken v. egg problem. If you don’t have paying subscribers, then it’s not worth the effort to make the features. Conversely, without the features, no one is going to pay. On second thought, I guess it’s not chicken and egg, it’s pretty clear: you need features or no one will pay.

Asking for payment means you will need to be able to accept it. Currency on the web is passed almost exclusively via credit cards (except for PayPal…), so that’s the direction we need to go in. That requires us to select a credit card processor. For today, we will be looking at TrustCommerce.

I won’t go into the details of how credit card processing works, mainly because I don’t really understand it myself. Suffice it to say, there are a lot of middle-men, and they are all trying to take a cut. Each cut is either a percentage of the total charge or a flat fee or both. So, a typical fee structure might be $0.30 flat fee plus 2.5% of the total transaction.

Note: If you don’t care about the analysis and just want to see a rundown of their prices, then jump to the pricing.

Go easy on me; it’s my first time

When selecting a processing agent, our first priority right now is ease of use. We don’t expect there will be a lot of people signing up for our premium service, so we don’t want to expend a lot of effort on a payment system only to never see it used. Also, we’re willing to pay a higher rate to the processor since 3% of $30/month is a lot different than 3% of $30,000/month. I’ll pay 3% vs 2.5% if the 3% service takes 2 hours to implement and the 2.5% service takes 10. So, for us, ease of use trumps competitive pricing.

Since we’re talking about subscriptions as opposed to purchases, there is a recurring element to the payments. Since we want easy-to-implement solutions, we are scoping our search to only include the payment processors that offer a recurring service. This is a very important thing to note, especially if you’re in the same boat. A 1-time payment processor model (like Google Checkout) just will not work if you want to do subscriptions. The main reason is that you will have to store the users’ credit card info on your server in order to pass it to the payment processor each billing cycle. Do not do this! There are actual laws and regulations detailing what sort of security procedures you have to maintain in order to hold that sort of sensitive data. It’s much easier to simply pay someone else to deal with that crap. If you do choose to store their info in your database, you should be looking for a lawyer right now, not a payment processor.

Just plug in your credit card info

In Rails, ease of use means finding a plugin. I write a lot about plugins on this blog, so why should credit card processing be any different? Doing a quick Google search led me to the TrustCommerce subscription payment plugin.

Finding this bit of code brought a smile to my face, as I thought I had just finished 90% of the work. Sign up for an account, drop in the plugin, and wait for the money to roll in. Too bad there were a few red flags that derailed the money train.

Sitting by the phone

TrustCommerce does not list any pricing on their website. Instead, they say you have to sign up for a test account, and then you’ll be contacted. Not a big deal, I guess. So, I signed up for a test account.

The first red flag went up when I did not get an immediate callback. Sure, I signed up at 11:00pm Eastern Time, but that’s normal business hours in Internet time. In other words, if you’re an Internet company that requires phone contact, you had better have someone manning the phone at all hours. A lot of Web jockeys like me have a regular 9-5 job that precludes us from doing our business dealings during normal business hours. I want to deal with companies that understand this and have staff available during my normal business hours.

Red flags: 1

The ball sits in my court

The second red flag went up at their lackluster eventual response. My cell is in a dead zone at work, so whenever I leave for lunch, I get all my messages. On the day after requesting contact, I had a voicemail message from TrustCommerce. Still no pricing info, just a short message to call them back. Seeing as how I was busy, I couldn’t do it right away. Then I forgot. Dead silence on their end. No e-mails, no more calls, nothing.

Now a lot of people may disagree with me on this, but I think they should have been hitting my inbox and voicemail pretty hard. “Mr. Wedemeyer, we’re still interested in talking to you about blah blah.” or “Send us an e-mail with the best time to call you.” That’s how the mortgage people behaved when I used LendingTree. Sure, it was annoying, but you knew they wanted your business. To me, an anemic response indicates that someone isn’t really serious about recruiting me as a customer.

Red flags: 2

Little fish: prepare to get fried

When I finally did get in touch with someone from TrustCommerce, he was quite happy to answer my pricing questions. I don’t know if I’m allowed to post that info, but since they didn’t expressly forbid it, here you go:

Basic pricing

• $95 1-time fee
• $20 / month
• $0.20 / transaction

Citadel (recurring payments)

• $145 1-time fee
• $10 / month
• $0.10 / month / billing id (ie. subscription)

Holy crap! $240 just to get started, plus an additional $30 per month, just to be allowed to use their service? Seeing as how I expect Obsidian Portal to be making around $10 / month, at least until we can recruit more people, this is insane! I politely said thank you to the salesman, hung up the phone, and started writing this post.

I guess I see these huge front-loaded fees like this: If you’re making enough money that the fees don’t matter, then you already have a lot of subscribers, which means you’re already handling credit cards. Maybe their service is so great compared to the competition that it’s worth it for the big boys. But, if you’re a small time operator like me, forget about it.

Red flags: 240 + 30 / month

The search continues

Although I said pricing was not our top priority, the front loaded fees with TrustCommerce completely invalidate them as a viable option. It would be a very long time before we paid off the initial investment, and with our none-to-clear business prospects with Obsidian Portal, that’s a gamble I’m not willing to take.

In the next exciting chapter we will be looking at Amazon Flexible Payment System (FPS). This new web service from Amazon is meant to rival Google Checkout and PayPal. I’ve been extremely pleased with S3, and maybe they can do one better with FPS. Stay tuned to find out.

The correct use of singular vs plural nouns is something the Rails community is very familiar with. Your ‘User’ object resides in the “users” table. A User has_many ‘blogs’ while a Blog belongs_to a single ‘user’ You all know this already.

So with this much attention being paid to number in the back-end code, it’s pretty sloppy to disregard it in the views. “You have 1 friends” or “Last update: 1 weeks ago” just looks bad. Sure, we could write the code to fix it fairly easily, but we have better (ie. more fun) things to work on, right?

Luckily, Rails’s text helpers ride to the rescue once again. The pluralize method does exactly what we’re talking about. Simply give it a number and a singular spelling, and it will use either the singular or plural form based on the number. If it guesses the wrong plural form (ie. ‘mouse’ vs ‘mice’), you can explicitly supply the plural version.

You currently own < %= pluralize @blogs.size, 'blog' %>.

If you have a single blog, it will say “You currently own 1 blog.” For n > 1, it will say “You currently own n blogs.”

Simple and sweet, nothing more to say