Midnight Oil

We’re 95% done on RioFlexPay, a front-end graphical user interface (gui) to the Amazon Flexible Payment System. We’ve got a splash page up right now, and we’re getting ready to put the production server together in anticipation of a pre-June deployment.

It’s been a tough ride, but we’re almost there! In a few short days or weeks, managing your FPS account will get much, much easier.

Stay tuned!

If you’ve been following our blog you may have noticed that we’re using Solr and ActsAsSolr plugin for our searching.

Getting started with acts_as_solr
acts_as_solr for development and production in one Tomcat instance
Optimizing Solr and Rails - Index in the background

Unfortunately being Java, Solr is a bit of a memory hog. We’ve attempted all sorts of optimizations, but we’re going to take a leap and switch to Sphinx. Sphinx is a free open-source SQL full-text search engine.

First step is to get Sphinx itself installed. For that have a look at Rob’s post:
http://www.notch8.com/articles/2007/10/15/sphinx-and-ultrasphinx-and-eye-on-search

Ultrasphinx works pretty much out of the box with will_paginate; however, a lot of our work is currently compatible with paginating_find. So to keep rails memory footprint down by using one plugin we’re sticking with paginating_find. Unfortunately Ultrasphinx doesn’t work out of the box with paginating_find, so time for a little hacking…

Install paginating_find plugin:

piston import http://svn.cardboardrocket.com/paginating_find/paginating_find vendor/plugins/paginating_find

More information on paginating_find

Install ultrasphinx plugin:

piston import http://fauna.rubyforge.org/svn/ultrasphinx/trunk/ vendor/plugins/ultrasphinx

More information on Ultrasphinx

Now for hacking Ultrasphinx plugin to work with Paginating Find plugin. Piston is great for managing plugins; however, it can easily get confused if you’ve hacked a plugin directly. The solution: use a method called “Evil Twin” as mentioned on the Err the Blog: Evil Twin Plugin

Create a directory for the hack:

mkdir vendor/plugins/ultrasphinx_hacks touch vendor/plugins/ultrasphinx_hacks/init.rb

Now edit the vendor/plugins/ultrasphinx_hacks/init.rb file with the hacks for paginating_find:

# Hack to allow Ultrasphinx to work with pagination_find Ultrasphinx::Search.module_eval do def first_page 1 end def last_page self.page_count end def previous_page? previous_page ? true : false end def next_page? next_page ? true : false end end

Now for adding a method to your controller for the search:

app/controllers/posts_controller.rb

def search @query = h(params[:query]) begin @posts = Ultrasphinx::Search.new(:query => @query, :class_names => 'Post', :page => params[:page] || 1, :per_page => 20) @ posts.run rescue RuntimeError flash[:warning] = 'Search is currently disabled. Please try again in a few hours.' rescue NoMethodError flash[:notice] = 'No records found for this search' end render :action => 'list' end

app/views/posts/list.html.erb

<ul> < % @posts.each do |cog| %> <li>< %= posts.name %></li> < % end %> </ul> < %= paginating_links(@posts) %>

Paginating End Result

Bonus : will_paginate styled links

Unfortunately paginating_find doesn’t automatically have the “style” that will_paginate automatically comes with. So to fix that just add a little css for the style and a partial for the pervious and next buttons.

.pagination{ padding: 2px; } .pagination a, .pagination a:visited{ padding: 0 5px; border: 1px solid #9aafe5; text-decoration: none; color: #2e6ab1; } .pagination a:hover, .pagination a:active{ border: 1px solid #2b66a5; color: #000; } .pagination .currentpage{ font-weight: bold; padding: 0 5px; border: 1px solid navy; background-color: #2e6ab1; color: #FFF; } .pagination .disablepage{ padding: 0 5px; border: 1px solid #929292; color: #929292; display: inline; }

And now for the partial:

app/views/shared/_paginate.html.erb

< % if (collection.is_a?(PagingEnumerator) || collection.is_a?(Ultrasphinx::Search)) && (collection.page_count != collection.first_page) -%> <div class="pagination"> < % if collection.previous_page? -%> < %= link_to '« previous', url_for(:overwrite_params => { :page => collection.previous_page }) %> < % else -%> <div class="disablepage">« previous</div> < % end -%> < %= paginating_links(collection, {:window_size => 4, :always_show_anchors => true, :params => {:query=> params[:query]}} ) %> < % if collection.next_page? -%> < %= link_to 'next »', url_for(:overwrite_params => { :page => collection.next_page }) %> < % else -%> <div class="disablepage">next »</div> < % end -%> </div> < % end -%>

So in your code replace the code paginating_links call with a render partial:

# OLD VERSION < %= paginating_links(@posts) %> # NEW VERSION < %= render :partial => 'shared/paginate', :locals => {:collection => @posts} %>

Styled Paginating End Result

Resources:
ErrTheBlog: Evil Twin Plugin
Will Paginate Plugin
Paginating Find Plugin
Ultrasphinx
Sphinx

We’ve taken a small leap and added several microformats to Obsidian Portal. It was just so easy, there was no reason not to. For starters, we’re going with hCard and XFN.

Right from the start, we’re going whole-hog on hCard. I made a helper function to output an hCard every time we display a user’s avatar. So, every comment, every recent update, every friend link has an hCard on it. If the microformat-aware spiders cannot figure out who is who on Obsidian Portal, then they’re brain-dead.

Secondly, we’ve added XFN rels on the friend-links in the user’s profile. I don’t know of any (non-trivial) tools that are using XFN yet, but it’s easy enough to add, so we did. We’ve got rel=me on links to user’s other websites, and even self-referential rel=me links on the profile pages. We’re basically screaming “I am me” at the spiders.

Still on the TODO list is rel-tag for the NPC character tags. We’ve got a lot of plans for tags in the future, we just need to free up some time to work.

In the future, I might look at adding hAtom for the Adventure Logs, but honestly, we’ve already got RSS feeds, and I can’t see the value-add here. Am I missing something? Let me know.

Examples

• My profile - hCard and XFN enabled
• The homepage - Notice the hCard for each recent-update user
• An Adventure Log post - Notice the hCards for the comments

Resources

• Microformats.org - The definitive source for information.
• Operator - A microformat parser plugin for Firefox. Adds a nice little toolbar that lights up when microformats are detected.
• Tails Export - Another microformat parser plugin for Firefox. Seems to handle de-duping better than Operator.
• rel-lint - A validator / checker for XFN and rel-tag

I got a new job! Starting April 23rd, I will be a Senior Software Engineer at Vitrue, helping them develop their embeddable video platform. I honestly don’t know a lot of details yet, but it seems to be a themeable, embeddable YouTube-like video platform. The twist is that companies can keep complete control over the posted content. From a corporate standpoint, that seems a little easier to stomach than the wild-wild-west of YouTube.

Best of all, I’ll be networking and meeting other young entrepreneurs like myself. The best place to meet startup-oriented people is, you guessed it, at a startup.

My greatest fear is that I will have less and less time to work on Obsidian Portal and RioFlexPay (link coming soon…). Oh well, I can always sleep less.

Before we switched to Slicehost, we had our VPS on Rimuhosting. I must say that I was very pleased with Rimuhosting’s service, and we never had any problems with stability. We eventually had to switch to Slicehost anyway because it was simply cheaper and we’re on razor-thin margins here.

Still, switching hosts is not a trivial process, and you have to factor in the switching effort when deciding if the price differential is worth it. If you’ve found a cheaper host and are thinking about jumping ship, the very first thing you should do is contact your current provider and see if they’ll match the new host’s price. That’s what we did initially with Rimuhosting. They didn’t match the deal, but they did give us a little more RAM for free. That tided us over for another month or two before we decided we wanted to give Slicehost a shot.

So, for posterity, and to help other bargain hunters, here is the letter I sent, and Rimuhosting’s response.

I’ve been a Rimuhosting customer for several months now. In general I am happy with the service, although my server seems to be getting more and more sluggish in recent weeks.

Due to the performance issues, I began shopping around for alternatives and I came across Slicehost ( http://www.slicehost.com/ ). Since they’re a competitor, I’m sure you’re already familiar with them.

According to their price list, they will give me 256MB RAM plus more disk space for $20/mo, compared to the 160MB of RAM I’m getting for $29. For me, the RAM is the big deal, and the extra storage is just icing on the cake.

I am seriously considering moving to slicehost, as I am finding that the 160MB I currently have is eaten quickly by Tomcat, mongrel, and MySQL. Plus, as I mentioned earlier, I have been seeing some very serious performance issues lately, with slowdowns on ssh as well as terrible response times from the web server. This could be due to not enough memory on my VPS, but I’m concerned it has to do with too few CPU cycles being devoted to the VPS.

In all honesty, I would rather not switch hosting companies, as I have been pleased with the level of service from Rimuhosting. In addition, moving a VPS installation is no small task and I do not relish the idea of trying to replicate all my settings on a new machine. Still, Slicehost’s pricing plan is very attractive.

Before I make any decisions, I would like to ask what sort of plan you would be willing to offer in order to compete with Slicehost. If you could offer a combination of more RAM and a lower price, it would go a long way towards tipping the scales in Rimuhosting’s favor.

Thanks for your time and I look forward to hearing from you.

Micah Wedemeyer
http://www.aisleten.com

Their response to me:

Hi Micah,

I’ve added some additional memory to your VPS:

[root@obsidianportal ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           300        148        151          0         16         55
-/+ buffers/cache:         76        223
Swap:           95          0         95

Also moved your current pricing from 39.95, to where our current MiroVPS2 plans are, at 29.95 a month.

We feel that we offer a good value for the service we provide. We appreciate your business and hope you will stay with us.

I’m a big fan of Behavior Driven Development (BDD). It really illustrates how one little change in your code can have significant impact on the rest of you application, which you would never be able to catch without testing. Not to to mention the benefit of being able to write code and make sure it works even if you don’t have production data available.

So, upon completing a clean upgrade to Leopard I noticed that after installing Growl and growlnotify, that growlnotify would not work even through Growl itself was working.

1) First let’s set up the .autotest file. For this you’ll need the gems rspec (1.1.3), ZenTest (3.9.1), and redgreen since ZenTest change how it handled exceptions in 3.9.

sudo gem install rspec sudo gem install ZenTest sudo gem install redgreen

Now open up your ~/.autotest file in Textmate

mate ~/.autotest

Paste in the following code. Notice the exceptions at the bottom of the file that really helps speed up autotests as well as keeps your cpu usage low. My Macbook would get really hot really fast prior to using this execptions.

require 'autotest/redgreen' module Autotest::Growl def self.growl title, msg, img, pri=0, stick="" system "growlnotify -n autotest --image #{img} -p #{pri} -m #{ msg.inspect} #{title} #{stick}" end Autotest.add_hook :ran_command do |autotest| filtered = autotest.results.grep(/\d+\s.*examples?/) output = filtered.empty? ? “” : filtered.last.slice(/(\d+)\s.*examples?,\s(\d+)\s.*failures?(?:,\s(\d+)\s.*pending)?/) if output =~ /[1-9]\sfailures?/ growl "Test Results", "#{output}", "~/Library/autotest/rails_fail.png" elsif output =~ /pending/ growl "Test Results", "#{output}", "~/Library/autotest/rails_pending.png" else growl "Test Results", "#{output}", "~/Library/autotest/rails_ok.png" end end end Autotest.add_hook :initialize do |at| %w{.svn .hg .git vendor}.each {|exception| at.add_exception(exception)} end

Next, you’ll want to download the Pass, Fail, and Pending images below:

  

First, the prerequisite to growlnotify is Growl. You can download it from http://growl.info/. It’s a graphical installer, so you shouldn’t have an issues.

Once you’ve installed Growl, pop open the terminal and enter the following commands.

cd /Volumes/Growl 1.1.2/Extras/growlnotify sudo sh install.sh

Give it a test run:

growlnotify -m "Testing growlnotify" Test

So if that doesn’t work, it’s probably because of an issue with the default permissions and Mac OS X 10.5 Leopard. Here’s what my permssions were. (I have no idea why the @ symbol is there)

cd /usr/local/bin ls -l -rwxr-xr-x@ 1 ryan 501 130288 Jan 22 17:38 growlnotify

To fix the permissions, I ran the following:

sudo chown root:admin growlnotify

Resulting in the following permssions:

cd /usr/local/bin ls -l -rwxr-xr-x@ 1 root admin 130288 Jan 22 17:38 growlnotify

Give it a test run again:

growlnotify -m "Testing growlnotify" Test

Now you can change directories to your rails app and run autotest and growlnotify should be working now..

cd ~/myrailsapp autotest

References:
http://blog.codefront.net/2007/04/01/get-your-testing-results-via-growl-notifications/
http://www.danielfischer.com/2007/05/14/ruby-on-rails-bdd-with-autotest-growl-rspec/
http://railsontherun.com/2008/1/30/misc-tips-and-tricks/

Recently, I noticed that the LoneGM had put together a pretty good review of ObsidianPortal. He focuses on it from a play-by-post perspective, which is not exactly what it’s for, but still gives us a good rating.

Like me, it seems that most of the people using OP can see what it should be, even if it’s not there yet. We’ve all been searching for a good campaign management tool for a long time, and building it up to match that perfect ideal will be a long road.

Update: 2008-02-21 We’re looking into using ActiveMessaging and Amazon SQS to help with the workflow for background processing. Stay tuned for an updated post.

With before_save and after_save filters being so easy to use, it’s tempting to add more and more pre and post-processing to saving an ActiveRecord model. For Obsidian Portal, we update permissions, set timestamps of associated objects, and do all sorts of stuff. Unfortunately, all this extra work takes time, and can significantly slow down your application. The more work you do on the main execution thread, the more time Mongrel is tied up doing stuff unrelated to servicing requests. If something goes wrong in any of the filters, Rails will rollback the database transaction, and *poof* it’s all gone!

A while back, we started seeing ‘rbuf_fill’ timeout errors in the server logs. From what we could see, calls to acts_as_solr indexing were timing out, interrupting the save. For us, this was really bad. People would spend lots of time painstakingly crafting their perfect blog posting or wiki page, only to have it evaporate into nothing. All they saw was our default “Internal Server Error” page. Sure, it looks nice, but no one wants to see that

Tracing the timeout back to Solr was not hard, and the solution was clear: take the indexing out of the main execution thread and move it to a background process. Luckily, acts_as_solr made this a fairly easy refactoring process. Here’s what we did:

Add an :if clause to your acts_as_solr macro call

acts_as_solr supports an :if clause that will be used to determine whether or not the record will be indexed when save is called. We want this to always evaluate to false, except when we explicitly set it to true during off-line processing. Below is an example from one of our models:

acts_as_solr :fields => [:name, :body, :post_title, :post_tagline, :slug], :if => :solr_index? def solr_index? @solr_index end attr_writer :solr_index

Use rake/cron to do the indexing in the background.

Now that indexing does not happen on save, we need to make sure it happens at some point. Our solution was to move it to a rake task that gets executed by a periodic cron job. Rake + cron has worked well for us in the past, so we’ll stick with it.

The task itself is very simple. Find all the objects that have been updated since the last indexing, and push them to Solr.

Below is the rake task that I wrote. If I were more clever, I would probably come up with a neat trick for automatically finding all the models that support Solr indexing. Now that I’m an official committer on acts_as_solr, maybe I’ll try to figure something out and get it into the trunk. Still…I’m lazy

namespace :solr do namespace :index do desc "Indexes campaigns" task :campaigns => :environment do index_class(Campaign) end desc "Indexes wiki pages" task :wiki_pages => :environment do index_class(WikiPage) end desc "Indexes game contents" task :game_contents => :environment do index_class(GameContent) end desc "Indexes users" task :users => :environment do index_class(User) end desc "Indexes everything that we're storing in solr" task :all => [:campaigns, :wiki_pages, :game_contents, :users] def index_class(klass) # If REBUILD is set to "true" then we rebuild the entire index rebuild = ENV["REBUILD"] ? ENV["REBUILD"] == "true" : false interval = rebuild ? 100.years : 30.minutes objects = klass.find(:all, :conditions => ["updated_at > ?", Time.now - interval], :page => {:size => 20, :auto => true} ) objects.each do |o| puts("Indexing #{klass.to_s}: #{o.id}") o.solr_index = true o.solr_save end klass.solr_optimize end end end

Set up a cron job to run this every thirty minutes or so. For most sites, a half hour will be a good balance between keeping the load down and making sure the searching is fairly up to date.

By moving the indexing off the main thread, we’ve noticed a significant reduction in the number of Solr related exceptions. That means our users have seen a significant reduction in the number of “Sorry, we lost all your data” errors, and that is exactly what we were hoping for.

References

• Solr homepage
• acts_as_solr Trac instance
• ActiveMessaging in Rails
• Obsidian Portal - Create your own D&D campaign website!

I saw an article in the New York Times the other day about working 10 hours a week and making $10 million a year. The article profiles Markus Frind, who started an online dating website, Plenty of Fish. The main thrust of the article is that Markus barely works at all and just sits back raking in cash. Are there any other web entrepreneurs out there who are sick and tired of hearing this story over and over? This is all I ever hear, and I secretly dream about it while working on my projects, but it doesn’t match up well with the real world of late nights and no money.

Hmm, let’s look through the NY Times article and see what the keys are to creating a cash-cow website:

• Just sit down and create a site as an experiment in teaching yourself a new programming language.
• Ignore the interface and usability. If stuff looks bad, just say, “Users don’t care about that.”
• Forget about customer service and moderation. Crowdsource it to the forums.
• Don’t charge users for anything. Rely 100% on advertisements.
• Lie in your hammock and collect checks.

Ok, I’ll admit, this is the life I want to live. This is the dream of all us web guys. However, I’m starting to suspect that the vast majority of “successful” web entrepreneurs invest a lot more effort and reap much smaller rewards. How many of us out there are pulling in $50, $100, or dare I say it, $500 a month from our projects? If I could get Obsidian Portal to generate $100 a month, I would be ecstatic. After paying off the hosting fees and whatnot, my take-home would probably be less then $1/hr. Still, it’s a goal to shoot for.

We can’t all be Digg, YouTube, or Facebook, but that’s OK. Success has different levels, and if we can just generate enough income to justify the amount of time we spend working on the sites we love, then that’s success. I just wish the media would profile a few more of us who live on the wrong side of profitability, work until 3:00 in the morning, and jump for joy with every subscription or CafePress T-shirt we sell.

Are you like me, barely scraping by (or not, as it were)? Drop a comment when you take a break from furiously writing code or begging someone not to cancel their account…

Moving to Rails 2.0 or above will break Comatose 0.8.1 (the latest as of this writing), but luckily it’s very easy to fix. In order to get running again, all you need to do is install two plugins that used to be part of Rails core:

script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_list/
script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_tree/

Once again, a great big thanks to Matt McCray for putting out such a useful plugin.