Midnight Oil

Recently for Obsidian Portal, we decided that we wanted to move some particularly intensive graphic processing offline from the main server. It was simply consuming too much CPU time and making the entire site sluggish. Luckily for us, it was already set up as a background process that only needed to connect to the database and Amazon S3. So, to do it on a whole new server wouldn’t require changing any of the algorithms.

However, since the remote server would be connecting to the main database over the Internet, we decided that encrypting the communication was probably a good idea. None of the actual data is all that sensitive, but the database username/password definitely is. Plus, in general, our policy is that any communication to and from our server should be encrypted.

It turns out that enabling SSL in MySQL is not too hard, but there are a lot of steps to follow. Further, to Rails docs on using database.yml to set up the connection aren’t that great (big surprise there). So, to help out those who are in the same boat, here’s what I did.

Setting up the MySQL server

The first thing to do is read through the official MySQL docs on SSL connections. These provide a good overview on how to configure the server to allow for (or force) encrypted connections. However, they assume a little knowledge of SSL and CA’s and keys and whatnot. So, if you’re stumped, you can read the following steps on how to proceed.

Verify that SSL is supported

We’re running Ubuntu on our server, and the MySQL that comes with it has SSL support already compiled in, so that’s a big relief on our part. I hate compiling from source, especially something big and important like MySQL.

To verify that support is already compiled in, log in with the mysql client and try the following:

show variables like ‘have_ssl’;

If it says DISABLED, then you’re in the right place. If it says YES, then you’ve already set the server up and can skip to the client or Rails sections below. If it says anything else (like no variables are returned) then it’s time to recompile MySQL. That’s beyond the scope here, but I wish you the best of luck.

Create the required SSL keys and certificates and whatnot

I’m no security expert, so all the SSL / CA / certificate / key advice is at your own risk. I’m still learning a lot of this stuff.

In order to get your server set up, you will need 3 files: A certification authority (CA), a certificate, and a key. Like I said, I really don’t know what all these things are. I just have a vague understanding.

To create my necessary files, I used TinyCA2, which I heavily recommend. It provides a GUI for using OpenSSL. Otherwise, get ready for lots of arcane command lines. If you’re on debian/ubuntu, all you need to do is run the following:

sudo apt-get install tinyca
tinyca2

Using TinyCA2, the process is a snap. It will walk you through creating a CA, then generating a certificate and key from that CA. I don’t know what options are required, but I got away with specifying only a common name for the CA and the certificate. Plus, I also used 1024 bit encryption since I’m not sure what level MySQL supports. I also heard somewhere that the common name for your CA and the certificate should be different, so watch out for that.

Once you’ve created your CA, certificate, and key, then you need to export them as pem files. In order to get MySQL to read the key, I had to export the key without a password. This is generally very bad advice, since if anyone gets the key they can pose as you. However, if the key is password locked, then MySQL would have to get the password from you somehow (Apache does this on startup), and maybe that’s just not supported. Please correct me if I’m wrong.

Configure MySQL to use the generated files

Copy the 3 files into /etc/mysql and then edit /etc/mysql/my.cnf Add the following lines:

ssl-ca=/etc/mysql/cacert.pem
ssl-cert=/etc/mysql/my-new-server-cert.pem
ssl-key=/etc/mysql/my-new-server-key.pem

Note! Make sure you check the file permissions to ensure that the mysql user can read the files. If you followed my previous bad advice and exported your key without a password, then it’s extremely important to strictly control the read permissions on these files. Chown them to the mysql user or use groups. Do not just chmod 777 and blissfully continue.

Restart your MySQL server and make sure there are no startup errors.

Test that everything works

The first thing to do is log in on the mysql client and check the have_ssl variable. Just run the following from the mysql client.

show variables like ‘have_ssl’;

If it says YES, then you’re good to go. If not, something went wrong and you need to retrace your steps to find out what’s up.

Client setup and testing

Now that the server is setup, let’s verify that we can connect with a client. The first thing to do is create a new user that only has SSL connection availability. Connect as root (or someone with grant privileges) and run the following:

GRANT ALL on somedatabase.* TO ‘ssluser’@'localhost’ IDENTIFIED BY ‘some_password’ REQUIRE SSL;

This will create a user that can only connect from localhost and must use SSL. Now, to make sure that everything is working, try this:

mysql -ussluser -p –ssl-ca=/etc/mysql/cacert.pem

If you are able to log in and don’t get the dreaded SSL ERROR message, then everything is great! If you do get an error, the first thing to check is the read permissions on the cacert.pem file. It must be readable by the current user. If your read permissions are set correctly and you’re still getting errors…sorry I can’t help.

Before we move on, it’s important to note that we’re using the same CA pem file as we did to create the server’s certificate and key. I really don’t understand why MySQL clients are required to specify a CA certificate, and I don’t know which are allowed. Presumably, you can set the server to allow clients to specify one of the major CAs (like Verisign or GoDaddy). Still, that’s beyond my knowledge. I tried creating a second CA and specifying that on the client side, but the server refused the connection. For now, it seems that you will simply have to copy the CA certificate to every remote client that wishes to connect to this server. That seems strange to me, and I’m probably wrong here. Please correct me in the comments.

Connecting from Rails

A small bump in the road…

The official Rails MysqlAdapter documentation lists the SSL parameters to use in database.yml. Unfortunately, the docs seem to be out of date and missing a very important parameter, sslca, the certificate authority file we need for every SSL client connection.

Further, the actual adapter code will not set any of the ssl parameters unless the sslkey option is set. This parameter is unnecessary in many cases, such as where you only want to require SSL communication and not X509 authentication of the client.

I have created a ticket and patch at lighthouse to cover this issue, but for now, you’re going to have to take a few extra steps to get things working.

Create a client key and certificate

While not strictly necessary according to MySQL, due to the coding of the Rails MySQL Adapter, you will need a client key and certificate. Like I said, I’ve submitted a patch, but no sense waiting on that.

Fire up TinyCA2 again and create a new certificate (select new client certificate and key) using the same CA that you used to create the server’s certificate and key. Again, export both the certificate (call it something like mysql-client-cert.pem) and the key (mysql-client-key.pem). Also, remember to export the key without a password!

Put these somewhere accessible to your Rails app. I will assume that you put them in the db directory. Make sure they are readable by your Rails app’s web server user.

Finally, place a copy of your cacert.pem in the db directory as well. Using TinyCA2, just go to the CA tab and click the export button. Drop in the db directory and check the file permissions.

Update your database user permissions

Update your Rails app’s user in mysql to give them remote access permissions. Assuming they are currently set to access only from localhost, the following line will extend access to your remote client.

GRANT ALL on my_rails_app_db.* TO ‘my_rails_app_db_user’@'my.remote.client.com’ IDENTIFIED BY ‘somepassword’ REQUIRE SSL;

Check your MySQL options and firewall

Make sure your firewall is set to allow incoming connections on the MySQL port (defaul 3306) and that your MySQL server is set to allow connections from more than just localhost.

Test using the bare mysql client

At this point, I would test that MySQL is accepting outside SSL connections by trying to connect using the mysql client from the remote machine. Something like:

mysql -umy_rails_app_db_user -psomepassword -hmy.mysql.server –ssl-ca=/path/to/rails/app/db/cacert.pem

If you cannot successfully connect using this, then you’ll need to troubleshoot your remote connection before trying to do anything with the Rails connection.

Update your database.yml

Add the following lines to your database.yml

sslca: /path/to/rails/app/db/cacert.pem
sslkey: /path/to/rails/app/db/mysql-client-key.pem
sslcert: /path/to/rails/app/db/mysql-client-cert.pem

Fire it up!

Fire up a Rails console on your remote client. If all goes well, you will be presented with the standard console prompt. You can verify that everything is working by executing the following

ActiveRecord::Base.connection.execute(“show status like ‘Ssl_cipher’;”).fetch_row

If you see something like DHE-RSA-AES256-SHA, then you’re set!

Congratulations! Your communications are now encrypted!

File summary

Since we’re dealing with so darn many pem files, I thought it might be nice to have an index of exactly which files you need and where you need them.

On the MySQL server

/etc/mysql/cacert.pem
/etc/mysql/mysql-server-cert.pem
/etc/mysql/mysql-server-key.pem
All files must be readable by the MySQL user

On the remote client

/path/to/rails/app/db/cacert.pem
/path/to/rails/app/db/mysql-client-key.pem
/path/to/rails/app/db/mysql-client-cert.pem
All files must be readable by the Rails webserver user

The cacert.pem files must be the same, and all the certificates and keys must be generated using the cacert (CA). In addition, the keys must be exported without passwords. Divert from this at your own risk!

Correct me please!

As I have said, I am not a security expert. If there is anything in here that is bad practice or just blatantly wrong, please correct me! The official documentation isn’t great, so it’s up to us (the community) to help each other out.

Help us climb

If you like this article, please link to it with the text AisleTen’s guide to connecting to MySQL using SSL encryption in Ruby on Rails. A few incoming links like that will help this article be near the top when people google for MySQL and SSL and Rails.

If you’ve been following our blog you may have noticed that we’re using Solr and ActsAsSolr plugin for our searching.

Getting started with acts_as_solr
acts_as_solr for development and production in one Tomcat instance
Optimizing Solr and Rails – Index in the background

Unfortunately being Java, Solr is a bit of a memory hog. We’ve attempted all sorts of optimizations, but we’re going to take a leap and switch to Sphinx. Sphinx is a free open-source SQL full-text search engine.

First step is to get Sphinx itself installed. For that have a look at Rob’s post:
http://www.notch8.com/articles/2007/10/15/sphinx-and-ultrasphinx-and-eye-on-search

Ultrasphinx works pretty much out of the box with will_paginate; however, a lot of our work is currently compatible with paginating_find. So to keep rails memory footprint down by using one plugin we’re sticking with paginating_find. Unfortunately Ultrasphinx doesn’t work out of the box with paginating_find, so time for a little hacking…

Install paginating_find plugin:

piston import http://svn.cardboardrocket.com/paginating_find/paginating_find vendor/plugins/paginating_find

More information on paginating_find

Install ultrasphinx plugin:

piston import http://fauna.rubyforge.org/svn/ultrasphinx/trunk/ vendor/plugins/ultrasphinx

More information on Ultrasphinx

Now for hacking Ultrasphinx plugin to work with Paginating Find plugin. Piston is great for managing plugins; however, it can easily get confused if you’ve hacked a plugin directly. The solution: use a method called “Evil Twin” as mentioned on the Err the Blog: Evil Twin Plugin

Create a directory for the hack:

mkdir vendor/plugins/ultrasphinx_hacks
touch vendor/plugins/ultrasphinx_hacks/init.rb

Now edit the vendor/plugins/ultrasphinx_hacks/init.rb file with the hacks for paginating_find:

# Hack to allow Ultrasphinx to work with pagination_find
Ultrasphinx::Search.module_eval do
def first_page
1
end

def last_page
self.page_count
end

def previous_page?
previous_page ? true : false
end

def next_page?
next_page ? true : false
end
end

Now for adding a method to your controller for the search:

app/controllers/posts_controller.rb

def search
@query = h(params[:query])
begin
@posts = Ultrasphinx::Search.new(:query => @query,
:class_names => ‘Post’,
:page => params[:page] || 1, :per_page => 20)
@ posts.run
rescue RuntimeError
flash[:warning] = ‘Search is currently disabled. Please try again in a few hours.’
rescue NoMethodError
flash[:notice] = ‘No records found for this search’
end
render :action => ‘list’
end

app/views/posts/list.html.erb

< % @posts.each do |cog| %>
• < %= posts.name %>

< % end %>

< %= paginating_links(@posts) %>

Paginating End Result

Bonus : will_paginate styled links

Unfortunately paginating_find doesn’t automatically have the “style” that will_paginate automatically comes with. So to fix that just add a little css for the style and a partial for the pervious and next buttons.

.pagination{
padding: 2px;
}
.pagination a, .pagination a:visited{
padding: 0 5px;
border: 1px solid #9aafe5;
text-decoration: none;
color: #2e6ab1;
}

.pagination a:hover, .pagination a:active{
border: 1px solid #2b66a5;
color: #000;
}

.pagination .currentpage{
font-weight: bold;
padding: 0 5px;
border: 1px solid navy;
background-color: #2e6ab1;
color: #FFF;
}

.pagination .disablepage{
padding: 0 5px;
border: 1px solid #929292;
color: #929292;
display: inline;
}

And now for the partial:

app/views/shared/_paginate.html.erb

< % if (collection.is_a?(PagingEnumerator) || collection.is_a?(Ultrasphinx::Search)) && (collection.page_count != collection.first_page) -%>

< % end -%>

So in your code replace the code paginating_links call with a render partial:

# OLD VERSION
< %= paginating_links(@posts) %>
# NEW VERSION
< %= render :partial => ‘shared/paginate’, :locals => {:collection => @posts} %>

Styled Paginating End Result

Resources:
ErrTheBlog: Evil Twin Plugin
Will Paginate Plugin
Paginating Find Plugin
Ultrasphinx
Sphinx

I’m a big fan of Behavior Driven Development (BDD). It really illustrates how one little change in your code can have significant impact on the rest of you application, which you would never be able to catch without testing. Not to to mention the benefit of being able to write code and make sure it works even if you don’t have production data available.

So, upon completing a clean upgrade to Leopard I noticed that after installing Growl and growlnotify, that growlnotify would not work even through Growl itself was working.

1) First let’s set up the .autotest file. For this you’ll need the gems rspec (1.1.3), ZenTest (3.9.1), and redgreen since ZenTest change how it handled exceptions in 3.9.

sudo gem install rspec
sudo gem install ZenTest
sudo gem install redgreen

Now open up your ~/.autotest file in Textmate

mate ~/.autotest

Paste in the following code. Notice the exceptions at the bottom of the file that really helps speed up autotests as well as keeps your cpu usage low. My Macbook would get really hot really fast prior to using this execptions.

require ‘autotest/redgreen’

module Autotest::Growl
def self.growl title, msg, img, pri=0, stick=”"
system “growlnotify -n autotest –image #{img} -p #{pri} -m #{ msg.inspect} #{title} #{stick}”
end

Autotest.add_hook :ran_command do |autotest|
filtered = autotest.results.grep(/\d+\s.*examples?/)
output = filtered.empty? ? ” : filtered.last.slice(/(\d+)\s.*examples?,\s(\d+)\s.*failures?(?:,\s(\d+)\s.*pending)?/)
if output =~ /[1-9]\sfailures?/
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_fail.png”
elsif output =~ /pending/
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_pending.png”
else
growl “Test Results”, “#{output}”, “~/Library/autotest/rails_ok.png”
end
end

end

Autotest.add_hook :initialize do |at|
%w{.svn .hg .git vendor}.each {|exception| at.add_exception(exception)}
end

Next, you’ll want to download the Pass, Fail, and Pending images below and place them in ~/Library/autotest:

First, the prerequisite to growlnotify is Growl. You can download it from http://growl.info/. It’s a graphical installer, so you shouldn’t have an issues.

Once you’ve installed Growl, pop open the terminal and enter the following commands.

cd /Volumes/Growl 1.1.2/Extras/growlnotify
sudo sh install.sh

Give it a test run:

growlnotify -m “Testing growlnotify” Test

So if that doesn’t work, it’s probably because of an issue with the default permissions and Mac OS X 10.5 Leopard. Here’s what my permssions were. (I have no idea why the @ symbol is there)

cd /usr/local/bin
ls -l
-rwxr-xr-x@ 1 ryan 501 130288 Jan 22 17:38 growlnotify

To fix the permissions, I ran the following:

sudo chown root:admin growlnotify

Resulting in the following permssions:

cd /usr/local/bin
ls -l
-rwxr-xr-x@ 1 root admin 130288 Jan 22 17:38 growlnotify

Give it a test run again:

growlnotify -m “Testing growlnotify” Test

Now you can change directories to your rails app and run autotest and growlnotify should be working now..

cd ~/myrailsapp
autotest

References:
http://blog.codefront.net/2007/04/01/get-your-testing-results-via-growl-notifications/
http://www.danielfischer.com/2007/05/14/ruby-on-rails-bdd-with-autotest-growl-rspec/
http://railsontherun.com/2008/1/30/misc-tips-and-tricks/

Update: 2008-02-21 We’re looking into using ActiveMessaging and Amazon SQS to help with the workflow for background processing. Stay tuned for an updated post.

With before_save and after_save filters being so easy to use, it’s tempting to add more and more pre and post-processing to saving an ActiveRecord model. For Obsidian Portal, we update permissions, set timestamps of associated objects, and do all sorts of stuff. Unfortunately, all this extra work takes time, and can significantly slow down your application. The more work you do on the main execution thread, the more time Mongrel is tied up doing stuff unrelated to servicing requests. If something goes wrong in any of the filters, Rails will rollback the database transaction, and *poof* it’s all gone!

A while back, we started seeing ‘rbuf_fill’ timeout errors in the server logs. From what we could see, calls to acts_as_solr indexing were timing out, interrupting the save. For us, this was really bad. People would spend lots of time painstakingly crafting their perfect blog posting or wiki page, only to have it evaporate into nothing. All they saw was our default “Internal Server Error” page. Sure, it looks nice, but no one wants to see that

Tracing the timeout back to Solr was not hard, and the solution was clear: take the indexing out of the main execution thread and move it to a background process. Luckily, acts_as_solr made this a fairly easy refactoring process. Here’s what we did:

Add an :if clause to your acts_as_solr macro call

acts_as_solr supports an :if clause that will be used to determine whether or not the record will be indexed when save is called. We want this to always evaluate to false, except when we explicitly set it to true during off-line processing. Below is an example from one of our models:

acts_as_solr :fields => [:name, :body, :post_title, :post_tagline, :slug],
:if => :solr_index?

def solr_index?
@solr_index
end
attr_writer :solr_index

Use rake/cron to do the indexing in the background.

Now that indexing does not happen on save, we need to make sure it happens at some point. Our solution was to move it to a rake task that gets executed by a periodic cron job. Rake + cron has worked well for us in the past, so we’ll stick with it.

The task itself is very simple. Find all the objects that have been updated since the last indexing, and push them to Solr.

Below is the rake task that I wrote. If I were more clever, I would probably come up with a neat trick for automatically finding all the models that support Solr indexing. Now that I’m an official committer on acts_as_solr, maybe I’ll try to figure something out and get it into the trunk. Still…I’m lazy

namespace :solr do
namespace :index do
desc “Indexes campaigns”
task :campaigns => :environment do
index_class(Campaign)
end

desc “Indexes wiki pages”
task :wiki_pages => :environment do
index_class(WikiPage)
end

desc “Indexes game contents”
task :game_contents => :environment do
index_class(GameContent)
end

desc “Indexes users”
task :users => :environment do
index_class(User)
end

desc “Indexes everything that we’re storing in solr”
task :all => [:campaigns, :wiki_pages, :game_contents, :users]

def index_class(klass)
# If REBUILD is set to “true” then we rebuild the entire index
rebuild = ENV["REBUILD"] ? ENV["REBUILD"] == “true” : false

interval = rebuild ? 100.years : 30.minutes

objects = klass.find(:all,
:conditions => ["updated_at > ?", Time.now - interval],
:page => {:size => 20, :auto => true}
)

objects.each do |o|
puts(“Indexing #{klass.to_s}: #{o.id}”)
o.solr_index = true
o.solr_save
end
klass.solr_optimize

end
end
end

Set up a cron job to run this every thirty minutes or so. For most sites, a half hour will be a good balance between keeping the load down and making sure the searching is fairly up to date.

By moving the indexing off the main thread, we’ve noticed a significant reduction in the number of Solr related exceptions. That means our users have seen a significant reduction in the number of “Sorry, we lost all your data” errors, and that is exactly what we were hoping for.

References

• Solr homepage
• acts_as_solr Trac instance
• ActiveMessaging in Rails
• Obsidian Portal – Create your own D&D campaign website!

Moving to Rails 2.0 or above will break Comatose 0.8.1 (the latest as of this writing), but luckily it’s very easy to fix. In order to get running again, all you need to do is install two plugins that used to be part of Rails core:

script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_list/
script/plugin install http://svn.rubyonrails.org/rails/plugins/acts_as_tree/

Once again, a great big thanks to Matt McCray for putting out such a useful plugin.

My podcasts / vidcasts at SD Ruby have been posted to the podcast section.

Simple Sidebar Plugin

How to use Simple Sidebar plugin to DRY up sidebar content in applications.

Related Blog Posts:

http://blog.aisleten.com/2007/06/03/simplesidebar-if-you-have-sidebars-you-need-this-plugin/

Ajax CSS Star Rating with ActsAsRateable

How to build an Ajax-powered, CSS star rater using the ActsAsRateable plugin and Komodo Media’s CSS Star Rating Redux technique.

Related Blog Posts:

http://blog.aisleten.com/2007/05/03/ajax-css-star-rating-with-acts_as_rateable/
http://blog.aisleten.com/2007/05/17/find-the-top-5-highest-rated-objects-with-acts_as_rateable/

At this month’s meeting we’re going to be having our first Rails Roundtable so come and check it out.
SD.rb December Meeting Schedule

We have finally gotten to the point where we are ready to start offering subscriptions to Obsidian Portal. We don’t expect there will be a lot of interest, but it’s always a sort of chicken v. egg problem. If you don’t have paying subscribers, then it’s not worth the effort to make the features. Conversely, without the features, no one is going to pay. On second thought, I guess it’s not chicken and egg, it’s pretty clear: you need features or no one will pay.

Asking for payment means you will need to be able to accept it. Currency on the web is passed almost exclusively via credit cards (except for PayPal…), so that’s the direction we need to go in. That requires us to select a credit card processor. For today, we will be looking at TrustCommerce.

I won’t go into the details of how credit card processing works, mainly because I don’t really understand it myself. Suffice it to say, there are a lot of middle-men, and they are all trying to take a cut. Each cut is either a percentage of the total charge or a flat fee or both. So, a typical fee structure might be $0.30 flat fee plus 2.5% of the total transaction.

Note: If you don’t care about the analysis and just want to see a rundown of their prices, then jump to the pricing.

Go easy on me; it’s my first time

When selecting a processing agent, our first priority right now is ease of use. We don’t expect there will be a lot of people signing up for our premium service, so we don’t want to expend a lot of effort on a payment system only to never see it used. Also, we’re willing to pay a higher rate to the processor since 3% of $30/month is a lot different than 3% of $30,000/month. I’ll pay 3% vs 2.5% if the 3% service takes 2 hours to implement and the 2.5% service takes 10. So, for us, ease of use trumps competitive pricing.

Since we’re talking about subscriptions as opposed to purchases, there is a recurring element to the payments. Since we want easy-to-implement solutions, we are scoping our search to only include the payment processors that offer a recurring service. This is a very important thing to note, especially if you’re in the same boat. A 1-time payment processor model (like Google Checkout) just will not work if you want to do subscriptions. The main reason is that you will have to store the users’ credit card info on your server in order to pass it to the payment processor each billing cycle. Do not do this! There are actual laws and regulations detailing what sort of security procedures you have to maintain in order to hold that sort of sensitive data. It’s much easier to simply pay someone else to deal with that crap. If you do choose to store their info in your database, you should be looking for a lawyer right now, not a payment processor.

Just plug in your credit card info

In Rails, ease of use means finding a plugin. I write a lot about plugins on this blog, so why should credit card processing be any different? Doing a quick Google search led me to the TrustCommerce subscription payment plugin.

Finding this bit of code brought a smile to my face, as I thought I had just finished 90% of the work. Sign up for an account, drop in the plugin, and wait for the money to roll in. Too bad there were a few red flags that derailed the money train.

Sitting by the phone

TrustCommerce does not list any pricing on their website. Instead, they say you have to sign up for a test account, and then you’ll be contacted. Not a big deal, I guess. So, I signed up for a test account.

The first red flag went up when I did not get an immediate callback. Sure, I signed up at 11:00pm Eastern Time, but that’s normal business hours in Internet time. In other words, if you’re an Internet company that requires phone contact, you had better have someone manning the phone at all hours. A lot of Web jockeys like me have a regular 9-5 job that precludes us from doing our business dealings during normal business hours. I want to deal with companies that understand this and have staff available during my normal business hours.

Red flags: 1

The ball sits in my court

The second red flag went up at their lackluster eventual response. My cell is in a dead zone at work, so whenever I leave for lunch, I get all my messages. On the day after requesting contact, I had a voicemail message from TrustCommerce. Still no pricing info, just a short message to call them back. Seeing as how I was busy, I couldn’t do it right away. Then I forgot. Dead silence on their end. No e-mails, no more calls, nothing.

Now a lot of people may disagree with me on this, but I think they should have been hitting my inbox and voicemail pretty hard. “Mr. Wedemeyer, we’re still interested in talking to you about blah blah.” or “Send us an e-mail with the best time to call you.” That’s how the mortgage people behaved when I used LendingTree. Sure, it was annoying, but you knew they wanted your business. To me, an anemic response indicates that someone isn’t really serious about recruiting me as a customer.

Red flags: 2

Little fish: prepare to get fried

When I finally did get in touch with someone from TrustCommerce, he was quite happy to answer my pricing questions. I don’t know if I’m allowed to post that info, but since they didn’t expressly forbid it, here you go:

Basic pricing

• $95 1-time fee
• $20 / month
• $0.20 / transaction

Citadel (recurring payments)

• $145 1-time fee
• $10 / month
• $0.10 / month / billing id (ie. subscription)

Holy crap! $240 just to get started, plus an additional $30 per month, just to be allowed to use their service? Seeing as how I expect Obsidian Portal to be making around $10 / month, at least until we can recruit more people, this is insane! I politely said thank you to the salesman, hung up the phone, and started writing this post.

I guess I see these huge front-loaded fees like this: If you’re making enough money that the fees don’t matter, then you already have a lot of subscribers, which means you’re already handling credit cards. Maybe their service is so great compared to the competition that it’s worth it for the big boys. But, if you’re a small time operator like me, forget about it.

Red flags: 240 + 30 / month

The search continues

Although I said pricing was not our top priority, the front loaded fees with TrustCommerce completely invalidate them as a viable option. It would be a very long time before we paid off the initial investment, and with our none-to-clear business prospects with Obsidian Portal, that’s a gamble I’m not willing to take.

In the next exciting chapter we will be looking at Amazon Flexible Payment System (FPS). This new web service from Amazon is meant to rival Google Checkout and PayPal. I’ve been extremely pleased with S3, and maybe they can do one better with FPS. Stay tuned to find out.

Following Ryan’s experiences at Barcamp San Diego, I’ve decided to attend and (hopefully) present at Barcamp Atlanta. If I get the chance, I will be doing a presentation on custom Google Maps using S3. I will center on Ruby on Rails, of course, but most of the idea is language agnostic.

I’m hoping to use BarCamp as an opportunity to network with other Atlanta area entrepreneurs and hackers. I’ve got a lot of ideas and the skills to execute them, but I’m looking for people that are better than me in the marketing department. If you’re the kind of person who can sell water to a fish, then look me up at BarCamp!

Here’s a picture of me, looking as I probably will at the conference (lost and confused). Please don’t be shy, just come up and introduce yourself.

As we have mentioned numerous times, we here at Aisle Ten prefer vendor branches for installing our plugins. We have found that most plugins require some tweaking or modification in order to put them use exactly as we want. Rather than being a strike against the plugin architecture, I count this as one of its greatest strengths. Plugins are usually simple enough that it is an easy task to understand what they do, and then modify them to support exactly what you need.

In my previous post, I gave directions on how to create a vendor branch for a plugin. What was missing was a good explanation of how to actually upgrade when a new release version comes from the vendor. Today I’ll cover that, using the real-life example of moving from acts_as_solr 0.7 to 0.8.5. All examples will relate to acts_as_solr where the version we currently have is 0.7 and the new version we want is 0.8.5.

Update:Thanks to Chris in the comments, there may be a way to do all of this “the correct way” Skip down to the bottom to see.

What the SVN book recommends

The definitive source for info regarding subversion is the book, Version Control with Subversion. It’s available online for free. If you haven’t skimmed through it, now’s the time.

In the section on vendor branches, they give a very terse explanation of how to go about this:

To perform this upgrade, we checkout a copy of our vendor branch, and replace the code in the current directory with the new libcomplex 1.1 source code. We quite literally copy new files on top of existing files, perhaps exploding the libcomplex 1.1 release tarball atop our existing files and directories. The goal here is to make our current directory contain only the libcomplex 1.1 code, and to ensure that all that code is under version control. Oh, and we want to do this with as little version control history disturbance as possible.

After replacing the 1.0 code with 1.1 code, svn status will show files with local modifications as well as, perhaps, some unversioned or missing files. If we did what we were supposed to do, the unversioned files are only those new files introduced in the 1.1 release of libcomplex–we run svn add on those to get them under version control. The missing files are files that were in 1.0 but not in 1.1, and on those paths we run svn delete. Finally, once our current working copy contains only the libcomplex 1.1 code, we commit the changes we made to get it looking that way.

Why it doesn’t work

The main problem with this approach has to do with files and directories that are deleted between versions. At the end of the branch upgrade process, two things you want to have are:

• current contains all and only the code from 0.8.5. In other words, current is an identical copy of 0.8.5 from the acts_as_solr repository
• Our repository contains the history of the transition from 0.7 to 0.8.5.

Getting the second one is a little tough, and getting them both together is very tricky. From the subversion book:

We quite literally copy new files on top of existing files, perhaps exploding the libcomplex 1.1 release tarball atop our existing files and directories. The goal here is to make our current directory contain only the libcomplex 1.1 code

What about deleted files? Copying the new files in or exploding the tarball will cover changed and added files, but any files or directories that were deleted will show up in svn as unmodified Using this method as described, it is impossible to know whether a file listed as unmodified is actually present in 0.8.5 without manually verifying its existence. For acts_as_solr this would be an annoyance. For a project with thousands of files, it would be a nightmare, and the human operator would surely make mistakes, meaning what you have in your repository would not be an exact copy of the vendor’s release. That’s about the worst possible outcome.

Solution 1: Strip out all except directories and .svn

One possible solution is to “clean” your working copy before exploding the tarball. In this case, you write a script that walks your working copy deleting everything except the directories and their .svn subdirectories. Now, when you explode the tarball and get a list of changes, it will tell you all the files that are now missing (and therefore are not part of the release). Then it’s easy to follow the book’s instructions and remove them from svn.

Why it doesn’t work

Besides the fact that I have not been able to find a script that does what I’m describing, it doesn’t solve the problem of deleted directories. If, during the change in versions, entire directories have been deleted, this method will not detect that. After running the script, you’re left with an empty skeleton of your current version. Exploding the tarball will overlay the directory structure of the new version, but any deleted directories will still be present. They’ll be empty, but they’ll still be there. Again, it is impossible, without manual intervention, to determine if any particular empty directory still belongs. This is a much better situation than before, where we had to check every unmodified file, but it’s still an annoyance, and allows for human error. On to the next solution…

Solution 2: Merge against the vendor’s repository

When I thought of this one, I was very excited. It should conceivably allow me to use svn to handle all the dirty work, which is what it does best. If your vendor allows you read access to their svn repository, you should be able to do a merge (diff) between your current version, and their newest version, and apply that to the current. It’s a standard merge operation saying “What do I have to do to make my copy look like theirs?”

Why it doesn’t work

Simply enough, svn does not allow you to merge between two different repositories. Why this is, I have no idea, but they must have their reasons. So, this idea takes a bullet to the head.

Solution 3: Import, then merge against your repository

When I realized that Solution 2 would not work, it occurred to me that I could just import the newest release into my repository and then merge with that.

Step by step:

1. Do an export from the vendor’s repository (or explode the tarball) of the latest version to somewhere local (let’s say temp_latest)
2. Import this code into your repository: svn import temp_latest http://your/repo/here/vendor/some_package/latest
3. Check out current to somewhere: svn co http://your/repo/here/vendor/some_package/current temp_current
4. Merge and apply: svn merge http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/latest temp_current
5. Commit the changes: svn ci temp_current
6. Bonus points – verify the integrity: svn diff http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/latest If there are no differences, then current now contains an exact copy of the new release. Good job!
7. Tag the new current: svn copy http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/2.0

At this point, you can copy the new tag to your trunk and deal with any conflicts. This is all covered in the SVN book.

Why it works…but kind of sucks

This is my preferred solution, but it’s not perfect. My main issue is that each time it’s done, you have to import a brand new copy of the code into your repository in order to do the merge. This is pretty wasteful from a storage perspective, and can be prohibitive if the import is sizeable and/or the vendor puts out new releases on a fast schedule. It won’t be a problem for (most) plugins, but for larger projects it is probably impractical.

Update: In hindsight, I don’t think this actually works the way I intended. When you import the new files, svn does not realize that they’re in any way related to the files in current. Therefore, when you do the merge, rather than applying changes to the old files, it simply deletes them and adds the new ones. In other words, the merge simply replaces all the old files with the new ones, rather than inspecting and recording the changes that occurred. This process will discard any changes that you’ve made, making the whole process fairly worthless.

Going from 0.7 to 0.8.5

We chose solution 3, import and merge, for handling the move from 0.7 to 0.8.5 of acts_as_solr. I am happy with the result, as I noticed during the process that the plugin had been seriously re-organized, resulting in the deleting and moving of many files and directories. Had we tried any of the other solutions, it would have been a painstaking process figuring out where things had moved. With the route we took, svn handled all of that.

Once I decided to go that route, the entire process took about 15 minutes. There were no conflicts to deal with when merging to trunk, but that is probably due to the fact that the only changes I made resulted in patches that were accepted back into the acts_as_solr trunk.

Think you can do better?

These are the only solutions I could come up with. Some flat-out don’t work, while others have their plusses and minuses. If you have a suggestion for a better way, please post a comment and I’ll try to include your idea into the list.

Update: svn_load_dirs

What I haven’t mentioned up until now was svn_load_dirs. This is a perl script that is supposed to help manage vendor branches. I have fumbled with it in the past and simply gotten frustrated. I just couldn’t find good documentation.

However, Chris in the comments has posted a link to an excellent post detailing svn_load_dirs, and I recommend everyone check it out. The next time I have to do a vendor branch update, I will try following these directions.