Midnight Oil

As we have mentioned numerous times, we here at Aisle Ten prefer vendor branches for installing our plugins. We have found that most plugins require some tweaking or modification in order to put them use exactly as we want. Rather than being a strike against the plugin architecture, I count this as one of its greatest strengths. Plugins are usually simple enough that it is an easy task to understand what they do, and then modify them to support exactly what you need.

In my previous post, I gave directions on how to create a vendor branch for a plugin. What was missing was a good explanation of how to actually upgrade when a new release version comes from the vendor. Today I’ll cover that, using the real-life example of moving from acts_as_solr 0.7 to 0.8.5. All examples will relate to acts_as_solr where the version we currently have is 0.7 and the new version we want is 0.8.5.

Update:Thanks to Chris in the comments, there may be a way to do all of this “the correct way” Skip down to the bottom to see.

What the SVN book recommends

The definitive source for info regarding subversion is the book, Version Control with Subversion. It’s available online for free. If you haven’t skimmed through it, now’s the time.

In the section on vendor branches, they give a very terse explanation of how to go about this:

To perform this upgrade, we checkout a copy of our vendor branch, and replace the code in the current directory with the new libcomplex 1.1 source code. We quite literally copy new files on top of existing files, perhaps exploding the libcomplex 1.1 release tarball atop our existing files and directories. The goal here is to make our current directory contain only the libcomplex 1.1 code, and to ensure that all that code is under version control. Oh, and we want to do this with as little version control history disturbance as possible.

After replacing the 1.0 code with 1.1 code, svn status will show files with local modifications as well as, perhaps, some unversioned or missing files. If we did what we were supposed to do, the unversioned files are only those new files introduced in the 1.1 release of libcomplex–we run svn add on those to get them under version control. The missing files are files that were in 1.0 but not in 1.1, and on those paths we run svn delete. Finally, once our current working copy contains only the libcomplex 1.1 code, we commit the changes we made to get it looking that way.

Why it doesn’t work

The main problem with this approach has to do with files and directories that are deleted between versions. At the end of the branch upgrade process, two things you want to have are:

• current contains all and only the code from 0.8.5. In other words, current is an identical copy of 0.8.5 from the acts_as_solr repository
• Our repository contains the history of the transition from 0.7 to 0.8.5.

Getting the second one is a little tough, and getting them both together is very tricky. From the subversion book:

We quite literally copy new files on top of existing files, perhaps exploding the libcomplex 1.1 release tarball atop our existing files and directories. The goal here is to make our current directory contain only the libcomplex 1.1 code

What about deleted files? Copying the new files in or exploding the tarball will cover changed and added files, but any files or directories that were deleted will show up in svn as unmodified Using this method as described, it is impossible to know whether a file listed as unmodified is actually present in 0.8.5 without manually verifying its existence. For acts_as_solr this would be an annoyance. For a project with thousands of files, it would be a nightmare, and the human operator would surely make mistakes, meaning what you have in your repository would not be an exact copy of the vendor’s release. That’s about the worst possible outcome.

Solution 1: Strip out all except directories and .svn

One possible solution is to “clean” your working copy before exploding the tarball. In this case, you write a script that walks your working copy deleting everything except the directories and their .svn subdirectories. Now, when you explode the tarball and get a list of changes, it will tell you all the files that are now missing (and therefore are not part of the release). Then it’s easy to follow the book’s instructions and remove them from svn.

Why it doesn’t work

Besides the fact that I have not been able to find a script that does what I’m describing, it doesn’t solve the problem of deleted directories. If, during the change in versions, entire directories have been deleted, this method will not detect that. After running the script, you’re left with an empty skeleton of your current version. Exploding the tarball will overlay the directory structure of the new version, but any deleted directories will still be present. They’ll be empty, but they’ll still be there. Again, it is impossible, without manual intervention, to determine if any particular empty directory still belongs. This is a much better situation than before, where we had to check every unmodified file, but it’s still an annoyance, and allows for human error. On to the next solution…

Solution 2: Merge against the vendor’s repository

When I thought of this one, I was very excited. It should conceivably allow me to use svn to handle all the dirty work, which is what it does best. If your vendor allows you read access to their svn repository, you should be able to do a merge (diff) between your current version, and their newest version, and apply that to the current. It’s a standard merge operation saying “What do I have to do to make my copy look like theirs?”

Why it doesn’t work

Simply enough, svn does not allow you to merge between two different repositories. Why this is, I have no idea, but they must have their reasons. So, this idea takes a bullet to the head.

Solution 3: Import, then merge against your repository

When I realized that Solution 2 would not work, it occurred to me that I could just import the newest release into my repository and then merge with that.

Step by step:

1. Do an export from the vendor’s repository (or explode the tarball) of the latest version to somewhere local (let’s say temp_latest)
2. Import this code into your repository: svn import temp_latest http://your/repo/here/vendor/some_package/latest
3. Check out current to somewhere: svn co http://your/repo/here/vendor/some_package/current temp_current
4. Merge and apply: svn merge http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/latest temp_current
5. Commit the changes: svn ci temp_current
6. Bonus points - verify the integrity: svn diff http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/latest If there are no differences, then current now contains an exact copy of the new release. Good job!
7. Tag the new current: svn copy http://your/repo/here/vendor/some_package/current http://your/repo/here/vendor/some_package/2.0

At this point, you can copy the new tag to your trunk and deal with any conflicts. This is all covered in the SVN book.

Why it works…but kind of sucks

This is my preferred solution, but it’s not perfect. My main issue is that each time it’s done, you have to import a brand new copy of the code into your repository in order to do the merge. This is pretty wasteful from a storage perspective, and can be prohibitive if the import is sizeable and/or the vendor puts out new releases on a fast schedule. It won’t be a problem for (most) plugins, but for larger projects it is probably impractical.

Update: In hindsight, I don’t think this actually works the way I intended. When you import the new files, svn does not realize that they’re in any way related to the files in current. Therefore, when you do the merge, rather than applying changes to the old files, it simply deletes them and adds the new ones. In other words, the merge simply replaces all the old files with the new ones, rather than inspecting and recording the changes that occurred. This process will discard any changes that you’ve made, making the whole process fairly worthless.

Going from 0.7 to 0.8.5

We chose solution 3, import and merge, for handling the move from 0.7 to 0.8.5 of acts_as_solr. I am happy with the result, as I noticed during the process that the plugin had been seriously re-organized, resulting in the deleting and moving of many files and directories. Had we tried any of the other solutions, it would have been a painstaking process figuring out where things had moved. With the route we took, svn handled all of that.

Once I decided to go that route, the entire process took about 15 minutes. There were no conflicts to deal with when merging to trunk, but that is probably due to the fact that the only changes I made resulted in patches that were accepted back into the acts_as_solr trunk.

Think you can do better?

These are the only solutions I could come up with. Some flat-out don’t work, while others have their plusses and minuses. If you have a suggestion for a better way, please post a comment and I’ll try to include your idea into the list.

Update: svn_load_dirs

What I haven’t mentioned up until now was svn_load_dirs. This is a perl script that is supposed to help manage vendor branches. I have fumbled with it in the past and simply gotten frustrated. I just couldn’t find good documentation.

However, Chris in the comments has posted a link to an excellent post detailing svn_load_dirs, and I recommend everyone check it out. The next time I have to do a vendor branch update, I will try following these directions.