Midnight Oil

In my previous post on acts_as_solr, I noted that as of 0.7, acts_as_solr did not support placing the development and production Solr instances inside the same servlet containers. This is because it differentiates between the two based on the host and port, and a single servlet container is always going to be at a single host/port.

Luckily, the author did a great job of architecting the plugin, and there is only one line where it references the servlet path. So, to configure it to talk to a different servlet in the same container takes only a single change to a single line

Without further ado, here’s how to do it.

1. Get the patch (see update below)

First, download and apply the patch I made to acts_as_solr. This patch is for version 0.7, and it makes the changes I described above. You can get the patch here.

All the patch does is add another configuration option to the solr.yml file, then use this option when constructing the URL for the request to Solr.

Note: This patch is distinct from my earlier patch regarding acts_as_solr and single table inheritance. Feel free to apply either one or both, as per your needs.

Update: acts_as_solr was recently updated to 0.8, and this patch was included as one of the changes. So, if you’re using 0.8 or higher, it is unnecessary. The single table inheritance patch was included in 0.8 as well, so both are unnecessary.

2. Install Tomcat

Although the patch should work for any servlet container, the remainder of this HOWTO will deal with Tomcat 5.5 since that is the only servlet container I am really familiar with. For reference, I am using Tomcat 5.5.23, so if all else fails, grab that exact release and see if things work.

I will assume that you already have Tomcat installed and can get to the Tomcat welcome page at http://localhost:8080/. If not, there are plenty of other tutorials explaining these initial stages. Try this one I found with a quick Google search. Come back here once you can see the kittie-cat

3. Create the dual solr home directories

Each instance of Solr is going to need its own solr home directory. This directory will contain the configuration options as well as the index data itself. In our case, the configurations will be identical, but we want to maintain distinct indices.

Make two copies the solr directory that came as part of the Solr distribution:

cp -r apache_solr_directory/example/solr /some_path/solr_dev
cp -r apache_solr_directory/example/solr /some_path/solr_prod

The base path for these directories is up to you. Put them somewhere you’ll remember.

You will also need to copy the schema.xml file from acts_as_solr into the two directories as follows:

cp myrailsapp/vendor/plugins/acts_as_solr/schema.xml /some_path/solr_dev/conf
cp myrailsapp/vendor/plugins/acts_as_solr/schema.xml /some_path/solr_prod/conf

4. Create the dual solr webapps

Each instance of Solr is going to need its own web application archive (WAR) in the Tomcat webapps directory. We will name them in such a way as to make their servlet paths distinct.

Move two copies of the Solr war file to the Tomcat webapps directory:

cp apache_solr_directory/dist/apache-solr-XXXX.war /my/tomcat/home/webapps/solr_dev.war
cp apache_solr_directory/dist/apache-solr-XXXX.war /my/tomcat/home/webapps/solr_prod.war

At this point, we have two copies of the Solr war in the webapps directory, each with a distinct name. This will allow Tomcat to create two separate instances, each with its own path.

Note: At this point, you will not be able to get to the servlets if you start up Tomcat, so don’t bother testing yet. Do the next step first.

5. Create the Tomcat context files

Each Solr instance needs to know how to find its home directory. The Solr wiki describes this in general, but I’ll try to be explicit.

Place the following two context files in the /my/tomcat/home/conf/Catalina/localhost/ directory:

solr_dev.xml

<Context path="solr_dev" docBase="solr_dev.war" debug="0" crossContext="true">
  <Environment name="solr/home" type="java.lang.String"
    value="/some_path/solr_dev" override="false" />
</Context>

solr_prod.xml

<Context path="solr_prod" docBase="solr_prod.war" debug="0" crossContext="true">
  <Environment name="solr/home" type="java.lang.String"
    value="/some_path/solr_prod" override="false" />
</Context>

Just make sure that the two value attributes point to the two copies of the solr directory that you made in step 3.

If you restart Tomcat, you should be able to test your dual installations. There should be a Solr installation at two addresses now: http://localhost:8080/solr_dev/admin and http://localhost:8080/solr_prod/admin.

6. Update the solr.yml config file

If you look closely at the patch, you can see the new servlet_path option that needs to be set. I left the setting to solr for backward compatibility, but we’re going to need to change that.

Here is an example of a solr.yml config file that works with Tomcat/Solr as we have currently set it up.

solr.yml

# Config file for the acts_as_solr plugin.

development:
  host: localhost
  port: 8080
  servlet_path: solr_dev

production:
  host: localhost
  port: 8080
  servlet_path: solr_prod

7. Finished

That’s it! At this point, you have two separate instances of Solr running inside a single Tomcat container. Your development and production indices will remain distinct, and you don’t have to worry about running multiple servlet containers on multiple ports.

I recently downloaded and installed acts_as_solr, and I must say that I am quite impressed. I was dreading trying to include any sort of search service, but acts_as_solr made it a snap. As always with Rails plugins, there were a few snafus along the way, but I’ll try to help people get over them.

Note: This writeup deals with acts_as_solr 0.7, so be wary if you’re using a different version.

Config/Setup

Get the plugin

As always, the first step is to get the plugin. You can use the built in plugin installation scripts like so:

script/plugin install svn://svn.railsfreaks.com/projects/acts_as_solr/trunk

However, I will urge you to use a vendor drop instead. This plugin is unstable enough that you will probably want to edit the source, and a vendor drop will allow you to do it a little easier. Instructions for performing a vendor drop in Rails can be found in one of my earlier posts.

If you do a vendor drop, it will skip one small installation step, so you’ll have to manually copy the lib/solr.yml file in the acts_as_solr plugin into your main config directory. No big deal.

Get Solr

Solr is a Java app that is deployed inside a servlet container, for example, Tomcat. If you’re like me, that fact brings an unhappy frown to your face. One of the main reasons I switched to Rails was because I was tired of editing arcane servlet config files. Still, over the years, there have been a lot of great projects written as servlets, so sometimes we just have to grin and bear it.

Anyways, to get started fastest, just grab the latest Solr distribution from the Solr homepage. This will come with an embedded Jetty servlet container which will get you going super fast.

After you unzip the Solr distribution somewhere, you’ll need to copy the schema.xml file into the right place:

cp myrailsproject/vendor/plugins/acts_as_solr/schema.xml apache_solr_directory/example/solr/conf/

At this point, you’re ready to rock.

cd apache_solr_directory/example
java -jar start.jar


Test it out

The basic installation given in the acts_as_solr documentation have you using Jetty on port 8983. So, to test it out, point your browser to http://localhost:8983/solr/admin If that brings up the Solr admin interface, you’re 99% home. If not, it’s time to start debugging your servlet container. Good luck

Using it in a model

There’s really no point to me explaining this part, as there is an excellent tutorial movie on the acts_as_solr homepage. Just look for Up and running in less than 5 minutes.

Still, there’s one special point I want to make at this stage. After everything is set up and you have Solr running, you might find that you cannot use it to find any of your model objects. This is because, by default, acts_as_solr only indexes your objects on save. This makes good sense, but it also means that all objects currently in your database will have to be loaded and re-saved in order to show up in the index. There is probably an easier way to do it, but as of right now, I don’t know what it is.

Update: In the comments section, Jason posted a great way to get all your model objects into Solr quickly using rebuild_solr_index. Assuming your model is called Book, you can index all the Book objects in your database with the following call:

Book.rebuild_solr_index

Note, however, that with the current behavior, acts_as_solr calls a commit after indexing each object, which can be a costly operation, according to the Solr docs. I will see if I can find a way to modify rebuild_solr_index to only call commit after all objects are committed, or perhaps at user-specified intervals along the way.

To verify that objects are being indexed, watch the output from Jetty in the terminal window where you started it from. As soon as an object is indexed, Jetty should scroll some output about it. At that point, use the find_by_solr command to see if it was successfully placed in the index.

Single Table Inheritance

Because of the way that acts_as_solr names the indices, version 0.7 is incompatible with single table inheritance (STI), and therefore does not work with model object heirarchies. However, it’s a very small bug, and I am guessing that it was simply an oversight on the part of the author. I looked into it and was able to create a fix in about 10 minutes.

Attached is my acts_as_solr STI patch to enable indexing of STI models. (see update)

Update: This patch has been merged into the trunk as part of acts_as_solr 0.8. So, if you’re using 0.8 or higher, the patch is unnecessary.

Development vs Production environment – Howto

WARNING! Do not have your development and production environments pointing to the same Solr installation. They will step all over each other, and find_by_solr will return incorrect results. It is roughly equivalent to using the same database for production and development. So, a general rule of thumb is that each separate database needs its own Solr index.

The easiest way to do this is to set up multiple servet containers on different ports. You can have Jetty for production on 8983 and Jetty for development on 8984, for example. To enable this, you will want two separate copies of the Solr distribution. To change the port Jetty is listening on, edit the jetty.xml file at apache_solr_directory/example/etc/jetty.xml. Look for the line that says:

<Set name="Port"><SystemProperty name="jetty.port" default="8983"/></Set>

Even though it’s easy, just adding more servlet containers is usually not the best solution, due to memory and sysadmin overhead. Instead, it’s probably a better idea to deploy multiple Solr servlets to the same container. Heck, it may even be possible to have a single Solr instance handle multiple distinct indices.

Update: I have found a way to deploy multiple Solr instances inside a single Tomcat container. The details are available in another blog post.

An example in the wild

I have acts_as_solr up and running on my site, Obsidian Portal. For a good example of it in action, Take a look at the characters list page and search for ‘Nent’. If you look at the individual characters, you’ll see that each of them has ‘Nent’ in their content somewhere. This is the sort of functionality that acts_as_solr gives out of the box.

Gotchas

Coming soon…
Until then: did you forget to copy schema.xml into the Solr conf directory?

I went to a Dungeons and Dragons meetup today in order to, well, meetup with other Dungeons and Dragons players. Of course, my ulterior motive was to show of Obsidian Portal a little in the hopes that I could garner a few more subscribers/beta testers.

I have never been a sales person before, so I was a little worried about trying to sell something. Luckily, my wife Sarah, has been through some of this before with her website, Southern Spaces, and had some good advice. Rather than approaching it as a sales opportunity, I should approach the meeting as an opportunity to network with other gamers and talk about our campaigns. Naturally, any discussion of my campaign would lead me to show off the Adventure Log. Then, I could use this as a jumping off point for describing the website.

Sarah’s advice was quite helpful and put me much more at ease. Rather than being a corporate shill trying to sleaze my way into this group, I would be a fellow gamer there to talk about my game. If the opportunity to talk about the site presented itself, then I would take it, but I would not otherwise try to force it.

Still, I think it is useful to set goals, so I told myself that I would try to get five people to sign up for the site, and at least one person to create a new campaign.

Unfortunately, only two other people showed up to the meetup. I blame it on the fact that it was on Easter Sunday. It was truly disheartening. Of the two that showed up, only one was really much of a hardcore player. His name is Logan, and he is currently playing in one campaign and getting ready to start another. I tried to sell him on the idea of hosting his new campaign on Obsidian Portal, but he seemed sort of on the fence. Perhaps if I keep at him, maybe I can convince him to give it a try. I don’t want to be too pushy, but being not pushy enough really isn’t getting me anywhere.

Although the meetup was fun, and I really enjoyed talking to Logan, the final tally of signups was a big, fat zero. That’s not good.

The next Atlanta area D&D meetup is one month away. Hopefully we can get the homepage working by then and perhaps make things a little more usable, with better page flow and perhaps some documentation. Then, assuming the meeting is much larger, I can give it another go at recruiting some local players.