Comments on: Optimizing Solr and Rails – Index in the background http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/ Late nights eventually pay off Wed, 08 Feb 2012 19:15:48 +0000 hourly 1 http://wordpress.org/?v= By: shokal_s http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2285 shokal_s Wed, 06 Aug 2008 08:13:02 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2285 Modifying the mergeFactor in the solrconfig.xml from 10 (default on installation) to 20, made the solr index 10 times faster (3 sec instead of 30sec) and the query time wasn't significantly higher. Modifying the mergeFactor in the solrconfig.xml from 10 (default on installation) to 20, made the solr index 10 times faster (3 sec instead of 30sec) and the query time wasn’t significantly higher.

]]> By: shokal_s http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2284 shokal_s Wed, 06 Aug 2008 07:56:13 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2284 Modifying the in the solrconfig.xml file from 10 (default value on installation) to 20 has made the index action 10 times faster for me (reduced from 30sec to 3sec), while the query time is not significantly slower. Modifying the in the solrconfig.xml file from 10 (default value on installation) to 20 has made the index action 10 times faster for me (reduced from 30sec to 3sec), while the query time is not significantly slower.

]]> By: Midnight Oil » Blog Archive » Hacking the Ultrasphinx plugin to work with paginating_find http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2201 Midnight Oil » Blog Archive » Hacking the Ultrasphinx plugin to work with paginating_find Sat, 19 Apr 2008 22:59:02 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2201 [...] Getting started with acts_as_solr acts_as_solr for development and production in one Tomcat instance Optimizing Solr and Rails - Index in the background [...] [...] Getting started with acts_as_solr acts_as_solr for development and production in one Tomcat instance Optimizing Solr and Rails – Index in the background [...]

]]> By: Geoff http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2126 Geoff Sat, 01 Mar 2008 18:23:13 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2126 preventing solr from indexing on every save is a good practice. But I suspect that the solr has even deeper problems after reading this: http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html preventing solr from indexing on every save is a good practice. But I suspect that the solr has even deeper problems after reading this:

http://headius.blogspot.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html

]]> By: Micah http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2097 Micah Tue, 05 Feb 2008 14:28:56 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2097 I may have to write a follow up to this article covering the use of message queue servers. That seems to be a common theme with Solr. On that note, what is the memory usage like for ActiveMQ? Our VPS is already swapping like mad thanks to Solr and RMagick. We may have to disable search entirely just to make sure that MySQL doesn't have to swap to disk all the time. From a few mailing list posts I've read, ActiveMQ (like most things Java) is not shy about grabbing memory, which is a big no-no for those of us trying to squeeze everything in a limited size. Which message server has the absolute smallest memory footprint? That means more to me than fancy features. I may have to write a follow up to this article covering the use of message queue servers. That seems to be a common theme with Solr.

On that note, what is the memory usage like for ActiveMQ? Our VPS is already swapping like mad thanks to Solr and RMagick. We may have to disable search entirely just to make sure that MySQL doesn’t have to swap to disk all the time. From a few mailing list posts I’ve read, ActiveMQ (like most things Java) is not shy about grabbing memory, which is a big no-no for those of us trying to squeeze everything in a limited size.

Which message server has the absolute smallest memory footprint? That means more to me than fancy features.

]]> By: Alex Payne http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2096 Alex Payne Tue, 05 Feb 2008 01:02:57 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2096 We use Solr for our "people search" feature at Twitter. We use acts_as_modified to determine whether or not an ActiveRecord object has changed and thus should be updated in the Solr index. Our Starling queue server stores the list of objects that need to be re-indexed. A simple daemon reads from that queue and fires off messages to Solr. Two cron jobs periodically commit and optimize the index. We use Solr for our “people search” feature at Twitter. We use acts_as_modified to determine whether or not an ActiveRecord object has changed and thus should be updated in the Solr index. Our Starling queue server stores the list of objects that need to be re-indexed. A simple daemon reads from that queue and fires off messages to Solr. Two cron jobs periodically commit and optimize the index.

]]> By: Micah http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2093 Micah Mon, 04 Feb 2008 14:04:34 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2093 Jacob, Thanks. This is all great stuff. I'll have to look into ActiveMessaging, as the rake + cron solution is workable, but seems inelegant. This might actually solve another problem we're looking at: capturing deletes. Currently, deletes from Solr still take place in the web request, because once they're gone from the database, we have no way to figure out what exactly was deleted. From your post, it sounds like we could use ActiveMessaging to handle this. Jacob,

Thanks. This is all great stuff. I’ll have to look into ActiveMessaging, as the rake + cron solution is workable, but seems inelegant.

This might actually solve another problem we’re looking at: capturing deletes. Currently, deletes from Solr still take place in the web request, because once they’re gone from the database, we have no way to figure out what exactly was deleted. From your post, it sounds like we could use ActiveMessaging to handle this.

]]> By: Jacob Stetser http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2088 Jacob Stetser Sat, 02 Feb 2008 23:21:50 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2088 We had the same issue with solr, and also traced it to acts_as_solr committing after every request. Eventually, we figured out the same thing - stop committing every save. Then the problem became - newer stuff was not showing up in the index, until we figured out we needed to send a commit command every so often! The newer versions of Solr also support autocommitting by time elapsed or number of documents uncommitted, so now we let Solr handle its own reindexing. That solved a lot of our issues, but we've also moved to ActiveMQ and ActiveMessaging to offload the process of sending updates to the index. Each update goes into an indexing queue and then a separate processor (written in ruby with active_messaging) grabs from the queue and ships it off to Solr. Since then, we've had much faster saves, up-to-date indexing and no save timeouts - even if the Solr service was offline. When we restart it, the processors resume where they left off. Once you get to this point, being able to offload stuff that doesn't need to happen in the space of a web request is vital to performance. The hardest part of ActiveMessaging with ActiveMQ is getting ActiveMQ running (java!!!) - but there are extremely simple-to-use ruby message queuing solutions around. The main idea is to move to a message queue architecture where you rapidly hand off the work request (update Solr with this record) to a queue, instead of slowly wait for Solr to respond and possibly lose the input because of a timeout error. As far as backgroundRB, I wouldn't recommend using it at this point - ActiveMessaging or 'bj' are much newer and more reliable, and the author of bdrb has publicly recommended against using it any more. We had the same issue with solr, and also traced it to acts_as_solr committing after every request. Eventually, we figured out the same thing – stop committing every save. Then the problem became – newer stuff was not showing up in the index, until we figured out we needed to send a commit command every so often!

The newer versions of Solr also support autocommitting by time elapsed or number of documents uncommitted, so now we let Solr handle its own reindexing.

That solved a lot of our issues, but we’ve also moved to ActiveMQ and ActiveMessaging to offload the process of sending updates to the index. Each update goes into an indexing queue and then a separate processor (written in ruby with active_messaging) grabs from the queue and ships it off to Solr.

Since then, we’ve had much faster saves, up-to-date indexing and no save timeouts – even if the Solr service was offline. When we restart it, the processors resume where they left off.

Once you get to this point, being able to offload stuff that doesn’t need to happen in the space of a web request is vital to performance. The hardest part of ActiveMessaging with ActiveMQ is getting ActiveMQ running (java!!!) – but there are extremely simple-to-use ruby message queuing solutions around. The main idea is to move to a message queue architecture where you rapidly hand off the work request (update Solr with this record) to a queue, instead of slowly wait for Solr to respond and possibly lose the input because of a timeout error.

As far as backgroundRB, I wouldn’t recommend using it at this point – ActiveMessaging or ‘bj’ are much newer and more reliable, and the author of bdrb has publicly recommended against using it any more.

]]> By: Nazar http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2080 Nazar Fri, 01 Feb 2008 11:38:12 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2080 Check out backgroundDRB if you need to run one off or scheduled background tasks. It offers a far more elegant solution compared to rake + cron. Check out backgroundDRB if you need to run one off or scheduled background tasks. It offers a far more elegant solution compared to rake + cron.

]]> By: Micah http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2076 Micah Mon, 28 Jan 2008 05:01:14 +0000 http://blog.aisleten.com/2008/01/26/optimizing-solr-and-rails-index-in-the-background/#comment-2076 Hi Nima, I've never heard of sphinx. I'll have to take a closer look sometime. As to the active_messaging, that's also new to me. We've been using rake + cron, and while it works, it seems a little ungainly. So, if there's a better way to do it (that's not a total pain to get working), then I'm all over that. Hi Nima,

I’ve never heard of sphinx. I’ll have to take a closer look sometime.

As to the active_messaging, that’s also new to me. We’ve been using rake + cron, and while it works, it seems a little ungainly. So, if there’s a better way to do it (that’s not a total pain to get working), then I’m all over that.

]]>