The Mechanical Turk

I'm testing some concurrency issues with a Singleton Active Record class. The Singleton ActiveRecord in question should create only one row when the table is empty, and all subsequent instances should use that row.

Note: I'll rewrite this post soon with a more common rails example - updating a record which uses the very latest results from a select.

I want to test the case where a number of concurrent processes do this at once. What's the best way to do this in rails?

(To solve the concurrency issues I'm using pessimistic locking that recently made it into rails - here I'm focussing on testing my solution. I suggest reading this scrap from Ryan for illumination on Rails and locking). Oh yeah, I'm also using RSpec awesomeness in the tests.

First attempt: fork (doesn't work)

The following test fires up 30 processes all asking for a singleton row, and makes sure only one was created.

MySingleton.delete_all

pids = (1..30).to_a.collect do
  fork { MySingleton.instance }
end
pids.each {|pid| Process.waitpid pid} # wait for them all to finish

MySingleton.count.should == 1

The above code doesn't work - the database connection kept dropping out. Why is that? Because a database connection is for one single threaded process - if you simultaneously bombard a single connection with a bunch of stuff, it's going to get confused.

Second attempt: threads (not quite what we want)

ActiveRecord allows you to specify that you'll be using the database concurrently:

ActiveRecord::Base.allow_concurrency = true

This allows the database connection to be handled properly when spawning new threads. (The difference between a forked process and a thread is that forked process runs a new process (a copy of the the forking process) with it's own memory space, whereas a thread shares the memory space, but may run asynchronously).

After adding ActiveRecord::Base.allow_concurrency = true and ActiveRecord::Base.allow_concurrency = false to setup and teardown we change the test to:

MySingleton.delete_all

threads = (1..30).to_a.collect do
  Thread.new { MySingleton.instance }
end
threads.each {|thread| thread.join } # wait for the threads to complete

MySingleton.count.should == 1

Which passes! However, this is not be the case I'm trying to test. Allowing concurrency, and using threads, is a case where one process manages the single database connection intelligently for a concurrent application.

I want to test the case where a number of different processes, all with their own connection, are accessing the database. And I want to be sure there's no intelligent handling of the connection within a class. This is the usual real-world case - lots of distinct rails processes all garnering the database's attention.

(I have still kept this test, as it shows my Singleton class works in a multi-threaded application)

Third attempt: fork mk II (success!)

So I want to establish a new connection in each forked process. Here's a method that does that, taking a db config (we'll see how to get that in a second).

def fork_with_new_connection(config, klass = ActiveRecord::Base)
  fork do
    begin
      klass.establish_connection(config)
      yield
    ensure
      klass.remove_connection
    end
  end
end

To get the config, we remove the current connection, the return value of that method is the config. We can pass this to the new fork method, and also use it to re-establish the parent connection when everything's done. The test now looks like this:

MySingleton.delete_all

config = ActiveRecord::Base.remove_connection
pids = (1..30).to_a.collect do
  fork_with_new_connection(config) { MySingleton.instance }
end
ActiveRecord::Base.establish_connection(config)

pids.each {|pid| Process.waitpid pid}

MySingleton.count.should == 1

Roundup

So now I've got two techniques for concurrency testing:

• when one ActiveRecord connection is being used concurrently, and
• when multiple ActiveRecord connections are accessing the database

The Singleton example is just one case of a model with concurrency issues, another would an update which absolutely must be based on the very latest result of a select.

A word of warning: if you're running something like rcov with your tests, each forked process will also be running rcov, along with all of the associated overhead.

Testing a Singleton class is a bit tricker than you might expect. When testing, you typically need to reset the state of the objects in question in between tests. But this is something that a Singleton, by its nature, should never allow.

To solve this problem, you can make use of Ruby's open classes and add the following before your tests:

require 'singleton'

class <<Singleton
  def included_with_reset(klass)
    included_without_reset(klass)
    class <<klass
      def reset_instance
        Singleton.send :__init__, self
        self
      end
    end
  end
  alias_method :included_without_reset, :included
  alias_method :included, :included_with_reset
end

This code adds a class method when Singleton is mixed into your target class. This method reset_instance effectively removes the instance, so you can start again.

In your tests, when you need to start over with a Singleton, just send reset_instance to the singleton class. The following demonstrates this (note that the instance is a different object after MySingleton.reset_instance):

irb(main):026:0> MySingleton.instance
=> #<MySingleton:0x396fc>
irb(main):027:0> MySingleton.reset_instance
=> MySingleton
irb(main):028:0> MySingleton.instance
=> #<MySingleton:0x1cd04>