Background

I was working on the test suite for active_record_doctor when I run into an interesting problem. Many active_record_doctor tasks are applied at the model-level which means they iterate over all Active Record models and process them one-by-one.

I had to replace the dummy test app (that the Rails generator had created) with test cases based on dynamically-defined Active Record models. This improved test cohesion and readability but unexpectedly resulted in unpredictable test failures.

After spending way more time than I’d like on diagnosing the root cause, it turned out that Rails is leaking these dynamically-defined models. They were not garbage collected even after removing all references from my code. This caused the tests to break because active_record_doctor would iterate over these redundant classes.

The most likely culprit was ActiveSupport::DescendantsTracker. I didn’t want to spend more time on a solution especially that I wanted to support Rubies from 1.9.3 to 2.5.1 and Rails 4.2. to 5.2. Fixing the problem in one setup would make it reoccur in a different one.

I was left with the simplest solution: run one test per process. This would guarantee no leaks between test cases. I could have changed the way the test suite was run by listing all test files and passing them to rails test via xargs but I wanted to avoid leaking such implementation issues to the outside.

This forced me to create a custom Minitest runner that I packed up and published as minitest-fork_executor. In this article, we’ll go through the reasoning behind the gem’s design.

Understanding Minitest

Minitest implements high-level logic in the Minitest module in minitest/minitest.rb. The entry point to the test suite is Minitest.run. The comment above the method shows a call stack driving the test suite execution which will be helpful in a moment. It also does some housekeeping, part of which is starting and shutting down the parallel executor. parallel_executor is defined via an accessor on Minitest and is expected to be configured before starting the test suite.

The executor should implement #shutdown and may implement #start. Except for starting and shutting down, Minitest doesn’t reference the executor explicitly due to the assumption that calling #start modifies Minitest in-place. This means we should use #start to modify Minitest and make it fit our needs.

Knowing we need to implement an executor with a #start method, let’s focus on what we need to change. The comment above Minitest.run shows the call stack behind a test run:

  ##
  # This is the top-level run method. Everything starts from here. It
  # tells each Runnable sub-class to run, and each of those are
  # responsible for doing whatever they do.
  #
  # The overall structure of a run looks like this:
  #
  #   Minitest.autorun
  #     Minitest.run(args)
  #       Minitest.__run(reporter, options)
  #         Runnable.runnables.each
  #           runnable.run(reporter, options)
  #             self.runnable_methods.each
  #               self.run_one_method(self, runnable_method, reporter)
  #                 Minitest.run_one_method(klass, runnable_method)
  #                   klass.new(runnable_method).run

We can see it iterates over runnables and runs test methods one at a time. A runnable is usually a subclass of Minitest::TestCase but can also be a benchmark, etc. Running a single test method is implemented in Minitest.run_one_method. This is the method we need to modify in order to make forking the default behavior. The last step is figuring out what code to write to make forking work seamlessly with Minitest.

To sum up, we need to:

1. Implement an executor.
2. Make the executor override Minitest.run_one_method
3. Run a test method in a forked process.

Let’s tackle these problems one by one.

Step 1: Implementing an Executor

The executor is a simple two-method class that looks like this:

module Minitest
  class ForkExecutor
    def start
      # We'll implement this in the next section.
    end

    def shutdown
      # Nothing to do here but required by Minitest.
    end
  end
end

#shutdown is required as Minitest always calls it. #start is the method where we’ll modify Minitest. Before providing method bodies, let’s quickly cover how the executor is supposed to be used.

In a Rails app, we should configure the executor in test/test_helper.rb in the following way:

require 'minitest/fork_executor'
Minitest.parallel_executor = Minitest::ForkExecutor.new

That’s it!

Step 2: Overriding .run_one_method

Minitest.run_one_method is defined on the Minitest module. It means it’s an instance method on the singleton class of Minitest. It’d be ideal to be able to still call the original .run_one_method to make our executor robust in face of changes to Minitest. The initial implementation used module prepending to achieve that but I had to use a different technique in order to support Ruby 1.9.3.

Overriding .run_one_method is a three-step process:

1. Get a method object corresponding to .run_one_method. We need to reuse the original implementation from our implementation. You can think of as more elaborate super.
2. Remove .run_one_method in order to avoid warning when redefining it.
3. Define a new .run_one_method on Minitest.

Step 2 has one caveat. There’s define_singleton_method but no remove_singleton_method. This means we need to call remove_method on the singleton class of Minitest. Other than that, a straightforward translation of these steps into code looks like this:

def start
  original_run_one_method = Minitest.method(:run_one_method)

  class << Minitest
    remove_method(:run_one_method)
  end

  Minitest.define_singleton_method(:run_one_method) do |klass, method_name|
    # This is where we'll put our custom implementation in the next section.
  end
end

We could store original_run_one_method in an attribute and use it to reinstate the original method in #shutdown but we’ll skip that for now. Time to focus on building the forking mechanism.

Step 3: Fork and Run

Forking is a UNIX concept that means creating another copy of the calling process. In Ruby, we can use #fork which returns true to the parent process and false to the child process. In our case, the child process will run original_run_one_method and the parent will simply receive the result from the child and return to Minitest.

Forking is easy but we also need a way of sending the result object returned by original_run_one_method back to the parent process. We can easily send Ruby objects over IO by using the Marshal module. In order to do that, we need to somehow create these IO objects. This is where UNIX pipes enter the scene.

Without diving into low-level details, a pipe can be created by calling IO.pipe. It returns a pair of IO objects – one for reading, one for writing. They’re connected to each other in the sense that whatever is written to the write object can be read from the read object. On top of that, fork will cause the child process to inherit the pipe so that both the child and the parent will be able to read from and write to the same pipe.

We’ve got all the ingredients for the algorithm:

1. Open a pipe.
2. Fork the process.
3. Make the child run original_run_one_method and marshal the result back via the pipe.
4. Make the parent process unmarshal the result object via the pipe.

The code to do that is below. Please notice we also have extra steps in form of enabling the binary mode on the pipe and closing unused IOs.

Minitest.define_singleton_method(:run_one_method) do |klass, method_name|
  read_io, write_io = IO.pipe
  read_io.binmode
  write_io.binmode

  if fork
    # Parent: load the result sent from the child

    write_io.close
    result = Marshal.load(read_io)
    read_io.close

    Process.wait
  else
    # Child: just run normally, dump the result, and exit the process to avoid double-reporting.
    result = original_run_one_method.call(klass, method_name)

    read_io.close
    Marshal.dump(result, write_io)
    write_io.close
    exit
  end

  result
end

Mission Accomplished

I recommend you look at the complete gem. If you ever run into a test isolation problem at the process level then the gem can be a quick solution. This may be especially useful in projects with heavy meta-programming or lots of global state.