How to Reduce Test Interference in Minitest
Global state can easily lead to interference between test cases and cause random failures. In this article, we’ll discuss a technique for alleviating this problem when reducing the global state is infeasible.
I was working on the test suite for
active_record_doctor when I run into an interesting problem. Many
active_record_doctor tasks are applied at the model-level which means they iterate over all Active Record models and process them one-by-one.
I had to replace the dummy test app (that the Rails generator had created) with test cases based on dynamically-defined Active Record models. This improved test cohesion and readability but unexpectedly resulted in unpredictable test failures.
After spending way more time than I’d like on diagnosing the root cause, it turned out that Rails is leaking these dynamically-defined models. They were not garbage collected even after removing all references from my code. This caused the tests to break because
active_record_doctor would iterate over these redundant classes.
The most likely culprit was
ActiveSupport::DescendantsTracker. I didn’t want to spend more time on a solution especially that I wanted to support Rubies from 1.9.3 to 2.5.1 and Rails 4.2. to 5.2. Fixing the problem in one setup would make it reoccur in a different one.
I was left with the simplest solution: run one test per process. This would guarantee no leaks between test cases. I could have changed the way the test suite was run by listing all test files and passing them to
rails test via
xargs but I wanted to avoid leaking such implementation issues to the outside.
This forced me to create a custom Minitest runner that I packed up and published as
minitest-fork_executor. In this article, we’ll go through the reasoning behind the gem’s design.
Minitest implements high-level logic in the
Minitest module in
minitest/minitest.rb. The entry point to the test suite is
Minitest.run. The comment above the method shows a call stack driving the test suite execution which will be helpful in a moment. It also does some housekeeping, part of which is starting and shutting down the parallel executor.
parallel_executor is defined via an accessor on
Minitest and is expected to be configured before starting the test suite.
The executor should implement
#shutdown and may implement
#start. Except for starting and shutting down, Minitest doesn’t reference the executor explicitly due to the assumption that calling
Minitest in-place. This means we should use
#start to modify
Minitest and make it fit our needs.
Knowing we need to implement an executor with a
#start method, let’s focus on what we need to change. The comment above
Minitest.run shows the call stack behind a test run:
## # This is the top-level run method. Everything starts from here. It # tells each Runnable sub-class to run, and each of those are # responsible for doing whatever they do. # # The overall structure of a run looks like this: # # Minitest.autorun # Minitest.run(args) # Minitest.__run(reporter, options) # Runnable.runnables.each # runnable.run(reporter, options) # self.runnable_methods.each # self.run_one_method(self, runnable_method, reporter) # Minitest.run_one_method(klass, runnable_method) # klass.new(runnable_method).run
We can see it iterates over runnables and runs test methods one at a time. A runnable is usually a subclass of
Minitest::TestCase but can also be a benchmark, etc. Running a single test method is implemented in
Minitest.run_one_method. This is the method we need to modify in order to make forking the default behavior. The last step is figuring out what code to write to make forking work seamlessly with Minitest.
To sum up, we need to:
- Implement an executor.
- Make the executor override
- Run a test method in a forked process.
Let’s tackle these problems one by one.
Step 1: Implementing an Executor
The executor is a simple two-method class that looks like this:
module Minitest class ForkExecutor def start # We'll implement this in the next section. end def shutdown # Nothing to do here but required by Minitest. end end end
#shutdown is required as
Minitest always calls it.
#start is the method where we’ll modify
Minitest. Before providing method bodies, let’s quickly cover how the executor is supposed to be used.
In a Rails app, we should configure the executor in
test/test_helper.rb in the following way:
require 'minitest/fork_executor' Minitest.parallel_executor = Minitest::ForkExecutor.new
Step 2: Overriding
Minitest.run_one_method is defined on the
Minitest module. It means it’s an instance method on the singleton class of
Minitest. It’d be ideal to be able to still call the original
.run_one_method to make our executor robust in face of changes to
Minitest. The initial implementation used module prepending to achieve that but I had to use a different technique in order to support Ruby 1.9.3.
.run_one_method is a three-step process:
- Get a method object corresponding to
.run_one_method. We need to reuse the original implementation from our implementation. You can think of as more elaborate
.run_one_methodin order to avoid warning when redefining it.
- Define a new
Step 2 has one caveat. There’s
define_singleton_method but no
remove_singleton_method. This means we need to call
remove_method on the singleton class of
Minitest. Other than that, a straightforward translation of these steps into code looks like this:
def start original_run_one_method = Minitest.method(:run_one_method) class << Minitest remove_method(:run_one_method) end Minitest.define_singleton_method(:run_one_method) do |klass, method_name| # This is where we'll put our custom implementation in the next section. end end
We could store
original_run_one_method in an attribute and use it to reinstate the original method in
#shutdown but we’ll skip that for now. Time to focus on building the forking mechanism.
Step 3: Fork and Run
Forking is a UNIX concept that means creating another copy of the calling process. In Ruby, we can use
#fork which returns
true to the parent process and
false to the child process. In our case, the child process will run
original_run_one_method and the parent will simply receive the result from the child and return to
Forking is easy but we also need a way of sending the result object returned by
original_run_one_method back to the parent process. We can easily send Ruby objects over
IO by using the
Marshal module. In order to do that, we need to somehow create these
IO objects. This is where UNIX pipes enter the scene.
Without diving into low-level details, a pipe can be created by calling
IO.pipe. It returns a pair of
IO objects – one for reading, one for writing. They’re connected to each other in the sense that whatever is written to the write object can be read from the read object. On top of that,
fork will cause the child process to inherit the pipe so that both the child and the parent will be able to read from and write to the same pipe.
We’ve got all the ingredients for the algorithm:
- Open a pipe.
- Fork the process.
- Make the child run
original_run_one_methodand marshal the result back via the pipe.
- Make the parent process unmarshal the result object via the pipe.
The code to do that is below. Please notice we also have extra steps in form of enabling the binary mode on the pipe and closing unused IOs.
Minitest.define_singleton_method(:run_one_method) do |klass, method_name| read_io, write_io = IO.pipe read_io.binmode write_io.binmode if fork # Parent: load the result sent from the child write_io.close result = Marshal.load(read_io) read_io.close Process.wait else # Child: just run normally, dump the result, and exit the process to avoid double-reporting. result = original_run_one_method.call(klass, method_name) read_io.close Marshal.dump(result, write_io) write_io.close exit end result end
I recommend you look at the complete gem. If you ever run into a test isolation problem at the process level then the gem can be a quick solution. This may be especially useful in projects with heavy meta-programming or lots of global state.