Merck Developer Portal Case Study

A glimpse into our process from start to finish on a developer portal
project for Merck.

My experience with Minitest and RSpec

January 23, 2015

I’ve been using RSpec in earnest for the past 6 months now, so I thought it’s time to write a blurrrgh poast comparing RSpec with Minitest. I’ve used Minitest for years, and RSpec only for 6 months, so please keep that in mind when reading!

PLEASE REMEMBER, I don’t care what test framework you use, as long as you are testing your code. This is a post just about my experience with these two frameworks. In other words, this is an opinion piece.

INTRO!!!

I really believe that all test frameworks are essentially the same. No test framework can do something that’s impossible for another to do. It’s just code. So what differentiates test frameworks? I would say the main difference would be user interface. So I’m really going to compare user interfaces between these frameworks.

Things I like about RSpec.

By far and away, my favorite thing about RSpec is that if there is a failure, it prints how to run just that one test at the bottom of the output. I can easily copy and paste that line and run just the one test. For people who don’t know, here is an example:

describe "something" do
  it "works" do
    expect(10).to equal(11)
  end

  it "really works" do
    expect(11).to equal(11)
  end
end

When you run this, the output looks like this:

[aaron@TC tlm.com (master)]$ rspec code/fail_spec.rb 
F.

Failures:

  1) something works
     Failure/Error: expect(10).to eq(11)
       
       expected: 11
            got: 10
       
       (compared using ==)
     # ./code/fail_spec.rb:3:in `block (2 levels) in <top (required)>'

Finished in 0.00504 seconds (files took 0.20501 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./code/fail_spec.rb:2 # something works
[aaron@TC tlm.com (master)]$

All you have to do to rerun just the one failing spec is to copy and paste that one line like this:

I don’t have to think or type, I just “copy, paste, run”. I like to think of this as a “learn as you play” feature.

The other thing I like about RSpec is that it ships with a colored output formatter, so if you do rspec --color, you’ll get colored output. I’ve used Minitest for years without colored output, but I’ve come to really like the colored output from RSpec. It helps me quickly see the most important parts of the test output, the assertion failure, and stack traces (though I highly suspect that the RSpec core team all use black backgrounds in their terminals because sometimes the colors aren’t so nice on my white terminal).

Things I like about Minitest

The main thing that I like about Minitest is that a Minitest test is just a Ruby class. Here is an example test similar to the RSpec sample I showed earlier:

require 'minitest/autorun'

class Something < Minitest::Test
  def test_works
    assert_equal 11, 10
  end

  def test_really_works
    assert_equal 11, 11
  end
end

I like this because I know exactly where the test_works method is defined. It’s no different than any other class in Ruby, so I don’t have to learn anything new. I like it because I am a heavy user of CTags, so I can easily jump to test methods or classes inside my editor. Another thing that’s nice about using regular Ruby classes is that when I need to refactor my tests, I just use my normal refactoring skills: I extract methods, classes, modules, change to inheritance, etc. I use the same refactoring techniques on my test files as I do on my library files. I think that is where Minitest really shines.

Things I dislike about RSpec

RSpec is a DSL for writing tests. I think this is probably RSpec’s largest weakness. I am a programmer, I can read code, so I don’t particularly care whether or not my test code “reads like English”. I don’t understand the value of the DSL, especially when I have a nearly 4000 line specfile that I’d like to refactor. How do I go about refactoring? I can extract methods:

describe "something" do
  def helper
    11
  end

  it "works" do
    expect(10).to eq(helper)
  end

  it "really works" do
    expect(11).to eq(helper)
  end
end

But where is the helper method defined? What’s its visibility? Can I put it in a module? Can I use inheritance? Who can call it? Can I call super from the extracted method? If so, where does super go? These are not the types of questions I want to be pondering when I have 3000 test failures and really long spec files to read through. My mind spins off in to thoughts like “I wonder how RSpec works?”, and “I wonder what my cat is doing?”

From what I can gather, calling describe in RSpec essentially defines a class. But if that’s the case, why not just use Ruby classes? Then I don’t need to guess about method visibility, modules, inheritance, etc. Not to mention, the library code that I’m working with is just using Ruby classes. Why do I need to switch to some other language for testing?

Nesting a describe seems to perform inheritance, but only for certain things like before blocks. If you run this code:

describe "something" do
  before do
    puts "hi!"
  end

  it "works" do
    expect(10).to eq(helper)
  end

  it "really works" do
    expect(11).to eq(helper)
  end

  context "another thing" do
    before do
      puts "hello!"
    end

    it "really really works" do
      expect(11).to eq(helper)
    end
  end

  def helper
    11
  end
end

You’ll see “hi!” printed 3 times, once for each it, and “hello!” printed once. For the nested context, I would like to print “hello!” before it prints “hi!”. In a normal inheritance situation, I would change the call to super. I am not sure how to do what I just described without learning a new thing (I think I would need let but I don’t know). For now, I’m coping with this by refactoring to classes outside of the specs themselves, then calling the objects from the specs. Essentially “composition” in my tests.

Another thing that bugs me is that if I do try to refactor my RSpec tests, and I change any lines, then the line that I copy and pasted to run just the one test won’t work anymore. I have to figure out the new line. I can use the name of the test by specifying the -e flag on RSpec, but then I lose the “copy, paste, run” feature that I love so much.

Anyway, I will stop picking on RSpec now. To me, RSpec seems to have some sort of identity crisis and is caught somewhere between “a language for writing tests” and “just Ruby”. That is RSpec’s biggest weakness.

Things I dislike about Minitest

If RSpec does one thing better than Minitest, I would say that the command line UI easily outshines Minitest. Here is one example.

I can run one test at a time like so:

$ ruby code/fail_test.rb -n Something#test_works

This will run the test_works method on the Something class. The problem for me is that it’s not clear to do that when you see the failing test output. If we run the whole suite, the output looks like this:

[aaron@TC tlm.com (master)]$ ruby code/fail_test.rb 
Run options: --seed 5990

# Running:

.F

Finished in 0.002047s, 977.0396 runs/s, 977.0396 assertions/s.

  1) Failure:
Something#test_works 1:
Expected: 11
  Actual: 10

2 runs, 2 assertions, 1 failures, 0 errors, 0 skips
[aaron@TC tlm.com (master)]$

Looking at this, how would you know how to run just that one test? With RSpec, it’s no problem. I was able to learn how to run just one test the first time I ever used RSpec. I didn’t read any docs, just ran the tests, and the info was right there. Coming from RSpec, you might need to think you could do this:

$ ruby code/fail_test.rb:5

But of course that doesn’t work because you’re just using Ruby on the command line.

Here is a short video of me running the single test:

The disadvantages of this over RSpec’s output is I have to type a few characters and know a commandline interface in advance. You might think “it’s just a few characters”, but every time I have to type something is a potential for a mistake that I’ll have to correct, which takes even more time. I really think the advantages of “copy, paste, run” can’t be understated. (Yes, I know there are tools like autotest for both Minitest and RSpec that will run tests for you, but sometimes I can’t / don’t use them)

The other thing I wish Minitest would do is to ship with a color formatter like RSpec does. I am used to the color output from RSpec, and I don’t really want to go back. It really helps me focus on what’s important about my test failures.

Conclusion

Both of these frameworks have things I like and dislike. Being able to reuse my debugging and refactoring skills in my tests is pretty important to me, especially since I deal with lots of legacy code. This is why I’ll stick with Minitest in my personal projects. However, I am sure I will get better at refactoring and debugging in RSpec tests, and when I get better at it, I will share what I learn.

Now, do I care which one you use? No. As long as you test your code, I am happy. A professional developer should be able to work in either one of these because they essentially do the same thing: test your code.

[…]

The Ruby Rails Aggregator