Kicking Aruba Into A Bin

ODI Head of Robots Sam Pikesley uncovers some new features in Rspec in his search for alternative tools for building Ruby command-line interface apps

‘Doing it all with Rspec.’ Photo: Sam Pikesley

Once upon a time, I was a UNIX SysAdmin, which means I’ve spent a lot of my working life deep in the command-line. As a consequence, many of the Ruby tools I’ve built in recent years have been CLI ones.

My weapon-of-choice for building Ruby command-line interface (CLI) apps has long been the mighty Thor, and up until now I’ve always used Aruba to test my Thor apps (while sticking with Rspec to TDD the actual workings of my gems). This has mostly worked OK, but I’m also (for better or worse) a big fan of VCR, and these things really do not play nicely together.

Oh hi, technical debt

Because Aruba spawns a separate Ruby process to run its tests, it’s all invisible to VCR. There are a number of (now deprecated) hacks to get around this problem, but I was finding that I had to write my features in very contrived ways (which definitely defeats the purpose of Cucumber), and it still behaved unexpectedly. And when Aruba also started to interfere with some of my other favourite tools, I decided it was Different Solution time.

Doing it all with Rspec

I came across this blogpost which mentions this capture method in Thor’s spec_helper. Turns out this is kinda generic, and we can paste it right into our Gem’s own spec_helper.

But wait, don’t call yet, because then I was pointed towards this Stack Overflow post about validating exits in Rspec, and that led me to Rspec’s output matcher which appears to make all of the foregoing redundant.

So how does this all fit together?

Simple match

module Banjaxer
  describe CLI do
    let :subject do
      described_class.new
    end

    it 'has a version' do
      expect { subject.version }.to output(/^banjaxer version #{VERSION}$/).to_stdout
    end
  end
end


module Banjaxer
  class CLI < Thor
    desc 'version', 'Print banjaxer version'
    def version
      puts "banjaxer version #{VERSION}"
    end
    map %w(-v --version) => :version
  end
end

In the spec, we set up a instance of our class, which is a Thor, then we call its #version method and inspect whatever lands on STDOUT.

There is a certain amount of sleight-of-hand going on in this: note that our argument to the output matcher is a regex, even though we really want to match a string. That’s because the actual output string will have a "\n" on the end of it, so we’d have to match that explicitly.

With an argument

module Banjaxer
  describe CLI do
    let :subject do
      described_class.new
    end

    it 'gets the url', :vcr do
      expect { subject.get_url 'http://uncleclive.herokuapp.com/banjax' }.to output(/^Content-Length is 808$/).to_stdout
    end
  end
end

module Banjaxer
  class CLI < Thor
    desc 'get url', 'GET a url and tell us the Content-Length'
    def get_url url
      h = HTTParty.get url, headers: { 'Accept' => 'application/json' }
      puts "Content-Length is #{h.headers['Content-Length']}"
    end
  end
end

We might notice some more prestidigitation here, when we consider how Thor works: it takes something like banjaxer get_url http://uncleclive.herokuapp.com/banjax from STDIN, and turns that (via the ./exe/banjaxer executable) into a call to #version('http://uncleclive.herokuapp.com/banjax') - we’re bypassing that step and making the method call directly. The corresponding Aruba:

Scenario: Get url        
  When I successfully run `banjaxer get_url http://uncleclive.herokuapp.com/banjax`        
  Then the output should contain "Content-Length is 808"

… will do exactly what it says, which may be a more accurate test, but notice that we’ve dropped a :vcr into the Rspec version and it worked as expected, which simply would not happen with Aruba.

With options

module Banjaxer
  describe CLI do
    let :subject do
      described_class.new
    end

    context 'with options' do
      it 'can handle an option' do
        subject.options = {json: true}
        expect { subject.embiggen 'the smallest man' }.to output(/^{"embiggening":"the smallest man"}/).to_stdout
      end
    end
  end
end

module Banjaxer
  class CLI < Thor
    desc 'embiggen', 'Embiggen something'
    method_option :json,
                  type: :boolean,
                  aliases: '-j',
                  description: 'Return JSON on the console'
    def embiggen value
      if options[:json]
        puts({ embiggening: value }.to_json)
      else
        puts "embiggening #{value}"
      end
    end
  end
end

Some more trickery here, which took me a little while to figure out: when we pass options on the command-line, Thor shoves them into the options hash on the instance. So in our spec, we set up that hash ourselves with subject.options = {json: true} and then call the method.

Testing exit statuses

module Banjaxer
  describe CLI do
    let :subject do
      described_class.new
    end

    context 'deal with exit codes' do
      it 'exits with a zero by default' do
        expect { subject.cromulise }.to exit_with_status 0
      end
    end
  end
end

module Banjaxer
  class CLI < Thor
    desc 'cromulise', 'Exit with the supplied status'
    def cromulise status = 'zero'
      lookups = {
        'zero' => 0,
        'one' => 1
      }
      code = lookups.fetch(status, 99)

      puts "Exiting with a #{code}"
      exit code
    end
  end
end

Checking the exit status is supported out-of-the-box in Aruba:

Scenario: Get version
  When I run `banjaxer -v`
  Then the exit status should be 0

… but for Rspec, we have to cook up our own custom matcher:

RSpec::Matchers.define :exit_with_status do |expected|
  match do |actual|
    expect { actual.call }.to raise_error(SystemExit)

    begin
      actual.call
    rescue SystemExit => e
      expect(e.status).to eq expected
    end
  end

  supports_block_expectations
end

This is surprisingly simple: we just #call the method passed in as actual, trap the exception it raises, and check its #status against the expectation. That supports_block_expectations is apparently required because this matcher actually calls a block (but this is a bit magical and I don’t fully understand it, I just know that it didn’t work without it).

Inspecting output files

module Banjaxer
  describe CLI do
    let :subject do
      described_class.new
    end

    context 'read output files' do
      it 'writes the expected output file' do
        subject.say 'monorail'
        expect('said').to have_content (
        """
        The word was:
          monorail
        """
        )
      end
    end
  end
end

module Banjaxer
  class CLI < Thor
    desc 'say', 'Say the word'
    def say word, outfile = 'said'
      File.open outfile, 'w' do |f|
        f.write "The word was:\n#{word}"
      end
    end
  end
end

Replicating this (very useful) feature of Aruba:

Scenario: Write file
  When I run `banjaxer say monorail`
  Then a file named "said" should exist
  And the file named "said" should contain:  
  """
  The word was:
    monorail
  """

… required considerably more work. The full code for the have_content custom matcher (and its supporting bits and pieces) can be seen here. There’s quite a bit going on in this, so let’s dig in:

Temporary output directory

Presumably our CLI app would generate any output files in its Present Working Directory, but we can get Rspec to make us a temporary directory and switch to that before each test (and then bounce back out of it afterwards). Notice that it deletes the tmp/ directory before it starts, not at the end of the run. I stole this idea from Aruba and it means that in the event of a spec failure, we can run just the failing test and then debug by having a look at exactly the output it produced.

Custom matcher

This matcher takes the expected string from the spec and reads the actual file, then splits them both into lines and compares them – if it finds a mismatch, then pass becomes false and we get a failure. The clever stuff is in the next section, though:

Monkey-patching String

I originally wrote these as normal static methods, but it occurred to me that everything would be a lot more elegant if they were String instance methods. The interesting (and possibly brittle) thing here is the #is_regex stuff: if the string looks like a Regular Expression (ie with leading and trailing slashes) then we take the body of it and turn it into an actual regular expression and then do our comparison against that. I think this may bite me somewhere down the road.

This matcher is significantly more sophisticated than the exit_with_status one - so much so that it became necessary to generate Rspec with Rspec.

Suppressing console output

Any self-respecting CLI app is likely to be generating feedback on the command line, but this is going to pollute our Rspec output. We can suppress it with something like this in the spec_helper:

RSpec.configure do |config|
  # Suppress CLI output. This *will* break Pry
  original_stderr = $stderr
  original_stdout = $stdout
  config.before(:all) do
    # Redirect stderr and stdout
    $stderr = File.new '/dev/null', 'w'
    $stdout = File.new '/dev/null', 'w'
  end

  config.after(:all) do
    $stderr = original_stderr
    $stdout = original_stdout
  end
end

Before each test, we redirect STDOUT and STDERR to /dev/null, then bring them back afterwards. Note that this is not platform-independent, you need to something different on Windows, but I don’t know what. Also note that this causes pry to do really odd things - disable this if you want to reliably pry into your code (maybe this should be wrapped in an unless ENV['PRY'] guard of some sort).

Running the code

The code is all here, please feel free to have a look at it and run it:

git clone https://github.com/theodi/banjaxer
cd banjaxer
bundle
bundle exec rake

As always, feedback, PRs etc are welcome.

Next steps

I seem to have replicated quite a lot of the functionality of Aruba, but with the added benefit of not using Aruba. I think the thing to do now might be to package this up into a Gem and use it on a real project.

I sincerely hope somebody else finds this useful – I certainly did.

Sam Pikesley is Head of Robots at the ODI. Follow @pikesley on Twitter.

If you have ideas or experience in open data that you’d like to share, pitch us a blog or tweet us at @ODIHQ.