ActiveTest: Examination, Part I: Removing Transparency

Posted by Mathew Abonyi Mon, 01 Jan 2007 03:34:58 GMT

The first abstraction I made was to change assert_get_success and its equivalents (get_failure, post_success, etc.) into an abstract idea of common test cases. Obviously every application is going to be treated differently, but to make a suite of tools to remove the most common tests seemed like a useful thing to extract—it’s DRY.

Consequently, I had the following in Test::Unit::TestCase:

  class << self
    def test_should_index_items
      define_method(:test_should_index_items) do
        assert_get_success :index
      end
    end
  end

I then abstracted it a little:

  class << self
    def test_should_get_with_success(action = :index, parameters = {})
      define_method("test_should_#{action}_item") do
        assert_get_success action, parameters
      end
    end
  end

This is basic DRYing up. However, I have at this point already lost sight of the simplest benefit of writing out test cases with Test::Unit: documentation. If someone else uses this, the only thing they know is that their GET succeeds… but in what way? I began documenting what each method does and extracting more and more complex cases which may be ‘common’. Documenting these methods hid the general fault, which was making the entire testing process opaque to the user. Tests started to look like this:

class TestSomething < ActiveTest::Base
  test_should_get_with_success :index
  test_should_get_with_success :show, :id => 1
  test_should_get_with_success :new
  test_should_get_with_success :edit, :id => 1
end

So my first mistake was removing transparency without thinking about what I was beginning to lose. This is quite project-specific. In most cases, making methods opaque and useful is a good thing, but in the case of ActiveTest, it needs to be flexible and transparent. It is hard to beat a series of asserts.[1]

This mistake is probably my least offensive, because some, including myself, may like this feature just to remove all those irritating test_should_index_items, especially when you are writing new controllers left and right.

What Was I Thinking?

I thought that the documenting aspect of testing was purely outlining what everything should do, rather than describing what everything should do. My original idea was to make it easier to document and test at the same time by outlining the cases in each suite. Just in terms of line ratios, I now had a 5-line method which let me write a test case in 1 line. Not bad. It quickly paid for itself because it catered to :index, :new:, :show and :edit. However, I lost all the expressiveness of showing what is being tested.

Coming Up Next: Abstracting Without Basis…

Footnotes

1 It can be done. In fact, the next generation of Active Test, while rather different, frees up the tester to think completely about the behaviour of his code. I will trade in the brevity of test cases for the flexibility of defining them (for anything, whether Rails application, plugin, gem or 60-line file).

Posted in , , ,  | no comments | 1 trackback

ActiveTest: Examination, Introduction: A Mistake and How To Fix It

Posted by Mathew Abonyi Sun, 31 Dec 2006 22:21:17 GMT

In trying to bring ActiveTest to a state resembling my original article about ActiveTest, I realised that … it’s a piece of crap. It just isn’t the kind of code which many but myself would find useful and now even I don’t find it that helpful anymore. A total cowpat of a project, sadly.

To help both myself and the community, I will be analysing my mistake in full and mercilessly revealing my thinking process, development and design along the way. Because it may get a little… lengthy, I’ll break it down into a number of articles. The last article will be dedicated entirely to making Active Test useful (which itself is three parts: salvaging useful ideas, redesigning weak areas, and changing purpose).

Obituary to Active Test 0.1.0 Beta

In its death-throes, the Active Test Plugin can still be useful: as an example of what not to do. Before I start laying into my sad miscarriage of an idea, I’ll outline useful ideas which came out of it:

  • Finding a way to extend Test::Unit which is safe and allows inheritance
  • Learning different ways of metaprogramming (especially ways to wrap define_method)
  • Extracting a more granular set of things to test in an application
  • Learning techniques for self-testing a test library

As with many mistakes, more than half of it is about learning rather than providing something useful. That’s kind of the point, I guess! I didn’t receive a single support e-mail or comment that it has helped anyone. Then I began to realise what I was doing was wrong and had managed to air my dirty laundry in the process.

An overview of the mistakes

With that small obituary to ActiveTest in its current incarnation, let’s look at the problems in the order in which I’ll be discussing them:

  • Removing test case transparency (an advantage of Test::Unit)
  • Extracting and abstracting without a real-world need or basis1
  • Excessive metaprogramming, which introduces ambiguity into tests (the cardinal sin)
  • Code bloat, plain and simple

The Idea: the original test case

I first came up with ActiveTest when I looked at this test case (from an old revision of one of my projects):

  def test_should_index_items
     assert_get_success :index
  end

At first glance, this looks almost identical to what ActiveTest currently does. I have a case where I want to ensure that a typical GET request receives a HTTP 200 response. I extrapolated HTTP 200 to HTTP GET into a request-response ‘success’ condition. The method was given a parameter (an action to call) that was passed to get and assert_template. All perfectly fine, a little DRY, but not unreadable. However, this test case and a bunch of others exactly like it formed almost the entire basis of my initial version of Active Test.

Coming Up Next: My First Mistake…

Footnotes

1 For a milder case of this, called premature extraction, see this article

Posted in , , ,  | no comments | no trackbacks

Older posts: 1 2 3 4 5 6