Testing & What Needs Improvement
Posted by Mathew Abonyi Sat, 12 Aug 2006 13:00:00 GMT
Since Nicolas Paton commented earlier about Simple Access Control having no tests, I’ve held off on my other plugin, Simple Scope, which I’ll release only after it is thoroughly tested. This situation has brought me back to testing as a whole for anything I’m coding – where I was quite happy for a while just to get things working.
In a suite of tests, there are an unending slew of cases, most of them very similar, that are ensuring basic functionality is being implemented. For example, if you are following CRUD/REST conventions you will have lots of these:
def test_should_show_blah
get :show
assert_response :success
assert_template 'show'
end
def test_should_restrict_new
get :new
assert_response :redirect
assert_redirected_to login_url
end
def test_should_list_blah
get :index
assert_response :success
assert_template 'index'
endYou may have even refactored these across multiple test cases to get something like this:
class SomethingControllerTest < Test::Unit::TestCase
def setup
...
end
basic_crud_assertions
def test_should_do_something_special
... etc ...But there’s usually a slight difference between them and it is the minor variations for each case that makes these ‘core’ tests rather time consuming. We can type them very fast, no problem, but sometimes they throw up an unexpected failure or error. Not because of a bug…. because of a typo or wrong assert. Or maybe there is a bug, but it’s missed because of a lack of some assert or another. If this was the case with ActiveRecord, everyone would be up in arms: “Oi! Why can’t we move to higher levels? What’s this connection.quote rubbish? I don’t want to deal with that; give me asssociations, aggregations, calculations, validations…”
But, and this is very important, Test::Unit::TestCase is NOT very extensible. If you look at all those Acts::As::Hippos and ActiveForms, you’ll see they’ve got hooks galore into ActiveRecord::Base, Calculations, Aggregations and the rest. There are methods just there to be hooked into, aliased, overloaded and re-evaluated. Anyone who wants to write a quick plugin just need turn to ActiveRecord or ActionController’s highly extensible nature. Test::Unit::TestCase has methods so Spartan that you have two options: overloading setup/teardown (pointless because it is usually overwritten) or you have to rewrite run()... and you can guess what that’s for.
I have also looked into ZenTest and feel rather ambivalent about the work they are doing. The concept behind it is very similar to what I am discussing. Rails needs its own higher level testing framework than just Test::Unit, which is fantastic because it is basic, but also insufficient for the same reason. Test::Rails in ZenTest is, from my first glance, a little too brittle and invasive. There must be a middle road where Test::Unit::TestCase can be overloaded and rewritten within Rails terms while still using what is in the Rails core for Test::Unit::TestCase. A couple of beefs I have with ZenTest:
- requires additional setup instead of additional methods
- requires a little arcane knowledge to get it working
- zentest replicates some of the work in rcov (which is more complete and useful)
- purpose of unit_diff is also mostly superseded by rcov
- Test::Rails doesn’t work out of the box (for 1.1.4 – 1.1.6 at least)
I suppose it’s time for yet another plugin (or gem). If what I’m saying here is genuinely felt by other Rails developers, just give a little shout. If there are enough out there, I’ll seriously pursue a plugin or gem that creates a full layer over Test::Unit to make it extensible with an eye to getting Core to implement it. I think that addition in itself will allow the work of ZenTest, rcov and others to work on test extensions without individually rewriting aspects and being incompatible with each other.
Have you looked at RSpec? http://rspec.rubyforge.org/
You might find it more to your liking. I don’t know how well it integrates with Rails, but it’s worth looking into.
I have looked into RSpec briefly and it seemed to be a redefinition of Test::Unit from suites and cases into contexts and behaviours, but you still encounter the same verbosity: context “an empty stack”, context “a stack with one item”, etc.
Making tests DRY would not only open it up to more people, but make agile development much faster—faster than any single tool could. I envisage something much like Test::Unit::Base, expanding from which would be modules like Assertions, Statistics, Suites and Authentication (comparable to modules for ActiveRecords), and Rails-specific classes like Controllers, Helpers, Views, Models, Stress and Integration (comparable to ActiveRecord descendants). Much of Controllers, Models and Integration are already done by the Rails core, so it is just a matter of making it all consistent. Any aspect of the Rails environment can then be extracted into simple one-liners.
For example: index_should_list_items. That class method will dynamically create a test method to check whether you can call the index action, whether it returns 200 and whether an array of items are set. You can pass options to each class method, just like in ActiveRecord or ActionController, to define more specifically your needs for each test or group of tests. The next aspect would be grouping tests within suites to make it easier to read and making the eval dump out its own test code in case something is wrong with it. As you can see, it gets quite complicated, but I think creating the foundation, if done correctly, will open up a whole world of plugins… for tests!
So, to answer your question, it’s more a question of metaprogramming than style; RSpec I would say is more about the style of test rather than making them shorter and quicker to write.
I do, however, like the style of RSpec. You may wish to look at the Rails plugin: http://lachie.info/svn/projects/rails_plugins/rspec_on_rails/
Okay, I see what you mean now. Quickly, an anecdote:
I recently worked on a project which had a nice suite of tests built up. The problem was they were extremely DRY. Tests should (of course) test your code but also should display how your code is to be interfaced with. If you are spending time digging through test files and test helpers in order to understand what is going on, you might as well just dig through the code proper.
I’ve always felt it’s way okay to repeat yourself in tests because one can then look at each individual test and get the Big Picture without having to sift through other elements of the testing suite. Would introducing the library you propose further complicate what should be a simple and useful part of Rails?
Another anecdote: just yesterday I walked my company’s CSS-wiz/designer through some unit tests for a plugin. He instantly understood – I was explaining how the plugin worked and only had to say “when you run a test file, each test_ method gets run once. The assert method tests what you expect against what you got. If there’s a discrepency, the script tells you.” That’s as simple as it gets.
Okay, that said, I really like your proposal. I’d be willing to use it and contribute. I’m just not convinced a big testing library like this would actually make things easier.
I agree with you about Test::Unit::TestCase not being very extensible. The modifications made by ActiveRecord for fixture setup/teardown don’t help.
While developing Mocha I wanted to add multiple setup and teardown methods into the standard TestCase without subclassing it. I’ve come up with the MultipleSetupAndTeardown module. Just include this into TestCase and call the class methods :add_setup_method & :add_teardown_method. It should be compatible with Rails (at least v1.0)
Chris, I completely agree, but to show exactly what I think by making tests more DRY without losing their documentative qualities, I think it is better to have an example. Have a peek at the code for my latest plugin, Active Test.
James, I had a look at Mocha and I’m quite impressed. It is well documented, tested and designed. A good gem, all round, but with regards to the MultipleSetupAndTeardown, have you considered this method? Both have their faults, I suppose. The problem is that both those extending Test::Unit and those in Rails Core are just monkey patching Test::Unit::TestCase rather than wrapping and hiding it from view. That creates problems for anyone extending it after those files are required, like the notorious active_record/fixtures.
I’m glad you like Mocha. People seem particularly interested in the Stubba part of it which allows you to mock and stub methods on concrete instances and classes.
Thanks for pointing out your solution to the multiple setup/teardown conundrum. Definitely some food for thought. One thing I realised we are both missing is to ensure all teardowns are run even if one raises an exception.
I totally agree with you regarding the monkey-patching – in fact I’ve just responded to a post from Nathaniel Talbott (author of Test::Unit) on the rails-core mailing list. It sounds like he may be about to solve our problem.
Perhaps Nate will fix setup/teardown, but there’s also the general problem of modularity. For example, making Test::Unit’s filtering system more open so arbitrary namespaces can be properly ignored. For the most part, Test::Unit will stay as it is, I think. Fast, inflexible, simple. If Nate does make it more extensible, excellent and all kudos to him for doing something much needed by the Rails community.
But between Mocha and ActiveTest (which still needs finishing touches before final release), a lot of the issues with Test::Unit are dealt with for Rails developers. Not so sure that teardown or setup should force their way through an exception, though. It’s quite easily done, for sure. Perhaps an option passed to either of these testing frameworks?