Unit Testing and Purity

Tim Lucas has a fine article on mocking and Rails testing which touches on some themes that I also hit on in an earlier post.

There is a tendency towards keeping functional tests pure in Rails.

Now, I find myself in a different position. I need to exercise my code as much as possible with the little time I have, so I like to get lots of testing bang for my buck. That means that the pragmatist in me wins over the purist who'd like to see each strata tidily tested in its own appropriate testing layer which does not so much as touch the code stink of another layer, let alone the putrescent code fart-bomb that is the database.

Frankly, I regard it as one of the strengths of Rails that I'm again close to my database. Unlike my magnificent Tapestry+Struts+Spring+Hibernate architectures of old, I'm again within earshot of something that actually has implications for my users, no longer in that level of coding hell where I was which is testing Data Transport Objects and testing the XML configuration of my DTOs and testing my database schema declaration so that it would not be altered while I was busy testing all those other things that I had to be testing to, ya know, save a record in a database -- all of course in perfect TDD abstraction from my database, database connection, web controller, and views, and pretty much in abstraction from the 6,000 things that can and will go wrong. But at least I know my DTO code is impeccable! A bullet-proof POJO! My business logic is flawless, portable. Oh, wait. I didn't write any business logic, which is properly abstracted away into a business logic container framework. Whatever. Come break me!

Testing purists say the solution to this is just more tests, and they are certainly right. The problem is that a small startup simply doesn't have the resources. All this code does come at a cost. Can we do things less purely but more efficiently?

What is crucial to a smallish application is code coverage, not purity of testing style. This is particularly true for uncompiled languages, where there is no such thing as compilation to give you even a smoke test on your code.

Five layers of well-segregated tests are just great, but one layer of impure tests is far, far better than any number of layers of pure tests where some layer somewhere has gone uncovered.

But this alternative approach of a single round-trip from user to database and back has costs of its own. You pay a price in test fragility. Fragility is a symptom of concerns that are improperly coupled.

Lucas has hit on this problem: Rails controller tests basically change each time you change your validations. Repairing them involves duplicating validation concerns within controller tests. Fixtures seemed originally intended to resolve this problem, but they don't lighten the load any, they just hide it off in another file, which can become a curse of its own.

He proposes using mocha to stub out ActiveRecord objects during controller testing. Problem is that views will make calls on all kinds of properties, and stubbing each property just recreates the same coupling problem that stubs were supposed to get around.

One solution is to turn rendering off, but it's just very very useful to have something exercising the rendering code as a basic sanity check.

So my suggestion? What if each model object kept a class method for creating a single valid instance of itself? Something like Object.fixture? At least the responsibility then remains with the model object itself, close to the declaration of its validations. And the controller tests stay uncluttered. You can change properties on that single instance in your test itself, if that particular value is what you are testing. This way, the controller tests do not break. And if you add validations, changes only need to be made in one other method.

A low-fi suggestion, to be sure.