Thursday, 23 January 2014

MicroSpec

In the last two or three years I have been doing a lot of BDD, and I have really enjoyed it. Introducing BDD in a project for my latest client has been particularly rewarding. This company has previously relied heavily on manual regression testing and the majority of its QA staff were not trained sufficiently in development to be able to write their own automated tests. As mentioned this lead to a heavy reliance on manual testing but it also lead to something else: the unit and integration tests that were written by developers were largely ignored by the QAs because they did not understand them. "Not understanding" meant more than that they couldn't read the test source code and work out what the tests did. The tests were inconsistently named and poorly organised, and the QAs felt that they could not assess whether the appropriate test coverage had been achieved (real coverage, not a percentage of lines covered). This lead to unit and integration tests not being run as part of builds.

I've oversimplified this scenario - there are many reasons other than the QAs' inability to write and understand automated tests that led to this situation. However, the point is that that was the state of affairs - and things needed to be done. The reliance on manual regression testing meant releases were incredibly infrequent, and a full regression test cycle could easily take six weeks. If that's not madness, I don't know what is.

The company had brought in a new head of QA to sort out this mess, and his directive was fairly simple. Automate all tests that can be automated without excessive cost, write all tests as specifications (preferably with a Gherkin-style language and/or SpecFlow), and prefer BDD over TDD for new development work. Following this directive many existing manual regression tests were created as UI automation tests using SpecFlow and Selenium, existing integration and unit tests were cleaned up and made part of the build, and the company as a whole (well, more or less) attempted to embrace the BDD style of specifying test cases.

However, there was still the problem of new unit and integration tests. Developers across the organisation were used to writing their tests with NUnit and Moq. A typical, contrived, test could look something like:

[Test]
[ExpectedException(typeof(InvalidOperationException))]
public void ExceptionIfOrderIsNotDispatched()
{
  var order = new Order(){
                           Id= 12345;
                           //... other initialisation
                         };
  var refundAmount = 12.50;
  _orderService = new Mock();
  _orderService.Setup(x => x.Retrieve(order.Id)).Returns(order);

  var billingService = new BillingService(_orderService.Object);

  //throws because we haven't dispatched the order yet
  billingService.Refund(order.Id, refundAmount);
}

This test is simple and if you're a developer you can probably read it just fine. But if you're not a developer this is can be hard to understand. What are we testing, exactly? There is a comment in the test that says the order hasn't been dispatched and that's why an exception will be thrown - but what if the comment wasn't there? Even if you are a developer and you come across this test you're likely to have to go digging to really understand what the test is all about. Whether or not you think this style of writing tests is problematic is a matter of taste and preference, but what we realised in our project was that the non-developer QA who wanted to assess whether the right level of test coverage had been achieved would often be completely at a loss as to what had been tested.

In order to address this particular problem we decided that we should write all our unit and integration tests as specifications. We decided that the test we've already discussed should be written like this:
  
  Given I have an order
    And the order has not yet been dispatched
  When I try to refund an amount against the order
  Then an InvalidOperationException is thrown

This is a test that anybody can read and understand, regardless of technical ability, so we decided we'd write all our tests in this way so that anyone could look at any test, be it UI Automation test, database integration test or unit test and easily understand not only what it does, what it tests and why it failed (if that were the case).

The first, and rather obvious, choice of tooling for definig the tests was SpecFlow. However, we soon realised that it wasn't a good choice for this organisation. There was (not surprisingly) some resistance to having to use a new tool from the developers - why couldn't they just keep using NUnit like they had? Also, we quickly realised that although non-technical QAs now certainly could help define unit tests using Gherkin they typically didn't. So we decided to take matters in our own hands and create our own tiny framework, one that would give the best of both worlds: Specification style tests written using nothing but NUnit and Moq. I've called this framework MicroSpec.

MicroSpec is essentially one code file (I've taken a leaf out of Dapper's book) containing a single class, the Specification class, which implements four interfaces. The Specification class is an abstract class which executes your tests and prints the specification to the console as the tests run. You write tests by specifying each step in the test as a separate method and passing these steps as delegates to the Specification class' Given-When-Then methods. The test we've already looked at, above, could be written like this:

[Test]
public void ThrowsExceptionIfIssuingRefundBeforeOrderIsDispatched()
{
  Given(i_have_an_order).
  And(the_order_has_not_yet_been_dispatched).
  When(i_try_to_refund_an_amount_against_the_order).
  ThenAn().IsThrown();
}

Here we have three methods which define the steps in our test definition. They are all called by the specification class:
  • i_have_an_order
    creates a new order for use by our test
  • the_order_has_not_yet_been_dispatched
    ensures that the order created in the previous step has not been dispatched. This might execute some logic on the order or just assert that the state of the order is correct.
  • i_try_to_refund_an_amount_against_the_order
    would issue a refund against the order.
The last step in the test is a call to ThenAn which is a method that's used to capture expected exceptions. Chaining ThenAn (or it's brother ThenA for those who care about grammar) with IsThrown() asserts that a specific exceptions has been thrown during the course of the test (if you want to make sure an exception was _not_ thrown, you'd end the chain with IsNotThrown() instead).

As the test runs the Specification class creates console output before each step is executed. I'll repeat the output of this test here so you don't have to scroll:

  Given I have an order
    And the order has not yet been dispatched
  When I try to refund an amount against the order
  Then an InvalidOperationException is thrown

As you can see, the console output takes the names of the delegates/steps and prefixes them with the type of step: "Given", "When", "Then", "And", "Then an", "Then a", "is thrown" or "is not thrown". But the console output does not have to be constrained to just the name of the steps within the test. You can also include test parameters by following the following naming conventions:
  • integers surrounded by underscores, e.g. _0_, are replaced by the value of the corresponding parameter passed to the step.
  • integers preceded by the letter "g" and surrounded by underscores, e.g. "_g0_", are replaced by the name of the corresponding generic parameter defined by the step.
To illustrate, imagine that the value of the refund issued in the above test matters. You could then rewrite the test like this:

[Test]
public void ThrowsExceptionIfIssuingRefundBeforeOrderIsDispatched()
{
  var refundAmount = 20;
  var currencySymbol = "GBP";

  Given(i_have_an_order).
  And(the_order_has_not_yet_been_dispatched).
  When(i_try_to_refund_0__1_against_the_order, refundAmount, currencySymbol).
  ThenAn().IsThrown();
}

When run the output of the test would be:

  Given I have an order
    And the order has not yet been dispatched
  When I try to refund 20 GBP against the order
  Then an InvalidOperationException is thrown

Writing unit tests in this manner takes a little getting used to, and I find that it generally takes a little longer to complete the first couple of tests than when following a more traditional approach. That's because you've got a little more "infrastructure" to create in order for your tests to run. Managing state between test runs can also be a little trickier sometimes. But the tests themselves become infinitely more understandable and readable than tests written in a traditional way, and I think that the extra effort you have to put into defining your steps clearly in English language not only helps make your tests better but also aids in designing the software you're building. Describing the pictures you have in your mind with words while you're in the early stages of coding up a solution is a good way of vetting your design ideas.

Anyway - MicroSpec works for me, and it's bridged the gap between non-technical QAs who want to read and understand low-level (tests) and the developers that write them. It's the way I currently like to write my unit tests. If you want to give it a try, you can download it from BitBucket. Any thoughts and comments are most welcome.

1 comment:

Rohit P said...

Very useful information , and i was much in need of it
thank for this kind of real good information

and Thanks for share

We give best seo training across the nation digital marketing training in bangalore
seo training in bangalore