Paul Moser

Code coverage is not all that it might seem.

Paul — Thu, 17 Sep 2009 21:27:11 +0000

Don't get me wrong, code coverage is a useful tool, but you need to understand what it is telling you if you aren't going to allow it to lull you into a false sense of security.

This surprises some people at first:

Code coverage does not tell you what has been tested

Code coverage tells you what code has been executed by the tests, it does not tell you that the code has run as expected. As it tells you what code has been executed the inverse is also true, it tells you what code hasn't been executed. Code that hasn't been executed you can be sure hasn't been tested.

What can be said is:

Code coverage tells you what certainly isn't tested.
Code coverage tells you at best how much of the code is tested.

Code coverage of 80% doesn't mean 80% of your code is tested, it means at least 20% of your code isn't tested.

A simple example to demonstrate this. A very simple Calculator class which can perform basic arithmetic, its tests and the coverage report.

PLAIN TEXT

JAVA:

package uk.co.paulmoser.examples.codecoverage;
public class Calculator {
public int add(int a, int b) {
return a + b;
}
public int subtract(int a, int b) {
return a - b;
}
public int multiply(int a, int b) {
return a + b;
}
public int divide(int a, int b) {
return a / b;
}
}

PLAIN TEXT

JAVA:

package uk.co.paulmoser.examples.codecoverage;
import org.junit.Test;
import static org.junit.Assert.assertThat;
import static org.hamcrest.CoreMatchers.*;
public class CalculatorTest {
private Calculator calculator = new Calculator();
@Test
public void addsTwoNumbers() {
assertThat(calculator.add(2, 2), is(4));
}
@Test
public void multipliesTwoNumbers() {
assertThat(calculator.multiply(2, 2), is(4));
}
@Test
public void dividesTwoNumbers() {
calculator.divide(5, 2);
}
}

The coverage report shows that the subtract() method isn't covered, and checking the tests there are no tests for that method. It's true that at least 20% of our code isn't tested but if we assume that the other 80% works fine as it is covered then we could be in for a nasty surprise in the future.

The coverage report shows that the divide() method is covered and the dividesTwoNumbers() test does indeed execute it, however the test makes no assertions about what the behaviour of the method, the code may or may not function correctly. Whilst the code does divide one number by another it is not clear whether it functions as expected, it is performing integer arithmetic, did the author expect the result of 5/2 to be 2, rather than 2.5?

It's not just missing assertions that mean that covered code isn't correctly functioning tested code. The code can be wrong or the test could be wrong and the test will still pass. There is a test for multiplication that does check for a correct result, however because of an unfortunate choice of test data the test still passes despite the multiple() method incorrectly returning the sum of the two parameters rather than the product.

From over coverage of 80% it seems that only 40% is actually working correctly, the remainder is either untested, faulty or behaves ambiguously. Whilst the Calculator is a simple class and it's fairly obvious where omissions and mistakes are in more complex classes if can be far less clear.

Code coverage also means nothing with regards to the quality of the tests, how readable, maintainable and communicative they are, but that's another subject.

Ruby-on-Rails on Ubuntu 8.04

Paul — Fri, 31 Oct 2008 12:59:56 +0000

Every time I build an Ubuntu machine I have to figure out all over again how to get it to work - so this time I'll write the steps down!

Install the packages: postgresql, rake, ruby, rubygems, rcov, git-core

Update the RubgyGems distribution by running
sudo gem update --system

If you run gem now you'll get the nasty looking error:
/usr/bin/gem:10:Warning: Gem::manage_gems is deprecated and will be removed on or after March 2009 /usr/bin/gem:23: uninitialized constant Gem::GemRunner (NameError)

You need to edit /usr/bin/gem and add an extra require:
require 'rubygems/gem_runner'

Install the Postgres driver:
sudo gem install postgres-pr

You'll need to change where Postgres writes it's socket file, if you leave things unchanged you'll see the error:
No such file or directory - /tmp/.s.PGSQL.5432

when attempting to connect to the database.

Edit /etc/postgresql/8.3/main/postgresql.conf and alter the unix_socket_directory parameter:
unix_socket_directory = '/tmp'

Restart Postgres using the init script /etc/init.d/postgresql-8.3

You should be able to leave your pg_hba.conf file unchanged - it should already look like:

# Database administrative login by UNIX sockets local all postgres ident sameuser # "local" is for Unix domain socket connections only local all all ident sameuser # IPv4 local connections: host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5

You need to create a database user, and give that user a password and create a database. As the postgres user:
createuser -lqs db_username createdb -O db_username db_name psql -c "alter user db_username with password 'db_password';"
In the last command above make sure you don't miss the ';' character, if you miss it the statement isn't actually executed, you should see the command confirmed with ALTER ROLE.

In you Rails application database.yml you need to specify the database name, database username and password and the host (even if it's localhost)
development: adapter: postgresql encoding: unicode database: db_name host: localhost username: db_username password: db_password

White screen after login to X on Linux (Ubuntu)

Paul — Fri, 31 Oct 2008 12:59:27 +0000

Given it's taken me a while to sort this I thought I'd write down what I did in case it helps someone else (it might help me again in the future too, I'm sure if it happens again I won't remember what I did to fix it)

So yesterday I installed VirtualBox 2.0, removed VirtualBox 1.5.x OSE and its related dependencies, and probably installed a whole load of updates too - I really can't remember exactly what I did - at the time it all worked fine but after I switched the machine back on today and logged in I don't see my desktop just a white screen and the mouse cursor.

Hitting ctrl-alt-F1 gets me to a command shell, so the machine hasn't locked up.

From struggling to get my ATI based card to work in the past I know that dpkg-reconfigure xserver-org reverts X to a very basic configuration so I tried that from the shell and then rebooted. dpkg-reconfigure xserver-org obviously had an effect as the screen resolution of the login screen dropped down but I still got the white screen.

Rebooting again and choosing the recovery option from the bootloader prompt I chose the "Try to fix X server" menu option - *try* to fix X - I like that although I didn't find it funny at the time. Reboot, login in - still a white screen.

A Google search from another machine threw up suggestions to remove the likes of
~/.config/compiz ~/.gconf/apps/compiz
This didn't help either, but thinking about it it may have done if I'd gone about things differently. X obviously isn't completely broken as you can get the login screen and a safe session (see later) also works fully. I think only removing the above directories (i.e. not doing any of the above dpkg-reconfigure etc stuff) may have worked if I'd been in safe mode when I removed them, as it was X was running and I used ctrl-alt-F1 to get a shell. I suspect that when I issued the reboot command and X shut down it just rewrote the files with the same values in them.

Finally I stumbled on a blog post. I did something similar, logged in using a "Gnome FailSafe" session, reinstalled the ATI drivers - I just downloaded them from http://ati.amd.com/support/drivers/linux/linux-radeon.html rather than using EnvyNG - and re-enabled the proprietary drivers under "System -> Administration -> Hardware Drivers" menu (which if I hadn't run dpkg-reconfigure xserver-org in the first place I might not have needed to do), rebooted and everything was ok.

If nothing else a "Gnome FailSafe" session is your friend, at least you can get online to find answers.

Don’t write tests to test your code

Paul — Tue, 22 Apr 2008 18:45:39 +0000

Wisdom tells us what we should do, experience tells us what we should never do again

The theory goes that you should write tests to check the functionality of your code, then when you refactor your code or add new features you've got a safety net to check that what you've done hasn't broken existing functionality.

The theory is good, but in practice it doesn't work.*

If you write tests to test your code you won't write your tests, at least you won't write all of them and you'll cut corners. Writing tests to test a piece of code you already have is tedious. You'll end up compromising your tests. You won't test the simple stuff, because, well it's simple, what could go wrong? It may be simple, but that's just the sort of thing you'll change without a moments thought but somewhere that'll have a consequence you didn't think of. You might skip testing the obvious stuff, but what's obvious to you isn't obvious to the next person to look at the code. You won't test the difficult stuff, because that's too time consuming and you've tested most of the other stuff so you figure you're fairly safe. You won't test the stuff that's 'impossible' to test. That safety net that makes you feel secure, the holes are too big - just when you really need it you'll fall straight through it.

So don't write tests to test your code, write the tests before you write the code. Write tests to drive the design of your code (TDD/BDD), write tests so you know when you're done. Only write code to fix a failing test. Yes you'll still find your tests becoming complicated and stuff that's difficult or 'impossible' to test. Listen to your tests - they are telling you something about your design. And the great thing is at the end of it, a side effect almost, you end with that safety net that really does allow you to refactor or add functionality to your code.

I think TDD/BDD and automated acceptance testing has the potential to significantly change not only how we initially produce a piece of software but also more importantly how it is enhanced and maintained throughout its 'working life' and it's that which is radically going to change the value the software can deliver to the business and so the test become crucial to the success of your business. More on that later.

* when blogging you're suppose to go for big bold controversial statements aren't you?

Baby Steps

Paul — Wed, 12 Mar 2008 13:59:07 +0000

Taking baby steps is often advocated for peoples adoption of Agile.

If you're coming from a waterfall background there is a lot to take on when becoming Agile. To truly become Agile you cannot just blindly follow certain practices, you need to understand them so you can apply them appropriately, if you try to adopt all at once you'll overload yourself, you won't understand what's having what effect, so yes, start by taking some baby steps but....

A few things I'd like to point out about babies:

they grow up
within a couple of years they'll be walking, indeed running
when learning to walk they are bold, almost reckless
yes, sometimes they fall over, sometime they even hurt themselves, but they learn from it, stand up and try again

I also worry that by just constantly taking small steps you loose your point of reference, that every small step moves you such a short distance you cannot tell whether it's moving you to success or to failure. Don't be a frog in a slowly warmed pan of water, be the frog who is thrown in, you'll be able to tell if it's boiling and jump straight back out.

Improving Java unit test readability and maintainability

Paul — Tue, 11 Dec 2007 23:17:06 +0000

The last year has seen me working with Ruby rather than Java and I have really come to like the the way tests (or rather specs) are structured in RSpec and the general way BDD approaches tests (specs).

RSpec splits the behaviour of a class up into what happens in different contexts. An example probably explains things better, taking a trivial case of a stack (I'll get to Java in a minute but even if you don't know Ruby I think this code should be understandable):

PLAIN TEXT

RUBY:

require File.dirname(__FILE__) + "/stack"
context "A populated Stack" do
setup do
@stack = Stack.new
["a","b","c"].each { |x| @stack.push x }
end
specify "should add to the top when sent #push" do
@stack.push "d"
@stack.peek.should == "d"
end
specify "should return the top item when sent #peek" do
@stack.peek.should == "c"
end
specify "should NOT remove the top item when sent #peek" do
@stack.peek.should == "c"
@stack.peek.should == "c"
end
specify "should return the top item when sent #pop" do
@stack.pop.should == "c"
end
specify "should remove the top item when sent #pop" do
@stack.pop.should == "c"
@stack.pop.should == "b"
end
end
context "An empty stack" do
setup do
@stack = Stack.new
end
specify "should be empty" do
@stack.should be_empty
end
specify "should no longer be empty after #push" do
@stack.push "anything"
@stack.should_not be_empty
end
specify "should complain when sent #peek" do
lambda { @stack.peek }.should raise_error(StackUnderflowError)
end
specify "should complain when sent #pop" do
lambda { @stack.pop }.should raise_error(StackUnderflowError)
end
end
context "An almost empty stack (with one item)" do
setup do
@stack = Stack.new
@stack.push 3
end
specify "should not be empty" do
@stack.should_not be_empty
end
specify "should remain not empty after #peek" do
@stack.peek
@stack.should_not be_empty
end
specify "should become empty after #pop" do
@stack.pop
@stack.should be_empty
end
end

So a stack should behave in a particular way:

A populated Stack
- should add to the top when sent #push
- should return the top item when sent #peek
- should NOT remove the top item when sent #peek
- should return the top item when sent #pop
- should remove the top item when sent #pop

An empty stack
- should be empty
- should no longer be empty after #push
- should complain when sent #peek
- should complain when sent #pop

An almost empty stack (with one item)
- should not be empty
- should remain not empty after #peek
- should become empty after #pop

(incidentally the above is output generated from the spec tool, a bit like generating JavaDocs from Java code)

I think splitting the behaviour up into these different contexts greatly improves the readability and maintainability of both the tests themselves and the code under test. But how to do it in Java?

It's possible to use RSpec to test Java code because you can run RSpec using JRuby, however I'm not sure if it is wise to drive the development of code in one language with tests in a different language, Ruby and Java approach numerous things very differently.

Now it would be possible to have numerous JUnit test classes for a given class, each one for a different context but this has it's drawbacks, when you make a change to your class you'd need to run numerous test cases, which you could get around by having a master testcase with a suite method that includes all the others, but then you have to maintain the list of testcases in the suite() method and it's all too easy for the list to get out of sync with the actual test cases. JUnit 4 provides a solution, you can use the Enclosed runner. The code below is an equivalent example to the RSpec one in Java.

PLAIN TEXT

JAVA:

import static org.junit.Assert.assertThat;
import static org.hamcrest.CoreMatchers.is;
import java.util.EmptyStackException;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Enclosed;
@RunWith(Enclosed.class)
public class TestStack {
public static class AnEmptyStack {
private Stack stack = null;
@Before
public void setupAnEmptyStack() {
stack = new Stack();
}
@Test
public void shouldBeEmpty() {
assertThat(stack.isEmpty(), is(false));
}
@Test(expected = EmptyStackException.class)
public void shouldComplainWhenPeeked() {
stack.peek();
}
@Test(expected = EmptyStackException.class)
public void shouldComplainWhenPoped() {
stack.pop();
}
@Test
public void shouldNotBeEmptyAfterItemIsPushedOntoIt() {
stack.push(new Object());
assertThat(stack.isEmpty(), is(false));
}
}
public static class AStackThatIsNeitherEmptyNorFull {
private Stack stack = null;
@Before
public void setupAStackWithThreeItems() {
stack = new Stack();
stack.push("one");
stack.push("two");
stack.push("three");
}
@Test
public void shouldAddToTheTopWhenSentAPush() {
stack.push("four");
assertThat(stack.peek(), is("four"));
}
@Test
public void shouldReturnTheTopItemWhenSentAPeek() {
assertThat(stack.peek(), is("three"));
}
@Test
public void shouldNotRemoveTheTopItemWhenSentAPeek() {
assertThat(stack.peek(), is("three"));
assertThat(stack.peek(), is("three"));
}
@Test
public void shouldReturnTheTopItemWhenSentAPop() {
assertThat(stack.peek(), is("three"));
}
@Test
public void shouldRemoveTheTopItemWhenSentAPop() {
assertThat(stack.pop(), is("three"));
assertThat(stack.peek(), is("two"));
}
}
public static class AStackThatContainsOnlyOneItem {
private Stack stack = null;
@Before
public void setupAStackWithOneItem() {
stack = new Stack();
stack.push("one");
}
@Test
public void shouldNotBeEmpty() {
assertThat(stack.isEmpty(), is(false));
}
@Test
public void shouldNotBeEmptyAfterAPeek() {
stack.peek();
assertThat(stack.isEmpty(), is(false));
}
@Test
public void shouldBeEmptyAfterAPop() {
stack.pop();
assertThat(stack.isEmpty(), is(true));
}
}
}

As an aside I've also used the new assertThat() method and Hamcrest matchers as you get significantly better error messages when your tests fail that way.

To get a plain test output you could probably write a XSLT stylesheet that parses the test report, the stylesheet Kerry produced for use with Junit 3.x tests maybe a good starting point, or write a custom formatter and specify that in your JUnit Ant task.

There are also the JBehave and JDave BDD frameworks for Java which would be worth looking at if you like the look of BDD.

How does your team sit?

Paul — Tue, 11 Dec 2007 23:16:29 +0000

As in many UK offices we don't exactly have cubicles in my office, more like bays lined with, in our case, three desks on each side facing outwards, and between each bay is a partition which, unless you are particularly tall, you can't see over when sat down.

Typically a bay is assigned to a team so they can all sit together. However, what does it say about the importance of team work and communication that the way the office is set up you actually sit with your back to your colleagues? That's just wrong.

A few words of wisdom

Paul — Tue, 19 Jun 2007 22:34:35 +0000

Me: So much to learn but so little time.

Andy: But wouldn't it be so much worse if there was time and there was now nothing left to learn

They may have been his own words, or he may have been quoting someone else but in the absence of any evidence to the contrary (a quick Google search) I'll give Andy Settle the credit.

I was reminded of this little gem by a conversation with colleagues the other night in the pub where there was a regular expressions book lying around and I commented that sometime I really should get around to learning regular expressions properly (I can do basic ones with the help of a reference page and know the right people to ask for more complicated cases). A colleague then quoted my blog tag line to me, which was a little spooky as the above quote was from the end of a conversation where Andy (one of those 'right people') helped me out with a regular expression.

Can you use Agile for large projects?

Paul — Fri, 08 Jun 2007 06:45:03 +0000

So maybe Agile works for small projects but it couldn't possible work for large projects? You couldn't work in iterations for a multi-million, let alone billion, pound project with thousands, let alone tens of thousands, of people, you'd need to carefully gather all the requirements, draw up a detailed design, put together the project plan and carefully stick to the plan?

Well the biggest engineering project I can think of is the American space program of Mercury, Gemini and Apollo which ultimately put a man on the moon.

It is difficult now to appreciate how little was known about space. Assuming that the Soviets had not lied about Yuri Gagarin NASA knew it was possible for a human to briefly survive in space, but that is all they knew, they did not know even the basics, could a man drink in space, could a man eat, could he sleep let along the longer term physiological effects of being weightless for days. Then there was the mechanics of space flight, docking two space craft, men operating outside of a vehicle in space, the list is almost endless.

So what NASA do? Well they didn't come up with a grand design, built it and deployed it (sent it to the moon). Project Gemini was a series of 10 manned missions, each one building on the knowledge gained from the previous. Apollo then built on Gemini.

Gemini 3 - Cautious test flight, just 3 revolutions
Gemini 4 - First American space walk
Gemini 5 - Long duration 8 day flight,evaluated guidance and navigation system for future rendezvous missions
Gemini 6 - First space rendezvous
Gemini 7 - Long duration 14 day flight
Gemini 8 - First docking in space
Gemini 9 - 3 different rendezvous
Gemini 10 - First use of Agena's power
Gemini 11 - Altitude record using Agena propulsion system
Gemini 12 - Aldrin set an EVA record of 5 hours, 30 minutes
Apollo 7 - First test flight
Apollo 8 - 2nd manned flight, all the way to the moon
Apollo 9 - First flight of CSM/LM combination (Earth orbit)
Apollo 10 - Dress rehersal, LM taken within 50,000 feet of lunar surface
Apollo 11 - Armstrong and Aldrin walk on the moon

Typically Agile projects are time boxed with scope being variable, with the space program both time ("before the decade is out") and scope ("of landing a man on the moon") were fixed, as was quality ("and returning him safely to earth"), it was cost that was variable.

Ok, so the projects you've worked on haven't been as intimidating or as significant to mankind as sending someone to the moon but if you're honest with yourself there were probably a great many unknowns when you started and your future projects are likely to be the same.

So what prompted this post, after all Apollo ended over 3 years before I was born? I read Andrew Smiths "Moondust - In Search of the Men Who Fell to Earth" and "Carrying the Fire: An Astronaut's Journeys" by Michael Collins (he's the third astronaut on Apollo 11) , both highly recommended.

Unit testing interface implementations

Paul — Sun, 25 Feb 2007 23:41:21 +0000

It's bugged me for a while about how best to go about unit testing classes that implement multiple interfaces where those interfaces are implemented by multiple classes or where a class extends another class and implements one or more interfaces.

What I want to avoid is in each test class implementing the the same set of tests for the behavior specified by the interface. That's not very DRY.

If your classes only implement a single interface then you can have a base class with the tests for the interface and then separate test classes derived from that for each class implementing the interface. The problem arises when your classes implement multiple interfaces - Java doesn't support multiple inheritance for classes so you can't have a base test case for each interface.

One answer with JUnit 4 is to use the parameterized test runner - I haven't thought about JUnit 3, if you're not using JUnit 4 because you're still using a JVM that doesn't support annotations JSE 6 is out now, catch up!

You need to have a test case for your interface and configure this to run with the parameterized runner. You also have a static factory method that provides the various implementations of the interface to be used in each test.

As an example, say the container I'm writing (I'm not really - I know there is Apache Geronimo, JBoss, Hivemind etc - this is just an example) supports services that it can start and stop, so I have an interface Service and the various services that I have, e.g. caching, lookup, logging etc, implement this interface.

PLAIN TEXT

JAVA:

/**
* Represents a service that the container can run.
*
*/
public interface Service {
/**
* Starts the service, may only be called if the service is not started.
*
* @throws IllegalStateException
* if called when the service is already started.
*/
public void start();
/**
* Indicates whether the service is started
*
* @return true if the service is started, false otherwise
*/
public boolean isStarted();
}

I have a test case for Service that tests that the various implementations of Service adhere to the general contract. I might have the following three tests:

PLAIN TEXT

JAVA:

public class ServiceTest {
private Service service;
@Test
public void shouldNotIndicatedStartedForNewObject() {
assertFalse(service.isStarted());
}
@Test
public void shouldStartIfNotAlreadyStarted() {
service.start();
assertTrue(service.isStarted());
}
@Test(expected = IllegalStateException.class)
public void shouldThrowExceptionIfStartedCalledOnAStartedService() {
service.start();
assertTrue(service.isStarted());
service.start();
}
}

If I have a caching and lookup service and I want to test that both adhere to the contract of the Service interface then I have to annotate the ServiceTest to indicate that it should run with the parameterized runner and provide an annotated factory method that returns a collection of arguments to run the tests with.

So the full testcase is:

PLAIN TEXT

JAVA:

@RunWith(Parameterized.class)
public class ServiceTest {
private Service service;
public ServiceTest(final Service service) {
this.service = service;
}
@Parameters
public static Collection implementations() {
return Arrays.asList(new Object[][] { { new CachingService() }, { new LookupService() } });
}
@Test
public void shouldNotIndicatedStartedForNewObject() {
assertThat(service, is(not(started())));
}
@Test
public void shouldStartIfNotAlreadyStarted() {
service.start();
assertThat(service, is(started()));
}
@Test(expected = IllegalStateException.class)
public void shouldThrowExceptionIfStartedCalledOnAStartedService() {
service.start();
assertThat(service, is(started()));
service.start();
}
}

The test classes for CachingService and LookupService now only need deal with testing the specifics of caching or lookups rather than testing adherence to the service specification.

I have yet to use this in production code on a large scale so we'll have to see how it works out, but it seems far better than having to replicate test code amongst several test cases.