kill -3 to the rescue!

A basic and ancient weapon to keep close at hand when hunting down deadlocks and other weird threading issues in Java applications is the kill -3 command. I don't know of an equivalent on Windows, but if you are running on any flavour of Unix (including Mac OS X) it is available via the command line.

If you've never tried it, basically it gives you a dump of the stack traces of all living threads in the process. Despite the name, when you use the -3 option, it doesn't kill the process. This is handy if you want to take multiple snapshots a few seconds apart to see if anything changes.

Even better, it also shows you what objects have been locked, where, by which threads; and which objects are waiting for those locks. In no time at all you can find the source of the deadlock and do something about it.

I've only used this in development, but there's no reason you couldn't use it in production, provided your stdout and stderr are redirected somewhere you can see. Since it doesn't kill the process, it's entirely plausible you could find a workaround for some problems on the running system.

EasyMock Rock

Writing unit tests = good.

JUnit = good.

However...

Writing unit tests ≠ fun,

especially when dealing with classes that need a lot of stuff to work.

This is especially annoying when you are trying to test a relatively simple class, but it operates on a complex (in terms of data) class. You have to go through a lot of crap just to create the data that your relatively simple class needs to demonstrate that it works properly. This is the difference between unit testing examples in a book and unit tests in the real world.

Enter EasyMock.

Now very mature and well-liked in the Java community, EasyMock creates mock objects automagically and lets you get back to worrying about the class you're actually trying to test. Bravo!

Not only does it take HUGE amounts of pain out of writing JUnit tests, but it also has one of the most beautiful DSL-style APIs I've ever used. When you use it, your code reads like English. It's just a real pleasure to use, and thanks to the gorgeous API, the learning curve is very short.

I think every JUnit test I have written without EasyMock has shortened my life span.

Use EasyMock = Live Longer

Reflections on Quality

In October my team released a new version of our system, which represents several months of extremely hard work and is the culmination of everything we've learned so far. (It's also the reason I haven't posted for a long time.) We were all very proud of what we'd created, and confident that it was going to solve all of our (and our users') problems.

But from only 2 weeks after going live, we started having a string of major problems - almost every day something horrible happened. It was almost unbelievable, and I'm sure one day we'll look back and laugh, but right now it's an absolute nightmare.

After yet another disastrous week, I spent a weekend thinking about "quality" in software development, and trying to figure out where we went wrong. As I have done before, I made a pilgrimage to Joel Spolsky's site to get some perspective. I read and re-read The Joel Test - a simple list of 12 yes/no questions that works like a scorecard for software development teams. Joel says, "A score of 12 is perfect, 11 is tolerable, but 10 or lower and you've got serious problems."

We scored 3.

It was depressing, sure, but I was already so depressed anyway, I was actually just happy that here at least was a list of things I could do to try and correct the situation. Also, it was nice to know what we are already doing well.

The first thing that jumped out at me was #5 - "Do you fix bugs before writing new code?" I think this is really a fundamental part of where we went wrong. If you have bugs, that means you're not perfect (nobody is!), which means any new code you write will have bugs, too. So by not stopping dead and fixing the bugs you already have, the number of bugs can only increase. Obvious when you think about it like that, but easy to forget when there is a lot of pressure to implement new features. I can only imagine what it must feel like to know that there are no bugs (that your best efforts could find) in the production code. I'm sure it's a lot better than knowing that there are!

So the first action item: Stop all new development. Fix all the bugs we know of, then go & find some more and fix those too.

The next big one was #7 - "Do you have a spec?" To tell the truth, we have almost zero documentation. As Joel says, "Writing specs is like flossing: everybody agrees that it's a good thing, but nobody does it." I would expand that to cover all documentation. So far our team has been operating under a sort of modified XP credo, where anything that even faintly smells of an old-school waterfall methodology was implicitly rejected. But the reason #7 stood out to me was that in a couple of incidents, the users were asking us what the system does in this case, and we didn't know. The users didn't know, and neither did we. That's bad. If we had a spec, they wouldn't even have had to ask, and anyway there wouldn't have been a problem, because the system would have done what they wanted (instead of what it was doing!)

The worst thing I found when examining our methods was that we had all but abandoned our sacred Law One anyway. So it wasn't that our philosophy was wrong, it was that we had forgotten it, to our peril. But how do you know when you have enough testing? Code coverage tools are fantastic, if a little fiddly to set up, but you really need a human brain to imagine scenarios in which the code you're looking at might fail.

So the second action item: Build documentation and test coverage reviews into the process that puts code into production.

Speaking of reviews, our code review process was extremely weak. Basically it consisted of just me having a look at the code changes as I packaged them for release. But even then, with so much (perceived) pressure on time, I must admit there are huge swaths of code which has never been seen by anyone other than the developer who wrote it. Uncool.

So the third action item: Build a 2-level code review into the process that puts code into production.

The combination of documentation review, test review, and code review I've called the "critical review", meaning both that the reviewer should try to be as critical as possible, and that it's of critical importance. It's difficult to be critical of the work done by your friend, especially if they are relatively senior to you, but the important thing to remember is that every problem you find is saving them from the crippling shame and panic that grips you when code you wrote has caused a production problem. So I've told everyone to try as hard as they can to find a problem when it's their turn to review.

The final test that made me shudder was #6 - "Do you have an up-to-date schedule?" I've always hated giving estimates to the users. It's never much more than a wild guess, and yet they really hold you to it. You can't get away with saying, "it'll be done when it's done!" either.

I read Joel's article about Evidence-Based Scheduling (EBS), which quite frankly struck me as a shameless plug for his bug-tracking product, FogBugz, since it's virtually impossible to implement with anything else. I like his previous idea much better, mostly because of its simplicity, but also because it lends itself easily to constructing a burn-down chart to catch potential delays early.

Basically, the big message for me was we have to generate a new estimate every day. The moment before we start working on something is the moment we know the least about the work involved. Every day we work on it we know a little bit more, so it makes sense to re-invest that knowledge into the quality of the estimate - and if this is done for every in-progress task, the reliability of the whole schedule increases every day too. By giving the users a new schedule every day/week, they don't get any big shocks from huge and sudden delays, and every little slip can be explained in detail.

A key requirement for this is that the estimates are provided by the developer who will do the work. Again this is because nobody knows the work better, but it also prevents unrealistic schedules from being handed down from above.

Another important rule is that estimates and work logged must include all the time doing non-coding stuff - code reviews, talking to people, helping other developers, setting up environments, testing, documentation, picking your nose - everything. In essence the estimate and work logged for a task must be the total time spent in the office not working on a different task. That way everything adds up to a number of calendar days, which in the end is the only thing that matters to the schedule.

So the fourth action item: Estimate everything, and update the estimates every day.

The actual implementation of all this is in our issue tracking system (JIRA). I've added a few custom fields and worked out a process flow for each issue/task, so it gets estimated and re-estimated, reviewed, and scheduled. I'm sure that if we can stick to it, our quality problems will disappear - the trick is now to make sure the extra burden it places on us is more than repaid by a reduction in production panic.

At least it will get us a score of 6...

 

3 Laws of Software Development

One of the most important things to come out of the after-work beer-fuelled pub braindump sessions with John is our "3 Laws", which we thought of as we reviewed what we did right and what we did wrong on our project. We have them pinned up in big letters in the office. Here they are:

1. Test First

Yes, yes. Everyone has been saying this for ages. There must be a thousand blog entries about Test-Driven Development and Extreme Programming. Of course we had heard about XP and I even own Martin Fowler's great book on refactoring, but to actually follow the principles of XP takes a crapload more self discipline than we have. So I'm not necessarily talking about JUnit tests here, but I definitely insist on heavy, if not total, automation.

The "project" we're working on is to replace a legacy system that does something we didn't know a lot about, written by a guy who left the company, with no written spec. This is the perfect problem to solve with Law 1, because the basic idea is that your test cases are the spec. And the beautiful thing is, now we do follow this Law, we can test the test cases by running them against the legacy system, before running them on the new system. Brilliant.

This approach has also been great for benchmarking the old system vs the new system, so we can say things to my boss like, "the new system is 30 times faster than the old one - see, here's the test results to prove it."

2. Optimise Last

Dr Knuth said, "premature optimization is the root of all evil" - and it's true. I got that quote from an excellent book about Java performance, which encourages continuous benchmarking (see Law 1). Another book I have about NIO makes the important point that these days software tends to be IO bound rather than CPU bound. (That means your program probably spends much much more time talking to a DB or to the OS about files or network, than it does calculating or manipulating memory.) That point alone implies that most of the time optimising algorithms gains nothing at all!

That doesn't mean that you shouldn't ever optimise algorithms of course, but you should only do it if these two conditions are satisfied:

1. continuous benchmarking shows that something that used to be fast is suddenly very slow
2. your profiler software finds a hotspot

Most of the time however, I have found that performance improvements tend to come from the architecture level. For example, if you discover your JVM spends a lot of time garbage collecting, then there's probably some leaky code or unnecessary data copying going on somewhere. Usually the fix for that would be a micro design change that avoids the need to copy that data.

3. Write Code for Right Now

The phrase "I think Law 3 applies" has become interchangeable with "we'll cross that bridge when we come to it" in our team. This is the anti-framework Law. Object-oriented programmers have a natural (and evil) tendency to write über-generic classes, in the name of code reuse. Code reuse is a good thing, but too often we write a generic, reusable class, and then use it only once. In extreme cases that reusable class evolves into a framework, which just makes maintenance harder because every time you change it you have to consider both how it will fix the problem at hand and how it affects the other imaginary users of the framework.

The other natural (and evil) tendency that this Law avoids is adding unrequested features, and fulfilling fictional requirements. Yes, the users might ask for Feature X next quarter, but they haven't yet, so let's just write the simplest possible thing that will meet the requirements we know about Right Now.

This can be a tough Law to obey because let's face it, you only think of disobeying it when doing so would be fun. If I'm finding myself extra tempted my usual trick is to jot down the technical idea I have on a post it note, and tell myself that if I finish the real requirements quickly I might have time to do it later. Of course I never have time later - but that's the point: there's no time to waste writing code that isn't needed.

Summary

3 is a convenient number of Laws because you can memorise all of them easily, but in truth we've never felt a need to add a 4th Law. These seem to be the key to everything. When we obey them, good things happen. When we disobey them, bad things happen. Simple as that. I hope they will be useful for other people too.