Posts Tagged ‘metrics’

Misleading Metrics

February 15, 2011 Leave a comment

People like metrics. Numbers are facts: there’s no hiding behind excuses when what is being measured can be summarized in one nice, round, measurable number.


Wrong. So wrong.

Numbers on their own are meaningless, worrying and quite often distract from the bigger issue. Newspapers do it constantly; they may describe a particular action as doubling a certain risk, without saying if the original risk was 30% – affecting thousands of people – or 0.05% – affecting maybe one. One of the best books on the subject is The Tiger That Isn’t, which goes into far more detail than I possibly could; suffice to say, people understand significantly less about numbers than they think, and a number cannot stand alone.

A client recently came to us with a potential fault on a system we’d developed on their behalf, a system with around 600 individual users. Their CEO had logged on to take an online test, completed it and then discovered that none of his details had been stored; unfortunately, because of his diary commitments it wasn’t going to be possible for us to talk to him and try to discover any more details.

Rather than asking us to fix the fault, or even to track down a possible cause, the client wanted numbers and they wanted them immediately. We could tell within 5 minutes that hundreds of users were being shown as not having taken the course, but we also knew that this included users who really hadn’t taken the test.

If we had told the client that “up to 200” users were affected, it would have seriously damaged our credibility with them, whatever else followed. The number was a fact – a metric – but needed so many disclaimers as to be completely meaningless. Instead, we focussed on the knowns and the whys: we knew 400 people had successfully been through the system, so why was this account different? What was it about this single person’s account that meant their data had been lost while so many other people had been through successfully, some even at the same time as the CEO?

Half an hour’s investigation provided a more accurate metric, one which we were happy to pass on to the client. No record of the CEO’s test could be found, but we discovered that the CEO’s wife had also been added to the system, against the same email address. Our total number of affected records turned out to be “1”: the CEO had used his wife’s credentials to log in.

We know this client well, and by holding back from a metric that was based on pure speculation – namely, that there was a fault – we avoided causing panic. The only part we never comprehended was how the user himself never noticed. The username he had used was ‘Sandra5’. Maybe he assumed the ‘5’ meant something.

Categories: Development Tags: