All models are wrong (or: lies, damned lies and statistics)*

The statistician is seen with a certain amount of disdain (or possibly sympathy) by their pure mathematical brethren. And it is with that firmly in mind that I (as a fledgling statistician) take the reins of this worthy blog.

We have some idea of what mathematics is from Adam’s posts; but what is statistics?  Statistics is applied maths with uncertainty. In statistics mathematical techniques are used to model and quantify our uncertainty about reality. Modelling climate change, predicting the outcome of elections, wrecking the financial system and ensuring the casino always wins: statistics is everywhere. And uncertainty is the key to statistics.

In order to get across an understanding of what uncertainty is I will try to describe some of the different kinds we face and how statistics deals with them.  The five levels in the following taxonomy lie on a continuum running from complete certainty to complete uncertainty, and provide a means of measuring the range and limitations of statistics in different situations.** The further we go along this continuum the less effective statistics is at prediction and inference, and many problems in statistics and quantitative social sciences like economics come from not recognising just how far along the continuum we are.

Level 1: Complete Certainty

This is where most of maths resides. A priori truth. Facts are facts, largely unchanging and immutable. 1+1 is always going to be 2, $\pi$ is the ratio of the circumference of a circle to its diameter. You can be certain of it. Maths is the exploration and development of this land of hard facts, and aside from maths, not much other human endeavour lives here.  Some logic is certain (all men are mortal; Socrates was a man; therefore Socrates is mortal, etc), and perhaps some of the laws of physics (although since the advent of probability in quantum theory, complete certainty has become elusive in even this formerly so self-sure subject).  At this stage, mathematics rules.  And from the point of view of the statistician, this category is not very interesting.

Level 2: Certain Uncertainty

At this level we have events with uncertain outcomes that are governed by mechanisms which we understand completely. The canonical example might be an idealised fair coin toss, in which there is a 50/50 chance of each side of a coin landing. The statistics here are simple: if you toss a coin 100 times you would expect around 50 heads and 50 tails. But you might get 40 heads and 60 tails. Or even 0 heads and 100 tails. There is uncertainty here but it is fairly easy to quantify and predict outcomes. Using simple tools like the binomial distribution we can work out the probability of each of the outcomes above occurring (0.08, 0.01 and 7.8 $\times$10-31 respectively). This is like being in an honest casino where the rules are known by all in advance and they are always followed. This type of uncertainty can be very well predicted but is fairly rare. The real world tends to be more complex.

Level 3: Fully Reducible Uncertainty

At this stage events can be observed to follow the kind of patterns found in level 2 phenomena, and with enough data, and a bit of statistics, these type of events can be predicted with as much certainty as level 2 events. This is like being in a casino which follows rules, but in which the only way to determine what those rules are is through observing gamblers and inferring the odds. As long as the rules stay the same – and we can make enough observations – we can determine what the rules are and reduce the uncertainty to level 2.

This is the region of uncertainty where a lot of physical sciences and some human activity takes place. It is the ordered world of controlled experiments and predictable outcomes: test tubes, rats in cages and production lines. At this level of uncertainty the rules exist; we just have to discover them. This region relies on regularity, which seems to be present in a lot of the natural world. If you take the average weight of 100 rats and then add the weight of the world’s heaviest rat, the average will not change by very much (you don’t get rats the size of buses). If you plot the distribution of the weights of rats, it will follow a similar bell curve as that of the historical number of rainy days in November in Manchester, or of the height of Scottish men. Because we know this we can be confident that, with a bit of care, certain statistical models like linear regression and analysis of variance will work reasonably well to predict outcomes.

This is the stage where the tools of statistics are one of the best ways to describe uncertain events. Here statistics rules. The only trouble is recognising when you are in this level and not in Level 4.

Level 4: Partially Reducible Uncertainty

Here Be Dragons. And money, and humanity, and disaster. This is where the bits of existence which are not (yet) neat enough to fit into level 3 sit. Generally, if there is an event with lots of people involved, or a rare event with catastrophic implications, or anything we don’t understand very well, it should be in Level 4. Think: financial markets, earthquakes, technological change, 100 year floods and flashes of genius. Success in this field is as much a function of luck as it is statistical understanding (cf. millions of rich fools in the City of London). Some models will have success here but most will fail.  Here, for now, there is no certainty. But there is money to be made in claiming that we have it.  Mathematically, the probability distributions of events at this level have ‘fat tails’: extreme events are common.

There are some nice solutions here. For example Bayesian statistics – incorporating prior beliefs based on theoretical assumptions – can reduce uncertainty. Noise reduction, dimension reduction: many statistical techniques at this level involve some attempt to reduce the uncertainty to  level 3. Level 4 is where the demand for answers is highest, and so answers are provided.  Unfortunately it is very difficult to ascertain the validity of those answers; they could easily be right for 15 years and then become disastrously wrong, as in the recent financial meltdown.

Level 5: Irreducible Uncertainty

At this level, facts are abstractions. This is the realm of philosophers and religious leaders. Gods and Ghosts. The foundations of level 1, rather confusingly lie in this level, giving a vicious cycle of uncertainty (or perhaps a virtuous circle of certainty, depending on your temperament). For more on this level see Wittgenstein’s On Certainty.  As he said of this level: “Whereof one cannot speak, thereof one must be silent.”

* This is a guest post by my friend Tom Liptrot.
** These ideas have been to some extent lifted from this paper. There the distinction being drawn was between physics and economics, but the parallels hold between maths and statistics.
† At least in the ring of integers