Evening the odds

Note: in reading the following it is important to bear in mind that we are discussing a statistical issue arising from press coverage of the Sally Clarke/Roy Meadows case. We do not enter into the medical or legal issues of this particular case, and you should note that Ms Clarke was not convicted (or subsequently acquitted) solely on Prof. Meadows testimony.

There was a lot of media attention here in the UK in the days after Professor Roy Meadows was struck off the medical register by the GMC. For those who may not have been following the story, Prof. Meadows was called as an expert witness for the prosecution in the case of Sally Clarke, a solicitor who was accused of murdering her two baby sons. His evidence played a key role in her conviction, which was overturned on appeal – after she had served some time in prison. Unfortunately, his evidence was based on faulty statistical reasoning.

He said:

“Even when an infant dies suddenly and unexpectedly in early life and no cause is found at autopsy, and the reason for death is thought to be an unidentified natural cause (SIDS) it is extremely rare for that to happen again within a family. For example such a happening may occur 1:1,000 infants, therefore the chance of it happening twice within a family is 1:1,000,000.”

Teacher: What is wrong with that?

Student: Well it looks alright to me. We are supposed to multiply probabilities together like that, and 1,000 x 1,000 is 1,000,000.

Teacher: You are right – up to a point. But you can only square the original probability if the chance of the second event is the same as the probability of the first.

Student: I suppose that would change things. But how could the probability be different if the event is the same?

Teacher: Sometimes the first event itself changes things – for example, if you cut a pack of cards, the probability of getting an ace is 4:52. If you put the card back and do it again, it is still 4:52. You can multiply these together, giving a probability of 1:169 for getting two consecutive aces. But if you set the first card aside – and it is an ace – the probability the second time is 3:51. And also, if the first card wasn’t an ace, the chance is 4:51. Both are different; the first event has affected the second event.

Student: OK, I can see how something like that could change things but this situation is quite different.

Teacher: Yes, it is. SIDS by definition is more difficult to quantify, since it is unknown what causes it: “the reason for death is thought to be an unidentified natural cause”. Here’s another example: an entirely fictional police spokesman is talking about the frequency of traffic accidents. Apparently there are 103 a year in the High Street area. According to prosecution figures the drivers often lost control because they were drunk or speeding.

Student: I don’t see what this has to do with anything.

Teacher: Bear with me for a little. There is a journalist in the audience from the local paper. He asks about a day the previous February when there were five accidents in three hours. The spokesman replies “Sometimes there are odd clusters of all kinds of events. We have worked out that the probability of an accident is 1 in 1,000 journeys, but it’s just due to chance when they happen. These figures are very authoritative because we have been collecting them over several years. To combat this we have instigated road safety and drink-driving awareness campaigns.”

Rising to his feet, and aware that his moment has come, the journalist asks whether the spokesman can suggest any cause of the accidents other than bad driving. He replies “Yes, of course. Our job is law enforcement, so we separate the figures into dangerous driving where the law is broken – and the rest, which are insurance matters and could be caused by any number of things.”

The journalist says “So presumably the chance of five accidents in such a short period is 1-in-1000 x 1-in-1000 x 1-in-1000 x 1-in-1000 x 1-in-1000?”

The spokesman hesitates while he does the maths, and then agrees that you multiply the probabilities together, so that is right. The journalist asks “Are you really saying that the chance of those five accidents happening in the same time period is 1 in a million billion?” “I suppose I am.”

“Well, I was involved in two of those five accidents – and I can tell you that it was -6C and icy that day, and the road was treacherous. The first time, I was breathalysed at the roadside as a precaution, and after the car was checked I was free to go. I had not choice but to go on, as my journey was extremely important. However, the second time – presumably because of the first incident – I spent four hours being interviewed at the police station. I was asked if I was on medication, or a drug user, how much sleep I had had in the previous 24 hours, I was breathalysed again and blood was taken, and I was tested on the highway code – when given the road conditions it was quite unnecessary to presume I had broken the law! As I understand it, the chance of my having two accidents like this according to your statistics is 1:1,000 x 1:1,000 = 1 in a million. If something appears to happen which only has a chance of 1 in a million, either it is almost a miracle or the cause lies elsewhere. Isn’t it a lot more likely that at certain times it isn’t a driving offence, it is something else which is making this happen; in this case appalling road conditions?” The spokesman is forced to agree.

Once there have been two or more accidents, you have more evidence to find out the cause. When our police spokesman goes back and incorporates data from the weather bureau into his figures, he finds that the chance of accidents on icy days is much higher than on non-icy days. The first accident may not mean anything – it may be just a tiny bit of ice which proves too much for an inexperienced driver; and it might just melt away. But once subsequent accidents occur it begins to suggest that the ice has built up rather than melting and there is likely to be a spate of other accidents until there’s a thaw. In fact accidents caused by sober drivers losing control on ice are much more likely than those caused by drunk drivers in normal weather.

Student: So you are saying that there is something else going on – that once two accidents have happened, you should look to see if there is a reason for it other than your default assumption?

Teacher: Yes, because once there have been two or more accidents so close together it is probably not just bad driving or drunkenness, but something else entirely – maybe road conditions.

Student: So presumably this means that given that the weather conditions are icy, the chance of this series of accidents is actually not that rare?

Teacher: That’s right. Once you have evidence that the usual rate of accidents has changed, it is more likely than not that the original assumptions on which your probability is based have also changed. You are working in a different set of circumstances – in this case bad weather instead of fair weather.

Student: But we were talking about Prof Meadows… ?

Teacher: The point about SIDS is that no-one really knows what causes it. There is no direct equivalent of drunk-drivers or icy roads. However, there has been some research (of which Prof Meadows was aware) which shows that in fact SIDS is more likely to occur a second time once there has been one death of this kind in a family. “…the study provided evidence that the chance of a second SIDS death in one family is not independent of the occurrence of the first and that there is an elevated risk of a second SIDS death in one family after there has been one such death.”

Student: Are you saying there is an underlying cause we can look for after there have been two SIDS deaths in a family?

Teacher: It may be more than one thing, but these deaths do happen as a result of something. Isn’t it possible that SIDS are caused by some genetic problem which hasn’t yet been pinpointed? If you have two in a family, you have a good place to start looking – and after that you can see if anything you find also applied in the other single-death cases.

Student: So if the parents were carrying the faulty genes, it would stand to reason that their children would be more at risk than anyone else’s children.

Teacher: That’s right. A genetic cause hasn’t been found yet, but if it does exist then a family which has already exhibited signs of the faulty gene is more likely to suffer again than another family which hasn’t.

Student: And I suppose if it isn’t genetic, it could still be something else which would have the same effect – like, I don’t know, sleeping position, family habits, allergies, ventilation…

Teacher: History is full of unidentified illnesses; once the cause has been pinned down we can look back and explain previously inexplicable events. SIDS is currently one of those illnesses. Clusters, as happened in the 19th century cholera outbreaks in London, tell you where to look for evidence and data to identify what the causes are. It took years for the authorities to accept that it was a waterborne infection.

Prof Meadows’ first mistake was in quoting an overall probability averaged over the whole population rather than one that was more refined according to particular conditions. Once the second death had occurred, it should have been a signal to look back and say that the probability originally assigned to the first death was no longer valid and should be much smaller. His second mistake was not to recognise that in these changed circumstances, we no longer know that the events were independent. For instance, if one child has large feet it is more likely that another sibling will have large feet. If one child has SIDS it is more likely that another sibling will also have SIDS.

His third mistake was a more general one – that given that two SIDS deaths are unlikely, he jumped to the conclusion that murder must be the cause rather than any number of other explanations: such as that the pathologists could have made a mistake, or that there is a legitimate cause of death not yet identified, such as a genetic problem or a disease. This is his most fundamental mistake. It is almost as if you were told that a particular card in a pack is not the ten of clubs, and you then say “Ah, it must be the ace of diamonds.” and everyone believes you.

Links

More information on John Snow and cholera – a very full and informative site.

If you would like to follow up the issues relating to the use of statistics in court, the Royal Statistical Society has a page on Statistics and the Law which includes a link to a discussion of the Sally Clarke case.

Comments

Please contact us with any comments you may have on this debate.

“I use the Meadows case also in teaching. Briefly:

I first point out that apparently very unlikely events occur quite often. E.g., something as rare as 1 in 14 million occurs nearly every week (someone wins the lottery). Even though this is a rare event, we don’t accuse the winner of cheating.

If perinatal mortality is about 1 in 1000, then suppose a million babies are born every year (this is an overestimate, but would apply if UK + France was used). 1000 of them will result in perinatal death.

Those 1000 women will, quite likely, go on to try for another baby. Even with unrelated events, we would expect one of these to lose their second baby. Just by bad luck.

What happens, though is that on losing their second child, huge attention is focussed on the unfortunate parents post-hoc. The prior probabilities shouldn’t be used raw as Meadows did. So his fourth mistake is not allowing for the post-hoc selection practiced by the medical profession/social services by seeking out parents with more than one perinatal mortality. The raw odds, are, essentially, largely meaningless in this case as evidence for or against guilt of murder.”

Martin Le Voi
Open University
UK