David Papineau, Professor of philosophy of science at King’s College London, writes in Aeon:
Imagine that 100 prisoners are exercising in the prison yard, and suddenly 99 of them attack the guard, carrying out a plan that the 100th prisoner is no part of. Now one of these prisoners is in the dock. No further evidence is available. Guilt is 99 per cent likely, innocence 1 per cent. Should the court convict?… The court has no information that rules out the defendant being the one innocent prisoner. You can’t convict someone solely on statistical evidence.
Papineau points out that “[i]n both civil and criminal trials, defendants can be found responsible only if the evidence relates specifically to them.” Eyewitness testimony does relate specifically to the defendant but tends to be unreliable in certain situations. When judged acceptable, it is commonly seen as 95% reliable, in contrast to the 99% likelihood described above:
So we are often ready to convict on eyewitness testimony, but never on purely statistical evidence. You might well wonder why, if 95-per-cent reliable eyewitnesses are more likely to lead us astray than 99-per-cent reliable statistics.
I’m not even trying to answer this. I’d rather look into a class of evidence that is both statistical in essence and defendant-specific. DNA evidence appears to provide an exception to Papineau’s maxims:
Across the board, we trust presumed sources of direct knowledge more than indirect reasoning…
We keep favouring weaker direct evidence over good statistics…
To quote an early forensic DNA primer (published by the National Academy of Sciences in 1992):
Because any two human genomes differ at about 3 million sites, no two persons (barring identical twins) have the same DNA sequence…
However, the DNA typing systems used today examine only a few sites of variation and have only limited resolution for measuring the variability at each site. There is a chance that two persons might have DNA patterns (i.e., genetic types) that match at the small number of sites examined.
Today’s DNA typing methods allow for a greater number of sites (“loci”) to be examined – 10-20 vs. 3-5 back then – but it’s nowhere near the complete set. This means that a DNA match narrows the set of potential actors, but seldom if ever to just one person. When an analyst finds that the defendant’s reference sample matches the crime-scene DNA, she must estimate the probability of the latter belonging to the defendant rather than coming from someone else with similar alleles.
DNA “exclusions” are easy to interpret… a nonmatch is definitive proof that two samples had different origins. But DNA “inclusions” cannot be interpreted without knowledge of how often a match might be expected to occur in the general population.
This means using statistical methods to estimate the required probabilities. In theory, a state hellbent on surveillance could collect DNA samples from all residents and visitors. In practice, forensic scientists have to rely on limited samples to estimate the true frequency of DNA patterns in the general population.
Some methods based on simple counting produce modest frequencies, whereas some methods based on assumptions about population structure can produce extreme frequencies. The difference can be striking: In one Manhattan murder investigation, the reported frequency estimates ranged from 1 in 500 to 1 in 739 billion, depending on how the statistical calculations were performed.
Things get much more complicated nowadays as data from mixed-DNA samples gets analyzed using likelihood-ratio testing. These results can provide precious leads while the case is being investigated but juries are almost guaranteed to get them wrong.
But let’s ignore these recent developments for the time being, and limit ourselves to the old-fashioned DNA identification as practiced in the 1990s. As I’ve said, it’s based on statistical inference and yet it’s widely used in criminal and civil trials alike all over the world, above all in the US and the UK. Perhaps courts and juries don’t quite appreciate the method’s statistical foundations.