In a post a few weeks ago, I pointed out that Mississippi dentist Dr. Michael West, a bite mark analyst who often gives expert testimony in criminal trials, isn’t much of a scientist. I based my conclusion on Radley Balko’s Reason article, which included this description of West’s methodology:
But even in an already imprecise field, Dr. Michael West has taken forensic odontology to bizarre, megalomaniacal depths. West claims to have invented a system he modestly calls “The West Phenomenon.” In it, he dons a pair of yellow goggles and with the aid of a blue laser, he says he can identify bite marks, scratches, and other marks on a corpse that no one else can see—not even other forensics experts.
Conveniently, he claims his unique method can’t be photographed or reproduced, which he says makes his opinions unimpeachable by other experts.
That’s nonsense. If it’s not repeatable, it’s not science. Dr. West might as well be claiming divine revelation because, after all, no one can reproduce that either.
I’ve been thinking about this a lot, trying to explain why West’s assertion that he alone can do these tests is so wrong-headed. I can’t quite explain it in a way that completely satisfies me, but I think it’s fair to say that his method has problems with validity and robustness.
Forensic analysis starts with simple observable (or measurable) facts. Then you apply some general rules about the world to to derive new facts. When police find fingerprints on a murder weapon that are a match to a suspect’s fingerprints, they can use the general rule that no two people have the same fingerprints to derive a new fact: The suspect touched the murder weapon.
It should be obvious that in order for the analysis to be valid, the underlying general rule must be valid: If fingerprints were not unique, then a print match would be much less useful.
What’s less obvious is that the methodology for determining the observable facts must also be valid. If you don’t have a good way to determine that two fingerprints match, then you can’t be sure you’ve proven anything new. In other words, it’s not enough that the idea behind the analysis is sound. The the procedures for gathering and analyzing the data must also be valid and repeatable.
Consider the science of forensic bloodstain pattern analysis. Blood from a wound will be spattered around a crime scene in different ways depending on how the wound is inflicted and the victim’s post-wounding position and movement. The effect is a routine application of the physical laws governing the behavior of fluids and has been proven experimentally.
When a forensic scientist analyzes the scene of a bloody crime, however, he’s actually working backwards: Instead of figuring out how the blood will spatter for a particular crime, he’s looking at the resulting spatter and trying to figure out what caused it.
That seems like it ought to work, but we can also imagine reasons why it might not: Spattered blood may be difficult to classify properly, it may be hard to define the edges of a smeared droplet, different scenarios may produce identical droplets, and so on. Even if we know all about how a violent act produces blood spatter, whether or not a forensic analyst can work backwards to understand a violent act is a separate question that is itself subject to testing.
Complicating matters further, there’s also always a question how well the analysis works on the margins and where to set the limits. How large does a blood spatter pattern have to be for a forensic analyst to be able to reliably differentiate between high-velocity and medium velocity impact spatter? A square foot? A square inch? How much of a fingerprint is needed to get a reliable match? 14 points on a thumb? 4 points on a partial index finger?
As far as I know, these and other questions of validity have been investigated and answered quite thoroughly for fingerprint analysis, and a little less so for bloodstain pattern analysis. Since the “West Phenomenon” can’t be performed by anyone else, I doubt it’s ever been subject to a real scientific investigation of its validity.
Scientific tests can also be analyzed in terms of their robustness. A robust test isn’t very sensitive to changes in the testing conditions. For example, a robust chemical test will be accurate across a wide range of room temperatures, and will yield the same results regardless of how fast the technician stirs the beaker. An even more robust test will work even if the technician doesn’t use enough of one reagent, and it may be resistant to the most likely contaminants. A robust test methodology produces accurate results under a wide variety of conditions.
Since the conditions of the test are largely under the control of the person performing the test, this way of thinking about robustness leads us to an interesting and counter-intuitive conclusion: The most reliable scientific tests are those that can be performed by the stupidest people.
For example, I’ve had no medical training whatsoever, but with simple medical equipment from Walgreen’s I can measure my father’s blood pressure and blood sugar levels well enough for his doctor to use those numbers to assess his health. These are very robust tests because lots of people can learn to do them.
Not robust enough for you? How about measuring body temperature? Body weight? The tests for those are so robust that literally billions of people know how to do them. I’m no lawyer, but I doubt anyone has ever been prevented from testifying about body weight because of lack of expertise.
For an example of the middle ground in robustness, think about radar speed detection. Have you ever seen one of those electronic “Your Speed” signs along the road that supposedly shows you if you’re speeding? The numbers jump all over the place, and you can’t tell if it’s showing your speed or the speed of another car near you. That’s because it’s just a radar gun bolted in place and pointed down the road. With no one to aim it and no way to tell where it’s pointing, it’s probably the ultimate in unskilled operation, and it fails miserably. Clearly, radar speed detection requires a trained operator to get accurate results.
Other forensic tests, such as fingerprinting and bloodstain pattern analysis are even more complicated, requiring lots of training and practice. The most complicated forensic procedure of all is probably a forensic autopsy, which is why the people who do them have to be trained for years. Yet all of these forensic procedures can be performed by hundreds if not thousands of people in this country alone.
When Dr. West claims he’s the only one who can perform his method of testing, he’s saying that he’s the only person with the talent to do the test. But that’s just another way of saying that his method is totally lacking in robustness: Anyone else who tries his method will introduce enough variation into the process to make it fail.
I can’t help but wonder how often Dr. West himself introduces enough variation to cause the test to fail. Maybe Dr. West’s estimate of the number of people who can use his methods is too high by one…
I should add that just because a test procedure is difficult and not robust doesn’t mean it’s useless. It’s still possible to get great results by going to the expense and effort of controlling the test conditions very carefully or repeating the test enough times to build up confidence in the results. However, any such activity would surely either require or result in the ability of other people to reproduce the “West Phenomenon”
Finally, while I have some training in science and statistics, I have no legal training, so while I’m somewhat confident that I’ve accurately described the scientific approach to thinking about these things, I have no idea if a court would care about any of this.
Leave a Reply