9 - Major Sources of Uncertainty in Masses of Evidence

Summary

There are five major reasons why conclusions reached in intelligence analysis based on evidence will be necessarily probabilistic in nature: Our evidence is always incomplete, usually inconclusive, frequently ambiguous, commonly dissonant, and with various degrees of believability. Any one of these reasons can lead to uncertain conclusions, but an analyst drawing conclusions based on masses of different forms and combinations of evidence will likely encounter all five of these reasons at the same time. Intelligence analysts who report conclusions with varying degrees of uncertainty are often unjustly criticized for doing so. Persons providing such criticisms are doubtless unaware of many or most of the reasons why it is necessary to hedge conclusions in probabilistic terms. One main reason we have for providing a careful account of these five reasons is that no single view of probability we know about captures all five of these sources of uncertainty. Each view of probability we will mention provides useful insights about some of these sources of uncertainty, but no single view says all there is to be said.

INCOMPLETENESS

What Is Incompleteness of Evidence?

We may have all heard someone say, “I am going to wait until I have all the evidence before I draw a conclusion or make a decision.” This person faces an infinitely long wait because there is no situation in which we can say we have all the possible evidence. The first way of showing that this statement is true is to consider the distinction we have made between directly relevant and ancillary evidence. In doing so, we also referred to ancillary evidence as being meta-evidence, or evidence about evidence. The trouble here is that we face an infinite regression in which we have evidence, ancillary evidence about this evidence, ancillary evidence about this ancillary evidence, and so on, ad infinitum. Suppose we have an HUMINT asset who provides us with some interesting evidence; call this asset our primary source. But then we have a secondary source who provides us with ancillary evidence about the believability of our primary source. But then we have a tertiary source who provides ancillary evidence about the believability of our secondary source.