British Academy: The UK's National Academy for the Humanities and Social Sciences
Enquiry, Evidence and Facts: An Interdisciplinary Conference
Some Evidence Issues in Intelligence Analysis
Professor David A. Schum
School of Law, George Mason University,
4400 University Drive, Fairfax, Virginia 22030, U.S.
An abstract presented to the conference
‘Enquiry, Evidence and Facts: An Interdisciplinary conference’
at the British Academy, London, on 14 December 2007
Biography
David Schum joined the faculty of George Mason University in 1985. He holds the rank of professor in the Systems Engineering and Operations Research Department in the Volgenau School of Information Technology and Engineering, and in the George Mason University School of Law. During the years 1966 - 1985 he was a member of the faculty at Rice University, Houston, Texas. There he held the rank of professor in the Departments of Mathematical Sciences and Psychology. During this time he was also a member of the adjunct faculty of Baylor College of Medicine. Professor Schum earned the BA and MA degrees from Southern Methodist University [in 1956 and 1961] and the PhD degree [in 1964] from Ohio State University. Professor Schum has followed a career-long interest in study of the properties, uses, discovery and marshaling of evidence in probabilistic reasoning.
Abstract
I have been very honored these past four years to be a part of the University College London research program on Enquiry, Evidence and Facts that has been supported by the Leverhulme Foundation and the Economic and Social Research Council. Many important and difficult issues arising in a variety of disciplines have been addressed in this research. Some of these issues have also been very timely in light of recent world events. Since September 11, 2001 we have witnessed terrorist actions in New York City, Washington, DC and London, as well as in many other places. Our efforts to predict and prevent further terrorist actions depend in large measure on the questions we ask, the evidence we can gather, and conclusions we draw from this evidence. This is just one of the many contexts in which we are faced with the task of drawing defensible and persuasive conclusions from masses of different kinds of evidence that come to us from many different sources.
There is no shortage of criticism these days of the efforts of our intelligence analysts and those who serve them in various ways. One metaphor we all have heard since September 11, 2001 is that our analysts have often failed to "connect the dots" appropriately. But critics employing this metaphor seem to have few insights themselves about how astonishingly difficult is the task of connecting dots when conclusions are based on masses of evidence of various kinds and come to us from many different sources. On this occasion I will argue that the task of connecting the dots is an epic in complexity by mentioning various attributes of the difficulty of this task. But I will also mention ways in which we can assist persons in any context in which we have the task of drawing defensible conclusions based upon emerging masses of evidence.
I will discuss six major attributes of the task of connecting the dots. The first is that there is more than one kind of dot to be connected. The first kind of dot concerns details in the information we gather and may later argue that these details constitute relevant evidence. Sherlock Holmes referred to these evidential details as "trifles", saying that his inferential methods rested on his observance of trifles. But the second kind of dot refers to sources of doubt we imagine that are interposed between evidential dots [or trifles] and hypotheses we are trying to prove or disprove. Arguments in defense of the relevance of evidential dots consist of chains of reasoning in which every link consists of a source of doubt. These sources of doubt must be arranged in a logically consistent order so that the relevance of an evidential dot can be defended.
The second attribute of the complexity of connecting dots concerns which potential evidential dots should analysts try to connect. Here we immediately encounter a combinatorial explosion since the number of possible combinations of two or more dots increases exponentially with the number of dots we have. For example, if we had just fifty dots there would be over a million billion combinations of two or more of them. Potential evidential dots continually emerge and the problem of deciding which combination of dots to connect becomes exceedingly difficult. Clearly, it would make no sense to try to examine all possible dot combinations, even if it could be done. This would be the act of looking through everything in the hope of finding something of interest. As I will mention later, there are ways of helping analysts decide which dot combinations would be most productive to examine.
The third attribute concerns the question: which of the selected evidential dots can we believe? This is a credibility-related issue and is made difficult by the fact that we must ask different credibility questions depending upon the nature of the evidence being considered. Here I will briefly mention our development of a system called MACE [Method for Assessing the Credibility of Evidence]. This system rests upon a five hundred year-old legacy of experience and scholarship concerning assessing the credibility of witnesses who appear at trial and of various forms of tangible evidence. The fourth attribute concerns the question: whose dots should we consider connecting? Intelligence information is collected by a variety of different organizations. Unless this information is routinely shared, important evidential dots will never be connected. The result is that important evidential synergisms will never be recognized and exploited. As we all know, two or more items of evidence, taken together or connected, can have much greater inferential force that they would have when they are considered separately or independently.
The fifth attribute concerns the construction of complex arguments and involves the question: what specific dot connections should be made? Deciding which dot connections will be most inferentially productive calls for considerable imaginative and critical reasoning. We are again assisted by our colleagues in the field of law. Years ago, in 1913, an American legal scholar named John H. Wigmore developed an analytic and synthetic method for constructing complex arguments based on masses of evidence. Today these complex arguments are called "inference networks". We have been fortunate in our work at UCL to have with us the two leading authorities on Wigmorean inference networks: Professor William Twining [UCL] and Professor Terrence Anderson [University of Miami Florida Law School].
The final attribute of the complexity of connecting the dots involves the basic question: what does some collection of the different kinds of dots in a complex argument mean? Stated another way, what conclusion can be drawn from a mass of evidence that is defensible and persuasive? An important element of this task involves ways for assessing the inferential force or weight of the evidence. There are several possible strategies for doing so, each one providing important insights regarding what we should consider in assessing the force or weight of evidence.
I hope I have been convincing in showing that the task of connecting the dots is not child's play, which is often suggested by critics of our intelligence services. But it is one thing to show how complex something is but quite another to show how we might cope with this complexity. For quite some time now, I have been privileged to work with Professor Peter Tillers [Cardozo School of Law, New York City]. Tillers and I have studied how our strategies for marshaling evidence helps us respond to the six attributes of the complexity of "connecting the dots". Our work rests on another metaphor: the concept of a "marshaling magnet". What we should like to have is a conceptual "magnet" that would attract interesting evidential dots from the mass of dots that is continually emerging and then organize them in productive ways. But investigation or discovery in intelligence, law and other contexts is a dynamic process. Each episode of discovery is unique and we ask different questions and have different objectives at different times. So the result is that we will need different kinds of "magnets", each one representing a different marshaling operation or strategy. What I will show is how the "marshaling magnets" we have designed allow us to at least begin to respond to the obvious complexity of the task of "connecting the dots", whether in intelligence analysis, law or any other activity where such complexity occurs.