British Academy: The UK's National Academy for the Humanities and Social Sciences
Enquiry, Evidence and Facts: An Interdisciplinary Conference
Are Expert Opinions Telling Us Both Sides of the Story?
Dr Gianluca Baio
University College London, Department of Statistical Science, Gower Street, London WC1E 6BT
Dr Amanda B. Hepler
University College London, Department of Statistical Science, Gower Street, London WC1E 6BT
Dr Grant Fisher
University College London, Department of Science & Technology Studies, Gower Street
An abstract presented to the conference
‘Enquiry, Evidence and Facts: An Interdisciplinary conference’
at the British Academy, London, on 13 December 2007
Biographies
Dr Gianluca Baio
Gianluca Baio graduated with a degree in Statistics and Economics from the University of Florence (Italy), after spending a period studying at the Sheffield Hallam University, Sheffield (UK). Subsequently, he has been a Research Fellow in the University of Siena (Italy), where he started to work on applied Bayesian statistics with particular reference to the economic evaluation of health systems. He completed his PhD in Applied Statistics at the University of Florence, after spending a period at the Program on the Pharmaceutical Industry at the MIT Sloan School of Management, Cambridge, Massachusetts, and currently is a Research Fellow at University College London (UK). His main interests are in Bayesian statistical modelling, probabilistic expert systems, and applications to decision-making problems, particularly in health care.
Dr Grant Fisher
Grant Fisher obtained a PhD in Philosophy from the University of Leeds in 2004. He has taught philosophy of science at the University of Leeds and history of science at Durham University. He is currently a Research Fellow in the Department of Science & Technology Studies at University College London. He works on the Evidence in the Natural Sciences project within the Evidence, Inference & Enquiry programme. His interests are in philosophy of science, especially confirmation, theories and models, scientific representation and history and philosophy of chemistry.
Dr Amanda Hepler
Amanda Hepler is currently working as a Research Fellow for the Evidence Programme, and is based at University College London. After obtaining her B.S. from Towson University, she earned her Masters degree and doctorate while at North Carolina State University. Trained as a forensic statistician, she specializes in the application of Bayesian network technology to aid evidence analysis. Her dissertation research explored modeling the complexities involved in DNA evidence interpretation when individuals arise from inbred populations. In addition to continuing this work, Dr. Hepler is also investigating formal models for other types of forensic evidence, including alibis, expert witness testimony and eyewitness credibility.
Abstract
1. Introduction
The role of expert opinion as evidence to be used for statistical inference is increasingly important, for example in the field of forensic science. Often, decisions are taken on the basis of the assessment made by experts that report on the plausibility of a hypothesis of interest (H0). For example, in the legal framework, this might involve circumstances where two samples are compared in order to discern whether their alleged similarity is “consistent with” a causal relation between the crime and the suspect. This decision problem can be rationally solved in the Bayesian framework. Accordingly, based on his or her experience, the expert needs to assess the true unknown probability distributions of the observed variables under two (or more) possible explanations. However, it often happens that expert opinions only provide partial information. The actual process by which an expert declares a “match” between the crime and the suspect sample (i.e. favours H0) might not take into account properly how more likely this hypothesis is in comparison to the alternative one(s), explicitly violating Bayesian decision making axioms. We explore this issue in section 2 with particular reference to its philosophical significance in relation to the nature of evidence broadly and in section 3 presenting a hypothetical example of the process underlying the formation of the expert opinion.
2. Philosophical aspects
Explanation has a function in statistical inference, and philosophical accounts of explanation might provide fruitful perspectives on the problem of the likelihood of possible candidate hypotheses. However, introducing philosophical accounts of explanation to the Bayesian framework is encumbered by two notable philosophical problems. First, according to an influential traditional account from Hempel, explanation consists purely in the logical relationships between statements. But this account seems ill-equipped to accommodate the Bayesian framework, wherein non-logical relations between propositions are admitted and what constitutes a good explanation may vary according to audience (other experts, juries of lay-people, etc.). Second, that Bayesian inference and inference to the best explanation (according to which one infers some hypothesis which, if true, would constitute the best explanation of the evidence) are typically construed as incompatible. Our aim is to explore the significance of these problems by focusing on the ideas that, within the Bayesian decision-making framework, a) explanation is contextual, that is, explanation does not consist purely in the logical relationships between statements; and b) that while Bayesian inference and inference to the best explanation are compatible, what seems to be absent is a normative function for explanation in legal cases that might direct expert witnesses to consider the likelihood of alternative hypotheses.
Within Hempel’s deductive-nomological account of explanation, an event is deductively subsumed under appropriate laws and initial conditions such that the event to be explained is to be expected in a deterministic sense. Hempel also offered an account of explanation of events by statistical generalizations, the inductive-statistical view of explanation, according to which the event to be explained is expected in a stochastic sense. These accounts have suffered from many objections. Here we will note that they do not provide any normative decision-making framework to deal with competing explanations, nor indeed a decision strategy (such as the maximization of expected utility), when confronted with alternative explanations. One only needs to offer explanations that fulfil Hempel’s ‘condition of adequacy’ of explanation and such adequacy need only be a matter of the logical properties of an explanation. However, what constitutes an effective explanation within a legal framework is not a matter of logical subsumption of explanadum by explanans; it depends on pragmatic and contextual factors related to who requests an explanation, who provides it and for what reasons. And relations lying outside of the logical relationships between propositions (i.e. probabilistic associations) are required in order to gain a better understanding of the matter, and the Bayesian decision-making framework offers that facility in legal (and other) situations.
Our second issue concerns a normative function for explanation in legal frameworks. It is a widely held view that Bayesian inference and inference to the best explanation are incompatible. In explanatory frameworks, we might all too easily be led to fallacious judgements by violating the axioms of probability. More positively, explanation and Bayesian inference may coincide such that the ‘best’ explanation of some item of evidence is the most probable explanation. Explanation would therefore be epistemically posterior to Bayesian conditionalization on the observed evidence. That there is indeed a compatible relationship between explanation and Bayesianism has been defended by Lipton (2004). However, he argues that explanatory considerations are not epistemically posterior to Bayesian inference, that explanatory considerations are integral to Bayesian conditionalization itself and required in order to provide a descriptive account of our inductive practices: how we actually go about reasoning within a Bayesian framework. We broadly support this idea. However, Lipton does not address a normative function for scientific explanation in Bayesian inference. We argue that in legal frameworks, a normative function for explanation is required in order to justify maximising utility by considering all candidate explanations.
3. Expert opinions in court
Consider a crime case where some evidence from the crime scene is available (the crime sample); a suspect is apprehended and some evidence is measured from this individual (the suspect sample). The working hypothesis H is whether the suspect is actually guilty of the crime or not and, in order to help the jurors make a decision, an expert is called to give his or her opinion on the nature of the two pieces of evidence being compared. By definition, the expert has knowledge of the following two processes. The first one generates related instances of observations of the same nature as that of the crime and the suspect samples. The second one is the process describing some specific knowledge of variables of the same nature as the crime and the suspect sample, independently. As an example, suppose that the crime sample is represented by a bite mark on the victim and that the suspect sample is represented by the teeth configuration of the suspect. Knowledge of the first process should come from the fact that the expert has worked in previous criminal cases, where he or she had the opportunity to evaluate similar traces allegedly related to each other. Conversely, knowledge of the second process is presumably derived from the general expertise of dental characteristics that the expert has, being a dentist.
The relationships used by the expert to link the various items of evidence and the working hypothesis can very rarely be purely logical, since there generally is imperfect knowledge of the causal mechanism underlying the problem (one notable exception is represented by DNA evidence, although even in this case it might be necessary to include stochastic relationships, for instance to account for the possibility of mutations). Moreover, the expert has only access to a sample of realisations of the relevant processes (his or her past experience) and therefore only a limited, yet superior to the single jurors, understanding of them. This circumstance justifies the more general modelling procedure, which under the Bayesian framework is based on probabilistic relationships, as opposed to strictly logical relationships, to describe the level of uncertainty about the occurrence of the events of interest. In addition, the explanation is explicitly a function of contextual factors, such as the level of expertise of the person giving the testimony and the background of the recipients of the explanations (the jury). A second, related problem is that in this case the expert is called to produce a deductive inference, i.e. the objective of his or her testimony is to measure the probability of the (unobservable) hypothesis of guilt, given the information provided by the observed data. Ideally, this estimation will be in the form of a Bayesian posterior distribution, given the previous cases that the expert has had access to. Overcoming one of the limitations of Hempel’s account of explanation, the Bayesian machinery allows the expert to actually perform such a deductive inferential process based on more general non deterministic assumptions.
Finally, combined with the estimation of the underlying processes, the observation of the current samples will determine the expert’s testimony as to whether the crime and the suspect samples match. In particular, the testimony will be implicitly determined by a subjective criterion of the expert, and we regard this feature of Bayesian reasoning as highly relevant to our claim that context has a particularly relevant role in explanation. Ideally, from the precepts of Bayesian decision theory, a rational individual should take into account both the (posterior) evaluations of the underlying processes, compute a likelihood ratio and compare this value to a suitable threshold. This represents an extra feature, with respect to Lipton’s model, in that it provides a normative guideline, which is particularly relevant in applied cases, such as the legal one. Moreover, this final step provides a more general “strategy for explanation”, because it forces the expert to take into account (or even to look for) other competing hypothesis, and their likelihood, in a comparative way, in order to obtain a rational evaluation of the evidential reasoning.