British Academy: The UK's National Academy for the Humanities and Social Sciences
Enquiry, Evidence and Facts: An Interdisciplinary Conference
Formal Modes of Argument
John Fox
University of Oxford, Department of Engineering Science, University of Oxford, Parks Road, Oxford, OX1 3PJ, and
Department of Oncology, Royal Free Hospital, UCL School of Medicine
An abstract presented to the conference
‘Enquiry, Evidence and Facts: An Interdisciplinary conference’
at the British Academy, London, on 13 December 2007
Biography
John Fox was educated at Durham and Cambridge Universities, with post-doctoral stints at Carnegie-Mellon and Cornell Universities in the USA. In 1975 he joined the MRC to work on clinical decision making and computer-based decision support systems. In 1981 he move to the Imperial Cancer Research Fund in London, (now Cancer Research UK) and joined the Department of Engineering Science at Oxford University in 2007.
Prof Fox’s research straddles computer science, artificial intelligence and medical software engineering. He has published widely in computing, biomedical engineering and cognitive science, and was founding editor of the Knowledge Engineering Review. Recent publications include "Safe and Sound: Artificial Intelligence in Hazardous Applications" (AAAI and MIT Press, July 2000). He has led a variety of informatics development projects funded by UK research councils and European scientific programmes.
Abstract
Evidence is an important human preoccupation. It is important, for example, when we want to decide whether some hypothesis or claim is true; when we are considering how best to act in circumstances where there is uncertainty about the outcomes of our actions, and when one individual’s opinion might be shown to be superior to that of someone else. Given that a sound rationale for our beliefs and actions is fundamental to success - as individuals, societies and even as a species - it is surprising that we do not seem to be very good at evidence-based thinking. Psychological studies of human reasoning and decision-making suggest that our inferences are subject to fallacies and biases, and our decisions are rough and ready rather than rigorous and precise. The movement for “evidence-based medicine” and increasing demands for scientific evidence in formulating social policy suggest that the traditional authority for human action, expert opinion, is not trusted as much as it was. Indeed this conference is a manifestation of this change.
Recognition of shortcomings in human judgement has also led to a growing interest in formalising mathematical and rational foundations for evaluating claims and the evidence for those claims. In particular we have seen an increasing concern with quantifying evidence and, in recent years, a great deal of work on computer techniques for assessing evidence in fields like medicine, science and even public policy and the law. This interest seems closely related to the desire for a “science of evidence” in the UCL project.
Mathematical views of evidence are often assumed to begin with the concept of probability. As P Achinstein puts it “It is argued that evidence must supply a good reason for belief, and that the latter requires that the objective epistemic probability of the hypothesis on the evidence be greater than half.” Unfortunately, however, our ability to estimate “objective epistemic probabilities” depends upon being able to observe samples of previous cases. This is frequently not practical. For example, complex legal and medical cases may involve rare combinations of circumstances and science is often concerned with trying to explain new phenomena or predict events that have never been observed at all.
A modern interpretation of probability, sometimes called Bayesianism, tries to sidestep this difficulty by introducing the concept of “subjective probability”. A subjective probability can be obtained, say, from an expert’s estimation or, more indirectly, through someone’s willingness to engage in a bet. However the notion of subjective probability is subject to a number of scientific, philosophical and mathematical objections. We still need a practical approach to evidential reasoning which is not exposed to these objections and is not dependent on the availability of objective estimates of probabilities.
While recognising the value of quantitative evidence many researchers believe that quantification is sometimes neither necessary nor sufficient for interpreting evidence. As J Franklin says “Even now, the degree to which evidence supports hypotheses in law or science is not usually quantified, and it is debatable whether it is quantifiable even in principle. Attempts to give numbers to grades [such as "almost certain", "more likely than not" and so on] are not necessarily to be praised. One should not give in to the easy assumption that numbers are good, talk bad.”
Logicians, computer scientists and others have made many attempts to understand formally some of the things that Franklin calls “talk”. Their goals include formalising ideas that we express naturally, but somewhat vaguely in English and other human languages. They want to capture notions like “concepts”, “sentences”, “reasoning” and so on. Computer science offers many languages that have a more precise meaning for specific purposes than natural language offers. Furthermore some of these languages can be used to program computers to do the sorts of things that people do when they are assessing evidence; potentially automating human forms of reasoning while avoiding human forms of error.
For example logic programming is based on a computationally practical variant of classical (first-order) logic. First-order logic is a surprisingly expressive language for formalising concepts, rules of inference and similar ideas, and logic programming is an extremely powerful and versatile way of automating many different kinds of reasoning. However classical logic is built around the notion of proof (if P is true and P implies Q then Q is true) not evidence. It is debatable whether the standard concept of proof has much resemblance to human reasoning. It does not allow for uncertainty, for example, which is key here; indeed we seek evidence precisely when proof is not possible.*
Newer non-monotonic logics capture an important aspect of human belief which on the face of it classical logic doesn’t, viz our beliefs may be tentative. Non-monotonic logics allow for the possibility of reversing a conclusion, as when we first conclude that P is true but in the light of more information conclude that P is false. Despite greater flexibility than standard logic, however, these methods remain unsatisfactory for practical applications because confidence in the conclusion is still all-or-none. In assessing evidence for competing conclusions in science, medicine, legal proceedings and much of everyday life, we frequently need to be able to allow for different levels of confidence in some way if we are to make the best decisions about what to believe or what actions to take.
Recent developments in cognitive science and computer science have led to a new approach to the problem of uncertain reasoning, stimulated by observation of natural patterns of human reasoning, in medical and legal settings. They formalise the everyday concept of argumentation that we see in human decision-making and debate. The basic idea was first proposed in a clear form by the philosopher Stephen Toulmin. He emphasised the importance of making background knowledge and justification explicit in reasoning about hypotheses or “claims”. He also points out the ubiquitous use of qualifications in our claims (such as “it is possible that…”; “it is probable/improbable that…” and so on). He also made the idea of rebuttal explicit in his discussion of argumentation, a fore-runner of the technique of reasoning explicitly about contradiction which is exploited in non-monotonic reasoning.
Toulmin’s views have been increasingly influential since he published his book in 1958, but his framework is largely informal rather than mathematical or computational so the practical value of the approach has been limited. However, formal models for argumentation are now developing rapidly, with at least two general approaches emerging so far.
Our approach, which I will call “evidential” argumentation, is inspired by decision-making rather than logical reasoning. In order to make decisions between competing hypotheses or actions we must assess all the relevant and distinct lines of argument for each competing option, the pros and cons as it were. This entails making the background knowledge and rationale for every argument explicit as well as the claim (the option). Each distinct supporting argument increases our confidence in a hypothesis, for example, while each distinct opposing argument increases doubt, reducing overall confidence. Lastly, we can express grades of confidence by interpreting qualifiers in terms of argument patterns. For example we may say that “P is possible if there is at least one argument in support of P and no argument that categorically excludes P”, or “P is probable if there are more arguments in support of P than against P” and so on. This is a simple interpretation of qualifiers, but there are many other approaches to grading confidence, including some based on natural language, some on standardised logical modalities, and of course quantitative grading schemes such as probability.
On this approach even if we do not know the quantitative strength of each argument we can still compare the overall persuasiveness of competing claims. For this we require some kind of aggregation function which maps from the complete set of pros and cons into a summary assessment of confidence. Again, these summaries can be represented linguistically, logically or quantitatively requiring corresponding qualitative or quantitative aggregation functions. In the quantitative case we might use a probabilistic function, and arguably we should. However, quantification of the strength of arguments and/or overall grades of confidence is not a prerequisite to making assessments of plausibility based on evidence. Qualitative and semi-quantitative functions have been found to be surprisingly effective for assessing evidence and making decisions in many practical situations.
A different approach to formalising argumentation that is currently receiving a great deal of attention is “dialectical” argumentation. As in the evidential mode the logical rationale for a claim is made explicit but the dialectical approach emphasises conditions under which an argument is acceptable. The central idea here draws on another feature of human deliberation in which we “attack” and hope to “defeat” arguments in debates and disputes. One way of defining this is to say that an argument is acceptable so long as it has not been defeated by another argument (e.g. by proving that the assumptions on which an argument rests are false, or arguing that the rules which are used in its construction are inappropriate, irrelevant or unreliable). If an argument is defeated but the attacking argument can itself be attacked and defeated, the original argument will be reinstated. This is (arguably) the mode of argumentation that seems most appropriate when trying to resolve non-evidential issues, such as whether or not probability is the best way to quantify grades of uncertainty!
Recent approaches to dialectical theory have also tried to introduce an evidential mode. This is like aggregation in that dialectical arguments are said to accrue (all other things being equal two independent arguments are more convincing than one). However the relationships between accrual and standard quantitative evidence measures such as probability and other aggregation functions have not been established.
Both these modes of argument have a clear rationale and well understood mathematical foundations. Like probability methods both can be implemented in practical computer software that will automatically construct, evaluate and compare the arguments for claims (e.g. see www.argumentation.org).
Rather than seeing the dialectical and evidential approaches as alternatives it seems probable (sic) that they can be combined. For example we may construct individual arguments about a claim using the dialectical mode and then aggregate (accrue) all the acceptable arguments in an evidential mode, in order to grade our overall confidence in the claim. If we can also quantify the strengths of all the arguments, based on objective probabilities, for example, we may be able combine the naturalness and expressiveness of logical languages with the power and precision of quantitative methods. But even if we cannot give numerical strengths to our arguments we still have a versatile and robust formalism within which to represent and evaluate competing claims and analyse the supporting evidence. Furthermore argumentation theory provides a framework in which to capture many human notions, like “persuasiveness” of argument, “rules of evidence” and “relevance” of background knowledge, in a surprisingly intuitive way.
Note
* It can be claimed, coorectly, the LP is so powerful that we can implement any uncertainty calculus using it. However, this does not change the fact that the native form of reasoning in LP is based on a categorical concept of truth.