Hostname: page-component-5b777bbd6c-7sgmh Total loading time: 0 Render date: 2025-06-20T09:52:42.459Z Has data issue: false hasContentIssue false
  • English
  • Français

Explaining reasoning: an overview of explanation in knowledge-based systems

Published online by Cambridge University Press:  07 July 2009

Richard W. Southwick
Richard W. Southwick
Affiliation:
Department of Computing, Imperial College, London, UK
Get access Rights & Permissions [Opens in a new window]

Abstract

There seems to be general agreement amongst those involved in KBS research that in order to be useful, a system must be able to explain its reasoning to a user. This paper reviews the development of explanation facilities in knowledge-based systems. It differentiates between explanation as a problem-solving process, and that which explains a reasoning process. This review concentrates on the latter, identifying and giving examples of three categories of reasoning explanation.

We then look at user requirements for explanation. What makes an explanation useful depends on the expectations of a user, which in turn depends on such issues as user background and system context. Several techniques are examined that have been applied to the problem of producing explanations that are appropriately structured and conveyed.

Finally, we discuss some of the work that has been done in describing theories of human discourse and explanation, and some issues that will become increasingly important for future explanation systems.

An extensive annotated bibliography is provided.

Type
Research Article
Information
The Knowledge Engineering Review , Volume 6 , Issue 1 , March 1991 , pp. 1 - 19
Copyright
Copyright © Cambridge University Press 1991

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Clancey, WJ, 1983. “The epistemology of a rule-based expert system: a framework for explanation.” Artificial Intelligence 20 215–251. Knowledge can play one of three roles: structure (relations and structure of data); strategy (procedure for applying rules); and support (justification for applying rules; deep knowledge). Mycin's rules have compiled out much of this information and as a result, although optimized, the rule base is difficult to maintain, non-transferable, and not useful for explanation. Strategic, structural and support knowledge should be made explicit.Google Scholar
Davis, R, 1982. “Teiresias: applications of meta-level knowledge”. In: Davis, R and Lenat, DB (eds.), Knowledge-Based Systems in Artificial Intelligence McGraw-Hill. In his chapter on explanation, Davis views explanations as being necessary to monitor the performance and output of a program. Information must be detailed, but not too detailed, and must be complete. The program keeps a trace of rules that it uses during the deductive process. HOW and WHY commands are used to move up and down the reasoning chain. The confidence values attached to rules are used as an information metric to assist in providing the right level of detail in explanations.Google Scholar
Hammond, P and Sergot, MJ, 1983. “A PROLOG shell for logic based expert systems” Proc. 3rd BCS Expert Systems Conf. Cambridge, 95–104. Description of APES, Augmented Prolog for Expert Systems. APES builds upon Prolog execution and representation by offering a user interface that includes HOW and WHY explanations, and Query-the-User, wherein the user is treated as a data-base, and can be queried for missing information.Google Scholar
Hasling, DW, Clancey, WJ and Rennels, G, 1984. “Strategic explanations for a diagnostic consultation system” International Journal of Man-Machine studies 20 3–19. Hasling's main thesis is that to be useful, an expert system must be able to explain its problem solving strategies, the NEOMYCIN knowledge base includes meta-level knowledge in the form of tasks (metalevel goals and subgoals) and metarules (methods for performing tasks). This information is used in making clear the plans and methods used in reaching a goal. The general approach to problem solving is mentioned, as well as the particular action taken. The user can examine the strategy, using normal How and Why questions to explore the meta-rule space.Google Scholar
Hollan, JD, Hutchins, EL and Weitzman, L, 1984. “STEAMER: an interactive inspectable simulation-based training system” AI Magazine 5(2) 15–27. Description of the STEAMER system.Google Scholar
Joshi, A and Webber, B, 1984. “Living up to expectations: computing expert responses” Proc. AAAI-84, Austin, Texas, 169–175. In cooperative man–machine interactions, it is necessary but not sufficient for a system to respond truthfully to a user's question. If the system believes that a planned response might mislead the user, it must block that conclusion by modifying its response. This paper characterizes tractable cases in which the system can anticipate the possibility of the user drawing false conclusions, and develops a formal method for computing the inferences that a user might draw from a response from the system.Google Scholar
Kosy, DW and Wise, BP, 1984. “Self-explanatory financial planning models” Proc. AAAI-84, Austin, Texas, 176–181. When explaining the results of a computation presented by a financial modelling system, simply presenting the formulas and the input data is not sufficient. Kosey's strategy is to: distinguish the relevant parts of the model by determining the focus of the question; distinguish the significant effects by finding a set of variables in the model that are applicable; translate quantitative information into qualitative (e.g., X goes up because Y went down); and present the explanation using output templates.Google Scholar
McKeown, KR, Wish, M and Matthews, K, 1985. “Tailoring explanations for the user” Proc. 9th International Joint Conference on Artificial Intelligence,Los Angeles, California, 794–798. An expert system should provide explanations that correspond to the concerns of the user. Explanations can be tailored by inferring the user's point of view and goal from a brief discourse segment. Builds on Allen (1980) to derive the user's goal based on a series of utterances (rather than a single one).Google Scholar
Neches, R, Swartout, WR and Moore, JD, 1985. “Enhanced maintenance and explanation of expert systems through explicit models of their development” IEEE Transactions on Software Engineering 11 1337–1351. Identifies some of the shortcomings of XPLAIN and attempts to improve them. Describes EES (Explainable Expert Systems) approach. The knowledge engineer produces a semantic model of the domain, which is used by an automatic program writer. XPLAIN had context-dependent terms, limited number or question types, and was limited to goal refinement. EES implements a mapping between terms and concepts, expanded question types, and goal reformulation if subgoal refinement fails. Explanations are produced according to question types, each with an associated strategy for answering.CrossRefGoogle Scholar
Richer, MH and Clancey, WJ, 1985. “GUIDON-WATCH: a graphic interface for viewing a knowledge-based system” IEEE Computer Graphics and Applications 5 51–64.Google Scholar
Rissland, EL, Valcarce, EM and Ashley, KD, 1984. “Explaining and arguing with examples” Proc. AAAl-84, Austin, Texas, 288–294. Explores the use of examples in two domains-on-line help and legal reasoning. On-line help can be made more intelligent by embedding examples in explanations. Legal argumentation can be strengthened through the use of hypotheticals. Examples can be customized to fit a user's ability or circumstance, using knowledge about the user's directory and files, for example.Google Scholar
Schlobohm, DA and Waterman, DA, 1987. “Explanation for an expert system that performs estate planning” Proc. First International Conference on AI and Law, Boston, Massachusetts. EPS consults with a client to create a will. Since the user is typically unknowledgeable about the domain, the system must explain its actions and educate the user. EPS provides several types of explanations. Definitional explanations are assembled from the frame hierarchy, using pop-up menus. How-concluded, similar to HOW, use justification procedures attached to rules which generate text. EPS will return a list of suggestions which match the user's needs. These are used for Alternative-plan explanations and Compare-and-contrast explanations.Google Scholar
Scott, AC, Clancey, WJ, Davis, R and Shortliffe, EH, 1977. “Explanation capabilities of knowledge-based production systems” in: Buchanan, BG and Shortliffe, EH, eds., Rule-Based Expert Systems, Addison-Wesley. Mycin's explanation capability was expanded to be able to answer questions. The Explanation Capability comprises two modules: the RSC (reasoning status checker) is used during consultation to allow the user to examine the reasoning chain (how and why questions). The GQA (general question answerer) uses natural language routines to allow the user to ask questions about the conclusions, the static knowledge base, facts, rules, etc. A set of answering specialists are capable of answering questions on a particular topic, e.g., static knowledge or judgmental knowledge.Google Scholar
Southwick, RW, 1988. “Topic explanation in expert systems” in: Kelly, B and Rector, A, eds., Research and Development in Expert Systems V Cambridge University Press. A system can explain its strategy through the selection of ‘landmarks’ that designate topics in the knowledge base.Google Scholar
Swartout, WR, 1981. “Explaining and justifying expert consulting programs” Proc. 7th International Joint Conference on Artificial IntelligenceVancouver,BC, 815–822.Google Scholar
Swartout, WR, 1983. “XPLAIN: a system for creating and explaining expert consulting programs” Artificial Intelligence 21 285–325. Good explanations should be capable of presenting the reasoning and justification behind the actions taken by a program. XPLAIN writes the domain program itself, thereby remembering why it did what it did. Knowledge is separated into a Domain Model, comprising of descriptive facts, and Domain Principles, “how-to” methods and rules. The Writer module starts from a top-level goal, and gradually refines it by creating more specific subgoals, until the level of system primitives is reached. Explanations are produced using knowledge in the domain model, domain principles, and the execution trace.CrossRefGoogle Scholar
Wallis, JW and Shortliffe, EH, 1982. “Explanatory power for medical expert systems: studies in the representation for clinical consultations” Stanford Dept of Computer Science Report CS-82–923. The goal of the research is the generation of customized explanations, aimed according to the experience and knowledge of the user. When providing explanations of causal reasoning, leave out the concepts that are not understood. User understanding of concepts can be determined through the use of a user-selected difficulty level. If the causal chain for a query has the form tl ⇒ t2 ⇒ t3 and t2 is not understood (according to the difficulty level), the explanation presented would be tl ⇒ t3.Google Scholar
Weiner, JL, 1980. “BLAH, a system which explains its reasoning” Artificial Intelligence 15 19–48. BLAH is primarily concerned with structuring explanations so that they do not appear too complex. It uses a TMS, where each rule has justifications for belief attached to it. The knowledge base is segmented into a system and user's view, allowing system to reason using one set of information. Explanations can then be generated using the other, so that details already known by the user can be deleted. The knowledge base is also split into partititions which contain rules that are related in some way. Explanations are assembled, using natural language templates, from a node in a reasoning tree in such a way that the underlying structure of the reasoning tree can be recovered from the explanation.CrossRefGoogle Scholar
Allen, JF and Perrault, CR, 1980. “Analyzing intention in utterances” Artificial Intelligence 15 143–178. Describes a model of cooperative behaviour and describes how such a model can be applied in a natural language understanding system. Discusses several types of speech acts, and the formulation of actions and plans to deal with them. Develops goal inference techniques using plausible inference rules, representation of domain plans, and representation of speech plans.CrossRefGoogle Scholar
Cawsey, A, 1989. “Explanatory dialogues” Interacting with Computers 1 69–92. Adopts a grammar-based approach for explanation construction.CrossRefGoogle Scholar
Draper, SW, 1987. “Explanation, paradox and abduction” Proc. 2nd Workshop of the Explanation Special Interest Group 10–14, Alvey Knowledge Based Systems Club. If external explanations are absent, people will construct their own explanations when forming abductive hypotheses, and when dealing with paradoxes. In this context, the process of explanation-seeking is driven by a need to fit data into preexisting models, but is more of an active constructive process.Google Scholar
Draper, SW, 1987. “A user-centred concept of explanation” Proc. 2nd Workshop of the Explanation Special Interest Group 15–23, Alvey Knowledge Based Systems Club. There is little linguistic marking of the role of an utterance, so it is difficult to determine what kind of explanation is desired by a user. Consequently, taxonomies of question types have little promise. Explanations must calculate the difference between the Explainer's and the Inquirer's belief sets, and must recognize the Inquirer's intention in asking. This last will usually require an extended dialogue.Google Scholar
Goguen, JA, Weiner, JL and Linde, C, 1983. “Reasoning and natural explanation” International Journal of Man-Machine Studies 19 521–559. Presents a precise and computationally effective model of the structure of human explanation. Natural explanations are represented in a tree structure whose nodes correspond to the three major types of justification: giving a reason, giving an example, and eliminating alternatives. Explanation production is represented by a sequence of transformations on the tree. Focus of attention is represented by pointers in the tree, and shifts of focus by pointer movement.CrossRefGoogle Scholar
Gumpertz, JJ and Hymes, D, 1972. Directions in Sociolinguistics: The Ethnography of Communication Holt, Rinehart and Winston. Ethnomethodology.Google Scholar
Hempel, CG, 1965. Aspects of Scientific Explanation Free Press. Philosophical investigations into scientific or deductive theories of explanation.Google Scholar
Hughes, S, 1986. “Question classification in rule-based systems” Proc. Expert Systems '86, British Computer Society Specialist Group on Expert Systems. An implementation of Lehnert's “question type” model for rule-based systems. Current expert systems are able to treat few of the 14 question types identified by Hughes.Google Scholar
Joshi, A, Webber, B and Sag, I (eds.), 1981. Elements of Discourse Understanding Cambridge University Press. Collection of essays on discourse understanding and processing.Google Scholar
Lewis, C and Mack, RL, 1982. The role of abduction in learning to use computer systems Human Factors Research Report RC 9433, IBM. Experiments in determining how people develop theories about computer operation.Google Scholar
Linde, C and Goguen, JA, 1978. “Structure of planning discourse” Journal of Social Biological Structure 1 219–251. Using Watergate transcripts, this paper studies the structure of planning discourse. Planning is a mode of discourse with a regular structure, and a cooperatively formulated plan can be represented as a tree structure.Google Scholar
Buchanan, BG and Shortliffe, EH (eds.), 1984. Rule-based Expert Systems Addison-Wesley. Collection of articles, previously published, about the Stanford Heuristic Programming project MYCIN.Google Scholar
Clancey, WJ and Letsinger, R, 1981. “NEOMYCIN: reconfiguring a rule-based expert system for applications to teaching” Proc. the 7th International Joint Conference on Artificial Intelligence,Vancouver B.C.,Canada, 829–836. Describes attempts to reuse the MYCIN rule base for a tutoring system.Google Scholar
Davis, R and Buchanan, BG, 1977. “Meta-level knowledge: overview and applications” Proc. 5th International Joint Conference on Artificial IntelligenceMIT,Massachusetts, 819–826.Google Scholar
de Kleer, J and Brown, JS, 1985. “A qualitative physics based on confluences” in: Hobbs, JR and Moore, RC (eds.), Formal Theories of the Common-Sense World 109–183, Ablex. Develops a theory of qualitative causal physics to describe the behaviour of systems. System variables do not take quantitative values, but are simply assigned one of the qualitative values +, - or 0. A confluence is a qualitative differential equation, used as a modelling tool. Using this qualitative physics, many of the concepts of classical physics are derived.Google Scholar
Klein, D and Finin, T, 1987. “What's in a deep model?” Proc. IJCAI-87, Milan, Italy, 559–562. Gives an operational definition of ‘knowledge depth’ intended to be useful to knowledge engineers in characterizing deep and shallow models. A definition of the relation deeper-than states that one model is deeper than another if there is implicit knowledge in the second that is explicit in the first.Google Scholar
Bobrow, DG and Stefik, M, 1983. The LOOPS Manual Xerox Corporation.Google Scholar
Chandrasekaran, AB and Mittal, S, 1983. “Deep versus compiled knowledge approaches to diagnostic problem-solving” International Journal of Man-Machine Studies 19 425–436. The authors discuss the relationship between deep and compiled knowledge in expert systems. They claim that most extant systems employ rules that are simply pattern-decision pairs. They refute the claim that deep knowledge is necessary for reasoning, when a compiled version (D) of that knowledge can handle all problems that a deep-knowledge system (U) could. If D fails to solve a case in some way, it is due to one of several reasons: the information is missing in U; D's problem-solving strategy is too weak; D is improperly compiled from U; the compilation process causes a combinational explosion, the reduction of which results in a loss of completeness. Satisfactory explanations can be generated from D; if further (deeper) explanations are required, then a text string summing up the knowledge in U can be added to each node in D. Deep knowledge is not necessarily causal in nature, as some have argued.CrossRefGoogle Scholar
Coombs, M and Alty, J, 1984. “Expert systems: an alternative paradigm” International Journal of Man–Machine Studies 20 21–43. Human experts are most often called upon to assist other experts in extending and refining their understanding of a problem at the junction of two domains of knowledge. The first section of the paper describes human interaction in the domain of computer advice. The strategy favoured by participants involved the generation and then critiquing of explanations for some set of problem phenomena. The MINDPAD system was implemented to aid novice Prolog programmers. The programmer enters a problem (in the form of a Prolog program), then an explanation. The system checks the user's idea of how the program will execute against its own, and then tells the user what is wrong, so the user can supply a new explanation.CrossRefGoogle Scholar
de Kleer, J, 1986. “An assumption-based truth maintenance system” Artificial Intelligence 28 127–162. Introduces a truth maintenance system that records the base assumptions that support some datum. The ATMS operates in a breadth-first manner, eliminating all backtracking, and permitting reasoning with several possibly inconsistent contexts.CrossRefGoogle Scholar
Josephson, JR, Chandrasekaran, B, Smith, JW and Tanner, MC, 1987. “A mechanism for forming composite explanatory hypotheses” IEEE Transactions on Systems, Man and Cybernetics 17(3) 445–454. In order to perform “abductive inference” (going from data to an explanatory hypothesis), a mechanism is presented that assembles hypothesis parts into a unified explanatory hypothesis. The criteria for “best” are internal consistency, explanatory power, plausibility, consistency with the evidence, and parsimony. The assembler uses the data to be explained, a set of sub-hypotheses, and a plausibility rating to select the best explanation.Google Scholar
Langlotz, CP and Shortliffe, EH, 1983. “Adapting a consultation system to critique user plans” International Journal of Man–Machine Studies 19 479–496. ONCOCIN (cancer therapy) is adapted to accept, analyse and critique a physician's plan; to explain the significant differences between the system's plan and the user's. Data are entered through the “Interviewer”, while the “Reasoner” uses a rule-based reasoning system to arrive at a recommendation. The user enters his plan, and the system employs hierarchical plan analysis to determine where the two plans differ. Because the domain is hierarchical, it is possible to find the most general set of differences. Explanations of the difference set are produced using an agenda of parameters that differ. The user may select an item from the agenda to be explained, and if in that explanation further parameters are encountered, they are added to the agenda.CrossRefGoogle Scholar
Lehnert, WG, 1978. The Process of Question Answering Lawrence Erlbaum Associates. Uses a taxonomy of question types to drive a question answering facility. The TEXT system operates in a conceptual dependency context, splitting a question into its query part and CD concept.Google Scholar
McDonald, DD, 1982. “Natural language generation as a computational problem, an introduction” in: Brady, M, ed., Computational Models of Discourse MIT Press. Describes Mumble, a full-size text generation system.Google Scholar
McKeown, KR, 1985. Text Generation Cambridge University Press. The use of discourse strategies and focus constraints to generate natural language text.CrossRefGoogle Scholar
Miller, PL, 1984. A Critique Approach to Expert Computer Advice: ATTENDING Pitman. Presents an alternative paradigm to the standard MYCIN-style expert system. In Miller's approach, the user presents his idea of a solution, and the system critiques it, using its expert knowledge base. Miller uses an ATN to represent knowledge in his domain, anesthetic management. Given a user-entered plan, the system can produce possible alternatives, weed out the high-risk options, and produce a comparison between the user's plan and the system's. Polished text is produced by attaching template parts to nodes in the ATN, so that text is built up as the network is traversed.Google Scholar
Rich, EA, 1979. “User modeling via stereotypes” Cognitive Science 3 329–354. User models may be built by using known user characteristics to select a “stereotype” model that partially fits the user, and then individualizing that model to match user details.Google Scholar
Rich, EA, 1983. “Users are individuals: individualizing user models” International Journal of Man–Machine Studies 18 199–214. This paper is concerned with building individual, implicit, long-term models. Techniques are discussed: identification of concepts used; gauging responses that satisfy the user; and using stereotypes to generate many facts from few.Google Scholar
Rissland, EL, 1983. “Examples in legal reasoning” Proc. 8th International Joint Conference on Artificial Intelligence,Karlsruhe,West Germany.Google Scholar
Self, JA, 1977. “Concept teaching” Artificial Intelligence 9 197–221. Illustration of some design principles for concept teaching in CAI. Program and human concept learning performance is compared, and the incorporation of a concept learning program into a teaching system is discussed.CrossRefGoogle Scholar
Sergot, MJ, 1983. “A query-the-user facility of logic programming” in: Degano, P and Sandewall, E, eds., Integrated Interactive Computer Systems 27–41, North-Holland. A model of the user as a logical database is presented. This model is useful in expert systems that request missing information from a user.Google Scholar
Shortliffe, EH, 1976. Computer-based medical consultations: MYCIN Elsevier. The MYCIN handbook.Google Scholar
Sleeman, D, 1984. “UMFE: a user modelling front end subsystem” Stanford Research Report. UMFE determines the user's level of sophistication by asking a few questions, then presents an answer to a question in terms of concepts understood by the user. The knowledge base includes a list of domain concepts, each with a difficulty and importance rating. The system sets the difficulty level by interactively asking the user whether a concept is understood. If it is, it is assumed that all of its siblings with the same importance value are also known. See Wallis (1982) for initial work that this paper builds on.Google Scholar
Southwick, RW, 1990. A reason maintenance system for backward reasoning systems Research report, DOC 90/11, Imperial College, London. Describes reason maintenance techniques for backward reasoning systems, in order to eliminate redundant processing and maintain a consistent set of beliefs.Google Scholar