Mildred L. G. Shaw and Brian R. Gaines
Knowledge
Science Institute
University of Calgary
Alberta, Canada T2N 1N4
One problem of eliciting knowledge from several experts is that experts may share only parts of their terminologies and conceptual systems. Experts may use the same term for different concepts, use different terms for the same concept, use the same term for the same concept, or use different terms and have different concepts. Moreover, clients who use an expert system have even less likelihood of sharing terms and concepts with the experts who produced it. This paper outlines a methodology for eliciting and recognizing such individual differences. It can be used to focus discussion between experts on those differences between them which require resolution, enabling them to classify them in terms of differing terminologies, levels of abstraction, disagreements, and so on. The methodology promotes the full exploration of the conceptual framework of a domain of expertise by encouraging experts to operate in a "brain-storming" mode as a group, using differing viewpoints to develop a rich framework. It reduces social pressures forcing an invalid consensus by providing objective analysis of separately elicited conceptual systems.
The elicitation of knowledge from one expert presents enough problems that elicitation from many might seem an unnecessary complication. However, a major reason for practical interest in expert systems is that in many domains, such as quality control, expertise is essentially distributed over many experts, and the purpose of the system is to bring it together (Hayes-Roth 1984). More fundamentally, the use of multiple experts in `brain-storming' or synectics sessions has been found valuable in problem-solving, and is an attractive technique to prevent individuals experts `blocking' and failing fully to explore and express their conceptual domains.
In a well-established scientific domain it is reasonable to suppose that there will be consensus among experts as to relevant distinctions and terms--that is, objective knowledge independent of individuals (Popper, 1968). However, the "expert systems" approach to system development has been developed for domains where such objective knowledge is not yet available, and the primary sources of knowledge are the conceptual structures of individual experts (Gaines, 1987b). When multiple experts are available for a domain where a consensus has not yet been reached, it is important to be able to compare their conceptual structures, both among themselves and with those of potential clients for the resultant knowledge-based system (Gaines & Shaw 1989).
It is important to note in dealing with multiple experts that consensual agreement upon domain concepts is only one of many significant possibilities. Experts may legitimately have different terminologies for the same domain concept. They may describe it at different levels of abstraction. One expert may describe a concept in operational terms and another in descriptive terms. They may also legitimately use the same terminology for different domain concepts. They may be using the same term distinguished by different contexts. One expert may have a very different conceptual framework or strategy from another. These differences may be carried through to an expert system design that allows users to obtain advice based on different and mixed sources of expertise. Compton and Jansen (1989) have found this important in practical system development, and suggest that the diversity of conceptual structures is fundamental to the way in which, through insight, individuals subsume data as knowledge. These differences may be carried through to an expert system design that allows users to obtain advice based on different and mixed sources of expertise. Boose & Bradshaw (1987) have incoporporated conflicting expertise in Aquinas enabling the user to ask for ask for dissenting opinions.
Thus, in a knowledge acquisition system, it is important not to attempt to force a false consensus on a group of experts on the assumption that there is some `correct' terminology and conceptual framework. However, it is also important to bring to light differences among the experts and make these clearly available for discussion. Some of them may reflect errors in elicitation, others differences in terminology, others differences in conceptual frameworks. In any event, the discussion of these differences is in itself a significant stage in the knowledge elicitation process.
The next section establishes a theoretical framework for knowledge elicitation from mutiple experts. The following sections describe a practical methodology based on this framework, a computer tool implementing this methodology, and results of using this tool to apply the methodology to practical knowledge elicitation.
Figure 1 shows a domain in which an expert system could be constructed from the knowledge of existing experts. The two "experts" shown, Expert 1 and Expert 2, are best thought of as roles in the individuals involved. These expert roles are distinguished by involving conceptual systems which, when applied to problem solving in the domain, lead to recognition of the individuals playing the roles as beings "experts" in the domain. The individuals will also have other roles involving other conceptual systems appropriate to other domains in which they may, or may not, be experts.
Figure 1 Experts acting in the same domain
The figure shows an overlap in the two conceptual systems indicating corresponding concepts. The experts will each have distinctions and terms for expressing the concepts involved. The possible overlaps between distinctions and terms leads to four relations which can occur between parts of the conceptual systems as summarized in Figure 2 and illustrated in Figures 3 through 6.
Figure 2 Consensus, conflict, correspondence and contrast among experts
In Figure 3: consensus arises if the conceptual systems assign the same term to the same distinction.
Figure 3 Consensus when the same terms are used for the same distinctions
In Figure 4: conflict arises if the conceptual systems assign the same term to different distinctions.
Figure 4 Conflict when the same terms are used for different distinctions
In Figure 5: correspondence arises if the conceptual systems assign different terms to the same distinction.
Figure 5 Correspondence when different terms are used for the same distinctions
In Figure 6: contrast arises if the conceptual systems assign different terms to different distinctions.
Figure 6 Contrast when different terms are used for different distinctions
The recognition of consensual concepts is important because it establishes a basis for communication using shared concepts and terminologies.
The recognition of conflicting concepts is important because it establishes a basis for avoiding confusion over the labeling of differing concepts with same term.
The recognition of corresponding concepts is important because it establishes a basis for mutual understanding of differing terms through the availability of common concepts.
The recognition of contrasting concepts is important because it establishes that there are aspects of the differing expertise about which communication and understanding may be very difficult, even though this should not lead to confusion. Such contrasts are more common than is generally realized. For example, it is possible to derive the same theorem in mathematics either by using an algebraic perspective, or a geometric one. There is nothing in common in these two approaches except the final result. It may still be possible to discuss the same domain using consensual and corresponding concepts that were not fundamental to the problem solving activities.
These considerations also apply to clients of the experts whose concepts and terminologies may be evaluated along these four dimensions in relation to those of the expert. The recognition of possible conflicts between the experts' and clients' use of terminology, and the provision of a variety of corresponding concepts, are major factors in the usability of an expert system.
In Figure 7, Client 1 is shown as having overlap only with Expert 1. He or she has no overlap whatsoever with any other individual in the figure, expert or user. Client 2 has overlap both with Client 3 and with the two experts. He or she is able to understand the concepts of Expert 2 and to a lesser extent those of Expert 1. Client 3 has no overlap with either expert, but has sufficient overlap with Client 2 that Client 2 can explain to him the advice given. In the development of an expert system, Client 1 may choose not to take advantage of Expert 2's expertise in the system. It would be particularly important to develop additional concepts and terminology to enable Client 3 to have adequate communication with, and understanding of, the system.
Figure 7 Experts and clients operating in the same domain
These are some of the possible relations which may occur between the conceptual systems of users and experts. The model is one of roles within an individual using a particular conceptual system and expressing it in a particular way to enter into an externally perceived and valued domain community. This model of coherent intellectual processes, each with its own individuality, each using its own conceptual system and its own terminology to enter into the domain community provides a cogent picture of the rationale behind, and the conflicts involved in, the development of communication and understanding in the community and in expert systems playing a role in that community.
Personal Construct Psychology (Kelly 1955, Shaw 1980) has been widely used as a basis for developing methods of knowledge acquisition from experts in a given domain (Shaw & Gaines 1983, Boose 1984, Diederich, Ruhmann & May 1987, Boose & Gaines 1988, Gaines & Boose 1988). Personal Construct Psychology is a psychology of the individual and many of its applications emphasize the idiosynchratic nature of conceptual systems (Mancuso & Shaw 1988). However, it is also a psychology of the individual interacting with the world and embedded in society (Shaw 1985). It encompasses shared concepts and conceptual systems, and their generation through experience. In particular it encompasses those systems that are so widely shared and so significant that they are construed as knowledge, and treated as having an existence virtually independent of their carriers.
Entity-attribute, or repertory grid, methodologies based on Personal Construct Psychology allow a significant part of the conceptual systems of experts to be elicited through manual or computer-based interactive interviewing techniques such as those of Knowledge Support System Zero (KSS0, Gaines 1987a, Gaines & Shaw 1987) and Aquinas (Boose and Bradshaw 1987). The resultant data structure is one in which the terms for entities and attributes in a domain have been specified by the expert, together with the values of those entities along the dimensions of the attributes. The entities are usually concrete items in the domain whose nature, definition and names can be agreed by experts and clients. The attributes reflect individual conceptual systems and may be used, and labeled, idiosynchratically.
Figure 8 shows the main tools in KSS0 relevant to the issues discussed in this paper:
Figure 8 Some tools in Knowledge Support System Zero
Figures 9 and 10 give examples of the interactive graphical elicitation of attributes using KSS0 in a study of the consistency of expertise across experts, and across time, in a small group of geographers specializing in mapping techniques and their application to geological exploration (Shaw & Woodward 1987). Figure 9 shows some of the mapping techniques used as entities through a rating screen from the program, Elicit, in which a geographer is rating the entity, trend surface analysis, on the attribute, doesn't incorporate geologic model--incorporates geologic model.
Figure 9 KSS0 attribute elicitation screen showing entities being rated
Figure 10 shows some of the characteristics derived as attributes through a match screen also from Elicit in which a geographer is being asked to distinguish punctual kriging from universal kriging.
Figure 10 KSS0 entity match screen showing ratings on attributes
Figure 11 shows the resultant entity-attribute grid.
Figure 11 Conceptual system in entity-attribute grid
Exchange methodologies were developed for the measurement of understanding and agreement between either two individuals, two roles or on two occasions (Shaw 1980). To do this two people, possibly experts with differing points of view, each elicit a grid in an area of common knowledge or experience. Each may choose his own entities independently of the other, and elicit and rate his or her attributes quite separately. Each then can Exchange his or her grid, that is use the other's entities and attributes but fill in his or her own the rating values. For example, in terms of Figure 9, the exchanging expert would see the terms doesn't incorporate geologic model--incorporates geologic model, but the entities would all be to the left and have to be dragged to the scale with no knowledge of where the other expert had previously placed them.
The Socio analysis in KSS0 allows members of a community to explore their agreement and understanding with other members, and to make overt the knowledge network involved (Shaw 1980, 1981, 1988). It is an extension of techniques such as SOCIOGRIDS (Shaw 1980) for deriving socionets and mode constructs from groups of individuals construing the same class of entities. The objective of Socio is to take different conceptual systems in the same domain and compare them for their structure, showing the similarities and differences. It may be regarded as the implementation of a simple form of analogical reasoning. Figure 12 shows the basis of operation of Socio--consider one set of data as being the base class defined by its entities, their attributes and values, and consider variant classes:
Figure 12 Possible comparisons between base and related systems
If, as in the lower right, neither entities nor attributes are common then no comparison is possible.
When a number of classes representing the same sub-domain are available, Socio also provides two other forms of analysis:
Using the concepts developed above it is possible to develop a complete methodology for eliciting and analyzing consensus, conflict, correspondence and contrast in a group of experts, and implement this as an automatic process using the tools in KSS0. The methodology has three phases shown in Figures 13 through 15.
Phase 1: Domain Discussion and Instantiation
In phase 1 a group of experts comes to an agreement over a set of entities which instantiate the relevant domain. This is the initial phase of any entity-attribute methodology, whether used with individuals or groups. However, with individuals the elicitation techniques may be used to elicit more entities as the exploration of the conceptual domain proceeds. When comparing multiple experts it is important that a set of entities is established at the start of the comparative study, and that the experts mutually agree on the definitions of these.
Figure 13 Conceptual systems comparison methodology Phase 1
A convenient way to generate this set of entities is have each expert individually use Elicit to enter his or her conceptual system for a domain, and then extract the elicited entities from all the grids for discussion and consolidation by the group. This has the advantage that the experts gain some experience in the use of the KSS0 tools and can take advantage of the full elicitation facilities.
Phase 2: Conceptualization and Feedback
In phase 2 each expert individually elicits attributes and values for the agreed entities. The resultant conceptual systems will have the same entities but different attributes and can be analyzed by the Attribute Compare component of Socio as shown in Figure 12. This takes each attribute in one grid and determines the best matching attribute in the other grid, if there is one. The result is a mapping from the attributes in one expert's grid to those in another's as shown in Figure 14.
Figure 14 Conceptual systems comparison methodology Phase 2
In evaluating this mapping we are not particularly interested in the terminology used but rather whether one expert has an attribute that can be used to make the same distinctions between the entities as does the other expert, regardless of whether these distinctions are called by the same terms. If such a correspondence occurs then the experts have a basis for mutual understanding of the underlying concept.
If an attribute in one system has no matching attribute in the other then it stands in contrast to all the other expert's attributes, and it may be very difficult for the other expert to understand the use of this term.
Note that the arrows in Figure 14 need not be symmetric. Attribute A in G1 may be best matched by attribute B in G2, but attribute B in G2 may be best matched by attribute C in G1. The only constraint is that if an attribute has an incoming arrow then it will have an outgoing arrow.
Thus, the second phase provides the basis for analysis of correspondence and contrast as already discussed.
Phase 3: Exchange and Compare
In phase 3 each expert individually exchanges elicited conceptual systems with every other expert, and fills in the values for the agreed entities on the attributes used by the other experts. The resultant conceptual systems will have the same entities and attributes and can be analyzed by the Entity-Attribute Compare component of Socio as shown in Figure 12. This takes each attribute in one grid and determines whether it matches the corresponding attribute in the other grid.
Figure 15 Conceptual systems comparison methodology Phase 3
The result is a map showing consensus when attributes with the same labels are used in the same way and conflict when they are not as shown in Figure 15. Thus, the third phase provides the basis for analysis of consensus and conflict as already discussed.
Figure 15 also shows the correspondence and contrast relations analyzed in phase 2 as a relations between two of the grids used in phase 3. Thus, for two experts, four grids obtained by one elicitation and one exchange each, are sufficient to classify the relations between attributes in terms of consensus, conflict , correspondence and contrast. The methodology scales up linearly for each expert, so that n experts will be involved in n elicitations, one base elicitation and n-1 exchanges.
These three phases result in the experts' conceptual systems having become overt and inter-related. They lead naturally to later phases in which classes, objects and rules can be developed incorporating consensual, corresponding, and some of the contrasting attributes as kernel knowledge and, possibly, the conflicting and remaining contrasting attributes as `other opinions.'
This section gives an example of the methodology in action using data from the study of geographers specializing in mapping techniques previously cited (Shaw & Woodward 1987).
Figure 16 shows an entity-attribute comparison from the geographic study in which expert B has filled in the values for a class defined by entities and attributes elicited from expert A. The print out shows the matches sorted with best first. The cumulative percentage is given of the number of attributes with matches greater than the value shown. In the list of attributes at the top, it can be seen that there is consensus on interval data--nominal data, but conflict on requires no model--requires a model and linear interpolation--nonlinear interpolation. There is clear consensus on the top four attributes, clear conflict on the lower five, and uncertainty about the remaining three.
Figure 16 Entity-attribute comparison of expert B with expert A
In the list of entities at the bottom, it can be seen that there is close agreement on vector trend analysis, but high disagreement on probability mapping. This output can be used to focus a discussion between the experts on why they differ in their views of probability mapping and the classification of mapping techniques in terms of linear or nonlinear interpolation. For example, the first attribute, requires no model--requires a model shows high disagreement, and looking further into the elicited data it can be seen that expert A thinks that probability mapping requires a model, whereas expert B thinks that probability mapping requires NO model. On inquiring into this, the explanation given was couched in terms of what one actually means by the term "model", indicating the conflicting use of terminology.
Figure 16 can be redrawn as a difference grid where rating values (in this case 1 to 5) for expert B's ratings of expert A's entities on his attributes are subtracted from expert A's similar rating values respectively. Figure 17 shows this with the entities and attributes about which they agree the most in the top right corner, shown by no difference or a difference of only 1; and those with most disagreement towards the bottom left, shown by the maximum difference of 4 or a large difference of 3. Hence from this difference grid, the consensus and conflicts can easily be identified and discussed by the experts.
Figure 17 The difference grid for experts A and B
Figure 18 shows an attribute comparison from the geographic study in which expert B has specified attributes and filled in the values for a class defined by entities elicited from expert A. The print out shows the matches sorted with best first. The cumulative percentage is given of those with matches greater than the value shown, and the attribute from B which best matches each from A is shown beneath it.
Figure 18 Attribute comparison of expert B with expert A
It can be seen that:
Figure 19 shows each of the attributes from Figures 16 and 18 put into the appropriate quadrant of Figure 2. There is no significant example of contrast in this data, possibly because the two experts work very closely together. Such examples do arise in a full analysis of the three experts in the original study.
Figure 19 Consensus, conflict , correspondence and contrast from Figures 16 and 18
Figure 20 shows a mode attributes analysis of data from the three geographers which extends the clusters noted above. These mode attributes are essentially inter-conceptual-system clusters of corresponding attributes from the three experts. The first may be interpreted as centering around Global-Local encompassing a variety of concepts related to this; the second as around autocorrelation techniques and their consequences; the third as around the complexity of the technique; the fourth around the type of data; and the fifth around the number of variables considered.
These five mode attributes can be interpreted as indicating stereotypical lines of reasoning most used by these experts. This output can be used as a basis for discussion among the experts on whether these conceptual clusters should be split because they confound different concepts expressed in apparently corresponding attributes, or retained as being the same concept expressed in different terminologies. Once this form of analysis has been discussed by the group it is readily edited and extended.
Figure 20 Mode attributes from three experts
Figure 21 shows a socionets analysis of the same data based on a set of comparisons like that shown in full in Figure 14. It can be seen from the first two links that the conceptual system of expert B encompasses the majority of attributes used by experts A and C. This indicates that expert B has a deeper knowledge of the topic than either expert A or expert C. That of expert A encompasses the majority of the attributes of B, shown in the third link. However, that of expert C does not encompass many of those of A and B, and that of A does not encompass many of those of C indicating that C has a different point of view, or a background of different experiences from those of A and B. This could then be explored in detail with expert C.
Figure 21 Socionet analysis of three experts
Figure 22 shows this information expressed in the socionet links of relations between the experts' conceptual systems for the domain. Each new link is shown by a black arrow as it is added into the sequence.
Figure 22 Socionet of relations between experts' conceptual systems
Phase 1: Domain Discussion and Instantiation
Phase 2: Conceptualization and Feedback
Phase 3: Exchange and Compare
Phase 4: Rule Derivation and Validation
Any comparison of conceptual systems necessarily involves approximation since a complete conceptual system may involve indefinitely complex relations and different concepts will never be identical in all respects. However, in the initial phases of knowledge acquisition, highlighting gross similarities and differences is itself valuable in promoting directed discussion among experts and clients that can lead to the exposure of more subtle relationships. As a start one wishes to elicit the major distinctions that an individual uses in a domain, the terminology used for them, and the relation of such distinctions and terminology to those of others.
Entity-attribute grid elicitation is an extensional approach in that individuals are asked to specify a set of entities in a domain, then make distinctions among them, naming the distinctions and classifying all the specified entities in terms of them. The extension of a distinction determined in this way is only an approximation to the underlying concept since critical entities may be missing in the classification. However, both manual and computer-based elicitation techniques attempt to prompt the individual for additional entities to discriminate between extensionally related distinctions (that is making the same, or similar, classifications).
Group comparisons, as developed in this paper, have similar dynamics--an extensionally apparent consensus or correspondence may be accepted or rejected, and the rejection may be based on the specification of additional entities as counter-examples. Knowledge acquisition is essentially a negotiation process leading to approximations to conceptual structures that are adequate for some practical purpose such as system development.
The methodology described in this paper provides facilities for revealing the similarities and differences in the concept systems of different experts, or the same experts at different times, construing a domain defined through common entities or attributes. It can be used to focus discussion between experts on those differences between them which require resolution, enabling them to classify them in terms of differing terminologies, levels of abstraction, disagreements, and so on.
Note that the derivation of consensual, conflicting, corresponding and contrasting attributes is completely algorithmic, based solely on the data in the grids. This derivation is done by a computer program, not a knowledge engineer, and its basis can be demonstrated clearly to the experts through computer output such as the difference grid of Figure 17. Thus, there are no opinons being expressed about the correctness of the use of the attributes and terminology, that the differences highlighted are `right' or `wrong.' It is open to the experts to consider, discuss and explain these differences, changing or retaining them as they wish. Conflicts can be retained in the final system if desired by tagging classes, objects and rules with the sources from which they derive.
Note also that the methodology described applies equally to the relations between the conceptual structures of experts and representative clients for the system. The differences in conceptual systems shown in Figure 7 and the problems that may arise from them may be made overt, and the methods for overcoming these for the different classes of clients, may be analyzed in detail using the derivation of consensual, corresponding, conflicting and contrasting attributes from experts and clients.
The methodology promotes the full exploration of the conceptual framework of a domain of expertise by encouraging experts to operate in a "brain-storming" mode as a group, using differing viewpoints to develop a rich framework. It reduces social pressures forcing an invalid consensus by providing objective analysis of separately elicited conceptual systems.
Financial assistance for this work has been made available by the Natural Sciences and Engineering Research Council of Canada. The KSS0 system was made available by the Centre for Person Computer Studies. We are grateful to many colleagues at the knowledge acquisition workshops for discussions which have highlighted many of the issues raised in this paper.
Boose, J.H. (1984). Personal construct theory and the transfer of human expertise. Proceedings AAAI-84, 27-33. California: American Association for Artificial Intelligence.
Boose, J.H. & Bradshaw, J.M. (1987) Expertise transfer and complex problems: using AQUINAS as a knowledge acquisition workbench for knowledge-based systems. International Journal of Man-Machine Studies 26, 3-28.
Boose, J.H. & Gaines, B.R., Eds. (1988). Knowledge Acquisition Tools for Expert Systems. London, Academic Press.
Compton, P. & Jansen, R. (1989). A philosophical basis for knowledge acquisition. Proceedings of the Third European Workshop on Knowledge Acquisition for Knowledge-Based Systems (EKAW'89). Paris, France (June).
Diederich, J., Ruhmann, I. & May, M. (1987) KRITON: a knowledge-acquisition tool for expert systems. International Journal of Man-Machine Studies 26, 29-40.
Gaines, B.R. (1987a) Rapid prototyping for expert systems. Oliff, M. (Ed). Intelligent Manufacturing: Proceedings from First International Conference on Expert Systems and the Leading Edge in Production Planning and Control. pp.45-73. Menlo Park, California, Benjamin Cummins.
Gaines, B.R. (1987b) An overview of knowledge acquisition and transfer. International Journal of Man-Machine Studies 26(4), 453-472 (April).
Gaines, B.R. (1989) An Ounce of Knowledge is Worth a Ton of Data: Quantitative Studies of the Trade-Off between Expertise and Data based on Statistically Well-Founded Empirical Induction. Proceedings of 6th International Workshop on Machine Learning, Cornell University, Ithaca.
Gaines, B.R. & Boose, J.H., Eds. (1988). Knowledge Acquisition for Knowledge-Based Systems. London, Academic Press.
Gaines, B.R. & Shaw, M.L.G. (1987). Knowledge support systems. ACM MCC-University Research Symposium. Austin, Texas: MCC. pp.47-66.
Gaines, B.R. & Shaw, M.L.G. (1989). Comparing the conceptual structures of experts. Proceedings IJCAI-89. California: American Association for Artificial Intelligence.
Hayes-Roth, F. (1984). The industrialization of knowledge engineering. Reitman, W., Ed. Artificial Intelligence Applications for Business. pp. 159-177. Norwood, New Jersey: Ablex.
Kelly, G.A. (1955). The Psychology of Personal Constructs. New York: Norton.
Mancuso, J.C. & Shaw, M.L.G., Eds. (1988). Cognition and Personal Structure: Computer Access and Analysis. New York: Praeger Press.
Popper, K.R. (1968). Epistemology without a knowing subject. Van Rootselaar, B. (Ed.). Logic, Methodology and Philosophy of Science III. pp. 333-373. Amsterdam, Holland: North-Holland.
Shaw, M.L.G. (1980). On Becoming a Personal Scientist. London: Academic Press.
Shaw, M.L.G., Ed. (1981). Recent Advances in Personal Construct Technology. London: Academic Press.
Shaw, M.L.G. (1985). Communities of knowledge. Epting, F. & Landfield, A.W., Eds. Anticipating Personal Construct Psychology. pp. 25-35. Lincoln: University of Nebraska Press.
Shaw, M.L.G. (1988). An interactive knowledge-based system for group problem-solving. IEEE System, Man and Cybernetics 18, 610-617.
Shaw, M.L.G. & Gaines, B.R. (1983). A computer aid to knowledge engineering. Proceedings of British Computer Society Conference on Expert Systems, 263-271 (December). Cambridge.
Shaw, M.L.G. & Woodward, J.B. (1987) Validation in a knowledge support system: construing and consistency with multiple experts. International Journal of Man-Machine Studies 29, 329-350.