Path. Res. Pract. 188,396-404 (1992)

Construction of the Knowledge File for an Image Understanding System P. H. Bartels, D. Thompson and J. E. Weber Optical Sciences Center, Department of Pathology, and Department of Statistics, University of Arizona, USA

SUMMARY

To enable an image understanding system to provide an automated interpretation of diagnostic imagery it must have access to all of the concepts, procedures and methods used by human experts. The paper describes information elicitation from experts of different domains and the construction of a knowledge file. Uncertainty management is based on Bayesian belief network methods.

Introduction Image understanding systems are designed to provide an automated assessment of imagery based on information offered by the imagery itself and on wide ranging additional knowledge about the domain of application. The design and implementation of such a system for interpreting histopathologic images was described earlier 2,4. It is based on three interacting expert system modules. Communication with pathologists is carried out by a diagnostic expert system module13. This module interacts with an interpretive expert system that relates diagnostic knowledge to histometric measures. The interpretive expert system, in turn, calls on a scene segmentation expert system 11 to provide the required image information. The image understanding system is designed to process images that are difficult to segment. It follows a reasoning process based on a model of the histologic domain. Also, the image understanding system has built-in knowledge about the capabilities of each segmentation algorithm to resolve given segmentation problems 2, 11. Thus, the system can adaptively apply different segmentation algorithms at different locations. The scene segmentation capabilities recently were tested on an extensive set of histopathologic sections of prostate lesions. Correct segmentation was achieved for 58 of 60 sections; more than 8·5 % of approximately 17000 nuclei were correctly delineated 6 . 0344-0338/92/0188-0396$3.50/0

At this time, work is progressing to augment the capabilities of the interpretive expert system. The principle of knowledge representation as structured objects and of the processing strategy with its top-down model-based reasoning, and bottom-up local validation processes have been described earlier3,5, 12. It is the objective of this paper to describe procedures followed in the construction of the knowledge file, specifically, procedures used in knowledge elicitation and uncertainty management.

Material and Methods The clinical materials used in this study were histopathologic sections of prostate, colon and thyroid, and FNA cytologic preparations of thyroid. All were Feulgen stained. Preparation, scanning and digitizing have been described earlier2. Also described earlier were the design rationale of each expert system module, the programming approach and knowledge representation2, 5, II, 13. The uncertainty management methodology described here is based on Pearl's formulation of Bayesian belief networks 10. The implementation followed the programming guidelines given by Morawski 8,9. His PASCAL program was rewritten in "c" in an i"nteractive windowing environment. An increased functionality permits cycles in the Bayesian network. In C or PASCAL it is easy to represent a network with nodes and links and to access each data element of a node by name. This made it convenient to incorporate the belief network into the structured objects' © 1992 by Gustav Fischer Verlag, Stuttgart

Knowledge File for an Image Understanding System· 397 knowledge representation used in the image understanding system. Stored at each node of the belief network are a prior probability vector, a belief vector and a relative likelihood vector. Also stored, at all descendent nodes, is a conditional probability matrix (CP matrix) relating the different outcomes in the descendent node to the belief vector of the parent node. In addition, pointers to parent and descendent nodes are stored. Updating of the belief vectors at each node requires that the parent node, as well as descendent nodes, be revised; however, care must be taken that updating the parent node will not lead to a revision of belief at the node under consideration, since this would initiate an endless loop. Thus, all updates in the belief network traverse the network only once, proceeding along the links and never affecting the node from which updating started. Also, the algorithm must consider that the root node has no parent node. The pointer scheme facilitating access to each node in the belief network is structured according to a common scheme for representing tree graphs in memory. Each node has a pointer to its parent node and a pointer to the first of its descendent nodes. The pointer to the next descendent node is stored at the first descendent node.

Results

1. Knowledge Elicitation The interpretation of diagnostic imagery is based on knowledge of the histopathologic domain, observation of the image information, and application of procedures. Knowledge elicitation is aimed at establishing the necessary set of concepts and their mutual dependencies which an automated reasoning process would require to interpret images from a given domain. The knowledge is elicited from experts in different disciplines: diagnostic pathologists, image analysts, statisticians and computer scientists. The image understanding system mentioned above is in operation now. Its three expert system modules communicate at a basic level. It was found that extension to another histopathologic domain, e.g., to the interpretation of FNA cytologic preparations of the thyroid, becomes more efficient if the process of knowledge elicitation is

:1··) •

Fig. 1. a: Binary image of FNA cytologic preparation of thyroid. - b: Iconic representation of the scene showing a follicle and free-lying nuclei. - c-f: Iconic representations used in the iteration of a complete description. Minimum number of nuclei required to constitute a follicle; criteria for accepting an object as nucleus, size and shape constraints; nuclear center and nuclear center-to-center connect, and distance constr;,tint; closed contour formed by nuclear eenter-to-eenter connects.

systematically taken through a number of stages, as described below. The first stage is aimed at defining all diagnostic clues and their histometric transforms. A histometric transform is a set of histometric variables that the image understanding system can compute and that correspond to and represent a given visual diagnostic clue. This leads to a representation invariant to shift, rotation and even magnification, and a definition of both the topographic and topologic properties of the diagnostic pattern 1. Gray level images of representative fields are clipped to binary form. After inspection, a schematic, iconic representation is constructed, as shown for a thyroid follicle in Figures la and b. To elicit all defining entities, a list is established. The iconic representation reveals two entities: objects and background. The objects fall into two categories: follicles and free lying nuclei. What distinguishes a follicular nucleus from a free lying nucleus is the constraint that follicular nuclei have neighbors. To define a neighboring nucleus, a distance function needs to be defined, e.g., the distance from one nuclear center to the next. A constraint has to be added for a maximum distance, and the number of neighbors. One also has to define how "nuclear center" is to be established. This, with the distance constraint, allows one to define a "nuclear center-to-center connect". A follicle is characterized by a set of such nuclear center-to-center connects that return to the origin, i.e., form a closed contour. Thus, a function is needed that verifies whether a set of center-to-center connects forms a closed contour. Next, the need for further constraints is explored by driving each entity to extreme values: is a follicle acceptable if it is formed by only three follicular nuclei? Four is specified as the required minimum. How small an area may be accepted as a follicular interior? If an object is very small it should not be accepted as a single nucleus. If it is too large, and does not satisfy certain shape constraints it should be classified as a "segmentation problem", to be processed further. Figures 1 c-f render iconic representations of these considerations. Also, at this stage of the knowledge elicitation the needed topologic functions are defined. A diagnostician may use the terms "next to" or "inside of". To define a

a

• • ••••• •• •• •• •••• • ••

r

. '.

b

Q

"'....

e

....

IJ}

....

c

0

, I

" d

398 . P. H. Bartels, D. Thompson and

J.

E. Weber

Fig. 2. Entities required by the topological "next to" function.

term such as "next to", one requires a subject entity and a measure of its location. Further, one requires a reference entity to which the function applies, and its location. One requires a distance function, and possibly a reference direction. There may be a need for a minimization constraint. Figure 2 (from an image of a section stained with Weigerts hematoxylin/van Gieson picrofuchsin) shows a glandular nucleus "next to" the basement membrane. The subject entity is the glandular nucleus; its nuclear center coordinates supply the location. The reference entity is the basement membrane; its location is described by a set of coordinates. The orthogonal projection of the nuclear center to the basement membrane provides the minimization constraint needed to define the location of this entity. It is also necessary to establish that no other nuclear center falls into the region between the center of the subject nucleus and the basement membrane. The second stage of the knowledge elicitation establishes the relationships between the entities defined in the first stage. The iconic representation is replaced by a graph representation, as shown in Fig. 3. Fal ! i c I e ' _ f - - - - - - - - - , nuclei Foil icular

Nuclei

---~-------------

inter ior

,

in

foil icuJar pattern ........ - - - - - - - - - ' -

Set at neighboring nuclei

t

Single nuclei wi th constraints======n

nuclear center to center distance .........



nuclear center to center connect nuclear center

Set of nuclear center to center connects ___ with return to starting center

Closed contour

. t

Follicular pattern

Fig. 3. Relations graph.

The graph may contain loops and need not necessarily be a strictly directional graph. The construction of the relational graph allows one to check the knowledge representation for logical adequacy. In the example given here it is difficult to establish a relations structure without introducing the concept of "follicular pattern". So, this is added to the list of entities in the model. The graph representation undergoes three cycles of iteration. First, completeness and correctness of the model with respect to its histologic components and concepts and to their dependencies is sought. Then, the required functions are checked and entered, such as "find nuclear center. fct". Finally, constraints are entered. The entities identified so far primarily concerned histologic knowledge and the image processing knowledge. that is required to relate histologic entities. In order to attam an automated interpretation of the images a rather wide range of knowledge from other domains is required, such as from image processing, descriptive statistics, management of data structures, and uncertainty management. It is necessary to establish complete lists of entities from all domains of ancillary knowledge required to process the imagery and reason about the histopathologic domain. Examples for entities from the image processing domain are: image, grey scale image, binary image, objects, background; filtering algorithms, thresholding algorithms, chain coding algorithms. All of these are needed to condition and segment the image. Then, there are entities from the image analysis domain: object categories, segmentation problems, distance functions, numerical constraints on number of objects, size of objects, shape factors, directions, locations. There are the entities defining the structure of the histologic scene and the interpretive histometric transforms, as mentioned above; these form the basis for the model-based reasoning. In addition one may have to enter histopathologic knowledge, e.g., likelihoods to observe certain changes in karyometric features, which require information on their statistical distributions. The prognostic interpretation of the scene demands an additional set of entities, some of which allow selected retrieval of data from external data bases. Closely related to the latter capability are entries into the knowledge file that allow the system to establish a context for its image interpretation. For example, in systems where an automated grading of a lesion is the goal, low grade lesions would be represented by a different model than high grade lesions. To decide which model should be used for the scene segmentation and analysis, i.e., which context applies, the system has to rely on procedures that work reliably on the imagery before segmentation. Such procedures may involve measures of the texture of binary imagery as approximations to the nuclear placement pattern. The knowledge file must specify all entities needed for such a preliminary model choice. Finally, the knowledge file must list all entItIes and functions required in uncertainty management, as described below. The point to be made is that every single entity directly encountered in the interpretive process and every implied

Knowledge File for an Image Understanding System . 399

entity that might be used by any of the human experts contributing to the processing, analysis and interpretation of the scene must be made available to the automated reasoning process of an image understanding system. 2. Construction of the Knowledge File The knowledge file is entered in the form of a spread sheet, as described earlier2, and is shown in Tables 1 and 2. The expert system's interpreter (not to be confused here with the interpretive expert system) reads in the knowledge file and assembles the entities into an inference network. An example for the entity declarations covering the different areas of expertise is given in Table 1. The second section of the knowledge file to be entered are the definition statements, following a strict grammar and using as extensive a collection of key words as the designer of the system feels is necessary. The key words are used later, by the expert system's interpreter, to set up pointers berween entities. Table 2 provides examples for declarative definition statements. An example for the inference nerwork is shown in Fig. 4. It contains different kinds of nodes: entity nodes, definition nodes and constraint nodes. The inference nerwork is structured according to pointers that the interpreter sets up when it reads in the definition statements. A node has an internal organization as shown in Table 3. Once the knowledge file is read in and the inference nerwork is established, 'the system is ready to process

scenes from the domain. This involves forward chaining through the nerwork. At each node the model specifications are compared to the corresponding information extracted from the image, and pointers to the next node are followed. Table 1. Entity declarations in the knowledge file Knowledge file Entity declarations Image, binaty image, object, background, find object. fct, threshold.fct, closure. fet, erosion. fct, skeletonization. fct, gradient search. fet, ... pixel group, object group, boundary, segmentation problem, chaincode, threshold, shape factor, cusp, compute area. fct, compute total O.D . .fct, compute shape factor. fct, ... follicle, follicular pattern, follicular nucleus, single nucleus, follicular interior, neighboring nucleus, nuclear count, nuclear area, nuclear center, nuclear center to center distance, nuclear center connect, closed contour, find nuclear center. fct, compute center to center distance. fct, establish contour closure. fct, ...

Table 2. Definition statements in the knowledge file Knowledge file Definition statements Follicle IS A SET OF follicular nuclei Follicular nuclei FORM a follicular pattern Follicular pattern IS FORMED BY SET OF neighboring nuclei Neighboring nuclei ARE nuclei WITH (constraint) Nuclear center REQUIRES compute nuclear center. fct Nuclear center to center distance REQUIRES compute nuclear center distance. fct Closed contour IS A SET OF nuclear center connects WITH (constraints) Artifact IS A object WITH (constraints)

Table 3. Internal organization of information at a node in the inference network

Fig. 4. Inference network.

Entity node name of entity pointer to definition node pointer to belief function node prior probability vector belief vector relative likelihood ratio vector CP matrix pointer to parent node pointer to first descendent node feature list instances ID location feature values

400 . P. H. Bartels, D. Thompson and

J.

E. Weber

3. Uncertainty Management While comparing findings in a given scene with model it is necessary to provide for handling uncertamty: In expert systems, in general, this is achieved at several dIfferent levels and by different formalisms. At the simplest level, there is the "granularity" of descriptive values, e.g., for the description of nuclear area "small" "medium" or "large" with three membership functions: This formalism also is used in the image understanding system. For the r:nanagement ?f accumulating evidence during the r~asonm~ process, It was decided to implement a BayesIa.n. ~elIef network for the following reasons. A p:obabIlIstI~ met.h?d has a solid foundation in diagnostic hlstometry smce It IS always possible to obtain estimates of prior probabilities for events from frequency counts. Also, the dep~~dence struct~~e. of events and entities expressed by condItIonal probabIlItIes can be determined experimentally on samples of adequate magnitude. Belief networks as for~ulated by Pearl keep the computational burden feaSIble. They may readily be embedded in the inference network p~ovided by the structured objects knowledge represent~tIOn. There are only minor incompatibilities. Bay~s belIef networks are directed acyclic graphs. Loops in the mfe~ence network, therefore, have to be avoided in the unc~rtamty .assessment. After some experimentation, we decIded to Implement a number of local, "mini" belief networks at different points in both the interpretive and the scene segmentation expert systems. In the in~erence network, as shown in Fig. 4, an entity node c~mst~;utes an asserti?n, e.g., "this object represents an artIf~ct . In the practICal example of FNA thyroid preparatIOns a node A representing the entity "object category" may have three different values: artifact, object group, or single nucleus. For the occurrence of each the prior probabilities are known for scenes from' the domain. A belief network node incorporated into the inference network's entity node stores the prior probability vector for these three outcomes Jt (a). It also stores a pointer to the next node in the belief network. The prior probabilities Jt (a) are given, and remain constant. The effect of observed evidence on the belief that one or the other of the possible outcomes fo: the value of the entity "object category" is ~rue, as prov~ded, e.g., by ~ .descendent node "object size", IS expressed m two quantities. First, there is a conditional pro~ability matrix Mbla relating every outcome in node A (o~Ject category),.to each outcome in node B (object size). ThIS CP matnx IS stored at node B, for the link A-B. Second, the ac~al observation of the object size by the scene segmentation expert system provides a numerical value. This results in the relative likelihood vector A(b) that the observed value could have been a sample from th~ object size outcome small, moderate or large. Table 4 shows the information stored at nodes A and B. To illustrate the process of updating beliefs in the network, as experimental results are obtained consider the instant when the inference network has ~ctivated the function "object size":This results in a numerical value for a given object. Since the distribution functions for the three specific~tions,

outcomes "small", "moderate" and "large" are known, so are the relative likelihood ratios for membership of the observed value. They are entered at node B as vector A (b).

"

Th.e. relative lik~l~hood vector is multiplied by the condItIonal probabIlIty matrix Mbla, and normalized to update the likelihood vector for object categories, A'(a). The pr??uct of this likelihood vector with the prior probabIlIty vector Jt (a), normalized, results in an update of the values of the belief vector BEL (a) that is stored in node

A.

If additional evidence were provided from another descendent of node A, say node C, then that evidence would further update BEL (A). The process of updating the belief in an assertion at a given node X, in response to observed evidence entered at a desce.nden~ node Y is shown schematically in Fig. 5. EVIdential support for a given belief thus propagates upwards, from descendent nodes to parent nodes as described, and as this occurs the belief vectors at all traversed nodes are updated. . Updating beliefs in a Bayesian network, though, mvolves propagation not only upwards, from descendent nodes to parent nodes, but downwards as well, from parent nodes to descendent nodes. This propagation also is ref~rred to as t~at. o.f. "causal evidence". As a Bayesian belIef network IS IllltIally set up, the prior probability vector Jt (x) of the root node, multiplied by the conditional probability matrix MYlx of the link to the next descendent node Y yiel?s the prior probability vector Jt (y). This process contmues throughout the network at initial setup. The belief BEL (Y) at node Y is obtained by the product BEl (Y) = aJt (y) A(y), where), (y) is the relative likelihood vector, i!litially se~ to unity, and later obtained by the obser:ratIOn of .evIden~e. This means that before any expenmental eVIdence IS entered, all belief vectors and all prior probability vectors are derived from the root node and the conditional probability matrices at each link. It has been mentioned that updating of a belief BEL (X) in light of evidence observed at a descendent node Y should have no effect on the belief BEL (Y) at node Y. However,

4. Prior IJrobability vectors and conditional probability matrIX of the lmk between two nodes in the Bayesian belief network

Tabl~

node B

node A object size in scene

conditional probability matrix

object category

P (a)

Mbla

P (b)

small moderate large

lJ

[080 001 019] 0.20 0.20 0.60 0.20 0.70 0.10

[:J

artifact object group single nucleus

-

Knowledge File for an Image Understanding System· 401

causal suppor t

1r( x)

BEL (xl

xl

xl

x2

x2

x3

x3

A( x)

1r(y)

Myl x

= aJt (z)

Y2

Y3

Y3

Y3

support

...... _ ,\ ( y)

for other descendent nodes, say Z, Jt (z) has to be updated

BEL(Z)

Y2

Y2

Fig. 5. Propagation of evidential support A. message; and of causal support, Jt message.

= Jt (z)

.Ny)

Yl

evidential

BEL (X) Mzl x

t3EL(y)

node the entity "threshold", and the associated belief network node specifies the three outcomes "low", "correct" and "high". There are three independent descendent nodes "pixels" (for number of pixels above threshold),

A(z)

The simplest case is shown in Fig. 6. Figure 7 shows this two-directional propagation process by which the effect of every observation is considered throughout the belief network. White arrows represent the messages due to new evidence, spreading upwards in the network to the root node. Black arrows represent the effect on prior probabilities, the Jt vectors. This spreads peripherally to the descending nodes, with the exception of the branch that generated the relative likelihood vector that started the update cycle. In the image understanding system one such mini Bayesian belief network is used to determine automatically the best segmentation threshold. The network has as a root

root node

node Y new data A(yl

Fig. 6. Update of prior probability and of belief at two descendent nodes.

402 . P. H. Bartels, D. Thompson and

J. E. Weber "number of objects", and "object size". For each of these, experimental evidence is provided by the "Find object. fct". This function is an entity in the inference network. Figure 8 a shows the network in its initial state, with prior probabilities and CP matrices estimated from file data. Figure 8b shows the network update after the pixel entity was evaluated experimentally. Figure 8c shows the update resulting from considering the count of obtained objects for a given threshold. Figure 8d shows the state of the belief network after the relative likelihood vector for the object size node has been entered. The final belief indicates that the tested threshold is in the correct range, based on evidence from all three descendent nodes. The range of each membership function was previously established by visual observation of the effect of different thresholds on the three entities. Other mini-Bayesian belief networks are in the testing stage. There is a network for determining context in the automated grading of prostate lesions as well-differentiated, moderately differentiated or poorly differentiated. The outcome of this assessment determines which model will be followed in the image interpretation. There is a network for determining the best parameters for defining a lumen in glandular tissues; and for accumulating evidence during the model-based reasoning in scene interpretation. Discussion

Fig. 7. Updating of beliefs in a Bayesian network. White arrows indicate upward propagation of A messages. Black arrows indicate updates of prior probabilities and beliefs by it messages.

There are several aspects to the development of knowledge based systems for the automated diagnostic interpretation of histopathologic sections. There is the challenge of man-machine communication, and of the transfer of the complex visual capability of human experts to a machine vision system. The basic structure of a design for such a system has been described earlier with its reliance on several interacting expert system modules. The results reported in this article detail how two important aspects of the practical implementation are handled, the elicitation of knowledge and the management of uncertainty. Solutions to both of these problems became necessary as the development of the image understanding system evolved from the basic interacting of the expert system modules to application to clinical data. In the knowledge elicitation, specification of procedural stages provides necessary guidance and checks for adequacy and completeness. Responsibility for defining the problem remained with the histopathologists and team members with image analysis and computer science background. However, establishing definitive procedural steps is of help. The consideration of uncertainties in the automated reasoning process is an absolutely necessary capability. For our domain, and the chosen implementation in the form of a small number of local belief networks, the conceptualization by Pearl has proven to be very workable. Two observations may be made. The belief network is slow to respond to new evidence when prior probabilities are in an extreme range. If relative likelihood ratios are set very high

Knowledge File for an Image Understanding System . 403 Threshold

Pi[low] = 0.39 Pl[qood] = 0.38 Pi[hiqh] = 0.33 lambda[low] = 1.00 lambda[qood] = 1.00 lalllbda[high] = 1.00 Belief[low] = 0.35 Belief[good] 0.35 Belief [high] = 0.30

__-----------1-----------_

Pixels

Pi[%_classified>35] = 0.38 Pi[25

Construction of the knowledge file for an image understanding system.

To enable an image understanding system to provide an automated interpretation of diagnostic imagery it must have access to all of the concepts, proce...
792KB Sizes 0 Downloads 0 Views