The Clinical Utility of META: An Analysis for Hypertension James R. Campbell University of Nebraska, Omaha, NE
Gene A. Kallenberg Robert C. Sherrick George Washington University, Washington, DC Although designed as a resource for access to infonnation, a stated purpose of the development is to provide knowledge access for clinical care. Since the questions posed by the clinician should be outlined in the medical record, it is reasonable to claim that UMLS must be sufficient to index the content of that document in order to plunb the questions that reside therein. The additional benefits of an accepted standard for medical data recording in the computerized medical record cannot be understated. Despite these aspirations, and recent suggestions that the NLM is moving to make META more clinically robust,[21 little has been written describing the ability of the META resource to completely represent the content of the medical record. We undertook to examine whether META possessed sufficient conceptual detail to codify the process of medical care in the most common adult ambulatory problem -- hypertension.
ABSTRACT To evaluate the clinical compkteness of the NationalLibrary ofMedicine Metathesaurus(META), we coded the conceptual informationfound in 2000problem orented (SOAP) notes for hypertension from one COSTAR site. To minimize the effects of practce idiosyncracy, we analyzed an additonal 500 notes from a second, geographicaly remote site. Concepts occuming at either site numbered 1337. We classified concepts occumng at both sites as core concepts and these numbered 121. We attempted to find a matching concept of the proper semantic type in META for each of the items. AU matching was done by program with a manual review by a physician. The overaU success rate for matching was: Core AU Concepts Concepts 76% SUBJECTIVE 68% 38% 20% OBJECTIVE 72% ASSESSMENT 75% 80% 64% PLAN We observed the greatest frequency of unmatched concepts in physical examination, medications, symptoms, personal behavior, non-medical therapies and
METHIODS Medical records were reviewed and collated from two clinical sites. The Internal Medicine clinic at the University of Nebraska is a resident training clinic serving an ambulatory population with an average age of 56 and a predominance of Medicaid and Medicare third party coverage. Most patients at this site are receiving ongoing care for multiple medical problems. COSTAR medical records have been serving ambulatory care delivery at this site since 1983. The George Washington University Health Plan, Department of Health Care Sciences provides care for the University owned Health Plan, a 50,000 member health maintenance organization. The patient population at this site has an average age of 45, eleven years younger than the Nebraska patients. The practice serves as a training site for both medical students and primary care residents. COSTAR is used as the medical record for one of the practice teams consisting of 4 physicians, 2 physician assistants and a resident.
counseling. We conclude that the current release of META is not sufficiently rch to describe the process of care in the ambulatory management ofhypertension. However, the construction and breadth of the current scheme holds promise for medical knowledge representaton and translation.
INTRODUCTION The National Library of Medicine(NLM) has undertaken a monumental project in order to develop a unifonn medical language system (UMLS) as a "gateway" to the medical literature and medical information in generaI4l] This consists of the metathesaurus (META) which is an expanded conceptual and semantic representation scheme, a semantic network which links the concepts functionally and hierarchically, and an information sources map which identifies computer resources available to the medical professional.
0195-4210/92/$5.00O©1993 AMIA, Inc.
397
This mapping was then used to define a valid match from the hypertensive notes into the NLM metathesaurus. If a concept coded from the hypertensive note was found to have a matching concept in META, and that META entry was of an allowable santic type, then a "match" was declared. If the concept could not be located, or if the semantic type of the META concept was not in the mapping, then the concept was declared to have no match in META (even though an attractive concept might exist as an alternative semantic type). We defined the union of all the semantic types listed in Table 1 as META "clinical" types. A MUMPS alphabetic cross-reference of all clinical META codes was built for review of the hypertension notes. A manually reviewed, machine assisted matching of all clinical concepts from the original 2000 notes was then done by the primary author. In order to minimize the chances of missing important concepts due to regional variation, 500 hypertension notes were then identified from the Washington, DC site. These notes were textual, problem oriented docunents detailing encounters only for hypertension. These were manually reviewed by the authors for concepts consistent with the granularity of the Nebraska coding scheme. Tallies of code frequency were not kept because of differences in the volumes of notes reviewed between the sites, and in patient mix. Concepts identified by the manual review at the Washington site were added to the master list of concepts and a final matching against META concepts was done using the procedure outlined above.
As a part of a controlled trial of computerized records undertaken at the Nebraska site, all problem notes for a one year period were encoded from transribed source documents. This coding was done conceptually with a manual review of all records by the primary author and a spially trained coder. Synonyms of concepts that we identified, as wel as lexical variants, were ignored for this analysis. AU coding maintained a strict problem oriented "SOAP' hierarchy so that the coded element was crossrefemeed by problem and the note "segment" (for example - SUBJECTIVE). For this project, 2000 records coded for the problem of hypertension were taken from the Nebraska site. These were selected only for completeness, and were included in the evaluation if they contained at least 3 coded elements. A compilation of the primary concepts, expanded to include drug trade names, was then matched by program against concepts in the metathesaurus provided by NLM. Each main concept in META can have multiple senantic types. The senantic types defined by NLM denote allowable "contexts" for each concept. From our initial results, the santic types of matches were correlated with the SOAP segment and compared against the definitions of semantic types provided by NLM. In some cases, confusing areas were discussed with NLM staff and a final "mapping" of an idealized SOAP note into the NLM semantic types was done. This is summarized in Table 1.
Trabie 1 398
Following machine and manual matching into META, the codes from all sites without matches were summarized by the authors into generic classes of information. Table 3 is a summary of those areas without match. For each segment of the problem note, any conceptual group judged by the authors to have 5 or more failed matches is listed along with the total nunber of concepts outstanding for that class. For example, 42 patient symptoms could not be found in META. Finally, Table 4 summarizes the 121 core concepts found independently at both sites. For each segment of the SOAP note, those terms matching and failing to match META are listed for review.
Concepts appearing at either site will be called hypertension concepts. Concepts appearing in clinic notes at both sites were declared to be core concepts. We judged that core concepts were more representative of management variables for hypertension as an ambulatory problem, although conceptual outliers in patients with many clinical The frequency of problems were inevitable. appearance of hypertension concepts in META was calculated and patterns of "misses" and "hits" were reviewed by the authors. Similar tallies were done for core concepts.
RESULTS From the Nebraska sample of 2000 coded clinical concepts were identified. These 1186 notes, are broken down by SOAP segment in Table 2, showing a slight predominance of concepts in "Subjective" and "Plan" segments. This may be weighted somewhat by virtue of a decision to include drug trade names as concepts where they occurred in the medical record. In parentheses following each tabulation are listed the frequency with which matching concepts were found in META. Column 2 of Table 2 summarizs the clinical concepts found on review of notes at the Washington, DC site. The smaller number of concepts relates to the smaller number of notes available, differences in recording habits and the heavy orientation of this site towards primary care. The third column summarizes total numbers of unique concepts and the cumulative frequency of matching into the Metathesaurus. The final column tallies number of concepts and matching frequency for core concepts found on review at both
DISCUSSION The development and evolution of the NLM metathesaurus is a matter of strong interest to those working in medical informatics. Conventional coding schemes such as ICD-9-CM or CPI-V have the strengths of standardization and acceptance but are not clinically comprehensive. Schemes that were developed for billing patient care generally reflect a granularity and organization that supports the billing function but rarely are useful to describe the process of medical care delivery, much less serve as a medical record. Although META and the related resources published by NLM are designed for access to the medical literature, pressure for clinical utility and relevance of this representational scheme is building. Problems of transportability of medical data, nonuniformity of representation and confusion regarding medical meaning would be greatly ameliorated if META developed into a coding resource which developers could employ with confidence.
sites.
lanle z 399
Table 3 outlines several other areas that could benefit from concept additions. Personal symptoms, concepts attributed to nonnal behavior and function, some diagnoses, patient education and counselling, and non-medical therapies are the learest examples. From the foregoing results, it should be dear that META in its current release is not adequate to conceptually represent the process of ambulatory hypertensive management. Despite this conclusion, META has many stts including a rich and growing scheme of symptoms, diagnoses and treatments. The organization of META, and the relationship of the semantic types to the process of clinical care can be confusing at first. The mapping of META types into a SOAP note, summarized in Figure 2, is an example of that complexity. Clearly the area of greatest variety, and therefore confusion, exists in the "subjective" area where the types of information entering the record may be very diverse. However, the authors found that this was a minor point of confusion. We believe that this approach should ultimately prove useful for textual analysis where the organization of the semantic net will aid the dissection of content of structured medical documents. Finally, this paper makes no attempt to deal with the issues inherent in the structure and utility of the semantic network. Clearly, the accuracy and uniformity of the network relationships will provide a large benefit to the clinical developer who is trying to develop knowledge based programs which effectively deal with incomplete clinical data and uncertainty.
This paper represents an initial attempt to examine the conceptual content of META for a common ambulatory medical problem. Ignoring completely lexical issues involved in interpreting natural language, we only attempted to determine whether basic conceptual representation exsts in META for this arena of ambulatory medical process. Clearly, our comments cannot reflect on the ability of META to serve as a resource for text parsing or other textual analysis since we did not analyze for frequency of synonyms or lexical variants in this project. Nonetheless, we found similar patterns of discrepancy when matching META against core concepts and the larger set of hypertension concepts. This suggests that we can make useful observations regarding the dinical utility of the current release of META. The area of greatest clinical concen from our analysis is clearly the objective portion of the note specifically the physical examination. Better than 30% of all concepts not existing in META were physical tion. Even findings or portions of the physical ex though physical examination concepts were only 20% of the total hypeensive concepts reviewed, this portion of META is woefully incomplete. The second most frequent pattern of matching failure was attributed to medicinals - both trade and generic drugs. Although META does have many drug names and most medication classes, the sheer volume of clinical pharmaceuticals used in hypertension and related diseases points to a second area which requires additional work if META is to serve as a dinical coding scheme.
lTaDLe 3 400
........
....................... ......
._.
.........
.. .......
...
..........
co
......
........_...................
ww-.....I.........
PM...-.,
.............
..........
.......... .....................
__ ....... .............
..........._.
co
-.R
-.'-
.M
pance-tono..
...*...
..
:.1..!.F.:..:.'6.!. .:.
OYS
........
.......
...
............ tum
p
..
S-Vncirom gqW.`:'M
...
...
.................
...
:::.a
... .............
.............
V
.....................
................................. I'L ............
.......
...
..L.........
.....
ifiiaIefilu, .............................................
.pro
....
1115,
.........
...
-d
......... .......... ........
......
....
O ...
... .................
...............................
InMW0
to
......O..... .........................
..
t
Ow...I]1,11v
.............-............
.........._... ._........_.............
consw..p.xo
..
lik t
4)
........
..
.............
....
.....
..
.....................
..........
.....
pressp. :p4pie
i(CWM917=...Mim
...........L...... .....
......
--
......
_.......
............. ......L..............
.............
......
............
....
..
......
......
novenow,
..
`:X.W r" "crossing::::::044es",:.:::::ext".'
......_...............
.......
__
'iti'es "S1... """
.....
..
..
...ra
.j
"'W,. ...
...
......
... ...
..
..
..
... ..
..
..
..
..
p
PO
la ..
..
a........
jL ..
.... ..
....
.. ..
.ttwc-
:::::Mpo I.O.:::. ..
..
-Y0
.. ..
..
W"
clenpye
ig
..
..............
pain
-exercise
9.
i6.d i .M......
An examination of the paucity of the relationships between META physical examination concepts points to an area of our concern relative to the clinical utility of the semantic network. With all of the preceding comments in mind, we observe that the breadth and depth of META in its current release is still a wonderful enrichment over existing coding schemes. Since it tries to handle the full scope of medical care, the current release is already better than many coding resources. If the NLM continues to enhance the clinical areas we have discussed, we are convinced this will have substantial utility for knowledge engineers and medical data base developers. We applaud this initiative by NLM and can only hope as clinician infonnation specialists that their emphasis will continue to develop META in ways more helpful for clinical medicine.
tilIh"
.
be
[1] Humphreys BL, Lindberg DA. Building the Unified Medical Language System. Proceedings of the Thirteenth Annual SCAMC. IEEE Computer Society Press; 1989; 475480. [2] Humphreys BL, Lindberg DA, Hole WT. Assessing and Enhancing the Value of the UMLS Knowledge Sources. Proceedings of the Fifteenth Annual SCAMC. IEEE Computer Society Press, 1991; 78-82.
401
Gene A. Kallenberg Robert C. Sherrick George Washington University, Washington, DC Although designed as a resource for access to infonnation, a stated purpose of the development is to provide knowledge access for clinical care. Since the questions posed by the clinician should be outlined in the medical record, it is reasonable to claim that UMLS must be sufficient to index the content of that document in order to plunb the questions that reside therein. The additional benefits of an accepted standard for medical data recording in the computerized medical record cannot be understated. Despite these aspirations, and recent suggestions that the NLM is moving to make META more clinically robust,[21 little has been written describing the ability of the META resource to completely represent the content of the medical record. We undertook to examine whether META possessed sufficient conceptual detail to codify the process of medical care in the most common adult ambulatory problem -- hypertension.
ABSTRACT To evaluate the clinical compkteness of the NationalLibrary ofMedicine Metathesaurus(META), we coded the conceptual informationfound in 2000problem orented (SOAP) notes for hypertension from one COSTAR site. To minimize the effects of practce idiosyncracy, we analyzed an additonal 500 notes from a second, geographicaly remote site. Concepts occuming at either site numbered 1337. We classified concepts occumng at both sites as core concepts and these numbered 121. We attempted to find a matching concept of the proper semantic type in META for each of the items. AU matching was done by program with a manual review by a physician. The overaU success rate for matching was: Core AU Concepts Concepts 76% SUBJECTIVE 68% 38% 20% OBJECTIVE 72% ASSESSMENT 75% 80% 64% PLAN We observed the greatest frequency of unmatched concepts in physical examination, medications, symptoms, personal behavior, non-medical therapies and
METHIODS Medical records were reviewed and collated from two clinical sites. The Internal Medicine clinic at the University of Nebraska is a resident training clinic serving an ambulatory population with an average age of 56 and a predominance of Medicaid and Medicare third party coverage. Most patients at this site are receiving ongoing care for multiple medical problems. COSTAR medical records have been serving ambulatory care delivery at this site since 1983. The George Washington University Health Plan, Department of Health Care Sciences provides care for the University owned Health Plan, a 50,000 member health maintenance organization. The patient population at this site has an average age of 45, eleven years younger than the Nebraska patients. The practice serves as a training site for both medical students and primary care residents. COSTAR is used as the medical record for one of the practice teams consisting of 4 physicians, 2 physician assistants and a resident.
counseling. We conclude that the current release of META is not sufficiently rch to describe the process of care in the ambulatory management ofhypertension. However, the construction and breadth of the current scheme holds promise for medical knowledge representaton and translation.
INTRODUCTION The National Library of Medicine(NLM) has undertaken a monumental project in order to develop a unifonn medical language system (UMLS) as a "gateway" to the medical literature and medical information in generaI4l] This consists of the metathesaurus (META) which is an expanded conceptual and semantic representation scheme, a semantic network which links the concepts functionally and hierarchically, and an information sources map which identifies computer resources available to the medical professional.
0195-4210/92/$5.00O©1993 AMIA, Inc.
397
This mapping was then used to define a valid match from the hypertensive notes into the NLM metathesaurus. If a concept coded from the hypertensive note was found to have a matching concept in META, and that META entry was of an allowable santic type, then a "match" was declared. If the concept could not be located, or if the semantic type of the META concept was not in the mapping, then the concept was declared to have no match in META (even though an attractive concept might exist as an alternative semantic type). We defined the union of all the semantic types listed in Table 1 as META "clinical" types. A MUMPS alphabetic cross-reference of all clinical META codes was built for review of the hypertension notes. A manually reviewed, machine assisted matching of all clinical concepts from the original 2000 notes was then done by the primary author. In order to minimize the chances of missing important concepts due to regional variation, 500 hypertension notes were then identified from the Washington, DC site. These notes were textual, problem oriented docunents detailing encounters only for hypertension. These were manually reviewed by the authors for concepts consistent with the granularity of the Nebraska coding scheme. Tallies of code frequency were not kept because of differences in the volumes of notes reviewed between the sites, and in patient mix. Concepts identified by the manual review at the Washington site were added to the master list of concepts and a final matching against META concepts was done using the procedure outlined above.
As a part of a controlled trial of computerized records undertaken at the Nebraska site, all problem notes for a one year period were encoded from transribed source documents. This coding was done conceptually with a manual review of all records by the primary author and a spially trained coder. Synonyms of concepts that we identified, as wel as lexical variants, were ignored for this analysis. AU coding maintained a strict problem oriented "SOAP' hierarchy so that the coded element was crossrefemeed by problem and the note "segment" (for example - SUBJECTIVE). For this project, 2000 records coded for the problem of hypertension were taken from the Nebraska site. These were selected only for completeness, and were included in the evaluation if they contained at least 3 coded elements. A compilation of the primary concepts, expanded to include drug trade names, was then matched by program against concepts in the metathesaurus provided by NLM. Each main concept in META can have multiple senantic types. The senantic types defined by NLM denote allowable "contexts" for each concept. From our initial results, the santic types of matches were correlated with the SOAP segment and compared against the definitions of semantic types provided by NLM. In some cases, confusing areas were discussed with NLM staff and a final "mapping" of an idealized SOAP note into the NLM semantic types was done. This is summarized in Table 1.
Trabie 1 398
Following machine and manual matching into META, the codes from all sites without matches were summarized by the authors into generic classes of information. Table 3 is a summary of those areas without match. For each segment of the problem note, any conceptual group judged by the authors to have 5 or more failed matches is listed along with the total nunber of concepts outstanding for that class. For example, 42 patient symptoms could not be found in META. Finally, Table 4 summarizes the 121 core concepts found independently at both sites. For each segment of the SOAP note, those terms matching and failing to match META are listed for review.
Concepts appearing at either site will be called hypertension concepts. Concepts appearing in clinic notes at both sites were declared to be core concepts. We judged that core concepts were more representative of management variables for hypertension as an ambulatory problem, although conceptual outliers in patients with many clinical The frequency of problems were inevitable. appearance of hypertension concepts in META was calculated and patterns of "misses" and "hits" were reviewed by the authors. Similar tallies were done for core concepts.
RESULTS From the Nebraska sample of 2000 coded clinical concepts were identified. These 1186 notes, are broken down by SOAP segment in Table 2, showing a slight predominance of concepts in "Subjective" and "Plan" segments. This may be weighted somewhat by virtue of a decision to include drug trade names as concepts where they occurred in the medical record. In parentheses following each tabulation are listed the frequency with which matching concepts were found in META. Column 2 of Table 2 summarizs the clinical concepts found on review of notes at the Washington, DC site. The smaller number of concepts relates to the smaller number of notes available, differences in recording habits and the heavy orientation of this site towards primary care. The third column summarizes total numbers of unique concepts and the cumulative frequency of matching into the Metathesaurus. The final column tallies number of concepts and matching frequency for core concepts found on review at both
DISCUSSION The development and evolution of the NLM metathesaurus is a matter of strong interest to those working in medical informatics. Conventional coding schemes such as ICD-9-CM or CPI-V have the strengths of standardization and acceptance but are not clinically comprehensive. Schemes that were developed for billing patient care generally reflect a granularity and organization that supports the billing function but rarely are useful to describe the process of medical care delivery, much less serve as a medical record. Although META and the related resources published by NLM are designed for access to the medical literature, pressure for clinical utility and relevance of this representational scheme is building. Problems of transportability of medical data, nonuniformity of representation and confusion regarding medical meaning would be greatly ameliorated if META developed into a coding resource which developers could employ with confidence.
sites.
lanle z 399
Table 3 outlines several other areas that could benefit from concept additions. Personal symptoms, concepts attributed to nonnal behavior and function, some diagnoses, patient education and counselling, and non-medical therapies are the learest examples. From the foregoing results, it should be dear that META in its current release is not adequate to conceptually represent the process of ambulatory hypertensive management. Despite this conclusion, META has many stts including a rich and growing scheme of symptoms, diagnoses and treatments. The organization of META, and the relationship of the semantic types to the process of clinical care can be confusing at first. The mapping of META types into a SOAP note, summarized in Figure 2, is an example of that complexity. Clearly the area of greatest variety, and therefore confusion, exists in the "subjective" area where the types of information entering the record may be very diverse. However, the authors found that this was a minor point of confusion. We believe that this approach should ultimately prove useful for textual analysis where the organization of the semantic net will aid the dissection of content of structured medical documents. Finally, this paper makes no attempt to deal with the issues inherent in the structure and utility of the semantic network. Clearly, the accuracy and uniformity of the network relationships will provide a large benefit to the clinical developer who is trying to develop knowledge based programs which effectively deal with incomplete clinical data and uncertainty.
This paper represents an initial attempt to examine the conceptual content of META for a common ambulatory medical problem. Ignoring completely lexical issues involved in interpreting natural language, we only attempted to determine whether basic conceptual representation exsts in META for this arena of ambulatory medical process. Clearly, our comments cannot reflect on the ability of META to serve as a resource for text parsing or other textual analysis since we did not analyze for frequency of synonyms or lexical variants in this project. Nonetheless, we found similar patterns of discrepancy when matching META against core concepts and the larger set of hypertension concepts. This suggests that we can make useful observations regarding the dinical utility of the current release of META. The area of greatest clinical concen from our analysis is clearly the objective portion of the note specifically the physical examination. Better than 30% of all concepts not existing in META were physical tion. Even findings or portions of the physical ex though physical examination concepts were only 20% of the total hypeensive concepts reviewed, this portion of META is woefully incomplete. The second most frequent pattern of matching failure was attributed to medicinals - both trade and generic drugs. Although META does have many drug names and most medication classes, the sheer volume of clinical pharmaceuticals used in hypertension and related diseases points to a second area which requires additional work if META is to serve as a dinical coding scheme.
lTaDLe 3 400
........
....................... ......
._.
.........
.. .......
...
..........
co
......
........_...................
ww-.....I.........
PM...-.,
.............
..........
.......... .....................
__ ....... .............
..........._.
co
-.R
-.'-
.M
pance-tono..
...*...
..
:.1..!.F.:..:.'6.!. .:.
OYS
........
.......
...
............ tum
p
..
S-Vncirom gqW.`:'M
...
...
.................
...
:::.a
... .............
.............
V
.....................
................................. I'L ............
.......
...
..L.........
.....
ifiiaIefilu, .............................................
.pro
....
1115,
.........
...
-d
......... .......... ........
......
....
O ...
... .................
...............................
InMW0
to
......O..... .........................
..
t
Ow...I]1,11v
.............-............
.........._... ._........_.............
consw..p.xo
..
lik t
4)
........
..
.............
....
.....
..
.....................
..........
.....
pressp. :p4pie
i(CWM917=...Mim
...........L...... .....
......
--
......
_.......
............. ......L..............
.............
......
............
....
..
......
......
novenow,
..
`:X.W r" "crossing::::::044es",:.:::::ext".'
......_...............
.......
__
'iti'es "S1... """
.....
..
..
...ra
.j
"'W,. ...
...
......
... ...
..
..
..
... ..
..
..
..
..
p
PO
la ..
..
a........
jL ..
.... ..
....
.. ..
.ttwc-
:::::Mpo I.O.:::. ..
..
-Y0
.. ..
..
W"
clenpye
ig
..
..............
pain
-exercise
9.
i6.d i .M......
An examination of the paucity of the relationships between META physical examination concepts points to an area of our concern relative to the clinical utility of the semantic network. With all of the preceding comments in mind, we observe that the breadth and depth of META in its current release is still a wonderful enrichment over existing coding schemes. Since it tries to handle the full scope of medical care, the current release is already better than many coding resources. If the NLM continues to enhance the clinical areas we have discussed, we are convinced this will have substantial utility for knowledge engineers and medical data base developers. We applaud this initiative by NLM and can only hope as clinician infonnation specialists that their emphasis will continue to develop META in ways more helpful for clinical medicine.
tilIh"
.
be
[1] Humphreys BL, Lindberg DA. Building the Unified Medical Language System. Proceedings of the Thirteenth Annual SCAMC. IEEE Computer Society Press; 1989; 475480. [2] Humphreys BL, Lindberg DA, Hole WT. Assessing and Enhancing the Value of the UMLS Knowledge Sources. Proceedings of the Fifteenth Annual SCAMC. IEEE Computer Society Press, 1991; 78-82.
401
