Symptom assessment in pediatric oncology: how should concordance between children's and parents' reports be evaluated?

Copyright B 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins

Christina Baggott, PhD, RN, CPON Bruce A. Cooper, PhD Neyssa Marina, MD Katherine K. Matthay, MD Christine Miaskowski, PhD, RN, FAAN

Symptom Assessment in Pediatric Oncology How Should Concordance Between Children’s and Parents’ Reports Be Evaluated?

K E Y

W O R D S

Background: Clinical evaluations in pediatric oncology are often triadic, involving

Cancer

children or adolescents, parents, and clinicians. However, few studies have

Chemotherapy

evaluated the concordance between children’s and parents’ reports of symptom

Children

occurrence. Objectives: The purposes of this study were to evaluate the

Concordance

concordance between children’s and parents’ symptom reports during the week of

Symptom assessment

chemotherapy administration using 5 statistical approaches and determine which factors are associated with higher levels of dyadic concordance. Methods: Independent assessments of symptom occurrence were obtained from children and adolescents with cancer (n = 107) and their parents using the Memorial Symptom Assessment Scale 10-18. Concordance was assessed using (1) percentage of overall agreement, (2) Cohen . coefficients, (3) McNemar tests, (4) positive percentage agreement, and (5) negative percentage agreement. Results: For each dyad, an average of 20 of the 31 symptom reports were concordant. Using children’s reports as the ’’gold standard,‘‘ parents rarely underestimated the children’s symptoms. However, compared with children’s reports, parents overestimated 7 symptoms. Advantages and disadvantages of each of the statistical approaches used to evaluate concordance are described in this article. Conclusions: A variety of statistical approaches are needed to obtain a thorough evaluation of the concordance

Author Affiliations: Departments of Physiological Nursing (Drs Baggott and Miaskowski) and Community Health Systems (Dr Cooper), University of California San Francisco; Department of Pediatrics, Stanford University, Palo Alto, California (Dr Marina); and Department of Pediatrics, University of California San Francisco (Dr Matthay). This research was supported by the National Institute of Nursing Research (NR010600). Dr Baggott is funded by the American Cancer Society on a Mentored Research Scholar Grant (MRSG-12-01-PCSM) and received funding from an American Cancer Society Doctoral Degree Scholarship in Cancer

252 n Cancer NursingTM, Vol. 37, No. 4, 2014

Nursing, the Betty Irene Moore Doctoral Fellowship in Nursing, and an Oncology Nursing Foundation Doctoral Scholarship in Nursing. Dr Miaskowski is funded by the American Cancer Society as a Clinical Research Professor. The authors have no conflicts of interest to disclose. Correspondence: Christina Baggott, PhD, RN, CPON, Department of Physiological Nursing, University of California San Francisco, 2 Koret Way, Box 0610, San Francisco, CA 94143-0610 ([email protected]). Accepted for publication October 2, 2013. DOI: 10.1097/NCC.0000000000000111

Baggott et al

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

between symptom reports. Discordance was most common for symptoms that children refuted, particularly psychosocial symptoms. Implications for Practice: Clinicians need to interview children and adolescents along with their parents about the occurrence of symptoms and evaluate discrepant reports. Effective approaches are needed to improve communication between children and parents to improve symptom assessment and management.

I

n recent years, policymakers and the pharmaceutical industry have promoted the use of patient-reported outcomes (PROs) as an important indicator of the quality of cancer care.1,2 Patients’ self-report of symptom occurrence and severity has long been the ‘‘gold standard’’ in symptom assessment. In pediatric oncology, most children can describe their symptom experiences with parental assistance.3,4 However, when children and adolescents experience multiple symptoms as a result of their cancer and its treatment, it becomes more difficult for them to provide these self-reports. In fact, in a review of 9 studies, children’s participation in their own clinical consultations ranged from 3% to 14.2% among children 6 to 12 years of age.5 One can hypothesize that factors that may decrease children’s ability to self report symptoms include children’s developmental level and understanding of the reporting process, family dynamics and personalities, clinicians’ abilities to engage young patients, and the nature of symptoms themselves (eg, debilitating physical or neuropsychological sequelae of therapy). However, additional research on these issues is warranted. When children are unable to report symptoms, their parents’ assessments are needed to ensure effective symptom management. Full concordance between children’s and parents’ reports of symptoms is not expected, as parents do not experience what their children experience. In addition, parents view their children’s behaviors and interactions with others from the perspective of an outsider. At times, this perspective can provide critical insight. Hinds and colleagues6 noted that the ‘‘child’s voice’’ should not replace information from parents and clinicians. The child’s voice and those of family members can be compiled to create a ‘‘family voice.’’ Therefore, a thorough understanding of the factors that affect the degree of concordance between children’s and their parents’ reports is needed to guide clinicians’ interactions with patients and their family members. Although comparisons between quality of life ratings of children with cancer and their parents were reported in several studies,7Y11 only 2 studies have evaluated concordance between symptom reports of pediatric oncology patients and those of their parents.12,13 As part of the psychometric evaluation of the Memorial Symptom Assessment Scale 10Y18 (MSAS 10Y18), Collins and colleagues12 published a summary of the concordance analyses of symptom occurrence reported by children (n = 160) and their parents. The researchers indicated that . coefficients were strong for 24 of the 30 symptoms evaluated, but they did not define ‘‘strong.’’ In addition, they did not provide concordance data for each of the 31 MSAS symptoms. In a subsequent evaluation of the MSAS for 7- to 12-yearolds, Collins and colleagues13 reported concordance between children’s and parents’ reports of symptom occurrence. In this

Symptom Assessment in Pediatric Oncology

study, . coefficients were provided for the 8 symptoms evaluated. . Coefficients were described as poor for itching (. = 0.11) and worry (. = 0.16); fair for insomnia (. = 0.20), sadness (. = 0.33), and anorexia (. = 0.35); and moderate for lethargy (. = 0.42), nausea (. = 0.46), and pain (. = 0.46). Ninety percent of these children were evaluated in the outpatient setting, but the children’s treatment status was not reported. A wide range of statistical approaches is available to evaluate the degree of concordance between reports. Collins and colleagues12,13 used the . coefficient to compare children’s and parents’ ratings in both of their psychometric studies. In addition, in the second study,13 although they used McNemar tests to evaluate the direction of discordance, these data were not reported. The rationale for the selection of 1 or more statistical approaches is not included routinely in the literature. Careful consideration should be given to the selection of a particular approach to evaluate concordance based on the goals of the study and various factors that can influence an investigator’s conclusions. Given the paucity of research on the concordance between children’s and parents’ assessment of symptoms associated with cancer treatment, we sought to evaluate concordance in ratings of symptom occurrence between children and adolescents receiving myelosuppressive chemotherapy and their parents. Although the commonly used . coefficient provides detail on the degree of concordance between ratings, this statistical test does not elucidate patterns in the discordance. Furthermore, as described in this article, based on the distribution of responses, the . statistic may either overcorrect or undercorrect the data. For these reasons, concordance between children’s and parents’ symptom occurrence ratings was evaluated during the week of chemotherapy administration using varied statistical approaches. In addition, an evaluation was done to determine which demographic and clinical characteristics were associated with higher levels of dyadic concordance. These varied statistical approaches provide different insights into how children and parents evaluate the child’s symptom experience during cancer treatment. The occurrence data in this study are dichotomous ratings. Additional statistical tests are available to analyze concordance, including the Bland-Altman technique14 and the concordance correlation coefficient.15 However, these approaches use continuous data. It should be noted that the Pearson correlation should not be used to measure concordance because it is a measure of association.14,16 For example, consider the following hypothetical set of scores on a patient-reported outcome measure: 6 scores from children of 20, 30, 19, 20, 32, and 20 and 6 scores from parents of 100, 150, 100, 110, 150, and 120. Although the Pearson correlation coefficient is both high and statistically Cancer NursingTM, Vol. 37, No. 4, 2014


n

253

significant (r = 0.95, P = .004), the parents’ scores differ greatly from those of the children.

Conceptual Framework The Symptom Management Theory, developed by the faculty at the University of California, San Francisco (UCSF), served as the conceptual framework for this study. The key constructs of the theory are the symptom experience, symptom management strategies, and symptom status outcomes. The symptom experience dimension is central to the evaluation of concordance because it focuses on patients’ perceptions and evaluations of specific symptoms. The underlying factors that may affect these constructs are categorized into 3 domains (ie, person, health and illness, environment).17 Factors that may affect concordance, such as demographic characteristics (eg, age, gender, educational level) and patients’ clinical characteristics (eg, diagnosis, treatment status, time since diagnosis), can be evaluated in a straightforward fashion. Other factors, such as social and cultural issues, may impact symptom reporting but require indepth analysis. The theory purports that the patients’ report should be considered the gold standard. The Symptom Management Theory notes that if a patient is not able to report symptoms, a caregiver’s proxy report should be used.18 However, as currently proposed, the theory does not provide guidance on the common scenario of overlapping but incongruent reports obtained jointly from children and their parents. Clearly, when rating symptoms, children and their parents have different vantage points. Children experience and evaluate their symptoms, whereas parents rate their children’s symptoms based on their observations. Nevertheless, children’s and parents’ ratings of symptoms can be compared to delineate patterns of agreement and discordance.

n

Methods

Participants In this descriptive study, which is part of a larger study of symptom clusters in pediatric oncology patients,19 self-report questionnaires were administered to a convenience sample of children and adolescents with cancer (10Y18 years of age, hereafter referred to as children) as well as a parent/guardian (6 participants were legal custodial guardians; this group is hereafter referred to as parents) who were able to understand English or Spanish and gave consent/assent to participate. The children were receiving chemotherapy either as their initial therapy or for relapsed or refractory disease, had received chemotherapy within the preceding 4 weeks, and were scheduled for additional myelosuppressive chemotherapy on the day of enrollment into the study.

Instruments The revised version of the MSAS 10Y18 was used to assess 31 symptoms on 3 dimensions. If participants reported the 254 n Cancer NursingTM, Vol. 37, No. 4, 2014

occurrence of a symptom over the previous week, they rated the symptom’s frequency during the past week (ranges from 1, almost never, to 4, almost always), severity (ranges from 1, slight, to 4, very severe), and distress (ranges from 0, not at all, to 4, very much) using Likert scales. Positive responses for the occurrence of each symptom were summed to determine the total number of symptoms reported by participants. In this study, children and parents took approximately 10 minutes to complete the MSAS. The MSAS has established validity and reliability in pediatric oncology patients12 and adults.20 In this sample, the internal consistency ratings for the MSAS 10Y18 for the children’s data were 0.86, 0.79, and 0.75 for the total scale and the physical and psychological subscales, respectively. The comparable internal consistency ratings for the parents’ data were 0.88, 0.83, and 0.79. The Karnofsky Performance Status (KPS) scale was used to assess children’s functional status. Scores on the KPS scale can range from 0 (dead) to 100 (normal function) in 10-point increments.21 The KPS scale has well-established validity and reliability in adults22,23 and has been used in pediatric studies as an indication of patients’ clinical status.24Y27 All instruments were translated from English to Spanish using forward and backward translation procedures.28

Study Procedures For this study, myelosuppressive chemotherapy was defined as treatment that was expected to cause a significant drop in the absolute neutrophil count to less than 500 cells/6L, with subsequent blood count recovery expected to occur within 3 to 4 weeks. Among common cancer diagnoses, a standard chemotherapy regimen was targeted (eg, enrolled patients with acute lymphoblastic leukemia during delayed intensification and patients with osteosarcoma during cisplatin treatment). Children were excluded if they were receiving concurrent radiation therapy. Children were recruited from 3 pediatric oncology settings in the San Francisco Bay area. A total of 144 child-parent dyads were approached to participate, and 131 provided consent or assent (response rate of 91%). The primary reason for refusal was that children were not interested in completing questionnaires. The study was approved by the Human Subjects Committee at UCSF and at each of the study sites. Children and parents who were 18 years or older signed written, informed consent. Children aged 10 to 17 years gave either written or verbal assent as per each institution’s guidelines. During the enrollment visit, children and 1 parent independently completed the MSAS 10Y18.12 Children and parents completed this symptom survey weekly for 3 weeks. Although parents and children were not always in separate rooms to complete the MSAS, they were instructed to complete the forms independently and not to share responses. In addition, children completed a modified version of the KPS scale21 in layman’s terms weekly and a form that captured other data not uniformly included in the medical record (eg, number of hospitalizations, fever). Parents completed a demographic form at enrollment.

Baggott et al


For this article, analyses of concordance between children’s and parents’ ratings of symptom occurrence were done at the second assessment, when participants were asked to respond to each question based on their experiences during the week of chemotherapy administration. Complete data were available from 107 child-parent dyads. Participants received support from the research assistants to complete the study questionnaires as needed. Research staff reviewed children’s medical records for disease and treatment information. Children received a gift card to compensate them for their time.

Data Analysis Descriptive statistics were generated on the sample characteristics. Descriptions of commonly used statistical approaches to evaluate concordance are listed in Table 1. In this study, multiple statistical approaches were used to evaluate concordance, as each approach has advantages and disadvantages. Although detailed ratings of the frequency, severity, and distress for each of the 31 symptoms were available in Likert format, the analyses for this article were limited to the ratings of symptom occurrence.

Table 1 & Description of Statistical Analysis of Concordance Statistic

Level of Measurement

Cohen . coefficient29,30

Categorical: nominal or ordinal

Positive and negative percentage agreement (PPA and NPA)32

Categorical: dichotomous only

McNemar test33

Categorical: dichotomous only

Bland-Altman diagrams14,16,34

Continuous

Concordance correlation coefficient

Continuous


Description The overall percentage agreement between 2 reports is frequently calculated. However, the percentage agreement can be falsely elevated, as its calculation does not correct for chance agreement. For this reason, researchers often use Cohen . coefficient, an omnibus test of agreement, to evaluate concordance with nominal data. Weighted . coefficients can be used if levels of disagreement are viewed differentially, as with some ordinal scales. The interpretation of the . coefficient is based on the magnitude of agreement rather than the significance of the statistic. A commonly used criterion is that proposed by Landis and Koch29: poor, G0; slight, 0Y0.20; fair, 0.21Y0.40; moderate, 0.41Y0.60; substantial, 0.61Y0.80; almost perfect, 0.81Y1.0. However, Feinstein and Cicchetti31 report that Cohen . can either overcorrect or undercorrect the level of concordance in certain circumstances. Some researchers present PPA and NPA. These values are helpful to interpret data for which discrepancies in marginal totals affect Cohen . coefficients as described above. Applying this concept to symptom research, 2 raters (eg, child, parent) may have a high NPA and clearly agree on the absence of a symptom. Even with a high level for NPA, they may have difficulties agreeing on the presence of the symptom and the overall percentage agreement may be low. When children’s reports are considered to be the ‘‘gold standard,’’ if NPA is low, the parent would have overestimated the child’s symptom. If the PPA is low, the parent would have underestimated the symptom. The McNemar test assesses the significance of the difference between 2 paired proportions (ie, to evaluate variations in how raters disagree). When translating these concepts to symptom concordance research, the McNemar test would be significant if (1) most children deny a symptom that their parents report and (2) parents rarely respond with no when the children report the symptom (or vice versa). Bland and Altman14 proposed a method to evaluate concordance when data are continuous referred to as the 95% limits of agreement method. Assumptions for the analysis are that the mean and standard deviation of the differences are constant (checked by evaluation of a scatterplot with average of the ratings is plotted on the x-axis and the difference between the ratings is plotted on the y-axis) and that the differences are approximately normal (checked by evaluation of a histogram of the differences). The limits of agreement are determined by calculating the values for the mean of the differences T2 standard deviations (or the more precise value of 1.96 standard deviations can be used). The expectation is that 95% of the differences fall between these lines of agreement. Judgment of the values for these limits is related to clinical significance, set a priori. Interpretation: High level of agreement is noted if the mean of all differences lies close to 0 and the distance between the limits of agreement lines is small, with few outliers. While the Pearson correlation is inappropriate for use in concordance assessment, as it measures precision but not accuracy, the concordance correlation coefficient (CCC), calculated by using a bias correction factor in conjunction with the Pearson correlation, provides a measure of accuracy as well as precision. The correction factor incorporates the data’s deviation from a 45- line through the origin. A description of the calculations, validation of the test supported by Monte Carlo simulations, and illustrative examples of the CCC are found in a manuscript by Lin.15

Cancer NursingTM, Vol. 37, No. 4, 2014


n

255

If symptom occurrence rates differ between children and their parents, the evaluation of differences in other symptom dimensions (eg, severity) is not warranted. First, the overall agreement at the instrument level (ie, total number of concordant symptoms) was calculated. The total number of concordant symptoms within dyads based on ratings of symptom occurrence was calculated as a count of the number of symptoms for which both the child and parent endorsed the presence or the absence of the symptom. Five additional approaches were used to evaluate concordance for individual symptoms: percentage agreement, positive percentage agreement (PPA), negative percentage agreement (NPA), Cohen . coefficients, and McNemar tests. As noted by Landis and Koch,29 . coefficients were classified according to the magnitude of agreement (ie, poor, G0; slight, 0Y0.20; fair, 0.21Y0.40; moderate, 0.41Y0.60; substantial, 0.61Y0.80; almost perfect, 0.81Y1.00). Children’s reports were considered to be the gold standard. To define the category of ‘‘parents underestimated’’ symptoms, we used a cutoff of less than 80% for PPA (ie, Q20% of parents disagreed when the child reported the occurrence of a symptom), similar to the 80% level of agreement recommended by von Baeyer.35 However, at times, these marginal values were due to low symptom occurrence rates. Therefore, the PPA was restricted to values that represent at least 20% of the sample (ie, at least 21 patients). Conversely, the category of ‘‘parents overestimated’’ symptoms applied to cases for which the NPA was less than 80%, when sufficient numbers of responses were available for analysis (ie, at least 21 responses). Percentages of agreement, Cohen . coefficients, and McNemar tests were calculated using STATA version 11.1. All other data were analyzed using IBM SPSS Statistics version 19. Independentsamples t tests and Pearson correlations were used to evaluate the relationships of demographic characteristics (ie, gender, primary language, race/ethnicity, child’s age, number of children in the home, household income, parental education) and clinical characteristics (ie, child’s diagnosis, KPS score, number of relapses) with the number of concordant symptoms based on children’s and parents’ reports of symptom occurrence. A P value of G.05 was considered statistically significant.

n

Results

Demographic and Clinical Characteristics The demographic and clinical characteristics of the children (n = 107) and their parents are summarized in Table 2. Most children were male (55.1%) and a member of a racial/ethnic minority (ie, patients with self-reported race/ethnicity other than non-Hispanic white; 63.6%), with a mean age of 14.6 years and an average KPS score of 80.5. Children’s mean time since diagnosis was 12.8 months. A few children had experienced late relapses, which contributed to a standard deviation of 25.5 months. The median time since diagnosis was 3.2 months. Children were diagnosed with a wide range of cancers, and 23.4% were receiving treatment for a relapse or disease progression. Most treatment was delivered in the inpatient setting (68.9%). 256 n Cancer NursingTM, Vol. 37, No. 4, 2014

Most of the parents who participated were mothers (79.4%) and were from a racial or ethnic minority (57.9%). Their mean age was 44.9 years and, on average, had 14 years of education.

Symptom Characteristics Children’s and parents’ reports of symptom occurrence are listed in Table 3. Children reported significantly fewer symptoms (mean [SD], 10.5 [5.0]) than their parents did (mean [SD], 12.1 [5.6]; P = 0.024). The 5 most common symptoms reported by children were nausea, lack of energy, lack of appetite, pain, and less hair than usual. The most common symptoms reported by parents were the same, but in a different order.

Concordance of Symptom Reports On average, children’s and parents’ reports of symptom occurrence were concordant for 19.6 (SD, 4.7) of the 31 symptoms. Additional analyses of symptom concordance are found in Table 3. Overall percentage agreement ranged from 61% to 97%. . Statistics ranged from 0.15 to 0.65. Based on criteria for . statistics published by Landis and Koch,29 1 symptom was classified as having poor agreement; 14 symptoms, as having fair agreement; 11 symptoms, as having moderate agreement (ie, lack of energy, lack of appetite, pain, change in the way food tastes, weight loss, dry mouth, numbness and tingling in the hands/feet, mouth sores, constipation, diarrhea, and difficulty swallowing); and 5 symptoms, as having substantial agreement (ie, less hair than usual, vomiting, headache, cough, and problems with urination). Based on our criteria of a PPA of less than 80% for at least 20% of the sample, no symptoms were classified as being underestimated by parents. Based on our criteria of an NPA of less than 80% for at least 20% of the sample, 7 symptoms were classified as being overestimated by parents (ie, change in the way food tastes, feeling irritable, worrying, insomnia, lack of concentration, feeling sad, and ‘‘I don’t look like myself’’). Statistically significant differences were noted among the McNemar tests for 8 of the symptoms (ie, change in the way food tastes, feeling irritable, worrying, lack of concentration, feeling sad, ‘‘I don’t look like myself,’’ sweating, and difficulty swallowing). The relationships between the number of concordant symptoms and demographic and clinical factors are reported in Table 4. The only significant relationships with the number of concordant symptoms were parent race (ie, the concordance for dyads was higher if the parents were non-Hispanic white compared with another race) and the child’s diagnosis (ie, the concordance for dyads was lower if the children were diagnosed with acute lymphoblastic leukemia compared with other cancer diagnoses).

n

Discussion

Each of the statistical approaches used in this study to evaluate concordance between children’s and parents’ reports of

Baggott et al


Table 2 & Demographic and Clinical Characteristics of Children and Parents Characteristic Age, mean (SD), y Education, mean (SD), y Karnofsky Performance Status score, mean (SD)a Time since diagnosis, mean (SD), mo Number of children in household (including patient), mean (SD) Gender Female Male Race/ethnicity HispanicVany race Non-Hispanic white Asian Mixed race Black Native Hawaiian, other Pacific Islander American Indian Diagnosis Acute lymphoblastic leukemia Osteosarcoma Ewing sarcoma Hodgkin lymphoma Non-Hodgkin lymphoma Central nervous system tumor Rhabdomyosarcoma Acute myelogenous leukemia Other solid tumor Other leukemia Number of relapses None 1 Q2 Relation to child Mother Father Other Child lives in 91 household (% yes) Annual income G$10 000 $10 000Y$39 999 $40 000Y$69 999 $70 000Y$99 999 Q$100 000 Unknown

Children (n = 107) 14.6 8.6 80.5 12.8 2.4

(2.5) (2.5) (17.4) (25.5) (1.1)

Parents/Guardians (n = 107) 44.9 (7.3) 14.0 (3.8)

48 (44.9) 59 (55.1)

89 (83.2) 18 (16.8)

47 (43.9) 39 (36.4) 8 (7.5) 6 (5.6) 4 (3.7) 2 (1.9) 1 (0.9)

41 45 14 0 3 2 2

29 14 10 10 9 9 5 3 12 6

(38.3) (42.1) (13.1) (0) (2.8) (1.9) (1.9)

(27.1) (13.1) (9.3) (9.3) (8.4) (8.4) (4.7) (2.8) (11.2) (5.6)

82 (76.6) 20 (18.7) 5 (4.7) 85 (79.4) 16 (15.0) 6 (5.6) 10 (9.4) 13 18 17 8 35 16

(12.1) (16.9) (15.8) (7.5) (32.7) (15)

Data are presented as n (%), unless otherwise indicated. a A modified version of the Karnofsky Performance Status (in layman’s terms) was used, scored from 30 (I feel severely disabled and need to be hospitalized) to 100 (I feel normal; I have no complaints or symptoms) in 10-point increments.

symptoms has advantages and disadvantages. Although overall percentage agreement can be used as an estimate of agreement, this value is rarely cited because inflated values can occur simply because of chance. However, in symptom research, concordance between raters rarely has a chance component because the ratings are based on the patient’s condition. The percentage agreement and . coefficients provide similar insights into concordance, namely, the degree of similarity in responses. The PPA, NPA, and McNemar values provide insights into the direction and degree of discrepancies in the discordant values.


Taken together, these data provide insights to the typical patterns of responses. In addition to ‘‘Are the responses similar?’’ the question of ‘‘Who is more likely to say yes?’’ is answered. The . statistic can be used to correct for chance agreement. However, it has the potential to either overcorrect or undercorrect based on the distribution of responses. For example, 2 symptoms may have the same value for percentage agreement. However, if 1 symptom occurs either very frequently (or very infrequently), it will have a lower . statistic than one with an occurrence rate of approximately 50%. Therefore, awareness Cancer NursingTM, Vol. 37, No. 4, 2014


n

257

Table 3 & Child and Parent Reports of Symptom Occurrence and Analyses of Concordance

Symptom Nausea Lack of energy Lack of appetite Pain Less hair than usual Feeling drowsy Vomiting Changes in the way food tastes Weight loss Feeling irritable Headache Dry mouth Cough Worrying Dizziness Insomnia Feeling nervous Lack of concentration Feeling sad Skin changes ‘‘I don’t look like myself ’’ Numbness/tingling in hands/feet Itching Mouth sores Constipation Dyspnea Sweating Diarrhea Difficulty swallowing Swelling of arms/legs Problems with urination

Parent Child . Occurrence Occurrence PPA, Parent NPA, Parent McNemar Rate, % % Agreement Statistica Rate, % % Underestimated % Overestimated P 78.5 81.3 62.6 62.3 66.0 52.8 47.2 53.3

78 84 76 78 83 63 82 71

0.34 0.54 0.49 0.55 0.64 0.26 0.64 0.42

86 94 82 84 92 66 83 81

38.7 35.8 35.8 32.7 32.7 31.1 31.1 30.8 30.8 29.9

46.2 58.5 37.7 29.9 31.8 54.2 21.7 41.1 37.4 45.8

76 63 84 77 84 61 75 67 70 69

0.51 0.29 0.65 0.46 0.64 0.25 0.37 0.29 0.33 0.36

78 78 79 60 74 76 45 61 61 75

29.2 27.1 26.7

54.3 33.7 44.8

67 73 66

0.37 0.36 0.29

87 61 71

25.2

28.3

81

0.51

67

c

86

.655

23.4 20.8 20.6 19.8 19.6 17.9 11.3

22.4 24.3 22.9 19.8 9.3 23.6 19.8

75 83 86 79 79 83 87

0.29 0.52 0.59 0.35 0.15 0.47 0.49

44 68 73 48 19 63 82

c

84 87 89 87 93 87 87

.847 .346 .439 1.000 .022 .346 .008

7.5 3.7

6.7 4.7

91 97

0.35 0.65

38 75

c

96 98

.739 .564

c

c c c c c c c c c c

c c

c c c c c c

c

48 56 67 70 71 60 81 64

b

78.5 74.8 60.7 59.8 56.6 56.1 46.2 39.6

75 54 87 85 89 55 89 70 74 67 59 78 64

b b b b b

(

b (

(

( b (

( b (

1.000 .090 .695 .532 .059 .873 .491 .007 .162 G.001 .808 .549 .808 G.001 .050 .128 .289 .003 G.001 .257 .001

Abbreviations: NPA, negative percentage agreement; PPA, positive percentage agreement. ( indicates symptoms that the parents overestimated. a P values for all . statistics G.05. b NPA less than 80% but not classified as ‘‘parent overestimated’’ because of distribution of responses. c PPA less than 80% but not classified as ‘‘parent underestimated’’ because of distribution of responses.

of the influence of response distributions and review of the percentage agreement need to be done before one concludes that a low . statistic indicates poor concordance. For example, in our study, weight loss and itching had similar percentages of agreement (ie, 76% and 75%, respectively). However, the . coefficients were quite different (0.51 and 0.29, respectively). Although one can suggest that chance would be equally involved in the determination of agreement for itching and weight loss, the differences in . coefficients are based on the mathematical processes involved in this calculation. While one might strive for substantial agreement between raters, . coefficients of lower magnitude may not be indicative of poor agreement. The PPA 258 n Cancer NursingTM, Vol. 37, No. 4, 2014

and NPA can be used to supplement the . coefficient32 to provide the ‘‘big picture’’ of the level of agreement and direct researchers to the patterns of discordance. When dyadic responses differ, the McNemar test, in addition to the . statistic, can be used to provide a more complete picture of the level and type of agreement present. A McNemar result cannot be evaluated in isolation because, although it provides the direction of the discrepancy, it does not evaluate the degree of discordance. The McNemar test needs to be used with a measure of agreement like the . statistic. In this data set, we evaluated whether the discordance in symptom reports occurred because the parents overestimated (low NPA) or underestimated (low

Baggott et al


Table 4 & Comparison of the Mean Number of Concordant Symptoms Between Child and Parent Reports of Symptom Occurrence With Demographic and Clinical Characteristics

Number of Concordant Symptoms Between Children and Parents Characteristic Child’s gender Male Female Parent’s gender Male Female Child’s booklet language English Spanish Parent’s booklet language English Spanish Child’s race Non-Hispanic white Other Parent’s race Non-Hispanic white Other Child’s ethnicity Hispanic Non-Hispanic Parent’s ethnicity Hispanic Non-Hispanic Child’s diagnosis Acute lymphocytic leukemia Other Child’s Karnofsky Performance Status score Q80 G80

Mean (SD)

Statistic t = j0.98; P = .33

19.2 (4.6) 20.1 (4.8) t = j0.45; P = .65 19.1 (5.6) 19.7 (4.5) t = j0.68; P = .50 19.7 (4.6) 18.6 (5.7) t = 1.69; P = .09 19.9 (4.6) 17.8 (5.0) t = 1.37; P = .18 20.4 (5.0) 19.1 (4.0) t = 2.18; P = .03 20.7 (4.2) 18.7 (4.9) t = j1.27; P = .21 18.9 (5.5) 20.1 (3.9) t = j1.60; P = .11 18.7 (5.2) 20.1 (4.3) t = j2.62; P = .01 17.9 (4.5) 20.4 (4.6) t = 1.19; P = .24 19.9 (4.6) 18.8 (4.6) Correlation Between Number of Concordant Symptoms and Demographic and Clinical Characteristics

Characteristic Child’s age Number of relapses Time since diagnosis Number of children in the home Household income Parent’s years of education

Correlation j0.13 j0.11 j0.15 0.01 0.05 0.14

PPA) the children’s reports of symptom occurrence. Although the McNemar test provides the level of significance for these differences, one must review the cross-tabulated data to determine the direction of the discrepancy. In our study, the 5 most common symptoms were the same for both children and their parents. With the exception of headache, most of the symptoms with high levels of overall agreement (ie, 980%) have some visual cues or have an associated visible sign (eg, clumsiness associated with numbness in feet). Although the complaint of headache lacks visible signs, the inclusion of pain as ‘‘the fifth vital sign’’ in recent years may have


Statistic P P P P P P

= = = = = =

.17 .25 .12 .89 .64 .15

promoted improved communication regarding headache and other complaints of pain.36 Narrowing the focus further, the symptoms with the highest NPA (ie, children and parents agreed on their absence) were problems with urination, swelling of the arms or legs, sweating, cough, and dizziness. Likewise, most of these symptoms have visible manifestations. In contrast, the symptoms with the lowest NPA (ie, the child did not report the symptom, but the parent did) were nausea, feeling irritable, worrying, lack of energy, feeling sad, feeling drowsy, ‘‘I don’t look like myself,’’ changes in the way food tastes, lack of concentration, lack of appetite. Cancer NursingTM, Vol. 37, No. 4, 2014


n

259

These symptoms represent ‘‘hidden concerns,’’ with few visible cues. The symptom of altered self-perception (ie, ‘‘I don’t look like myself’’) might seem to be associated with visual cues (eg, hair loss). However, Woodgate37 reported that although adolescents with cancer acknowledged that their disease had a significant impact on their lives, these teenagers described themselves as ‘‘still being very much the same person.’’ What are the lessons learned from these analyses of dyadic concordance? The ability to distinguish symptoms with more visible manifestations from those symptoms without visual cues adds credibility to the use of NPA as a measure of symptom concordance. The separation between more and less visible symptoms was not so distinct when symptoms were evaluated using PPA. With the evaluation of less visible symptoms, one needs to confirm their presence rather than assume their absence. Although one must recall that the calculation for NPA does not correct for chance agreement, chance agreement may have little relevance in the evaluation of concordance between symptom reports. In contrast, although . coefficients are more commonly used to evaluate concordance, we noted inconsistent patterns in the . coefficients when they are related to percentages of agreement. Some symptoms in pediatric oncology occur infrequently but can be very severe or distressing when present.12,38 It is difficult to determine the degree of concordance between children’s reports and those of their parents with the use of . coefficients for these symptoms because of the distribution of the cross-tabulated responses. Most of the symptoms that parents tended to overestimate were psychosocial concerns (ie, feeling irritable, worrying, insomnia, lack of concentration, feeling sad, ‘‘I don’t look like myself’’). At least 30% of the time that the children denied these symptoms, their parents reported them as a problem. These findings are consistent with the literature in which family caregivers of adult cancer patients tend to overestimate psychosocial symptoms.39Y41 Of note, with the exception of insomnia, most of these symptoms had high PPAs (ie, 970%) such that if children reported the symptom, their parents rarely disagreed. However, if children do not overtly demonstrate evidence of a particular symptom, it may be difficult for parents to discern their children’s experience. In such cases, parents, who often experience distress while their children undergo cancer treatment,42 may infer the occurrence of a symptom based on their own level of distress or their own expectations (eg, feeling sad is expected during cancer treatment). In our study, when children reported psychosocial concerns, the parents agreed with their reports. However, clinicians may need to perform a more detailed assessment of children who deny these symptoms because parents may have a different perception of their children’s psychological health. Determining the underlying factors associated with dyadic discordance may be challenging. Children may repress difficult emotional concerns. Like adult cancer patients,43,44 children may conceal symptoms to avoid burdening family members. However, parents may project emotions based on their own expectations. For example, in a study by Poder and colleagues42 in which parents of pediatric oncology patients rated their children’s symptoms using the MSAS, parents who were at risk for posttraumatic 260 n Cancer NursingTM, Vol. 37, No. 4, 2014

stress disorder (PTSD) reported that their children experienced significantly more symptoms than did parents who were not at risk for PTSD. When between-group differences in MSAS total and subscale scores were evaluated, parents at risk for PTSD reported higher MSAS scores. Of note, the largest between-group difference was the subscale score for psychosocial symptoms.42 Regardless of the circumstances, clinicians need to be aware of the potential for discordance in children’s and parents’ assessments of children’s symptom experiences. Clinicians need to perform detailed investigations during and after cancer treatment. In symptom research, established criteria do not exist to determine acceptable levels of agreement. Clearly, the stringent levels of concordance established in engineering or other ‘‘hard sciences’’ are not necessary. As described by von Baeyer,35 the common approach devised by Bland and Altman14 that uses the 95% level of agreement may be overly stringent when one considers the subjective nature of symptoms and children’s developmental level. If children and parents are included in symptom assessments, discordant assessments can be investigated. However, if the child’s voice is not elicited, clinicians may not have an accurate picture of the child’s experience. Based on our data, in such cases, few symptoms are likely to be missed, as parents rarely underreported symptoms when compared with children’s accounts. In the clinical setting, in general, both child and parent viewpoints are elicited, but children may remain taciturn. Awareness of typical patterns of symptom reporting (eg, parents tend to overestimate children’s psychosocial concerns) may encourage clinicians to probe quiet children for responses rather than accept parents’ viewpoints as representative. The UCSF Symptom Management Theory places the patient’s symptom experience within the domains of nursing science (ie, person, health and illness, environment). Although we evaluated the associations between a number of demographic and clinical characteristics and dyadic discordance in number of symptoms reported (see Table 4), only the parents’ race and the child’s diagnosis were significantly associated with the number of concordant symptoms. Notably, no gender differences (ie, fathers’ vs mothers’ ratings) were identified in this analysis. The racial and ethnic diversity in this study may have affected symptom reporting. Cultural issues and other factors that may influence symptom concordance warrant investigation in future studies.

n

Study Limitations

Limitations of this study include the relatively small sample size and the use of a heterogeneous sample with regard to cancer diagnoses, the chemotherapy agents used, and the time elapsed since the patients’ diagnoses. All of these factors could influence the occurrence of symptoms and/or children’s and parents’ communication styles. In addition, a 1-week recall period was used to evaluate symptoms. Some symptoms may have occurred intermittently during the week, which may have influenced children’s and parents’ ratings of symptom occurrence.

Baggott et al


n

Clinical Implications

Findings from this study suggest that clinicians need to interview both children and parents about the occurrence of symptoms. Discrepancies between reports should be assessed to better understand the children’s symptom experiences. Clinicians need to encourage children and parents to communicate symptoms more clearly. Patient-centered tools are available to encourage children to share their ‘‘voice’’ regarding symptoms.45 Better communication will improve parents’ understanding of their children’s experiences. Clinician-directed educational programs to promote effective communication with children and their parents can lead to improved outcomes.46,47 When discordant reports are encountered, clinicians need to obtain more information from the child.

n

Research Implications

Clearly a ‘‘1-size-fits-all’’ approach is not appropriate when one is evaluating the concordance between symptom reports. The methods of analysis should match the aims and hypotheses of the study. Although research was identified on the concordance between patient-proxy symptom reports among adult cancer patients48 and among children with other chronic illnesses (eg, sickle cell disease,49 chronic pain,50Y52 rheumatoid arthritis,53 cystic fibrosis),54 only 2 studies were found in pediatric oncology. Additional investigations are needed to determine the factors that contribute to higher levels of concordance between children’s and parents’ reports of symptoms, the consequences of discordant reports, and the most effective strategies to improve concordance. When symptoms are collected as clinical trial end-points, whenever possible, the respondent must be the child. If parental concerns do not reflect children’s true symptom experiences, inappropriate conclusions may be drawn, which could adversely impact overall symptom management strategies.

n

Conclusions

The importance of eliciting PROs was affirmed by researchers and policymakers with the inclusion of PROs in clinical trials in the United States.1,2 Despite these mandates, and a more recent philosophical push to promote children’s active participation in healthcare,55 children may remain passive bystanders in their care.5 To our knowledge, this study is the first to provide an in-depth report of the level of concordance between children’s and parents’ reports of symptoms during cancer chemotherapy. Based on our findings, few symptoms are likely to be missed when clinicians rely on parental report of symptoms. However, the true symptom experience may be difficult to ascertain if parents overestimate symptom occurrence, particularly those of a psychosocial nature. Unfortunately, at times, in pediatric oncology, clinicians must rely on parental proxy, and a clear understanding of patterns of concordance in symptom reporting can guide decisions in symptom management.


References 1. Scoggins JF, Patrick DL. The use of patient-reported outcomes instruments in registered clinical trials: evidence from ClinicalTrials.gov. Contemp Clin Trials. 2009;30(4):289Y292. 2. Food and Drug Administration. Guidance for industry patient-reported outcome measures: use in medical product development to support labeling claims. 2009; http://www.fda.gov/downloads/Drugs/Guidances/UCM193282 .pdf. Accessed February 12, 2013. 3. Riley AW. Evidence that school-age children can self-report on their health. Ambul Pediatr. 2004;4(4 suppl):371Y376. 4. Van Cleve L, Munoz CE, Riggs ML, Bava L, Savedra M. Pain experience in children with advanced cancer. J Pediatr Oncol Nurs. 2012;29(1):28Y36. 5. Cahill P, Papageorgiou A. Triadic communication in the primary care paediatric consultation: a review of the literature. Br J Gen Pract. 2007;57:904Y911. 6. Hinds PS, Menard JC, Jacobs SS. The child’s voice in pediatric palliative and end-of-life care. Prog Palliat Care. 2012;20(6):337Y342. 7. Hinds PS, Billups CA, Cao X, et al. Health-related quality of life in adolescents at the time of diagnosis with osteosarcoma or acute myeloid leukemia. Eur J Oncol Nurs. 2009;13(3):156Y163. 8. Levi RB, Drotar D. Health-related quality of life in childhood cancer: discrepancy in parent-child reports. Int J Cancer Suppl. 1999;12:58Y64. 9. Matziou V, Perdikaris P, Feloni D, Moshovi M, Tsoumakas K, Merkouris A. Cancer in childhood: children’s and parents’ aspects for quality of life. Eur J Oncol Nurs. 2008;12(3):209Y216. 10. Russell KM, Hudson M, Long A, Phipps S. Assessment of health-related quality of life in children with cancer: consistency and agreement between parent and child reports. Cancer. 2006;106(10):2267Y2274. 11. Sawyer M, Antoniou G, Toogood I, Rice M. A comparison of parent and adolescent reports describing the health-related quality of life of adolescents treated for cancer. Int J Cancer Suppl. 1999;12:39Y45. 12. Collins JJ, Byrnes ME, Dunkel IJ, et al. The measurement of symptoms in children with cancer. J Pain Symptom Manage. 2000;19(5):363Y377. 13. Collins JJ, Devine TD, Dick GS, et al. The measurement of symptoms in young children with cancer: the validation of the Memorial Symptom Assessment Scale in children aged 7Y12. J Pain Symptom Manage. 2002;23(1): 10Y16. 14. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307Y310. 15. Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45(1):255Y268. 16. Kwiecien R, Kopp-Schneider A, Blettner M. Concordance analysis: part 16 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2011;108(30):515Y521. 17. Humphreys J, Lee KA, Carrieri-Kohlman V, et al. A middle range theory of symptom management. In: Smith MJ, Liehr PR, eds. Middle Range Theory for Nursing. New York, NY: Springer Publishing Company; 2008: 145Y158. 18. Dodd MJ, Janson S, Facione N, et al. Advancing the science of symptom management. J Adv Nurs. 2001;33(5):668Y676. 19. Baggott C, Cooper BA, Marina N, Matthay KK, Miaskowski C. Symptom cluster analyses based on symptom occurrence and severity ratings among pediatric oncology patients during myelosuppressive chemotherapy. Cancer Nurs. 2012;35(1):19Y28. 20. Portenoy RK, Thaler HT, Kornblith AB, et al. The Memorial Symptom Assessment Scale: an instrument for the evaluation of symptom prevalence, characteristics and distress. Eur J Cancer. 1994;30A(9):1326Y1336. 21. Karnofsky D, Burchenal J. The clinical evaluation of chemotherapeutics in cancer. In: Macleod CM, ed. Evaluation of Chemotherapeutic Agents; Symposia of the Section on Microbiology, the New York Academy of Medicine. New York, NY: Columbia University Press; 1949:191Y205. 22. Schaafsma J, Osoba D. The Karnofsky Performance Status Scale re-examined: a cross-validation with the EORTC-C30. Qual Life Res. 1994;3(6):413Y424. 23. Schag CC, Heinrich RL, Ganz PA. Karnofsky Performance Status revisited: reliability, validity, and guidelines. J Clin Oncol. 1984;2(3):187Y193. 24. Forinder U, Lof C, Winiarski J. Quality of life following allogeneic stem cell transplantation, comparing parents’ and children’s perspective. Pediatr Transplant. 2006;10(4):491Y496.

Cancer NursingTM, Vol. 37, No. 4, 2014


n

261

25. Jalali R, Dutta D, Kamble R, et al. Prospective assessment of activities of daily living using modified Barthel’s Index in children and young adults with low-grade gliomas treated with stereotactic conformal radiotherapy. J Neurooncol. 2008;90(3):321Y328. 26. Tseng TH, Cleeland CS, Wang XS, Lin CC. Assessing cancer symptoms in adolescents with cancer using the Taiwanese version of the M.D. Anderson Symptom Inventory. Cancer Nurs. 2008;31(3):E9YE16. 27. Williams PD, Schmideskamp J, Ridder EL, Williams AR. Symptom monitoring and dependent care during cancer treatment in children: pilot study. Cancer Nurs. 2006;29(3):188Y197. 28. Acquadro C, Conway K, Girourdet C, Mear I. Linguistic Validation Manual for Patient-Reported Outcomes (PRO) Instruments. Lyon, France: Mapi Research Institute; 2004. 29. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159Y174. 30. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37Y46. 31. Feinstein AR, Cicchetti DV. High agreement but low kappa, I: the problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543Y549. 32. Cicchetti DV, Feinstein AR. High agreement but low kappa, II: resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551Y558. 33. Pett MA. Nonparametric Statistics for Health Care Research: Statistics for Small Samples and Unusual Distributions. Thousand Oaks, CA: Sage Publications; 1997. 34. Bland JM, Altman DG. Applying the right statistics: analyses of measurement studies. Ultrasound Obstet Gynecol. 2003;22(1):85Y93. 35. von Baeyer CL. Reported lack of agreement between self-report pain scores in children may be due to a too strict criterion for agreement. Pain. 2012; 153(10):2152Y2153. Author reply 2153-2154. 36. Neale KL. The fifth vital sign: chronic pain assessment of the adolescent oncology patient. J Pediatr Oncol Nurs. 2012;29(4):185Y198. 37. Woodgate RL. A different way of being: adolescents’ experiences with cancer. Cancer Nurs. 2005;28(1):8Y15. 38. Baggott C, Dodd M, Kennedy C, et al. Changes in children’s reports of symptom occurrence and severity during a course of myelosuppressive chemotherapy. J Pediatr Oncol Nurs. 2010;27(6):307Y315. 39. Tang ST, McCorkle R. Use of family proxies in quality of life research for cancer patients at the end of life: a literature review. Cancer Invest. 2002; 20(7Y8):1086Y1104. 40. McPherson CJ, Addington-Hall JM. Judging the quality of care at the end of life: can proxies provide reliable information? Soc Sci Med. 2003;56(1):95Y109. 41. Lobchuk MM, Degner LF. Patients with cancer and next-of-kin response comparability on physical and psychological symptom well-being: trends and measurement issues. Cancer Nurs. 2002;25(5):358Y374.

262 n Cancer NursingTM, Vol. 37, No. 4, 2014

42. Poder U, Ljungman G, von Essen L. Parents’ perceptions of their children’s cancer-related symptoms during treatment: a prospective, longitudinal study. J Pain Symptom Manage. 2010;40(5):661Y670. 43. McPherson CJ, Wilson KG, Lobchuk MM, Brajtman S. Self-perceived burden to others: patient and family caregiver correlates. J Palliat Care. 2007;23(3):135Y142. 44. Larsson A, Wijk H. Patient experiences of pain and pain management at the end of life: a pilot study. Pain Manag Nurs. 2007;8(1):12Y16. 45. Ruland CM, Starren J, Vatne TM. Participatory design with children in the development of a support system for patient-centered care in pediatric oncology. J Biomed Inform. 2008;41(4):624Y635. 46. Kurtz SM, Silverman J, Draper J. Teaching and Learning Communication Skills in Medicine. 2nd ed. Oxford, United Kingdom: Radcliffe Pub; 2005. 47. Howells R, Lopez T. Better communication with children and parents. Paediatr Child Health. 2008;18(8):381Y385. 48. McPherson CJ, Wilson KG, Lobchuk MM, Brajtman S. Family caregivers’ assessment of symptoms in patients with advanced cancer: concordance with patients and factors affecting accuracy. J Pain Symptom Manage. 2008; 35(1):70Y82. 49. Barakat LP, Simon K, Schwartz LA, Radcliffe J. Correlates of pain-rating concordance for adolescents with sickle cell disease and their caregivers. Clin J Pain. 2008;24(5):438Y446. 50. Garber J, Van Slyke DA, Walker LS. Concordance between mothers’ and children’s reports of somatic and emotional symptoms in patients with recurrent abdominal pain or emotional disorders. J Abnorm Child Psychol. 1998;26(5):381Y391. 51. Cohen LL, Vowles KE, Eccleston C. Adolescent chronic pain-related functioning: concordance and discordance of mother-proxy and self-report ratings. Eur J Pain. 2010;14(8):882Y886. 52. Youssef NN, Murphy TG, Langseder AL, Rosh JR. Quality of life for children with functional abdominal pain: a comparison study of patients’ and parents’ perceptions. Pediatrics. 2006;117(1):54Y59. 53. Ennett ST, DeVellis BM, Earp JA, Kredich D, Warren RW, Wilhelm CL. Disease experience and psychosocial adjustment in children with juvenile rheumatoid arthritis: children’s versus mothers’ reports. J Pediatr Psychol. 1991;16(5):557Y568. 54. Britto MT, Kotagal UR, Chenier T, Tsevat J, Atherton HD, Wilmott RW. Differences between adolescents’ and parents’ reports of health-related quality of life in cystic fibrosis. Pediatr Pulmonol. 2004;37(2):165Y171. 55. Gabe J, Olumide G, Bury M. ‘It takes three to tango’: a framework for understanding patient partnership in paediatric clinics. Soc Sci Med. 2004; 59(5):1071Y1079.

Baggott et al


Should parents be allowed in pediatric RR?

International pediatric radiology education: who should be trained, and how?

How should the completeness and quality of curated nanomaterial data be evaluated?

Gastroschisis--what should be told to parents?

How should parents be informed about major procedures? An exploratory trial in the neonatal period.

How ARDS should be treated.

How should "ambidexterity" be estimated?

What components should be evaluated in a psychiatry residency.

Nailfold Capillaroscopy in Rheumatic Diseases: Which Parameters Should Be Evaluated?

How should novelty be valued in science?

How can myocarditis be diagnosed and should it be treated?

Thrombosis with Behçet's disease should be evaluated different conditions.

Pediatric psycho-oncology care: standards, guidelines, and consensus reports.

Assessment of Parents' Perception of Quality of Pediatric Oncology Inpatient Care at Kenyatta National Hospital.

How should continence and incontinence after radical prostatectomy be evaluated? A prospective study of patient ratings and changes with time.

[Should pediatric parenteral nutrition be individualized?].

Research groups: How big should they be?

Just how wide should 'wide reading' be?

How should medical care be rationed?

How frequently should asymptomatic patients be dilated?

How should prostate specific antigen be interpreted?

How social health insurance should be financed.

How far should blood pressure be lowered?

How should surgical safety be taught?