FROM THE ACADEMY
Developing and Assessing Nutrition Education Handouts (DANEH): Testing the Validity and Reliability of the New Tool Rosa K. Hand, MS, RDN, LD; Lisa Medrow, RDN, LD; Katie Brown, EdD, RDN, LD
W
RITTEN NUTRITION EDucation handouts are useful tools for registered dietitian nutritionists (RDNs) and nutrition and dietetics technicians, registered, to reinforce key messages, enhance knowledge, influence attitudes, and promote behavior change.1-3 Best-practice recommendations for developing effective nutrition education handouts for specific audiences are available, but many lack references and are lengthy, which limits their usefulness.4 The Centers for Disease Control and Prevention recently released a research-based checklist, the CDC Clear Communication Index, to plan and assess communication materials for the public. 4 This checklist is applicable for a wide variety of health messages, topics, and materials. Yet, a useful, reliable, checklist tool specifically for the development and assessment of effective nutrition education handouts has not been published. This gap led to the development of the Designing and Assessing Nutrition Education Handouts (DANEH) checklist tool (Figure) by the Academy of Nutrition and Dietetics Foundation. The DAN E H to o l wa s d e ve l o p e d f o r t h e purposes of screening existing nutrition education handouts to establish the inclusion and exclusion of important quality components, and develop quality nutrition education handouts. DANEH incorporates 21 constructs identified as quality indicators to be included in nutrition education handouts based on a literature review. DANEH was developed as an objective tool to determine which nutrition education handouts would be 2212-2672/Copyright ª 2015 by the Academy of Nutrition and Dietetics. http://dx.doi.org/10.1016/j.jand.2015.01.010 Available online 18 March 2015
816
included on the Healthy Food Bank Hub (hereafter the Hub; www. healthfoodbankhub.org). The Hub was developed as part of the Academy of Nutrition and Dietetic Foundation’s Future of Food Initiative, a collaboration with Feeding America and the National Dairy Council. The goal of the Hub was to provide a platform to promote high-quality, applicable materials for RDNs and hunger-relief professionals to use with individuals and families facing food insecurity. DANEH was developed to identify high-quality handouts for inclusion in the Hub. While DANEH was developed with the food-bank client/food-insecure audience in mind, its criteria have the potential to be applied to nutrition education handouts for other populations. This article describes the development and reliability testing of DANEH in the context of the Hub.
DEVELOPING THE CHECKLIST A review of the literature was conducted to identify best-practice recommendations for quality indicators that should be included in effective nutrition and health education handouts. Eighteen articles were reviewed.1-19 A list of 21 constructs from the articles was created and organized into five categories: content5-12; behavior focus8,11-14; 1-6,8-12 cultural sensitivity ; written word1-3,5,7-15; and organization/readability.1-3,5,6,8-19 To use DANEH to assess or score a nutrition education handout, one would determine whether each of the DANEH criteria is present or absent. The presence of each construct represents 1 point (1¼present; 0¼absent), for a possible total DANEH score of 21. The first content category (current and accurate information consistent with the 2010 Dietary Guidelines for Americans [DGA]20 and MyPlate21) was considered critical; therefore, it was
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
decided a priori that this category would require a “present” for approval.
Initial Testing and Revisions A DANEH draft, which had been reviewed and tested by the Academy’s Nutrition Solutions Working Group volunteers, was first tested by a group of 21 dietetics interns. The interns identified nutrition education handouts that they believed might be relevant to the food-insecure population (and therefore the Hub). The interns were provided six web links from the US Department of Agriculture’s Supplemental Nutrition Assistance Program Education, Cooking Matters, and state extension agencies/health departments as starting places for their search and were asked to each identify at least 10 unique handouts to which DANEH would be applied. Through this process, >200 handouts were identified. The interns were placed in groups of five and each group was assigned 10 of the identified handouts to assess with DANEH. Based on inter-rater reliability testing and comments from this group, significant changes were made to criteria wording for clarity. In addition, multiple options for reading-level assessment were identified. Based on the interns work with DANEH, 39 handouts were included in the Hub.
VALIDATING THE TOOL The goals of the testing were to determine the inter-rater reliability of DANEH, determine whether amount of training changes inter-rater reliability, determine a cut point for total DANEH score at which handouts can be considered high quality (and therefore be included in the Hub), and determine whether DANEH is valid using several concurrent markers of validity. Using the estimates of Walter and ª 2015 by the Academy of Nutrition and Dietetics.
FROM THE ACADEMY Present (1 point)
Absent (0 points)
Current, accurate, and consistent with Dietary Guidelines for Americans and MyPlate (required ‘yes’ for approval)
The content promotes current, evidencebased recommendations, and is consistent with Dietary Guidelines for Americans and MyPlate.
The content is based on outdated information (including MyPyramid).
Promotes relevant health issues for target audience
The content addresses a relevant health issue of the target audience (eg, reducing sodium for African-American audiences).
The content is not relevant to the target audience (eg, eating organic fruits and vegetables for low-income audiences).
Clear purpose
It is immediately clear what the handout will tell you or how it can help you.
You have to read much of the content before you realize what the handout will tell you or how it can help you.
One or two main themes
The handout contains no more than two main themes. For example, a handout with 10 strategies to shop on a budget has one main theme, shopping on a budget.
The handout contains three or more main themes. For example, a handout that covers tips for healthy snacks, physical activity, and bedtime routines contains three main themes.
Specific examples of desired behavior
The handout clearly instructs a person what to do, and how to do it through specific examples. For example, “Play active games with your kids, like hide-and-go-seek, double Dutch jump rope, or tag.”
The handout provides vague recommendations without specific examples of desired behavior. For example, “Be more physically active.”
Culturally appropriate content for target audience
The handout is intended for a general audience and includes many cultural food and behavior practices. Check “yes” if the target audience is a specific culture, and the handout includes appropriate food and behavior examples for that culture.
The handout is intended for a general audience and does not include culturally diverse food and behavior practices. Check “no” if the target audience is a specific culture and the handout does not represent appropriate food and behaviors for that culture.
Culturally appropriate images for target audience
The handout is intended for a general audience and includes images that respectfully represent many cultures, including the people, places and foods pictured. Check “yes” if the target audience is a specific culture, and the handout includes images of people, places, and foods from that culture.
The handout is intended for a general audience and does not include images that respectfully represent many cultures. Check “no” if the target audience is a specific culture and the handout does not include appropriate images representing that culture.
Simple, common words are used frequently, with limited use of abbreviations, acronyms, and technical jargon.
Complex and unfamiliar words are used frequently or if abbreviations, acronyms, and technical jargon appear excessively.
Content
Behavior Focus
Cultural Sensitivity
Written Word Simple, common words
(continued on next page) Figure. Designing and Assessing Nutrition Education Handouts scoring criteria and defining characteristics to help reviewers determine whether each criterion is present or absent in a nutrition education handout.
May 2015 Volume 115 Number 5
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
817
FROM THE ACADEMY Present (1 point)
Absent (0 points)
Positive messages
Messages focus mostly on the positive behavior desired (eg, “do this”).
Messages focus mostly on the negative behavior to change (eg, “don’t do this”).
Active voice, second person (you/your), conversational tone
The handout refers to the second person (eg, “you/your”), stays in the present tense, and uses a conversational/friendly tone. For example, “Offer healthy and tasty afterschool snacks for your kids, like grapes or cheese sticks.”
The handout refers to the third person, uses the past tense, and/or is too formal. For example, “Parents need to provide nutrientdense foods in between meals for their children and adolescents.”
Repetition of key words and/ or new concepts
Key words and/or new concepts are repeated effectively for reinforcement, or if repeating key words is not necessary (eg, for a very short handout).
Repeating key words and/or new concepts would be effective for reinforcement, but are missing.
Logical order, most important message first
The information is displayed in a logical order, with the most important messages listed first.
The information is disorganized, and/or the most important messages do not appear until later in the handout.
Short paragraphs
Each paragraph is short (60 words or less) and only contains one topic. Check “yes” if the handout does not contain paragraphs.
Each paragraph is long (over 60 words) and/ or contains two or more topics.
Space around headings and text
There is a 1/2-inch margin around the perimeter of the handout and has at least a double space before headings.
The margins are less than 1/2 inch around the perimeter of the handout and/or there is little space before headings.
Blocks of text are leftjustified
Blocks of text are left-justified.
Blocks of text are centered or right-justified.
Bullets, numbers, and tables
Bullets, numbers, and/or tables appear often.
The handout contains mostly text, and few bullets, numbers, or tables.
Several informative headings/subheadings
Several headings/subheadings are present to help identify what the text will tell you next.
Few headings/subheadings are present, and/or if the headings/subheadings do not inform you what the text will tell you next.
Easy to read font
Font is at least 12 point, and is serif (with feet, like Times Roman) or sans serif (without feet, like Arial).
Font is smaller than 12 point, and is fancy or curly.
Important text is bolded or underlined if necessary
Emphasizing text, bolding and underlining are used. Check “yes” if no emphasis of text is needed (eg, for a very short handout).
Emphasizing text uses all caps or italics.
Purposeful and relevant images
Images show instruction or the desired behavior, are placed near relevant text, and include captions when necessary to describe the behavior or emphasize a point.
Images do not show instruction or the desired behavior, are not placed near relevant text, or do not include captions as necessary.
5th-grade reading level
Handout is written at or below a 5th-grade reading level.
Handout is written at a 6th-grade reading level or higher.
Written Word
Organization and Readability
Figure. (continued) Designing and Assessing Nutrition Education Handouts scoring criteria and defining characteristics to help reviewers determine whether each criterion is present or absent in a nutrition education handout.
818
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
May 2015 Volume 115 Number 5
FROM THE ACADEMY colleagues22 assuming a P0 of 0.5 (a¼.05 and b¼.2), we determined that four reviewers would each need to review 33 handouts to detect an interclass correlation (ICC) of 0.6. In order to account for possible incomplete entries, we rounded the number of handouts to 35 and included two groups of four to test two training methods. Handouts for this assessment were selected from the group identified by the dietetics interns but not yet assessed with DANEH.
Identifying and Training Reviewers Eight RDNs who were Kids Eat Right members of the Academy of Nutrition and Dietetics were recruited to participate in the validity and reliability testing out of 132 eligible members based on participation in at least one Kids Eat Right project, years of experience, and activities with the Kids Eat Right program. After two groups of four were determined, the first group was randomly assigned to be trained via webinar. One of the groups of four was assigned to be trained via a 1-hour webinar led by one of the DANEH tool’s developers. The webinar trainer discussed each DANEH criterion and provided the attendees with examples of handouts that did or did not meet each criterion. The second group of four received short, written instructions on using DANEH. Both trainings included instructions on three methods of reading-level assessment: Microsoft Word, the www. readability-score.com website, and a hand calculation. All participants were sent the same group of 35 numbered handouts and a link to an online datacollection tool using SurveyMonkey and were instructed to work independently on their reviews, although project staff were available for procedural questions. The data-collection tool identified each handout by number, and asked for the reviewer’s assessment of the presence or absence of each DANEH criterion. Reading level was the only criterion for which additional information was required; reviewers had to enter the FleschKincaid grade level and which of the three methods they had used to obtain that score. The reviewers also entered their assessment of whether May 2015 Volume 115 Number 5
the handout was high quality in their opinion (to be used as the gold standard for criterion validity), and were able to make comments about each criterion and the handout overall. They had 3 weeks to complete the reviews using DANEH. Institutional Review Board approval was obtained from the American Academy of Family Physicians. Reviewers agreed to a contract that contained the elements of an informed consent. They were each provided a $500 stipend for completing the reviews.
score and the probability that a handout would be rated as high quality. Multiple-rater k and percent agreement were used to determine the consistency in evaluation of each of the criteria by reviewers and the quality assessment, both within the groups (to determine the effect of training method) and in the sample as a whole. ICC was used to assess the consistency in total DANEH score, again within groups and in the entire sample. Statistical analysis was completed using SPSS software (version 20.0, IBM SPSS), except for the multiple-rater k, which was executed in JMP software (version 11, SAS Institute Inc).
Analysis The mean DANEH scores of assessments conducted by each reviewer and reviewer group were compared using analysis of variance (reviewers) or t test (reviewer group). The number of handouts considered high quality by each reviewer was also examined. The mean and range of scores were compared based on the number of reviewers who rated the handouts high quality using analysis of variance, and compared via t test when all highquality ratings were compared to all low-quality ratings. Logistic regression was used to determine the relationship between a 1-point increase in DANEH
RESULTS All eight reviewers completed the reviews in the assigned time period. One review was missing for one of the handouts so the final number included in the sample was 34 handouts, each with eight reviews (four from each group). Table 1 shows the characteristics of the final eight reviewers. Matching on degrees and experience with Kids Eat Right programs (Energy Balance 4 Kids with Play and RD Parent Empowerment Program) was good; however, the years of experience varied between the two groups.
Table 1. Educational characteristics and experience of registered dietitian nutritionist reviewersa participating in the Developing and Assessing Nutrition Education Handouts validation testing
Education and experience
Webinar training Written training group (n[4) group (n[4) nb !
Master’s degree
2
2
Doctoral degree
1
1
Certified Diabetes Educator
2
0
Energy Balance 4 Kids with Play registered 1 dietitian nutritionist
1
RDc Parent Empowerment Program registered dietitian nutritionist
1
1
30 y experience
1
1
a
Reviewers were divided into two groups that were trained by webinar or via written materials (reading group). Number of reviewers in the two groups with each characteristic. c RD¼registered dietitian. b
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
819
FROM THE ACADEMY Mean total scores given by individual reviewers for the combination of the 34 handouts ranged from 16.7 to 20.3 (Table 2). The mean total score varied among reviewers and significantly between the reviewer groups (P
Developing and Assessing Nutrition Education Handouts (DANEH): Testing the Validity and Reliability of the New Tool Rosa K. Hand, MS, RDN, LD; Lisa Medrow, RDN, LD; Katie Brown, EdD, RDN, LD
W
RITTEN NUTRITION EDucation handouts are useful tools for registered dietitian nutritionists (RDNs) and nutrition and dietetics technicians, registered, to reinforce key messages, enhance knowledge, influence attitudes, and promote behavior change.1-3 Best-practice recommendations for developing effective nutrition education handouts for specific audiences are available, but many lack references and are lengthy, which limits their usefulness.4 The Centers for Disease Control and Prevention recently released a research-based checklist, the CDC Clear Communication Index, to plan and assess communication materials for the public. 4 This checklist is applicable for a wide variety of health messages, topics, and materials. Yet, a useful, reliable, checklist tool specifically for the development and assessment of effective nutrition education handouts has not been published. This gap led to the development of the Designing and Assessing Nutrition Education Handouts (DANEH) checklist tool (Figure) by the Academy of Nutrition and Dietetics Foundation. The DAN E H to o l wa s d e ve l o p e d f o r t h e purposes of screening existing nutrition education handouts to establish the inclusion and exclusion of important quality components, and develop quality nutrition education handouts. DANEH incorporates 21 constructs identified as quality indicators to be included in nutrition education handouts based on a literature review. DANEH was developed as an objective tool to determine which nutrition education handouts would be 2212-2672/Copyright ª 2015 by the Academy of Nutrition and Dietetics. http://dx.doi.org/10.1016/j.jand.2015.01.010 Available online 18 March 2015
816
included on the Healthy Food Bank Hub (hereafter the Hub; www. healthfoodbankhub.org). The Hub was developed as part of the Academy of Nutrition and Dietetic Foundation’s Future of Food Initiative, a collaboration with Feeding America and the National Dairy Council. The goal of the Hub was to provide a platform to promote high-quality, applicable materials for RDNs and hunger-relief professionals to use with individuals and families facing food insecurity. DANEH was developed to identify high-quality handouts for inclusion in the Hub. While DANEH was developed with the food-bank client/food-insecure audience in mind, its criteria have the potential to be applied to nutrition education handouts for other populations. This article describes the development and reliability testing of DANEH in the context of the Hub.
DEVELOPING THE CHECKLIST A review of the literature was conducted to identify best-practice recommendations for quality indicators that should be included in effective nutrition and health education handouts. Eighteen articles were reviewed.1-19 A list of 21 constructs from the articles was created and organized into five categories: content5-12; behavior focus8,11-14; 1-6,8-12 cultural sensitivity ; written word1-3,5,7-15; and organization/readability.1-3,5,6,8-19 To use DANEH to assess or score a nutrition education handout, one would determine whether each of the DANEH criteria is present or absent. The presence of each construct represents 1 point (1¼present; 0¼absent), for a possible total DANEH score of 21. The first content category (current and accurate information consistent with the 2010 Dietary Guidelines for Americans [DGA]20 and MyPlate21) was considered critical; therefore, it was
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
decided a priori that this category would require a “present” for approval.
Initial Testing and Revisions A DANEH draft, which had been reviewed and tested by the Academy’s Nutrition Solutions Working Group volunteers, was first tested by a group of 21 dietetics interns. The interns identified nutrition education handouts that they believed might be relevant to the food-insecure population (and therefore the Hub). The interns were provided six web links from the US Department of Agriculture’s Supplemental Nutrition Assistance Program Education, Cooking Matters, and state extension agencies/health departments as starting places for their search and were asked to each identify at least 10 unique handouts to which DANEH would be applied. Through this process, >200 handouts were identified. The interns were placed in groups of five and each group was assigned 10 of the identified handouts to assess with DANEH. Based on inter-rater reliability testing and comments from this group, significant changes were made to criteria wording for clarity. In addition, multiple options for reading-level assessment were identified. Based on the interns work with DANEH, 39 handouts were included in the Hub.
VALIDATING THE TOOL The goals of the testing were to determine the inter-rater reliability of DANEH, determine whether amount of training changes inter-rater reliability, determine a cut point for total DANEH score at which handouts can be considered high quality (and therefore be included in the Hub), and determine whether DANEH is valid using several concurrent markers of validity. Using the estimates of Walter and ª 2015 by the Academy of Nutrition and Dietetics.
FROM THE ACADEMY Present (1 point)
Absent (0 points)
Current, accurate, and consistent with Dietary Guidelines for Americans and MyPlate (required ‘yes’ for approval)
The content promotes current, evidencebased recommendations, and is consistent with Dietary Guidelines for Americans and MyPlate.
The content is based on outdated information (including MyPyramid).
Promotes relevant health issues for target audience
The content addresses a relevant health issue of the target audience (eg, reducing sodium for African-American audiences).
The content is not relevant to the target audience (eg, eating organic fruits and vegetables for low-income audiences).
Clear purpose
It is immediately clear what the handout will tell you or how it can help you.
You have to read much of the content before you realize what the handout will tell you or how it can help you.
One or two main themes
The handout contains no more than two main themes. For example, a handout with 10 strategies to shop on a budget has one main theme, shopping on a budget.
The handout contains three or more main themes. For example, a handout that covers tips for healthy snacks, physical activity, and bedtime routines contains three main themes.
Specific examples of desired behavior
The handout clearly instructs a person what to do, and how to do it through specific examples. For example, “Play active games with your kids, like hide-and-go-seek, double Dutch jump rope, or tag.”
The handout provides vague recommendations without specific examples of desired behavior. For example, “Be more physically active.”
Culturally appropriate content for target audience
The handout is intended for a general audience and includes many cultural food and behavior practices. Check “yes” if the target audience is a specific culture, and the handout includes appropriate food and behavior examples for that culture.
The handout is intended for a general audience and does not include culturally diverse food and behavior practices. Check “no” if the target audience is a specific culture and the handout does not represent appropriate food and behaviors for that culture.
Culturally appropriate images for target audience
The handout is intended for a general audience and includes images that respectfully represent many cultures, including the people, places and foods pictured. Check “yes” if the target audience is a specific culture, and the handout includes images of people, places, and foods from that culture.
The handout is intended for a general audience and does not include images that respectfully represent many cultures. Check “no” if the target audience is a specific culture and the handout does not include appropriate images representing that culture.
Simple, common words are used frequently, with limited use of abbreviations, acronyms, and technical jargon.
Complex and unfamiliar words are used frequently or if abbreviations, acronyms, and technical jargon appear excessively.
Content
Behavior Focus
Cultural Sensitivity
Written Word Simple, common words
(continued on next page) Figure. Designing and Assessing Nutrition Education Handouts scoring criteria and defining characteristics to help reviewers determine whether each criterion is present or absent in a nutrition education handout.
May 2015 Volume 115 Number 5
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
817
FROM THE ACADEMY Present (1 point)
Absent (0 points)
Positive messages
Messages focus mostly on the positive behavior desired (eg, “do this”).
Messages focus mostly on the negative behavior to change (eg, “don’t do this”).
Active voice, second person (you/your), conversational tone
The handout refers to the second person (eg, “you/your”), stays in the present tense, and uses a conversational/friendly tone. For example, “Offer healthy and tasty afterschool snacks for your kids, like grapes or cheese sticks.”
The handout refers to the third person, uses the past tense, and/or is too formal. For example, “Parents need to provide nutrientdense foods in between meals for their children and adolescents.”
Repetition of key words and/ or new concepts
Key words and/or new concepts are repeated effectively for reinforcement, or if repeating key words is not necessary (eg, for a very short handout).
Repeating key words and/or new concepts would be effective for reinforcement, but are missing.
Logical order, most important message first
The information is displayed in a logical order, with the most important messages listed first.
The information is disorganized, and/or the most important messages do not appear until later in the handout.
Short paragraphs
Each paragraph is short (60 words or less) and only contains one topic. Check “yes” if the handout does not contain paragraphs.
Each paragraph is long (over 60 words) and/ or contains two or more topics.
Space around headings and text
There is a 1/2-inch margin around the perimeter of the handout and has at least a double space before headings.
The margins are less than 1/2 inch around the perimeter of the handout and/or there is little space before headings.
Blocks of text are leftjustified
Blocks of text are left-justified.
Blocks of text are centered or right-justified.
Bullets, numbers, and tables
Bullets, numbers, and/or tables appear often.
The handout contains mostly text, and few bullets, numbers, or tables.
Several informative headings/subheadings
Several headings/subheadings are present to help identify what the text will tell you next.
Few headings/subheadings are present, and/or if the headings/subheadings do not inform you what the text will tell you next.
Easy to read font
Font is at least 12 point, and is serif (with feet, like Times Roman) or sans serif (without feet, like Arial).
Font is smaller than 12 point, and is fancy or curly.
Important text is bolded or underlined if necessary
Emphasizing text, bolding and underlining are used. Check “yes” if no emphasis of text is needed (eg, for a very short handout).
Emphasizing text uses all caps or italics.
Purposeful and relevant images
Images show instruction or the desired behavior, are placed near relevant text, and include captions when necessary to describe the behavior or emphasize a point.
Images do not show instruction or the desired behavior, are not placed near relevant text, or do not include captions as necessary.
5th-grade reading level
Handout is written at or below a 5th-grade reading level.
Handout is written at a 6th-grade reading level or higher.
Written Word
Organization and Readability
Figure. (continued) Designing and Assessing Nutrition Education Handouts scoring criteria and defining characteristics to help reviewers determine whether each criterion is present or absent in a nutrition education handout.
818
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
May 2015 Volume 115 Number 5
FROM THE ACADEMY colleagues22 assuming a P0 of 0.5 (a¼.05 and b¼.2), we determined that four reviewers would each need to review 33 handouts to detect an interclass correlation (ICC) of 0.6. In order to account for possible incomplete entries, we rounded the number of handouts to 35 and included two groups of four to test two training methods. Handouts for this assessment were selected from the group identified by the dietetics interns but not yet assessed with DANEH.
Identifying and Training Reviewers Eight RDNs who were Kids Eat Right members of the Academy of Nutrition and Dietetics were recruited to participate in the validity and reliability testing out of 132 eligible members based on participation in at least one Kids Eat Right project, years of experience, and activities with the Kids Eat Right program. After two groups of four were determined, the first group was randomly assigned to be trained via webinar. One of the groups of four was assigned to be trained via a 1-hour webinar led by one of the DANEH tool’s developers. The webinar trainer discussed each DANEH criterion and provided the attendees with examples of handouts that did or did not meet each criterion. The second group of four received short, written instructions on using DANEH. Both trainings included instructions on three methods of reading-level assessment: Microsoft Word, the www. readability-score.com website, and a hand calculation. All participants were sent the same group of 35 numbered handouts and a link to an online datacollection tool using SurveyMonkey and were instructed to work independently on their reviews, although project staff were available for procedural questions. The data-collection tool identified each handout by number, and asked for the reviewer’s assessment of the presence or absence of each DANEH criterion. Reading level was the only criterion for which additional information was required; reviewers had to enter the FleschKincaid grade level and which of the three methods they had used to obtain that score. The reviewers also entered their assessment of whether May 2015 Volume 115 Number 5
the handout was high quality in their opinion (to be used as the gold standard for criterion validity), and were able to make comments about each criterion and the handout overall. They had 3 weeks to complete the reviews using DANEH. Institutional Review Board approval was obtained from the American Academy of Family Physicians. Reviewers agreed to a contract that contained the elements of an informed consent. They were each provided a $500 stipend for completing the reviews.
score and the probability that a handout would be rated as high quality. Multiple-rater k and percent agreement were used to determine the consistency in evaluation of each of the criteria by reviewers and the quality assessment, both within the groups (to determine the effect of training method) and in the sample as a whole. ICC was used to assess the consistency in total DANEH score, again within groups and in the entire sample. Statistical analysis was completed using SPSS software (version 20.0, IBM SPSS), except for the multiple-rater k, which was executed in JMP software (version 11, SAS Institute Inc).
Analysis The mean DANEH scores of assessments conducted by each reviewer and reviewer group were compared using analysis of variance (reviewers) or t test (reviewer group). The number of handouts considered high quality by each reviewer was also examined. The mean and range of scores were compared based on the number of reviewers who rated the handouts high quality using analysis of variance, and compared via t test when all highquality ratings were compared to all low-quality ratings. Logistic regression was used to determine the relationship between a 1-point increase in DANEH
RESULTS All eight reviewers completed the reviews in the assigned time period. One review was missing for one of the handouts so the final number included in the sample was 34 handouts, each with eight reviews (four from each group). Table 1 shows the characteristics of the final eight reviewers. Matching on degrees and experience with Kids Eat Right programs (Energy Balance 4 Kids with Play and RD Parent Empowerment Program) was good; however, the years of experience varied between the two groups.
Table 1. Educational characteristics and experience of registered dietitian nutritionist reviewersa participating in the Developing and Assessing Nutrition Education Handouts validation testing
Education and experience
Webinar training Written training group (n[4) group (n[4) nb !
Master’s degree
2
2
Doctoral degree
1
1
Certified Diabetes Educator
2
0
Energy Balance 4 Kids with Play registered 1 dietitian nutritionist
1
RDc Parent Empowerment Program registered dietitian nutritionist
1
1
30 y experience
1
1
a
Reviewers were divided into two groups that were trained by webinar or via written materials (reading group). Number of reviewers in the two groups with each characteristic. c RD¼registered dietitian. b
JOURNAL OF THE ACADEMY OF NUTRITION AND DIETETICS
819
FROM THE ACADEMY Mean total scores given by individual reviewers for the combination of the 34 handouts ranged from 16.7 to 20.3 (Table 2). The mean total score varied among reviewers and significantly between the reviewer groups (P
