Journal of Community Health Nursing
ISSN: 0737-0016 (Print) 1532-7655 (Online) Journal homepage: http://www.tandfonline.com/loi/hchn20
Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program Merry Ann Pearson & Saundra L. Theis To cite this article: Merry Ann Pearson & Saundra L. Theis (1991) Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program, Journal of Community Health Nursing, 8:1, 25-31, DOI: 10.1207/s15327655jchn0801_3 To link to this article: http://dx.doi.org/10.1207/s15327655jchn0801_3
Published online: 07 Jun 2010.
Submit your article to this journal
Article views: 2
View related articles
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=hchn20 Download by: [York University Libraries]
Date: 06 November 2015, At: 15:41
JOURNAL OF COMMUNITY HEALTH NURSING, 1991,8(1), 25-31 Copyright O 1991, Lawrence Erlbaum Associates, Inc.
Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program Downloaded by [York University Libraries] at 15:41 06 November 2015
Merry A n n Pearson, PhD, RN Chicago Saundra L. Theis, PhD, RN University of Illinois at Chicago INTRODUCTION
Comprehensive evaluation has implications for community-based program planning. Many community programs lack an evaluation plan that is initiated at the outset of the program. Founders or participants may believe a program to be effective, but without an ongoing systematic evaluation plan it is difficult to determine outcomes objectively. The use of a comprehensive plan provides the framework and basic direction for the evaluation effort. It incorporates formative evaluation which allows for feedback to participants and stakeholders. Information can be shared on a continuous basis and decision making can be spread over the life of the program. Evaluation is imperative to secure future funding and sustain the viability of a program. Many program planners neglect the evaluation component and realize too late that there is insufficient data to convince funding agencies and supporters that the program has been effective. The development of a comprehensive plan provides an opportunity to explore all aspects of a program. Often this examination process generates questions not previously considered or reconfirms the presence of conditions that are out of the participants' control. The targeting of the evaluation to specific audiences narrows the focus and eliminates frivolous data collection efforts. The purpose of this article is to describe the application of a comprehensive evaluation plan for a community-based respite program. COMPREHENSIVE EVALUATION PLAN
The decision-making evaluation plan is comprehensive. The plan considers: (a) the context into which the program is placed, (b) the processes involved in meeting the objectives of the program, and (c) the actual program outcomes. Data are collected Requests for reprints should be sent to Merry Ann Pearson, PhD, RN, 1501 North State Parkway #23-G, Chicago, IL 60610.
26
Pearson and Theis PROGRAM DESCRIPTION
In~ut
process
1
I I
~utcomes
RATIONALE
REPORTING
Downloaded by [York University Libraries] at 15:41 06 November 2015
II .
\I
DATA COLLECTION
METHODOLQGY
FIGURE 1 Model for an evaluation plan. From "A Protocol for Program Evaluation" by W. L. Holzemer, 1976, Journal of Medical Education, 51(2), p. 103. Copyright 1976 by Holzemer. Reprinted by permission.
in a planned manner to provide thorough descriptions so that future decisions can be made for program improvement. The evaluation plan is based on the countenance framework developed by Stake (1973) who emphasized that an integrated model which includes antecedent conditions and transactions, as well as outcomes, provides a more comprehensive approach to program evaluation than merely examining outcomes. This process for evaluation is supported by Ervin and Chen (1986) who said that ". . . the evaluation focuses on the health program's quality, operation, cost, and outcomes. Based on the stated objectives and data obtained, evaluators judge the worth of the health program" (p. 358). A model based on this comprehensive approach was designed by Holzemer (1976). The model, which shows the seven aspects of the evaluation plan, is shown in Figure 1.
RESPITE CARE PROGRAM To provide an example of the application of the model, a Respite Care Program (RCP) was selected. Respite care is defined by the RCP as the planned, ongoing, or periodic provision of caregiving support services to a dependent elderly person and caregiver in the family home or in an institutional setting (Metropolitan Chicago Coalition on Aging, 1986, p. 1). The RCP, located in a suburban area of Chicago, is nurse managed and coordinates services for caregivers of the frail elderly. The program is collaborative among a community hospital, a visiting nurse association, and a college of nursing. Management of the program is by dual governing bodies (Advisory Board, Steering Committee), representative of the collaborating agencies and the community. The program has been privately funded for 3 years and is currently ending the 2nd year of operation. The RCP provides three major services: a referral center, in-home respite, and institutional respite. The referral center screens and coordinates inquiries about respite care. The in-home component incorporates volunteers who provide up to 4 hr a week of care to allow the caregiver time away from caregiving responsibilities. The institutional component coordinates admission of care recipients to a long-term care facility for a short-term, temporary stay. These
Program Evaluation
27
three services are managed by professional nurses who make the initial assessment of the caregiver and recipient; train, place, and supervise the volunteers; and arrange for the institutional care. APPLICATION OF THE EVALUATION PLAN
Downloaded by [York University Libraries] at 15:41 06 November 2015
Context
The first phase of the evaluation plan was to describe the context'of the RCP. This was a time-consuming process which involved the identification of inputs, processes, and planned outcomes of the program. However, examination of these three aspects aided in the design of the evaluation plan by ensuring that all parts of the RCP had been considered. (Only a few examples of the input, processes, and outcomes are presented.) Input considered antecedent conditions and phenomena that were present when the program started and that were highly relevant to the nature of the program and its outcomes. They were parts of the program over which the participants had little or no control and included the context into which the program was placed. Examples of input in the RCP included the existing philosophies of the collaborating institutions and the makeup of the communities in which the program serves. The program objectives were also part of the input or antecedent conditions that needed to be addressed from an evaluation perspective. Processes included the activities of the program which needed to be monitored in order to deliver the respite services. Examples included the time frame for the program (3 years) and the clients who were to use the services. These clients needed to be described in terms of variety, number, type of service utilized, and interactions with staff. Another process was the identification of existing community resources so that duplication of services could be avoided. The outcomes were the goals for which the program was designed. Examples of outcomes included satisfaction of caregivers with the respite services, and the impact of respite on caregiver burden and quality of life. Another outcome was lack of duplication and improved coordination of community services. Rationale
After the inputs, processes, and outcomes had been outlined to more clearly and objectively describe the program, the rationale for the evaluation was addressed. The basic question asked was "Why is this evaluation being undertaken?" For this RCP, the evaluation was mandated by the funding agency and the three collaborating agencies. The results of the evaluation would determine if there was a need to continue the program. There was interest in knowing if respite care had an impact on caregivers. Were they relieved of some of their caregiving burden? Evaluation would also examine the effectiveness of the collaborative approach and would determine if a nurse-managed center is a viable option for competent, humane, economical respite care.
28
Pearson and Theis
Audience
Downloaded by [York University Libraries] at 15:41 06 November 2015
The third phase was to identify the target audiences for the results of the evaluation. The plan identified the groups who were interested in the evaluation, examined their reasons for being interested, and determined what kind of data were important for their decision-making processes. In this RCP, the primary audiences were the funding agencies and the three collaborating agencies. Others included potential funding agencies and other communities who might be interested in establishing respite care services. Evaluation Questions The next phase in the evaluation process was to delineate the evaluation questions. The questions were phrased so that they specified the focus of concern of the various audiences. The evaluation questions specific to this RCP were divided into the areas of (a) program implementation and collaboration, (b) caregiver/recipient dyad, and (c) community impact. Table 1 presents examples of evduation questions related to these three areas. Specific variables were identified that reflected each evaluation question. Methodology and Data Collection Following question and variable identification, the methodology to answer the questions was delineated. Both quantitative and qualitative methods were found to be appropriate to answer the evaluation questions. A helpful way to operationalize the methodology was to develop a matrix including the time when the data collection should occur, the activity to be measured, the data source, and the measurement techniques. Excerpts from the matrix for this evaluation plan are shown in Table 2. To support the methodology, data collection involved the use of numerous sources and instruments that emanated from the variables identified in the evaluation questions. For example, issues involved with program implementation and collaboration have been identified annually. The sources of this information were the heads of the three agencies, RCP staff, and governing body minutes. The methods to evaluate inTABLE 1 Respite Care Program Evaluation Questions
Program Implementation and Collaboration 1 . Is the collaborative process effective for a nurse-managed respite care program? 2. What are the issues involved with collaboration? 3. Is a nurse-managed delivery system an effective strategy for providing community respite care services? Caregiver and Care Recipient 1 . DOrespite care services impact on caregivers' attitudes, stress, and psychosocial functioning? 2. Do clients use respite care services and are they satisfied with them? Community Impact 1. Has the awareness of respite issues among community leaders been heightened? 2. Is there an improved coordination and range of respite services?
Program Evaluation
29
TABLE 2 Matrix for Evaluation Methodology
Timetable
Activity
Source
Methods/Imtruments
-
Determine wegiver/recipient attitudes/perceptions that might be influenced by exposure to respite care services. Assessment of volunteer satisfaction with role.
Caregivers/ recipients
Moods, perceived burden, quality of life.
Volunteers
Yearly
Examination of demand/ utilization of Respite Program Services
Program staff
Yearly
Determine community potential for supplying volunteers.
Program staff
Years 2 and 3
Identify issues involved with interagency collaboration. Examine what agencies provide to and receive from program.
Years 2 and 3
Determine program staff satisfaction.
Program staff, three collaborating agencies, Governing body minutes. Program staff
Satisfaction questionnaire, evaluation of volunteer education program, focus group interviews, retention log. Content analysis of log of requests for respite services; documentation of use of Respite Program Services. Documentation of recruitment strategies and issues, numbers of volunteers. Interviews with agency heads and staff, content analysis of minutes.
Onset of service and Yearly
Downloaded by [York University Libraries] at 15:41 06 November 2015
Yearly
Semistructured interviews.
teragency collaboration were structured interviews with the agency heads and staff, and content analysis of the governing body minutes. In regard to the caregiverhecipient dyad, attitudes and perceptions that might be influenced by exposure to respite care services (mood, burden, quality of life) were variables to be examined. Data on these variables were collected at the onset of services and annually thereafter. Sources of the data were the caregivers and care recipients themselves. Instruments to collect data were specific to the variables identified and had previously established reliability and validity. Where possible, it is best to use measurement tools which have been administered to the population under study. An example of data collection regarding community impact is an ongoing determination of the ability of the community to provide volunteers for the in-home component of the respite program. Staff provided data about recruitment strategies and issues, and actual numbers of volunteers recruited. Reporting
The last phase of the evaluation will be to prepare a summative report which considers both the data collected and the intended audiences. The final report should include a clear description of the program's goals in regard to the evaluation data. It should also provide judgments about the worth of the program based on the data. This report will be presented to the respective audiences for their deliberation about the future of the program.
Downloaded by [York University Libraries] at 15:41 06 November 2015
30
Pearson and Theis
Formative evaluation. It is important that community-based service programs do not wait until the summative evaluation phase to consider what the data reveals about the effectiveness of a program. Formative evaluation is that which occurs as an ongoing part of a program. As new information is learned, it should be shared with the program staff and governing boards to make necessary modifications. This respite care program was evaluated at the end of the 1st year and several modifications were made based on the evaluation data. In the area of project implementation and collaboration, several problems were identified through structured interviews with the collaborating agencies. The role of the college of nursing and its relationship with the RCP was less clear than that of the other two collaborating agencies. As a result, changes were made in the college's representation at the governing board level. Also, student experiences in the respite care program were encouraged to enhance the relationship with the college. Another example of formative evaluation was a pilot study that was completed to examine the effectiveness of a caregiver intake assessment tool, and several caregiver outcome measures such as burden and quality of life. Results of this pilot study were used to revise the intake assessment tool and to modify the number of data collection instruments.
IMPLICATIONS FOR COMMUNITY HEALTH NURSES (CHNs) This evaluation plan has several implications for nurses who deliver services to clients in the community. One is the realization of the importance of the context within which a program is based. This realization should help nurses to understand what aspects they can control and which are beyond their control. There may be times when nurses wonder why it is important to collect so much data related to a program. Often this collection takes their time and/or the time of their clients. The evaluation plan should be devised with input from practitioners to determine the feasibility of data collection within a particular agency and with a particular client population. Once the plan has been determined, nurses can enhance data collection by encouraging clients to be as cooperative as possible, and by being complete in their own data reporting. The formative evaluation can be helpful in enhancing the delivery of service to the client. Formative evaluation allows for ongoing improvement in service and provides an opportunity for the nurse to witness the results of the data collection efforts. One challenge to measuring the impact of services on caregivers is the limited availability of reliable instruments to measure changes in the caregiving relationship. The nurse may provide additional insight in this area to those evaluating the program that can aid in the revision of instruments to make them more sensitive to the effects of respite on family caregivers. Many times nurses and other health professionals realize what a positive effect a service has on clients, but the service may be discontinued because the evaluation plan was not comprehensive, and therefore, the measures did not capture significant client outcomes.
Program Evaluation
31
REFERENCES
Downloaded by [York University Libraries] at 15:41 06 November 2015
Ervin, N. E., & Chen, S. (1986). Evaluation of family and community health nursing practice. In B. B. Logan & C. E. Dawkins (Eds.), Family-centered nursing in the community (pp. 355-371). Menlo Park, CA: Addison-Wesley. Holzemer, W. L. (1976). A protocol for program evaluation. Journal of Medical Education, 51(2), 101-108. Metropolitan Chicago Coalition on Aging. (1986). Respite care surveys examine effectiveness. Metropolitan Chicago Coalition on Aging Focus, 6(1), 1. Stake, R. E. (1973). The countenance of education evaluation. In B. R. Worthen & J. R. Sanders (Eds.), Education evaluation: Theory andpmctice @p. 106-128). Belrnont, CA: Wadsworth.
ISSN: 0737-0016 (Print) 1532-7655 (Online) Journal homepage: http://www.tandfonline.com/loi/hchn20
Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program Merry Ann Pearson & Saundra L. Theis To cite this article: Merry Ann Pearson & Saundra L. Theis (1991) Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program, Journal of Community Health Nursing, 8:1, 25-31, DOI: 10.1207/s15327655jchn0801_3 To link to this article: http://dx.doi.org/10.1207/s15327655jchn0801_3
Published online: 07 Jun 2010.
Submit your article to this journal
Article views: 2
View related articles
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=hchn20 Download by: [York University Libraries]
Date: 06 November 2015, At: 15:41
JOURNAL OF COMMUNITY HEALTH NURSING, 1991,8(1), 25-31 Copyright O 1991, Lawrence Erlbaum Associates, Inc.
Program Evaluation Application of a Comprehensive Model for a Community-Based Respite Program Downloaded by [York University Libraries] at 15:41 06 November 2015
Merry A n n Pearson, PhD, RN Chicago Saundra L. Theis, PhD, RN University of Illinois at Chicago INTRODUCTION
Comprehensive evaluation has implications for community-based program planning. Many community programs lack an evaluation plan that is initiated at the outset of the program. Founders or participants may believe a program to be effective, but without an ongoing systematic evaluation plan it is difficult to determine outcomes objectively. The use of a comprehensive plan provides the framework and basic direction for the evaluation effort. It incorporates formative evaluation which allows for feedback to participants and stakeholders. Information can be shared on a continuous basis and decision making can be spread over the life of the program. Evaluation is imperative to secure future funding and sustain the viability of a program. Many program planners neglect the evaluation component and realize too late that there is insufficient data to convince funding agencies and supporters that the program has been effective. The development of a comprehensive plan provides an opportunity to explore all aspects of a program. Often this examination process generates questions not previously considered or reconfirms the presence of conditions that are out of the participants' control. The targeting of the evaluation to specific audiences narrows the focus and eliminates frivolous data collection efforts. The purpose of this article is to describe the application of a comprehensive evaluation plan for a community-based respite program. COMPREHENSIVE EVALUATION PLAN
The decision-making evaluation plan is comprehensive. The plan considers: (a) the context into which the program is placed, (b) the processes involved in meeting the objectives of the program, and (c) the actual program outcomes. Data are collected Requests for reprints should be sent to Merry Ann Pearson, PhD, RN, 1501 North State Parkway #23-G, Chicago, IL 60610.
26
Pearson and Theis PROGRAM DESCRIPTION
In~ut
process
1
I I
~utcomes
RATIONALE
REPORTING
Downloaded by [York University Libraries] at 15:41 06 November 2015
II .
\I
DATA COLLECTION
METHODOLQGY
FIGURE 1 Model for an evaluation plan. From "A Protocol for Program Evaluation" by W. L. Holzemer, 1976, Journal of Medical Education, 51(2), p. 103. Copyright 1976 by Holzemer. Reprinted by permission.
in a planned manner to provide thorough descriptions so that future decisions can be made for program improvement. The evaluation plan is based on the countenance framework developed by Stake (1973) who emphasized that an integrated model which includes antecedent conditions and transactions, as well as outcomes, provides a more comprehensive approach to program evaluation than merely examining outcomes. This process for evaluation is supported by Ervin and Chen (1986) who said that ". . . the evaluation focuses on the health program's quality, operation, cost, and outcomes. Based on the stated objectives and data obtained, evaluators judge the worth of the health program" (p. 358). A model based on this comprehensive approach was designed by Holzemer (1976). The model, which shows the seven aspects of the evaluation plan, is shown in Figure 1.
RESPITE CARE PROGRAM To provide an example of the application of the model, a Respite Care Program (RCP) was selected. Respite care is defined by the RCP as the planned, ongoing, or periodic provision of caregiving support services to a dependent elderly person and caregiver in the family home or in an institutional setting (Metropolitan Chicago Coalition on Aging, 1986, p. 1). The RCP, located in a suburban area of Chicago, is nurse managed and coordinates services for caregivers of the frail elderly. The program is collaborative among a community hospital, a visiting nurse association, and a college of nursing. Management of the program is by dual governing bodies (Advisory Board, Steering Committee), representative of the collaborating agencies and the community. The program has been privately funded for 3 years and is currently ending the 2nd year of operation. The RCP provides three major services: a referral center, in-home respite, and institutional respite. The referral center screens and coordinates inquiries about respite care. The in-home component incorporates volunteers who provide up to 4 hr a week of care to allow the caregiver time away from caregiving responsibilities. The institutional component coordinates admission of care recipients to a long-term care facility for a short-term, temporary stay. These
Program Evaluation
27
three services are managed by professional nurses who make the initial assessment of the caregiver and recipient; train, place, and supervise the volunteers; and arrange for the institutional care. APPLICATION OF THE EVALUATION PLAN
Downloaded by [York University Libraries] at 15:41 06 November 2015
Context
The first phase of the evaluation plan was to describe the context'of the RCP. This was a time-consuming process which involved the identification of inputs, processes, and planned outcomes of the program. However, examination of these three aspects aided in the design of the evaluation plan by ensuring that all parts of the RCP had been considered. (Only a few examples of the input, processes, and outcomes are presented.) Input considered antecedent conditions and phenomena that were present when the program started and that were highly relevant to the nature of the program and its outcomes. They were parts of the program over which the participants had little or no control and included the context into which the program was placed. Examples of input in the RCP included the existing philosophies of the collaborating institutions and the makeup of the communities in which the program serves. The program objectives were also part of the input or antecedent conditions that needed to be addressed from an evaluation perspective. Processes included the activities of the program which needed to be monitored in order to deliver the respite services. Examples included the time frame for the program (3 years) and the clients who were to use the services. These clients needed to be described in terms of variety, number, type of service utilized, and interactions with staff. Another process was the identification of existing community resources so that duplication of services could be avoided. The outcomes were the goals for which the program was designed. Examples of outcomes included satisfaction of caregivers with the respite services, and the impact of respite on caregiver burden and quality of life. Another outcome was lack of duplication and improved coordination of community services. Rationale
After the inputs, processes, and outcomes had been outlined to more clearly and objectively describe the program, the rationale for the evaluation was addressed. The basic question asked was "Why is this evaluation being undertaken?" For this RCP, the evaluation was mandated by the funding agency and the three collaborating agencies. The results of the evaluation would determine if there was a need to continue the program. There was interest in knowing if respite care had an impact on caregivers. Were they relieved of some of their caregiving burden? Evaluation would also examine the effectiveness of the collaborative approach and would determine if a nurse-managed center is a viable option for competent, humane, economical respite care.
28
Pearson and Theis
Audience
Downloaded by [York University Libraries] at 15:41 06 November 2015
The third phase was to identify the target audiences for the results of the evaluation. The plan identified the groups who were interested in the evaluation, examined their reasons for being interested, and determined what kind of data were important for their decision-making processes. In this RCP, the primary audiences were the funding agencies and the three collaborating agencies. Others included potential funding agencies and other communities who might be interested in establishing respite care services. Evaluation Questions The next phase in the evaluation process was to delineate the evaluation questions. The questions were phrased so that they specified the focus of concern of the various audiences. The evaluation questions specific to this RCP were divided into the areas of (a) program implementation and collaboration, (b) caregiver/recipient dyad, and (c) community impact. Table 1 presents examples of evduation questions related to these three areas. Specific variables were identified that reflected each evaluation question. Methodology and Data Collection Following question and variable identification, the methodology to answer the questions was delineated. Both quantitative and qualitative methods were found to be appropriate to answer the evaluation questions. A helpful way to operationalize the methodology was to develop a matrix including the time when the data collection should occur, the activity to be measured, the data source, and the measurement techniques. Excerpts from the matrix for this evaluation plan are shown in Table 2. To support the methodology, data collection involved the use of numerous sources and instruments that emanated from the variables identified in the evaluation questions. For example, issues involved with program implementation and collaboration have been identified annually. The sources of this information were the heads of the three agencies, RCP staff, and governing body minutes. The methods to evaluate inTABLE 1 Respite Care Program Evaluation Questions
Program Implementation and Collaboration 1 . Is the collaborative process effective for a nurse-managed respite care program? 2. What are the issues involved with collaboration? 3. Is a nurse-managed delivery system an effective strategy for providing community respite care services? Caregiver and Care Recipient 1 . DOrespite care services impact on caregivers' attitudes, stress, and psychosocial functioning? 2. Do clients use respite care services and are they satisfied with them? Community Impact 1. Has the awareness of respite issues among community leaders been heightened? 2. Is there an improved coordination and range of respite services?
Program Evaluation
29
TABLE 2 Matrix for Evaluation Methodology
Timetable
Activity
Source
Methods/Imtruments
-
Determine wegiver/recipient attitudes/perceptions that might be influenced by exposure to respite care services. Assessment of volunteer satisfaction with role.
Caregivers/ recipients
Moods, perceived burden, quality of life.
Volunteers
Yearly
Examination of demand/ utilization of Respite Program Services
Program staff
Yearly
Determine community potential for supplying volunteers.
Program staff
Years 2 and 3
Identify issues involved with interagency collaboration. Examine what agencies provide to and receive from program.
Years 2 and 3
Determine program staff satisfaction.
Program staff, three collaborating agencies, Governing body minutes. Program staff
Satisfaction questionnaire, evaluation of volunteer education program, focus group interviews, retention log. Content analysis of log of requests for respite services; documentation of use of Respite Program Services. Documentation of recruitment strategies and issues, numbers of volunteers. Interviews with agency heads and staff, content analysis of minutes.
Onset of service and Yearly
Downloaded by [York University Libraries] at 15:41 06 November 2015
Yearly
Semistructured interviews.
teragency collaboration were structured interviews with the agency heads and staff, and content analysis of the governing body minutes. In regard to the caregiverhecipient dyad, attitudes and perceptions that might be influenced by exposure to respite care services (mood, burden, quality of life) were variables to be examined. Data on these variables were collected at the onset of services and annually thereafter. Sources of the data were the caregivers and care recipients themselves. Instruments to collect data were specific to the variables identified and had previously established reliability and validity. Where possible, it is best to use measurement tools which have been administered to the population under study. An example of data collection regarding community impact is an ongoing determination of the ability of the community to provide volunteers for the in-home component of the respite program. Staff provided data about recruitment strategies and issues, and actual numbers of volunteers recruited. Reporting
The last phase of the evaluation will be to prepare a summative report which considers both the data collected and the intended audiences. The final report should include a clear description of the program's goals in regard to the evaluation data. It should also provide judgments about the worth of the program based on the data. This report will be presented to the respective audiences for their deliberation about the future of the program.
Downloaded by [York University Libraries] at 15:41 06 November 2015
30
Pearson and Theis
Formative evaluation. It is important that community-based service programs do not wait until the summative evaluation phase to consider what the data reveals about the effectiveness of a program. Formative evaluation is that which occurs as an ongoing part of a program. As new information is learned, it should be shared with the program staff and governing boards to make necessary modifications. This respite care program was evaluated at the end of the 1st year and several modifications were made based on the evaluation data. In the area of project implementation and collaboration, several problems were identified through structured interviews with the collaborating agencies. The role of the college of nursing and its relationship with the RCP was less clear than that of the other two collaborating agencies. As a result, changes were made in the college's representation at the governing board level. Also, student experiences in the respite care program were encouraged to enhance the relationship with the college. Another example of formative evaluation was a pilot study that was completed to examine the effectiveness of a caregiver intake assessment tool, and several caregiver outcome measures such as burden and quality of life. Results of this pilot study were used to revise the intake assessment tool and to modify the number of data collection instruments.
IMPLICATIONS FOR COMMUNITY HEALTH NURSES (CHNs) This evaluation plan has several implications for nurses who deliver services to clients in the community. One is the realization of the importance of the context within which a program is based. This realization should help nurses to understand what aspects they can control and which are beyond their control. There may be times when nurses wonder why it is important to collect so much data related to a program. Often this collection takes their time and/or the time of their clients. The evaluation plan should be devised with input from practitioners to determine the feasibility of data collection within a particular agency and with a particular client population. Once the plan has been determined, nurses can enhance data collection by encouraging clients to be as cooperative as possible, and by being complete in their own data reporting. The formative evaluation can be helpful in enhancing the delivery of service to the client. Formative evaluation allows for ongoing improvement in service and provides an opportunity for the nurse to witness the results of the data collection efforts. One challenge to measuring the impact of services on caregivers is the limited availability of reliable instruments to measure changes in the caregiving relationship. The nurse may provide additional insight in this area to those evaluating the program that can aid in the revision of instruments to make them more sensitive to the effects of respite on family caregivers. Many times nurses and other health professionals realize what a positive effect a service has on clients, but the service may be discontinued because the evaluation plan was not comprehensive, and therefore, the measures did not capture significant client outcomes.
Program Evaluation
31
REFERENCES
Downloaded by [York University Libraries] at 15:41 06 November 2015
Ervin, N. E., & Chen, S. (1986). Evaluation of family and community health nursing practice. In B. B. Logan & C. E. Dawkins (Eds.), Family-centered nursing in the community (pp. 355-371). Menlo Park, CA: Addison-Wesley. Holzemer, W. L. (1976). A protocol for program evaluation. Journal of Medical Education, 51(2), 101-108. Metropolitan Chicago Coalition on Aging. (1986). Respite care surveys examine effectiveness. Metropolitan Chicago Coalition on Aging Focus, 6(1), 1. Stake, R. E. (1973). The countenance of education evaluation. In B. R. Worthen & J. R. Sanders (Eds.), Education evaluation: Theory andpmctice @p. 106-128). Belrnont, CA: Wadsworth.
