Journal of Consulting and Clinical Psychology 1975, Vol. 43, No. 5, 700-707

Process and Outcome in the Evaluation of Telephone Counseling Referrals Karl A. Slaikeu Steven R. Tulkin University of Texas at Austin State University of New fork at Buffalo David C. Speer Suicide Prevention and Crisis Service, Buffalo, New fork The purpose of this study was to replicate an earlier finding that two process variables ("concreteness" of volunteers' verbal expressions and "motivation" of callers for receiving help) would predict whether callers to a suicide and crisis service would show for appointments for face-to-face counseling. An attempt was also made to operationalize Carkhuff's concreteness construct by use of a new coding system for specificity focus of volunteer talk. In addition, a follow-up questionnaire was administered to shows, no shows, and cancels. It was found that caller motivation ("referral responsiveness") predicted referral outcome but that volunteer concreteness did not, and that the specificityfocus coding system did not correlate with concreteness. Callers' responses on the follow-up questionnaire challenged the assumption that not showing for an appointment is a sign of "failure" in telephone counseling.

The 1960s and early 1970s have witnessed the emergence of numerous telephone counseling services across the United States (Haughton, 1968; Lester & Brockopp, 1973; Musgrave, 1971). There have been some attempts to measure the effectiveness of these centers via follow-up of suicidal callers to determine whether or not the centers have in fact been instrumental in preventing suicide, but such research presents a number of problems, such as inaccurate reporting and recording of suicidal deaths, difficulties in contacting callers for follow-up, lack of control groups, etc. More important, it has become clear that the goals of telephone counseling services are of much greater breadth than the prevention of suicide. Research on the outcome of telephone crisis intervention is notably scarce primarily due to difficulty in the follow-up of callers. Slaikeu, Lester, and Tulkin (1973) took iniThe authors wish to express their appreciation to the volunteers and staff members of the Erie County Suicide Prevention and Crisis Service for their assistance and cooperation in conducting this research, to Lewis Leitner and Laurie Feldman for assistance in coding, and to the Institutional Funds Committee of the State University of New York at Buffalo for financial assistance. Requests for reprints should be sent to Karl A. Slaikeu, Counseling-Psychological Services Center, University of Texas at Austin, Austin, Texas 78712. 700

tial steps toward examining process and outcome in telephone crisis intervention in an exploratory content analysis of taped "referral" calls to a suicide prevention and crisis service. Showing up for the face-to-face interview was selected as an outcome measure, and a sample of 40 calls—a call resulting in "show" and a call resulting in "no show" for each of 20 volunteers—was coded on a number of dimensions. Shows appeared to be more "motivated" to receive help than no shows, as evidenced by a significant difference on a "referral responsiveness" scale. The coding of this scale is presented later. A second factor that correlated with showing for an appointment was the level of volunteer "concreteness" (Carkhuff, 1969), which is congruent with the optimal style for a worker attempting to help a person in crisis—that is, playing a directive, active, "focusing" role. The present investigation was an attempt to replicate and expand the findings of Slaikeu et al. (1973) regarding referral responsiveness and concreteness as predictors of show versus no show. An additional outcome category was also added: those clients who called to "cancel" their appointments. Also, an attempt was made to operationalize the concreteness construct using guidelines found in the literature (Carkhuff, 1969). A

EVALUATION OF TELEPHONE COUNSELING REFERRALS new coding system for "specificity focus" of counselor talk was developed. This endeavor came from the realization that in spite of the attractiveness of the concreteness construct in crisis intervention the scale presents problems for scientific research in that it is notably vague, as evidenced by low interrater reliability in some studies (Carkhuff & Truax, 1965), and thus has only minimal utility for training of counselors. Finally, in an attempt to broaden our understanding of the outcome category of show/no show/cancel, a follow-up questionnaire was administered. The questionnaire was a variation of one used by Speer and Schultz (1975) and focused on callers' assessment of the initial call during which the referral was made, as well as problem resolution since the initial call. It was hypothesized that callers who showed for appointments would rate the volunteers' performance in the initial call more favorably than callers who did not show. Follow-up data were also used to determine if ratings of clinical effectiveness (Carkhuff, 1969) were correlated with clients' own perception of counselor effectiveness along a variety of dimensions. The Carkhuff scales of empathy, positive regard, genuineness, concreteness, and self-disclosure have been used extensively in telephone counseling research (Bleach & Claiborn, 1974; Knickerbocker & McGee, 1973). The present investigation examined correlations between ratings of counselors on these five dimensions and callers' evaluations of counselors from the follow-up questionnaire. In particular, it was hypothesized that the Carkhuff empathy ratings would correlate positively with callers' evaluation of counselors' "empathy" and "understanding of the problem" (two follow-up scales) and that Carkhuff concreteness ratings would correlate positively with callers' evaluations of counselors' "summarization of the problem," "surveying of resources," and "development of action plan" (three follow-up scales). METHOD Sample The sample included all referral calls to the Erie County Suicide Prevention and Crisis Service during one month. "Referral calls" were those during which

701

the caller was given an appointment for face-to-face counseling at the suicide prevention and crisis service. In order to determine the representativeness of the sample, a comparison was made between referral characteristics during the sample month and the previous year, specifically: (a) number of completed calls; (b) number of referrals; (c) show rate (shows divided by shows plus no shows); (d) number of volunteers who made referrals;. (e) number of referrals per volunteer; and (f) show rate of individual volunteers. There were no significant differences between the sample month and the previous year on Dimensions d, e, and f. There were significant differences, however, on the other dimensions. There were fewer calls to the suicide prevention and crisis service during the sample month than in previous months (1,302 vs. 1,713, 2 = —1.77, p < .04). This difference accounts for the fact that there were also fewer referrals (89 vs. 124, z = —1.54, p < .06). However, when percentage of referrals per month was calculated (number of referrals divided by number of calls), the sample month's percentage (1%) fell within the range for the previous year (range^ 6%-ll%, M—7%), supporting the representativeness of the sample. It is noteworthy that the percentage of callers who showed for their appointments during the sample month was significantly less than in previous months (46 vs. SS, z = — 2.04, /> < .10. **p < .05.

contacted, all said they would be willing to fill out the follow-up questionnaires. However, only 70% of the no shows who were contacted and 93% of the cancels who were contacted returned the questionnaire.3 Results comparing shows, no shows, and cancels on follow-up scales are presented in Table 5. The hypothesis that shows would evaluate volunteer performance in the initial call more favorably than no shows was tested in two ways. When shows and no shows were compared on Scales 1-7 of the questionnaire, no significant differences between groups were found, suggesting that shows and no shows do not differ in the evaluation of volunteer performance on seven relatively specific di3 While some of these percentages are lower than one might wish, they are relatively high considering the population and the method of data collection. The fact that a number of questionnaires were not returned from the no shows, however, does limit the generalization of findings. It is noteworthy that no shows as a group distinguished themselves from cancels in that they were more difficult to contact for follow-up and less cooperative in returning questionnaires, even though procedures in dealing with both groups were identical.

EVALUATION or TELEPHONE COUNSELING REFERRALS mensions. Shows, however, did rate the "overall helpfulness" of the call (Scale 8) higher than no shows. Cancels fell midway between shows and no shows in their evaluations of the overall helpfulness of the call. The correlation between Scale 8 and an average of Scales 1-7 was calculated (r = .68, p < .001). Thus, the data suggest that while specific volunteer behaviors do not differentiate between the groups, when given an opportunity to evaluate the global helpfulness of the call, shows rated the call more favorably than no shows. The remainder of the follow-up scales, plus several open-ended questions, were included to gather descriptive data on shows, no shows, and cancels. It was found that the three groups did not differ on how badly they felt immediately prior to calling the center (Scale 11), how they felt immediately after the call (Scale 9), and how they felt at the time they filled out the questionnaire (Scale 10). A greater number of shows than no shows and cancels said that the decision to show/not show/cancel was not difficult for them, X 2 (2) = 11.75, p < .01. Shows also reported that they were sure of their decision (i.e., knew that they would come in) at the time of the call, as opposed to deciding (a) sometime between the call and the appointment or (b) at the time of the appointment. No shows and cancels, on the other hand, indicated that their decision was not made until sometime between the call and the time of the appointment, x 2 ( 4 ) = 21.15, p < .001. No shows and cancels were also asked an open-ended question: "For what reasons did you finally decide not to come in?" It is interesting to note that a number of no shows and cancels reported going to another agency (e.g., Alcoholics Anonymous), that "things got better," etc. The ^V for this item is too small to analyze statistically, but the responses may be useful in generating questionnaire items for future follow-up research. Implicit in many of the responses is the idea that these callers were working on their problems even though they did not come into the center for the appointment. The implications of this finding for the meaning of the outcome measure show versus no show are discussed below.

705

A final objective of the present study was to examine the relationships between follow-up ratings of counselor performance and clinical effectiveness ratings based on Carkhuffs (1969) system. The hypothesized correlations regarding Carkhuff's empathy and concreteness ratings and follow-up scales were not supported. An average of Scales 1 (empathy) and 2 (understanding of the problems) did not correlate with volunteers' empathy rating (r = —.08), nor did an average of Scales 3 (summarization of the problem), 4 (surveying resources), and 5 (development of action plan) correlate with therapist's concreteness rating (r=~.\2). Correlations between each of the individual scales and the related rating from the Carkhuff system were also nonsignificant. DISCUSSION A major impetus for the present investigation was an attempt to replicate the finding of Slaikeu et al. (1973). The present results corroborated the earlier finding that callers high in "referral responsiveness" are more likely to show for the appointments, but the finding that shows would have higher levels of volunteer concreteness was not replicated. Another major goal of this investigation was to operationalize concreteness, that is, to further understand this construct by exploring its correlates. This endeavor also produced no significant findings. The newly developed specificity-focus coding system—a rigorous attempt to operationalize concreteness based on guidelines found in the literature and on conversations with raters who use the Carkhuff system—did not produce significant correlations with concreteness ratings. At this point then, the only reliable initialcall difference between show and no-show referrals is referral responsiveness of the caller, or more simply, who mentions the referral question first and when it is asked. One possible explanation for the failure to replicate the concreteness finding was the different sampling procedures. The present sample consisted of 1-month's referral calls, whereas the sample used by Slaikeu et al. (1973) included the first 20 volunteers, who produced a tape-recorded show and no-show referral. The earlier "matched calls" design

706

K. A. SLAIKEU, S. R. TULKIN, AND D. C. SPEER

resulted in a sample that included more experienced volunteers (in terms of hours worked) since their chances for inclusion in the sample by getting a show and no-show call were greater than those who worked fewer hours per month. These differences in sampling might suggest that the concreteness finding holds for experienced volunteers but breaks down when a wider spectrum of volunteers is included. An attempt was made to test this hypothesis on the present sample by examining those volunteers who had both a show and a no-show referral, that is, a sample similar to that originally used by Slaikeu et al. In this small sample (eight pairs), shows did not differ from no shows on volunteer concreteness. While a small N precludes viewing this as a valid test of the hypothesis, it does lend support to the idea that failure to replicate the concreteness finding may be due to something other than differences in sampling procedures. The authors' experience with the concreteness coding system suggests that the failure to replicate may be due to the vagueness of the construct. In developing the specificityfocus coding system, the first author interviewed coders trained by Carkhuff. It was difficult to determine precisely what went into a rating. One rater stated that it was "hard to put into words; you have to get a feel for it." For the purposes of behavioral science (especially regarding behavioral definition and reliability), this is a troublesome state of affairs. Further, the specificity-focus coding system did not produce significant correlations with concreteness ratings, nor did the follow-up scales. Also, Tapp, Slaikeu, and Tulkin (1974) found that only two of nine items of the Fowler Technical Effectiveness Scale (which examines specific information elicited by volunteers) correlated significantly with concreteness. Finally, Muehlberg, Pierce, and Drasgow (1969) found high positive correlations between the five scales of empathy, positive regard, genuineness, concreteness, and self-disclosure. A therapist who was high on one scale tended to be high on the others, suggesting that there was one major factor underlying the five scales. These data, then, raise questions as to precisely what concrete-

ness is and how it is to be coded. While the construct has face validity and appeal, it has proven most elusive in terms of the specificity that needs to accompany behavioral research. The original impetus for developing the specificity-focus coding system came from interest in operationalizing Carkhuff's concreteness construct. While this effort was not fruitful, it should be noted that the specificity-focus system may still be useful in crisis intervention research. The system presents a codable means of examining a range of volunteer speech behaviors that is frequently mentioned in the literature as important in crisis intervention. The system allows for coding of "focusing techniques" along four dimensions. Use, of this coding system in future research may contribute to the understanding of volunteer "effectiveness" in the early stages of crisis interventions. Results from follow-up of shows, no shows, and cancels suggest that these outcome categories are more complex than has been previously understood. While shows gave higher ratings of the "overall helpfulness" of the call than the other two groups, some of the no shows and cancels reported that they either took some specific action to solve their problem or that they found help at another agency. It is possible, then, that the actions of no shows and cancels might also reflect "successful" telephone crisis intervention. This interpretation of preliminary findings must remain tentative until there is crossvalidation on other samples. According to available data, it cannot be concluded that showing for an agreed-upon appointment is necessarily a sign of successful telephone crisis intervention, or conversely that not showing is a sign of "failure." Adequate testing of hypotheses requires greater representation of no shows and cancels in samples than in the present study. Designing telephone follow-up inquiries would be one way of doing this, since it was found in the present study that higher percentages of no shows and cancels could be reached by telephone than would return questionnaires. The fact that few researchers had attempted follow-up of callers and that none had elicited feedback from no shows and cancels on evaluation of the helpfulness of a

EVALUATION OF TELEPHONE COUNSELING REFERRALS specific intervention was a primary impetus for our follow-up attempt. It was interesting to note that while no shows were difficult to follow-up cancels were relatively easy to contact and most cooperative in returning questionnaires. Follow-up of this population of callers is apparently quite possible. Much telephone counseling literature purports to define "effectiveness" of telephone therapy without tying notions of effectiveness to outcome, apparently assuming that these definitions of effectiveness are perceived similarly by callers (Knickerbocker & McGee, 1972; Heaton, Powell, & Ashton, Note 1). The fact that some callers did not respond to follow-up inquiries, or could not be contacted, plus the fact that all were recalling a telephone conversation of several days prior may reduce the validity of the callers' evaluation of "what happened during the call." Also there are problems in assuming the helpfulness of a particular aspect of the call simply because the caller reports it as such. Even less valid, however, is research that purports to measure volunteer effectiveness without consulting clients as to what they feel was helpful or effective. Caller feedback about the intervention is important. While the present exploratory follow-up did not produce conclusive results, it did demonstrate the feasibility of such research and highlights directions for future studies. The correlation between callers' response on overall helpfulness of the call and their evaluation of volunteers on Speer et al.'s (1975) seven scales lends some support to the idea that these behaviors are perceived by callers to be appropriate and helpful. The fact that some no shows and cancels stated (in response to an open-ended question) their reasons for not coming in were that "things got better" or they did something about the problem, provides new ideas for questionnaire items in subsequent follow-up research. REFERENCE NOTE 1. Heaton, M. E., Powell, E. R,, & Ashton, P. T. Multiple roles of crisis center workers. Paper presented at the meeting of the Southeastern Psychological Association, Atlanta, April 1972.

707

REFERENCES Bleach, G., & Claiborn, W. L. Initial evaluation of hotline telephone crisis centers. Community Mental Health Journal, 1974, 10, 387-394. Carkhuff, R. R. Helping and human relations: A primer for lay and professional helpers (Vol. 2): Practice and research. New York: Holt, Rinehart & Winston, 1969. Carkhuff, R. R., & Truax, C. B. Training in counseling and psychotherapy: An evaluation of an integrated didactic and experimental approach. Journal o) Consulting Psychology, 1965, 29, 333-336. Collingwood, T. R. Differential effects of large and small group training and retraining on the longterm retention of jacilitative communication. Unpublished doctoral dissertation, State University of New York at Buffalo, 1970. Gurney, D. V. Client listening as a factor in perceptual reorganisation. Unpublished doctoral dissertation, State University of New York at Buffalo, 1970. Haughton, A. B. Suicide prevention programs: The current scene. American Journal of Psychiatry, 1968, 124, 1692-1696. Knickerbocker, D, A., & McGee, R. K. Clinical effectiveness of volunteer crisis workers on the telephone. In D. Lester & G. W. Brockopp (Eds,), Crisis intervention and counseling by telephone. Springfield, 111.: Charles C Thomas, 1973. Lester, D. Steps toward the evaluation of a suicide prevention center: Part one. Crisis Intervention, 1970, 2, 42-45. Lester, D., & Brockopp, G. W. Crisis intervention and counseling by telephone. Springfield, 111.: Charles C Thomas, 1973. Muehlberg, N., Pierce, R., & Drasgow, J. A factor analysis of therapeutically facilitative conditions. Journal of Clinical Psychology, 1969, 25, 93-95. Musgrave; L. C. Hot lines take the heat off. American Journal of Nursing, 1971, 71, 756-759. Nichols, R. C. MAPS; A multiple analysis program system for behavioral science research. Buffalo, N.Y.: Behavioral Science Research Associates, 1974. Slaikeu, K. A., Lester, D., & Tulkin, S, R. Show vs. no show: A comparison of referral calls to a suicide prevention and crisis service. Journal of Consulting and Clinical Psychology, 1973, 40, 481486. Speer, D. C., & Schultz, M. An instrument for assessing caller-reported benefits of calls to a telephone crisis service. Journal of Consulting and Clinical Psychology, 1975, 43, 102. Tapp, J., Slaikeu, K. A., & Tulkin, S. R. Toward the evaluation of telephone counseling services. American Journal of Community Psychology 1974, 4, 357-364. (Received February 3, 1975)

Process and outcome in the evaluation of telephone counseling referrals.

Journal of Consulting and Clinical Psychology 1975, Vol. 43, No. 5, 700-707 Process and Outcome in the Evaluation of Telephone Counseling Referrals K...
726KB Sizes 0 Downloads 0 Views