Developing a quality assurance program for online services* By Anne Wood Humphries, M.L.S. Head Reference Librarian Gretchen V. Naisawald, M.L.S. Assistant Director for Public Services
The Claude Moore Health Sciences Library Box 234 University of Virginia Health Sciences Center Charlottesville, Virginia 22908
A quality assurance (QA) program provides not only a mechanism for establishing training and competency standards, but also a method for continuously monitoring current service practices to correct shortcomings. The typical QA cycle includes these basic steps: select subject for review, establish measurable standards, evaluate existing services using the standards, identify problems, implement solutions, and reevaluate services. The Claude Moore Health Sciences Library (CMHSL) developed a quality assurance program for online services designed to evaluate services against specific criteria identified by research studies as being important to customer satisfaction. These criteria include reliability, responsiveness, approachability, communication, and physical factors. The application of these criteria to the library's existing online services in the quality review process is discussed with specific examples of the problems identified in each service area, as well as the solutions implemented to correct deficiencies. The application of the QA cycle to an online services program serves as a model of possible interventions. The use of QA principles to enhance online service quality can be extended to other library service areas.
Quality service is a much-publicized selling point in commercial industries and is often the factor that stays in the buyer's mind when evaluating how well a company did in providing a superior product. Li-
As more libraries find it necessary to recover costs through fee-based services, however, library services may also be viewed as consumer products, and the issue of responsibility for providing a quality service then becomes significant. braries have traditionally provided customer services, but such assistance has not been viewed as a product in the same sense that a manufactured item is viewed * Based on a paper presented May 23, 1989, at the Eighty-Ninth Annual Meeting of the Medical Library Association, Boston, Massachusetts.
Bull Med Libr Assoc 79(3) July 1991
as a product. As more libraries find it necessary to recover costs through fee-based services, however, library services may also be viewed as consumer products, and the issue of responsibility for providing a quality service then becomes significant. Online bibliographic searching is one such service that is often accompanied by fees and surcharges, and needs to be justified in terms of cost and quality.
BACKGROUND Evaluating library services Evaluating any library service is difficult because it requires judgment about processes that are not readily quantified. Methods for measuring performance in libraries include user surveys, unobtrusive studies, quantitative analysis of collected statistical data, and individual performance evaluation. Cronin pointed out that no matter what method of evaluation is used, the quality of service is always defined in terms of 263
Humphries and Naisawald Figure 1 The quality assurance cycle SELECr SUBJECIT
_
EVALUATE SERVICES USING STANDARDS
ESTABLISH MEASURABLE STANDARDS
l
l REEVALUATE | SERVICES
|
IMPLEMENT
SOLUTIONS
|
|
IDENTIFY PROBLEMS
the needs of the library user and the skills of the staff in assessing and meeting those needs [1]. The level of service required depends entirely on the circumstances of the individual library and its particular clientele and, therefore, is highly variable. Fredenburg discussed establishing a quality assurance (QA) program in special libraries and related it to the development of specific performance plans [2]. QA was described as a practical tool for proactively establishing goals for the library, justifying the library's worth to administration, and getting everyone in the organization, from the support staff to the administration, involved in problem recognition and resolution. Blood gave examples of user survey questions recommended by an American Library Association (ALA) committee formed to evaluate the effectiveness of online search results [3]. Blood discussed factors that can affect the user's perception, including previous use of online services, comparisons with manual searches, awareness of the capabilities of database searching, and lack of criteria on which to base evaluation of the search intermediary. Blood concluded that if a survey is used, it must be short, simple, anonymous, and convenient. The survey should have clear goals for what is to be evaluated and should result in modifications to improve service. If no clear goals for raising the level of service are established, the evaluation process is futile. In predicting the cost-effectiveness of an evaluation effort, Lancaster warned: "Thorough evaluation tends to be expensive, and can only be justified if the evaluation program is likely to lead to significant improvement in the performance of the system" [4]. Libraries also attempt to maintain quality output for online search services through peer review or inhouse training programs. Glunz described weekly meetings of online searchers held to discuss searches and identify ways to improve search results through collegial examination of search strategies judged by specific criteria [5]. Glunz emphasized the need to 264
make the peer review program ongoing, nonthreatening, and a supplement to formal training classes. Similarly, Matthews and Jackson described conducting in-house training programs and using mentors to complement formal training at vendor workshops [6-7]. Jackson also noted that "surveys of users are often futile and are likely to detect problems only when a searcher is grossly deficient ... if it is impossible to measure the quality of output, we must do more to regulate the quality of input" [8]. Training programs that allow novice searchers to test their skills through practice exercises and to work with experienced searchers establish a solid base for quality searching. These studies indicated that issues of establishing, improving, and maintaining quality are prominent concerns in the management of library services. However, there is no discussion of the development of a QA program designed to monitor the interrelated service features of online services, including not only the technical quality of the end product or list of references, but also service quality in the process of delivering online services.
The QA cycle QA is a routine concept in production-line processes where output can be well controlled. QA programs have also been developed in service industries, including health care delivery systems. Self reviewed the application of the QA process in health sciences libraries [9]. The basic steps in the QA cycle are (Figure 1): * Select the subject for review. Limit the focus of the review to crucial aspects of service delivery. * Establish measurable standards. Select performance standards based on achievable goals that can be measured for their effectiveness in providing quality service. Standards must be relevant, understandable, measurable, and achievable. * Evaluate existing services using the standards. Identify various components of existing services and compare them to the goals that have been established. * Identify problems. Problems exist when actual outcome or compliance is below expected performance levels. Causes of the problems may involve many factors including gaps in staff knowledge or skills, problems with policies or procedures, environmental obstacles, and lack of materials or equipment. * Implement solutions. Once problems have been identified, devise positive solutions to correct the deficiency. The staff must be committed to the QA process to implement the solutions successfully. * Reevaluate services. A successful QA program is ongoing and periodically reaudits its selected criteria. Self pointed out the positive impact a QA program can have: "Skillful use of the quality assurance proBull Med Libr Assoc 79(3) July 1991
Quality assurance cess can upgrade services, maintain quality service, provide a mechanism for goal setting, make library staff more responsive to user needs, and influence behavior practices in the library" [10].
Determinants of customer services quality Traditionally, libraries have been evaluated by quantitative measures including size of collections, number of users, circulation figures, allocation of resources and personnel, and similar aspects. None of these factors guarantee that library services have the quality of excellence. Berry pointed out that the definition of service quality is the correspondence of performance or perceived performance to the customer's specifications [11]. Usually, the most important performance values are those most dependent on the human factor, which is difficult to control. In the study conducted by Berry and colleagues to evaluate important service features in banking, credit card, and machine repair businesses, the five elements most frequently reported by customers in order of importance were: * Reliability-the ability to perform dependably, accurately, and consistently to carry out what was promised to the customer * Responsiveness-the willingness to provide prompt service and to help customers * Assurance-evidence of employees' knowledge of the product, courtesy, and ability to convey trust and confidence * Empathy-provision of caring, individualized attention to customers * Tangibles-physical facilities, equipment, appearance of personnel [12]. All of these attributes are dependent on the behavior of personnel, making it critical that control over service be as close to the customer as possible and that personnel be empowered to attain quick and effective problem resolution. The QA cycle not only identifies and solves problems, it also provides the
The QA cycle not only identifies and solves problems, it also provides the platform to proactively create an environment that corrects deficiencies before they become problems. platform to proactively create an environment that corrects deficiencies before they become problems. F. J. McDonald, a president of General Motors, noted: "The environment has to be one that is error-friendly. Quality is error driven. It depends on each individual's ability to recognize error trends and to make corrections quickly" [13]. Employees must not feel the Bull Med Libr Assoc 79(3) July 1991
necessity to cover up errors in order to preserve their jobs. Shaughnessy proposed that if an organization is to be successful in consistently delivering service of high standards, quality must become a valued characteristic of every customer interaction [14]. Shaughnessy further described the type of team building and group cohesiveness necessary in the process: "Every culture develops unwritten norms which influence the behaviors and attitudes of its members ... this value system or culture determines how well the organization responds to challenges and opportunities" [15]. The QA process must be an integral part of the organization's mission and the employees' personal performance standards. Cronin pointed out that benefits of the performance measurement process include increased staff discussion and involvement, better staff awareness of major public services objectives within the library, increased user awareness of library activities, expanded accountability, and the requirement that ambiguous terms like "high quality service" be clearly defined [16].
QA FOR ONLINE SERVICES The Claude Moore Health Sciences Library (CMHSL) began a review of its online services program in 1987 as part of its long-range plan. The review was instituted not because of a concern that the program was deficient, but because of a commitment on the part of the library to provide high quality services rather than just acceptable ones. In the long-range plan several strategies were identified to improve online searches, including auditing each searcher's output, comparing results against a checklist of the components of a good online search, and redesigning forms. However, when work began on developing these strategies, it became clear that technical proficiency in online searching represented only one component and that online services encompass a whole range of service elements in an interrelated process. At this point the concept of creating a QA program began to form, and a committee composed primarily of online searchers was appointed within the Public Services Division to develop the program. The committee operated on the premise that online services would move beyond the mechanical output of a product to a more professional service that emphasized the consultative relationship between the user and the search analyst. Quality determinants The committee selected the library's online search services as the subject of the QA cycle. Developing measurable criteria for quality services proved more challenging and complex than originally anticipated. In developing criteria for online services, the ele265
Humphries and Naisawald
Table 1 Examples of quality determinants for online services Reliability/consistency/competence/credibility * standard training program * current awareness mechanisms * search sample audit * staff development plan * customer satisfaction with results _- .r . . ._ . .
. .. .
.. .
._...._-
* promised tumaround time * priority for patient care * daily searching schedule Access/approachability * information sheet for requesters * user-friendly request forms * search status log Courtesy/communication/understanding customer needs * guidelines for search interview * search summary letter for requester * information on related services
User feedback. No method had been developed to evaluate the quality of search results for each individual searcher. From 1983 to 1987, the library attached a search evaluation form to all search results generated by the department. This form served as a feedback measurement for extremes of praise and complaint but did not give an adequate systematic analysis of searchers' output based on measurable standards.
Security
Program management. The responsibilities for online search services lay solely with the head reference librarian, who had other obligations that competed with the responsibilities of online searching. Also, the library had an informal policy of rerunning searches at no charge if the results were unsatisfactory to the user, but the policy was not actively advertised to clientele.
* confidentiality of search topic * professional presentation of results
Physical factors * ergonomic computer equipment * high-quality paper * privacy for search appointments
ments of quality identified in customer service studies by Parasuraman [17] were gathered into related subject groups: * Reliability/consistency/competence/credibility * Responsiveness/ timeliness * Access /approachability * Courtesy/communication/understanding customer needs * Security * Physical factors. Once the criteria were examined and found to be appropriate and applicable to an online services system, the committee began the process of evaluating the library's online searching program using these determinants of quality. The next steps were to identify problems and to implement solutions. The problems and solutions were particular to this library and were not comprehensive, but illustrate the types of intervention that are possible (Table 1).
RELIABILITY/CONSISTENCY/COMPETENCE/ CREDIBILITY The first criteria are concerned with the development of equivalent skills for all searchers and the consistent application of policies in the delivery of online search results.
Problems Training. The review indicated that the department lacked a uniform base-level training program for all 266
searchers. While searchers were given opportunities to attend courses and updates on online searching, there was no formal plan or program that systematically coordinated these courses on an individual and departmental level. Similarly, there was no mechanism in place to ensure that searchers kept up-to-date except through self-initiated development and journal reading.
Solutions Uniform training. To solve training problems, a detailed plan outlining the steps of a base-level training program for all searchers was produced. This plan stipulates that all searchers attend the initial National Library of Medicine (NLM) online training course within the first six months of training. It also required attendance at specified database vendor workshops for frequently searched databases and systems in the first year. In addition to formal training, an in-house training program was established, with opportunities for practice sessions and gradual independent searching once experienced searchers observe that new searchers are uniformly fulfilling the requirements of the quality searching checklist compiled by the QA Committee.
Searcher audit. One of the key monitoring processes to come out of the QA review was a regular sampling period for online searches. Schnaitman described the details of the CMHSL review process, including twiceyearly periods during which searchers print out all searches [18]. The searches are then reviewed according to the checklist of quality search criteria developed by the QA Committee through literature reviews and staff input. Each searcher meets with the online coordinator to discuss searching strengths, weaknesses, and areas for development. The results Bull Med Libr Assoc 79(3) July 1991
Quality assurance
of the review meeting are summarized in a memorandum, which includes a proposed training program for the next six months, including formal courses. Searchers are also scheduled to present in-house review sessions of databases or systems for staff meetings. The memo serves as an adjunct report for performance evaluation as well as a management planning tool for scheduling staff development activities.
Each searcher meets with the online coordinator to discuss searching strengths, weaknesses, and areas for development. The results of the review meeting are summarized in a memorandum, which includes a proposed training program for the next six months, including formal courses. Current awareness. Online searching became a regular agenda item at department meetings to update staff on new databases, changes in systems, new features, and search tips. A staff member is designated to regularly print and route important system news. Electronic mail was recommended as an additional medium to alert searchers to important online searching issues. Program coordinator. The committee further recommended establishing a position for online services coordinator to ensure necessary managerial oversight and constant and focused attention to the issues and policies relevant to online services. In addition, a patron complaint policy was written to provide uniform guidelines for determining when a search should be rerun at no additional charge to the patron. The policy is advertised on all search request forms.
Solutions Promised delivery. A standard policy of two working days for turnaround time was established for all searches from primary clientele submitted during the business week, and a same-day turnaround policy was established for urgent patient care searches.To accommodate varying search loads, searching periods are scheduled throughout the day for searchers to complete any pending search requests. A search log records the date and time when the search is received, and the date and time when the search is completed and the patron is notified.
ACCESS/APPROACHABILITY Access and approachability relate to the ease with which requesters can submit search requests, and the help they receive in communicating their search need and obtaining results. Problems Search requests. A better mechanism was needed to assist patrons who wished to request searches, particularly after the reference desk was closed. The search form was not user friendly or self-explanatory. The library had no informative fact sheet covering general information about computer searches and potential costs that could be given to requesters when the department was not staffed. Finally, there was no systematic method of determining the current status of a search once it was in process.
Problems Turnaround time. The existing informal policy for
Solutions Improved forms. An informational checklist to accompany search request forms was written. The checklist gives patrons who request searches after hours tips on improving their search requests. Because no search interview is possible, it is helpful to suggest that patrons write topics in sentence form, be as specific as possible, include synonyms, and list known articles. The search request form itself was revised to make better use of space, following a simple and logical format. Frequently used limiting criteria, such as English, human, review, and years to be covered, are incorporated into check-off boxes for the benefit of both the requester and the searcher. Reference desk staff routinely check during the search interview to make sure that all required parameters have been identified.
turnaround time for searches was one to two working days, with priority given to patient care questions. However, no promised turnaround time was advertised and the priority of patient care was not clearly defined. Searches were processed whenever time was available to individual staff members, with no systematic schedule.
Search status log. To track the status of a search request at any point in the process, a search request log was designed for following the progress of the search. The log indicates the time of the initial request; the time, date, and initials of the searcher running the search; and the date the search was completed. The
RESPONSIVENESS/TIMELINESS These criteria relate to the customer's expectation that the searcher will provide search results promptly and will try to accommodate the requester's needs and time schedules.
Bull Med Libr Assoc 79(3) July 1991
267
Humphries and Naisawald
log also indicates whether the search is for patient care, whether a search has been picked up by the requester, and whether a search has been mailed to an outside client.
COURTESY/COMMUNICATION/ UNDERSTANDING CUSTOMER NEEDS Consumers rate caring, individualized attention highly. The courtesy and communication offered to search requesters has an important impact on their perception of quality service.
Problems Search interview. The department lacked a standard guide for conducting a search interview. Searchers relied on their own knowledge and training but did not have a common method for best ascertaining the requester's information need. In addition, at the completion of the search, any comments or concerns about search output were informally conveyed to patrons, either by searchers' handwritten notes or verbal comments.
Solutions Interview guidelines. To remedy these concerns, standard guidelines for the search request interview were included in a policy and procedure manual. Important points for the interview were determined through literature review. The search interview was incorporated into a department staff development program, which required reading articles about search interviews [19-20]. A copy of the search request checklist is readily available at the reference desk for review.
Search summary letter. A standardized memo was written to enable searchers to produce search summaries for patrons on their personal computers. The search memo is provided for every search and in-
Any concerns or possible alternative strategies are discussed in the memo to fully inform the patron of the issues encountered by the searcher when running the search. This provides the patron with a better understanding of the search process and offers an opportunity for feedback and improved results. cludes information on databases searched, years covered, and the search method used to obtain the final results. Any concerns or possible alternative strate268
gies are discussed in the memo to fully inform the patron of the issues encountered by the searcher when running the search. This provides the patron with a better understanding of the search process and offers an opportunity for feedback and improved results. The summary also includes information about related library services such as document delivery, interlibrary loan, and regular updates on a search topic.
SECURITY "Security" refers here to the patron's trust in the confidentiality of the patron-searcher interaction.
Problems Subject confidentiality. Although the department adhered to a policy of confidentiality of search topics and results, a number of steps could still be taken to ensure greater security. On the billing form, the topic of the search was written out with the patron's name and cost. Searches were left at the circulation desk for pickup, with the invoices indicating search topic and patron name evident to all staff and to other patrons at the desk. Solutions Numbered searches. Revised search request forms include prenumbered sheets so that searches can be recorded in the log, billed, and tracked according to a search number rather than by topic and requester name. In addition, each search is now packaged in a manila envelope when left at the circulation desk for pickup. This provides both an attractive and professional package and preserves security. The online training program now routinely reinforces the concept of confidentiality. PHYSICAL FACTORS Although physical surroundings, condition of equipment, and appearance of personnel are rated as less significant than some other criteria, customers rate physical factors or tangibles as an important facet of quality service.
Problems Search environment. Personal computers had been installed at each searcher's desk, but lacked appropriate equipment to provide ergonomically designed computer workstations. The computer screen was often hard to adjust to a position easily viewed by the patron during search appointments. The standard stock paper used to print search results was lightweight and somewhat translucent and tore easily. The desks of individual searchers were separated by freestanding divider walls, but the area was not enclosed for privacy during the search process. Bull Med Libr Assoc 79(3) July 1991
Quality assurance Solutions Professional workstations. New equipment was ordered to solve the design problems and logistics of using the computer easily and effectively. To improve the appearance and utility of the search printout, heavier paper was ordered for printing search results. Enhancing the look and ease of use of the searcher's workstation served to increase the patron's comfort and confidence in the searcher's expertise. A future space analysis and design project was planned that would result in the installation of an entire office system.
REEVALUATION OF SERVICES Approximately six months after devising and beginning implementation of the proposed solutions for improving services, the committee met to evaluate the results. Most of the items outlined in the plan had been accomplished, including the important process of individually auditing each searcher's results during a predetermined review period. The newly appointed online coordinator reported the positive results of the first review cycle. The program successfully established a baseline of knowledge for each searcher, provided a method for checking the quality of individual and departmental work, and included a training program to develop skills and correct deficiencies. In keeping with the ongoing nature of the QA process, the online coordinator made suggestions for further improving services and also for streamlining the QA process itself. During the first year the review period was held twice. The process was quite timeconsuming for both searchers and reviewers and has since been modified to a once-yearly assessment for current searchers. Experienced searchers new to the department are evaluated immediately upon joining
Experienced searchers new to the department are evaluated immediately upon joining the searching staff. Novice searchers are trained according to an in-house training plan which covers the skills, systems, and databases identified as being critical to quality searching in this environment.
sampling periods. The steps to be followed both during the formal review period and during spot checks were outlined in policy and procedure documents by
the online coordinator. A timetable was developed for implementing those suggested solutions that could not be executed immediately. Other service enhancements that have been instituted include providing patrons with diskettes and mailers for downloaded searches, and developing a brochure describing the online services program. The online services coordinator reports the results of the search sampling periods on a regular basis and makes recommendations for improving the quality of online services delivery.
DEVELOPMENT AND IMPLEMENTATION COSTS The development and ongoing support of a QA program is not without significant costs in terms of both staff time and equipment. The development of the QA program for online services at CMHSL involved staff time in research, committee work, administrative review, staff audits, and the writing of an extensive policy and procedure manual to establish measurable criteria to guide the program application. Staff hours spent in initial program development over a two-year period are estimated at 1,575 hours. Ongoing costs include searcher training, search audits, forms, and supplies. Additional funds were spent on equipment and supplies to enhance services.
QA PROCESS RESULTS The QA cycle has successfully moved the library's online services toward a more professional, customeroriented program. Many changes were implemented after the initial review because the various components of service had not previously been carefully evaluated. With many of the service enhancements in place, fewer areas require major change. However, providing quality service and meeting high standards have become common goals integrated into the daily functioning of the department; staff can easily generate course-correcting ideas outside of the formal review process. Pride, teamwork, and collegiality are generated by efforts to provide the best results possible.
COST-BENEFIT ANALYSIS the searching staff. Novice searchers are trained according to an in-house training plan which covers the skills, systems, and databases identified as being critical to quality searching in this environment. In addition, spot checks of search results ensure that shortcomings are not overlooked between search Bull Med Libr Assoc 79(3) July 1991
The problems identified as being important in the QA cycle were primarily issues selected by searchers through informal feedback from requesters and through observations of users' responses to online services. The QA Committee judged the program to be successful. No formal survey or evaluation of on269
Humphries and Naisawald
line services from the user's viewpoint has as yet been conducted to determine whether requesters view the service changes to be significant and worth the cost. The library's long-range plan calls for a general user survey addressing library services as a whole, with an online search services component studying users' views on search services in particular. Such problem identification and resolution from the customer's viewpoint deserves further investigation; the study should determine what changes are truly beneficial in relation to the cost of implementation.
CONCLUSION The QA model is a useful tool for the evaluation of library services and can be applied to specific library service areas to improve performance and achieve high quality results. A QA program provides not only a mechanism to proactively establish training and competency standards, but also a method for continuous monitoring of current service practices to correct deficiencies. Future studies should include customer expectations for particular services and the objective criteria required to evaluate fulfillment of expectations. Such research will provide models for effective and successful library programs.
ACKNOWLEDGMENT The authors gratefully acknowledge the extensive contributions of Lenore K. Schnaitman, Jonquil D. Feldman, and John Patruno in the development of the QA program.
REFERENCES 1. CRONIN MJ. Performance measurement for public services in academic and research libraries. Washington, DC: Office of Management Studies, Association of Research Li-
2. FREDENBURG AM. Quality assurance: establishing a program for special libraries. Spec Libr 1988 Fall;79(4):277-84. 3. BLOOD RW. Evaluation of online searches. RQ 1983 Spring; 22(3):266-77. 4. LANCASTER FW. Evaluation and testing of information retrieval systems. In: Kent A, Lancour H, eds. Encyclopedia of library and information science. New York: Marcel Dekker, 1972:234. 5. GLUNZ D, WAKIJI E. Maximizing search quality through a program of peer review. Online 1983 Sep;7(5):100-10. 6. MATTHEWS CA. Supplemental training program for MEDLARS searchers. Med Ref Serv Q 1983 Winter;2(4):21-9. 7. JACKSON WJ. Staff selection and training for quality online searching. RQ 1982 Fall;22(1):48-54. 8. IBID., 51. 9. SELF PC. A quality assurance process in health sciences libraries. Bull Med Libr Assoc 1980 Jul;68(3):288-92. 10. IBID., 291. 11. BERRY LL, PARASURAMAN A, ZEITHAML VA. The servicequality puzzle. Bus Horiz 1988 Sep-Oct;31(5):35-43. 12. IBID., 37. 13. MCDONALD FJ. Creating a quality culture. Qual Circles J 1987 Jun;10(2):57. 14. SHAUGHNESSY TW. The search for quality. J Libr Adm 1987 Spring;8(1):5-10. 15. IBID., 9. 16. CRONIN, op. cit., 14. 17. PARASURAMAN A. A conceptual model of service quality and its implications for future research. J Market 1985 Fall;
49(4):41-50. 18. SCHNAITMAN LK. Implementation of the search review cycle: a practical approach. Paper presented at the EightyNinth Annual Meeting of the Medical Library Association, Boston, Massachusetts, May 1989. 19. KOLNER SJ. Improving the MEDLARS search interview: a checklist approach. Bull Med Libr Assoc 1981 Jan;69(1): 26-33. 20. CALABRETA N. Educating the online search requester: a checklist. Med Ref Serv Q 1983 Winter;2(4):31-9.
Received May 1990; accepted August 1990
braries, 1985. (Occasional Paper no. 9).
270
Bull Med Libr Assoc 79(3) July 1991
The Claude Moore Health Sciences Library Box 234 University of Virginia Health Sciences Center Charlottesville, Virginia 22908
A quality assurance (QA) program provides not only a mechanism for establishing training and competency standards, but also a method for continuously monitoring current service practices to correct shortcomings. The typical QA cycle includes these basic steps: select subject for review, establish measurable standards, evaluate existing services using the standards, identify problems, implement solutions, and reevaluate services. The Claude Moore Health Sciences Library (CMHSL) developed a quality assurance program for online services designed to evaluate services against specific criteria identified by research studies as being important to customer satisfaction. These criteria include reliability, responsiveness, approachability, communication, and physical factors. The application of these criteria to the library's existing online services in the quality review process is discussed with specific examples of the problems identified in each service area, as well as the solutions implemented to correct deficiencies. The application of the QA cycle to an online services program serves as a model of possible interventions. The use of QA principles to enhance online service quality can be extended to other library service areas.
Quality service is a much-publicized selling point in commercial industries and is often the factor that stays in the buyer's mind when evaluating how well a company did in providing a superior product. Li-
As more libraries find it necessary to recover costs through fee-based services, however, library services may also be viewed as consumer products, and the issue of responsibility for providing a quality service then becomes significant. braries have traditionally provided customer services, but such assistance has not been viewed as a product in the same sense that a manufactured item is viewed * Based on a paper presented May 23, 1989, at the Eighty-Ninth Annual Meeting of the Medical Library Association, Boston, Massachusetts.
Bull Med Libr Assoc 79(3) July 1991
as a product. As more libraries find it necessary to recover costs through fee-based services, however, library services may also be viewed as consumer products, and the issue of responsibility for providing a quality service then becomes significant. Online bibliographic searching is one such service that is often accompanied by fees and surcharges, and needs to be justified in terms of cost and quality.
BACKGROUND Evaluating library services Evaluating any library service is difficult because it requires judgment about processes that are not readily quantified. Methods for measuring performance in libraries include user surveys, unobtrusive studies, quantitative analysis of collected statistical data, and individual performance evaluation. Cronin pointed out that no matter what method of evaluation is used, the quality of service is always defined in terms of 263
Humphries and Naisawald Figure 1 The quality assurance cycle SELECr SUBJECIT
_
EVALUATE SERVICES USING STANDARDS
ESTABLISH MEASURABLE STANDARDS
l
l REEVALUATE | SERVICES
|
IMPLEMENT
SOLUTIONS
|
|
IDENTIFY PROBLEMS
the needs of the library user and the skills of the staff in assessing and meeting those needs [1]. The level of service required depends entirely on the circumstances of the individual library and its particular clientele and, therefore, is highly variable. Fredenburg discussed establishing a quality assurance (QA) program in special libraries and related it to the development of specific performance plans [2]. QA was described as a practical tool for proactively establishing goals for the library, justifying the library's worth to administration, and getting everyone in the organization, from the support staff to the administration, involved in problem recognition and resolution. Blood gave examples of user survey questions recommended by an American Library Association (ALA) committee formed to evaluate the effectiveness of online search results [3]. Blood discussed factors that can affect the user's perception, including previous use of online services, comparisons with manual searches, awareness of the capabilities of database searching, and lack of criteria on which to base evaluation of the search intermediary. Blood concluded that if a survey is used, it must be short, simple, anonymous, and convenient. The survey should have clear goals for what is to be evaluated and should result in modifications to improve service. If no clear goals for raising the level of service are established, the evaluation process is futile. In predicting the cost-effectiveness of an evaluation effort, Lancaster warned: "Thorough evaluation tends to be expensive, and can only be justified if the evaluation program is likely to lead to significant improvement in the performance of the system" [4]. Libraries also attempt to maintain quality output for online search services through peer review or inhouse training programs. Glunz described weekly meetings of online searchers held to discuss searches and identify ways to improve search results through collegial examination of search strategies judged by specific criteria [5]. Glunz emphasized the need to 264
make the peer review program ongoing, nonthreatening, and a supplement to formal training classes. Similarly, Matthews and Jackson described conducting in-house training programs and using mentors to complement formal training at vendor workshops [6-7]. Jackson also noted that "surveys of users are often futile and are likely to detect problems only when a searcher is grossly deficient ... if it is impossible to measure the quality of output, we must do more to regulate the quality of input" [8]. Training programs that allow novice searchers to test their skills through practice exercises and to work with experienced searchers establish a solid base for quality searching. These studies indicated that issues of establishing, improving, and maintaining quality are prominent concerns in the management of library services. However, there is no discussion of the development of a QA program designed to monitor the interrelated service features of online services, including not only the technical quality of the end product or list of references, but also service quality in the process of delivering online services.
The QA cycle QA is a routine concept in production-line processes where output can be well controlled. QA programs have also been developed in service industries, including health care delivery systems. Self reviewed the application of the QA process in health sciences libraries [9]. The basic steps in the QA cycle are (Figure 1): * Select the subject for review. Limit the focus of the review to crucial aspects of service delivery. * Establish measurable standards. Select performance standards based on achievable goals that can be measured for their effectiveness in providing quality service. Standards must be relevant, understandable, measurable, and achievable. * Evaluate existing services using the standards. Identify various components of existing services and compare them to the goals that have been established. * Identify problems. Problems exist when actual outcome or compliance is below expected performance levels. Causes of the problems may involve many factors including gaps in staff knowledge or skills, problems with policies or procedures, environmental obstacles, and lack of materials or equipment. * Implement solutions. Once problems have been identified, devise positive solutions to correct the deficiency. The staff must be committed to the QA process to implement the solutions successfully. * Reevaluate services. A successful QA program is ongoing and periodically reaudits its selected criteria. Self pointed out the positive impact a QA program can have: "Skillful use of the quality assurance proBull Med Libr Assoc 79(3) July 1991
Quality assurance cess can upgrade services, maintain quality service, provide a mechanism for goal setting, make library staff more responsive to user needs, and influence behavior practices in the library" [10].
Determinants of customer services quality Traditionally, libraries have been evaluated by quantitative measures including size of collections, number of users, circulation figures, allocation of resources and personnel, and similar aspects. None of these factors guarantee that library services have the quality of excellence. Berry pointed out that the definition of service quality is the correspondence of performance or perceived performance to the customer's specifications [11]. Usually, the most important performance values are those most dependent on the human factor, which is difficult to control. In the study conducted by Berry and colleagues to evaluate important service features in banking, credit card, and machine repair businesses, the five elements most frequently reported by customers in order of importance were: * Reliability-the ability to perform dependably, accurately, and consistently to carry out what was promised to the customer * Responsiveness-the willingness to provide prompt service and to help customers * Assurance-evidence of employees' knowledge of the product, courtesy, and ability to convey trust and confidence * Empathy-provision of caring, individualized attention to customers * Tangibles-physical facilities, equipment, appearance of personnel [12]. All of these attributes are dependent on the behavior of personnel, making it critical that control over service be as close to the customer as possible and that personnel be empowered to attain quick and effective problem resolution. The QA cycle not only identifies and solves problems, it also provides the
The QA cycle not only identifies and solves problems, it also provides the platform to proactively create an environment that corrects deficiencies before they become problems. platform to proactively create an environment that corrects deficiencies before they become problems. F. J. McDonald, a president of General Motors, noted: "The environment has to be one that is error-friendly. Quality is error driven. It depends on each individual's ability to recognize error trends and to make corrections quickly" [13]. Employees must not feel the Bull Med Libr Assoc 79(3) July 1991
necessity to cover up errors in order to preserve their jobs. Shaughnessy proposed that if an organization is to be successful in consistently delivering service of high standards, quality must become a valued characteristic of every customer interaction [14]. Shaughnessy further described the type of team building and group cohesiveness necessary in the process: "Every culture develops unwritten norms which influence the behaviors and attitudes of its members ... this value system or culture determines how well the organization responds to challenges and opportunities" [15]. The QA process must be an integral part of the organization's mission and the employees' personal performance standards. Cronin pointed out that benefits of the performance measurement process include increased staff discussion and involvement, better staff awareness of major public services objectives within the library, increased user awareness of library activities, expanded accountability, and the requirement that ambiguous terms like "high quality service" be clearly defined [16].
QA FOR ONLINE SERVICES The Claude Moore Health Sciences Library (CMHSL) began a review of its online services program in 1987 as part of its long-range plan. The review was instituted not because of a concern that the program was deficient, but because of a commitment on the part of the library to provide high quality services rather than just acceptable ones. In the long-range plan several strategies were identified to improve online searches, including auditing each searcher's output, comparing results against a checklist of the components of a good online search, and redesigning forms. However, when work began on developing these strategies, it became clear that technical proficiency in online searching represented only one component and that online services encompass a whole range of service elements in an interrelated process. At this point the concept of creating a QA program began to form, and a committee composed primarily of online searchers was appointed within the Public Services Division to develop the program. The committee operated on the premise that online services would move beyond the mechanical output of a product to a more professional service that emphasized the consultative relationship between the user and the search analyst. Quality determinants The committee selected the library's online search services as the subject of the QA cycle. Developing measurable criteria for quality services proved more challenging and complex than originally anticipated. In developing criteria for online services, the ele265
Humphries and Naisawald
Table 1 Examples of quality determinants for online services Reliability/consistency/competence/credibility * standard training program * current awareness mechanisms * search sample audit * staff development plan * customer satisfaction with results _- .r . . ._ . .
. .. .
.. .
._...._-
* promised tumaround time * priority for patient care * daily searching schedule Access/approachability * information sheet for requesters * user-friendly request forms * search status log Courtesy/communication/understanding customer needs * guidelines for search interview * search summary letter for requester * information on related services
User feedback. No method had been developed to evaluate the quality of search results for each individual searcher. From 1983 to 1987, the library attached a search evaluation form to all search results generated by the department. This form served as a feedback measurement for extremes of praise and complaint but did not give an adequate systematic analysis of searchers' output based on measurable standards.
Security
Program management. The responsibilities for online search services lay solely with the head reference librarian, who had other obligations that competed with the responsibilities of online searching. Also, the library had an informal policy of rerunning searches at no charge if the results were unsatisfactory to the user, but the policy was not actively advertised to clientele.
* confidentiality of search topic * professional presentation of results
Physical factors * ergonomic computer equipment * high-quality paper * privacy for search appointments
ments of quality identified in customer service studies by Parasuraman [17] were gathered into related subject groups: * Reliability/consistency/competence/credibility * Responsiveness/ timeliness * Access /approachability * Courtesy/communication/understanding customer needs * Security * Physical factors. Once the criteria were examined and found to be appropriate and applicable to an online services system, the committee began the process of evaluating the library's online searching program using these determinants of quality. The next steps were to identify problems and to implement solutions. The problems and solutions were particular to this library and were not comprehensive, but illustrate the types of intervention that are possible (Table 1).
RELIABILITY/CONSISTENCY/COMPETENCE/ CREDIBILITY The first criteria are concerned with the development of equivalent skills for all searchers and the consistent application of policies in the delivery of online search results.
Problems Training. The review indicated that the department lacked a uniform base-level training program for all 266
searchers. While searchers were given opportunities to attend courses and updates on online searching, there was no formal plan or program that systematically coordinated these courses on an individual and departmental level. Similarly, there was no mechanism in place to ensure that searchers kept up-to-date except through self-initiated development and journal reading.
Solutions Uniform training. To solve training problems, a detailed plan outlining the steps of a base-level training program for all searchers was produced. This plan stipulates that all searchers attend the initial National Library of Medicine (NLM) online training course within the first six months of training. It also required attendance at specified database vendor workshops for frequently searched databases and systems in the first year. In addition to formal training, an in-house training program was established, with opportunities for practice sessions and gradual independent searching once experienced searchers observe that new searchers are uniformly fulfilling the requirements of the quality searching checklist compiled by the QA Committee.
Searcher audit. One of the key monitoring processes to come out of the QA review was a regular sampling period for online searches. Schnaitman described the details of the CMHSL review process, including twiceyearly periods during which searchers print out all searches [18]. The searches are then reviewed according to the checklist of quality search criteria developed by the QA Committee through literature reviews and staff input. Each searcher meets with the online coordinator to discuss searching strengths, weaknesses, and areas for development. The results Bull Med Libr Assoc 79(3) July 1991
Quality assurance
of the review meeting are summarized in a memorandum, which includes a proposed training program for the next six months, including formal courses. Searchers are also scheduled to present in-house review sessions of databases or systems for staff meetings. The memo serves as an adjunct report for performance evaluation as well as a management planning tool for scheduling staff development activities.
Each searcher meets with the online coordinator to discuss searching strengths, weaknesses, and areas for development. The results of the review meeting are summarized in a memorandum, which includes a proposed training program for the next six months, including formal courses. Current awareness. Online searching became a regular agenda item at department meetings to update staff on new databases, changes in systems, new features, and search tips. A staff member is designated to regularly print and route important system news. Electronic mail was recommended as an additional medium to alert searchers to important online searching issues. Program coordinator. The committee further recommended establishing a position for online services coordinator to ensure necessary managerial oversight and constant and focused attention to the issues and policies relevant to online services. In addition, a patron complaint policy was written to provide uniform guidelines for determining when a search should be rerun at no additional charge to the patron. The policy is advertised on all search request forms.
Solutions Promised delivery. A standard policy of two working days for turnaround time was established for all searches from primary clientele submitted during the business week, and a same-day turnaround policy was established for urgent patient care searches.To accommodate varying search loads, searching periods are scheduled throughout the day for searchers to complete any pending search requests. A search log records the date and time when the search is received, and the date and time when the search is completed and the patron is notified.
ACCESS/APPROACHABILITY Access and approachability relate to the ease with which requesters can submit search requests, and the help they receive in communicating their search need and obtaining results. Problems Search requests. A better mechanism was needed to assist patrons who wished to request searches, particularly after the reference desk was closed. The search form was not user friendly or self-explanatory. The library had no informative fact sheet covering general information about computer searches and potential costs that could be given to requesters when the department was not staffed. Finally, there was no systematic method of determining the current status of a search once it was in process.
Problems Turnaround time. The existing informal policy for
Solutions Improved forms. An informational checklist to accompany search request forms was written. The checklist gives patrons who request searches after hours tips on improving their search requests. Because no search interview is possible, it is helpful to suggest that patrons write topics in sentence form, be as specific as possible, include synonyms, and list known articles. The search request form itself was revised to make better use of space, following a simple and logical format. Frequently used limiting criteria, such as English, human, review, and years to be covered, are incorporated into check-off boxes for the benefit of both the requester and the searcher. Reference desk staff routinely check during the search interview to make sure that all required parameters have been identified.
turnaround time for searches was one to two working days, with priority given to patient care questions. However, no promised turnaround time was advertised and the priority of patient care was not clearly defined. Searches were processed whenever time was available to individual staff members, with no systematic schedule.
Search status log. To track the status of a search request at any point in the process, a search request log was designed for following the progress of the search. The log indicates the time of the initial request; the time, date, and initials of the searcher running the search; and the date the search was completed. The
RESPONSIVENESS/TIMELINESS These criteria relate to the customer's expectation that the searcher will provide search results promptly and will try to accommodate the requester's needs and time schedules.
Bull Med Libr Assoc 79(3) July 1991
267
Humphries and Naisawald
log also indicates whether the search is for patient care, whether a search has been picked up by the requester, and whether a search has been mailed to an outside client.
COURTESY/COMMUNICATION/ UNDERSTANDING CUSTOMER NEEDS Consumers rate caring, individualized attention highly. The courtesy and communication offered to search requesters has an important impact on their perception of quality service.
Problems Search interview. The department lacked a standard guide for conducting a search interview. Searchers relied on their own knowledge and training but did not have a common method for best ascertaining the requester's information need. In addition, at the completion of the search, any comments or concerns about search output were informally conveyed to patrons, either by searchers' handwritten notes or verbal comments.
Solutions Interview guidelines. To remedy these concerns, standard guidelines for the search request interview were included in a policy and procedure manual. Important points for the interview were determined through literature review. The search interview was incorporated into a department staff development program, which required reading articles about search interviews [19-20]. A copy of the search request checklist is readily available at the reference desk for review.
Search summary letter. A standardized memo was written to enable searchers to produce search summaries for patrons on their personal computers. The search memo is provided for every search and in-
Any concerns or possible alternative strategies are discussed in the memo to fully inform the patron of the issues encountered by the searcher when running the search. This provides the patron with a better understanding of the search process and offers an opportunity for feedback and improved results. cludes information on databases searched, years covered, and the search method used to obtain the final results. Any concerns or possible alternative strate268
gies are discussed in the memo to fully inform the patron of the issues encountered by the searcher when running the search. This provides the patron with a better understanding of the search process and offers an opportunity for feedback and improved results. The summary also includes information about related library services such as document delivery, interlibrary loan, and regular updates on a search topic.
SECURITY "Security" refers here to the patron's trust in the confidentiality of the patron-searcher interaction.
Problems Subject confidentiality. Although the department adhered to a policy of confidentiality of search topics and results, a number of steps could still be taken to ensure greater security. On the billing form, the topic of the search was written out with the patron's name and cost. Searches were left at the circulation desk for pickup, with the invoices indicating search topic and patron name evident to all staff and to other patrons at the desk. Solutions Numbered searches. Revised search request forms include prenumbered sheets so that searches can be recorded in the log, billed, and tracked according to a search number rather than by topic and requester name. In addition, each search is now packaged in a manila envelope when left at the circulation desk for pickup. This provides both an attractive and professional package and preserves security. The online training program now routinely reinforces the concept of confidentiality. PHYSICAL FACTORS Although physical surroundings, condition of equipment, and appearance of personnel are rated as less significant than some other criteria, customers rate physical factors or tangibles as an important facet of quality service.
Problems Search environment. Personal computers had been installed at each searcher's desk, but lacked appropriate equipment to provide ergonomically designed computer workstations. The computer screen was often hard to adjust to a position easily viewed by the patron during search appointments. The standard stock paper used to print search results was lightweight and somewhat translucent and tore easily. The desks of individual searchers were separated by freestanding divider walls, but the area was not enclosed for privacy during the search process. Bull Med Libr Assoc 79(3) July 1991
Quality assurance Solutions Professional workstations. New equipment was ordered to solve the design problems and logistics of using the computer easily and effectively. To improve the appearance and utility of the search printout, heavier paper was ordered for printing search results. Enhancing the look and ease of use of the searcher's workstation served to increase the patron's comfort and confidence in the searcher's expertise. A future space analysis and design project was planned that would result in the installation of an entire office system.
REEVALUATION OF SERVICES Approximately six months after devising and beginning implementation of the proposed solutions for improving services, the committee met to evaluate the results. Most of the items outlined in the plan had been accomplished, including the important process of individually auditing each searcher's results during a predetermined review period. The newly appointed online coordinator reported the positive results of the first review cycle. The program successfully established a baseline of knowledge for each searcher, provided a method for checking the quality of individual and departmental work, and included a training program to develop skills and correct deficiencies. In keeping with the ongoing nature of the QA process, the online coordinator made suggestions for further improving services and also for streamlining the QA process itself. During the first year the review period was held twice. The process was quite timeconsuming for both searchers and reviewers and has since been modified to a once-yearly assessment for current searchers. Experienced searchers new to the department are evaluated immediately upon joining
Experienced searchers new to the department are evaluated immediately upon joining the searching staff. Novice searchers are trained according to an in-house training plan which covers the skills, systems, and databases identified as being critical to quality searching in this environment.
sampling periods. The steps to be followed both during the formal review period and during spot checks were outlined in policy and procedure documents by
the online coordinator. A timetable was developed for implementing those suggested solutions that could not be executed immediately. Other service enhancements that have been instituted include providing patrons with diskettes and mailers for downloaded searches, and developing a brochure describing the online services program. The online services coordinator reports the results of the search sampling periods on a regular basis and makes recommendations for improving the quality of online services delivery.
DEVELOPMENT AND IMPLEMENTATION COSTS The development and ongoing support of a QA program is not without significant costs in terms of both staff time and equipment. The development of the QA program for online services at CMHSL involved staff time in research, committee work, administrative review, staff audits, and the writing of an extensive policy and procedure manual to establish measurable criteria to guide the program application. Staff hours spent in initial program development over a two-year period are estimated at 1,575 hours. Ongoing costs include searcher training, search audits, forms, and supplies. Additional funds were spent on equipment and supplies to enhance services.
QA PROCESS RESULTS The QA cycle has successfully moved the library's online services toward a more professional, customeroriented program. Many changes were implemented after the initial review because the various components of service had not previously been carefully evaluated. With many of the service enhancements in place, fewer areas require major change. However, providing quality service and meeting high standards have become common goals integrated into the daily functioning of the department; staff can easily generate course-correcting ideas outside of the formal review process. Pride, teamwork, and collegiality are generated by efforts to provide the best results possible.
COST-BENEFIT ANALYSIS the searching staff. Novice searchers are trained according to an in-house training plan which covers the skills, systems, and databases identified as being critical to quality searching in this environment. In addition, spot checks of search results ensure that shortcomings are not overlooked between search Bull Med Libr Assoc 79(3) July 1991
The problems identified as being important in the QA cycle were primarily issues selected by searchers through informal feedback from requesters and through observations of users' responses to online services. The QA Committee judged the program to be successful. No formal survey or evaluation of on269
Humphries and Naisawald
line services from the user's viewpoint has as yet been conducted to determine whether requesters view the service changes to be significant and worth the cost. The library's long-range plan calls for a general user survey addressing library services as a whole, with an online search services component studying users' views on search services in particular. Such problem identification and resolution from the customer's viewpoint deserves further investigation; the study should determine what changes are truly beneficial in relation to the cost of implementation.
CONCLUSION The QA model is a useful tool for the evaluation of library services and can be applied to specific library service areas to improve performance and achieve high quality results. A QA program provides not only a mechanism to proactively establish training and competency standards, but also a method for continuous monitoring of current service practices to correct deficiencies. Future studies should include customer expectations for particular services and the objective criteria required to evaluate fulfillment of expectations. Such research will provide models for effective and successful library programs.
ACKNOWLEDGMENT The authors gratefully acknowledge the extensive contributions of Lenore K. Schnaitman, Jonquil D. Feldman, and John Patruno in the development of the QA program.
REFERENCES 1. CRONIN MJ. Performance measurement for public services in academic and research libraries. Washington, DC: Office of Management Studies, Association of Research Li-
2. FREDENBURG AM. Quality assurance: establishing a program for special libraries. Spec Libr 1988 Fall;79(4):277-84. 3. BLOOD RW. Evaluation of online searches. RQ 1983 Spring; 22(3):266-77. 4. LANCASTER FW. Evaluation and testing of information retrieval systems. In: Kent A, Lancour H, eds. Encyclopedia of library and information science. New York: Marcel Dekker, 1972:234. 5. GLUNZ D, WAKIJI E. Maximizing search quality through a program of peer review. Online 1983 Sep;7(5):100-10. 6. MATTHEWS CA. Supplemental training program for MEDLARS searchers. Med Ref Serv Q 1983 Winter;2(4):21-9. 7. JACKSON WJ. Staff selection and training for quality online searching. RQ 1982 Fall;22(1):48-54. 8. IBID., 51. 9. SELF PC. A quality assurance process in health sciences libraries. Bull Med Libr Assoc 1980 Jul;68(3):288-92. 10. IBID., 291. 11. BERRY LL, PARASURAMAN A, ZEITHAML VA. The servicequality puzzle. Bus Horiz 1988 Sep-Oct;31(5):35-43. 12. IBID., 37. 13. MCDONALD FJ. Creating a quality culture. Qual Circles J 1987 Jun;10(2):57. 14. SHAUGHNESSY TW. The search for quality. J Libr Adm 1987 Spring;8(1):5-10. 15. IBID., 9. 16. CRONIN, op. cit., 14. 17. PARASURAMAN A. A conceptual model of service quality and its implications for future research. J Market 1985 Fall;
49(4):41-50. 18. SCHNAITMAN LK. Implementation of the search review cycle: a practical approach. Paper presented at the EightyNinth Annual Meeting of the Medical Library Association, Boston, Massachusetts, May 1989. 19. KOLNER SJ. Improving the MEDLARS search interview: a checklist approach. Bull Med Libr Assoc 1981 Jan;69(1): 26-33. 20. CALABRETA N. Educating the online search requester: a checklist. Med Ref Serv Q 1983 Winter;2(4):31-9.
Received May 1990; accepted August 1990
braries, 1985. (Occasional Paper no. 9).
270
Bull Med Libr Assoc 79(3) July 1991
