Journal of Biomedical Informatics 47 (2014) 11–17
Contents lists available at ScienceDirect
Journal of Biomedical Informatics journal homepage: www.elsevier.com/locate/yjbin
Pre-launch evaluation checklist for online health-promoting communities Joakim Ekberg a,⇑, Elin A. Gursky b, Toomas Timpka a,c a
Department of Medical and Health Sciences, Linköping University, SE-581 83 Linköping, Sweden Analytic Services Inc., Arlington, VA, USA c Centre for Public Health Sciences, Linköping, Sweden b
a r t i c l e
i n f o
Article history: Received 22 October 2012 Accepted 16 October 2013 Available online 29 October 2013 Keywords: Community-based participatory research Health promotion Adolescents Health service
a b s t r a c t Background: Despite the apparent potential of online health-promoting communities (OHPC), there is limited guidance available for developers on the basic design features that contribute to successful applications. The aim of this study was to develop a checklist for a pre-launch evaluation of OHPCs incorporating the perspectives of both the user and the health services communities. Methods: The study was based on an action research design. Constructs previously applied to evaluate information system success were used as the basis for checklist development. The constructs were adapted for the OHPC context and formatively evaluated in a case study project. Evaluation data were collected from participatory observations and analyzed using qualitative methods. Results: The initial OHPC checklist included the constructs information quality, service quality, and subjective norms. The contextual adaptation of the information quality construct resulted in items for content area, trust, and format; the adaptation of the service quality construct in items for staff competence, prompt service and empathy; and the adaptation of the subject norms construct in items for social facilitation, interconnectivity and communication. The formative evaluation demonstrated the critical need to balance the autonomy of the online community with the professional control of health services quality expressed in the information and service quality constructs. Conclusions: A pre-launch OHPC evaluation checklist has been designed for use in practical development of health promotion web resources. Research on instruments for OHPC evaluations is warranted. Ó 2013 Elsevier Inc. All rights reserved.
1. Background As a result of the prevalent use of the Internet and social media, the community concept is no longer limited to physical locations [1]. An online community refers to a gathering of individuals in a virtual space that form a network and participate in long-term public discussions on common interests or experiences [2]. In contrast to traditional communities, online communities are not connected through a shared focus on physical tasks and activities [3]. Rather, emphasis is placed on cognitive tasks such as learning and planning, which are suitable for the virtual space [4]. Medical services have also embraced the opportunities and challenges of such virtual settings in the form of, e.g., Internet-mediated treatments
Abbreviations: CCS, cascading style sheet; CMS, content management system; IS, information system; OHPC, online health-promoting community. ⇑ Corresponding author. E-mail addresses: [email protected] (J. Ekberg), [email protected] (E.A. Gursky), [email protected] (T. Timpka). 1532-0464/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jbi.2013.10.004
[5,6], patient-governed clinical information websites [7] and support groups for disease self-management [8]. Prevention of specific diseases via web-based interventions has matured into a relatively well-documented intervention alternative. Unfortunately, few studies on web-based health promotion have been published [9,10], although across the prevention spectrum, health promotion may offer the greatest return on investment by sustaining quality of life and avoiding disease and injury. The development of open source and easy-to-use content management systems (CMSs) makes it possible to start up an online community. CMSs are flexible enough to host information resources addressing a wide range of topics related to lifestyle and healthy living conditions without the need for specialized computer science competence. Despite the apparent potential of online health-promoting communities (OHPC), limited guidance is available for developers on the basic design features that contribute to successful applications. For instance, the success of health promotion interventions in virtual environments is contingent on the sustained use of intervention resources, although use alone does not guarantee the intervention’s intended effect. The
12
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
development of a health information system is usually informed by requirements derived from both primary end-users and health service managers [11,12]. If end-user requirements are given precedence in the design of an OHPC, it may stimulate use but fail to lead to beneficial health effects. Conversely, if requirements derived from the health service are given precedence; the online community may fail to attract the intended end-users [13]. An online community can flourish only when it is perceived useful and is embraced by a critical amount of users sustaining online activity. Consumer health informatics refers to the provision of health information resources to consumers via the Internet. It has been noted that, in reaching out to health consumers, the field should make use of health information resources ranging from static informational Web pages to patient forums and virtual reality environments. Additionally, consumers’ information needs and usability concerns should be given specific consideration [14]. An individual’s motivation to use an online community is contingent on his/her perception of the usefulness of the online community and its specific cultural, social and technical context. This interplay between information system (IS) design and success has been associated with at least five constructs: systems quality, information quality, service quality, user satisfaction and net benefit [15]. Moreover, it has been found that these constructs need to be highly contextualized to specified information resources in order to be useful [16]. Previous research on online communities has pointed to several constructs shown to be associated with their success, both in terms of use rate and net benefit [17]. Service quality has been shown to be particularly important for user satisfaction [18]. The aim of this study was to develop a checklist for pre-launch evaluations of OHPCs that covers the perspectives of both the user community and the health services sector. The purpose of such a pre-launch checklist is to utilize experiences from previous evaluations of ISs to identify potential problems in an OHPC under development. In the present case, the checklist was used to evaluate an OHPC before its introduction to end-users. 2. Methods The study was based on an action research design. Action research involves the process of actively participating in an organization change situation while conducting research [19]. In the current study, development of a prototype OHPC evaluation checklist was initiated by researchers reviewing and analyzing the literature on IS success. The resultant checklist was applied as a formative evaluation in a case study implementation of an OHPC, where the researchers participated both as agents in the development process and as observers of the same process. Data from this process were collected and analyzed using qualitative methods. 2.1. Case study setting The case study county (total population 420,000) is situated in the south east of Sweden. To counteract the growing incidence of obesity among young adults, the county recognized the need for health policy and health promotion initiatives specific to this age group [20]. Consequently, both public health practitioners and researchers embarked on a public health intervention initiative aimed at adolescents and obesity. A needs analysis was performed using data from focus group interviews with adolescents, which was moderated by a researcher and a local public health representative. A requirements analysis performed by an interdisciplinary expert team consisting of two computer scientists, an interaction designer, a cognitive scientist, and a sample of young adults. A prototype OHPC was then drafted using the website content management system Joomla! (http://www.joomla.org). The resulting
OHPC design was demonstrated and discussed in meetings and through telephone conferences with representatives from regional public health services, public health researchers and senior public health officials. The OHPC design was subsequently modified as a result of this input. 2.2. Contextual adaption of OHPC evaluation checklist Constructs used in evaluations of IS success served as the basis for the formation of a prototype OHPC evaluation checklist. The inclusion criteria for concepts in the checklist were that (1) they had been reported valid for evaluation of IS success (ISS), and (2) were available for application before the introduction of the OHPC to the end users. A meta-narrative review [21] was performed on the literature on evaluation methods for IS success. The meta-narrative review method was chosen because it is pluralistic rather than normative (i.e., it asks not ‘what is the best approach to researching this topic?’ but ‘what can we learn from the range of different approaches?’). A web search on ISS research, a search of the PubMed database using the search terms ‘information system success’, and a seminal overview article of the IS success framework [15] were approaches used to identify a basic set of publication records (Fig. 1). The abstracts of these publications were assessed for relevance, and 14 publications were used in the final analysis of constructs [15– 18,22–31]. From these publications, six instrument categories were identified: the DeLone and McLean model of ISS [15], the SERVQUAL instrument [22], Mirani and Lederer’s instrument [24], adaptations of the Balanced Scorecard [25,26], instruments for measuring the individual impact of ISs [27], and the user satisfaction model [30]. Finally, constructs applicable to the pre-launch evaluation of OHPCs were selected from these instrument categories. The constructs were adapted to the OHPC context and combined into a preliminary checklist to be used in the formative evaluation. The justifications for each construct adaptation were recorded by the researchers. 2.3. Formative evaluation of test checklist Formative evaluation data were collected from a process whereby the test checklist was applied to evaluate the case study OHPC. The OHPC, not yet introduced to its end-users, was assessed by the researchers using each construct in the checklist. The
Web search on ISS research
Seminal overview arcle on ISS [15]
Electronic search “Informaon system success” on PubMed (n=11) and Scopus (n=305)
33 records
60 records
316 records
Citaon tracking
450 Abstracts
References of references
14 ISS instrument arcles [15-18, 22-31]
Six ISS instrument categories
ISS constructs for preliminary OHPC checklist
Fig. 1. Meta-narrative review on evaluation methods for IS success.
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
assessment data were collected from the OHPC requirements and design specifications. Field notes were made during meetings with the OHPC development team, as well as feedback reports from invited users. The specifications were represented as design rationales whereby the identified user needs were connected to alternative OHPC design solutions. The rationale behind the design choices made during the implementation, including why alternative solutions were rejected. The data from the formative evaluation of the checklist were categorized and interpreted by the researchers. The data were first categorized by the first author, and the interpretations thereafter established by the other two authors. Disagreements were resolved by discussions involving all authors. In the final step, the researchers consolidated their analysis with the regional public health representatives.
13
the website was about the right subject area; (2) trust, in that the method of developing content was sound and clearly communicated to the users; and (3) format, or that the content was presented in a way expected by the end-users. 3.1.3. Evaluation of information quality in the case study project One concern was that the maintenance of information quality was contingent on the continuous involvement of health professionals. To gain a long-term commitment from the health services organization, the health benefit of the OHPC needs to be demonstrated. Paradoxically, and unexpectedly, the long-term sustainability of information quality was thus found to be associated with measurements of health outcomes, in turn dependent on long-term sustainability. 3.2. Service quality
3. Results The adjusted checklist for pre-launch evaluation of OHPCs included the following constructs: information quality, service quality and subjective norms (Table 1). In the following, each construct and its formative evaluation are reported in the form of the definition of the construct, contextual adaptation of the construct, formative evaluation in the case study project, and finally the constructed checklist applicable to both the end-user community and the health care services. After using the preliminary checklist, some changes in terms of the actual OHPC and the routines for the OHPC editorial work were made (Tables 2, 4 and 6). The application of the checklist to the study OHPC is provided in Appendix A, with excepts in Tables 3, 5 and 7. 3.1. Information quality 3.1.1. Construct definition Information quality is defined as the degree to which information produced by the website has the attributes of content, accuracy, and format required by the users [17]. 3.1.2. Contextual adaptation to OHPCs Information quality indicates that the information is relevant and needed by the users with respect to content, accuracy and format. The content is mainly determined from the needs of the users, but this has to be balanced by the intended health services purpose of the OHPC. The perception of trustworthiness is important for users, but quality assurance is a more pressing priority for the health services community. The contextual adaptation of the construct resulted in check points for (1) content area, indicating that
3.2.1. Construct definition Service quality is defined as the degree of alignment between users’ expected and perceived service experiences [31]. 3.2.2. Contextual adaptation to OHPCs User preferences of service quality are not uniform, and it is important that the health services community have sustainable funding and personnel for the management of the website. The contextual adaptation of the construct resulted in check points for (1) staff competence, e.g. that the staff were qualified for their task, and that this expectation was properly communicated; (2) prompt service, that the health service had the capacity to manage the online activity; and (3) empathy, that the staff acknowledged that their online activities would be unbiased and honest. 3.2.3. Evaluation of service quality in the case study project A concern was raised regarding having moderators from the community working voluntarily in the OHPC. It is difficult to demand professionalism from voluntary adolescent moderators and it was found that their skills and level of professionalism must be closely monitored by supervisors. 3.3. Subjective norms 3.3.1. Construct definition Subjective norms is defined as the degree to which individuals perceive that others believe they should use the website [17]. 3.3.2. Contextual adaptation to OHPCs Subjective norms are the effect on use that comes from networking offline and online. As a prerequisite, the OHPC needs be
Table 1 OHPC evaluation constructs. Construct: definition
Construct evaluation questions
Information quality: Degree to which information produced by the OHPC has attributes of content, accuracy, and format required by the users
The site provides. . . . . .output that is exactly what you need . . .accurate information . . .information that is useful
Service quality: Degree to which users’ perceived and expected service align
OHPC staff give prompt service to users OHPC staff have the knowledge to do their job well OHPC has users’ best interest at heart
Subjective norms: Degree to which individuals perceive that others believe they should use the website
People who influence my behaviour think I should use the site People who are important to me think I should use the site I use the site because of the number of people I know who use it
14
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
Table 2 Information quality checklist. Information quality
End-user community
Health services
Content area
Is the content (subject areas and framing) of the OHPC in sync with the needs of the users? Is the framing of the content in sync with the preferences of the users?
Is the content (subject areas) of the OHPC in sync with the intent of the health service? Is the framing of the content in sync with the intent of the health service?
Trust
Is it possible for the users to assess the accuracy of the content? Is the source of the content clearly communicated?
Is there a method for quality assurance of the content? Is there an established guideline to resolve misinformation and conflicting information? Is the responsibility for the accuracy of the information clear? Is it clear by what authority, or role, content is provided from the health service? Is it possible to overview the extent of discussions, commentaries and remarks about content?
Is it possible to challenge the accuracy of the information? Is there a clear distinction between user-generated content and authorized content? Is there an established method for filtering relevant content? Format
Is the content presented in formats that are in sync with the user preferences?
Is the format for the content suitable for the content provider to work with?
Table 3 Except from Appendix A: information quality checklist application. Information quality
Health service
Application of checklist
Content area
Is the framing of the content in sync with the intent of the health service?
The project is high risk, in the sense that the health service invests considerable commitment and resources, hoping that the project will prevent obesity and empower the end users. It is unknown if such a project will generate the intended benefits. For the project to transform into routine practice, the health outcome needs to be demonstrated
End-user community Trust
Is it possible for the users to assess the accuracy of the content?
The participating health professionals have profiles online, but essentially there is no authority that guarantees the accuracy of the content, especially regarding the user-generated content. The project is based on the concept of apomediary quality control [43] where the community filters and directs itself to relevant content
Table 4 Service quality checklist. Service quality
End-user community
Health services
Staff competence
Are the rules for the staff transparent? Are the responsibilities of the staff clearly communicated? Are the tasks of the staff clearly communicated Is the agenda for the staff and website clearly communicated
Are the rules of the staff reasonable and sustainable? Is the website suitable to facilitate the expertise of the staff Are the tasks of the staff reasonable and sustainable? Is the agenda of the staff and website unbiased and honest?
Prompt service Empathy
Table 5 Except from Appendix A: Service quality checklist application. Information quality
Health service
Application of checklist
Empathy
Is the agenda for the staff and website unbiased and honest?
Partly. The professionals can be assumed to have the users’ best interests at heart, but there may be alternate motives among the voluntary moderators. This situation is going to have to be re-assessed periodically.
Table 6 Subjective norms checklist. Subjective norms
End-user community
Health services
Social facilitation
Is the OHPC tied to a geographic area or organization?
Interconnectivity
Is there functionality to connect content and interaction with other social media? Is it possible to interact with other users privately and publicly?
Are the OHPC staff tied to a region, organization, or adequate profession? Are there any issues with content being shared over social media?
Communication
Is there functionality to connect with the staff, for questions or private messaging?
Is there functionality to connect with individual users, for referral or moderation? Is there authority to connect with individual users, for referral or moderation?
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
15
Table 7 Except from Appendix A: Subjective norms checklist application. Information quality
Health service
Application of checklist
Interconnectivity
Are there any issues with content being shared over social media?
Communication
Is there authority to connect with a user for referral or moderation?
Possibly. The content may be challenged and discussed outside of the online community where there no longer is possible to respond and overview the discussion. This has be continue to be an on-going editorial discussion Not formally. The health professionals can recommend and, through their professional context, make a referral, but the site has no authority by itself. There is also a lack of routines and practice on how to assess individuals possibly in need of care online. This is something that needs to be re-assessed periodically
able to facilitate social connectivity. The contextual adaptation construct resulted in check points for (1) social facilitation, where users and the corresponding staff are tied together by some special interest, organization or region; (2) interconnectivity, whereby the site is social media friendly and problems are not introduced when content is moved outside the OHPC; and (3) communication, whereby users and staff are not hindered in their communication with each other, but communication is also manageable. 3.3.3. Evaluation of subjective norms in the case study project An important issue was identified regarding what happens to statements made by health professionals when shared in a new context over social media. In the world of social media, it is impossible to monitor how content is presented, framed, and used; subsequent exchange may lead to misinformation that can be dispersed unchallenged. Also, outdated or faulty information corrected through the community is not necessarily similarly updated on third party social platforms. Additionally, the website staff has no authority to act in situations when users may require interventional health care. Moreover, there was a lack of preparedness for how to assess serious health issues online. 4. Discussion This study set out to develop a checklist for pre-launch evaluation of OHPCs. The use of checklists as design tools has a long tradition in design methodology [32,33], and checklists have also been used in the development of health information websites [34] and in consumer health informatics [35]. Checklists are not intended for use as research tools, but are heuristic guidelines for system development practice. Neither is the objective of checklists to strive for conformity, but instead to ascertain that central design questions are addressed and that there are clear arguments supporting each divergent design decision [36]. The resulting checklist includes the following constructs: information quality, service quality and subjective norms. The most important result from its formative evaluation was that the delicate balance between community autonomy and quality control had to be clearly recognized in the formulation of the information and service quality constructs. In order to empower an online community, some control over content and moderation needs to be surrendered from the health services to the community in a managed process. There are several different uses for the checklist developed in this study. For example, it can be used in conjunction with practical development guides for health promotion web resources, [37] to cross-validate results from requirements analyses, and to ensure that design specifications have a reasonable scope. In website design, a site can be redesigned or re-imagined after implementation, based on the availability of technologies such as cascading style sheets (CCS) and CMSs, without having to alter the underlying technical architecture [38]. However, first implementations become more or less fixed once the basic style of the website has become known and used. The logotype, layout, and color scheme
are intimately associated with a website image and radical reimagination is more consequential the longer the time lapse after launch. The combined use of the checklist and participatory design methods [39] can overcome the problem of website image ‘‘stickiness’’ in online community design. The checklist can be used to adjust the OHPC design in order to prevent adverse effects of a bad first impression, while participatory design methods and technical flexibility allow for ongoing fine-tuning of technical functions. There are important limitations that must be taken into account when considering application of the results of this study. First, the checklist must not be mistaken for a research instrument; it is intended for use as a tool in design practice. Moreover, a checklist based on measures of IS success [15] risks focusing OHPC design on tangible features that characterize successful online communities, rather than promoting health. Focusing on IS success rather than use alone may be one way to lessen this problem. Research on online communities has identified several constructs shown to be associated with both the use and perceived net benefit of online communities as an IS [17]. The contextual adaptation of the constructs in this study resulted in items for content area, trust, format, staff competence, prompt service, empathy, social facilitation, interconnectivity, and communication. Nonetheless, these constructs should not be regarded as a final set. For instance, no construct presently addresses online moral conduct, although the service quality construct covers the consequences of biases and lack of transparency. Vibrant communities exist today where the discourse is characterized as uncivil and destructive. It is currently held that online conduct is less civil than the offline counterpart, and the anonymity of the Internet has been proposed as an explanation [40]. More importantly, no construct addresses health benefits per se. Health outcomes from an OHPC are not possible to appraise in pre-launch evaluations. In contrast to online disease prevention efforts [41], OHPC outcomes are also challenging to define in the summative evaluation context. The checklist presented in this study is based on the tradition of IS success [15], with emphasis on providing benefit to communities as defined by the community. However, the benefits determined by the online community may or may not correspond to the health outcomes stated in public health policies. The purpose of a meta-narrative review is to transcend terms and traditions when a phenomenon is studied in several disciplines [21]. In our study, the literature on ISS was reviewed in the search for checklist constructs. The strategy to discern candidate constructs was limited to this tradition and may thus have introduced a selection bias. Moreover, action research is a contextual research approach [19] and therefore generalizations, if done at all, should be done with caution. To ensure contextual soundness, health service practitioners participated in this study in their actual professional roles, and young people participated as OHPC users, rather than merely informants. Both the practitioners and young people provided data about and enacted the infrastructure and processes of the Swedish health services and general society. The needs anal-
16
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
ysis, design, and validation of the OHPC were embedded in its health services context. Although the development of the initial version of the checklist was influenced by the experience and prejudices of the researchers, the formative evaluation data comprised factual decisions and adjustments made in the development process, thus limiting researcher influence. It may, thus, be possible that vital aspects of a pre-launch OHPC evaluation checklist eluded analysis in this study, but it is less likely that unfounded design arguments were introduced as data. Before using the checklist, a comparison of the infrastructure and processes of the study context from which the checklist was abstracted with the target context is needed in order to determine what aspects of the checklist are irrelevant [42].
5. Conclusions A checklist for pre-launch evaluation of OHPCs including the constructs: information quality, service quality and subjective norms has been developed. A formative evaluation demonstrated the critical need to balance the autonomy of the online community with the professional control of health services quality expressed in the information and service quality constructs. Future studies addressing health outcome constructs for use in OHPC evaluations are warranted.
Authors’ contributions JE participated in the design of the study, performed the analysis and drafted the manuscript. EAG participated in the design of the study, performed the analysis and drafted the manuscript. TT participated in the design of the study, performed the analysis and drafted the manuscript. All authors read and approved the final manuscript. Acknowledgments The study was funded by a research grant from the Research Council for South-East Sweden (FORSS). The funding agency did not influence the research process in any aspect. We are grateful to all public health officials and adolescents in the region who made this study possible.
Appendix A. Supplementary material Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.jbi.2013.10.004. References [1] Johnson CM. A survey of current research on online communities of practice. Internet Higher Edu 2001;4:45–60. [2] Rheingold H. The virtual community. New York: Addison-Wesley; 1993. [3] Lave J, Wenger E. Situated learning: legitimate peripheral participation. Cambridge: Cambridge University Press; 1991. [4] Ekberg J, Ericson L, Timpka T, Eriksson H, Nordfeldt S, Hanberger L, et al. Web 2.0 systems supporting childhood chronic disease management: design guidelines based on information behaviour and social learning theories. J Med Syst 2010;34:107–17. [5] Delbanco T, Berwick DM, Boufford JI, Edgman-Levitan S, Ollenschläger G, Plamping D, et al. Healthcare in a land called PeoplePower: nothing about me without me. Health Expect 2001;4:144–50. [6] March S, Spence SH, Donovan CL. The efficacy of an Internet-based cognitivebehavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 2009;34:474–87. [7] Frost JH, Massagli MP. Social uses of personal health information within PatientsLikeMe, an online patient community: what can happen when patients have access to one another’s data. J Med Internet Res 2008;10:e15.
[8] Afrin LB, Greenland WJ, Finch NJ. Support group webcasting: a feasibility assessment. AMIA Annu Symp Proc 2005:880. [9] Schneider F, van Osch L, de Vries H. Identifying factors for optimal development of health-related websites: a Delphi study among experts and potential future users. J Med Internet Res 2012;14:e18. [10] Williamson DA, Walden HM, White MA, York-Crowe E, Newton Jr RL, Alfonso A, et al. Two year internet-based randomized controlled trail for weight loss in African American girls. Obesity 2006;14:1231–43. [11] Hallberg N, Timpka T, Eriksson H. The medical software quality deployment method. Methods Inf Med 1999;38:66–73. [12] Pilemalm S, Timpka T. Third generation participatory design in health informatics – making user participation applicable to large-scale information system projects. J Biomed Inform 2008;41:327–39. [13] Adams SA. Revisiting the online health information reliability debate in the wake of ‘‘web 2.0’’: an inter-disciplinary literature and website review. Int J Med Inform 2010;79:391–400. [14] Keselman A, Logan R, Smith CA, Leroy G, Zeng-Treitler Q. Developing informatics tools and strategies for consumer-centered health communication. J Am Med Inform Assoc 2008;15(4):473–83. http:// dx.doi.org/10.1197/jamia.M274. [15] DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manage Inform Syst 2003;19:9–30. [16] Molla A, Licker PS. E-commerce systems success: an attempt to extend and respecify the DeLone and McLean model of IS success. J Electronic Commerce Success 2001;2:131–41. [17] Schaupp LC, Belanger F, Fan W. Examining the success of websites beyond ecommerce: an extension of the IS success model. J Computer Inform Syst 2009;summer:42–52. [18] In KJ, Hye AS, Lee CC. A study on service quality determinants that influence continued success of portal online community service. In: Proceedings of the 2nd IEEE international conference on computer service and information technology, ICCSIT; 2009. p. 171–5. [19] Reason P, Bradbury-Huang H. Handbook of action research. 2nd ed. London: Sage; 2007. [20] Ekberg J, Angbratt M, Valter L, Nordvall M, Timpka T. History matters: childhood weight trajectories as a basis for planning community-based obesity prevention to adolescents. Int J Obes (Lond) 2012;36:524–8. [21] Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med 2013;11:20. [22] Kettinger WJ, Lee CC. Perceived service quality and user satisfaction with the information services function. Decis Sci 1995;25:737–65. [23] Pitt LF, Watson RT, Kavan CB. Service quality: a measure of information systems effectiveness. MIS Q 1995;19:173–88. [24] Mirani R, Lederer AL. An instrument for assessing the organizational benefits of IS projects. Decis Sci 1988;29:803–38. [25] Martinsons M, Davison MR, Tse D. The balanced scorecard: a foundation for the strategic management of information systems. Decis Support Syst 1999;25:71–88. [26] Kaplan RS, Norton DP. Translating strategy into action: the balanced scorecard. Boston (MA): Harvard Business School Press; 1996. [27] Tokzadeh G, Doll WJ. The development of a tool for measuring the perceived impact of information technology on work. Omega 1999;27:327–39. [28] Bailey JE, Pearson SW. Development of a tool for measuring and analysing computer user satisfaction. Manage Sci 1983;29:530–45. [29] Ives B, Olsen M, Baroudi JJ. The measurement of user information satisfaction. Commun ACM 1983;26:785–93. [30] Shirani A, Aiken M, Reithel B. A model of user information satisfaction. Data Base 1994;25:17–23. [31] Parasuraman A. Customer service in business-to-business markets: an agenda for research. J Bus Ind Market 1998;13(4/5):309–21. [32] Tetzlaff L, Schwartz DR. The use of guidelines in interface design. In: Butler K, Thomas J, editors. Human factors in computing systems. CHI ‘91 conference proceedings. Reading (MA): Addison Wesley Publishing; 1991. p. 329–33. [33] Martz W. Validating an evaluation checklist using a mixed method design. Eval Program Plan 2010;33:215–22. [34] Freund T, Freund M, Mahler C, Schwantes U, Ose D, Joos S, et al. Family practice web sites: new perspectives for patient care and health information? Fam Med 2010;42:501–6. [35] Misra R, Mark JH, Khan S, Kukafka R. Using design principles to foster understanding of complex health concepts in consumer informatics tools. AMIA Annu Symp Proc 2010(November):492–6. [36] Sjöberg C. Voices in design: argumentation in participatory development. PhD thesis no 436. Linköping (Sweden): Linköping University; 1994. [37] Valaitis R, O’Mara L, Bezaire S. Tailoring online health promotion: a toolkit for communities. Produced by Huron County Health Unit and McMaster University, School of Nursing; 2007. [38] Hirschheim R. Assessing participative systems design: some conclusions from an exploratory study. Inform Manage 1983;6:13–32. [39] Clemensen J, Larsen SB, Kyng M, Kirkevold M. Participatory design in health sciences: using cooperative experimental methods in developing health services and computer technology. Qual Health Res 2007;17:122–30. [40] Suler J. The online disinhibition effect. Cyberpsychol Behav 2004;7:321–6. [41] Wieland LS, Falzon L, Sciamanna CN, Trudeau KJ, Brodney S, Schwartz JE, et al. Interactive computer-based interventions for weight loss or weight maintenance in overweight or obese people. Cochrane Database Syst Rev 2012;8:CD007675. http://dx.doi.org/10.1002/14651858.CD007675.pub.
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17 [42] Mendel P, Meredith LS, Schoenbaum M, Sherborne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health service research. Adm Policy Ment Health 2008;35:21–37.
17
[43] Eysenbach G. From intermediation to disintermediation and apomediation: new models for consumers to access and assess the credibility of health information in the age of Web2.0. Stud Health Technol Inform 2007;129(Pt 1):162–6.
Contents lists available at ScienceDirect
Journal of Biomedical Informatics journal homepage: www.elsevier.com/locate/yjbin
Pre-launch evaluation checklist for online health-promoting communities Joakim Ekberg a,⇑, Elin A. Gursky b, Toomas Timpka a,c a
Department of Medical and Health Sciences, Linköping University, SE-581 83 Linköping, Sweden Analytic Services Inc., Arlington, VA, USA c Centre for Public Health Sciences, Linköping, Sweden b
a r t i c l e
i n f o
Article history: Received 22 October 2012 Accepted 16 October 2013 Available online 29 October 2013 Keywords: Community-based participatory research Health promotion Adolescents Health service
a b s t r a c t Background: Despite the apparent potential of online health-promoting communities (OHPC), there is limited guidance available for developers on the basic design features that contribute to successful applications. The aim of this study was to develop a checklist for a pre-launch evaluation of OHPCs incorporating the perspectives of both the user and the health services communities. Methods: The study was based on an action research design. Constructs previously applied to evaluate information system success were used as the basis for checklist development. The constructs were adapted for the OHPC context and formatively evaluated in a case study project. Evaluation data were collected from participatory observations and analyzed using qualitative methods. Results: The initial OHPC checklist included the constructs information quality, service quality, and subjective norms. The contextual adaptation of the information quality construct resulted in items for content area, trust, and format; the adaptation of the service quality construct in items for staff competence, prompt service and empathy; and the adaptation of the subject norms construct in items for social facilitation, interconnectivity and communication. The formative evaluation demonstrated the critical need to balance the autonomy of the online community with the professional control of health services quality expressed in the information and service quality constructs. Conclusions: A pre-launch OHPC evaluation checklist has been designed for use in practical development of health promotion web resources. Research on instruments for OHPC evaluations is warranted. Ó 2013 Elsevier Inc. All rights reserved.
1. Background As a result of the prevalent use of the Internet and social media, the community concept is no longer limited to physical locations [1]. An online community refers to a gathering of individuals in a virtual space that form a network and participate in long-term public discussions on common interests or experiences [2]. In contrast to traditional communities, online communities are not connected through a shared focus on physical tasks and activities [3]. Rather, emphasis is placed on cognitive tasks such as learning and planning, which are suitable for the virtual space [4]. Medical services have also embraced the opportunities and challenges of such virtual settings in the form of, e.g., Internet-mediated treatments
Abbreviations: CCS, cascading style sheet; CMS, content management system; IS, information system; OHPC, online health-promoting community. ⇑ Corresponding author. E-mail addresses: [email protected] (J. Ekberg), [email protected] (E.A. Gursky), [email protected] (T. Timpka). 1532-0464/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jbi.2013.10.004
[5,6], patient-governed clinical information websites [7] and support groups for disease self-management [8]. Prevention of specific diseases via web-based interventions has matured into a relatively well-documented intervention alternative. Unfortunately, few studies on web-based health promotion have been published [9,10], although across the prevention spectrum, health promotion may offer the greatest return on investment by sustaining quality of life and avoiding disease and injury. The development of open source and easy-to-use content management systems (CMSs) makes it possible to start up an online community. CMSs are flexible enough to host information resources addressing a wide range of topics related to lifestyle and healthy living conditions without the need for specialized computer science competence. Despite the apparent potential of online health-promoting communities (OHPC), limited guidance is available for developers on the basic design features that contribute to successful applications. For instance, the success of health promotion interventions in virtual environments is contingent on the sustained use of intervention resources, although use alone does not guarantee the intervention’s intended effect. The
12
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
development of a health information system is usually informed by requirements derived from both primary end-users and health service managers [11,12]. If end-user requirements are given precedence in the design of an OHPC, it may stimulate use but fail to lead to beneficial health effects. Conversely, if requirements derived from the health service are given precedence; the online community may fail to attract the intended end-users [13]. An online community can flourish only when it is perceived useful and is embraced by a critical amount of users sustaining online activity. Consumer health informatics refers to the provision of health information resources to consumers via the Internet. It has been noted that, in reaching out to health consumers, the field should make use of health information resources ranging from static informational Web pages to patient forums and virtual reality environments. Additionally, consumers’ information needs and usability concerns should be given specific consideration [14]. An individual’s motivation to use an online community is contingent on his/her perception of the usefulness of the online community and its specific cultural, social and technical context. This interplay between information system (IS) design and success has been associated with at least five constructs: systems quality, information quality, service quality, user satisfaction and net benefit [15]. Moreover, it has been found that these constructs need to be highly contextualized to specified information resources in order to be useful [16]. Previous research on online communities has pointed to several constructs shown to be associated with their success, both in terms of use rate and net benefit [17]. Service quality has been shown to be particularly important for user satisfaction [18]. The aim of this study was to develop a checklist for pre-launch evaluations of OHPCs that covers the perspectives of both the user community and the health services sector. The purpose of such a pre-launch checklist is to utilize experiences from previous evaluations of ISs to identify potential problems in an OHPC under development. In the present case, the checklist was used to evaluate an OHPC before its introduction to end-users. 2. Methods The study was based on an action research design. Action research involves the process of actively participating in an organization change situation while conducting research [19]. In the current study, development of a prototype OHPC evaluation checklist was initiated by researchers reviewing and analyzing the literature on IS success. The resultant checklist was applied as a formative evaluation in a case study implementation of an OHPC, where the researchers participated both as agents in the development process and as observers of the same process. Data from this process were collected and analyzed using qualitative methods. 2.1. Case study setting The case study county (total population 420,000) is situated in the south east of Sweden. To counteract the growing incidence of obesity among young adults, the county recognized the need for health policy and health promotion initiatives specific to this age group [20]. Consequently, both public health practitioners and researchers embarked on a public health intervention initiative aimed at adolescents and obesity. A needs analysis was performed using data from focus group interviews with adolescents, which was moderated by a researcher and a local public health representative. A requirements analysis performed by an interdisciplinary expert team consisting of two computer scientists, an interaction designer, a cognitive scientist, and a sample of young adults. A prototype OHPC was then drafted using the website content management system Joomla! (http://www.joomla.org). The resulting
OHPC design was demonstrated and discussed in meetings and through telephone conferences with representatives from regional public health services, public health researchers and senior public health officials. The OHPC design was subsequently modified as a result of this input. 2.2. Contextual adaption of OHPC evaluation checklist Constructs used in evaluations of IS success served as the basis for the formation of a prototype OHPC evaluation checklist. The inclusion criteria for concepts in the checklist were that (1) they had been reported valid for evaluation of IS success (ISS), and (2) were available for application before the introduction of the OHPC to the end users. A meta-narrative review [21] was performed on the literature on evaluation methods for IS success. The meta-narrative review method was chosen because it is pluralistic rather than normative (i.e., it asks not ‘what is the best approach to researching this topic?’ but ‘what can we learn from the range of different approaches?’). A web search on ISS research, a search of the PubMed database using the search terms ‘information system success’, and a seminal overview article of the IS success framework [15] were approaches used to identify a basic set of publication records (Fig. 1). The abstracts of these publications were assessed for relevance, and 14 publications were used in the final analysis of constructs [15– 18,22–31]. From these publications, six instrument categories were identified: the DeLone and McLean model of ISS [15], the SERVQUAL instrument [22], Mirani and Lederer’s instrument [24], adaptations of the Balanced Scorecard [25,26], instruments for measuring the individual impact of ISs [27], and the user satisfaction model [30]. Finally, constructs applicable to the pre-launch evaluation of OHPCs were selected from these instrument categories. The constructs were adapted to the OHPC context and combined into a preliminary checklist to be used in the formative evaluation. The justifications for each construct adaptation were recorded by the researchers. 2.3. Formative evaluation of test checklist Formative evaluation data were collected from a process whereby the test checklist was applied to evaluate the case study OHPC. The OHPC, not yet introduced to its end-users, was assessed by the researchers using each construct in the checklist. The
Web search on ISS research
Seminal overview arcle on ISS [15]
Electronic search “Informaon system success” on PubMed (n=11) and Scopus (n=305)
33 records
60 records
316 records
Citaon tracking
450 Abstracts
References of references
14 ISS instrument arcles [15-18, 22-31]
Six ISS instrument categories
ISS constructs for preliminary OHPC checklist
Fig. 1. Meta-narrative review on evaluation methods for IS success.
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
assessment data were collected from the OHPC requirements and design specifications. Field notes were made during meetings with the OHPC development team, as well as feedback reports from invited users. The specifications were represented as design rationales whereby the identified user needs were connected to alternative OHPC design solutions. The rationale behind the design choices made during the implementation, including why alternative solutions were rejected. The data from the formative evaluation of the checklist were categorized and interpreted by the researchers. The data were first categorized by the first author, and the interpretations thereafter established by the other two authors. Disagreements were resolved by discussions involving all authors. In the final step, the researchers consolidated their analysis with the regional public health representatives.
13
the website was about the right subject area; (2) trust, in that the method of developing content was sound and clearly communicated to the users; and (3) format, or that the content was presented in a way expected by the end-users. 3.1.3. Evaluation of information quality in the case study project One concern was that the maintenance of information quality was contingent on the continuous involvement of health professionals. To gain a long-term commitment from the health services organization, the health benefit of the OHPC needs to be demonstrated. Paradoxically, and unexpectedly, the long-term sustainability of information quality was thus found to be associated with measurements of health outcomes, in turn dependent on long-term sustainability. 3.2. Service quality
3. Results The adjusted checklist for pre-launch evaluation of OHPCs included the following constructs: information quality, service quality and subjective norms (Table 1). In the following, each construct and its formative evaluation are reported in the form of the definition of the construct, contextual adaptation of the construct, formative evaluation in the case study project, and finally the constructed checklist applicable to both the end-user community and the health care services. After using the preliminary checklist, some changes in terms of the actual OHPC and the routines for the OHPC editorial work were made (Tables 2, 4 and 6). The application of the checklist to the study OHPC is provided in Appendix A, with excepts in Tables 3, 5 and 7. 3.1. Information quality 3.1.1. Construct definition Information quality is defined as the degree to which information produced by the website has the attributes of content, accuracy, and format required by the users [17]. 3.1.2. Contextual adaptation to OHPCs Information quality indicates that the information is relevant and needed by the users with respect to content, accuracy and format. The content is mainly determined from the needs of the users, but this has to be balanced by the intended health services purpose of the OHPC. The perception of trustworthiness is important for users, but quality assurance is a more pressing priority for the health services community. The contextual adaptation of the construct resulted in check points for (1) content area, indicating that
3.2.1. Construct definition Service quality is defined as the degree of alignment between users’ expected and perceived service experiences [31]. 3.2.2. Contextual adaptation to OHPCs User preferences of service quality are not uniform, and it is important that the health services community have sustainable funding and personnel for the management of the website. The contextual adaptation of the construct resulted in check points for (1) staff competence, e.g. that the staff were qualified for their task, and that this expectation was properly communicated; (2) prompt service, that the health service had the capacity to manage the online activity; and (3) empathy, that the staff acknowledged that their online activities would be unbiased and honest. 3.2.3. Evaluation of service quality in the case study project A concern was raised regarding having moderators from the community working voluntarily in the OHPC. It is difficult to demand professionalism from voluntary adolescent moderators and it was found that their skills and level of professionalism must be closely monitored by supervisors. 3.3. Subjective norms 3.3.1. Construct definition Subjective norms is defined as the degree to which individuals perceive that others believe they should use the website [17]. 3.3.2. Contextual adaptation to OHPCs Subjective norms are the effect on use that comes from networking offline and online. As a prerequisite, the OHPC needs be
Table 1 OHPC evaluation constructs. Construct: definition
Construct evaluation questions
Information quality: Degree to which information produced by the OHPC has attributes of content, accuracy, and format required by the users
The site provides. . . . . .output that is exactly what you need . . .accurate information . . .information that is useful
Service quality: Degree to which users’ perceived and expected service align
OHPC staff give prompt service to users OHPC staff have the knowledge to do their job well OHPC has users’ best interest at heart
Subjective norms: Degree to which individuals perceive that others believe they should use the website
People who influence my behaviour think I should use the site People who are important to me think I should use the site I use the site because of the number of people I know who use it
14
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
Table 2 Information quality checklist. Information quality
End-user community
Health services
Content area
Is the content (subject areas and framing) of the OHPC in sync with the needs of the users? Is the framing of the content in sync with the preferences of the users?
Is the content (subject areas) of the OHPC in sync with the intent of the health service? Is the framing of the content in sync with the intent of the health service?
Trust
Is it possible for the users to assess the accuracy of the content? Is the source of the content clearly communicated?
Is there a method for quality assurance of the content? Is there an established guideline to resolve misinformation and conflicting information? Is the responsibility for the accuracy of the information clear? Is it clear by what authority, or role, content is provided from the health service? Is it possible to overview the extent of discussions, commentaries and remarks about content?
Is it possible to challenge the accuracy of the information? Is there a clear distinction between user-generated content and authorized content? Is there an established method for filtering relevant content? Format
Is the content presented in formats that are in sync with the user preferences?
Is the format for the content suitable for the content provider to work with?
Table 3 Except from Appendix A: information quality checklist application. Information quality
Health service
Application of checklist
Content area
Is the framing of the content in sync with the intent of the health service?
The project is high risk, in the sense that the health service invests considerable commitment and resources, hoping that the project will prevent obesity and empower the end users. It is unknown if such a project will generate the intended benefits. For the project to transform into routine practice, the health outcome needs to be demonstrated
End-user community Trust
Is it possible for the users to assess the accuracy of the content?
The participating health professionals have profiles online, but essentially there is no authority that guarantees the accuracy of the content, especially regarding the user-generated content. The project is based on the concept of apomediary quality control [43] where the community filters and directs itself to relevant content
Table 4 Service quality checklist. Service quality
End-user community
Health services
Staff competence
Are the rules for the staff transparent? Are the responsibilities of the staff clearly communicated? Are the tasks of the staff clearly communicated Is the agenda for the staff and website clearly communicated
Are the rules of the staff reasonable and sustainable? Is the website suitable to facilitate the expertise of the staff Are the tasks of the staff reasonable and sustainable? Is the agenda of the staff and website unbiased and honest?
Prompt service Empathy
Table 5 Except from Appendix A: Service quality checklist application. Information quality
Health service
Application of checklist
Empathy
Is the agenda for the staff and website unbiased and honest?
Partly. The professionals can be assumed to have the users’ best interests at heart, but there may be alternate motives among the voluntary moderators. This situation is going to have to be re-assessed periodically.
Table 6 Subjective norms checklist. Subjective norms
End-user community
Health services
Social facilitation
Is the OHPC tied to a geographic area or organization?
Interconnectivity
Is there functionality to connect content and interaction with other social media? Is it possible to interact with other users privately and publicly?
Are the OHPC staff tied to a region, organization, or adequate profession? Are there any issues with content being shared over social media?
Communication
Is there functionality to connect with the staff, for questions or private messaging?
Is there functionality to connect with individual users, for referral or moderation? Is there authority to connect with individual users, for referral or moderation?
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
15
Table 7 Except from Appendix A: Subjective norms checklist application. Information quality
Health service
Application of checklist
Interconnectivity
Are there any issues with content being shared over social media?
Communication
Is there authority to connect with a user for referral or moderation?
Possibly. The content may be challenged and discussed outside of the online community where there no longer is possible to respond and overview the discussion. This has be continue to be an on-going editorial discussion Not formally. The health professionals can recommend and, through their professional context, make a referral, but the site has no authority by itself. There is also a lack of routines and practice on how to assess individuals possibly in need of care online. This is something that needs to be re-assessed periodically
able to facilitate social connectivity. The contextual adaptation construct resulted in check points for (1) social facilitation, where users and the corresponding staff are tied together by some special interest, organization or region; (2) interconnectivity, whereby the site is social media friendly and problems are not introduced when content is moved outside the OHPC; and (3) communication, whereby users and staff are not hindered in their communication with each other, but communication is also manageable. 3.3.3. Evaluation of subjective norms in the case study project An important issue was identified regarding what happens to statements made by health professionals when shared in a new context over social media. In the world of social media, it is impossible to monitor how content is presented, framed, and used; subsequent exchange may lead to misinformation that can be dispersed unchallenged. Also, outdated or faulty information corrected through the community is not necessarily similarly updated on third party social platforms. Additionally, the website staff has no authority to act in situations when users may require interventional health care. Moreover, there was a lack of preparedness for how to assess serious health issues online. 4. Discussion This study set out to develop a checklist for pre-launch evaluation of OHPCs. The use of checklists as design tools has a long tradition in design methodology [32,33], and checklists have also been used in the development of health information websites [34] and in consumer health informatics [35]. Checklists are not intended for use as research tools, but are heuristic guidelines for system development practice. Neither is the objective of checklists to strive for conformity, but instead to ascertain that central design questions are addressed and that there are clear arguments supporting each divergent design decision [36]. The resulting checklist includes the following constructs: information quality, service quality and subjective norms. The most important result from its formative evaluation was that the delicate balance between community autonomy and quality control had to be clearly recognized in the formulation of the information and service quality constructs. In order to empower an online community, some control over content and moderation needs to be surrendered from the health services to the community in a managed process. There are several different uses for the checklist developed in this study. For example, it can be used in conjunction with practical development guides for health promotion web resources, [37] to cross-validate results from requirements analyses, and to ensure that design specifications have a reasonable scope. In website design, a site can be redesigned or re-imagined after implementation, based on the availability of technologies such as cascading style sheets (CCS) and CMSs, without having to alter the underlying technical architecture [38]. However, first implementations become more or less fixed once the basic style of the website has become known and used. The logotype, layout, and color scheme
are intimately associated with a website image and radical reimagination is more consequential the longer the time lapse after launch. The combined use of the checklist and participatory design methods [39] can overcome the problem of website image ‘‘stickiness’’ in online community design. The checklist can be used to adjust the OHPC design in order to prevent adverse effects of a bad first impression, while participatory design methods and technical flexibility allow for ongoing fine-tuning of technical functions. There are important limitations that must be taken into account when considering application of the results of this study. First, the checklist must not be mistaken for a research instrument; it is intended for use as a tool in design practice. Moreover, a checklist based on measures of IS success [15] risks focusing OHPC design on tangible features that characterize successful online communities, rather than promoting health. Focusing on IS success rather than use alone may be one way to lessen this problem. Research on online communities has identified several constructs shown to be associated with both the use and perceived net benefit of online communities as an IS [17]. The contextual adaptation of the constructs in this study resulted in items for content area, trust, format, staff competence, prompt service, empathy, social facilitation, interconnectivity, and communication. Nonetheless, these constructs should not be regarded as a final set. For instance, no construct presently addresses online moral conduct, although the service quality construct covers the consequences of biases and lack of transparency. Vibrant communities exist today where the discourse is characterized as uncivil and destructive. It is currently held that online conduct is less civil than the offline counterpart, and the anonymity of the Internet has been proposed as an explanation [40]. More importantly, no construct addresses health benefits per se. Health outcomes from an OHPC are not possible to appraise in pre-launch evaluations. In contrast to online disease prevention efforts [41], OHPC outcomes are also challenging to define in the summative evaluation context. The checklist presented in this study is based on the tradition of IS success [15], with emphasis on providing benefit to communities as defined by the community. However, the benefits determined by the online community may or may not correspond to the health outcomes stated in public health policies. The purpose of a meta-narrative review is to transcend terms and traditions when a phenomenon is studied in several disciplines [21]. In our study, the literature on ISS was reviewed in the search for checklist constructs. The strategy to discern candidate constructs was limited to this tradition and may thus have introduced a selection bias. Moreover, action research is a contextual research approach [19] and therefore generalizations, if done at all, should be done with caution. To ensure contextual soundness, health service practitioners participated in this study in their actual professional roles, and young people participated as OHPC users, rather than merely informants. Both the practitioners and young people provided data about and enacted the infrastructure and processes of the Swedish health services and general society. The needs anal-
16
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17
ysis, design, and validation of the OHPC were embedded in its health services context. Although the development of the initial version of the checklist was influenced by the experience and prejudices of the researchers, the formative evaluation data comprised factual decisions and adjustments made in the development process, thus limiting researcher influence. It may, thus, be possible that vital aspects of a pre-launch OHPC evaluation checklist eluded analysis in this study, but it is less likely that unfounded design arguments were introduced as data. Before using the checklist, a comparison of the infrastructure and processes of the study context from which the checklist was abstracted with the target context is needed in order to determine what aspects of the checklist are irrelevant [42].
5. Conclusions A checklist for pre-launch evaluation of OHPCs including the constructs: information quality, service quality and subjective norms has been developed. A formative evaluation demonstrated the critical need to balance the autonomy of the online community with the professional control of health services quality expressed in the information and service quality constructs. Future studies addressing health outcome constructs for use in OHPC evaluations are warranted.
Authors’ contributions JE participated in the design of the study, performed the analysis and drafted the manuscript. EAG participated in the design of the study, performed the analysis and drafted the manuscript. TT participated in the design of the study, performed the analysis and drafted the manuscript. All authors read and approved the final manuscript. Acknowledgments The study was funded by a research grant from the Research Council for South-East Sweden (FORSS). The funding agency did not influence the research process in any aspect. We are grateful to all public health officials and adolescents in the region who made this study possible.
Appendix A. Supplementary material Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.jbi.2013.10.004. References [1] Johnson CM. A survey of current research on online communities of practice. Internet Higher Edu 2001;4:45–60. [2] Rheingold H. The virtual community. New York: Addison-Wesley; 1993. [3] Lave J, Wenger E. Situated learning: legitimate peripheral participation. Cambridge: Cambridge University Press; 1991. [4] Ekberg J, Ericson L, Timpka T, Eriksson H, Nordfeldt S, Hanberger L, et al. Web 2.0 systems supporting childhood chronic disease management: design guidelines based on information behaviour and social learning theories. J Med Syst 2010;34:107–17. [5] Delbanco T, Berwick DM, Boufford JI, Edgman-Levitan S, Ollenschläger G, Plamping D, et al. Healthcare in a land called PeoplePower: nothing about me without me. Health Expect 2001;4:144–50. [6] March S, Spence SH, Donovan CL. The efficacy of an Internet-based cognitivebehavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 2009;34:474–87. [7] Frost JH, Massagli MP. Social uses of personal health information within PatientsLikeMe, an online patient community: what can happen when patients have access to one another’s data. J Med Internet Res 2008;10:e15.
[8] Afrin LB, Greenland WJ, Finch NJ. Support group webcasting: a feasibility assessment. AMIA Annu Symp Proc 2005:880. [9] Schneider F, van Osch L, de Vries H. Identifying factors for optimal development of health-related websites: a Delphi study among experts and potential future users. J Med Internet Res 2012;14:e18. [10] Williamson DA, Walden HM, White MA, York-Crowe E, Newton Jr RL, Alfonso A, et al. Two year internet-based randomized controlled trail for weight loss in African American girls. Obesity 2006;14:1231–43. [11] Hallberg N, Timpka T, Eriksson H. The medical software quality deployment method. Methods Inf Med 1999;38:66–73. [12] Pilemalm S, Timpka T. Third generation participatory design in health informatics – making user participation applicable to large-scale information system projects. J Biomed Inform 2008;41:327–39. [13] Adams SA. Revisiting the online health information reliability debate in the wake of ‘‘web 2.0’’: an inter-disciplinary literature and website review. Int J Med Inform 2010;79:391–400. [14] Keselman A, Logan R, Smith CA, Leroy G, Zeng-Treitler Q. Developing informatics tools and strategies for consumer-centered health communication. J Am Med Inform Assoc 2008;15(4):473–83. http:// dx.doi.org/10.1197/jamia.M274. [15] DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manage Inform Syst 2003;19:9–30. [16] Molla A, Licker PS. E-commerce systems success: an attempt to extend and respecify the DeLone and McLean model of IS success. J Electronic Commerce Success 2001;2:131–41. [17] Schaupp LC, Belanger F, Fan W. Examining the success of websites beyond ecommerce: an extension of the IS success model. J Computer Inform Syst 2009;summer:42–52. [18] In KJ, Hye AS, Lee CC. A study on service quality determinants that influence continued success of portal online community service. In: Proceedings of the 2nd IEEE international conference on computer service and information technology, ICCSIT; 2009. p. 171–5. [19] Reason P, Bradbury-Huang H. Handbook of action research. 2nd ed. London: Sage; 2007. [20] Ekberg J, Angbratt M, Valter L, Nordvall M, Timpka T. History matters: childhood weight trajectories as a basis for planning community-based obesity prevention to adolescents. Int J Obes (Lond) 2012;36:524–8. [21] Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med 2013;11:20. [22] Kettinger WJ, Lee CC. Perceived service quality and user satisfaction with the information services function. Decis Sci 1995;25:737–65. [23] Pitt LF, Watson RT, Kavan CB. Service quality: a measure of information systems effectiveness. MIS Q 1995;19:173–88. [24] Mirani R, Lederer AL. An instrument for assessing the organizational benefits of IS projects. Decis Sci 1988;29:803–38. [25] Martinsons M, Davison MR, Tse D. The balanced scorecard: a foundation for the strategic management of information systems. Decis Support Syst 1999;25:71–88. [26] Kaplan RS, Norton DP. Translating strategy into action: the balanced scorecard. Boston (MA): Harvard Business School Press; 1996. [27] Tokzadeh G, Doll WJ. The development of a tool for measuring the perceived impact of information technology on work. Omega 1999;27:327–39. [28] Bailey JE, Pearson SW. Development of a tool for measuring and analysing computer user satisfaction. Manage Sci 1983;29:530–45. [29] Ives B, Olsen M, Baroudi JJ. The measurement of user information satisfaction. Commun ACM 1983;26:785–93. [30] Shirani A, Aiken M, Reithel B. A model of user information satisfaction. Data Base 1994;25:17–23. [31] Parasuraman A. Customer service in business-to-business markets: an agenda for research. J Bus Ind Market 1998;13(4/5):309–21. [32] Tetzlaff L, Schwartz DR. The use of guidelines in interface design. In: Butler K, Thomas J, editors. Human factors in computing systems. CHI ‘91 conference proceedings. Reading (MA): Addison Wesley Publishing; 1991. p. 329–33. [33] Martz W. Validating an evaluation checklist using a mixed method design. Eval Program Plan 2010;33:215–22. [34] Freund T, Freund M, Mahler C, Schwantes U, Ose D, Joos S, et al. Family practice web sites: new perspectives for patient care and health information? Fam Med 2010;42:501–6. [35] Misra R, Mark JH, Khan S, Kukafka R. Using design principles to foster understanding of complex health concepts in consumer informatics tools. AMIA Annu Symp Proc 2010(November):492–6. [36] Sjöberg C. Voices in design: argumentation in participatory development. PhD thesis no 436. Linköping (Sweden): Linköping University; 1994. [37] Valaitis R, O’Mara L, Bezaire S. Tailoring online health promotion: a toolkit for communities. Produced by Huron County Health Unit and McMaster University, School of Nursing; 2007. [38] Hirschheim R. Assessing participative systems design: some conclusions from an exploratory study. Inform Manage 1983;6:13–32. [39] Clemensen J, Larsen SB, Kyng M, Kirkevold M. Participatory design in health sciences: using cooperative experimental methods in developing health services and computer technology. Qual Health Res 2007;17:122–30. [40] Suler J. The online disinhibition effect. Cyberpsychol Behav 2004;7:321–6. [41] Wieland LS, Falzon L, Sciamanna CN, Trudeau KJ, Brodney S, Schwartz JE, et al. Interactive computer-based interventions for weight loss or weight maintenance in overweight or obese people. Cochrane Database Syst Rev 2012;8:CD007675. http://dx.doi.org/10.1002/14651858.CD007675.pub.
J. Ekberg et al. / Journal of Biomedical Informatics 47 (2014) 11–17 [42] Mendel P, Meredith LS, Schoenbaum M, Sherborne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health service research. Adm Policy Ment Health 2008;35:21–37.
17
[43] Eysenbach G. From intermediation to disintermediation and apomediation: new models for consumers to access and assess the credibility of health information in the age of Web2.0. Stud Health Technol Inform 2007;129(Pt 1):162–6.
