COMMENTARY
Caveat Spectator: Digital Imaging and Data Manipulation Somjade J. Songcharoen, MD; Uldis Bite, MD; and Ricky P. Clay, MD
From the Division of Plastic and Reconstructive Surgery, University of Mississippi Medical Center, Jackson, MS (S.J.S., R.P.C.); and Division of Plastic and Reconstructive Surgery, Mayo Clinic, Rochester, MN (U.B.).
1036
I
n 2011, Tim Piper created a video clip featuring a short time lapse of a beauty photo shoot and postprocessing session on Photoshop showing a radical transformation.1 Although the video was created for the website GlobalDemocracy.com to support the need for a mandatory disclaimer for airbrushed models, it touched many more cultural issues surrounding the media and methods of advertising today. It has been used as part of a growing social outcry against unrealistic expectations and the objectification of women in the media. This video has been shared and uploaded through multiple social platforms, resulting in more than 10 million cumulative views on YouTube alone. The culture of image manipulation has been perpetuated in colloquialism by the word Photoshop and continues to be the subject of heated debate. For example, photographer and artist Anna Hill2 created a viral parody of this Photoshop culture using techniques that are rampant in mainstream advertising to create a series featuring herself as the model (Figure 1), touting Photoshop as the “all-in-one” kit that even performs plastic surgery. Indeed, image manipulation in the media is not a new concept. Even the supermodel Cindy Crawford once said, “I wish I looked like Cindy Crawford.”3 However, with the growing market in digital imaging technology, high-resolution digital cameras and editing software like Photoshop are quickly becoming more affordable and more attainable. In a recent marketing report published by BCC Research, the global digital photography market was valued at $68.4 billion in 2011 and projected to reach $82.5 billion in 2016.4 Similarly, digital imaging continues to grow in the scientific community, and with this growth, issues regarding improper and even fraudulent use of digital images have multiplied. Digital media have revolutionized the creation and consumption of information in the scientific community. In the education realm, they are used for teaching. In the scientific realm, digital images are considered data.5 In the medical
realm, they are a part of the patient record and can be used as evidence in medicolegal proceedings. Image manipulation is a complex issue in which the line separating the enhancement of information from its misrepresentation, whether intentional or not, remains obscure. Individuals,5 organizations,6,7 government departments, scientific journals,8 and research centers9 have all proposed methods to standardize guidelines and to provide tools for the detection of image manipulation. Despite these efforts, cases of scientific misconduct involving image manipulation continue to increase. In fact, the United States Department of Health and Human Services, Office of Research Integrity, has reported that the percentage of their integrity review cases involving image manipulation has grown from 4.1% in 1993 to 68% in 2007.10 In studies conducted by the Journal of Cell Biology (JCB),11 American Journal of Respiratory and Critical Care Medicine,12 and Blood,13 despite explicit instructions published for authors, 20% to 25% of reviewed and accepted articles at these journals had at least one figure that was noncompliant with the instructions. One of the most cited articles related to scientific image manipulation was written by Rossner and Yamada8 for JCB. Their characterization of image manipulation has been echoed in many other publications. These manipulations include gross misrepresentation, brightness/ contrast adjustments, cleaning up the background, splicing lanes, enhancing specific features, and employing linear vs nonlinear adjustments.8 Given the multitude of possible methods for manipulating high-quality digital images, it becomes problematic to detect those that meaningfully alter image content and accuracy. The growing market for digital imaging has introduced technological innovation faster than new products and techniques can be mastered by end users. Although modern digital imaging sensors are extremely sensitive and accurate, they will never have the same contextual information that we understand at a very
Mayo Clin Proc. n August 2014;89(8):1036-1041 n http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org n ª 2014 Mayo Foundation for Medical Education and Research
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
FIGURE 1. Photoshop parody. Reproduced with permission from Anna Hill.2
basic level. A simple example of this is evident in the difficulty that a camera might have resolving a snowy scene on a cloudy day, trying to find the color black when there is none, and searching for a point of reference to call white. Even without adjustments from the end user, linear and nonlinear adjustments are inevitably made from within the camera with its internal processor. These inaccuracies are further multiplied en route from the imaging sensor and camera, through computer hardware and software, and finally to a monitor. Each component assigns and handles unique ranges of colors known as color profiles. On the way to the publisher, an image can be compressed, decoded, and recompressed multiple times. Many of the guidelines published in scientific journals distinguish the enhancement of specific features in an image from global enhancements, with the latter being more generally accepted.14 Global changes, however, are not without consequences. Prasad et al14 recently reported a study in which digital images of 15 Papanicolaoustained specimens were manipulated using a widely available, free, open-source imaging Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
application called GIMP (GNU Image Manipulating Program); the images were then presented to 5 observers for evaluation. Although simple adjustments including brightness, contrast, and color were used globally, 41% of those altered images resulted in a different diagnosis.14 Although most incidences regarding the misuse of digital images are due to improper handling and manipulation,10 not all are completely innocent. In fact, in 1% of instances studied by JCB, the journal concluded that there was evidence of scientific fraud.5 These cases have even made their mark in high-profile studies, often fraught with controversy that spreads throughout the scientific community. One notable example relates to 2 retracted articles published in Science in 2004 and 2005.15,16 In these articles, a group led by Woo Suk Hwang published data claiming the successful derivation of a pluripotent embryonic stem cell line from a cloned human blastocyst. These cells were established by somatic cell nuclear transfer of human skin cells into donated oocytes, promising the potential for stem cell transplant. However, through a series of events sparked by suspicions
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1037
MAYO CLINIC PROCEEDINGS
surrounding egg donor misrepresentations followed by whistle-blowing by research associates, postings on a Korean website serving young scientists led to the discovery that the images of embryoid bodies were actually digital clones. These allegations led to investigations by Seoul National University and the use of independent DNA fingerprinting and analysis of mitochondrial DNA.17 In 2006, Hwang admitted to falsifying data and later was found guilty of buying human eggs in violation of the country’s bioethics law and of embezzling 830 million won (US $700,000) of government money.18 Although many issues regarding ethics, legality, and scientific fraud have come to light in the Hwang case, it illustrates an important point about how technology is driving change in the way that scientific information is both shared and manipulated. The imaging data in this case underwent scrutiny in a small social network that, in turn, led to the eventual investigations. This occurred in 2 very high-profile articles that nominally adhered to published journal guidelines and passed the journal’s stringent peer-review process. This issue raises an important point about how images are not given appropriate attention in journals. In a review of 399 high-impact biomedical journals, Bosch et al19 reported that only 35.1% provided explicit published definitions of misconduct, and 44.9% of those had published procedures for responding to misconduct. Of the reviewed journals, only 38.6% mentioned image manipulation as misconduct. Image processing programs like ImageJ20 used at the National Institutes of Health, as well as the Office of Research Integrity’s Advanced Forensic Actions,21 are examples of systems used to detect image manipulation. Although they are publicly available, they lack a user interface that is easy to use.22 Furthermore, detection software is used routinely by only a few journals. Figure 2 shows how some simple techniques can be used to detect some digital manipulation employed in the article by Hwang et al. Although image processing software packages are widely regarded as useful tools for the detection of inconsistencies, the sensitivity of detection will necessarily decrease with the ever-evolving methods by which files are handled by imaging software. When images are discussed in journal guidelines, they usually focus on postprocess manipulation of photos. However, there are many 1038
Mayo Clin Proc.
n
ways that an image can be misrepresented by the techniques used to obtain them. Examples of this are readily available in surgical fields. With the growing market for digital photographic equipment, medical photographers are being used to a lesser extent. As a consequence, there currently are more photographs published in scientific journals that were taken by someone with no formal training in photography, let alone medical photography. In a recent study published in the Journal of Oral and Maxillofacial Surgery, 1317 photos were analyzed for quality and scored on a 10-point scale for specific parameters.23 Of these parameters, technical aspects (including sharpness, depth of field, exposure, and white balance) scored lower than average when compared with other parameters such as positioning and environment. Certainly, deficiencies in technical ability can lead to a misrepresentation of any subject. However, even the most technically sound photograph can be limited in terms of perspective and positioning. Lens distortion is a ubiquitous problem. Most reports in the literature recommend a 90- to 105-mm lens24 for facial photography; however, cameras that are most widely available are marketed toward the consumer, featuring wider-angle lenses. These wider focal lengths (50 mm) will introduce barrel distortion, seen as central bulging and bent lines at the edge of the frame. This issue is further complicated by the many different sensor sizes, with lens specifications marketed on digital cameras as “35-mm equivalent.” Positioning and lighting are important factors that also fall outside the realm of postprocessing image manipulation. In 1990, Daniel et al25 reported that small changes in the position of lighting can change the reflection of light on the nose, thus modifying the perception of the size and position of the nasal tip. In another study, Riml et al26 found that the nose was affected most by positioning mistakes and camera angles (Figure 3). They found that a 10-degree lift of the head shortens the nasal dorsum by an average of 32.4% and decreases the angle of the jaw by 10.2%, while a 10degree lowering of the head elongates the nasal septum by 11.9% and shortens the upper lip and chin by 22.5% and 13.6%, respectively. Even a slight smile shortens the upper lip by 23.2% and lifts the cheek, decreasing the nasolabial angle by 14.4%.26 A slight smile is often
August 2014;89(8):1036-1041
n
http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
FIGURE 2. Supplemental Figure S1B in the article by Hwang et al16 was originally intended to show negative staining for a particular cell surface marker in 4 different cell lines. An adjustment of tonal range in Photoshop published in The Scientist indicates that the 2 middle images are identical. The minor differences in pixel structure are due to image compression. From The Scientist,11 with permission from the American Association for the Advancement of Science.
seen in postoperative photography and is usually passed off as the “happy patient.” In many ways, these are simple photographic techniques that can be used to replicate some of the digital work mentioned in the beginning of this commentary. One study by Scomação et al27 featured “pre” and “post” photos of 29 patients that were presented to a cohort of 119 plastic surgeons. The surgeons were led to believe that the patients had undergone a surgical procedure between the 2 images, but no procedure was actually performed. In the survey, the surgeons were asked to rate the change on a positive or negative 10-point scale, with zero indicating no change. “Post” photos featured differences in the background color or the addition of flash, moisture to the skin, makeup, a slight smile, or distance. Based on these changes alone, 45.3% of the scores were above zero, and the score change for the addition of a flash was statistically significant (Figure 4).27 Indeed, there are multiple ways to use and manipulate an image, and the ramifications go far beyond scientific journals. One company caught the attention of the media in 2012. Lifestyle Lift was founded in 2001, marketing a less invasive approach to the face-lift. Adoption of the techniques and the brand quickly grew to Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
include more than 50 affiliated surgical consultation offices nationally.28 With its growing popularity, the company was met with an increasing amount of criticism published anonymously on various review websites. In response, Lifestyle Lift allegedly asked its employees to post fabricated reviews online utilizing before and after photos of real patients.29 In a 2009 settlement with the New York State attorney general, Lifestyle Lift was fined $300,000 and ordered to cease specific marketing practices said to be “deceptive commercial practices, false advertising, and fraudulent and illegal conduct.”29 This was followed by a probe into marketing practices of Lifestyle Lift in Florida, in settlement of which Lifestyle Lift “agreed to change its marketing materials and practices to eliminate any possible consumer confusion about its services” by disclosing compensation to models and complying with Federal Trade Commission guidelines concerning the use of before and after photographs of models.30 Still, within the broader consumer market, websites other than that of Lifstyle Lift, and that include user-generated before and after pictures of patients, persist.31 When evaluating visual information within these websites, viewers must take into account positioning, camera
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1039
MAYO CLINIC PROCEEDINGS
the digital manipulation of properly acquired data for the purpose of making a figure clearer, more perfect and more consistent with the best images yielded in such experiments. Removing dust from a digitized photo with the erasure tool, cropping bands from gels, and playing with fluorescence micrographs to enhance a particular effect are all attempts to show better results than were actually achieved in that run.
FIGURE 3. Deviations from the standardized photo document in positioning of the head. Left, 10-Degree head lift. Right, 10-Degree lowering of the head. From J Plast Reconstr Aesthet Surg,26 with permission from Elsevier.
angle, facial expression, and whether there was standardization and consistency of lighting. Image manipulation in this digital age encompasses many different issues. Although it is important to scrutinize and screen for manipulated images before publication, it is also important to realize that image manipulation is here to stay and is ever changing. In a 2006 editorial, the scientific journal Nature coined the data alteration term beautification.32 They define the term as
FIGURE 4. No flash used on the left, only flash used on the right. From Aesthetic Plast Surg,27 with permission from Springer Science and Business Media.
1040
Mayo Clin Proc.
n
Nature further remarked that in “all these cases the data are legitimately acquired but then processed to yield an idealized image.”32 This is certainly similar to the culture of image manipulation that is catching the attention of the mass media. Similarly, Nature called for the full disclosure of instrument settings and the software manipulation performed on figures in an attempt to get “a little more real” and to find ways to regain trust in scientific images.32 In the editorial, Nature also announced the development of a concise guide for appropriate image handling.32 The fact remains, however, that the technology driving digital imaging will always be changing and improving. It is unlikely that any established or standardized technique or any detection method could remain perpetually relevant in the face of this change. Finally, the editorial also discussed how all researchers have always tended to “aspire to perfection.” Science, however, has never been in the business of absolutes as the word perfection might imply. The scientific method is, instead, an ongoing process of questioning, experimenting, and interpreting the data.33 How should we then combat the problem of data manipulation via the altering of images? Ideally, the widespread adoption of software to detect image manipulation would help, but, as we have noted, technological advancements will always offer an edge to the “photo hacker.” In addition, such expenditure may not be financially viable for smaller journals. Short of this, we feel that it is incumbent on every ethical researcher to disclose any image manipulation that has been performed on any figure or photo, even when it has been done only for the sake of clarity, and we further propose that it should be the stated policy of peer-reviewed journals that such disclosure is made and is considered on a par with the currently standard financial
August 2014;89(8):1036-1041
n
http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
disclosures. Although it will not deter the unethical to falsify a disclosure, such a policy would force authors to take a second look at their own techniques of image manipulation and evaluate for themselves how this may unintentionally mislead the reader. Finally, with the new technologies being adopted for photography and photographic manipulation comes a greater need to uphold the basic principles of the scientific method, such as reproducibility. A healthy skepticism will always remain the best defense for the data consumer. If other researchers cannot duplicate an outstanding photographic result, its validity should be openly questioned. We have now entered the age of “caveat spectator”d“let the viewer beware.” When you see something that appears too good to be true, it probably isn’t true. Correspondence: Address to Ricky P. Clay, MD, Division of Plastic and Reconstructive Surgery, University of Mississippi Medical Center, 2500 N State St, Jackson, MS 39216 ([email protected]).
REFERENCES 1. Piper T. Body evolutiondmodel before and after Photoshop. http://www.youtube.com/watch?v¼xKQdwjGiF-s&feature¼ youtube_gdata_player. Published November 2, 2013. Accessed December 30, 2013. 2. Hill, A. All In One. Nebulae Decay. Available at: http:// nebulaedecay.com/digi/4rbopin30pgrakdlpm0zim17u6ukod. Accessed December 30, 2013. 3. Kilbourne J. Killing Us Softly 4: Advertising’s Image of Women [DVD]. Northampton, MA: Media Education Foundation; 2010. 4. Dueck J, Elder M. Digital Photgraphy: Global Markets. Wellesley, MA: BCC Research; March 2012. http://www.bccresearch. com/market-research/information-technology/digital-photographyglobal-markets-ift030c.html. Accessed December 9, 2013. 5. Cromey DW. Digital images are data: and should be treated as such. Methods Mol Biol. 2013;931:1-27. 6. Scott-Lichter D; Editorial Policy Committee, Council of Science Editors. CSE’s White Paper on Promoting Integrity in Scientific Journal Publications, 2012 Update. 3rd rev ed. Wheat Ridge, CO: Council of Science Editors; 2012. 7. Iverson C; American Medical Association. AMA Manual of Style: A Guide for Authors and Editors. 10th ed. New York, NY: Oxford University Press; 2007. 8. Rossner M, Yamada KM. What’s in a picture? the temptation of image manipulation. J Cell Biol. 2004;166(1):11-15. 9. University of Alabama at Birmingham. Online Learning Tool for Research Integrity and Image Processing. http://www.uab. edu/researchintegrityandimages/default.html. Published 2008. Accessed December 30, 2013. 10. Cromey DW. Avoiding twisted pixels: ethical guidelines for the appropriate use and manipulation of scientific digital images. Sci Eng Ethics. 2010;16(4):639-667. 11. Rossner M. How to guard against image fraud. The Scientist website. http://www.the-scientist.com/?articles.view/articleNo/ 23749/title/How-to-Guard-Against-Image-Fraud/. Published March 1, 2006. Accessed December 13, 2013. 12. Abraham E, Adler KB, Shapiro SD, Leff AR. The ATS Journals’ policy on image manipulation. Am J Respir Cell Mol Biol. 2008;39(5):499. 13. Shattil SJ. A digital exam for hematologists [editorial]. Blood. 2007;109(6):2275.
Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
14. Prasad H, Wanjari S, Parwani R. Global manipulation of digital images can lead to variation in cytological diagnosis. J Pathol Inform. 2011;2:20. 15. Hwang WS, Ryu YJ, Park JH, et al. Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst [published correction appears in Science. 2005; 310(5755):1769; retracted in: Science. 2006;311(5759):335]. Science. 2004;303(5664):1669-1674. 16. Hwang WS, Roh SI, Lee BC, et al. Patient-specific embryonic stem cells derived from human SCNT blastocysts [published correction appears in Science. 2005;310(5755):1769; retracted in: Science. 2006;311(5759):335]. Science. 2005;308(5729): 1777-1783. 17. Beautification and fraud [editorial]. Nat Cell Biol. 2006;8(2): 101-102. 18. Cyranoski D. Woo Suk Hwang convicted, but not of fraud. Nature. 2009;461(7268):1181. 19. Bosch X, Hernández C, Pericas JM, Doti P, Marusic A. Misconduct policies in high-impact biomedical journals. PloS One. 2012;7(12):e51928. 20. Wayne Rasband. ImageJ: Image Processing and Analysis in Java. February 1, 2014: National Institute of Health. Available at: http:// rsbweb.nih.gov/ij/download.html. Accessed February 12, 2014. 21. US Department of Health and Human Services, Office of Research Integrity. Advanced Forensic Actions. Office of Research Integrity website. http://ori.hhs.gov/advanced-forensicactions. Updated July 26, 2012. Accessed February 12, 2014. 22. Parrish D, Noonan B. Image manipulation as research misconduct. Sci Eng Ethics. 2009;15(2):161-167. 23. Wu T, Chen S, Xiong X. Evaluation of the clinical photographs in the Journal of Oral and Maxillofacial Surgery: from readers’ perspectives. J Oral Maxillofac Surg. 2013;72(3):449-455. 24. Archibald DJ, Carlson ML, Friedman O. Pitfalls of nonstandardized photography. Facial Plast Surg Clin North Am. 2010; 18(2):253-266. 25. Daniel RK, Hodgson J, Lambros VS. Rhinoplasty: the light reflexes. Plast Reconstr Surg. 1990;85(6):859-866. 26. Riml S, Piontke A, Larcher L, Kompatscher P. Quantification of faults resulting from disregard of standardised facial photography. J Plast Reconstr Aesthet Surg. 2011;64(7):898-901. 27. Scomação IR, Graf RM, Maluf I Jr, Forte AJ, da Silva Freitas R. Evaluation of photographic variables in patients with indication for facial rejuvenation. Aesthetic Plast Surg. 2013;37(6): 1114-1119. 28. Lifestyle Lift founder Dr. David Kent named as Ernst & Young Entrepreneur Of The Year [press release]. Phoenix Business Journal website. http://www.bizjournals.com/phoenix/prnewswire/ press_releases/Georgia/2012/06/15/DA25773?ana¼prnews. Published June 15, 2012. Accessed February 12, 2014. 29. Schneiderman ET, New York Office of the Attorney General. Attorney General Cuomo secures settlement with plastic surgery franchise that flooded Internet with false positive reviews [press release]. http://www.ag.ny.gov/press-release/attorneygeneral-cuomo-secures-settlement-plastic-surgery-franchiseflooded-internet. Published July 14, 2009. Accessed February 12, 2014. 30. Bondi P, Florida Office of the Attorney General. Attorney General Pam Bondi reaches agreement with Lifestyle Lift regarding facial rejuvenation services [press release]. http://www.myfloridalegal. com/newsrel.nsf/newsreleases/8EB42E172E84A2D285257B8D 006B4E44. Published June 17, 2013. Accessed February 12, 2014. 31. RealSelf, Inc. Lifestyle Lift pictures. realself website. http://www. realself.com/lifestyle-lift/before-and-after-photos. Accessed February 14, 2014. 32. Not picture-perfect [editorial]. Nature. 2006;439(7079): 891-892. 33. Lanier WL, Rajkumar SV. Empiricism and rationalism in medicine: can 2 competing philosophies coexist to improve the quality of medical care [editorial]? Mayo Clin Proc. 2013;88(10): 1042-1045.
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1041
Caveat Spectator: Digital Imaging and Data Manipulation Somjade J. Songcharoen, MD; Uldis Bite, MD; and Ricky P. Clay, MD
From the Division of Plastic and Reconstructive Surgery, University of Mississippi Medical Center, Jackson, MS (S.J.S., R.P.C.); and Division of Plastic and Reconstructive Surgery, Mayo Clinic, Rochester, MN (U.B.).
1036
I
n 2011, Tim Piper created a video clip featuring a short time lapse of a beauty photo shoot and postprocessing session on Photoshop showing a radical transformation.1 Although the video was created for the website GlobalDemocracy.com to support the need for a mandatory disclaimer for airbrushed models, it touched many more cultural issues surrounding the media and methods of advertising today. It has been used as part of a growing social outcry against unrealistic expectations and the objectification of women in the media. This video has been shared and uploaded through multiple social platforms, resulting in more than 10 million cumulative views on YouTube alone. The culture of image manipulation has been perpetuated in colloquialism by the word Photoshop and continues to be the subject of heated debate. For example, photographer and artist Anna Hill2 created a viral parody of this Photoshop culture using techniques that are rampant in mainstream advertising to create a series featuring herself as the model (Figure 1), touting Photoshop as the “all-in-one” kit that even performs plastic surgery. Indeed, image manipulation in the media is not a new concept. Even the supermodel Cindy Crawford once said, “I wish I looked like Cindy Crawford.”3 However, with the growing market in digital imaging technology, high-resolution digital cameras and editing software like Photoshop are quickly becoming more affordable and more attainable. In a recent marketing report published by BCC Research, the global digital photography market was valued at $68.4 billion in 2011 and projected to reach $82.5 billion in 2016.4 Similarly, digital imaging continues to grow in the scientific community, and with this growth, issues regarding improper and even fraudulent use of digital images have multiplied. Digital media have revolutionized the creation and consumption of information in the scientific community. In the education realm, they are used for teaching. In the scientific realm, digital images are considered data.5 In the medical
realm, they are a part of the patient record and can be used as evidence in medicolegal proceedings. Image manipulation is a complex issue in which the line separating the enhancement of information from its misrepresentation, whether intentional or not, remains obscure. Individuals,5 organizations,6,7 government departments, scientific journals,8 and research centers9 have all proposed methods to standardize guidelines and to provide tools for the detection of image manipulation. Despite these efforts, cases of scientific misconduct involving image manipulation continue to increase. In fact, the United States Department of Health and Human Services, Office of Research Integrity, has reported that the percentage of their integrity review cases involving image manipulation has grown from 4.1% in 1993 to 68% in 2007.10 In studies conducted by the Journal of Cell Biology (JCB),11 American Journal of Respiratory and Critical Care Medicine,12 and Blood,13 despite explicit instructions published for authors, 20% to 25% of reviewed and accepted articles at these journals had at least one figure that was noncompliant with the instructions. One of the most cited articles related to scientific image manipulation was written by Rossner and Yamada8 for JCB. Their characterization of image manipulation has been echoed in many other publications. These manipulations include gross misrepresentation, brightness/ contrast adjustments, cleaning up the background, splicing lanes, enhancing specific features, and employing linear vs nonlinear adjustments.8 Given the multitude of possible methods for manipulating high-quality digital images, it becomes problematic to detect those that meaningfully alter image content and accuracy. The growing market for digital imaging has introduced technological innovation faster than new products and techniques can be mastered by end users. Although modern digital imaging sensors are extremely sensitive and accurate, they will never have the same contextual information that we understand at a very
Mayo Clin Proc. n August 2014;89(8):1036-1041 n http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org n ª 2014 Mayo Foundation for Medical Education and Research
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
FIGURE 1. Photoshop parody. Reproduced with permission from Anna Hill.2
basic level. A simple example of this is evident in the difficulty that a camera might have resolving a snowy scene on a cloudy day, trying to find the color black when there is none, and searching for a point of reference to call white. Even without adjustments from the end user, linear and nonlinear adjustments are inevitably made from within the camera with its internal processor. These inaccuracies are further multiplied en route from the imaging sensor and camera, through computer hardware and software, and finally to a monitor. Each component assigns and handles unique ranges of colors known as color profiles. On the way to the publisher, an image can be compressed, decoded, and recompressed multiple times. Many of the guidelines published in scientific journals distinguish the enhancement of specific features in an image from global enhancements, with the latter being more generally accepted.14 Global changes, however, are not without consequences. Prasad et al14 recently reported a study in which digital images of 15 Papanicolaoustained specimens were manipulated using a widely available, free, open-source imaging Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
application called GIMP (GNU Image Manipulating Program); the images were then presented to 5 observers for evaluation. Although simple adjustments including brightness, contrast, and color were used globally, 41% of those altered images resulted in a different diagnosis.14 Although most incidences regarding the misuse of digital images are due to improper handling and manipulation,10 not all are completely innocent. In fact, in 1% of instances studied by JCB, the journal concluded that there was evidence of scientific fraud.5 These cases have even made their mark in high-profile studies, often fraught with controversy that spreads throughout the scientific community. One notable example relates to 2 retracted articles published in Science in 2004 and 2005.15,16 In these articles, a group led by Woo Suk Hwang published data claiming the successful derivation of a pluripotent embryonic stem cell line from a cloned human blastocyst. These cells were established by somatic cell nuclear transfer of human skin cells into donated oocytes, promising the potential for stem cell transplant. However, through a series of events sparked by suspicions
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1037
MAYO CLINIC PROCEEDINGS
surrounding egg donor misrepresentations followed by whistle-blowing by research associates, postings on a Korean website serving young scientists led to the discovery that the images of embryoid bodies were actually digital clones. These allegations led to investigations by Seoul National University and the use of independent DNA fingerprinting and analysis of mitochondrial DNA.17 In 2006, Hwang admitted to falsifying data and later was found guilty of buying human eggs in violation of the country’s bioethics law and of embezzling 830 million won (US $700,000) of government money.18 Although many issues regarding ethics, legality, and scientific fraud have come to light in the Hwang case, it illustrates an important point about how technology is driving change in the way that scientific information is both shared and manipulated. The imaging data in this case underwent scrutiny in a small social network that, in turn, led to the eventual investigations. This occurred in 2 very high-profile articles that nominally adhered to published journal guidelines and passed the journal’s stringent peer-review process. This issue raises an important point about how images are not given appropriate attention in journals. In a review of 399 high-impact biomedical journals, Bosch et al19 reported that only 35.1% provided explicit published definitions of misconduct, and 44.9% of those had published procedures for responding to misconduct. Of the reviewed journals, only 38.6% mentioned image manipulation as misconduct. Image processing programs like ImageJ20 used at the National Institutes of Health, as well as the Office of Research Integrity’s Advanced Forensic Actions,21 are examples of systems used to detect image manipulation. Although they are publicly available, they lack a user interface that is easy to use.22 Furthermore, detection software is used routinely by only a few journals. Figure 2 shows how some simple techniques can be used to detect some digital manipulation employed in the article by Hwang et al. Although image processing software packages are widely regarded as useful tools for the detection of inconsistencies, the sensitivity of detection will necessarily decrease with the ever-evolving methods by which files are handled by imaging software. When images are discussed in journal guidelines, they usually focus on postprocess manipulation of photos. However, there are many 1038
Mayo Clin Proc.
n
ways that an image can be misrepresented by the techniques used to obtain them. Examples of this are readily available in surgical fields. With the growing market for digital photographic equipment, medical photographers are being used to a lesser extent. As a consequence, there currently are more photographs published in scientific journals that were taken by someone with no formal training in photography, let alone medical photography. In a recent study published in the Journal of Oral and Maxillofacial Surgery, 1317 photos were analyzed for quality and scored on a 10-point scale for specific parameters.23 Of these parameters, technical aspects (including sharpness, depth of field, exposure, and white balance) scored lower than average when compared with other parameters such as positioning and environment. Certainly, deficiencies in technical ability can lead to a misrepresentation of any subject. However, even the most technically sound photograph can be limited in terms of perspective and positioning. Lens distortion is a ubiquitous problem. Most reports in the literature recommend a 90- to 105-mm lens24 for facial photography; however, cameras that are most widely available are marketed toward the consumer, featuring wider-angle lenses. These wider focal lengths (50 mm) will introduce barrel distortion, seen as central bulging and bent lines at the edge of the frame. This issue is further complicated by the many different sensor sizes, with lens specifications marketed on digital cameras as “35-mm equivalent.” Positioning and lighting are important factors that also fall outside the realm of postprocessing image manipulation. In 1990, Daniel et al25 reported that small changes in the position of lighting can change the reflection of light on the nose, thus modifying the perception of the size and position of the nasal tip. In another study, Riml et al26 found that the nose was affected most by positioning mistakes and camera angles (Figure 3). They found that a 10-degree lift of the head shortens the nasal dorsum by an average of 32.4% and decreases the angle of the jaw by 10.2%, while a 10degree lowering of the head elongates the nasal septum by 11.9% and shortens the upper lip and chin by 22.5% and 13.6%, respectively. Even a slight smile shortens the upper lip by 23.2% and lifts the cheek, decreasing the nasolabial angle by 14.4%.26 A slight smile is often
August 2014;89(8):1036-1041
n
http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
FIGURE 2. Supplemental Figure S1B in the article by Hwang et al16 was originally intended to show negative staining for a particular cell surface marker in 4 different cell lines. An adjustment of tonal range in Photoshop published in The Scientist indicates that the 2 middle images are identical. The minor differences in pixel structure are due to image compression. From The Scientist,11 with permission from the American Association for the Advancement of Science.
seen in postoperative photography and is usually passed off as the “happy patient.” In many ways, these are simple photographic techniques that can be used to replicate some of the digital work mentioned in the beginning of this commentary. One study by Scomação et al27 featured “pre” and “post” photos of 29 patients that were presented to a cohort of 119 plastic surgeons. The surgeons were led to believe that the patients had undergone a surgical procedure between the 2 images, but no procedure was actually performed. In the survey, the surgeons were asked to rate the change on a positive or negative 10-point scale, with zero indicating no change. “Post” photos featured differences in the background color or the addition of flash, moisture to the skin, makeup, a slight smile, or distance. Based on these changes alone, 45.3% of the scores were above zero, and the score change for the addition of a flash was statistically significant (Figure 4).27 Indeed, there are multiple ways to use and manipulate an image, and the ramifications go far beyond scientific journals. One company caught the attention of the media in 2012. Lifestyle Lift was founded in 2001, marketing a less invasive approach to the face-lift. Adoption of the techniques and the brand quickly grew to Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
include more than 50 affiliated surgical consultation offices nationally.28 With its growing popularity, the company was met with an increasing amount of criticism published anonymously on various review websites. In response, Lifestyle Lift allegedly asked its employees to post fabricated reviews online utilizing before and after photos of real patients.29 In a 2009 settlement with the New York State attorney general, Lifestyle Lift was fined $300,000 and ordered to cease specific marketing practices said to be “deceptive commercial practices, false advertising, and fraudulent and illegal conduct.”29 This was followed by a probe into marketing practices of Lifestyle Lift in Florida, in settlement of which Lifestyle Lift “agreed to change its marketing materials and practices to eliminate any possible consumer confusion about its services” by disclosing compensation to models and complying with Federal Trade Commission guidelines concerning the use of before and after photographs of models.30 Still, within the broader consumer market, websites other than that of Lifstyle Lift, and that include user-generated before and after pictures of patients, persist.31 When evaluating visual information within these websites, viewers must take into account positioning, camera
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1039
MAYO CLINIC PROCEEDINGS
the digital manipulation of properly acquired data for the purpose of making a figure clearer, more perfect and more consistent with the best images yielded in such experiments. Removing dust from a digitized photo with the erasure tool, cropping bands from gels, and playing with fluorescence micrographs to enhance a particular effect are all attempts to show better results than were actually achieved in that run.
FIGURE 3. Deviations from the standardized photo document in positioning of the head. Left, 10-Degree head lift. Right, 10-Degree lowering of the head. From J Plast Reconstr Aesthet Surg,26 with permission from Elsevier.
angle, facial expression, and whether there was standardization and consistency of lighting. Image manipulation in this digital age encompasses many different issues. Although it is important to scrutinize and screen for manipulated images before publication, it is also important to realize that image manipulation is here to stay and is ever changing. In a 2006 editorial, the scientific journal Nature coined the data alteration term beautification.32 They define the term as
FIGURE 4. No flash used on the left, only flash used on the right. From Aesthetic Plast Surg,27 with permission from Springer Science and Business Media.
1040
Mayo Clin Proc.
n
Nature further remarked that in “all these cases the data are legitimately acquired but then processed to yield an idealized image.”32 This is certainly similar to the culture of image manipulation that is catching the attention of the mass media. Similarly, Nature called for the full disclosure of instrument settings and the software manipulation performed on figures in an attempt to get “a little more real” and to find ways to regain trust in scientific images.32 In the editorial, Nature also announced the development of a concise guide for appropriate image handling.32 The fact remains, however, that the technology driving digital imaging will always be changing and improving. It is unlikely that any established or standardized technique or any detection method could remain perpetually relevant in the face of this change. Finally, the editorial also discussed how all researchers have always tended to “aspire to perfection.” Science, however, has never been in the business of absolutes as the word perfection might imply. The scientific method is, instead, an ongoing process of questioning, experimenting, and interpreting the data.33 How should we then combat the problem of data manipulation via the altering of images? Ideally, the widespread adoption of software to detect image manipulation would help, but, as we have noted, technological advancements will always offer an edge to the “photo hacker.” In addition, such expenditure may not be financially viable for smaller journals. Short of this, we feel that it is incumbent on every ethical researcher to disclose any image manipulation that has been performed on any figure or photo, even when it has been done only for the sake of clarity, and we further propose that it should be the stated policy of peer-reviewed journals that such disclosure is made and is considered on a par with the currently standard financial
August 2014;89(8):1036-1041
n
http://dx.doi.org/10.1016/j.mayocp.2014.04.021 www.mayoclinicproceedings.org
MANIPULATION OF DIGITAL IMAGING AND SCIENTIFIC FRAUD
disclosures. Although it will not deter the unethical to falsify a disclosure, such a policy would force authors to take a second look at their own techniques of image manipulation and evaluate for themselves how this may unintentionally mislead the reader. Finally, with the new technologies being adopted for photography and photographic manipulation comes a greater need to uphold the basic principles of the scientific method, such as reproducibility. A healthy skepticism will always remain the best defense for the data consumer. If other researchers cannot duplicate an outstanding photographic result, its validity should be openly questioned. We have now entered the age of “caveat spectator”d“let the viewer beware.” When you see something that appears too good to be true, it probably isn’t true. Correspondence: Address to Ricky P. Clay, MD, Division of Plastic and Reconstructive Surgery, University of Mississippi Medical Center, 2500 N State St, Jackson, MS 39216 ([email protected]).
REFERENCES 1. Piper T. Body evolutiondmodel before and after Photoshop. http://www.youtube.com/watch?v¼xKQdwjGiF-s&feature¼ youtube_gdata_player. Published November 2, 2013. Accessed December 30, 2013. 2. Hill, A. All In One. Nebulae Decay. Available at: http:// nebulaedecay.com/digi/4rbopin30pgrakdlpm0zim17u6ukod. Accessed December 30, 2013. 3. Kilbourne J. Killing Us Softly 4: Advertising’s Image of Women [DVD]. Northampton, MA: Media Education Foundation; 2010. 4. Dueck J, Elder M. Digital Photgraphy: Global Markets. Wellesley, MA: BCC Research; March 2012. http://www.bccresearch. com/market-research/information-technology/digital-photographyglobal-markets-ift030c.html. Accessed December 9, 2013. 5. Cromey DW. Digital images are data: and should be treated as such. Methods Mol Biol. 2013;931:1-27. 6. Scott-Lichter D; Editorial Policy Committee, Council of Science Editors. CSE’s White Paper on Promoting Integrity in Scientific Journal Publications, 2012 Update. 3rd rev ed. Wheat Ridge, CO: Council of Science Editors; 2012. 7. Iverson C; American Medical Association. AMA Manual of Style: A Guide for Authors and Editors. 10th ed. New York, NY: Oxford University Press; 2007. 8. Rossner M, Yamada KM. What’s in a picture? the temptation of image manipulation. J Cell Biol. 2004;166(1):11-15. 9. University of Alabama at Birmingham. Online Learning Tool for Research Integrity and Image Processing. http://www.uab. edu/researchintegrityandimages/default.html. Published 2008. Accessed December 30, 2013. 10. Cromey DW. Avoiding twisted pixels: ethical guidelines for the appropriate use and manipulation of scientific digital images. Sci Eng Ethics. 2010;16(4):639-667. 11. Rossner M. How to guard against image fraud. The Scientist website. http://www.the-scientist.com/?articles.view/articleNo/ 23749/title/How-to-Guard-Against-Image-Fraud/. Published March 1, 2006. Accessed December 13, 2013. 12. Abraham E, Adler KB, Shapiro SD, Leff AR. The ATS Journals’ policy on image manipulation. Am J Respir Cell Mol Biol. 2008;39(5):499. 13. Shattil SJ. A digital exam for hematologists [editorial]. Blood. 2007;109(6):2275.
Mayo Clin Proc. n August 2014;89(8):1036-1041 www.mayoclinicproceedings.org
n
14. Prasad H, Wanjari S, Parwani R. Global manipulation of digital images can lead to variation in cytological diagnosis. J Pathol Inform. 2011;2:20. 15. Hwang WS, Ryu YJ, Park JH, et al. Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst [published correction appears in Science. 2005; 310(5755):1769; retracted in: Science. 2006;311(5759):335]. Science. 2004;303(5664):1669-1674. 16. Hwang WS, Roh SI, Lee BC, et al. Patient-specific embryonic stem cells derived from human SCNT blastocysts [published correction appears in Science. 2005;310(5755):1769; retracted in: Science. 2006;311(5759):335]. Science. 2005;308(5729): 1777-1783. 17. Beautification and fraud [editorial]. Nat Cell Biol. 2006;8(2): 101-102. 18. Cyranoski D. Woo Suk Hwang convicted, but not of fraud. Nature. 2009;461(7268):1181. 19. Bosch X, Hernández C, Pericas JM, Doti P, Marusic A. Misconduct policies in high-impact biomedical journals. PloS One. 2012;7(12):e51928. 20. Wayne Rasband. ImageJ: Image Processing and Analysis in Java. February 1, 2014: National Institute of Health. Available at: http:// rsbweb.nih.gov/ij/download.html. Accessed February 12, 2014. 21. US Department of Health and Human Services, Office of Research Integrity. Advanced Forensic Actions. Office of Research Integrity website. http://ori.hhs.gov/advanced-forensicactions. Updated July 26, 2012. Accessed February 12, 2014. 22. Parrish D, Noonan B. Image manipulation as research misconduct. Sci Eng Ethics. 2009;15(2):161-167. 23. Wu T, Chen S, Xiong X. Evaluation of the clinical photographs in the Journal of Oral and Maxillofacial Surgery: from readers’ perspectives. J Oral Maxillofac Surg. 2013;72(3):449-455. 24. Archibald DJ, Carlson ML, Friedman O. Pitfalls of nonstandardized photography. Facial Plast Surg Clin North Am. 2010; 18(2):253-266. 25. Daniel RK, Hodgson J, Lambros VS. Rhinoplasty: the light reflexes. Plast Reconstr Surg. 1990;85(6):859-866. 26. Riml S, Piontke A, Larcher L, Kompatscher P. Quantification of faults resulting from disregard of standardised facial photography. J Plast Reconstr Aesthet Surg. 2011;64(7):898-901. 27. Scomação IR, Graf RM, Maluf I Jr, Forte AJ, da Silva Freitas R. Evaluation of photographic variables in patients with indication for facial rejuvenation. Aesthetic Plast Surg. 2013;37(6): 1114-1119. 28. Lifestyle Lift founder Dr. David Kent named as Ernst & Young Entrepreneur Of The Year [press release]. Phoenix Business Journal website. http://www.bizjournals.com/phoenix/prnewswire/ press_releases/Georgia/2012/06/15/DA25773?ana¼prnews. Published June 15, 2012. Accessed February 12, 2014. 29. Schneiderman ET, New York Office of the Attorney General. Attorney General Cuomo secures settlement with plastic surgery franchise that flooded Internet with false positive reviews [press release]. http://www.ag.ny.gov/press-release/attorneygeneral-cuomo-secures-settlement-plastic-surgery-franchiseflooded-internet. Published July 14, 2009. Accessed February 12, 2014. 30. Bondi P, Florida Office of the Attorney General. Attorney General Pam Bondi reaches agreement with Lifestyle Lift regarding facial rejuvenation services [press release]. http://www.myfloridalegal. com/newsrel.nsf/newsreleases/8EB42E172E84A2D285257B8D 006B4E44. Published June 17, 2013. Accessed February 12, 2014. 31. RealSelf, Inc. Lifestyle Lift pictures. realself website. http://www. realself.com/lifestyle-lift/before-and-after-photos. Accessed February 14, 2014. 32. Not picture-perfect [editorial]. Nature. 2006;439(7079): 891-892. 33. Lanier WL, Rajkumar SV. Empiricism and rationalism in medicine: can 2 competing philosophies coexist to improve the quality of medical care [editorial]? Mayo Clin Proc. 2013;88(10): 1042-1045.
http://dx.doi.org/10.1016/j.mayocp.2014.04.021
1041
