REVIEWS

Real-World Data in the Molecular Era—Finding the Reality in the Real World DJ Dickson1 and JD Pfeifer2 Real-world data (RWD) promises to provide a pivotal element to the understanding of personalized medicine. However, without true representation (or the reality) of the patient-disease biosystem and its molecular contributors, RWD may hamper rather than help this advancement. In this review article, we discuss RWD vs. clinical reality and the disconnects that exist currently (emphasizing molecular medicine), and methods of closing the gaps between RWD and reality.

REAL-WORLD DATA, CLINICAL REALITY, AND THE IMPORTANCE OF MOLECULAR MEDICINE IN BOTH

“Real-world data” (RWD) is a basket term for information collected outside of the structured data gathering inside of a clinical trial or clinical trial-like registry. RWD can be obtained from various sources, including patient records (both electronic and hard copy), outside ancillary records (hospital, laboratory, radiology, pathology), pharmacy records, billing and payment data, patientreported information (such as questionnaires, blogs, or social media), etc. The aggregation of RWD into consolidated datasets is often termed as “big data.”

mation, but it may not completely or even partially represent what is actually happening in a given clinical situation. Often, the documented RWD is missing crucial pieces of information which are necessary to get a complete understanding of the patient or system. Sometimes, we erroneously attribute RWD as being reality. This is not necessarily the case. Outside of medicine, this confusion is also apparent. For example, what is generally called “reality TV” should probably be better termed as “realworld TV” in as much as, frequently, that what is shared is nothing more than an observed perspective on something that may or may not have any bearing on what is really happening in most individuals’ lives.

PROMISE OF BIG DATA (AGGREGATED REAL-WORLD DATA)

In 2012, the Obama administration announced the Big Data Research and Development Initiative explaining, “By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help accelerate the pace of discovery in science and engineering . . . and transform teaching and learning.” We live in an era where, theoretically, most everything can be collected, connected, packaged, parsed, mined, and refined to develop an image of a given individual or system. In health care, the promise of big data is that with larger and larger datasets, we can begin to see and identify signals, and develop previously nonapparent interventions that lead to improved outcomes.1–3 REAL-WORLD DATA DOES NOT NECESSARILY REPRESENT REALITY

Reality can be defined as that which constitutes the real or actual item, as distinguished from something that is merely observed, reported, or otherwise apparent. RWD can have valuable infor-

RWD IN THE MOLECULAR ERA

Over the last decade, what was traditionally observed (and often captured in paper or electronic records as RWD) were clinicopathological characteristics, medical and social history, physical examination findings, ancillary studies (laboratory, radiology, etc.), treatments, and clinical outcomes. Although valuable, these elements may not give the entire or even a significant portion of the clinical story of a given patient. Two patients with the same diagnosis, clinical prognosis, and treatment may have entirely distinct outcomes. In these cases, it is clear that something is missing that can identify the difference in the patients. In other words, the reality of these given patients is not predicted in the documented RWD. Over the last decades, we have learned that often the outcome disparity from two nearly identical patients is because of characteristics not routinely found in the traditional medical record. The molecular basis of disease or the “omics” of a patient and associated disease state are often a crucial but missing aspect of a given clinical scenario. The intersection of the

1 Molecular Evidence Development Consortium, Rexburg, Idaho, USA; 2Department of Pathology, Washington University School of Medicine, St. Louis, Missouri, USA. Correspondence: DJ Dickson ([email protected])

Received 13 October 2015; accepted 10 November 2015; advance online publication 00 Month 2015. doi:10.1002/cpt.300 186

VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS

Figure 1 Relationship between reality and RWD. RWD is generated when a patient undergoes some sort of interaction with someone or something that makes an observation, interprets this observation, and reports it in such a manner that it can (and will be) collected as RWD. Of note, the observation can be done by an individual alone, an instrument alone, or a combination of both.

genomics, proteomics, metabolomics, and epigenetics of a given system causes much of the difficulty of predicting the true clinical course or reality of a patient. Although environmental factors also cause disconnects, it is likely that these interact directly or indirectly with the molecular factors to modify disease and course. DEFINING REALITY

In the molecular era, reality can be described as “that which is actually occurring in a specific patient, which, if understood and appropriately treated, will achieve an expected outcome with little variation.” To the degree that RWD can predict the reality as defined in the previous paragraph is very valuable, and can begin to answer the promise of transforming medicine. Yet, just as the collected works of Shakespeare and all the commentaries ever written on such constitute a wonderful example of big data, it is unlikely that a dramatic advance in health care will come out of this dataset. It is not the act of collecting more and more RWD, but the collecting of what is really happening (reality) that advances medicine. IDENTIFYING METHODS OF IMPROVING THE DISCONNECT BETWEEN REALITY AND RWD

In this article, we discuss the basic hurdles that exist in finding the reality in the RWD of the molecular era. We highlight specific challenges and areas of concern. We also discuss methods of improving RWD and big data to allow a more rapid advancement of health care. Although much of the exploration of these disconnects will focus on oncology, the principles identified can readily be extrapolated to other disease states.

server consistency), and unless the resultant RWD is based on what is really occurring in the patient, we have developed a disconnect between reality and RWD. Claudius Galen, also known as Galen of Pergamon, the famous Greek physician born in AD 129, did much to codify the way that a patient should be approached. In his era, reality was that the patient had an ailment. To Galen and his followers, the observation, interpretation, documentation, and treatment of that reality was based on the erroneous belief that the patient had an imbalance of humors. Although the work of Galen was seminal in the beginning of Western physiology, it is unlikely that a vast collection of RWD generated during his era would have much value because of the marked disconnect between what was really happening and the RWD that would have been documented.10 RWD based on Galen’s work would be considered a gross distortion of reality (see Figure 2). Molecular medicine and health reality

Widespread molecular medicine and its relationship to health reality is a new phenomenon. It was only at the turn of the millennium that several groups reported separate gene expression patterns in tumors that beforehand had been classified as being similar.11–13 Later in the decade, molecular signatures of certain tumors were shown to correlate with prognosis.14–19 In many disease states, molecular markers are the most important element of a patient’s health reality, being key for diagnosis, treatment, and prognosis.20–24 As such, the molecular aspect of patient and disease is likely to become one of the most valuable aspects of a given patient’s history and treatment in almost all subspecialties of medicine and associated collected data.25–27

Health reality and the disconnects of RWD

The comprehensive and integrated nature of all aspects of an individual’s health at a distinct time can be called their point health reality. Over longer periods of time, the integration of an individual’s point health reality can be called the overall health reality or just health reality. When a patient’s point health reality is observed, interpreted, and then documented, RWD is generated (see Figure 1). Defining disconnects between health reality and RWD

The observing, interpreting, and then documenting of a patient may or may not capture reality. The act of observation, even with what seems to be strict criteria for interpretation, can be fraught with problems4–7 or it can be very consistent and accurate.8,9 Even if there is wide consensus on what to observe (intraobserver consistency), near-perfect interpretation and reporting (interob-

Deconstructing health reality and understanding molecular contributors

Before we can analyze disconnects between RWD and reality, it is helpful to break up what we generally consider the components of a patient’s health reality (see Figure 3). It is readily apparent that molecular medicine is a prominent component in each area. The interplay of these areas creates the outcome (or reality) for a given patient. In some cases, the molecular characteristics of the patient overwhelm everything else (trisomy 21, also known as Down syndrome), or the disease characteristics that are the driving force (hepatitis C). Both environment and treatment affect the patient or disease or the combined interaction for better or worse. In the treatment category, certain disease biomarkers mandate a specific treatment (BCRABL mutation in chronic myelogenous leukemia). Given the

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

187

REVIEWS

Figure 2 Example of common disconnection between reality and RWD. This depicts a graphical representation of reality, as demonstrated by the person holding a piece of paper representing an illness. In the act of observing, interpreting, and reporting the reality of the given patient, there are several examples of data disconnects. (a) Oversimplification: although the resulting data looks similar to the original, it is missing essential detail and depth. (b) Missing information: essential information that is necessary to understand the entire picture is missing entirely. (c) Binary transformation: taking data that has inherent nuances and variability and assigning a result of present or not. This is a common occurrence with molecular medicine and biomarkers. (d) Added information: seeing things that are not really present or inferring relationships or associations that do not really exist. (e) Distortion: fundamentally changing some aspects of reality in such a way that it alters true nature of the patient-disease interplay. This often comes from focusing on one element of reality that may distract from the overall picture. (f) Gross alteration: complete misunderstanding, misinterpreting, or misreporting reality. Note: Many disconnects are a combination of these examples and others not defined in this graphic.

ever-increasing complexity of the interactions of these areas, reliable understanding of molecular medicine will be needed to better unlock the reality of these systems.

Reality and reliability of molecular medicine

In order for RWD sets and big data to unlock the complicated interplay of molecular elements, reliable data points are needed.

Figure 3 General components of health reality. Health reality can be broken down into separate parts, which interact with varying influence on each other. In this representation, four major categories are represented, namely patient-specific characteristics, disease characteristics, treatment, and environmental effects. Each of the major categories are modified or driven by molecular factors to differing degrees. As such, the molecular basis of health reality is a crucial element to understand health reality. 188

VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS This is especially important when it comes to nonphenotypically manifested biomarkers or biomarkers that warrant a specific treatment. A genomic biomarker, such as trisomy 21, even if the testing were unreliable, would be manifested phenotypically, and any specific treatment for a patient is not based on the biomarker alone. The reliability of the testing is not as important as the understanding of the history and physical examination findings. The testing is still important but more for confirmation, prognostic, and counseling reasons, rather than essential for initial treatment. In contrast, a somatic translocation in an anaplastic lymphoma kinase oncogene in a patient with advanced nonsmall cell lung cancer is an example of a setting where treatment and prognosis both are dependent on reliable and accurate testing. If the analysis of the specimen is not done correctly and the alteration is missed, then the patient will never have access to important standard of care treatment. In both cases, reality is a fundamental molecular change that is a powerful driver of disease. In Down syndrome, the physical examination findings are enhanced little by the molecular testing other than to confirm the diagnosis, and any RWD collected on the patient is dominated by nonmolecular disease-specific information. In contrast, in the patient with anaplastic lymphoma kinase-translocated nonsmall cell lung cancer, the molecular testing is crucial to treatment and prognosis, and nonreliable testing will have a very adverse outcome on the patient. Given that most of the phenotypically manifested molecularly driven disease states have likely been identified (phenotype suggesting genotype), the future of medicine is determining how genotype (and broader -omics) determines phenotype. Companion diagnostics and their role in reality

Companion diagnostics (CDx) have a special role in defining the reality in molecular medicine. Although not perfect, these standardized tests and associated results are directly tied to a specific treatment and outcome, thereby coupling the testing-treatmentoutcome together. The US Food and Drug Administration (FDA) approves the drug and the device simultaneously based on the test identifying the group of patients who receive a specific intervention that leads to an improved outcome.28 Much of what we know about the reality of molecularly based disease treatment come from these interventions.29–31 Frequently, the same test will be used in other disease states; in these cases, the relationship between the diagnosis and the clinical results may be unknown, but at least the testing is standardized. If clinical benefit from treatment based on the results of the testing occur in other disease states, then the reproducibility of the diagnostic part of the system is crucial to being able to expand clinical knowledge and help define reality. PROBLEMS WITH MOLECULARLY CENTERED RWD

RWD hopes to understand the complex relationships between phenotype and genotype. However, if the genotype (or -omics) that make up the core of the RWD are not consistent between groups, then the chances of seeing true relationships is difficult or impossible without a great deal of detailed data that explores all

the perturbations. Even if one institution or group is able to be reliably consistent in observations, interpretations, and reporting, unless another group is acting in a very similar fashion, their datasets cannot be readily combined to answer these questions. In addition, unless the collected data are compared back to known phenotype-genotype testing results, it is unknown if the new method of genotyping can provide similar or improved results. There are significant areas of concerns with much of the molecularly focused RWD that currently exists. Without clear understanding of the results being collected and how they relate back to the clinical reality, a significant disconnect between RWD and reality is occurring. The following areas help define some of these major disconnects. 1. Lack of comparison with companion diagnostic

Although the well-studied CDx testing in its approved disease state and subsequent treatment is how clinical benefit has been proven (defines known reality), there has been no control as to the introduction of other testing that is meant to replace the CDx. These laboratory developed tests (LDTs), for the most part, are not compared with the corresponding CDx, let alone verified to demonstrate the same (or improved) outcomes when used in place of the CDx. In 2014, the FDA announced to the US Congress its intent to strengthen its oversight of LDTs. Margret Hamburg, then Commissioner of the FDA said “Ensuring that doctors and patients have access to safe, accurate and reliable diagnostic tests to help guide treatment decisions is a priority for the FDA. Inaccurate test results could cause patients to seek unnecessary treatment or delay and sometimes forgo treatment altogether. (This) action demonstrates the agency’s commitment to personalized medicine, which depends on accurate and reliable tests to get the right treatment to the right patient.” The National Comprehensive Cancer Network has also raised concerns regarding the lack of comparison of LDTs with the companion diagnostics.31 LDTs may not have the same sensitivity or specificity as the CDx. As such, it is possible for these LDTs to give false-negative results that would keep a patient from therapy, or false-positive results that would allow a patient to receive a treatment that is ineffectual. Furthermore, the pace that information is being generated on genetic causes of disease, coupled with the time required to gain FDA approval for a CDx, has created the situation in which only a small number of diseases have a CDx (and can even be expected to have a CDx). Because this situation guarantees that, for many diseases, the only option for testing is an LDT, the lack of control over the introduction of other testing virtually ensures that disconnects between RWD and reality will persist in the setting of molecularly centered disease analysis (as discussed in more detail below). 2. Binary capture of nonbinary reality estrogen receptor status

In 2010, estrogen receptor standards, in spite of being used for many years, were determined to be wrong in as many as 20% of patients, and new guidelines were put into place. These guidelines also decreased the threshold of estrogen receptor positivity to

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

189

REVIEWS Table 1 Example of reality vs. RWD in identical patients with breast cancer diagnosed in 2012 and then 2013 Reality

Grade 3

HER2-positive

Observation/interpretation/reporting

RWD

Year

2012

2013

Grade

Grade 3

Grade 3

IHC findings

11

11

Molecular (FISH) findings

Not indicated

FISH 1

Reporting

HER2-negative

HER2-positive

Intervention Not treated with HER2-based therapy

Treat with HER2-based therapy

FISH, fluorescence in situ hybridization; HER2, human epidermal receptor 2; IHC, immunohistochemical; RWD, real-world data. This table outlines how RWD is dependent on definitions and testing parameters that evolve over time. Reality: Two data elements that we think are important in determining a patient’s prognosis and affects treatment (reality), namely tumor grade and HER2 status. Although this is current reality, it is possible that this reality will be supplanted by newer understanding of what is really happening. Observation/intervention/reporting: This would lead to different results depending on the year and the definitions used at the time. RWD: Results by year for two different patients, diagnosed a year apart. In both patients, the reality and the RWD are connected in grade, but disconnected for HER2. Intervention: The patient in 2012 would not have received HER2-based therapy, which is important in both the adjuvant and metastatic setting.

>1% rather than >10%. Although there are some data to suggest that patients may respond even with the lower estrogen receptor thresholds, most oncologists will agree that there is a fundamental difference in women with estrogen receptor of 1% compared to women with a much greater percentage. In most medical records, the only data point for receptor status is a binary positive or negative. By changing the standard and not collecting the quantifiable (although imperfect) percentage either before or after, there is a group of women who are likely phenotypically different who have abruptly changed groups. Data collected before 2010 and the patients whose records were not updated are negative who now should be positive in the EMR and RWD. Women presently who are positive would have been negative. This binary approach to a biomarker could have been solved if results were captured in a quantitative manner rather than a qualitative one.32–35 3. Changing definitions without version updates

The evolution of the human epidermal growth factor receptor-2 (HER2) biomarker analyses is similar to the estrogen receptor story. Notwithstanding the early struggles to find standards for HER2 testing, in 2013, the American Society of Clinical Oncology (ASCO), in connection with the College of American Pathologists, released updated guidelines, recommending fluorescence in situ hybridization analysis for HER2 when an initial immunohistochemical workup was negative or 11 in women with grade 3 tumors.36 Based on these new guidelines, a hypothetical patient before 2013 would not have been tested by fluorescence in situ hybridization and any RWD collected for that patient would consider them HER2-negative, when, in reality, they were HER2-positive. Even if this patient were retested with the new regulation, it is unclear how the results could be aggregated with other patients, either HER2-positive or HER2-negative, given that they would not have received first-line treatment with HER2 therapy in the right sequencing, which may or may not change the prognosis (see Table 1). Even with the 2013 standards in HER2 testing there are still controversies.37 This example shows the problems of changing 190

versions of testing without detailing how this change creates possible disconnects in RWD. 4. Lack of testing standards and comparisons: same technology—no consistency

There are several examples of problems that have arisen when strict central standards for testing as part of clinical studies have not been required, even with the same technology. For example, immunohistochemical HER2 testing in several pivotal studies was initially allowed to be done using an institution’s own Clinical Laboratory Improvement Amendment accredited laboratory, but after low confirmation with oversight review, central testing or validation was required.38 The agreement with the central laboratory was only 80%.39 NEXT GENERATION SEQUENCING DIFFERENT PLATFORMS

Recently, several groups have compared next generation sequencing (NGS) platforms using same specimens and have showed differences.40–42 Boland et al.40 used three NGS platforms to analyze the exact same cell lines and look for concordance. Even when only requiring two of three platforms to find the same alteration to call positive, the concordance is only at 71% for single nucleotide polymorphisms (SNPs), and a dismal 25% for insertion/deletions. In contrast, if a false-negative result is defined as only one of the platforms finding an alteration that was not identified by the other two, the false-positive rate is as high as 21% for SNPs, and 62% for insertion/deletions (see Figure 4). Although there are differences in results of different platforms, it is unclear how or if these differences translate into clinical benefit. Different biomarker profiles and higher sensitivities may lead to identifying more technically treatable, but clinically irrelevant passenger mutations or subclones that may have no relationship with reality. Burghel et al.43 showed much better concordance between NGS platforms, but still identified significant differences because of false-positive results and inherent technological differences. VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS

Figure 4 False-positives and false-negatives from comparing three NGS platforms. SNP, single nucleotide polymorphisms. Indels, insertions and deletions. These figures are adapted from Boland et al.40 showing the concordance of three distinct NGS platforms using the same human cell lines, preparation methods, and analysis as recommended by the manufacture of the platform. (a) True-positives are defined as where two of three or three of three of the platforms found the same results. These numbers are averages of the findings. (b) False-positives are defined as where only one of the platforms found an alteration. These numbers are additive given that they contribute to the total number of results found in the given specimens.

NGS variant call software and sample preparation

One of the reasons for the difference in NGS platforms is due to the software used to interpret the raw data given by the instrument. Analysis of molecular data is complex. There are various ways to look at the data using either single or sequential software programs, each giving different results, with some generating higher numbers of false-positive results than others.44–48 Sample preparation before analysis is also crucial. Various techniques have been shown to either improve or degrade specimen quality in analysis.49 Each method has its own challenges that need to be overcome.50 Although a great deal of work has been done, most of it has been in the preclinical space and much is needed to be done before we can begin to compare NGS data points (RWD) to what is really happening with the patient.51,52 NEW TECHNOLOGY—LACK OF BACKWARD COMPARISON TO PREVIOUS CLINICAL OUTCOMES

When different technology has been compared, there are also differences in results. A patient that was negative for a mutation using older technology may be found to harbor an alteration by the use of newer technology. For example, Drilon et al.53 identified patients that were lifetime light or never smokers and reanalyzed tumor specimens that had already undergone a very extensive testing methodology but were negative. Using NGS, they identified 65% of patients who had mutations for which there was either a clinical trial or identified agent for which they would not have been eligible previously. Tuononen et al.54 showed similar results, comparing traditional polymerase chain reaction to NGS. There was high concordance of the genes tested (96.3% to 100%) between the platforms, but there were more alterations found by NGS than polymerase chain reaction. Although Drilon et al.53 had a few case reports of patients whose

increased sensitivity of the test led to early positive clinical effect, it generally is not clear if increased sensitivity translates to an improved clinical benefit in either study. 5. Lack of clinical outcome data: actionable means target, not necessary clinical benefit (or reality)

In the molecular era, the term “actionable” has been applied to any biomarker for which there is a drug that can target any part of the pathway linked to this finding. This term encompasses trial drugs for which there is still no proven benefit, or for wellestablished drugs that have not shown consistent or widespread benefit when used outside of one major tissue type (e.g., tamoxifen in melanoma; B-Raf inhibitors in colon cancer). Reality in this setting can be defined as is when a certain biomarker is observed, intervention done to target that biomarker/pathway, and a consistent response is demonstrated. Reality is a small subset of actionability. Therefore, although tamoxifen targets the estrogen receptor pathway, which is usually a driver pathway in estrogen receptor positive breast cancer, it is usually a passenger mutation in melanoma. In both cases, estrogen receptor positivity would be actionable, but only in breast cancer does it show benefit (reality). Unless there is appropriate clinical benefit surrounding the biomarker’s “actionability,” it is possible that the patient will receive either unproven or even ineffectual treatment. Germline vs. somatic mutations

Jones et al.55 compared alterations found in tumor tissue with paired normal tissue to determine overlap. Using two different NGS methods, looking at both targeted DNA sequences, or using a broader exome analysis, they identified alterations in somatic tumor tissue that were called actionable yet were present in the underlying germline analysis 31% of the time with targeted analysis and 65% of with broader exome sequencing. These findings

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

191

REVIEWS were interpreted as evidence of so-called overlap “false-positives” and lead to a recommendation of strong consideration of testing somatic tissue along with host germline. However, without clinical outcomes in these patients it is impossible to determine clinical reality of the findings. If the germline mutation acted as a driver that contributed to the formation of the tumor and has the capability to respond to therapy, then the finding in the tumor is a true-positive. In converse, if the alteration is simply a passenger gene that has no major role in the molecular pathways, treatments, or outcomes that determine reality, then the identification of this alteration is considered a nonclinically relevant (not real) alteration. RWD, REALITY, AND FALSE-POSITIVES

In some settings, false-positive results have perhaps the greatest potential for creating a rift between reality and RWD. If our current reality has been defined by information obtained through a companion diagnostic and associated drug, then anything that changes the testing portion of this reality has the possibility of creating a disconnect from the current outcome-based reality. There are two general arenas in which clinical false-positive results can be generated, and these errors can compound. CLINICAL TRUE NEGATIVE (REALITY)—ANALYTICALLY TRUE POSITIVE (RWD) Low allele frequency/tumor heterogeneity/increased sensitivity

Many tumors are composed of genetically and phenotypically distinct subpopulations that are continually evolving.56 This tumor heterogeneity often lends to clonal response, which may or may not translate to clinical benefit.57 For example, a lung cancer specimen may only have 1% of all sampled and analyzed cells harboring a treatable mutation, such as epidermal growth factor receptor. A very sensitive test (which includes much of the newer molecular technology) may rightly pick up this clone, and analytically call it a true-positive (which will be recorded in the RWD as being epidermal growth factor receptor-positive), but given that it is such a small cell population, targeted therapy, even if completely effective, would unlikely show clinical benefit. It is unclear at what allele frequency a patient will receive clinical benefit when treated with an associated proven therapeutic, although we know that in some disease states, such as Her2-positive breast cancer, that overexpression is associated with response.58 The increasing sensitivity of advanced testing identifying biomarkers is most certainly increasing the number of clinically true-negative and analytically true-positive results. Weiss et al.41 recently compared their commercially available NGS platform to another well-respected commercial laboratory. Differences were striking in that clinically actionable alterations were identified in 40% vs. 18% of cases between the laboratories, but whether the increased number of mutations identified resulted in different patient outcomes is unknown. If we assume that both laboratories are correct and there are no false-positives in either dataset, then the added numbers can be attributed to increasing sensitivity. 192

Preexisting germline mutations

As mentioned in the previous section, Jones et al.55 identified preexisting germline mutations in 31% and 65% of patients who had the same alteration in somatic tissue and tested by targeted and whole exome sequencing, respectively. It is unclear how or if many of these alterations served as a precursor to or drivers of the malignancy, and, as such, although the mutation may be present and a true-positive by RWD standards, it has no bearing on the clinical reality of the patient. Variants of uncertain significance

As whole exome sequencing becomes more commonplace, certain genes will be found to have alterations, but unless there is consistent identification and strong phenotype associated with these alterations, the clinical impact (if any) of these genes may be missed. Depending on the platform, the regions of analytic interest, and findings, a mutation may be classified as deleterious or even as a variant of uncertain significance but may be reported as a true-positive mutation, although in reality it may be a nondeleterious mutation clinically (negative in reality). CLINICAL (REALITY) TRUE-NEGATIVE—ANALYTICALLY (RWD) FALSE-POSITIVE

Newer testing to replace a CDx is often able to obtain superior sensitivity, but it is unclear if this higher qualitative sensitivity translates into higher clinical response. In addition, as sensitivity increases, there is generally an increase in the number of falsepositives. The false-positive rate for treatable biomarkers is difficult to estimate, but when looking at generalities, on average, the analytical false-positive rate for SNPs correlates with the platform, sensitivity of the analysis, and allele frequency.40,59 When insertion and deletions are added to the SNPs, the rate is even greater. CUMULATIVE NATURE OF FALSE-POSITIVES IN REALITY

Each of the previous areas is mutually exclusive of each other. As such, their effects are additive. In each case, the patient would be catalogued as being biomarker-positive when in truth they have no chance of responding to therapy. An example of how this can take place is demonstrated in Figure 5; of significant note, every platform will have its own set of ratios, and some platforms may sacrifice sensitivity to remove the false-positives. The national trend has been to increase sensitivity, which likely leaves Figure 5 as a close representation of much of what is happening at the current time. The exact rate of false-positives is unknown but it is almost certain that the number of false-positives is quite substantial. Statistical significance of RWD false-positives

With a substantial false-positive rate, hypothesis can only be generated and definitively evaluated when they are based on larger patient numbers where there is consistency in the testing. A definitive evaluation may require testing of dozens of patients to detect even a small signal unless the treatment effect is very large or the specificity of the testing limits the number of patients VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS

Figure 5 The estimated disconnect between reality and real-world reported biomarkers in NGS. Left: In the real world only a portion of reported positives actually have any clinical meaning and much of the currently reported information is likely clinically erroneous. This chart estimates the disconnect between RWD and reality, with over half of the reported values actually not having any clinical correlation. True-positives (reality): these are the biomarkers have clinical significance. False-positives (reality): biomarkers reported as being positive but actually have no clinical meaning. Right: Identifies the three main categories that make up the reality based false-positives on the left. Real-world false-positives (RWFPs) are mainly due to analytical and processing limitations calling a biomarker that is not really there. Real-world true-positives (RWTPs) are broken up into two categories, namely alterations that are present, but are not clinically significant for the given system due to underlying germline alterations or variants of unknown significance (VUS) and low allele frequency alterations that do not control the clinical behavior of the malignancy.

entering into study to those whose reported testing results closely match clinical reality. SILOS OF DATA

No discussion of RWD and reality would be complete without discussing the lack of interconnectivity and sharing amongst different stakeholders. Data have become a commodity, with many nonprofit groups partnering with for-profit entities to develop shared revenue models in exchange for access to their proprietary

data. There has been some reticence to work together on developing joint approaches to standards to testing, data collection, and reporting. The reasons for these are myriad, including privacy, complexity, and at least some concern for losing possible future revenue or competitive advantage.60 Standards for testing, and methods of sharing are starting to develop, but substantial hurdles exist. Recently ASCO has requested of the United States Congress legislative action to “ensure widespread interoperability of electronic health records and prohibit information blocking.”61 This is, in part, because of lack of access to records that ASCO desires to build their shared, although ASCO proprietary CancerLinQ initiative. Methods of improving RWD (closing disconnects with reality)

To close the disconnects between RWD and health reality, there are several key steps. These are detailed below.

Figure 6 The interplay of factors (including genotype) determine phenotype. Although genotype and genome are almost identical words, genotype is a much broader term that encompasses the individual genetics and downstream characteristics and interactions. The interplay of molecular factors, such as genome, transcriptome, proteome, metabolome, which are being altered by epigenetic, environmental, and medical interventions, all modulated by the immune system makes up a more complicated picture of what is a genotype. It is this “genotypic system” that then creates the specific downstream characteristics known as phenotype.

1. Understand and deconstruct the complexity of molecular medicine. As molecular medicine has advanced, the complexity of the interplay of contributing factors has become better known (Figure 6). However, this complexity and impact has been oversimplified at times.62 In order to unlock the reality of disease and impact on patients, we need to first begin to deconstruct the component parts that we believe represent these interactions, and then begin to build our knowledge based on stepwise clinical application of sound scientific hypothesis, starting from what we have studied and know, and then advancing into what we suspect we know but have not examined yet. We cannot jump to the proteome without at least beginning to consider its interaction with the transcriptome and genome before it. 2. Establish testing standards. Although no standard is perfect, by having one standard that can be applied to a given technology (i.e., NGS), and to which we can begin to collect outcomes, we can start, in a stepwise fashion, to collect the genotypic data

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

193

REVIEWS Table 2 Examples of RWD in oncology and key characteristics RWD oncology initiatives ASCO CancerLinQ

Foundation Medicine PMEC

MED-C NGS Registry

NCCN-FlatIron

ORIEN – M2Gen

Crucial partner tax status

Nonprofit

For-profit

Nonprofit

For-profit

For-profit

Build status

Announced and building

Announced and building

Announced and building

Announced and building

Ongoing

How addresses complexity

Likely passive

Unknown – likely passive

Actively planned as part of its strategy

Likely passive

Unknown – likely passive

Establishes testing standards

No

No

Yes

No

No

Compare to companion diagnostics

No

No

Yes

No

No

Quantitative data with binary

No

Unknown

Yes

Unknown

Unknown

Tracking versions

No

Unknown

Yes

Unknown

Unknown

Clinical outcomes defining sensitivity

No

Unknown

Yes

Unknown

Unknown

Unifying efforts

Yes

Unknown

Yes

No

No

Data siloing

No

Unknown

No

Yes (Flatiron – commercializing)

Yes (M2Gen – commercializing)

ASCO, American Society of Clinical Oncology; MED-C, Molecular Evidence Development Consortium; NCCN, National Comprehensive Cancer Network; NGS, next generation sequencing; ORIEN, Oncology Research Information Exchange; PMEC, Precision Medicine Exchange Consortium; RWD, real-world data. Some rows are self-explanatory, others are detailed below. Crucial partner tax status: This row helps identify likely a key factor of the initiative. Many of the “unknowns” in the columns of the for-profit entities are due to the proprietary nature of the data that is being collected. In most of the for-profit entities, one of the key endpoints of the initiative is commercialization. How addressing complexity: Often RWD sets collect information passively and the complexity of molecular medicine is gleaned on the back end rather than being guided on the front end. The only group that has announced its intention to actively approach the complexity of molecular medicine in a stepwise fashion is MED-C. Compare with companion diagnostics: Only MED-C is addressing this issue actively. Quantitative data with binary/version tracking: Unless detailed collection of laboratory data takes place, it is near impossible to obtain the data necessary to do this. CancerLinQ is collecting data from electronic medical records where the data are likely to detail the necessary information to be able to track these items. The proprietary nature of the other datasets makes it impossible to know the detail of collected data. Unifying efforts: Most groups have announced that they want multiple groups to come together to support a broad initiative, yet, in many cases, the proprietary nature of some venture makes it difficult for implementation. Data siloing: In the cases of NCCN-Flatiron and ORIEN, the information technology partner plans on commercializing the proprietary data and share revenue back with the nonprofit entity. In each case, the data are not open to any outside groups. ASCO has announced that it wants to share results for joint learning, but some groups have questioned how transparent or detailed this sharing will be.

that can be reliably connected with the phenotypic outcomes. As standards improve, new testing should be compared to old to show clinical benefit and we can advance incrementally, without skipping over important comparisons needed to verify better technology advances better outcomes (or in other words, better understanding of molecular reality and associated treatment). 3. Compare the testing standard to companion diagnostics (where they exist). By comparing the new testing standards to the previous reality of molecular alteration and treatment leading to outcome, we can begin to understand the benefits and limitations of new technology. 4. Collect quantitative data along with binary data values. Wherever possible, collect the quantitative molecular data before the qualitative operator is applied, this will allow backtesting of certain assumptions if the definitions change. 5. Apply versions and details to RWD. Although reality ultimately is constant, our understanding of it is evolving. As of such, previous assumptions or beliefs will change with time. If we understand these changes and apply them at certain time 194

points along with a version change, then it allows crosswalking between versions if we understand the relationship between the old and the new. For example, it may be reasonable to discuss not only what type of testing, but all component parts of the testing that can lead to differences in results. This may include sample preparation methods, analysis techniques, and bioinformatics interpretation. 6. Clinical outcomes should define sensitivity and specificity. As part of standardization, and in connection with collecting outcomes based in reality, modify testing standards in such a way that we are reliably identifying biomarkers that give us consistent outcomes when applied in a patient. Do not sacrifice quality of testing for quantity of testing results. 7. Unify efforts and collect directed outcomes. Once testing has been unified, it will be crucial to bring the many disparate groups together to collect outcomes and advance science through larger trial and registry networks. Many groups are working independently on the component parts of personalized medicine, but personalized medicine will require learning from a great deal more patients. VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS 8. Stop the siloing of patient data. Some groups have considered RWD collected from patients as a commodity. As of such, there is little desire to share this data with others for fear that its commercial value may be diminished. In order to identify unusual alterations and disease responses, it requires looking at a great deal of patients and hopefully enrolling many of these in national and international clinical trials. The more fractioned the data sources, the harder it will be to advance personalized medicine. For research purposes, data should be centrally aggregated and made available.

SELECTED EXAMPLES OF RWD AND SPECIFIC EFFORTS TO ADDRESS THESE DISCONNECTS

1. Selected RWD in Oncology: Although there are many RWD sets outside of oncology, the complexity of dealing with both somatic and germline mutations, as well as the constantly changing tumor microenvironment, creates substantial challenges in the design and implementation of these datasets. In Table 2, five initiatives (most of which are in the building phase) have been highlighted. This is not a comprehensive list, but serves to show some of the differences between approaches and how well disconnects are being addressed. 2. Notable Efforts to Close Disconnects: a. Testing Standards: i. FDA: LDTs (including NGS). The FDA has been working on implementing its LDT guidance, but method and timing of this momentous effort is still unknown. Specifically with NGS, FDA has held several public workshops discussing standards and their role in oversight of NGS. With this said, a specific standard has not been released as of yet and it is not clear when this will happen. ii. The College of American Pathologists and the Association of Molecular Pathologists. Both groups are working on guidance documents for NGS, and the College of American Pathologists is planning on developing certification methodology. It is unclear however, how closely this certification will approach the standards chosen by some of the major clinical trials relying on sequencing for inclusion. For example, lung-MAP requires that testing for inclusion take place at one specific laboratory using illumine-based technology, yet, NCI’s LungMATCH has decided to use a Thermo Fischer platform run at designated laboratories that have been cross-certified. iii. Palmetto’s Molecular Diagnostics (MolDX) Group. Although not a traditional certification body, MolDX published minimum standards that it would consider necessary to pay for the testing. In essence, these requirements were the first nonclinical trial standards that were published. b. MED-C. This initiative is somewhat unique in as much as it has announced its plans to address many of the disconnects listed above. It is bringing a broad group of stakeholders together in a shared consortium, including laboratory,

industry, pharma, academia, regulators, and payors to develop prospective clinical outcome registries based on standardized testing, comparison with existing companion diagnostics, and with continual updating of the testing parameters as they evolve and technology advances. Last, MED-C has said its database will not be proprietary. Because this endeavor is being built it is unclear how well MED-C will be able to achieve its lofty goals. SUMMARY

RWD has the potential to become a powerful tool in understanding patients and disease, but it is only as valuable as its ability to detail the clinical reality of the interaction. The disconnects that currently exist between RWD and reality are daunting, and unless we close the gaps between these two areas in a unified fashion, we will continue to move in a haphazard direction rather than jointly find a unified and direct method to unlock personalized medicine. CONFLICT OF INTEREST The authors declared no conflict of interest. C 2015 American Society for Clinical Pharmacology and Therapeutics V

1.

Raghupathi, W. & Raghupathi, V. Big data analytics in healthcare: promise and potential. Health Inf. Sci. Syst. 2014 Feb 7. . Accessed 18 August 2015. 2. Hansen, M.M., Miron-Shatz, T., Lau, A.Y. & Paton, C. Big data in science and healthcare: a review of recent literature and perspectives. Contribution of the IMIA Social Media Working Group. Yearb. Med. Inform. 9, 21–26 (2014). 3. Belle, A., Thiagarajan, R., Soroushmehr, S.M.R., Navidi, F., Beard, D.A. & Najarian, K. Big data analytics in healthcare. BioMed Research International [Internet]. . Accessed 15 August 2015. 4. Chindamo, M.C. et al. Intermediate fibrosis staging in hepatitis C: a problem not overcome by optimal samples or pathologists’ expertise. Ann. Hepatol. 14, 652–657 (2015). 5. Elmore, J.G. et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313, 1122–1132 (2015). 6. Gomes, D.S., Porto, S.S., Balabram, D. & Gobbi, H. Inter-observer variability between general pathologists and a specialist in breast pathology in the diagnosis of lobular neoplasia, columnar cell lesions, atypical ductal hyperplasia and ductal carcinoma in situ of the breast. Diagn. Pathol. 9, 121 (2014). 7. van den Brekel, M.W., Lodder, W.L., Stel, H.V., Bloemena, E., Leemans, C.R. & van der Waal, I. Observer variation in the histopathologic assessment of extranodal tumor spread in lymph node metastases in the neck. Head Neck 34, 840–845 (2012). 8. Krajewski, K.M., Nishino, M., Franchetti, Y., Ramaiya, N.H., Van den Abbeele, A.D. & Choueiri, T.K. Intraobserver and interobserver variability in computed tomography size and attenuation measurements in patients with renal cell carcinoma receiving antiangiogenic therapy: implications for alternative response criteria. Cancer 120, 711–721 (2014). 9. Malpica, A. et al. Interobserver and intraobserver variability of a twotier system for grading ovarian serous carcinoma. Am. J. Surg. Pathol. 31, 1168–1174 (2007). 10. West, J.B. Galen and the beginnings of Western physiology. Am. J. Physiol. Lung Cell Mol. Physiol. 307, L121–L128 (2014). 11. Golub, T.R. et al. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999).

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

195

REVIEWS 12. Ben-Dor, A., Bruhn, L., Friedman, N., Nachman, I., Schummer, M. & Yakhini, Z. Tissue classification with gene expression profiles. J. Comput. Biol. 7, 559–583 (2000). 13. Alizadeh, A.A. et al. Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling. Nature 403, 503–511 (2000). 14. Yanaihara, N. et al. Unique microRNA molecular profiles in lung cancer diagnosis and prognosis. Cancer Cell 9, 189–198 (2006). 15. Chan, K.S. et al. Identification, molecular characterization, clinical prognosis, and therapeutic targeting of human bladder tumor-initiating cells. Proc. Natl. Acad. Sci. USA 106, 14016–14021 (2009). 16. Schaefer, A. et al. Diagnostic and prognostic implications of microRNA profiling in prostate carcinoma. Int. J. Cancer 126, 1166– 1176 (2010). 17. Marisa, L. et al. Gene expression classification of colon cancer into molecular subtypes: characterization, validation, and prognostic value. PLoS Med. 10, e1001453 (2015). 18. Wilkerson, M.D. et al. Lung squamous cell carcinoma mRNA expression subtypes are reproducible, clinically important, and correspond to different normal cell types. Clin. Cancer Res. 16, 4864–4875 (2010). 19. Zhong, Q., Peng, H.L., Zhao, X., Zhang, L. & Hwang, W.T. Effects of BRCA1- and BRCA2-related mutations on ovarian and breast cancer survival: a meta-analysis. Clin. Cancer Res. 21, 211–220 (2015). 20. Preciado, M.V. et al. Hepatitis C virus molecular evolution: transmission, disease progression and antiviral therapy. World J. Gastroenterol. 20, 15992–16013 (2014). 21. Kantarjian, H. et al. Hematologic and cytogenetic responses to imatinib mesylate in chronic myelogenous leukemia. N. Engl. J. Med. 346, 645–652 (2002). 22. Huntly, B.J. et al. Imatinib improves but may not fully reverse the poor prognosis of patients with CML with derivative chromosome 9 deletions. Blood 102, 2205–2212 (2003). 23. Michor, F. et al. Dynamics of chronic myeloid leukaemia. Nature 435, 1267–1270 (2005). ~ones-Mateu, M.E., Avila, S., Reyes-Teran, G. & Martinez, M.A. 24. Quin Deep sequencing: becoming a critical tool in clinical virology. J. Clin. Virol. 61, 9–19 (2014). 25. Ramos, P.S., Shedlock, A.M. & Langefeld, C.D. Genetics of autoimmune diseases: insights from population genetics. J. Hum. Genet. (2015); e-pub ahead of print. 26. Kaddurah-Daouk, R., Weinshilboum, R.; Pharmacometabolomics Research Network. Metabolomic signatures for drug response phenotypes: pharmacometabolomics enables precision medicine. Clin. Pharmacol. Ther. 98, 71–75 (2015). 27. Dumas, M.E. Metabolome 2.0: quantitative genetics and network biology of metabolic phenotypes. Mol. Biosyst. 8, 2494–2502 (2012). 28. Mansfield, E.A. FDA perspective on companion diagnostics: an evolving paradigm. Clin. Cancer Res. 20, 1453–1457 (2014). 29. Shaw, A.T. et al. Crizotinib in ROS1-rearranged non-small-cell lung cancer. N. Engl. J. Med. 371, 1963–1971 (2014). 30. Herbst, R.S. et al. TRIBUTE: a phase III trial of erlotinib hydrochloride (OSI-774) combined with carboplatin and paclitaxel chemotherapy in advanced non-small-cell lung cancer. J. Clin. Oncol. 23, 5892–5899 (2005). 31. Engstrom, P.F. et al. NCCN molecular testing white paper: effectiveness, efficiency, and reimbursement. J. Natl. Compr. Cancer Netw. 9 (suppl. 6), S1–S16 (2011). 32. Ogawa, Y. et al. Immunohistochemical assessment for estrogen receptor and progesterone receptor status in breast cancer: analysis for a cut-off point as the predictor for endocrine therapy. Breast Cancer 11, 267–275 (2004). 33. Hammond, M.E. et al. American Society of Clinical Oncology/College of American Pathologists guideline recommendations for immunohistochemical testing of estrogen and progesterone receptors in breast cancer. J. Clin. Oncol. 28, 2784–2795 (2010). 34. Hammond, M.E. et al. American Society of Clinical Oncology/College of American Pathologists guideline recommendations for immunohistochemical testing of estrogen and progesterone receptors in breast cancer (unabridged version). Arch. Pathol. Lab. Med. 134, e48–e72 (2010). 35. Raghav, K.P. et al. Impact of low estrogen/progesterone receptor expression on survival outcomes in breast cancers previously classified as triple negative breast cancers. Cancer 118, 1498–1506 (2012). 196

36. Wolff, A.C. et al. Recommendations for human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology/College of American Pathologists clinical practice guideline update. J. Clin. Oncol. 31, 3997–4013 (2013). , C. Current challenges 37. Sapino, A., Goia, M., Recupero, D. & Marchio for HER2 testing in diagnostic pathology: state of the art and controversial issues. Front. Oncol. 3, 129 (2013). 38. Wolff, A.C. et al. American Society of Clinical Oncology/College of American Pathologists guideline recommendations for human epidermal growth factor receptor 2 testing in breast cancer. J. Clin. Oncol. 25, 118–145 (2007). 39. Paik, S. et al. Real-world performance of HER2 testing—National Surgical Adjuvant Breast and Bowel Project experience. J. Natl. Cancer Inst. 94, 852–854 (2002). 40. Boland, J.F. et al. The new sequencer on the block: comparison of Life Technology’s Proton sequencer to an Illumina HiSeq for wholeexome sequencing. Hum. Genet. 132, 1153–1163 (2013). 41. Weiss, G.J. et al. Evaluation and comparison of two commercially available targeted next-generation sequencing platforms to assist oncology decision making. Onco. Targets Ther. 8, 959–967 (2015). 42. O’Brien, T.D. et al. Inconsistency and features of single nucleotide variants detected in whole exome sequencing versus transcriptome sequencing: a case study in lung cancer. Methods 83, 118–127 (2015). 43. Burghel, G.J. et al. Towards a next-generation sequencing diagnostic service for tumour genotyping: a comparison of panels and platforms. Biomed. Res. Int. 2015, 478017 (2015). 44. O’Rawe, J. et al. Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med. 5, 28 (2013). 45. Cheng, A.Y., Teo, Y.Y. & Ong, R.T. Assessing single nucleotide variant detection and genotype calling on whole-genome sequenced individuals. Bioinformatics 30, 1707–1713 (2014). 46. Li, H. Toward better understanding of artifacts in variant calling from high-coverage samples. Bioinformatics 30, 2843–2851 (2014). 47. Warden, C.D., Adamson, A.W., Neuhausen, S.L. & Wu, X. Detailed comparison of two popular variant calling packages for exome and targeted exon studies. PeerJ. 2, e600 (2014). 48. Qi, Y. et al. Reproducibility of variant calls in replicate next generation sequencing experiments. PLoS One 10, e0119230 (2015). 49. Roy-Chowdhuri, S. et al. Factors affecting the success of nextgeneration sequencing in cytology specimens. Cancer Cytopathol. (2015); e-pub ahead of print. 50. Head, S.R. et al. Library construction for next-generation sequencing: overviews and challenges. Biotechniques 56, 61–64 (2014). 51. van Dijk, E.L., Jaszczyszyn, Y. & Thermes, C. Library preparation methods for next-generation sequencing: tone down the bias. Exp. Cell Res. 322, 12–20 (2014). 52. Bennett, N.C. & Farah, C.S. Next-generation sequencing in clinical oncology: next steps towards clinical validation. Cancers (Basel) 6, 2296–2312 (2014). 53. Drilon, A. et al. Broad, hybrid capture-based next-generation sequencing identifies actionable genomic alterations in lung adenocarcinomas otherwise negative for such alterations by other genomic testing approaches. Clin. Cancer Res. 21, 3631–3639 (2015). 54. Tuononen, K. et al. Comparison of targeted next-generation sequencing (NGS) and real-time PCR in the detection of EGFR, KRAS, and BRAF mutations on formalin-fixed, paraffin-embedded tumor material of non-small cell lung carcinoma-superiority of NGS. Genes Chromosomes Cancer 52, 503–511 (2013). 55. Jones, S. et al. Personalized genomic analyses for cancer mutation discovery and interpretation. Sci. Transl. Med. 7, 283ra53 (2015). 56. Nik-Zainal, S. et al. The life history of 21 breast cancers. Cell 149, 994–1007 (2012). zquez-Garcıa, I., Illingworth, C.J. & Mustonen, V. High57. Fischer, A., Va definition reconstruction of clonal composition in cancer. Cell Rep. 7, 1740–1752 (2014). 58. Seidman, A.D. et al. Randomized phase III trial of weekly compared with every-3-weeks paclitaxel for metastatic breast cancer, with trastuzumab for all HER-2 overexpressors and random assignment to trastuzumab or not in HER-2 nonoverexpressors: final results of Cancer and Leukemia Group B protocol 9840. J. Clin. Oncol. 26, 1642–1649 (2008). VOLUME 99 NUMBER 2 | FEBRUARY 2016 | www.wileyonlinelibrary/cpt

REVIEWS 59. Wang, Q. et al. Detecting somatic point mutations in cancer genome sequencing data: a comparison of mutation callers. Genome Med. 5, 91 (2013). 60. Shaywitz, D. Data silos: healthcare’s silent shame. Forbes. . Accessed 12 September 2015. 61. ASCO calls on congress to enact legislation to ensure widespread interoperability of electronic health records and prohibit information

blocking. ASCO.org. . Accessed 24 September 2015. 62. Kolata, G. Hope in the lab: a special report. A cautious awe greets drugs that eradicate tumors in mice. The New York Times. May 3, 1998. . Accessed 17 September 2015.

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 99 NUMBER 2 | FEBRUARY 2016

197