Considerations for observational research using large data sets in radiation oncology.

International Journal of

Radiation Oncology biology

physics

www.redjournal.org

Considerations for Observational Research Using Large Data Sets in Radiation Oncology Reshma Jagsi, MD, DPhil,* Justin E. Bekelman, MD,y Aileen Chen, MD,z Ronald C. Chen, MD, MPH,x Karen Hoffman, MD,jj Ya-Chen Tina Shih, PhD,{ Benjamin D. Smith, MD,# and James B. Yu, MD** *Department of Radiation Oncology, University of Michigan, Ann Arbor, Michigan; yDepartments of Radiation Oncology and Medical Ethics and Health Policy, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania; zDepartment of Radiation Oncology, Harvard Medical School, Boston, Massachusetts; xDepartment of Radiation Oncology, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina; jjDepartment of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas; {Department of Medicine, Section of Hospital Medicine, The University of Chicago, Chicago, Illinois; #Department of Radiation Oncology, Division of Radiation Oncology, and Department of Health Services Research, The University of Texas MD Anderson Cancer Center, Houston, Texas; and **Yale School of Medicine, New Haven, Connecticut Received Mar 1, 2014, and in revised form May 10, 2014. Accepted for publication May 12, 2014.

The radiation oncology community has witnessed growing interest in observational research conducted using large-scale data sources such as registries and claims-based data sets. With the growing emphasis on observational analyses in health care, the radiation oncology community must possess a sophisticated understanding of the methodological considerations of such studies in order to evaluate evidence appropriately to guide practice and policy. Because observational research has unique features that distinguish it from clinical trials and other forms of traditional radiation oncology research, the International Journal of Radiation Oncology, Biology, Physics assembled a panel of experts in health services research to provide a concise and well-referenced review, intended to be informative for the lay reader, as well as for scholars who wish to embark on such research without prior experience. This review begins by discussing the types of research questions relevant to radiation oncology that large-scale databases may help illuminate. It then describes major potential data sources for such endeavors, including information regarding access and insights regarding the strengths and limitations of each. Finally, it provides guidance regarding the analytical challenges that observational studies must confront, along with discussion of the techniques that have been developed to help minimize the impact of certain common analytical issues in observational analysis. Features characterizing a well-designed observational study include clearly defined research questions, careful selection of an

Reprint requests to: Dr Reshma Jagsi, Department of Radiation Oncology, University of Michigan, UHB2C490, SPC 5010, 1500 East Medical Ctr Dr, Ann Arbor, MI 48109-5010; Tel: (734) 936-7810; E-mail: [email protected] Dr Bekelman is supported by a K award (K07-CA163616) is from the NCI. Dr Jagsi receives funding from the American Cancer Society (MRSG-09-145-01). Dr Shih is supported by grants R21CA165092, R01 Int J Radiation Oncol Biol Phys, Vol. 90, No. 1, pp. 11e24, 2014 0360-3016/$ - see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.ijrobp.2014.05.013

HS018535, and R01 HS020263 from the National Cancer Institute, Agency for Healthcare Research and Quality, and the University of Chicago Cancer Research Foundation Women’s Board. Conflict of interest: Dr Smith receives research funding from Varian Medical Systems. Dr Jagsi received support for a phase 1 trial from Abbvie Pharmaceuticals and is a consultant for the Eviti Medical Advisory Board.

12

Jagsi et al.

International Journal of Radiation Oncology Biology Physics

appropriate data source, consultation with investigators with relevant methodological expertise, inclusion of sensitivity analyses, caution not to overinterpret small but significant differences, and recognition of limitations when trying to evaluate causality. This review concludes that carefully designed and executed studies using observational data that possess these qualities hold substantial promise for advancing our understanding of many unanswered questions of importance to the field of radiation oncology. Ó 2014 Elsevier Inc.

Introduction The radiation oncology community has witnessed growing interests in observational research conducted using largescale data sources such as registries and claims-based data sets. Given the low barriers to accessing certain sources of such data and the recent emphasis on understanding the “real-world” outcomes of medical interventions, studies relying on registry and/or claims data have proliferated in the International Journal of Radiation Oncology, Biology, Physics (Red Journal) and the general oncology literature in recent years. Although the literature of health services research contains detailed resources to guide the design and interpretation of such studies (1-13) and some reviews have considered specific topics of interest to the radiation oncologist (14-19), we are aware of no single, comprehensive overview specifically targeted toward the radiation oncologist who seeks to conduct or interpret such studies. Because observational research has unique features that distinguish it from clinical trials and other forms of traditional radiation oncology research, the Red Journal assembled a panel of experts in health services research to provide a concise and well-referenced overview, intended to be informative for the lay reader of the Red Journal, as well as for scholars who wish to embark on such research without prior experience. In this paper, we begin by discussing the types of research questions relevant to radiation oncology that largescale registry data may help illuminate. We then describe major potential data sources for such endeavors, including information regarding access and insights regarding the strengths and limitations of each. Finally, we provide guidance regarding the analytical dilemmas that observational studies must confront, along with discussion of the techniques that have been developed to help minimize the impact of certain common challenges in observational analysis.

Research Questions Many different kinds of studies can be performed with registry and/or claims data. These data have 2 distinct advantages over clinical trial data: the large sample size of the database and the real-world nature of the data. This section outlines several types of studies that can be explored with

such data. This is by no means an exhaustive list, and other novel and creative uses are certainly possible.

Rare cancers and rare events The characterization of the incidence and survival of patients diagnosed with rare cancers is one potential project that can be performed with large observational data sets. In particular, cancers which arise in a rare site (20) or a rare histology (21) benefit from the characterization of relatively straightforward concepts such as survival and patterns of presentation that would otherwise be difficult to obtain in single-institution experiences. Large observational databases may also be useful in studies exploring the potential role of radiation therapy in the management of rare diseases. However, it is important to recognize that these studies are limited by variations in the expertise of the diagnosing pathologist, as there may be significant limitations for both sensitivity and specificity when attempting to accurately determine a rare diagnosis. In addition, these data are useful for characterizing rare events like radiationinduced second malignancies (22).

Changes in practice patterns As radiation oncology is a rapidly changing field, the investigation of changes in radiation practice patterns over time is a particularly fertile area of research. For example, whether treatment patterns have changed in response to clinical evidence (23), randomized trials (24, 25), or clinical guidelines (26) is important in order to identify areas where further education is needed. In particular for radiation therapy, investigating the dissemination of new technologies is also important, although it is dependent on the source of the data to record whether new technology has been delivered. Dissemination of intensity modulated radiation therapy (IMRT) (27) and adoption of computed tomographic (CT) treatment planning (28) for lung cancers are 2 examples of areas where registry data have led to useful insights into the sociodemographic factors associated with the diffusion of important technology. Often, studies of new technology dissemination are limited by whether the data collected by registries or present in administrative claims are specific enough to identify the treatment to be studied. For example, studies of the use of accelerated partial breast irradiation using claims data have had to focus on the use of

Volume 90 Number 1 2014

brachytherapy because the receipt of external beam partial breast irradiation cannot reliably be differentiated from incomplete courses of whole-breast irradiation in claims data alone.

Prediction models and nomograms The creation of predictive models and nomograms is an area of potential research that is also dependent on a data source with diverse patient information. Some of the earliest pioneering work using cancer registry data was the characterization of survival of men following conservative management of prostate cancer, the so-called Albertson tables (29). This work has subsequently been updated (30); and as the manner and stage in which patients present over time continues to change, updating of these tables will be needed. Nomograms predicting benefit from postlumpectomy radiation therapy for older patients with breast cancer have been developed (31) to define subpopulations of patients who would have larger benefit from treatment (32). These types of studies are valuable and give patients more specific insights into the outcomes of patients similar to themselves.

Comparative effectiveness Perhaps the most interesting area of current research using observational data in radiation oncology is the exploration of the comparative effectiveness of new technologies in the treatment of cancer. Observational studies using registry and/or claims data are particularly important where clinical trials are not forthcoming, will not be available during a time of rapid technology adoption, or will not be conducted at all. Furthermore, randomized trials can sometimes prioritize internal validity (the measurement of cause and effect with minimized bias within the study sample) over external validity (the application of study results to patients in the source population who are not included in the study sample). The external validity of randomized studies may be limited if the treatment arms, outcomes, or follow-up periods tested are different or unrelated to current clinical practice (33) or if the participants are not representative of the patient population, a concern frequently voiced for elderly cancer patients (34). Large observational studies provide the advantage of drawing information from a diverse array of patients, providers, and treatment facilities in real-world practice. Examples of comparative effectiveness research where clinical trials already exist include comparisons of IMRT to 3D conformal radiation therapy for head and neck cancers (35). As for randomized patients, survival of patients treated with IMRT for head and neck cancers in community practice has been shown to be equivalent to treatment with 3D conformal RT (CRT) (36), although continued diligence about complications from treatment is needed (37).

Observational research using large data sets

13

Examples of comparative effectiveness where relevant randomized trials are ongoing include the comparative effectiveness of different types of radiation therapy for prostate cancer, including comparisons of 3D CRT and IMRT (38) to proton radiation therapy (39-41) and comparisons of breast brachytherapy to standard breast radiation therapy (42,43). It is notable that these examinations of comparative effectiveness are limited by how up-to-date the data are, whether specific treatment codes are available, and whether complications are adequately captured by administrative claims. In particular, it is important to note that the diffusion of certain technologies may be limited in the early years of use, in which case even the experiences in a multiinstitutional registry may reflect treatment at only 1 or a few select centers. In summary, there are a wide variety of potential studies to be performed using registry or claims data. These studies typically take advantage of the large sample size and treatment diversity available in such data. These studies are helpful in delineating the treatment and presentation of cancers both rare and common and in understanding changes and variations in treatment patterns, and are particularly helpful in investigating the comparative effectiveness of new technologies, both where randomized trials have not yet been completed and where external validation is needed.

Data Sources Numerous potential data sets exist that are suitable for the investigation of questions relevant to the practice of radiation oncology (Table 1). In this section, we describe several commonly used data sources, with attention to the relative strengths and limitations of each.

Surveillance, Epidemiology, and End Results data The Surveillance, Epidemiology, and End Results (SEER) program of the National Cancer Institute (NCI) is a collection of population-based cancer registries which began collecting data on January 1, 1973, in Connecticut, Iowa, New Mexico, Utah, Hawaii, Detroit, and San Francisco/Oakland, California (44). Over time, additional registries were added, and SEER program now includes a total of 20 registries covering approximately 28% of the US population (44). The data collected include: demographics (race and ethnicity, age at diagnosis, month and year of diagnosis, sex, marital status, state and county of residence at time of diagnosis), tumor site, tumor morphology, stage at diagnosis, first course of treatment (information on surgery and radiation but not systemic therapy), and survival (including cause of death from the death certificate). SEER data can be obtained free of charge; a signed data use agreement form is required, and data requests can be found on the SEER website (http://seer.cancer.gov/data/). The website also provides data analysis tools.

14


Jagsi et al.

Table 1

Data sources* Data source

SEER

SEER-Medicare Medicare

Private insurer claims databases (eg, Marketscan) Survey data sets NCDB

Registries from centers of excellence (eg, National Comprehensive Cancer Network) Radiation oncology-specific registries (eg, National Radiation Oncology Registry, Michigan Radiation Oncology Quality Consortium)

Strengths

Weaknesses

Large, NCI-funded; population-based; regional cancer registries that cover 26% of US population (reasonably representative, although more urban, racial differences) Claims data may better capture treatments received Claims represent entire population of Medicare beneficiaries rather than subset included in SEER

Limited information about key factors; inaccuracy, such as underascertainment of outpatient treatments, including RT; migration/loss to follow-up Generalizability (age), misclassification errors, failure to capture all morbidity Generalizability (age), lack of information on pathology and staging, misclassification, failure to capture all morbidity Costly, selection of only privately insured patients Costly to fund, some selection bias due to nonresponse Generalizability (CoC accredited centers)

Includes claims data on younger patients than Medicare Detailed measures; some have populationbased target population Large, joint project of ACS/ACoS, captures 70% of all cancers diagnosed in US More detailed information, potentially greater accuracy

Generalizability (patient selection effects), more limited numbers

Very detailed dosimetric and other data

Costly to establish, still in development

Abbreviations: ACoS Z American College of Surgeons; ACS Z American Cancer Society; CoC Z Commission on Cancer; NCDB Z National Cancer Data Base; NCI Z National Cancer Institute; RT Z radiation therapy; SEER Z Surveillance, Epidemiology, and End Results. * Table includes only those sources commonly used to study US patients. Additional valuable sources exist in other countries, including robust nationwide, population-based registries.

The population-based nature of SEER makes it a powerful data source with which to examine temporal changes in cancer incidence, patterns of care, and survival, with results generalizable to the US population. The limitations mainly relate to the lack of certain data elements, including systemic therapy, treatment after the first course, disease recurrence, comorbidities and functional status, and health-related quality of life (HRQOL). Further, recent data suggest that SEER may underascertain postoperative radiation therapy (45, 46). This can lead to spurious findings of underutilization or variation in the use of radiation therapy that are in fact artifacts of incorrect data (47). Therefore, there has been interest in combining the detailed pathologic information in SEER with other data, such as claims or surveys, as described in subsequent sections.

SEER-Medicare Data from the SEER cancer registries have been linked with Medicare administrative claims data to provide enriched information about Medicare beneficiaries with cancer (48, 49). The linked SEER-Medicare data set contains information from the SEER registries (patient demographics, tumor characteristics, initial treatment, and vital status) as well as billing claims for fee-for-service

Medicare beneficiaries. The claims contain diagnostic and billing codes generated for services covered by Medicare, including tests, procedures, office visits, admissions, durable medical equipment, home health, hospice care, and prescription drugs. The most recent linkage, performed in 2012, includes cancer diagnoses through 2009 and Medicare claims through 2010 (49). Medicare covers individuals younger than 65 if they are disabled or have end-stage renal disease, but most investigators use SEER-Medicare data to study individuals age 65 and older. Because Medicare insures nearly all individuals age 65 and older, the data are fairly representative of national cancer care for older cancer patients enrolled in fee-for-service Medicare in the United States. The linked SEER-Medicare claims provide insight about patient medical conditions, pretreatment evaluation, initial treatment, and subsequent medical care that is not available in SEER data alone. Medical diagnosis codes in claims generated the year prior to cancer diagnosis are commonly used to characterize patient comorbid medical conditions (48). Claims for imaging services can be used to evaluate assessment at diagnosis and surveillance after treatment. For example, procedure codes have been used to study the impact of breast magnetic resonance imaging (MRI) on type of breast cancer surgery and to evaluate the frequency of surveillance colonoscopies in colorectal cancer survivors


(50, 51). Claims can identify treatments not captured in SEER, including chemotherapy administration, androgen deprivation therapy administration, and type of reconstructive surgery (52-54). Although SEER collects data regarding administration of radiation during the first course of treatment, the linked Medicare claims can identify radiation administration not captured in SEER (46) and provide additional granularity regarding type of radiation administered (55). The linked SEER-Medicare database has been used to compare the use and outcomes of 3D conformal radiation therapy, IMRT, proton radiation therapy, and stereotactic radiosurgery, which would not be possible using SEER alone (28, 40, 56, 57). Because multiple years of claims data are available, beneficiaries can be followed longitudinally over time. The longitudinal nature of claims data facilitates the study of events that occur after initial diagnosis and treatment such as procedures for posttreatment complications, surveillance follow-up evaluation, cancer treatments for recurrence, and end-of-life care (5, 58, 59). Because claims data reflect billed medical services, they are also uniquely suited to study the cost of care (2). The linked Medicare claims provide additional information about the health, treatment, and outcomes of cancer patients; however, research using the linked claims has distinct limitations as well. First, it is important to remember that claims data are administrative data not clinical data. Claims contain only services covered by and billed to the insurer. Medical care not billed to Medicare, such as services provided to beneficiaries by the Veterans Administration, Military Medical System, or communitysponsored screening programs are not captured. Second, claims are generated for billing. Information needed to obtain payment, such as the procedure performed, is of higher quality than information not required to obtain payment, such as secondary diagnoses. Third, claims reflect care that patients received rather than care that patients may have needed. Therefore, studies that use administration of salvage cancer treatment as a surrogate for cancer recurrence may miss patients whose cancer recurred but who declined salvage treatment. Fourth, there is limited information on why a patient did or did not receive a medical service. Fifth, although posttreatment complications can be inferred from diagnostic and billing codes, there are no patient-reported outcomes in claims data. Sixth, data for individual claims are available only for fee-for-service Medicare beneficiaries. Therefore, most studies using SEER-Medicare are limited to fee-for-service beneficiaries with continuous enrollment, despite the fact that patients and their care may differ between the fee-for-service and managed care settings (60). Seventh, prescription drug data are available only since 2006 onward and for those beneficiaries with Part D Medicare coverage. Eighth, there is no follow-up for cancer recurrence from the SEER registry data. Ninth, researchers can identify receipt of tests covered by Medicare (eg, Prostate-Specific Antigen (PSA), KRAS) but do not know the results of the tests.


15

Finally, Medicare claims provide information about older patients with cancer, but patterns of care and outcomes may be different for younger patients (61). Despite these limitations, SEER-Medicare data offer a unique opportunity to evaluate medical care for older cancer patients prior to diagnosis, during initial treatment, and after treatment, as well as the cost and quality of care. The data can be linked to other sources as well, including the Area Resource File to study the impact of health resources on medical care, the American Medical Association Physician Masterfile to study physician characteristics, and the American Hospital Association Annual Survey Database to study hospital characteristics (3). Of note, physician characteristic linkages are imperfect (4), and 1 study identified potential misclassification in particular for radiation oncologists (62). Restricted geographic identifiers (ie, census tract of residence) also are available for linkage to US Census tract-level data. A random sample of Medicare beneficiaries who do not have cancer is also available for comparative analyses. Researchers interested in conducting studies utilizing SEER-Medicare data must obtain approval from the NCI for specific research questions and complete a data use agreement (DUA). It should be noted that the DUA is strictly tied to the specific questions approved by the NCI, and researchers planning to use the same data to study questions outside the scope of the DUA must submit a new proposal to seek approval for a new DUA. In addition, papers using SEER-Medicare data must be approved by the NCI prior to submitting to a journal. The cost of the data depends on the number of files requested, and a separate level of approval is necessary to obtain access to data like restricted geographic identifiers. Separate files contain data on hospital inpatient services, hospital outpatient services, physician payments, home health, hospice, skilled nursing facilities, and outpatient prescription drugs. The data files are large and complex and require computing power and advanced data management skills to create an analytical data set. Given the complexities of the data, it is helpful to work with someone who has experience with the data when first using SEER-Medicare data. Of note, new measures are added to SEER-Medicare over time (eg, American Joint Committee on Cancer M1 substaging for site of metastasis and HER 2 status in breast cancer) to permit more granular sample definition; typically, the year in which the measure was first made available is excluded when conducting analyses using the measure. The NCI periodically offers videocasts of training workshops that may be particularly useful resources for beginners interested in working with this data source (63).

Claims-based data sets Medical claims data that have not been linked to cancer registry information encompass a broader population. In contrast to the linked SEER-Medicare database (64),

16

Jagsi et al.

which is limited to beneficiaries diagnosed in a SEER registry (approximately 26% of the population), the Chronic Conditions Warehouse is a nationally comprehensive database that contains 100% of all Medicare fee-for-service claims for all beneficiaries. Claims for younger patients can be obtained from several private sources, such as the MarketScan database, which contains inpatient, outpatient, and pharmaceutical claims from more than 100 payors in all 50 states (60). Other proprietary commercial claims data include IMS LifeLink data, United Healthcare claims data, and Humana claims data, among others. Although commercial claims data provide information for a younger population, only privately insured individuals are included, and long-term follow-up can be limited due to frequent changes in insurance plans in the younger population. Claims-based data sets are subject to the limitations of SEER-Medicare data as outlined above. Additionally, claims-based algorithms must be applied to these data sets to identify incident cancer cases (65). Although some such algorithms have been validated, they inevitably have certain limitations and thus will omit some patients with the cancer to be studied and also include certain patients without a true diagnosis of cancer or mistaken prevalent cases as incident cases (66). Accordingly, great care should be applied when posing study questions with claims-only data to ensure that findings will not be sensitive to this inherent limitation. Additionally, there is no information about cancer stage or cancer histology. Algorithms to derive cancer stage from claims data have been developed for breast cancer but are limited in their predictive power (67). In certain cases, biological information from the tumor may be derived, for example, if a prescription for endocrine therapy is identified, it may be reasonable to infer that the patient’s breast cancer was estrogen receptor positive. Another limitation of some claims-based data sets is that they may not include race/ethnicity data as a matter of company policy, and this precludes those data from use in studies of race disparities. Certain limitations of claims-based algorithms can pose particular challenges for those investigating issues in radiation oncology. For example, algorithms to identify delivery of palliative radiation to the bone have also been developed using claims but have yet to be validated (6870). Studies of skeleton-related events using claims data are limited by absence of clinically confirmed diagnosis of bone metastasis, and prior literature documents concerns with misclassification when using billing codes to identify cancer recurrence or metastasis (71, 72). In addition, although claims data do allow for the identification of certain events relative to diagnosis or another index date (such as timing of face-to-face contact with a radiation oncologist, timing of receipt of radiation therapy), other measures that may be of interest to radiation oncologists cannot be determined, including site of radiation delivery (eg, to the bone or organ) and length of radiation therapy (eg, 4 week course or 6 week course, although the total number of treatment fractions billed can be useful in this regard). Researchers using claims data will often create


episodes of care in order to properly characterize use and costs associated with treatment for a given event or purpose. For example, a patient experiencing a fracture may generate claims related to the initial fracture event over the course of several weeks or months as they receive treatment and return to visit their physicians for follow-up care. Although claims data present opportunities to investigate real-world utilization patterns, it is not often straightforward to isolate utilization and costs associated with specific events or conditions using claims data. Medicare claims data can be obtained from Centers for Medicare and Medicaid Services (CMS) through the Research Data Assistance Center (www.resdac.org) and require an extensive data security plan and approval from the CMS Privacy Board. Commercial claims data can be obtained from their respective data vendor (eg, Truven Health Analytics [www.truvenhealth.com] for MarketScan database) and licensing fees typically differ between researchers in academic settings and those in nonacademic settings.

Survey data sets Some researchers have sought to enhance population-based sources of data with surveys that elicit information from patients themselves. Strengths of survey data include the ability to collect information reflecting patients’ own perceptions and experiences with care, as well as data relating to domains such as quality of life that are best evaluated using patient-reported outcomes measures. Growing interest in patient-reported outcomes (73-75) has led many registries to begin exploring ways in which to incorporate patient-reported data in their data sets. Limitations relate to the selection bias that can be introduced by nonresponse, and studies relying on survey data sets with low response rates should be viewed with considerable caution (76). The SEER-Medicare Health Outcomes Survey (MHOS)linked database resulted from a collaboration between NCI and CMS (77, 78). MHOS is a survey administered to a sample of Medicare managed-care beneficiaries: 1200 beneficiaries are randomly selected per managed care plan each year (1000 per year from 1998 to 2006). The latest data set available at the time of press includes cases diagnosed through 2009, with follow-up through 2011. For each selected beneficiary, a baseline survey is mailed and followed by a phone call if no response was received; a followup survey 2 years later is administered the same way. Data collected include participant-reported demographics (age, sex, race and ethnicity, marital status, household income, education, smoking status), comorbid conditions, functional status, and general HRQOL (assessed using Short-Form 36 from 1998 to 2005 and Veterans RAND 12-item health survey (79) from 2006 and later). To obtain linked SEER-MHOS data, a data request and proposal must be submitted for review (http://appliedresearch.cancer.gov/ surveys/seer-mhos/obtain/req.docs.html). If approved,


there is a cost to obtain the data set and other procedures that must be followed. SEER-MHOS is a unique, population-based data source used to examine HRQOL of cancer patients, which also includes a number of self-reported comorbid conditions and functional status measures; thus, this linked data set improves on some important limitations of the SEER database alone. However, the SEER-MHOS data source contains only patients 65 years or older and, overall, has a relatively limited sample size. Because MHOS was not designed specifically to study HRQOL in cancer patients, sampled participants may have no history of cancer; in participants with cancer, survey administration is often not coincident with the time of cancer diagnosis or treatment. Furthermore, no disease-specific HRQOL measures are used, and long-term longitudinal assessment of HRQOL changes is not possible. In addition, data quality for longitudinal assessment of HRQOL changes is poor because a follow-up survey is administered to the same cohort 2 years after the baseline survey, and the followup survey is limited to beneficiaries who are still enrolled in Medicare managed care plans. Other sources of SEER data linked to surveys also exist, and access to them may be available by request to the investigators who have developed these initiatives. For example, the Cancer Surveillance Outcomes Research Team (CanSORT) is a collaborative group funded by the NCI to survey cohorts of patients with newly diagnosed breast cancer in selected SEER regions in order to evaluate their decisionmaking experiences, satisfaction, and quality of life (80). Detailed patient data for treatments received are linked to SEER data for pathologic and clinical disease characteristics and have been used to illuminate patterns of radiation therapy receipt as well as potential causes for variations in care (81). The Cancer Care Outcomes Research and Surveillance Consortium (CanCORS), funded by the NCI in collaboration with the Department of Veterans Affairs, represents another valuable source of patient survey data. It focuses on patients with newly diagnosed colorectal or lung cancer from geographically diverse populations and health care delivery systems nationwide and includes data from medical records, as well as from physician and patient surveys (82). CanCORS data have been used for a number of purposes, including comparison of practice patterns for palliative radiation to clinical evidence and evaluation of metastatic cancer patients’ perceptions of the intent of radiation therapy (83-85).

National Cancer Data Base The National Cancer Data Base (NCDB) is a joint program of the Commission on Cancer (CoC) of the American College of Surgeons and the American Cancer Society and collects data from over 1400 CoC-accredited cancer hospitals throughout the United States. The NCDB contains data on approximately 70% of incident US cancer patients (86). The NCDB was started in 1989, and by 1996, reporting of cancer cases


17

was required of all CoC-accredited hospitals, which include teaching, community, and Veterans Health Administration hospitals. At each hospital, certified tumor registrars abstract data from patient medical records, and registrars are required to obtain and submit patient treatment and follow-up data even if part of the care is received at another (eg, non-CoCaccredited) hospital. Annually, registrars upload data to the NCDB on incident cancer cases as well as follow-up information on existing patients. Data contained in the NCDB include demographics (age, sex, race, marital status, medical insurance), comorbidity status, stage, treatments received, recurrence, and survival. NCDB data can be obtained by investigators from CoCaccredited programs through an application process (http:// www.facs.org/cancer/ncdb/participantuserfiles.html). If applications are approved, data are provided free of charge. The NCDB is a powerful source of data for research given its large size and inclusion of information on use of systemic therapy, comorbidities, and recurrence. However, an important limitation is that the NCDB is not populationbased, thus limiting generalizability; data are entered from accredited hospitals, and this selection may bias findings toward higher quality care and patient outcomes (87). Specifically, although cancer diagnoses at hospitals approved by the CoC constitute 70% of all new cancer diagnoses in the United States, CoC-approved hospitals are larger, more frequently situated in urban locations, and have more cancer-related services available to patients. Furthermore, patients who receive diagnoses and all of their cancer care in the outpatient setting, without entering a CoC-accredited hospital, are not included in the NCDB.

Centers of excellence Many large institutions maintain their own hospital registries, and some have joined together in collaborative registries, such as the one maintained by the National Comprehensive Cancer Network (NCCN) (88). Registries from centers of excellence are often restricted to investigators with an affiliation with a member site. The strengths of such data sets include the potential for access to highly detailed clinical information that may even extend to the level of dosimetric or other potentially relevant granular data. For example, the NCCN database contains more than 300 data elements that track the continuum of care longitudinally, including information on all clinical interventions. The primary limitation of these data sets is their failure to include a population-based sample. Selection effects can be highly problematic; estimates based on registries that include primarily centers of excellence are vulnerable to significant bias.

Radiation oncologyespecific registries In recent years, radiation oncologists have begun to assemble registries that include both academic and

18

Jagsi et al.

community sites from which to gather detailed radiation treatment planning information that may be particularly relevant for the investigation of the comparative effectiveness of radiation therapy and that other registries have traditionally lacked. American Society for Radiation Oncology and the Radiation Oncology Institute are embarking on the nascent National Radiation Oncology Registry (NROR), which aims to monitor and improve the quality of care for patients treated with radiation (89, 90). In the state of Michigan, Blue Cross Blue Shield has funded a Collaborative Quality Initiative known as the Michigan Radiation Oncology Quality Consortium (MROQC). MROQC collects detailed dosimetric, clinical, and patientreported outcomes data from multiple academic and community sites across the state on all patients treated with adjuvant whole-breast irradiation after lumpectomy and radiation therapy for lung cancer with curative intent (91). As these endeavors grow and mature, they will serve as unique sources of information for those wishing to compare different forms of radiation treatment. Their primary limitation is the particularly resource-intensive nature of collecting detailed dosimetric information relating to radiation treatment plans and the fact that they are still works in progress.


favorable selection into the more aggressive treatment will thus overestimate the benefit of the treatment. For example, 1 study (92) of patients with prostate and colon cancer in the SEER-Medicare database found that analysis of these observational data produced improbable results. The study found that men with locally advanced prostate cancer who underwent androgen deprivation therapy had higher prostate cancer mortality (hazard ratio, 1.5; 95% confidence interval, 1.29-1.92). Clinical trials have provided strong evidence that androgen deprivation therapy improves cancer mortality, so this finding almost certainly reflects selection bias.

Traditional regression-based methods to address confounding

Although the preceding sections have focused on data sources assembled in the United States, many countries outside the United States have robust registries or administrative claims data at the provincial or even national level, and these registries may be promising sources for the investigation of key research questions. A listing of some of these non-US oncology databases can be found at http://www.ispor.org/ OncologyORResources/SearchOcologyResources.aspx.

To deal with the issue of confounding due to nonrandom treatment assignment, several statistical and econometric techniques are commonly employed, including multivariable regression analyses that attempt to control for potential confounders in observational studies. Researchers often start with a series of descriptive bivariable analyses to explore the unadjusted correlation between each explanatory variable and the outcomes of interest and then follow with multivariable models to include the treatment variable (quantified as a binary variable with 1 indicating the treatment received by patients in the case group and 0 otherwise), plus other potential confounders in the same model. Variations on multivariable regression models, including stepwise selection algorithms and survival models for time-dependent effects, are also frequently used. In theory, if all important confounders were observed and controlled for, multiple regression techniques could produce similar results from an observational study that would be observed in a randomized study.

Analytical Issues

Propensity score models

Nonrandom assignment

Standard multiple regression techniques are limited, however, by specific assumptions about the relationship between the covariates and outcome and also by the fact that not all confounders may be observable. In these situations, propensity score models and instrumental variable analysis can play an important role in improving the accuracy of statistical inferences. Propensity score methods assign a probability of receiving the new treatment or the intervention received by patients in the case group to each individual based on observed covariates and can be implemented in several ways, including covariate adjustment and matching (93, 94). In covariate adjustment, propensity scores (predicted from a logistic regression) are simply entered in the regression model as an additional explanatory variable for the outcome. In propensity matching, strata are created by grouping individuals by propensity scores and calculating the treatment effect within those strata. Unlike standard

Canadian, European, and Asian oncology databases

In oncology, we are often interested in comparing the effect of different interventions or treatments. Randomized studies are the gold standard for determining whether one treatment is superior to another. However, for practical or ethical reasons, randomized studies are not always possible. In these situations, well-conducted observational studies can provide useful information to influence decision making. One common challenge in analyzing observational data is that treatments are not randomly assigned. Therefore, it is possible that characteristics of patients, providers, or healthcare systems, both observed and unobserved, can influence not only treatment assignment but also outcomes. For example, sicker patients might be less likely to receive aggressive treatment, and they may also be more likely to have a poor outcome, regardless of the treatment received. Analyses not accounting for such


multiple regression models, propensity scores do not assume a specific relationship between the covariates and outcome (eg, linear or log linear) but rather assign a probability or “propensity” for each individual to be assigned to a treatment group, based on observed characteristics. Therefore, like multiple regression models, propensity score methods are limited by the assumption that all factors affecting treatment assignment are accounted for by observable covariates. Furthermore, propensity score matched models do not include patients who remain unmatched and, therefore, require large data sets with significant overlap in patient characteristics. Such approaches have been used previously in numerous studies of topics relevant to radiation oncology (28, 40, 57). Of note, in the study described above, in which locally advanced prostate cancer patients who received androgen deprivation therapy fared worse, controlling for comorbidity, extent of disease, and other characteristics by multivariable analyses or by propensity analyses had remarkably small impact on the improbable results (92). Thus, caution must be employed when attempting to use observational data to evaluate the impact of treatments on patient outcomes, even when techniques to address measured confounding factors have been applied. Other methods that arise from the causal inference literature, including instrumental variable analysis, inverse probability treatment weights (95), and marginal structural models (96), have also been proposed for observational comparative effectiveness research.

Instrumental variable analysis Instrumental variable analysis is an econometric technique to remove the effects of hidden bias (97) and, unlike propensity scores, does not assume that all potential confounders are observed. The key assumption is that the instrumental variable does not affect the outcome directly, except through treatment assignment. Therefore, a good instrumental variable should be strongly correlated with treatment assignment (referred to as “instrument relevancy” in the econometric literature), yet lacking an independent effect on the outcome (referred to as “instrument exogeneity”). If an instrument is weak (ie, the proportion of treatment assignments in the instrument groups are about the same) then the estimate of treatment effect becomes unstable (98). In practice, it is often very difficult to identify a good instrumental variable for a particular treatment and outcome, which can limit the use of this technique. In certain cases, use of geographic variables, such as local area treatment rate (99) or differential distance to a treating facility (43, 100), have been appropriate instruments. Upon identification of a set of candidate instrumental variables, it is important to check the validity of these instruments. Once a set of valid instruments is identified, it is also important to verify whether the treatment variable is indeed “endogenous” (ie, the treatment variable is correlated with the error term). If


19

treatment variable is in fact exogenous, the use of instrumental variable method is unnecessary and can lead to efficiency loss. Instrumental variable approaches do not estimate the average effect of treatment in a population but rather the average effect for patients whose treatment was induced by the instrument. The treatment effect produced by instrumental variable analysis applies to what has been called the “marginal” or “complying” patient population, defined as patients whose treatment status depends strongly on the instrument. Research should describe which patients comprise the marginal patient population to inform the generalizability of findings from instrumental variable methods (101). Several examples of recent articles in radiation oncology using these techniques exist (40, 99, 100, 102-104).

Handling missing data Missing data is a common problem in randomized and nonrandomized studies. Data can be missing because patients are lost to follow-up, centers are unable to collect certain variables for all patients, or myriad other reasons. The main concern is whether missing data could bias study results; this concern is heightened in studies of comparative effectiveness when the level of missing data is different between study groups. Therefore, all studies must address how missing data were handled in analyses. Many approaches exist to handle missing data; we mention three here. The first approach involves omitting everyone without complete data, and is known as complete case analysis. However, if a large proportion of patients have missing data, their exclusion could result in a substantial loss of statistical power and biased results, especially if the discarded cases differ systematically from those retained. A second approach involves entering missing data into regression models as dummy variables. The downsides of this approach are similar to complete case analysis and, when levels of missing data are greater than 3% to 5% (as a general rule of thumb), this approach has also been shown to yield biased results (105). Multiple imputation, the third approach, is commonly available in statistical packages and is accepted as the most valid method for handling missing data. Multiple imputation can be thought of as a method to represent the uncertainty of missing data in regression models and involves the creation and combination of multiple data sets produced through the imputation process. As with any statistical approach, the validity of the results of multiple imputation depends on careful and appropriate modeling. In summary, we recommend the following best practices for handling missing data in observational research (106). First, reports should describe the number of patients excluded for missing data and the number and proportion of missing values for each variable of interest. Second, reports should examine differences between patients with complete or incomplete data across exposure and outcome variables. Third, reports should describe the methods used to handle

20

Jagsi et al.

missing data and the assumption or rationale for the approach. Fourth, reports should discuss whether and how missing data might bias study results, even in the context of valid approaches used to handle missing data (particularly if “missingness” may differ between study groups in comparative analyses). Several articles can be reviewed for further background (107-109).

Analysis of medical costs data Studies examining the economic impact of an illness or a new treatment often are interested in estimating medical costs associated with the illness or treatment. When preparing the data for cost analysis, it is important to normalize the cost data to the same year of currency (eg, in 2013 US dollars) by applying the appropriate inflation adjustment index, such as the use of Medicare Prospective Payment System adjuster and the Medicare Economic Index for US costs data extracted from Medicare Part A and B claims, respectively. It should also be noted that the type of cost information available varies by data sets. Some provide information on payments as well as charges (eg, Medicare claims), while other record only payment (eg, MarketScan) or charges (eg, Nationwide Inpatient Sample [NIS]). When both charges and payments are reported in the data, it is recommended that researchers use the payment variable for medical costs as charges are often highly inflated from the actual costs. If only charge information is provided, cost-to-charge ratio should be applied to obtain better cost measure. Medical cost data tend to be highly skewed, with a small proportion of very ill patients incurring excessively high costs; therefore, when costs are compared between 2 interventions, the commonly used t test for continuous variables is often inappropriate because cost variable is unlikely to be normally distributed. The recommended approach to compare mean medical costs when cost data are highly skewed is the nonparametric bootstrap method suggested by Barber and Thompson (110). The violation of normality assumption also complicates the multivariable regression models because estimates based on the standard ordinary least squares (OLS) technique can be severely biased if the dependent variable is highly skewed. Historically, researchers applied the logtransformed model to deal with this issue. However, more recent econometric literature has pointed out that despite its ease of implementation, the log-transformed model must take into consideration of the role of heteroscedasticity when retransforming the estimate from log(cost) back to cost (111). Generalized linear models (GLM) offer a nice alternative to the log-transformed model and can avoid the retransformation issue. The challenge for the GLM model is that one must specify a mean and variance function and the best model to use depends on the characteristics of the data (112). More advanced methods, such as the extended estimating


equations approach (113), have been developed to relax the requirement of GLM models. In general, applied econometricians agree that no one model universally dominates the others under all circumstances (113, 114). Given these complexities, it is highly advisable that studies of medical costs include individuals with relevant econometric expertise among the investigator team. Furthermore, medical costs data can be censored due to reasons such as loss of follow-up (eg, switch from a feefor-service plan to an HMO plan) or end of data collection period, resulting in partial observation of costs for patients whose data are censored. Methods such as the inverse probability weighted least squares method (115), Lin’s regression method (116, 117), and the 2-stage method by Carides et al (118) have been proposed to address the censoring issue in medical costs data. A comprehensive review can be found in the report by Huang (119).

Other biases Finally, no discussion of the analytical challenges of observational research using large data sets would be complete without mention of the risk of spurious findings when many exploratory analyses are performed without prespecification or theoretical justification. A statistically significant P value does not guarantee that the relationship observed is not due to chance, and the risk of finding chance associations increases when there is a “greater number and lesser preselection of tested relationships” (120). It is also important to note that the large size of observational data sets allows for the detection of relatively small differences; statistical significance does not imply clinical significance, and these issues must be addressed in the interpretation of any large-scale study, including observational studies.

Conclusions In summary, carefully designed and executed studies using observational data hold substantial promise for advancing our understanding of many unanswered questions of importance to the field of radiation oncology. Several common features characterize a well-designed observational study. These include clearly defined research questions, careful selection of an appropriate data source, collaboration or at least consultation with investigators with relevant methodological expertise, inclusion of sensitivity analyses, caution not to overinterpret small but statistically significant differences as clinically meaningful differences, and recognition of limitations when trying to evaluate causality. Experts have published guidelines for the conduct of comparative effectiveness research, and radiation oncologists who seek to use registry and/or claims data for such studies should consider their advice. For example, Table 2

Volume 90 Number 1 2014 Table 2

21


Checklist: Guidance and key considerations for study design for an observational CER protocol* Guidance

Key considerations

Check

Provide a rationale for study design choice and describe key design features

Cohort study proposals should clearly define cohort entry date (baseline date), employ a new user design (or provide rationale for including prevalent users), and include plans for reporting losses to follow-up. Case-control study proposals should clearly describe the control sampling method, employ a new user design (or provide a rationale for assessing confounders at index date), and assess potential for recall bias (if applicable). Case-cohort study proposals should include how the sampling scheme will be accounted for during analysis. Case-crossover study proposals should discuss the potential for confounding by time-varying factors and clearly state how the resulting effect estimate can be interpreted. Caseetime controlled study proposals should clearly weigh the pros and cons of accounting for calendar trends in the prevalence of exposure. The time point for start of follow-up should be clearly defined and meaningful, ideally anchored to the time of a medical intervention (eg, initiation of drug use). If alternative approaches are proposed, the rationale should be provided and implications discussed. Exclusion and inclusion criteria should be defined at the start of follow-up (baseline) and should be based solely on information available at this point in time (ie, ignoring potentially known events after baseline). The definition should include the time window for assessment (usually the same for all cohort members).

⁯

Define start of follow-up (baseline)

Define inclusion and exclusion criteria at start of follow-up (baseline)

Define exposure (treatments) of interest at start of follow-up Define outcome(s) of interest Define potential confounders

Information should be provided on measures of accuracy if possible. Potential confounders known to be associated with treatment and outcome should be prespecified when possible. Confounders should be assessed prior to exposure or treatment initiation to ensure they are not affected by the exposure. Approaches to empirical identification of confounders should be described if planned.

⁯

⁯

⁯ ⁯ ⁯

* Reprinted with permission from: Velentgas P, Dreyer NA, Nourjah P, et al (eds). Developing a protocol for observational comparative effectiveness research: A user’s guide. AHRQ Publication No. 12(13)-EHC099. Rockville, MD: Agency for Healthcare Research and Quality; January 2013; 31. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/440/1166/User-Guide-to-Observational-CER-1-10-13.pdf. Accessed June 16, 2014.

reproduces a checklist for study design of an observational comparative effectiveness research protocol from the Agency for Healthcare Research and Quality (1). Other checklists are also available and may prove useful in evaluating the rigor of a study’s approach (6-8,121-123). For those seeking to evaluate the strengths and limitations of the increasing volume of observational research in radiation oncology, we hope that the information in this paper and the references provided will prove useful and instructive. Understanding these issues is critical because ongoing developments in the social and policy environments are likely to lead to new opportunities and interest in

observational research in radiation oncology in the coming years. For example, the creation of health information exchanges (secure repositories of patient health records) could support the development of a true “learning healthcare system” (124-126) in oncology care. The introduction of new automated techniques for data extraction from medical records is likely also to introduce new challenges and the need for ongoing evaluation of data quality and limitations. Other changes, such as modifications of payment structures, will create opportunities for leveraging the methods of observational research to evaluate the impact of policy changes on outcomes. With the growing emphasis on observational analyses in health care, a sophisticated understanding of the

22

Jagsi et al.

issues discussed herein are of critical importance to the radiation oncology community if we are to evaluate evidence appropriately to guide practice and policy.

References 1. Velentgas P, Dreyer NA, Nourjah P, et al., editors. Developing a protocol for observational comparative effectiveness research: A user’s guide. AHRQ publication No. 12(13)-EHC099. Rockville, MD: Agency for Healthcare Research and Quality; 2013. Available at, http://www.effectivehealthcare.ahrq.gov/ehc/products/440/1166/ User-Guide-to-Observational-CER-1-10-13.pdf. Accessed June 16, 2014. 2. Brown ML, Riley GF, Schussler N, et al. Estimating health care costs related to cancer treatment from SEER-Medicare data. Med Care 2002;40(suppl 8 8):IV-104-IV-117. 3. Schrag D, Bach PB, Dahlman C, et al. Identifying and measuring hospital characteristics using the SEER-Medicare data and other claims-based sources. Med Care 2002;40(suppl 8 8):IV-96-IV103. 4. Baldwin LM, Adamache W, Klabunde CN, et al. Linking physician characteristics and Medicare claims data: Issues in data availability, quality, and measurement. Med Care 2002;40(suppl 8):IV-82-IV-95. 5. Earle CC, Nattinger AB, Potosky AL, et al. Identifying cancer relapse using SEER-Medicare data. Med Care 2002;40(suppl 8):IV-75-IV-81. 6. Berger ML, Mamdani M, Atkins D, et al. Good research practices for comparative effectiveness research: Defining, reporting, and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force reportdpart I. Value Health 2009;12:1044-1052. 7. Cox E, Martin BC, Staa TV, et al. Good research practices for comparative effectiveness research: Approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: The International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Reportdpart II. Value Health 2009;12:1053-1061. 8. Johnson ML, Crown W, Martin BC, et al. Good research practices for comparative effectiveness research: Analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Reportdpart III. Value Health 2009;12:1062-1073. 9. Korn EL, Freidlin B. Methodology for comparative effectiveness research: Potential and limitations. J Clin Oncol 2012;30:4185-4187. 10. Lyman GH, Levine M. Comparative effectiveness research in oncology: An overview. J Clin Oncol 2012;30:4181-4184. 11. Hershman DL, Wright JD. Comparative effectiveness research in oncology methodology: Observational data. J Clin Oncol 2012;30: 4215-4222. 12. Lyman GH. Comparative effectiveness research in oncology. Oncologist 2013;18:752-759. 13. Ramsey SD, Sullivan SD, Reed SD, et al. Oncology comparative effectiveness research: A multistakeholder perspective on principles for conduct and reporting. Oncologist 2013;18:760-767. 14. Virnig BA, Warren JL, Cooper GS, et al. Studying radiation therapy using SEER-Medicare-linked data. Med Care 2002;40(suppl 8):IV49-IV-54. 15. Aneja S, Yu JB. Comparative effectiveness research in radiation oncology: Stereotactic radiosurgery, hypofractionation, and brachytherapy. Semin Radiat Oncol 2014;24:35-42. 16. Chen AB. Comparative effectiveness research in radiation oncology: Assessing technology. Semin Radiat Oncol 2014;24:25-34. 17. Meyer AM, Wheeler SB, Weinberger M, et al. An overview of methods for comparative effectiveness research. Semin Radiat Oncol 2014;24:5-13.

International Journal of Radiation Oncology Biology Physics 18. Chen RC. Comparative effectiveness research in oncology: The promise, challenges, and opportunities. Semin Radiat Oncol 2014;24:1-4. 19. Bekelman JE, Shah A, Hahn SM. Implications of comparative effectiveness research for radiation oncology. Pract Radiat Oncol 2011;1:72-80. 20. Urdaneta AI, Yu JB, Wilson LD. Population based cancer registry analysis of primary tracheal carcinoma. Am J Clin Oncol 2011;34:32-37. 21. Yu JB, Blitzblau RC, Patel SC, et al. Surveillance, Epidemiology, and End Results (SEER) database analysis of microcystic adnexal carcinoma (sclerosing sweat duct carcinoma) of the skin. Am J Clin Oncol 2010;33:125-127. 22. Beard CJ, Travis LB, Chen MH, et al. Outcomes in stage I testicular seminoma: A population-based study of 9193 patients. Cancer 2013; 119:2771-2777. 23. Bekelman JE, Epstein AJ, Emanuel EJ. Single- vs multiple-fraction radiotherapy for bone metastases from prostate cancer. JAMA 2013;310:1501-1502. 24. Soulos PR, Yu JB, Roberts KB, et al. Assessing the impact of a cooperative group trial on breast cancer care in the Medicare population. J Clin Oncol 2012;30:1601-1607. 25. Hoffman KE, Nguyen PL, Chen MH, et al. Recommendations for post-prostatectomy radiation therapy in the United States before and after the presentation of randomized trials. J Urol 2011;185:116-120. 26. Kuykendal AR, Hendrix LH, Salloum RG, et al. Guideline-discordant androgen deprivation therapy in localized prostate cancer: Patterns of use in the Medicare population and cost implications. Ann Oncol 2013;24:1338-1343. 27. Shirvani SM, Jiang J, Gomez DR, et al. Intensity modulated radiotherapy for stage III non-small cell lung cancer in the United States: Predictors of use and association with toxicities. Lung Cancer 2013;82:252-259. 28. Chen AB, Neville BA, Sher DJ, et al. Survival outcomes after radiation therapy for stage III non-small-cell lung cancer after adoption of computed tomography-based simulation. J Clin Oncol 2011;29: 2305-2311. 29. Albertsen PC, Fryback DG, Storer BE, et al. Long-term survival among men with conservatively treated localized prostate cancer. JAMA 1995;274:626-631. 30. Lu-Yao GL, Albertsen PC, Moore DF, et al. Outcomes of localized prostate cancer following conservative management. JAMA 2009; 302:1202-1209. 31. Albert JM, Liu DD, Shen Y, et al. Nomogram to predict the benefit of radiation for older patients with breast cancer treated with conservative surgery. J Clin Oncol 2012;30:2837-2843. 32. Breast cancer nomogram to predict benefit of radiation for older patients with breast cancer treated with conservative surgery. Available at: www.mdanderson.org/RadiationBenefitPredictor. Accessed June 16, 2014. 33. Presley CJ, Soulos PR, Herrin J, et al. Reply to LW Cuttino, et al [letter to the editor]. J Clin Oncol 2013;31:2227-2229. 34. Levit LA, Balogh EP, Nass SJ, et al. editors. Delivering high-quality cancer care: Charting a new course for a system in crisis. Washington, DC: Institute of Medicine; 2013. 35. Nutting CM, Morden JP, Harrington KJ, et al. Parotid-sparing intensity modulated versus conventional radiotherapy in head and neck cancer (PARSPORT): A phase 3 multicentre randomised controlled trial. Lancet Oncol 2011;12:127-136. 36. Yu JB, Soulos PR, Sharma R, et al. Patterns of care and outcomes associated with intensity-modulated radiation therapy versus conventional radiation therapy for older patients with head-and-neck cancer. Int J Radiat Oncol Biol Phys 2012;83:e101-e107. 37. Beadle BM, Liao KP, Chambers MS, et al. Evaluating the impact of patient, tumor, and treatment characteristics on the development of jaw complications in patients treated for oral cancers: A SEERMedicare analysis. Head Neck 2013;35:1599-1605. 38. Bekelman JE, Mitra N, Efstathiou J, et al. Outcomes after intensitymodulated versus conformal radiotherapy in older men with nonmetastatic prostate cancer. Int J Radiat Oncol Biol Phys 2011;81: e325-e334.

Volume 90 Number 1 2014 39. Kim S, Shen S, Moore DF, et al. Late gastrointestinal toxicities following radiation therapy for prostate cancer. Eur Urol 2011;60: 908-916. 40. Sheets NC, Goldin GH, Meyer AM, et al. Intensity-modulated radiation therapy, proton therapy, or conformal radiation therapy and morbidity and disease control in localized prostate cancer. JAMA 2012;307:1611-1620. 41. Yu JB, Soulos PR, Herrin J, et al. Proton versus intensity-modulated radiotherapy for prostate cancer: Patterns of care and early toxicity. J Natl Cancer Inst 2013;105:25-32. 42. Smith GL, Xu Y, Buchholz TA, et al. Association between treatment with brachytherapy vs whole-breast irradiation and subsequent mastectomy, complications, and survival among older women with invasive breast cancer. JAMA 2012;307:1827-1837. 43. Presley CJ, Soulos PR, Herrin J, et al. Patterns of use and short-term complications of breast brachytherapy in the national Medicare population from 2008-2009. J Clin Oncol 2012;30:4302-4307. 44. Surveillance Epidemiology and End Results Program. Available at: http://seer.cancer.gov. Accessed June 16, 2014. 45. Jagsi R, Abrahamse P, Hawley ST, et al. Underascertainment of radiotherapy receipt in Surveillance, Epidemiology, and End Results registry data. Cancer 2012;118:333-341. 46. Walker GV, Giordano SH, Williams M, et al. Muddy water? Variation in reporting receipt of breast cancer radiation therapy by population-based tumor registries. Int J Radiat Oncol Biol Phys 2013;86:686-693. 47. Smith GL, Eifel PJ. In regard to Han et al. Int J Radiat Oncol Biol Phys 2014;88:459-460 [letters to the editor]. 48. Warren JL, Klabunde CN, Schrag D, et al. Overview of the SEERMedicare data: Content, research applications, and generalizability to the United States elderly population. Med Care 2002;40:IV-3IV-18. 49. SEER-Medicare: About the data files. Oct 2013. Available at: http:// appliedresearch.cancer.gov/seermedicare/aboutdata/. Accessed June 16 2014. 50. Fortune-Greeley AK, Wheeler SB, Meyer AM, et al. Preoperative breast MRI and surgical outcomes in elderly women with invasive ductal and lobular carcinoma: A population-based study. Breast Cancer Res Treat 2014;143:203-212. 51. Brawarsky P, Neville BA, Fitzmaurice GM, et al. Surveillance after resection for colorectal cancer. Cancer 2013;119:1235-1242. 52. In H, Jiang W, Lipsitz SR, et al. Variation in the utilization of reconstruction following mastectomy in elderly women. Ann Surg Oncol 2013;20:1872-1879. 53. Shahinian VB, Kuo YF, Freeman JL, et al. Characteristics of urologists predict the use of androgen deprivation therapy for prostate cancer. J Clin Oncol 2007;25:5359-5365. 54. Warren JL, Harlan LC, Fahey A, et al. Utility of the SEER-Medicare data to identify chemotherapy use. Med Care 2002;40:IV-55-IV-61. 55. Smith BD, Pan IW, Shih YC, et al. Adoption of intensity-modulated radiation therapy for breast cancer in the United States. J Natl Cancer Inst 2011;103:798-809. 56. Guadagnolo BA, Huo J, Liao KP, et al. Changing trends in radiation therapy technologies in the last year of life for patients diagnosed with metastatic cancer in the United States. Cancer 2013;119:1089-1097. 57. Shirvani SM, Jiang J, Chang JY, et al. Comparative effectiveness of 5 treatment strategies for early-stage non-small cell lung cancer in the elderly. Int J Radiat Oncol Biol Phys 2012;84:1060-1070. 58. Butler Nattinger A, Schapira MM, Warren JL, et al. Methodological issues in the use of administrative claims data to study surveillance after cancer treatment. Med Care 2002;40:IV-69-IV-74. 59. Potosky AL, Warren JL, Riedel ER, et al. Measuring complications of cancer treatment using the SEER-Medicare data. Med Care 2002; 40:IV-62-IV-68. 60. Jagsi R, Jiang J, Momoh AO, et al. Trends and variation in use of breast reconstruction in breast cancer patients undergoing mastectomy in the United States. J Clin Oncol 2014;32:919-926. http://dx.doi.org/10. 1200/JCO.2013.52.2284. Epub 2014 Feb 18. Accessed June 16, 2014.


23

61. Pan IW, Smith BD, Shih YC. Factors contributing to underuse of radiation among younger women with breast cancer. J Natl Cancer Inst 2014;106:340. 62. Pollack LA, Adamache W, Eheman CR, et al. Enhancement of identifying cancer specialists through the linkage of Medicare claims to additional sources of physician specialty. Health Serv Res 2009; 44(2 Pt 1):562-576. 63. SEER-Medicare Training. National Cancer Institute. Updated Oct 2013 Available at: http://appliedresearch.cancer.gov/seermedicare/ considerations/training.html. Accessed June 16, 2014. 64. Centers for Medicare & Medicaid Services (CMS). Chronic Conditions Data Warehouse (CCW). 2014. Available at: https://www. ccwdata.org/web/guest/home 65. Schulman KL, Berenson K, Shih YCT, et al. A checklist for ascertaining study cohorts in oncology health services research using secondary data: Report of the ISPOR oncology good outcomes research practices working group. Value Health 2013;16:655-669. 66. Nattinger AB, Laud PW, Bajorunaite R, et al. An algorithm for the use of Medicare claims data to identify women with incident breast cancer. Health Serv Res 2004;39:1733-1749. 67. Smith GL, Shih YC, Giordano SH, et al. A method to predict breast cancer stage using Medicare claims. Epidemiol Perspect Innov 2010;7:1. 68. Sathiakumar N, Delzell E, Morrisey MA, et al. Mortality following bone metastasis and skeletal-related events among men with prostate cancer: a population-based analysis of US Medicare beneficiaries, 1999-2006. Prostate Cancer Prostatic Dis 2011;14:177-183. 69. Sathiakumar N, Delzell E, Morrisey MA, et al. Mortality following bone metastasis and skeletal-related events among women with breast cancer: A population-based analysis of US Medicare beneficiaries, 1999-2006. Breast Cancer Res Treat 2012;131:231-238. 70. Lage MJ, Barber BL, Harrison DJ, et al. The cost of treating skeletalrelated events in patients with prostate cancer. Am J Manag Care 2008;14:317-322. 71. Hassett MJ, Ritzwoller DP, Taback N, et al. Validating billing/encounter codes as indicators of lung, colorectal, breast, and prostate cancer recurrence using 2 large contemporary cohorts. Med Care 2012. December 6 [Epub], http://dx.doi.org/10.1097/MLR. 0b013e318277eb6f. 72. Nordstrom BL, Whyte JL, Stolar M, et al. Identification of metastatic cancer in claims data. Pharmacoepidemiol Drug Saf 2012;21(suppl 2):21-28. 73. Basch E, Abernethy A, Mullins CD, et al. Recommendations for incorporating patient-reported outcomes into the design of clinical trials in adult oncology. Center for Medical Technology Policy; 2010. 74. Bruner DW, Hanisch LJ, Reeve BB, et al. Stakeholder perspectives on implementing the National Cancer Institute’s patient-reported outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Transl Behav Med 2011;1:110-122. 75. Abernethy AP, Ahmad A, Zafar SY, et al. Electronic patient-reported data capture as a foundation of rapid learning cancer care. Med Care 2010;48(suppl 6):S32-S38. 76. Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA 2012;307:1805-1806. 77. Ambs A, Warren JL, Bellizzi KM, et al. Overview of the SEERMedicare Health Outcomes Survey linked dataset. Health Care Finance Rev 2008;29:5-21. 78. SEER-Medicare Health Outcomes Survey (SEER-MHOS) linked database. Available at: http://appliedresearch.cancer.gov/surveys/ seer-mhos/. Accessed June 16, 2014. 79. Veterans RAND survey is correct as written. See: http://www. hosonline.org/surveys/hos/download/veterans_rand_12_item_health_ survey_white_paper_summary.pdf 80. Cancer Surveillance and Outcomes Research Team website. Available at: http://www.med.umich.edu/cansort/. Accessed June 16, 2014. 81. Jagsi R, Abrahamse P, Morrow M, et al. Patterns and correlates of adjuvant radiotherapy receipt after lumpectomy and after mastectomy for breast cancer. J Clin Oncol 2010;28:2396-2403.

24

Jagsi et al.

82. Ayanian JZ, Chrischilles EA, Fletcher RH, et al. Understanding cancer treatment and outcomes: The Cancer Care Outcomes Research and Surveillance Consortium. J Clin Oncol 2004;22:2992-2996. 83. Chen AB, Cronin A, Weeks JC, et al. Expectations about the effectiveness of radiation therapy among patients with incurable lung cancer. J Clin Oncol 2013;31:2730-2735. 84. Chen AB, Cronin A, Weeks JC, et al. Palliative radiation therapy practice in patients with metastatic non-small-cell lung cancer: A Cancer Care Outcomes Research and Surveillance Consortium (CanCORS) study. J Clin Oncol 2013;31:558-564. 85. National Cancer Institute. Cancer care outcomes research and surveillance consortium (CanCORS). March 2014. Available at: http:// appliedresearch.cancer.gov/cancors/. Accessed June 16 2014. 86. Bilimoria KY, Stewart AK, Winchester DP, et al. The National Cancer Data Base: A powerful initiative to improve cancer care in the United States. Ann Surg Oncol 2008;15:683-690. 87. Bilimoria KY, Bentrem DJ, Stewart AK, et al. Comparison of commission on cancer-approved and -nonapproved hospitals in the United States: Implications for studies that use the National Cancer Data Base. J Clin Oncol 2009;27:4177-4181. 88. NCCN oncology outcomes database. Available at: http://www.nccn. org/network/business_insights/outcomes_database/outcomes.aspx. Accessed June 16, 2014. 89. Palta JR, Efstathiou JA, Bekelman JE, et al. Developing a national radiation oncology registry: From acorns to oaks. Pract Radiat Oncol 2012;2:10-17. 90. Efstathiou JA, Nassif DS, McNutt TR, et al. Practice-based evidence to evidence-based practice: Building the National Radiation Oncology Registry. J Oncol Pract 2013;9:e90-e95. 91. Vainshtein JM, Hayman JA, Moran JM, et al. Collaborative quality initiative in the treatment of breast and lung cancer: An important step toward high quality cost-effective care. Int J Radiat Oncol Biol Phys 2013;87(suppl 2):S498-S499. 92. Giordano SH, Kuo YF, Duan Z, et al. Limits of observational data in determining outcomes from cancer therapy. Cancer 2008;112: 2456-2466. 93. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983;70:41-55. 94. Lunceford JK, Davidian M. Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Stat Med 2004;23:2937-2960. 95. Hogan JW, Lancaster T. Instrumental variables and inverse probability weighting for causal inference from longitudinal observational studies. Stat Methods Med Res 2004;13:17-48. 96. Robins JM, Herna´n MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000;11:550-560. 97. Newhouse JP, McClellan M. Econometrics in outcomes research: The use of instrumental variables. Annu Rev Public Health 1998;19:17-34. 98. Staiger D, Stock J. Instrumental variables regression with weak instruments. Econometrica 1997;65:557-586. 99. Bekelman JE, Handorf EA, Guzzo T, et al. Radical cystectomy versus bladder-preserving therapy for muscle-invasive urothelial carcinoma: Examining confounding and misclassification bias in cancer observational comparative effectiveness research. Value Health 2013;16:610-618. 100. Punglia RS, Saito AM, Neville BA, et al. Impact of interval from breast conserving surgery to radiotherapy on local recurrence in older women with breast cancer: Retrospective cohort analysis. BMJ 2010;340:c845. 101. Brookhart MA, Rassen JA, Schneeweiss S. Instrumental variable methods in comparative safety and effectiveness research. Pharmacoepidemiol Drug Saf 2010;19:537-554. 102. Beadle BM, Liao KP, Elting LS, et al. Improved survival using intensitymodulated radiation therapy in head and neck cancers: A SEERMedicare analysis. Cancer 2014;120:702-710.

International Journal of Radiation Oncology Biology Physics 103. Zeliadt SB, Potosky AL, Penson DF, et al. Survival benefit associated with adjuvant androgen deprivation therapy combined with radiotherapy for high- and low-risk patients with nonmetastatic prostate cancer. Int J Radiat Oncol Biol Phys 2006;66: 395-402. 104. Hadley J, Polsky D, Mandelblatt JS, et al. An exploratory instrumental variable analysis of the outcomes of localized breast cancer treatments in a Medicare population. Health Econ 2003;12: 171-186. 105. Jones MP. Indicator and stratification methods for missing explanatory variables in multiple linear regression. J Am Stat Assoc 1996;91: 222-230. 106. Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ 2009;338:b2393. 107. Schafer JL. Multiple imputation: A primer. Stat Methods Med Res 1999;8:3-15. 108. Barladi AN, Enders CK. An introduction to modern missing data analyses. J School Psychol 2010;48:5-37. 109. Little RJA, Rubin DB. Statistical Analysis With Missing Data. New York: Wiley; 1987. 110. Barber JA, Thompson SG. Analysis of cost data in randomized trials: An application of the non-parametric bootstrap. Stat Med 2000;19: 3219-3236. 111. Manning WG. The logged dependent variable, heteroscedasticity, and the retransformation problem. J Health Econ 1998;17:283-295. 112. Manning WG, Mullahy J. Estimating log models: To transform or not to transform? J Health Econ 2001;20:461-494. 113. Basu A, Rathouz PJ. Estimating incremental and marginal effects on health outcomes using flexible link and variance function models. Biostatistics 2005;6:93-109. 114. Basu A, Manning WG, Mullahy J. Comparing alternative models: Log vs. Cox proportional hazard? Health Econ 2004;13:749-765. 115. Bas‚er O, Gardiner JC, Bradley CJ, et al. Longitudinal analysis of censored medical cost data. Health Econ 2006;15:513-525. 116. Lin DY. Proportional means regression for censored medical costs. Biometrics 2000;56:775-778. 117. Lin DY. Regression analysis of incomplete medical cost data. Stat Med 2003;22:1181-1200. 118. Carides GW, Heyse JF, Iglewicz B. A regression-based method for estimating mean treatment cost in the presence of right-censoring. Biostatistics 2000;1:299-313. 119. Huang Y. Cost analysis with censored data. Med Care 2009;47(suppl 7):S115-S119. 120. Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124. 121. The STROBE checklist. Available at: http://www.equator-network. org. Accessed June 16, 2014. 122. Motheral B, Brooks J, Clark MA, et al. A checklist for retrospective database studiesereport of the ISPOR Task Force on Retrospective Databases. Value Health 2003;6:90-97. 123. Garrison LP Jr., Neumann PJ, Erickson P, et al. Using real-world data for coverage and payment decisions: The ISPOR Real-World Data Task Force report. Value Health 2007;10:326-335. 124. Institute of Medicine. The learning health system series. Available at: http://www.iom.edu/Activities/Quality/w/media/85DAF51E8463421 0B05C1317FFF94D22.pdf. Accessed June 16, 2014. 125. Okun S, McGraw D, Stang P, et al. Making the case for continuous learning from routinely collected data. Discussion paper, Institute of Medicine, Washington, DC. 2013. Available at: http://www.iom.edu/ Global/Perspectives/2013/MakingtheCaseforContinuousLearning.aspx. Accessed June 16, 2014. 126. Abernethy AP, Etheredge LM, Ganz PA, et al. Rapid-learning system for cancer care. J Clin Oncol 2010;28:4268-4274.

Challenges Associated With Using Large Data Sets for Quality Assessment and Research in Clinical Settings.

Considerations for using research data to verify clinical data accuracy.

A new tool called DISSECT for analysing large genomic data sets using a Big Data approach.

How efficient is translational research in radiation oncology? The example of a large Dutch academic radiation oncology department.

Considerations for observational research: in regard to Jagsi et al.

An effective filter for IBD detection in large data sets.

BigDataViewer: visualization and processing for large image data sets.

STEME: a robust, accurate motif finder for large data sets.

Comparative effectiveness research in practice and policy for radiation oncology.

Big Data and Comparative Effectiveness Research in Radiation Oncology: Synergy and Accelerated Discovery.

Implications of comparative effectiveness research for radiation oncology.

Using administrative data for your research project: 10 considerations before you begin.

Methods for comparing Salmonella mutagenicity data sets using nonlinear models.

Sustainability considerations for health research and analytic data infrastructures.

Fast Nonparametric Density-Based Clustering of Large Data Sets Using a Stochastic Approximation Mean-Shift Algorithm.

The uses and abuses of large data sets.

Rescuing failures: can large data sets provide the answer?

Comparative effectiveness research in radiation oncology: assessing technology.

Computational derivation of structural alerts from large toxicology data sets.

Design considerations for nanotherapeutics in oncology.

An automated workflow for quantifying RNA transcripts in individual cells in large data-sets.

Using Large Diabetes Databases for Research.

Using observational methods in nursing research.

A Scalable Approach for Protein False Discovery Rate Estimation in Large Proteomic Data Sets.