Practical Radiation Oncology (2012) 2, 296–305
www.practicalradonc.org
Original Report
Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems Benjamin E. Nelms PhD a,b,⁎, Greg Robinson CMD c , Jay Markham CMD c , Kyle Velasco CMD c , Steve Boyd CMD c , Sharath Narayan CMD c , James Wheeler MD, PhD d , Mark L. Sobczak MD e a
Canis Lupus LLC, Merrimac, Wisconsin Department of Human Oncology, University of Wisconsin, Madison, Wisconsin c Radiation Oncology Resources, Goshen, Indiana d Department of Radiation Oncology, Goshen Health System, Goshen, Indiana e Fox Chase Cancer Center, Philadelphia, Pennsylvania b
Received 10 October 2011; revised 18 November 2011; accepted 28 November 2011
Abstract Purpose: This study quantifies variation in radiation treatment plan quality for plans generated by a population of treatment planners given very specific plan objectives. Methods and Materials: A “Plan Quality Metric” (PQM) with 14 submetrics, each with a unique value function, was defined for a prostate treatment plan, serving as specific goals of a hypothetical “virtual physician.” The exact PQM logic was distributed to a population of treatment planners (to remove ambiguity of plan goals or plan assessment methodology) as was a predefined computed tomographic image set and anatomic structure set (to remove anatomy delineation as a variable). Treatment planners used their clinical treatment planning system (TPS) to generate their best plan based on the specified goals and submitted their results for analysis. Results: One hundred forty datasets were received and 125 plans accepted and analyzed. There was wide variability in treatment plan quality (defined as the ability of the planners and plans to meet the specified goals) quantified by the PQM. Despite the variability, the resulting PQM distributions showed no statistically significant difference between TPS employed, modality (intensity modulated radiation therapy versus arc), or education and certification status of the planner. The PQM results showed negligible correlation to number of beam angles, total monitor units, years of experience of the planner, or planner confidence. Conclusions: The ability of the treatment planners to meet the specified plan objectives (as quantified by the PQM) exhibited no statistical dependence on technologic parameters (TPS, modality, plan complexity), nor was the plan quality statistically different based on planner demographics (years of experience, confidence, certification, and education). Therefore, the wide variation in plan quality could be attributed to a general “planner skill” category that would lend itself to processes of
Conflicts of interest: None. ⁎ Corresponding author. E13624 Grace St, Merrimac, WI 53561. E-mail address: [email protected] (B.E. Nelms). 1879-8500/$ – see front matter © 2012 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.prro.2011.11.012
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
297
continual improvement where best practices could be derived and disseminated to improve the mean quality and minimize the variation in any population of treatment planners. © 2012 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved.
Introduction Variation has a negative impact on quality Variation is inherent in manufacturing systems, yet it is widely accepted that variation degrades quality and that the removal (or at least the minimization) of variation increases quality and decreases costs. 1,2 Radiation therapy can be considered a system of major sub-components, such as the following: diagnosis, imaging, anatomy delineation, treatment plan design, pretreatment quality assurance, and per fraction treatment, which itself includes distinct sub-components such as patient setup, image guidance, and dose delivery. Two of these major components, anatomy delineation (also called “contouring”) and treatment plan design, make up what is commonly called “radiation treatment planning.”
Radiation treatment planning Radiation treatment planning is performed by trained clinicians; specifically, physicians, dosimetrists, and in some cases medical physicists (for convenience and generality, we will from here on use the term “treatment planner” or “planner”). Radiation treatment planning tasks are largely computer-driven. The planner first contours patient anatomy structures, including target volumes and organs-at-risk (OAR), based on a volumetric imaging dataset, and these contours form the basis for the design of the treatment plan. Using dose prescriptions and dose constraints defined by the physician for targets and OARs, respectively, the planner will design an arrangement of radiation sources that will meet the objectives. In external beam radiation therapy, the planner can manipulate parameters such as number of beams, beam modalities (ie, photon, electron, etc), and beam geometries, in addition to per-beam parameters such as energy, modulation, and beam intensity or monitor units. Modern treatment planning systems (TPS) automate many of these parameters, in particular the per-beam modulation, via inverse planning computations that create intensity modulated radiation therapy (IMRT) or volume modulated arc therapy (VMAT) beams. The TPS estimates the dose distribution and the plan results are reviewed using 3-dimensional (3D) dose information and clinical analytics such as the dose volume histogram (DVH) per structure. Thus, treatment planning is really performed by a system comprised of the treatment planner, the TPS, and the plan assessment clinician who will ultimately approve a plan.
Given that each treatment plan is highly customized for each patient, it is important to understand sources of variability in the system that could detract from the overall quality of treatment plan output. There have been studies quantifying significant variability in anatomy contouring 3-6 with 1 study showing potentially large impact of contouring variability on the dose and DVH. 6 Regarding treatment plan variation, Matsuo et al 7 did a study of inter-institutional variability in treatment plan output, and Wu et al 8 investigated a post-planning assessment of OAR dose metrics versus a population of prior, similar plans. However, in general there is a lack of attention in the literature on the important focus of treatment plan variability. Specifically, there is a need for carefully designed experiments to measure relative abilities of treatment planners and planning systems to achieve specific quantifiable plan objectives. It is important to quantify and understand the variability of treatment plan quality because doing so provides guidance on the need for action plans, the goals of which would be the following: (1) to decrease the variability; (2) to improve quality; and (3) to reduce overall costs.
Quantifying plan quality for this study: The “Plan Quality Metric” A controlled experiment on treatment plan quality must remove as many variables as possible, including the following: (1) potential variability in anatomy and contours; and (2) inter-clinician variability in judgments of the quality of a plan. The former is easily removed by providing a single patient image set with contours (targets and OARs) already defined. The latter is removed by giving very specific and common treatment planning objectives to all planners along with a mechanism to “score” the treatment plan quality objectively. In this work, we introduce a sophisticated scoring mechanism called the Plan Quality Metric (PQM) that removes any ambiguity of the plan objectives and provides a fair platform to compare plan results. It is important to note that the PQM algorithm employed does not purport to describe the best clinical plan, as this would be debated and difficult to prove; rather, the PQM algorithm used here serves as a an objective method to quantify a plan's quality, specifically in terms of meeting clear and specific treatment plan goals. The origin of the PQM was by request from treatment planners who had participated in previous studies where the judgments of plan quality were more subjective, and the usage of the clear PQM methodology was well received by the base of treatment planners.
298
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Purpose of this work
CT images and anatomy contours
The main purposes of this work are to quantify the following: (1) Inter-clinician variability in treatment plan quality (ie, variability in achieving very specific plan objectives); (2) inter-TPS variability in treatment plan quality; (3) correlation of treatment plan quality to methodsrelated factors such as number of beam geometries, total monitor units, and modality; and (4) correlation of treatment plan quality to clinician-related factors such as experience, certification, and planner confidence.
The patient was scanned at 3-mm slice spacing/thickness with a 550-mm-bore CT scanner (Philips Healthcare, Andover, MA). A series of 90 axial images was acquired, covering a total range of 270-mm superiorinferior. Critical OARs and target volumes were precontoured and reviewed by the clinical team of authors, and these contours were provided to all participating treatment planners. It was imperative to this study to keep the contours fixed, given the known variability in contouring 3-6 that we needed to control (ie, avoid) in this experimental design. In total, 12 critical contour volumes were defined: gross tumor volume (GTV; prostate bed), rectum, bladder, left femoral head, right femoral head, lymph nodes, penile bulb, body (external contour), planning target volume (PTV) 68, PTV 56, target volume outside PTV 68 (PTV 56-PTV 68), and volume outside PTV 68 (body-PTV 68). Figure 1 shows anatomy planes and contour reconstructions through the center of PTV 68.
Methods and materials Patient case Background This study employed a prostate cancer patient's data. The patient had already undergone treatment. This patient had a rising prostate-specific antigen after radical prostatectomy. A computed tomography (CT) of the pelvis showed an enlarged lymph node (N0 at time of surgery), and a CT-guided biopsy of the suspicious node was positive. The patient was offered hormones and external beam radiation therapy.
Figure 1
Data provided to, or collected from, planners The patient data were anonymized and made available for download as DICOM3 images (CT) and DICOM radiotherapy (RT) structure set (contours). Treatment planners
Sagittal, axial, and coronal views of the CT and contour data provided to the treatment planners.
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
299
worldwide were notified of the study and invited to participate. As wide a possible invitation list was formed and included members of the American Association of Medical Dosimetrists and the Medical Dosimetrist Certification Board. All treatment planners were provided with the exact treatment plan objectives and PQM scoring logic (detailed in the next section) via public website and given 5 weeks to submit their plan(s). In addition to patient data and plan objectives, a survey questionnaire was also distributed and included queries such as the following: treatment planning system (and software version); experience (number of years); certified medical dosimetrist (CMD) or non-CMD; treatment planning education institution: accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT), non-JRCERT, or on-the-job-training; and a self-appointed treatment planning confidence rank (1 to 5, 1 = beginner, 5 = master). Essential treatment plan data were collected from all planners in the form of DICOM RT plan and dose files. The additional questionnaire responses were collected via electronic form.
which was posted on a public website alongside the data download center. As such, there was no ambiguity about the objectives or the relative importance of each. There were 14 sub-components of the PQM for this study, each having a unique metric quantity and PQM value function used to calculate a point value based on the submetric. A description of each PQM metric is shown in Table 1, and the value functions are plotted in Fig 2. In addition to publishing the scoring mechanism, we required all planners to submit plans that were both practical and deliverable, specifically fitting into a conventional 15-minute treatment time. With patient setup and image guidance required in the 15 minutes, the treatment time (first beam start to last beam finish) was not allowed to be more than an estimated 10 minutes (a liberal allowance for a prostate plan). The treatment times were estimated using the submitted RT plans’ beams’ control point monitor units and dose rates, combined with interbeam transition time (for IMRT) or gantry rotation speed (for VMAT). For tomotherapy plans, the estimated treatment time was taken directly from the beam's total cumulative meterset, which was recorded in minutes (not monitor units).
Plan objectives and plan quality scoring mechanism
Data analysis: PQM scoring
Treatment plan goals and relative values of each goal were debated and defined by a team of 5 professional treatment planners, reviewed by a radiation oncologist, and rendered as a quantitative PQM scoring mechanism,
A third-party software system called 3DVH (Sun Nuclear Corporation, Melbourne, FL) was used to generate the 14 metric components required as inputs into the PQM calculation. Using a common DVH calculator
Table 1 The 14 metric components derived from each treatment plan along with their minimum and maximum possible point values; maximum sum of values = 150 Structure
Metric
Definition
PQM value range Minimum
PTV 68 PTV 68 PTV 56 GTV PTV 56-PTV 68
V68 Gy (%) D0.03 cc (Gy) V56 Gy (%) V68 Gy (%) V58.8 Gy (%)
Body-PTV 68 PTV 68
V68 Gy (cc) Conformity index
Rectum Rectum Rectum Bladder Bladder –
V68 Gy (cc) V65 Gy (%) V40 Gy (%) V65 Gy (%) V40 Gy (%) Global maximum location
Rectum
Serial rectum
Maximum
Percent of PTV 68 volume ≥ 68 Gy Dose (Gy) covering highest 0.03 cc of PTV 68 Percent of PTV56 volume ≥ 56 Gy Percent of GTV (prostate bed) volume ≥ 68 Gy Percent of the (PTV 56-PTV 68) volume ≥ 58.8 Gy, (ie, percent above 105% of 56 Gy) Volume (cc) of tissue outside PTV 68 ≥ 68 Gy
0 0 0 0 0
30 10 30 10 10
0 0
10 5
Volume (cc) of rectum ≥ 68 Gy Percent of rectum volume ≥ 65 Gy Percent of rectum volume ≥ 40 Gy Percent of bladder volume ≥ 65 Gy Percent of bladder volume ≥ 40 Gy Anatomic location of global maximum: GTV, PTV 68, or elsewhere Number of axial planes with all rectum voxels exceeding 34 Gy
0 0 0 0 0 0
10 10 10 7 3 5
−10
0
½PTV68 V64:6Gy ðccÞ2 ½Total Volume PTV68 ðccÞ T Volume 64:6Gy Isosurface
GTV, gross tumor volume; PQM, Plan Quality Metric; PTV, planning target volume.
300
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Figure 2 The 14 metric components (see Table 1) were assigned numeric values based on Plan Quality Metric (PQM) value functions as detailed in this figure. The final PQM was the sum of all components (maximum possible PQM = 150.0).
removed a potential variable of differences in how each TPS computes DVH statistics. (It should be mentioned that TPS-generated metrics and 3DVH-generated metrics were spot checked for differences over all TPS, and the impact to total PQM scores was generally within 1% with only a few exceptions.) DICOM RT plan and RT dose objects (from each planner) were processed by 3DVH along with the standard and invariable DICOM RT structure set object, and a special software module was created by one author (B.F.N.) to automate the derivation and output of the 14 metric components. These component values were input into a spreadsheet template (Microsoft Excel, Redmond, WA) in which the PQM value functions were
programmed. The sum of the 14 components’ values was recorded as the PQM for each planner.
Results Data received Data itemization A breakdown of the data received is shown in Table 2, including itemization by TPS, modality and technique, total monitor units, and certain demographics collected in the planner survey.
Practical Radiation Oncology: October-December 2012 Table 2
Variation in treatment plan quality
301
Breakdown of data received from treatment planners
Category
No.
Comments/details
Datasets received Datasets accepted
140 125 Two were rejected due to impractical treatment times, others due to multiple plans submitted by one planner. TPS = Eclipse 40 Versions: 6.5 (n = 1), 8.1 (4), 8.2 (2), 8.6 (11), 8.9 (12), 9.0 (1), 10.0 (3), unspecified (4) TPS = Pinnacle 70 Versions: 7.6 (n = 1), 8.× (24), 9.× (37), unspecified (8) TPS = XiO 7 Versions: 4.5 (n = 1), 4.6 (4), unspecified (2) TPS = tomotherapy 5 Versions: 3.1.2.9 (n = 1), 4.0.4 (3), unspecified (1) TPS = Monaco 3 Versions: 2.0 (n = 2), 3.0 (1) IMRT (fixed gantry) 106 Number of unique beam geometries ranged from 5 to 11. Arc (VMAT or tomotherapy) 19 VMAT (n = 14), tomotherapy (5) Total monitor units (MU) 125 Range: 412-1951 Experience (y) 125 Range: 1-38 Certification 125 CMD (n = 101), Non-CMD (24) Education 125 JRCERT (N = 27), Non-JRCERT (15), OJT (75), Unspecified (8) Planner confidence 125 Levels: 1-beginner (N = 3), 2 (2), 3 (27), 4 (59), 5-master (42), unspecified (2) CMD, certified medical dosimetrist; IMRT, intensity modulated radiation therapy; OJT, on-the-job-training; JRCERT, Joint Review Committee on Education in Radiologic Technology; TPS, treatment planning system; VMAT, volume modulated arc therapy.
Rejected plan(s) A total of 15 plans were rejected. Thirteen of these were rejected only because they were multiple entries from individual planners, and in each planner's case we retained their plan with the highest PQM and removed the others to avoid potential bias in the data caused by a single planner having multiple data points. The only other rejections were 2 tomotherapy plans that had long treatment times (11.5 and 38.5 minutes, respectively, as recorded in the DICOM RT plan) that would render the plan untreatable within the 15-minute overall treatment time slot specified in the initial instructions.
PQM distributions and correlations Total population distribution The PQM distribution for all plans is shown in Fig 3A. There was a wide range of PQM scores (58.2-142.5) with a mean of 116.9 and standard deviation of 16.4, resulting in a coefficient of variation (CV) of 0.14. Distribution per TPS PQM distributions for the 5 TPS represented in this study are shown in Figures 3E – 3I. The PQM averages from highest to lowest were: Eclipse (Varian Medical Systems, Palo Alto, CA), Tomotherapy (Tomotherapy, Madison, WI), Pinnacle (Philips, Madison, WI), followed by XiO and Monaco (Elekta, St. Louis, MO); however, there was no statistically significant difference between any of the PQM distributions using the 2-sample t test at the 95% and 90% confidence levels. The PQM maximum scores from highest to lowest by TPS were: Pinnacle, Eclipse, Monaco, Tomotherapy, and XiO. Each TPS distribution
shows a fairly wide range of PQM scores, with the lowest coefficient of variation = 0.088 (tomotherapy plans). Distributions: IMRT versus arc Figure 3B shows the PQM distributions filtered by IMRT (static gantry beams) versus VMAT and tomotherapy (dynamic arc beams). The means were similar (116.6 for IMRT vs 118.8 for arc) and not statistically different at the 95% or 90% confidence level. The variability of the dynamic arc plans was less (standard deviation of 11.1 vs 17.1 for IMRT).
Correlation versus planning techniques There was a weak correlation (Pearson r value = 0.313) of PQM with the number of unique beam angles for IMRT plans, as seen in Fig 4A. There was negligible correlation (r value = 0.152) with total monitor units, as seen in Fig 4B.
Correlation versus planner demographics PQM distributions for both CMDs and noncertified planners are shown in Fig 3C. The non-CMD planners had a higher mean (122.5 vs 115.6 for CMD) though the differences were not statistically significant at the 95% or 90% levels (2-sample t test). It was observed that all PQM scores below 100 were for plans submitted by CMDs. Figure 3D shows 3 distributions, this time filtered by whether the planner was educated at a JRCERT-accredited program, non-JRCERT, or via on-the-job-training. The results are not statistically different. Figure 4C plots PQM versus years of professional experience of the planner, and there was negligible correlation (r value = 0.168), although it is observed that
302
B.E. Nelms et al
there are no very low scores (PQM b 100) for planners with 20+ years of experience. Finally, there was negligible correlation of PQM with self-rated planner “confidence” (r value = 0.184), as shown in Fig 4D.
Discussion PQM: A useful methodology? For this work, it was essential to use methodology like the PQM mechanism to ensure that all planners were planning to specific and exact objectives. In this sense, the PQM acts as the “virtual physician” communicating plan goals and constraints to the treatment planner, along with the relative weights of each objective. As stated in the Introduction, this does not imply that the specific PQM logic used here is necessarily clinically optimal, though it was certainly based on realistic objectives of a highly conformal prostate plan. The PQM method could be generalized as a 3D plan and DVH reduction technique using weighted objectives with customizable objective functions. Besides being useful for a study on planning variability, the PQM method might be useful in unifying the following: (1) optimization functions used in inverse planning; and (2) the approach of plan review by physicians. The goal would be to further remove variability not only from the plan outputs, but also in the judgment and scoring of resulting plans by physicians. Of course, physician task groups would need to design and approve the PQM components and their associated value functions, which could perhaps be verified by retrospective analysis of clinical trials. At minimum, the PQM method serves as a “check list” to ensure that all critical objectives are considered by the planner and monitored in plan review.
PQM variability and lack of correlation to TPS, modality, and planner demographics Two observations stand out from these results: (1) high variability in inter-planner treatment plan quality; and (2) lack of correlation to the variables studied. Prior to this study, potential hypotheses for common causes of variation in treatment plan quality might have included the following: usage of different TPS (ie, varying TPS performance); number of beam angles (for IMRT); plan complexity; total monitor units; or different techniques such as fixed gantry IMRT versus dynamic arc plans. However, breaking the total PQM distribution into per-TPS distributions does not show
Practical Radiation Oncology: October-December 2012
statistical difference between any 2 TPS. Breaking into IMRT and arc distributions also shows no statistical difference. As far as the correlation with the number of beam angles or total monitor units (both somewhat indicative of a plan's complexity), there was negligible to weak correlation. In fact, it is worth noting that some plans in the top 10 percentile were achieved with 6- and 7-field IMRT plans, and that the highest PQM achieved (142.5) was for a plan with one of the lower total monitor units counts (b800 MU). Factors related to planner demographics (rather than techniques or technology) also did not correlate with differences in plan quality. Professional certification, years of experience, education accreditation, and even planner confidence are not good predictors of plan quality.
Limitations of this study Uncontrolled variables Not all sources of variation were removed from the experimental design. For example, the dose calculation algorithm and the accuracy of the TPS beam model were not designed out as variables; however, the planners used clinically accepted beam models for their plans so at the minimum we know the algorithm and beam models all underwent some form of acceptance testing. Another variable that was not restricted to a constant (but could have been) was the dose grid resolution (ie, the distance between dose points on the exported RT dose grid). In the submitted plans, this variable ranged from 2 mm to 4 mm as defined by the user in their TPS as per their clinical preferences. However, spot checks of DVH results of 2-mm calculations and 4-mm calculations of the same plan yield similar DVH results (±1%) with larger differences occurring for small volume structures or metrics such as “max dose.” In the future, PQM studies should require 3 mm or smaller dose grid resolution to control this variability. Perhaps the most important variable that was not fixed was a limit on the cumulative planning time (ie, the time spent by the planner start-to-finish). The trial was open for a long time (∼5 weeks) compared with a common time limit on generating a treatment plan for a patient (∼2 hours-2 days, depending on the complexity. Ideally, a fixed time limit would have been imposed to ensure that a large time allowance was not biasing results toward higher-than-normal quality metrics. However, in order to get a large number of data points (ie, plans submitted) we
N
Figure 3 Plan Quality Metric (PQM) distributions. (A) All accepted plans (n = 125). (B) Fixed gantry intensity modulated radiation therapy (IMRT) plans (n = 106) and arc plans made up of volume modulated arc therapy and tomotherapy plans (n = 19). (C) Certification level: certified medical dosimetrist (CMD) (n = 101) and non-CMD (n = 24). (D) Professional education: Joint Review Committee on Education in Radiologic Technology (JRCERT) accredited institution (n = 27), non-JRCERT (n = 15), and on-the-job-training (n = 75). For the following 5 commercial treatment planning systems used by participating treatment planners: (E) Eclipse (n = 40); (F) Pinnacle (n = 70); (G) XiO (n = 7); (H) Tomotherapy (n = 5); and (I) Monaco (n = 3).
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
303
304
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Figure 4 Plan Quality Metric (PQM) correlation graphics. (A) PQM versus number of unique beam angles for intensity modulated radiation therapy (IMRT) plans; Pearson r value = 0.313. (B) PQM versus total monitor units over all plans (r value = 0.152). (C) PQM versus planner's years of experience (r value = 0.168). (D) PQM versus planner's confidence level (1-5), where 1 = beginner and 5 = master (r value = 0.184).
had to give flexibility of the participants to fit this test plan into their schedules. For a first study, we determined that a large sample was imperative.
Other limitations There were other limitations to this study. First, the number of plans submitted for 3 of the 5 treatment planning systems was low (b10 each), and future PQM surveys would benefit from a larger sample size per TPS. Second, this was an international study, and because the definition of “certified” medical dosimetrist is not used outside the United States, all non-USA planners were in the non-CMD category, as were any planners that were medical physicists by trade. Finally, an indirect limitation of this study is a potential bias in the results given that the planner participation was voluntary; ie, it could be postulated that only planners who had some degree of satisfaction with their plan were likely to submit their results for analysis.
Removing variation in treatment plan quality: Next steps We hypothesize that the observed variation in plan quality is due to variability in what could be generalized into a category of “planning skills” that are not necessarily
guaranteed by details of certification, education, or years of experience. This offers the opportunity to introduce processes of continual improvement such as the Shewhart or the Deming cycle 9,10 (sometimes called “plan-docheck-act” or “plan-do-study-act”). Such a process of continual improvement requires objective measures of performance in order to quantify the effects of each improvement plan attempted, and the PQM method introduced here fulfills this need. Continual improvement initiatives might help identify the best practices of the high performers and enable training the population of planners on the skills or techniques that allow them to better meet any specified plan objectives. The goals would be to remove variation in plan quality from the population of treatment planners, and to raise the mean quality toward the levels of highest performance. In addition, there is a clear opportunity to implement a suite of assessment tests of planning ability (measured by a system such as PQM or other fair measures) as a part of treatment planning education programs and certification. It would be advantageous to introduce such systems prior to clinical treatment planning to ensure a base level of performance of each treatment planner, but then continually invest in training (best practices, new techniques, etc) throughout the treatment planner's career, using these objective measures to gauge their performance and also the effectiveness of the training strategies.
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
Conclusions 4.
There is a large inter-planner variation in plan quality as defined by a quantitative PQM score that measures the ability of the planner to meet very specific plan objectives. Plan quality was not statistically different between different TPS or delivery techniques and was not correlated to metrics of plan complexity. Certification and education demographics, experience, and confidence level of the planner were not good predictors of plan quality.
References 1. Deming WE. Out of the Crisis. Cambridge, MA: MIT Press. 1986. 2. Aguayo R. Dr. Deming: The American who taught the Japanese about quality. New York, NY: Simon and Schuster; 1991. 3. Li XA, Tai A, Arthur DW, et al. Variability of target and normal structure delineation for breast cancer radiotherapy: An RTOG
5.
6.
7.
8.
9. 10.
305
multi-institutional and multiobserver Study. Int J Radiat Oncol Biol Phys. 2009;73:944-951. Hurkmans CW, Borger JH, Pieters BR, Russell NS, Jansen EP, Mijnheer BJ. Variability in target volume delineation on CT scans of the breast. Int J Radiat Oncol Biol Phys. 2001;50:1366-1372. Yamamoto M, Nagata Y, Okajima K, et al. Differences in target outline delineation from CT scans of brain tumours using different methods and different observers. Radiother Oncol. 1999;50:151-156. Nelms BE, Tomé WA, G Robinson, Wheeler J. Variations in the contouring of organs at risk: Test case from a patient with oropharyngeal cancer [E-pub ahead of print November 29, 2010]. Int J Radiat Oncol Biol Phys. doi:10.1016/j.ijrobp.2010.10.019. Matsuo Y, Takayama K, Nagata Y, et al. Interinstitutional variations in planning for stereotactic body radiation therapy for lung cancer.. Int J Radiat Oncol Biol Phys. 2007;68:416-425. Wu B, Ricchetti F, Sanguineti G, et al. Patient geometry-driven information retrieval for IMRT treatment plan quality control.. Med Phys. 2009;36:5497-5505. Shewhart WA. 1939. Statistical method from the viewpoint of quality control. Department of Agriculture. Dover; 1986:45. Deming WE. Elementary principles of the statistical control of quality, Union of Japanese Scientists and Engineers (JUSE).. 1950.
www.practicalradonc.org
Original Report
Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems Benjamin E. Nelms PhD a,b,⁎, Greg Robinson CMD c , Jay Markham CMD c , Kyle Velasco CMD c , Steve Boyd CMD c , Sharath Narayan CMD c , James Wheeler MD, PhD d , Mark L. Sobczak MD e a
Canis Lupus LLC, Merrimac, Wisconsin Department of Human Oncology, University of Wisconsin, Madison, Wisconsin c Radiation Oncology Resources, Goshen, Indiana d Department of Radiation Oncology, Goshen Health System, Goshen, Indiana e Fox Chase Cancer Center, Philadelphia, Pennsylvania b
Received 10 October 2011; revised 18 November 2011; accepted 28 November 2011
Abstract Purpose: This study quantifies variation in radiation treatment plan quality for plans generated by a population of treatment planners given very specific plan objectives. Methods and Materials: A “Plan Quality Metric” (PQM) with 14 submetrics, each with a unique value function, was defined for a prostate treatment plan, serving as specific goals of a hypothetical “virtual physician.” The exact PQM logic was distributed to a population of treatment planners (to remove ambiguity of plan goals or plan assessment methodology) as was a predefined computed tomographic image set and anatomic structure set (to remove anatomy delineation as a variable). Treatment planners used their clinical treatment planning system (TPS) to generate their best plan based on the specified goals and submitted their results for analysis. Results: One hundred forty datasets were received and 125 plans accepted and analyzed. There was wide variability in treatment plan quality (defined as the ability of the planners and plans to meet the specified goals) quantified by the PQM. Despite the variability, the resulting PQM distributions showed no statistically significant difference between TPS employed, modality (intensity modulated radiation therapy versus arc), or education and certification status of the planner. The PQM results showed negligible correlation to number of beam angles, total monitor units, years of experience of the planner, or planner confidence. Conclusions: The ability of the treatment planners to meet the specified plan objectives (as quantified by the PQM) exhibited no statistical dependence on technologic parameters (TPS, modality, plan complexity), nor was the plan quality statistically different based on planner demographics (years of experience, confidence, certification, and education). Therefore, the wide variation in plan quality could be attributed to a general “planner skill” category that would lend itself to processes of
Conflicts of interest: None. ⁎ Corresponding author. E13624 Grace St, Merrimac, WI 53561. E-mail address: [email protected] (B.E. Nelms). 1879-8500/$ – see front matter © 2012 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.prro.2011.11.012
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
297
continual improvement where best practices could be derived and disseminated to improve the mean quality and minimize the variation in any population of treatment planners. © 2012 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved.
Introduction Variation has a negative impact on quality Variation is inherent in manufacturing systems, yet it is widely accepted that variation degrades quality and that the removal (or at least the minimization) of variation increases quality and decreases costs. 1,2 Radiation therapy can be considered a system of major sub-components, such as the following: diagnosis, imaging, anatomy delineation, treatment plan design, pretreatment quality assurance, and per fraction treatment, which itself includes distinct sub-components such as patient setup, image guidance, and dose delivery. Two of these major components, anatomy delineation (also called “contouring”) and treatment plan design, make up what is commonly called “radiation treatment planning.”
Radiation treatment planning Radiation treatment planning is performed by trained clinicians; specifically, physicians, dosimetrists, and in some cases medical physicists (for convenience and generality, we will from here on use the term “treatment planner” or “planner”). Radiation treatment planning tasks are largely computer-driven. The planner first contours patient anatomy structures, including target volumes and organs-at-risk (OAR), based on a volumetric imaging dataset, and these contours form the basis for the design of the treatment plan. Using dose prescriptions and dose constraints defined by the physician for targets and OARs, respectively, the planner will design an arrangement of radiation sources that will meet the objectives. In external beam radiation therapy, the planner can manipulate parameters such as number of beams, beam modalities (ie, photon, electron, etc), and beam geometries, in addition to per-beam parameters such as energy, modulation, and beam intensity or monitor units. Modern treatment planning systems (TPS) automate many of these parameters, in particular the per-beam modulation, via inverse planning computations that create intensity modulated radiation therapy (IMRT) or volume modulated arc therapy (VMAT) beams. The TPS estimates the dose distribution and the plan results are reviewed using 3-dimensional (3D) dose information and clinical analytics such as the dose volume histogram (DVH) per structure. Thus, treatment planning is really performed by a system comprised of the treatment planner, the TPS, and the plan assessment clinician who will ultimately approve a plan.
Given that each treatment plan is highly customized for each patient, it is important to understand sources of variability in the system that could detract from the overall quality of treatment plan output. There have been studies quantifying significant variability in anatomy contouring 3-6 with 1 study showing potentially large impact of contouring variability on the dose and DVH. 6 Regarding treatment plan variation, Matsuo et al 7 did a study of inter-institutional variability in treatment plan output, and Wu et al 8 investigated a post-planning assessment of OAR dose metrics versus a population of prior, similar plans. However, in general there is a lack of attention in the literature on the important focus of treatment plan variability. Specifically, there is a need for carefully designed experiments to measure relative abilities of treatment planners and planning systems to achieve specific quantifiable plan objectives. It is important to quantify and understand the variability of treatment plan quality because doing so provides guidance on the need for action plans, the goals of which would be the following: (1) to decrease the variability; (2) to improve quality; and (3) to reduce overall costs.
Quantifying plan quality for this study: The “Plan Quality Metric” A controlled experiment on treatment plan quality must remove as many variables as possible, including the following: (1) potential variability in anatomy and contours; and (2) inter-clinician variability in judgments of the quality of a plan. The former is easily removed by providing a single patient image set with contours (targets and OARs) already defined. The latter is removed by giving very specific and common treatment planning objectives to all planners along with a mechanism to “score” the treatment plan quality objectively. In this work, we introduce a sophisticated scoring mechanism called the Plan Quality Metric (PQM) that removes any ambiguity of the plan objectives and provides a fair platform to compare plan results. It is important to note that the PQM algorithm employed does not purport to describe the best clinical plan, as this would be debated and difficult to prove; rather, the PQM algorithm used here serves as a an objective method to quantify a plan's quality, specifically in terms of meeting clear and specific treatment plan goals. The origin of the PQM was by request from treatment planners who had participated in previous studies where the judgments of plan quality were more subjective, and the usage of the clear PQM methodology was well received by the base of treatment planners.
298
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Purpose of this work
CT images and anatomy contours
The main purposes of this work are to quantify the following: (1) Inter-clinician variability in treatment plan quality (ie, variability in achieving very specific plan objectives); (2) inter-TPS variability in treatment plan quality; (3) correlation of treatment plan quality to methodsrelated factors such as number of beam geometries, total monitor units, and modality; and (4) correlation of treatment plan quality to clinician-related factors such as experience, certification, and planner confidence.
The patient was scanned at 3-mm slice spacing/thickness with a 550-mm-bore CT scanner (Philips Healthcare, Andover, MA). A series of 90 axial images was acquired, covering a total range of 270-mm superiorinferior. Critical OARs and target volumes were precontoured and reviewed by the clinical team of authors, and these contours were provided to all participating treatment planners. It was imperative to this study to keep the contours fixed, given the known variability in contouring 3-6 that we needed to control (ie, avoid) in this experimental design. In total, 12 critical contour volumes were defined: gross tumor volume (GTV; prostate bed), rectum, bladder, left femoral head, right femoral head, lymph nodes, penile bulb, body (external contour), planning target volume (PTV) 68, PTV 56, target volume outside PTV 68 (PTV 56-PTV 68), and volume outside PTV 68 (body-PTV 68). Figure 1 shows anatomy planes and contour reconstructions through the center of PTV 68.
Methods and materials Patient case Background This study employed a prostate cancer patient's data. The patient had already undergone treatment. This patient had a rising prostate-specific antigen after radical prostatectomy. A computed tomography (CT) of the pelvis showed an enlarged lymph node (N0 at time of surgery), and a CT-guided biopsy of the suspicious node was positive. The patient was offered hormones and external beam radiation therapy.
Figure 1
Data provided to, or collected from, planners The patient data were anonymized and made available for download as DICOM3 images (CT) and DICOM radiotherapy (RT) structure set (contours). Treatment planners
Sagittal, axial, and coronal views of the CT and contour data provided to the treatment planners.
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
299
worldwide were notified of the study and invited to participate. As wide a possible invitation list was formed and included members of the American Association of Medical Dosimetrists and the Medical Dosimetrist Certification Board. All treatment planners were provided with the exact treatment plan objectives and PQM scoring logic (detailed in the next section) via public website and given 5 weeks to submit their plan(s). In addition to patient data and plan objectives, a survey questionnaire was also distributed and included queries such as the following: treatment planning system (and software version); experience (number of years); certified medical dosimetrist (CMD) or non-CMD; treatment planning education institution: accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT), non-JRCERT, or on-the-job-training; and a self-appointed treatment planning confidence rank (1 to 5, 1 = beginner, 5 = master). Essential treatment plan data were collected from all planners in the form of DICOM RT plan and dose files. The additional questionnaire responses were collected via electronic form.
which was posted on a public website alongside the data download center. As such, there was no ambiguity about the objectives or the relative importance of each. There were 14 sub-components of the PQM for this study, each having a unique metric quantity and PQM value function used to calculate a point value based on the submetric. A description of each PQM metric is shown in Table 1, and the value functions are plotted in Fig 2. In addition to publishing the scoring mechanism, we required all planners to submit plans that were both practical and deliverable, specifically fitting into a conventional 15-minute treatment time. With patient setup and image guidance required in the 15 minutes, the treatment time (first beam start to last beam finish) was not allowed to be more than an estimated 10 minutes (a liberal allowance for a prostate plan). The treatment times were estimated using the submitted RT plans’ beams’ control point monitor units and dose rates, combined with interbeam transition time (for IMRT) or gantry rotation speed (for VMAT). For tomotherapy plans, the estimated treatment time was taken directly from the beam's total cumulative meterset, which was recorded in minutes (not monitor units).
Plan objectives and plan quality scoring mechanism
Data analysis: PQM scoring
Treatment plan goals and relative values of each goal were debated and defined by a team of 5 professional treatment planners, reviewed by a radiation oncologist, and rendered as a quantitative PQM scoring mechanism,
A third-party software system called 3DVH (Sun Nuclear Corporation, Melbourne, FL) was used to generate the 14 metric components required as inputs into the PQM calculation. Using a common DVH calculator
Table 1 The 14 metric components derived from each treatment plan along with their minimum and maximum possible point values; maximum sum of values = 150 Structure
Metric
Definition
PQM value range Minimum
PTV 68 PTV 68 PTV 56 GTV PTV 56-PTV 68
V68 Gy (%) D0.03 cc (Gy) V56 Gy (%) V68 Gy (%) V58.8 Gy (%)
Body-PTV 68 PTV 68
V68 Gy (cc) Conformity index
Rectum Rectum Rectum Bladder Bladder –
V68 Gy (cc) V65 Gy (%) V40 Gy (%) V65 Gy (%) V40 Gy (%) Global maximum location
Rectum
Serial rectum
Maximum
Percent of PTV 68 volume ≥ 68 Gy Dose (Gy) covering highest 0.03 cc of PTV 68 Percent of PTV56 volume ≥ 56 Gy Percent of GTV (prostate bed) volume ≥ 68 Gy Percent of the (PTV 56-PTV 68) volume ≥ 58.8 Gy, (ie, percent above 105% of 56 Gy) Volume (cc) of tissue outside PTV 68 ≥ 68 Gy
0 0 0 0 0
30 10 30 10 10
0 0
10 5
Volume (cc) of rectum ≥ 68 Gy Percent of rectum volume ≥ 65 Gy Percent of rectum volume ≥ 40 Gy Percent of bladder volume ≥ 65 Gy Percent of bladder volume ≥ 40 Gy Anatomic location of global maximum: GTV, PTV 68, or elsewhere Number of axial planes with all rectum voxels exceeding 34 Gy
0 0 0 0 0 0
10 10 10 7 3 5
−10
0
½PTV68 V64:6Gy ðccÞ2 ½Total Volume PTV68 ðccÞ T Volume 64:6Gy Isosurface
GTV, gross tumor volume; PQM, Plan Quality Metric; PTV, planning target volume.
300
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Figure 2 The 14 metric components (see Table 1) were assigned numeric values based on Plan Quality Metric (PQM) value functions as detailed in this figure. The final PQM was the sum of all components (maximum possible PQM = 150.0).
removed a potential variable of differences in how each TPS computes DVH statistics. (It should be mentioned that TPS-generated metrics and 3DVH-generated metrics were spot checked for differences over all TPS, and the impact to total PQM scores was generally within 1% with only a few exceptions.) DICOM RT plan and RT dose objects (from each planner) were processed by 3DVH along with the standard and invariable DICOM RT structure set object, and a special software module was created by one author (B.F.N.) to automate the derivation and output of the 14 metric components. These component values were input into a spreadsheet template (Microsoft Excel, Redmond, WA) in which the PQM value functions were
programmed. The sum of the 14 components’ values was recorded as the PQM for each planner.
Results Data received Data itemization A breakdown of the data received is shown in Table 2, including itemization by TPS, modality and technique, total monitor units, and certain demographics collected in the planner survey.
Practical Radiation Oncology: October-December 2012 Table 2
Variation in treatment plan quality
301
Breakdown of data received from treatment planners
Category
No.
Comments/details
Datasets received Datasets accepted
140 125 Two were rejected due to impractical treatment times, others due to multiple plans submitted by one planner. TPS = Eclipse 40 Versions: 6.5 (n = 1), 8.1 (4), 8.2 (2), 8.6 (11), 8.9 (12), 9.0 (1), 10.0 (3), unspecified (4) TPS = Pinnacle 70 Versions: 7.6 (n = 1), 8.× (24), 9.× (37), unspecified (8) TPS = XiO 7 Versions: 4.5 (n = 1), 4.6 (4), unspecified (2) TPS = tomotherapy 5 Versions: 3.1.2.9 (n = 1), 4.0.4 (3), unspecified (1) TPS = Monaco 3 Versions: 2.0 (n = 2), 3.0 (1) IMRT (fixed gantry) 106 Number of unique beam geometries ranged from 5 to 11. Arc (VMAT or tomotherapy) 19 VMAT (n = 14), tomotherapy (5) Total monitor units (MU) 125 Range: 412-1951 Experience (y) 125 Range: 1-38 Certification 125 CMD (n = 101), Non-CMD (24) Education 125 JRCERT (N = 27), Non-JRCERT (15), OJT (75), Unspecified (8) Planner confidence 125 Levels: 1-beginner (N = 3), 2 (2), 3 (27), 4 (59), 5-master (42), unspecified (2) CMD, certified medical dosimetrist; IMRT, intensity modulated radiation therapy; OJT, on-the-job-training; JRCERT, Joint Review Committee on Education in Radiologic Technology; TPS, treatment planning system; VMAT, volume modulated arc therapy.
Rejected plan(s) A total of 15 plans were rejected. Thirteen of these were rejected only because they were multiple entries from individual planners, and in each planner's case we retained their plan with the highest PQM and removed the others to avoid potential bias in the data caused by a single planner having multiple data points. The only other rejections were 2 tomotherapy plans that had long treatment times (11.5 and 38.5 minutes, respectively, as recorded in the DICOM RT plan) that would render the plan untreatable within the 15-minute overall treatment time slot specified in the initial instructions.
PQM distributions and correlations Total population distribution The PQM distribution for all plans is shown in Fig 3A. There was a wide range of PQM scores (58.2-142.5) with a mean of 116.9 and standard deviation of 16.4, resulting in a coefficient of variation (CV) of 0.14. Distribution per TPS PQM distributions for the 5 TPS represented in this study are shown in Figures 3E – 3I. The PQM averages from highest to lowest were: Eclipse (Varian Medical Systems, Palo Alto, CA), Tomotherapy (Tomotherapy, Madison, WI), Pinnacle (Philips, Madison, WI), followed by XiO and Monaco (Elekta, St. Louis, MO); however, there was no statistically significant difference between any of the PQM distributions using the 2-sample t test at the 95% and 90% confidence levels. The PQM maximum scores from highest to lowest by TPS were: Pinnacle, Eclipse, Monaco, Tomotherapy, and XiO. Each TPS distribution
shows a fairly wide range of PQM scores, with the lowest coefficient of variation = 0.088 (tomotherapy plans). Distributions: IMRT versus arc Figure 3B shows the PQM distributions filtered by IMRT (static gantry beams) versus VMAT and tomotherapy (dynamic arc beams). The means were similar (116.6 for IMRT vs 118.8 for arc) and not statistically different at the 95% or 90% confidence level. The variability of the dynamic arc plans was less (standard deviation of 11.1 vs 17.1 for IMRT).
Correlation versus planning techniques There was a weak correlation (Pearson r value = 0.313) of PQM with the number of unique beam angles for IMRT plans, as seen in Fig 4A. There was negligible correlation (r value = 0.152) with total monitor units, as seen in Fig 4B.
Correlation versus planner demographics PQM distributions for both CMDs and noncertified planners are shown in Fig 3C. The non-CMD planners had a higher mean (122.5 vs 115.6 for CMD) though the differences were not statistically significant at the 95% or 90% levels (2-sample t test). It was observed that all PQM scores below 100 were for plans submitted by CMDs. Figure 3D shows 3 distributions, this time filtered by whether the planner was educated at a JRCERT-accredited program, non-JRCERT, or via on-the-job-training. The results are not statistically different. Figure 4C plots PQM versus years of professional experience of the planner, and there was negligible correlation (r value = 0.168), although it is observed that
302
B.E. Nelms et al
there are no very low scores (PQM b 100) for planners with 20+ years of experience. Finally, there was negligible correlation of PQM with self-rated planner “confidence” (r value = 0.184), as shown in Fig 4D.
Discussion PQM: A useful methodology? For this work, it was essential to use methodology like the PQM mechanism to ensure that all planners were planning to specific and exact objectives. In this sense, the PQM acts as the “virtual physician” communicating plan goals and constraints to the treatment planner, along with the relative weights of each objective. As stated in the Introduction, this does not imply that the specific PQM logic used here is necessarily clinically optimal, though it was certainly based on realistic objectives of a highly conformal prostate plan. The PQM method could be generalized as a 3D plan and DVH reduction technique using weighted objectives with customizable objective functions. Besides being useful for a study on planning variability, the PQM method might be useful in unifying the following: (1) optimization functions used in inverse planning; and (2) the approach of plan review by physicians. The goal would be to further remove variability not only from the plan outputs, but also in the judgment and scoring of resulting plans by physicians. Of course, physician task groups would need to design and approve the PQM components and their associated value functions, which could perhaps be verified by retrospective analysis of clinical trials. At minimum, the PQM method serves as a “check list” to ensure that all critical objectives are considered by the planner and monitored in plan review.
PQM variability and lack of correlation to TPS, modality, and planner demographics Two observations stand out from these results: (1) high variability in inter-planner treatment plan quality; and (2) lack of correlation to the variables studied. Prior to this study, potential hypotheses for common causes of variation in treatment plan quality might have included the following: usage of different TPS (ie, varying TPS performance); number of beam angles (for IMRT); plan complexity; total monitor units; or different techniques such as fixed gantry IMRT versus dynamic arc plans. However, breaking the total PQM distribution into per-TPS distributions does not show
Practical Radiation Oncology: October-December 2012
statistical difference between any 2 TPS. Breaking into IMRT and arc distributions also shows no statistical difference. As far as the correlation with the number of beam angles or total monitor units (both somewhat indicative of a plan's complexity), there was negligible to weak correlation. In fact, it is worth noting that some plans in the top 10 percentile were achieved with 6- and 7-field IMRT plans, and that the highest PQM achieved (142.5) was for a plan with one of the lower total monitor units counts (b800 MU). Factors related to planner demographics (rather than techniques or technology) also did not correlate with differences in plan quality. Professional certification, years of experience, education accreditation, and even planner confidence are not good predictors of plan quality.
Limitations of this study Uncontrolled variables Not all sources of variation were removed from the experimental design. For example, the dose calculation algorithm and the accuracy of the TPS beam model were not designed out as variables; however, the planners used clinically accepted beam models for their plans so at the minimum we know the algorithm and beam models all underwent some form of acceptance testing. Another variable that was not restricted to a constant (but could have been) was the dose grid resolution (ie, the distance between dose points on the exported RT dose grid). In the submitted plans, this variable ranged from 2 mm to 4 mm as defined by the user in their TPS as per their clinical preferences. However, spot checks of DVH results of 2-mm calculations and 4-mm calculations of the same plan yield similar DVH results (±1%) with larger differences occurring for small volume structures or metrics such as “max dose.” In the future, PQM studies should require 3 mm or smaller dose grid resolution to control this variability. Perhaps the most important variable that was not fixed was a limit on the cumulative planning time (ie, the time spent by the planner start-to-finish). The trial was open for a long time (∼5 weeks) compared with a common time limit on generating a treatment plan for a patient (∼2 hours-2 days, depending on the complexity. Ideally, a fixed time limit would have been imposed to ensure that a large time allowance was not biasing results toward higher-than-normal quality metrics. However, in order to get a large number of data points (ie, plans submitted) we
N
Figure 3 Plan Quality Metric (PQM) distributions. (A) All accepted plans (n = 125). (B) Fixed gantry intensity modulated radiation therapy (IMRT) plans (n = 106) and arc plans made up of volume modulated arc therapy and tomotherapy plans (n = 19). (C) Certification level: certified medical dosimetrist (CMD) (n = 101) and non-CMD (n = 24). (D) Professional education: Joint Review Committee on Education in Radiologic Technology (JRCERT) accredited institution (n = 27), non-JRCERT (n = 15), and on-the-job-training (n = 75). For the following 5 commercial treatment planning systems used by participating treatment planners: (E) Eclipse (n = 40); (F) Pinnacle (n = 70); (G) XiO (n = 7); (H) Tomotherapy (n = 5); and (I) Monaco (n = 3).
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
303
304
B.E. Nelms et al
Practical Radiation Oncology: October-December 2012
Figure 4 Plan Quality Metric (PQM) correlation graphics. (A) PQM versus number of unique beam angles for intensity modulated radiation therapy (IMRT) plans; Pearson r value = 0.313. (B) PQM versus total monitor units over all plans (r value = 0.152). (C) PQM versus planner's years of experience (r value = 0.168). (D) PQM versus planner's confidence level (1-5), where 1 = beginner and 5 = master (r value = 0.184).
had to give flexibility of the participants to fit this test plan into their schedules. For a first study, we determined that a large sample was imperative.
Other limitations There were other limitations to this study. First, the number of plans submitted for 3 of the 5 treatment planning systems was low (b10 each), and future PQM surveys would benefit from a larger sample size per TPS. Second, this was an international study, and because the definition of “certified” medical dosimetrist is not used outside the United States, all non-USA planners were in the non-CMD category, as were any planners that were medical physicists by trade. Finally, an indirect limitation of this study is a potential bias in the results given that the planner participation was voluntary; ie, it could be postulated that only planners who had some degree of satisfaction with their plan were likely to submit their results for analysis.
Removing variation in treatment plan quality: Next steps We hypothesize that the observed variation in plan quality is due to variability in what could be generalized into a category of “planning skills” that are not necessarily
guaranteed by details of certification, education, or years of experience. This offers the opportunity to introduce processes of continual improvement such as the Shewhart or the Deming cycle 9,10 (sometimes called “plan-docheck-act” or “plan-do-study-act”). Such a process of continual improvement requires objective measures of performance in order to quantify the effects of each improvement plan attempted, and the PQM method introduced here fulfills this need. Continual improvement initiatives might help identify the best practices of the high performers and enable training the population of planners on the skills or techniques that allow them to better meet any specified plan objectives. The goals would be to remove variation in plan quality from the population of treatment planners, and to raise the mean quality toward the levels of highest performance. In addition, there is a clear opportunity to implement a suite of assessment tests of planning ability (measured by a system such as PQM or other fair measures) as a part of treatment planning education programs and certification. It would be advantageous to introduce such systems prior to clinical treatment planning to ensure a base level of performance of each treatment planner, but then continually invest in training (best practices, new techniques, etc) throughout the treatment planner's career, using these objective measures to gauge their performance and also the effectiveness of the training strategies.
Practical Radiation Oncology: October-December 2012
Variation in treatment plan quality
Conclusions 4.
There is a large inter-planner variation in plan quality as defined by a quantitative PQM score that measures the ability of the planner to meet very specific plan objectives. Plan quality was not statistically different between different TPS or delivery techniques and was not correlated to metrics of plan complexity. Certification and education demographics, experience, and confidence level of the planner were not good predictors of plan quality.
References 1. Deming WE. Out of the Crisis. Cambridge, MA: MIT Press. 1986. 2. Aguayo R. Dr. Deming: The American who taught the Japanese about quality. New York, NY: Simon and Schuster; 1991. 3. Li XA, Tai A, Arthur DW, et al. Variability of target and normal structure delineation for breast cancer radiotherapy: An RTOG
5.
6.
7.
8.
9. 10.
305
multi-institutional and multiobserver Study. Int J Radiat Oncol Biol Phys. 2009;73:944-951. Hurkmans CW, Borger JH, Pieters BR, Russell NS, Jansen EP, Mijnheer BJ. Variability in target volume delineation on CT scans of the breast. Int J Radiat Oncol Biol Phys. 2001;50:1366-1372. Yamamoto M, Nagata Y, Okajima K, et al. Differences in target outline delineation from CT scans of brain tumours using different methods and different observers. Radiother Oncol. 1999;50:151-156. Nelms BE, Tomé WA, G Robinson, Wheeler J. Variations in the contouring of organs at risk: Test case from a patient with oropharyngeal cancer [E-pub ahead of print November 29, 2010]. Int J Radiat Oncol Biol Phys. doi:10.1016/j.ijrobp.2010.10.019. Matsuo Y, Takayama K, Nagata Y, et al. Interinstitutional variations in planning for stereotactic body radiation therapy for lung cancer.. Int J Radiat Oncol Biol Phys. 2007;68:416-425. Wu B, Ricchetti F, Sanguineti G, et al. Patient geometry-driven information retrieval for IMRT treatment plan quality control.. Med Phys. 2009;36:5497-5505. Shewhart WA. 1939. Statistical method from the viewpoint of quality control. Department of Agriculture. Dover; 1986:45. Deming WE. Elementary principles of the statistical control of quality, Union of Japanese Scientists and Engineers (JUSE).. 1950.
