526981
research-article2014
CPJXXX10.1177/0009922814526981Clinical PediatricsFalcone
Article
City Population Size Is Associated with Examinee Outcomes on the American Board of Pediatrics Certifying Examination
Clinical Pediatrics 2014, Vol. 53(8) 751–757 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0009922814526981 cpj.sagepub.com
John L. Falcone, MD, MS1
Abstract Objective. The purpose of this article is to evaluate American Board of Pediatrics Certifying Examination performance based on city population, hypothesizing that larger cities are associated with higher exam performance. Methods. Pediatric residency programs were retrospectively evaluated from 2010 to 2012. The associated city populations were found using census data. Simple linear regression was performed for continuous variables, and χ2 tests were performed for categorical variables, using α = .05. Results. For 188 programs, the median pass rate was 79.5% (interquartile range = 69% to 90%). Regression showed that city size explains program size (P < .001). Examinees from larger cities had a higher pass rates than examinees from smaller cities, and examinees who are in cities that share residency programs had a higher pass rate than examinees from solo city programs (both P < .05). Conclusions. City size is associated with examinee performance on the American Board of Pediatrics Certifying Examination. City size should be a consideration of future residency applicants. Keywords certification, educational measurement, geographic factors, pediatrics, specialty boards
Introduction While there are a few individual predictors of success on the American Board of Pediatrics (ABP) Certifying Examination (CE), there are only a few program demographics that are associated with higher examination pass rates.1-3 It is clear that there is a positive association of program size and program pass rates on the ABP CE.1 Moreover, larger programs are more likely than smaller programs to meet the Accreditation Council for Graduate Medical Education minimum program pass rate standards.1-3 Now, while program size is associated with ABP CE performance, program size is likely a confounding variable for other program attributes. The strength of the curriculum, the quality of the faculty, the faculty recruitment process, incentives for resident examinees, and the ability to select and draw more knowledgeable residents in the resident selection process are such variables that are potentially confounded by residency program size.1,2,4 As a direct corollary, it has been suggested in Internal Medicine that programs with higher certifying examination performance are able to attract more knowledgeable residents into the training program.5 Moreover,
some program characteristics affect examination performance over and above the resident baseline test scores. These changes have been associated with the ability to provide an extensive educational experience involving clinical exposure.5 At the program level, there is also clear regional variation in performance. It is evident that examinees from residency training programs in the Western United States do better than examinees from training programs in the Northeast, Midwest, and South.1 The reasons behind this find are also likely multifactorial and are unknown. One programmatic variable yet to be studied in Pediatrics with its relation to ABP CE performance is the size of the residency city. One would surmise that training in a smaller city may not give the breadth or 1
Owensboro Health, Owensboro, KY, USA
Corresponding Author: John L. Falcone, Owensboro Health Surgical Specialists, Ridgecrest Medical Park, 2801 New Hartford Road, Owensboro, KY 42303, USA. Email: [email protected]
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
752
Clinical Pediatrics 53(8)
depth that a large city clinical experience might offer due to the literal size of the population. Larger cities might also have the ability to have larger residency programs. The purpose of this study is to further evaluate the nature of population base and ABP CE performance. The hypothesis is that programs and examinees related to larger cities and populations would have higher performance on the ABP CE than programs located in smaller cities.
this two-group comparison. To evaluate examinee outcomes and city demographics, χ2 tests were performed across the same groups. In this study, there were nonnormal distributions of the data. So, nonparametric statistics were performed for the analyses using Stata 13 statistical software (StataCorp, College Station, TX). All statistical tests were performed utilizing α =.05.
Results
Methods In this retrospective study from 2010 to 2012, residency program city and ABP CE performance were obtained from the ABP Web site for all the foreign and domestic residency training programs.4 Programs from Canada and Puerto Rico were excluded from the analyses. To determine the associated program city size, the 2010 population census data were obtained from the US Census Bureau.6 Programs with cities not found in the database were also excluded from the analyses. Any program with both native city census data and at least one examinee that attempted the ABP CE from 2010 to 2012 were included. Double data entry was performed to ensure accurate data extrapolation. Permission to use electronically published data for education research had been previously confirmed in the past with a personal communication with the psychometrician for the ABP. To evaluate the relationship between city size and program pass rates, a scatterplot was performed, and simple linear regression was performed using the city population size as the independent variable and the 3-year first-attempt program pass rate as the dependent variable. This was performed with raw population numbers and on a log scale due to the distribution of city population data, and the exponential relationship of city area and population. To control for city size, the city population to city eligible resident ratio was determined. This is analogous to examinee density. Simple linear regression was performed for this metric. Regression analysis was also performed using the number of eligible examinees to see if larger cities have more pediatric residents. Spearman’s rank correlations were also determined. To further evaluate the relationship between city size and residency program performance, residency programs were sorted by relation to other programs. Programs were first sorted by tertile according to city size. A Kruskal–Wallis test was then performed to evaluate differences in program pass rates by city size tertile. Programs were also sorted based on if they were a solo program in the city or if there were more than one program in the city. Mann–Whitey U tests were used for
There were 188/210 (89.5%) residency training programs that satisfied the inclusion criteria for this study. Of the 22 excluded residency training programs, 17 (77%) were located in Canada, 3 (14%) were located in Puerto Rico, and 2 (9%) were Army Medical Centers without associated city population census data. Of these 188 programs, 188 (100%) had ABP CE data for the study period. The median pass rate on the ABP CE per program during the study period was 79.5% (interquartile range [IQR] = 69% to 90%). The median number of examinees per program during the study period was 37 (IQR = 25 to 59.25). The median number of eligible examinees per program during the study period was 39.5 (IQR = 28 to 64.5). There were 8362 examinees that attempted the ABP CE during the study period, with an overall pass rate of 81.8% (6842/8362). The median residency city population size was 312 499 (IQR = 112 575 to 766 576). The linear regression analyses are shown in Figures 1 to 3. In Figure 1, there were no significant positive or negative linear slopes between residency program pass rates and city population (P = .31) or log city population (P = .08) (both ρ = 0.13 [P = .08]). In Figure 2, there were no significant positive or negative linear slopes between residency program pass rates and city population/eligible resident ratio (P = .25) or log city population/eligible resident ratio (P = .95) (both ρ = 0.02 [P = .83]). Additionally, while the linear regression of the number of eligible examinees did not have a positive or negative slope when considering city population (P = .10), there was a positive slope when considering the log city population (P < .001), as shown in Figure 3 (both ρ = 0.26 [P < .001]). In comparative analyses, a Kruskal–Wallis test of program pass rate per tertile of city population showed no differences in the median program pass rates between cities in the first tertile, 75% (IQR = 68% to 88%); second tertile, 81% (IQR = 71% to 90%); or third tertile, 84.5% (IQR = 68.5% to 91%) (P = .16). There were 119 (63%) residency training programs that were solo programs in the associated city, and there were 69 (37%)
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
753
Falcone
A
100 90
Residency Program Pass Rate (%)
80 70
y = 3E-07x + 78.18 R² = 0.001
60 50 40 30 20 10 0 0
2,000,000
4,000,000
6,000,000
8,000,000
City Popula on (n)
B 100 90
Residency Program Pass Rate (%)
80 70 60 50 40 y = 0.910x + 66.99 R² = 0.010
30 20 10 0 5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Log (City Popula on)
Figure 1. Simple linear regression of residency program pass rate on the American Board of Pediatrics Certifying Examination by city population (A) and log city population (B).
residency training programs that were located in a city with at least one other residency training program. A Mann–Whitney U test showed no differences in the median program pass rate for solo programs, 78% (IQR = 70% to 89%), and for programs that shared cities with
at least one other program, 83% (IQR = 68% to 91%) (P = .43). The χ2 examinee outcomes by associated residency training demographics are shown in Table 1. Examinees from larger cities had a higher pass rate
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
754
Clinical Pediatrics 53(8)
A
100 90
y = -0.000x + 79.44 R² = 0.007
Residency Program Pass Rate (%)
80 70 60 50 40 30 20 10 0 0
B
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
11
12
City Popula on / Eligible Resident Ra o 100 90
Residency Program Pass Rate (%)
80 70 60 50
y = -0.059x + 78.96 R² = 2E-05
40 30 20 10 0 4
5
6
7
8
9
10
Log (City Popula on / Eligible Resident Ra o)
Figure 2. Simple linear regression of residency program pass rate on the American Board of Pediatrics Certifying Examination by city population/resident ratio (A) and log city population/resident ratio (B).
than examinees from smaller cities, and examinees that are in cities that share residency programs had a higher pass rate than examinees from solo city programs (both P < .05).
Discussion In this study, residency program and individual examinee performance on the ABP CE were evaluated with
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
755
Falcone
A
160
Number of Eligible Residents per Program (n)
140 120 100 80
y = 2E-06x + 46.09 R² = 0.009
60 40 20 0 0
2,000,000
4,000,000
6,000,000
8,000,000
City Populaon (n)
B
160
Number of Eligible Residents per Program (n)
140 120 100
y = 4.102x-4.149 R² = 0.057
80 60 40 20 0 5
7
9
11
13
15
17
19
Log (City Populaon)
Figure 3. Simple linear regression of eligible residency examinees for the American Board of Pediatrics Certifying Examination by city population (A) and log city population (B).
regard to the residency program native city population size. The hypothesis was that larger cities would be associated with higher pass rates both due to the depth of clinical exposure and that these programs would be
more likely to be larger programs. There are numerous findings in this study that support the study hypothesis. To support a basis behind the hypothesis, the number of eligible examinees per log city population size has a
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
756
Clinical Pediatrics 53(8)
Table 1. Performance of Examinees (N = 8362) on the American Board of Pediatrics Certifying Examination Based on Residency Program Demographics. Residency Program Characteristics
Passing Examinees, n (%)
Failing Examinees, n (%)
P Valuea
City size in first tertile City size in second tertile City size in third tertile
1691 (79%) 2327 (82%) 2824 (84%)
461 (21%) 508 (18%) 551 (16%)
research-article2014
CPJXXX10.1177/0009922814526981Clinical PediatricsFalcone
Article
City Population Size Is Associated with Examinee Outcomes on the American Board of Pediatrics Certifying Examination
Clinical Pediatrics 2014, Vol. 53(8) 751–757 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0009922814526981 cpj.sagepub.com
John L. Falcone, MD, MS1
Abstract Objective. The purpose of this article is to evaluate American Board of Pediatrics Certifying Examination performance based on city population, hypothesizing that larger cities are associated with higher exam performance. Methods. Pediatric residency programs were retrospectively evaluated from 2010 to 2012. The associated city populations were found using census data. Simple linear regression was performed for continuous variables, and χ2 tests were performed for categorical variables, using α = .05. Results. For 188 programs, the median pass rate was 79.5% (interquartile range = 69% to 90%). Regression showed that city size explains program size (P < .001). Examinees from larger cities had a higher pass rates than examinees from smaller cities, and examinees who are in cities that share residency programs had a higher pass rate than examinees from solo city programs (both P < .05). Conclusions. City size is associated with examinee performance on the American Board of Pediatrics Certifying Examination. City size should be a consideration of future residency applicants. Keywords certification, educational measurement, geographic factors, pediatrics, specialty boards
Introduction While there are a few individual predictors of success on the American Board of Pediatrics (ABP) Certifying Examination (CE), there are only a few program demographics that are associated with higher examination pass rates.1-3 It is clear that there is a positive association of program size and program pass rates on the ABP CE.1 Moreover, larger programs are more likely than smaller programs to meet the Accreditation Council for Graduate Medical Education minimum program pass rate standards.1-3 Now, while program size is associated with ABP CE performance, program size is likely a confounding variable for other program attributes. The strength of the curriculum, the quality of the faculty, the faculty recruitment process, incentives for resident examinees, and the ability to select and draw more knowledgeable residents in the resident selection process are such variables that are potentially confounded by residency program size.1,2,4 As a direct corollary, it has been suggested in Internal Medicine that programs with higher certifying examination performance are able to attract more knowledgeable residents into the training program.5 Moreover,
some program characteristics affect examination performance over and above the resident baseline test scores. These changes have been associated with the ability to provide an extensive educational experience involving clinical exposure.5 At the program level, there is also clear regional variation in performance. It is evident that examinees from residency training programs in the Western United States do better than examinees from training programs in the Northeast, Midwest, and South.1 The reasons behind this find are also likely multifactorial and are unknown. One programmatic variable yet to be studied in Pediatrics with its relation to ABP CE performance is the size of the residency city. One would surmise that training in a smaller city may not give the breadth or 1
Owensboro Health, Owensboro, KY, USA
Corresponding Author: John L. Falcone, Owensboro Health Surgical Specialists, Ridgecrest Medical Park, 2801 New Hartford Road, Owensboro, KY 42303, USA. Email: [email protected]
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
752
Clinical Pediatrics 53(8)
depth that a large city clinical experience might offer due to the literal size of the population. Larger cities might also have the ability to have larger residency programs. The purpose of this study is to further evaluate the nature of population base and ABP CE performance. The hypothesis is that programs and examinees related to larger cities and populations would have higher performance on the ABP CE than programs located in smaller cities.
this two-group comparison. To evaluate examinee outcomes and city demographics, χ2 tests were performed across the same groups. In this study, there were nonnormal distributions of the data. So, nonparametric statistics were performed for the analyses using Stata 13 statistical software (StataCorp, College Station, TX). All statistical tests were performed utilizing α =.05.
Results
Methods In this retrospective study from 2010 to 2012, residency program city and ABP CE performance were obtained from the ABP Web site for all the foreign and domestic residency training programs.4 Programs from Canada and Puerto Rico were excluded from the analyses. To determine the associated program city size, the 2010 population census data were obtained from the US Census Bureau.6 Programs with cities not found in the database were also excluded from the analyses. Any program with both native city census data and at least one examinee that attempted the ABP CE from 2010 to 2012 were included. Double data entry was performed to ensure accurate data extrapolation. Permission to use electronically published data for education research had been previously confirmed in the past with a personal communication with the psychometrician for the ABP. To evaluate the relationship between city size and program pass rates, a scatterplot was performed, and simple linear regression was performed using the city population size as the independent variable and the 3-year first-attempt program pass rate as the dependent variable. This was performed with raw population numbers and on a log scale due to the distribution of city population data, and the exponential relationship of city area and population. To control for city size, the city population to city eligible resident ratio was determined. This is analogous to examinee density. Simple linear regression was performed for this metric. Regression analysis was also performed using the number of eligible examinees to see if larger cities have more pediatric residents. Spearman’s rank correlations were also determined. To further evaluate the relationship between city size and residency program performance, residency programs were sorted by relation to other programs. Programs were first sorted by tertile according to city size. A Kruskal–Wallis test was then performed to evaluate differences in program pass rates by city size tertile. Programs were also sorted based on if they were a solo program in the city or if there were more than one program in the city. Mann–Whitey U tests were used for
There were 188/210 (89.5%) residency training programs that satisfied the inclusion criteria for this study. Of the 22 excluded residency training programs, 17 (77%) were located in Canada, 3 (14%) were located in Puerto Rico, and 2 (9%) were Army Medical Centers without associated city population census data. Of these 188 programs, 188 (100%) had ABP CE data for the study period. The median pass rate on the ABP CE per program during the study period was 79.5% (interquartile range [IQR] = 69% to 90%). The median number of examinees per program during the study period was 37 (IQR = 25 to 59.25). The median number of eligible examinees per program during the study period was 39.5 (IQR = 28 to 64.5). There were 8362 examinees that attempted the ABP CE during the study period, with an overall pass rate of 81.8% (6842/8362). The median residency city population size was 312 499 (IQR = 112 575 to 766 576). The linear regression analyses are shown in Figures 1 to 3. In Figure 1, there were no significant positive or negative linear slopes between residency program pass rates and city population (P = .31) or log city population (P = .08) (both ρ = 0.13 [P = .08]). In Figure 2, there were no significant positive or negative linear slopes between residency program pass rates and city population/eligible resident ratio (P = .25) or log city population/eligible resident ratio (P = .95) (both ρ = 0.02 [P = .83]). Additionally, while the linear regression of the number of eligible examinees did not have a positive or negative slope when considering city population (P = .10), there was a positive slope when considering the log city population (P < .001), as shown in Figure 3 (both ρ = 0.26 [P < .001]). In comparative analyses, a Kruskal–Wallis test of program pass rate per tertile of city population showed no differences in the median program pass rates between cities in the first tertile, 75% (IQR = 68% to 88%); second tertile, 81% (IQR = 71% to 90%); or third tertile, 84.5% (IQR = 68.5% to 91%) (P = .16). There were 119 (63%) residency training programs that were solo programs in the associated city, and there were 69 (37%)
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
753
Falcone
A
100 90
Residency Program Pass Rate (%)
80 70
y = 3E-07x + 78.18 R² = 0.001
60 50 40 30 20 10 0 0
2,000,000
4,000,000
6,000,000
8,000,000
City Popula on (n)
B 100 90
Residency Program Pass Rate (%)
80 70 60 50 40 y = 0.910x + 66.99 R² = 0.010
30 20 10 0 5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Log (City Popula on)
Figure 1. Simple linear regression of residency program pass rate on the American Board of Pediatrics Certifying Examination by city population (A) and log city population (B).
residency training programs that were located in a city with at least one other residency training program. A Mann–Whitney U test showed no differences in the median program pass rate for solo programs, 78% (IQR = 70% to 89%), and for programs that shared cities with
at least one other program, 83% (IQR = 68% to 91%) (P = .43). The χ2 examinee outcomes by associated residency training demographics are shown in Table 1. Examinees from larger cities had a higher pass rate
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
754
Clinical Pediatrics 53(8)
A
100 90
y = -0.000x + 79.44 R² = 0.007
Residency Program Pass Rate (%)
80 70 60 50 40 30 20 10 0 0
B
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
11
12
City Popula on / Eligible Resident Ra o 100 90
Residency Program Pass Rate (%)
80 70 60 50
y = -0.059x + 78.96 R² = 2E-05
40 30 20 10 0 4
5
6
7
8
9
10
Log (City Popula on / Eligible Resident Ra o)
Figure 2. Simple linear regression of residency program pass rate on the American Board of Pediatrics Certifying Examination by city population/resident ratio (A) and log city population/resident ratio (B).
than examinees from smaller cities, and examinees that are in cities that share residency programs had a higher pass rate than examinees from solo city programs (both P < .05).
Discussion In this study, residency program and individual examinee performance on the ABP CE were evaluated with
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
755
Falcone
A
160
Number of Eligible Residents per Program (n)
140 120 100 80
y = 2E-06x + 46.09 R² = 0.009
60 40 20 0 0
2,000,000
4,000,000
6,000,000
8,000,000
City Populaon (n)
B
160
Number of Eligible Residents per Program (n)
140 120 100
y = 4.102x-4.149 R² = 0.057
80 60 40 20 0 5
7
9
11
13
15
17
19
Log (City Populaon)
Figure 3. Simple linear regression of eligible residency examinees for the American Board of Pediatrics Certifying Examination by city population (A) and log city population (B).
regard to the residency program native city population size. The hypothesis was that larger cities would be associated with higher pass rates both due to the depth of clinical exposure and that these programs would be
more likely to be larger programs. There are numerous findings in this study that support the study hypothesis. To support a basis behind the hypothesis, the number of eligible examinees per log city population size has a
Downloaded from cpj.sagepub.com at Bobst Library, New York University on June 5, 2015
756
Clinical Pediatrics 53(8)
Table 1. Performance of Examinees (N = 8362) on the American Board of Pediatrics Certifying Examination Based on Residency Program Demographics. Residency Program Characteristics
Passing Examinees, n (%)
Failing Examinees, n (%)
P Valuea
City size in first tertile City size in second tertile City size in third tertile
1691 (79%) 2327 (82%) 2824 (84%)
461 (21%) 508 (18%) 551 (16%)
