NEWS & VIEWS TRANSPLANTATION
histological information to determine the clinical phenotype used to validate the assay. In addition to the lack of prospectively collected and precisely phenotyped samples of subclinical AR obtained using protocol biopsy, another reason for the use of this new algorithm seems to be that the number of proposed genes has expanded to 17 from the 5–gene set previously reported for a 2–way classifier to assess AR.8 The new algorithm is further complicated by the fact that rather than using all 17 genes, the new algorithm uses varying combinations of smaller subsets (as few as three) of these genes. Much to our chagrin, the information contained in the comprehensive supplement to the manuscript is inadequate to assess and validate the researchers’ computational strategy, which is specifically a form of model fitting that trains multiple models to determine the ‘best fit’. Use of this approach raises the potential concern that the model will be self-fulfilling since the output has no constraints, and as such will eventually find a combination of genes that will correlate to the end point. Accordingly, kSAS takes a 17 gene model, tests it, and if it finds that it does not correlate well, tries 16 genes, then 15 genes, and so on until it finds a good correlation to best-fit the data to a known clinical phenotype. This strategy is inconsistent with the industry standard to create a laboratory developed test or standardized clinical kit, which demands strict workflows and quality controls whereby any algorithm that uses a ‘best-fit’ model has to be locked down before being tested on independent samples. Furthermore, the cut-off or threshold for a correlation is not defined, even though this is a key element of the
Biomarkers in transplantation —the devil is in the detail Michael Abecassis and Bruce Kaplan Refers to: Roedder, S. et al. The kSORT assay to detect renal transplant patients at high risk for acute rejection: results of the multicenter AART study. PLoS Med. 11, e1001759 (2014)
Numerous studies have suggested the utility of non-invasive molecular biomarkers to monitor recipients of kidney transplants. A new correlationbased algorithm using stepwise analysis of gene expression data from peripheral blood samples, claiming to detect subclinical, and predict clinical acute allograft rejection, requires corroboration before testing in prospective validation studies.
‘‘
The authors…candidly admit a number of limitations of the study design
’’
validation of the kSORT assay to identify patients at high risk of acute transplant rejection,7 but candidly admit a number of limitations of the study design. These limitations include the heterogeneity of study samples, which were aggregated from a number of small cohorts of adult and paediatric recipients and collected using various protocols (367 cross-sectional samples from 338 patients at eight centres, and 191 serially collected samples from 98 patients [
histological information to determine the clinical phenotype used to validate the assay. In addition to the lack of prospectively collected and precisely phenotyped samples of subclinical AR obtained using protocol biopsy, another reason for the use of this new algorithm seems to be that the number of proposed genes has expanded to 17 from the 5–gene set previously reported for a 2–way classifier to assess AR.8 The new algorithm is further complicated by the fact that rather than using all 17 genes, the new algorithm uses varying combinations of smaller subsets (as few as three) of these genes. Much to our chagrin, the information contained in the comprehensive supplement to the manuscript is inadequate to assess and validate the researchers’ computational strategy, which is specifically a form of model fitting that trains multiple models to determine the ‘best fit’. Use of this approach raises the potential concern that the model will be self-fulfilling since the output has no constraints, and as such will eventually find a combination of genes that will correlate to the end point. Accordingly, kSAS takes a 17 gene model, tests it, and if it finds that it does not correlate well, tries 16 genes, then 15 genes, and so on until it finds a good correlation to best-fit the data to a known clinical phenotype. This strategy is inconsistent with the industry standard to create a laboratory developed test or standardized clinical kit, which demands strict workflows and quality controls whereby any algorithm that uses a ‘best-fit’ model has to be locked down before being tested on independent samples. Furthermore, the cut-off or threshold for a correlation is not defined, even though this is a key element of the
Biomarkers in transplantation —the devil is in the detail Michael Abecassis and Bruce Kaplan Refers to: Roedder, S. et al. The kSORT assay to detect renal transplant patients at high risk for acute rejection: results of the multicenter AART study. PLoS Med. 11, e1001759 (2014)
Numerous studies have suggested the utility of non-invasive molecular biomarkers to monitor recipients of kidney transplants. A new correlationbased algorithm using stepwise analysis of gene expression data from peripheral blood samples, claiming to detect subclinical, and predict clinical acute allograft rejection, requires corroboration before testing in prospective validation studies.
‘‘
The authors…candidly admit a number of limitations of the study design
’’
validation of the kSORT assay to identify patients at high risk of acute transplant rejection,7 but candidly admit a number of limitations of the study design. These limitations include the heterogeneity of study samples, which were aggregated from a number of small cohorts of adult and paediatric recipients and collected using various protocols (367 cross-sectional samples from 338 patients at eight centres, and 191 serially collected samples from 98 patients [
