CONGENITAL HEART DISEASE IN CHILDREN AND ADULTS
Application of four complexity stratification tools (Aristotle Basic Score, RACHS-1, STAT Mortality Score, and STAT Mortality Categories) to evaluate early congenital heart surgery outcomes over 16 years at a single institution

Introduction

The most universally used measures of outcome in congenital heart surgery are operative, 30-day, and hospital mortalities [1, 2]. Two large international multicenter registries for congenital heart surgery, the Society of Thoracic Surgery (STS) Congenital Heart Surgery Database and the European Association for Cardio-Thoracic Surgery (EACTS) Congenital Database, facilitate the collection, analysis, and comparison of the early outcomes of congenital cardiac surgery [3, 4]. Both databases use the same nomenclature for diagnosis, procedures, and complications, as well as the same datasets. After collecting a huge amount of procedural data and outcomes, it became obvious that the next step was to develop validated methods of risk adjustment to facilitate the analysis and reporting of outcomes. Reporting raw unadjusted mortality data can be risky, especially for the surgeons and centers that undertake the challenge of treating the most complex cases [5]. At the beginning of the last decade, two groups of researchers independently developed two tools based primarily on the subjective opinion of experts. The Aristotle Basic Complexity Score (ABC Score) consists in the sum of three components (potential for mortality, potential for morbidity, and technical difficulty) and is able to score 98% of coded operations [6-11]. The Risk Adjustment for Congenital Heart Surgery-1 Categories (RACHS-1 Categories) groups procedures into 6 levels of increasing mortality risk. RACHS-1 scores fewer procedures (92%), but appears to be a slightly stronger predictor of early death [12-15]. In 2009, O’Brien and colleagues derived a new mortality score and mortality categories using the combined data of the STS and EACTS databases. In the Society of Thoracic Surgeons - European Association for Cardio-Thoracic Surgery Congenital Heart Surgery Mortality Score (STAT Mortality Score), each procedure was assigned a numeric score by shifting and rescaling the estimated procedure-specific mortality rates to fit the range from 0.1 to 5.0, and then rounding the values to one decimal place. In the STAT Mortality Categories, procedures are sorted by increasing estimated risk of mortality and divided into 5 relatively homogeneous categories. With this data-driven approach, procedures in the same category are similar with respect to their estimated risk of mortality. The STAT Mortality Categories consist of 5 groups from 1 to 5, with higher numbers indicating higher risk of mortality. The STAT Mortality Score and STAT Mortality Categories appear to be the strongest predictors of early death after congenital heart surgery. O’Brien and colleagues showed strong positive associations between the STAT Mortality Score and STAT Mortality Categories and the observed mortality in the validation sample (the C-Index for the STAT Mortality Score was 0.816, while for the STAT Mortality Categories it was 0.812) [5]. Since 2010, the STAT Mortality Score and STAT Mortality Categories have been implemented in both the STS and EACTS Congenital Heart Surgery Databases as new tools for case-mix adjustment [3, 4].

The aim of our retrospective study was not only to analyze the results achieved by our institution, but also to evaluate the ability of each of the four complexity stratification tools to predict mortality when applied to our single-institution data.

Material and methods

The study was approved by the ethical committee of The Children’s Memorial Health Institute. Since the individual patients were not identified, the need for parental consent was waived.

For our retrospective analysis, we used complete and verified data on hospital mortalities that occurred after 8404 operations for congenital heart disease performed in our institution between 1995 and 2010. Patients who were 18 years old and younger were included in the study. For all operations involving combinations of procedures that were not included in the STS-EACTS procedure list, the operation was classified according to the most technically complex procedure, as determined by the difficulty component of the ABC score. Procedures with less than 25 occurrences were excluded.

The ABC score covered the largest number of procedures – 40. The STAT Mortality Score and STAT Mortality Categories covered 37. The number of specific procedures for which the RACHS-1 Categories were defined was 33.

STAT estimated mortality was compared with the mortality observed in our institution. Each of the four tools was examined with regard to the correlation between the mortality observed in our patients and the mortality predicted by each of the scores.



Statistical analysis



Statistical analysis was performed using Statistica for Windows and MedCalc Software. Descriptive statistics were presented as mean values and 95% confidence intervals (95% CI). Each score was modeled as a continuous variable, while hospital mortality was modeled as a binary variable. Univariate and multivariate logistic regression was used to assess the risk factors for hospital mortality in our dataset. The discrimination of the model was assessed by calculating the c-statistic. The ability of the scores to predict the risk of individual procedures was quantified by calculating the Pearson correlation coefficient between the score and the actual calculated procedure-specific mortality rate in our patients. Because of sampling variations, procedures with fewer than 40 occurrences were excluded when calculating the Pearson correlation coefficient.

Results

In our experience between 1995 and 2010, the overall mean hospital mortality was 4.38%, 95% CI (3.9–4.8). After exclusion of procedures with incidence lower than 25, 7493 operations remained for further analysis (89%).

The ABC Score was applied to 89% of operations. The STAT Mortality Score and Categories covered 83.2% of operations. The RACHS-1 Categories covered 74.4%.

The frequency of in-hospital mortality for the different procedures ranged from 0 to 33.3%; there were 8 procedures with no deaths. The highest mortality occurred after Norwood operations (33.3%). The comparison of the mortality rates observed in our institution and the mortality rates estimated by the STAT risk model is presented in Table I.

Examples of the regression models of the scores and categories are summarized in Table II.

The STAT Mortality Score had the highest discrimination for predicting mortality (C-index = 0.768). The STAT Mortality Categories, RACHS-1, and ABC Score were weaker predictors of death (C-index: 0.750, 0.765, and 0.746 respectively).

To assess which of the tools discriminate mortality better, each of them was evaluated in the sample using the subset of procedures to which all four tools were applicable. This sample contained 6534 operations. As summarized in Table III, discrimination was highest for the STAT Mortality Score (C-index = 0.777), followed by the STAT Mortality Categories (C-index = 0.772), RACHS-1 Categories (C-index = 0.765), and ABC Score (C-index = 0.721), all differences – p < 0.0001.

Receiver operating characteristic curves for the four tools are displayed in Fig. 1.

The pairwise comparisons of ROC curves between the four tools are summarized in Table IV. Only the difference between the areas of the STAT Mortality Score and the ABC Score was significant, p < 0.0001.

With regard to procedures with at least 40 occurrences, the Pearson correlation coefficient between the scores of a procedure and its actual observed mortality rate was highest for the STAT Mortality Score: r = 0.8379 (Fig. 2).

Comments

The evaluation of the predictability of operative mortality provided by each of the complexity stratification tools was one of the two major aims of our study. The strength of our study is that it was performed using complete and validated data.

This study began in 1995, which was before many of the major improvements of outcomes of the most complex operations for congenital heart disease took place. Nevertheless, in-hospital mortality in our cohort of patients is similar to that reported by O’Brien and colleagues (4.38% vs. 4.3%). In some complex procedures, the mortality in our institution was higher. The cause of the higher mortality in patients undergoing these more complex operations could be related to both the early study time frame and the small sample size of several individual procedures.

All four stratification tools have good discrimination for predicting mortality. This discrimination is higher when the sample of procedures is limited to those scored by all tools. The tool with the highest discrimination is the STAT Mortality Score followed by the STAT Mortality Categories, RACHS-1 Categories, and ABC Score. The STAT Mortality Score has the highest correlation with the observed operative mortality in our study. This finding is consistent with the data reported by O’Brien and colleagues [5]. The differences in discrimination in the four stratification tools are small, and only the difference between the STAT Mortality Score and the ABC Score is statistically significant.

Conclusions

In this single-institution analysis, the STAT Mortality Score had the strongest correlation with the observed mortality prior to discharge from the hospital. This study demonstrates a strategy for the application of complexity stratification tools, based on multi-institutional data, to single-institution results.

References

1. Jacobs JP, Benavidez OJ, Bacha EA, Walters HL, Jacobs ML. The nomenclature of safety and quality of care for patients with congenital cardiac diseases: a report of Society of Thoracic Surgeons Congenital Database Taskforce Subcommittee on Patient Safety. Cardiol Young 2008; 18 suppl. 2: 81-91.

2. Maruszewski B, Lacour-Gayet FG, Monro JL, Keogh B, Tobota Z, Kansy A. An attempt at data verification in the EACTS Congenital Database. Eur J Cardiothorac Surg 2005; 28: 400-404.

3. STS Congenital Heart Surgery Database. Available at: www.sts.org.

4. EACTS Congenital Database. Available at: www. eactscongenitaldb.org.

5. O’Brien SM, Clarke DR, Jacobs JP, Jacobs ML, Lacour-Gayet FG, Pizarro C, Welke KF, Maruszewski B, Tobota Z, Miller WJ, Hamilton L, Peterson ED, Mavroudis C, Edwards FH. An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg 2009; 138: 1139-1153.

6. Lacour-Gayet F, Clarke D, Jacobs J, Comas J, Daebritz S, Daenen W, Gaynor W, Hamilton L, Jacobs M, Maruszsewski B, Pozzi M, Spray T, Stellin G, Tchervenkov C, Mavroudis And C; Aristotle Committee. The Aristotle Score: a complexity – adjusted method to evaluate surgical results. Eur J Cariothorac Surg 2004; 25: 911-924.

7. Lacour-Gayet F, Clarke D, Jacobs J, Gaynor W, Hamilton L, Jacobs M, Maruszewski B, Pozzi M, Spray T, Tchervenkov C, Mavroudis C; Aristotle Committee. The Aristotle score for congenital heart surgery. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu 2004; 7: 185-191.

8. Maruszewski B, Tobota Z, Kansy A, Lacour-Gayet F, Jacobs JP, Clarke D, Elliott MJ. The Aristotle Score methogology for evaluation of outcomes in Congenital Heart Surgery. Standardy Med Pediatria 2005; 7 supl. 22: 29-33.

9. Kansy A, Maruszewski B, Burczyński P, Pastuszko A, Birbach M, Lipiński W, Mirkowicz-Małek M, Mozol K, Kozłowski M, Manowska M, Tobota Z. Analiza wczesnych wyników 2542 operacji wykonanych w referencyjnym ośrodku kardiochirurgii dziecięcej w latach 2001–2005. Kardiochir Torakochir Pol 2006; 4: 372-376.

10. Kansy A, Tobota Z, Burczyński P, Birbach M, Maruszewski P, Maruszewski B. Analiza wyników leczenia wad wrodzonych serca u noworodków z niską masą ciała na podstawie danych EACTS. Przegl Chir Dziec 2008; 3: 81-86.

11. Kansy A, Tobota Z, Maruszewski P, Maruszewski B. Analysis of 14843 neonatal congenital heart surgical procedures in the European Association for Cardiothoracic Surgery Congenital Database. Ann Thorac Surg 2010; 89: 1255-1259.

12. Jenkins KJ. Risk adjustment for congenital heart surgery: the RACHS-1 method. Semi Thorac Cardiovasc Surg Pediatr Card Surg Annu 2004; 7: 180-184.

13. Jenkins KJ, Gauvreau K. Center-specific differences in mortality: preliminary analyses using the Risk Adjustment in Congenital Heart Surgery (RACHS-1) method. J Thorac Cardiovasc Surg 2002; 124: 97-104.

14. Al-Radi OO, Harrell FE Jr, Caldarone CA, McCrindle BW, Jacobs JP, Williams MG, Van Arsdell GS, Williams WG. Case complexity scores in congenital heart surgery: a comparative study of the Aristotle Basic Complexity score and the Risk Adjustment in Congenital Heart Surgery (RACHS-1) system. J Thorac Cardiovasc Surg 2007; 133: 865-875.

15. Kang N, Tsang VT, Elliott MJ, de Leval MR, Cole TJ. Dose the Aristotle score predict outcome in congenital heart surgery? Eur J Cardiothorac Surg 2006; 29: 986-988.