Predicting 3-month poor functional outcomes of acute ischemic stroke in young patients using machine learning

Abstract

Background

Prediction of short-term outcomes in young patients with acute ischemic stroke (AIS) may assist in making therapy decisions. Machine learning (ML) is increasingly used in healthcare due to its high accuracy. This study aims to use a ML-based predictive model for poor 3-month functional outcomes in young AIS patients and to compare the predictive performance of ML models with the logistic regression model.

Methods

We enrolled AIS patients aged between 18 and 50 years from the Third Chinese National Stroke Registry (CNSR-III), collected between 2015 and 2018. A modified Rankin Scale (mRS) ≥ 3 was a poor functional outcome at 3 months. Four ML tree models were developed: The extreme Gradient Boosting (XGBoost), Light Gradient Boosted Machine (lightGBM), Random Forest (RF), and The Gradient Boosting Decision Trees (GBDT), compared with logistic regression. We assess the model performance based on both discrimination and calibration.

Results

A total of 2268 young patients with a mean age of 44.3 ± 5.5 years were included. Among them, (9%) had poor functional outcomes. The mRS at admission, living alone conditions, and high National Institutes of Health Stroke Scale (NIHSS) at discharge remained independent predictors of poor 3-month outcomes. The best AUC in the test group was XGBoost (AUC = 0.801), followed by GBDT, RF, and lightGBM (AUCs of 0.795, 0, 794, and 0.792, respectively). The XGBoost, RF, and lightGBM models were significantly better than logistic regression (P < 0.05).

Conclusions

ML outperformed logistic regression, where XGBoost the boost was the best model for predicting poor functional outcomes in young AIS patients. It is important to consider living alone conditions with high severity scores to improve stroke prognosis.

Background

Stroke is the second leading cause of death worldwide and the leading cause of death and disability in China [1, 2]. Acute ischemic stroke (AIS) or transient ischemic attack (TIA) accounts for approximately 80% of all strokes. Stroke was once a disease of the elderly, although there is no universally agreed-upon definition for young adult patients, most research specifies this age group as individuals between 18 and 50 years, which is the definition used in our study [3, 4]. Recent studies suggest that 10% to 15% of all strokes occur in young adults between the ages of 18 and 50, resulting in approximately 2 million young adults worldwide having a stroke each year, with the incidence increasing over the past decade [5, 6]. Severe functional outcomes affect about 20–25% of stroke patients [7]. As certainty, the degree of disability/dependence after a stroke was measured using the modified Rankin Scale (mRS), which ranges from 0 to 6, with an mRS of 6 indicating death. Young patients with poor functional outcomes have a significant impact on health due to high medical costs and reduced work productivity [8]. Therefore, accurate prediction of functional outcomes after stroke will facilitate post-stroke management and improve the distribution of healthcare services.

In recent years, machine learning (ML) techniques have been increasingly used to solve a variety of research problems, including diagnostic and clinical research, such as stroke [9, 10]. ML is a subfield of artificial intelligence in which a computer extracts past information and uses it to predict new information. It can self-optimize by learning complex systems containing many variables and data [11]. Various algorithms have been used in previous studies, such as logistic regression, random forest classifier (RF) [12], support vector machine (SVM) [13], fully connected deep neural network (DNN) [9], Catboost [14] and extreme gradient boosting (XGBoost) [15] have been used to predict poor functional outcomes in general patients with AIS. However, the identifying factors that predict disability in young patients (under 50 years) are unclear.

In this study, four ML models (XGBoost, Light Gradient Boosting Machine (lightGBM), RF, and Gradient Boosting Decision Trees (GBDT)) were developed and compared with logistic regression to predict poor 3-month functional outcomes in young AIS patients. ML models are increasingly utilized in healthcare due to their ability to handle complex, non-linear relationships within the data and their potential for high accuracy [16]. XGBoost, a powerful and popular gradient-boosting algorithm, was chosen for its ability to handle missing data, scalability, and high predictive performance [17]. LightGBM, another gradient-boosting framework, was selected for its efficiency and speed in handling large datasets [18]. The RF, known for its robustness and ability to handle noisy data [19], was included for comparison. The GBDT model, which is similar to XGBoost but with some differences in the underlying algorithm [20], was also employed. We select logistic regression as a benchmark to illustrate the strengths and weaknesses of the more complex models utilized in our study, thereby providing deeper insights into their comparative advantages. The rationale for choosing these models in our study was based on their proven effectiveness in handling complex relationships, capturing interactions within the data, and providing accurate predictions. Each model was selected for its specific strengths, such as computational efficiency, scalability, robustness against overfitting, and the ability to capture complex patterns in the data. By comparing these models with logistic regression and evaluating discrimination based on the area under the receiver operating characteristic curve (AUC), our study aimed to determine the most effective approach for predicting poor functional outcomes in young AIS patients using our third China National Stroke Registry (CNSR-III).

Patients and methods

Study population

The CNSR-III is a large-scale prospective registry of acute ischemic cerebrovascular events in China, enrolled patients with AIS or TIA between August 2015 and March 2018. CNSR-III encompasses 201 sites distributed across 22 provinces and four municipalities across China. Specifically, 163 central teaching hospitals and 38 urban hospitals were selected based on their comprehensive assessment, adequate research personnel, relevant experience, and qualified equipment. The study design and methods of CNSRIII have been previously published [21]. We include young patients aged between 18 and 50 in this study. Asymptomatic patients with cerebral infarction who had no signs or symptoms or refused to participate in the registry were excluded.

Predictors and data processing

This study presented a comprehensive list of the 55 variables summarized in Supplementary Table 1, including demographic characteristics (sex, age, BMI, living condition, marital status, education level), thrombolytic therapy (alteplase), history of smoking, history of alcohol consumption, medical history (hypertension, diabetes, dyslipidemia, migraine, stroke, TIA, heart disease, arterial fibrillation), family history (hypertension, diabetes, dyslipidemia, stroke, heart disease, cancer), laboratory data, neurological severity such as the National Institutes of Health Stroke Scale (NIHSS) at admission and discharge, mRS score, the different etiology classified according to the trial of ORG 10172 in acute stroke treatment (TOAST), and secondary prevention treatment at discharge. We had 24 variables with missing values, and the rates were below 5%. We used linear interpolation to impute missing values for continuous variables and mode imputation for categorical variables. We randomly divided the total dataset into a training set and a test set in an 80:20 ratio. Feature selection, parameter tuning, and model training were performed on the training set, while validation was conducted on the test set. We utilized the Sequential Forward Selection (SFS) combined with the K Nearest Neighbors (KNN) technique for feature selection.

Patient follow-up and outcome evaluation

Patients were followed up with a face-to-face interview at 3 months. Clinical data were collected using an electronic data capture system by trained research coordinators based on a standardized interview protocol. The clinical outcome of this study was poor functional outcome, defined as an mRS ≥ 3 within 3 months confirmed by the treating hospital after AIS onset.

Model algorithms

We use ML models as follows to predict poor functional outcomes after 3 months in young patients:

• The XGBoost: is a scalable ML system for trees supported by Chen Tianqi of Washington College in 2016. The system runs more than ten times faster than existing popular solutions on a single machine and scales to millions of instances in distributed or memory-based environments [17].

• The GBTD: is a regression tree created using the gradient boosting method. It uses the gradient descent method and the function before loss is the squared error [20].

• The RF: is a combination of various trees identified by Leo Breiman et al. in 2001. It allows variables to be selected in the design, making it more robust to overcome the overfitting problem in the estimation [22].

• The LightGBM: It is an open-source library developed by Microsoft. It extends the gradient boosting algorithm with automatic selection and focuses on boosting samples with large gradients [18].

• Logistic regression: is a statistical method used to analyze the relationship between a categorical dependent variable and one or more independent variables. It predicts the probability of a categorical outcome, making it a powerful tool for understanding and predicting binary outcomes

Statistical analysis

Continuous variables were reported as mean and Standard deviation and group differences were tested for differences using the t test and the Mann–Whitney U test. In contrast, categorical variables are expressed as the number of cases and percentage divided by the number of cases, excluding missing and unknown cases. Fisher’s exact test or the χ² test was used to compare categorical variables. Following the preliminary data, all patients were randomly divided into training and testing in a ratio of 80:20. The first group included 2070 patients with good functional outcomes (mRS ≤ 2), and the second group included 198 patients with poor functional outcomes (mRS ≥ 3). Tenfold cross-validation was used for feature selection and parameter fitting on the training set. The training process is used for modeling, while light testing is used only for model evaluation. We use the calibration plots to evaluate the calibration, the SFS technique for feature selection, and the GridSearch CV for hyperparameter tuning. The tuned hyperparameters for the ML models are listed in Supplementary Table 2.

The primary evaluation metric for discrimination was the AUC, while accuracy, positive predictive value (PPV), negative predictive value (NPV), and F1 score were considered secondary metrics. The differences between the logistic regression and other ML models were tested using the Delong test. The calibration of the models was evaluated using calibration curve plots. Statistical analysis was performed using SAS software (SAS 9.4) and Python software (python v3.9.7). Two-sided probability values < 0.05 are considered significant.

Results

Baseline characteristics

A total of 2268 young patients were included in our study after excluding 12,874 patients from 15,166, who are older than 50 years, and patients with missing data, as presented in Fig. 1. We had 24 variables with missing values, but the missing proportions for all variables are less than 5%, as shown in Supplemental Table 3. The mean age of our included patients was 44.3 ± 5.5 years, and 1787 (76%) patients were male. Table 1 presents the clinical characteristics of the young patients grouped into good and poor functional outcomes. After 3 months, 2070 patients (91%) had a good functional outcome (mRS 0–2) and 198 patients (9%) had poor functional outcomes (mRS 0–5), the mean age in each group was 44.3 ± 5.5 and 44.5 ± 5.3 respectively. The rate of males was 78.6% and 80.3% in the two groups respectively. Poor functional outcome in young patients was associated with numerous factors notably: living alone condition (P = 0.06), marital status (P = 0.002), education level (P = 0.08), history of smoking (P = 0.02), heavy drinking (P = 0.06) and stroke (P < 0.001), including arterial atrial fibrillation (P = 0.02), in addition to NIHSS score (P < 0.001) and mRS score in admission (P < 0.001) TOAST classification (P < 0.001), secondary prevention, and laboratory values of lymphocytes (P < 0.001) and neutrophils (P < 0.001).

Fig. 1

Flow chart of the patients included in the study

Table 1 Clinical characteristics of young patients according to modified Rankin score

Feature selection

The feature selection was made by tenfold cross-validation and SFS–KNN. For the feature selection, choosing mRS in admission, living alone, and NIHSS in discharge remained independent predictors of a 3-month poor outcome (Supplementary Fig. 1).

Performance of the model

Table 2 shows the AUC, accuracy, PPV, NPV, and F1 scores of the different models. The XGBboost model achieved the highest AUC (AUC = 0.801), followed by GBDT (AUC = 0.795), RF (AUC = 0.794), LightGBM (AUC = 0.792), and logistic regression (AUC = 0.789). The ROC curve and AUC of each machine-learning method compared with the logistic model are shown in Fig. 2. The predictive performance of the XGBoost (P = 0.03), RF (P = 0.01), and LightGBM (P = 0.04) models was better than the logistic regression model. The calibration plots curve of XGBoost, GBDT, RF, and lightGBM are shown in Supplementary Fig. 2.

Table 2 Test sets result of machine learning models at 3-month stroke outcome prediction

Fig. 2

Receiver Operating Characteristic curves for machine learning models. a: ROC curve of XGBoost model; b: ROC curve of LightGBM model; c: ROC curve of Random Forest model; d: ROC curve of GBDT model

Discussion

In this study, the XGBoost model was identified as the optimal predictive model. This study aimed to determine the factors that lead to poor functional outcomes 3 months after an AIS in young patients. It also aimed to compare the predictive performance of the ML algorithm and the logistic model. The main findings of this study were:

• In 2268 young patients, poor functional outcome was significantly associated with a high mRS score at admission, living alone conditions, and a high NIHSS score at discharge.

• ML is superior to logistic regression, with XGBoost being the best model.

The lifelong impact of stroke in young adults is associated with significant costs for patients themself, their families, and society. The long-term medical, psychosocial, and socioeconomic consequences are particularly severe at younger ages [23]. Therefore, there is a need to identify risk factors and develop and validate predictive scores for post-AIS outcomes. Recently, many ML models have been designed to predict adverse outcomes using algorithms that can learn from large amounts of complex data. In a recent study, the RF method using a combination of Random Under-Sampling (RUS) and biomarkers was found to be the best stroke prediction model in Chinese adult patients with hypertension [24]. A multidisciplinary study of atherosclerosis found that of nine predictive tests, RF was the best model for predicting cardiovascular disease risk including AIS [25].

In addition, results from the China Longitudinal Health and Longevity Study, show that red light running (RLR) applied to the Synthetic Minority Over-sampling Technique (SMOTE) is superior to other test models in predicting stroke in the elderly [22]. Also, the study by Hao et al. showed that a deep neural network model could improve the prediction of long-term outcomes in 2604 AIS patients aged 66.2 ± 12.6 years [9].

Our study shows that the XGBoost model has good discrimination (AUC = 0.81), and is better than other algorithms in predicting poor functional outcomes in young AIS patients within 3 months, followed by RF, lightGBM, and GBDT. Among them, XGBoost, RF, and lightGBM were better than logistic regression. Choosing the right ML model for disease prediction is critical for optimization. Various ML models have already been developed to predict clinical outcomes after stroke in both general and elderly patients. The study of Chen et al. suggested that the CatBoost algorithm had the best predictive performance compared to logistic regression and other ML models [14], and found that gender, age, stroke history, heart rate, d-dimer, creatinine, TOAST classification, mRS at admission and discharge, and NIHSS score at discharge predicts poor outcomes at 90 days in patients with TIA [14]. In addition, the study by Xiang et al. [26] showed that the RF model could better predict 6-month outcomes of Chinese AIS patients than the Houston intra-arterial therapy (HIAT) score, the total health risks in vascular events (THRIVE) score, as well, the NIHSS score on admission, age, previous Diabetes mellitus and crEatinine (NADE) Nomogram. This study found that NIHSS at admission, age, premorbid mRS, fasting glucose, and creatinine were significant predictor factors. Moreover, the study by Xio et al. proved that the XGB model is a reliable predictive model, and also showed that hypertension, cancer, congestive heart failure, chronic lung, and peripheral vascular disease may be closely associated with stroke in elderly patients [27]. However, predicting risk factors for poor functional impairment in young patients using different types of ML remains unclear.

Feature selection from ML has shown that a high mRS score at admission and a high NIHSS score at discharge, as well as, the patient living alone remained independent predictors of poor 3-month outcomes in young patients with AIS. The NIHSS and mRS are quantitative tools used to efficiently and effectively assess the degree of neurological impairment in patients with AIS. In addition, these neurological severity scores are closely related to the patient’s brain necrosis volume, location, type, perfusion, and injury [28, 29]. On the other hand, our results are consistent with Waje-Andreassen et al. who found that living alone was a predictor of long-term mortality in 232 young stroke patients [30]. Additionally, in the Riks-Stroke-based study, living alone condition was an independent predictor of short-term mortality after stroke [31]. In addition, a recent study suggests that stroke severity is associated with living alone [32]. Mathew et al. showed that individuals living alone at home were much less likely to arrive at the hospital early than those living with others and that this delay resulted in a much lower thrombolysis treatment rate [33]. The Swedish stroke registry also showed that treatment rates were ≈ 50% lower in patients living alone [34], which may explain the association between the condition of living alone and poor functional outcomes after stroke in young patients. Moreover, other studies have demonstrated that living alone can be considered a proxy for low social support, and for coronary heart disease, biological processes such as inflammatory and prothrombotic disorders, and mental disorders [35].

Our finding shows that the XGboost model can better predict the risk of 3-month poor functional outcomes in young patients with AIS. These results are similar to the study by Chung et al. which suggested that the XGBoost model is a reliable predictive power for AIS and also demonstrates the validity of the model for use in patients receiving various AIS treatments [15]. In addition, Yuan et al. have shown that the XGBoost model has better performance in predicting the 90-day readmission risk in AIS patients [36]. XGBoost is a new integrated learning method that boosts gradient. It implements a ML algorithm in the context of gradient boosting and is efficient, flexible, and portable. XGBoost is an efficient gradient-boosting algorithm capable of handling large-scale datasets, outperforming many other ML algorithms in terms of performance. It features built-in regularization, effectively preventing overfitting and enhancing the model's generalization ability. Overall, XGBoost excels in processing large-scale data, high-dimensional features, and complex tasks. The XGBoost classification method is more suitable for clinical predictive analysis than other ML techniques because it is effective and can combine the classification and regression tree process, allowing the processing of different, complex, and nonlinear models (such as multiple cases, and medical conditions). The potential of ML to significantly improve health care by automating routine processes and improving clinical decision-making is tantalizing today [37]. The future is likely to be characterized by augmented intelligence, in which computers become indispensable tools for patient care, and allow physicians to spend more time on patient care [38].

In the future, we could use the XGBoost model accessible via an online web page or integrated into clinical decision support systems (CDSS). This would allow clinicians to conveniently use the model in their daily work. Additionally, providing clear expectations to patients and their families can help them better understand the illness and actively participate in the treatment and rehabilitation process. Our prediction model will require further validation in prospective studies to confirm its effectiveness. We believe that with additional research and validation, the XGBoost model has the potential to be widely applied in clinical practice, enhancing the treatment and prognosis of young stroke patients.

Using the smallest variables to achieve better predictions is our strength. The simpler a model is, the easier it is to validate. Second, the predictors used in our study were comprehensive and included demographic, lifestyle, and clinical variables, which allowed us to examine the relationship between risk factors and stroke from multiple perspectives. In addition, the data used in this study were from large a Chinese cohort with high-quality data representing AIS patients in China.

Our study also has some limitations, first, there is some level of missing values, but, our all-missing values are < 5%. We used imputations to fill in missing laboratory data, and no statistical differences were observed between the data before and after the imputation process. Second, this study does not include genomic and imaging data, which may have limited predictive power. Third, external validation is absent, and this will be conducted in an independent external cohort population in the future.

Conclusion

Our results suggest that employing ML methods particularly XGBoost may improve upon conventional logistic regression models in identifying young stroke patients at risk of poor functional outcomes within 3 months.