Prediction of lateral lymph node metastasis with short diameter less than 8 mm in papillary thyroid carcinoma based on radiomics

Abstract

Objective

The aim of this study was to establish an ensemble learning model based on clinicopathological parameter and ultrasound radomics for assessing the risk of lateral cervical lymph node with short diameter less than 8 mm (small lymph nodes were used instead) metastasis in patients with papillary thyroid cancer (PTC), thereby guiding the selection of surgical methods.

Methods

This retrospective analysis was conducted on 454 patients diagnosed with papillary thyroid carcinoma who underwent total thyroidectomy and lateral neck lymph node dissection or lymph node intraoperative frozen section biopsy at the First Hospital of China Medical University between January 2015 and April 2022. In a ratio of 8:2, 362(80%) patients were assigned to the training set and 92(20%) patients were assigned to the test set. Clinical pathological features and radomics features related to ultrasound imaging were extracted, followed by feature selection using recursive feature elimination (RFE). Based on distinct feature sets, we constructed ensemble learning models comprising random forest (RF), extreme gradient boosting (XGBoost), categorical boosting (CatBoost), gradient boosting decision tree (GBDT), and light gradient boosting machine (Lightgbm) to develop clinical models, radiomics models, and clinical-radiomic models. Through the comparison of performance metrics such as area under curve (AUC), accuracy (ACC), specificity (SPE), precision (PRE), recall rate, F1 score, mean squared error (MSE) etc., we identified the optimal model and visualized its results using shapley additive exPlanations (SHAP).

Results

In this study, a total of 454 patients were included, among whom 342 PTC patients had small lymph node metastasis in the lateral neck region, while 112 did not have any metastasis. A total of 1035 features were initially considered for inclusion in this study, which were then narrowed down to 10 clinical features, 8 radiomics features, and 17 combined clinical-omics features. Based on these three feature sets, a total of fifteen ensemble learning models were established. In the test set, RF model in the clinical model is outperforms other models (AUC = 0.72, F1 = 0.75, Jaccard = 0.60 and Recall = 0.84), while CatBoost model in the radiomics model is superior to other models (AUC = 0.91, BA = 0.83 and SPE = 0.76). Among the clinical-radiomic models, Catboost exhibits optimal performance (AUC = 0.93, ACC = 0.88, BA = 0.87, F1 = 0.91, SPE = 0.83, PRE = 0.88, Jaccard = 0.83 and Recall = 0.92). Using the SHAP algorithm to visualize the operation process of the clinical-omics CatBoost model, we found that clinical omics features such as central lymph node metastasis (CLNM), Origin_Shape_Sphericity (o_shap_sphericity), LoG-sigma3_first order_ Skewness (log-3_fo_skewness), wavelet-HH_first order_Skewness (w-HH_fo_skewness) and wavelet-HH_first order_Skewness (sqr_gldm_DNUN) had the greatest impact on predicting the presence of lateral cervical small lymph node metastasis in PTC patients.

Conclusions

(1) In this study, among the ensemble learning models established based on clinicopathological features and radiomics features for predicting PTC lateral small lymph node metastasis, the clinical-radiomic CatBoost model has the best performance. (2) SHAP can visualize how the clinical and radiomics features affect the results and realize the interpretation of the model. (3) The combined CatBoost model can improve the diagnostic accuracy of suspicious lymph nodes with short diameter < 8 mm that are difficult to obtain accurate puncture results. The combined application of radiomics features is more accurate and reasonable than the prediction of clinical data alone, which helps to accurately evaluate the surgical scope and provide support for individual clinical decision making.

Introduction

Thyroid cancer is the most common endocrine malignancy, among which differentiated thyroid cancer (DTC) accounts for about 95%, and papillary thyroid cancer (PTC) is the most common, accounting for about 90% of adult thyroid cancer [1]. The prevalence of lateral lymph node metastasis (LLNM) in papillary thyroid carcinoma (PTC) patients has been reported to range from 21 to 63% based on scientific studies [2, 3]. Lymph node metastasis is a major risk factor for PTC recurrence [4], and when recurrence occurs, it may increase the risk of second surgery and increase the probability of postoperative complications [5]. Therefore, effective preoperative evaluation can make reasonable inferences and predictions for thyroid lateral lymph node metastasis (LLNM). It is of great significance for the selection of PTC staging and treatment [6].

Ultrasound is widely acknowledged as the preferred imaging modality for thyroid diseases [7], enabling effective evaluation of the status of lateral neck lymph nodes with a sensitivity ranging from 60 to 80%. However, it should be noted that the diagnostic accuracy of ultrasonography is significantly influenced by the expertise and experience of the operator [8, 9]. The indications for lateral neck lymph node dissection include ultrasound-guided lymph node puncture biopsy or intraoperative frozen biopsy, both of which require pathologically confirmed metastasis [10, 11]. According to the American Thyroid Association( ATA) in 2015, fine needle aspiration(FNA) is not recommended for lymph nodes with a short diameter of < 8 mm [11]. The European Thyroid Association (ETA) pointed out that lymph node diameter greater than 5 mm is considered to be a predictor of malignancy. Similarly, Xia et al. proposed that ultrasound-guided FNA has high accuracy in diagnosing lateral lymph node metastases with a diameter smaller than 6 mm [12, 13]. Clinically, machine learning models based on ultrasonographic features have been applied to evaluate central or lateral cervical lymph node metastasis of PTC, and have high diagnostic efficacy [14, 15]. However, there is no reliable machine learning model to evaluate the lateral cervical small lymph nodes [16](short diameter < 8 mm).

Compared with a single machine learning model, ensemble learning model can combine the advantages of multiple classifiers to make the prediction results more accurate. The shapley additive explanation (SHAP), proposed by Lundberg and Lee in 2017, is an additive feature attribution method that introduces SHAP values as a unified measure of feature importance, realizing local and global visualization, and improving the explainability of the model [17,18,19]. SHAP has been successfully applied to predict sentinel lymph node metastasis of breast cancer and distant metastasis of osteosarcoma [20, 21]. The purpose of this study was to establish ensemble learning models based on ultrasound image radiomics to predict the metastasis of lateral cervical small lymph nodes of PTC, and to visualize the prediction results of the ensemble learning model using the SHAP algorithm. This provides guidance for the management of suspicious lymph nodes that are difficult to obtain accurate puncture results.

Material and method

Population

This study follows the Declaration of Helsinki and was approved by the Medical Science Research Ethics Committee of the First Affiliated Hospital of China Medical University (AF-SOP-07-1.2-01). The requirement for written informed consent was waived due to the retrospective nature of this study. A retrospective analysis was made of 1386 patients who underwent thyroidectomy, lateral cervical lymph node dissection or intraoperative frozen biopsy in the Department of Thyroid Surgery, the First Affiliated Hospital of China Medical university from January 2015 to April 2022. Inclusion criteria: (1) patients had complete clinical data, and ultrasound examinations of thyroid and cervical lymph nodes were performed within one week before surgery; (2) the initial thyroid surgery should be accompanied by lateral neck lymph node dissection or intraoperative frozen biopsy of III/IV lymph nodes; (3) postoperative histopathological results confirmed PTC; (4) ultrasonography revealed that the short diameter of suspicious lymph node(SLN) in the lateral neck was less than 8 mm. SLN were defined as those exhibiting at least one of the following five ultrasonic features: round shape, focal strong echo, cystic degeneration, microcalcification, and peripheral blood vessels [11]. Exclusion criteria: (1) a medical history of prior cervical radiation exposure or treatment; (2) history of previous malignancies in the cervical region; (3) absence of conventional ultrasound images; (4) the US image failed to fully depict the lesion due to its considerable size. Out of these, a total of 932 patients were excluded from the study, while 454 patients were included in the final analysis. Out of these, a total of 932 patients were excluded from the study, while 454 patients were included in the final analysis. Among them, 362 patients were randomly assigned to the training set, and the remaining 92 patients were allocated to the test set (Fig. 1).

Fig. 1

Study flow chart

Instrument

EPIQ7 (linear array probe = 5–12 MHz; Philips, Amsterdam, Netherlands), Preirus (linear array probe = 5–12 MHz; Hitachi, Tokyo, Japan) and AixPlorer (linear array probe = 4–15 MHz; Supersonic, France).

Ultrasonic image characteristics and clinicopathological parameters

All participants underwent preoperative ultrasound examination within one week prior to the surgery. Then, the lesions were evaluated by ultrasound by physicians A and B with more than 10 years of experience in ultrasound examination. The clinical characteristics of the patients were retrospectively analyzed, encompassing demographic information, ultrasonic features of the primary thyroid lesion, and postoperative pathological findings. The basic information of patients includs: sex and age. Ultrasound features includes: tumor location (upper left, middle left, lower left, upper right, middle right or lower right); diameter; texture(even or uneven); echogenicity (hyperechoic, isoechoic, hypoechoic, very hypoechoic or anechoic); composition (solid, solid dominate, cystic, cystic dominate or spongy); boundary(clear or dim); orientation(wider-than-tall or taller-than-wide); posterior features(enhancement, attenuation, unaltered, mixed changes); mulifocality; halo; microcalcification; benign nodules and SLN. Pathology information includes: prelaryngeal lymph node (PLN), central lymph node (CLN), hashimoto’s thyroiditis (HT), capsular invasion.

Division of areas of interest

A physician with a decade of expertise in thyroid ultrasound meticulously delineates regions of interest (ROI) on ultrasound images for the purpose of identifying thyroid lesions.

Data preprocessing and feature screening

The ultrasonic image data is subjected to mixup image amplification, followed by the application of various filters on the amplified image data. These filters include laplacian of gaussian (LoG, σ = 1, 3), wavelet, square, square root, exponential, and gradient. The utilization of pyradiomic (v3.0.1, https://github.com/Radiomics/pyradiomics) in ultrasound images facilitates the extraction of ROI characteristics pertaining to radiomics analysis. Firstly, the non-counting features undergo max-min normalization, followed by a subsequent screening of these features through recursive feature elimination (RFE).

Model construction

Random forest (RF), extreme gradient boosting (XGBoost), categorical boosting (CatBoost), gradient boosting decision tree (GBDT), and light gradient boosting machine (Lightgbm). Classifiers were utilized to construct clinical models, radiomics models, and clinical-radiomic combined models based on different feature sets obtained after pretreatment. In the process of model construction, a 5-fold cross-validation is performed on the training set by randomly partitioning it based on patients, and subsequently evaluating the final performance of this predictive model using the test set. Data analysis incorporates five machine learning algorithms implemented in Python (version 3.10), namely xgboost (https://github.com/dmlc/xgboost/), lightgbm (https://github.com/microsoft/LightGBM/), catboost (https://github.com/catboost/), as well as RF and GBDT from Scikit-learn (https://scikit-learn.org/stable/).

Selection and validation of Ensemble Learning models

The performance of the model was assessed on both training and validation datasets using established clinical statistical measures, including area under the curve (AUC), accuracy, specificity, precision, sensitivity, F1 score, and mean squared error(MSE). The diagnostic performance was compared, and the best-performing model set on the test dataset was identified and evaluated to select the classifier with the optimal predictive performance.

SHAP visualization

The SHAP Python framework (version 0.41.0) was utilized for visualizing the optimal model, employing SHAP (https://github.com/slundberg/shap). Figure 2 illustrates the process of constructing a model based on clinical and radiomics features, presented using SHAP.

Fig. 2

The flow diagram of the research

Statistical analysis

SPSS (v. 27.01; IBM Corp., Armonk, NY, USA) statistical software was used. The independent samples t-test is employed for continuous variables that adhere to a normal distribution, while the wilcoxon rank-sum test is utilized for those that do not. For categorical variables, both the Pearson chi-square test and Fisher’s exact probability test are applied. P < 0.05 indicates statistical significance of the observed differences.

Result

Clinical features of papillary thyroid carcinoma

This study included a total of 454 patients diagnosed with PTC, among whom 342 cases exhibited small lymph node metastasis in the lateral neck region, while 112 cases showed no evidence of metastasis. The patients were randomly divided into training and testing sets in an 8:2 ratio (Fig. 1). A total of 362 patients (80%) were assigned to the training set, comprising of 90 males and 272 females, with a median age of 37 years (range: 30–48 years). A total of 92 patients (20%) were allocated to the testing set, comprising 31 males and 61 females, with a median age of 36.5 years (range: 29.5–47 years). The mean tumor diameter was 1.59 ± 0.89 centimeters, as determined by postoperative pathology analysis. Preoperative ultrasound findings indicated SLN in 387 patients, and subsequent pathological examination confirmed CLNM in 309 patients, including 243 in the training set and 64 in the testing set. Additionally, concurrent Hashimoto’s thyroiditis was observed in 125 patients. For categorical variables, inter-observer agreement analysis was conducted for radiologists A and B, with K values > 0.8. All ultrasound feature evaluation results in this study are based on the assessments of radiologist B.

Table 1 presents a comprehensive inventory of the clinical characteristics associated with 454 patients diagnosed with PTC, encompassing both the training and testing cohorts. Significant differences (P < 0.05) were found in the training set between patient gender, age, multifocality, SLN, PLN and CLN in terms of the existence of lymph node metastases in the lateral neck. Age, diameter, posterior feature, and CLN all showed statistically significant variations in the testing set. (p < 0.05)

Table 1 Univariate analysis of lateral cervical micro-lymph node metastases of papillary thyroid carcinoma was concentrated in the training and testing sets

Evaluation of model performance and clinical utility

The study encompassed a total of 1035 features, and feature selection was conducted using RFE, resulting in the identification of 10 individual clinical features, 8 individual radiomics features, and 17 combined clinical-radiomic features (supplementary table). Based on this rationale, a total of 15 models were constructed, encompassing RF, XGBoost, CatBoost, GBDT and Lightgbm models (Table 2). By comparing the AUC values of different models, it can be observed that the RF model outperforms other models in terms of clinical model (Fig. 3A), with an AUC = 0.72, f1 = 0.75, Jac = 0.60, and recall = 0.84. In radiomics model, the CatBoost model demonstrated superior performance compared to other models (Fig. 3B), achieving a bac = 0.83, spe = 0.76, and AUC = 0.91. Among all the models, the clinical-radiomic CatBoost model demonstrates superior performance in predicting lateral cervical small lymph node metastasis in PTC (Fig. 3C), exhibiting an AUC = 0.93, ACC = 0.88, Ba = 0.87, f1 = 0.91, spe = 0.83, pre = 0.88, Jac = 0.83 and recall = 0.92; additionally, it achieves the lowest MSE at 0.33 (Table 2). Compared to individual clinical or radiomics CatBoost models, the combined model exhibits superior predictive performance with the highest area under the ROC curve (Fig. 4), while also demonstrating optimal precision-recall characteristics (Fig. 5). Compared to individual clinical or radiomics CatBoost models, the combined model exhibits superior predictive performance with the highest area under the ROC curve (Fig. 4), while also demonstrating optimal precision-recall characteristics (Fig. 5). The calibration curve of the clinical-radiomic model of CatBoost exhibits remarkable consistency with bias-corrected predicted values (Fig. 6). The detection error trade-off (DET) curves reveal that all five datasets exhibit a concentration in the third quadrant, with the CatBoost model demonstrating the most favorable error rejection rate and error acceptance rate (Fig. 7).

Table 2 Predictive performance of ensemble learning models in training and test sets

Fig. 3

(A) A ROC curve for clinical model. (B) B ROC curve of radiomics model set. (C) ROC curve of clinico-radiomics model

Fig. 4

ROC curve based on CatBoost model of three data

Fig. 5

Precision-recall characteristics of clinico-radiomics model

Fig. 6

The calibration curve of the clinical-radiomic model of CatBoost

Fig. 7

Detection error trade-off curves of clinico-radiomics model

Visualization analysis of clinical-radiomic CatBoost model

We employ Shapley values to visualize the impact of specific clinical and radiomics features on diagnostic outcomes in a clinical-radiomic CatBoost model. The SHAP bar chart (Fig. 8) illustrates the mean SHAP scores for features, highlighting CLN, o_shap_sphericity, log-3_fo_skewness, w-HH_fo_skewness, and sqr_gldm_DNUN as the most influential factors impacting the outcomes. The SHAP scatter plot (Fig. 9) visually represents the distribution of LLNM and non-LLNM by utilizing red and blue colors to indicate positive or negative effects of features on prediction outcomes. Specifically, o_shap_sphericity, log-3_fo_skewness, and w-HH_fo_skewness exhibit a detrimental impact on the results based on Shapley values, while CLN and sqr_gldm_DNUN demonstrate a favorable influence.

Fig. 8

The bar chart in the SHAP summary plot illustrates the impact of each feature on the CatBoost model

Fig. 9

The scatter plot in SHAP summary graph visually depicts the relationship between feature values and predicted probabilities through color

The SHAP decision plot (Fig. 10) visually depicts the impact of each key feature on the final decision, where individual-colored lines represent predicted outcomes for patients. The SHAP values of each element are incrementally added to the baseline of the CatBoost model, illustrating visually how individual features contribute to predictions. Ultimately, at the apex, each line intersects with its corresponding predicted value on the X-axis, representing the final probability prediction of the model.

Fig. 10

The SHAP decision plot for all patients with PTC

The SHAP heat map (Fig. 11) visualizes how different features impact the model output. Each row in the heatmap represents a specific feature, while each column presents an individual instance. Each cell’s color represents the SHAP value, which measures the contribution of a feature to the prediction for that instance. A red cell indicates that the feature increases the prediction, while a blue cell indicates that the feature decreases it. A white cell means that the feature has no impact on the prediction. The line graph depicted above the heat map illustrates the values of output f(x) for each individual case. The x-axis is labeled with instance numbers, while the y-axis is scaled based on the output range. Line charts facilitate the visualization of prediction variations across diverse instances and their correlation with the SHAP values of features.

Fig. 11

The SHAP heat map effectively visualizes the impact of different features on model output

Discussion

The present study retrospectively analyzed a cohort of 454 patients diagnosed with PTC who had undergone either lateral neck lymph node dissection or intraoperative frozen biopsy. Ensemble learning models have demonstrated outstanding performance in predicting lateral cervical small lymph node metastasis of PTC. Notably, the clinical-radiomic CatBoost model exhibited superior performance with AUC values of 0.95 and 0.94 for the training and testing sets, respectively. The radiomics model exhibited significantly higher AUC values compared to the clinical model in both the training set (0.89 vs. 0.81) and the testing set (0.90 vs. 0.71), within a single model. Compared to a single model, the clinical-radiomic integrated model not only ensures higher sensitivity (0.926 vs. 0.90) based on the radiomics model but also demonstrates improved specificity (0.83 vs. 0.76). The influential parameters affecting the judgment of LLNM were extracted based on this analysis, with the top five parameters identified as CLN, o_shap_sphericity, log-3_fo_skewness, w-HH_fo_skewness and sqr_gldm_DNUN. We employed SHAP to visualize explanations for both individuals and the overall population, where SHAP values represent the significance of features. The red and blue colors indicate positive and negative impacts on the outcome, providing robust evidence for clinical decision-making [20].

Comprehensive clinical studies have been carried out to predict lymph node metastasis based on the clinical features and pathological classification of papillary thyroid carcinoma [22]. Additionally, there exist nomogram and machine learning models for predicting the metastatic status of cervical lymph nodes on the PTC side based on ultrasound characteristics [3, 23,24,25]. However, there is a scarcity of established models utilizing radiomics features extracted from ultrasound images [9]. Recently, Zhu et al. [26]. developed a clinical-ultrasound nomogram based on multicenter data to predict lateral cervical lymph node metastasis in patients with PTC. In their internal validation set, the AUC of their study was higher than that of our clinical model (0.82 vs. 0.71), but lower than that of the clinical-radiomic CatBoost model (0.82 vs. 0.94). The suboptimal performance of our study’s individual clinical model may be attributed to the relatively small size of our dataset compared to theirs. The incorporation of radiomics features significantly enhances prediction accuracy, potentially capturing underlying tissue heterogeneity in PTC that is imperceptible to the naked eye, such as nodule density and uniformity. These characteristics can be extracted through radiomics analysis and provide valuable evidence for model evaluation [14]. A study conducted by Feng et al. and Lai et al. investigated the utilization of machine learning models for predicting LLNM in PTC based on clinical ultrasound features. The findings demonstrated that the RF model exhibited superior performance, aligning with the outcomes of this investigation [3, 25]. Vivian Y et al. [9] constructed ROC curves by extracting radiomic features such as texture, wavelet, and first-order statistics from ultrasound images of primary tumors. In the validation cohort, the AUC value of this model was significantly lower than that of our clinical-radiomic model in this study (0.62 vs. 0.94). Building upon Vivian Y’s research, we incorporated additional clinical-pathological features into our model, which not only encompassed characteristics of the lesions themselves but also included extranodal features such as CLN status, PLN status, and ultrasound indications of SLN. This integration to some extent improved the performance of our model.

The CatBoost clinical-radiomic model employed in this study successfully identified 17 features through rigorous feature selection techniques and effectively visualized their impact using SHAP. This approach not only accounts for the individual effects of factors on the outcome but also considers the synergistic interactions among different variables [20]. The magnitude of the absolute value of SHAP indicates the influence of features on the results. Our findings reveal that a significant proportion is attributed to expressing grayscale distribution features, including LoG-sigma3_first order_Skewness, LoG-sigma3_glrlm_ShortRunLowGrayLevelEmphasis, wavelet-HH_first order_Skewness, and square_glrlm_ShortRunEmphasis. It can be observed that first-order statistical characteristics of pixel grayscale values and short run gray level intensity have a substantial impact on prediction accuracy, reflecting both overall grayscale distribution patterns and local grayscale intensity conditions. Furthermore, a significant correlation exists between the sphericity of shape features and the predictive outcomes. As the sphericity approaches 1, indicating smoother and more circular primary lesions in PTC, there is a decrease in the likelihood of metastasis. Conversely, when the sphericity is below 0.5 or close to 0, there is an increase in the probability of metastasis. After conducting experimental analysis, this study identified several effective ultrasound image filtering methods for predicting lateral neck lymph node metastasis in PTC, including the original image, wavelet transform, LoG-σ = 3, square transformation, and square root transformation. The radiomics feature, wavelet-HH_first order_Skewness is derived from the application of wavelet transform. The high-pass filtering employed in wavelet transform effectively enhances diagonal edges and image details, as well as provides valuable edge and shape information. In addition to omic features, a plethora of research has demonstrated that CLNM exerts a substantial influence on LLNM, which aligns with the findings of this study. This not only augments the risk of ipsilateral LLNM but also escalates the likelihood of contralateral metastasis [27]. Furthermore, risk stratification will incorporate the quantity of CLNM, with commonly utilized thresholds in clinical practice being 2, 3 and 5 [4, 17, 28]. Currently, a plethora of clinical studies substantiate the pivotal role of CLN status in guiding follow-up and treatment strategies. For patients afflicted with CLNM, specialized examination techniques should be employed to assess the LLN, while regular surveillance suffices for patients without CLNM [29].

The combined CatBoost model can help in the early detection of metastasis, leading to timely and potentially more effective treatment, and it is a non-invasive method which could reduce the need for more invasive procedures like biopsies. Meanwhile by accurately predicting metastasis, the model can aid in tailoring personalized treatment plans for patients. However, this study also has the following limitations. First of all, the data collection is based on a single-center retrospective study, which lacks external verification of the model, and the generalization ability of the model cannot be improved. Meanwhile, ultrasound doctors may have different evaluation criteria for suspicious small lymph nodes, so it is necessary to unify evaluation indicators and conduct prospective multi-center exploration on its practicability. Secondly, our reliance on preoperative fine needle puncture or intraoperative grade III/IV lymph node biopsy as the sole basis for lateral cervical lymph node dissection may lead to an increase in the incidence of false negatives. Finally, while the initial setup might add cost-related efforts, the long-term use of these models can be cost-effective by reducing unnecessary treatments and procedures. At the same time, the software is being developed, and it is reasonable and effective to put into the clinic, and it does not require much cost in future application and maintenance.

Conclusions

• 1. This study presents an ensemble learning model to predict the presence of small lymph node metastasis in the lateral cervical of PTC based on clinical and radiomic features.

• 2. Among these models, the CatBoost model demonstrates superior performance, with its results effectively visualized using SHAP for enhanced interpretability.

• 3. The combined CatBoost model can improve the diagnostic accuracy of suspicious lymph nodes with short diameter < 8 mm that are difficult to obtain accurate puncture results. The combined application of radiomics features is more accurate and reasonable than the prediction of clinical data alone, which helps to accurately evaluate the surgical scope and provide support for individual clinical decision making.