Skip to main content
Springer Nature Link
Log in

Multi-modal clear cell renal cell carcinoma grading with the segment anything model

  • Regular Paper
  • Published:
Save article
View saved research
Multimedia Systems Aims and scope Submit manuscript

Abstract

Clear cell renal cell carcinoma (ccRCC) is a prevalent kidney disease, accounting for more than 75% of renal cell carcinoma (RCC) and approximately 3.8% of human malignancies. Early grading of ccRCC is crucial in guiding personalized treatment plans, and it is of great significance in clinical decision-making and prognosis evaluation, but ccRCC grade is usually obtained by postoperative pathology. The purpose of this study is to develop an automated non-invasive approach based on deep learning methods for accurate segmentation and preoperative grading of ccRCC tumors into low-grade and high-grade categories. CT images, radiomics features, and patient data are enrolled for our ccRCC grading framework. Firstly, we train a model for kidney tumor segmentation on CT images based on the recently released Segment Anything Model (SAM) with simple yet effective strategies. Secondly, we utilize the pretrained image encoder from the segmentation model to propose a contrastive learning-based approach, in order to learn the mutual information between the images and radiomics. Lastly, we train machine learning-based models for ccRCC grading, using image features, radiomics features and patient data. Our proposed segmentation method has demonstrated superior performance in kidney tumor segmentation, outperforming the original SAM with an average improvement of 6.03% in Dice Similarity Coefficient (DSC) and 16.24% in surface Dice Similarity Coefficient (sDSC). While for ccRCC grading, our method achieves an accuracy of 82.50%, superior to other comparative methods. The experimental results demonstrate the efficacy of our methods in accurately segmenting ccRCC tumors and providing reliable preoperative grading information. These findings highlight the significant clinical potential of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
The alternative text for this image may have been generated using AI.
Fig. 2
The alternative text for this image may have been generated using AI.
Fig. 3
The alternative text for this image may have been generated using AI.
Fig. 4
The alternative text for this image may have been generated using AI.
Fig. 5
The alternative text for this image may have been generated using AI.
Fig. 6
The alternative text for this image may have been generated using AI.
Fig. 7
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data availability

Data is provided within the manuscript or supplementary information files.

References

  1. Capitanio, U., Montorsi, F.: Renal cancer. Lancet. 387(10021), 894–906 (2016). https://doi.org/10.1016/S0140-6736(15)00046-X

    Article  Google Scholar 

  2. Motzer, R.J., Carducci, M.A., Fishman, M., et al.: Kidney cancer. Clinical practice guidelines. J. Natl. Compr. Canc Netw. 3(1), 84–93 (2005)

    Google Scholar 

  3. Moch, H., Cubilla, A.L., Humphrey, P.A., Reuter, V.E., Ulbright, T.M.: The 2016 WHO classification of tumours of the urinary system and male genital organs-part A: Renal, Penile, and testicular tumours. Eur. Urol. 70(1), 93–105 (2016). https://doi.org/10.1016/j.eururo.2016.02.029

    Article  Google Scholar 

  4. Delahunt, B., Eble, J.N., Egevad, L., Samaratunga, H.: Grading of renal cell carcinoma. Histopathology. 74, 4–17 (2019)

    Article  Google Scholar 

  5. Motzer, R.J., Agarwal, N., Beard, C., et al.: NCCN clinical practice guidelines in oncology: Kidney cancer. J. Natl. Compr. Canc Netw. 7(6), 618–630 (2009). https://doi.org/10.6004/jnccn.2009.0043

    Article  Google Scholar 

  6. Novara, G., Martignoni, G., Artibani, W., Ficarra, V.: Grading systems in renal cell carcinoma. J. Urol. 177, 430–436 (2007). https://doi.org/10.1016/j.juro.2006.09.034

    Article  MATH  Google Scholar 

  7. Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2016, 2424–2433 (2016). https://doi.org/10.1109/CVPR.2016.266

    Article  Google Scholar 

  8. Gao, Z., Puttapirat, P., Shi, J., Li, C.: Renal cell carcinoma detection and subtyping with minimal point-based annotation in whole-slide images. MICCAI. 439–448 (2020). https://doi.org/10.48550/arXiv.2008.05332

  9. Tabibu, S., Vinod, P.K., Jawahar, C.V.: Pan-renal cell carcinoma classification and survival prediction from histopathology images using deep learning. Sci. Rep. 9(1), 10509 (2019). https://doi.org/10.1038/s41598-019-46718-3

    Article  MATH  Google Scholar 

  10. Pedersen, M., Andersen, M.B., Christiansen, H., Azawi, N.H.: Classification of renal tumour using convolutional neural networks to detect oncocytoma. Eur. J. Radiol. 133, 109343 (2020). https://doi.org/10.1016/j.ejrad 2020.109343

    Article  Google Scholar 

  11. McGillivray, P.D., Ueno, D., Pooli, A., et al.: Distinguishing benign renal tumors with an oncocytic gene expression (onex) classifier. Eur. Urol. 79(1), 107–111 (2021). https://doi.org/10.1016/j.eururo.2020.09.017

    Article  Google Scholar 

  12. Zabihollahy, F., Schieda, N., Krishna, S., Ukwatta, E.: Automated classification of solid renal masses on contrast-enhanced computed tomography images using convolutional neural network with decision fusion. Eur. Radiol. 30(9), 5183–5190 (2020). https://doi.org/10.1007/s00330-020-06787-9

    Article  Google Scholar 

  13. Shu, J., Tang, Y., Cui, J., et al.: Clear cell renal cell carcinoma: CT-based radiomics features for the prediction of Fuhrman grade. Eur. J. Radiol. 109, 8–12 (2018). https://doi.org/10.1016/j.ejrad.2018.10.005

    Article  MATH  Google Scholar 

  14. Wang, W., Cao, K., Jin, S., Zhu, X., Ding, J., Peng, W.: Differentiation of renal cell carcinoma subtypes through MRI-based radiomics analysis. Eur. Radiol. 30(10), 5738–5747 (2020). https://doi.org/10.1007/s00330-020-06896-5

    Article  MATH  Google Scholar 

  15. Liu, S., Deng, J., Dong, D., et al.: Deep learning-based radiomics model can predict extranodal soft tissue metastasis in gastric cancer. Med. Phys. (2023). https://doi.org/10.1002/mp.16647

    Article  Google Scholar 

  16. Kirillov, A., Mintun, E., Ravi, N., et al.: Segment Anything Preprint ArXiv ; (2023). arXiv:2304.02643

  17. Deng, R., Cui, C., Liu, Q., et al.: Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. Preprint ArXiv. (2023). arXiv:2304.04155

  18. He, S., Bao, R., Li, J., Grant, P.E., Ou, Y.: Computer-vision benchmark segment-anything model (SAM) in medical images: Accuracy in 12 datasets. Preprint ArXiv ; (2023). arXiv:2304.09324

  19. Hu, C., Li, X.: When Sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation. Preprint ArXiv. (2023). arXiv:2304.08506

  20. Maciej, A.M., Haoyu, D., Hanxue, G., et al.: Segment anything model for medical image analysis: An experimental study. Preprint ArXiv ; (2023). arXiv:2304.10517

  21. Mohapatra, S., Gosai, A., Schlaug, G.: Sam vs bet: A comparative study for brain extraction and segmentation of magnetic resonance images using deep learning. Preprint ArXiv ; (2023). arXiv:2304.04738

  22. Roy, S., Wald, T., Koehler, G., et al.: Sam. Md: Zero-shot medical image segmentation capabilities of the segment anything model. Preprint ArXiv ; (2023). arXiv:2304.05396

  23. Ma, J., He, Y., Li, F., et al.: Segment anything in medical images. Nat. Commun. 15, 654 (2024). https://doi.org/10.1038/s41467-024-44824-z

    Article  MATH  Google Scholar 

  24. Tianrun, C., Lanyun, Z., Chaotao Ding, et al.: SAM fails to segment anything? -SAM-adapter: Adapting SAM in underperformed scenes: Camouflage, shadow, medical image segmentation, and more. Preprint ArXiv ; (2023). arXiv:2304.09148

  25. Yizhe, Z., Tao, Z., Peixian, L., Danny, Z.C.: Input augmentation with SAM: Boosting medical image segmentation with segmentation foundation model. Preprint ArXiv ; (2023). arXiv:2304.11332

  26. Hussain, M.A., Hamarneh, G., Garbi, R.: Learnable image histograms-based deep radiomics for renal cell carcinoma grading and staging. Comput. Med. Imaging Graph. 90, 101924 (2021). https://doi.org/10.1016/j.compmedimag.2021.101924

    Article  MATH  Google Scholar 

  27. Shu, J., Wen, D., Xi, Y., et al.: Clear cell renal cell carcinoma: Machine learning-based computed tomography radiomics analysis for the prediction of WHO/ISUP grade. Eur. J. Radiol. 121, 108738 (2019). https://doi.org/10.1016/j.ejrad.2019.108738

    Article  Google Scholar 

  28. Zheng, M., Chen, Q., Ge, Y., et al.: Development and validation of CT-based radiomics nomogram for the classification of benign parotid gland tumors. Med. Phys. 50(2), 947–957 (2023). https://doi.org/10.1002/mp.16042

    Article  MATH  Google Scholar 

  29. Ge, G., Zhang, J.: Feature selection methods and predictive models in CT lung cancer radiomics. J. Appl. Clin. Med. Phys. 24(1), e13869 (2023). https://doi.org/10.1002/acm2.13869

    Article  Google Scholar 

  30. Gillies, R.J., Kinahan, P.E., Hricak, H., Radiomics: Images are more than pictures, they are data. Radiology. 278(2), 563–577 (2016). https://doi.org/10.1148/radiol.2015151169

    Article  Google Scholar 

  31. Khalvati, F., Zhang, Y., Wong, A., Haider, M.A.: Radiomics Encyclopedia Biomedical Eng. ;597–603. (2019)

  32. Nazari, M., Shiri, I., Zaidi, H.: Radiomics-based machine learning model to predict risk of death within 5-years in clear cell renal cell carcinoma patients. Comput. Biol. Med. 129, 104135 (2021). https://doi.org/10.1016/j.compbiomed.2020.104135

    Article  MATH  Google Scholar 

  33. Hassani, C., Varghese, B.A., Nieva, J., Duddalwar, V.: Radiomics in pulmonary lesion imaging. AJR Am. J. Roentgenol. 212(3), 497–504 (2019). https://doi.org/10.2214/AJR.18.20623

    Article  Google Scholar 

  34. Lin, R.Y., Zheng, Y.N., Lv, F.J., et al.: A combined non-enhanced CT radiomics and clinical variable machine learning model for differentiating benign and malignant sub-centimeter pulmonary solid nodules. Med. Phys. 50(5), 2835–2843 (2023). https://doi.org/10.1002/mp.16316

    Article  Google Scholar 

  35. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. ArXiv. (2020). arXiv:2002.05709

  36. Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. ArXiv. (2020). arXiv:2010.00747

  37. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. ArXiv. (2019). arXiv:1911.05722v2

  38. Chen, K., Wang, Q., Ma, Y.: Cervical optical coherence tomography image classification based on contrastive self-supervised texture learning. Med. Phys. 49(6), 3638–3653 (2022). https://doi.org/10.1002/mp.15630

    Article  MATH  Google Scholar 

  39. Hu, E.J., Shen, Y., Wallis, P., et al.: Lora: Low-rank adaptation of large language models. ArXiv. (2021). arXiv:2106.09685

  40. van Griethuysen, J.J.M., Fedorov, A., Parmar, C., et al.: Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77(21), e104–e107 (2017). https://doi.org/10.1158/0008-5472.CAN-17-0339

    Article  Google Scholar 

  41. Zhao, B., Li, X., Lu, X.: CAM-RNN: Co-attention model based RNN for video captioning. IEEE Trans. Image Process. 28(11), 5552–5565 (2019). https://doi.org/10.1109/TIP.2019.2916757

    Article  MathSciNet  MATH  Google Scholar 

  42. Nazari, M., Shiri, I., Zaidi, H.: Radiomics-based machine learning model to predict risk of death within 5-years in clear cell renal cell carcinoma patients. Comput. Biol. Med. 129, 104135 (2021)

    Article  MATH  Google Scholar 

  43. Hassani, C., Varghese, B.A., Nieva, J., Duddalwar, V.: Radiomics in pulmonary lesion imaging. Am. J. Roentgenol. 212, 497–504 (2019). https://doi.org/10.2214/AJR.18.20623

    Article  Google Scholar 

  44. Heller, N., Sathianathen, N., Kalapara, A., et al.: The KiTS19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes. ArXiv ; (2019). arXiv:1904.00445

  45. Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-net: A self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods. 18(2), 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z

    Article  Google Scholar 

  46. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. IEEE. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  47. Huang, G., Liu, Z., Laurens, V.D.M., Van Maaten, L.D., Weinberger, K.Q.: Densely connected convolutional networks. IEEE Comput. Soc. 4700–4708 (2016). https://doi.org/10.1109/CVPR.2017.243

  48. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision (pp. 405–421). Springer, Cham (2020)

  49. Zhao, Z., Yang, G.: Unsupervised contrastive learning of radiomics and deep features for label-efficient tumor classification, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 252–261 (2021)

  50. Ravi, N., Gabeur, V., Hu, Y.T., et al.: Sam 2: Segment anything in images and videos[J]. (2024). arXiv preprint arXiv:2408.00714

  51. He, Y., Guo, P., Tang, Y., et al.: A Short Review and Evaluation of SAM2’s Performance in 3D CT Image Segmentation[J]. (2024). arXiv preprint arXiv:2408.11210

  52. Chen, T., Lu, A., Zhu, L., et al.: Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more[J]. arXiv preprint arXiv:2408.04579, 2024.

  53. Zhu, J., Qi, Y., Wu, J.: Medical sam 2: Segment medical images as video via segment anything model 2[J]. arXiv preprint arXiv:2408.00874, 2024.

  54. Bai, Y., Yu, Q., Yun, B., et al.: FS-MedSAM2: Exploring the potential of SAM2 for few-Shot Medical Image Segmentation without Fine-tuning[J]. (2024). arXiv preprint arXiv:2409.04298

  55. Xiong, X., Wu, Z., Tan, S., et al.: SAM2-UNet: Segment anything 2 makes strong encoder for natural and medical image Segmentation[J]. (2024). arXiv preprint arXiv:2408.08870

Download references

Acknowledgements

This study was funded by the National Natural Science Foundation (U23A20461), the Clinical Medical Research Center of Shandong Province (2021LCZX04), the Academic promotion program of Shandong First Medical University (2019LJ004), Major Basic Research Project of Shandong Natural Science Foundation (ZR2022ZD31), Special Fund Project of Shandong Central Government to Guide Local Science and Technology Development (YDZX2022010), 2021 Shandong Medical Association Clinical Research Fund -- Qilu Special Project (YXH2022DZX02002) .

Author information

Authors and Affiliations

  1. Key Laboratory of Image Science and Technology, School of Cyber Science and Engineering, Southeast University, Nanjing, China

    Yunbo Gu

  2. Key Laboratory of Image Science and Technology, School of Computer Science and Engineering, Sipailou Campus of Southeast University, Chenxian Street, Xuanwu District, Nanjing, Jiangsu province, China

    Qianyu Wu, Baosheng Li, Yudong Zhang & Yang Chen

  3. Department of Radiology, Affiliated Hospital of Medical School, Nanjing Drum Tower Hospital, Nanjing University, Nanjing, China

    Xiaoli Mai

  4. Nanjing Drum Tower Hospital Clinical College of Nanjing Medical University, Zhongshan Road 321, Gulou District, Nanjing, Jiangsu province, China

    Junting Zou & Xiaoli Mai

Authors
  1. Yunbo Gu
    View author publications

    Search author on:PubMed Google Scholar

  2. Qianyu Wu
    View author publications

    Search author on:PubMed Google Scholar

  3. Junting Zou
    View author publications

    Search author on:PubMed Google Scholar

  4. Baosheng Li
    View author publications

    Search author on:PubMed Google Scholar

  5. Xiaoli Mai
    View author publications

    Search author on:PubMed Google Scholar

  6. Yudong Zhang
    View author publications

    Search author on:PubMed Google Scholar

  7. Yang Chen
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Yunbo Gu: Conceptualization, Formal Analysis, Investigation, Methodlogy, Software, Writing and EditingQianyu Wu: Investigation and MethodlogyJunting Zou: Data Curation and Formal AnalysisBaosheng Li: Funding Acquisition and SupervisionXiaoli Mai: Conceptualization and SupervisionYundong Zhang: Review and EditingYang Chen: Supervision.

Corresponding authors

Correspondence to Baosheng Li or Xiaoli Mai.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Y., Wu, Q., Zou, J. et al. Multi-modal clear cell renal cell carcinoma grading with the segment anything model. Multimedia Systems 31, 19 (2025). https://doi.org/10.1007/s00530-024-01602-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s00530-024-01602-7

Keywords

Profiles

  1. Baosheng Li View author profile