Skip to main content
Springer Nature Link
Log in

Collaborative learning of supervision and correlation for generalized zero-shot extreme multi-label learning

  • Published:
Save article
View saved research
Applied Intelligence Aims and scope Submit manuscript

Abstract

Generalized zero-shot extreme multi-label learning (GZXML) aims to predict relevant labels for unknown instances from a set of seen and unseen labels and is widely used in engineering applications. Since the supervisory information of the instances is incomplete in this task, the existing methods classify such instances based on the semantic relationships between the instances and labels. However, the supervisory information of the seen labels is also crucial for achieving high prediction performance. To bridge this gap, we propose collaborative learning of supervision and correlations for GZXML (CLSC-XML). CLSC-XML leverages both the semantic relationships between instances and labels and the supervisory information of the seen labels to enhance the prediction results for unseen labels. Specifically, CLSC-XML extracts discriminative and representational features, which are then fed into classification and correlation modules for collaborative learning. Furthermore, to enrich the incomplete supervised information, we propose the generation of features for unseen labels (GFUL) algorithm. The classifier is trained alternately with the GFUL algorithm. The classifier provides semantic guidance to the GFUL algorithm, and in turn, the GFUL algorithm helps the classification model enrich the supervised information. Experimental results show that CLSC-XML outperforms the state-of-the-art methods and requires less training time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
The alternative text for this image may have been generated using AI.
Fig. 2
The alternative text for this image may have been generated using AI.
Algorithm 1
The alternative text for this image may have been generated using AI.
Fig. 3
The alternative text for this image may have been generated using AI.
Fig. 4
The alternative text for this image may have been generated using AI.
Fig. 5
The alternative text for this image may have been generated using AI.
Fig. 6
The alternative text for this image may have been generated using AI.
Fig. 7
The alternative text for this image may have been generated using AI.
Fig. 8
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Availability of data and materials

The authors confirm that the data supporting the findings of this study are available.

Notes

  1. https://www.kaggle.com/nltkdata/reuters

  2. https://manikvarma.org/downloads/XC/XMLRepository.html

Abbreviations

GZXML:

Generalized zero-shot extreme multi-label learning

CLSC-XML:

Collaborative learning of supervision and correlation for generalized zero-shot extreme multi-label learning

GFUL:

Generation of available features for unseen labels

VAE:

Variational autoencoder

nDCG:

Normalized discounted cumulative gain

PLT:

Probabilistic label tree

GCN:

Approximate nearest-neighbor

ANN:

Graph convolutional network

RTS:

Randomized text segmentation

ICT:

Inverse cloze task

References

  1. Jung G, Shin J, Lee S (2023) Impact of preprocessing and word embedding on extreme multi-label patent classification tasks. Appl Intell 53(4):4047–4062

    Article  Google Scholar 

  2. Tang P, Jiang M, Xia BN, Pitera JW, Welser J, Chawla NV (2020) Multi-label patent categorization with non-local attention-based graph convolutional network. Proceedings of the AAAI conference on artificial intelligence, pp 9024–9031

  3. Prabhu Y, Kusupati A, Gupta N, Varma M (2020) Extreme Regression for Dynamic Search Advertising. Proceedings of the 13th international conference on web search and data mining, pp 456–464

  4. Chang W-C, Yu H-F, Zhong K, Yang Y, Dhillon IS (2020) Taming Pretrained Transformers for Extreme Multi-label Text Classification. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 3163–3171

  5. Gupta N, Bohra S, Prabhu Y, Purohit S, Varma M (2021) Generalized Zero-Shot Extreme Multi-label Learning. Proceedings of the 27th ACM SIGKDD international conference on knowledge discovery & data mining, pp 527–535

  6. Xiong Y, Chang W-C, Hsieh C-J, Yu H-F, Dhillon I (2022) Extreme Zero-Shot Learning for Extreme Text Classification. Proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies, pp 5455–5468

  7. Zhang T, Xu Z, Medini T, Shrivastava A (2022) Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification. Find Assoc Comput Linguis EMNLP, pp 4937–4947

  8. Aggarwal P, Deshpande A, Narasimhan KR (2023) SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification. Int Conf Mach Learn pp 228–247

  9. Simig D, Petroni F, Yanki P, Popat K, Du C, Riedel S, Yazdani M (2022) Open Vocabulary Extreme Classification Using Generative Models. Find Assoc Comput Linguis ACL, pp 1561–1583

  10. You R, Zhang Z, Wang Z, Dai S, Mamitsuka H, Zhu S (2019) AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification. Adv Neural Inform Process Syst pp 5820–5830

  11. Jiang T, Wang D, Sun L, Yang H, Zhao Z, Zhuang F (2021) Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. Proceedings of the AAAI conference on artificial intelligence, pp 7987–7994

  12. Zong D, Sun S (2023) Bgnn-xml: Bilateral graph neural networks for extreme multi-label text classification. IEEE Trans Knowl Data Eng 35(7):6698–6709

    Google Scholar 

  13. Xiong J, Yu L, Niu X, Leng Y (2023) Xrr: Extreme multi-label text classification with candidate retrieving and deep ranking. Inf Sci 622:115–132

    Article  Google Scholar 

  14. Wang J, Chen Z, Qin Y, He D, Lin F (2023) Multi-aspect co-attentional collaborative filtering for extreme multi-label text classification. Knowl-Based Syst 260:110110

  15. Yu H-F, Zhong K, Zhang J, Chang W-C, Dhillon IS (2022) Pecos: Prediction for enormous and correlated output spaces. J Mach Learn Res 23(98):1–32

  16. Xu P, Xiao L, Liu B, Lu S, Jing L, Yu J (2023) Label-Specific Feature Augmentation for Long-Tailed Multi-Label Text Classification. Proceedings of the AAAI conference on artificial intelligence, pp 10602–10610

  17. Qaraei M, Babbar R (2024) Meta-classifier free negative sampling for extreme multilabel classification. Mach Learn 113(2):675–697

    Article  MathSciNet  Google Scholar 

  18. Schultheis E, Babbar R (2022) Speeding-up one-versus-all training for extreme classification via mean-separating initialization. Mach Learn 111(11):3953–3976

    Article  MathSciNet  Google Scholar 

  19. Huang X, Chen B, Xiao L, Yu J, Jing L (2022) Label-aware document representation via hybrid attention for extreme multi-label text classification. Neural Process Lett 54(5):3601–3617

    Article  Google Scholar 

  20. Li Q, Peng H, Li J, Xia C, Yang R, Sun L, Yu PS, He L (2022) A survey on text classification: From traditional to deep learning. Acm Trans Intell Syst Technol 13(2):1–41

    Google Scholar 

  21. Etter PA, Zhong K, Yu H-F, Ying L, Dhillon I (2022) Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees. Proceedings of the ACM Web Conference 2022:452–461

  22. Vu H-T, Nguyen M-T, Nguyen V-C, Pham M-H, Nguyen V-Q, Nguyen V-H (2023) Label-representative graph convolutional network for multi-label text classification. Appl Intell 53(12):14759–14774

  23. Basabain S, Cambria E, Alomar K, Hussain A (2023) Enhancing arabic-text feature extraction utilizing label-semantic augmentation in few/zero-shot learning. Expert Syst 40(8):13329

    Article  Google Scholar 

  24. Liu W, Pang J, Li N, Yue F, Liu G (2023) Few-shot short-text classification with language representations and centroid similarity. Appl Intell 53(7):8061–8072

    Article  Google Scholar 

  25. Fan W, Liang C, Wang T (2022) Contrastive semantic disentanglement in latent space for generalized zero-shot learning. Knowl-Based Syst 257:109949

    Article  Google Scholar 

  26. Zhang C, Liang C, Zhao Y (2022) Exemplar-based, semantic guided zero-shot visual recognition. IEEE Trans Image Process 31:3056–3065

    Article  Google Scholar 

  27. Wang X, Jing L, Lyu Y, Guo M, Wang J, Liu H, Yu J, Zeng T (2022) Deep generative mixture model for robust imbalance classification. IEEE Trans Pattern Anal Mach Intell 45(3):2897–2912

    Google Scholar 

  28. Mishra A, Reddy SK, Mittal A, Murthy HA (2018) A Generative Model for Zero Shot Learning Using Conditional Variational Autoencoders. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 2269–22698

  29. Schonfeld E, Ebrahimi S, Sinha S, Darrell T, Akata Z (2019) Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8239–8247

  30. Liu Y, Gao X, Han J, Shao L (2023) A discriminative cross-aligned variational autoencoder for zero-shot learning. IEEE Trans Cybern 53(6):3794–3805

  31. Liu Y, Dang Y, Gao X, Han J, Shao L (2022) Zero-shot learning with attentive region embedding and enhanced semantics. IEEE Trans Neural Netw Learn Syst, pp 1–12

  32. Luo Y, Wang X, Pourpanah F (2021) Dual vaegan: A generative model for generalized zero-shot learning. Appl Soft Comput 107:107352

    Article  Google Scholar 

  33. Tang C, He Z, Li Y, Lv J (2022) Zero-shot learning via structure-aligned generative adversarial network. IEEE Trans Neural Netw Learn Syst 33(11):6749–6762

    Article  Google Scholar 

  34. Fan C, Chen W, Tian J, Li Y, He H, Jin Y (2023) Accurate use of label dependency in multi-label text classification through the lens of causality. Appl Intell 53:21841–21857

    Article  Google Scholar 

  35. Ai Q, Li F, Li X, Zhao J, Wang W, Gao Q, Zhao F (2023) An improved mltsvm using label-specific features with missing labels. Appl Intell 53(7):8039–8060

    Article  Google Scholar 

  36. Hang J-Y, Zhang M-L (2021) Collaborative learning of label semantics and deep label-specific features for multi-label classification. IEEE Trans Pattern Anal Mach Intell 44(12):9860–9871

    Article  Google Scholar 

  37. Zhao W, Kong S, Bai J, Fink D, Gomes C (2021) HOT-VAE: Learning High-Order Label Correlation for Multi-Label Classification via Attention-Based Variational Autoencoders. Proceedings of the AAAI conference on artificial intelligence, pp 15016–15024

  38. Loza Mencía E, Fürnkranz J (2008) Efficient pairwise multilabel classification for large-scale problems in the legal domain. Joint European conference on machine learning and knowledge discovery in databases, pp 50–65

  39. McAuley J, Leskovec J (2013) Hidden factors and hidden topics: understanding rating dimensions with review text. Proceedings of the 7th ACM conference on Recommender systems, pp 165–172

  40. Prabhu Y, Varma M (2014) Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 263–272

  41. Yang Y (1999) An evaluation of statistical approaches to text categorization. Inf Retr 1(1–2):69–90

    Article  Google Scholar 

  42. Wang W, Wei F, Dong L, Bao H, Yang N, Zhou M (2020) MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. Adv Neural Inform Process Syst pp 5776–5788

Download references

Funding

This research was funded in part by the Natural Science Foundation of Liaoning Province in China (2020-MS-281) and the Basic Research Project of Education Department of Liaoning Province in China (JYTMS20230929).

Author information

Author notes

  1. Fei Zhao and Ran Tao contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Software Engineering, University of Science and Technology Liaoning, Anshan, 114051, China

    Fei Zhao, Bo Cui, Yuting Xu & Qing Ai

  2. Sales Department, Shanghai Maruka Computer Information Technology Co., Ltd, Shanghai, 200050, China

    Ran Tao

  3. Beijing Synchrotron Radiation Facility, Chinese Academy of Sciences, Beijing, 100049, China

    Wenhui Wang

  4. Chinese Spallation Neutron Source Science Center, Chinese Academy of Sciences, Dongguan, 523808, China

    Wenhui Wang

Authors
  1. Fei Zhao
    View author publications

    Search author on:PubMed Google Scholar

  2. Ran Tao
    View author publications

    Search author on:PubMed Google Scholar

  3. Wenhui Wang
    View author publications

    Search author on:PubMed Google Scholar

  4. Bo Cui
    View author publications

    Search author on:PubMed Google Scholar

  5. Yuting Xu
    View author publications

    Search author on:PubMed Google Scholar

  6. Qing Ai
    View author publications

    Search author on:PubMed Google Scholar

Contributions

The authors confirm contribution to the paper as follows: study conception and design: Fei Zhao, Qing Ai; data collection: Fei Zhao, Ran Tao; analysis and interpretation of results: Fei Zhao, Qing Ai; draft manuscript preparation: Fei Zhao, Qing Ai, Bo Cui, Yuting Xu; Supervision: Qing Ai, Ran Tao, Wenhui Wang. The final version of the manuscript approved by all authors.

Corresponding author

Correspondence to Qing Ai.

Ethics declarations

Ethical and informed consent for data used

The data used in this study were obtained through publicly available sources, and no ethical or informed consent considerations were required.

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, F., Tao, R., Wang, W. et al. Collaborative learning of supervision and correlation for generalized zero-shot extreme multi-label learning. Appl Intell 54, 6285–6298 (2024). https://doi.org/10.1007/s10489-024-05498-8

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s10489-024-05498-8

Keywords