{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T01:08:32Z","timestamp":1759972112425,"version":"build-2065373602"},"reference-count":40,"publisher":"Wiley","issue":"9","license":[{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["asistdl.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Asso for Info Science &amp; Tech"],"published-print":{"date-parts":[[2022,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>LEarning TO Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document\u2010topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document\u2010topic pairs instead of single\u2010valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provided by relevance judgments distributions and show how they can be applied both to neural and gradient boosting machine (GBM) architectures. Moreover, we show how training a LETOR model on a sampled version of the relevance judgments from certain probability distributions can improve its performance when relying either on traditional or probabilistic loss functions. Finally, we validate our hypothesis on real\u2010world crowdsourced relevance judgments distributions. Overall, we observe that relying on relevance judgments distributions to train different LETOR models can boost their performance and even outperform strong baselines such as LambdaMART on several test collections.<\/jats:p>","DOI":"10.1002\/asi.24629","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T05:20:34Z","timestamp":1646112034000},"page":"1236-1252","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Learning to rank from relevance judgments distributions"],"prefix":"10.1002","volume":"73","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1701-7805","authenticated-orcid":false,"given":"Alberto","family":"Purpura","sequence":"first","affiliation":[{"name":"Department of Information Engineering University of Padua  Padua"}]},{"given":"Gianmaria","family":"Silvello","sequence":"additional","affiliation":[{"name":"Department of Information Engineering University of Padua  Padua"}]},{"given":"Gian Antonio","family":"Susto","sequence":"additional","affiliation":[{"name":"Department of Information Engineering University of Padua  Padua"}]}],"member":"311","published-online":{"date-parts":[[2022,3]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2020.2996134"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00904ED1V01Y201903ICR066"},{"key":"e_1_2_9_4_1","first-page":"3438","volume-title":"Proceedings of the 28th international conference on neural information processing systems","author":"Anoop K.","year":"2015"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-45528-0"},{"key":"e_1_2_9_6_1","unstructured":"Bruch S.(2019).An alternative cross entropy loss for learning\u2010to\u2010rank. arXiv:1911.09798."},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371844"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331347"},{"volume-title":"From RankNet to LambdaRank to LambdaMART: An overview","year":"2010","author":"Burges C.","key":"e_1_2_9_9_1"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"volume-title":"Proceedings of the 22nd international conference on neural information processing systems","year":"2009","author":"Chen W.","key":"e_1_2_9_11_1"},{"key":"e_1_2_9_12_1","unstructured":"Devlin J. Chang M. Lee K. &Toutanova K.(2018).Bert: Pre\u2010training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805."},{"key":"e_1_2_9_13_1","first-page":"289","article-title":"How do interval scales help us with better understanding IR evaluation measures?","volume":"23","author":"Ferrante M.","year":"2020","journal-title":"IR Journal"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58219-7_2"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"volume-title":"Proceedings of 25th international conference of the CIKM","year":"2016","author":"Guo J.","key":"e_1_2_9_16_1"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_16"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2866571"},{"key":"e_1_2_9_19_1","first-page":"3149","volume-title":"Proceedings of the 31st international conference on neural information processing systems","author":"Ke G.","year":"2017"},{"volume-title":"Joint European conference on machine learning and knowledge discovery in databases","year":"2019","author":"K\u00f6ppel M.","key":"e_1_2_9_20_1"},{"key":"e_1_2_9_21_1","unstructured":"Lease M.&Kazai G.(2011).Overview of the TREC 2011 crowdsourcing track. Paper presented at the proceedings of TREC National Institute of Standards and Technology."},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00799"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331317"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9321-y"},{"key":"e_1_2_9_25_1","unstructured":"Pang L. Lan Y. Guo J. Xu J. &Cheng X.(2016).A study of matchpyramid models on ad\u2010hoc retrieval. arXiv:1606.04648."},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3409256.3409819"},{"key":"e_1_2_9_27_1","unstructured":"Pobrotyn P. Bartczak T. Synowiec M. Bia\u0142obrzeski R. &Bojar J.(2020).Context\u2010aware learning to rank with self\u2010attention. arXiv:2005.10084."},{"key":"e_1_2_9_28_1","unstructured":"Qin T.&Liu T.(2013).Introducing letor 4.0 datasets. arXiv:1306.2597."},{"key":"e_1_2_9_29_1","first-page":"375","article-title":"A general approximation framework for direct optimization of information retrieval measures","volume":"4","author":"Qin T.","year":"2010","journal-title":"IR Journal"},{"key":"e_1_2_9_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9123-y"},{"volume-title":"Proceedings of the 20th Australasian database conference","year":"2009","author":"Ravana S. D.","key":"e_1_2_9_31_1"},{"key":"e_1_2_9_32_1","doi-asserted-by":"crossref","unstructured":"Smucker M. Kazai G. &Lease M.(2012).Overview of the TREC 2012 crowdsourcing track. Paper presented at the proceedings of TREC.","DOI":"10.6028\/NIST.SP.500-298.crowd-overview"},{"key":"e_1_2_9_33_1","doi-asserted-by":"crossref","unstructured":"Smucker M. Kazai G. &Lease M.(2013).Overview of the TREC 2013 crowdsourcing track. Paper presented at the proceedings of TREC.","DOI":"10.6028\/NIST.SP.500-302.crowd-overview"},{"key":"e_1_2_9_34_1","unstructured":"Sun S.&Duh K.(2020).Modeling document interactions for learning to rank with regularized self\u2010attention. arXiv:2005.03932."},{"key":"e_1_2_9_35_1","first-page":"757","article-title":"A cross\u2010benchmark comparison of 87 learning to rank methods","volume":"51","author":"Tax N.","year":"2015","journal-title":"IP&M"},{"volume-title":"31st conference on neural information processing system","year":"2017","author":"Vaswani A.","key":"e_1_2_9_36_1"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390306"},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3234944.3234968"},{"volume-title":"Proceedings of ICLR","year":"2021","author":"Zhen Q.","key":"e_1_2_9_39_1"},{"key":"e_1_2_9_40_1","doi-asserted-by":"crossref","unstructured":"Zhuang H. Wang X. Bendersky M. Grushetsky A. Wu Y. Mitrichev P. Sterling E. Bell N. Ravina W. &Qian H.(2020).Interpretable learning\u2010to\u2010rank with generalized additive models. arXiv:2005.02553.","DOI":"10.1145\/3437963.3441796"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401333"}],"container-title":["Journal of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/asi.24629","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/asi.24629","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/asi.24629","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T19:25:47Z","timestamp":1759951547000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/asi.24629"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3]]},"references-count":40,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2022,9]]}},"alternative-id":["10.1002\/asi.24629"],"URL":"https:\/\/doi.org\/10.1002\/asi.24629","archive":["Portico"],"relation":{},"ISSN":["2330-1635","2330-1643"],"issn-type":[{"type":"print","value":"2330-1635"},{"type":"electronic","value":"2330-1643"}],"subject":[],"published":{"date-parts":[[2022,3]]},"assertion":[{"value":"2021-05-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-11","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}