{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T17:05:35Z","timestamp":1773594335691,"version":"3.50.1"},"reference-count":20,"publisher":"Wiley","issue":"11","license":[{"start":{"date-parts":[[2004,4,20]],"date-time":"2004-04-20T00:00:00Z","timestamp":1082419200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Am. Soc. Inf. Sci."],"published-print":{"date-parts":[[2004,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In this paper, we focus on the effect of graded relevance on the results of interactive information retrieval (IR) experiments based on assigned search tasks in a test collection. A group of 26 subjects searched for four Text REtrieval Conference (TREC) topics using automatic and interactive query expansion based on relevance feedback. The TREC\u2010 and user\u2010suggested pools of relevant documents were reassessed on a four\u2010level relevance scale. The results show that the users could identify nearly all highly relevant documents and about half of the marginal ones. Users also selected a fair number of irrelevant documents for query expansion. The findings suggest that the effectiveness of query expansion is closely related to the searchers' success in retrieving and identifying highly relevant documents for feedback. The implications of the results on interpreting the findings of past experiments with liberal relevance thresholds are also discussed.<\/jats:p>","DOI":"10.1002\/asi.20046","type":"journal-article","created":{"date-parts":[[2004,4,20]],"date-time":"2004-04-20T20:01:02Z","timestamp":1082491262000},"page":"963-969","source":"Crossref","is-referenced-by-count":43,"title":["The influence of relevance levels on the effectiveness of interactive information retrieval"],"prefix":"10.1002","volume":"55","author":[{"given":"Pertti","family":"Vakkari","sequence":"first","affiliation":[]},{"given":"Eero","family":"Sormunen","sequence":"additional","affiliation":[]}],"member":"311","published-online":{"date-parts":[[2004,4,20]]},"reference":[{"key":"e_1_2_6_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<85::AID-ASI8>3.0.CO;2-Z"},{"key":"e_1_2_6_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3166.3197"},{"key":"e_1_2_6_4_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb050097"},{"key":"e_1_2_6_5_1","volume-title":"Experimental studies of relevance judgements: Final Report","author":"Cuadra C.","year":"1967"},{"key":"e_1_2_6_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(98)00041-7"},{"key":"e_1_2_6_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q"},{"key":"e_1_2_6_8_1","unstructured":"Hawking D.(2002). Overview of the TREC\u20109 Web Track. Retrieved January 20 2003 fromhttp:\/\/trec.nist.gov\/pubs\/trec9\/papers\/web9.pdf"},{"key":"e_1_2_6_9_1","unstructured":"Hersh W. &Over P.(2001). TREC\u20109 Interactive Track Report. In E.M. Voorhees & D.K. Harman (Eds.) The Ninth Text Retrieval Conference (TREC\u20109). Retrieved January 11 2003 fromhttp:\/\/trec.nist.gov\/pubs\/trec9\/papers\/t9irep.pdf"},{"key":"e_1_2_6_10_1","unstructured":"2000 ACM New York K. J\u00e4rvelin J. Kek\u00e4l\u00e4inen N. Belkin P. Ingwersen M.\u2010K. Leong IR evaluation methods for retrieving highly relevant documents 41 48"},{"key":"e_1_2_6_11_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10137"},{"key":"e_1_2_6_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00053-4"},{"key":"e_1_2_6_13_1","first-page":"199","volume-title":"Emerging frameworks and methods","author":"Ruthven I.","year":"2002"},{"key":"e_1_2_6_14_1","series-title":"Advances in Librarianship","first-page":"79","author":"Saracevic T.","year":"1976"},{"key":"e_1_2_6_15_1","unstructured":"2002 ACM New York E. Sormunen M. Beaulieu R. Baeza\u2010Yates S. Myaeng K. J\u0101rvelin Liberal relevance criteria of TREC\u2014Counting on negligible documents? 324 330"},{"key":"e_1_2_6_16_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007087"},{"key":"e_1_2_6_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"e_1_2_6_18_1","series-title":"Annual review of information science and technology","first-page":"413","author":"Vakkari P.","year":"2002"},{"key":"e_1_2_6_19_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007127"},{"key":"e_1_2_6_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_2_6_21_1","unstructured":"2001 ACM New York E. Vorhees W. Croft D. Harper D. Kraft J. Hobel Evaluation by highly relevant documents 74 82"}],"container-title":["Journal of the American Society for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fasi.20046","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/asi.20046","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,16]],"date-time":"2023-10-16T17:51:57Z","timestamp":1697478717000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/asi.20046"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2004,4,20]]},"references-count":20,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2004,9]]}},"alternative-id":["10.1002\/asi.20046"],"URL":"https:\/\/doi.org\/10.1002\/asi.20046","archive":["Portico"],"relation":{},"ISSN":["1532-2882","1532-2890"],"issn-type":[{"value":"1532-2882","type":"print"},{"value":"1532-2890","type":"electronic"}],"subject":[],"published":{"date-parts":[[2004,4,20]]}}}