{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:13:58Z","timestamp":1755839638887,"version":"3.41.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,1,22]],"date-time":"2024-01-22T00:00:00Z","timestamp":1705881600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ARC Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]},{"name":"ARC Training Centre for Information Resilience","award":["IC200100022"],"award-info":[{"award-number":["IC200100022"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Misinformation has been rapidly spreading online. The common approach to dealing with it is deploying expert fact-checkers who follow forensic processes to identify the veracity of statements. Unfortunately, such an approach does not scale well. To deal with this, crowdsourcing has been looked at as an opportunity to complement the work done by trained journalists. In this article, we look at the effect of presenting the crowd with evidence from others while judging the veracity of statements. We implement variants of the judgment task design to understand whether and how the presented evidence may or may not affect the way crowd workers judge truthfulness and their performance. Our results show that, in certain cases, the presented evidence and the way in which it is presented may mislead crowd workers who would otherwise be more accurate if judging independently from others. Those who make appropriate use of the provided evidence, however, can benefit from it and generate better judgments.<\/jats:p>","DOI":"10.1145\/3637872","type":"journal-article","created":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T11:50:18Z","timestamp":1702986618000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["On the Impact of Showing Evidence from Peers in Crowdsourced Truthfulness Assessments"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2654-6891","authenticated-orcid":false,"given":"Jiechen","family":"Xu","sequence":"first","affiliation":[{"name":"The University of Queensland, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7777-3592","authenticated-orcid":false,"given":"Lei","family":"Han","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6739-4145","authenticated-orcid":false,"given":"Shazia","family":"Sadiq","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7311-3693","authenticated-orcid":false,"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}]}],"member":"320","published-online":{"date-parts":[[2024,1,22]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abf4393"},{"key":"e_1_3_2_3_2","first-page":"9","volume-title":"ACM SIGIR Forum","author":"Alonso Omar","year":"2008","unstructured":"Omar Alonso, Daniel E. Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. In ACM SIGIR Forum, Vol. 42. ACM, New York, NY, USA, 9\u201315."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_2_5_2","first-page":"207","volume-title":"ECIR","author":"Barbera David La","year":"2020","unstructured":"David La Barbera, Kevin Roitero, Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 2020. Crowdsourcing truthfulness: The impact of judgment scale and assessor bias. In ECIR. Springer, 207\u2013214."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"2334","DOI":"10.1145\/3025453.3026044","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Chang Joseph Chee","year":"2017","unstructured":"Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2334\u20132346."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1108\/MD-03-2012-0156"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_3_2_9_2","first-page":"159","volume-title":"ASLD@ISWC","author":"Denaux Ronald","year":"2020","unstructured":"Ronald Denaux, Flavio Merenda, and Jos\u00e9 Manu\u00e9l G\u00f3mez-P\u00e9rez. 2020. Towards crowdsourcing tasks for accurate misinformation detection. In ASLD@ISWC. 159\u2013167."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1145\/2736277.2741685","volume-title":"Proceedings of the 24th International Conference on World Wide Web","author":"Difallah Djellel Eddine","year":"2015","unstructured":"Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, and Philippe Cudr\u00e9-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web. 238\u2013247."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/978-3-030-02922-7_14","volume-title":"Web Information Systems Engineering\u2013WISE 2018: 19th International Conference, Dubai, United Arab Emirates, November 12-15, 2018, Proceedings, Part I 19","author":"Dong Manqing","year":"2018","unstructured":"Manqing Dong, Lina Yao, Xianzhi Wang, Boualem Benatallah, Quan Z. Sheng, and Hao Huang. 2018. DUAL: A deep unified attention model with latent relation representations for fake news detection. In Web Information Systems Engineering\u2013WISE 2018: 19th International Conference, Dubai, United Arab Emirates, November 12-15, 2018, Proceedings, Part I 19. Springer, 199\u2013209."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_3_2_13_2","first-page":"32","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"4","author":"Drapeau Ryan","year":"2016","unstructured":"Ryan Drapeau, Lydia Chilton, Jonathan Bragg, and Daniel Weld. 2016. MicroTalk: Using argumentation to improve crowdsourcing accuracy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 4. 32\u201341."},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","first-page":"2114","DOI":"10.1145\/3531146.3534629","volume-title":"2022 ACM Conference on Fairness, Accountability, and Transparency","author":"Draws Tim","year":"2022","unstructured":"Tim Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, and Stefano Mizzaro. 2022. The effects of crowd worker biases in fact-checking tasks. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2114\u20132124."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512239"},{"key":"e_1_3_2_16_2","article-title":"A topic modeling comparison between LDA, NMF, Top2Vec, and BERTopic to demystify Twitter posts","volume":"7","author":"Egger Roman","year":"2022","unstructured":"Roman Egger and Joanne Yu. 2022. A topic modeling comparison between LDA, NMF, Top2Vec, and BERTopic to demystify Twitter posts. Frontiers in Sociology 7 (2022).","journal-title":"Frontiers in Sociology"},{"key":"e_1_3_2_17_2","first-page":"162","volume-title":"Proceedings of WSDM","author":"Eickhoff Carsten","year":"2018","unstructured":"Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. In Proceedings of WSDM (Marina Del Rey, CA, USA). ACM, New York, NY, USA, 162\u2013170."},{"key":"e_1_3_2_18_2","first-page":"1","volume-title":"Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI","author":"Finnerty Ailbhe","year":"2013","unstructured":"Ailbhe Finnerty, Pavel Kucherbaev, Stefano Tranquillini, and Gregorio Convertino. 2013. Keep it simple: Reward and task design in crowdsourcing. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI. 1\u20134."},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1145\/3078714.3078715","volume-title":"Proceedings of the 28th ACM Conference on Hypertext and Social Media","author":"Gadiraju Ujwal","year":"2017","unstructured":"Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2017. Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media. 5\u201314."},{"key":"e_1_3_2_20_2","article-title":"BERTopic: Neural topic modeling with a class-based TF-IDF procedure","author":"Grootendorst Maarten","year":"2022","unstructured":"Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).","journal-title":"arXiv preprint arXiv:2203.05794"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371857"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3476073"},{"key":"e_1_3_2_23_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI","author":"Hube Christoph","year":"2019","unstructured":"Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI. 1\u201312."},{"key":"e_1_3_2_24_2","first-page":"1113","volume-title":"Proceedings of the 52nd Annual Meeting of the ACL (Volume 1: Long Papers)","author":"Iyyer Mohit","year":"2014","unstructured":"Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the ACL (Volume 1: Long Papers). 1113\u20131122."},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1145\/2009916.2009947","volume-title":"Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Kazai Gabriella","year":"2011","unstructured":"Gabriella Kazai, Jaap Kamps, Marijn Koolen, and Natasa Milic-Frayling. 2011. Crowdsourcing for book search evaluation: Impact of hit design on comparative system ranking. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 205\u2013214."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531677"},{"key":"e_1_3_2_27_2","first-page":"43","article-title":"Computing Krippendorff\u2019s alpha reliability","author":"Krippendorff Klaus","year":"2007","unstructured":"Klaus Krippendorff. 2007. Computing Krippendorff\u2019s alpha reliability. Departmental Papers (ASC) (2007), 43.","journal-title":"Departmental Papers (ASC)"},{"key":"e_1_3_2_28_2","volume-title":"Proceedings of the 6th Workshop on Natural Language for Artificial Intelligence (NL4AI 2022) Co-located with 21st International Conference of the Italian Association for Artificial Intelligence (AI* IA 2022)","author":"Barbera David La","year":"2022","unstructured":"David La Barbera, Kevin Roitero, and Stefano Mizzaro. 2022. A hybrid human-in-the-loop framework for fact checking. In Proceedings of the 6th Workshop on Natural Language for Artificial Intelligence (NL4AI 2022) Co-located with 21st International Conference of the Italian Association for Artificial Intelligence (AI* IA 2022)."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445507"},{"key":"e_1_3_2_30_2","volume-title":"CIKM Workshops","author":"Maddalena Eddy","year":"2018","unstructured":"Eddy Maddalena, Davide Ceolin, and Stefano Mizzaro. 2018. Multidimensional news quality: A comparison of crowdsourcing and nichesourcing. In CIKM Workshops."},{"key":"e_1_3_2_31_2","first-page":"139","volume-title":"HCOMP 2016","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? Collecting annotator rationales for relevance judgments. In HCOMP 2016. 139\u2013148."},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1142\/9789812836304_0002","volume-title":"Handbook of Metadata, Semantics and Ontologies","author":"M\u00e9ndez Eva","year":"2014","unstructured":"Eva M\u00e9ndez and Seth van Hooland. 2014. Metadata typology and metadata uses. In Handbook of Metadata, Semantics and Ontologies. World Scientific, 9\u201339."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797620939054"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806781116"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1109\/CSCWD.2019.8791903","volume-title":"2019 IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD)","author":"Pinto Marcos Rodrigues","year":"2019","unstructured":"Marcos Rodrigues Pinto, Yuri Oliveira de Lima, Carlos Eduardo Barbosa, and Jano Moreira de Souza. 2019. Towards fact-checking through crowdsourcing. In 2019 IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD). IEEE, 494\u2013499."},{"key":"e_1_3_2_36_2","first-page":"45","volume-title":"Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks","author":"\u0158eh\u016f\u0159ek Radim","year":"2010","unstructured":"Radim \u0158eh\u016f\u0159ek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45\u201350. http:\/\/is.muni.cz\/publication\/884893\/en"},{"key":"e_1_3_2_37_2","first-page":"439","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference","author":"Roitero Kevin","year":"2020","unstructured":"Kevin Roitero, Michael Soprano, Shaoyang Fan, Damiano Spina, Stefano Mizzaro, and Gianluca Demartini. 2020. Can the crowd identify misinformation objectively? The effects of judgment scale and assessor\u2019s background. In Proceedings of the 43rd International ACM SIGIR Conference. 439\u2013448."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3290994"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102710"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1109\/DSAA.2017.61","volume-title":"2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA)","author":"Syed Shaheen","year":"2017","unstructured":"Shaheen Syed and Marco Spruit. 2017. Full-text or abstract? Examining topic coherence scores using latent Dirichlet allocation. In 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 165\u2013174."},{"key":"e_1_3_2_41_2","article-title":"Some like it hoax: Automated fake news detection in social networks","author":"Tacchini Eugenio","year":"2017","unstructured":"Eugenio Tacchini, Gabriele Ballarin, Marco L. Della Vedova, Stefano Moret, and Luca De Alfaro. 2017. Some like it hoax: Automated fake news detection in social networks. arXiv preprint arXiv:1704.07506 (2017).","journal-title":"arXiv preprint arXiv:1704.07506"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3371925"},{"key":"e_1_3_2_43_2","article-title":"\u201cLiar, liar pants on fire\u201d: A new benchmark dataset for fake news detection","author":"Wang William Yang","year":"2017","unstructured":"William Yang Wang. 2017. \u201cLiar, liar pants on fire\u201d: A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648 (2017).","journal-title":"arXiv preprint arXiv:1705.00648"},{"issue":"1","key":"e_1_3_2_44_2","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1109\/MIC.2020.3032410","article-title":"Effect of conformity on perceived trustworthiness of news in social media","volume":"25","author":"Wijenayake Senuri","year":"2020","unstructured":"Senuri Wijenayake, Danula Hettiachchi, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves. 2020. Effect of conformity on perceived trustworthiness of news in social media. IEEE Internet Computing 25, 1 (2020), 12\u201319.","journal-title":"IEEE Internet Computing"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33015644"},{"key":"e_1_3_2_46_2","first-page":"1445","volume-title":"Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing","author":"Zhu Haiyi","year":"2014","unstructured":"Haiyi Zhu, Steven P. Dow, Robert E. Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing. 1445\u20131455."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637872","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637872","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:18Z","timestamp":1750286958000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637872"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,22]]},"references-count":45,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3637872"],"URL":"https:\/\/doi.org\/10.1145\/3637872","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2024,1,22]]},"assertion":[{"value":"2023-03-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}