{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T17:58:39Z","timestamp":1765821519375,"version":"3.48.0"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Digital Threats"],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>National Computer Security Incident Response Teams (CSIRTs) are established worldwide to coordinate responses to cyber security incidents at the national level. It is known that software tools (including open source ones) and public data are routinely used to facilitate incident response in national CSIRTs. However, there is a lack of an authoritative set of criteria that can be used for a systematic evaluation to decide which software tools and data sources should be used by national CSIRTs for incident response. A prior study identified a set of potential candidate criteria for such an evaluation. The study presented in this article aims to validate these candidate criteria empirically by asking staff members of several national CSIRTs how they perceive the candidate criteria\u2019s practical usefulness and readiness for deployment in national CSIRTs\u2019 operations. The study involved online semi-structured interviews with nine interviewees from nine national CSIRTs in Asia-Pacific, Africa and Europe. After validating the candidate criteria using semi-structured interviews with these nine interviewees, we applied the criteria to evaluate a selection of software tools and data sources by converting each criterion into one or more relevant metrics, such as \u2018measuring the time taken by a tool to produce results\u2019. Results from the study led to the following main findings: (1) all interviewees perceived the candidate criteria as practically useful for evaluating tools and data sources in the operations of national CSIRTs; (2) all interviewees agreed that the candidate criteria could be deployed in national CSIRTs and other types of CSIRTs and (3) the candidate criteria can be applied relatively easily in practice. These criteria are envisaged to help national CSIRTs select the most appropriate tools and data sources to facilitate effective incident response, improve their operational practices and improve the quality of wider security operations.<\/jats:p>","DOI":"10.1145\/3748267","type":"journal-article","created":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T14:04:03Z","timestamp":1757513043000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Validating a Set of Candidate Criteria for Evaluating Software Tools and Data Sources for National CSIRTs\u2019 Cyber Incident Responses"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-8603-9583","authenticated-orcid":false,"given":"Sharifah Roziah Binti","family":"Mohd Kassim","sequence":"first","affiliation":[{"name":"Institute of Cyber Security for Society (iCSS) and School of Computing, University of Kent, Canterbury, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5628-7328","authenticated-orcid":false,"given":"Shujun","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Cyber Security for Society (iCSS) and School of Computing, University of Kent, Canterbury, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1830-1587","authenticated-orcid":false,"given":"Budi","family":"Arief","sequence":"additional","affiliation":[{"name":"Institute of Cyber Security for Society (iCSS) and School of Computing, University of Kent, Canterbury, UK"}]}],"member":"320","published-online":{"date-parts":[[2025,12,15]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2015.08.001"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2022.01.364"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.npls.2016.01.001"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2001.0483"},{"key":"e_1_3_2_6_2","volume-title":"Enabling the Sustainability and Success of a National Computer Security Incident Response Team","author":"Bills Tracy","year":"2022","unstructured":"Tracy Bills, Brittany Manley, and James Lord. 2022. Enabling the Sustainability and Success of a National Computer Security Incident Response Team. Handbook. Carnegie Mellon University. Retrieved from https:\/\/resources.sei.cmu.edu\/asset_files\/Handbook\/2022_002_001_885865.pdf"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.5555\/800253.807736"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1280680.1280693"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.7748\/nr.4.3.5.s2"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2022.3145265"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X103884"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2458-14-53"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.345830"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.34190\/iccws.17.1.66"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.afjem.2017.08.001"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1361-3723(17)30013-1"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.46743\/2160-3715\/2015.2281"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Lisa M. Given. 2008. The SAGE Encyclopedia of Qualitative Research Methods. SAGE. Retrieved from https:\/\/uk.sagepub.com\/en-gb\/eur\/the-sage-encyclopedia-of-qualitative-research-methods\/book229805","DOI":"10.4135\/9781412963909"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.5555\/140207"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRACOS53680.2021.9701982"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1177\/1049732316665344"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/3187205"},{"key":"e_1_3_2_24_2","unstructured":"International Organization for Standardization (ISO). 2008. Software Engineering\u2014Software Product Quality Requirements and Evaluation (SQuaRE)\u2014Data Quality Model. Web Page. Retrieved from https:\/\/www.iso.org\/standard\/35736.html"},{"key":"e_1_3_2_25_2","unstructured":"International Organization for Standardization (ISO). 2011. Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014System and Software Quality Models. Web Page. Retrieved from https:\/\/www.iso.org\/standard\/35733.html"},{"key":"e_1_3_2_26_2","unstructured":"International Organization for Standardization (ISO). 2021. Information Technology\u2014Software Measurement\u2014Software Quality Measurement\u2014Automated Source Code Quality Measures. Web Page. Retrieved from https:\/\/www.iso.org\/en\/contents\/data\/standard\/08\/06\/80623.html"},{"key":"e_1_3_2_27_2","unstructured":"International Telecommunication Unit. 2023. Global Cybersecurity Index 2020. Web Page. Retrieved from https:\/\/www.itu.int\/epublications\/publication\/D-STR-GCI.01-2021-HTM-E"},{"key":"e_1_3_2_28_2","unstructured":"Internet Governance Forum (IGF). 2014. Internet Governance Forum (IGF) 2014: Best Practice Forum on Establishing and Supporting Computer Security Incident Response Teams (CSIRT) for Internet Security. Online Document. Retrieved from https:\/\/www.intgovforum.org\/cms\/documents\/best-practice-forums\/establishing-and-supporting-computer-emergency-response-teams-certs-for-internet-security\/409-bpf-2014-outcome-document-computer-security-incident-response-teams\/file"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/IMF.2009.13"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSC54232.2022.9888803"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27937-9_5"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-49059-1_33"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TELFOR48224.2019.8971040"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11379-1_1"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/IACS.2017.7921994"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Jim A. McCall Paul K. Richards and Gene F. Walters. 1977. Factors in Software Quality. Volume I. Concepts and Definitions of Software Quality. Technical Report ADA049014. General Electric Company. Retrieved from https:\/\/apps.dtic.mil\/sti\/citations\/ADA049014","DOI":"10.21236\/ADA049014"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISI.2008.4565058"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.5121\/ijsea.2014.5603"},{"key":"e_1_3_2_39_2","volume-title":"Qualitative Data Analysis: An Expanded Sourcebook","author":"Miles Matthew B.","year":"1994","unstructured":"Matthew B. Miles and A. Michael Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. SAGE."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.69554\/NFNP6432"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3609230"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.22364\/bjmc.2020.8.4.04"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TENCON.2016.7848750"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485983.3494872"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-78551-2_15"},{"key":"e_1_3_2_46_2","unstructured":"Briony J. Oates. 2005. Researching Information Systems and Computing. SAGE. Retrieved from https:\/\/us.sagepub.com\/en-us\/nam\/researching-information-systems-and-computing\/book226898"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/iThings-GreenCom-CPSCom-SmartData-Cybermatics50389.2020.00111"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1177\/160940690900800301"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2663887.2663899"},{"key":"e_1_3_2_50_2","unstructured":"Pawe\u0142 Pawlinski and Andrew Kompanek. 2016. Evaluating Threat Intelligence Feeds. Presentation Slides. Retrieved from https:\/\/www.first.org\/resources\/papers\/munich2016\/kompanek-pawlinski-evaluating-threat-ntelligence-feeds.pdf"},{"key":"e_1_3_2_51_2","volume-title":"The Threat Intelligence Handbook: Moving Toward a Security Intelligence Program","author":"Pokorny Zane","year":"2019","unstructured":"Zane Pokorny (Ed.). 2019. The Threat Intelligence Handbook: Moving Toward a Security Intelligence Program (2nd ed.). CyberEdge Group, LLC. Retrieved from https:\/\/go.recordedfuture.com\/book-2","edition":"2"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3429789.3429867"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-017-0574-8"},{"key":"e_1_3_2_54_2","unstructured":"Shadowserver Foundation. 2023. Shadowserver. Website. Retrieved from https:\/\/www.shadowserver.org\/"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3427787"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2015.71"},{"issue":"4","key":"e_1_3_2_57_2","first-page":"5634","article-title":"A comparative study of software quality models","volume":"5","author":"Suman Manoj Wadhwa","year":"2014","unstructured":"Manoj Wadhwa Suman and M. D. U. Rohtak. 2014. A comparative study of software quality models. International Journal of Computer Science and Information Technologies 5, 4 (2014), 5634\u20135638. Retrieved from https:\/\/ijcsit.com\/docs\/Volume%205\/vol5issue04\/ijcsit20140504177.pdf","journal-title":"International Journal of Computer Science and Information Technologies"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315067339"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/W-FiCloud.2018.00007"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00045"},{"key":"e_1_3_2_61_2","unstructured":"Martijn van der Heide. 2017. Establishing a CSIRT Version 1.2. Technical report. ThaiCERT and ETDA. Retrieved from https:\/\/www.first.org\/resources\/guides\/Establishing-CSIRT-v1.2.pdf"},{"key":"e_1_3_2_62_2","unstructured":"Michel van Eeten Qasim Lone Hadi Asghari TUD and Hadi Asghari. 2015. WP4 Evaluating and Incentivizing Botnet Mitigation 67 pages. Retrieved from https:\/\/www.acdc-project.eu\/wp-content\/uploads\/2015\/05\/ACDC_D4.2_Statistical_Evaluation_final.pdf"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-51974-2_58"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1353\/lib.2006.0053"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/IDAACS53288.2021.9660957"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.12928\/TELKOMNIKA.v18i5.16061"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372297.3420013"}],"container-title":["Digital Threats: Research and Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3748267","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T17:57:28Z","timestamp":1765821448000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3748267"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,15]]},"references-count":66,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3748267"],"URL":"https:\/\/doi.org\/10.1145\/3748267","relation":{},"ISSN":["2692-1626","2576-5337"],"issn-type":[{"type":"print","value":"2692-1626"},{"type":"electronic","value":"2576-5337"}],"subject":[],"published":{"date-parts":[[2025,12,15]]},"assertion":[{"value":"2025-05-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}