{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T00:39:08Z","timestamp":1768351148604,"version":"3.49.0"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,4,17]],"date-time":"2020-04-17T00:00:00Z","timestamp":1587081600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA8750-15-2-0104"],"award-info":[{"award-number":["FA8750-15-2-0104"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007515","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1801545, EDU-1319147"],"award-info":[{"award-number":["CNS-1801545, EDU-1319147"]}],"id":[{"id":"10.13039\/100007515","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000161","name":"National Institute of Standards and Technology","doi-asserted-by":"publisher","award":["70NANB15H330"],"award-info":[{"award-number":["70NANB15H330"]}],"id":[{"id":"10.13039\/100000161","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Priv. Secur."],"published-print":{"date-parts":[[2020,5,31]]},"abstract":"<jats:p>Typical security contests focus on breaking or mitigating the impact of buggy systems. We present the Build-it, Break-it, Fix-it (BIBIFI) contest, which aims to assess the ability to securely build software, not just break it. In BIBIFI, teams build specified software with the goal of maximizing correctness, performance, and security. The latter is tested when teams attempt to break other teams\u2019 submissions. Winners are chosen from among the best builders and the best breakers. BIBIFI was designed to be open-ended\u2014teams can use any language, tool, process, and so on, that they like. As such, contest outcomes shed light on factors that correlate with successfully building secure software and breaking insecure software. We ran three contests involving a total of 156 teams and three different programming problems. Quantitative analysis from these contests found that the most efficient build-it submissions used C\/C++, but submissions coded in a statically type safe language were 11\u00d7 less likely to have a security flaw than C\/C++ submissions. Break-it teams that were also successful build-it teams were significantly better at finding security bugs.<\/jats:p>","DOI":"10.1145\/3383773","type":"journal-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T07:04:40Z","timestamp":1588575880000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Build It, Break It, Fix It"],"prefix":"10.1145","volume":"23","author":[{"given":"James","family":"Parker","sequence":"first","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Michael","family":"Hicks","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Andrew","family":"Ruef","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Michelle L.","family":"Mazurek","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Dave","family":"Levin","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Daniel","family":"Votipka","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]},{"given":"Piotr","family":"Mardziel","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University Silicon Valley, CA, USA"}]},{"given":"Kelsey R.","family":"Fulton","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,4,17]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"ICPC Foundation. 2018. ACM-ICPC International Collegiate Programming Contest. Retrieved from http:\/\/icpc.baylor.edu.  ICPC Foundation. 2018. ACM-ICPC International Collegiate Programming Contest. Retrieved from http:\/\/icpc.baylor.edu."},{"key":"e_1_2_1_2_1","unstructured":"BSIMM. 2020. Building Security In Maturity Model (BSIMM). Retrieved from http:\/\/bsimm.com.  BSIMM. 2020. Building Security In Maturity Model (BSIMM). Retrieved from http:\/\/bsimm.com."},{"key":"e_1_2_1_3_1","unstructured":"DEF CON Communications. 2018. Capture the Flag Archive. Retrieved from https:\/\/www.defcon.org\/html\/links\/dc-ctf.html.  DEF CON Communications. 2018. Capture the Flag Archive. Retrieved from https:\/\/www.defcon.org\/html\/links\/dc-ctf.html."},{"key":"e_1_2_1_4_1","volume-title":"DEF CON Hacking Conference.","author":"Communications DEF CON","year":"2010","unstructured":"DEF CON Communications . 2010 . DEF CON Hacking Conference. Retrieved from http:\/\/www.defcon.org. DEF CON Communications. 2010. DEF CON Hacking Conference. Retrieved from http:\/\/www.defcon.org."},{"key":"e_1_2_1_5_1","unstructured":"Git. 2020. Git \u2013 distributed version control management system. Retrieved from http:\/\/git-scm.com.  Git. 2020. Git \u2013 distributed version control management system. Retrieved from http:\/\/git-scm.com."},{"key":"e_1_2_1_6_1","unstructured":"Google. 2020. Google Code Jam. Retrieved from http:\/\/code.google.com\/codejam.  Google. 2020. Google Code Jam. Retrieved from http:\/\/code.google.com\/codejam."},{"key":"e_1_2_1_7_1","unstructured":"ICFP Programming Contest. 2019. Retrieved from http:\/\/icfpcontest.org.  ICFP Programming Contest. 2019. Retrieved from http:\/\/icfpcontest.org."},{"key":"e_1_2_1_8_1","unstructured":"Federal Business Council. 2012. Maryland Cyber Challenge 8 Competition. Retrieved from http:\/\/www.fbcinc.com\/e\/cybermdconference\/competitorinfo.aspx.  Federal Business Council. 2012. Maryland Cyber Challenge 8 Competition. Retrieved from http:\/\/www.fbcinc.com\/e\/cybermdconference\/competitorinfo.aspx."},{"key":"e_1_2_1_9_1","unstructured":"TOPCODER. 2020. Top Coder competitions. Retrieved from http:\/\/apps.topcoder.com\/wiki\/display\/tc\/Algorithm+Overview.  TOPCODER. 2020. Top Coder competitions. Retrieved from http:\/\/apps.topcoder.com\/wiki\/display\/tc\/Algorithm+Overview."},{"key":"e_1_2_1_10_1","unstructured":"Michael Snoyman. 2020. Yesod Web Framework for Haskell. Retrieved from http:\/\/www.yesodweb.com.  Michael Snoyman. 2020. Yesod Web Framework for Haskell. Retrieved from http:\/\/www.yesodweb.com."},{"key":"e_1_2_1_11_1","unstructured":"American Fuzzing Lop (AFL). 2018. Retrieved from http:\/\/lcamtuf.coredump.cx\/afl\/.  American Fuzzing Lop (AFL). 2018. Retrieved from http:\/\/lcamtuf.coredump.cx\/afl\/."},{"key":"e_1_2_1_12_1","article-title":"Control-flow integrity principles, implementations, and applications","volume":"13","author":"Abadi Mart\u00edn","year":"2009","unstructured":"Mart\u00edn Abadi , Mihai Budiu , \u00dalfar Erlingsson , and Jay Ligatti . 2009 . Control-flow integrity principles, implementations, and applications . ACM Trans. Info. Syst. Secur. 13 , 1 (2009), 4:1--4:40. Mart\u00edn Abadi, Mihai Budiu, \u00dalfar Erlingsson, and Jay Ligatti. 2009. Control-flow integrity principles, implementations, and applications. ACM Trans. Info. Syst. Secur. 13, 1 (2009), 4:1--4:40.","journal-title":"ACM Trans. Info. Syst. Secur."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the IEEE Symposium on Security and Privacy (SP\u201917)","author":"Acar Y.","unstructured":"Y. Acar , M. Backes , S. Fahl , S. Garfinkel , D. Kim , M. L. Mazurek , and C. Stransky . 2017. Comparing the usability of cryptographic APIs . In Proceedings of the IEEE Symposium on Security and Privacy (SP\u201917) . Y. Acar, M. Backes, S. Fahl, S. Garfinkel, D. Kim, M. L. Mazurek, and C. Stransky. 2017. Comparing the usability of cryptographic APIs. In Proceedings of the IEEE Symposium on Security and Privacy (SP\u201917)."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201917)","author":"Acar Yasemin","year":"2017","unstructured":"Yasemin Acar , Christian Stransky , Dominik Wermke , Michelle L. Mazurek , and Sascha Fahl . 2017 . Security developer studies with GitHub users: Exploring a convenience sample . In Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201917) . Yasemin Acar, Christian Stransky, Dominik Wermke, Michelle L. Mazurek, and Sascha Fahl. 2017. Security developer studies with GitHub users: Exploring a convenience sample. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201917)."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE Secure Development Conference (SecDev\u201917)","author":"Acar Yasemin","unstructured":"Yasemin Acar , Christian Stransky , Dominik Wermke , Charles Weir , Michelle L. Mazurek , and Sascha Fahl . [n.d.]. Developers need support too: A survey of security advice for software developers . In Proceedings of the IEEE Secure Development Conference (SecDev\u201917) . Yasemin Acar, Christian Stransky, Dominik Wermke, Charles Weir, Michelle L. Mazurek, and Sascha Fahl. [n.d.]. Developers need support too: A survey of security advice for software developers. In Proceedings of the IEEE Secure Development Conference (SecDev\u201917)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140241.3140253"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 2nd European Workshop on Usable Security (EuroUSEC\u201917)","author":"Becker Ingolf","unstructured":"Ingolf Becker , Simon Parkin , and M. Angela Sasse . 2017. Finding security champions in blends of security culture . In Proceedings of the 2nd European Workshop on Usable Security (EuroUSEC\u201917) . Internet Society. Ingolf Becker, Simon Parkin, and M. Angela Sasse. 2017. Finding security champions in blends of security culture. In Proceedings of the 2nd European Workshop on Usable Security (EuroUSEC\u201917). Internet Society."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33481-8_9"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the USENIX Workshop on Advances in Security Education (ASE\u201918)","author":"Bock Kevin","year":"2018","unstructured":"Kevin Bock , George Hughey , and Dave Levin . 2018 . King of the hill: A novel cybersecurity competition for teaching penetration testing . In Proceedings of the USENIX Workshop on Advances in Security Education (ASE\u201918) . Kevin Bock, George Hughey, and Dave Levin. 2018. King of the hill: A novel cybersecurity competition for teaching penetration testing. In Proceedings of the USENIX Workshop on Advances in Security Education (ASE\u201918)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00265-010-1029-6"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)","author":"Chapman Peter","year":"2014","unstructured":"Peter Chapman , Jonathan Burket , and David Brumley . 2014 . PicoCTF: A game-based computer security competition for high school students . In Proceedings of the USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914) . Peter Chapman, Jonathan Burket, and David Brumley. 2014. PicoCTF: A game-based computer security competition for high school students. In Proceedings of the USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)."},{"key":"e_1_2_1_23_1","volume-title":"Secure Programming with Static Analysis","author":"Chess Brian","unstructured":"Brian Chess and Jacob West . 2007. Secure Programming with Static Analysis . Addison-Wesley . Brian Chess and Jacob West. 2007. Secure Programming with Static Analysis. Addison-Wesley."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14215-4_8"},{"key":"e_1_2_1_25_1","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"Cohen Jacob","unstructured":"Jacob Cohen . 1988. Statistical Power Analysis for the Behavioral Sciences . Lawrence Erlbaum Associates . Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1107622.1107627"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2006.110"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2011.51"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2076732.2076740"},{"key":"e_1_2_1_30_1","volume-title":"CanSecWest Applied Security Conference.","unstructured":"dragostech. com inc. [n.d.] . CanSecWest Applied Security Conference. Retrieved from http:\/\/cansecwest.com. dragostech.com inc. [n.d.]. CanSecWest Applied Security Conference. Retrieved from http:\/\/cansecwest.com."},{"key":"e_1_2_1_31_1","volume-title":"Computer security competitions: Expanding educational outcomes. Secur. Privacy 11, 4","author":"Eagle Chris","year":"2013","unstructured":"Chris Eagle . 2013. Computer security competitions: Expanding educational outcomes. Secur. Privacy 11, 4 ( 2013 ). Chris Eagle. 2013. Computer security competitions: Expanding educational outcomes. Secur. Privacy 11, 4 (2013)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36563-8_14"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508859.2516693"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508859.2516655"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/2002168.2002177"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382196.2382204"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2010.51"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2005.120"},{"key":"e_1_2_1_39_1","volume-title":"Writing Secure Code","author":"Howard Michael","unstructured":"Michael Howard and David LeBlanc . 2003. Writing Secure Code . Microsoft Press . Michael Howard and David LeBlanc. 2003. Writing Secure Code. Microsoft Press."},{"key":"e_1_2_1_40_1","volume-title":"The Security Development Lifecycle","author":"Howard Michael","unstructured":"Michael Howard and Steve Lipner . 2006. The Security Development Lifecycle . Microsoft Press . Michael Howard and Steve Lipner. 2006. The Security Development Lifecycle. Microsoft Press."},{"key":"e_1_2_1_41_1","unstructured":"Queena Kim. 2014. Want to learn cybersecurity? Head to Def Con. Retrieved from http:\/\/www.marketplace.org\/2014\/08\/25\/tech\/want-learn-cybersecurity-head-def-con.  Queena Kim. 2014. Want to learn cybersecurity? Head to Def Con. Retrieved from http:\/\/www.marketplace.org\/2014\/08\/25\/tech\/want-learn-cybersecurity-head-def-con."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_1_43_1","volume-title":"Software Security: Building Security In","author":"McGraw Gary","year":"2006","unstructured":"Gary McGraw . 2006 . Software Security: Building Security In . Addison-Wesley . Gary McGraw. 2006. Software Security: Building Security In. Addison-Wesley."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the USENIX Security Symposium.","author":"Molnar David","unstructured":"David Molnar , Xue Cong Li , and David A. Wagner . 2009. Dynamic test generation to find integer bugs in x86 binary Linux programs . In Proceedings of the USENIX Security Symposium. David Molnar, Xue Cong Li, and David A. Wagner. 2009. Dynamic test generation to find integer bugs in x86 binary Linux programs. In Proceedings of the USENIX Security Symposium."},{"key":"e_1_2_1_45_1","volume-title":"A note on a general definition of the coefficient of determination. Biometrika 78, 3 (09","author":"Nagelkerke N. J. D.","year":"1991","unstructured":"N. J. D. Nagelkerke . 1991. A note on a general definition of the coefficient of determination. Biometrika 78, 3 (09 1991 ), 691--692. N. J. D. Nagelkerke. 1991. A note on a general definition of the coefficient of determination. Biometrika 78, 3 (09 1991), 691--692."},{"key":"e_1_2_1_46_1","unstructured":"National Collegiate Cyber Defense Competition. [n.d.]. Retrieved from http:\/\/www.nationalccdc.org.  National Collegiate Cyber Defense Competition. [n.d.]. Retrieved from http:\/\/www.nationalccdc.org."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2664243.2664254"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Oliveira Daniela Seabra","year":"2018","unstructured":"Daniela Seabra Oliveira , Tian Lin , Muhammad Sajidur Rahman , Rad Akefirad , Donovan Ellis , Eliany Perez , Rahul Bobhate , Lois A. DeLong , Justin Cappos , and Yuriy Brun . 2018 . API blindspots: Why experienced developers write vulnerable code . In Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201918) . Daniela Seabra Oliveira, Tian Lin, Muhammad Sajidur Rahman, Rad Akefirad, Donovan Ellis, Eliany Perez, Rahul Bobhate, Lois A. DeLong, Justin Cappos, and Yuriy Brun. 2018. API blindspots: Why experienced developers write vulnerable code. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS\u201918)."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Oliveira Daniela Seabra","year":"2018","unstructured":"Daniela Seabra Oliveira , Tian Lin , Muhammad Sajidur Rahman , Rad Akefirad , Donovan Ellis , Eliany Perez , Rahul Bobhate , Lois A. DeLong , Justin Cappos , and Yuriy Brun . 2018 . API blindspots: Why experienced developers write vulnerable code . In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918) . Daniela Seabra Oliveira, Tian Lin, Muhammad Sajidur Rahman, Rad Akefirad, Donovan Ellis, Eliany Perez, Rahul Bobhate, Lois A. DeLong, Justin Cappos, and Yuriy Brun. 2018. API blindspots: Why experienced developers write vulnerable code. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)."},{"key":"e_1_2_1_50_1","unstructured":"OWASP. 2010. Secure Coding Practices - Quick Reference Guide. Retrieved from https:\/\/www.owasp.org\/images\/0\/08\/OWASP_SCP_Quick_Reference_Guide_v2.pdf.  OWASP. 2010. Secure Coding Practices - Quick Reference Guide. Retrieved from https:\/\/www.owasp.org\/images\/0\/08\/OWASP_SCP_Quick_Reference_Guide_v2.pdf."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290388"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-54494-5_3"},{"key":"e_1_2_1_53_1","unstructured":"Polytechnic Institute of New York University. [n.d.]. CSAW\u2014CyberSecurity Competition 2012. Retrieved from http:\/\/www.poly.edu\/csaw2012\/csaw-CTF.  Polytechnic Institute of New York University. [n.d.]. CSAW\u2014CyberSecurity Competition 2012. Retrieved from http:\/\/www.poly.edu\/csaw2012\/csaw-CTF."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.22"},{"key":"e_1_2_1_55_1","unstructured":"psql [n.d.]. PostgreSQL: The world\u2019s most advanced open source database. Retrieved from http:\/\/www.postgresql.org.  psql [n.d.]. PostgreSQL: The world\u2019s most advanced open source database. Retrieved from http:\/\/www.postgresql.org."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978382"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201915)","author":"Ruef Andrew","year":"2015","unstructured":"Andrew Ruef , Michael Hicks , James Parker , Dave Levin , Atif Memon , Jandelyn Plane , and Piotr Mardziel . 2015 . Build it break it: Measuring and comparing development security . In Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201915) . Andrew Ruef, Michael Hicks, James Parker, Dave Levin, Atif Memon, Jandelyn Plane, and Piotr Mardziel. 2015. Build it break it: Measuring and comparing development security. In Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201915)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1975.9939"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2013.6698898"},{"key":"e_1_2_1_60_1","volume-title":"Secure Coding in C and C++","author":"Seacord Robert C.","unstructured":"Robert C. Seacord . 2013. Secure Coding in C and C++ . Addison-Wesley . Robert C. Seacord. 2013. Secure Coding in C and C++. Addison-Wesley."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2034675.2034688"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201917)","author":"Stransky Christian","year":"2017","unstructured":"Christian Stransky , Yasemin Acar , Duc Cuong Nguyen , Dominik Wermke , Doowon Kim , Elissa M. Redmiles , Michael Backes , Simson Garfinkel , Michelle L. Mazurek , and Sascha Fahl . 2017 . Lessons learned from using an online platform to conduct large-scale, online controlled security experiments with software developers . In Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201917) . Christian Stransky, Yasemin Acar, Duc Cuong Nguyen, Dominik Wermke, Doowon Kim, Elissa M. Redmiles, Michael Backes, Simson Garfinkel, Michelle L. Mazurek, and Sascha Fahl. 2017. Lessons learned from using an online platform to conduct large-scale, online controlled security experiments with software developers. In Proceedings of the International Conference on Cyber Security for Emerging Technologies (CSET\u201917)."},{"key":"e_1_2_1_63_1","volume-title":"ATM logic attacks: scenarios","author":"Technologies Positive","year":"2018","unstructured":"Positive Technologies . 2018. ATM logic attacks: scenarios , 2018 . Retrieved from https:\/\/www.ptsecurity.com\/upload\/corporate\/ww-en\/analytics\/ATM-Vulnerabilities-2018-eng.pdf. Positive Technologies. 2018. ATM logic attacks: scenarios, 2018. Retrieved from https:\/\/www.ptsecurity.com\/upload\/corporate\/ww-en\/analytics\/ATM-Vulnerabilities-2018-eng.pdf."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127005.3127014"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201917)","author":"Trickel Erik","year":"2017","unstructured":"Erik Trickel , Francesco Disperati , Eric Gustafson , Faezeh Kalantari , Mike Mabey , Naveen Tiwari , Yeganeh Safaei , Adam Doup\u00e9 , and Giovanni Vigna . 2017 . Shell we play a game? CTF-as-a-service for security education . In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201917) . Erik Trickel, Francesco Disperati, Eric Gustafson, Faezeh Kalantari, Mike Mabey, Naveen Tiwari, Yeganeh Safaei, Adam Doup\u00e9, and Giovanni Vigna. 2017. Shell we play a game? CTF-as-a-service for security education. In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201917)."},{"key":"e_1_2_1_66_1","unstructured":"\u00dalfar Erlingsson. 2012. personal communication stating that CFI was not deployed at Microsoft due to its overhead exceeding 10%.  \u00dalfar Erlingsson. 2012. personal communication stating that CFI was not deployed at Microsoft due to its overhead exceeding 10%."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238200"},{"key":"e_1_2_1_68_1","volume-title":"Building Secure Software: How to Avoid Security Problems the Right Way","author":"Viega John","unstructured":"John Viega and Gary McGraw . 2001. Building Secure Software: How to Avoid Security Problems the Right Way . Addison-Wesley . John Viega and Gary McGraw. 2001. Building Secure Software: How to Avoid Security Problems the Right Way. Addison-Wesley."},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIXSecurity\u201920)","author":"Votipka Daniel","year":"2020","unstructured":"Daniel Votipka , Kelsey R. Fulton , James Parker , Matthew Hou , Michelle L. Mazurek , and Michael Hicks . 2020 . Understanding security mistakes developers make: Qualitative analysis from build it, break it, fix it . In Proceedings of the 29th USENIX Security Symposium (USENIXSecurity\u201920) . USENIX Association. Daniel Votipka, Kelsey R. Fulton, James Parker, Matthew Hou, Michelle L. Mazurek, and Michael Hicks. 2020. Understanding security mistakes developers make: Qualitative analysis from build it, break it, fix it. In Proceedings of the 29th USENIX Security Symposium (USENIXSecurity\u201920). USENIX Association."},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the IEEE IEEE Symposium on Security and Privacy (S8P\u201918)","author":"Votipka D.","unstructured":"D. Votipka , R. Stevens , E. Redmiles , J. Hu , and M. Mazurek . 2018. Hackers vs. testers: A comparison of software vulnerability discovery processes . In Proceedings of the IEEE IEEE Symposium on Security and Privacy (S8P\u201918) . D. Votipka, R. Stevens, E. Redmiles, J. Hu, and M. Mazurek. 2018. Hackers vs. testers: A comparison of software vulnerability discovery processes. In Proceedings of the IEEE IEEE Symposium on Security and Privacy (S8P\u201918)."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.32"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.14722\/eurousec.2017.23002"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201918)","author":"Wi SeongIl","year":"2018","unstructured":"SeongIl Wi , Jaeseung Choi , and Sang Kil Cha . 2018 . Git-based CTF: A simple and effective approach to organizing in-course attack-and-defense security competition . In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201918) . SeongIl Wi, Jaeseung Choi, and Sang Kil Cha. 2018. Git-based CTF: A simple and effective approach to organizing in-course attack-and-defense security competition. In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201918)."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595676.1595691"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing.","author":"Xie J.","unstructured":"J. Xie , H. R. Lipford , and B. Chu . 2011. Why do programmers make security errors? In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing. Retrieved from http:\/\/ieeexplore.ieee.org\/xpls\/abs_all.jsp?arnumber=6070393. J. Xie, H. R. Lipford, and B. Chu. 2011. Why do programmers make security errors? In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing. Retrieved from http:\/\/ieeexplore.ieee.org\/xpls\/abs_all.jsp?arnumber=6070393."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3284557.3284743"},{"key":"e_1_2_1_77_1","volume-title":"Proceedings of the International Conference on Big Data and Smart Computing (BigComp\u201916)","author":"Yang Joonseok","year":"2016","unstructured":"Joonseok Yang , Duksan Ryu , and Jongmoon Baik . 2016 . Improving vulnerability prediction accuracy with secure coding standard violation measures . In Proceedings of the International Conference on Big Data and Smart Computing (BigComp\u201916) . Joonseok Yang, Duksan Ryu, and Jongmoon Baik. 2016. Improving vulnerability prediction accuracy with secure coding standard violation measures. In Proceedings of the International Conference on Big Data and Smart Computing (BigComp\u201916)."}],"container-title":["ACM Transactions on Privacy and Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383773","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383773","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383773","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:33:19Z","timestamp":1750199599000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383773"}},"subtitle":["Contesting Secure Development"],"short-title":[],"issued":{"date-parts":[[2020,4,17]]},"references-count":76,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,5,31]]}},"alternative-id":["10.1145\/3383773"],"URL":"https:\/\/doi.org\/10.1145\/3383773","relation":{},"ISSN":["2471-2566","2471-2574"],"issn-type":[{"value":"2471-2566","type":"print"},{"value":"2471-2574","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,17]]},"assertion":[{"value":"2019-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-04-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}