{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T09:28:58Z","timestamp":1772270938144,"version":"3.50.1"},"reference-count":65,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2017,9,4]],"date-time":"2017-09-04T00:00:00Z","timestamp":1504483200000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Tekes, the Finnish Funding Agency for Technology and Innovation"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>Software engineering education is under constant pressure to provide students with industry-relevant knowledge and skills. Educators must address issues beyond exercises and theories that can be directly rehearsed in small settings. Industry training has similar requirements of relevance as companies seek to keep their workforce up to date with technological advances. Real-life software development often deals with large, software-intensive systems and is influenced by the complex effects of teamwork and distributed software development, which are hard to demonstrate in an educational environment. A way to experience such effects and to increase the relevance of software engineering education is to apply empirical studies in teaching. In this paper, we show how different types of empirical studies can be used for educational purposes in software engineering. We give examples illustrating how to utilize empirical studies, discuss challenges, and derive an initial guideline that supports teachers to include empirical studies in software engineering courses. Furthermore, we give examples that show how empirical studies contribute to high-quality learning outcomes, to student motivation, and to the awareness of the advantages of applying software engineering principles. Having awareness, experience, and understanding of the actions required, students are more likely to apply such principles under real-life constraints in their working life.<\/jats:p>","DOI":"10.7717\/peerj-cs.131","type":"journal-article","created":{"date-parts":[[2017,9,4]],"date-time":"2017-09-04T03:08:54Z","timestamp":1504494534000},"page":"e131","source":"Crossref","is-referenced-by-count":7,"title":["Guidelines for using empirical studies in software engineering education"],"prefix":"10.7717","volume":"3","author":[{"given":"Fabian","family":"Fagerholm","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Helsinki, Helsinki, Finland"}]},{"given":"Marco","family":"Kuhrmann","sequence":"additional","affiliation":[{"name":"Institute for Applied Software Systems Engineering, Clausthal University of Technology, Goslar, Germany"}]},{"given":"J\u00fcrgen","family":"M\u00fcnch","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Helsinki, Helsinki, Finland"},{"name":"Herman Hollerith Center (HHZ), Reutlingen University, B\u00f6blingen, Germany"}]}],"member":"4443","published-online":{"date-parts":[[2017,9,4]]},"reference":[{"key":"10.7717\/peerj-cs.131\/ref-1","first-page":"121","article-title":"Exploring machine learning methods to automatically identify students in need of assistance","author":"Ahadi","year":"2015"},{"issue":"4","key":"10.7717\/peerj-cs.131\/ref-2","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1002\/spip.391","article-title":"Experiences and results from tailoring and deploying a large process standard in a company","volume":"13","author":"Armbrust","year":"2008","journal-title":"Software Process: Improvement and Practice"},{"key":"10.7717\/peerj-cs.131\/ref-3","article-title":"Classroom experiments","author":"Ball","year":"2012"},{"key":"10.7717\/peerj-cs.131\/ref-4","volume-title":"Problem-based learning: an approach to medical education","author":"Barrows","year":"1980"},{"issue":"7","key":"10.7717\/peerj-cs.131\/ref-5","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1109\/TSE.1986.6312975","article-title":"Experimentation in software engineering","volume":"12","author":"Basili","year":"1986","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"6","key":"10.7717\/peerj-cs.131\/ref-6","doi-asserted-by":"publisher","first-page":"758","DOI":"10.1109\/32.6156","article-title":"The TAME project: towards improvement-oriented software environments","volume":"14","author":"Basili","year":"1988","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"4","key":"10.7717\/peerj-cs.131\/ref-7","first-page":"544","article-title":"Qualitative case study methodology: study design and implementation for novice researchers","volume":"13","author":"Baxter","year":"2008","journal-title":"Qualitative Report"},{"key":"10.7717\/peerj-cs.131\/ref-8","article-title":"The four steps to the epiphany","author":"Blank","year":"2006"},{"issue":"3\u20134","key":"10.7717\/peerj-cs.131\/ref-9","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1080\/00461520.1991.9653139","article-title":"Motivating project-based learning: sustaining the doing, supporting the learning","volume":"26","author":"Blumenfeld","year":"1991","journal-title":"Educational Psychologist"},{"issue":"4","key":"10.7717\/peerj-cs.131\/ref-10","doi-asserted-by":"publisher","first-page":"17:1","DOI":"10.1145\/2732155","article-title":"Software engineering project courses with industrial clients","volume":"15","author":"Br\u00fcgge","year":"2015","journal-title":"Transactions on Computing Education"},{"key":"10.7717\/peerj-cs.131\/ref-11","first-page":"239","article-title":"Issues in using students in empirical studies in software engineering education","author":"Carver","year":"2003"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-12","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/s10664-009-9109-9","article-title":"A checklist for integrating student empirical studies with research and teaching goals","volume":"15","author":"Carver","year":"2010","journal-title":"Empirical Software Engineering"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-13","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/S0742-051X(02)00091-4","article-title":"Learning and unlearning: the education of teacher educators","volume":"19","author":"Cochran-Smith","year":"2003","journal-title":"Teaching and Teacher Education"},{"key":"10.7717\/peerj-cs.131\/ref-14","volume-title":"Research design: qualitative, quantitative, and mixed methods approaches","author":"Creswell","year":"2009","edition":"3"},{"key":"10.7717\/peerj-cs.131\/ref-15","doi-asserted-by":"crossref","DOI":"10.1109\/ICGSE.2011.26","article-title":"GloSE-Lab: teaching global software engineering","author":"Deiters","year":"2011"},{"key":"10.7717\/peerj-cs.131\/ref-16","volume-title":"How we think: a restatement of the relation of reflective thinking to the educative process","author":"Dewey","year":"1935"},{"key":"10.7717\/peerj-cs.131\/ref-17","article-title":"A review of the research on practical work in school science","author":"Dillon","year":"2008","journal-title":"Technical report"},{"key":"10.7717\/peerj-cs.131\/ref-18","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-84800-044-5_11","article-title":"Selecting empirical methods for software engineering research","volume-title":"Guide to advanced empirical software engineering","author":"Easterbrook","year":"2008"},{"issue":"4","key":"10.7717\/peerj-cs.131\/ref-19","doi-asserted-by":"crossref","first-page":"532","DOI":"10.5465\/amr.1989.4308385","article-title":"Building theories from case study research","volume":"14","author":"Eisenhardt","year":"1989","journal-title":"The Academy of Management Review"},{"key":"10.7717\/peerj-cs.131\/ref-20","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1016\/j.jss.2016.03.034","article-title":"The RIGHT model for continuous experimentation","volume":"123","author":"Fagerholm","year":"2017","journal-title":"Journal of Systems and Software"},{"key":"10.7717\/peerj-cs.131\/ref-21","doi-asserted-by":"crossref","DOI":"10.1109\/CTGDSD.2013.6635237","article-title":"A platform for teaching applied distributed software development: the ongoing journey of the Helsinki software factory","author":"Fagerholm","year":"2013"},{"key":"10.7717\/peerj-cs.131\/ref-22","first-page":"26","article-title":"Building blocks for continuous experimentation","author":"Fagerholm","year":"2014"},{"key":"10.7717\/peerj-cs.131\/ref-23","doi-asserted-by":"crossref","DOI":"10.1145\/2652524.2652540","article-title":"The role of mentoring and project characteristics for onboarding in open source software projects","author":"Fagerholm","year":"2014"},{"issue":"4","key":"10.7717\/peerj-cs.131\/ref-24","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1111\/j.1465-7295.1997.tb01962.x","article-title":"The impact of classroom experiments on the learning of economics: an empirical investigation","volume":"35","author":"Frank","year":"1997","journal-title":"Economic Inquiry"},{"key":"10.7717\/peerj-cs.131\/ref-25","first-page":"25:1","article-title":"On the effects of programming and testing skills on external quality and productivity in a test-driven development context","author":"Fucci","year":"2015"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-26","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/0742-051X(94)00012-U","article-title":"Reflection in teacher education: towards definition and implementation","volume":"11","author":"Hatton","year":"1995","journal-title":"Teaching and Teacher Education"},{"key":"10.7717\/peerj-cs.131\/ref-27","article-title":"Energizing software engineering education through real-world projects as experimental studies","author":"Hayes","year":"2002"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-28","doi-asserted-by":"crossref","first-page":"75","DOI":"10.2307\/25148625","article-title":"Design science in information systems research","volume":"28","author":"Hevner","year":"2004","journal-title":"MIS Quarterly"},{"key":"10.7717\/peerj-cs.131\/ref-29","first-page":"170","article-title":"Introducing empirical software engineering methods in education","author":"H\u00f6st","year":"2002"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-30","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1080\/08878730.2012.740153","article-title":"Teaching reflective practice: implementation in the teacher-education setting","volume":"48","author":"Jones","year":"2013","journal-title":"The Teacher Educator"},{"key":"10.7717\/peerj-cs.131\/ref-31","first-page":"60","article-title":"Replication of software engineering experiments","volume-title":"LASER summer school 2008\u20132010. Lecture Notes in Computer Science","volume":"vol. 7007","author":"Juristo","year":"2012"},{"issue":"3","key":"10.7717\/peerj-cs.131\/ref-32","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/s10664-010-9141-9","article-title":"The role of non-exact replications in software engineering experiments","volume":"16","author":"Juristo","year":"2011","journal-title":"Empirical Software Engineering"},{"key":"10.7717\/peerj-cs.131\/ref-33","doi-asserted-by":"crossref","DOI":"10.1109\/ICGSE.2010.31","article-title":"Simulating global software development in a course environment","author":"Keenan","year":"2010"},{"key":"10.7717\/peerj-cs.131\/ref-34","doi-asserted-by":"crossref","DOI":"10.1201\/b19467","volume-title":"Evidence-based software engineering and systematic reviews","author":"Kitchenham","year":"2015"},{"key":"10.7717\/peerj-cs.131\/ref-35","first-page":"786","article-title":"Trustworthy online controlled experiments: five puzzling outcomes explained","author":"Kohavi","year":"2012"},{"issue":"1","key":"10.7717\/peerj-cs.131\/ref-36","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s11191-005-3199-6","article-title":"Generative role of experiments in physics and in teaching physics: a suggestion for epistemological reconstruction","volume":"15","author":"Koponen","year":"2006","journal-title":"Science & Education"},{"key":"10.7717\/peerj-cs.131\/ref-37","doi-asserted-by":"crossref","DOI":"10.1109\/ICCSE.2012.35","article-title":"A practical approach to align research with master\u2019s level courses","author":"Kuhrmann","year":"2012"},{"issue":"2","key":"10.7717\/peerj-cs.131\/ref-38","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1002\/smr.1703","article-title":"Crafting a software process improvement approach\u2014a retrospective systematization","volume":"27","author":"Kuhrmann","year":"2015","journal-title":"Journal of Software: Evolution and Process"},{"key":"10.7717\/peerj-cs.131\/ref-39","doi-asserted-by":"crossref","DOI":"10.4018\/978-1-4666-5800-4.ch010","article-title":"Controlled experiments as means to teach soft skills in software engineering","volume-title":"Overcoming challenges in software engineering education: delivering non-technical knowledge and skills","author":"Kuhrmann","year":"2014"},{"key":"10.7717\/peerj-cs.131\/ref-40","article-title":"Who cares about software process modeling? A first investigation about the perceived value of process engineering and process consumption","author":"Kuhrmann","year":"2013"},{"key":"10.7717\/peerj-cs.131\/ref-41","doi-asserted-by":"crossref","DOI":"10.1109\/ICSE.2013.6606665","article-title":"Teaching software process modeling","author":"Kuhrmann","year":"2013"},{"key":"10.7717\/peerj-cs.131\/ref-42","doi-asserted-by":"crossref","DOI":"10.1109\/ICGSEW.2016.13","article-title":"Distributed software development with one hand tied behind the back: a course unit to experience the role of communication in GSD","author":"Kuhrmann","year":"2016"},{"key":"10.7717\/peerj-cs.131\/ref-43","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1145\/2889160.2889194","article-title":"When teams go crazy: an environment to experience group dynamics in software project management courses","author":"Kuhrmann","year":"2016","journal-title":"Proceedings of the 38th international conference on software engineering companion, ICSE \u201916"},{"key":"10.7717\/peerj-cs.131\/ref-44","first-page":"60","article-title":"Identification of programmers from typing patterns","author":"Longi","year":"2015"},{"key":"10.7717\/peerj-cs.131\/ref-45","doi-asserted-by":"publisher","DOI":"10.1109\/SEKE.1992.227898","article-title":"SESAM-simulating software projects","author":"Ludewig","year":"1992"},{"key":"10.7717\/peerj-cs.131\/ref-46","article-title":"Creating an advanced software engineering laboratory by combining empirical studies with process simulation","author":"M\u00fcnch","year":"2003"},{"key":"10.7717\/peerj-cs.131\/ref-47","doi-asserted-by":"crossref","DOI":"10.1109\/CSEET.2007.14","article-title":"Comprehensive evaluation of an educational software engineering simulation environment","author":"Navarro","year":"2007"},{"key":"10.7717\/peerj-cs.131\/ref-48","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-39259-7_19","article-title":"Identifying potential risks and benefits of using cloud in distributed software development","author":"Oza","year":"2013"},{"issue":"3","key":"10.7717\/peerj-cs.131\/ref-49","doi-asserted-by":"publisher","first-page":"189","DOI":"10.3152\/147154404781776400","article-title":"What is the value of replicating other studies?","volume":"13","author":"Park","year":"2004","journal-title":"Research Evaluation"},{"key":"10.7717\/peerj-cs.131\/ref-50","article-title":"Using laboratory experiments to teach introductory economics","author":"Parker"},{"key":"10.7717\/peerj-cs.131\/ref-51","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-44930-7_11","article-title":"Feature prioritization based on mock purchase: a mobile case study","author":"Rein","year":"2013"},{"key":"10.7717\/peerj-cs.131\/ref-52","article-title":"Distributed development\u2014an education perspective on the global studio project","author":"Richardson","year":"2006"},{"key":"10.7717\/peerj-cs.131\/ref-53","volume-title":"The lean startup: how today\u2019s entrepreneurs use continuous innovation to create radically successful businesses","author":"Ries","year":"2011"},{"issue":"5","key":"10.7717\/peerj-cs.131\/ref-54","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1016\/j.jss.2007.06.004","article-title":"Teaching disciplined software development","volume":"81","author":"Rombach","year":"2008","journal-title":"International Journal of Systems and Software"},{"key":"10.7717\/peerj-cs.131\/ref-55","article-title":"Using students as experiment subjects\u2014an analysis on graduate and freshmen student data","author":"Runeson","year":"2003"},{"issue":"2","key":"10.7717\/peerj-cs.131\/ref-56","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","article-title":"Guidelines for conducting and reporting case study research in software engineering","volume":"14","author":"Runeson","year":"2009","journal-title":"Empirical Software Engineering"},{"key":"10.7717\/peerj-cs.131\/ref-57","doi-asserted-by":"crossref","DOI":"10.1002\/9781118181034","volume-title":"Case study research in software engineering: guidelines and examples","author":"Runeson","year":"2012"},{"key":"10.7717\/peerj-cs.131\/ref-58","volume-title":"The reflective practitioner: how professionals think in action","author":"Sch\u00f6n","year":"1983"},{"key":"10.7717\/peerj-cs.131\/ref-59","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-84800-044-5","volume-title":"Guide to advanced empirical software engineering","author":"Shull","year":"2008"},{"key":"10.7717\/peerj-cs.131\/ref-60","volume-title":"The art of case study research","author":"Stake","year":"1995"},{"key":"10.7717\/peerj-cs.131\/ref-61","first-page":"221","article-title":"Using Students as subjects in experiments\u2014a quantitative analysis of the influence of experimentation on students\u2019 learning proces","author":"Staron","year":"2007"},{"issue":"6","key":"10.7717\/peerj-cs.131\/ref-62","doi-asserted-by":"publisher","first-page":"384","DOI":"10.1037\/h0022100","article-title":"Developmental sequence in small groups","volume":"63","author":"Tuckman","year":"1965","journal-title":"Psychological Bulletin"},{"key":"10.7717\/peerj-cs.131\/ref-63","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering","author":"Wohlin","year":"2012"},{"issue":"7384","key":"10.7717\/peerj-cs.131\/ref-64","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1136\/bmj.326.7384.328","article-title":"Problem based learning","volume":"326","author":"Wood","year":"2003","journal-title":"BMJ"},{"key":"10.7717\/peerj-cs.131\/ref-65","volume-title":"Case study research: design and methods","author":"Yin","year":"2009","edition":"4"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-131.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-131.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-131.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-131.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,10,16]],"date-time":"2020-10-16T20:43:54Z","timestamp":1602881034000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-131"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,9,4]]},"references-count":65,"alternative-id":["10.7717\/peerj-cs.131"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.131","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.2\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.2\/reviews\/1","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.1\/reviews\/1","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.1\/reviews\/3","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.1\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.131v0.2\/reviews\/3","asserted-by":"object"}]},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,9,4]]},"article-number":"e131"}}