Computing Education Research in Software Security

2025

1. WIP: How Effective Are LLM-Implemented Autograders for Programming Assignments Compared to Human Graders?

   Kevin Lewis and Hui Chen

   In Proceedings of the 2025 IEEE Frontiers in Education Conference (FIE), 2025

   Accepted and to appear

   Abs Bib Preprint PDF

   This research-to-practice WIP paper describes the development and evaluation of a generative Large Language Model (gLLM)-based autograder for computer programming assignments. Manual grading is becoming increasingly unsustainable due to growing student enrollment and the demand for timely, high-quality feedback. To address these challenges, this study explores the use of automated grading tools to reduce instructors’ workload and improve scalability. The proposed autograder takes a “reverse-engineering” approach, i.e., it converts student code into structured natural language summaries, which are then compared against predefined grading rubrics. An evaluation is performed using an external dataset (the Menagerie dataset), which contains real student submissions graded by four human graders. The objective is to assess the alignment between grades assigned by the autograder and those assigned by human graders. Findings indicate that the autograder closely matches human grading when letter grades are considered, though it performs less accurately with fine-grained numerical scores. While not yet a complete substitute for human assessment, the autograder shows strong potential as a scalable, efficient tool for supporting grading in programming education.

   @inproceedings{fie2025,
     author = {Lewis, Kevin and Chen, Hui},
     title = {{WIP}: How Effective Are LLM-Implemented Autograders for Programming Assignments Compared to Human Graders?},
     booktitle = {Proceedings of the 2025 IEEE Frontiers in Education Conference (FIE)},
     volume = {},
     number = {},
     pages = {LC:1--LC:5},
     year = {2025},
     organization = {IEEE},
     keywords = {Grades, Grading Systems, Automated Grading, Student Assessment, Computing Skills},
     preprint = {preprint/fie2025autograder.pdf},
     note = {Accepted and to appear}
   }

2024

1. Utilizing Real-World Software Vulnerabilities to Enhance Secure Programming Education

   Denise Daniels, Joon Suk Lee, Hui Chen, and 1 more author

   In 2024 IEEE Frontiers in Education Conference (FIE), 2024

   Abs DOI Bib Preprint PDF

   This research paper describes a study of using real-world vulnerabilities to motivate computer science students to- wards learning secure programming. Given the rise in cybersecurity incidents due to programming errors, there is a pressing need to improve programmers’ secure programming skills. Despite educators’ numerous efforts towards this goal, communicating the importance of this training to students remains a challenge. Grounding on the theory of intrinsic motivation, we propose that exposing students to authentic, relatable vulnerabilities can significantly enhance their learning orientation towards secure programming. Our approach involves selecting vulnerabilities from the National Vulnerability Database that are both relatable to students and understandable without extensive external context. These vulnerabilities are transformed into comprehensive course modules, each featuring a demonstrative video, source code snippets of the vulnerability and its patch, and associated developer communications about the vulnerability. We assess the impact of one of our course modules on students’ learning disposition through a study conducted in two universities in an identical setting. The study results indicate that students appreciate seeing real-world vulnerabilities in detail, especially the video we recorded reproducing the vulnerability, and that they gain in self-efficacy after completing the module.

   @inproceedings{fie2024,
     author = {Daniels, Denise and Lee, Joon Suk and Chen, Hui and Damevski, Kostadin},
     title = {Utilizing Real-World Software Vulnerabilities to Enhance Secure Programming Education},
     booktitle = {2024 IEEE Frontiers in Education Conference (FIE)},
     volume = {},
     number = {},
     pages = {DLCD:1--DLCD:8},
     year = {2024},
     organization = {IEEE},
     keywords = {Training;Surveys;Databases;Grounding;Source coding;Pressing;Software;Computer security;Programming profession;5.b.vii. Computer science;8.c. Computing skills;8.u. Student perception;12.d.iii. Mixed methods research},
     preprint = {preprint/fie2024vul.pdf},
     doi = {10.1109/FIE61694.2024.10893584}
   }

2019

1. Using Automated Prompts for Student Reflection on Computer Security Concepts

   Hui Chen, Agnieszka Ciborowska, and Kostadin Damevski

   In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, Aberdeen, Scotland Uk, 2019

   DOI Bib

   @inproceedings{Chen_2019_UAP_3304221_3319731,
     author = {Chen, Hui and Ciborowska, Agnieszka and Damevski, Kostadin},
     title = {Using Automated Prompts for Student Reflection on Computer Security Concepts},
     booktitle = {Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education},
     series = {ITiCSE '19},
     year = {2019},
     isbn = {978-1-4503-6301-3},
     location = {Aberdeen, Scotland Uk},
     pages = {506--512},
     numpages = {7},
     url = {http://doi.acm.org/10.1145/3304221.3319731},
     doi = {10.1145/3304221.3319731},
     acmid = {3319731},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {automated reflection, reflection, reflection prompt}
   }