NCME 2022 Presentation

item response theory R Stan item preknowledge detecting test misconduct 2022

I was not able to go to NCME 2022 this year. Our session organizers, Huijuan Meng and Anjali Weber, kindly let me make an asynchronous video presentation. The session provided a collaborative exercise in which five independent research groups each propose a method that could help effectively and efficiently detect cheaters in the operational setting. Each group used the same data from two linear fixed-form IT certification exams with known security breaches. The five approaches were evaluated regarding their accuracy in detecting cheating and feasibility to implement. Here, I provide the slides, a Github repository for the code, and a video for my presentation.

Cengiz Zopluoglu (University of Oregon)
04-22-2022

My NCME 22 presentation was a supplemental analysis for another blog post I wrote about a year ago.

https://cengiz.me/posts/dglnrt2/

In this post, I played with an idea to simultaneously detect compromised items and examinees with item preknowledge using the response time. The model worked fine with a simulated dataset and a real dataset with known item preknowledge. When I was invited to submit a paper for this session, I thought the IT certification exam datasets offered to be used in the session would be a great test of the model in the wild. I applied the model to a random subset of examinees, and I think the results were encouraging.

I hope to turn this accumulated knowledge on this model into a formal paper in the coming months! Here is some more information and additional results until it is published.

Download the Slides

R code used for the analysis

Video Presentation

Session Information

Cheating Detection: A Collaborative Case Study using IT Certification Exams

Coordinated Paper Session

Friday, April 22

4:15 to 5:45 pm PT

Westin San Diego Gaslamp: Del Mar

Cheating damages the integrity of a testing program and can cause testing organizations significant losses. Security breaches can arise from individuals memorizing and sharing items, the concerted efforts of a test preparation company to harvest items and teach them to their customers, and answer copying or collusion among examinees during a testing event. Without proper detection, these types of cheating could remain undetected until their presence becomes significant enough to threaten test-score validity. It is crucial for a test sponsor to accurately identify cheaters and invalidate their scores to effectively deter cheating behaviors. However, many cheating detection techniques developed so far are based on complicated mathematical models and extensive ad-hoc data analyses and thus cannot be practically conducted on daily basis. Therefore, we propose a collaborative exercise in which five independent research groups each propose a method that could help effectively and efficiently detect cheaters in the operational setting. Each group will use the same data from two linear fixed-form IT certification exams with known security breaches. The five approaches will be comparatively evaluated regarding their accuracy in detecting cheating and feasibility to implement.

Session Organizer:

Participants:

Discussant:

Citation

For attribution, please cite this work as

Zopluoglu (2022, April 22). Cengiz Zopluoglu: NCME 2022 Presentation. Retrieved from https://github.com/czopluoglu/website/tree/master/docs/posts/ncme22/

BibTeX citation

@misc{zopluoglu2022ncme,
  author = {Zopluoglu, Cengiz},
  title = {Cengiz Zopluoglu: NCME 2022 Presentation},
  url = {https://github.com/czopluoglu/website/tree/master/docs/posts/ncme22/},
  year = {2022}
}