New strategies for distinguishing between students who know and students who guess

Dec. 28 (UPI) — Research economists at the University of Leuven have developed new strategies for more accurately assessing the academic performance of students in online classes.

According to their new paper on the subject — published in the journal Heliyon — there are a variety of problems with the assessments currently used for many so-called massive open online courses, or MOOCs.

First, MOOC assessments generally feature too few questions. Tests featuring just 10 to 15 questions are insufficient, researchers argue. Additionally, the common use of multiple choice rewards guessing, distorting the test results. Finally, online tests used by MOOCs frequently rely on the same sets of answers, minimizing their ability to measure changes in knowledge as tests are updated.

For solutions to the assessment problems, researchers looked to the Rasch model, a series of models designed to derive measurements from categorical data. The model calls for assessors to account for additional factors — not just the final answers — including multiple attempts by test-takers, as well as a student’s interaction with video lectures and instructional tasks.

“First, our expanded approach includes the effect of multiple attempts, making it possible to distinguish between students who guess and those who know the answers,” researcher Dmitry Abbakumov said in a news release. “Second, because the knowledge metrics obtained with this expanded approach are expressed on a single scale, they can be compared, even when the test questions are changed significantly.”

“Finally, we calculate metrics based not only on test results, but also by taking into account the student’s experience — their activity when watching videos and performance in hands-on sessions — providing a more comprehensive understanding of the student’s competence,” Abbakumov said.

Without accurate assessments, it’s difficult to measure the efficacy of online courses. Researchers hope their solutions will help course designers measure and improve the educational benefits provided by MOOCs.