Pusan National University researchers assess the impact of repeated item development faculty training on item difficulty prediction

Evaluation is an essential part of the learning process, as it helps to determine whether students have truly grasped the concepts being taught. The item difficulty index (the difficulty level of items or questions) plays a key role in evaluation and represents the ratio of the number of students who chose the correct response to the total number of students who responded to each question. This index can therefore provide a general measure of the difficulty level of tests. Furthermore, the ability of educators to create items and predict their difficulty indices significantly affects the evaluation process.

A recurring challenge is that educators may overestimate item difficulty, highlighting the need for improved predictive skills. However, research on faculty development programs aimed at improving the ability of teachers to predict and adjust item difficulty is limited.

To address this gap, a team of researchers at Pusan National University led by Professor Sang Lee, Vice President for Medical Affairs, Professor of Medical Education, and Professor of Family Medicine, conducted a study, the findings of which were published online in BMC Medical Education on May 30, 2024. The study investigated whether repeated item development training for medical school faculty improved their ability to predict and adjust the difficulty level of multiple-choice questions (MCQs).

Explaining the background of their study, Prof. Lee elucidates, “Just like the final stroke completes and perfects a painting, education is perfected through evaluation. Medical school faculty members cannot create high-quality items without proper training. Item development training is essential, and this study demonstrates that the effectiveness of such training increases with repetition.”

Item development workshops were conducted with 62 participants, first in 2016 and later in 2018, and the estimated accuracies of item difficulty predictions were compared. Before the workshop, the teachers developed newly drafted items which were then reviewed. An item development committee trained the faculty members by offering continuous feedback and helping them revise the newly drafted items according to the national exam standards, with an ideal difficulty range and an application-based focus. Furthermore, the difficulty indices predicted by the participants were compared with fourth-year medical student evaluation analyses.

The study found that before the training, significant agreement between the predicted and actual item difficulty indices was observed for only one subject, i.e., cardiology. In contrast, significant agreement was observed for four subjects, namely, cardiology, neurology, internal medicine, and preventative medicine. These findings suggest that systematic and effective training can improve the quality of MCQ assessments in medical education.

Repeated training sessions significantly enhanced faculty members' ability to predict and adjust item difficulty levels accurately, leading to effective assessments and better educational outcomes. Despite the benefits of the workshop, sustaining them might be challenging owing to its three-day duration and hectic schedules of participating faculty members. However, educators receiving item development and modification training will be better equipped to create items and make precise adjustments to difficulty levels thereby improving assessment practices. Moreover, these training programs can be applied across all academic fields. In conclusion, this study advocates for continuous faculty development programs to ensure the creation of appropriate items aligned with the purpose of evaluation.

Talking about the potential applications of their study, Prof. Lee shares, “Repeated item development training not only helps adjust the difficulty level but also enhances the construction of the items, increases their discriminating power, and properly addresses the issue of validity.” He further adds, “Soon there will be an era of item development using AI. For that, studies like ours are important for providing necessary information about existing items and students' answer data, which will help in developing an AI-powered automated item development program.”

***

Reference

Title of original paper: The impact of repeated item development training on the prediction of medical faculty members’ item difficulty index

Journal: BMC Medical Education

DOI: https://doi.org/10.1186/s12909-024-05577-x

About the institute

Pusan National University, located in Busan, South Korea, was founded in 1946 and is now the No. 1 national university of South Korea in research and educational competency. The multi-campus university also has other smaller campuses in Yangsan, Miryang, and Ami. The university prides itself on the principles of truth, freedom, and service, and has approximately 30,000 students, 1200 professors, and 750 faculty members. The university is composed of 14 colleges (schools) and one independent division, with 103 departments in all.

Website: https://www.pusan.ac.kr/eng/Main.do

About the author

Sang Yeoup Lee currently holds several prominent positions at Pusan National University, including Vice President for Medical Affairs, Professor of Medical Education, and Professor of Family Medicine. He has been recognized as the University's Best Teaching Professor. Lee's teaching expertise covers areas like evidence-based medicine, item development, and clinical reasoning. His research focuses on assessment, learning theory, academic stress, and student writing in medical education. He also conducts research on primary care, obesity, nutrition, and metabolic syndrome, with a particular emphasis on conducting clinical trials on functional foods.

Lab Website: https://www.researchgate.net/profile/Sang-Yeoup-Lee

ORCID ID: 0000-0002-3585-9910