(Boston)—Medical, dental and master’s students in biomedical sciences frequently take standardized, multiple-choice question tests to assess their foundational knowledge. Reasons for its widespread use include reliability, efficiency, low cost and when the questions are well-constructed, high accuracy. However, multiple choice questions may present challenges for test-takers when they contain item writing flaws (flawed, ambiguous, or poorly constructed test questions) that can potentially compromise the fairness and validity of the assessment.

New research, led by scientists at Boston University Chobanian & Avedisian School of Medicine, suggests that an increased number of total flaws in questions could lead to disadvantages among lower-performing students, not because the content is more difficult, but rather that these students may be less able to navigate item writing flaws.

“Our study highlights the importance of careful multiple-choice question construction to ensure assessments are fair and accurately reflect student abilities,” explains corresponding author Marisol E. Lopez, PhD, assistant professor of pharmacology, physiology & biophysics at the school. “Based on our findings, we recommend the identification and elimination of item writing flaws to promote greater equity.”

To study if poorly written questions affect student performance, the researchers created a physiology test and demographic survey that was given to 31 BU dental as well as graduate medical science master’s students and 54 University of Central Florida medical students.

After analyzing the results, they found test performance did not differ among demographic groups (sex, race/ethnicity, English language proficiency, birth country, and primary home language) and question difficulty was not affected by the number of flaws present in the question. However, they did find that questions with more flaws have a greater impact on low performing students. Specifically, certain flaws such as when questions have unclear stems (prompts) affect weaker students disproportionately. These findings suggest that high performing students can better navigate question flaws.

According to the researchers, these findings highlight the need for improved faculty training and rigorous review of test questions before exams are administered. “Fixing unclear wording, simplifying stems, and eliminating common structure mistakes could help ensure that all students, no matter their background, are evaluated fairly,” adds Lopez.

These findings appear online in the journal Medical Science Educator.

Partial funding for this study was from the University of Central Florida’s FIRE (Focused Inquiry and Research Experience) to L.D. Larissa Dixon was supported by the Fogarty International Center of the National Institutes of  Health under grant #D43TW009345 awarded to the Northern Pacific Global Health Fellows Program.