According to Gram Research analysis, an AI program called SimConverse can grade medical students’ simulated patient conversations about as well as expert teachers do for straightforward tasks, achieving agreement scores of 0.71 or higher on 7 out of 11 assessment items. However, the AI struggles with complex judgments requiring understanding of full conversation context, with agreement scores as low as 0.42 on those items. After adjusting the AI’s instructions, one problematic item improved dramatically from 0.56 to 0.89 agreement, showing that AI grading can be refined but requires ongoing human expertise and guidance.
Medical schools are using artificial intelligence to grade how well students interact with simulated patients, especially when it comes to treating patients with sensitivity about their weight. Researchers tested whether an AI program called SimConverse could score student performance the same way expert teachers do. They found that AI worked well for straightforward tasks, like checking if students asked specific questions, but struggled with more complex judgments that required understanding the full context of a conversation. After adjusting the AI’s instructions multiple times, most scores matched expert ratings, suggesting AI can help with medical training but needs careful setup and ongoing human guidance.
Key Statistics
A 2026 research study of 60 medical students found that an AI-based simulation program achieved good or very good agreement with expert graders on 7 out of 11 assessment items, with agreement scores of 0.71 or higher.
One assessment item in the study improved from poor agreement (0.56) to very good agreement (0.89) after researchers adjusted the AI program’s instructions, demonstrating that iterative refinement can enhance AI grading accuracy.
Three assessment items in the study remained problematic even after instruction adjustments, with final agreement scores ranging from 0.42 to 0.66, indicating that AI struggles with context-dependent clinical judgments.
Medical students in the study reported mostly positive reactions to using the AI-based simulation program, suggesting that AI-generated practice opportunities can be engaging even when AI grading has limitations.
The Quick Take
- What they studied: Whether an AI program can grade medical students’ simulated patient conversations as accurately as expert teachers, focusing on how students handle conversations about weight and weight bias.
- Who participated: Medical students (60 in the main study) and medical school faculty members who tested an AI-based simulation program called SimConverse and compared its grades to expert human graders.
- Key finding: AI successfully matched expert grading on 7 out of 11 evaluation items, with agreement scores of 0.71 or higher. One item improved dramatically from poor agreement (0.56) to very good agreement (0.89) after adjusting the AI’s instructions. However, three items remained difficult for AI to grade consistently.
- What it means for you: AI can help medical schools provide instant feedback to students practicing patient interactions, but it works best for specific, clear tasks. For complex judgments requiring understanding of context, human teachers still need to review AI grades. This technology could expand practice opportunities without replacing expert oversight.
The Research Details
Researchers used a two-phase approach to test whether an AI program could grade medical students fairly and accurately. In phase one, they had medical faculty and volunteers create 31 practice conversations with the AI program, then compared how the AI scored these conversations against how an expert human reviewer scored them. When they found areas where AI and humans disagreed, they rewrote the AI’s instructions (called prompt engineering) to improve accuracy. They then tested the revised AI with 10 more conversations.
In phase two, they had 60 medical students use the AI program as part of their training. The researchers randomly selected 30 of these student conversations and compared the AI’s grades to expert human grades. They also collected feedback from students about their experience using the AI system.
The researchers used two statistical methods to measure agreement between AI and human graders: percent agreement (how often they gave the same score) and Gwet’s AC1 (a more sophisticated measure that accounts for chance agreement). They looked at 11 different items on a checklist called the Weight Sensitivity Instrument, which measures whether students show bias or sensitivity when discussing weight with patients.
This research approach is important because AI is being used more and more in medical education to provide instant feedback, but we don’t yet have clear guidelines for making sure AI grades fairly and accurately. By testing AI against expert human graders and then adjusting the AI based on disagreements, the researchers developed a practical method that other medical schools could use. This iterative testing process (testing, finding problems, fixing, testing again) is crucial because it shows that AI needs human guidance to work well in education.
The study’s strengths include testing the AI with real medical students and using established statistical methods to measure agreement. The researchers were transparent about where AI struggled and didn’t overstate AI’s capabilities. However, the sample size of 30 student transcripts in the final testing phase is relatively small, and the study focused on one specific AI program and one specific assessment tool. The findings may not apply to other AI systems or different types of medical training. Additionally, the study didn’t report how long it took to adjust the AI’s instructions or how much expert time was needed for the iterative refinement process.
What the Results Show
The AI program performed very well on straightforward assessment items. Seven out of eleven checklist items showed consistently good or very good agreement between AI and expert graders, with agreement scores (Gwet’s AC1) of 0.71 or higher. This means that for about 64 percent of the evaluation criteria, the AI graded student conversations almost as reliably as expert teachers.
One item that initially showed poor agreement improved dramatically after the researchers adjusted the AI’s instructions. This item asked whether students specifically inquired if weight loss was a priority for the patient. Before adjustment, agreement was only 0.56 (poor), but after prompt engineering, it improved to 0.89 (very good). This demonstrates that AI can be improved through careful refinement of its instructions.
However, three items remained problematic even after adjustment. These items required the AI to make broader judgments about the context of the entire conversation, such as assessing whether students demonstrated overall sensitivity or understanding. Final agreement scores for these items were 0.42, 0.66, and 0.64, which range from poor to fair. These findings suggest that AI struggles more with complex, context-dependent judgments than with specific, concrete tasks.
Student feedback about the AI simulation experience was mostly positive. Medical students reported favorable reactions to using SimConverse, suggesting that the AI program was engaging and useful for practice. This indicates that even though AI has limitations in grading, students still find value in practicing with it. The positive student feedback is important because it shows that AI-based simulations could increase practice opportunities without frustrating learners, which addresses one of the main benefits researchers hoped to achieve.
This study builds on growing interest in using AI for medical education and assessment. Previous research has shown that AI can provide instant feedback and reduce the burden on teachers, but there has been limited research on whether AI grades as accurately as human experts. This study is among the first to systematically test AI grading reliability using established statistical methods and to show that iterative refinement of AI instructions can improve performance. The findings align with broader research suggesting that AI works best for well-defined tasks but struggles with nuanced, context-dependent judgments.
The study has several important limitations. First, it tested only one AI program (SimConverse) and one assessment tool (the Weight Sensitivity Instrument), so results may not apply to other AI systems or different types of medical training. Second, the final testing phase involved only 30 student transcripts, which is a relatively small sample for drawing broad conclusions. Third, the study did not measure how much time and expert effort was required to adjust the AI’s instructions, which is important for schools considering whether to implement this approach. Fourth, the study focused on one specific clinical scenario (weight sensitivity), so it’s unclear whether these findings would apply to other medical topics. Finally, the research did not explore whether students actually learned better from AI feedback compared to traditional methods, only whether AI could grade accurately.
The Bottom Line
Medical schools can use AI programs like SimConverse to provide additional practice opportunities and instant feedback for students, particularly for straightforward assessment tasks. However, schools should not rely entirely on AI grading for complex, context-dependent judgments. Instead, schools should use AI as a tool to supplement human expert assessment, with human teachers reviewing AI grades, especially for items that require understanding of broader context. Schools implementing AI-based simulations should plan for an iterative refinement process, working with experts to adjust AI instructions based on comparison with human grading. This approach requires initial investment but can lead to improved AI performance over time.
Medical schools and other health professions education programs should pay attention to these findings, as they provide practical guidance for implementing AI assessment tools. Faculty members who teach clinical skills and assessment specialists should understand both the capabilities and limitations of AI grading. Students may benefit from knowing that AI-based practice tools can help them prepare, but that human expert feedback remains important for complex clinical judgments. Healthcare administrators considering AI investments should understand that successful implementation requires ongoing human expertise and iterative refinement, not just purchasing software.
Students using AI-based simulations can expect to receive instant feedback on their practice conversations, which is faster than traditional methods. However, improvements in clinical skills typically develop over weeks to months of consistent practice, not from a single interaction. Schools should plan for several months of iterative refinement before AI grading becomes reliable enough to use with confidence, particularly for complex assessment items.
Frequently Asked Questions
Can artificial intelligence grade medical student performance as accurately as human teachers?
AI can grade straightforward clinical tasks very accurately, matching expert teachers 71 percent of the time or better on 7 out of 11 assessment items. However, AI struggles with complex judgments requiring understanding of full conversation context, achieving only 42 to 66 percent agreement on those items.
How can AI grading be improved for medical education?
Researchers found that adjusting the AI program’s instructions based on comparison with expert grading significantly improves accuracy. One item improved from 56 percent to 89 percent agreement after refinement, though context-dependent items remain challenging even after adjustment.
Should medical schools replace human teachers with AI assessment tools?
No. This research shows AI works best as a supplement to human expertise, not a replacement. AI can provide instant feedback and increase practice opportunities, but human experts should review AI grades, especially for complex clinical judgments that require understanding context.
What types of clinical skills can AI assess reliably?
AI assesses specific, concrete tasks very well, such as whether students asked particular questions. AI struggles with broader assessments requiring judgment about overall clinical sensitivity or understanding demonstrated throughout a conversation.
Do medical students find AI-based simulations helpful for learning?
Yes. Most students in this study reported positive reactions to the AI simulation program, indicating that AI-based practice tools are engaging and useful for preparing for clinical interactions, even with AI grading limitations.
Want to Apply This Research?
- Track the number of simulated patient interactions completed each week and monitor which specific assessment items (like asking about weight loss priorities) you consistently perform well on versus areas needing improvement. Compare your AI-generated scores across multiple conversations to identify patterns in your clinical communication.
- Use the AI simulation tool at least twice weekly to practice patient conversations about sensitive topics like weight. After each simulation, review the AI’s feedback on specific items where you scored lower, and deliberately focus on those areas in your next practice session. For example, if the AI notes you didn’t ask about weight loss priorities, make that question a priority in your next interaction.
- Keep a simple log of your simulation scores over 4 to 8 weeks, tracking your performance on the specific checklist items. Look for upward trends in your scores, particularly on items the AI initially flagged as weak. Share your progress with a mentor or instructor monthly to get human feedback on whether the AI’s assessments match their observations of your actual clinical skills.
This research describes how one AI program performed in a specific educational setting and should not be interpreted as a comprehensive evaluation of all AI assessment tools. Medical schools and educators should conduct their own testing before implementing AI-based assessment systems. AI grading should supplement, not replace, expert human assessment, particularly for complex clinical judgments. Individual student learning outcomes may vary, and this study does not establish whether AI-based practice improves actual clinical performance compared to traditional training methods. Consult with medical education specialists and assessment experts before adopting AI grading systems in your institution.
This research translation is published by Gram Research, the science division of Gram, an AI-powered nutrition tracking app.