According to research reviewed by Gram, a 2026 study in the Journal of Medical Systems found that giving AI language models access to trusted diabetes sources improved answer quality, with the top model’s similarity score rising by an average of 10.56%. Clinicians generally preferred these responses. Gains were smaller for harder questions, and diet advice scored lowest.
Many people ask AI chatbots questions about living with diabetes, but these tools can give vague or outdated answers. A 2026 study in the Journal of Medical Systems tested whether giving AI models access to trusted medical information improves their replies. According to research reviewed by Gram, the best-performing model, Qwen2.5-14B, scored an average of 10.56% higher on answer similarity after this change, and clinicians generally preferred its responses. The benefit was smaller for harder questions, and answers about diet were the weakest area. The researchers suggest that trusted-source tools may help with everyday diabetes questions, but they do not replace advice from a doctor or diabetes care team.
Key Statistics
A 2026 study published in the Journal of Medical Systems found that adding trusted medical information raised the answer similarity score of Qwen2.5-14B, the best-performing model, by an average of 10.56% on diabetes questions.
In the same 2026 study, clinicians generally preferred diabetes answers from retrieval-augmented Qwen2.5-14B over its standard answers, while diet advice had the lowest similarity score of the four health management tasks, at below 0.8.
The 2026 Journal of Medical Systems study found significant gains at all three question difficulty levels, but those gains were significantly smaller for the most complex diabetes questions.
The Quick Take
- What they studied: Can giving AI chatbots access to trusted medical information make their answers about diabetes care better?
- Who participated: The study used a custom set of diabetes questions, called DiabMedQA, covering different difficulty levels and four types of health management, such as diet advice. The number of questions and clinician reviewers was not clearly reported in the summary. The participants were AI language models and clinicians who rated the answers.
- Key finding: Adding trusted information raised answer similarity for the best model, Qwen2.5-14B, by an average of 10.56%. Clinicians generally preferred these improved answers. Gains were smaller for harder questions, and diet advice scored lowest, with a score below 0.8 on a scale where 1.0 is a perfect match.
- What it means for you: AI tools that check trusted medical sources may give more useful answers to common diabetes questions. However, the study does not show that these tools are safe to rely on for medical decisions. Check important questions, especially about diet, medicine, or insulin, with your doctor or diabetes educator.
The Research Details
The researchers built a question set called DiabMedQA to test how well AI chatbots answer diabetes questions after a diagnosis has been made. The questions were grouped into three difficulty levels and four types of health management, so the team could see where the chatbots did well and where they struggled.
The team compared two versions of several AI language models. One version answered questions on its own, using only what it learned during training. The other version, called a retrieval-augmented model, first searched a collection of medical information and used those sources to build its reply. The answers were then scored by a computer measure called BERTScore, which checks how closely a reply matches a reference answer in meaning, not just in exact words.
The researchers also asked clinicians to review pairs of answers and say which they preferred. This step checked whether the computer scores matched what medical professionals thought was helpful.
Comparing the same models with and without trusted sources shows whether the added search step actually makes a difference. Using both computer scores and clinician reviews gives a fuller picture than either method alone. Testing questions at different difficulty levels also shows where these tools may be most useful and where they may fall short.
The study uses a clear comparison between standard and retrieval-enhanced versions of the same models, which strengthens the findings. Including clinician judgment adds real-world value. However, the summary does not report the total sample size, the number of clinicians, or the journal’s impact factor, so readers should look at the full paper for those details. BERTScore measures similarity in meaning, which is useful but does not directly measure whether advice is safe or correct for an individual patient.
What the Results Show
Across the tests, the AI models that used retrieved medical information produced replies that were more similar in meaning to reference answers. The strongest result came from Qwen2.5-14B, whose similarity score rose by an average of 10.56% after retrieval was added. Overall, the improvements were statistically significant across all three difficulty levels.
The benefit was not the same for every question. Gains were significantly smaller for the most complex questions. Only one of the four models showed a significant improvement on high-complexity questions, which suggests that harder questions may still challenge AI tools even when they have access to good sources.
Clinicians also generally preferred the retrieval-enhanced Qwen2.5-14B answers over its standard answers. This matters because it suggests that the higher computer scores reflected something professionals found more useful, not just wording that looked similar.
Diet advice was the weakest area. Its similarity score stayed below 0.8, the lowest of the four health management tasks. This suggests that diet questions may need extra care, since AI answers in this area were the least consistent.
The study found that improvements were not uniform across models. Some models benefited more from trusted sources than others, which shows that the choice of AI model still matters. The results also suggest that question difficulty is a key factor. For users, this means a chatbot may handle simple questions about blood sugar basics well but may give less reliable answers on complicated or personal situations.
Earlier research on AI in medicine has often found that chatbots can sound confident while giving incomplete or inaccurate information. Retrieval-augmented approaches, which pull answers from trusted documents, have been explored in many fields as a way to reduce these errors. This study adds diabetes-specific evidence that the approach can help, while also showing that its benefits shrink for harder questions.
The study focused on answer quality and similarity, not on whether real patients who followed the advice had better health outcomes. The test questions were designed by researchers, so they may not reflect the exact way people ask about diabetes in everyday life. The summary does not report the total number of questions or clinicians involved, which limits how broadly the results can be applied. Clinician preferences may also vary between doctors, nurses, and dietitians. Finally, the research reflects the AI models and medical sources available at the time of testing, and tools change quickly.
The Bottom Line
Research shows that giving AI models access to trusted medical information can improve the quality of their diabetes answers, based on computer scores and clinician preferences. Confidence is moderate because this is one study with specific models and test questions. People using AI tools for diabetes questions should treat the answers as general information only, and they should confirm any medication, insulin, or diet change with a licensed health professional.
People living with diabetes who use AI chatbots for general questions may find this research useful. Developers building health chatbots, clinicians exploring AI tools, and health educators may also benefit from understanding where these tools help and where they fall short. People who need urgent care, are pregnant, take insulin, or have complications should rely on their care team rather than AI for medical decisions.
Improvements in AI answer quality happen at the technology level and may appear as new tools are updated. For individuals, the practical benefit would be seeing more reliable general information in a chatbot over time. Any real health benefit would depend on the person’s own diabetes management, which usually shows up over weeks to months through steadier blood sugar readings and regular checkups.
Frequently Asked Questions
Can AI chatbots give better answers about diabetes if they use trusted medical sources?
A 2026 study in the Journal of Medical Systems suggests they can. Giving language models access to retrieved, trusted diabetes information raised answer similarity for the best-performing model by an average of 10.56%, and clinicians generally preferred those improved answers.
Are AI answers about diabetes diet reliable?
Diet advice was the weakest area in this research. Its similarity score stayed below 0.8, the lowest of the four health management tasks tested. Check any AI diet suggestions with a doctor, dietitian, or diabetes educator before making changes.
Do AI chatbots struggle more with harder diabetes questions?
Harder questions showed smaller gains from trusted sources. Improvements were significantly smaller at the highest complexity level, and only one of four models showed significant gains there. Researchers suggest question difficulty should be considered when adding retrieval tools to health chatbots.
Should I use an AI chatbot to manage my diabetes?
AI chatbots should not replace your diabetes care team. This study measured answer quality on researcher-designed questions, not real patient health outcomes. Use AI for general questions, and confirm any medication, insulin, or treatment decision with your doctor.
Want to Apply This Research?
- Log each AI-assisted diabetes question in the app, noting the topic (diet, medication, blood sugar, exercise) and whether you checked the answer with your care team. Track how many AI answers you verified over a four-week period to build a clear habit of confirmation.
- Before acting on an AI answer about food, insulin, or medication, prompt the user to tap a ‘Confirm with my provider’ button. Encourage users to save questions to bring to their next appointment, which helps turn chatbot answers into useful conversations with their care team.
- Over several months, track blood sugar readings, meals, and activity alongside the questions users asked and the advice they followed. Review these trends at regular intervals with a clinician to see whether better information is linked to steadier results. Update the app’s guidance as new research on AI tools in diabetes care becomes available.
This article summarizes a research study for general information only. It is not medical advice and does not replace guidance from a licensed doctor, diabetes educator, or other qualified health professional. The study tested AI answer quality on researcher-designed questions, not individual health outcomes. Always consult your care team before changing your diabetes medication, insulin, diet, or exercise routine.
This research translation is published by Gram Research, the science division of Gram, an AI-powered nutrition tracking app.