Researchers tested Microsoft Copilot, an AI search engine, to see how well it answers questions about dietary supplements and whether they work for common health problems like diabetes and joint pain. They asked 180 questions about 30 different supplements across three different response modes. The results were concerning: the AI only gave accurate answers about 31-36% of the time, and it often cited unreliable sources like blogs and sales websites instead of scientific research. The study suggests that while AI search engines seem authoritative and helpful, they may actually spread misleading health information about supplements.

The Quick Take

  • What they studied: How accurate and reliable is Microsoft Copilot (an AI search engine) when answering questions about whether dietary supplements actually work for treating common diseases?
  • Who participated: No human participants. Instead, researchers asked the AI 180 different questions about 30 dietary supplements and their effectiveness for treating 6 common conditions: cancer, diabetes, obesity, constipation, joint pain, and high blood pressure.
  • Key finding: Microsoft Copilot gave accurate answers only 31-36% of the time, regardless of which response mode was used. Additionally, 73% of the sources it cited came from unreliable places like blogs and sales websites rather than scientific studies.
  • What it means for you: If you’re using AI search engines to research dietary supplements, be cautious. The information may sound official but could be inaccurate or based on unreliable sources. Always verify supplement claims with your doctor or a registered dietitian before making decisions.

The Research Details

Researchers created a test to evaluate how well Microsoft Copilot answers health questions about supplements. They wrote 180 questions (6 questions for each of 30 different supplements) asking whether each supplement could treat one of six common health conditions. They asked these same questions three different ways by using Copilot’s three different response modes: creative (more imaginative), balanced (middle ground), and precise (more factual). The AI’s answers were then compared against expert evaluations from licensed physicians who reviewed scientific evidence about each supplement’s actual effectiveness. Researchers also analyzed what sources Copilot cited and checked whether those sources actually supported what the AI claimed.

This research approach is important because it shows how AI systems perform in real-world situations where people actually use them. By testing multiple response modes and comparing answers to expert medical knowledge, the researchers could identify specific problems with how the AI handles health information. This helps us understand whether we can trust AI search engines for important health decisions.

This study has several strengths: it tested a large number of questions (180), used expert physician review for accuracy assessment, and examined actual sources cited by the AI. However, the study only tested one AI search engine (Copilot) and only in Japanese, so results may differ in other languages or with other AI systems. The study didn’t involve human participants, so it shows what the AI does but not how real people might misuse or misunderstand the information.

What the Results Show

Microsoft Copilot’s accuracy was disappointingly low across all three response modes. In creative mode, it was accurate 36% of the time. In balanced mode, accuracy dropped to 32%. In precise mode, it was also only 32% accurate. This means that roughly two-thirds of the time, the AI gave wrong or misleading information about supplements. Surprisingly, the ‘precise’ mode—which sounds like it should be most accurate—performed no better than the other modes. The AI claimed supplements were effective 29-48% of the time depending on the mode, but these claims were often not supported by scientific evidence. When doctors reviewed the answers, they found that the AI frequently made statements that weren’t backed up by reliable research.

A major problem was the sources Copilot cited. About 73% of all sources came from unreliable places like personal blogs, sales websites, and social media posts rather than peer-reviewed scientific journals or medical organizations. In 10% of responses, the AI cited sources that didn’t actually support the claims being made—a problem called ‘hallucination’ where the AI appears to make up or misrepresent information. Only about half of the responses (48.5%) included a recommendation to talk to a healthcare professional before using supplements. The accuracy varied by health condition: questions about cancer had the highest accuracy (likely because there’s less misinformation about cancer treatments), while other conditions had lower accuracy rates.

This is the first study to specifically evaluate Microsoft Copilot’s performance on dietary supplement information. Previous research has shown that AI language models sometimes struggle with health information accuracy and tend to cite unreliable sources. This study confirms those concerns in the specific area of supplements, where scientific evidence is often limited and misinformation is common online. The findings align with other research showing that AI systems can appear authoritative while actually providing inaccurate health information.

The study only tested Microsoft Copilot and only in Japanese, so results may be different for other AI systems or in other languages. The researchers didn’t test how real people would use this information or whether they would follow up with healthcare providers. The study examined 30 supplements and 6 conditions, which is a good sample but doesn’t cover all possible supplements or health issues. Additionally, the study didn’t evaluate how the AI’s responses might change over time as the system is updated.

The Bottom Line

Do not rely solely on AI search engines for information about dietary supplements. If you’re considering taking a supplement, discuss it with your doctor or a registered dietitian who can review scientific evidence and consider your personal health situation. When you do use AI search engines for health information, check whether the sources cited are from reputable organizations (like medical schools, government health agencies, or peer-reviewed journals) rather than blogs or sales websites. Look for responses that recommend consulting a healthcare professional. Confidence level: High—this recommendation is based on clear evidence that AI systems currently provide inaccurate supplement information.

Anyone considering taking dietary supplements should care about this research, especially people managing chronic conditions like diabetes, high blood pressure, or cancer. Parents researching supplements for their children should be particularly cautious. Healthcare providers should be aware that patients may be getting inaccurate information from AI search engines. People in countries where supplements are popular (like Japan) should be especially careful. However, this doesn’t mean AI search engines are useless—they can be helpful for general information if you verify important health claims with professionals.

You won’t see immediate changes. The improvements needed are in how AI systems are developed and regulated, which could take months to years. In the meantime, you should expect that using AI for supplement research will require additional verification steps. If you do decide to try a supplement based on AI information, give it at least 4-8 weeks to show effects (depending on the supplement), but only after consulting with your healthcare provider.

Want to Apply This Research?

  • Track which supplements you research using AI versus professional sources. Create a simple log noting: supplement name, health condition, source of information (AI vs. doctor vs. scientific database), and whether you consulted a healthcare provider before use.
  • When researching supplements in your app, add a required step: before saving any supplement information, users must note whether they’ve scheduled a conversation with their doctor about it. This creates accountability and ensures professional consultation happens.
  • Implement a monthly reminder asking users to review their supplement sources and verify that any new supplements were discussed with healthcare providers. Track the percentage of supplement decisions that included professional consultation as a key health literacy metric.

This research evaluates how well AI search engines provide information about dietary supplements, but it does not provide medical advice. Dietary supplements are not regulated the same way as medications and are not proven to treat, cure, or prevent diseases. Before starting any supplement, consult with your doctor or a registered dietitian, especially if you have existing health conditions, take medications, or are pregnant or breastfeeding. Do not use AI search engines as your primary source for health decisions. The findings in this study apply specifically to Microsoft Copilot and may not reflect other AI systems or future versions of this technology.