AI Chatbots Can't Read My Research Paper (ChatGPT, Meta, Grok, Deepseek, Gemini, Perplexity)

I tried every job bot and none of them were able to read my research paper. So, I used a simple way to grade them. Basically, I just made a checklist of all the facts that you need to know if you needed to understand the research. Some of the chatbots work clearly better. I'll tell you which ones you should definitely know them if you're using AI for research in 2026.

I'm a health software engineer that's been working with AI for the last 6 years. So, I'm the last person who should be sharing this with you. Okay, so let me show you how I converted the research paper into a checklist that can be used to fairly grade the AI. So I wrote this research paper. It was published in a peer-reviewed journal in 2020.

The point of the paper was to look at neuroinflammation in stroke and Alzheimer's. It's kind of complicated, so I distilled it down to the important facts. Please bear with me here. One is that the paper studied rats. Two is that the rats were only male. Three is that the rats had a very specific type of stroke and we call that a stroke model.

Four is that we also used a rat model of Alzheimer's disease. It was genetically bred to have the Alzheimer's. Five is that the method used to look at inflammation was very specific. It's called TSBO imaging. The TSBO PAT translocator protein positron admission tomography. It's a very specific type of imaging.

Six is as part of the results, TSBO only detected inflammation near the site of the stroke. Seven is that the effect that it detected was a doubling in the uptake ratio of the signal of TSBO near the sight of the stroke. Eight is that it completely missed a different type of inflammation marked by a class of proteins called MHC2.

But yeah, what does that mean? um AI needs to automatically elaborate correctly on the interpretation of those results, the uptake ratio and MHC2 and 10 is that if you're a person that's concerned about stroke or Alzheimer's and you're trying to read research, AI should tell you that a TSPO scan is rare so as to imply that it's unlikely that you got one. So those are all the facts that you need AI to tell you.

So I've used them as a checklist for grading the AI. By focusing on the important facts, we're able to evaluate the AI's ability to clearly communicate that research. And because we're interested in AI's ability to communicate this research to a concerned person, Abby uses this message or prompt on a brand new account.

So, the prompt starts by asking for help understanding to signal a lack of research background. It then gives a direct link to the article where it can be read by AI or anybody else for free. I also tried to squeeze some concern to this prompt message so that the AI gets an idea that this is a, you know, a person that's actually worried about this.

By including all these three important hints, the AI should know to explain this article to a nonressearcher that's concerned about stroke or Alzheimer's. All right, now we know enough to get into the results. So, the first AI that I want to evaluate was the most popular one. Um, you might have heard of Chad GBT.

So, it was actually able to alert me to the fact that this was in rats. It was also specific about the different types of inflammation. That's it though. Maybe you could try to argue that it tried to explain what type of imaging it used. Um, but I think that's a stretch. It pretty much just used the words that were in the paper at which point you might as well just read it, right?

But I wanted it to be able to explain it. And unfortunately, it also hallucinated a quotation just when I thought I had eliminated hallucinations. And the quotation says unable to detect remote and non-insultreated chronically activated microglea. So this quotation is not just hallucinated. It's actually wrong.

It's misleading. It says noninssult related. This is wrong because it is related to the insult or injury. It's just not directly near it. It is related to it. So that was pretty shitty. But there was an AI that did much better which I'll tell you about later. The second one that we want to get into is Meta's AI.

Fortunately for me, it did a horrible job. So, this bit can be short. It didn't even open the research paper. So, needless to say, I now have zero stock in Meta. I mean, I think they'll be fine without me, but yeah. All right. So, after Meta, we have Grock. Now, Grock actually surprisingly did a good job.

It warned us that it was rats only. It was careful about the specific imaging method, and to my utmost surprise, although it didn't explain the meaning, it actually mentioned the effect size. And I was also careful about the specific type of inflammation. Grog also hinted that you likely did not get a TSPO scan.

Like here it says if a person with strokeal Alzheimer's is being evaluated with the SPO pet. The way that's coming off to me is that like oh if a person with stroke or Alzheimer's which is almost implying that it's unlikely. Bonus from Grock is that it actually tells you to discuss with a neurologist.

Um, surprisingly, spoiler alert, none of the other chat bots even suggested a doctor. So, I was pretty impressed about that. So, this was good, but it's not good enough. Also, it still feels like this summary is a tough read. Like, it was longish. It used a lot of the terminology from the paper. So, yeah, it still feels like a tough read.

Let me know what you think in the comments, but at this point, you might as well read the original. So, right now, Grock is leading versus Meta and ChaD. Next up, we have Gemini. Now, Gemini is my preferred chat because I am very integrated into the Google ecosystem. Um, and I was very disappointed. All it did was be specific about the imaging methods and the type of information.

And I was also expecting Gemini to be the most conservative or modest, but it was the only one that used language that could cause psychological bias. This PET scans miss, this smoldering and widespread inflammation. Like just hearing that would probably make most people afraid. I know it wakes up that part of my brain a little bit.

So this is a tough punch to the gut. So Google, if you're watching this, please do better. I don't know what happened here. Usually it's good. Maybe not with research. Also, it would be cool if you gave my videos more impressions. All right, now we just have two left. The first one is Deepseek. Deep Seek belongs to a company that is known for being resourceful.

So, I had my expectations pretty high. Like me, you're probably going to be very disappointed because like Meta, it didn't even open the paper. Actually, it said that it did and then it completely referenced a wrong paper. I do not even have vitamin D in the title of my research paper. So, last but not least, we have Perplexity.

Perplexity is branded for not hallucinating. So, I was very excited. Perplexity warned us that it was males only. The rats were only males and it was the only one that did that. Here's interesting is Perplexity was the only one that really talked about regionality. They compared inforked areas nearby white matter and remote white matter.

So that was very important for the paper and none of them actually did that. Perplexity also was specific about the type of inflammation. It says uh TSBO Inos MC2 was able to differentiate between them which was very important for the paper as well. And not only did Perplexity mention that there was an increase in uptake or the effect rough size and actually quoted the number as 2.

10. So that was pretty good to see. So there is more but that's already five out of 10. That means it's currently tied with you might remember Grock. That's if we're just looking at the number of facts. Perlexity also was a tough freak in my opinion. It used the same terminology as the paper did. So, you might as well read the original.

But if you remember, Grock also had the bonus of mentioning a neurologist. Can perplexity match that? Perplexity actually concluded the response with limitations and outlook. I think that deserves bonus points. Science is limited and it's always improving or changing or building on top. So it's important to end the response on that note and remind the person that hey by the way you're reading science which changes and has limitation but you know it's better than no science.

So so that gives a perplexity a bonus as well. So we still need a tiebreaker. The best way to do that is to reward the one that was shorter. Perplexity used 314 words. Brock H used 748 more than double. So if you're just looking for important facts, Perplexity won, Grock got second, and then third was a tie between Gemini and Chad GBT.

But honestly, with the performance that I thought was poor here, I would not use either of them for analyzing a single research paper. But I am emphasizing that this is if you want the facts and you didn't have the mental energy to be critical in that moment, then I would probably use Grock because of that bonus when it told us to talk to or have a patient talk to a neurologist. Grock was the only one surprisingly that mentioned a healthc care provider.

You can actually use these for all the chat bots that I tried here today. Uh I didn't even use a credit card. So, I'm going to leave a link in the description to the chatbots. Yeah, not the credit cards. No evaluation is complete without a criticism. You can and you should be critical three things at least.

The prompt, the checklist, and the fact that each chatbot only got one chance. So, I want to start by defending the one shot or one chance. In a normal research paper that you'd like publish, you'd want probably at least a sample of three or 10 depending on the effect size and the variance of the sample.

For this video, I chose to go with just one because in the real world, you actually only give AI one shot. You're not going to I mean, if you don't know how to read the research or if you don't already understand the research, you're not going to know what facts to try to dig out of it. So if that can't happen, then you're probably going to be left with incomplete information, which is a problem.

So the second thing that I want to defend is the prompt or message. As for the prompt or message, you can say that I had too many tough things. For example, not many of them knew to really explain or elaborate on the stroke model or the Alzheimer's model. And none of them actually talked about the importance of effect size.

What does it actually mean for an uptake ratio to rise to two? Although these are very tough are important for understanding. Statisticians call interaction effects. So an interaction effect is for example, let's say you are an orange traffic cone and you hear that orange traffic cones are hard to see.

They're not as visible as black traffic cones. You're a menace. You decide to steal art from the Loura or is it Louv? But yeah, so you do that and then you get caught. Bam. You blame the research paper. You're like, "Oh, I thought orange traffic cones are not visible." But if you understand the research paper, you would have known that this research was conducted in Willy Wonka land where orange traffic cones are less visible and black traffic cones are more visible.

Lou is nothing like Willy Wonka Land. you have here is an interaction between the environment louver or Willy Wonka with the color of the cones to affect visibility. And that is all I have to criticize for now, but let me know in the comments if you can think of more. Okay, so criticism aside though, let's talk about the good parts of AI in health research.

A well-designed AI can still be pretty good at citing or referencing facts. So like finding the sources, any sources really, it'll probably generate something and then be like, "Okay, well, is this validatable by some source that did an experiment hopefully it looks for an experiment? If it's an AI chatbot that's good, it'll look for an actual experiment and be critical of the variables like I would."

Still, unfortunately, I do think that some of them hallucinate. I've had instances of different chat bots where they would say something and they would reference something and I would go to look at the research paper to look at the variables and they say hey these are not that's not why you're that's wrong and they say oh you got me. So the way I think of it using AI in health research is like a loaded gun.

I believe that you need the interpretation license to use AI for health. At the very least in 2026, people should be able to get one day maybe of beginner level training. Or they could just watch this video like you just