Is OpenEvidence actually the best clinician research platform?

Doctors can now make decisions based on research produced by AI. There are many popular platforms for doing that. For example, there's evidence, elicit, consensus, and sight. Some of them are really good while others are bad. In this video, I'm going to use three questions to help you find out which ones and how.

We're mostly going to focus on accuracy, but we'll also look at user experience. So, some things like speed, how readable it is, etc. So, for the first question, I wanted to see if they could all agree on something simple, but they didn't. Basically, I asked them to provide evidence-based guidelines on warfin dosing in a 70-year-old adult with nonvalular atrial fibrillation.

I asked about the starting dose, titration strategy, and how you would change the dosage. To make this sharable, I put everything in a table. So, the first difference between the four is whether or not they warned the doctor about a 5 milligram dose of warfin potentially being too high. Site did not.

consensus and illicit did that said open evidence did suggest a low dose it said two to three milligrams per day but I can't say the same for site because it suggested 2. 5 to 5 milligrams per day consensus and elicit were both around the 2. 5 to 3. 5 4 milligrams daily another place where they differed is their guidance on the monitoring of international normalized ratio while Open evidence said that it should be monitored daily illicit and sites said it should be monitored frequently.

Consensus didn't mention it. That said, consensus had a target INR of 1. 6 to 2. 6, which is lower than the 2. 0 to 3. 0 suggested by elicit and sight. Another important difference between the four is consensus was the only one that mentioned a time in therapeutic range or really emphasized that it should be kept above 70%.

And along those lines, open evidence said things that would make the doctor not want to modify the frequency of the warin dose too much. Concessants didn't really mention anything. Elicit insight said things that may want the doctor want to frequently modify the dosage. What's nice is that all four of them said if there is minor bleeding you could hold the orphan dose.

So temporarily skip a dose. Another thing for minor bleeding is vitamin K. Open evidence elicit antsai. All three of those said do not use vitamin K. If there is minor bleeding reserve it for serious issues. Consensus did not even mention vitamin K. The last difference between all of those was open evidence was the only one that explicitly said consider a direct oral anti-coagulant at DOAC.

I thought they all did an okay job and there is no really clear winner here. So for example, open evidence and site both missed the 5 mgram orphan warning. Consensus was the only one that mentioned time and therapeutic range but it did not mention dosage modifying frequency or vitamin K. elicit did a good job, but it did not mention the time and therapeutic range.

Uh, it also didn't really mention in DOAC. I thought they did too good for the first one. Actually, you won't believe what they did for the second one. So, specifically for the question, I gave a lot of context. A 74 year old male with HFRF LVF 20% and CKD stage 4 based on EGFR25. uh it's admitted with acute decompensated HF and remains clinically congested despite 48 hours of IV ferosomide with a rise of creatine from 2.

1 to 3. 5 milligrams per deciliter and potassium of 5. 7 meq per liter and then explicitly asked for the immediate pharmacological management plan addressing the diuretic strategy how to adjust or hold elements of his quadruple GDMT and potassium lowering measures and monitoring parameters over the next 24 hours.

At the end, I asked to support what it says with specific evidence. So for the IV loop diuretic, open evidence said double. Consensus said initiate, elicit said escalate, site said continue. For the theide, open evidence said add metalon zone. They all did that. Uh elicit also suggested zolomide. They all recommended something for lowering potassium and monitoring it closely.

What they really differed is their stance on bisoporal versus impactly. So open evidence said reduce bisoplural site said you know assess skipping whereas consensus said and elicit said continue bisoplural. On the other hand for the implied flossen open said continue so did consensus. So consensus is you know continuing both of those but elicit and sai said skip those or temporarily hold them.

So obviously on the places where they agreed they did good on the disagreement with the medication that is not as good but it's possible that doctors as well would disagree on that. Um if you're a doctor mention something in the comments about that. So the last example is the most exciting I think in general u especially it is for me it's from the New England Journal of Medicine Nejm.

So I split the case into four questions and we're going to look at the responses to each of those. So for the first question I took the fact that it's a 27year-old man with a history of bipolar one disorder and previous lithium use. He presented with a two-month history of subaccute confusion, paranoia, memory deficits, and a recent generalized tonic colonic seizure.

In the ED, his heart rate is 116. Other vitals are normal. The physical exam shows disorganized thought processes and limited recall uh but no focal neurological deficits. And so I asked them, what is your differential diagnosis? And what tests would you do to narrow it down? From this first question already, they differed a lot.

So the first question is whether or not he should get an EEG. Open evidence told that and said it was urgent. Consensus analysis said yes. Uh site did not mention EEG. For imaging both open evidence and site mentioned an MRI with contrast. Elysa also suggested adding a CT. What was really different is that consensus asked for a flare acquisition.

The second feature that was important to mention is whether or not the CSF should be sampled. Uh they all did. So open evidence did sample the CSF. So did consensus. So did illicit. So did site. There was no real difference except with respect to what you'd be looking for in the CSF. So consensus specifically recommends looking at cell count and protein oligoconal veins for autoimmune and sephilitis and infections.

All the other ones are also looking for those um focusing on an antibbody panel. For other testing they are pretty similar. They all recommended looking at the lithium. Some of them recommended blood panel urine toxicology looking for infections autoimmuneitis etc. What was uniquely different about the four is in terms of medication, open evidence was the only one that recommended starting them on IV acic.

So that was really interesting. For the second question in this case, I gave them the context that his heart rate remains elevated at 129. Toxicology is positive only for benzioipines which were administered in the ED. Blood's alcohol is undetectable. CBC,CMP and renal liver functions are normal. Your analysis shows keoneria, head CT, brain MRI, EEG and CSF analysis are all completely unremarkable.

The patients medical history includes remote hypothyroidism treated only non- selective beta blockers. Based on this data, what diagnostic category is most probable and what specific labs do you want next or require next? Actually see a big difference between them. Open evidence in site seemed to get really concerned.

Um site specifically mentioned a life-threatening thyroid storm and preparing to initiate beta blockers and theomides and iodine solution and gluccocorticoids. Uh same thing for open evidence theomides, beta blockers, corticosteroids and organic iodide. uh consensus or elicit did not mention preparing to do that.

In line with that both open evidence and site mentioned using the Burj Wowski point scale um so did consensus elicit did not mention that for the same reasons site ordered a stat thyroid panel. They all ordered it but not stat. They also differed on what other type of additional testing they wanted to do.

So open evidence wanted to look at cortisol. also did site elicit and consensus um did not seem to be concerned with that as much. So elicit said glucose and ketones whereas consensus said liver panel and cardiac markers. So yeah a lot of difference there but that's not surprising because it is a case worthy of being in the New England Journal of Medicine.

So after they gave me those responses, I told them about the thyroid panel results, the thyrorotropin receptor antibodies being negative and I asked what do the specific laboratory values suggest about the mechanism of the disease and what functional imaging is required to confirm the ethology. Elicit insight suggested that it might be destructive thyroiditis. Consensus actually said that it might be Graves disease.

Um so it actually contradicted illicit insight who explicitly said not grave disease or rather than graves disease. In terms of testing they all asked for an iodine scan so radioactive iodine uptake raiu scan. Open evidence also asked for antiPO antibodies serum thyrolyoglobin and ESPCP. Uh consensus doubled down on imaging and asked for thyroid cintigraphy.

Elicid also doubled down on imaging but instead it looked at technesium 999m protectate uptake. Site light consensus wanted to look at thyrointigraphy. With respect to medication there is an important difference. Uh site and open evidence said use beta blockers don't use comides. Open evidence also suggested cortical steroids.

So another part that was interesting about this is that elillicit was the only one to look at or look for low or absent uptake to confirm destructive thyroiditis. Um but also said be cautious of a recent iodine exposure possibly making it complicated to interpret the test results. The last part of this question is the most exciting um because you get to the diagnosis and this is the stuff that I gave them.

This is the information I gave them. Third ultrasound is normal. There's no excess blood flow. Iodine 123 centigraphy shows diffusely suppressed uptake at just 1. 2%. The patient denies taking exogenous thyroid hormone. What is the final primary diagnosis and a diagnosis for the psychiatric presentation?

Open evidence did not really provide a diagnosis. Uh said to continue to roll officious cytotoxosis and Hashimoto andphilopathy. uh consensus illicit and sight all said um that it's probably destructive thyroiditis likely painless thyroiditis. They all agreed on the psychiatric presentation that it was thyrotoxic psychiatric.

Open evidence was continuing the suggestion to use beta blockers and avoid theomides but site also suggested using temporary antiscychotics and anti-epileptics um to control the psychiatric presentation. elicit and consensus focused mostly on just correcting the thyroid state and did not really uh offer specifics there. Consensus had a similar thought that the treatment would correct the psychiatric state.

What was interesting about open evidence and site is that they said to expect a hypothyroid phase. Open evidence did not really address or resolve the psychiatric part of this. So I thought it was interesting to see the differences between all of them. I think there is room for interpretation, but obviously some of them are better than the others.

I'm not going to say because I'm not really sponsored or anything here. That's the benefit of being a small creator. And uh I'll leave it up to you to judge. So, all that was pretty good for judging accuracy, but it doesn't tell you much about the user experience. Um how it feels, how readable it feels.

I can't be unbiased about this, so I'm just going to tell you what I think and why I think that site was the slowest. It spent a lot of time checking what it was saying. There's probably a good reason for that. Site was sort of doing a systematic review. Its answers were really long even though I did not really ask for that.

On the other side of the spectrum, elicit was pretty fast and its answers were really clean. The UI was really clean. I think um I think they're really good. In my opinion, honestly, they're the best. Um again, not sponsored. The product is expensive. I don't think anybody's going to want to spend that much on the subscription without even really trying it.

Just to do this test, I basically had to ask it two questions per day. In the middle of the spectrum was consensus and open evidence. Honestly, from a user experience, uh they were pretty much the same. Uh open evidence is free and available for doctors. It's also marketed as exclusive to doctors and they're growing pretty fast.

so they can make their product a lot better in the future. Maybe they can even surprise me and make it better than elicit. We'll see. If you have any feedback or ideas for future videos, just let me know in the comments.