The Red Flag That Marks AI YouTube Doctors

YouTube has lots of doctors and now we're getting some that are actually AI. The data makes it obvious and I'm here to share it with you. So, I myself started out doing research in health where I worked with some of the world's best and now I'm a health software engineer where I process all kinds of data all the time.

So, in this video we're going to copy paste the scripts the YouTube video scripts of 100 videos of three different doctors each. Then we'll calculate three numbers that can be used to identify whether the doctor is AI or not. Those three numbers are perplexity, burstiness, and structural predictability.

We'll define them as we go along. So the first score that we're calculating is perplexity. You can think of perplexity as predictability. So usually humans make texts that are unpredictable, so high perplexity. Whereas AI makes text that's pretty predictable, at least to an AI, and that's going to be low perlexity.

So to show you what I mean, here's an example of a low perplexity sentence and a high perplexity sentence. The low perplexity sentence is, and as always, stay happy and healthy. Um, this is the signature from Dr. Mike's videos, the goodbye signature. So AI is actually able to predict that pretty well because from the start of the sentence, it knows how it's going to end just because he said it so many times.

So the sentence that has a hyperplexity is you can't have a green forest of red trees. So we're also using a Dr. Mike example here. And this sentence is a hyperplexity because he could have used any color besides red. He could have said blue, purple, white, whatever. There was no way of predicting. So it is a hyperplexity sentence.

So to calculate the plexity for all these 100 videos, you know, obviously didn't do it by hand. I took all the transcripts 100 of each from each doctor and then I ran them through a llama model to get the probabilities. So the probability that the AM model was going to spit out those words in that order.

And then once you get the probabilities, it's pretty easy to calculate the perplexity. and you just run them through this equation right here. It's pretty straightforward, right? All right, so let's get into the results. The results for perplexity were perplexing. The YouTube channel for Dr. Riberly, which I believe is AI, actually got the highest perplexity.

And that was again very perplexing. So I looked deeper into that and it became obvious. The sentences that Dr. Riberly's channel says are higher in length, abnormally high in length. So the average or median, so the median sentence was in the 10 word range. And if you have sentences that are longer, with each word, you add some amount of uncertainties.

If the word was totally out of the blue, then you'll increase the sentences perplexity by a massive amount. So most of the sentences in Dr. Robert's channel are long like this one. And yet sarcopenia, the age related loss of muscle mass and strength is one of the most predictive factors for disability, for falls, for metabolic disease, and early death.

Whereas Dr. Mike, for example, has some sentences that are long like this. I think this is one of those scenarios much like intermittent fasting. People think there's some sort of miraculous benefit to doing intermittent fasting when in reality you're just shrinking the window the time period in which you are able to eat.

Therefore, you consume less calories simply because there's less time. And he also has some that are short like this. It's natural, tastes yummy, and it works. Evidence-based. We also have Dr. Burke in our comparison. So, we're going to look at his sentences. They're also short and long like this. But, let's just dive right in.

A lot of people have a lot of questions. Absolutely. So, that was perplexity. It was pretty interesting. But the one that I really liked is burstiness. And so I'm going to show you that now. So you can think of burstiness as how much the sentence is vary in perplexity from one sentence to the other. So if you have a sentence that's very predictable and then you have a sentence that's not predictable is going to have a high burstiness.

For like an extreme example, if you had a whole video that had just three these three sentences and these sentences are identical, they're very similar. So we'll get a very low burstiness. will actually get a burstiness of zero, right? Because they have the same complexity. For reference, I just calculated burstiness as a standard deviation or variance of perplexities between the sentences.

Whereas, if your sentences were varied like these three, then god help you because that is bursty af. Okay, now we can dig into the results. So, Dr. Wal's channel had a low burstiness, whereas Dr. Mike and Dr. Berg had a high burstiness. So, that's exactly what you'd expect from a channel that's AI. AI models when they spit out text, they want to be able to maintain a persona or character and not surprise you too much.

So, they're going to maintain about the same perplexity and therefore they'll they'll tend to have a low burstiness. The Dr. Oberly channel had a lot of sentences like this. But what if I told you that the way your skin ages is actually determined far more by what happens inside your body than what you put on it?

Whereas for Dr. Mike and Dr. It was all over the place, which is good, right? That's how you know it's human. Okay, so lowers means AI. Well, yes, but I would still be cautious. And let me tell you why. So, you can see that as time passed, Dr. Orberly's channel actually went up in burstiness. It's still well below the other doctors, but it is trending upwards.

So, that is a thing that I'm going to keep a pulse on. But, if it does get to a point where the burstiness is high, like you'd expect from humans, what can we do at that point? Well, at that point, we need more data. So, right now, we looked at like within sentence perplexity, and then we looked at within video burstiness.

So, you can predict what we're going to do next, right? We're going to look at within channel structural predictability. So, how similar are the videos? Basically, the results for this one were the most surprising, and you'll know why in just a bit. So, I calculated this using an AI prototype that I just whipped up.

So, I'm like fighting fire with fire here. Dr. Mike was the least predictable with an average score of 65% predictability, which again is good. And it makes sense because he experiments with a lot of different formats. He's also a very lively person, so you'll often have things that are one-off. And I, you know, I would love to see the bloopers as well.

And here's the good news. Dr. Wley's channel had 80% predictability. So, so the videos were very similar from one another. That's what you'd expect from AI. I honestly thought it would be 100%. But the health topics were different enough to go undetected from this prototype that I made. I think we can make something better.

Leave a comment if you have any ideas. Okay. But the really surprising part about the social predictability is that Dr. Berg had a very high predictability. So that was surprising. But then I looked into the channel and I realized that a lot of them were very same format by board style videos that are edited with some B-rolls.

So that so that's fine, but in my opinion it's dangerous, especially for new creators. Dr. Burke doesn't have to worry because he's got 14 million subscribers. But I don't think this type of content will last because the point of YouTube is to have a connection with the creator. So people are eventually going to want to see humans.

So they're going to want more variety because that is what works. Okay. So out of all the three that we just covered, perplexity, burstiness, and structural predictability, burstiness was the most sensitive. I think we can make it more sensitive with structural predictability.