Today, The Washington Post published an article titled "ChatGPT is little help for doctors in diagnosing diseases, study finds."
Yesterday, ChatGPT couldn't give me the name of a movie about a one-armed baseball player.
The fact that it couldn't give me the correct answer wasn't concerning, but the frequency in which it gave incorrect answers, the absolute confidence in which they did so, and the fabricated synopses for the movies they gave, was. While my example might be a little ridiculous, the WaPo article was enough for my eyes to pop out a little, considering here at Interactually, we only trust AI to perform some limited digital marketing tasks, where the stakes are quite a bit lower than medical diagnoses.
A bit of backstory: I love baseball movies. For close to 30 years, I've had this vague recollection of a movie about a baseball player with one arm. In this movie, a different boy loses his arm in an accident (traumatizing and probably why I remembered the movie). There was a scene in which the one-armed player shows the boy how to transfer the ball from his glove to his hand. That's it. That's all I remembered.
Over the years, I've tried Googling the title, but no luck. I've begun to think I imagined it. But alas! This is the age of AI! I finally realized the wonderful world of large language models could bring this movie to light. Here's a bit of our conversation:
OK, let's recap:
- The Champ (1979) is neither about baseball or anyone with only one arm.
- The Champ (1989) doesn't seem to exist. I couldn't find anything matching it's description. The only thing I found from 1989 is a New Zealand TV series called The Champion with one season. Also not about baseball or anyone missing an arm. It seems like they just took the title of their first recommendation and made up a movie synopsis out of whole cloth, based on my prompts.
- The Winning Team (1952) is a least about a baseball player, starring Ronald Reagan ("The actor!?"). But, sadly(?) it appears he has all his limbs.
- Pride of the Yankees (1942): I own this movie and have a picture of Lou Gehrig's farewell speech hanging by my desk, so I took this one personally. Especially the "...if you're thinking of a one-armed baseball player" bit. I'm pretty sure I would have remembered that piece of Gehrig trivia.
- The Legend of the Lone Ranger (1981): Yeah, not even close.
- One in a Million: The Ron LeFlore Story (1978): I do like the wording of "He did not have a single arm.." Not even one? Spoiler: He has two.
This is just a bit of the pointless back-and-forth I experienced as I looked up the actual synopses of several movies to see if they were correct, but it seemed like ChatGPT was pulling theirs out of thin air.
Finally, I took to Google, and after entering in the exact same initial query, both their AI Overviews feature and their featured snippet gave me the following answer:
This looks promising... and, what's this in the Videos section? The full movie on YouTube? This is the one! I'm NOT crazy!
A detail I'd forgotten was that this movie was based on a real person. Funny enough, in one of my questions to ChatGPT, it mentioned Pete Gray but immediately threw me off his trail:
So, as much as I don't want to give Google credit for their search results considering the state of things over the past year, they nailed it on the first try.
This has, admittedly, been a tremendous waste of time. Was I looking for an excuse to rant about AI? Possibly. Am I always looking for an excuse to weave baseball into the discussion? Definitely. Is this indicative of the quality and usefulness of ChatGPT for medical diagnoses? Not quite.
According to the full study referenced by The Washington Post, "The LLM alone outperformed physicians even when the LLM was available to them, indicating that further development in human-computer interactions is needed to realize the potential of AI in clinical decision support systems."
But also, according to one of the co-first authors of the study, "All the information was prepared in a way that doesn’t mimic real life."
In other words, ChatGPT and other LLMs could be useful tools in aiding doctors, but doctors need to work on how to use them. And, hopefully, never rely on them completely. In my case, even when I added more context to my initial question, ChatGPT seemingly doubled-down on their incorrect answers with what seemed like increasing confidence.
In a medical setting, one could consider that, "concerning." But, worry not, at least you can sit back, relax, and watch A Winner Never Quits on YouTube (Spoiler: It's not great).
