Can You Trust AI Answers About Your Health?

tl;dr – My experience with kidney stones and AI Answers about them from Google, Microsoft, ChatGPT, and Perplexity.

During the summer of 2020, when the entire world was focused on the pandemic, getting treatment for other health issues became a challenge.

I started experiencing a lot of back pain, but I shrugged it off as an injury and took Advil to cope.

A week later, I thought I had food poisoning.

I tried an at-home service where they pumped me full of IV fluids.

I went to urgent care. They sent me to get scans. I paid hundreds of dollars out of pocket to get them quickly.

The imaging place never sent the scans to the urgent care.

A day later, I had the worst chills. It was July, in Phoenix. Most likely 100 F. I went outside and was still freezing.

At that point, I was taken to the ER. I ended up in the ICU in one of those rooms they zipped up in plastic.

While most of the focus was on COVID, I had something else: a large kidney stone. The kind that doesn’t pass on its own.

I was in septic shock and acute renal failure according to discharge papers.

The hospital stay itself wasn’t too long, but the treatment with specialists took three months to complete.

Surgeries during COVID were extra special because if you tested positive, your surgery was delayed.

Two years later, in 2022, I ended up in the same hospital for the same reason.

And now, I’m a few days into aggressive antibiotic treatment for my kidneys, yet again. Wondering if I make it to the next followup or have to Lyft off to the ER.

So what does this have to do with Google AI Answers?

In 2020 and 2022, I spent a lot of time perusing Google Search results on kidney stones.

Now, I get AI Answers above at the top of SERPs (search engine results pages).

This wouldn’t be a bad thing if one could trust the AI to accurately summarize its sources.

That’s the big if.

In the first screenshot, you’ll find an AI Answer from Google Search results for the phrase how to pass kidney stone.

The first mistake involved an error with paraphrasing a source with legitimate information.

Because I can assure you after four years of seeing urology specialists, no one has ever suggested drinking two quarts or liters of urine.

The basil leaves suggestion, on the other hand, is suspect. I’ve never come across that as a suggestion. It would have stood out, because I have a lot of basil growing around the garden.

I don’t even remember that website from my previous Google searches. I had to check Wikipedia and other sources to find out what the company even was.

Much like the search quality raters and AI Answer checkers do…

In the second screenshot, Microsoft Bing with Copilot offered ads for supplements and advice from five sources, two of which are the MSN Health Hub.

The hub includes a section where you can Ask a health professional questions.

In the third screenshot, ChatGPT using GPT-4 with browsing offered a concise response based on its training data, but claimed not to have access to external sources.

In the fourth screenshot, Perplexity provided the best AI Answer with 19 sources I recognized from my previous research into this health issue.

Moral of the story:

You can’t trust generative AI with your money or your life issues.

But if you do, start with the right AI Answer engine. One that cites sources you trust and doesn’t suggest drinking your own pee.

Think of Perplexity as a better starting point for more in-depth research that you can discuss during your next doctor’s visit.

Not as a definitive answer.

---