Deskilling Healthcare

Will doctors and nurses remember how to care for patients?

Source: ChatGPT

There has been a rise in concern about de-skilling and loss of critical, thorough thinking in various fields because of AI. I have written about aspects of this issue, and Princeton professor Kwame Anthony Appiah wrote about it more generally in this Atlantic article.

I am most concerned about this issue in healthcare—perhaps obviously, because de-skilled clinicians could endanger human health. The other general reason why healthcare professionals might let AI de-skill themselves is that they are under considerable pressure to be more productive—at least in the U.S. The average patient visit with a physician is only about 15 minutes, and there are temptations galore to take AI-enabled shortcuts. If AI can easily summarize patient visits in clinical notes, diagnose diseases and communicate with patients about them, and help navigate highly bureaucratic insurance processes, why shouldn’t doctors and nurses take advantage of this capability?

My Family’s Experience with AI Shortcuts

My wife and I have experienced several issues with shortcuts (or the absence of them in one case, I think) in the medical field over the last several weeks.

1. The first event was with my primary care physician (PCP) in California. I had experienced an upper respiratory infection for several weeks that just wasn’t getting better. I was beginning to think that I had a sinus infection, but wasn’t sure if it was viral or bacterial. So I emailed my PCP about it, asking whether I should take something or see a doctor (I was far away from him at the time). He almost immediately emailed me back, and the message began:

Tom [in a different font than the rest of the message],

Thanks for reaching out about the patient with a lingering cough after a recent viral illness.

The emailed response went on to tell me that I should just wait out the infection and cough. But it certainly appeared that my doctor had used generative AI to prepare his response to my message, and that he hadn’t bothered to edit out the incriminating introduction. I asked him, “So what AI model are you using?,” and he admitted to using OpenEvidence, the fast-growing “medical information platform.” I don’t object to my doctor using that tool, but I would like to think that he read it enough to edit out the part revealing that the message didn’t really come from him.

2. Shortly thereafter, I was at a conference on AI in Boston. One of the speakers was Dr. John Halamka, a former prominent medical CIO in Boston and now head of the Mayo Clinic Platform. It is gathering patient information from a variety of sources and incubating healthcare startups, particularly in AI. One of its incubated companies is OpenEvidence. In his talk Halamka described a glowing future for healthcare AI, and mentioned that physicians were becoming increasingly dependent upon it. So I asked a somewhat cheeky question when audience Q&A time came around: “How does this AI-enabled future relate to the events of November 13, 2002?” I guessed that Halamka would know what happened on that date, and he did—even remembering the exact time of day it happened. He was CIO of Beth Israel Deaconess hospital system at the time, and the entire computer network at the hospital went down—for four days. During that time the hospital and its employees couldn’t use any electronic medical records, order labs or procedures electronically, or use any sort of decision support. Were it not for some old paper forms—some accounts suggest they were taken out of a dumpster—and some old modems, all care processes might have broken down. Halamka said in answer to my question that those events point out the importance of “business continuity” approaches, but what happens when people forget how to diagnose diseases and write up clinical notes?

3. My wife and I had patient checkups a few weeks later at a different health provider in Massachusetts. Like many clinical practices these days, the physician makes use of “ambient AI” to capture all conversation in the exam room. Fine, I thought, and we agreed to the use of it. However, the clinical notes for our visits were both inaccurate. My wife’s said that she had chronic fatigue, sleep problems, and a child enrolled at Northeastern University—none of which she said anything about. My notes said that I was visiting the doctor for a chronic cough (yes, the one I mentioned above—although it had passed several weeks earlier, and I made the mistake of telling this doctor about the AI-generated diagnosis message) and shoulder pain, which the doctor mentioned from my California patient records but also was resolved more than a year ago. As a result of this text in the clinical notes—which appeared not to have been reviewed by the doctor—our visits were coded as medical problem visits rather than routine checkups, which resulted in $1200 of medical bills. We’re still trying to get that problem reversed.

4. My last story was not a medical shortcut (to my knowledge) but an admission that they are a problem. I had a routine colonoscopy a couple of weeks ago (I recommend the pills for the awful prep process rather than the exceedingly nasty liquid). It went fine, but as I was chatting with my gastroenterologist before the procedure I asked him if he planned to use AI to identify potential polyps. He was quite familiar with the de-skilling issue revealed in this article, which found that gastroenterologists’ ability to identify polyps declined after only 3 months of AI use. He said he’s worried about it as AI use among doctors performing colonoscopies grows quickly. My doctor’s operating room did have a screen using AI, but he said he tried not to rely on it to identify polyps. Of course, I was sleeping during the procedure, so I have to take his word for it.

So How to Avoid De-Skilling?

These were all relatively minor problems, and our health was never in danger. But it’s easy to see how big problems could result. What can we do to avoid AI-related deskilling? Some degree of regular recurrent training in non-AI-based clinical decision-making—perhaps making use of simulations—may be necessary. Doctors do already have continuing education requirements (particularly to retain board certification in specialty areas), but they don’t generally address this de-skilling issue.

There is another field that has a similar problem—aviation. De-skilling because of automation in aircraft has already been blamed for at least one horrible airline crash. Commercial pilots do have regular recurrent training, but their simulators often use the same automated systems that real planes do. The de-skilling issue is probably greater for general aviation pilots, who have no recurrent training and whose planes are being increasingly automated over time.

In both cases, people can die if AI or automation leads to the loss of critical skills by those in charge of safety. Given the rate at which AI is advancing, professional associations in these safety-critical domains had better move fast to address the problem.