Surgical checklists have little effect on patient outcomes, study finds

Share

When medicine first borrowed the concept of safety checklists from aviation, the idea was that surgeons, like pilots, could reduce the risk of fatal complications with some simple verbal confirmations and the ticking of boxes.

The vast majority of research on the checklist effect backed this up, leading as many as 6,000 hospitals around the world, including 88 per cent of Canadian hospitals, to adopt what seemed like a cheap and easy route to safer operations.

But now a large new Canadian study is calling the checklist orthodoxy into question.

Researchers in Ontario found surgical checklists introduced in the province between 2008 and 2010 did not measurably reduce deaths or surgical complications.

Plumbing a rich trove of health-care billing data collected by the Institute for Clinical Evaluative Sciences in Toronto, the investigators compared procedures performed in the three months before and after the checklists were adopted – for a total of more than 215,000 operations – and found the decrease in deaths and complications was too slight to count as significant.

"There was always a bit of a sense on the part of some people that the concrete safety benefits seemed a little too good to be true," said David Urbach, a general surgeon at Toronto's University Health Network and a co-author of the study, published Wednesday in the New England Journal of Medicine. "It just didn't seem plausible that these tools could be as effective as they were shown to be in these studies."

Dr. Urbach was referring to a wealth of research, including an oft-cited 2009 study, also published in the NEJM, which found the World Health Organization's 19-point safe surgical checklist program nearly halved the rate of death in eight hospitals in eight cities, including Toronto.

Further research tended to back that up, though not always as dramatically, including a study of 74 Veterans Affairs facilities in the U.S. that found an 18-per-cent reduction in deaths one year after an extensive surgical checklist training program was put in place.

Two of the authors of the original WHO study are raising questions about the new research – namely, how can anyone be sure the checklists were not ticked off robotically and meaninglessly?

One of those authors is Atul Gawande, a surgeon, Harvard professor and New Yorker staff writer whose book The Checklist Manifesto helped spread the surgical checklist program around the world.

He said checklists are nearly useless if not hammered home with extensive training, the effect of which takes longer than three months to show up.

"It's a huge culture change," Dr. Gawande said. "When aviation came up with checklists – they came up with it in the 1930s – but right into the seventies and eighties, captains routinely said, 'eh, that's not for me.' After, you know, a gazillion airplane crashes, they finally began to change the culture, which required that people really be trained in how to use [checklists] from the get-go."

An NEJM editorial accompanying the new study echoes this, saying that without monitors watching the surgical teams at work, "the likely reason for the failure of the surgical checklist in Ontario is that it was not actually used."

Another author of the 2009 study, Thomas Weiser, a surgeon and assistant professor at the Stanford University School of Medicine in California, said his biggest concern about the new study is that it could "arm the naysayers and keep people from implementing the checklist or spreading the checklist in a meaningful way."

In Ontario's case, 97 of the 101 hospitals analyzed in the new study said they offered special education and training when they introduced checklists.

Most used a version of the list designed by the Canadian Patient Safety Institute that includes checking anesthesia equipment, verifying the patient's identity, taking a timeout before incision and debriefing after surgery, among other steps, for a total score out of 100.

The organization that accredits Canadian hospitals made surgical checklists a required organizational practice in 2011; by the following year, 88 per cent of Canadian hospitals had adopted the lists.

"Checklists are here to stay. No one is advocating that they be done away with," said Dr. Urbach, adding his new study is a reminder that the medical profession needs to be "realistic" about how difficult it is to improve patient safety.

Training and teamwork are key, he added.

"It's not as easy as a checklist."

Follow me on Twitter: @kellygrant1