Office of Research & Development

print icon sign up for VA Research updates
thumbnail thumbnail

Amy Binns-Calvey, who today works at VA's Center for Innovation for Complex Chronic Healthcare, had previously parlayed her acting experience into a stint as an unannounced standardized patient, part of research led by Dr. Saul Weiner. (Photo by Jenny Fontaine/UIC)

'Mystery shopper' model being used to boost VA care

March 21, 2018

By Mitch Mirkin
VA Research Communications

See below

It may not have won her an Oscar, but it was a 'calling'


Amy Binns-Calvey is today a coding manager at the Center for Innovation for Complex Chronic Healthcare, assisting with Dr. Saul Weiner’s research. A few years ago, she was on the front lines, working as an unannounced standardized patient in one of Weiner’s studies. Read more





Highlights of work with standardized and real patients


2006—VA’s Health Research and Development Service (HSR&D) funds Dr. Saul Weiner’s proposal to use actors portraying Veteran patients to help identify and predict “contextual errors in medical decision-making.” Read more

If you’ve ever worked in retail, you may be familiar with the concept: A “mystery” or “secret” shopper comes through the store, pretending to be just another customer, and later reports back to the company on how good the service was.

Thanks to the research of Dr. Saul J. Weiner and colleagues, based at the Jesse Brown VA Medical Center in Chicago, the same model is being used to improve VA care and services.

One version uses what Weiner’s team calls “unannounced standardized patients,” or, in the case of non-medical settings, “unannounced standardized Veterans.” These are actors trained to appear as real Veteran patients or clients. There’s a lot that goes into orchestrating this; more on that below.

Another version relies on actual VA patients who agree to audio-record their visits with VA doctors—ideally, surreptitiously—to help improve the care they and their fellow Veterans receive.  

At first glance, these methods might seem to raise ethical flags. Don’t these participants—whether actors pretending to be Veterans, or real Veterans secretly recording their visits—have to lie to, or at least be less than open with, the staff who are earnestly working to take care of them?

One point to counter the concern is that VA staff know ahead of time that a “mystery shopper” will be coming through their clinic or office. They have agreed to it as part of research or quality improvement. They just don’t know who it will be, or when exactly.

There are other important factors that have not only made the approach acceptable to VA doctors and other providers, but also caught the attention of top policymakers in the agency who are excited about using it to get an honest, real-world picture of how VA is doing, and to detect problems that need fixing.  

First off, says Weiner, doctors know that “this will never come back to bite them. It’s confidential.” His team scrubs the real identities—both the patient’s and the doctor’s—from the data, and reports gets aggregated across multiple providers. “The only way we share provider-identifiable data is that we send providers individualized reports for their own benefit.”

Second, in an age when doctors often feel overwhelmed by administrative chores, they are happy that Weiner’s approach to quality improvement does not add to their workload. “It’s not burdensome,” he says. “We’re not asking them to do anything other than provide usual care.”

Dr. Saul J. Weiner is deputy director of the Center for Innovation for Complex Chronic Healthcare, based at the Jesse Brown VA Medical Center in Chicago. (Photo by Jenny Fontaine/UIC)
Dr. Saul J. Weiner is deputy director of the Center for Innovation for Complex Chronic Healthcare, based at the Jesse Brown VA Medical Center in Chicago. (Photo by Jenny Fontaine/UIC)

Dr. Saul J. Weiner is deputy director of the Center for Innovation for Complex Chronic Healthcare, based at the Jesse Brown VA Medical Center in Chicago. (Photo by Jenny Fontaine/UIC)

Third, providers can use the activity to help maintain their board certifications, which typically require many hours of seminars or other learning. “We’ve worked out an arrangement with the American Board of Internal Medicine and the American Board of Family Medicine to give them credit toward their board re-certifications,” notes Weiner. “This is great for them, because they don’t have to do any extra work, and they get 20 hours of credit.”

Concept has long history in medical world

The idea of unannounced standardized patients goes back several decades. A famous early example was the “Rosenhan experiment,” in which a Stanford University psychologist and his team posed as patients and feigned psychotic symptoms to test how well doctors were diagnosing such conditions.  The study appeared in 1973 in the journal Science.  

Later work was spearheaded in VA by Dr. John Peabody.

“He published a series of papers based on bringing unannounced standardized patients into VA,” says Weiner. “I communicated with him extensively about 15 years ago when I started doing my research, to understand his methods.”

The approach has gotten more complicated in the era of electronic health records, explains Weiner, who is deputy director of VA’s Center for Innovation for Complex Chronic Healthcare.

“Initially, it was really easy to create a fake patient. The recordkeeping was paper-based. Someone could show up, and it was not hard to portray yourself as being a VA patient. We could create fake notes.

“But, as the electronic health record started to become increasingly sophisticated, it required partnering with various offices to create the fake patient record in the database. Then, after the visit, you have to make sure this fake data does not corrupt downstream databases. So I work with another office that removes the data.”

To get a phony ID badge for a standardized patient, Weiner’s team works with yet another VA office.

The actors, for the most part, have come from the University of Illinois at Chicago, which has a center that trains people to be “standardized patients” for the medical school. That idea—without the “unannounced” part—is a staple of medical education. The actors learn how to present like patients with various health conditions, to help future doctors build their interviewing and diagnostic skills.

"Our actors are thoroughly trained. They have a comprehensive script."

“These actors are simply role-playing cases,” says Weiner. “Everyone knows they’re role-playing. You do it in medical school to train medical students and residents in interviewing different types of patients. Nearly every medical school in the country does it now, and many have been doing it for decades.”

Taking those actors and teaching them to present as Veteran patients—to the point where VA staff won’t be able to tell them from real patients—involves additional steps. The actors have to look and sound like the real thing. Whatever the researchers are doing to prepare them has apparently been working, as “detection rates are low,” says Weiner.

“Our actors are thoroughly trained. They have a comprehensive script. We actually pair them with a Veteran, so that they have a whole backstory.  It’s just like actors in a movie. If someone is portraying a Vietnam Veteran, we pair them with a Vietnam Veteran so they have a comprehensive backstory, and are prepared to have conversations.”

And what if a doctor—who in some cases will be a Veteran, or from a military family—starts asking too many questions about particular people, places, or deployments?

“We give our actors techniques to deal with that type of situation,” says Weiner. “If they feel they’re getting into uneasy territory, they say things like, ‘I was somewhat traumatized, and I’d rather not talk about that.’”

Targeting ‘contextual errors’ in medicine

In Weiner’s research at VA medical centers, the focus has been on what he calls “contextual errors.” He and colleague Dr. Alan Schwartz, a psychologist, wrote a book about the concept, titled Listening for What Matters: Avoiding Contextual Errors in Health Care. The duo also maintain a website called Contextualizing Care.

Every actor-patient has a “story” that he or she is prepared to tell, offering clues about life challenges relevant to their health care that the doctor should pick up on and take into account when deciding what services and support the patient needs. But will the doctor listen well, read between the lines, ask the right questions, and tease out those facts? Weiner’s research indicates it’s a skill that’s lacking in many physicians; his mission has been to close the gap.

The scenarios he equips his actors with come from real life, not textbooks. They are based on his actual experiences seeing patients, or on the experiences of colleagues.

In an article in Chicago Medicine in 2016, Weiner and Schwartz wrote about one of the scripts they had used in their research, and the results. The tale is interesting not only because of how few doctors picked up on the contextual clues, but also because of the costs that patients and health care systems incurred when doctors didn’t get it right:

“One, for instance, portrayed an elderly man who came to the doctor’s office complaining of weight loss. His pants were loose. He also gave several clues that he was homeless and not eating regularly. Before we hired an actor in his 70s to learn the script and start visiting doctors in our study, we showed it to a group of board-certified physicians and asked them how they would manage such a patient, after confirming he was food insecure and probably malnourished. Four out of four said they would focus on strategies for resolving his food insecurity, such as referring him to Meals on Wheels.

“However, when we deployed the actor into the field as an ‘unannounced standardized patient,’ meaning that he stayed in character during medical visits and followed the same script at every encounter, we discovered that most of the time physicians did not address the evidence that their patient’s weight loss was related to a particular life challenge (lack of access to food). Only a minority addressed his social situation. The few who picked up on the food insecurity and delved into the situation rarely ordered expensive tests. The rest ordered an extensive malignancy workup, including CT scans, colonoscopy, and other tests.”

Aside from contextual errors, Weiner’s research has also looked, more simply, at whether doctors are doing the routine things they are supposed to be doing during medical visits.

“One thing we’ve found,” says Weiner, “is that what doctors put in the chart is often not what actually happened during a visit. A physician might enter, ‘I helped the patient on smoking cessation,’ whereas maybe all they did was say, ‘you need to stop smoking.’ But they didn’t use an evidence-based approach.

“Or, a physician might say he did a diabetic foot exam, when in fact he never had the patient take off his shoes. We can tell that from the audio and the checklist the actors completes. Another doctor who says he did one may have actually gone through all the proper steps.”

Screening for depression is another area where providers commonly fall short, says Weiner.

“They say they screen for depression, but we can hear whether they did or not. Unfortunately, very often they ask only a few basic questions that don’t really constitute a screen. Others will really go through the PHQ-9, which is the formal VA screening tool for depression.”

VA RESEARCH
TOPIC PAGES

Weiner’s group has also looked at contextual errors involving real patients. The way it usually works, the researchers will have a table set up at a primary care clinic, where they invite patients awaiting their appointments to become involved in a quality-improvement project. Those who agree are given an encrypted recorder, which they can either conceal or keep out in the open when they meet with their provider. They hand the recorder back to the researchers after the visit.

Given the ethically sensitive nature of the work, it is overseen by a national advisory board that VA set up in 2015.

Unlike the standardized patient method, the one involving real patients does not allow researchers to compare how the same person is treated by different doctors, across different settings. But it has its own advantages, and offers a unique window on the actual care Veterans receive.

"We've built and published a coding system for listening to those recordings and scoring the physicians' performance in a variety of areas,” says Weiner. "We've been able to show that what we measure is significant. It can predict how Veterans are going to do."

When real patients choose to keep the recorder out in the open, the data are tagged accordingly by the research team, versus data collected without the doctor’s awareness. But the recordings are still useful in improving quality.

Over the past decade, thousands of real patients have taken part in Weiner’s research, providing observational audio data from their doctor visits, and the team currently has VA operational and research funding to use and study the method at VA primary care clinics in six cities, with the goal of improving care.

Weiner: “We generate reports based on what we heard, we feed that back to the [primary care] teams, and based on that, they try to improve the care they provide to the next round of patients.”

Method now in use to curb Veteran homelessness

Building on his studies around medical care, Weiner was invited by senior VA executives to bring the method to VA’s homeless programs. Here, the focus is not on contextual errors, but on whether these “homeless Veterans” (as played by actors) are getting the services they are supposed to get, and with the right level of respect and caring.

During 2017, “unannounced standardized Veteran” visits took place at 30 Community Resource and Referral Centers (CRRCs) nationwide. These are VA sites, separate from VA medical centers, that help Veterans who are homeless or at risk of becoming so. The sites are not shelters, although they do directly provide food, showers, and laundry facilities. Mainly, they offer assessment and case-management and partner with community agencies, and with other VA programs, to help Veterans avoid or come out of homelessness.

The actors pretending to be homeless or at-risk Veterans didn’t audio-record their encounters with VA staff, but they filled out questionnaires right afterward to document the experience.

As with Weiner’s medical-center work, the key was transparency vis-à-vis all the staff involved.

“We told all the [Veterans Integrated Service Network] directors, the facility directors, all the unions, so they would understand this is not going to be used in any punitive way. We had a phone call with all the CRRC directors and told them this would happen, although we didn’t say when. We assured them it would never be used in a negative way—only to help improve care.” 

The actors selected for the project represent the diversity of Veterans experiencing homeless, in terms of age, skin color, military service, and health conditions, such as  PTSD and addiction.

In the tradition of the unannounced standardized patient, each actor visited several sites. The goal is to create a rigorously controlled, scientific experiment, to the extent possible.

Weiner’s team scored the data from the visits, with an eye primarily on access: Did the “Veterans” receive housing referrals, food, counseling, or other services they needed? The team also assessed how streamlined the process was, and whether staff displayed VA values such as integrity and respect. The goal was not to catch and penalize poor-performing sites or individuals, but to identify actionable issues that could be addressed system-wide.

The project illuminated a number of barriers to care. For example, at several sites, Veterans seeking services were asked to show a DD214—military discharge papers—even if they already had a valid VA ID card. Thus, one quality-improvement goal became doing away with this requirement.

A second wave of “unannounced standardized Veteran” visits is now underway at CRRCs to gauge how well this change and many others have been adopted, and to possibly reveal other issues to address. The actors are also visiting community service organizations that contract with the VA through the Supportive Services for Veterans Families program.

‘What VA is doing … is extraordinary’

Weiner sees this all as a remarkable application of the research.

“I was so impressed when the national homeless program office sought me out. They’re taking our most vulnerable population, Veterans who are homeless, and adopting a methodology that’s usually used for high-end shopping and retail, where money flows in, to optimally serve those who have almost no voice. It’s an incredible sign of commitment to these Veterans.”

And, says Weiner, it represents an unparalleled level of accountability for an organization.

“We’re holding a bright light to ourselves. How can we be absolutely sure that we’re providing the best services to these vulnerable Veterans? We’re not just interested in what is being said up the chain of command. We want to know what’s actually happening on the front lines. The unannounced standardized Veteran [method] is the gold standard. It doesn’t deceive. It’s ground truth.”

The method can serve to uncover innovative best practices, says Weiner. He gives an example from the CRRC project:

“There were situations where [the actors] would show up at a CRRC and were told they would have to go to a local VA facility to register. ‘I’m sorry, but it’s six miles away, and we don’t have a shuttle. You’ll have to find your way over there. Once you register, come back.’ Now, that’s not very helpful. Not a good way to do it. We actually found that more than half the sites were doing that.

“Other sites said, ‘You have to register at the local VA, but we have a shuttle to get you over there. It leaves every hour. Then you can come back and we’ll register you.’ That’s better. But then we had other sites that said, ‘Hey, we’ve worked out a process here. We’re going to just pick up the phone, call over to the local VA facility, and we’ll have you registered in 15 minutes.’ We collected all that. The exact same actor going to all these different sites had these different experiences.”

Based on the finding, the VHA Homeless Programs Office decided to disseminate this best practice—streamlined phone registration—across all the CRRCs.

Weiner points out that the improved approach required no extra resources on the part of VA, “because another facility had already solved the problem,” simply by being more resourceful and creative.

He sees a big advantage to disseminating best practices based on standardized-patient projects, such as those he’s conducted with the homeless programs or various medical centers:

“What’s compelling about it is, other facilities can’t say, well you don’t understand, our patients are different. No they’re not—we sent you the exact same patient!”

It may not have won her an Oscar, but it was a ‘calling’

Amy Binns-Calvey holds a photo of her late father, James
Amy Binns-Calvey holds a photo of her late father, James "Budd" Binns. An actor who became involved through his daughter in working for VA research as an unannounced standardized patient, he was also a World War II Coast Guard Veteran and actual VA patient. (Photo by Jenny Fontaine/UIC)

Amy Binns-Calvey holds a photo of her late father, James "Budd” Binns. An actor who became involved through his daughter in working for VA research as an unannounced standardized patient, he was also a World War II Coast Guard Veteran and actual VA patient. (Photo by Jenny Fontaine/UIC)

Amy Binns-Calvey is today a coding manager at the Center for Innovation for Complex Chronic Healthcare, assisting with Dr. Saul Weiner’s research. A few years ago, she was on the front lines, working as an unannounced standardized patient in one of Weiner’s studies.

Binns-Calvey had long been an actress, as well as a playwright and director, so the acting came easy. “I grew up in a theater family,” says the affable baby boomer. “My father was an actor his whole life.”

At one point, Binns-Calvey involved her dad, James “Bud” Binns, a World War II Coast Guard Veteran, in the research, too. He passed away in 2011, and had been an actual VA patient when he took on the unannounced standardized patient role.

Bud Binns, shares his daughter, had lied about his age to serve during the war. “He was actually too young, but he went ahead and enlisted.”

The lying was for a good cause—a theme that perhaps resonates just a tad with the family’s work aiding medical research.

‘It feels like a calling’

“I’ve worked with a lot of standardized patients and unannounced standardized patients, and I’ve been one,” says Binns-Calvey. “When you are working in the medical profession in this way, you almost feel like it’s much more important than acting. It feels like a calling. The standardized patients who are able to give feedback to medical students feel incredibly important, and it’s the same thing with the unannounced standardized patients. You feel you are helping patients.”

It wasn’t that she didn’t feel a bit guilty now and then, seeing how caring and empathetic some providers were toward her fictitious patient character. But she says the tinges of guilt were assuaged by the knowledge that none of the data she collected would ever be used to call out specific providers to others, and none of it would ever be punitive.

“I think a lot of actors would have trouble doing this work if you felt like you were ratting out somebody, or spying on them, or playing a ‘gotcha’ game. Because the projects I’ve worked on are not at all punitive, I can do it with a clean conscience.”

Binns-Calvey also appreciates that providers aren’t kept in the dark for long about which of their recent patients was not real.

“As soon as we collect the data that we need, usually within a couple of days, that provider is notified that the patient they saw was an unannounced standardized patient. So that way they are not expending a lot of energy trying to follow up with that person.”

Still, when she was an actor-patient herself, she sought opportunities to express her gratitude directly to the doctors she had seen.

“I was able to write them a letter when the project was done, just to say thank-you for taking such good care of the patient.”

She adds: “I actually met one of the physicians that I had visited, at a conference, and I was able to talk to her and express in person how much I appreciated the care she gave to my character. So that was satisfying.”

Past role: A patient considering hip replacement

Binns-Calvey’s character was a woman in her 40s who was considering hip replacement surgery, but was also the sole caregiver to a son dying of muscular dystrophy. The character was unaware of the extensive recovery time from the surgery; had she known, she would have delayed it for the sake of her son. It was the doctors’ job to get at that context, so as to best advise the patient about the surgery. The case was based on a real one Weiner had seen.

“It was pretty shocking how few providers picked up on the clue. And I was so ready to give them my story,” recalls Binns-Calvey.

During the course of some 40 visits as a fake patient, to 40 different providers, Binns-Calvey was able to keep her cool, and respond appropriately to whatever questions came up. She says not every run-of-the-mill actor is up to the job. Her experience doing improv—which is big in Chicago—helped, she says.

“It takes a certain kind of person—someone who can think on their feet and remain calm in stressful situations. It’s stressful enough to go to a doctor, just as a patient, but as an unannounced standardized patient, you don’t know that the phlebotomist isn’t going to walk in any minute and want to take your blood. It can be very stressful.”

And was she ever detected? Well, almost.

“Once, I ended up getting an appointment with a provider who would never see women patients. He was sort of a character. He said to me, ‘You’re not a fake patient, are you?’ And I was like, ‘Oh my gosh, no. Who would do that for a job?’”



Highlights of work with standardized and real patients

2006—VA’s Health Research and Development Service (HSR&D) funds Dr. Saul Weiner’s proposal to use actors portraying Veteran patients to help identify and predict “contextual errors in medical decision-making.”

2010—Main study results published in Annals of Internal Medicine, showing that “inattention to contextual information, such as a patient’s transportation needs, economic situation, or caretaker responsibilities, can lead to contextual error, which is not currently measured in assessments of physician performance. “

2013—Veterans Integrated Service Network 12, in the Midwest, funds quality-improvement program based on patient-collected audio to reduce contextual errors during medical visits.

2016—HSR&D funds project to advance Weiner’s work with real patients into the implementation stage. VA’s Homeless Programs Office engages Weiner’s team to improve services at Community Resource and Referral Centers (CRRCs) nationwide.

2017—First wave of unannounced standardized Veteran visits at CRRCs.

2018—Second wave of visits at CRRCs, to measure progress and identify further areas for improvement.


VA Research Currents archives || Sign up for VA Research updates


Questions about the R&D website? Email the Web Team.

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.