Office of Research & Development |
  |
Talk to the Veterans Crisis Line now
An official website of the United States government
The Stratification Tool for Opioid Risk Mitigation, which prioritizes review of patients receiving opioids based on their risk for overdose, accident, or suicide, is one of four VA programs being rigorously evaluated with the help of VA researchers. (Photo: ©iStock/DNY59)
September 13, 2018
By Mike Richman
VA Research Communications
In 2016, a bill that was aimed at making the U.S. government more effective and efficient became law. The Evidence-Based Policymaking Commission Act was enacted to improve the ability of researchers, evaluators, and statisticians to use information collected by the government to make important policy decisions.
The legislation directed the Office of Management and Budget (OMB) to ensure that government agencies become more rigorous in how they evaluate existing programs. In the case of VA, OMB is providing more than $10 million annually to create, in part, a funding mechanism for randomized program evaluations (RPEs) of the care Veterans receive and the efficiency with which it's delivered. The RPEs would be carried out in much the same way a randomized clinical trial is used to determine which medical treatments work best.
VA has allocated some of those dollars to support its first RPEs, with the agency's Health Services Research & Development service (HSR&D) funding randomized evaluations of four VA programs. VA's Quality Enhancement Research Initiative (QUERI), which is under HSR&D, funds the resource center that is consulting on the design, the Partnered Evidence-based Policy Resource Center (PEPReC). PEPReC is devoted to methods that facilitate evidence-based policy development and practice change across VA.
A VA-funded paper whose lead author was Dr. Austin Frakt, a health economist at the VA Boston Healthcare System, reported key lessons learned in the design of the four randomized program evaluations. He and his team hope that conveying the challenges they faced will help other VA researchers who may want to conduct similar work.
"We were engaged in a systematic effort to design and initiate four large randomized program evaluations across VA."
PEPReC investigators conducted the evaluations, with help from researchers elsewhere in VA. They analyzed the design of RPEs of a home- and community-based care program for Veterans at risk of nursing home placement; an opioid risk mitigation program; and an initiative for suicide risk assessment and prevention.
In developing the RPEs, Frakt and his team tried to overcome a host of issues that often surface in designing randomized controlled trials.
They explained that the successful development of randomized evaluations requires more than attention to traditional design issues, such as ensuring the availability of data. Success also calls for stable financing, a commitment to adhering to a randomized roll-out, a degree of acceptance from staff, and the ability to manage situations when officials show opposition to how the evaluation is being carried out.
Some of the findings in the paper were not at all surprising. Case in point: The researchers noted that successful program implementation and rigorous evaluation require resources, specialized expertise, and careful planning. If the health care system's ability to innovate and grow is to be sustained, organizations will need dedicated programs to prioritize resources and continuously adapt evaluation designs, the researchers indicated.
"We were engaged in a systematic effort to design and initiate four large randomized program evaluations across VA," says Frakt, who is the director of PEPReC and an associate professor at Boston University. "It's a challenging undertaking even to do one. The research community in general doesn't have an opportunity to do these kinds of studies very often. Randomization is hard. We thought that conveying some of the challenges we encountered and some of what we overcame would be useful for the research community."
He adds: "We don't want people to have to have to reinvent the wheel. If we can suggest ways they can avoid some problems and delays and therefore make their program come off more efficiently, that would be a good thing."
AI to Maximize Treatment for Veterans with Head and Neck Cancer
VA opens new research center to seek novel arthritis treatments
VA investigator brings diversity into autoimmune disease research
Under Secretary of Health, panel of experts discusses PACT Act impacts for Veterans
These four VA programs were selected for randomized evaluation of the design. Three of the four evaluations were designed with a randomized arm and a control arm:
In his paper, Frakt and his team listed examples of lessons learned in the design stage of the randomized evaluations, including these two:
QUERI Director Dr. Amy Kilbourne says the information produced by the evaluations is key to determining if a program works and how to get the most out of it.
"It's a very efficient way of doing pragmatic research and to get a question answered in the time frame required by many of our operations leaders, who work at a very fast pace," Kilbourne says. "The randomized program evaluation mechanism also solves another problem: Things sometimes get rolled out quickly without planning or adequate attention to the details of how you design a program to not only make sure it's implementable and can be used in routine practice, or that it's a policy that can be applied in VA practices around the country, but also that the program can be sustained. We always come up with these ideas for programs or policies. They sound good. But we never know if they actually work. The only way to really determine that is to randomize them."
VA is one of the first health care systems, including those in the private sector, to systematically adopt a funding mechanism for RPEs in response to the evidence-based policy plan that OMB has been pushing, Kilbourne notes.
"It's been a government-wide priority through OMB for at least 20 years," she says. "And VA has been the only health care system and probably one of the only government agencies with a national mechanism to enable researchers to do randomized program evaluations to ultimately help VA improve its functioning and organization."
She says the "closest cousin" to VA in this regard is the NIH Health Care Systems Research Collaboratory. In part, the endeavor supports the design and rapid execution of pragmatic clinical trial demonstration projects that address questions of major public health importance and engage health care systems in research partnerships. These projects help to establish best practices and provide proof of concept for innovative designs in clinical research.
"But NIH doesn't have its own embedded health care system like VA," Kilbourne says. "So we're pretty much the model of the health care system by doing these randomized program evaluations. That's why it's so crucial to have embedded internal-funding programs like HSR&D and QUERI to do scientifically supported and scientifically reviewed randomized program evaluations that address top priorities in the VA system."
The randomized evaluations featured in Frakt's study were chosen by HSR&D and QUERI leadership, after HSR&D put out a national call for topics. VA national program offices submitted the names of more than 35 programs, four of which were selected based on rigorous criteria that QUERI developed with PEPReC. The criteria included whether a randomized program evaluation would be feasible, given the nature of the program; whether the program was a high-priority VA topic; and whether the program officials provided a clear statement of how they'd apply the results.
"Would they tweak the program, would they design a better one, would they implement it nationally, things like that," Kilbourne says.
Frakt's paper has limitations. Most notably, the investigators could assess and try to overcome challenges that arose only in the design phase of the four randomized evaluations. "It's possible that other challenges will arise in subsequent phases of implementation and evaluation that we cannot anticipate at this stage," they write.
According to Kilbourne, VA program officials are starting to implement the findings from the randomized program evaluations for the STORM, REACH VET, and VD-HCBS programs. She says it's too early to cite any of the results from the implementation stage, noting that a few other randomized evaluations of VA programs are in the process of starting.
"Now that VA program offices are aware of this opportunity," she says, "we opened the door for any VA investigator to apply if he or she has an operational partner, a program office lead that's interested in doing a randomized evaluation. We just finished our last cycle of reviewing them. We're going to continue soliciting for innovative randomized program evaluation ideas through this funding mechanism."
The Veteran Directed Home and Community Based Services program, recently evaluated with the help of VA investigators, provides Veterans with opportunities to self-direct their support system so they can continue to live at home. (Photo for illustrative purposes only: ©iStock/FredFroese)
A decade ago, the U.S. Department of Health and Human Services, through its agency the Administration for Community Living, began a partnership with VA to serve Veterans of all ages who may be at risk of nursing home placement.
VA and the Administration for Community Living (ACL) launched the Veteran Directed Home and Community Based Services (VD-HCBS) program, which provides Veterans with opportunities to self-direct their support system so they can continue to live at home.
The VD-HCBS program is one of the four VA initiatives that was highlighted in a VA-funded paper that reported lessons learned in the design of four randomized program evaluations. In the case of VD-HCBS, the researchers had determined that every VA medical center that didn't offer the program as of March 2017 would receive it as part of the evaluation. That means 77 VA sites would be included in the design evaluation.
To obtain VD-HCBS and control data from every site, while still allowing each site to receive the program, the investigators decided to randomize the time at which the sites would be referring patients to the program—rather than assign those sites to provide VD-HCBS or not. They designed a plan in which 14 sites would be randomized to refer patients to VD-HCBS every 6 to 10 months. The rest would continue offering their standard method of care to Veterans who may need referral to a nursing home.
Staggering the start date is a way of preventing bias, says Dr. Melissa Garrido, the associate director of VA's Partnered Evidence-based Policy Resource Center, which designed and initiated the evaluation of the VD-HCBS program. She explains that staggering, as a form of randomization, helps investigators ensure that any effects they observe are due to VD-HCBS, rather than to unrelated circumstances occurring at the same time the program is implemented. Those circumstances could involve a policy change, for example, or another program for Veterans who may need nursing homes.
The VA-funded paper cited challenges that the investigators faced in designing the four randomized program evaluations. Regarding VD-HCBS, they could not determine a roll-out schedule for each of the 77 sites up front. Randomization could occur only for sites that had buy-in from on-site officials and funding to hire a local coordinator to oversee the program, and that had gone through a VHA-ACL interagency preparation process lasting several months. In other words, sites will be selected in the order in which they are ready to provide the program.
"The initiation and completion of this process were outside our control," the researchers write. "For these reasons, randomization was to occur every 6-10 months and limited to sites that were ready for the intervention. With a small number of sites that were available for randomization in each wave, we were concerned that simple randomization (by a coin flip, for instance) may lead to imbalance across study arms."
The VD-HCBS program calls, in part, for giving Veterans a monthly budget to pay for the services or support they need to stay in their home. That may involve hiring a caregiver or home health aide, or installing handlebars or other modifications to the home to improve accessibility and help avoid injury. Standard care could mean referral to a medical foster home for Veterans where a caretaker oversees their daily needs.
Based on preliminary evidence, the VD-HCBS program should improve Veteran and caregiver satisfaction and reduce nursing home admissions, according to Garrido. She says the evidence is from VA sites that have implemented the program, not including those in the group of 77.
"However, we need rigorous evidence of whether VD-HCBS improves care for Veterans who are at risk of nursing home placement," says Garrido, who is also a health services researcher at VA Boston. "We're trying to obtain evidence to see whether the program reduces nursing home admissions that are unwanted, whether it reduces emergency department visits or inpatient care, and whether it increases caregiver and Veteran satisfaction. We have preliminary evidence suggesting that this program will do these kinds of things, but this is the first time we've tried to study it in such a rigorous, staggered, randomized manner."
--- Mike Richman
VA Research Currents archives || Sign up for VA Research updates