In part two of our discussion about measurement and metrics and antimicrobial stewardship, we're going to talk about how to put the data that you generate in context, so that you can use it to take action. We're going to focus on the measures of days of therapy and antimicrobial susceptibility as a period prevalence. So, if you haven't seen part one, you might want to view it to familiarize yourself with how the data is generated. Let's talk about what we do with this data that we've gathered including some of the challenges in separating the signal from the noise, as well as national initiatives that aim to make benchmarking much easier. Once you obtain your data, whatever it is, it's important to put it in context. After all, knowing you use X number of days of therapy per a thousand patient days of an antibiotic, isn't intrinsically meaningful. One level of comparison is to sum absolute standard. What should that number be? In a number of quality improvement contexts, we can set an absolute standard goal, no central line associated infections or 100% hand hygiene compliance. Those standards may be high, but they represent a clear goal. However, for some measurements there is not a reasonable absolute standard. We certainly don't want our antibiotic use to be zero and currently there is no magic number that represents the correct amount of total antibiotic use, thus we may be forced to use other standards for comparison. One way is to use your own institutions prior performance as a reference standard. When doing so, you'll want to make sure there's adequate data to ensure that you're seeing a real effect rather than just some random variation. That you use the best techniques to estimate the effects of any interventions that might have occurred over the time period. Another means of contextualizing the data is through comparison to other groups. These might be other hospitals or their teams or services within an institution. For this comparison, challenges can include obtaining data from comparators, and ensuring that a comparator really represents a good benchmark for your institution. Okay. So, first let's talk about how do you use your institution's own data to provide context through antimicrobial stewardship. That particular emphasis are the common mistakes that can be made when analyzing this data longitudinally. Mistakes that can lead to substantially overstating the effects of your program you're wasting your efforts on problems that are just random variability. On first glance, this graph appears to be an example of stewardship metrics demonstrating the success of an Antimicrobial Stewardship Program. Over an eight month period, vancomycin use in adults as measured by days of therapy per thousand patient days seems to be definitively trending down. However, there are substantial variations in month-to-month antimicrobial use, such that what may seem to be patterns over a period, even as long as eight month, are part of the background variability. Note as well the impact that the axis scale has on the perceived magnitude of the fluctuations. Tracking usage over longer time periods can account for seasonal and other fluctuations, and reveal long-term trends that may indicate targets for stewardship intervention. Tracking stewardship metrics over longer time periods as in the example here with meropenem separability and pseudomonas, can collapse some of that month-to-month noise, and may give confidence that a trend that is apparent is actually real. However, it's important to remember that variability still exists that should be accounted for, especially when organisms that are less commonly isolated are being tracked. Applying confidence intervals or error bars to the data points can indicate the degree of uncertainty around estimates and reduce the likelihood of overstating trends, again, most helpfully when combined with a greater range of data points. For more common organisms, there's less uncertainty and significant trends and shifts over time can be detected with a greater degree of confidence that these are areas to definitively target. What about when an intervention is actually put into place? What's the best way to measure its effect? Here's an example of antimicrobial use data over a three-year period in an institution, although the concepts apply to any data that can be measured at repeated intervals over time. Let's consider this data is being collected to evaluate the impact of an Antimicrobial Stewardship Program, which is given one year to show its effect on utilization. We can ask what the best comparisons to perform on this data set might be, what might compare the use just before the program was implemented to the utilization at the end of the study period is indicated by A, but in this case, it would give a misleading story that there is little effect of the program. Even worse, would be comparison of the utilization immediately before to immediately after the initiation of the stewardship program. It's unlikely there would be enough time to see a true effect and instead the random variation might lead to the conclusion that the program increased utilization. Many studies would report the mean use in the period before the intervention, and mean use during the intervention period indicated by C. But this doesn't count for the underlying trend in utilization, which was clearly increasing before the intervention, and which flattened out afterwards. More accurate comparison would be to compare the observed trend in antimicrobial use after the intervention, to the projected trend in utilization if the intervention had not occurred noted by D. Although slightly more complicated statistically, this interrupted time series approach is recognized as the most valid way to analyze and present such data. There are a number of real life examples of studies where comparing the mean value of a metric before and after an intervention gives an incomplete or even false interpretation. In this case, the effect of an infection control intervention seems to have clearly reversed an outbreak of ceftazidime resistant Klebsiella, yet the mean rate of ceftazidime resistant Klebsiella wasn't statistically significantly different. Interrupted time series analysis, however, would indicate that both the immediate effect of the intervention noted by the change in level, and longer-term effect of the intervention noted by the change in slope, were positive. The opposite effect can also be seen, in this case the mean number of doses of vancomycin per one thousand patient days decreased after an intervention to restrict its use. But the utilization was clearly down trending before the intervention. While associated with a modest immediate reduction in use, the longer-term trend after the intervention was actually an increase in the slope of vancomycin utilization. Our example show that actionable data can be gleaned from looking at trends in antimicrobial stewardship metrics over time, but that a lot of data points in a long period of time are required to see an effect. Many institutions will want a more immediate snapshot of what areas they're performing well in, and what areas are targets for improvement. For this, obtaining data from peer institutions can be invaluable, but fraught with its own potential for error. This figure reports the aggregate antimicrobial use in terms of days of therapy per thousand patient days across 70 university hospitals. Even though these are all academic medical centers, there's nearly a twofold variation usage from the lowest to highest users. In order to properly benchmark your own institutional data to this data, we'd like to be able to isolate what component of the variability comes from potentially improvable practice patterns, and what is the result of different mixes of patients across these institutions. This figure illustrates the impact that patient mix can have on utilization. In this hypothetical, and somewhat extreme for illustration purposes, example, two hospitals, A and B, have different percentages of patients on medicine, psychiatry, and transplants services. Although each service has the same utilization rate per patient for each service, five days of therapy per thousand patient needs for medicine, one for site patients and 10 for transplant patients, the total utilization at institution A, is much higher because they're mix includes higher use patients. Thus, these hospitals have similar rates of modifiable antibiotic use but much different overall usage rates. This graph illustrates the same principle. Here, instead of comparing entire hospitals, antibiotic usage among patients on particular services in each hospital is plotted. This allows institutions to see where they compare in treating similar groups of patients, removing the effect of different patient mixes. Obtaining and analyzing the data for peer bench-marking can be a significant challenge. However, the CDC has made a huge step forward with the antimicrobial use module of the National Health Care Safety Network surveillance program. Institutions that electronically submit their antimicrobial use data through this platform, contribute to statistical models that adjust for patient mix and other inter-institutional variables. These models compare the measured antimicrobial days in a facility, to the predicted antimicrobial yet days to come up with a Standardized Antimicrobial Administration Ratio. This SAAR can be used as an indicator as to whether usage during that time period was above or below that which would be predicted for the institution, based on its specific characteristics. Analysis for all antimicrobial as well as for specific groups of drugs and hospital locations are available. Ask your infection control and IT folks today about participating in the antimicrobial use in resistance module. What follows is an example of how antimicrobial stewardship metrics might be used to identify a high yield opportunity for stewardship intervention. It combines longitudinal analysis with analysis of peer bench-marking data. This is data from our institution as to the use of common antimicrobial and groups measured in days of therapy per thousand patient days, along with the mean values for a group of nine comparator institutions. This is a more useful way to view the same data, as differences from the comparator mean. In a number of cases, differences can be explained by substitution effects. For example, while UCSF uses far more piperazine tazobactam than the average, it uses much less cefepime, such that there is little difference in anti-pseudomonas penicillins and cephalosporins as a group. We were surprised that the next biggest outlier was vancomycin. So, we decided to look into it more. It turned out that UCSF, in red here, had the second highest usage in the group of hospitals, substantially more than institutions at the lower end. This effect wasn't readily explained away by substitution with alternative agents. Total anti-gram positive use was still high driven by vancomycin use. Perhaps we're seeing something that was actually improving. We had made efforts to monitor vancomycin use. But, looking at the data longitudinally, it was clear that we had seen a steady increase in vancomycin over time, even as the mean usage at the peer group of hospitals, indicated by the dotted red line, was essentially flat. Perhaps the increased use of vancomycin was clinically justified, based on an increase in MRSA. It can be tricky to answer this question, since potentially the MRSA rate could be held down by administering vancomycin widely before our cultures are drawn, but our surveillance data suggested that while MRSA had been previously increasing, during the period we're analyzing our usage data, it was actually flat. A useful bit of information was figuring out what was driving the increased number of days of therapy of vancomycin. We are able to separate out the mean duration of vancomycin the patients received, noted with the dotted dark line, from the number of new starts of vancomycin, noted by the red line. Over the time period, the mean duration of vancomycin among patients who received it actually went down, but the number of new starts increased substantially, such that 25 percent of all adult admissions for any indication were receiving vancomycin during their hospitalization. There's clearly lots of vancomycin use in the hospital. It can be a challenge to know whether to implement a hospital-wide intervention or targeted intervention. So, we wanted to see whether or not there was a substantial difference by service in this increased utilization of vancomycin. We had anticipated that the bone-marrow transplant unit would be a large user based on the inclusion of vancomycin into the febrile neutropenia pathway. But we're surprised to identify the general medicine service, not only has a high volume user, but one substantially above our peer hospital mean. This analysis allowed us to identify vancomycin as a real outlier in terms of utilization, and to target a specific service for initial intervention. We've made some changes and we'll be analyzing the effect when, of course, we've gathered enough longitudinal data points. So, in summary, key metrics for antimicrobial stewardship have lots of noise obscuring the signal. So you will need to gather lots of data points and estimate the uncertainty. There's a best way to measure the effect of a stewardship intervention. Graphs make the visual case, an interrupted time-series analysis makes this statistical case. Bench-marking to peers has availability and apples to apples comparison problems, but the development of the CDC national database is a significant step forward. Finally, I hope our example should the combining longitudinal and benchmark analysis can be very powerful. I hope this has been a useful module for you, happy stewarding.
