In this section, we'll talk about evaluation, which is what you did any good? Why should you care about evaluation? Well, we're going to tackle that in a few minutes. First of all, let's understand what do I mean by evaluation. I'm surprise I'm using the stack here. Remember that the top of the stack is what you want to accomplish, the bottom is how you accomplish it. Well, evaluation is checking, if what you put together in lower half the stack actually accomplish what you wanted to accomplish. So, checking that whether the thing below the line is working at all, it's called testing in my language. Checking whether it's accomplishing what you want, that's evaluation. The most difficult thing about evaluation is this issue about, are you looking at what you really care about? This is the best picture we can put together of a drunk, walking, looking for other keys under the light. I think you may have heard; the drunk is looking for the keys and you go and start helping them. After a few minutes, you say, "Where did you lose your keys?" The drunk says, "Two blocks away" You say, "Why are you looking here? " They say, "Because that's where the light is" This is a big danger with all systems in general and IT in particular that we measure what's easy to measure in order to evaluate what's going on, but we don't necessarily measure what needs to be measured. So, why do we evaluate? On the left hand side, we see the for, right? You want to know where you're starting from. If you're building, if you're going to spend hundreds of billions of dollars in electronic health working systems, in order to accomplish certain goals, you'd like to know where you started from before you say whether or not you accomplished those goals. You also want, let's say you're in the middle of a deployment of something, maybe there's a midcourse correction, maybe things are not going the way they're supposed to be going. Maybe they're having an unintended consequence, maybe you need to make some sort of a fix or you finished applying, everything is in, but you know they're not the best they could be. Well, where should you optimize? How should you optimize? If you don't evaluate what's going on, you don't know. Then going to the next step, if you want to know the next new thing, now is it time to get that next new thing. The bottom line is that, evaluation is what defines the should between can and should. At the right hand side though, there are very strong reasons that mitigate against evaluating. One is, it takes effort and we still want to do. If you want spend money optimizing something or you want to spend money figuring out whether you should optimize something. Very often, you need to go outside the system to figure out whether the system is working. The only way you can find out that piece whether or not people are happy with the system is to survey them outside the system. Finally, people don't want to direct because they don't want to get fired. You maybe spenting $20 million on CHR, it's [inaudible] evaluation shows it's horrible, you're fired. The best way of preventing that from happening is don't evaluate. So, there's a real tension. People who get informatics training are generally called upon to design the evaluation or to pass judgment about an evaluation were to look what's happened somewhere else. So, it helps to have some facility with what we mean by evaluation in informatics. So, I'm going to give you the real complicated notions of evaluation. Here, it requires knowing three letters x, y and z. So the first thing we start about is with y. Why? Well, because until these x's and y's in statistics. Well, we start with y. Y is the thing you care about. So, if you put an HR in order to improve efficiency or to maximize efficiency, then y is efficiency. If you put in a system to minimize mortality, then y is mortality. If you put in a system to maximize revenue, then y is revenue. If you get the point that y is the thing that you care about, that sounds obvious. But very often, people jump into a whole bunch of things and forgetting to simply characterize the theme that they care about. What is mortality? What is efficiency? What do we mean by efficiency? There are different types of efficiency, how are you going to assess them without getting into any causes? Just what is the efficiency that we care about? Once you get the y, it's no surprise, now we need x. So, x are the things that lead to y. We are now allowed to call them causes not just correlations. So, if y is bad, if mortality went up, x may be a thing that you can fix to lower mortality. If y is good, then x might be a thing you may want to celebrate or publish or sell or simply disseminate more broadly. Of course, the problem is that there are many x's for any given y and we have a number of frameworks of evaluation to help you think about what x's should you be looking for. Now, I did mention z, and z are things that sometimes can confound the association between x and y. x might be how unkempt a patient is and y could be the mortality or morbidity. It looks like how unkempt they are is causing them to die. But of course, what's really going on is the poverty that's causing both the unkemptness in the poverty. The other one where z is a common result of two things is a bit more complicated, and I couldn't go into tha right now. So, we have frameworks to help us figure out what the x's, y's and z's are, and the data collection that you want should be no more than what you need to go into that model. Very often, as you'll see, one of them is not in the system itself, sometimes you have to go outside the system, and you'll see what I mean by the end of this in a couple of minutes. So, an easy framework is called a prism framework, developed for looking at the impact of health IT in developing country environments. They focus on three things that you can see at the vertices, organization, technology and behavior, and then the inputs coming in are IT components, and what's coming out are outcomes performance and improved outcomes. Sometimes desired outcomes are called outputs but you can see what's going in. The light is being refracted and it's affected by those three things. So, the y's could be either the outcomes, the performance of the other outcomes, and the x's can be one or more of all those things that are going into the prism. I already mentioned you can measure those x's in the system itself, but you might have to measure things from outside the system, like what do you mean by the organization? what you mean by behavior? That might require a questionnaire. Not all deaths are recorded in the hospital. You might have to go to some other registry or health information exchange to find out what the outcomes with your patients. Another framework, very popular, it's called technology organization environment. Very similar to the prism. You can see it's kind of the same three types of components. Another is user acceptance of information technology. If you're caring more about that workful level and you want to see whether people are using the system, then it's interesting that you have this little thing called intention to use and something to measure. So, yes. You can measure through a questionnaire. You'll need a questionnaire to see what the reactions are to using the information technology and what you needed questionnaire to find out what their intentions are. You can get used to the system for the system log. Theory plan behavior. Yet another one. Again, focusing more at the work flow level of things, you can see, again subjectivity and intervention becomes important. The behavior is the outcome. The y and then you can see the definitions of these different components. The popular is the Technology Acceptance Model. You see that these all have to do about the people using the system, and the people use the system as critical to the system having the impact, the system's success model. The [inaudible] human organization technology sounds exactly like the prism framework. There's a lot more lines and arrows, and then this can get arbitrarily complicated. Though, obviously, you're not here to memorize this but it shows. The real use of this is if you're doing an evaluation, you may come back to a slide like this and say "Which one of these do I really care about in this environment?", and then, "how am I going to measure it in this environment?". Here's a fun one. Website satisfaction with intermediate z called socialness. I apologize for the destruction of the English language in this case. Here's one that Paulina cycled and I put together many years ago, integrating health services research and EHR-based research into one common model. Again, the system use is crucial as well as the outcomes that come out of the use of the system. So, these are all evaluations that you would do in an institution or it could be possible across institutions. The WHO put together a very nice schema for a series of evaluations for different stages of maturity of your interventions. So there's that first piloting phase, then always scaling up to national and talks about you would evaluate or why you would evaluate at each level. So, our take home is not to memorize all those diagrams but to come back to them at some point in the future when you feel you may need to evaluate or to judge an evaluation, where somebody has a comment on a protocol or a plan. Any one of these schemas can be used to help you figure out, what did I miss and what's my y and what's my x and are there any z's? If your understanding of the x's, y's and z's is the same as a proposal or the article that you're reading, you will feel very good. If they are at odds with each other, you might say maybe this person doesn't know what they're talking about, assuming of course that you do. So, evaluation is a little bit of the most academic thing that we can do in this course. At the same time, I hope I've given you some reasons to take it seriously and some tools to use it if you ever do.
