The primary goal of user testing is to see where users have trouble with the system being tested and define the problems that caused the trouble. In this lecture, I'll talk about how to observe user tests to get the most out of them. The first thing you're going to want to do is record your test sessions. You need to do this because it's very challenging to pay close attention during the test especially when you're moderating the tests, you're interacting with the participant, and you're just trying to do too many things at once. It's basically impossible to take the detailed notes that you'll need in order to do your analysis later. You also want to record your test sessions because you're going to need that evidence that you get from the test for your findings. The evidence can be in the form of videos, sometimes it can be user quotes, sometimes it can be screenshots, and you're going to need the recordings in order to produce that evidence when you're trying to report your results later. The most typical kind of recording that you're going to produce from a user test session would be a recording that would look something like this, a screen capture. Essentially, a video of the screen that the user interacts with as they go through the test, and then often a small video of the user's face as they're interacting with the site. So, you can see their expressions, see what they're thinking about, and doing at the time, and then you'll also have audio of the user thinking aloud as they go through your tasks. In order to capture these recordings, you're going to have to install some software on the laptop or computer that you use for conducting your user tests. There are a whole bunch of options out there. So, rather than me telling you what the best option is, your best bet is actually to just search for terms like screen recorder, screen casting, screen capture, and find something that works for your platform, and suits your price preferences. There are a bunch of free options. There's a bunch of options that have 30-day free trials and things like that. There is another class of software that is sometimes used for this kind of thing and this is software that's designed specifically for usability testing. Probably, the best known of this type of software is Morae by TechSmith. When you use specialized software, you get some additional features. It can automatically capture and calculate the time people take for particular tasks, capture all the keyboard entry, the mouse clicks that users enter during the test. It also usually will have facilities for remote observation, so for other people to actually watch the test in real time from a remote location and things like that. Generally speaking, you don't need all of these features although they can be nice to have in certain situations. This approach to capturing user test sessions works great if you're using a laptop or a desktop. If you're using another kind of device like a smartphone, for example, the options are a little bit more limited. We're starting to see screen capture software for mobile devices, however, it's generally not very mature and you don't know what you're getting. People often use other approaches for capturing mobile devices. This is an example, an overhead camera. The downside of this is that, of course, it requires the user to keep their hands in the screen during the user task that can be a little bit unnatural, but unfortunately, it's pretty much the best approach that we have for mobile testing at the moment. Capturing the video and audio of the test is going to give you access to everything that happened during the test, allow you to go back, and analyze it later, and of course, the reason we do this is that it's just too difficult to pay attention to everything that's happening during a test. However, usually you don't want to have to go back and review all of the video if you can possibly avoid it because it can be very time consuming to go back and watch from start to finish all of your user test sessions, take notes, do the logging of what happened, and so on, and so forth. In fact, a general rule of thumb is that for every hour of video that you have, it will take two to four hours of time to analyze that video. You can save yourself a great deal of work if you can take notes during the test to index into the recording. Now, I said before, it's difficult to take notes and I'll talk a little bit more about how you can structure the test in order to do that. But one thing that you want to do is you want to make sure that your logging or your note-taking is as streamlined as possible. So, here's an example of a logging sheet that you might use in a usability test to structure that notes that are taken. So, in this example, this is a logging sheet for a particular task in a user test. It indicates the success criteria, so that when the task is completed, the logger can just check off whether it's successful or not. Then, you can see that the time of each event is indicated in one column, the screen that the user was on, the type, in this case, a code where CU stands for a comment by the user, E stands for error, and C! stands for a critical incident, indicates what type of thing this was, and then just some brief notes about what happened. What this does is it basically gives you a set of timestamps into the video that you can use to go back and just re-watch parts of the video where interesting stuff happens. So that way, you don't have to watch through all of the places where the person was successfully going through the task or where there was a delay between tasks or all of the dead space and you can just focus on the places where interesting things happen that you're going to need to report on when you're generating your usability test report. Wherever possible, it's best or at least most convenient to do user testing with a research team that will help to conduct the test. When you have a team, you want to assign people into different roles. So, you typically will have one person who's the moderator, who's interacting with the participant, moving them through the tasks, conducting interviews, administering questionnaires, and things like that. Then, you'll typically have a different person who's responsible for observing and logging. That would be the person who's sitting with the logging sheet, making the notes, noting down the time code, and things like that. You might additionally have people that are responsible just for managing the recording, making sure that the recording starts and stops, is archived onto a backup disk, and things like that. Perhaps, also offering tech support in case the system that's being tested needs certain types of maintenance or the system unstuck if it gets into stock position or something like that. In some cases, there will be additional people who are interested. These might be designers, these might be developers, product managers, and things like that, who would benefit from observing the test. So, in those cases, you would have an additional role for people that are just passive observers, that are just watching what goes on, but don't necessarily have any role in actually conducting or analyzing the test. But the reality is there's going to be a lot of situations where you essentially have to fly solo, where you're the only person that's going to be available for conducting a user test. In that case, you're just going to have to bite the bullet, and plan to review the videos in depth after the fact, and you're going to also want to make sure that things like recording and tech support actions that you need to do, the tech support actions might be things like clearing out the database in between runs, resetting the user accounts, and things like that. You're going to want to make sure that you include those in a detailed checklist, so that you make sure you don't miss anything. You will find that there's just a lot to keep track of when you're conducting a user test and checklists can really be your friends to make sure that you don't miss anything. It would be terrible to run a test session and realize at the end that you hadn't started the recording or you hadn't saved the recording or whatever it is and essentially, you've lost that data, you no longer have it in order to conduct your analysis later. One question you might have is, where should I conduct my tests? Well, the most important things are that you conduct the test in an environment that is private, it's quiet, and distraction-free. You want to make sure that your participants feel comfortable, that they don't feel like their actions are being exposed to other people outside of the testing team, and you also want to make sure that they're not distracted by other things that are going on in the environment. There are some testing situations that might require different types of spaces. For example, if you have multiple observers, if you have advanced recording needs, for example, if you're trying to capture mobile device interaction or if you have special concerns over the amount or degree of distraction that might be available. If you want to really, really reduce the number of possible distractions, you might need to pick an environment that does that for you. In those situations where you have more elaborate concerns around your testing, you will typically need to be testing in a formal usability testing lab such as the one that's pictured here. Sorry, the picture is a little bit dark, but what you can see is that there are three people in a control room that are sitting behind a one-way mirror, and observing a moderator, and a test participant in the testing room that's right next door. This setup basically allows a larger testing team to be participating. It also allows for more elaborate recording setup and it can reduce the number of distractions that a participant has while conducting a test, especially when you have more people involved in the testing team. As shown in this picture, which is the same room from a different angle, you can have a fairly large number of observers and members of the testing team who are involved. It would be infeasible to have this large testing team in the same room as a test participant because it would just freak them out really to have a whole bunch of people sitting and watching over their shoulders while they're trying to do these tasks. But for most of the tests that you'll conduct, you don't need an elaborate set up like that. You can use basically any meeting room or any kind of private space that you can get access to including people's private offices, your own office, your own home, if that's appropriate, and you just will basically be sitting alongside participant administering tasks and watching what they do when they perform the test. Why would you need to use a formal user testing lab? Well, basically, if you want to run tests where you have multiple observers, where you have these special recording needs like mobile or other kinds of devices, or you want to use special equipment like eye trackers, or where you have special concerns about distraction, especially when you're going to be having larger testing teams. It's also worth noting that these labs are quite expensive to set up and maintain. So, in small organizations or where you're doing testing outside of an organization, you're probably not going to have access to this type of resource and you'll be just fine without it. Most of the time, you don't need a fancy room or fancy equipment to do a user test. If you can find a private distraction-free space and use simple recording software, in most cases, you'll be just fine. It's great if you can have another person help with observing and logging, but if you can't, it just means you might spend more time reviewing the recordings later. Careful thorough observations are key to a successful user test. If you make sure to capture good recordings and take the time to review them, you can be sure to get the information you need from your user testing efforts.