Now, with that general introduction to tomography, now let's turn our attention to data collection and reconstruction. Let's start by looking at this diagram again, which shows the specimen holder in the column of the microscope. So this, this is the microscope column. And these are the coils of the objective lens, and this is the sample, okay? So there's part of the coils and objective lens that are above the sample and part of it are below the sample. And here's the specimen, and the specimen is held by the specimen holder. Now, the specimen holder rests within a tube. And this tube is permanently mounted into the column of the microscope. And this tube has an outer sheath that's mounted in the microscope. And then inside, there's the goniometer, which can rotate back and forth to tilt the sample. So imagine this tube that can rotate, and it's fixed in space. Now, the specimen holder itself rests within the goniometer. It's supported by an o-ring here. This is an o-ring around the specimen holder. And then the specimen holder goes through a ball valve that allows, it access to the high-vacuum of the microscope column. So the situation is that we have the goniometer as a tube, and then we have the specimen holder that inserts within that, and it's supported by this o-ring. Now this allows the specimen holder to be moved both out and in within the goniometer. It can also be tilted up or down around, braced by that o-ring, and also left and right. All within the goniometer tube which can rotate back and forth to tilt the sample. Now, if we were to rotate that view, so we were looking straight down on the specimen, this is what I'm trying to depict in this diagram here. And let the blue circle represent the tube of the goniometer. And then imagine that the specimen holder is this blue rod, and the specimen itself is this flat plate going across the middle. Now ideally, the specimen plate, when you rotated the tube of the goniometer, the specimen and the target of interest, this is the beam coming down to the target of interest. Would stay exactly in place as it was rotated back and forth. And if we removed the specimen from the goniometer we would see that there is an axis of the goniometer that comes right through the middle. And let's put its axis of rotation right here. Now, what would happen if instead of having the sample right here along the axis of rotation, what if our sample were higher than that? So here was our target of interest. Now, imagine that we rotated this. The sample would then go through a trajectory like this around the axis of rotation. And because of that, we would see the image of the sample move back and forth across the screen. But as we lowered the sample, we could mo, we could move the sample down. And eventually, when the sample was right here and we rotated the goniometer, it wouldn't move at all. And that position then is called the eucentric height. And the way you can set your specimen at eucentric height is to wobble the specimen tilt, and move the specimen height up and down until the movement of the image is minimized on the screen. And that's because your sample is right at that eucentric height where it doesn't move when you tilt it. Now, what I just described was an ideal situation. And unfortunately, nothing mechanical is actually ideal or perfect, at least not down to the 10 to 50 nanometer level. And so in fact, the goniometer tube is nowhere exactly coincident with the optical axis of the objective lens. And so imagine that the tube is actually a little off centered. And that's what I've tried to draw here. Here, I've moved the goniometer tube in a terribly gross exaggeration. I've moved it way far away from the optical axis of the objective lens. Now in this case, after aligning the microscope, the beam will be passing straight down here, which is the optical axis of the objective lens. And because of that, if we were looking at a sample, we would have the sample moved over here so that it was under the beam. Obviously, if you're looking at a target, it has to be underneath the electron beam. So it'll be in this position over here. And that's a problem because now when you rotate the sample, it's going to go through a trajectory like this. Around the axis of the goniometer. And your sample, as we rotate it, will end up moving, say, in that way and in a negative, in negative tilt angles down this way. And so now, our target is moving underneath the beam while we tilt it. Now, this distance from here to there, we can call the tilt axis offset, and it can be calibrated beforehand. You can measure the tilt axis offset of your microscope and know that before you start. Now unfortunately, the real situation is even worse. Here, let me draw again the electron beam going down the optical axis of the objective lens. And in fact, in addition to our tilt axis offset, it's also always the case that we never can actually get our grid at exactly eucentric height. And so there's always some non-eucentricity. And so the distance between here and there is what we would call the non-eucentricity. And that's always finite as well. And as a result of non-eucentricity and the tilt axis offset, our sample when we rotate it, instead of being stationary, it's always going to follow some trajectory like this. And as you can imagine, as we rotate it, it will go through a series of positions in the microscope. Because of this, in order to keep the target in the beam during the tilt series, we have to apply, first of all, beam shifts. Now remember that above the objective lens, we have the objective deflectors. And the objective deflectors give us the ability to deflect the beam somewhat so that we can hit the sample even as it's moving within the microscope. And after we take its picture, then we use, lower in the column we have another set of deflectors, which are called the image deflectors. These were the beam deflectors. And we can use the image deflectors then to undo the shift that we induced with the beam deflectors. So that the image goes straight down the optical axis of the later lenses and hits the center of our detector. And as our sample moves further and further during the tilt series, we can deflect this further and further. So that we image that sample, and then we bring it back always to the optical axis of the lower column. In order to keep the image on the detector. So as the sample moves back and forth we apply beam shifts and compensatory image shifts to keep our target within the image. Now in addition to shifting left and right, we also see that our image is rising in the microscope. As it moves up here, it rises. And so in order to maintain a constant focus we have to change the current in the objective lens, which is focus. So that even though the sample is moving up and down in the column, each of our pictures of the tilt series will have the same focus value. So to record a tilt series, in between each image, there's a number of deflector and lens adjustments that have to be made. And there's a couple different strategies to go about this. The first strategy we'll talk about is the so called predictive method. In the predictive method, one first centers the object and you do this with a low dose, low magnification image. So that the dose that's applied to center the object underneath the beam is a tiny fraction of the dose that you'll apply to record the tilt series. That way this tiny fraction won't damage the sample much while you get it centered underneath the beam. Once it's centered you go ahead and record the untilted image, with a zero degree tilt, at whatever desired magnification you want for the tilt series, and we'll talk about that later. Then you tilt it the first tilt increment, like for instance one degree. And you record the first tilted image. Now these images are recorded digitally and so a computer attached to the microscope can quickly calculate through cross correlation how much did the target of interest shift between the untilted image and the first tilted image. And so you can get a measurement for the shift that took place. Then you tilt again. Say, to two degrees, and you record the next tilted image. And once again determine the shifts between the first and the second. And once you have a few images recorded, you can then fit these shifts to a model of the tilt axis offset and the specimen height. In other words, the non-uccentricity there. And what this means is that you simply acknowledge that your sample isn't going to be exactly along the tilt axis. And so as you begin to move it, tilt it in one direction. You expect to see the sample moving slightly. And you begin with an estimate of the tilt axis offset. And you can now fit the shifts that you see from here to here to here, you fit it to a model of how you'd expect it to move given the tilt axis offset. And then you begin to find the specimen height or the non-uccentricity by its movement in the first few images. Then you predict where you think it's going to move for the next image. After predicting where you think it's going to move, you then apply the appropriate beam shifts, image shifts, and the focus changes that would be needed. And then you tilt the sample and record the next tilted image and determine the shifts. Now you can refine your model of what the tilt axis offset was and the specimen height was because now you have more data points, and using that refine model you again predict where it's going to move for the next image. You apply the appropriate beam shifts, image shifts, and focus changes as necessary and you move forward, tilt it, and record the next image. And you loop through this all the way from, say, an untilted image all the way to a highly tilted image, say, 65 degrees. Then you typically go back to zero degrees, recenter the object to make sure that you still have it in the field of view. And then you begin to tilt in the negative direction. And perhaps go to minus 65 degrees. Obviously there's variations of where you start and end. But the idea is you start at low tilt angles so that the movements will be small as you gather the information you need to be able to predict the movements later. When you're highly tilted at 55 or 60 degrees tilt, then these errors cause much greater movements, and so by then your model is well refined and you have good predictions. And we call the process of keeping your object of interest in the field of view and imaged on the detector throughout the tilt series while it's moving. That process we call tracking. This predictive method can be very effective, especially for high quality goniometers. But there's a lot of goniometers that don't actually behave in a very predictable pattern. So for instance you might start with a specimen here and the predictive model is that as you tilt it it would rotate in a trajectory like this. But maybe in certain positions there's something that slips within the goniometer and the stage actually moves. Or jiggles forward or backward, or maybe it doesn't actually follow a nice, smooth circular pattern. Maybe it has a much rougher trajectory that's difficult to predict. In that case a more robust procedure is what's called the focus position method. And here, to explain this I'm showing an EM grid from the top. And there's the holes in the carbon film in irregular pattern. And lets imagine, for instance, that we have a cell that we're trying to image. And here it is, right there. And, now, let's assume that the tilt axis of the goniometer. Lies the projection of the tilt axis lies there on the sample. Now we can choose a couple positions that say here, and here where the line between these focus positions and the sample, this line is parallel to the tilt axis. And because of that, as the grid tilts back and forth, these positions should always stay at the same height as the specimen itself. And if the grid moves up or back, these should be moving to the left or to the right, and up and down in the same way that the sample. In the same way that the target is moving left and right and up and down. And now in this situation the procedure goes as follows. First of all you center the object with a low dose, low mag image. So you're looking here on the object and you center on that object. Then you shift the beam to a focus position. And you focus at that position and you record a reference image. Now one of the principles here is that we have to minimize the dose applied to the target itself because the principal resolution limitation is radiation damage. And focusing requires lots of dose to be applied. And, also tracking the sample to see how it moves, requires dose. And, so we want to minimize the dose applied to the sample, and move and use all of that dose in some remote focus position that doesn't matter if it gets damaged. So the idea is you beam shift off of your target over to some focus position, say over here. And here you focus the microscope and record a reference image of what this region of the grid looks like. You might then move to another focus position over here and measure the focus here and record a reference image of what that part of the grid looks like. Then at this point, you can blank the beam here. You blank the beam and then you unshift it, so now it's back on top of your target. Then, you unblank it for the exposure time to record a clean image of your sample and then you immediately reblank it. Then you tilt the grid, and move again with beam shifts back to one of these focus positions. And at this focus position you refocus, because now that you've tilted, the grid might have moved up or down or to the left or right a little bit. And you can detect that by refocusing here on the focal position and use as much dose as you want. Furthermore, if you see that it actually moved during the tilt, now you have a prediction of how that part of the grid moved, and you might move to the other tilt position. Let me change color so that it's easier to see, and you might focus over here again, and you might notice that the grid moved a little bit when you tilted it. And now because you know how the grid has moved, you can apply the appropriate beam shifts, and image shifts, and focal changes. Blank the beam again, and then move back to the target region. Record an image, and hopefully after that procedure your target will be directly under the beam again. Then you tilt again, and you move to the focus position. And, so after moving to the focus position, you, you refocus. And maybe you'll see that the grid seems to have moved over here. And then you might go to the other focus position and see that your grid has moved a little bit. And so by tracking the movement at the focal position, you prevent overdosing your sample, but you can still track its movement. So every time you take another image in the tilt series, your object of interest is centered under the beam and the image is centered on the detector. And so in this case, the iterative cycle goes from here back to there, and you iterate that as many times as you have images in the tilt series. Now most projects require that we record lots of tomograms of lots of different objects. Maybe they're different cells undergoing the same process, or they're different examples of a macromolecular complex. In any case, we're usually taking lots of tomograms. In order to facilitate that, all the processes of tracking that I just described are done automatically using specialized software. And the software packages also allow you to record a tilt series of one target after another after another after another, all automatically. To facilitate that, you usually start by recording a very low magnification image of a large region of your grid. Now we call this an atlas, and here's an example atlas that we recorded of a grid. And if you look carefully you can see that it's composed of a number of squares. This is because the software used a low magnification and took images of each part of the grid, one by one, as it was shifting the grid around in the microscope. And then the software put all of these individual images together, stitched them together to form this very large atlas of a large region of the grid. And based on an atlas like this, you can identify certain regions of the grid like say here, which look like they had the right thickness of ice and they look like they would have some good targets. And having identified a region of interest, the microscope can then move to a slightly higher magnification and record a higher magnification atlas of just that region. And at this point, you can make better decisions about which grid squares, say, this one and maybe this one, are the right ones to do further imaging. Again, because the ice thickness looks right, because there doesn't look to be a lot of contamination blocking a lot of the grid square. This grid square here, the carbon has been ripped so there's no there's no substrate here. So you choose the grid squares that look the best. And finally, you take an even higher magnification atlas of say, a particular grid square here that looked the best. And at this level, you can begin to see individual cells. This grid had bacterial cells frozen on it, so there was one cell. Here's another cell. Here's another cell. You can see the individual holes in the carbon film. So at this level, you can actually see individual targets. But if you notice, you can still see these squares superimposed on the image, where the microscope took a whole series of higher magnification images, and then stitched them together. So this is called an atlas series. And once you've recorded an intermediate magnification atlas you can then look at the potential targets and select the ones that you would like to record tilt series of. So, for instance, this is an example intermediate magnification atlas, again of some bacterial cells. And as you can see, several fields of view have been chosen for imaging. This one here this one. [COUGH] This had the number two associated with it. Here's field number three that was chosen. Here's field number four that was chosen. And among these potential targets, there was also a focus position chosen. And an example, automatic sequential tilt-series acquisition protocol would proceed like this. First you would mark the targets and any focused positions that you'd like to use on a low mag atlas image like this one to the left. Then you would stage shift, and it could be potentially pretty far away on the grid. You would stage shift to the first target in the list. Here you'd record a low dose, low mag image to be used to make sure that you're centered on that first target of interest. You'd determine the shifts and you'd re-shift the stage as necessary until the target that you had previously chosen, like for instance this cell, really is in the center of your field of view. Then you could stage or beam shift to a nearby focus position, for instance, over here. And here you would perform an auto-focusing procedure and an auto-eucentric height. And you might even center the slit on your energy filter at this focus position where you can apply as much dose as you need. Then you un-shift [COUGH] the stage back to the target of interest and record a tilt series. After the tilt series is recorded you back, go back to stage shift to the next target on the list. And you repeat for as many targets as you want, and we call the process of going from one target to the next targeting. And using software to automatically target and track objects of interest during a tilt series, a typical recording session then involves first recording an atlas series to find the right region of the grid. Then, once you've found a favorable region of the grid, then choosing your targets and the other data collection parameters. And then you can simply hit go, and leave. And the microscope will go from one target to the next, adjusting parameters as necessary. And recording the tilt series of each target as it moves along. Now, in such an automatic data collection protocol, several functions have to be done automatically. The first one we'll talk about is auto-eucentric height. So let's imagine this is the electron beam, and the tilt axis of the goniometer is, let's say, right there. That's the eucentric height within the microscope. Now, to automatically set eucentric height, auto-eucentric height. One way to do this, imagine that your sample is not at the eucentric height, would be to simply wobble the specimen tilt so that it rolls around that tilt axis and simply record images. And what you'll see is that the sample will begin in the middle, and then it'll move to the left, and it'll move to the right as you're wobbling it back and forth. And the microscope can adjust specimen height in order to minimize this movement. And that will pull the sample down to the eucentric height. That's one way to set eucentric height automatically. Now, the other imaging parameter that we'd like to be able to set automatically is focus. So how can we automatically focus an object of interest? And in order to explain that, let me call our attention back to this diagram where imagine we have a sample, and there's an incoming beam. And this beam is scattered by the sample. Here's the objective lens which focuses that scattering. And scattering that goes straight on through, that will be collected on to a particular focus spot here. And then it moves forward through that spot down on to the image plane. And scattering in this direction, say, that is focused on to a spot here, and it also moves on through and re-interferes with the unscattered beam. We've been through this before, scattering it to this direction, is focused to a spot here. And that moves to re-interfere. Okay, and so this is the picture of how scattering is refocused by the objective lens to form a magnified image here on the image plane. But now, let's think about what would happen if instead of the beam coming down, strait down on to this sample, what if it were coming from a tilted direction? Now, this of course is grossly exaggerated, but what if it was coming down in this tilted direction? Because biological objects are weak scatterers, the maximum of the wave front as it comes through the sample, would be in the same direction as the unscattered beam. And that unscattered beam in this direction, because of the lens, it would be focused into a position, let's say, over here on the back focal plane. And from there, it would proceed to once again fill this region of the image. And in addition, there would be scattered beams that were scattered to a higher angle here. And those scattered beams would be focused to a different spot here on the back focal plane. And they would also carry on through [NOISE] to fill the image plane. And finally, scattering on to, in this, to this side, might be focused to a position on the back focal plane, say, here. And then carry on through to fill the image plane. Now, the key thing to notice is that while the images that are formed in these two situations are in the same location, at the plane where there's a focused image. In other words, if the beam is coming straight on through, you have an image formed from here to there on your detector. And likewise, even if there's a beam tilt applied, the focused image still appears from here to here on the detector. That's true for the image plane, but it's not true for any other plane in this system. For instance, if the detector happened to be conjugate to some plane, let's say, here. Let's say that is the plane that's being imaged. Then what we would observe is that the scattering from the untilted beam would form a blurry image from here to here. This is where the major scattering is in that situation. But from the tilted beam, now they made, the bulk of the scattering is from here to here. And so we would have an out of focus image that was from here over to here. And so as we wobble the beam tilt coming into the sample, we would see these out of focus images moving from left to right, back and forth. Because of this, we can automatically find focus by wobbling the beam tilt and then adjusting focus. In other words, adjusting the strength of this objective lens, more or less, until our detector becomes conjugate with this true image plane, this ideal image plane. Where even through you're wobbling the beam tilt, the image itself doesn't shift either to the left or to the right. In a focused image, you can beam tilt slightly and the image won't actually move to the left or the right, it'll stay where it is. But if your detector is conjugate to any other plane above or below true focus, then when you beam tilt, you'll see the out of focus image shifting back and forth to the left and the right. So you just change focus until that shift is minimized. Now, it turns out that you can actually do an auto-focus and an auto-eucentric height simultaneously. If you know what the current is that is required to ne, take an object at eucentric height and bring it to perfect focus on the plane that is conjugate to your detector, okay. Your detector is fixed in space. It is bolted into the microscope where it is. And there is a fixed eucentric height where the, the tilt axis of the goniometer goes through. And so the eucentric height is a fixed position in the microscope, and so is the detector, it's also fixed. And so it's possible to determine beforehand what is current in the objective lens that's required to cause an object at eucentric height to be in focus on your detector. And let's say, you know, that's a known current i. Then when you insert a new sample into the microscope, you can set the current of the objective lens to this known current i. And then you can start wobbling the beam tilt and simply adjust the specimen height until the movement of the image is minimized on the detector. That will be minimized when you've moved the specimen to view eucentric height. Because your objective lens is set with the current that's required to take an object at eucentric height and produce a focused image on the detector. If your sample is not at eucentric height, then the image will not be in focus. And if you beam tilt, you'll see the image rocking back and forth. And so that's a way to simultaneously set auto focus and auto eucentric height. Now that we've talked about how tilt series are collected, let's talk about how they're reconstructed. Now here again is the example tilt series that I showed earlier of a videlo vibrio cell. And you can see that in this tilt series, the cell itself is always present in the middle of the images. But it jumps left and right a little bit as the images are recorded. And that's the error in the tracking procedure that was used. This was recorded using the predictive method. And as it predicted the x and y shifts that would be occurring for each new image, there are small errors in that. And so that's why you see it jumping around. For this reason, we add these gold fiducial beads into the sample. And they're frozen in the ice, and so we can track the shifts using those fiducial beads. And so the first task in calculating a 3D reconstruction is to align this tilt series. And for each image, we first determine the X and Y shifts that apply. Also, the rotation or the position of the tilt axis, because remember EM lenses cause images to actually rotate as they come down the column slightly. And the rotation is dependent on exactly how high the sample is in the objective lens. And so because of that, even though we are shifting focus along the way to keep our sample in focus as it moves within the microscope. Nevertheless there are subtle shifts in the rotation of the image because the sample was at slightly different heights within the objective lens. And so the images that appear are slightly rotated, one with respect to another. So we have to determine the position of the tilt axis in each of the images. Next we have to determine the tilt angle. Of course, as we're recording the tilt series, we tell the goniometer to tilt to say one degree or maybe two degrees or three degrees, and it tries to do that. But it's never perfect and so the actual angle that the sample might have been tilted is not exactly one degree, it might have been 1.1 degrees, or maybe 0.9 degrees. And you can detect that from the positions of the gold beads in the images. And so for each image, we refine the tilt angle. Next, again, because the sample is actually moving slightly within the objective lens, and it's moving to slightly higher or lower positions as the tilt series is recording, the magnification of each of those images will vary subtly, maybe 1%, 2%, 3%. And so before we can merge all of these tilted images into a reconstruction, we have to determine the magnification to higher precision. Finally, as we'll talk more a little bit later, if we want to correct for the contrast transfer function in our images, then we need to determine the defocus of each image precisely. And again, because the sample is moving actually up and down within the objective lens during the tilt series, unless we adjust the focus just right, each image can have a subtly different focal value. So what the software does is it tries to identify the locations of individual gold beads throughout the tilt series. And it might try to identify this one and say this one and this one and this one, maybe 20 or even 40 different individual gold beads. And remember that this is a cell frozen in a 3D block of ice and the gold beads are scattered throughout this object. And so some of them are high in the ice and some are low in the ice. Some are right next to the cell. Some are far away from the cell. And it's easy enough to get an initial estimation for the x and the y coordinate of each gold bead here, horizontal and vertical, in the image. But what's unknown at the beginning is where that gold bead is in the ice. It could be either higher or it could be lower. So the software begins with the knowns are the X, Y position of each gold bead as it moves through the tilt series. And it has an estimate for what the tilt angle of each of those images is. The software solves a set of equations, where the knowns are the X, Y coordinates of the beads in each of the tilted images. And the unknowns are the height of each bead within the ice as well as all of these parameters about each image. For instance, the X, Y shift of each image and the rotation about which tilt axis, a refined value of the tilt angle, and a magnification. Now the gold beads are not used to determine defocus, it's just these parameters. And a least squares fit is performed to best match all these parameters. Then a new aligned tilt series can be produced, which is corrected for the various X, Y shifts between the images, any rotations that are going on, and any magnification changes. Then also using the refined values of the tilt angle, a reconstruction is produced. Now, this is the method that uses fiducial markers to determine these various parameters. Now, it's not always possible to add gold beads to a sample. For instance, it's difficult to add gold beads to the surface of a FIB milled lamellae, for instance, and so in this case the alignment can be done fiducial-less. In this case, you simple use the features within the images themselves to determine the shifts between images, the possible rotations and magnification changes that could be happening, and to refine the value of the tilt angle. And this can also be very effective. Now once the images are aligned, the three dimensional reconstruction can either be done using a weighted back projection which is done in reciprocal space as I explained previously, or there are, are also real space algorithms. For instance, an iterative real space reconstruction that can also be used. Then this whole process of image alignment and then reconstruction can also be done automatically, so that once the user inserts the grid into the microscope and chooses the targets of interest on the low mag atlas, then the microscope can record the images and the computers can cal, can align the images and calculate the reconstruction all automatically. And many labs will have in place some kind of a pipeline where the results can be stored in a database. For instance in our lab we have a tomography database, where all of the tilt series and the reconstructions that are calculated are stored, as well as a thumbnail image showing a central slice of that reconstruction. And it's organized like a YouTube page, where you can see a central slice, and if you click on this one, you see a movie of that reconstruction. And, it can be organized by the species or by the investigator that collected that data or whichever one is most popular. And so you're likely to work with some kind of a database system that stores the results.
