[SOUND]. Okay, so we've had Marco explain to you about rotation and translation and other types of methods for changing the display of certain types of graphics and characteristics of graphics on the screen. And we've had Matt explaining to you, explaining really how we analyze audio among other things. Now what we going to do is we're going to try and combine those two things together. We're going to do a little bit of what we call graphical synthesis to create algorithmic graphics. And we're going to do a little bit of audio analysis and combine that to make an audio visualizer. Audio visualizers are really popular and you find them all over the place remarkably so in pretty much every media player and every major console. So for example in the xbox Jeff Minter's audio visualizer is the main visualizer when you see when you put a CD in. although other visual artist like I think it's flight 404, their visualizer used for iTunes, so there is a whole range of people who just make computer graphics, abstract computer graphics. And their work is a major part of everybody's listening experience. the audo analysis aspect of doing audio-visualization is quite good to understand and is quite nice to know how to do some of those tricks. I'm going to take you through basic analysis and how we use it to change graphic parameters. And I'm also going to give you a really quick introduction to beat detection. so let's start at the beginning. Building an audio visualiser. So, the first thing we need to learn is how to do algorithmic drawings. There are so many different ways to doing this and so many different methods and different algorithms you can apply. I'm going to teach you how to do some really simple stuff that uses rotation and translation, and we're going to start with rotation. Now, fundamentally the key to almost all algorithmic drawing is a for loop. And Marco will explain that to you if you're not sure what for loops are you should go back and check the programming lectures that Marco supplied for you in case you're a little rusty. It's actually really, really easy to understand the for loop and I'm sure you'll be fine with it. But all you really need to know is this is a method for drawing lots and lots of things in different ways on the screen. Let me give you an example. Here's a basic rotation sketch which we can look at. You can see this would run fine on an iPad or an iPhone. We've got a bunch of rectangles and they're all spinning round. You'll notice that when they spin round, they seem to, line up at certain points. Like just then. Also you'll notice if you're looking, that you get four, a group of four, then a group of five, then a group of three, and then it spins out into one long arm and goes all the way around. Now the reason this works is because every individual shape I'm drawing is spinning. Based on its position in a system of a certain number of object. So if I want, If I had 20 objects and I was spinning them all, and I was spinning an object in the middle, I, I would be spinning it 20 times slower than the object on the object on the edge. And when you do that, you get this interesting symmetrical pattern. This has been used by lots of artists over the years and it's an interesting way of getting feedback type effects that are really symmetrical. Let's have a quick look at how it works. So it doesn't use any trigonometry although we are going to look at some basic trig real soon. But what we do have is a so it doesn't use any trigonometry, but what we do have is a simple rotation. And I'm going to show you how that works. This is the sketch. So ther's no sine, there's no cosine, there's no tangents or stuff, and there's no pi. We'll be looking at pi a little bit later. Young blah etc. But it has a similar effect, I'll just used a variable called time. This variable is initialized at zero at the top of the sketch. Let's have a quick look at what's going on underneath here if we go back. So we have time and time is this value where we're incrementing bit by bit. Every, every time that we ask the computer to draw a frame that increments. And then what we do is we say okay, here's a for loop. basically I want to do this 20 times times, and it uses this variable i, which is an integer, to do it 20 times. We're going to rotate given the current time so as time progresses and as time is incremented by this value of 0.01, it's going to change the rotation. And then what we're going to do is we're going to draw a rectangle based on its position, i, it's position in the system, whichever one it is. And then we're also going to change the signs based on their position. and that's what we have here and that's what gives us this spinning set of squares. So you can see here the normal sort processing stuff size and frame rate, we set the background to white. We translated everything to the middle of the screen so that zero and zero the 0x and 0y is right in the center. This is useful, and for all the sketches I'm doing today, that why, that's how I'm working. I've turned the fill of and then here's the bit we just looked at. I'm incrementing time. I've got 20 of these and then I'm just spinning them around. Each one is being spun around differently based on what the time is. And then I'm drawing it but the join functions been parameterized with the number i. The number i represents the number in the system, so remember what I said before. So if I'm spinning the one in the center around, I was spinning it 20 times slower than the one on the edge. So as I count each one out rectangle here, rectangle here, rectangle here this one spins like this and this one spins like this. So this one for every single rotation this one does this one does 20 rotations, the one on the other end. Okay, so that's the basic idea. Let's try something a bit more complicated. Let's do a for loop with rotation and a bit of trigonometry. It's a really, really simple algorithmic process. This one, I always use, I always used this example when I'm explaining about algorithmic drawing because it's a really good algorithm to start with because it's one of the earliest computer graphics algorithms we're creating. animations and it was used by John Whitney, Sr to produce a range of really famous abstract animated pieces. He also used a similar technique for generating what's called Lissajous figures for doing the title sequence to Vertigo. Which I think there's just been a, in fact the day we're recording this there's a Google doodle which gives you an indication of when we're recording this. There's a Google doodle which celebrates the work of Saul Bass and on that there's quite a lot of stuff about John Whitney because Saul Bass went with John Whitney to do the vertigo type of sequence using exactly this algorithm. And it's one of the earliest pieces of computer graphics that Whitney did on a computer that he built himself, in fact. So get this to work, we need to use some trigonometry, and we need some elements. We've got some variables and some elements. The elements is the number of objects that we want to draw and the variables are the way we want to draw them. And we change these all inside the loop. So we have a variable float spacing a variable called rotation a variable called radius and a variable called magnify. And the we have int the number of elements. So in the for loop we say. As many elements as we've got, we keep looping around until we've drawn them all. That's what it says int i equals 0, i less than element i plus plus. And then we're using pushMatrix and popMatrix, which Marco will have explained to you about already. Now what we do is instead of rotating given our time value, we rotate given spacing, i and rotation. So there are three numbers that we use to calculate what the current rotation's going to be. So we have the spacing, which is the distance between all the objects. We have i, which is the number of the object in the system. And we have rotation, which is how much rotation there currently is, how much we're rotating into our system. And then what we do is we're translating it as well at the same time. So that's the main difference. Although we're parametrizing rotate differently, we're still just rotating it. The key thing is here we're translating at the same time. Which means It doesn't just spin round, it goes in and out. And we are using sine to do this which is a trigonometric function which squeezes all the, all the shapes in and out. Now, let's have a quick look. Basically, we're drawing an ellipse every time we do this. I'm going to show you what that looks like. So this is the basic rotation with sine. Now, as always with these examples, they're accessible in week three, lesson three of the coursera documents that you should have. and here we have it. So this is this is the algorithm that I was just talking about. It's useful for all sorts of things. And, it's neat so I'll pair, I'll parametrize it with mouseX and mouseY so you can see I'm just exploring with the mouse at the moment. you'll notice that while I'm just traveling down the diagonal, we'll always have it well it always gives us circles. But if I move in the y dimension, it rotates and if I move in the x dimension, it translates. And you see we get these interesting symmetries. Let's have a, a kind of closer look at that. I'm going to go back to here. So how are we doing that? Well, we're mapping the radius and the rotation, given mouseX and mouseY, exactly as I'm showing you on this slide. So, we're going, okay. Well, we'll take the mouseX as input. And we presume that's from zero, which is the leftmost side of the screen, to the width of the screen. And then we're mapping that to values between 0 and 10. And we're doing the same for the rotation. So, because of the way we've organized these elements to be drawn, radius and rotation are in the same range. The other thing, the other variable which we introduced in the last slide which we're using to make this Whitney algorithm, is, spacing. Now spacing is how far apart, as I've said, how far apart the different elements are drawn. And this is really about how far around a circle it is. So in this case, we're getting TWO_PI. Now TWO_PI is basically how we measure around a circle. So we start here zero pi and we measure around and we get to, TWO_PI. Okay? So Marco will have explained a little bit about pi. We're using pi divided by elements to figure out how we get, say we have 1000 elements. We're dividing TWO_PI by 1000. And we're saying that's how far apart all the, all the different elements are. So it's a bit like saying, well, you know, I've got a box which is one meter long. And I want to get ten sandwiches in it. So I'm going to stick each one in at a tenth of a meter, apart. That's basically what we're doing. Although they're not sandwiches. There's no box. It's just a circle and some graphics. So radius and rotation, they're the two variables which are how far around it's going, and how far away from the center it is, and spacing is how far apart we are around the circle. So quickly let's have a look at this. So here they are. We're in the draw loop. We set the background to black. We've mapped mouseX to ray x. We've mapped mouseY to rotation. We've created spacing which is TWO_PI elements. We translated the center of the screen which is what we're doing for all these sketches. I've explained that to you before. We turn fill off and then what we do is we say for every element pick a color. I'll be talking about a color in a minute. PushMatrix just like Marco said we needed to. Now rotate based on how far apart they're supposed to be, which element and system they are and what the rotation currently is. And then translate but when we translating we're getting the sine of the spacing which is how far apart they are. The i which is the number which particular element it is and the radius which is how far from the center it is. So we get the sine of that and then we magnify it. Now this is what creates that squeezing, cause the, what sine produces is this waveform like this. And because we're running those values into a sine wave and then magnifying them out, then we're basically not just going around the circle, we're going through the circle as well. As we are squeezing the radius to and from the centerfold. And that's how it works. Then we draw our circles. Now, you'll see, if I show you the sketch, all the circles are the same size. I can do some interesting things quite fast just by times-ing them by i. Let's try again. And you'll notice that immediately I've got a very very different sketch with, which looks a lot more interesting. You see, I've got, I think they're far too big so I should probably change the size. Let's just make them, I just, I'll say i times 10 instead of 20, cause they're a bit off the screen. let's try that again. Yeah, that's a bit more like it. So you can see I've got a lot more control, and you get this weird sort of pseudo 3D effect. And it's worrying as all the different circles interact, exactly like they did in the first sketch. You get this kind of weird mesh. Now, when I'm all the way over here, it looks a bit messy. But really when both the values are quite near to zero, you get some quite nice effects, nice 3D effects. Also what we can do is we can change the number of circles. So if I change the number of elements to something a bit ridiculous, like 1,000 is a lot. Let's try 256, just double them, I wonder if it can handle that? Yeah, there you go, well that's 256 and you see that immediately. It's starting to look a lot more interesting, and all we've done is taken a few of the variables and parameterized them okay. So we're going to use this, in a bit to show you the final example. But lets move on. I want to talk a bit about hue and rotation and then I'm going to talk about how we organize, so we do color and then I'm going to talk about how we organize sound and beat detection. So instead of using red, green, and blue colors like we normally do I'm using hue saturation and brightness, HSB. Sometimes called HSV color where the V stands for Value instead of brightness. Sometimes called HSL, where it stands for Luminance. So basically, hue is the color tone. Rather than having a mixture of red, green and blue, we can say, well, I want the color tone, whether it's red, blue or yellow to be one number. And that's normally expressed between zero and 360 degrees. And it's, it's as if it was going around a circle. saturation is the color depth which is how strong the color is. And brightness is self explanatory, it's how bright it is. So bright it is if so if saturation is zero. The color is, is either you know, is a shade of gray. And if brightness is zero, it's black. And you can see by this triangle, how it works. Now so what we do here in processing, in order to make this work is we change the color mode to HSB. And then we can control what we would normally like the Fill command, or the Stroke command to set the hue saturation and brightness. And in that example I was showing you, you saw that the color was going through a range of different colors from one circle to the other. Because in my Stroke command in my fit, yeah, it was the Stroke command. In that Stroke command I was setting it based on the number of the elements in the system. So it went around the hue circle, you can see the hue circle here, which is on the top of this, on the bottom of this cone. It went round and round and round that cone, based on which number in the graphical system we were drawing. So, if we use that hue saturation and brightness, and then we do some analysis, we can create some interesting visuals. I'm going to start with a really simple visualizer, based on analyzing sound. Exactly what Matt said. We get, we analyze the audio by getting a float value called power in this example. And that's that's done by using the audio method, get average power. So, we go float power equals plat pow Float power equals player.getAveragePower and then we use the variable power to change the animation. I've got an example that I want to show you. Which is really, really simple. what it does is, it's, uses HSV, as I've already said, and it produces. The form, it produces it, you load the sound, you analyze the sound, and then we just change the color. You can see I'm changing the color here, before it draws this circle. And it looks like this. And we should get some [MUSIC] and just to prove that I'm actually analyzing it if I change the speed you see that a hue rotation changes speed too. So, I'm clicking and dragging, [SOUND] clicking and dragging to change the speed and you gotta be a bit careful with this cause if you stare at it to long it can give you a funny feeling. I guess I better move on. So, if I show you a bit of that code, it's really simple. We have another for loop with a number of elements which is exactly the same as before. I'm literally just drawing 20 circles, and those 20 circles are spaced in the screen from the outside in, just as if I was laying sugar paper circles one on top of the other. And then, what I'm doing, is I'm using the hue and I'm rotating the hue based on the power, and also the also then the number of the element that we're currently in. So, the outer circle is a different hue to the inner circle. Which is why it seems to cycle from the outside to in. That's basic color cycling. Okay. So, we're just to go through that again, we're changing the color of the elements depending on the power. We get the float value power from the getAveragePower method. We also have another float called Go which. Go is basically how much energy we have. So we keep adding the extra power and I've got a multiplier plus 50 which you can change. Have a mess with that. See how it effects how fast it moves. So Go is a value which we use to keep cycling through, through all the different color values. So, it's the, It's the additive power as we keep going. And it's kind of a number that keeps getting bigger and bigger. Now, I'm using Modulo here to make sure that whatever that value is, it's constrained to between 0 and 255. And we go, basically we have a fill command, i plus go, modulo 255. And that's the hue, power times 512, that's the saturation, and 255 is the brightness, so it's always very bright. And then we draw our ellipses, and the size of the ellipse is dependent on which circle we're actually drawing. So the circle go from the outside in. So that's the most simple kind of audio visualiser. Let's take another look at it because it's good for the eyes [MUSIC]. Really simple and really effective. Now let's say you want to do something more complicated, we can control the visualizer with sound through beat detection. Beat detection is something which is useful when you're doing visualizers. And I've got a simple example here. Let me show you the example. I'll tell you how it works. As I've said before, all of these can be found in the materials which you should have week three. So um, [MUSIC] we've got the same track here and you'll notice we've got circle in the middle, which is showing us the average power. And then there's a square appearing whenever it, it detects a beat, and the square is appearing in the top left hand corner. Let me show you how I did that. So how did I do that? Basically, what I'm doing is I'm getting the average power, I'm running a beat detector. Let me talk you through how the beat detector works. So quickly, what we do is we need to have two variables for beat detector. In addition to power, we need to have threshold, and we need to have wait. So the threshold is what the power needs to be for us to detect that there's been a beat. It's basically a level that we set and if it goes over that level, we call it a beat. And then wait, tells us to wait a certain amount of time before we go looking for more beats. Now, so we have a little method here, in each statement. Power, if the power is greater than the threshold, and also weight is less than zero so if we finished waiting and the power is over the threshold then we can do something to change the animation. So in the case that I just showed you we're just drawing a rectangle on the screen but we could also do something else. We could get this value go or our other value, we'll call it amplitude for example and we can add the power to that amplitude to create a running graph of the current amplitude. And then what we do is we add ten to the wait value. So we wait for another ten frames before we detect another beat. And then immediately we close that bracket we do wait minus minus which starts the countdown. So every drawing frame we count down and we're counting down until wait is less than zero again and then we go okay we can look for another beat. And that's basically how it works. Let me just show it working again so you can see. [MUSIC] And you can see that [MUSIC] you should actually be able to see the numbers flashing on as it detects a beat here. You've got the average here and the square is being drawn when we get a beat. [NOISE] So that's really simple. Okay so this is the last example I'm showing you, it's just a way of showing you how we put all those things together. it's not a fully In a worked example you cold do a lot. So for example, I'm, I'm not really using the color cycling. And also I've only done a little bit to change the animation. I just made it look slightly nice. But I'm drawing lines instead of drawing circles. also I'm not changing the visualizer through out. You could have a number of different drawing methods and change them based on. How many beats are being detected or any number of different ideas. So let me just show you a bit what that looks like and then you should take this away as a starting point with all of the other examples and try and build your own. So this is what it ends up looking like. [MUSIC] You can see that we've got the position of the lines changes based on [MUSIC] the beat's being protected, but they're also rotating based on the energy, so they're really closely linked to the music. Also, because the lines are all being rotated. They're being rotated twice, once around the origin and then again given a separate value, and you'll see sometimes we get nice symmetries, and sometimes we get more noisy output. [MUSIC] I'm also using a hue to rotate all of the different colors, and so that's it. So in summary, what we've done is we're created a simple visualizer, a really basic one. We could do a lot more by using alpha blending, by doing more color cycling. But essentially we are taking some algorithmic approaches to drawing. We are parameterizing them so that we can control them in various ways. We're analyzing the audio and we're taking the output of the audio analysis and plugging it into the sound, into the graphics parameters. So the sounds being plugged straight into the graphics as if we were taking a Jack plug out of a guitar and sticking it in. In fact it's more complex than that because we're doing spectral analysis and we're doing weighted average spectral analysis. So this is a really good way a good method of generating visual material and generating interesting visual applications [MUSIC]
