Daniel Bienstock: Okay?
Daniel Bienstock: Alright,
Daniel Bienstock: Before we get started there's a chance
Daniel Bienstock: that I'm going to get
Daniel Bienstock: called by the electricity company. They have to come and do some service, and I'm just. I'm waiting for them to call back.
Daniel Bienstock: Okay, let me open this up alright. So now let's review where we were last time. We have a bit of ground to cover today.
Daniel Bienstock: we're going to need at least one more lecture in these Ml tools, if not 2 more lectures
Daniel Bienstock: and a quick review. Last lecture.
Daniel Bienstock: we saw deep learning. I encourage you to. Look at the little Pdf.
Daniel Bienstock: That I had uploaded. This is a deep learning using gradient descent
Daniel Bienstock: to me, minimize mean square error
Daniel Bienstock: use training data
Daniel Bienstock: that the data consists in the example that we considered a vector each. Each element of the data consists of a vector and a value.
Daniel Bienstock: And now we are trying to use the vectors to predict the values, and the network will predict the number. And the error is squared. And now, over all the data samples, we take the average of these square errors, the mean square error
Daniel Bienstock: and the the variables and the optimization problem are the network weights.
Daniel Bienstock: And we were using, or in the community gradient descent. First order methods are used to try to minimize this function
Daniel Bienstock: and thus build a network or put voids on the network, and then that gets used to make predictions.
Daniel Bienstock: Then we described how to use a variation, a variation on this
Daniel Bienstock: to do classification.
Daniel Bienstock: and then classification. What you have is you have a network
Daniel Bienstock: and the the last layer, the last layer of the network.
Daniel Bienstock: outputs
Daniel Bienstock: one number per each class that we trying to use to classify.
Daniel Bienstock: So one number of those one number per class. So if you're trying to use images as inputs, and then, describe what is in the image. It could be different types of animals. And so let's say, there are 5 choices. So the last layer of the network will have 5 neurons outputting numbers.
Daniel Bienstock: And then, for example, you can take the maximum of those numbers, and that is the one that is the that is the prediction. And the the network is trained the same way. More generally you take these 5 numbers, and out of them you fashion a probability.
Daniel Bienstock: discrete probability distribution.
Daniel Bienstock: for example, using this what I call the soft Max, this exponential, divided by the sum of the exponentials. So you take the exponential for each of the numbers you divide by the sum of these 5 exponentials, and that gives you a probability distribution. And now you use that probability distribution to make a stochastic prediction.
Daniel Bienstock: And again, you can, you can, use this metric
Daniel Bienstock: as compared to the truth.
Daniel Bienstock: to set up an optimization problem to train the network.
Daniel Bienstock: Okay? Now, let's move on. Past all of this we were discussing the application which is the application of interest here, which is how to play a game like chess, or in particular play go, which is considered more, more difficult.
Daniel Bienstock: then then tests.
Daniel Bienstock: And so the first topic that we started to look at in the last lecture is what we call supervised learning of policy network. So this is a a very loaded terminology.
Daniel Bienstock: let's see what it what it means, and then we'll see how it goes was used
Daniel Bienstock: by the Alphago people. By the deepmind people to create the first element in this setup that they call Alphago. That proved very successful. Not the most successful we get. We'll get to that in a future lecture, maybe next lecture or the one after that.
Daniel Bienstock: But this is supervised learning
Daniel Bienstock: of policy networks.
Daniel Bienstock: Okay, the learning is what we have been discussing before. This is the AI term networks is because we have a network supervised here is because we have input data which is considered to be a kind of ground truth.
Daniel Bienstock: In the case of the goal playing algorithms, the supervision
Daniel Bienstock: was provided by data coming from games played by masters of the game.
Daniel Bienstock: And policy is yet another loaded term. Okay, will be exposed to this term multiple times today. And we'll come to understand what that means. This is a term that predates to some extent this AI community.
Daniel Bienstock: And so what is the setup? What is the setup? You know we have pairs. This is the data that we have.
Daniel Bienstock: We have pairs
Daniel Bienstock: of the form action state. And so here, state is the state of a system.
Daniel Bienstock: and the system, for example, could be the board for the game.
Daniel Bienstock: My understanding is for for the goal playing algorithms the State was actually a picture
Daniel Bienstock: of the board with the pieces in place. A picture, okay? And a is action
Daniel Bienstock: taken, let's say, by an expert.
Daniel Bienstock: the correct action to take when the system is in this given state?
Daniel Bienstock: And then how does how? What is a supervised learning of a policy network. So here's our network.
Daniel Bienstock: Okay? And we have all these layers and so on multiple layers. We have. These are parameters. let's not call it Theta, actually, because they they chose not to call theta theta. Let's call it that Greek letter. So we have the state
Daniel Bienstock: the state coming in.
Daniel Bienstock: And it's going to be mapped as a vector into the input layer. And out of the output layer comes, not an action con comes up. Probability distribution.
Daniel Bienstock: Okay? The probability that we should take action. A given state. S,
Daniel Bienstock: and we use the the data that we have the pairs that we have to provide the training data.
Daniel Bienstock: Okay? And so we run this classification network.
Daniel Bienstock: it makes a prediction
Daniel Bienstock: for the correct action to take.
Daniel Bienstock: and then we compare it to the recommended action recommended by the expert, and thus we obtain a metric.
Daniel Bienstock: Okay? However, the way that this would got was got used was by doing gradient ascent gradient ascent. And what do I mean by that gradient ascent to maximize maximize. What maximize the probability that we make the correct action given any any of the States?
Daniel Bienstock: Okay, so let me write the formula for what would amount to the gradient ascent, and we'll explain what we did. I uploaded the first paper by the Alphago people. Oop.
Daniel Bienstock: There are more people now by the Alphago people. There's a section toward the end of the paper that's called results or methods, pardon methods, and in it they describe in somewhat more detail what exactly it is that they did. Okay, so let me. And it's best to look at what they did in terms of the formulas which we're going to do to understand exactly what it is that we are doing.
Daniel Bienstock: Okay, this is gradient ascent
Daniel Bienstock: rather than decent.
Daniel Bienstock: Okay, gradient ascent.
Daniel Bienstock: And let me write the formula.
Daniel Bienstock: And I'll explain what it is that that we are doing. So each iteration
Daniel Bienstock: is as follows, so delta of the segment, these are the network weights. Okay, these are the numbers that go in the network. Okay, which is our optimization problem is what? Okay? So it's a number. Okay, Alpha, Alpha is what we will call the learning rate. It's the step size, basically for a gradient method.
Daniel Bienstock: But I have this. M. Now, I have a son.
Daniel Bienstock: Okay of what? And I take the gradient the notation I'm using is more standard that what they have in their paper. But you'll understand what I mean. Gradient, with respect to the network weight is our our variables. This is the gradient part of the gradient descent method. Gradient of what function, and I'm going to write the function here. The log, the log of the probability
Daniel Bienstock: of a K given SK.
Daniel Bienstock: So what is all of this. So ak comma should should do it the other way around. Aksk.
Daniel Bienstock: SKAK. This is the training data.
Daniel Bienstock: These are, you know, this is a state.
Daniel Bienstock: and this is the action
Daniel Bienstock: by a master of it. Again.
Daniel Bienstock: okay. And that K is what K. Is roughly 30 million.
Daniel Bienstock: Well, roughly, actually, 1 million
Daniel Bienstock: or 1.5 something like that.
Daniel Bienstock: We write it the standard way. So we have.
matias: Is that M. Then.
Daniel Bienstock: No, that's not them. Oh, no, no good point
Daniel Bienstock: one so good point. Okay. What is m is M is a mini batch. Okay? So let me write all the data this is. If you look at the paper, you'll see how highly dimensional they paper is too many training wheels. This is training data training data
Daniel Bienstock: is about that large training data is about that large M
Daniel Bienstock: is Mini batch
Daniel Bienstock: batch, which was 16.
Daniel Bienstock: So this is a case of stochastic gradient. Okay, we take a mini batch of size or 2 people enter the room that'd be in that'd be allowed.
Daniel Bienstock: But both of them
Daniel Bienstock: okay, my network is kind of slow. Admit all they are getting admitted. Okay, joining joining fine. we. We have roughly 1.5 million
Daniel Bienstock: data points. We take many batches of size. They said, size 16. I'm not ex exactly sure. Y, 16. Okay, but the the sum that we see here this is I. I'm taking an average right? An average of the of the 16 gradients that I'm computing. And this is stochastic gradient descent.
Daniel Bienstock: Okay, so.
Daniel Bienstock: but really, what are we trying to do? You know, we are trying to maximize the probability that the that the system, the train network predicts the correct action. Okay, P of Ak, given sk is the action that the sorry ak given sk is the action that the expert took given State. Sk
Daniel Bienstock: p of ak, sk, let me let me further highlight. This is the probability that our train system gives to this correct action.
Daniel Bienstock: And so if we didn't have the log we would just that we would have something. It looks natural or right with that for each sample. If we had only one example instead of M. Equals to 16, we have M equal to one we are at that point. We are trying to maximize the probability of the action taken by the expert.
Daniel Bienstock: And then we are taking an average which is a stochastic gradient descent part. But it's not that the P. It's a log of the P. Okay? Why the log? Okay. Now, log is consistent with probability. If you maximize the log, you're also maximizing the probability. So that's okay. But still, why the log? Okay. So we'll get to that in a minute. Okay, we'll get to that in a little bit.
Daniel Bienstock: So this is what these characters did. Okay, they took. Let's say, a training data set of one and a half or close to 2 million moves taken by experts.
Daniel Bienstock: And they use that to train the network using gradient descent exactly with these rules. What else did they have? Hold on! Hold on! Wait a minute what is Alpha? Alpha is the training rate. Okay, this is the step size for those of us who remember first order methods. What did they do? They use Alpha equals to 3 times 10 to the minus 3 very small.
Daniel Bienstock: and reduced
Daniel Bienstock: by a factor of a half
Daniel Bienstock: every. How many, how many steps, every what, every 80 million steps.
Daniel Bienstock: This tells you that it took a lot of steps
Daniel Bienstock: they are not shy. Okay, what else? No momentum we learned about momentum
Daniel Bienstock: in previous lectures. No momentum, straight, fixed step, fixed learning rate grade in the center. Simple as possible. Okay. They say, now in the network, in the, in the network, in the paper. They describe some details about the network
Daniel Bienstock: in a future lecture in a future lecture.
Daniel Bienstock: We'll do something practical. But discuss, you know, what are the network architectures?
Daniel Bienstock: There's some terminology. If you read this Google paper in that methods section. They have a subsection, a few paragraphs describing the network architecture. The main thing that that we can say is, it has 13 layers
Daniel Bienstock: if you keep reading, and that they tell you something about the number of revenues that they have. You'll see the term kernel and the term stride. These are AI, or I should say Ml. Terms that describe to some extent the architecture
Daniel Bienstock: of the network, how the layers, what the layers look like. The last layer was what we called a convolutional layer, or the last between the last 2 layers is a convolutional layer means it's fully connected.
Daniel Bienstock: Okay, what is the size of these layers. How about the first layer? Well, they you have to be able to input a state. Okay? A ste, the go board has an is like 19 by 19.
Daniel Bienstock: And so and you have to have that dimensionality, but a little more, because you have to say where the pieces are. But even 19 by 19. And it's roughly, you know, almost 400 dimensional.
Daniel Bienstock: Okay, 13 layers. So alright it, they say, hold on.
Daniel Bienstock: They say that it took some time to train this. How long did it take? I think they say 3 weeks, 3 weeks
Daniel Bienstock: to do the training they have this. They have this word in there. It took 3 weeks this term. I'm not exactly sure what it means. Does it mean 3 weeks of continuous running of their their computing. That would be the direct interpretation. I don't know if that's what they mean, or if they they, this is the entire length of time that they spent, you know, with the.
Daniel Bienstock: you know, making mistakes and correcting them and changing parameters and hyper parameters. I do not know. Okay? And roughly, roughly, it's it it oh, they furthermore say, Oh, and I I cannot see. I'm computing impaired today because of various issues. Sorry. 14 layers, 14 layers, and in terms of the total runtime
Daniel Bienstock: it works to this works to roughly 100 and I forget. I forget the number of steps that they took. These are gradient steps. Okay? But I did the computation. And it works roughly, about 187 steps per second.
Daniel Bienstock: Okay, and I don't know whether this is very fast or not. Very fast. Okay, and so so this is the training, the training data set.
Daniel Bienstock: Okay, they you. They approximately solve this, a gradient descent problem to try to maximize the probability.
Daniel Bienstock: that you pick the right action in the in the given the the input state just maximize that probability, and then they tested it, testing data
Daniel Bienstock: on what? Roughly 29 million
Daniel Bienstock: cases.
Daniel Bienstock: So the testing data was much, much bigger than the training data and the accuracy that they got
Daniel Bienstock: accuracy was about how much? 57%.
Daniel Bienstock: Okay, so a little bit better than half the time.
Daniel Bienstock: Okay? And apparently this was much better than what was available in the state of the art.
Daniel Bienstock: Okay? Now, what other information can we?
Daniel Bienstock: can we provide? I lost the numbers here. So roughly.
Daniel Bienstock: So this is training. Okay in deployment when you want to use it. And you want to evaluate the state
Daniel Bienstock: in order to make a prediction. How fast was that? And they said roughly, 3 ms.
Daniel Bienstock: They also drained a less accurate network where the accuracy was a lot less than 57% only about 25.
Daniel Bienstock: But the advantage is that the making a prediction was much faster, only a few microseconds.
Daniel Bienstock: and we will see that later on, perhaps in the next lecture. Why, that is important.
Daniel Bienstock: Okay, so this is supervised learning of of policy networks. The word policy has to do with the fact that, given a state.
Daniel Bienstock: the trained network amounts to a policy. It tells you what to do.
Daniel Bienstock: Okay.
Daniel Bienstock: what do you do? That's a policy. This is a a old language really in in in decision sciences.
Daniel Bienstock: Just a policy. Alright.
Daniel Bienstock: and it's supervised. Because well, they we had the data provided by the masters, the other supervisors.
Daniel Bienstock: And it's a network. And we learned it. Okay.
Daniel Bienstock: the next element and what they had okay, moving on with the the machine learning hierarchy is what
Daniel Bienstock: they would call, or they call reinforcement reinforce cement learning
Daniel Bienstock: of policy networks.
Daniel Bienstock: Okay? And what do I mean by that. So before we had supervised learning
Daniel Bienstock: reinforcement, learning means that that will use an algorithm to try to correct our errors. Okay.
Daniel Bienstock: so what is the setting.
Daniel Bienstock: we start
Daniel Bienstock: from the previously trained network.
Daniel Bienstock: Okay, and now we want to want to improve, want to improve
Daniel Bienstock: on the weights
Daniel Bienstock: on the trained weights. Sigma. These are the weights that define the network.
Daniel Bienstock: And what is the goal? Now? All right. Look before we had the system that tries to predict what a master would do.
Daniel Bienstock: You could use that to play a game
Daniel Bienstock: every time that you are in a certain board position.
Daniel Bienstock: Well, you use the network to tell you to predict what a master would do.
Daniel Bienstock: And you play that move.
Daniel Bienstock: Okay?
Daniel Bienstock: well, now, the goal is to here in reinforcement learning. We want to get better at that.
Daniel Bienstock: We want to start from the system we trained before that predicted what a master would do.
Daniel Bienstock: and starting from there.
Daniel Bienstock: improve on that system to win games
Daniel Bienstock: when games
Daniel Bienstock: okay? And what is the method?
Daniel Bienstock: So the method we could call it self play.
Daniel Bienstock: Okay, you play. This is how it's going to work play a game.
Daniel Bienstock: play a game
Daniel Bienstock: against an opponent.
Daniel Bienstock: actually, an opponent, an opponent is going to be one of our prior algorithms. And in the very first iteration it'll be the the previously trained network.
Daniel Bienstock: But because do it over and over again.
Daniel Bienstock: every time that we run this exercise, we're going to get a better network.
Daniel Bienstock: And now we'll play again against that network.
Daniel Bienstock: Okay, against a prior
Daniel Bienstock: algorithm.
Daniel Bienstock: Okay, and so what are the States that we see the States
Daniel Bienstock: going to be? S. One s. 2. These are board states. Okay, as we play the game.
Daniel Bienstock: And what is the T. They put a finite ending time for the game. Okay, if the game was taken too long. Well, it was given value 0. We're going to set up another maximization problem. So value 0 means that we didn't play the game too effectively. I forget what the T was. Some maximum number of moves that they allowed. Okay? And
Daniel Bienstock: What are the actions? What are the actions?
Daniel Bienstock: So these are the actions that that we took in playing the game. And let's say, these are a one, a 2. These are the moves that we took.
Daniel Bienstock: Okay? And what is the outcome.
Daniel Bienstock: the outcome. I'm going to denote it as a Z, which is going to be equal to plus one or minus one win or lose.
Daniel Bienstock: okay, or 0. If the game gets end up terminated, and then we applying gradient descent. So now again, let me write the formula gradient ascent ascent.
Daniel Bienstock: We want to basically win games. We want to increase the score that we can get.
Daniel Bienstock: So let me write the formula, and then we'll puzzle it out. It's going to look similar to what we had before. We're using exactly the same network, you know. So the the very first time that we we do this self play, we play against the previously trained network.
Daniel Bienstock: Okay? And now, this is one game, one game. Okay, we'll change this in a minute to use multiple games like in less than a minute. But let's do it for one game. Okay, Alpha, this is again, that learning rate the gradient the size of the gradient step.
Daniel Bienstock: And now I take the sun.
Daniel Bienstock: and we'll have to explain this again. You know why this? So the gradient, with respect to the weights of the logarithm
Daniel Bienstock: of the the of the probability. Pardon
Daniel Bienstock: action. T
Daniel Bienstock: given state T.
Daniel Bienstock: All of this
Daniel Bienstock: times. Z.
Daniel Bienstock: Alright. So what does this say? Let's say that Z is plus one.
Daniel Bienstock: So we won the game.
Daniel Bienstock: Okay, does this make sense? Yes, we want the game. And so now I'm doing something that at least superficially, is consistent with maximizing the probabilities of the various moves that we did.
Daniel Bienstock: Okay.
Daniel Bienstock: Alright.
Daniel Bienstock: but why the log again? Okay. Why the log? Again, we have to try to understand this. Well, actually, this is not quite right. We did.
Daniel Bienstock: Stochastic, gradient descent, actually
Daniel Bienstock: stochastic
Daniel Bienstock: great dissent.
Daniel Bienstock: which is what? Well, it's going to be alpha divided by the number of games. So we play multiple games. We don't play one game, we play multiple games. And now we take this sum. Okay? So it's an average. This is a stochastic part. We take an average
Daniel Bienstock: of steps
Daniel Bienstock: and then we have something. It looks like what I had above
Daniel Bienstock: T equals one. Well, each game ends up at a different time. Maybe.
Daniel Bienstock: Okay. Ti is at most as capital T. And now I have. Now I have the gradient log of P. Sigma of a T sub. I is, and king game, IST sub i, and now I have a Z sub. I, the outcome output of game. I outcome of game. I, okay, so this is just an average of terms of what I had before. So really, it's like a very similar
Daniel Bienstock: and so and what what was them? I forgot. They they told me. I mean, it's in the paper. Pardon, and there's a hundred 28.
Daniel Bienstock: So they play mini batches. They they, they play sequences of mini batches of a hundred 28 games
Daniel Bienstock: against a prior opponent.
Daniel Bienstock: From that they compute at one gradient step
Daniel Bienstock: for all the network weights. The gradient step is a gradient descent step. It's consistent with increasing the probability that we win again.
Daniel Bienstock: We are just looking at each sample, each trajectory, as it were.
Daniel Bienstock: Again, when Z is one, we want to make the probability big when Z is minus one, we lost
Daniel Bienstock: so and we make we want to decrease the probability.
Daniel Bienstock: And then, okay, how long did this take? They say that it took how long it took? One day one day.
Daniel Bienstock: in terms of the the training
Daniel Bienstock: with a world with 50 gpus.
Daniel Bienstock: Okay, so think about the Gpu as being one big computer.
Daniel Bienstock: So if they had 50 of them running in parallel for one day.
Daniel Bienstock: Okay, running stochastic, gradient descent, they say, asynchronously. So they're doing the different steps in parallel.
Daniel Bienstock: in this Mini batch computation that we have in here alright. But now we want to understand, you know why the log finally, why the log? Okay, why the log before we go there, you know. Notice that this the term with that has the gradient inside the sun. This is really the same. This is really the same
Daniel Bienstock: sorry I I I. This is correct. This is correct. What's inside inside inside the sum here? This is the same as the gradient of the sun. It's a sum of gradients, which is the gradient of the sum
Daniel Bienstock: T equals one to Ti.
Daniel Bienstock: Sigma's no sigma is gone
Daniel Bienstock: of log
Daniel Bienstock: of P. Sigma of action. T sub. I given state Z, sub, I,
Daniel Bienstock: okay, that's what that is.
Daniel Bienstock: And a log. A sum of logs is the log of of the product.
Daniel Bienstock: Okay, so this is really, this is really just the log of the probability of the entire sequence. Okay, because the different moves are independent of one another.
Daniel Bienstock: You know, if we find ourselves in state. Sti. Well, we'll take the move that the network the train network dictates at that point. The past is in is gone, it's independent.
Daniel Bienstock: And so this is a little bit reminiscent. Again, of dynamic programming.
Daniel Bienstock: Okay, so I'm just taking the the the log of the probability of the entire trajectory.
Daniel Bienstock: and I'm taking the gradient of that. So again, this is all very, very makes sense. But why the log? All right. So now let's get to that.
Daniel Bienstock: How much time do I have? Well, they don't have much, all right. So here's some classical literature that they cite.
Daniel Bienstock: Did they cite? They think they sign. I'm not sure how they cited it. Classical literature going back to the dawn of the AI. Okay, there's a one of their gods is this Guy Williams.
Daniel Bienstock: 1992.
Daniel Bienstock: And then even, and he and other people today, they cite even earlier work very famous work in very early work in AI Barton and Sutton.
Daniel Bienstock: and roughly, 1983.
Daniel Bienstock: Okay, I can tell you that back then nobody took AI seriously.
Daniel Bienstock: everybody would say, this is this is such a joke!
Daniel Bienstock: It's a complete joke. These are charlatans, you know. People who are, you know, should be thrown away out of universities and such.
Daniel Bienstock: Okay.
Daniel Bienstock: so now let's look at a generic setup for reinforcement learning.
Daniel Bienstock: I take it that some of you know I'm I'm I'm told in in good faith
Daniel Bienstock: that some of you know what a mark of decision process is.
Daniel Bienstock: but not everybody does.
Daniel Bienstock: So let me give you an example of a markup decision process and what it is. And and then we'll take the conversation elsewhere.
Daniel Bienstock: Okay, so what is a mark of decision process.
Daniel Bienstock: Okay, so we have a number of states. Okay, here we have a little 4 States.
Daniel Bienstock: and out of each State, we have possible transitions.
Daniel Bienstock: Let's say there and they are to there, and they are to there, I don't know. And and and that's it.
Daniel Bienstock: Okay. Now, in, in each state, in each State.
Daniel Bienstock: We have
Daniel Bienstock: some number of actions, possible actions, you know. Action one
Daniel Bienstock: 2, and 3 possible actions at that state. Let's call it State a.
Daniel Bienstock: And now there are 2 things that happen. One is that we get a reward.
Daniel Bienstock: If we take an action, we get a reward. So here's the reward.
Daniel Bienstock: And let's say the it's a 5 and minus 10 and and 6. Okay.
Daniel Bienstock: so at this state there are 3 actions.
Daniel Bienstock: and then for each action. If we take that action.
Daniel Bienstock: we get a reward.
Daniel Bienstock: What else? Well, in addition to getting a reward, we're going to get a probability distribution. Okay, one third and 2 thirds, or one half and one half, and 0 and one. So what is that probability? Distribution? Well, notice
Daniel Bienstock: that, you know there are 2 numbers in each case.
Daniel Bienstock: blue and green.
Daniel Bienstock: Okay, and there are 2 arcs going out blue and green.
Daniel Bienstock: Okay, so if we take action. The action one, then the probability distribution indicates a probability that we will switch
Daniel Bienstock: to States blue or green. From
Daniel Bienstock: okay.
Daniel Bienstock: And we have such information for every node of this network.
Daniel Bienstock: So we choose the action that we want to take.
Daniel Bienstock: and then we get the reward immediately.
Daniel Bienstock: and then we transition to the other States according to the probability distribution.
Daniel Bienstock: Another question is, okay, what policy should we have? What do I mean by a policy? A policy tells you in each state which action you should take.
Daniel Bienstock: Okay? And what is the goal? The goal is to maximize the total reward. Think about this. Well, if you run this infinitely many times, of course. Then you're going to get some kind of infinite, infinite reward. But let's make it time discounted. Okay, with some discounting factor, you know. Let's say a half.
Daniel Bienstock: You know we want to choose a but is there such a thing as a policy that maximizes them? The the maximum that maximizes the expected reward. Discounted expected reward.
Daniel Bienstock: Okay, a fixed policy.
Daniel Bienstock: And the answer is, yes, there is. There's a whole slew of theorems. And one nice thing that we don't have time to go over today is that this this particular problem can be solved as a small linear program.
Daniel Bienstock: Okay, as a small linear program. There are several methods for solving this problem efficiently. This markup decision model efficiently. And I can see that the blue is not showing up. Do you see the blue Matthias.
matias: Yes. Done.
Daniel Bienstock: Okay. Good.
matias: Oh, so you have.
matias: I I we only see 2 colors like tail and green, or.
Daniel Bienstock: Yeah. Okay, so let me let me highlight. I mean, whatever the color was that this is the other color.
Daniel Bienstock: Okay, now, it's a little more visible. Alright.
Daniel Bienstock: right! And then, for every node, you know, like the this node. On the other hand, you know they are. Notice there are no, no exiting states wrong thing
Daniel Bienstock: so once you're there, you're stuck there forever. Okay.
Daniel Bienstock: and you get the reward. You don't get anything.
Daniel Bienstock: And so on. This can happen
Daniel Bienstock: all right
Daniel Bienstock: now.
Daniel Bienstock: later. During the summer I have some some meetings with with the the students here. We'll go over Markov. Decision processes in a little more detail. Okay?
Daniel Bienstock: But now, alright
Daniel Bienstock: now that we understand what a system is and what a policy is. And so on. Let's look at a generic setup, not for mark of decision processes, for reinforcing reinforcement learning.
Daniel Bienstock: Okay, which is basically something that Williams was looking at.
Daniel Bienstock: So we control a system. Okay? And now we can talk in generalities. We control a system
Daniel Bienstock: by a policy.
Daniel Bienstock: Let's call it pi, okay, in the network learning case, the policy really was the network. Okay.
Daniel Bienstock: the network instantiates a policy
Daniel Bienstock: and given a state.
Daniel Bienstock: It tells you what to do.
Daniel Bienstock: You pull a lever, a big lever. Given a state that policy. Pi tells you what to do.
Daniel Bienstock: Alright. Now the policy
Daniel Bienstock: is controlled
Daniel Bienstock: by parameters.
Daniel Bienstock: Sigma, which is a high dimensional vector and in the network case well, these are the weights of the network.
Daniel Bienstock: Okay, the system.
Daniel Bienstock: Given an application
Daniel Bienstock: of the policy.
Daniel Bienstock: this system
Daniel Bienstock: will follow
Daniel Bienstock: a stochastic trajectory.
Daniel Bienstock: Then I'm going to call Tao.
Daniel Bienstock: Okay? In the network case, it's stochastic. Well, because at the very end the very last layer, this soft Max layer, and so on. That's a probability.
Daniel Bienstock: Okay, it's stochastic.
Daniel Bienstock: We we assemble an action to take according to the probabilities that the network has produced.
Daniel Bienstock: We run as we run the state as the input you run it. It produces these numbers, which are probabilities.
Daniel Bienstock: We sample from the distribution, and we get the desired action.
Daniel Bienstock: Okay? And let's use a notation
Daniel Bienstock: pi of Tau is the probability.
Daniel Bienstock: Oh, that trajectory
Daniel Bienstock: in a more general reinforcement learning setup. There could be stochastics at multiple steps.
Daniel Bienstock: Okay? And so
Daniel Bienstock: at termination, at termination
Daniel Bienstock: we get a reward.
Daniel Bienstock: Then I'm going to call R of Tau.
Daniel Bienstock: That depends on the entire trajectory. Okay? And the network setup I was describing before. The the reward does not depend on the entire trajectory. We just collect a reward at the very end if we match if we win the game. Pardon or not. Okay.
Daniel Bienstock: it was plus or minus or minus one alright. And so
Daniel Bienstock: maximize. The goal goal
Daniel Bienstock: is to choose policy, choose policy
Daniel Bienstock: to maximize
Daniel Bienstock: the expected reward.
Daniel Bienstock: which is what that's the integral. So that'd be a little bit sloppy, the integral. Let me write that the notation and then we will debug it. Okay, it's a little, just a little sloppy, you know, it's a elegantly sloppy
Daniel Bienstock: okay, this is our notation for saying what we are saying. Okay, look, we are sampling trajectories
Daniel Bienstock: with some distribution. So this is like a stochastic integral pi is the probability of sampling trajectory Tau, and then we get the reward. So this is the expected reward. And we want to maximize this.
Daniel Bienstock: Okay, it's only a little bit sloppy.
Daniel Bienstock: It's a sloppy enough to scandalize somebody who's serious about stochastic processes.
Daniel Bienstock: But it's correct enough that we understand what it says.
Daniel Bienstock: We want to maximize this. How do we maximize this? Well, Williams, back, you know, like I don't know. 100 years ago, whatever 1992, he said. Look, I have no clue. What we are doing. Let's take gradient steps. Okay, this is long before modern AI and Ml. And Gpus, and this and that, and you know anything existed, he said. Look, this is so hard to do.
Daniel Bienstock: I know how to take gradients, you know. Let's take a small gradient steps and then start with a policy and correct it. Okay, and now that the thing is well.
Daniel Bienstock: how do we take the gradient of something like this?
Daniel Bienstock: So we want to take the gradient.
Daniel Bienstock: Oh, boy, it seems like we actually going to make it through the lecture on time? Excellent! The gradient of the expected reward, the gradient of this.
Daniel Bienstock: Let me copy this
Daniel Bienstock: paste.
Daniel Bienstock: and I? Well, you know, let's not review how what this means in in in the case of the prior slide, this.
Daniel Bienstock: Okay? let's put aside
Daniel Bienstock: or let's ignore for the time being, the details toward the bottom. Here
Daniel Bienstock: now we are sampling. If we have a policy, if we have a policy which is a trained network.
Daniel Bienstock: We are sampling games. Okay? That's those are our trajectories. A game is a trajectory.
Daniel Bienstock: Okay? And we want to maximize the expected reward that we would get. You know it's a it's a if we if we keep playing many games over and over and over again. You know we are sampling from the same distribution for a given train network.
Daniel Bienstock: It's you know, we're playing a game at each move of the game. We're getting a certain move with a certain probability.
Daniel Bienstock: and at termination of the game we get a reward.
Daniel Bienstock: And so we keep sampling games.
Daniel Bienstock: and we want to somehow maximize the expected reward.
Daniel Bienstock: Buy a first order methods.
Daniel Bienstock: Okay? So we need to be able to compute something like that. A gradient of of an integral, you know, like combines 2 things that probably many of my students don't like. Alright so how do we do that? Okay, so let me copy that and move on to the next page.
Daniel Bienstock: So we want to do this.
Daniel Bienstock: Okay? And let's refresh our memory. The tiles of the the we are sampling trajectories. According to our policy, R. Of Tau is the terminal reward that we get, or or the reward that we get as we go through the trajectory and pi of Tau is the probability that we sample that particular trajectory.
Daniel Bienstock: Okay, so alright, let's see if we can do some math over here. Okay, you want to take the gradient. So let's break some eggs. This is we can immediately say, this is equal to this.
Daniel Bienstock: Okay, so I switch the gradient with the integral. I'm sure that there there's there are hundreds of pages of mathematics that say, well, in order for you to do that, you have to satisfy A, BCD, and E, and so on. And I'm saying, Yes, I satisfy everything. I'm I'm happy with us. Okay.
Daniel Bienstock: alright.
Daniel Bienstock: So how do we do that? Okay.
Daniel Bienstock: alright. So now let's be creative. Okay, there's a a proof. What am I doing here? Okay, what am I doing here. I'm doing something that is called the policy gradient theorem.
Daniel Bienstock: which I think was proved by this guy Williams that I had before. But it's possible. Some people say that actually, these 2 other guys, Barto and Sutton had already proved or outlined or something. Okay. But this a policy gradient. Theorem.
Daniel Bienstock: Okay policy, gradient theorem or gradient policy. Theorem is like one of the golden one of the the. This is the eleventh commandment of the AI community, the traditional AI community.
Daniel Bienstock: This is what they believe in. So alright. So let's let's do this. Okay.
Daniel Bienstock: now, alright, I want you to think about this and and what I'm going to be doing here. I'm sampling in in the integral that is fully written up at the top on the right. I'm sampling trajectories.
Daniel Bienstock: So let's say that the towel has been sampled. Okay.
Daniel Bienstock: I have a given trajectory.
Daniel Bienstock: It has a certain reward.
Daniel Bienstock: So
Daniel Bienstock: let me write this, and then let's see if we agree. This is equal to the gradient of the probability.
Daniel Bienstock: Okay.
Daniel Bienstock: time's a reward.
Daniel Bienstock: And again, I want you to think of the integral at the top, right? As one where you're just. It's like a an integral is always like a big sum. Okay
Daniel Bienstock: for any given cow. It's a trajectory. Its reward is is a well given.
Daniel Bienstock: The only function of the parameters, sigma, are the probabilities, not the not the reward of the of the trajectory, and this is why I can write the the integral at the bottom left.
Daniel Bienstock: The rewards do not depend on the probabilities given the trajectory.
Daniel Bienstock: Okay, it took me like 5 days to understand this. And so what does this equal to? So now let me multiply and divide
Daniel Bienstock: by that.
Daniel Bienstock: And I have this.
Daniel Bienstock: Okay, I multiplied and divided. Truth be told, the pi, the probability pi depends on the Sigmas. I'm I'm I'm I'm skipping that to make the notations lighter. Remember, in the network training case, the probability is output by the network.
Daniel Bienstock: And the network depends on the segments. Okay? And that's what we are changing times the reward.
Daniel Bienstock: Times. D pi, okay, detail. And now equal. And now what is the sequel? So this animal here, this animal here, that is delta sigma of the log. Finally.
Daniel Bienstock: okay. Times, pi of Tau
Daniel Bienstock: times are of Tao
Daniel Bienstock: detail. Okay? And now, what is this animal? Okay, remember, pi is the probability of trajectory
Daniel Bienstock: Tau. This is the expectation. This is the expectation of the log of Sigma of the log, of the probability
Daniel Bienstock: times. R.
Daniel Bienstock: That's it.
Daniel Bienstock: It's the expectation of that quantity. The pi entered this pie here entered because we are taking an expectation. That's all.
Daniel Bienstock: Okay, that thing, the integral and the pi means that I'm taking an expectation.
Daniel Bienstock: And this is this famous gradient policy, theorem
Daniel Bienstock: or policy grading policy grading.
Daniel Bienstock: Okay.
Daniel Bienstock: alright. And so if we go back to what we were doing before
Daniel Bienstock: you can see here in in this term over here in this term, over here in the stochastic term, taking this average, this average. Let me leave out. Let me leave out the alpha.
Daniel Bienstock: Let me leave that out, or rather, let me write that outside.
Daniel Bienstock: Okay, the stuff that is in yellow. This is an estimate of that expectation. Think about the central limit, theorem. Okay? Applied to a function
Daniel Bienstock: to take an average. You just take a large number of samples, and then you average them.
Daniel Bienstock: And so here I'm taking a large well, a hundred 28
Daniel Bienstock: samples of the gradient
Daniel Bienstock: of what of of the Lord? Times a reward?
Daniel Bienstock: Okay.
Daniel Bienstock: The reward does not depend on on on anything really given. Given that the the trajectory. Given the trajectory.
Daniel Bienstock: Okay? And, in fact.
Daniel Bienstock: here I could have done that right if the expectation with respect to the weights, or the sigma or the towel. Pardon the trajectory. It's the same thing but the the if we think again in terms of the sampling interpretation of an expectation there the rewards don't depend okay
Daniel Bienstock: on on the weights.
Daniel Bienstock: Alright, but I should put it in there otherwise it it's not clear what it is that I'm doing.
Daniel Bienstock: But I wanted to highlight that alright.
Daniel Bienstock: And this is the policy gradient algorithm. And that's what we are doing in here. We are sampling games. Okay, that's the towel. The game. A full game is a towel.
Daniel Bienstock: I'm taking the the gradient of each of the samples, and I'm averaging them, and then the alpha is the learning rate, the step size.
Daniel Bienstock: And so they're doing gradient descent consistent, consistent with with maximizing. Well, maximize, Max maximizing the the the expected reward, maximizing the expected reward which is winning the game
Daniel Bienstock: alrighty. And this was a step 2 of what they did. But you know it's something that is standard. This a policy gradient. Theorem.
Daniel Bienstock: Okay? Alright
Daniel Bienstock: alright. And what was the outcome of all of us? With these, you know, 128 Mini batches one day, and 50 gpus and all that. So they had started with the original train system that was supposed to mimic what the masters did.
Daniel Bienstock: Then they took the weights from from that, and then they applied this reinforcement learning algorithm
Daniel Bienstock: to actually improve their chances of winning games.
Daniel Bienstock: So now they had a better game playing system. Okay? And what were the the hitting? The statistics
Daniel Bienstock: for this? And so they said that they beat
Daniel Bienstock: the original. The masters
Daniel Bienstock: trained
Daniel Bienstock: system
Daniel Bienstock: 80% of the time.
matias: Before it was.
Daniel Bienstock: Loud.
matias: Before it was 7, 57, right.
Daniel Bienstock: No, no, so the the 57 was accuracy, and predicting individual moves.
Daniel Bienstock: Now they use that system. Now they played games against that correct.
matias: You see.
Daniel Bienstock: Now the part that I'm that I that I skipped here. Sorry. Let me go back because it is important. Here is that every how many, every 500 steps or so. Okay.
Daniel Bienstock: they would take the network that that they had just computed.
Daniel Bienstock: and they would make that into a new opponent.
Daniel Bienstock: Okay? And after a while they would have this set of previously developed opponents.
Daniel Bienstock: And then, the next time that they played a game like here play a game against a prior adversary. They would pick a random one
Daniel Bienstock: of the ones that they had previously trained.
Daniel Bienstock: So now they are doing self play.
Daniel Bienstock: Okay?
Daniel Bienstock: So what else? Now there was a system. It was an existing open source system, open source.
Daniel Bienstock: Code.
Daniel Bienstock: there was call. I think it was called Pachy. I have no idea what Pachy means. Okay, and I don't know what language that's supposed to be. This was a a simulation Monte Carlo.
Daniel Bienstock: Monte Carlo
Daniel Bienstock: system.
Daniel Bienstock: It would evaluate
Daniel Bienstock: a hundred 1,000
Daniel Bienstock: moves per second.
Daniel Bienstock: according to some criterion.
Daniel Bienstock: Okay? And choose what it decided was the best.
Daniel Bienstock: Okay? And so it beat this guy. It beat
Daniel Bienstock: Pachy
Daniel Bienstock: 85% of the time.
Daniel Bienstock: Okay, Pachy was, I guess, considered the best
Daniel Bienstock: at that point. Okay? And they beat it. 85% of the time and the previous
Daniel Bienstock: best
Daniel Bienstock: from any other system was 12% of the time.
Daniel Bienstock: So up to then, a anybody who played against Pachy. At best. They beat Pachy 12% of the time. But this system, this Alphago system I I mean, up to this point we are missing.
Daniel Bienstock: We're missing 2 big things.
Daniel Bienstock: It was already beating the best 85% of the time.
Daniel Bienstock: Alright, which is great. But not the best that one can do.
Daniel Bienstock: Okay. Now, there's more that we have to go into the next lecture. Okay. there are 2 things. Let me outline them both. They are both very important. They go together. Okay, One thing is called what it's called reinforce reinforcement
Daniel Bienstock: learning.
Daniel Bienstock: So we know what that means by now of something networks.
Daniel Bienstock: I'm going to call it value. This is new. What do I mean by this.
Daniel Bienstock: So the last thing that I described
Daniel Bienstock: was a system for playing a game. It would play games for you.
Daniel Bienstock: Okay, what does this do? This predicts
Daniel Bienstock: win or loss, win or lose
Daniel Bienstock: quickly.
Daniel Bienstock: Okay, so this is no longer a system for playing games. This tells you given a state of the board.
Daniel Bienstock: If you play really? Well, will you win?
Daniel Bienstock: Okay or not?
Daniel Bienstock: Okay.
Daniel Bienstock: We have to describe next lecture what this is. Okay? it has to be, what are the goals?
Daniel Bienstock: It should be very fast, should be very fast. Goal.
Daniel Bienstock: Make it very fast.
Daniel Bienstock: Question is, why? Well, there's going to be another algorithm that's going to use this as a sub routine many times.
Daniel Bienstock: many, many times. Why? Well, we'll have to explain. Okay, does not need
Daniel Bienstock: to be super accurate.
Daniel Bienstock: So for those of you who play any kind of card game or or anything like that, that requires many moves.
Daniel Bienstock: many moves
Daniel Bienstock: or the stock market.
Daniel Bienstock: Okay, if you know that you're going to be correct, at least some fraction of the time, and you can use that to reinforce your decisions that when you, when you actually do, when you reinforce that.
Daniel Bienstock: then maybe you can actually boost your chances of winning. So this is the spirit.
Daniel Bienstock: Alright. So this is only one thing. So this output went into the last thing, and probably the most important thing where we want to spend a little more time. Okay, maybe the single most important component of the entire setup that they did. All of that we did until now was a way to prepare to to develop the next. The final phase.
Daniel Bienstock: Everything that we did so far is very, very important, so that the next phase gets off the ground properly.
Daniel Bienstock: But it's going to be the most important phase of all. And that's called Monte Carlo tree search.
Daniel Bienstock: Okay? And usually abbreviated like that.
Daniel Bienstock: And what is that. Okay? So we'll definitely go through these 2 things next lecture.
Daniel Bienstock: we have to. So what is that? Okay, what is the tree that we are talking about here.
Daniel Bienstock: So imagine a game like chess or go, you know, and you play first.
Daniel Bienstock: And now there are many moves that you can make.
Daniel Bienstock: There are many moves that you can make. And so you can have a picture. You know, this is the beginning. And they all these different moves. There's a whole bunch of moves that you can make. And now you don't know. Okay, which which one is is a good move to make at the very beginning.
Daniel Bienstock: Well, then you could simulate. For each of these moves. You could simulate what your opponent would do. Okay, and your opponent in each case will have a bunch of moves.
Daniel Bienstock: So if you assume that your opponent is very intelligent. Okay, how would they choose it?
Daniel Bienstock: Well, they could simulate what you would do in each case.
Daniel Bienstock: And so if you continue like this, you're you're going to grow this humongous tree.
Daniel Bienstock: It's going to be both very broad and probably way too deep.
Daniel Bienstock: If somehow you could evaluate this tree all the way. Eventually each branch, each branch, is going to terminate in some terminal state where somebody wins.
Daniel Bienstock: Okay? And if you could. If you could visualize the entire tree all at once, then you could pick the best move in some sense
Daniel Bienstock: right, and the the worth of a move. That move would be dependent upon 2 things right? The it's going to be a blend, you know. In in terms of a binary game like chess or go. You know you want to know. Okay, in the subtree, you know. How likely is it that I win? That is to say, ho! How many times do I win.
Daniel Bienstock: and how many times do I not win?
Daniel Bienstock: So we have to combine the 2 things. There's some some kind of a stochastic interpretation and some kind of a value interpretation. Okay.
Daniel Bienstock: and you have to somehow balance the 2 of them.
Daniel Bienstock: So Monte Carlo tree search is a way to to heuristically, heuristically, evaluate a subtree of this tree
Daniel Bienstock: where you're evaluating. You are repeatedly evaluating some branch of the tree, and then you're adjusting your estimate of probabilities in particular.
Daniel Bienstock: of a success.
Daniel Bienstock: If you had a game that was not a binary game, it's the game that, in addition that gives you a a number, a value not plus minus one, but some number. Okay? Then it's a more delicate process, because you don't just want to to take the branch that gives you. You know, th has the single node the single leaf with the highest possible value. It's a blend of that. And how many times you know some kind of expectation, and so on.
Daniel Bienstock: And so, Monte Carlo tree search is a collection of heuristics
Daniel Bienstock: to try to narrow down the tree, both in terms of its breadth.
Daniel Bienstock: and also rapidly getting to the bottom in some sense
Daniel Bienstock: and so these last 2 elements is a reinforcement learning of value networks which builds upon the very first thing that we saw here, or it starts from from
Daniel Bienstock: from that supervised learning
Daniel Bienstock: and then it also uses this to give you good moves.
Daniel Bienstock: Give you good moves that are likely to win games. So the the first tool the the prediction of what a good move would be that in fast, in a fast mode is going to be used to develop a quick estimate as to whether given a state of the board, you're likely to win or not very quickly.
Daniel Bienstock: and the second one, the reinforcement learning of policy networks will be used to begin the Monte Carlo tree, search
Daniel Bienstock: and as we take a dive, you know, we'll be taking Dives down this tree. Okay.
Daniel Bienstock: then, occasionally, occasionally we may terminate the dive early. Okay, we. The game is not over yet.
Daniel Bienstock: But then we evaluate the chances of winning using this first component using that
Daniel Bienstock: very quickly. Okay. And we're going to take many, many such dives.
Daniel Bienstock: We have to discuss how this reinforcement, learning, this reinforcement learning of value networks actually works, and how Monte Carlo Tree search actually works.
Daniel Bienstock: Okay, there's a ton of heuristics in there. This is a very, very interesting.
Daniel Bienstock: They got it to work.
Daniel Bienstock: so we'll see next time how they did this, and then beginning next time, and probably finishing the time after. We'll see how the Deepmind people move from from this setup.
Daniel Bienstock: Which, incorporating all of these things they called Alphago.
Daniel Bienstock: where they moved to something, where they let go. Of the masters there were no more masters.
Daniel Bienstock: Okay, no more, no more training of their initial step.
Daniel Bienstock: using masters data There we go. Instead, they they kind of combine the 2 steps, this one and that one by doing self play.
Daniel Bienstock: They just kept playing against themselves over and over again.
Daniel Bienstock: And as they improved their algorithms they would
Daniel Bienstock: send these improved algorithms into this, a a basket of previously developed algorithms against which they keep playing games.
Daniel Bienstock: okay and always improving.
Daniel Bienstock: And when they completed that task, and we'll see that second, maybe next lecture or the lecture. After that. That's when they got this system. They beat everybody. Okay. They beat all the human masters, you know, by some ridiculous margins like, you know, they would have a championship. I don't know 5 games, and they would win 5 to nothing.
Daniel Bienstock: They beat all their all all other programs, a hundred percent of the time.
Daniel Bienstock: I'm not sure there's any competition today.
Daniel Bienstock: My understanding is that this technology is still getting developed, not necessarily to play silly games, but to do other things, and it's certainly not open source.
Daniel Bienstock: and how much we can divine by reading their papers is probably going to be somewhat limited, but it's still very entertaining.
Daniel Bienstock: Alright.
Daniel Bienstock: that's it for today. And I can see that the electricity company never called me. Hopefully, they're not going to cut power. And
Daniel Bienstock: I'll see you guys on Tuesday. I guess the semester is over, probably for for everybody.
Daniel Bienstock: But we have some ground to cover. You know there are some of these Ml. Related lectures, and then we should do something about Cuda.
Daniel Bienstock: because I promised that we would.
Daniel Bienstock: Okay, that's like.
Daniel Bienstock: Hey, yeah, take care.
Daniel Bienstock: and I'll stay on
Daniel Bienstock: with a blink.
Blake: Yeah, we're.
Daniel Bienstock: We are. We are on next right, Blake.
Blake: Yeah. Yep.
Daniel Bienstock: Alright! Give me 1 min.
Blake: Yeah, that's alright. I was gonna go grab a jacket.
Blake: Sorry. Okay.
Daniel Bienstock: Alright, I'm gonna get a drink.
Blake: Okay. Be right back.
Daniel Bienstock: Let me stop recording.