1 00:00:14,000 --> 00:00:18,000 Good morning everyone. Today we are going to do some 2 00:00:18,000 --> 00:00:23,000 algorithms, back to algorithms, and we are going to use a lot 3 00:00:23,000 --> 00:00:27,000 of the, well, some of the simpler mathematics 4 00:00:27,000 --> 00:00:31,000 that we developed last class like the master theorem for 5 00:00:31,000 --> 00:00:35,000 solving recurrences. We are going to use this a lot. 6 00:00:35,000 --> 00:00:39,000 Because we are going to talk about recursive algorithms 7 00:00:39,000 --> 00:00:41,000 today. And so we will find their 8 00:00:41,000 --> 00:00:43,000 running time using the master theorem. 9 00:00:43,000 --> 00:00:46,000 This is just the same as it was last time, I hope, 10 00:00:46,000 --> 00:00:49,000 unless I made a mistake. A couple of reminders. 11 00:00:49,000 --> 00:00:52,000 You should all go to recitation on Friday. 12 00:00:52,000 --> 00:00:54,000 That is required. If you want to, 13 00:00:54,000 --> 00:00:56,000 you can go to homework lab on Sunday. 14 00:00:56,000 --> 00:00:59,000 That may be a good excuse for you to actually work on your 15 00:00:59,000 --> 00:01:04,000 problem set a few hours early. Well, actually, 16 00:01:04,000 --> 00:01:07,000 it's due on Wednesday so you have lots of time. 17 00:01:07,000 --> 00:01:10,000 And there is no class on Monday. 18 00:01:10,000 --> 00:01:13,000 It is the holiday known as Student Holiday, 19 00:01:13,000 --> 00:01:16,000 so don't come. Today we are going to cover 20 00:01:16,000 --> 00:01:20,000 something called Divide and Conquer. 21 00:01:27,000 --> 00:01:31,000 Also known as divide and rule or divide et impera for those of 22 00:01:31,000 --> 00:01:34,000 you who know Latin, which is a tried and tested way 23 00:01:34,000 --> 00:01:39,000 of conquering a land by dividing it into sections of some kind. 24 00:01:39,000 --> 00:01:43,000 It could be different political factions, different whatever. 25 00:01:43,000 --> 00:01:47,000 And then somehow making them no longer like each other. 26 00:01:47,000 --> 00:01:50,000 Like starting a family feud is always a good method. 27 00:01:50,000 --> 00:01:53,000 You should remember this on your quiz. 28 00:01:53,000 --> 00:01:56,000 I'm kidding. And if you can separate this 29 00:01:56,000 --> 00:02:00,000 big power structure into little power structures such that you 30 00:02:00,000 --> 00:02:03,000 dominate each little power structure then you can conquer 31 00:02:03,000 --> 00:02:06,000 all of them individually, as long as you make sure they 32 00:02:06,000 --> 00:02:09,000 don't get back together. That is divide and conquer as 33 00:02:09,000 --> 00:02:11,000 practiced, say, by the British. 34 00:02:11,000 --> 00:02:14,000 But today we are going to do divide and conquer as practiced 35 00:02:14,000 --> 00:02:17,000 in Cormen, Leiserson, Rivest and Stein or every other 36 00:02:17,000 --> 00:02:20,000 algorithm textbook. This is a very basic and very 37 00:02:20,000 --> 00:02:22,000 powerful algorithm design technique. 38 00:02:22,000 --> 00:02:27,000 So, this is our first real algorithm design experience. 39 00:02:27,000 --> 00:02:30,000 We are still sort of mostly in the analysis mode, 40 00:02:30,000 --> 00:02:33,000 but we are going to do some actual design. 41 00:02:33,000 --> 00:02:37,000 We're going to cover maybe only three or four major design 42 00:02:37,000 --> 00:02:39,000 techniques. This is one of them, 43 00:02:39,000 --> 00:02:43,000 so it is really important. And it will lead to all sorts 44 00:02:43,000 --> 00:02:45,000 of recurrences, so we will get to use 45 00:02:45,000 --> 00:02:49,000 everything from last class and see why it is useful. 46 00:02:49,000 --> 00:02:51,000 As you might expect, the first step in 47 00:02:51,000 --> 00:02:57,000 divide-and-conquer is divide and the second step is conquer. 48 00:02:57,000 --> 00:03:01,000 But you may not have guessed that there are three steps. 49 00:03:01,000 --> 00:03:04,000 And I am leaving some blank space here, so you should, 50 00:03:04,000 --> 00:03:06,000 too. Divide-and-conquer is an 51 00:03:06,000 --> 00:03:10,000 algorithmic technique. You are given some big problem 52 00:03:10,000 --> 00:03:14,000 you want to solve, you don't really know how to 53 00:03:14,000 --> 00:03:18,000 solve it in an efficient way, so you are going to split it up 54 00:03:18,000 --> 00:03:22,000 into subproblems. That is the divide. 55 00:03:22,000 --> 00:03:26,000 You are going to divide that problem, or more precisely the 56 00:03:26,000 --> 00:03:30,000 instance of that problem, the particular instance of that 57 00:03:30,000 --> 00:03:32,000 problem you have into subproblems. 58 00:03:32,000 --> 00:03:36,000 And those subproblems should be smaller in some sense. 59 00:03:36,000 --> 00:03:41,000 And smaller means normally that the value of N is smaller than 60 00:03:41,000 --> 00:03:45,000 it was in the original problem. So, you sort of made some 61 00:03:45,000 --> 00:03:47,000 progress. Now you have one, 62 00:03:47,000 --> 00:03:51,000 or more likely you have several subproblems you need to solve. 63 00:03:51,000 --> 00:03:55,000 Each of them is smaller. So, you recursively solve each 64 00:03:55,000 --> 00:04:00,000 subproblem. That is the conquer step. 65 00:04:00,000 --> 00:04:04,000 You conquer each subproblem recursively. 66 00:04:04,000 --> 00:04:11,000 And then somehow you combine those solutions into a solution 67 00:04:11,000 --> 00:04:14,000 for the whole problem. 68 00:04:23,000 --> 00:04:26,000 So, this is the general divide-and-conquer paradigm. 69 00:04:26,000 --> 00:04:29,000 And lots of algorithms fit it. You have already seen one 70 00:04:29,000 --> 00:04:33,000 algorithm that fits this paradigm, if you can remember. 71 00:04:33,000 --> 00:04:36,000 Merge sort, good. Wow, you are all awake. 72 00:04:36,000 --> 00:04:39,000 I'm impressed. So, we saw merge sort. 73 00:04:39,000 --> 00:04:42,000 And, if I am clever, I could fit it in this space. 74 00:04:42,000 --> 00:04:44,000 Sure. Let's be clever. 75 00:04:44,000 --> 00:04:48,000 A quick review on merge sort. Phrased in this 1, 76 00:04:48,000 --> 00:04:52,000 2, 3 kind of method. The first step was to divide 77 00:04:52,000 --> 00:04:56,000 your array into two halves. This really doesn't mean 78 00:04:56,000 --> 00:04:59,000 anything because you just sort of think, oh, 79 00:04:59,000 --> 00:05:05,000 I will pretend my array is divided into two halves. 80 00:05:05,000 --> 00:05:07,000 There is no work here. This is zero time. 81 00:05:07,000 --> 00:05:11,000 You just look at your array. Here is your array. 82 00:05:11,000 --> 00:05:14,000 I guess maybe you compute n/2 and take the floor. 83 00:05:14,000 --> 00:05:17,000 That takes constant time. And you say OK, 84 00:05:17,000 --> 00:05:21,000 I am pretending my array is now divided into the left part and 85 00:05:21,000 --> 00:05:24,000 the right part. And then the interesting part 86 00:05:24,000 --> 00:05:27,000 is that you recursively solve each one. 87 00:05:27,000 --> 00:05:32,000 That's the conquer. We recursively sort each 88 00:05:32,000 --> 00:05:37,000 subarray. And then the third step is to 89 00:05:37,000 --> 00:05:44,000 combine those solutions. And so here we really see what 90 00:05:44,000 --> 00:05:49,000 this means. We now have a sorted version of 91 00:05:49,000 --> 00:05:55,000 this array by induction. We have a sorted version of 92 00:05:55,000 --> 00:06:02,000 this array by induction. We now want the sorted version 93 00:06:02,000 --> 00:06:07,000 of the whole array. And we saw that was the merge 94 00:06:07,000 --> 00:06:10,000 problem, merging two sorted arrays. 95 00:06:10,000 --> 00:06:15,000 And that we saw how to do in linear time, order n time. 96 00:06:15,000 --> 00:06:20,000 I am not going to repeat that, but the point is it falls into 97 00:06:20,000 --> 00:06:24,000 that framework. I want to write the running 98 00:06:24,000 --> 00:06:29,000 time and merge sort as a recurrence. 99 00:06:29,000 --> 00:06:33,000 You have already seen the recurrence, you have already 100 00:06:33,000 --> 00:06:37,000 been told the solution, but now we actually know how to 101 00:06:37,000 --> 00:06:39,000 solve it. And, furthermore, 102 00:06:39,000 --> 00:06:43,000 every algorithm that follows the divide-and-conquer paradigm 103 00:06:43,000 --> 00:06:47,000 will have a recurrence of pretty much the same form, 104 00:06:47,000 --> 00:06:51,000 very much like our good friend the master method. 105 00:06:51,000 --> 00:06:56,000 Let's do it for merge sort where we sort of already know 106 00:06:56,000 --> 00:07:00,000 the answer and get a bit of practice. 107 00:07:00,000 --> 00:07:02,000 This is the merge sort recurrence. 108 00:07:02,000 --> 00:07:06,000 You should know and love this recurrence because it comes up 109 00:07:06,000 --> 00:07:09,000 all over the place. It comes from this general 110 00:07:09,000 --> 00:07:13,000 approach by just seeing what are the sizes of the subproblems you 111 00:07:13,000 --> 00:07:17,000 are solving and how many there are and how much extra work you 112 00:07:17,000 --> 00:07:20,000 are doing. You have here the size of the 113 00:07:20,000 --> 00:07:22,000 subproblems. It happens here that both 114 00:07:22,000 --> 00:07:25,000 subproblems have the same size roughly. 115 00:07:25,000 --> 00:07:29,000 There is this sloppiness that we have, which really should be 116 00:07:29,000 --> 00:07:34,000 T of floor of n over 2 plus T of ceiling of n over 2. 117 00:07:34,000 --> 00:07:37,000 And when you go to recitation on Friday you will see why that 118 00:07:37,000 --> 00:07:40,000 is OK, the floors and ceilings don't matter. 119 00:07:40,000 --> 00:07:43,000 There is a theorem you can prove that that's happy. 120 00:07:43,000 --> 00:07:47,000 You can assume that N is a power of 2, but we are just 121 00:07:47,000 --> 00:07:51,000 going to assume that for now. We just have two problems with 122 00:07:51,000 --> 00:07:53,000 size n over 2. This 2 is the number of 123 00:07:53,000 --> 00:07:55,000 subproblems. 124 00:08:00,000 --> 00:08:03,000 And this order n is all the extra work we are doing. 125 00:08:03,000 --> 00:08:07,000 Now, what is the extra work potentially? 126 00:08:07,000 --> 00:08:10,000 Well, the conquering is always just recursion. 127 00:08:10,000 --> 00:08:14,000 There is sort of no work there except this lead part. 128 00:08:14,000 --> 00:08:19,000 The dividing in this case is trivial, but in general it might 129 00:08:19,000 --> 00:08:23,000 involve some work. And the combining here involves 130 00:08:23,000 --> 00:08:25,000 linear work. So, this is the 131 00:08:25,000 --> 00:08:29,000 divide-and-conquer running times. 132 00:08:29,000 --> 00:08:31,000 So, this is the nonrecursive work. 133 00:08:31,000 --> 00:08:36,000 And that is generally how you convert a divide-and-conquer 134 00:08:36,000 --> 00:08:41,000 algorithm into a recurrence. It's really easy and you 135 00:08:41,000 --> 00:08:44,000 usually get to apply the master method. 136 00:08:44,000 --> 00:08:47,000 Here we are in Case 2. Very good. 137 00:08:47,000 --> 00:08:49,000 This is Case 2. And k is zero here. 138 00:08:49,000 --> 00:08:54,000 And so in the recursion tree, all of the costs are roughly 139 00:08:54,000 --> 00:08:58,000 the same. They are all n to the log base 140 00:08:58,000 --> 00:09:02,000 b of a. Here n to the log base 2 of 2 141 00:09:02,000 --> 00:09:04,000 is just n. So these are equal. 142 00:09:04,000 --> 00:09:07,000 We get an extra log factor because of the number of levels 143 00:09:07,000 --> 00:09:11,000 in the recursion tree. Remember the intuition behind 144 00:09:11,000 --> 00:09:13,000 the master method. This is n log n, 145 00:09:13,000 --> 00:09:16,000 and that is good. Merge sort is a fast sorting 146 00:09:16,000 --> 00:09:20,000 algorithm n log n. Insertion sort was n squared. 147 00:09:20,000 --> 00:09:23,000 In some sense, n log n is the best you can do. 148 00:09:23,000 --> 00:09:26,000 We will cover that in two lectures from now, 149 00:09:26,000 --> 00:09:30,000 but just foreshadowing. Today we are going to do 150 00:09:30,000 --> 00:09:32,000 different divide-and-conquer algorithms. 151 00:09:32,000 --> 00:09:36,000 Sorting is one problem. There are all sorts of problems 152 00:09:36,000 --> 00:09:39,000 we might want to solve. It turns out a lot of them you 153 00:09:39,000 --> 00:09:42,000 can apply divide-and-conquer to. Not every problem. 154 00:09:42,000 --> 00:09:45,000 Like how to wake up in the morning, it's not so easy to 155 00:09:45,000 --> 00:09:48,000 solve a divide-and-conquer, although maybe that's a good 156 00:09:48,000 --> 00:09:51,000 problem set problem. 157 00:09:59,000 --> 00:10:03,000 The next divide-and-conquer algorithm we are going to look 158 00:10:03,000 --> 00:10:09,000 at is even simpler than sorting, even simpler than merge sort, 159 00:10:09,000 --> 00:10:13,000 but it drives home the point of when you have only one 160 00:10:13,000 --> 00:10:16,000 subproblem. How many people have seen 161 00:10:16,000 --> 00:10:19,000 binary search before? Anyone hasn't? 162 00:10:19,000 --> 00:10:22,000 One, OK. I will go very quickly then. 163 00:10:22,000 --> 00:10:27,000 You have some element X. You want to find X in a sorted 164 00:10:27,000 --> 00:10:31,000 array. How many people had not seen it 165 00:10:31,000 --> 00:10:33,000 before they saw it in recitation? 166 00:10:33,000 --> 00:10:34,000 No one, OK. Good. 167 00:10:34,000 --> 00:10:38,000 You have seen it in another class, probably 6.001 or 168 00:10:38,000 --> 00:10:39,000 something. Very good. 169 00:10:39,000 --> 00:10:42,000 You took the prerequisites. OK. 170 00:10:42,000 --> 00:10:46,000 I just want to phrase it as a divide-and-conquer because you 171 00:10:46,000 --> 00:10:50,000 don't normally see it that way. The divide step is you compare 172 00:10:50,000 --> 00:10:53,000 X with the middle element in your array. 173 00:10:53,000 --> 00:10:56,000 Then the conquer step. Here is your array. 174 00:10:56,000 --> 00:11:02,000 Here is the middle element. You compare X with this thing 175 00:11:02,000 --> 00:11:06,000 if let's say X is smaller than the middle element in your 176 00:11:06,000 --> 00:11:09,000 array. You know that X is in the left 177 00:11:09,000 --> 00:11:13,000 half because it is sorted, a nice loop invariant there, 178 00:11:13,000 --> 00:11:17,000 whatever, but we are just going to think of that as recursively 179 00:11:17,000 --> 00:11:22,000 I am going to solve the problem of finding X in this subarray. 180 00:11:22,000 --> 00:11:26,000 We recurse in one subarray, unlike merge sort where we had 181 00:11:26,000 --> 00:11:31,000 two recursions. And then the combined step we 182 00:11:31,000 --> 00:11:35,000 don't do anything. I mean if you find X in here 183 00:11:35,000 --> 00:11:38,000 then you've found X in the whole array. 184 00:11:38,000 --> 00:11:42,000 There is nothing to bring it back up really. 185 00:11:42,000 --> 00:11:46,000 So, this is just phrasing binary search in the 186 00:11:46,000 --> 00:11:51,000 divide-and-conquer paradigm. It is kind of a trivial 187 00:11:51,000 --> 00:11:56,000 example, but there are lots of circumstances where you only 188 00:11:56,000 --> 00:12:02,000 need to recurse in one side. And it is important to see how 189 00:12:02,000 --> 00:12:06,000 much of a difference making one recursion versus making two 190 00:12:06,000 --> 00:12:09,000 recursions can be. This is the recurrence for 191 00:12:09,000 --> 00:12:13,000 binary search. We start with a problem size n. 192 00:12:13,000 --> 00:12:16,000 We reduce it to 1. There is an implicit 1 factor 193 00:12:16,000 --> 00:12:19,000 here. One subproblem of size n over 2 194 00:12:19,000 --> 00:12:21,000 roughly. Again, floors and ceilings 195 00:12:21,000 --> 00:12:24,000 don't matter. Plus a constant which is to 196 00:12:24,000 --> 00:12:28,000 compare X with the middle element, so it is actually like 197 00:12:28,000 --> 00:12:32,000 one comparison. This has a solution, 198 00:12:32,000 --> 00:12:35,000 log n. And you all know the running 199 00:12:35,000 --> 00:12:38,000 time of binary search, but here it is at solving the 200 00:12:38,000 --> 00:12:41,000 recurrence. I mean, there are a couple of 201 00:12:41,000 --> 00:12:44,000 differences here. We don't have the additive 202 00:12:44,000 --> 00:12:48,000 order n term. If we did, it would be linear, 203 00:12:48,000 --> 00:12:51,000 the running the time. Still better than n log n. 204 00:12:51,000 --> 00:12:55,000 So, we are getting rid of the 2, bringing it down to a 1, 205 00:12:55,000 --> 00:13:00,000 taking the n and bringing it down to a 1. 206 00:13:00,000 --> 00:13:02,000 That is making the running time a lot faster, 207 00:13:02,000 --> 00:13:05,000 the whole factor of n faster. No big surprise there. 208 00:13:05,000 --> 00:13:08,000 Let's do some more interesting algorithms. 209 00:13:18,000 --> 00:13:24,000 The powering a number problem is I give you a number X. 210 00:13:24,000 --> 00:13:31,000 I give that as like a real number or floating point number 211 00:13:31,000 --> 00:13:35,000 or whatever. And I give you an integer n, 212 00:13:35,000 --> 00:13:40,000 at least zero. I want to compute X to the 213 00:13:40,000 --> 00:13:44,000 power n. So, it's a very simple problem. 214 00:13:44,000 --> 00:13:49,000 It is, in some sense, even easier than all of these. 215 00:13:49,000 --> 00:13:54,000 But here it is. And divide-and-conquer is sort 216 00:13:54,000 --> 00:14:00,000 of the right thing to do. So, the naÔve algorithm is very 217 00:14:00,000 --> 00:14:04,000 simple. How do you compute X to the 218 00:14:04,000 --> 00:14:07,000 power n? Well, the definition of X to 219 00:14:07,000 --> 00:14:11,000 the power n is I take X and I multiply by X n times. 220 00:14:11,000 --> 00:14:16,000 So, I take X times X times X times X where there are n copies 221 00:14:16,000 --> 00:14:19,000 of X totally. And that's X to the n. 222 00:14:19,000 --> 00:14:22,000 No big surprise. That is n multiplications, 223 00:14:22,000 --> 00:14:27,000 or n minus 1 multiplications, theta n time. 224 00:14:34,000 --> 00:14:39,000 But that's not the best you can do for this problem. 225 00:14:39,000 --> 00:14:44,000 Any suggestions on what we might do using 226 00:14:44,000 --> 00:14:49,000 divide-and-conquer? Has anyone seen this algorithm 227 00:14:49,000 --> 00:14:51,000 before? A few, OK. 228 00:14:51,000 --> 00:14:56,000 For the rest? Testing on the spot creativity, 229 00:14:56,000 --> 00:15:04,000 which is very difficult, but I always like a challenge. 230 00:15:04,000 --> 00:15:08,000 I mean random ideas. What could we possibly do to 231 00:15:08,000 --> 00:15:12,000 solve this problem in less than linear time? 232 00:15:12,000 --> 00:15:16,000 How is this sort of a divide-and-conquer problem? 233 00:15:16,000 --> 00:15:19,000 We have to inputs, X and n. 234 00:15:19,000 --> 00:15:22,000 Yeah? We could try to divide on X. 235 00:15:22,000 --> 00:15:25,000 It seems a bit hard. It is just some number. 236 00:15:25,000 --> 00:15:30,000 Or, we could try to divide on n. 237 00:15:30,000 --> 00:15:32,000 Any guesses? Look at X to the n over 2, 238 00:15:32,000 --> 00:15:35,000 very good. That is exactly the idea of the 239 00:15:35,000 --> 00:15:38,000 divide-and-conquer algorithm. 240 00:15:48,000 --> 00:15:50,000 We would like to look at X to the n over 2. 241 00:15:50,000 --> 00:15:53,000 This is going to be a little bit tricky. 242 00:15:53,000 --> 00:15:56,000 Now we are going to have to pay attention to floors and 243 00:15:56,000 --> 00:15:59,000 ceilings. What I would like to say is 244 00:15:59,000 --> 00:16:04,000 while X to the n is X to the n over 2 times X to the n over 2. 245 00:16:04,000 --> 00:16:07,000 And this is true if n is even. If it is odd then I need to be 246 00:16:07,000 --> 00:16:11,000 a little bit more careful. But let's just think about the 247 00:16:11,000 --> 00:16:14,000 intuition why this is a good divide-and-conquer algorithm. 248 00:16:14,000 --> 00:16:17,000 We have a problem of size n, let's say. 249 00:16:17,000 --> 00:16:20,000 We convert it into, it looks like two subproblems 250 00:16:20,000 --> 00:16:23,000 of size n over 2, but in fact they are the same 251 00:16:23,000 --> 00:16:25,000 subproblems. So, I only have to solve one of 252 00:16:25,000 --> 00:16:28,000 them. If I compute X to the n over 2. 253 00:16:28,000 --> 00:16:33,000 Yeah, I know X to the n over 2. So, there is one conversive 254 00:16:33,000 --> 00:16:38,000 call, problem of size n over 2, then I square that number. 255 00:16:38,000 --> 00:16:44,000 And that is one computation. So, exactly the same recurrence 256 00:16:44,000 --> 00:16:48,000 as binary search, log n time much better than n. 257 00:16:48,000 --> 00:16:51,000 Cool. I also have to solve the odd 258 00:16:51,000 --> 00:16:53,000 case. So, n is odd. 259 00:16:53,000 --> 00:00:02,000 I will look at n minus 1 over 260 00:16:56,000 --> 00:17:02,000 N minus 1 better be even. And then I am missing another 261 00:17:02,000 --> 00:17:04,000 factor of X. If n is odd, 262 00:17:04,000 --> 00:17:09,000 I am going to have to do one recursive call and two 263 00:17:09,000 --> 00:17:12,000 multiplications. The same recurrence. 264 00:17:12,000 --> 00:17:17,000 One recursive problem of size n over 2, plus constant time. 265 00:17:17,000 --> 00:17:21,000 The dividing work here is dividing by 2 and the 266 00:17:21,000 --> 00:17:27,000 combination work is doing one or possibly two multiplications. 267 00:17:27,000 --> 00:17:32,000 And this is lg n. And if all you are allowed to 268 00:17:32,000 --> 00:17:37,000 do is multiply numbers, lg n is the best you can do. 269 00:17:37,000 --> 00:17:40,000 Good. Simple but powerful. 270 00:17:40,000 --> 00:17:44,000 Whenever you want to compute a power of a number, 271 00:17:44,000 --> 00:17:48,000 now you know what to do. 272 00:17:55,000 --> 00:17:58,000 Does anyone not know the definition of Fibonacci numbers 273 00:17:58,000 --> 00:18:01,000 and is willing to admit it? OK, so this is a good old 274 00:18:01,000 --> 00:18:02,000 friend. I will write down the 275 00:18:02,000 --> 00:18:05,000 definition as just a reminder. And, in particular, 276 00:18:05,000 --> 00:18:07,000 the base cases. 277 00:18:17,000 --> 00:18:18,000 Fibonacci numbers, I will claim, 278 00:18:18,000 --> 00:18:21,000 are very important because it appears throughout nature. 279 00:18:21,000 --> 00:18:24,000 You look at certain fruits, you see the Fibonacci sequence. 280 00:18:24,000 --> 00:18:28,000 If you count the number of little bumps around each ring. 281 00:18:28,000 --> 00:18:31,000 If you look at the sand on the beach and how the waves hit it, 282 00:18:31,000 --> 00:18:34,000 it's the Fibonacci sequence I am told. 283 00:18:34,000 --> 00:18:39,000 If you look all over the place, Fibonacci sequence is there. 284 00:18:39,000 --> 00:18:43,000 How does nature compute the Fibonacci sequence? 285 00:18:43,000 --> 00:18:46,000 Well, that is a different class. 286 00:18:46,000 --> 00:18:52,000 But how are we going to compute the Fibonacci sequence as fast 287 00:18:52,000 --> 00:18:56,000 as possible? You have probably seen two 288 00:18:56,000 --> 00:18:59,000 algorithms. The most naÔve algorithm is the 289 00:18:59,000 --> 00:19:04,000 recursive algorithm. Where you say OK, 290 00:19:04,000 --> 00:19:06,000 f of n. I say well, if n is zero, 291 00:19:06,000 --> 00:19:09,000 return zero, if n is 1, return one. 292 00:19:09,000 --> 00:19:13,000 Otherwise, recursively compute f of n minus 1 and f of n minus 293 00:19:13,000 --> 00:19:17,000 2, add them together. How much time does this 294 00:19:17,000 --> 00:19:20,000 algorithm take, for those who have seen it 295 00:19:20,000 --> 00:19:22,000 before? This is not obvious to guess. 296 00:19:22,000 --> 00:19:26,000 It doesn't have to be exact. 297 00:19:31,000 --> 00:19:33,000 OK. And how many people have seen 298 00:19:33,000 --> 00:19:36,000 this algorithm before and analyzed it? 299 00:19:36,000 --> 00:19:39,000 Half, OK. So what is the running time? 300 00:19:39,000 --> 00:19:41,000 Really, really long, very good. 301 00:19:41,000 --> 00:19:44,000 Anymore precise answers? What's that? 302 00:19:44,000 --> 00:19:45,000 Exponential, yes. 303 00:19:45,000 --> 00:19:48,000 That is also correct and more precise. 304 00:19:48,000 --> 00:19:52,000 I will be even more precise. Maybe you haven't seen this 305 00:19:52,000 --> 00:19:56,000 analysis before. It's phi to the n where phi is 306 00:19:56,000 --> 00:20:01,000 the Golden Ratio. Again, Golden Ratio appears 307 00:20:01,000 --> 00:20:03,000 throughout the world in mathematics. 308 00:20:03,000 --> 00:20:07,000 This is probably the only time in this class, 309 00:20:07,000 --> 00:20:11,000 I'm afraid, but there we go. It made its cameo so we are 310 00:20:11,000 --> 00:20:14,000 happy. This is called exponential 311 00:20:14,000 --> 00:20:16,000 time. This is bigger than one, 312 00:20:16,000 --> 00:20:20,000 that's all you need to know. This is exponential time. 313 00:20:20,000 --> 00:20:25,000 Exponential time means basically some constant to the 314 00:20:25,000 --> 00:20:28,000 power n. Exponential time is a very long 315 00:20:28,000 --> 00:20:30,000 time. It's bad. 316 00:20:30,000 --> 00:20:35,000 Polynomial time is good. [LAUGHTER] This is what we 317 00:20:35,000 --> 00:20:38,000 want, polynomial time algorithms. 318 00:20:38,000 --> 00:20:42,000 This class is basically entirely about polynomial time 319 00:20:42,000 --> 00:20:44,000 algorithms. Question? 320 00:20:44,000 --> 00:20:47,000 Oh, say what the algorithm does again. 321 00:20:47,000 --> 00:20:53,000 Define function Fibonacci of n? I check for the base cases. 322 00:20:53,000 --> 00:20:59,000 Otherwise, I recursively call Fibonacci of n minus 1. 323 00:20:59,000 --> 00:21:02,000 I recursively call Fibonacci of n minus 2 and add those two 324 00:21:02,000 --> 00:21:04,000 numbers together. So, you get this branching 325 00:21:04,000 --> 00:21:06,000 tree. You are solving two subproblems 326 00:21:06,000 --> 00:21:09,000 of almost the same size, just additively smaller by one 327 00:21:09,000 --> 00:21:10,000 or two. I mean you are almost not 328 00:21:10,000 --> 00:21:13,000 reducing the problem size at all, so that's intuitively why 329 00:21:13,000 --> 00:21:16,000 it is exponential. You can draw a recursion tree 330 00:21:16,000 --> 00:21:18,000 and you will see how big it gets and how quickly. 331 00:21:18,000 --> 00:21:21,000 I mean by n over two levels, you've only reduced on one 332 00:21:21,000 --> 00:21:23,000 branch the problem from n to n over 2. 333 00:21:23,000 --> 00:21:26,000 The other one, maybe you've gotten from n down 334 00:21:26,000 --> 00:21:29,000 to one, but none of the branches have stopped after n of two 335 00:21:29,000 --> 00:21:32,000 levels. You have at least 2 to the 336 00:21:32,000 --> 00:21:37,000 power n over 2 which is like square root of 2 to the power n, 337 00:21:37,000 --> 00:21:40,000 which is getting close to phi to the n. 338 00:21:40,000 --> 00:21:42,000 So, this is definitely exponential. 339 00:21:42,000 --> 00:21:46,000 And exponential is bad. We want polynomial. 340 00:21:46,000 --> 00:21:49,000 N squared, n cubed, log n would be nice. 341 00:21:49,000 --> 00:21:53,000 Anything that is bounded above by a polynomial is good. 342 00:21:53,000 --> 00:21:57,000 This is an old idea. It goes back to one of the main 343 00:21:57,000 --> 00:22:01,000 people who said polynomial is good, Jack Edmonds who is famous 344 00:22:01,000 --> 00:22:06,000 in the combinatorial optimization world. 345 00:22:06,000 --> 00:22:12,000 He is my academic grandfather on one side. 346 00:22:12,000 --> 00:22:18,000 He is a very interesting guy. 347 00:22:32,000 --> 00:22:34,000 OK, so that's a really bad algorithm. 348 00:22:34,000 --> 00:22:37,000 You have probably seen a somewhat better algorithm, 349 00:22:37,000 --> 00:22:41,000 which you might think of as the bottom-up implantation of that 350 00:22:41,000 --> 00:22:43,000 recursive algorithm. 351 00:22:49,000 --> 00:22:51,000 Or, another way to think of it is if you build out the 352 00:22:51,000 --> 00:22:54,000 recursion tree for Fibonacci of n, you will see that there are 353 00:22:54,000 --> 00:22:58,000 lots of common subtrees that you are just wasting time on. 354 00:22:58,000 --> 00:23:01,000 When you solve Fibonacci of n minus 1, it again solves 355 00:23:01,000 --> 00:23:04,000 Fibonacci of n minus 2. Why solve it twice? 356 00:23:04,000 --> 00:23:07,000 You only need to solve it once. So, it is really easy to do 357 00:23:07,000 --> 00:23:11,000 that if you do it bottom-up. But you could also do it 358 00:23:11,000 --> 00:23:14,000 recursively with some cache of things you have already 359 00:23:14,000 --> 00:23:16,000 computed. So, no big surprise. 360 00:23:16,000 --> 00:23:19,000 You compute the Fibonacci numbers in order. 361 00:23:19,000 --> 00:23:22,000 And each time, when you compute Fibonacci of 362 00:23:22,000 --> 00:23:24,000 n, let's say, you have already computed the 363 00:23:24,000 --> 00:23:27,000 previous two, you add them together, 364 00:23:27,000 --> 00:23:32,000 it takes constant time. So, the running time here is 365 00:23:32,000 --> 00:23:36,000 linear, linear in n, and as our input. 366 00:23:36,000 --> 00:23:39,000 Great. Is that the best we can do? 367 00:23:39,000 --> 00:23:42,000 No. Any ideas on how we could 368 00:23:42,000 --> 00:23:47,000 compute Fibonacci of n faster than linear time? 369 00:23:47,000 --> 00:23:52,000 Now we should diverge from what you have already seen, 370 00:23:52,000 --> 00:23:56,000 most of you. Any ideas using techniques you 371 00:23:56,000 --> 00:24:00,000 have already seen? Yes? 372 00:24:00,000 --> 00:24:02,000 Yes. We can use the mathematical 373 00:24:02,000 --> 00:24:05,000 trick of phi and psi to the nth powers. 374 00:24:05,000 --> 00:24:09,000 In fact, you can just use phi, phi, pi, pho, 375 00:24:09,000 --> 00:24:13,000 phum, whatever you want to call this Greek letter. 376 00:24:13,000 --> 00:24:16,000 Good. Here is the mathematical trick. 377 00:24:16,000 --> 00:24:18,000 And, indeed, this is cheating, 378 00:24:18,000 --> 00:24:21,000 as you have said. This is no good, 379 00:24:21,000 --> 00:24:24,000 but so it is. We will call it naÔve recursive 380 00:24:24,000 --> 00:24:30,000 squaring and say well, we know recursive squaring. 381 00:24:30,000 --> 00:24:32,000 Recursive squaring takes log n time. 382 00:24:32,000 --> 00:24:36,000 Let's use recursive squaring. And if you happen to know lots 383 00:24:36,000 --> 00:24:40,000 of properties of the Fibonacci numbers, you don't have to, 384 00:24:40,000 --> 00:24:44,000 but here is one of them. If you take phi to the n 385 00:24:44,000 --> 00:24:48,000 divided by root 5 and you round it to the nearest integer that 386 00:24:48,000 --> 00:24:51,000 is the nth Fibonacci number. 387 00:24:58,000 --> 00:25:01,000 This is pretty cool. Fibonacci of n is basically phi 388 00:25:01,000 --> 00:25:03,000 to the n. We could apply recursive 389 00:25:03,000 --> 00:25:07,000 squaring to compute phi to the n in log n time, 390 00:25:07,000 --> 00:25:11,000 divide by root 5, assume that our computer has an 391 00:25:11,000 --> 00:25:16,000 operation that rounds a number to its nearest integer and poof, 392 00:25:16,000 --> 00:25:19,000 we are done. That doesn't work for many 393 00:25:19,000 --> 00:25:22,000 different reasons. On a real machine, 394 00:25:22,000 --> 00:25:26,000 probably you would represent phi and root 5 as floating point 395 00:25:26,000 --> 00:25:32,000 numbers which have some fixed amount of precise bits. 396 00:25:32,000 --> 00:25:35,000 And if you do this computation, you will lose some of the 397 00:25:35,000 --> 00:25:37,000 important bits. And when you round to the 398 00:25:37,000 --> 00:25:39,000 nearest integer you won't get the right answer. 399 00:25:39,000 --> 00:25:42,000 So, floating point round off will kill you on a floating 400 00:25:42,000 --> 00:25:45,000 point machine. On a theoretical machine where 401 00:25:45,000 --> 00:25:48,000 we magically have numbers that can do crazy things like this, 402 00:25:48,000 --> 00:25:51,000 I mean it really takes more than constant time per 403 00:25:51,000 --> 00:25:53,000 multiplication. So, we are sort of in a 404 00:25:53,000 --> 00:25:56,000 different model. You cannot multiply phi times 405 00:25:56,000 --> 00:25:58,000 phi in constant time. I mean that's sort of outside 406 00:25:58,000 --> 00:26:03,000 the boundaries of this course, but that's the way it is. 407 00:26:03,000 --> 00:26:06,000 In fact, in a normal machine, some problems you can only 408 00:26:06,000 --> 00:26:09,000 solve in exponential time. In a machine where you can 409 00:26:09,000 --> 00:26:13,000 multiply real numbers and round them to the nearest integers, 410 00:26:13,000 --> 00:26:15,000 you can solve them in polynomial time. 411 00:26:15,000 --> 00:26:19,000 So, it really breaks the model. You can do crazy things if you 412 00:26:19,000 --> 00:26:22,000 were allowed to do this. This is not allowed. 413 00:26:22,000 --> 00:26:25,000 And I am foreshadowing like three classes ahead, 414 00:26:25,000 --> 00:26:28,000 or three courses ahead, so I shouldn't talk more about 415 00:26:28,000 --> 00:26:32,000 it. But it turns out we can use 416 00:26:32,000 --> 00:26:35,000 recursive squaring in a different way if we use a 417 00:26:35,000 --> 00:26:38,000 different property of Fibonacci numbers. 418 00:26:38,000 --> 00:26:42,000 And then we will just stick with integers and everything 419 00:26:42,000 --> 00:26:45,000 will be happy. Don't forget to go to 420 00:26:45,000 --> 00:26:48,000 recitation and if you want to homework lab. 421 00:26:48,000 --> 00:26:51,000 Don't come here on Monday. That is required. 422 00:26:51,000 --> 00:26:57,000 This is sort of the right recursive squaring algorithm. 423 00:26:57,000 --> 00:27:02,000 And this is a bit hard to guess if you haven't already seen it, 424 00:27:02,000 --> 00:27:07,000 so I will just give it to you. I will call this a theorem. 425 00:27:07,000 --> 00:27:12,000 It's the first time I get to use the word theorem in this 426 00:27:12,000 --> 00:27:15,000 class. It turns out the nth Fibonacci 427 00:27:15,000 --> 00:27:18,000 number is the nth power of this matrix. 428 00:27:18,000 --> 00:27:21,000 Cool. If you look at it a little bit 429 00:27:21,000 --> 00:27:24,000 you say oh, yeah, of course. 430 00:27:24,000 --> 00:27:29,000 And we will prove this theorem in a second. 431 00:27:29,000 --> 00:27:31,000 But once we have this theorem, we can compute f of n by 432 00:27:31,000 --> 00:27:34,000 computing the nth power of this matrix. 433 00:27:34,000 --> 00:27:36,000 It's a two-by-two matrix. You multiply two two-by-two 434 00:27:36,000 --> 00:27:39,000 matrixes together, you get a two-by-two matrix. 435 00:27:39,000 --> 00:27:41,000 So that is constant size, four numbers. 436 00:27:41,000 --> 00:27:44,000 I can handle four numbers. We don't have crazy precision 437 00:27:44,000 --> 00:27:46,000 problems on the floating point side. 438 00:27:46,000 --> 00:27:48,000 There are only four numbers to deal with. 439 00:27:48,000 --> 00:27:52,000 Matrixes aren't getting bigger. So, the running time of this 440 00:27:52,000 --> 00:27:55,000 divide-and-conquer algorithm will still be log n because it 441 00:27:55,000 --> 00:27:57,000 takes a constant time per two-by-two matrix 442 00:27:57,000 --> 00:28:00,000 multiplication. Yes? 443 00:28:00,000 --> 00:28:02,000 Oh, yes. Thank you. 444 00:28:02,000 --> 00:28:06,000 I have a type error. Sorry about that. 445 00:28:06,000 --> 00:28:12,000 F of n is indeed the upper left corner, I hope. 446 00:28:12,000 --> 00:28:17,000 I better check I don't have it off by one. 447 00:28:17,000 --> 00:28:21,000 I do. It's F_n upper right corner, 448 00:28:21,000 --> 00:28:24,000 indeed. That's what you said. 449 00:28:24,000 --> 00:28:27,000 F of n. I need more space. 450 00:28:27,000 --> 00:28:32,000 Sorry. I really ought to have a 451 00:28:32,000 --> 00:28:36,000 two-by-two matrix on the left-hand side there. 452 00:28:36,000 --> 00:28:40,000 Thank you. So, I compute this nth power of 453 00:28:40,000 --> 00:28:44,000 a matrix in log n time, I take the upper right corner 454 00:28:44,000 --> 00:28:48,000 or the lower left corner, your choice, 455 00:28:48,000 --> 00:28:51,000 that is the nth Fibonacci number. 456 00:28:51,000 --> 00:28:56,000 This implies an order log n time algorithm with the same 457 00:28:56,000 --> 00:29:01,000 recurrence as the last two, binary search and really the 458 00:29:01,000 --> 00:29:07,000 recursive squaring algorithm. It is log n plus a constant, 459 00:29:07,000 --> 00:29:10,000 so log n. Let's prove that theorem. 460 00:29:30,000 --> 00:29:34,000 Any suggestions on what techniques we might use for 461 00:29:34,000 --> 00:29:37,000 proving this theorem, or what technique, 462 00:29:37,000 --> 00:29:40,000 singular? Induction, very good. 463 00:29:40,000 --> 00:29:44,000 I think any time I ask that question the answer is 464 00:29:44,000 --> 00:29:47,000 induction. Hint for the future in this 465 00:29:47,000 --> 00:29:49,000 class. 466 00:29:54,000 --> 00:29:56,000 A friend of mine, when he took an analysis class, 467 00:29:56,000 --> 00:29:59,000 whenever the professor asked, and what is the answer to this 468 00:29:59,000 --> 00:30:01,000 question, the answer was always zero. 469 00:30:01,000 --> 00:30:04,000 If you have taken analysis class that is funny. 470 00:30:04,000 --> 00:30:10,000 [LAUGHTER] Maybe I will try to ask some questions whose answers 471 00:30:10,000 --> 00:30:13,000 are zero just for our own amusement. 472 00:30:13,000 --> 00:30:17,000 We are going to induct on n. It's pretty much the obvious 473 00:30:17,000 --> 00:30:20,000 thing to do. But we have to check some 474 00:30:20,000 --> 00:30:23,000 cases. So, the base case is we have 475 00:30:23,000 --> 00:30:27,000 this to the first power. And that is itself [(1, 476 00:30:27,000 --> 00:30:33,000 1), (1, 0)]. And I should have said n is at 477 00:30:33,000 --> 00:30:36,000 least 1. And you can check. 478 00:30:36,000 --> 00:30:40,000 This is supposed to be F_2, F_1, F_1, F_0. 479 00:30:40,000 --> 00:30:46,000 And you can check it is, F_0 is 0, F_1 is 1 and F_2 is 480 00:30:46,000 --> 00:00:01,000 Base case is correct, step case is about as exciting, 481 00:00:01,000 --> 00:30:46,000 Good. 482 00:30:52,000 --> 00:30:59,000 but you've got to prove that your algorithm works. 483 00:30:59,000 --> 00:31:03,000 Suppose this is what we want to compute. 484 00:31:03,000 --> 00:31:10,000 I am just going to sort of, well, there are many ways I can 485 00:31:10,000 --> 00:31:14,000 do this. I will just do it the fast way 486 00:31:14,000 --> 00:31:18,000 because it's really not that exciting. 487 00:31:18,000 --> 00:31:23,000 Which direction? Let's do this direction. 488 00:31:23,000 --> 00:31:29,000 I want to use induction on n. If I want to use induction on 489 00:31:29,000 --> 00:31:33,000 n, presumably I should use what I already know is true. 490 00:31:33,000 --> 00:31:37,000 If I decrease n by 1, I have this property that this 491 00:31:37,000 --> 00:31:40,000 thing is going to be [(1, 1), (1, 0)] to the power n 492 00:31:40,000 --> 00:31:42,000 minus 1. This I already know, 493 00:31:42,000 --> 00:31:45,000 by the induction hypothesis, [(1, 1), (1, 494 00:31:45,000 --> 00:31:48,000 0)] to the n minus 1. So, presumably I should use it 495 00:31:48,000 --> 00:31:51,000 in some way. This equality is not yet true, 496 00:31:51,000 --> 00:31:54,000 you may have noticed. So, I need to add something on. 497 00:31:54,000 --> 00:31:59,000 What could I possibly add on to be correct? 498 00:31:59,000 --> 00:32:01,000 Well, another factor of [(1, 1), (1, 0)]. 499 00:32:01,000 --> 00:32:05,000 The way I am developing this proof is the only way it could 500 00:32:05,000 --> 00:32:08,000 possibly be, in some sense. If you know its induction, 501 00:32:08,000 --> 00:32:12,000 this is all that you could do. And then you check. 502 00:32:12,000 --> 00:32:14,000 Indeed, this equality holds conveniently. 503 00:32:14,000 --> 00:32:17,000 For example, F_(n plus 1) is the product of 504 00:32:17,000 --> 00:32:20,000 these two things. It is this row times this 505 00:32:20,000 --> 00:32:22,000 column. So, it is F_n times 1 plus F_(n 506 00:32:22,000 --> 00:32:25,000 minus 1) times 1, which is indeed the definition 507 00:32:25,000 --> 00:32:30,000 of F_(n plus 1). And you could check four of the 508 00:32:30,000 --> 00:32:32,000 entries. This is true. 509 00:32:32,000 --> 00:32:36,000 Great. If that is true then I would 510 00:32:36,000 --> 00:32:39,000 just put these together. That is [(1, 511 00:32:39,000 --> 00:32:43,000 1), (1, 0)] to the n minus 1 times [(1, 1), 512 00:32:43,000 --> 00:32:47,000 (1, 0)], which is [(1, 1), (1, 0)] to the n, 513 00:32:47,000 --> 00:32:50,000 end of proof. A very simple proof, 514 00:32:50,000 --> 00:32:55,000 but you have to do that in order to know if this algorithm 515 00:32:55,000 --> 00:32:57,000 really works. Good. 516 00:32:57,000 --> 00:33:00,000 Question? Oh, yes. 517 00:33:00,000 --> 00:33:03,000 Thank you. This, in the lower right, 518 00:33:03,000 --> 00:33:08,000 we should have F_(n minus 1). This is why you should really 519 00:33:08,000 --> 00:33:12,000 check your proofs. We would have discovered that 520 00:33:12,000 --> 00:33:16,000 when I checked that this was that row times that column, 521 00:33:16,000 --> 00:33:20,000 but that is why you are here, to fix my bugs. 522 00:33:20,000 --> 00:33:25,000 That's the great thing about being up here instead of in a 523 00:33:25,000 --> 00:33:29,000 quiz. But that is a minor mistake. 524 00:33:29,000 --> 00:33:31,000 You wouldn't lose much for that. 525 00:33:31,000 --> 00:33:34,000 All right. More divide-and-conquer 526 00:33:34,000 --> 00:33:37,000 algorithms. Still, we have done relatively 527 00:33:37,000 --> 00:33:41,000 simple ones so far. In fact, the fanciest has been 528 00:33:41,000 --> 00:33:44,000 merge sort, which we already saw. 529 00:33:44,000 --> 00:33:48,000 So, that is not too exciting. The rest have all be log n 530 00:33:48,000 --> 00:33:51,000 time. Let's break out of the log n 531 00:33:51,000 --> 00:33:54,000 world. Well, you all have the master 532 00:33:54,000 --> 00:33:58,000 method memorized, right, so I can erase that. 533 00:33:58,000 --> 00:34:01,000 Good. This will be a good test. 534 00:34:01,000 --> 00:34:04,000 Next problem is matrix multiplication, 535 00:34:04,000 --> 00:34:08,000 following right up on this two-by-two matrix 536 00:34:08,000 --> 00:34:11,000 multiplication. Let's see how we can compute 537 00:34:11,000 --> 00:34:15,000 n-by-n matrix multiplications. Just for a recap, 538 00:34:15,000 --> 00:34:19,000 you should know how to multiply matrixes, but here is the 539 00:34:19,000 --> 00:34:22,000 definition so we can turn it into an algorithm. 540 00:34:22,000 --> 00:34:26,000 You have two matrixes, A and B, which are capital 541 00:34:26,000 --> 00:34:28,000 levels. The ijth entry. 542 00:34:28,000 --> 00:34:34,000 Ith row, jth column is called little a_ij or little b_ij. 543 00:34:34,000 --> 00:34:40,000 And your goal is to compute the products of those matrixes. 544 00:34:40,000 --> 00:34:45,000 I should probably say that i and j range from 1 to n. 545 00:34:45,000 --> 00:34:51,000 So, they are square matrixes. The output is to compute C 546 00:34:51,000 --> 00:34:57,000 which has entry c_ij which is the product of A and B. 547 00:34:57,000 --> 00:35:01,000 And, for a recap, the ijth entry of the product 548 00:35:01,000 --> 00:35:08,000 is the inner product of the ith row of A with the jth column of 549 00:35:08,000 --> 00:35:13,000 B. But you can write that out as a 550 00:35:13,000 --> 00:35:17,000 sum like so. We want to compute this thing 551 00:35:17,000 --> 00:35:22,000 for every i and j. What is the obvious algorithm 552 00:35:22,000 --> 00:35:27,000 for doing this? Well, for every i and j you 553 00:35:27,000 --> 00:35:32,000 compute the sum. You compute all the products. 554 00:35:32,000 --> 00:35:35,000 You compute the sum. So, it's like n operations here 555 00:35:35,000 --> 00:35:37,000 roughly. I mean like 2n minus 1, 556 00:35:37,000 --> 00:35:40,000 whatever. It is order n operations. 557 00:35:40,000 --> 00:35:43,000 There are n^2 entries of C that I need to compute, 558 00:35:43,000 --> 00:35:47,000 so that's n^3 time. I will write this out just for 559 00:35:47,000 --> 00:35:50,000 the programmers at heart. Here is the pseudocode. 560 00:35:50,000 --> 00:35:53,000 It's rare that I will write pseudocode. 561 00:35:53,000 --> 00:35:57,000 And this is a simple enough algorithm that I can write it in 562 00:35:57,000 --> 00:36:03,000 gory detail. But it gives you some basis for 563 00:36:03,000 --> 00:36:06,000 this analysis if you like to program. 564 00:36:06,000 --> 00:36:10,000 It is a triply nested for loop. 565 00:36:17,000 --> 00:36:23,000 And I made a coding error. Hopefully you haven't written 566 00:36:23,000 --> 00:36:27,000 that far yet. I need c_ij to be initialized 567 00:36:27,000 --> 00:36:31,000 to zero. And then I add to c_ij the 568 00:36:31,000 --> 00:36:36,000 appropriate product, a_ik, b_kj. 569 00:36:36,000 --> 00:36:39,000 That is the algorithm. And the point is you have a 570 00:36:39,000 --> 00:36:42,000 nesting of n for loops from 1 to n. 571 00:36:42,000 --> 00:36:46,000 That takes n^3 time because this is constant and that is 572 00:36:46,000 --> 00:36:48,000 constant. So, very simple, 573 00:36:48,000 --> 00:36:51,000 n^3. Let's do better. 574 00:36:57,000 --> 00:36:58,000 And, of course, we are going to use 575 00:36:58,000 --> 00:36:59,000 divide-and-conquer. 576 00:37:04,000 --> 00:37:06,000 Now, how are we going to divide a matrix? 577 00:37:06,000 --> 00:37:10,000 There are a lot of numbers in a matrix, n^2 of them in each one. 578 00:37:10,000 --> 00:37:12,000 There are all sorts of ways you could divide. 579 00:37:12,000 --> 00:37:15,000 So far all of the divide-and-conquers we have done 580 00:37:15,000 --> 00:37:19,000 have been problems of size n into some number of problems of 581 00:37:19,000 --> 00:37:21,000 size n over 2. I am going to say I start with 582 00:37:21,000 --> 00:37:25,000 some matrixes of size n-by-n. I want to convert it down to 583 00:37:25,000 --> 00:37:28,000 something like n/2-by-n/2. Any suggestions how I might do 584 00:37:28,000 --> 00:37:30,000 that? Yeah? 585 00:37:30,000 --> 00:37:33,000 Block form the matrix, indeed. 586 00:37:33,000 --> 00:37:38,000 That is the right answer. So, this is the first 587 00:37:38,000 --> 00:37:43,000 divide-and-conquer algorithm. This will not work, 588 00:37:43,000 --> 00:37:47,000 but it has the first idea that we need. 589 00:37:47,000 --> 00:37:51,000 We have a n-by-n matrix. We can view it, 590 00:37:51,000 --> 00:37:55,000 this equality is more, you can think of it as, 591 00:37:55,000 --> 00:38:01,000 it's really the thing, a two-by-two block matrix where 592 00:38:01,000 --> 00:38:06,000 each entry in this two-by-two block matrix is a block of 593 00:38:06,000 --> 00:38:11,000 n/2-by-n/2 submatrixes. 594 00:38:20,000 --> 00:38:27,000 I will think of C as being divided into three parts, 595 00:38:27,000 --> 00:38:32,000 r, s, t and u. Even though I write them as 596 00:38:32,000 --> 00:38:37,000 lower case letters they are really matrixes. 597 00:38:37,000 --> 00:38:42,000 Each is n/2-by-n/2. And A, I will split into a, 598 00:38:42,000 --> 00:38:46,000 b, c, d, times B, I will split into e, 599 00:38:46,000 --> 00:38:48,000 f, g, h. Why not? 600 00:38:48,000 --> 00:38:53,000 This is certainly true. And if you've seen some linear 601 00:38:53,000 --> 00:39:01,000 algebra, this is basically what you can do with matrixes. 602 00:39:01,000 --> 00:39:05,000 Now I can pretend these are two-by-two and sort of forget 603 00:39:05,000 --> 00:39:09,000 the fact that these little letters are matrixes and say 604 00:39:09,000 --> 00:39:14,000 well, r is the inner product of this row with this column. 605 00:39:14,000 --> 00:39:17,000 It is ae times bg. Let me not cheat or else it 606 00:39:17,000 --> 00:39:23,000 will be too easy. r=ae+bg, s=af+bh, 607 00:39:23,000 --> 00:39:34,000 t=ce+dh and u=cf+dg. It's nothing like making it too 608 00:39:34,000 --> 00:39:41,000 hard on yourself. OK, got them right. 609 00:39:41,000 --> 00:39:48,000 Good. I mean this is just a fact 610 00:39:48,000 --> 00:39:59,000 about how you would expand out this product. 611 00:39:59,000 --> 00:40:01,000 And so now we have a recursive algorithm. 612 00:40:01,000 --> 00:40:04,000 In fact, we have a divide-and-conquer algorithm. 613 00:40:04,000 --> 00:40:07,000 We start with our n-by-n matrix. 614 00:40:07,000 --> 00:40:09,000 Well, we have two of them actually. 615 00:40:09,000 --> 00:40:12,000 We divide it into eight little pieces, a, b, 616 00:40:12,000 --> 00:40:13,000 c, d, e, f, g, h. 617 00:40:13,000 --> 00:40:16,000 Then we compute these things and that gives us C, 618 00:40:16,000 --> 00:40:20,000 just by sticking them together. Now, how do we compute these 619 00:40:20,000 --> 00:40:23,000 things? Well, these innocent-looking 620 00:40:23,000 --> 00:40:27,000 little products between these two little numbers are actually 621 00:40:27,000 --> 00:40:31,000 recursive matrix multiplications. 622 00:40:31,000 --> 00:40:37,000 Because each of these little letters is an n/2-by-n/2 matrix 623 00:40:37,000 --> 00:40:42,000 so I have to recursively compute the product. 624 00:40:42,000 --> 00:40:49,000 There are like eight recursive multiplications of n/2-by-n/2 625 00:40:49,000 --> 00:40:52,000 matrixes. That is what bites us. 626 00:40:52,000 --> 00:40:59,000 And then there are like four additions, plus minor work of 627 00:40:59,000 --> 00:41:05,000 gluing things together. How long does it take to add 628 00:41:05,000 --> 00:41:06,000 two matrixes together? n^2. 629 00:41:06,000 --> 00:41:09,000 This is cheap. It just takes n^2. 630 00:41:09,000 --> 00:41:12,000 Remember, we are trying to beat n^3 for our matrix 631 00:41:12,000 --> 00:41:15,000 multiplication. Addition is a really easy 632 00:41:15,000 --> 00:41:17,000 problem. You just have to add every 633 00:41:17,000 --> 00:41:20,000 number. There is no way you can do 634 00:41:20,000 --> 00:41:23,000 better than n^2. So, that is not recursive. 635 00:41:23,000 --> 00:41:26,000 That is the nice thing. But the bad thing is we have 636 00:41:26,000 --> 00:41:31,000 eight of these recursions. We have 637 00:41:31,000 --> 00:41:39,000 T(n)=8T(n/2)+Theta(n^2). And I have erased the master 638 00:41:39,000 --> 00:41:46,000 method, but you should all have it memorized. 639 00:41:46,000 --> 00:41:52,000 What is the solution to this recurrence? 640 00:41:52,000 --> 00:41:56,000 Theta(n^3). That is annoying. 641 00:41:56,000 --> 00:42:01,000 All right. A is 8, b is 2, 642 00:42:01,000 --> 00:42:07,000 log base 2 of 8 is 3. Every computer scientist should 643 00:42:07,000 --> 00:42:09,000 know that. n^log_b(a)=n^3. 644 00:42:09,000 --> 00:42:15,000 That is polynomially larger than n^2, so we are in Case 1. 645 00:42:15,000 --> 00:42:19,000 Thank you. Let's get them upside down. 646 00:42:19,000 --> 00:42:23,000 This is n^3, no better than our previous 647 00:42:23,000 --> 00:42:26,000 algorithm. That kind of sucks. 648 00:42:26,000 --> 00:42:30,000 And now comes the divine inspiration. 649 00:42:30,000 --> 00:42:34,000 Let's go over here. 650 00:42:41,000 --> 00:42:44,000 There are some algorithms like this Fibonacci algorithm where 651 00:42:44,000 --> 00:42:47,000 if you sat down for a little while, it's no big deal, 652 00:42:47,000 --> 00:42:51,000 you would figure it out. I mean it is kind of clever to 653 00:42:51,000 --> 00:42:54,000 look at that matrix and then everything works happily. 654 00:42:54,000 --> 00:42:57,000 It is not obvious but it is not that amazingly clever. 655 00:42:57,000 --> 00:43:01,000 This is an algorithm that is amazingly clever. 656 00:43:01,000 --> 00:43:05,000 You may have seen it before which steals the thunder a 657 00:43:05,000 --> 00:43:10,000 little bit, but it is still really, really cool so you 658 00:43:10,000 --> 00:43:15,000 should be happy to see it again. And how Strassen came up with 659 00:43:15,000 --> 00:43:19,000 this algorithm, he must have been very clever. 660 00:43:19,000 --> 00:43:24,000 The idea is we've got to get rid of these multiplications. 661 00:43:24,000 --> 00:43:30,000 I could do a hundred additions. That only costs Theta(n^2). 662 00:43:30,000 --> 00:43:33,000 I have to reduce this 8 to something smaller. 663 00:43:33,000 --> 00:43:35,000 It turns out, if you try to split the 664 00:43:35,000 --> 00:43:39,000 matrices into three-by-three or something, that doesn't help 665 00:43:39,000 --> 00:43:41,000 you. You get the same problem 666 00:43:41,000 --> 00:43:44,000 because we're using fundamentally the same 667 00:43:44,000 --> 00:43:47,000 algorithm, just in a different order. 668 00:43:47,000 --> 00:43:51,000 We have got to somehow reduce the number of multiplications. 669 00:43:51,000 --> 00:43:55,000 We are going to reduce it to 7. The claim is that if we have 670 00:43:55,000 --> 00:43:59,000 two two-by-two matrices we can take their product using seven 671 00:43:59,000 --> 00:44:03,000 multiplications. If that were true, 672 00:44:03,000 --> 00:44:08,000 we would reduce the 8 to a 7 and presumably make things run 673 00:44:08,000 --> 00:44:11,000 faster. We will see how fast in a 674 00:44:11,000 --> 00:44:14,000 moment. You can compute it in your 675 00:44:14,000 --> 00:44:16,000 head. If you are bored and like 676 00:44:16,000 --> 00:44:21,000 computing logs that are non-integral logs then go ahead. 677 00:44:21,000 --> 00:44:23,000 All right. Here we are. 678 00:44:23,000 --> 00:44:28,000 This algorithm is unfortunately rather long, but it is only 679 00:44:28,000 --> 00:44:32,000 seven multiplications. 680 00:44:45,000 --> 00:44:49,000 Each of these P's is a product of two things which only 681 00:44:49,000 --> 00:44:54,000 involves addition or subtraction, the same thing. 682 00:45:38,000 --> 00:45:41,000 Those are seven multiplications. 683 00:45:41,000 --> 00:45:44,000 And I can compute those in 7T(n/2). 684 00:45:44,000 --> 00:45:47,000 Oh, indeed it is. Six was wrong. 685 00:45:47,000 --> 00:45:51,000 Six and seven are the same, very good. 686 00:45:51,000 --> 00:45:57,000 You know, you think that copying something would not be 687 00:45:57,000 --> 00:46:02,000 such a challenging task. But when you become an 688 00:46:02,000 --> 00:46:07,000 absent-minded professor like me then you will know how easy it 689 00:46:07,000 --> 00:46:07,000 is. OK. We have them all correct, hopefully. 690 00:46:10,000 --> 00:46:12,000 We continue. That wasn't enough. 691 00:46:12,000 --> 00:46:16,000 Of course we had seven things. Clearly we have to reduce this 692 00:46:16,000 --> 00:46:19,000 down to four things, the elements of C. 693 00:46:19,000 --> 00:46:22,000 Here they are, the elements of C, 694 00:46:22,000 --> 00:46:25,000 r, s, t, u. It turns out r=P_5+P_4-P_2+P_6. 695 00:46:25,000 --> 00:46:27,000 Of course. Didn't you all see that? 696 00:46:27,000 --> 00:46:33,000 [LAUGHTER] I mean this one is really easy, 697 00:46:33,000 --> 00:46:36,000 s=P_1+P2. t=P_3+P_4. 698 00:46:36,000 --> 00:46:41,000 I mean that is clearly how they were chosen. 699 00:46:41,000 --> 00:46:48,000 And then u is another tricky one, u=P_5+P_1-P_3-P_7. 700 00:46:48,000 --> 00:46:52,000 OK. Now, which one of these would 701 00:46:52,000 --> 00:46:57,000 you like me to check? Don't be so nice. 702 00:46:57,000 --> 00:47:03,000 How about s? I can show you s is right. 703 00:47:03,000 --> 00:47:06,000 Any preferences? u. 704 00:47:06,000 --> 00:47:09,000 Oh, no, sign errors. OK. 705 00:47:09,000 --> 00:47:14,000 Here we go. The claim that this really 706 00:47:14,000 --> 00:47:21,000 works is you have to check all four of them. 707 00:47:21,000 --> 00:47:25,000 And I did in my notes. u=P_5. 708 00:47:25,000 --> 00:47:31,000 P_5=(ae + ah + de + dh). That is P_5. 709 00:47:31,000 --> 00:47:35,000 Check me. If I screw up, 710 00:47:35,000 --> 00:47:40,000 I am really hosed. (af - ah) = P_1. 711 00:47:40,000 --> 00:47:48,000 P_3 has a minus sign in front, so that is (ce + de). 712 00:47:48,000 --> 00:47:55,000 And then we have minus P_7, which is a big one, 713 00:47:55,000 --> 00:48:00,000 (ae + af - ce - cf). OK. 714 00:48:00,000 --> 00:48:12,000 Now I need like the assistant that crosses off things in 715 00:48:12,000 --> 00:48:19,000 parallel like the movie, right? 716 00:48:19,000 --> 00:48:25,000 ah, de, af, ce, ae, thank you, 717 00:48:25,000 --> 00:48:37,000 and hopefully these survive, dh minus minus cf. 718 00:48:37,000 --> 00:48:40,000 And, if we are lucky, that is exactly what is written 719 00:48:40,000 --> 00:48:42,000 here, except in the opposite order. 720 00:48:42,000 --> 00:48:45,000 Magic, right? Where the hell did Strassen get 721 00:48:45,000 --> 00:48:47,000 this? You have to be careful. 722 00:48:47,000 --> 00:48:51,000 It is OK that the plus is in the wrong order because plus is 723 00:48:51,000 --> 00:48:54,000 commutative, but the multiplications better not be in 724 00:48:54,000 --> 00:48:58,000 the wrong order because multiplication over matrixes is 725 00:48:58,000 --> 00:49:01,000 not commutative. I check cf, OK, 726 00:49:01,000 --> 00:49:05,000 dh, they are in the right order. 727 00:49:05,000 --> 00:49:11,000 I won't check the other three. That is matrix multiplication 728 00:49:11,000 --> 00:49:17,000 in hopefully subcubic time. Let's write down the 729 00:49:17,000 --> 00:49:19,000 recurrence. T(n) is now 7. 730 00:49:19,000 --> 00:49:25,000 Maybe I should write down the algorithm for kicks. 731 00:49:25,000 --> 00:49:30,000 Why not? Assuming I have time. 732 00:49:30,000 --> 00:49:33,000 Lots of time. Last lecture I was ten minutes 733 00:49:33,000 --> 00:49:36,000 early. I ended ten minutes early. 734 00:49:36,000 --> 00:49:40,000 I apologize for that. I know it really upsets you. 735 00:49:40,000 --> 00:49:46,000 And I didn't realize exactly when the class was supposed to 736 00:49:46,000 --> 00:49:48,000 end. So, today, I get to go ten 737 00:49:48,000 --> 00:49:50,000 minutes late. OK. 738 00:49:50,000 --> 00:49:52,000 Good. I'm glad you all agree. 739 00:49:52,000 --> 00:49:55,000 [LAUGHTER] I am kidding. Don't worry. 740 00:49:55,000 --> 00:49:56,000 OK. Algorithm. 741 00:49:56,000 --> 00:50:02,000 This is Strassen. First we divide, 742 00:50:02,000 --> 00:50:07,000 then we conquer and then we combine. 743 00:50:07,000 --> 00:50:14,000 As usual, I don't have it written anywhere here. 744 00:50:14,000 --> 00:50:17,000 Fine. Divide A and B. 745 00:50:17,000 --> 00:50:27,000 This is sort of trivial. Then we compute the terms -- 746 00:50:36,000 --> 00:50:39,000 -- for the products. This means we get ready to 747 00:50:39,000 --> 00:50:42,000 compute all the P's. We compute a+b, 748 00:50:42,000 --> 00:50:45,000 c+d, g-e, a+d, e+h and so on. 749 00:50:45,000 --> 00:50:49,000 All of the terms that appear in here, we compute those. 750 00:50:49,000 --> 00:50:54,000 That takes n2 time because it is just a bunch of additions and 751 00:50:54,000 --> 00:50:56,000 subtractions. No big deal. 752 00:50:56,000 --> 00:51:04,000 A constant number of them. Then we conquer by recursively 753 00:51:04,000 --> 00:51:12,000 computing all the P_i's. That is each our product of 754 00:51:12,000 --> 00:51:17,000 seven of them. We have P_1, 755 00:51:17,000 --> 00:51:21,000 P_2 up to P_7. And, finally, 756 00:51:21,000 --> 00:51:30,000 we combine, which is to compute r, s, t and u. 757 00:51:36,000 --> 00:51:40,000 And those are just additions and subtractions again, 758 00:51:40,000 --> 00:51:43,000 so they take n^2 times. So, here we finally get an 759 00:51:43,000 --> 00:51:47,000 algorithm that is nontrivial both in dividing and in 760 00:51:47,000 --> 00:51:51,000 combining. Recursion is always recursion, 761 00:51:51,000 --> 00:51:54,000 but now we have interesting steps one and three. 762 00:51:54,000 --> 00:51:59,000 The recurrence is T(n) is seven recursive subproblems, 763 00:51:59,000 --> 00:52:03,000 each are size n/2 plus order n^2, to do all this addition 764 00:52:03,000 --> 00:52:07,000 work. Now we need to solve this 765 00:52:07,000 --> 00:52:11,000 recurrence. We compute n^log_b(a), 766 00:52:11,000 --> 00:52:16,000 which here is nlog_2(7). And we know log base 2 of 8 is 767 00:00:03,000 --> 00:52:20,000 Log base 2 of 7 is going be a 768 00:52:20,000 --> 00:52:26,000 little bit less than 3 but still bigger than 2 because log base 2 769 00:52:26,000 --> 00:52:30,000 of 4 is 2. So, it is going to be 770 00:52:30,000 --> 00:52:36,000 polynomially larger than n^2 but polynomially smaller than n^3. 771 00:52:36,000 --> 00:52:41,000 We are again in Case 1. And this is the cheating way to 772 00:52:41,000 --> 00:52:44,000 write n log base 2 of 7, nlg7. 773 00:52:44,000 --> 00:52:48,000 lg means log base 2. You should know that. 774 00:52:48,000 --> 00:52:53,000 It is all over the textbook and in our problem sets and what 775 00:52:53,000 --> 00:52:56,000 not, nlg7. And, in particular, 776 00:52:56,000 --> 00:53:02,000 if I have my calculator here. This is a good old-fashion 777 00:53:02,000 --> 00:53:04,000 calculator. No, that is wrong. 778 00:53:04,000 --> 00:53:07,000 Sorry. It is strictly less than 2.81. 779 00:53:07,000 --> 00:53:11,000 It is strictly less than 2.81. That is cool. 780 00:53:11,000 --> 00:53:14,000 I mean it is polynomially better than n^3. 781 00:53:14,000 --> 00:53:18,000 Still not as good as addition, which is n^2. 782 00:53:18,000 --> 00:53:23,000 It is generally believed, although we don't know whether 783 00:53:23,000 --> 00:53:29,000 you can multiply as fast as you can divide for matrices. 784 00:53:29,000 --> 00:53:32,000 We think you cannot get n^2, but who knows? 785 00:53:32,000 --> 00:53:36,000 It could still happen. There are no lower bounds. 786 00:53:36,000 --> 00:53:40,000 This is not the best algorithm for matrix multiplication. 787 00:53:40,000 --> 00:53:44,000 It is sort of the simplest that beats n^3. 788 00:53:44,000 --> 00:53:46,000 The best so far is like n^2.376. 789 00:53:46,000 --> 00:53:50,000 Getting closer to 2. You might think these numbers 790 00:53:50,000 --> 00:53:54,000 are a bit weird. Maybe the constants out here 791 00:53:54,000 --> 00:54:00,000 dominate the improvement you are getting in the exponent. 792 00:54:00,000 --> 00:54:03,000 It turns out improving the exponent is a big deal. 793 00:54:03,000 --> 00:54:06,000 I mean, as n gets larger exponents really come out to 794 00:54:06,000 --> 00:54:09,000 bite you. So, n^3 is pretty impractical 795 00:54:09,000 --> 00:54:13,000 for any very large values of n. And we known that Strassen will 796 00:54:13,000 --> 00:54:16,000 beat normal matrix multiplication if n is 797 00:54:16,000 --> 00:54:19,000 sufficiently large. The claim is that roughly at 798 00:54:19,000 --> 00:54:22,000 about 32 or so already you get an improvement, 799 00:54:22,000 --> 00:54:25,000 for other reasons, not just because the exponent 800 00:54:25,000 --> 00:54:30,000 gets better, but there you go. So, this is pretty good. 801 00:54:30,000 --> 00:54:33,000 This is completely impractical, so don't use whatever this 802 00:54:33,000 --> 00:54:36,000 algorithm is. I don't have the reference 803 00:54:36,000 --> 00:54:40,000 handy, but it is just trying to get a theoretical improvement. 804 00:54:40,000 --> 00:54:44,000 There may be others that are in between and more reasonable but 805 00:54:44,000 --> 00:54:46,000 that is not it. Wow, lots of time. 806 00:54:46,000 --> 00:54:49,000 Any questions? We're not done yet, 807 00:54:49,000 --> 00:54:52,000 but any questions before we move on for matrix 808 00:54:52,000 --> 00:54:53,000 multiplication? OK. 809 00:54:53,000 --> 00:54:56,000 I have one more problem. 810 00:55:15,000 --> 00:55:17,000 Divide-and-conquer is a pretty general idea. 811 00:55:17,000 --> 00:55:20,000 I mean, you can use it to dominate countries. 812 00:55:20,000 --> 00:55:22,000 You can use it to multiply matrices. 813 00:55:22,000 --> 00:55:26,000 I mean, who would have thought? Here is a very different kind 814 00:55:26,000 --> 00:55:30,000 of problem you can solve with divide-and-conquer. 815 00:55:30,000 --> 00:55:32,000 It is not exactly an algorithmic problem, 816 00:55:32,000 --> 00:55:35,000 although it is computer science. 817 00:55:35,000 --> 00:55:37,000 That is clear. This is very large-scale 818 00:55:37,000 --> 00:55:40,000 integration. The chips, they are very large 819 00:55:40,000 --> 00:55:44,000 scale integrated. Probably even more these days, 820 00:55:44,000 --> 00:55:47,000 but that is the catch phrase. Here is a problem, 821 00:55:47,000 --> 00:55:51,000 and it arises in VLSI layout. We won't get into too many 822 00:55:51,000 --> 00:55:54,000 details why, but you have some circuit. 823 00:55:54,000 --> 00:55:58,000 And here I am going to assume that the circuit is a binary 824 00:55:58,000 --> 00:56:02,000 tree. This is just part of a circuit. 825 00:56:02,000 --> 00:56:07,000 Assume for now here that it is just a complete binary tree. 826 00:56:07,000 --> 00:56:11,000 A complete binary tree looks like this. 827 00:56:11,000 --> 00:56:15,000 In all of my teachings, I have drawn this figure for 828 00:56:15,000 --> 00:56:18,000 sure the most. It is my favorite figure, 829 00:56:18,000 --> 00:56:21,000 the height four complete binary tree. 830 00:56:21,000 --> 00:56:25,000 OK, there it is. I have some tree like that as 831 00:56:25,000 --> 00:56:30,000 some height. I want to imbed it into some 832 00:56:30,000 --> 00:56:36,000 chip layout on a grid. Let's say it has n leaves. 833 00:56:36,000 --> 00:56:41,000 I want to imbed it into a grid with minimum area. 834 00:56:41,000 --> 00:56:47,000 This is a very cute problem and it really shows you another way 835 00:56:47,000 --> 00:56:53,000 in which divide-and-conquer is a useful and powerful tool. 836 00:56:53,000 --> 00:57:00,000 So, I have this tree. I like to draw it in this way. 837 00:57:00,000 --> 00:57:02,000 I want to somehow draw it on the grid. 838 00:57:02,000 --> 00:57:06,000 What that means is the vertices have to be imbedded onto dots on 839 00:57:06,000 --> 00:57:09,000 the grid, and I am talking about the square grid. 840 00:57:09,000 --> 00:57:12,000 It has to go to vertices of the grid. 841 00:57:12,000 --> 00:57:15,000 And these edges have to be routed as sort of orthogonal 842 00:57:15,000 --> 00:57:19,000 paths between one dot and another, so that should be an 843 00:57:19,000 --> 00:57:22,000 edge and they shouldn't cross and all these good things 844 00:57:22,000 --> 00:57:25,000 because wires do not like to cross. 845 00:57:25,000 --> 00:57:28,000 There is the obvious way to solve this problem and there is 846 00:57:28,000 --> 00:57:31,000 the right way. 847 00:57:36,000 --> 00:57:37,000 And let's talk about both of them. 848 00:57:37,000 --> 00:57:41,000 Neither of them is particularly obvious, but divide-and-conquer 849 00:57:41,000 --> 00:57:44,000 sort of gives you a hint in the right direction. 850 00:57:44,000 --> 00:57:47,000 So, the naÔve imbedding. I seem to like the word naÔve 851 00:57:47,000 --> 00:57:48,000 here. 852 00:57:54,000 --> 00:57:58,000 I am going to draw this bottom up because it is easier, 853 00:57:58,000 --> 00:58:02,000 so leave three grid lines and then start drawing. 854 00:58:02,000 --> 00:58:05,000 I don't know how big that is going to be. 855 00:58:05,000 --> 00:58:11,000 Here is the bottom of our tree. This is like the little three 856 00:58:11,000 --> 00:58:15,000 nodes there. And then I leave a blank column 857 00:58:15,000 --> 00:58:21,000 and then a blank column. I don't actually need to leave 858 00:58:21,000 --> 00:58:25,000 those blank columns, but it makes a prettier 859 00:58:25,000 --> 00:58:30,000 drawing. And then we work our way up. 860 00:58:37,000 --> 00:58:40,000 There is the tree, which should be aligned, 861 00:58:40,000 --> 00:58:42,000 on a grid. No crossings. 862 00:58:42,000 --> 00:58:45,000 Everything is happy. How much area does it take? 863 00:58:45,000 --> 00:58:49,000 By area, I mean sort of the area of the bounding box. 864 00:58:49,000 --> 00:58:54,000 So, I count this blank space even though I am not using it 865 00:58:54,000 --> 00:59:00,000 and I count all this blank space even though I am not using it. 866 00:59:00,000 --> 00:59:04,000 I want to look at the height. Let's call this H(n). 867 00:59:04,000 --> 00:59:08,000 And to look at the width, which I will call W(n). 868 00:59:08,000 --> 00:59:13,000 Now, it is probably pretty obvious that H(n) is like log n, 869 00:59:13,000 --> 00:59:18,000 W(n) is like n or whatever. But I want to write it as a 870 00:59:18,000 --> 00:59:22,000 recurrence because that will inspire us to do the right 871 00:59:22,000 --> 00:59:23,000 thing. H(n). 872 00:59:23,000 --> 00:59:27,000 Well, if you think of this as a recursion-tree, 873 00:59:27,000 --> 00:59:31,000 in some sense. We start with the big tree. 874 00:59:31,000 --> 00:59:35,000 We split it into two halves, two subtrees of size n/2 indeed 875 00:59:35,000 --> 00:59:40,000 because we are counting leaves. It is exactly n/2 on each side. 876 00:59:40,000 --> 00:59:43,000 Then for height they are in parallel so it is no big deal. 877 00:59:43,000 --> 00:59:46,000 The height is just the height of this thing, 878 00:59:46,000 --> 00:59:48,000 one of these subproblems plus one. 879 00:59:48,000 --> 00:59:52,000 The width, you have to add together the two widths and also 880 00:59:52,000 --> 00:59:55,000 add on 1. You don't have to add on 1 881 00:59:55,000 --> 01:00:00,000 here, but it doesn't matter. It is certainly at most 1. 882 01:00:00,000 --> 01:00:10,598 H(n) = H(n/2) + Theta(1), there you do have to add 1, 883 01:00:10,598 --> 01:00:20,178 and W(n) = 2W(n/2) + O(1). The usual base cases. 884 01:00:20,178 --> 01:00:32,000 I mean, these are recurrences we should know and love. 885 01:00:32,000 --> 01:00:36,155 This is log n, I sort of have already given 886 01:00:36,155 --> 01:00:40,409 away the answers, and this better be linear. 887 01:00:40,409 --> 01:00:45,356 This is again Case 1. And to the log base 2 of 2 is 888 01:00:45,356 --> 01:00:49,512 n, which is the answer, much bigger than 1. 889 01:00:49,512 --> 01:00:54,459 And here n to the log base 2 of 1 is n to the zero, 890 01:00:54,459 --> 01:01:00,000 which is 1, which is the same so we get log n. 891 01:01:00,000 --> 01:01:04,085 The area is n log n, but if you are making chips you 892 01:01:04,085 --> 01:01:08,331 want the area as small as possible so you can fit more 893 01:01:08,331 --> 01:01:12,256 good stuff in there. So, we would like to aim for, 894 01:01:12,256 --> 01:01:16,182 well, we certainly cannot do a better area than n. 895 01:01:16,182 --> 01:01:19,627 You've got to put the leaves down somewhere, 896 01:01:19,627 --> 01:01:22,110 but this is already pretty good. 897 01:01:22,110 --> 01:01:26,356 It is only a log factor off, but we want to aim for n. 898 01:01:26,356 --> 01:01:31,053 How could we get n? Any guesses on what needs to 899 01:01:31,053 --> 01:01:34,794 change in this layout? Not how to do it because that 900 01:01:34,794 --> 01:01:37,801 is not obvious, but in terms of height and 901 01:01:37,801 --> 01:01:41,616 width what should we do? It is pretty hard to get the 902 01:01:41,616 --> 01:01:44,696 height smaller than log n, I will tell you, 903 01:01:44,696 --> 01:01:48,584 because this is a tree. It cannot really get its width 904 01:01:48,584 --> 01:01:52,398 down to less than log n. What could we do to make the 905 01:01:52,398 --> 01:01:54,819 product linear? Just random ideas. 906 01:01:54,819 --> 01:01:59,000 What are two functions whose product is n? 907 01:02:07,000 --> 01:02:09,227 Square root of n and square root of n. 908 01:02:09,227 --> 01:02:12,298 That is a good choice. Were there other suggestions? 909 01:02:12,298 --> 01:02:15,188 n times constant. Yeah, n times constant would be 910 01:02:15,188 --> 01:02:17,054 nice. But I claim you cannot get 911 01:02:17,054 --> 01:02:19,704 either of these down to less than a constant. 912 01:02:19,704 --> 01:02:23,316 You could aim for n over log n by log n, that is more likely, 913 01:02:23,316 --> 01:02:25,544 but I think that is almost impossible. 914 01:02:25,544 --> 01:02:30,000 Root n by root n is the right answer, so let's go with that. 915 01:02:30,000 --> 01:02:36,050 So, root n by root n. We haven't seen any recurrences 916 01:02:36,050 --> 01:02:42,450 whose solution is root n, but surely they are out there. 917 01:02:42,450 --> 01:02:49,316 Let's say the goal is to get W(n) = Theta(root n) and to get 918 01:02:49,316 --> 01:02:54,785 H(n) = Theta(root n). If we did that we would be 919 01:02:54,785 --> 01:03:02,000 happy, because then the area is the product is linear. 920 01:03:02,000 --> 01:03:05,677 How? What is a recurrence that is in 921 01:03:05,677 --> 01:03:11,245 the usual master method form whose solution is root n? 922 01:03:11,245 --> 01:03:15,237 I mean, you could think of it that way. 923 01:03:15,237 --> 01:03:20,490 Recurrence is a bit tricky, but let's just think of 924 01:03:20,490 --> 01:03:24,377 n^log_b(a). When is log base b of a Ω? 925 01:03:24,377 --> 01:03:29,000 Because then n^log_b(a) is root n. 926 01:03:29,000 --> 01:03:33,726 And there is some hope that I could get a root n solution to 927 01:03:33,726 --> 01:03:36,770 recurrence. This is designed by knowing 928 01:03:36,770 --> 01:03:41,095 that it is divide-and-conquer, and therefore it must be 929 01:03:41,095 --> 01:03:44,940 something like this. It is easy once you know the 930 01:03:44,940 --> 01:03:49,186 approach you are supposed to take and you can try this 931 01:03:49,186 --> 01:03:51,989 approach. When is log base b of a Ω? 932 01:03:51,989 --> 01:03:54,633 Lots of solutions, shout them out. 933 01:03:54,633 --> 01:04:00,000 4 and 2, that is a good one. I better get this right. 934 01:04:00,000 --> 01:04:06,410 Log base 4 of 2 is Ω because the square root of 4 is 2. 935 01:04:06,410 --> 01:04:10,089 So, let's aim for this. Why not? 936 01:04:10,089 --> 01:04:14,007 When would we get log base 4 of 2? 937 01:04:14,007 --> 01:04:19,823 This is b, this is a, so it should be 2T(n/4) plus 938 01:04:19,823 --> 01:04:24,690 something. And if I want the n^log_b(a) to 939 01:04:24,690 --> 01:04:31,338 dominate, it has got to be polynomially smaller than root 940 01:04:31,338 --> 01:04:34,388 n. So, this should be 941 01:04:34,388 --> 01:04:37,320 n^1/2-epsilon. But it could be smaller. 942 01:04:37,320 --> 01:04:39,867 It could be 1. Zero would be nice, 943 01:04:39,867 --> 01:04:43,029 but that is probably too much to hope for. 944 01:04:43,029 --> 01:04:46,966 So, something smaller, strictly polynomially smaller 945 01:04:46,966 --> 01:04:49,203 than root n. That is our goal. 946 01:04:49,203 --> 01:04:53,293 And now comes the magic. If you played with this for a 947 01:04:53,293 --> 01:04:58,000 while you would find it, I think, at this point. 948 01:04:58,000 --> 01:05:02,560 When you know that you are somehow solve this problem of 949 01:05:02,560 --> 01:05:07,287 size n with two subproblems of size n/4 what could you do? 950 01:05:07,287 --> 01:05:11,268 Well, if you start thinking of things as squares, 951 01:05:11,268 --> 01:05:15,000 this is the natural thing that happens. 952 01:05:20,000 --> 01:05:30,000 This is called the H layout. You can imagine why. 953 01:05:30,000 --> 01:05:35,578 It would be much easier to draw if I had a grid board, 954 01:05:35,578 --> 01:05:40,000 a graph board, whatever, if that exists. 955 01:05:46,000 --> 01:05:48,983 This is a recursive layout. I am only going to draw a 956 01:05:48,983 --> 01:05:51,680 couple iterations, but hopefully you can imagine 957 01:05:51,680 --> 01:05:53,000 the generalization. 958 01:06:05,000 --> 01:06:08,578 I take four Hs, a good plan because I want 959 01:06:08,578 --> 01:06:12,156 problems of size n/4. This has n/4 leaves. 960 01:06:12,156 --> 01:06:15,385 This has n/4 leaves. This is the root, 961 01:06:15,385 --> 01:06:19,400 by the way, in the middle. This has n/4 leaves. 962 01:06:19,400 --> 01:06:23,501 This has n/4 leaves. So, I have four problems of 963 01:06:23,501 --> 01:06:26,904 size n/4. Somehow I have got to get that 964 01:06:26,904 --> 01:06:31,471 down to two. Thankfully, if I look at width 965 01:06:31,471 --> 01:06:35,920 or if I look at height, there are only two that matter. 966 01:06:35,920 --> 01:06:40,286 And these two matter and these two get along for free. 967 01:06:40,286 --> 01:06:44,818 They are going in parallel, just like we had with height 968 01:06:44,818 --> 01:06:48,031 over here. But I get that both in height 969 01:06:48,031 --> 01:06:50,173 and in width. If I measure, 970 01:06:50,173 --> 01:06:54,538 well, now they are equal, so I will just call them the 971 01:06:54,538 --> 01:06:58,000 length. We have L(n)-by-L(n). 972 01:06:58,000 --> 01:07:01,117 And if I compute well, what is L(n)? 973 01:07:01,117 --> 01:07:06,194 I have here L(n/4) because there are only a quarter of the 974 01:07:06,194 --> 01:07:11,271 leaves in this one or in that one, then I have a constant, 975 01:07:11,271 --> 01:07:15,813 Theta(1), no big deal, and then I have L(n/4) again. 976 01:07:15,813 --> 01:07:20,000 So, I get the recurrence that I wanted. 977 01:07:26,000 --> 01:07:35,170 L(n) = 2L(n/4) + Theta(1). And that has solution square 978 01:07:35,170 --> 01:07:39,976 root of n, as we claimed before. Again, we are in Case 1 of the 979 01:07:39,976 --> 01:07:41,759 master method. Cool, ha? 980 01:07:41,759 --> 01:07:44,395 This is a much more compact layout. 981 01:07:44,395 --> 01:07:47,108 Charles, did you invent this layout? 982 01:07:47,108 --> 01:07:49,589 No. But I know it appears on your 983 01:07:49,589 --> 01:07:53,620 PhD thesis and you extended it in various directions. 984 01:07:53,620 --> 01:07:58,271 So, this is sort of a classic cool layout of trees into grids 985 01:07:58,271 --> 01:08:03,000 and another application of divide-and-conquer. 986 01:08:03,000 --> 01:08:07,788 I mean this is not particularly useful for algorithms directly. 987 01:08:07,788 --> 01:08:10,647 It is useful for VLSI layout directly. 988 01:08:10,647 --> 01:08:14,663 But it gives you more flavor of how you should think. 989 01:08:14,663 --> 01:08:18,371 If you know what running time you are aiming for, 990 01:08:18,371 --> 01:08:23,237 like in problem sets in quizzes often we say here is the running 991 01:08:23,237 --> 01:08:27,563 time you have got to get, think about the recurrence that 992 01:08:27,563 --> 01:08:31,116 will get you there. And that could inspire you. 993 01:08:31,116 --> 01:08:33,665 And that is it. Recitation Friday. 994 01:08:33,665 --> 01:08:36,446 Homework lab Sunday. No class Monday. 995 01:08:36,446 --> 01:08:39,000 See you Wednesday.