1 00:00:07,000 --> 00:00:12,000 -- valuable experience. OK, today we're going to start 2 00:00:12,000 --> 00:00:18,000 talking about a particular class of algorithms called greedy 3 00:00:18,000 --> 00:00:22,000 algorithms. But we're going to do it in the 4 00:00:22,000 --> 00:00:27,000 context of graphs. So, I want to review a little 5 00:00:27,000 --> 00:00:32,000 bit about graphs, which mostly you can find in 6 00:00:32,000 --> 00:00:39,000 the textbook in appendix B. And so, if you haven't reviewed 7 00:00:39,000 --> 00:00:45,000 in appendix B recently, please sit down and review 8 00:00:45,000 --> 00:00:50,000 appendix B. It will pay off especially 9 00:00:50,000 --> 00:00:55,000 during our take-home quiz. So, just reminder, 10 00:00:55,000 --> 00:01:01,000 a digraph, what's a digraph? What's that short for? 11 00:01:01,000 --> 00:01:04,000 Directed graph, OK? 12 00:01:04,000 --> 00:01:07,000 Directed graph, G equals (V,E), 13 00:01:07,000 --> 00:01:13,000 OK, has a set, V, of vertices. 14 00:01:13,000 --> 00:01:17,000 And, I always get people telling me that I have one 15 00:01:17,000 --> 00:01:20,000 vertice. The singular is not vertice; 16 00:01:20,000 --> 00:01:21,000 it is vertex, OK? 17 00:01:21,000 --> 00:01:25,000 The plural is vertices. The singular is vertex. 18 00:01:25,000 --> 00:01:29,000 It's one of those weird English words. 19 00:01:29,000 --> 00:01:33,000 It's probably originally like French or something, 20 00:01:33,000 --> 00:01:37,000 right? I don't know. 21 00:01:37,000 --> 00:01:47,000 OK, anyway, and we have a set, E, which is a subset of V cross 22 00:01:47,000 --> 00:01:52,000 V of edges. So that's a digraph. 23 00:01:52,000 --> 00:02:02,000 And undirected graph, E contains unordered pairs. 24 00:02:12,000 --> 00:02:15,000 OK, and, sorry? It's Latin, OK, 25 00:02:15,000 --> 00:02:21,000 so it's probably pretty old, then, in English. 26 00:02:21,000 --> 00:02:28,000 I guess the vertex would be a little bit of a giveaway that 27 00:02:28,000 --> 00:02:34,000 maybe it wasn't French. It started to be used in 1570, 28 00:02:34,000 --> 00:02:39,000 OK. OK, good, OK, 29 00:02:39,000 --> 00:02:52,000 so the number of edges is, whether it's directed or 30 00:02:52,000 --> 00:03:02,000 undirected, is O of what? V^2, good. 31 00:03:02,000 --> 00:03:05,000 OK, and one of the conventions that will have when we're 32 00:03:05,000 --> 00:03:09,000 dealing, once we get into graphs, we deal a lot with sets. 33 00:03:09,000 --> 00:03:13,000 We generally drop the vertical bar notation within O's just 34 00:03:13,000 --> 00:03:16,000 because it's applied. It just makes it messier. 35 00:03:16,000 --> 00:03:18,000 So, once again, another abuse of notation. 36 00:03:18,000 --> 00:03:22,000 It really should be order the size of V^2, but it just messes 37 00:03:22,000 --> 00:03:25,000 up, I mean, it's just more stuff to write down. 38 00:03:25,000 --> 00:03:29,000 And, you're multiplying these things, and all those vertical 39 00:03:29,000 --> 00:03:33,000 bars. Since they don't even have a 40 00:03:33,000 --> 00:03:36,000 sense to the vertical bar, it gets messy. 41 00:03:36,000 --> 00:03:40,000 So, we just drop the vertical bars there when it's in 42 00:03:40,000 --> 00:03:44,000 asymptotic notation. So, E is order V^2 when it's a 43 00:03:44,000 --> 00:03:48,000 set of pairs, because if it's a set of pairs, 44 00:03:48,000 --> 00:03:52,000 it's at most n choose two, which is where it's at most n^2 45 00:03:52,000 --> 00:03:56,000 over 2, here it could be, at most, sorry, 46 00:03:56,000 --> 00:04:00,000 V^2 over 2, here it's at most V^2. 47 00:04:00,000 --> 00:04:07,000 And then, another property that sometimes comes up is if the G 48 00:04:07,000 --> 00:04:11,000 is connected, we have another bound, 49 00:04:11,000 --> 00:04:18,000 implies that the size of E is at least the size of V minus 50 00:04:18,000 --> 00:04:22,000 one. OK, so if it's connected, 51 00:04:22,000 --> 00:04:31,000 meaning, what does it mean to have a graph that's connected? 52 00:04:31,000 --> 00:04:37,000 Yeah, there's a path from any vertex to any other vertex in 53 00:04:37,000 --> 00:04:40,000 the graph. That's what it means to be 54 00:04:40,000 --> 00:04:44,000 connected. So if that's the case, 55 00:04:44,000 --> 00:04:50,000 that a number of edges is at least the number of vertices 56 00:04:50,000 --> 00:04:53,000 minus one, OK? And so, what that says, 57 00:04:53,000 --> 00:05:00,000 so one of the things we'll get into, a fact that I just wanted 58 00:05:00,000 --> 00:05:04,000 to remind you, is that in that case, 59 00:05:04,000 --> 00:05:09,000 if I look at log E, OK, log of the number of edges, 60 00:05:09,000 --> 00:05:14,000 that is O of log V. And by this, 61 00:05:14,000 --> 00:05:18,000 is omega of log V. So, it's equal to theta of log 62 00:05:18,000 --> 00:05:21,000 V. OK, so basically the number of, 63 00:05:21,000 --> 00:05:25,000 in the case of a connected graph, the number of edges, 64 00:05:25,000 --> 00:05:30,000 and the number of vertices are polynomially related. 65 00:05:30,000 --> 00:05:36,000 So, their logs are comparable. OK, so that's helpful just to 66 00:05:36,000 --> 00:05:41,000 know because sometimes I just get questions later on where 67 00:05:41,000 --> 00:05:45,000 people will say, oh, you showed it was log E but 68 00:05:45,000 --> 00:05:51,000 you didn't show it was log V. And I could point out that it's 69 00:05:51,000 --> 00:05:55,000 the same thing. OK, so there's various ways of 70 00:05:55,000 --> 00:05:59,000 representing graphs in computers, and I'm just going to 71 00:05:59,000 --> 00:06:04,000 cover a couple of the important ones. 72 00:06:04,000 --> 00:06:11,000 There's actually more. We'll see some more. 73 00:06:11,000 --> 00:06:20,000 So, the simplest one is what's called an adjacency matrix. 74 00:06:20,000 --> 00:06:30,000 An adjacency matrix of the graph, G, equals (V,E), 75 00:06:30,000 --> 00:06:40,000 where, for simplicity, I'll let V be the set of 76 00:06:40,000 --> 00:06:52,000 integers from one up to n, OK, is the n by n matrix A 77 00:06:52,000 --> 00:07:04,000 given by the ij-th at the entry is simply one if the edge, 78 00:07:04,000 --> 00:07:20,000 ij, is in the edge set and zero if ij is not in the edge set. 79 00:07:20,000 --> 00:07:22,000 OK, so it's simply the matrix where you say, 80 00:07:22,000 --> 00:07:25,000 the ij entry is one if it's in the matrix. 81 00:07:25,000 --> 00:07:27,000 So, this is, in some sense, 82 00:07:27,000 --> 00:07:32,000 giving you the predicate for, is there an edge from i to j? 83 00:07:32,000 --> 00:07:35,000 OK, remember, predicate is Boolean formula 84 00:07:35,000 --> 00:07:39,000 that is either zero or one, and in this case, 85 00:07:39,000 --> 00:07:45,000 you're saying it's one if there is an edge from i to j and zero 86 00:07:45,000 --> 00:07:48,000 otherwise. OK, sometimes you have edge 87 00:07:48,000 --> 00:07:52,000 weighted graphs, and then sometimes what people 88 00:07:52,000 --> 00:07:56,000 will do is replace this by edge weights. 89 00:07:56,000 --> 00:08:02,000 OK, it will be the weight of the edge from i to j. 90 00:08:02,000 --> 00:08:12,000 So, let's just do an example of that just to make sure that our 91 00:08:12,000 --> 00:08:22,000 intuition corresponds to our mathematical definitions. 92 00:08:22,000 --> 00:08:33,000 So, here's an example graph. Let's say that's our graph. 93 00:08:33,000 --> 00:08:37,000 So let's just draw the adjacency the matrix. 94 00:08:37,000 --> 00:08:42,000 OK, so what this says: is there's an edge from one to 95 00:08:42,000 --> 00:08:44,000 one? And the answer is no. 96 00:08:44,000 --> 00:08:47,000 Is there an edge from one to two? 97 00:08:47,000 --> 00:08:50,000 Yes. Is there an edge from one to 98 00:08:50,000 --> 00:08:51,000 three here? Yep. 99 00:08:51,000 --> 00:08:54,000 Is there an edge for one to four? 100 00:08:54,000 --> 00:08:58,000 No. Is there an edge from two until 101 00:08:58,000 --> 00:09:00,000 one? No. 102 00:09:00,000 --> 00:09:02,000 Two to two? No. 103 00:09:02,000 --> 00:09:04,000 Two to three? Yes. 104 00:09:04,000 --> 00:09:06,000 Two to four? No. 105 00:09:06,000 --> 00:09:13,000 No edges going out of three. Edge from four to three, 106 00:09:13,000 --> 00:09:19,000 and that's it. That's the adjacency matrix for 107 00:09:19,000 --> 00:09:23,000 this particular graph, OK? 108 00:09:23,000 --> 00:09:33,000 And so, I can represent a graph as this adjacency matrix. 109 00:09:33,000 --> 00:09:43,000 OK, when I represent it in this way, how much storage do I need? 110 00:09:43,000 --> 00:09:52,000 OK, n^2 or V^2 because the size is the same thing for V^2 111 00:09:52,000 --> 00:10:03,000 storage, OK, and that's what we call a dense representation. 112 00:10:03,000 --> 00:10:06,000 OK, it works well when the graph is dense. 113 00:10:06,000 --> 00:10:11,000 So, the graph is dense if the number of edges is close to all 114 00:10:11,000 --> 00:10:14,000 of the edges possible. OK, then this is a good 115 00:10:14,000 --> 00:10:18,000 representation. But for many types of graphs, 116 00:10:18,000 --> 00:10:23,000 the number of edges is much less than the possible number of 117 00:10:23,000 --> 00:10:26,000 edges, in which case we say the graph is sparse. 118 00:10:26,000 --> 00:10:32,000 Can somebody give me an example of a sparse graph? 119 00:10:32,000 --> 00:10:35,000 A class of graphs: so, I want a class of graphs 120 00:10:35,000 --> 00:10:38,000 that as n grows, the number of edges in the 121 00:10:38,000 --> 00:10:42,000 graph doesn't grow as the square, but grows rather as 122 00:10:42,000 --> 00:10:45,000 something much smaller. A linked list, 123 00:10:45,000 --> 00:10:48,000 so, a chain, OK, if you look at it from a 124 00:10:48,000 --> 00:10:52,000 graph theoretically, is a perfectly good example: 125 00:10:52,000 --> 00:10:56,000 only n edges in the chain for a chain of length n. 126 00:10:56,000 --> 00:10:59,000 So therefore, the number of edges would be 127 00:10:59,000 --> 00:11:02,000 order V. And in particular, 128 00:11:02,000 --> 00:11:07,000 you'd only have one edge per row here. 129 00:11:07,000 --> 00:11:10,000 What other graphs are sparse? Yeah? 130 00:11:10,000 --> 00:11:16,000 Good, a planar graph, a graph that can be drawn in a 131 00:11:16,000 --> 00:11:21,000 plane turns out that if it has V vertices has, 132 00:11:21,000 --> 00:11:25,000 and V is at least three, then it has, 133 00:11:25,000 --> 00:11:30,000 at most, three V minus six edges. 134 00:11:30,000 --> 00:11:34,000 So, it turns out that's order V edges again. 135 00:11:34,000 --> 00:11:38,000 What's another example of a common graph? 136 00:11:38,000 --> 00:11:42,000 Yeah, binary tree, or even actually any tree, 137 00:11:42,000 --> 00:11:49,000 you know, what's called a free tree if you read the appendix, 138 00:11:49,000 --> 00:11:54,000 OK, a tree that just is a connected graph that has no 139 00:11:54,000 --> 00:12:00,000 cycles, OK, is another example. What's an example of a graph 140 00:12:00,000 --> 00:12:04,000 that's dense? A complete graph, 141 00:12:04,000 --> 00:12:07,000 OK: it's all ones, OK, or if you have edge 142 00:12:07,000 --> 00:12:11,000 weights, it would be a completely filled in matrix. 143 00:12:11,000 --> 00:12:14,000 OK, good. So, this is good for dense 144 00:12:14,000 --> 00:12:17,000 representation. But sometimes you want to have 145 00:12:17,000 --> 00:12:22,000 a sparse representation so we don't have to spend V^2 space to 146 00:12:22,000 --> 00:12:26,000 deal with all of the, where most of it's going to be 147 00:12:26,000 --> 00:12:28,000 zeroes. OK, it's sort of like, 148 00:12:28,000 --> 00:12:33,000 if we know why it's zero, why bother representing it as 149 00:12:33,000 --> 00:12:40,000 zero? So, one such representation is 150 00:12:40,000 --> 00:12:46,000 an adjacency list representation. 151 00:12:46,000 --> 00:12:56,000 Actually, adjacency list of a given vertex is the list, 152 00:12:56,000 --> 00:13:08,000 which we denote by Adj of V, of vertices adjacent to V. 153 00:13:08,000 --> 00:13:12,000 OK, just in terms by their terminology, vertices are 154 00:13:12,000 --> 00:13:17,000 adjacent, but edges are incident on vertices. 155 00:13:17,000 --> 00:13:22,000 OK, so the incidence is a relation between a vertex and an 156 00:13:22,000 --> 00:13:25,000 edge. An adjacency is a relation 157 00:13:25,000 --> 00:13:31,000 between two vertices. OK, that's just the language. 158 00:13:31,000 --> 00:13:36,000 Why they use to different terms, I don't know, 159 00:13:36,000 --> 00:13:40,000 but that's what they do. So, in the graph, 160 00:13:40,000 --> 00:13:45,000 for example, the adjacency list for vertex 161 00:13:45,000 --> 00:13:52,000 one is just the list or the set of two three because one has 162 00:13:52,000 --> 00:13:57,000 going out of one are edges to two and three. 163 00:13:57,000 --> 00:14:02,000 The adjacency list for two is just three, four, 164 00:14:02,000 --> 00:14:07,000 three. It's the empty set, 165 00:14:07,000 --> 00:14:10,000 and for four, it is three. 166 00:14:10,000 --> 00:14:13,000 OK, so that's the representation. 167 00:14:13,000 --> 00:14:21,000 Now, if we want to figure out how much storage is required for 168 00:14:21,000 --> 00:14:26,000 this representation, OK, we need to understand how 169 00:14:26,000 --> 00:14:35,000 long the adjacency list is. So, what is the length of an 170 00:14:35,000 --> 00:14:40,000 adjacency list of a vertex, V? 171 00:14:40,000 --> 00:14:48,000 What name do we give to that? It's the degree. 172 00:14:48,000 --> 00:14:57,000 So, in an undirected graph, we call it the degree of the 173 00:14:57,000 --> 00:15:02,000 vertex. This is undirected. 174 00:15:02,000 --> 00:15:07,000 OK, about here, OK. 175 00:15:07,000 --> 00:15:12,000 So that's an undirected case. In the directed case, 176 00:15:12,000 --> 00:15:18,000 OK, actually I guess the way we should do this is say this. 177 00:15:18,000 --> 00:15:22,000 If the degree, we call it the out degree for a 178 00:15:22,000 --> 00:15:25,000 digraph. OK, so in a digraph, 179 00:15:25,000 --> 00:15:32,000 we have an out degree and an in degree for each vertex. 180 00:15:32,000 --> 00:15:36,000 So here, the in degree is three. 181 00:15:36,000 --> 00:15:41,000 Here, the out degree is two, OK? 182 00:15:41,000 --> 00:15:50,000 So, one of the important lemma that comes up is what's called 183 00:15:50,000 --> 00:15:56,000 the handshaking lemma. OK, it's one of these 184 00:15:56,000 --> 00:16:01,000 mathematical lemmas. 185 00:16:11,000 --> 00:16:16,000 And so, it comes from a story. Go to a dinner party, 186 00:16:16,000 --> 00:16:21,000 and everybody at the dinner party shakes other people's 187 00:16:21,000 --> 00:16:24,000 hands. Some people may not shake 188 00:16:24,000 --> 00:16:29,000 anybody's hand. Some people may shake several 189 00:16:29,000 --> 00:16:32,000 people's hands. Nobody shakes hands with 190 00:16:32,000 --> 00:16:37,000 themselves. And at some point during the 191 00:16:37,000 --> 00:16:41,000 dinner party, the host goes around and counts 192 00:16:41,000 --> 00:16:45,000 up how many, the sum, of the number of hands that 193 00:16:45,000 --> 00:16:48,000 each person has shaken. OK, so he says, 194 00:16:48,000 --> 00:16:52,000 how many did you shake? How many did you shake? 195 00:16:52,000 --> 00:16:55,000 How many did you shake? He adds them up, 196 00:16:55,000 --> 00:17:00,000 OK, and that number is guaranteed to be even. 197 00:17:00,000 --> 00:17:03,000 OK, that's the handshaking lemma. 198 00:17:03,000 --> 00:17:08,000 Or, stated a little bit more precisely, if I take for any 199 00:17:08,000 --> 00:17:13,000 graph the degree of the vertex, and sum them all up, 200 00:17:13,000 --> 00:17:20,000 that's how many hands everybody shook, OK, that's actually equal 201 00:17:20,000 --> 00:17:23,000 to always twice the number of edges. 202 00:17:23,000 --> 00:17:26,000 So, why is that going to be true? 203 00:17:26,000 --> 00:17:33,000 Why is that going to be twice the number of edges? 204 00:17:33,000 --> 00:17:33,000 Yeah? Yeah. Every time you put in an edge, you add one to the degree of 205 00:17:39,000 --> 00:17:44,000 each person on each end. So, it's just two different 206 00:17:44,000 --> 00:17:48,000 ways of counting up the same number of edges. 207 00:17:48,000 --> 00:17:52,000 OK, I can go around, and if you imagine that, 208 00:17:52,000 --> 00:17:56,000 that every time I count the degree of the node, 209 00:17:56,000 --> 00:18:01,000 I put a mark on every edge. Then, when I'm done, 210 00:18:01,000 --> 00:18:07,000 every edge has two marks on it, one for each end. 211 00:18:07,000 --> 00:18:17,000 OK: a pretty simple theorem. So, what that says is that for 212 00:18:17,000 --> 00:18:26,000 undirected graphs, that implies that the adjacency 213 00:18:26,000 --> 00:18:35,000 list representation, uses how much storage? 214 00:18:35,000 --> 00:18:39,000 OK, at most, 2E, so order E because that's 215 00:18:39,000 --> 00:18:42,000 not all. Yeah, so you have to have the 216 00:18:42,000 --> 00:18:47,000 number of vertices plus order the number of edges, 217 00:18:47,000 --> 00:18:53,000 OK, whether it's directed or undirected because I may have a 218 00:18:53,000 --> 00:18:59,000 graph, say it has a whole bunch of vertices and no edges, 219 00:18:59,000 --> 00:19:05,000 that's still going to cost me order V, OK? 220 00:19:05,000 --> 00:19:09,000 So, it uses theta of V plus E storage. 221 00:19:09,000 --> 00:19:15,000 And, it's basically the same thing asymptotically. 222 00:19:15,000 --> 00:19:22,000 In fact, it's easier to see in some sense for digraphs because 223 00:19:22,000 --> 00:19:27,000 for digraphs, what I do is I just add up the 224 00:19:27,000 --> 00:19:33,000 out degrees, and that equal to E, OK, if I add up the out 225 00:19:33,000 --> 00:19:39,000 degrees as equally. In fact, this is kind of like 226 00:19:39,000 --> 00:19:41,000 it amortized analysis, if you will, 227 00:19:41,000 --> 00:19:45,000 a book keeping analysis, that if I'm adding up the total 228 00:19:45,000 --> 00:19:48,000 number of edges, one way of doing it is 229 00:19:48,000 --> 00:19:50,000 accounting for a vertex by vertex. 230 00:19:50,000 --> 00:19:54,000 OK, so for each vertex, I basically can take each 231 00:19:54,000 --> 00:19:58,000 degree, and basically each vertex, look at the degree, 232 00:19:58,000 --> 00:20:02,000 and that allocating of account per edge, and then ending up 233 00:20:02,000 --> 00:20:06,000 with twice the number of edges, that's exactly accounting type 234 00:20:06,000 --> 00:20:12,000 of analysis that we might do for amortized analysis. 235 00:20:12,000 --> 00:20:17,000 OK, so we'll see that. So, this is a sparse 236 00:20:17,000 --> 00:20:22,000 representation, and it's often better than an 237 00:20:22,000 --> 00:20:25,000 adjacency matrix. For example, 238 00:20:25,000 --> 00:20:32,000 you can imagine if the World Wide Web were done with an 239 00:20:32,000 --> 00:20:37,000 adjacency matrix as opposed to, essentially, 240 00:20:37,000 --> 00:20:44,000 with an adjacency list type of representation. 241 00:20:44,000 --> 00:20:47,000 Every link on the World Wide Web, I had to say, 242 00:20:47,000 --> 00:20:50,000 here are the ones that I'm connected to, 243 00:20:50,000 --> 00:20:53,000 and here are all the ones I'm not connected to. 244 00:20:53,000 --> 00:20:57,000 OK, that list of things you're not connected to for a given 245 00:20:57,000 --> 00:20:59,000 page would be pretty dramatically, 246 00:20:59,000 --> 00:21:03,000 show you that there is an advantage to sparse 247 00:21:03,000 --> 00:21:06,000 representation. On the other hand, 248 00:21:06,000 --> 00:21:13,000 one of the nice things about an adjacency matrix representation 249 00:21:13,000 --> 00:21:19,000 is that each edge can be represented with a single bit, 250 00:21:19,000 --> 00:21:24,000 whereas typical when I'm representing things with an 251 00:21:24,000 --> 00:21:30,000 adjacency list representation, how many bits am I going to 252 00:21:30,000 --> 00:21:35,000 need to represent each adjacency? 253 00:21:35,000 --> 00:21:39,000 You'll need order log of V to be able to name each different 254 00:21:39,000 --> 00:21:41,000 vertex. OK, the log of the number is 255 00:21:41,000 --> 00:21:46,000 the number of bits that I need. So, there are places where this 256 00:21:46,000 --> 00:21:50,000 is actually a far more efficient representation. 257 00:21:50,000 --> 00:21:53,000 In particular, if you have a very dense graph, 258 00:21:53,000 --> 00:21:56,000 OK, this may be a better way of representing it. 259 00:21:56,000 --> 00:22:01,000 OK, the other thing I want you to get, and we're going to see 260 00:22:01,000 --> 00:22:05,000 more of this in particular next week, is that a matrix and a 261 00:22:05,000 --> 00:22:11,000 graph, there are two ways of looking at the same thing. 262 00:22:11,000 --> 00:22:14,000 OK, and in fact, there's a lot of graph theory 263 00:22:14,000 --> 00:22:17,000 that when you do things like multiply the adjacency matrix, 264 00:22:17,000 --> 00:22:20,000 OK, and so forth. So, there's a lot of 265 00:22:20,000 --> 00:22:24,000 commonality between graphs and matrices, a lot of mathematics 266 00:22:24,000 --> 00:22:28,000 that if it applies for one, it applies to the other. 267 00:22:28,000 --> 00:22:31,000 Do you have a question, or just holding your finger in 268 00:22:31,000 --> 00:22:33,000 the air? OK, good. 269 00:22:33,000 --> 00:22:37,000 OK, so that's all just review. Now I want to get onto today's 270 00:22:37,000 --> 00:22:40,000 lecture. OK, so any questions about 271 00:22:40,000 --> 00:22:42,000 graphs? So, this is a good time to 272 00:22:42,000 --> 00:22:45,000 review appendix B. there are a lot of great 273 00:22:45,000 --> 00:22:48,000 properties in there, and in particular, 274 00:22:48,000 --> 00:22:52,000 there is a theorem that we're going to cover today that we're 275 00:22:52,000 --> 00:22:56,000 going to talk about today, which is properties of trees. 276 00:22:56,000 --> 00:23:00,000 Trees are very special kinds of graphs, so I really want you to 277 00:23:00,000 --> 00:23:05,000 go and look to see what the properties are. 278 00:23:05,000 --> 00:23:08,000 There is, I think, something like six different 279 00:23:08,000 --> 00:23:11,000 definitions of trees that are all equivalent, 280 00:23:11,000 --> 00:23:16,000 OK, and so, I think a very good idea to go through and read 281 00:23:16,000 --> 00:23:20,000 through that theorem. We're not going to prove it in 282 00:23:20,000 --> 00:23:23,000 class, but really, provides a very good basis for 283 00:23:23,000 --> 00:23:27,000 the thinking that we're going to be doing today. 284 00:23:27,000 --> 00:23:30,000 And we'll see more of that in the future. 285 00:23:30,000 --> 00:23:33,000 OK, so today, we're going to talk about 286 00:23:33,000 --> 00:23:38,000 minimum spanning trees. OK, this is one of the world's 287 00:23:38,000 --> 00:23:42,000 most important algorithms. OK, it is important in 288 00:23:42,000 --> 00:23:46,000 distributed systems. It's one of the first things 289 00:23:46,000 --> 00:23:50,000 that almost any distributed system tries to find is a 290 00:23:50,000 --> 00:23:55,000 minimum spanning tree of the nodes that happened to be alive 291 00:23:55,000 --> 00:23:56,000 at any point, OK? 292 00:23:56,000 --> 00:24:01,000 And one of the people who developed an algorithm for this, 293 00:24:01,000 --> 00:24:04,000 we'll talk about this a little bit later, OK, 294 00:24:04,000 --> 00:24:09,000 it was the basis of the billing system for AT&T for many years 295 00:24:09,000 --> 00:24:16,000 while it was a monopoly. OK, so very important kind of 296 00:24:16,000 --> 00:24:21,000 thing. It's got a huge number of 297 00:24:21,000 --> 00:24:25,000 applications. So the problem is the 298 00:24:25,000 --> 00:24:31,000 following. You have a connected undirected 299 00:24:31,000 --> 00:24:35,000 graph, G equals (V,E), 300 00:24:35,000 --> 00:24:42,000 with an edge weight function, w, which maps the edges into 301 00:24:42,000 --> 00:24:50,000 weights that are real numbers. And for today's lecture, 302 00:24:50,000 --> 00:24:56,000 we're going to make an important assumption, 303 00:24:56,000 --> 00:25:02,000 OK, for simplicity. The book does not make this 304 00:25:02,000 --> 00:25:08,000 assumption. And so, I encourage you to look 305 00:25:08,000 --> 00:25:16,000 at the alternative presentation or, because what they do in the 306 00:25:16,000 --> 00:25:22,000 book is much more general, but for simplicity and 307 00:25:22,000 --> 00:25:29,000 intuition, I'm going to make this a little bit easier. 308 00:25:29,000 --> 00:25:37,000 We're going to assume that all edge weights are distinct. 309 00:25:37,000 --> 00:25:39,000 OK, all edge weights are distinct. 310 00:25:39,000 --> 00:25:42,000 So what does that mean? What does that mean that this 311 00:25:42,000 --> 00:25:46,000 function, w, what property does the function, 312 00:25:46,000 --> 00:25:48,000 w, have if all edge weights are distinct? 313 00:25:48,000 --> 00:25:51,000 Who remembers their discreet math? 314 00:25:51,000 --> 00:25:53,000 It's injective. OK, it's one to one. 315 00:25:53,000 --> 00:25:56,000 OK, it's not one to one and onto necessarily. 316 00:25:56,000 --> 00:26:00,000 In fact, it would be kind of hard to do that because that's a 317 00:26:00,000 --> 00:26:05,000 pretty big set. OK, but it's one to one. 318 00:26:05,000 --> 00:26:09,000 It's injective. OK, so that's what we're going 319 00:26:09,000 --> 00:26:14,000 to assume for simplicity. OK, and the book, 320 00:26:14,000 --> 00:26:19,000 they don't assume that. It just means that the way you 321 00:26:19,000 --> 00:26:24,000 have to state things is just a little more precise. 322 00:26:24,000 --> 00:26:28,000 It has to be more technically precise. 323 00:26:28,000 --> 00:26:33,000 So, that's the input. The output is-- 324 00:26:33,000 --> 00:26:44,000 The output is a spanning tree, T, and by spanning tree, 325 00:26:44,000 --> 00:26:52,000 we mean it connects all the vertices. 326 00:26:52,000 --> 00:27:02,000 OK, and it's got to have minimum weight. 327 00:27:02,000 --> 00:27:08,000 OK, so we can write the weight of the tree is going to be, 328 00:27:08,000 --> 00:27:14,000 by that, we meet the sum over all edges that are in the tree 329 00:27:14,000 --> 00:27:18,000 of the weight of the individual edges. 330 00:27:18,000 --> 00:27:24,000 OK, so here I'(V,E) done a little bit of abusive notation, 331 00:27:24,000 --> 00:27:31,000 which is that what I should be writing is w of the edge (u,v) 332 00:27:31,000 --> 00:27:37,000 because this is a mapping from edges, which would give me a 333 00:27:37,000 --> 00:27:42,000 double parentheses. And, you know, 334 00:27:42,000 --> 00:27:45,000 as you know, I love to abuse notation. 335 00:27:45,000 --> 00:27:48,000 So, I'm going to drop that extra parentheses, 336 00:27:48,000 --> 00:27:54,000 because we understand that it's really the weight of the edge, 337 00:27:54,000 --> 00:27:57,000 OK, not the weight of the ordered pair. 338 00:27:57,000 --> 00:28:02,000 So, that's just a little notational convenience. 339 00:28:02,000 --> 00:28:05,000 OK, so one of the things, when we do the take-home exam, 340 00:28:05,000 --> 00:28:09,000 notational convenience can make the difference between having a 341 00:28:09,000 --> 00:28:12,000 horrible time writing up a problem, and an easy time. 342 00:28:12,000 --> 00:28:16,000 So, it's worth thinking about what kinds of notation you'll 343 00:28:16,000 --> 00:28:19,000 use in writing up solutions to problems, and so forth. 344 00:28:19,000 --> 00:28:22,000 OK, and just in general, a technical communication, 345 00:28:22,000 --> 00:28:25,000 you adopt good notation people understand you. 346 00:28:25,000 --> 00:28:29,000 You adopt a poor notation: nobody pays attention to what 347 00:28:29,000 --> 00:28:34,000 you're doing because they don't understand what you're saying. 348 00:28:34,000 --> 00:28:38,000 OK, so let's do an example. 349 00:28:45,000 --> 00:28:52,000 OK, so here's a graph. I think for this, 350 00:28:52,000 --> 00:29:02,000 somebody asked once if I was inspired by biochemistry or 351 00:29:02,000 --> 00:29:09,000 something, OK, but I wasn't. 352 00:29:09,000 --> 00:29:11,000 I was just writing these things down, OK? 353 00:29:11,000 --> 00:29:15,000 So, here's a graph. And let's give us some edge 354 00:29:15,000 --> 00:29:16,000 weights. 355 00:29:31,000 --> 00:29:34,000 OK, so there are some edge weights. 356 00:29:34,000 --> 00:29:39,000 And now, what we want is we want to find a tree. 357 00:29:39,000 --> 00:29:45,000 So a connected acyclic graph such that every vertex is part 358 00:29:45,000 --> 00:29:49,000 of the tree. But it's got to have the 359 00:29:49,000 --> 00:29:55,000 minimum weight possible. OK, so can somebody suggest to 360 00:29:55,000 --> 00:30:03,000 me some edges that have to be in this minimum spanning tree? 361 00:30:03,000 --> 00:30:05,000 Yeah, so nine, good. 362 00:30:05,000 --> 00:30:09,000 Nine has to be in there because, why? 363 00:30:09,000 --> 00:30:14,000 It's the only one connecting it to this vertex, 364 00:30:14,000 --> 00:30:16,000 OK? And likewise, 365 00:30:16,000 --> 00:30:22,000 15 has to be in there. So those both have to be in. 366 00:30:22,000 --> 00:30:26,000 What other edges have to be in? Which one? 367 00:30:26,000 --> 00:30:33,000 14 has to be it. Why does 14 have to be in? 368 00:30:33,000 --> 00:30:40,000 Well, one of 14 and three has to be in there. 369 00:30:40,000 --> 00:30:50,000 I want the minimum weight. The one that has the overall 370 00:30:50,000 --> 00:30:57,000 smallest weight. So, can somebody argue to me 371 00:30:57,000 --> 00:31:04,000 that three has to be in there? Yeah? 372 00:31:04,000 --> 00:31:09,000 That's the minimum of two, which means that if I had a, 373 00:31:09,000 --> 00:31:14,000 if you add something you said was a minimum spanning tree that 374 00:31:14,000 --> 00:31:19,000 didn't include three, right, and so therefore it had 375 00:31:19,000 --> 00:31:23,000 to include 14, then I could just delete this 376 00:31:23,000 --> 00:31:28,000 edge, 14, and put in edge three. And, I have something of lower 377 00:31:28,000 --> 00:31:34,000 weight, right? So, three has to be in there. 378 00:31:34,000 --> 00:31:37,000 What other edges have to be in there? 379 00:31:37,000 --> 00:31:43,000 Do a little puzzle logic. Six and five have to be in 380 00:31:43,000 --> 00:31:46,000 there. Why do they have to be in 381 00:31:46,000 --> 00:31:48,000 there? 382 00:32:02,000 --> 00:32:05,000 Yeah, well, I mean, it could be connected through 383 00:32:05,000 --> 00:32:08,000 this or something. It doesn't necessarily have to 384 00:32:08,000 --> 00:32:11,000 go this way. Six definitely has to be in 385 00:32:11,000 --> 00:32:14,000 there for the same reason that three had to be, 386 00:32:14,000 --> 00:32:16,000 right? Because we got two choices to 387 00:32:16,000 --> 00:32:19,000 connect up this guy. And so, if everything were 388 00:32:19,000 --> 00:32:22,000 connected but it weren't, 12, I mean, and 12 was in 389 00:32:22,000 --> 00:32:24,000 there. I could always, 390 00:32:24,000 --> 00:32:27,000 then, say, well, let's connect them up this way 391 00:32:27,000 --> 00:32:31,000 instead. OK, so definitely that's in 392 00:32:31,000 --> 00:32:35,000 there. I still don't have everything 393 00:32:35,000 --> 00:32:37,000 connected up. 394 00:32:50,000 --> 00:33:03,000 What else has to be in there for minimum spanning tree? 395 00:33:03,000 --> 00:33:11,000 Seven, five, and eight, why seven, 396 00:33:11,000 --> 00:33:22,000 five, and eight? OK, so can we argue those one 397 00:33:22,000 --> 00:33:32,000 at a time? Why does five have to be in 398 00:33:32,000 --> 00:33:37,000 there? Yeah? 399 00:33:37,000 --> 00:33:41,000 OK, so we have four connected components because we have this 400 00:33:41,000 --> 00:33:43,000 one, this one, we actually have, 401 00:33:43,000 --> 00:33:46,000 yeah, this one here, and this one, 402 00:33:46,000 --> 00:33:49,000 good. We need at least three edges to 403 00:33:49,000 --> 00:33:53,000 connect them because each edge is going to reduce the connected 404 00:33:53,000 --> 00:33:57,000 components by one. OK, so we need three edges, 405 00:33:57,000 --> 00:33:59,000 and those are the three cheapest ones. 406 00:33:59,000 --> 00:34:04,000 And they work. That works, right? 407 00:34:04,000 --> 00:34:11,000 Any other edges are going to be bigger, so that works. 408 00:34:11,000 --> 00:34:15,000 Good. OK, and so, now do we have a 409 00:34:15,000 --> 00:34:19,000 spanning tree? Everything is, 410 00:34:19,000 --> 00:34:24,000 we have one big connected graph here, right? 411 00:34:24,000 --> 00:34:31,000 Is that what I got? Hey, that's the same as what I 412 00:34:31,000 --> 00:34:35,000 got. Life is predictable. 413 00:34:35,000 --> 00:34:41,000 OK, so, so everybody had the idea of what a minimum spanning 414 00:34:41,000 --> 00:34:44,000 tree is, then, out of this, 415 00:34:44,000 --> 00:34:49,000 OK, what's going on there? So, let's first of all make 416 00:34:49,000 --> 00:34:53,000 some observations about this puzzle. 417 00:34:53,000 --> 00:34:59,000 And what I want to do is remind you about the optimal 418 00:34:59,000 --> 00:35:06,000 substructure property because it turns out minimum spanning tree 419 00:35:06,000 --> 00:35:12,000 has a great optimal substructure property. 420 00:35:12,000 --> 00:35:17,000 OK, so the setup is going to be, we're going to have some 421 00:35:17,000 --> 00:35:22,000 minimum spanning tree. Let's call it T. 422 00:35:22,000 --> 00:35:27,000 And, I'm going to show that with the other edges in the 423 00:35:27,000 --> 00:35:32,000 graph, are not going to be shown. 424 00:35:32,000 --> 00:35:37,000 OK, so here's a graph. 425 00:35:54,000 --> 00:35:58,000 OK, so here's a graph. It looks like the one I have my 426 00:35:58,000 --> 00:36:01,000 piece of paper here. OK, so the idea is, 427 00:36:01,000 --> 00:36:05,000 this is some minimum spanning tree. 428 00:36:05,000 --> 00:36:09,000 Now, we want to look at a property of optimal 429 00:36:09,000 --> 00:36:13,000 substructure. And the way I'm going to get 430 00:36:13,000 --> 00:36:17,000 that, is, I'm going to remove some edge, (u,v), 431 00:36:17,000 --> 00:36:22,000 move an arbitrary edge, (u,v), in the minimum spanning 432 00:36:22,000 --> 00:36:26,000 tree. So, let's call this u and this 433 00:36:26,000 --> 00:36:29,000 V. And so, we're removing this 434 00:36:29,000 --> 00:36:33,000 edge. OK, so when I remove an edge in 435 00:36:33,000 --> 00:36:36,000 a tree, what happens to the tree? 436 00:36:36,000 --> 00:36:39,000 What's left? I have two trees left, 437 00:36:39,000 --> 00:36:41,000 OK? I have two trees left. 438 00:36:41,000 --> 00:36:45,000 Now, proving that, that's basically one of the 439 00:36:45,000 --> 00:36:50,000 properties in that appendix, and the properties of trees 440 00:36:50,000 --> 00:36:55,000 that I want you to read, OK, because you can actually 441 00:36:55,000 --> 00:37:00,000 prove that kind of thing rather than it just being obvious, 442 00:37:00,000 --> 00:37:05,000 which is, OK? OK, so we remove that. 443 00:37:05,000 --> 00:37:11,000 Then, T is partitioned into two subtrees. 444 00:37:11,000 --> 00:37:15,000 And, we'll call them T_1 and T_2. 445 00:37:15,000 --> 00:37:22,000 So, here's one subtree, and here's another subtree. 446 00:37:22,000 --> 00:37:29,000 We'(V,E) partitioned it. No matter what edge I picked, 447 00:37:29,000 --> 00:37:38,000 there would be two subtrees that it's partitioned into. 448 00:37:38,000 --> 00:37:40,000 Even if the sub tree is a trivial subtree, 449 00:37:40,000 --> 00:37:43,000 for example, it just has a single node in it 450 00:37:43,000 --> 00:37:45,000 and no edges. 451 00:37:58,000 --> 00:38:11,000 So, the theorem that we'll prove demonstrates a property of 452 00:38:11,000 --> 00:38:24,000 optimal substructure. T_1 is a minimum spanning tree 453 00:38:24,000 --> 00:38:31,000 for the graph, G_1, E_1, 454 00:38:31,000 --> 00:38:43,000 a subgraph of G induced by the vertices in T_1. 455 00:38:43,000 --> 00:38:55,000 OK, that is, V_1 is just the vertices in T_1 456 00:38:55,000 --> 00:39:09,000 is what it means to be induced. OK, so V_1 is the vertices in 457 00:39:09,000 --> 00:39:12,000 T_1. So, in this picture, 458 00:39:12,000 --> 00:39:16,000 I didn't label it. This is T_1. 459 00:39:16,000 --> 00:39:20,000 This is T_2. In this picture, 460 00:39:20,000 --> 00:39:27,000 these are the vertices of T_1. So, that's V_1, 461 00:39:27,000 --> 00:39:32,000 OK? And, E_1 is the set of pairs of 462 00:39:32,000 --> 00:39:39,000 vertices, x and y, that are the edges that are in 463 00:39:39,000 --> 00:39:47,000 E_1 such that both x and y belong to V_1. 464 00:39:47,000 --> 00:39:49,000 OK, so I haven't shown the edges of G here. 465 00:39:49,000 --> 00:39:52,000 But basically, if an edge went from here to 466 00:39:52,000 --> 00:39:57,000 here, that would be in the E_1. If it went from here to here, 467 00:39:57,000 --> 00:40:00,000 it would not. And if it went from here to 468 00:40:00,000 --> 00:40:04,000 here, it would not. OK, so the vertices, 469 00:40:04,000 --> 00:40:10,000 the subgraph induced by the vertices of T_1 are just those 470 00:40:10,000 --> 00:40:17,000 that connect up things in T_1, and similarly for T_2. 471 00:40:27,000 --> 00:40:33,000 So, the theorem says that if I look at just the edges within 472 00:40:33,000 --> 00:40:38,000 the graph here, G_1, those that are induced by 473 00:40:38,000 --> 00:40:41,000 these vertices, T_1 is, in fact, 474 00:40:41,000 --> 00:40:46,000 a minimum spanning tree for that subgraph. 475 00:40:46,000 --> 00:40:51,000 That's what the theorem says. OK, if I look over here 476 00:40:51,000 --> 00:40:58,000 conversely, or correspondingly, if I look at the set of edges 477 00:40:58,000 --> 00:41:05,000 that are induced by this set of vertices, the vertices in T_2, 478 00:41:05,000 --> 00:41:13,000 in fact, T_2 is a minimum spanning tree on that subgraph. 479 00:41:13,000 --> 00:41:17,000 OK, OK, we can even do it over here. 480 00:41:17,000 --> 00:41:21,000 If I took a look, for example, 481 00:41:21,000 --> 00:41:27,000 at these, let's see, let's say we cut out five, 482 00:41:27,000 --> 00:41:35,000 and if I cut out edge five, that T_1 would be these four 483 00:41:35,000 --> 00:41:40,000 vertices here. And, the point is that if I 484 00:41:40,000 --> 00:41:44,000 look at the subgraph induced on that, that these edges here. 485 00:41:44,000 --> 00:41:48,000 In fact, the six, eight, and three are all edges 486 00:41:48,000 --> 00:41:52,000 in a minimum spanning tree for that subgraph. 487 00:41:52,000 --> 00:41:54,000 OK, so that's what the theorem says. 488 00:41:54,000 --> 00:41:57,000 So let's prove it. 489 00:42:09,000 --> 00:42:19,000 OK, and so what technique are we going to use to prove it? 490 00:42:19,000 --> 00:42:28,000 OK, we learned this technique last time: hint, 491 00:42:28,000 --> 00:42:33,000 hint. It's something you do it in 492 00:42:33,000 --> 00:42:39,000 your text editor all the time: cut and paste, 493 00:42:39,000 --> 00:42:45,000 good, cut and paste. OK, so the weight of T I can 494 00:42:45,000 --> 00:42:51,000 express as the weight of the edge I removed, 495 00:42:51,000 --> 00:42:57,000 plus the weight of T_1, plus the weight of T_2. 496 00:42:57,000 --> 00:43:07,000 OK, so that's the total weight. So, the argument is pretty 497 00:43:07,000 --> 00:43:13,000 simple. Suppose that there were some 498 00:43:13,000 --> 00:43:20,000 T_1 prime that was better than T_1 for G_1. 499 00:43:20,000 --> 00:43:31,000 Suppose I had some better way of forming a spanning tree. 500 00:43:31,000 --> 00:43:42,000 OK, then I would make up a T prime, which just contained the 501 00:43:42,000 --> 00:43:48,000 edges, (u,v), and T_1 prime, 502 00:43:48,000 --> 00:43:53,000 union T_2. So, I would take, 503 00:43:53,000 --> 00:44:05,000 if I had a better spanning tree, a spanning tree of lower 504 00:44:05,000 --> 00:44:12,000 weight for T_1. And I call that T_1 prime. 505 00:44:12,000 --> 00:44:17,000 I just substitute that and make up a spanning tree that 506 00:44:17,000 --> 00:44:22,000 consisted of my edge, (u,v), whatever works well for 507 00:44:22,000 --> 00:44:26,000 T_1 prime and whatever works well for T. 508 00:44:26,000 --> 00:44:30,000 And, that would be a spanning tree. 509 00:44:30,000 --> 00:44:36,000 And it would be better than T itself was for G, 510 00:44:36,000 --> 00:44:44,000 OK, because the weight of these is just as the weight for this, 511 00:44:44,000 --> 00:44:50,000 I now just get to use the weight of T_1 prime, 512 00:44:50,000 --> 00:44:54,000 and that's less. And so, therefore, 513 00:44:54,000 --> 00:45:02,000 the assumption that T was a minimum spanning tree would be 514 00:45:02,000 --> 00:45:11,000 violated if I could find a better one for the subpiece. 515 00:45:11,000 --> 00:45:16,000 So, we have this nice property of optimal substructure. 516 00:45:16,000 --> 00:45:20,000 OK, I have subproblems that exhibit optimal, 517 00:45:20,000 --> 00:45:25,000 if I have a globally optimal solution to the whole problem 518 00:45:25,000 --> 00:45:31,000 within it, I can find optimal solutions to subproblems. 519 00:45:31,000 --> 00:45:36,000 So, now the question is, that's one hallmark. 520 00:45:36,000 --> 00:45:41,000 That's one hallmark of dynamic programming. 521 00:45:41,000 --> 00:45:45,000 What about overlapping subproblems? 522 00:45:45,000 --> 00:45:51,000 Do I have that property? Do I have overlapping 523 00:45:51,000 --> 00:45:58,000 subproblems over here for this type of problem? 524 00:46:19,000 --> 00:46:20,000 So, imagine, for example, 525 00:46:20,000 --> 00:46:22,000 that I'm removing different edges. 526 00:46:22,000 --> 00:46:26,000 I look at the space of taking a given edge, and removing it. 527 00:46:26,000 --> 00:46:30,000 It partitions it into two pieces, and now I have another 528 00:46:30,000 --> 00:46:32,000 piece. And I remove it, 529 00:46:32,000 --> 00:46:35,000 etc. Am I going to end up getting a 530 00:46:35,000 --> 00:46:38,000 bunch of subproblems that are similar in there? 531 00:46:38,000 --> 00:46:41,000 Yeah, I am. OK, if I take out this one, 532 00:46:41,000 --> 00:46:43,000 then I take out, say, this one here, 533 00:46:43,000 --> 00:46:46,000 and then I'll have another tree here and here. 534 00:46:46,000 --> 00:46:51,000 OK, that would be the same as if I had originally taken this 535 00:46:51,000 --> 00:46:53,000 out, and then taken that one out. 536 00:46:53,000 --> 00:46:57,000 If I look at simple ordering of taking out the edges, 537 00:46:57,000 --> 00:47:00,000 I'm going to end up with a whole bunch of overlapping 538 00:47:00,000 --> 00:47:04,000 subproblems. Yeah, OK. 539 00:47:04,000 --> 00:47:14,000 So then, what does that suggest we use as an approach? 540 00:47:14,000 --> 00:47:18,000 Dynamic programming, good. 541 00:47:18,000 --> 00:47:26,000 What a surprise! Yes, OK, you could use dynamic 542 00:47:26,000 --> 00:47:33,000 programming. But it turns out that minimum 543 00:47:33,000 --> 00:47:41,000 spanning tree exhibits an even more powerful property. 544 00:47:41,000 --> 00:47:48,000 OK, so we'(V,E) got all the clues for dynamic programming, 545 00:47:48,000 --> 00:47:57,000 but it turns out that there's an even bigger clue that's going 546 00:47:57,000 --> 00:48:05,000 to help us to use an even more powerful technique. 547 00:48:05,000 --> 00:48:11,000 And that, we call, the hallmark for greedy 548 00:48:11,000 --> 00:48:13,000 algorithms. 549 00:48:32,000 --> 00:48:41,000 And that is, we have a thing called the 550 00:48:41,000 --> 00:48:53,000 greedy choice property, which says that a locally 551 00:48:53,000 --> 00:49:03,000 optimal choice is globally optimal. 552 00:49:03,000 --> 00:49:05,000 And, of course, as all these hallmarks is the 553 00:49:05,000 --> 00:49:09,000 kind of thing you want to box, OK, because these are the clues 554 00:49:09,000 --> 00:49:12,000 that you're going to be able to do that. 555 00:49:12,000 --> 00:49:15,000 So, we have this property that we call the greedy choice 556 00:49:15,000 --> 00:49:18,000 property. I'm going to show you how it 557 00:49:18,000 --> 00:49:21,000 works in this case. And when you have a greedy 558 00:49:21,000 --> 00:49:24,000 choice property, it turns out you can do even 559 00:49:24,000 --> 00:49:29,000 better that dynamic programming. OK, so when you see the two 560 00:49:29,000 --> 00:49:33,000 dynamic programming properties, there is a clue that says 561 00:49:33,000 --> 00:49:36,000 dynamic programming, yes, but also it says, 562 00:49:36,000 --> 00:49:41,000 let me see whether it also has this greedy property because if 563 00:49:41,000 --> 00:49:46,000 it does, you're going to come up with something that's even 564 00:49:46,000 --> 00:49:49,000 better than dynamic programming, OK? 565 00:49:49,000 --> 00:49:53,000 So, if you just have the two, you can usually do dynamic 566 00:49:53,000 --> 00:49:56,000 programming, but if you have this third one, 567 00:49:56,000 --> 00:50:00,000 it's like, whoa! Jackpot! 568 00:50:00,000 --> 00:50:04,000 OK, so here's the theorem we'll prove to illustrate this idea. 569 00:50:04,000 --> 00:50:08,000 Once again, these are not, all these hallmarks are not 570 00:50:08,000 --> 00:50:09,000 things. They are heuristics. 571 00:50:09,000 --> 00:50:14,000 I can't give you an algorithm to say, here's where dynamic 572 00:50:14,000 --> 00:50:16,000 programming works, or here's where greedy 573 00:50:16,000 --> 00:50:20,000 algorithms work. But I can sort of indicate when 574 00:50:20,000 --> 00:50:23,000 they work, the kind of structure they have. 575 00:50:23,000 --> 00:50:32,000 OK, so here's the theorem. So let's let T be the MST of 576 00:50:32,000 --> 00:50:40,000 our graph. And, let's let A be any subset 577 00:50:40,000 --> 00:50:49,000 of V, so, some subset of vertices. 578 00:50:49,000 --> 00:51:04,000 And now, let's suppose that edge, (u,v), is the least weight 579 00:51:04,000 --> 00:51:17,000 edge connecting our set A to A complement, that is, 580 00:51:17,000 --> 00:51:27,000 V minus A. Then the theorem says that 581 00:51:27,000 --> 00:51:39,000 (u,v) is in the minimum spanning tree. 582 00:51:39,000 --> 00:51:43,000 So let's just take a look at our graph over here and see if 583 00:51:43,000 --> 00:51:45,000 that's, in fact, the case. 584 00:51:45,000 --> 00:51:49,000 OK, so let's take, so one thing I could do for A 585 00:51:49,000 --> 00:51:53,000 is just take a singleton node. So, I take a singleton node, 586 00:51:53,000 --> 00:51:56,000 let's say this guy here, that can be my A, 587 00:51:56,000 --> 00:52:00,000 and everything else is V minus A. 588 00:52:00,000 --> 00:52:04,000 And I look at the least weight edge connecting this to 589 00:52:04,000 --> 00:52:07,000 everything else. Well, there are only two edges 590 00:52:07,000 --> 00:52:10,000 that connect it to everything else. 591 00:52:10,000 --> 00:52:15,000 And the theorem says that the lighter one is in the minimum 592 00:52:15,000 --> 00:52:17,000 spanning tree. Hey, I win. 593 00:52:17,000 --> 00:52:21,000 OK, if you take a look, every vertex that I pick, 594 00:52:21,000 --> 00:52:25,000 the latest edge coming out of that vertex is in the minimum 595 00:52:25,000 --> 00:52:29,000 spanning tree. OK, the lightest weight edge 596 00:52:29,000 --> 00:52:35,000 coming out, but that's not all the edges that are in here. 597 00:52:35,000 --> 00:52:39,000 OK, or let's just imagine, let's take a look at these 598 00:52:39,000 --> 00:52:43,000 three vertices connected to this set of vertices. 599 00:52:43,000 --> 00:52:46,000 I have three edges is going across. 600 00:52:46,000 --> 00:52:50,000 The least weight one is five. That's the minimum spanning 601 00:52:50,000 --> 00:52:53,000 tree. Or, I can cut it this way. 602 00:52:53,000 --> 00:52:57,000 OK, the ones above one, the edges going down are seven, 603 00:52:57,000 --> 00:53:02,000 eight, and 14. Seven is the least weight. 604 00:53:02,000 --> 00:53:04,000 It's in the minimum spanning tree. 605 00:53:04,000 --> 00:53:08,000 So, no matter how I choose, I could make this one in, 606 00:53:08,000 --> 00:53:10,000 this one out, this one in, 607 00:53:10,000 --> 00:53:12,000 this one out, this one in, 608 00:53:12,000 --> 00:53:14,000 this one out, this one in, 609 00:53:14,000 --> 00:53:18,000 this one out, take a look at what all the 610 00:53:18,000 --> 00:53:21,000 edges are. Which ever one to the least 611 00:53:21,000 --> 00:53:24,000 weight: it's in the minimum spanning tree. 612 00:53:24,000 --> 00:53:28,000 So, in some sense, that's a local property because 613 00:53:28,000 --> 00:53:34,000 I don't have to look at what the rest of the tree is. 614 00:53:34,000 --> 00:53:38,000 I'm just looking at some small set of vertices if I wish, 615 00:53:38,000 --> 00:53:42,000 and I say, well, if I wanted to connect that set 616 00:53:42,000 --> 00:53:46,000 of vertices to the rest of the world, what would I pick? 617 00:53:46,000 --> 00:53:50,000 I'd pick the cheapest one. That's the greedy approach. 618 00:53:50,000 --> 00:53:53,000 It turns out, that wins, OK, 619 00:53:53,000 --> 00:53:57,000 that picking that thing that's locally good for that subset, 620 00:53:57,000 --> 00:54:04,000 A, OK, is also globally good. OK, it optimizes the overall 621 00:54:04,000 --> 00:54:09,000 function. That's what the theorem says, 622 00:54:09,000 --> 00:54:13,000 OK? So, let's prove this theorem. 623 00:54:13,000 --> 00:54:20,000 Any questions about this? OK, let's prove this theorem. 624 00:54:20,000 --> 00:54:27,000 So, we have (u,v) is the least weight edge connecting A to D 625 00:54:27,000 --> 00:54:32,000 minus A. So, let's suppose that this 626 00:54:32,000 --> 00:54:40,000 edge, (u,v), is not in the minimum spanning tree. 627 00:54:40,000 --> 00:54:45,000 OK, let's suppose that somehow there is a minimum spanning tree 628 00:54:45,000 --> 00:54:50,000 that doesn't include this least weight edge. 629 00:54:50,000 --> 00:54:55,000 OK, so what technique you think will use to prove to get a 630 00:54:55,000 --> 00:54:58,000 contradiction here? Cut and paste, 631 00:54:58,000 --> 00:55:04,000 good. Yeah, we're going to cut paste. 632 00:55:04,000 --> 00:55:08,000 OK, we're going to cut and paste. 633 00:55:08,000 --> 00:55:14,000 So here, I did an example. OK, so -- 634 00:55:40,000 --> 00:55:42,000 OK, and so I'm going to use the notation. 635 00:55:42,000 --> 00:55:44,000 I'm going to color some of these in. 636 00:56:05,000 --> 00:56:10,000 OK, and so my notation here is this is an element of A, 637 00:56:10,000 --> 00:56:14,000 and color it in. It's an element of V minus A. 638 00:56:14,000 --> 00:56:18,000 OK, so if it's not colored it, that's an A. 639 00:56:18,000 --> 00:56:21,000 This is my minimum spanning tree. 640 00:56:21,000 --> 00:56:27,000 Once again, I'm not showing the overall edges of all the graphs, 641 00:56:27,000 --> 00:56:30,000 but they're there, OK? 642 00:56:30,000 --> 00:56:33,000 So, my edge, (u,v), which is not my minimum 643 00:56:33,000 --> 00:56:38,000 spanning tree I say, let's say is this edge here. 644 00:56:38,000 --> 00:56:42,000 It's an edge from u, u as in A, v as in V minus A. 645 00:56:42,000 --> 00:56:48,000 OK, so everybody see the setup? So, I want to prove that this 646 00:56:48,000 --> 00:56:52,000 edge should have been in the minimum spanning tree, 647 00:56:52,000 --> 00:56:58,000 OK, that the contention that this is a minimum spanning tree, 648 00:56:58,000 --> 00:57:02,000 and does include (u,v) is wrong. 649 00:57:02,000 --> 00:57:05,000 So, what I want to do, that, is I have a tree here, 650 00:57:05,000 --> 00:57:08,000 T, and I have two vertices, u and v, and in a tree, 651 00:57:08,000 --> 00:57:12,000 between any two vertices there is a unique, simple path: 652 00:57:12,000 --> 00:57:16,000 simple path meaning it doesn't go back and forth and repeat 653 00:57:16,000 --> 00:57:18,000 edges or vertices. OK, there's a unique, 654 00:57:18,000 --> 00:57:23,000 simple path from u to v. So, let's consider that path. 655 00:57:42,000 --> 00:57:46,000 OK, and the way that I know that that path exists is because 656 00:57:46,000 --> 00:57:51,000 I'(V,E) read appendix B of the textbook, section B.5.1, 657 00:57:51,000 --> 00:57:56,000 OK, which has this nice theorem about properties of trees. 658 00:57:56,000 --> 00:58:00,000 OK, so that's how I know that there exists a unique, 659 00:58:00,000 --> 00:58:05,000 simple path. OK, so now we're going to do is 660 00:58:05,000 --> 00:58:09,000 take a look at that path. So in this case, 661 00:58:09,000 --> 00:58:13,000 it goes from here, to here, to here, 662 00:58:13,000 --> 00:58:16,000 to here. And along that path, 663 00:58:16,000 --> 00:58:22,000 there must be a point where I connect from a vertex in A to a 664 00:58:22,000 --> 00:58:25,000 vertex in V minus A. Why? 665 00:58:25,000 --> 00:58:32,000 Well, because this is in A. This is in V minus A. 666 00:58:32,000 --> 00:58:42,000 So, along the path somewhere, there must be a transition. 667 00:58:42,000 --> 00:58:52,000 OK, they are not all in A, OK, because in particular, 668 00:58:52,000 --> 00:58:58,000 V isn't. OK, so we're going to do is 669 00:58:58,000 --> 00:59:09,000 swap (u,v) with the first edge on this path that connects a 670 00:59:09,000 --> 00:59:18,000 vertex in A to a vertex in V minus A. 671 00:59:18,000 --> 00:59:20,000 So in this case, it's this edge here. 672 00:59:20,000 --> 00:59:24,000 I go from A to V minus A. In general, I might be 673 00:59:24,000 --> 00:59:28,000 alternating many times, OK, and I just picked the first 674 00:59:28,000 --> 00:59:32,000 one that I encounter. OK, that this guy here. 675 00:59:32,000 --> 00:59:36,000 And what I do is I put this edge in. 676 00:59:36,000 --> 00:59:38,000 OK, so then, what happens? 677 00:59:38,000 --> 00:59:42,000 Well, the edge, (u,v), is the lightest thing 678 00:59:42,000 --> 00:59:46,000 connecting something in A to something in V minus A. 679 00:59:46,000 --> 00:59:49,000 So that means, in particular, 680 00:59:49,000 --> 00:59:53,000 it's lighter than this edge, has lower weight. 681 00:59:53,000 --> 00:59:57,000 So, by swapping this, I'(V,E) created a tree with 682 00:59:57,000 --> 01:00:02,198 lower overall weight, contradicting the assumption 683 01:00:02,198 --> 01:00:08,000 that this other thing was a minimum spanning tree. 684 01:00:08,000 --> 01:00:14,219 OK: so, a lower weight spanning tree than T results, 685 01:00:14,219 --> 01:00:18,000 and that's a contradiction -- 686 01:00:25,000 --> 01:00:33,010 -- than T results. And that's a contradiction, 687 01:00:33,010 --> 01:00:36,570 OK? How are we doing? 688 01:00:36,570 --> 01:00:44,225 Everybody with me? OK, now we get to do some 689 01:00:44,225 --> 01:00:46,895 algorithms. Yea! 690 01:00:46,895 --> 01:00:55,439 So, we are going to do an algorithm called Prim's 691 01:00:55,439 --> 01:01:01,853 algorithm. Prim eventually became a very 692 01:01:01,853 --> 01:01:07,069 high-up at AT&T because he invented this algorithm for 693 01:01:07,069 --> 01:01:12,187 minimum spanning trees, and it was used in all of the 694 01:01:12,187 --> 01:01:15,730 billing code for AT&T for many years. 695 01:01:15,730 --> 01:01:21,438 He was very high up at Bell Labs back in the heyday of Bell 696 01:01:21,438 --> 01:01:24,784 Laboratories. OK, so it just shows, 697 01:01:24,784 --> 01:01:30,000 all you have to do is invent an algorithm. 698 01:01:30,000 --> 01:01:36,702 You too can be a president of a corporate monopoly. 699 01:01:36,702 --> 01:01:43,807 Of course, the government can do things to monopolies, 700 01:01:43,807 --> 01:01:49,438 but anyway, if that's your mission in life, 701 01:01:49,438 --> 01:01:55,202 invent an algorithm. OK, so here's the idea. 702 01:01:55,202 --> 01:02:03,648 What we're going to do is we're going to maintain V minus A as a 703 01:02:03,648 --> 01:02:11,923 priority queue. We'll call it Q. 704 01:02:11,923 --> 01:02:26,076 And each vertex, we're going to key each vertex 705 01:02:26,076 --> 01:02:39,923 in Q with the weight of the least weight edge, 706 01:02:39,923 --> 01:02:53,280 connecting it to a vertex in A. So here's the code. 707 01:02:53,280 --> 01:03:00,000 So, we're going to start out with Q being all vertices. 708 01:03:00,000 --> 01:03:03,873 So, we start out with A being, if you will, 709 01:03:03,873 --> 01:03:07,930 the empty set. OK, and what we're going to do 710 01:03:07,930 --> 01:03:13,095 it is the least weight edge, therefore, for everything in 711 01:03:13,095 --> 01:03:18,536 the priority queue is basically going to be infinity because 712 01:03:18,536 --> 01:03:23,700 none of them have any edges. The least weight edge to the 713 01:03:23,700 --> 01:03:29,325 empty set is going to be empty. And then, we're going to start 714 01:03:29,325 --> 01:03:33,958 out with one guy. We'll call him S, 715 01:03:33,958 --> 01:03:39,489 which will set to zero for some arbitrary S in V. 716 01:03:39,489 --> 01:03:45,135 And then, the main part of the algorithm kicks in. 717 01:03:45,135 --> 01:03:51,703 So that's our initialization. OK, when we do the analysis, 718 01:03:51,703 --> 01:03:58,271 I'm going to write some stuff on the left hand side of the 719 01:03:58,271 --> 01:04:04,406 board. So if you're taking notes, 720 01:04:04,406 --> 01:04:14,406 you may want to also leave a little bit of space on the left 721 01:04:14,406 --> 01:04:22,711 hand side of your notes. So, while Q is not empty, 722 01:04:22,711 --> 01:04:30,000 we get the smallest element out of it. 723 01:04:41,000 --> 01:04:43,000 And then we do some stuff. 724 01:05:19,000 --> 01:05:21,970 That's it. And the only thing I should 725 01:05:21,970 --> 01:05:25,503 mention here is, OK, so let's just see what's 726 01:05:25,503 --> 01:05:28,875 going on here. And then we'll run it on the 727 01:05:28,875 --> 01:05:32,256 example. OK, so what we do is we take 728 01:05:32,256 --> 01:05:36,609 out the smallest element out of the queue at each step. 729 01:05:36,609 --> 01:05:40,156 And then for each step in the adjacency list, 730 01:05:40,156 --> 01:05:43,783 in other words, everything for which I have an 731 01:05:43,783 --> 01:05:46,846 edge going from v to u, we take a look, 732 01:05:46,846 --> 01:05:51,440 and if v is still in our set V minus A, so things we'(V,E) 733 01:05:51,440 --> 01:05:54,261 taken out are going to be part of A. 734 01:05:54,261 --> 01:05:57,163 OK, every time we take something out, 735 01:05:57,163 --> 01:06:02,000 that's going to be a new A that we construct. 736 01:06:02,000 --> 01:06:04,258 At every step, we want to find, 737 01:06:04,258 --> 01:06:08,400 what's the cheapest edge connecting that A to everything 738 01:06:08,400 --> 01:06:11,035 else? We basically are going to take 739 01:06:11,035 --> 01:06:15,025 whatever that cheapest thing is, OK, add that edge in, 740 01:06:15,025 --> 01:06:19,242 and now bring that into A and find the next cheapest one. 741 01:06:19,242 --> 01:06:22,103 And we just keep repeating the process. 742 01:06:22,103 --> 01:06:25,567 OK, we'll do it on the example. And what we do, 743 01:06:25,567 --> 01:06:28,955 is every time we bring it in, I keep track of, 744 01:06:28,955 --> 01:06:34,000 what was the vertex responsible for bringing me in. 745 01:06:34,000 --> 01:06:43,947 And what I claim is that at the end, if I look at the set of 746 01:06:43,947 --> 01:06:52,209 these pairs that I'(V,E) made here, V and pi of V, 747 01:06:52,209 --> 01:06:58,279 that forms the minimum spanning tree. 748 01:06:58,279 --> 01:07:05,441 So let's just do this. And, what's that? 749 01:07:05,441 --> 01:07:12,191 We're all set up. So let's get rid of these guys 750 01:07:12,191 --> 01:07:20,234 here because we are going to recompute them from scratch. 751 01:07:20,234 --> 01:07:30,000 OK, so you may want to copy the graph over again in your notes. 752 01:07:30,000 --> 01:07:34,840 I was going to do it, but it turned out, 753 01:07:34,840 --> 01:07:40,797 this is exactly the board is going to erase this. 754 01:07:40,797 --> 01:07:47,127 OK, well let me just modify it. OK, so we start out. 755 01:07:47,127 --> 01:07:54,574 We make everything be infinity. OK, so that's where I'm going 756 01:07:54,574 --> 01:08:01,028 to keep the key value. OK, and then what I'm going to 757 01:08:01,028 --> 01:08:07,952 do is find one vertex. And I'm going to call him S. 758 01:08:07,952 --> 01:08:11,749 And I'm going to do this vertex here. 759 01:08:11,749 --> 01:08:15,018 We'll call that S. So basically, 760 01:08:15,018 --> 01:08:19,447 I now make him be zero. And now, what I do, 761 01:08:19,447 --> 01:08:23,453 is I execute extract min. So basically, 762 01:08:23,453 --> 01:08:28,198 what I'll do is I'll just shade him like this, 763 01:08:28,198 --> 01:08:34,000 indicating that he has now joined the set A. 764 01:08:34,000 --> 01:08:40,931 So, this is going to be A. And this is element of V minus 765 01:08:40,931 --> 01:08:44,644 A. OK, so then what we do is we 766 01:08:44,644 --> 01:08:47,986 take a look. We extract him, 767 01:08:47,986 --> 01:08:53,433 and then for each edge in the adjacency list, 768 01:08:53,433 --> 01:08:59,002 OK, so for each vertex in the adjacency lists, 769 01:08:59,002 --> 01:09:05,315 that these guys here, OK, we're going to look to see 770 01:09:05,315 --> 01:09:12,000 if it's still in Q, that is, in V minus A. 771 01:09:12,000 --> 01:09:16,795 And if so, and its key value is less than what the value is at 772 01:09:16,795 --> 01:09:20,254 the edge, there, we're going to replace it by 773 01:09:20,254 --> 01:09:22,770 the edge value. So, in this case, 774 01:09:22,770 --> 01:09:25,600 we're going to replace this by seven. 775 01:09:25,600 --> 01:09:30,317 We're going to replace this by 15, and we're going to replace 776 01:09:30,317 --> 01:09:33,854 this by ten, OK, because what we're interested 777 01:09:33,854 --> 01:09:39,608 in is, what is the cheapest? Now, notice that everything in 778 01:09:39,608 --> 01:09:43,782 V minus A, that is, what's in the priority queue, 779 01:09:43,782 --> 01:09:48,216 everything in there, OK, now has its cheapest way of 780 01:09:48,216 --> 01:09:53,086 connecting it to the things that I'(V,E) already removed, 781 01:09:53,086 --> 01:09:57,173 the things that are in A. OK, and so now I just, 782 01:09:57,173 --> 01:10:01,608 OK, when I actually do that update, there's actually 783 01:10:01,608 --> 01:10:07,000 something implicit going on in this priority queue. 784 01:10:07,000 --> 01:10:10,636 And that is that I have to do a decreased key. 785 01:10:10,636 --> 01:10:14,111 So, there's an implicit decrease of the key. 786 01:10:14,111 --> 01:10:19,121 So, decreased key is a priority queue operation that lowers the 787 01:10:19,121 --> 01:10:22,191 value of the key in the priority queue. 788 01:10:22,191 --> 01:10:26,878 And so, that's implicitly going on when I look at what data 789 01:10:26,878 --> 01:10:31,646 structure I'm going to use to implement that priority queue. 790 01:10:31,646 --> 01:10:36,171 OK, so common data structures for implementing a priority 791 01:10:36,171 --> 01:10:41,376 queue are a min heap. OK, so I have to make sure that 792 01:10:41,376 --> 01:10:43,905 I'm actually doing this operation. 793 01:10:43,905 --> 01:10:47,355 I can't just change it and not affect my heap. 794 01:10:47,355 --> 01:10:51,111 So, there is an implicit operation going on there. 795 01:10:51,111 --> 01:10:54,407 OK, now I repeat. I find the cheapest thing, 796 01:10:54,407 --> 01:10:58,547 oh, and I also have to set, now, a pointer from each of 797 01:10:58,547 --> 01:11:02,931 these guys back to u. So here, this guy sets a 798 01:11:02,931 --> 01:11:07,114 pointer going this way. This guy sets a pointer going 799 01:11:07,114 --> 01:11:11,298 this way, and this guy sets a pointer going this way. 800 01:11:11,298 --> 01:11:16,206 That's my pi thing that's going to keep track of who caused me 801 01:11:16,206 --> 01:11:20,873 to set my value to what it is. So now, we go in and we find 802 01:11:20,873 --> 01:11:22,885 the cheapest thing, again. 803 01:11:22,885 --> 01:11:25,620 And we're going to do it fast, too. 804 01:11:25,620 --> 01:11:32,361 OK, this is a fast algorithm. OK, so now we're going to go do 805 01:11:32,361 --> 01:11:36,481 this again. So now, what's the cheapest 806 01:11:36,481 --> 01:11:39,843 thing to extract? This guy here, 807 01:11:39,843 --> 01:11:42,987 right? So, we'll take him out, 808 01:11:42,987 --> 01:11:47,542 OK, and now we update all of his neighbors. 809 01:11:47,542 --> 01:11:51,771 So this guy gets five. This guy gets 12. 810 01:11:51,771 --> 01:11:56,542 This guy gets nine. This guy we don't update. 811 01:11:56,542 --> 01:12:02,722 We don't update him because he's no longer in the priority 812 01:12:02,722 --> 01:12:07,464 queue. And all of these guys now, 813 01:12:07,464 --> 01:12:12,297 we make point to where they're supposed to point to. 814 01:12:12,297 --> 01:12:17,983 And, we're done with that step. Now we find the cheapest one. 815 01:12:17,983 --> 01:12:22,437 What's the cheapest one now? The five over here. 816 01:12:22,437 --> 01:12:24,807 Good. So, we take him out. 817 01:12:24,807 --> 01:12:30,019 OK, we update the neighbors. Here, yep, that goes to six 818 01:12:30,019 --> 01:12:34,000 now. And, we have that pointer. 819 01:12:34,000 --> 01:12:39,684 And, this guy we don't do, because he's not in there. 820 01:12:39,684 --> 01:12:44,604 This guy becomes 14, and this guy here becomes 821 01:12:44,604 --> 01:12:47,774 eight. So, we update that guy, 822 01:12:47,774 --> 01:12:52,803 make him be eight. Did I do this the right way? 823 01:12:52,803 --> 01:12:57,395 Yeah, because pi is a function of this guy. 824 01:12:57,395 --> 01:13:00,675 So basically, this thing, then, 825 01:13:00,675 --> 01:13:04,938 disappears. Yeah, did I have another one 826 01:13:04,938 --> 01:13:09,258 that I missed? 12, yes, good, 827 01:13:09,258 --> 01:13:12,584 it's removed, OK, because pi is just a 828 01:13:12,584 --> 01:13:14,741 function. And now I'm OK. 829 01:13:14,741 --> 01:13:18,516 OK, so now what do I do? OK, so now my set, 830 01:13:18,516 --> 01:13:23,191 A, consists of these three things, and now I want the 831 01:13:23,191 --> 01:13:26,786 cheapest edge. I know it's in the minimum 832 01:13:26,786 --> 01:13:30,561 spanning tree. So let me just greedily pick 833 01:13:30,561 --> 01:13:34,554 it. OK, so what's the cheapest 834 01:13:34,554 --> 01:13:37,108 thing now? This guy appear? 835 01:13:37,108 --> 01:13:39,466 Yeah, six. So we take it. 836 01:13:39,466 --> 01:13:44,771 We go to update these things, and nothing matters here. 837 01:13:44,771 --> 01:13:50,175 OK, nothing changes because these guys are already in A. 838 01:13:50,175 --> 01:13:54,202 OK, so now the cheapest one is eight here. 839 01:13:54,202 --> 01:13:56,856 Good. So, we take eight out. 840 01:13:56,856 --> 01:14:01,656 OK, we update this. Nothing to be done. 841 01:14:01,656 --> 01:14:04,970 This: nothing to be done. This: oh, no, 842 01:14:04,970 --> 01:14:09,242 this one, instead of 14 we can make this be three. 843 01:14:09,242 --> 01:14:14,212 So, we get rid of that pointer and make it point that way. 844 01:14:14,212 --> 01:14:16,915 Now three is the cheapest thing. 845 01:14:16,915 --> 01:14:21,100 So, we take it out, and of course there's nothing 846 01:14:21,100 --> 01:14:24,239 to be done over there. And now, last, 847 01:14:24,239 --> 01:14:26,506 I take nine. And it's done. 848 01:14:26,506 --> 01:14:32,000 And 15: it's done. And the algorithm terminates. 849 01:14:32,000 --> 01:14:36,972 OK, and as I look at, now, all the edges that I 850 01:14:36,972 --> 01:14:43,135 picked, those are exactly all the edges that we had at the 851 01:14:43,135 --> 01:14:48,000 beginning. OK, let's do an analysis here. 852 01:14:58,000 --> 01:15:06,316 OK, so let's see, this part here costs me order 853 01:15:06,316 --> 01:15:11,197 V, right? OK, and this part, 854 01:15:11,197 --> 01:15:16,983 let's see what we are doing here. 855 01:15:16,983 --> 01:15:27,107 Well, we're going to go through this loop how many times? 856 01:15:27,107 --> 01:15:32,711 V times. It's V elements we put into the 857 01:15:32,711 --> 01:15:35,860 queue. We are not inserting anything. 858 01:15:35,860 --> 01:15:39,795 We're just taking them out. This goes V times, 859 01:15:39,795 --> 01:15:43,818 OK, and we do a certain number of extract Mins. 860 01:15:43,818 --> 01:15:47,492 So, we're going to do order V extract Mins. 861 01:15:47,492 --> 01:15:52,915 And then we go to the adjacency list, and we have some constant 862 01:15:52,915 --> 01:15:55,801 things. But we have these implicit 863 01:15:55,801 --> 01:16:00,000 decreased keys for this stuff here. 864 01:16:00,000 --> 01:16:07,412 That's this thing here. OK, and so how many implicit 865 01:16:07,412 --> 01:16:14,389 decreased keys do we have? That's going to be the 866 01:16:14,389 --> 01:16:18,459 expensive thing. OK, we have, 867 01:16:18,459 --> 01:16:25,000 in this case, the degree of u of those. 868 01:16:25,000 --> 01:16:31,309 OK, so overall, how many implicit decreased 869 01:16:31,309 --> 01:16:38,218 keys do we have? Well, we have V times through. 870 01:16:38,218 --> 01:16:43,025 How big could the degree of u be? 871 01:16:43,025 --> 01:16:48,433 OK, it could be as big as V, order V. 872 01:16:48,433 --> 01:16:56,995 So, that's V^2 decreased use. But we can do a better bound 873 01:16:56,995 --> 01:17:04,189 than that. How many do we really have? 874 01:17:04,189 --> 01:17:11,948 Yeah, at most order E, OK, because what am I doing? 875 01:17:11,948 --> 01:17:19,086 I'm summing up the degrees of all the vertices. 876 01:17:19,086 --> 01:17:27,000 That's how many times I actually execute that. 877 01:17:27,000 --> 01:17:34,322 So, I have order E, implicit decreased keys. 878 01:17:34,322 --> 01:17:44,028 So the time overall is order V times time for whatever the 879 01:17:44,028 --> 01:17:53,224 extract Min is plus E times the time for decreased key. 880 01:17:53,224 --> 01:18:02,931 So now, let's look at data structures, and we can evaluate 881 01:18:02,931 --> 01:18:14,000 for different data structures what this formula gives us. 882 01:18:14,000 --> 01:18:21,492 So, we have different ways of implementing a data structure. 883 01:18:21,492 --> 01:18:28,222 We have the cost of extract Min, and of decreased key, 884 01:18:28,222 --> 01:18:32,636 and total. So, the simplest way of 885 01:18:32,636 --> 01:18:38,369 implementing a data structure is an unsorted array. 886 01:18:38,369 --> 01:18:44,904 If I have an unsorted array, how much time does it take me 887 01:18:44,904 --> 01:18:51,668 to extract the minimum element? If I have an unsorted array? 888 01:18:51,668 --> 01:18:58,433 Right, order V in this case because it's an array of size V. 889 01:18:58,433 --> 01:19:06,000 And, to do a decreased key, OK, I can do it in order one. 890 01:19:06,000 --> 01:19:14,245 So, the total is V^2, good, order V^2 algorithm. 891 01:19:14,245 --> 01:19:22,666 Or, as people suggested, how about a binary heap? 892 01:19:22,666 --> 01:19:33,017 OK, to do an extract Min in a binary heap will cost me what? 893 01:19:33,017 --> 01:19:38,905 O of log V. Decreased key will cost me, 894 01:19:38,905 --> 01:19:44,932 yeah, it turns out you can do that in order log V because 895 01:19:44,932 --> 01:19:49,668 basically you just have to shuffle the value, 896 01:19:49,668 --> 01:19:54,295 actually shuffle it up towards the root, OK? 897 01:19:54,295 --> 01:19:58,708 Or at log V. And, the total cost therefore 898 01:19:58,708 --> 01:20:01,717 is? E log V, good. 899 01:20:01,717 --> 01:20:06,869 Which of these is better? It depends, good. 900 01:20:06,869 --> 01:20:12,758 When is one better, and when is the other better? 901 01:20:12,758 --> 01:20:18,401 Yeah, if it's a dense graph, E is close to V^2, 902 01:20:18,401 --> 01:20:24,166 the array is better. But if it's a sparse graph, 903 01:20:24,166 --> 01:20:33,000 and E is much smaller than V^2, then the binary heap is better. 904 01:20:33,000 --> 01:20:37,824 So that motivated the invention of a data structure, 905 01:20:37,824 --> 01:20:43,216 OK, called a Fibonacci Heap. So, Fibonacci Heap is covered 906 01:20:43,216 --> 01:20:47,851 in Chapter 20 of CLRS. We're not going to hold you 907 01:20:47,851 --> 01:20:53,148 responsible for the content, but it's an interesting data 908 01:20:53,148 --> 01:20:57,878 structure because it's an amortized data structure. 909 01:20:57,878 --> 01:21:01,851 And it turns out that it is data structure, 910 01:21:01,851 --> 01:21:08,000 you can do extract Min in order log V amortized time. 911 01:21:08,000 --> 01:21:12,747 And remarkably, you can do decreased key in 912 01:21:12,747 --> 01:21:17,834 order one amortized. So, when I plug those in, 913 01:21:17,834 --> 01:21:21,000 what do I get over here? 914 01:21:34,000 --> 01:21:42,088 What's that going to be? Plug that it here. 915 01:21:42,088 --> 01:21:52,296 It's going to be V times log V plus E: E plus V log V. 916 01:21:52,296 --> 01:22:00,000 These are amortized, so what's this? 917 01:22:00,000 --> 01:22:02,317 Trick question. It's worst-case. 918 01:22:02,317 --> 01:22:05,979 It's not amortized over here. These are amortized, 919 01:22:05,979 --> 01:22:08,745 but that's the beauty of amortization. 920 01:22:08,745 --> 01:22:13,006 I can say it's going to be worst case: E plus V log V over 921 01:22:13,006 --> 01:22:17,715 here, because when I add up the amortized cost of my operations, 922 01:22:17,715 --> 01:22:20,480 it's an upper bound on the true costs. 923 01:22:20,480 --> 01:22:24,291 OK, so that's why I say, one of the beauties of this 924 01:22:24,291 --> 01:22:27,058 amortized analysis, and in particular, 925 01:22:27,058 --> 01:22:31,692 being able to assign different costs to different operations is 926 01:22:31,692 --> 01:22:37,000 I can just add them up and I get my worst-case costs. 927 01:22:37,000 --> 01:22:40,565 So this is already V log V. There are a couple other 928 01:22:40,565 --> 01:22:43,012 algorithms just before I let you go. 929 01:22:43,012 --> 01:22:47,066 Kruskal's Algorithm in the book uses another amortized data 930 01:22:47,066 --> 01:22:50,282 structure called a disjoint set data structure, 931 01:22:50,282 --> 01:22:53,498 which also runs in E log V, that is, this time: 932 01:22:53,498 --> 01:22:56,574 runs in this time, the same as using a binary 933 01:22:56,574 --> 01:23:00,000 heap. So, I'll refer you to the book. 934 01:23:00,000 --> 01:23:04,934 The best algorithm to date with this problem is done by our own 935 01:23:04,934 --> 01:23:09,233 David Karger on the faculty here with one of our former 936 01:23:09,233 --> 01:23:12,975 graduates, Phil Kline, who is now a professor at 937 01:23:12,975 --> 01:23:17,353 Brown, and Robert Tarjan, who is sort of like the master 938 01:23:17,353 --> 01:23:22,368 of all data structures who was a professor at Princeton in 1993. 939 01:23:22,368 --> 01:23:26,189 OK, it's a randomized algorithm, and it gives you 940 01:23:26,189 --> 01:23:32,000 order V plus E expected time. OK, so that's the best to date. 941 01:23:32,000 --> 01:23:36,300 It's still open as to whether there is a deterministic, 942 01:23:36,300 --> 01:23:40,679 there is worst-case bound, whether there is a worst-case 943 01:23:40,679 --> 01:23:45,059 bound that is linear time. OK, but there is a randomized 944 01:23:45,059 --> 01:23:47,369 to linear time, and otherwise, 945 01:23:47,369 --> 01:23:51,509 this is essentially the best bound without additional 946 01:23:51,509 --> 01:23:54,058 assumptions. OK, very cool stuff. 947 01:23:54,058 --> 01:23:58,675 Next, we're going to see a lot of these ideas of greedy and 948 01:23:58,675 --> 01:24:01,000 dynamic programming in practice.