The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
PROFESSOR: OK, welcome back.
This is the second half of our two lectures on distributed
algorithms this week.
I could turn that on, OK.
All right, I'll start with a quick preview.
This week, we're just looking at synchronous and asynchronous
distributed algorithms, not worrying
about interesting stuff like failures.
Last time we looked at leader election,
maximal independent set, breadth-first spanning trees,
and we started looking at shortest paths trees.
We'll finish that today, and then we'll
move on to the main topic for today, which is asynchronous
distributed algorithms where things start getting much more
complicated because not everything is going on
in synchronous rounds.
And we'll revisit the same two problems, breadth-first
and shortest paths spanning trees.
Quick review, all of our algorithms
are using a model that's based on undirected graph.
Using this notation, gamma, for the neighbors of a vertex.
We'll talk about the degree of a vertex.
We have a process, then, associated
with every vertex in the graph.
And we associate communication channels
in both directions on every edge.
Last time we looked at synchronous distributed
algorithms in the synchronous model.
We have the processes of the ground vertices,
they communicate using messages.
The processes have local ports that they
know by some kind of local name, and the ports
connect to their communication channel.
The algorithm executes in synchronous rounds
where, in every around, every process
is going to decide what to send on all of its ports,
and the messages then get put into the channel delivered
to the processes at the other end.
And everybody looks at all the messages they get at that round
all at once, and they determine a new state.
They compute a new state based on all
of those arriving messages.
We started with leader election.
I won't repeat the problem definition,
but here's the results that we got last time.
We looked at a special case of a graph that's
just a simple clique.
And if the processes don't have any distinguishing information,
like unique identifiers, and they're deterministic,
then there's no way to break the symmetry and you can prove
an impossibility result that says that you
can't-- in a deterministic, indistinguishable case,
you can't guarantee to elect a leader in this kind of a graph.
Just as an aside, I should say that distributed computing
theory is just filled with impossibility results,
and they're all based on this limitation of distributed
computing where each node only knows
what's happening to itself and in its neighborhood.
Nobody knows what's happening in the entire system.
That's a very strong limitation, and as you might expect,
that would make a lot of things impossible or difficult.
Then we went on and got two positive results, a theorem
that-- well, an algorithm that is deterministic,
but the processes have unique identifiers,
and then you can elect a leader quickly.
Or if you don't have unique identifiers
but you have probability, so you can make random choices,
you can essentially choose an identifier,
and then it works almost as well with those identifiers.
Then we looked at maximal independent set.
Remember what it means, no two neighbors
should both be in the set, but you
don't want to be able to add any more nodes while keeping
these nodes independent.
In other words, every node is either in the set
or has a neighbor in the set.
And we gave this algorithm-- I'm just including it here
as a reminder-- Luby's algorithm, which basically goes
through a number of phases.
In each phase, some processes decide to join the MIS
and their neighbors decide not to join,
and you just repeat this.
You do this based on choosing random IDs again.
And we proved that the theorem correctly computes an MIS,
and with a good probability, all the nodes
decide within only logarithmic phases,
and here is the number of nodes.
All right, then went on to breadth-first spanning trees.
Here, we have now a graph that already has a leader.
It has a distinguished vertex.
And the process that's sitting there
knows that it's the leader.
The processes now are going to produce
a breadth-first spanning tree rooted at that vertex.
Now, for the rest of the time, we'll
assume unique identifiers, but the processes
don't know anything about the graph except that their own ID
and their neighbors' IDs.
And we want the processes to eventually output
the ID of their parent.
And here is, just to repeat, the simple algorithm that we used.
Basically, it's processes just mark themselves
as they get included in the tree.
It starts out with just the root being marked.
He sends a special search message out to his neighbors,
and soon as they get it they get marked and pass it on.
Everybody gets marked in a number
of rounds that corresponds to their depth in-- their distance
from the root of the tree, [INAUDIBLE] from the tree.
We talked about correctness in terms of invariance.
What this algorithm guarantees is,
at the end of any number r of rounds, exactly the processes
at distance up to r are marked, and the processes that
are marked, have their parents defined.
And if your parents defined-- it's
the UID of a process that has distance d minus 1
from the root.
If you're a distance d, your parent
should be somebody who's correct--
has the correct distance d minus 1.
We analyze the complexity.
Time is counting the number of rounds.
And that's going to be, at worst,
the diameter of the network.
It's really the distance from a particular vertex, v 0.
And the message complexity-- there's
only one message sent in each direction on each edge,
so that's going to be only order of the number of edges.
And we talked about how you can get child pointers.
This just gives you parents.
But if you want to also find out your children,
then every search message should get
a response either saying you're my parent
or you're not my parent.
And we can do a termination using convergecast,
and there's some applications we talked about as well.
All right, and then at the end of the hour,
we started talking about a generalization,
a breadth-first spanning trees, which adds weights so you
have shortest paths trees.
Now you have weights on the edges.
Processes still have unique identifiers.
They don't know anything about the graph,
except their immediate neighborhood information.
And they have to produce a shortest path spanning
tree rooted at vertex v 0.
You know what a spanning tree is and the shortest paths
are in terms of the total weight of the path.
Now we want each node each, process, to output its parent
in the shortest path.
And also the distance from the original vertex v 0.
At the end of the hour, we looked
at a version of Bellman-Ford, which you've already
seen as a sequential algorithm.
But as a distributed algorithm, everybody
keeps track of their best guess, their best current guess,
of the distance from the initial vertex.
And they keep track of their parent
on some path that gave it this distance estimate,
and they keep their unique identifier.
The complete algorithm now is, everybody's
going to keep sending their distance around-- I
mean we could optimize that, but this is simple.
At every round, everybody sends their current distance estimate
to all their neighbors.
They collect the distance estimates
from all their neighbors, and then they do a relaxation step.
If they get anything that's better
than what they had before, they look at the best new estimate
they could get.
They take the minimum of their old estimate-- stop shaking,
good-- and the minimum of all of the estimates
they would get by looking at the incoming information
plus adding the weight of the edge between the sender
and the node itself.
This way you may get a better estimate.
And if you do, you would reset your parent
to be the sender of this improved information.
And again, you can pick-- if there's a tie,
you can pick any of the nodes that
sense-- the information leading to the best new guess,
you can set your parent to any of those.
And then this just repeats.
At the very end of the hour, we showed an animation,
which I'm not going to repeat now,
which basically shows how you can get lots of corrections.
You can have many paths that are set up
that look good after just one round,
but then they get corrected as the rounds go on.
You get much lower weight paths by having
a roundabout, multi-hop path to a node.
You can get a better total cost.
Here's where we got to last time.
Now why does this work?
Well what we need is eventually every process
should get the correct distance.
And the parent should be its predecessor on some shortest
path.
In order to prove that-- you always look for an invariant,
something that's true, at intermediate steps
of the algorithm that you can show by induction will hold--
and that will imply the result that you want at the end.
Here, what's the key invariant?
At the end of some number r of rounds, what do processes have?
After r rounds have passed, in this kind of algorithm,
what do the estimates look like?
Well, after one round, everybody's
got their best estimate that could happen
on a single-- that could result from a single hop path
from v 0.
After two rounds, you also get the best guesses
from two hop paths, and after r rounds in general,
you have your distance and parent corresponding
to a shortest path chosen from among those
that have at most our hops.
Yes?
That makes sense?
No?
Yeah?
OK.
All right, and if there is no path of r hops or fewer
to get to a node, there's still going
to have their distance estimate be infinity after r rounds.
This is not a complete proof, but it's the key invariant
that makes this work.
You can see that after enough rounds corresponding
to the number of nodes, for example,
everybody will have the correct shortest path of any length.
OK?
The number of rounds until every estimate
stabilize-- all the estimates stabilize,
it's going to be n minus 1.
All right?
Because the longest simple path to any node
is going to be n minus 1, if it goes
through all the nodes of the network before reaching it.
Makes sense?
If you want to make sure you have the best estimate,
you have to wait n minus 1 rounds
to make sure the information has reached you.
The message complexity-- well, since there's
all these repeated estimates it's no longer just
proportional to the number of edges,
but you have to take into account
that there can be many new estimates sent on each edge.
In fact, the way I've written this,
you just keep sending your distance at every round.
It's going to be the number of edges
times the number of rounds.
You can do somewhat better than this,
but it's worse than just the simple BFS case.
This is more expensive, because BFS just had diameter rounds
and this now has n for n minus 1 rounds,
and BFS just had one message ever sent on each edge,
and now we have to send many.
Comments?
Questions?
Is it clear that the time bound really does depend on n and not
on the diameter?
For breadth-first search, it was enough just
to have enough rounds to correspond
to the actual distance and hops to each node,
but now you need enough rounds to take
care of these indirect paths that
might go through many nodes but still end up with a shorter--
with a smaller total weight.
Is it clear to everybody why the bound depends on n?
Actually the animation that I had last time
showed how there are lots of corrections,
and you had enough-- you had rounds that depended
on the total number of nodes.
Yes?
OK.
Yeah?
AUDIENCE: But could you keep track of each round
if the values-- if any of [INAUDIBLE]?
If you-- if everything stops changing after less than n
rounds, then you might not have to--
PROFESSOR: OK, so you're asking about termination?
AUDIENCE: Yeah.
PROFESSOR: Ah, OK, well so that's probably the next slide.
First, let's deal with the child pointers,
and then we'll come back to termination.
First, this just gives you your parents.
If you want your children, you can do the same thing
that we did last time.
When a process gets a message, and the message doesn't
make the sender its parent, this is not
giving it an improved distance.
Then the node can just respond, non-parent.
But if a process receives a message that
doesn't improve the distance, it says, OK, you are my parent,
but it might have also told another node--
it might have another parent, another node
that previously it thought was its parent.
It has to do more work, in this case,
to correct the erroneous parent information.
It has to send its previous parent a non-parent message
to correct the previous parent message.
Things are getting a little bit trickier here.
On the other end, if somebody is keeping track of its children,
it has to make adjustments too because things can
change during the algorithm.
Let's say a process keeps track of its children in some set,
it has a set children.
If it gets a non-parent message from a child,
even if that child might be from a neighbor.
That neighbor might be in its set children,
and this could be a correction, and then
the process has to take that neighbor out
of the set of children.
And suppose the process improves its own distance.
Well, now, it's kind of starting over.
It's going to send that distance to all of its neighbors
again and collect new information about
whether they're children or not.
The simple thing to do here is just empty--
zero out your children set and start over.
Now you send your new messages to your neighbors
and wait for them to respond again.
There's tricky bookkeeping to do to handle corrections
as the structure of this tree changes,
so getting child pointers is a little more complicated
than before.
Make sense?
All right, so now back to your question, termination.
How do all the processes know when the tree is complete?
In fact, we have a worse problem.
With this problem we hit for breadth first search,
but we have an even worse problem.
Now what is that?
Yeah?
AUDIENCE: [INAUDIBLE].
PROFESSOR: Yeah, before we had each individual node.
Once it figured out who its parent was,
it could just output that.
And now, you can figure out your parent, but it's just a guess
and you don't know when you can output this.
How does a process even-- an individual process
even figure out its own parent and distance?
There's two aspects here in termination.
One is how do you know the whole thing is finished,
but the other one is even how do you
know when you're done with your own parent and distance?
Well, if you knew something about the graph,
like an upper bound on the total number of nodes,
then you could just wait until that number of rounds
and be done.
But what if you don't have that information about the graph?
What might you do?
Yeah?
AUDIENCE: You want to BFS in parallel
and filter down the information when you've
reached the size of the graph.
PROFESSOR: Maybe.
I think-- what is the strategy we use for termination for BFS?
Let's start with that one.
It's a little easier.
You did the subtree thing that we call
convergecast the information.
When we had a leaf he knew he was a leaf,
and he could send his done information up to his parent
and that got convergecast up to the top of the tree.
Can we convergecast in this setting?
Turns out we can, but since things are changing,
you're going to be sending done messages
and then something might change.
You might be participating in the convergecast many times.
Since the tree structure is changing,
the main idea is anybody can send a done message
to it's current-- the node he believes is his parent,
provided he's received responses to all of its distance messages
so it thinks it knows who all its children are.
It has a current estimate of all the children
and-- so if it knows all its children
and they have all sent him done messages.
For all your current children, your current belief
or who your children are, if they've all
sent you done messages, then you can send a done message
up the tree.
But this can get a little more complicated
than it sounds, because you can change who your children are.
What this means is that the same process
can be involved several times in the convergecast,
based on improving the estimates.
Here's an example of the kind of thing that can happen.
Let's say you start out, you have these huge weights
and then you have a long path with small weights.
i 0 starts out and sends its distance information
on its three nodes, to its three neighbors.
And this guy at the bottom now has a distance estimate of 100,
and it's going to decide it's a leaf.
Why?
Because when it sends messages to its children,
to its neighbors, they're not able to improve their estimates
based on the new information that he's sending.
This guy decides that he's a leaf right away,
and he sends that information back to node i 0.
On the other hand, this guy has the same estimate of 100.
And he sends out his messages to try to find out if he's a leaf,
but he finds out when he sends a message this way
that he's actually able to improve that neighbor's
estimate, because that was infinity till then.
He doesn't think he's a leaf.
We have this one guy who thinks he's a leaf and responds,
so this i 0 is sitting there.
He has to wait to hear from his other children.
OK so far?
All right, in the meantime, the messages
are going to eventually creep around
and this node is going to get a smaller estimate based
on the length of that long many hop path.
Then he's going to decide he is not a child of i 0.
He's going to tell i 0 I'm really not
your child, which means that i 0 stops waiting for him.
But also, this guy decides he's not a child as well.
He becomes a child of the node right above him.
So i 0 now will realize he only has one child,
but this guy believes he's a leaf again after trying again
to find children.
And now the done information propagates all the way
up the tree, which is now just this one long path.
They start trying to convergecast,
but then, oops, they were wrong.
They have to make a correction, and they
are forming a new tree.
Eventually, the tree is going to stabilize,
and eventually the done information
will get all the way up to the top,
but there could be lots of false starts in the mean time.
It's sort of confusing, but that's
the kind of thing that happens.
Yeah?
AUDIENCE: There may be a process in which [INAUDIBLE]
and then it just switches its mind at the very end of it.
How do you make sure that, that propagation [INAUDIBLE]?
PROFESSOR: Yes, so the root node isn't
going to terminate until it hears from everybody.
You kind of have to close out the whole process.
It's always pending, waiting for somebody,
if it hasn't heard from someone.
If things switch, they'll join in another part of the tree.
I think the best thing to do here
is sort of construct some little examples by hand.
I mean we're not going to get into it how
you do formal proofs of things like this.
We don't even have right now a simulator
you can use to play with these algorithms.
Although if anybody's interested,
some students in my class last term,
actually, wrote a simulator that might be available.
OK, all right, so that's what you
get for a synchronous-- some example synchronous distributed
algorithms.
Now let's look at something more complicated
when you get into asynchronous algorithms.
OK.
So far complications that you've seen
over the rest of this course, you
have now processes that are acting concurrently.
And we had a little bit of non-determinism,
nothing important, but now things
are about to get much worse.
We don't have rounds anymore.
Now we're going to have processes taking steps,
messages getting delivered at absolutely arbitrary times,
arbitrary orders.
The processes can get completely out of sync,
and so you have lots and lots more non-determinism
in the algorithm.
The non-determinism has to do with who's
doing what in what order.
Understanding that type of algorithm
is really different from understanding the algorithms
that you've seen all term and even synchronous distributed
algorithms, because there isn't just one way
the algorithm is going to execute.
The execution can proceed in many different ways,
just depending on the order of the steps.
You can't ever hope to understand
exactly how this kind of algorithm is executing.
What can you do?
Well, you can play with it, but in the end,
you have to understand is some abstract properties.
Some properties of the executions, rather
than exactly what happens at every step, and that's a jump.
It's a new way of thinking.
We can look at asynchronous stuff,
if you want, from my book.
Now we still have processes at the nodes of a graph.
And now we have communication channels
associated with the edges.
Now, the processes are going to be some kind of automata,
but the channels will also be some kind of automata.
We'll have all these components, and we'll
be modeling all of them.
Processes still have their ports.
They need not, in general, have identifiers.
What's a channel?
A channel is-- it's a kind of an automaton, infinite state
automaton, that has inputs and some outputs.
Here, this is just a picture of a channel from node u
to node v. Channel uv is just this cloud
thing that delivers messages.
It has inputs.
The inputs are-- messages get sent on the channel.
You can have one process at one end sending a message m
and the outputs at the other end are the deliveries,
let's say receive message m, at the other end, node v.
To model this, the best thing is actually
to-- instead of just describing what
it does, to give an explicit model of its state
and what happens when the inputs and outputs occur.
If you want these messages to be delivered-- let's
say in FIFO order, fine.
You can make the state of the channel be an actual q.
mq would just be a FIFO queue of messages.
Starts out empty, and when messages get sent,
they get added to the end, and get delivered they
get removed from the beginning.
All that we need to describe-- this
is a sort of a pseudo code.
All we need to describe-- to write in order
to describe what this channel does,
is what happens when a send occurs,
and when can this channel deliver a message, and what
happens when it does that.
A send, which can just come in at any time, and the effect
is just to add this message to the end of the queue.
The recieve-- stop moving.
The receive-- that cannot be construction.
We'd hear noise.
It's gremlins.
We have-- a receive can occur only
when this message is at the head of the queue,
and when it occurs, it gets removed from the queue.
Does this make sense as a description
of what a channel does?
Messages come in, get added to it,
and then messages can get delivered
in certain situations, and they get removed.
That's a description of the channel.
to be using an asynchronous system.
A process-- the rest of the system
consists of processes associated with the graph vertices.
Let's say pu is a process that's associated with vertex u.
But I'm writing that just sort of as a shorthand, because u
is the vertex in the graph, and the process doesn't actually
know the name of the vertex.
It just has its own unique ID or something,
but I'm going to be a little sloppy about that now.
The process that at vertex u can perform
send outputs to put messages on the channel,
and it will receive inputs for messages
to come in on the channel.
But the processes might also have some external interface
where somebody is submitting some inputs to the process,
and the process has to produce some output at the end.
There can be other inputs and outputs.
And we'll model it with state variables.
Process is supposed to keep taking steps.
The channel is supposed to keep delivering messages.
It's a property called liveness.
You want to make sure that your components in your system
all keep doing things.
They don't just do some steps and then stop.
Here's a simple example.
A process that's remembering the maximum number it's ever seen.
There's a max process automaton.
It receives some messages m, some value,
and it will send it out.
It keeps track of the max-- that's max state variable.
it starts out with its own initial values,
so x for u. x of u is its initial value.
And then it has-- for every neighbor,
it has a little queue-- here, it's just a Boolean--
asking whether it's supposed to send to that neighbor.
This is the pseudocode for that max process.
What does it do when it receives a message?
Well, it sees if that value is bigger than what it had before,
and if so, it resets the max.
And it also makes a note that it's supposed to send this out
to all its neighbors.
Whenever it gets new information,
it will want to propagate it to its neighbors.
You see how this is written?
You just say, reset the max and then get ready
to send to all your neighbors, and the last part
is just sort of trivial code.
It says, if you are ready to send, then you can send,
and then you're done and you can set the send flags to false.
Yeah?
AUDIENCE: What study?
PROFESSOR: For every neighbor.
Oh, I wrote neighbor v before, and then I-- yeah.
I wrote neighbor-- oh, I know why I wrote w.
Here, I'm talking about if you receive a message
from a particular neighbor v, then
you have to send it to all your neighbors.
Before, I used v to denote a generic neighbor,
but now I can't do that anymore, because v
is the sender of the message.
Just technical-- OK?
We have these process automata.
We have this channel automata.
Now, we want to build the system.
We paste them together.
How we paste them is just we have
outputs from processes that can match up with inputs
to channels and vice versa.
If a process has a send output, let's
say send from u to v, that will match up with the channel that
has send uv as an input.
And the receive from the channel matches up
with the process that has that receive as an input.
All I'm doing is matching up these components.
I'm hooking together these components
by matching up their action names.
Does that make sense?
I'm saying how I build a system out of all these components,
and I just have a syntactic way of saying what actions match up
in different components.
Questions?
This is all new.
When this system takes a step, well,
if somebody's performing an action,
and someone else has that same action--
let's say a process is doing a send.
The channel has a send.
They both take their transitions at the same time.
The process sends a message, it gets
put into the channel at the end of its queue.
Make sense?
OK.
How does this thing execute?
Well, there's no synchronous rounds,
so the system just operates by the processes
and the channels just perform their steps in any order.
One at a time, but it can be in any order,
so it's a sequence model.
You just have a sequence of individual steps.
There's no concurrency here.
In the synchronous model, we had everybody taking their steps
in one big block.
And here, it's just they take steps one at a time,
but it could be in any order.
And we have to make sure that everybody keeps taking steps.
That every channel continues to deliver messages,
and every process always performs
some step that's enabled.
For the max processes, well, we can just
have a bunch of processes, each one now starting
with its initial value.
And what happens when we plug them together with all
their channels between them?
Corresponding to whatever graph they are in.
They just have channels on the edges.
What's the behavior of this?
If all the processes are like the ones I just showed you,
they wait till they hear some new max
and then they send it out.
Yeah?
AUDIENCE: All processes eventually
have a globally maximum value.
PROFESSOR: Yeah, they'll all eventually get the global max.
They'll keep propagating until everybody receives it.
Here's a animation if everybody starts with the values that
are written in these circles.
Now, remember, before they were all
sending it once, now no more.
Let's say the first thing that happens
is the process that started with five sends its message out
on one of its channels, so the five goes out.
The next thing that might happen is the other process
with the seven might send the seven out
on one of its channels.
And these are three more steps.
Somebody sends a 10.
Somebody sends a seven.
Somebody received a message and updated its value as a result,
and we continue.
I'm depicting several steps at once,
because it's boring to really do them one at a time,
but the model really says that they take steps in some order.
Everybody is propagating the largest thing it's seen,
and eventually, you wind up with everybody having
the maximum value, the 10.
All right, that's how an asynchronous system operates.
We can analyze the message complexity of this.
The total number of messages sent
during the entire execution, at worst, on every edge.
You can send the successively improved estimate,
so that's, again, what are n times the number of edges.
Time complexity is an issue.
When we had synchronous algorithms,
we just counted the number of rounds and that was easy.
But what do we measure now?
How do we count the time when you have all these processes
and channels taking steps whenever they want?
Yeah, so this really isn't obvious.
There's a solution that's commonly used, which is-- OK,
we're going to use real time.
And we're going to make some assumptions
about certain basic steps taking, at most,
a certain amount of time.
Let's say that local computational-- time
for a process to perform its next step, is little l.
You just give a local time bound.
And then you have d for a channel to deliver one message.
The first message that's currently in the channel.
If you have assumptions like that,
you could use those to infer a real time upper bound
for solving the whole problem.
I mean if you know it takes no longer than d to deliver one
message, then you can bound how long
it takes to deliver-- to empty out a queue, a channel, and how
long it takes for messages to propagate through the network.
It's tricky, but this is about the only thing
you can do in a setting where you don't actually
have rounds to measure.
Then for the max system, how long does it take?
Well, let's just ignore the local processing time.
Usually that's assumed to be very small,
so let's say it's 0.
We can get a very simple, pessimistic really,
upper bound that says the real time for finishing
the whole thing is of the order of the diameter of the network
times the number of nodes times little
d is the amount of time for a message queue
to deliver its first message.
As a naive way of analyzing this,
you just consider how long it takes for the max
to reach some particular vertex u along the shortest path.
Well, it has to go through all the hops on the path,
so that would be the diameter.
And how long does it have to wait in each channel
before it gets to move another hop?
Well, it might be at the end of a queue.
How big could the queue be?
Well, at worst end for the improved estimates.
Let's say it's n times the delivery time on the channel,
just to traverse one channel.
What I'm doing is modeling possible congestion
on the queue to see how long it takes for a message
to traverse one channel.
Yeah?
AUDIENCE: Are we just assuming that processes process things,
and messages are delivered as soon as they can possibly
have them?
PROFESSOR: Good.
Yeah, we normally have-- I'm sort of skipping over
some things in the model-- but you have a liveness assumption
that says that the process keeps taking steps as long
as it has anything to do, and so we
would be putting time bounds on how long
it takes between those steps.
That'll be the local processing time, here.
I'm saying that's 0.
AUDIENCE: You do that-- wouldn't you
be able to get some amount of information
about what were things happening?
PROFESSOR: Ah, OK.
Here is-- this is a very subtle point.
What do you think is that if I'm making all these timing
assumptions, the processes should
be able to figure that out.
But actually, you can't figure anything out just
based on these assumptions about these upper bounds.
Putting upper bounds on the time between steps
does not in any way restrict the orderings.
You still have all the same possible orderings of steps.
Nobody can see anything different.
These times are not visible in any sense.
They're not marked anywhere for the processes to read.
They're just something that we're
using to evaluate the cost, the time cost, of the execution.
And you're not restricting the execution
by putting just upper bounds on the time.
If you are restricting the execution,
like you had upper bounds and lower bounds on the time, then
the processes might know a lot more.
They might, in fact, be acting more like a synchronous system.
These are times that are just used for analyzing complexity.
They're not anything known to the processes in the system.
OK?
All right.
Now let's revisit the breadth-first spanning tree
problem.
We want to now compute a breadth-first spanning tree
in an asynchronous network.
Connected graph, distinguish root vertex,
processes have no knowledge of the graph.
They have you UIDs.
All this is the same as before in the synchronous case.
Everybody's supposed to output its parent information
when it's done.
Here's an idea.
Suppose we just take that nice simple synchronous algorithm
that I reviewed at the beginning of the hour where everybody
just sends a search message as soon as they get it
and they just adopt the first parent they see.
What happens if I run that asynchronously?
I just send it and I get a search message.
Then I send it out to my parents whenever
and everybody's doing this.
Whenever they get their first search message,
they decide the sender is its parent.
Yeah?
AUDIENCE: Could you possibly have a front tier
that doesn't keep expanding-- not obeying the defining
property of the BFS?
PROFESSOR: Yeah it could be that,
because we don't have any restriction on how
fast the messages might be sent and the order of steps,
it could be that some messages get sent
very quickly on some long path.
Someone sitting at the far end of the network might
get a message first on a long path
and later on a short path, and the first one to
gets it decides that's its parent, then it's stuck.
This is not an algorithm that makes corrections.
OK?
All right, well, before we had a little bit of non-determinism
when we had this algorithm in the synchronous case
because we could have two messages arriving at once,
and you have to pick one to be your parent.
Now that doesn't happen, because you only
get one message at a time, but you have a lot more
non-determinism now.
Now you have all this non-determinism
from the order in which the messages get sent and processes
take their steps.
There's plenty of non-determinism,
and remember, the way we treat non-determinism for distributed
algorithms is it's supposed to work
regardless of how those non-deterministic choices get
made.
How would we describe?
I'll just write some pseudo code here
for a process that's just mimicking
the trivial algorithm.
It can receive a search message, that's the inputs,
and it can send a search message as its output.
It can also output parent when it's ready to do that.
What does it have to keep track of?
Well, it keeps track of its parent,
keeps track of whether it's reported its parent,
and it has some send buffers with messages
it's ready to send to its neighbors,
could be search messages or nothing.
This bottom symbol is just a placeholder symbol.
What happens when the process receives a search message, just
following the simple algorithm?
Well, if it doesn't have a parent yet,
then it sets its parent to be the sender of that search
message, and it gets ready to send the search message
to all of its neighbors.
That's really the heart of the algorithm
that you saw for the simple algorithm
for that is the synchronous case so that's the same code.
The rest of the code is it just sends the search messages out
when it's got them in the send buffers,
and it can announce its parent once the parent is set,
and then it doesn't-- this flag just means it doesn't keep
announcing it over and over again.
It makes sense that this describes
that simple algorithm.
It's pretty concise, just what does
it do in all these different steps.
OK?
Now, if you run this asynchronously,
as you already noted, it isn't going
to necessarily work right.
You can have this guy sending search messages,
but some are going to arrive faster than others.
And you see you can have the search messages creeping around
in an indirect path, which causes
a spanning tree like this one to be created,
which is definitely not a breadth-first spanning tree.
The breadth-first tree is this one-- a breadth-first tree.
You have these roundabout paths.
This doesn't work.
What do we do?
Yeah?
AUDIENCE: You could have a child [INAUDIBLE].
PROFESSOR: You're going to try to synchronize them, basically.
AUDIENCE: [INAUDIBLE].
PROFESSOR: That is related to a homework question this week.
Very good.
We're going to do something different.
Other suggestions?
Yeah?
AUDIENCE: We could keep a variable in each process
that-- you can do something like Bellman-Ford.
PROFESSOR: Yeah, so you can do what we did for Bellman-Ford,
but now the setting is completely different.
We did it for Bellman-Ford when it was synchronous
and we had weights.
Now, it's asynchronous and there are no weights.
OK?
Oh, there's some remarks on a couple of slides here.
This is just belaboring the point,
that the paths that you get by this algorithm
can be longer than the shortest paths,
and-- yeah, you can analyze the message and time complexity.
The complexity here is order of the diameter times the message
delay on one link.
And why is it the diameter even though some paths
may be very long?
This is a real time upper bound that
depends on the actual diameter of the graph,
not on the total number of nodes.
Why would an upper bound on the running time for the simple
algorithm depend on the diameter?
Yeah?
AUDIENCE: Because you only have a longer path
if it's faster than--
PROFESSOR: Exactly.
The way we're modeling it-- it's a little strange,
maybe-- but we're saying that something
can travel on a long path only if it's going very fast.
But the actual shortest paths still move along, at worst,
at d time per hop.
At worst, the shortest path information
would get there within time d.
Something is going to get there within time d,
even though something else might get there faster.
OK?
All right.
Yes, we can set up a child pointers,
and we can do termination using convergecast.
It's just one tree.
There's nothing changing here.
And applications, same as before.
But that didn't work, so we're going
to-- back to the point we were talking
about a minute ago-- we're going to use a relaxation
algorithm like Bellman-Ford.
In the synchronous case, we corrected
for paths that had many hops but low weight,
but now we're going to correct for the asynchrony errors.
All the errors that you get because of things traveling
fast on long paths, we're going to correct for those
using the same strategy.
Everybody is going to keep track of the hop distance.
No weights now just the hop distance,
and they will change the parent when
they learn of a shorter path.
And then they will propagate the improved distance,
so it's exactly like Bellman-Ford.
And eventually, this will stabilize
to an actual breadth-first spanning tree.
Here's a description of this new algorithm.
Everybody keeps track of their parent,
and now they keep track of their hop distance,
and they have their channels.
Here's the key, when you get new information-- you receive
some new information, this m is a number
of hops that your neighbor is telling you.
Well, if m plus 1, which would be
your new estimate for a number of
hops-- if that's better than what you already have,
then you just replace your estimate
by this new number of hops.
And you set your-- to a new parent.
You set your parent pointer to the sender,
and you propagate this information.
It's exactly the same as what we had before,
but now we're correcting for the asynchrony.
We get shorter hop paths later.
Makes sense?
And the rest of this is just you send the message.
And notice we don't have any terminating actions.
Why?
Because we have the same problem that we
had before with having processes know when they're done.
If you keep get getting corrections,
how do you know when you're finished?
And how do you know this is going to work?
In the synchronous case, we could
get an exact characterization of what exactly
the situation is after any number of rounds.
And we can't do that now, because things
can happen in so many orders.
We have to, instead, state some higher level
abstract properties, either in variance
or you'll see some other kinds of properties as well.
We could say, for instance, as an invariant,
that all the distance information you get is correct.
If you ever have your distance set to something,
it's the actual distance on some path
and your parent is correctly set to be your predecessor on such
a path.
This is just saying whatever you get is correct information.
This doesn't say that eventually you're going to finish, though.
It just says what you get is correct.
If you want to show that, eventually, you
get the right answer, you have to do something
with the timing.
You have to say something like by a certain time that
depends on the distance.
If there is-- and at most our hop path to a node,
then it will learn about that by a certain time,
but that depends on the length of the path and the message
delivery time.
And the number of nodes, because they can be congestion.
You have to not only say things are correct,
but you have to say, eventually, you get the right result,
and here it will say by a certain time you get
the right result. Makes sense?
This is how you would understand an algorithm like this one.
Message complexity.
Since there's all these corrections,
you're back in number of edges times possibly
the number of nodes.
And the time complexity, till all the distances and parent
values stabilize, could be-- this
is pessimistic again-- the diameter
times the number of nodes times d, because there
can be congestion in each of the links because
of the corrections.
How do you know when this is done?
How can a process know when it can finish?
Idea?
Before we had said, well, if you knew n,
if you knew the number of nodes, you
could weight that number of rounds.
That doesn't even help you here.
Even if you know-- have a good upper bound
on the number of nodes in the network,
there's no rounds to count.
You can't tell.
Even knowing the number of nodes you can't tell,
so how might you detect termination?
Yep?
AUDIENCE: It could bound on the diameter of [INAUDIBLE].
PROFESSOR: Yeah, well, but even if you know
that, you can't count time.
See this is the thing about asynchronous algorithms,
you don't have-- although we're using time
to measure how long it's termination takes,
we-- the processes in there don't have that.
They're just these asynchronous guys who just take their steps.
They're ignorant of anything to do with time.
Other ideas?
Yeah.
AUDIENCE: Couldn't you use the same converge kind of thing--
PROFESSOR: Same thing.
We're just going to use convergecast again, same idea.
You just compute and your repute your child pointers.
You send a done to your current parent, after you've gotten
responses to all your messages, so you
think you know your children.
And they've all told you they're done
and-- but then you might have to make corrections,
so as in what we saw before, you can be involved
in this convergecast several times until it finally reaches
all the way to the root.
Once you have these, you can use them the same way as before.
You now have costs that are better
than this simple tree that didn't have shortest
paths, because it now takes you less time to use
the tree for computing functions or disseminating information
because the tree is shallower.
Finally, what happens when we want
to find shortest path trees in an asynchronous setting?
Now, we're going to add to the complications
that we just saw with all the asynchrony.
The complications of having weights on the edges.
All right, the problem is get a shortest path spanning tree,
now in an asynchronous network, weighted graph,
processes don't know about the graph again.
They have UIDs, and everybody's supposed to output its distance
and parent in the tree.
We're going to use another relaxation algorithm.
Now think about what the relaxation
is going to be doing for you.
We have two kinds of corrections to make.
You could have long paths that have small weight.
That showed up for Bellman-Ford in the synchronous setting,
so we have to correct for those.
But you could also have-- because of asynchrony,
you could have information travelling fast on many hops,
and you have to correct for that as well.
There's two kinds of things you're
going to be correcting for in one algorithm.
This is going to-- and it's pretty surprising-- it's
going to lead to ridiculously high complexity, message
and time complexity.
If you really have unbridled asynchrony and weights,
this is going to give you a very costly algorithm.
You're going to see some exponential is
going to creep in there.
Here's the algorithm for the asynchronous Bellman-Ford
algorithm.
Everyone keeps track of their parent.
Their conjecture distance, and they
have, now, messages that they're going to send to the neighbors.
Let's say you have a queue because there
could be successive estimates.
We'll have a queue there.
The key step, the relaxation step,
is what happens when you receive a new estimate of the best
distance.
This is weighted distance, now, from a neighbor.
Well, you look at that distance plus the weight
of the edge in between, and you see
if that's better than your current distance,
just like synchronous Bellman-Ford.
And if it is, then you improve your distance,
reset your parent, and send the distance out to your neighbors.
It's exactly like the synchronous case,
but we're going to be running this asynchronously.
And since you're going to be correcting every time you see
a new estimate, this is actually going
to handle both kinds of corrections,
whether it comes because of a many hop path with a smaller
weight, or whether it just comes because of the asynchrony.
Whenever you get a better estimate,
you're going to make the correction.
Is it clear this is all you really
need to do in the algorithm, just what's in this code?
That's the received, and then the rest of the algorithm
is just you send it out when you're ready to send.
And then we have the same issue about termination,
there's no terminating actions.
We'll come back to that.
It's really hard to come up with invariants and timing
properties, now, for this setting.
You can certainly have an invariant like the one
that we just had for asynchronous breadth-first
search.
You can say that at any point, whatever
distance you have is an actual distance that's
achievable along some path, and the parent is correct.
But we'd also like to have something
that says, eventually, you'll get the right distance.
You want to state a timing property that says,
fine, if you have an at most r hop path, by a certain time,
you'd like to know that your distance is at least as
good as what you could get on that path.
The problem is what are you going
to have here for the amount of time?
How long would it possibly take in order
to get the best estimate that you could
for a path of at most r hops?
A guess?
I was able to calculate something reasonable
for the breadth-first search case,
but now this is going to be much, much worse.
It's not obvious at all.
It's actually going to depend on how many messages could pile up
in a channel, and that can be an awful lot,
an exponential number-- exponential in the number
of nodes in the network.
I'm going to produce an execution for you that
can generate a huge number of messages, which then will take
a long time to deliver and delay the termination
of the algorithm.
First, let's look at an upper bound.
What can we say for an upper bound?
Well, there's many different paths
from v 0 to any other particular node.
We might have to traverse all the simple paths in the graph,
perhaps.
And how many are there?
Well, as an upper bound, you could say order n factorial
for the number of different paths
that you can traverse to get from v 0
to some particular other node.
That's exponential in n.
Certainly, it's order n to the n.
This says that the number of messages
that you might send on any channel
could correspond to doing that many corrections.
This can blow up your message complexity into n
to the n times the number of edges,
and your time complexity n to the n times, n times d,
because on every edge you might have
to wait for that many messages, corrected messages, sitting
in front of you.
That seems pretty awful.
Does it actually happen?
Can you actually construct an execution of this algorithm
where you get exponential bounds like that?
And we'll see that, yes, you can.
Any questions so far?
Here's a bad example.
This is a network, consists of a sequence of, let's say,
k plus 1-- k plus 2, I guess, nodes, in a line.
And I'm going to throw in some little detour nodes
in between each consecutive pair of nodes in this graph,
and now let me play with the weights.
Let's say on this path, this direct path from v 0
to vk plus 1, all the weights are 0.
That's going to be the shortest path, the best weight path,
from v 0 to vk plus 1.
But now I'm going to have some detours.
And on the detours I have two edges, one of weight 0
and the other one of weight that's a power of 2.
I'm going to start with high powers of 2,
2 to the k minus 1, and go down to 2 to the k minus 2,
down to 2 of the 1, to the 0.
See what this graph is doing?
It has a very fast path in the bottom,
which you'd like to hear about as soon as you can.
But, actually, there is detours which could give you
much worse paths.
Let's see how this might execute to make a lot of messages
pile up in a channel.
My claim is that there's an execution
of that network in which the last node, bk, sends 2
to the k messages to the next node vk plus 1.
He's really going to send an exponential number of messages
corresponding to his corrections.
He's going to keep making corrections for better
and better estimates.
And if all this happens relatively fast,
that just means you have a channel
with an exponential number of messages in it,
emptying that will take exponential time.
You have an idea how this might happen?
How could this node, bk, get so many successively improved
estimates, one after the other?
Well, what's the biggest estimate it might get?
Yeah?
AUDIENCE: 2 to the k minus 1.
PROFESSOR: It could get 2 to the k minus 1?
Well, let's see.
AUDIENCE: Or [INAUDIBLE].
PROFESSOR: It could do that.
AUDIENCE: Then it's 2 to the k.
PROFESSOR: And it could do that.
AUDIENCE: Plus 2 to the k.
PROFESSOR: Plus 2 to the k minus 2,
plus all the way down to plus 2 to the 0.
You could be following this really inefficient path, just
all the detours, before the messages actually arrive
on the edges on the spine.
AUDIENCE: [INAUDIBLE].
PROFESSOR: Yeah, so you follow all-- oh, you said 2 to the k,
minus 1, parenthesis.
Yeah, that's exactly right.
AUDIENCE: It's all right. [INAUDIBLE].
PROFESSOR: We can't parenthesize our speech.
All right, so the possible estimates that bk can take on
are actually 2 to the k-- as you said, 2 to the k minus 1, which
you would get by taking all of the detours
and adding up the powers of 2.
But it could also have an estimate,
which is 2 to the k minus 2 or 2 to the k minus 3.
All of those are possible.
Moreover, you can have a single execution
of this asynchronous algorithm in which node bk actually
acquires all those estimates in sequence, one at a time.
How might that work?
First, the messages travel all the detours.
Fine, bk gets 2 to the k minus 1.
Then, there's a message traveling-- well
this guy sends a message on the lower link.
This guy has only heard about the messages on the detours up
to that point.
But he sends that on the lower link,
which means you kind of bypass this weight of one.
bk gets a little bit of an improvement,
which gives it 2 to the k minus 2 as its estimate.
What happens next?
Well, we step one back, and the node from node k minus 2
can send a message on the lower link, which
has weight 0, to bk minus one.
But that corrected estimate might then traverse the detour
to get to node bk.
If you get the correction for node bk minus 1,
but then you follow the detour, you haven't improved that much.
You've just improved by one.
This way, you get 2 to the k minus 3 as the new estimate.
But then, again, on the lower link,
the message eventually arrives, which
gives you 2 to the k minus 4.
You see the pattern sort of?
You're going to be counting down in binary
by successively having nodes further to the left
deliver their messages, but then they
do the worst possible thing of getting the information to bk.
He has to deal with all those other estimates in between.
If this happens quickly, what you get
is a pile up of an exponential number of search messages
in one channel.
And then that information has to go
on to the next node or the rest of the network, whatever.
It's going to take an exponential amount of time,
in the worst case, to empty that all out.
This is pretty bad, but the algorithm is correct.
And so how do you learn when everything is finished,
and how does a process know when it
can output its own correct distance information?
How can we figure out when this is all stabilized?
Same thing as before, we can just do a convergecast.
I mean this is more corrections, but it's still
the same kind of corrections.
We can convergecast and, eventually, this
is going to convergecast all the way up to the root.
And then the root knows it's done
and can tell everyone else.
A moral here-- you've had a quick dose
of a lot of asynchrony-- yeah, if you don't do anything
about it and you just use unrestrained asynchrony,
in the worst case, you're going to have
some pretty bad performance.
The question is, what do you do about that?
There are various techniques.
And if you want to take my course next fall,
we'll cover some of those.
I'll say a little bit about the course.
It's a basic-- it's a TQE level, basic grad course.
We do synchronous, asynchronous, and some other stuff
where the nodes really know about something about time.
Here's some of the synchronous problems--
some like the ones you've already seen.
Building many other kinds of structures in graphs,
and then we get into fault tolerance.
There's a lot of questions about what
happens when some of the components can fail,
or they're even malicious, and you
have to deal with the effects of malicious processes
in your system.
And then for asynchronous algorithms,
we'll do not only individual problems
like the ones you've just seen, but some general techniques
like synchronizers, notion of logical time--
that's Leslie Lamport's first and biggest contribution.
He one the Turing Award last year-- other techniques,
like taking global snapshots of the entire system
while it's running.
In addition to talking about networks,
as we've been doing this week, we'll
talk about shared memory, multi-processors accessing
shared memory.
And solving problems that are of use in multiprocessors,
like mutual exclusion.
And again, fault tolerance.
Fault tolerance then gets us into a study of data objects
with consistency conditions, which
is the sort of stuff that's useful in cloud computing.
If you want to have coherent access to data that's
stored at many locations, you need to have some interesting
distributed algorithms.
Self stabilization-- if you plunge your system
into some arbitrary state, and you'd
like it to converge to a good state, that's the topic of self
stabilization.
And depending on time, there's things
that use time in the algorithms, and the newer work
that we're working on in research is very dynamic.
You have distributed algorithms where the network
itself is changing during the execution.
That comes up in wireless networks,
and lately we're actually looking
at insect colony algorithms.
What distributed algorithms do ants
use to decide on how to select a new nest when
the researchers smash their old nest in the laboratory?
That kind of question.
That's it for the distributed algorithms week.
And we'll see you-- next week is security?
OK, yeah.