Leader election

Raft enforces strong leadership. It elects one leader which is responsible for sending log entries to all other servers (peers). Log entries always flow from leader to followers.

Every raft server has an electionTimeout (stored persistently on disk) that is randomly set from within a range for e.g., between 150-300ms. Elections are managed using RequestVote RPCs. Below is the pseudo-code for how elections occur and how a leader is elected.

Every peer has a long running task (conceptually a cron job)

// The candidate sends a RequestVote RPC to the peer
// this method runs on the peer
// The result of the execution (whether vote was granted etc)
// is sent back via the reply
func RequestVoteRpc(args RequestVoteArgs, reply RequestVoteReply){
   if args.Term < currentTerm {
    reply.Success = false
    return
   }

   reply.Success = false
   if votedFor == null or votedFor == args.CandidateId && IsCandidateLogUpToDate(lastLogIdx, lastLogTerm int) {
    reply.Success = true
   }
}


func startElection(){
    if (time elapsed since receiving AppendEntries RPC or since voting for a candidate > electionTimeout){
        currentState := Candidate
        currentTerm += 1
        voteFor := myself
        votes := 1
        electionTimeout := 150 + (randInit()%150) // reset election timer
        // the RequestVote RPCs are sent in parallel to all peers by the candidate
        for p := peers {
            if p != myself {
                requestVoteReply := go RequestVoteRpc()
                if (requestVoteReply != successful) {
                    sleep(smallDuration)
                    continue // retry the rpc
                } else {
                    votes += 1
                }
            }
        }

        if votes > majority of peers {
            currentState := Leader
            for p := peers {
                if p != myself {
                    go SendAppendEntriesHeartbeat(p)
                }
            }
        }
    }
}

func electionTicker(){
    for {
        sleepDuration := 150 + (randInt()%150)
        Sleep(sleepDuration)
        startElection()
    }
}

At its core for a peer to be elected as a leader, first a peer transitions to candidate, then it sends RequestVote RPCs in parallel to all other peers. If it gets a majority of votes, it becomes a Leader. Not every candidate can become a leader, only candidates that have an up to date log can become leaders, if a candidate doesn’t have an up to date log it won’t receive a majority of votes. This restriction is implemented via the IsCandidateUpToDate method.

Upon election as a leader, the leader sends an AppendEntries RPC to all other peers to establish and maintain it leadership. This is to prevent other peers from attempting to become candidates and also to transfer logs from leader to followers. We will discuss this in the Log replication section.

Log replication

In Raft, all clients (that send logs) talk to the leader only. If a client connects to a non-leader its redirected to the leader. When a client sends a new log entry to be appended, the leader would add it to its own log and then the AppendEntries heartbeat would take care of replicating this to all the peers.

After a leader is elected, it sends an AppendEntries RPC to all other peers. This has two functions, to establish and maintain leadership, for which we send the RPC at a regular cadence and second, if there are new log entries added to the leader’s log this would be replicated by the RPC as well. Although the Raft paper maintains that for an AppendEntries heartbeat, the log entries sent should be empty. This is an implementation detail. In our case, when we send AppendEntries heartbeat and in between the heartbeats if there are logs added to the leader, we would send the newly added logs in the next heartbeat.

// this runs on the peer, args is sent by the leader and reply
// is set by the peer for the leader to consume
func AppendEntriesRpc(peer, appendEntriesArgs, appendEntriesReply){
    if appendEntriesArgs.Term < currentTerm {
        // rpc sent by outdated leader
        // decline the rpc (reply false)
        // Leader will convert to follower on receipt of this RPC response
    }

    if appendEntriesArgs.PrevLogIdx >= len(log) {
        // Leader's log is longer than the follower
        // Decline the RPC (reply false)
        // Leader will update nextIdx after receipt of this rpc response
    }

    if log[appendEntriesArgs.PrevLogidx] != appendEntriesArgs.PrevLogTerm {
        // There is a term mismatch at lastLogIdx b/w leader and follower
        // Decline the RPC (reply false)
        // Leader will update nextIdx after receipt of this rpc response.
    }

    // Terms match at lastLogIdx for leader and peer.
    i := appendEntriesArgs.PrevLogIdx + 1
    j := 0
    for i < len(log) && j < len(appendEntriesArgs.Entries) {
        // find out where leader and follower log disagree (if at all)
        // let that index be k
        // discard follower log after that index
    }

    // Append Log sent by leader starting from index k
    // since its not present on the follower.
    log = append(log, appendEntriesArgs.Entries[k:])

    // AppendEntriesRPC was successful
    // After leader receives this rpc response and updates
    // nextIdx and matchIdx for this peer, leader and follower logs match
    appendEntriesReply.Success = true

    // Update this peer's commit Index based on
    // leader's commitIndex and this peer's updated log
    if appendEntriesArgs.LeaderCommit > commitIndex {
        commitIndex = min(appendEntriesArgs.LeaderCommmit, len(log))
    }
}

// The leader sends an Append Entries RPC to every peer
// The leader runs this and handles the response from the peer
// This doesn't handle the scenarios where RPC responses
// might be dropped or delivered out of order
func SendAppendEntriesRPC(peer int){
    // After an election rf.nextIdx[peer] = len(log)
    lastLogIdx := rf.nextIdx[peer] - 1 // we are optimistically assuming that peer has
    // log up to lastLogIdx.
    lastLogTerm := rf.log[lastLogIdx].Term
    logEntries := rf.log[lastLogIdx+1:] // if peer has log up to lastLogIdx, we need to send
    // log starting from lastLogIdx+1 to be rpelicated on the peer.

    appendEntriesArgs := {
        Term : currentTerm,
        PrevLogIdx: lastLogIdx,
        PrevLogTerm: lastLogTerm,
        Entries: logEntries
    }

    appendEntriesResp := go AppendEntriesRpc(peer, appendEntriesArgs, appendEntriesReply)
    if appendEntriesResp.Status != Failed {
        //peer was down, resp was dropped, req timed out etc
        retry
    } else {
        if appendEntriesReply.Success {
            // log up to lastLogIdx is present on the peer and it added logEntries to its log
            // update nextIndex and matchIndex
            nextIdx[peer] = lastLogIdx + len(logEntries)
            matchIdx[peer] = lastLogIdx + len(logEntries) - 1
        } else {
            nextIdx[peer] = nextIdx[peer] - 1 // we know that log doesn't match at lastLogIdx
            // so decrement it by 1 and retry
            retry
        }
    }
}

func SendAppendEntriesPeriodically(peer int){
    for {
        // Send Append Entries RPC every 10ms to maintain leadership
        // and replicate logs
        go SendAppendEntriesRPC(peer)
        time.Sleep(10ms)
    }
}

func SendAppendEntriesHeartbeat(){
    for p := range peers {
        if p != leader {
            go SendAppendEntriesPeriodically(p)
        }
    }
}

The Raft protocol leaves a lot of room for interpretation from the sender’s side. The protocol doesn’t specify how to handle out of order RPC replies. Here is one bug that stumped me. In Figure 2 of the paper, its mentioned that matchIndex[] increases monotonically. This means that whenever we update matchIndex for a peer on a successful AppendEntries RPC response, we should ensure matchIndex can only increase.

We use matchIndex to update the commitIndex on the leader, which in the end decides whether log entries up to a certain index are committed and eventually applied to the state machine. When an operation is applied to the state machine, we return a successful response to the client and that means that the log entries up to an index submitted by the client are durable and won’t be overwritten in the future.

If there exists an N such that N > commitIndex, a majority
of matchIndex[i] ≥ N, and log[N].term == currentTerm:
set commitIndex = N

If we were to decrement matchIndex, we would potentially be decrementing commitIndex and uncommitting log entries which were already committed thereby violating Leader completeness and State machine safety properties. When we set commitIndex, it will eventually be used to apply the operation to the state machine via

If commitIndex > lastApplied: increment lastApplied, apply
log[lastApplied] to state machine (§5.3)

As to how the bug can manifest, assume leader sends an Append Entries RPC A and RPC B, both were successful. RPC-A was sent before RPC-B but the leader received them out of order i.e., it processed B before A.

AppendEntries RPC-B
PrevLogIdx: 10
PrevLogTerm: 8
Entries: [11, 12, 13] // indices of the entries

AppendEntriesReply
Success = true
nextIndex[peer] = 13
matchIndex[peer] = 12

When we process an older Append Entries response, we must ensure that it doesn’t decrement matchIndex from a newer Append Entries RPC response. If we process RPC-A’s response in this instance, we would be decrementing nextIndex and matchIndex.

AppendEntries RPC-A
PrevLogIdx: 5
PrevLogTerm: 8
Entries: [6, 7] // indices of the entries

AppendEntriesReply // received after B
Success = true

nextIndex[peer] = 8 // WRONG!
matchIndex[peer] = 7 // WRONG!
// Leader and peer already agree till index 12 (matchIndex[peer] = 12 from RPC-B)
// we are removing entries from the peer which were successfully replicated
// and possibly committed and applied to the state machine.

Also, this post will only go over how sequential scan would work in the above MVCC scheme. We will leave Insert, Updates and Deletes for a later post. The goal is to build a solid understanding via pseudo code on how how timestamps are assigned to transactions and how they are used to read data. The idea in Snapshot Isolation (SI) is that every transaction gets a consistent snapshot of data as and when it was created. They then make changes to the data in their own private workspace i.e., the changes aren’t visible to any other transactions. During commit time, transactions check for any possible conflicts in the data that was written wrt running or committed transactions and resolve the conflicts using some scheme such as first committer wins, first updater wins etc. This post will not go over the Commit phase.

The HyPer paper proposes that every row in the table stores a version vector. The version vector is conceptually a pointer to a Linked list (LL). Each node in this LL contains an undo buffer. This undo buffer just stores the change that was made for the row and not the entire modified row. As an example, assume we have the following tuple (1, 2, 3) and a transaction modified it to (11, 2, 3) i.e., only the first attribute was modified. Instead of storing the modified row, the undo buffer only stores (11). We also store metadata about which properties were and weren’t modified so that we can reconstruct the tuple later on (say during a read or sequential scan).

Every transaction gets the following properties,

$\texttt{uint64_t txn_id} \in [2^{62}, \infty)$ (unique per txn) $\texttt{uint64_t read_ts} \in [0, 2^{62})$ - when did this transaction start ? $\texttt{uint64_t commit_ts} \in [0, 2^{62}]$ - when did this transaction commit ? $\texttt{vector} \text{undoBuffer}$ - List of undo logs for this txn

$\texttt{commit_ts} > \texttt{read_ts}$

Every tuple has the following properties,

$\texttt{uint64_t ts}$ — stores the commit timestamp or the transaction id of the running transaction that modified it $\texttt{bool is_deleted}$

Every undo log or undo buffer has the following properties $\texttt{bool is_deleted}$ — whether this undo log corresponds to a deletion $\texttt{uint64_t ts}$ — timestamp corresponding to this undo log $\texttt{Tuple tuple}$ — stores the partial tuple corresponding to the change, not the entire tuple (additional metadata required to reconstruct the tuple is omitted for brevity) $\text{Link}$ // -> This is a pointer to the undo log before $\text{ts}$ if one exists

Assume we have transaction $\text{T}$ that starts at time $\text{t}$, we want to read the tuple $\text{X}$. In SI, every transaction gets its own snapshot of data. $\text{T}$ should see the world as it exists at time $\text{t}$.

When $\text{T}$ begins, it gets assigned a $\texttt{txn_id}$ that is unique and starts from ${2^{62}}$ and upwards. The $\text{read_ts}$ is non-negative and monotonically increasing counter that gets assigned at the start and is ${ < 2^{62}}$.

When $\text{T}$ modifies X, it does two things. It modifies the tuple in-place and sets the $\text{ts} = \text{txn_id}$ and it creates an undo log and stores it in its undo buffer. Lets plug in some numbers to make it more concrete. Initially, $\text{X = 200}$ and $\text{T}$ modifies it to $\text{X = 300}$ at time $\text{t = 9}$. Assume that $\text{txn_id}$ = ${2^{62} +12}$. We will ignore the ${2^{62}}$ and use the remaining number as the transaction identifier.

One mistake I remember doing for the alarm assignment was that somehow I had this notion in mind that I had to copy the alarm handler code into usertrap and execute it there. But that is clearly not the case. If you read the question multiple times (as I did), we just need the kernel to go the alarm handler code, execute that code and return back.

add-auto-load-safe-path ~/xv6-riscv-fall19/.gdbinit

target remote localhost:26000

add-symbol-file ~/xv6-riscv-fall19/user/_lazytests_

Add symbols for whatever files you want to debug.

• gdb shortcuts,

b *filename:line_number* (but bp in file at file number)

c -> continue

n -> step over (next line)

s -> step into

• Do make qemu for building without debugging. Do make qemu-gdb to be able to attach bp & debug.

On thread_exit, thread structures are cleaned up and thread switches to ZOMBIE state.

1. What function or functions handle(s) a context switch?

thread_switch.

1. What does it mean for a thread to be in each of the possible thread states?

S_RUN = Running

S_READY = Ready to Run

S_SLEEP = Sleeping

S_ZOMBIE = exited but not yet deleted.

You’re given three needles/pegs lettered A,B & C from left to right. You have \(n\) disks placed on peg A and you’re goal is to move all \(n\) disks from A to C with the following restrictions

1. You can only move disks one at a time onto peg B or from/to Peg A and C.
2. It is forbidden to place a larger disk onto a smaller one.
3. You can move any number of disks from peg B to any other peg.

The problem is to describe the algorithm and the number of steps it takes i.e. the Running time.

I was assisted by this figure below -

The important clue here is that with the restriction number three, we have only one recursive sub-problem instead of the usual two in the classical Tower of Hanoi problem. Say that you begin with \(n\) disks and you have moved \(n-1\) disks to peg B i.e. the Recursion Fairy has moved those \(n-1\) disks, after this you can move the \(n^{th}\) disk to peg C in one move and the rest of \(n-1\) disks onto peg C in one move (because of the restriction #3).

Consequently the recursion translates to the following equation \(T(n) = T(n-1) + 2 \thinspace \forall \thinspace n \geq 2 \) For the base case of when you have one disk, it only takes one move to move that one disk from any peg to any other. Solving this gives \(T(n) = 2n - 1 \thinspace \forall \thinspace n \geq 1 \)

Right here The paper states that

,

A Log entry is committed once the leader that created the entry has 
replicated it on a majority of the servers.

I misunderstood this statement to assume that whenever an entry is present on a majority of the servers, it is committed. This assumption doesn’t hold.

On Page-8 of the raft paper and in Figure-8

,

In Figure-8.c, It is Term 4 and S1 is leader. Even though the entry at Index 2 Term 2 is present on a majority of the servers, it is entirely possible that that entry can be wiped out as shown in Figure-8.d. In the next Term i.e. Term 5 (whenever new leader is elected term increases) S5 could overwrite entries at indices >= 2 with its own Term 3 entry (because of AppendEntries RPC #3)

If an existing entry conflicts with a new one(same index but different terms), 
delete the existing entry and all that follow it.

In order for this to not happen, the leader in the current term should replicate an entry that it received in the current term on a majority of the servers. In our case, S1 needs to replicate the entry it received in Term 4 on a majority of the servers. After doing that, if S1 crashes before committing the entries, S5 cannot be elected leader and hence won’t overwrite the entries with indices >= 2. The RequestVote RPC would fail for S5 as RequestVoteRPC #2 would not hold, S5’s logs aren’t up-to-date with S1,S2,S2 (i.e. a majority) and hence S5 cannot get votes from a majority of the servers and can’t be elected as leader.

There are two subtle learnings that I could discern -

1. Even if entries from older terms are present on a majority of the servers, that is no guarantee of commitment.
2. In order for an entry to be committed, an entry must be stored on a majority of the servers and the leader that commits it must have replicated an entry from its own current term to a majority of the servers.

To tell the reader (If there are any, comment below if you do exist!!), I have been doing a few things lately. I studied quite a bit of Math and got my basics cemented. I studied Linear Algebra from here and also studied Statistics 110 from here. I did this with the express intention of strengthening my Math background for studying Machine Learning/Artificial Intelligence and the like. Suffice to say, it helped tremendously. And I would gladly suggest this to anyone planning such an endeavour. I still have a few things left to study in Statistics though, Beta,Gamma distributions, Markov Chains and Law of Large Numbers. Which Reminds me that I need to finish them pretty soon. I am also studying Sheldon Axler to get a better grip over abstract linear algebra, I bought the hardcover from Amazon and intend to read it cover-to-cover, but alas the progress has been a bit slow on that front.

Lastly, I am also into Lecture 4 of CS 229. But I think the progress there will slow down a bit, courtesy of the upcoming placement session from Jan 2017. I will keep updating my plans here. See you soon.