opt(stream): add option to directly copy over tables from lower levels

[manishrjain]

This PR adds FullCopy option in Stream. This allows sending the table entirely to the writer. If this option is set to true we directly copy over the tables from the last 2 levels. This option increases the stream speed while also lowering the memory consumption on the DB that is streaming the KVs.
For 71GB, compressed and encrypted DB we observed 3x improvement in speed. The DB contained ~65GB in the last 2 levels while remaining in the above levels.

	Time taken	CPU usage (% user, system, io, idle)
master	22m15s	37.2, 5.4, 5.9, 53.4
PR	8m20s	1.2, 2.6, 7.8, 88.8
master-wo-write	7m30s	51.45, 4.1, 0.1, 44.34
PR-wo-write	2m50s	0.5, 3, 4, 92

To use this option, the following options should be set in Stream.

stream.KeyToList = nil
stream.ChooseKey = nil
stream.SinceTs = 0
db.managedTxns = true

If we use stream writer for receiving the KVs, the encryption mode has to be the same in sender and receiver. This will restrict db.StreamDB() to use same encryption mode in both input and output DB. Added TODO for allowing different encryption modes.

This change is 

[manishrjain]

Added comments.

Reviewed 3 of 8 files at r1, 1 of 6 files at r2, 8 of 8 files at r3.
Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @manishrjain)

---

key_registry.go, line 390 at r3 (raw file):

	if kr.lastCreated < dk.CreatedAt {
		kr.lastCreated = dk.CreatedAt

No need to update this. We didn't create this.

---

key_registry.go, line 393 at r3 (raw file):

	}

	kr.dataKeys[dk.KeyId] = &dk

Just check if you already have this keyID or not. If not, then set it. If you do, then just change the key ID?

for that matter, you can just always use the nextKeyId. So, basically never use the KeyId that was passed. It's just safer.

---

levels.go, line 1772 at r3 (raw file):

	var change pb.ManifestChange
	if err := proto.Unmarshal(kv.Key, &change); err != nil {
		return err

errors.Wrapf

---

levels.go, line 1777 at r3 (raw file):

	dk, err := lc.kv.registry.DataKey(change.KeyId)
	if err != nil {
		return err

errors.Wrapf

---

levels.go, line 1788 at r3 (raw file):

	// Create a copy of the kv.Value because it is owned by the z.buffer.
	tbl, err := table.CreateTableFromBuffer(fname, y.Copy(kv.Value), opts)

error?

---

levels.go, line 1792 at r3 (raw file):

	// TODO(ibrahim): Check all the increment refs are done correctly.
	// Tables are sent in the sorted order, so no need to sort them here.
	lc.levels[lev].addTable(tbl)

Can you add a TODO that encryption / decryption might be required for the table. In fact, it shouldn't be too hard -- just do inplace encrypt / decrypt.

Just ensure you're not doing that from within a level controller lock.

---

stream.go, line 90 at r3 (raw file):

	// CopyTables should be set to true only when the and encryption is enabled/disabled in both
	// the sender and receiver.
	CopyTables   bool

FullCopy bool

If FullCopy is set, then SinceTs == 0, and ChooseKey == nil, and KeyToList == nil. If any of them are set, do a panic.

---

stream.go, line 377 at r3 (raw file):

		level := i
		tables := tableMatrix[i]
		for _, t := range tables {

Add a TODO here to see if making this concurrent would be helpful. Most likely it won't. But, if it does work, then most like <3 goroutines might be sufficient.

---

stream.go, line 459 at r3 (raw file):

	// Picks up ranges from Badger, and sends them to rangeCh.
	go st.produceRanges(ctx)

Just put a comment here that just for simplicity, we'd still consider all the tables for range production.

---

value.go, line 791 at r3 (raw file):

// write is thread-unsafe by design and should not be called concurrently.
func (vlog *valueLog) write(reqs []*request) error {
	if vlog.db.opt.InMemory || vlog.db.opt.managedTxns {

If Badger starts in managed mode, can we not create a new value log?

[manishrjain]

Some comments. Nice work!

Reviewed 1 of 8 files at r1, 7 of 7 files at r4.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @jarifibrahim)

---

level_handler.go, line 355 at r4 (raw file):

		}
		return out
	}

Vertical space after this. To represent that we returned above.

---

levels.go, line 1795 at r4 (raw file):

	lc.levels[lev].addTable(tbl)
	// Release the ref held by OpenTable.

Mention in comment: addTable would add a reference.

---

stream_writer.go, line 54 at r4 (raw file):

	senderPrevLevel int
	keyId           map[uint64]*pb.DataKey // stores reader's keyId to data key map.
	processingKeys  bool                   // true if we have started processing keys.

Add a comment about the writer might receive tables first, and then receive keys. So, this would tell us which stage we are in.

---

value.go, line 791 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

If Badger starts in managed mode, can we not create a new value log?

Add a TODO.

[jarifibrahim]

Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @manishrjain)