forked from dgraph-io/badger
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BREAKING] opt(compactions): Improve compaction performance (dgraph-i…
…o#1574) Implement multiple ideas for speeding up compactions: 1. Dynamic Level Sizes: https://rocksdb.org/blog/2015/07/23/dynamic-level.html 2. L0 to L0 compactions: https://rocksdb.org/blog/2017/06/26/17-level-based-changes.html 3. Sub Compactions: Split up one compaction into multiple sub-compactions using key ranges, which can be run concurrently. 4. If a table being generated at Li overlaps with >= 10 tables at Li+1, finish the table. This helps avoid big overlaps and expensive compactions later. 5. Update compaction priority based on the priority of the next level prioritizing compactions of lower levels over upper levels, resulting in an always healthy LSM tree structure. With these changes, we can load 1B entries (160GB of data) into Badger (without the Stream framework) in 1h25m at 31 MB/s. This is a significant improvement over current master. Co-authored-by: Ibrahim Jarif <[email protected]>
- Loading branch information
1 parent
2b71a94
commit 45bca18
Showing
22 changed files
with
843 additions
and
406 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.