Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BREAKING]: Change how Badger handles WAL #1555

Merged
merged 90 commits into from
Oct 7, 2020
Merged

[BREAKING]: Change how Badger handles WAL #1555

merged 90 commits into from
Oct 7, 2020

Commits on Oct 1, 2020

  1. Initial logic written

    @manishrjain
    manishrjain committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    ff6d61a View commit details
    Browse the repository at this point in the history

  2. More changes

    @manishrjain
    manishrjain committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    13e9b19 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2020

  1. Remove head pointer

    @manishrjain
    manishrjain committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    91cf8da View commit details
    Browse the repository at this point in the history

  2. Use mmap to store discard stats. Move to a new discard.go file.

    @manishrjain
    manishrjain committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    f73f11e View commit details
    Browse the repository at this point in the history

  3. Remove discard stats key

    @manishrjain
    manishrjain committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    6fa80d6 View commit details
    Browse the repository at this point in the history

  4. add open wal and open skiplist

    @NamanJain8
    NamanJain8 committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    e50eb08 View commit details
    Browse the repository at this point in the history

  5. Fix skl test

    Ibrahim Jarif committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    4596dd6 View commit details
    Browse the repository at this point in the history

  6. Add test for skiplist open

    Ibrahim Jarif committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    bd756be View commit details
    Browse the repository at this point in the history

  7. Refactor openskiplist

    Ibrahim Jarif committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    628c31c View commit details
    Browse the repository at this point in the history

  8. Add replay code but it doesn't work

    Ibrahim Jarif committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    460501e View commit details
    Browse the repository at this point in the history

  9. more fixes to replay and some tests work

    @NamanJain8
    NamanJain8 committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    8c3a319 View commit details
    Browse the repository at this point in the history

  10. Add some TODOs

    @manishrjain
    manishrjain committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    85f547c View commit details
    Browse the repository at this point in the history

  11. add decrref, increref and function to read from logfile

    @martinmr
    martinmr committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    b4b627e View commit details
    Browse the repository at this point in the history

  12. add open memtable

    @NamanJain8
    NamanJain8 committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    f89448f View commit details
    Browse the repository at this point in the history

  13. More changes. code builds

    @martinmr
    martinmr committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    fca1e71 View commit details
    Browse the repository at this point in the history

  14. write entry go wal.

    @martinmr
    martinmr committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    84628a8 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2020

  1. Refactor code to split it into a new log.go file

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    2416ada View commit details
    Browse the repository at this point in the history

  2. Rename to memtable

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    096f237 View commit details
    Browse the repository at this point in the history

  3. simplify iterate

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    eb68bcc View commit details
    Browse the repository at this point in the history

  4. Deal with ref counting and WAL deletion.

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    125012a View commit details
    Browse the repository at this point in the history

  5. Somewhat working

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    03e0fbb View commit details
    Browse the repository at this point in the history

  6. Remove value log loading mode

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    28ac3c8 View commit details
    Browse the repository at this point in the history

  7. Seems to work.

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    4c39854 View commit details
    Browse the repository at this point in the history

  8. Set size correctly during truncation.

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    35946a4 View commit details
    Browse the repository at this point in the history

  9. Capture Truncate to set logFile.size

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    b739ff4 View commit details
    Browse the repository at this point in the history

  10. Fix in-memory mode and remove some tests

    Ibrahim Jarif committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    5bb2995 View commit details
    Browse the repository at this point in the history

  11. fix skl_test

    @NamanJain8
    NamanJain8 committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    5dd8125 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'mrjn/wal' of github.com:dgraph-io/badger into mrjn/wal

    @NamanJain8
    NamanJain8 committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    99d0ad7 View commit details
    Browse the repository at this point in the history

  13. Avoid zeroing out the entire discard stats file.

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    8290e85 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'master' into mrjn/wal

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    0ea920a View commit details
    Browse the repository at this point in the history

  15. Copmiles

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    c394424 View commit details
    Browse the repository at this point in the history

  16. Fix up MaxVersion function

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    93b612e View commit details
    Browse the repository at this point in the history

  17. Remove vlog sampling and trace.

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    a307270 View commit details
    Browse the repository at this point in the history

  18. fix some tests

    Ibrahim Jarif committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    c8de40a View commit details
    Browse the repository at this point in the history

  19. enable TestTxnReadTs test

    Ibrahim Jarif committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    5c53171 View commit details
    Browse the repository at this point in the history

  20. Empty builder should return slice

    Ibrahim Jarif committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    f21b48c View commit details
    Browse the repository at this point in the history

  21. Add assert to catch failure

    Ibrahim Jarif committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    8db3d11 View commit details
    Browse the repository at this point in the history

  22. Return 0 as MaxVersion if rcv is nil.

    @martinmr
    martinmr committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    a3e096c View commit details
    Browse the repository at this point in the history

  23. Check rcv is not nil in KeyCount.

    @martinmr
    martinmr committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    a3ea60b View commit details
    Browse the repository at this point in the history

  24. more nil checks.

    @martinmr
    martinmr committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    02afa26 View commit details
    Browse the repository at this point in the history

  25. Add assert that index is not nil.

    @martinmr
    martinmr committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    2cda2f9 View commit details
    Browse the repository at this point in the history

  26. Regenerate TableIndex.go

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    e7d85f5 View commit details
    Browse the repository at this point in the history

  27. Fix segfault in monitorCache.

    @martinmr
    martinmr committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    7055781 View commit details
    Browse the repository at this point in the history

  28. Set t.index to point to the mmapped table always, unless encryption i… 

    …s on. In that case, ensure there's a cache set.
    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    115242c View commit details
    Browse the repository at this point in the history

  29. Remove TableLoadingMode. Defaults to mmap

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    08cf171 View commit details
    Browse the repository at this point in the history

  30. Use z.MmapFile and z.Mmap

    @manishrjain
    manishrjain committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    d03573b View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2020

  1. Remove truncate option. Move to y.Wrap and y.Wrapf

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    249b006 View commit details
    Browse the repository at this point in the history

  2. Zero out the next entry when writing log file.

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    443d8ee View commit details
    Browse the repository at this point in the history

  3. Fix more tests

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    adc66b0 View commit details
    Browse the repository at this point in the history

  4. fix more tests

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    c7f7a7c View commit details
    Browse the repository at this point in the history

  5. fix another test

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    848a1bb View commit details
    Browse the repository at this point in the history

  6. Remove vloggc command

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    2c4e672 View commit details
    Browse the repository at this point in the history

  7. fix table tests

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    7e7f4f4 View commit details
    Browse the repository at this point in the history

  8. All tests work

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    9538a36 View commit details
    Browse the repository at this point in the history

  9. Merge remote-tracking branch 'origin/master' into mrjn/wal

    @martinmr
    martinmr committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    d80a2cc View commit details
    Browse the repository at this point in the history

  10. Fix uint32 overflow test

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    9dc23c0 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'mrjn/wal' of https://github.com/dgraph-io/badger into m… 

    …rjn/wal
    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    dff5229 View commit details
    Browse the repository at this point in the history

  12. Fix discard back

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    5bb651c View commit details
    Browse the repository at this point in the history

  13. Fix Swap

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    dc6904e View commit details
    Browse the repository at this point in the history

  14. Fix another test

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    a859f51 View commit details
    Browse the repository at this point in the history

  15. Add locks in MaxVersion

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    38b264c View commit details
    Browse the repository at this point in the history

  16. Merge branch 'mrjn/wal' of https://github.com/dgraph-io/badger into m… 

    …rjn/wal
    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    7d8b677 View commit details
    Browse the repository at this point in the history

  17. Add a CreateTable API, which internally just copies bytes from builde… 

    …r.Finish to mmap.
    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    01131cb View commit details
    Browse the repository at this point in the history

  18. Move Badger Flags to its own type

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    ad06fd6 View commit details
    Browse the repository at this point in the history

  19. Fix up shit ton of tests

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    8b32add View commit details
    Browse the repository at this point in the history

  20. fix(txn): Allow the first big entry to pass

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    92d5ef0 View commit details
    Browse the repository at this point in the history

  21. Set sync writes to false

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    be413df View commit details
    Browse the repository at this point in the history

  22. Merge remote-tracking branch 'origin/ibrahim/txn-size' into mrjn/wal

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    308c5a8 View commit details
    Browse the repository at this point in the history

  23. Set sync writes to false by default. Update magic number.v

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    53fe46b View commit details
    Browse the repository at this point in the history

  24. Comment out replace in go.mod

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    9fbb7d1 View commit details
    Browse the repository at this point in the history

  25. Revert "fix(txn): Allow the first big entry to pass" 

    This reverts commit 92d5ef0.
    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    ec49200 View commit details
    Browse the repository at this point in the history

  26. Fix picksorttable

    Ibrahim Jarif committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    9486089 View commit details
    Browse the repository at this point in the history

  27. Pull in latest ristretto.

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    6e9b66b View commit details
    Browse the repository at this point in the history

  28. tidy go.sum

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    7090cf8 View commit details
    Browse the repository at this point in the history

  29. Fix up a seg fault in levels.go

    @manishrjain
    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    4c28427 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2020

  1. Add check for nil logFile

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    b9f075a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mrjn/wal' of https://github.com/dgraph-io/badger into m… 

    …rjn/wal
    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    d74698d View commit details
    Browse the repository at this point in the history

  3. Revert "Fix(builder): Too many small tables when compression is enabl… 

    …ed (#1549)"

This reverts commit 5d1bab4.
We'reverting this commit because it seems to cause a strange issue while
writing data. Running `go run -tags main.go benchmark write --sorted
--compression=true --block-cache-mb=100` creates a directory which does
not have all the keys. I don't know why this would fix the issue but the
test works fine after reverting this commit.
    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    76a893f View commit details
    Browse the repository at this point in the history

  4. Fix discard stats cleanup

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    998f30c View commit details
    Browse the repository at this point in the history

  5. Add a test for discard stats

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    e9a6ec3 View commit details
    Browse the repository at this point in the history

  6. Add license

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    9555605 View commit details
    Browse the repository at this point in the history

  7. Do not GC files with discard less than discard ratio

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    8fae622 View commit details
    Browse the repository at this point in the history

  8. Don't rewrite the vlog file already marked for deletion

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    dd9e5d4 View commit details
    Browse the repository at this point in the history

  9. Revert "Revert "Fix(builder): Too many small tables when compression … 

    …is enabled (#1549)""

This reverts commit 76a893f.
    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    3db8512 View commit details
    Browse the repository at this point in the history

  10. Use go mem instead of z.Calloc for builder buffer

    Ibrahim Jarif committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    1082dbe View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. Set default conflict detection to false in write benchmark

    asdf committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    ef21567 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2020

  1. Self review

    @manishrjain
    manishrjain committed Oct 11, 2020
    Configuration menu
    Copy the full SHA
    6f3fad6 View commit details
    Browse the repository at this point in the history

  2. Remove KeepL0InMemory option.

    @manishrjain
    manishrjain committed Oct 11, 2020
    Configuration menu
    Copy the full SHA
    a5daea1 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2020

  1. Self review

    @manishrjain
    manishrjain committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    99f4f92 View commit details
    Browse the repository at this point in the history

  2. Self review

    @manishrjain
    manishrjain committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    ac7688c View commit details
    Browse the repository at this point in the history