Documentation
¶
Overview ¶
Package badger implements an embeddable, simple and fast key-value database, written in pure Go. It is designed to be highly performant for both reads and writes simultaneously. Badger uses Multi-Version Concurrency Control (MVCC), and supports transactions. It runs transactions concurrently, with serializable snapshot isolation guarantees.
Badger uses an LSM tree along with a value log to separate keys from values, hence reducing both write amplification and the size of the LSM tree. This allows LSM tree to be served entirely from RAM, while the values are served from SSD.
Usage ¶
Badger has the following main types: DB, Txn, Item and Iterator. DB contains keys that are associated with values. It must be opened with the appropriate options before it can be accessed.
All operations happen inside a Txn. Txn represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given key (which are returned inside an Item), or iterate over a set of key-value pairs using an Iterator (which are returned as Item type values as well). Read-write transactions can also update and delete keys from the DB.
See the examples for more usage details.
Index ¶
- Constants
- Variables
- type DB
- func (db *DB) Backup(w io.Writer, since uint64) (uint64, error)
- func (db *DB) Close() error
- func (db *DB) DropAll() error
- func (db *DB) DropPrefix(prefix []byte) error
- func (db *DB) Flatten(workers int) error
- func (db *DB) GetMergeOperator(key []byte, f MergeFunc, dur time.Duration) *MergeOperator
- func (db *DB) GetSequence(key []byte, bandwidth uint64) (*Sequence, error)
- func (db *DB) KeySplits(prefix []byte) []string
- func (db *DB) Load(r io.Reader, maxPendingWrites int) error
- func (db *DB) MaxBatchCount() int64
- func (db *DB) MaxBatchSize() int64
- func (db *DB) NewKVLoader(maxPendingWrites int) *KVLoader
- func (db *DB) NewStream() *Stream
- func (db *DB) NewStreamAt(readTs uint64) *Stream
- func (db *DB) NewStreamWriter() *StreamWriter
- func (db *DB) NewTransaction(update bool) *Txn
- func (db *DB) NewTransactionAt(readTs uint64, update bool) *Txn
- func (db *DB) NewWriteBatch() *WriteBatch
- func (db *DB) NewWriteBatchAt(commitTs uint64) *WriteBatch
- func (db *DB) PrintHistogram(keyPrefix []byte)
- func (db *DB) RunValueLogGC(discardRatio float64) error
- func (db *DB) SetDiscardTs(ts uint64)
- func (db *DB) Size() (lsm, vlog int64)
- func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, prefixes ...[]byte) error
- func (db *DB) Sync() error
- func (db *DB) Tables(withKeysCount bool) []TableInfo
- func (db *DB) Update(fn func(txn *Txn) error) error
- func (db *DB) View(fn func(txn *Txn) error) error
- type Entry
- type Item
- func (item *Item) DiscardEarlierVersions() bool
- func (item *Item) EstimatedSize() int64
- func (item *Item) ExpiresAt() uint64
- func (item *Item) IsDeletedOrExpired() bool
- func (item *Item) Key() []byte
- func (item *Item) KeyCopy(dst []byte) []byte
- func (item *Item) KeySize() int64
- func (item *Item) String() string
- func (item *Item) UserMeta() byte
- func (item *Item) Value(fn func(val []byte) error) error
- func (item *Item) ValueCopy(dst []byte) ([]byte, error)
- func (item *Item) ValueSize() int64
- func (item *Item) Version() uint64
- type Iterator
- type IteratorOptions
- type KVList
- type KVLoader
- type Logger
- type Manifest
- type MergeFunc
- type MergeOperator
- type Options
- func (opt *Options) Debugf(format string, v ...interface{})
- func (opt *Options) Errorf(format string, v ...interface{})
- func (opt *Options) Infof(format string, v ...interface{})
- func (opt *Options) Warningf(format string, v ...interface{})
- func (opt Options) WithBypassLockGuard(b bool) Options
- func (opt Options) WithCompactL0OnClose(val bool) Options
- func (opt Options) WithDir(val string) Options
- func (opt Options) WithEventLogging(enabled bool) Options
- func (opt Options) WithLevelOneSize(val int64) Options
- func (opt Options) WithLevelSizeMultiplier(val int) Options
- func (opt Options) WithLogRotatesToFlush(val int32) Options
- func (opt Options) WithLogger(val Logger) Options
- func (opt Options) WithMaxLevels(val int) Options
- func (opt Options) WithMaxTableSize(val int64) Options
- func (opt Options) WithNumCompactors(val int) Options
- func (opt Options) WithNumLevelZeroTables(val int) Options
- func (opt Options) WithNumLevelZeroTablesStall(val int) Options
- func (opt Options) WithNumMemtables(val int) Options
- func (opt Options) WithNumVersionsToKeep(val int) Options
- func (opt Options) WithReadOnly(val bool) Options
- func (opt Options) WithSyncWrites(val bool) Options
- func (opt Options) WithTableLoadingMode(val options.FileLoadingMode) Options
- func (opt Options) WithTruncate(val bool) Options
- func (opt Options) WithValueDir(val string) Options
- func (opt Options) WithValueLogFileSize(val int64) Options
- func (opt Options) WithValueLogLoadingMode(val options.FileLoadingMode) Options
- func (opt Options) WithValueLogMaxEntries(val uint32) Options
- func (opt Options) WithValueThreshold(val int) Options
- func (opt Options) WithVerifyValueChecksum(val bool) Options
- type Sequence
- type Stream
- type StreamWriter
- type TableInfo
- type TableManifest
- type Txn
- func (txn *Txn) Commit() error
- func (txn *Txn) CommitAt(commitTs uint64, callback func(error)) error
- func (txn *Txn) CommitWith(cb func(error))
- func (txn *Txn) Delete(key []byte) error
- func (txn *Txn) Discard()
- func (txn *Txn) Get(key []byte) (item *Item, rerr error)
- func (txn *Txn) NewIterator(opt IteratorOptions) *Iterator
- func (txn *Txn) NewKeyIterator(key []byte, opt IteratorOptions) *Iterator
- func (txn *Txn) ReadTs() uint64
- func (txn *Txn) Set(key, val []byte) error
- func (txn *Txn) SetEntry(e *Entry) error
- type WriteBatch
Examples ¶
Constants ¶
const (
// ManifestFilename is the filename for the manifest file.
ManifestFilename = "MANIFEST"
)
const ( // ValueThresholdLimit is the maximum permissible value of opt.ValueThreshold. ValueThresholdLimit = math.MaxUint16 - 16 + 1 )
Variables ¶
var ( // ErrValueLogSize is returned when opt.ValueLogFileSize option is not within the valid // range. ErrValueLogSize = errors.New("Invalid ValueLogFileSize, must be between 1MB and 2GB") // ErrValueThreshold is returned when ValueThreshold is set to a value close to or greater than // uint16. ErrValueThreshold = errors.Errorf( "Invalid ValueThreshold, must be less than %d", ValueThresholdLimit) // ErrKeyNotFound is returned when key isn't found on a txn.Get. ErrKeyNotFound = errors.New("Key not found") // ErrTxnTooBig is returned if too many writes are fit into a single transaction. ErrTxnTooBig = errors.New("Txn is too big to fit into one request") // ErrConflict is returned when a transaction conflicts with another transaction. This can // happen if the read rows had been updated concurrently by another transaction. ErrConflict = errors.New("Transaction Conflict. Please retry") // ErrReadOnlyTxn is returned if an update function is called on a read-only transaction. ErrReadOnlyTxn = errors.New("No sets or deletes are allowed in a read-only transaction") // ErrDiscardedTxn is returned if a previously discarded transaction is re-used. ErrDiscardedTxn = errors.New("This transaction has been discarded. Create a new one") // ErrEmptyKey is returned if an empty key is passed on an update function. ErrEmptyKey = errors.New("Key cannot be empty") // ErrInvalidKey is returned if the key has a special !badger! prefix, // reserved for internal usage. ErrInvalidKey = errors.New("Key is using a reserved !badger! prefix") // ErrRetry is returned when a log file containing the value is not found. // This usually indicates that it may have been garbage collected, and the // operation needs to be retried. ErrRetry = errors.New("Unable to find log file. Please retry") // ErrThresholdZero is returned if threshold is set to zero, and value log GC is called. // In such a case, GC can't be run. ErrThresholdZero = errors.New( "Value log GC can't run because threshold is set to zero") // ErrNoRewrite is returned if a call for value log GC doesn't result in a log file rewrite. ErrNoRewrite = errors.New( "Value log GC attempt didn't result in any cleanup") // ErrRejected is returned if a value log GC is called either while another GC is running, or // after DB::Close has been called. ErrRejected = errors.New("Value log GC request rejected") // ErrInvalidRequest is returned if the user request is invalid. ErrInvalidRequest = errors.New("Invalid request") // ErrManagedTxn is returned if the user tries to use an API which isn't // allowed due to external management of transactions, when using ManagedDB. ErrManagedTxn = errors.New( "Invalid API request. Not allowed to perform this action using ManagedDB") // ErrInvalidDump if a data dump made previously cannot be loaded into the database. ErrInvalidDump = errors.New("Data dump cannot be read") // ErrZeroBandwidth is returned if the user passes in zero bandwidth for sequence. ErrZeroBandwidth = errors.New("Bandwidth must be greater than zero") // ErrInvalidLoadingMode is returned when opt.ValueLogLoadingMode option is not // within the valid range ErrInvalidLoadingMode = errors.New("Invalid ValueLogLoadingMode, must be FileIO or MemoryMap") // ErrReplayNeeded is returned when opt.ReadOnly is set but the // database requires a value log replay. ErrReplayNeeded = errors.New("Database was not properly closed, cannot open read-only") // ErrWindowsNotSupported is returned when opt.ReadOnly is used on Windows ErrWindowsNotSupported = errors.New("Read-only mode is not supported on Windows") // ErrTruncateNeeded is returned when the value log gets corrupt, and requires truncation of // corrupt data to allow Badger to run properly. ErrTruncateNeeded = errors.New( "Value log truncate required to run DB. This might result in data loss") // ErrBlockedWrites is returned if the user called DropAll. During the process of dropping all // data from Badger, we stop accepting new writes, by returning this error. ErrBlockedWrites = errors.New("Writes are blocked, possibly due to DropAll or Close") // ErrNilCallback is returned when subscriber's callback is nil. ErrNilCallback = errors.New("Callback cannot be nil") )
var DefaultIteratorOptions = IteratorOptions{ PrefetchValues: true, PrefetchSize: 100, Reverse: false, AllVersions: false, }
DefaultIteratorOptions contains default options when iterating over Badger key-value stores.
var ErrUnsortedKey = errors.New("Keys not in sorted order")
ErrUnsortedKey is returned when any out of order key arrives at sortedWriter during call to Add.
Functions ¶
This section is empty.
Types ¶
type DB ¶
type DB struct { sync.RWMutex // Guards list of inmemory tables, not individual reads and writes. // contains filtered or unexported fields }
DB provides the various functions required to interact with Badger. DB is thread-safe.
func Open ¶
Open returns a new DB object.
Example ¶
dir, err := ioutil.TempDir("", "badger-test") if err != nil { panic(err) } defer removeDir(dir) db, err := Open(DefaultOptions(dir)) if err != nil { panic(err) } defer db.Close() err = db.View(func(txn *Txn) error { _, err := txn.Get([]byte("key")) // We expect ErrKeyNotFound fmt.Println(err) return nil }) if err != nil { panic(err) } txn := db.NewTransaction(true) // Read-write txn err = txn.SetEntry(NewEntry([]byte("key"), []byte("value"))) if err != nil { panic(err) } err = txn.Commit() if err != nil { panic(err) } err = db.View(func(txn *Txn) error { item, err := txn.Get([]byte("key")) if err != nil { return err } val, err := item.ValueCopy(nil) if err != nil { return err } fmt.Printf("%s\n", string(val)) return nil }) if err != nil { panic(err) }
Output: Key not found value
func OpenManaged ¶
OpenManaged returns a new DB, which allows more control over setting transaction timestamps, aka managed mode.
This is only useful for databases built on top of Badger (like Dgraph), and can be ignored by most users.
func (*DB) Backup ¶
Backup is a wrapper function over Stream.Backup to generate full and incremental backups of the DB. For more control over how many goroutines are used to generate the backup, or if you wish to backup only a certain range of keys, use Stream.Backup directly.
func (*DB) Close ¶
Close closes a DB. It's crucial to call it to ensure all the pending updates make their way to disk. Calling DB.Close() multiple times would still only close the DB once.
func (*DB) DropAll ¶
DropAll would drop all the data stored in Badger. It does this in the following way. - Stop accepting new writes. - Pause memtable flushes and compactions. - Pick all tables from all levels, create a changeset to delete all these tables and apply it to manifest. - Pick all log files from value log, and delete all of them. Restart value log files from zero. - Resume memtable flushes and compactions.
NOTE: DropAll is resilient to concurrent writes, but not to reads. It is up to the user to not do any reads while DropAll is going on, otherwise they may result in panics. Ideally, both reads and writes are paused before running DropAll, and resumed after it is finished.
func (*DB) DropPrefix ¶
DropPrefix would drop all the keys with the provided prefix. It does this in the following way:
- Stop accepting new writes.
- Stop memtable flushes before acquiring lock. Because we're acquring lock here and memtable flush stalls for lock, which leads to deadlock
- Flush out all memtables, skipping over keys with the given prefix, Kp.
- Write out the value log header to memtables when flushing, so we don't accidentally bring Kp back after a restart.
- Stop compaction.
- Compact L0->L1, skipping over Kp.
- Compact rest of the levels, Li->Li, picking tables which have Kp.
- Resume memtable flushes, compactions and writes.
func (*DB) Flatten ¶
Flatten can be used to force compactions on the LSM tree so all the tables fall on the same level. This ensures that all the versions of keys are colocated and not split across multiple levels, which is necessary after a restore from backup. During Flatten, live compactions are stopped. Ideally, no writes are going on during Flatten. Otherwise, it would create competition between flattening the tree and new tables being created at level zero.
func (*DB) GetMergeOperator ¶
GetMergeOperator creates a new MergeOperator for a given key and returns a pointer to it. It also fires off a goroutine that performs a compaction using the merge function that runs periodically, as specified by dur.
func (*DB) GetSequence ¶
GetSequence would initiate a new sequence object, generating it from the stored lease, if available, in the database. Sequence can be used to get a list of monotonically increasing integers. Multiple sequences can be created by providing different keys. Bandwidth sets the size of the lease, determining how many Next() requests can be served from memory.
GetSequence is not supported on ManagedDB. Calling this would result in a panic.
func (*DB) KeySplits ¶
KeySplits can be used to get rough key ranges to divide up iteration over the DB.
func (*DB) Load ¶
Load reads a protobuf-encoded list of all entries from a reader and writes them to the database. This can be used to restore the database from a backup made by calling DB.Backup(). If more complex logic is needed to restore a badger backup, the KVLoader interface should be used instead.
DB.Load() should be called on a database that is not running any other concurrent transactions while it is running.
func (*DB) MaxBatchCount ¶
MaxBatchCount returns max possible entries in batch
func (*DB) MaxBatchSize ¶
MaxBatchSize returns max possible batch size
func (*DB) NewKVLoader ¶
NewKVLoader returns a new instance of KVLoader.
func (*DB) NewStreamAt ¶
NewStreamAt creates a new Stream at a particular timestamp. Should only be used with managed DB.
func (*DB) NewStreamWriter ¶
func (db *DB) NewStreamWriter() *StreamWriter
NewStreamWriter creates a StreamWriter. Right after creating StreamWriter, Prepare must be called. The memory usage of a StreamWriter is directly proportional to the number of streams possible. So, efforts must be made to keep the number of streams low. Stream framework would typically use 16 goroutines and hence create 16 streams.
func (*DB) NewTransaction ¶
NewTransaction creates a new transaction. Badger supports concurrent execution of transactions, providing serializable snapshot isolation, avoiding write skews. Badger achieves this by tracking the keys read and at Commit time, ensuring that these read keys weren't concurrently modified by another transaction.
For read-only transactions, set update to false. In this mode, we don't track the rows read for any changes. Thus, any long running iterations done in this mode wouldn't pay this overhead.
Running transactions concurrently is OK. However, a transaction itself isn't thread safe, and should only be run serially. It doesn't matter if a transaction is created by one goroutine and passed down to other, as long as the Txn APIs are called serially.
When you create a new transaction, it is absolutely essential to call Discard(). This should be done irrespective of what the update param is set to. Commit API internally runs Discard, but running it twice wouldn't cause any issues.
txn := db.NewTransaction(false) defer txn.Discard() // Call various APIs.
func (*DB) NewTransactionAt ¶
NewTransactionAt follows the same logic as DB.NewTransaction(), but uses the provided read timestamp.
This is only useful for databases built on top of Badger (like Dgraph), and can be ignored by most users.
func (*DB) NewWriteBatch ¶
func (db *DB) NewWriteBatch() *WriteBatch
NewWriteBatch creates a new WriteBatch. This provides a way to conveniently do a lot of writes, batching them up as tightly as possible in a single transaction and using callbacks to avoid waiting for them to commit, thus achieving good performance. This API hides away the logic of creating and committing transactions. Due to the nature of SSI guaratees provided by Badger, blind writes can never encounter transaction conflicts (ErrConflict).
func (*DB) NewWriteBatchAt ¶
func (db *DB) NewWriteBatchAt(commitTs uint64) *WriteBatch
NewWriteBatchAt is similar to NewWriteBatch but it allows user to set the commit timestamp. NewWriteBatchAt is supposed to be used only in the managed mode.
func (*DB) PrintHistogram ¶
PrintHistogram builds and displays the key-value size histogram. When keyPrefix is set, only the keys that have prefix "keyPrefix" are considered for creating the histogram
func (*DB) RunValueLogGC ¶
RunValueLogGC triggers a value log garbage collection.
It picks value log files to perform GC based on statistics that are collected during compactions. If no such statistics are available, then log files are picked in random order. The process stops as soon as the first log file is encountered which does not result in garbage collection.
When a log file is picked, it is first sampled. If the sample shows that we can discard at least discardRatio space of that file, it would be rewritten.
If a call to RunValueLogGC results in no rewrites, then an ErrNoRewrite is thrown indicating that the call resulted in no file rewrites.
We recommend setting discardRatio to 0.5, thus indicating that a file be rewritten if half the space can be discarded. This results in a lifetime value log write amplification of 2 (1 from original write + 0.5 rewrite + 0.25 + 0.125 + ... = 2). Setting it to higher value would result in fewer space reclaims, while setting it to a lower value would result in more space reclaims at the cost of increased activity on the LSM tree. discardRatio must be in the range (0.0, 1.0), both endpoints excluded, otherwise an ErrInvalidRequest is returned.
Only one GC is allowed at a time. If another value log GC is running, or DB has been closed, this would return an ErrRejected.
Note: Every time GC is run, it would produce a spike of activity on the LSM tree.
func (*DB) SetDiscardTs ¶
SetDiscardTs sets a timestamp at or below which, any invalid or deleted versions can be discarded from the LSM tree, and thence from the value log to reclaim disk space. Can only be used with managed transactions.
func (*DB) Size ¶
Size returns the size of lsm and value log files in bytes. It can be used to decide how often to call RunValueLogGC.
func (*DB) Subscribe ¶
Subscribe can be used to watch key changes for the given key prefixes. At least one prefix should be passed, or an error will be returned. You can use an empty prefix to monitor all changes to the DB. This function blocks until the given context is done or an error occurs. The given function will be called with a new KVList containing the modified keys and the corresponding values.
func (*DB) Sync ¶
Sync syncs database content to disk. This function provides more control to user to sync data whenever required.
func (*DB) Tables ¶
Tables gets the TableInfo objects from the level controller. If withKeysCount is true, TableInfo objects also contain counts of keys for the tables.
func (*DB) Update ¶
Update executes a function, creating and managing a read-write transaction for the user. Error returned by the function is relayed by the Update method. Update cannot be used with managed transactions.
type Entry ¶
type Entry struct { Key []byte Value []byte UserMeta byte ExpiresAt uint64 // time.Unix // contains filtered or unexported fields }
Entry provides Key, Value, UserMeta and ExpiresAt. This struct can be used by the user to set data.
func NewEntry ¶
NewEntry creates a new entry with key and value passed in args. This newly created entry can be set in a transaction by calling txn.SetEntry(). All other properties of Entry can be set by calling WithMeta, WithDiscard, WithTTL methods on it. This function uses key and value reference, hence users must not modify key and value until the end of transaction.
func (*Entry) WithDiscard ¶
WithDiscard adds a marker to Entry e. This means all the previous versions of the key (of the Entry) will be eligible for garbage collection. This method is only useful if you have set a higher limit for options.NumVersionsToKeep. The default setting is 1, in which case, this function doesn't add any more benefit. If however, you have a higher setting for NumVersionsToKeep (in Dgraph, we set it to infinity), you can use this method to indicate that all the older versions can be discarded and removed during compactions.
type Item ¶
type Item struct {
// contains filtered or unexported fields
}
Item is returned during iteration. Both the Key() and Value() output is only valid until iterator.Next() is called.
func (*Item) DiscardEarlierVersions ¶
DiscardEarlierVersions returns whether the item was created with the option to discard earlier versions of a key when multiple are available.
func (*Item) EstimatedSize ¶
EstimatedSize returns the approximate size of the key-value pair.
This can be called while iterating through a store to quickly estimate the size of a range of key-value pairs (without fetching the corresponding values).
func (*Item) ExpiresAt ¶
ExpiresAt returns a Unix time value indicating when the item will be considered expired. 0 indicates that the item will never expire.
func (*Item) IsDeletedOrExpired ¶
IsDeletedOrExpired returns true if item contains deleted or expired value.
func (*Item) Key ¶
Key returns the key.
Key is only valid as long as item is valid, or transaction is valid. If you need to use it outside its validity, please use KeyCopy.
func (*Item) KeyCopy ¶
KeyCopy returns a copy of the key of the item, writing it to dst slice. If nil is passed, or capacity of dst isn't sufficient, a new slice would be allocated and returned.
func (*Item) KeySize ¶
KeySize returns the size of the key. Exact size of the key is key + 8 bytes of timestamp
func (*Item) UserMeta ¶
UserMeta returns the userMeta set by the user. Typically, this byte, optionally set by the user is used to interpret the value.
func (*Item) Value ¶
Value retrieves the value of the item from the value log.
This method must be called within a transaction. Calling it outside a transaction is considered undefined behavior. If an iterator is being used, then Item.Value() is defined in the current iteration only, because items are reused.
If you need to use a value outside a transaction, please use Item.ValueCopy instead, or copy it yourself. Value might change once discard or commit is called. Use ValueCopy if you want to do a Set after Get.
func (*Item) ValueCopy ¶
ValueCopy returns a copy of the value of the item from the value log, writing it to dst slice. If nil is passed, or capacity of dst isn't sufficient, a new slice would be allocated and returned. Tip: It might make sense to reuse the returned slice as dst argument for the next call.
This function is useful in long running iterate/update transactions to avoid a write deadlock. See Github issue: https://github.com/meeyio/badger/issues/315
type Iterator ¶
type Iterator struct {
// contains filtered or unexported fields
}
Iterator helps iterating over the KV pairs in a lexicographically sorted order.
func (*Iterator) Close ¶
func (it *Iterator) Close()
Close would close the iterator. It is important to call this when you're done with iteration.
func (*Iterator) Item ¶
Item returns pointer to the current key-value pair. This item is only valid until it.Next() gets called.
func (*Iterator) Next ¶
func (it *Iterator) Next()
Next would advance the iterator by one. Always check it.Valid() after a Next() to ensure you have access to a valid it.Item().
func (*Iterator) Rewind ¶
func (it *Iterator) Rewind()
Rewind would rewind the iterator cursor all the way to zero-th position, which would be the smallest key if iterating forward, and largest if iterating backward. It does not keep track of whether the cursor started with a Seek().
func (*Iterator) Seek ¶
Seek would seek to the provided key if present. If absent, it would seek to the next smallest key greater than the provided key if iterating in the forward direction. Behavior would be reversed if iterating backwards.
func (*Iterator) ValidForPrefix ¶
ValidForPrefix returns false when iteration is done or when the current key is not prefixed by the specified prefix.
type IteratorOptions ¶
type IteratorOptions struct { // Indicates whether we should prefetch values during iteration and store them. PrefetchValues bool // How many KV pairs to prefetch while iterating. Valid only if PrefetchValues is true. PrefetchSize int Reverse bool // Direction of iteration. False is forward, true is backward. AllVersions bool // Fetch all valid versions of the same key. // The following option is used to narrow down the SSTables that iterator picks up. If // Prefix is specified, only tables which could have this prefix are picked based on their range // of keys. Prefix []byte // Only iterate over this given prefix. InternalAccess bool // Used to allow internal access to badger keys. // contains filtered or unexported fields }
IteratorOptions is used to set options when iterating over Badger key-value stores.
This package provides DefaultIteratorOptions which contains options that should work for most applications. Consider using that as a starting point before customizing it for your own needs.
type KVLoader ¶
type KVLoader struct {
// contains filtered or unexported fields
}
KVLoader is used to write KVList objects in to badger. It can be used to restore a backup.
type Logger ¶
type Logger interface { Errorf(string, ...interface{}) Warningf(string, ...interface{}) Infof(string, ...interface{}) Debugf(string, ...interface{}) }
Logger is implemented by any logging system that is used for standard logs.
type Manifest ¶
type Manifest struct { Levels []levelManifest Tables map[uint64]TableManifest // Contains total number of creation and deletion changes in the manifest -- used to compute // whether it'd be useful to rewrite the manifest. Creations int Deletions int }
Manifest represents the contents of the MANIFEST file in a Badger store.
The MANIFEST file describes the startup state of the db -- all LSM files and what level they're at.
It consists of a sequence of ManifestChangeSet objects. Each of these is treated atomically, and contains a sequence of ManifestChange's (file creations/deletions) which we use to reconstruct the manifest at startup.
func ReplayManifestFile ¶
ReplayManifestFile reads the manifest file and constructs two manifest objects. (We need one immutable copy and one mutable copy of the manifest. Easiest way is to construct two of them.) Also, returns the last offset after a completely read manifest entry -- the file must be truncated at that point before further appends are made (if there is a partial entry after that). In normal conditions, truncOffset is the file size.
type MergeFunc ¶
MergeFunc accepts two byte slices, one representing an existing value, and another representing a new value that needs to be ‘merged’ into it. MergeFunc contains the logic to perform the ‘merge’ and return an updated value. MergeFunc could perform operations like integer addition, list appends etc. Note that the ordering of the operands is maintained.
type MergeOperator ¶
MergeOperator represents a Badger merge operator.
func (*MergeOperator) Add ¶
func (op *MergeOperator) Add(val []byte) error
Add records a value in Badger which will eventually be merged by a background routine into the values that were recorded by previous invocations to Add().
func (*MergeOperator) Get ¶
func (op *MergeOperator) Get() ([]byte, error)
Get returns the latest value for the merge operator, which is derived by applying the merge function to all the values added so far.
If Add has not been called even once, Get will return ErrKeyNotFound.
func (*MergeOperator) Stop ¶
func (op *MergeOperator) Stop()
Stop waits for any pending merge to complete and then stops the background goroutine.
type Options ¶
type Options struct { Dir string ValueDir string SyncWrites bool TableLoadingMode options.FileLoadingMode ValueLogLoadingMode options.FileLoadingMode NumVersionsToKeep int ReadOnly bool Truncate bool Logger Logger EventLogging bool MaxTableSize int64 LevelSizeMultiplier int MaxLevels int ValueThreshold int NumMemtables int NumLevelZeroTables int NumLevelZeroTablesStall int LevelOneSize int64 ValueLogFileSize int64 ValueLogMaxEntries uint32 NumCompactors int CompactL0OnClose bool LogRotatesToFlush int32 // When set, checksum will be validated for each entry read from the value log file. VerifyValueChecksum bool // BypassLockGaurd will bypass the lock guard on badger. Bypassing lock // guard can cause data corruption if multiple badger instances are using // the same directory. Use this options with caution. BypassLockGuard bool // contains filtered or unexported fields }
Options are params for creating DB object.
This package provides DefaultOptions which contains options that should work for most applications. Consider using that as a starting point before customizing it for your own needs.
Each option X is documented on the WithX method.
func DefaultOptions ¶
DefaultOptions sets a list of recommended options for good performance. Feel free to modify these to suit your needs with the WithX methods.
func LSMOnlyOptions ¶
LSMOnlyOptions follows from DefaultOptions, but sets a higher ValueThreshold so values would be colocated with the LSM tree, with value log largely acting as a write-ahead log only. These options would reduce the disk usage of value log, and make Badger act more like a typical LSM tree.
func (*Options) Errorf ¶
Errorf logs an ERROR log message to the logger specified in opts or to the global logger if no logger is specified in opts.
func (Options) WithBypassLockGuard ¶
WithBypassLockGuard returns a new Options value with BypassLockGuard set to the given value.
When BypassLockGuard option is set, badger will not acquire a lock on the directory. This could lead to data corruption if multiple badger instances write to the same data directory. Use this option with caution.
The default value of BypassLockGuard is false.
func (Options) WithCompactL0OnClose ¶
WithCompactL0OnClose returns a new Options value with CompactL0OnClose set to the given value.
CompactL0OnClose determines whether Level 0 should be compacted before closing the DB. This ensures that both reads and writes are efficient when the DB is opened later.
The default value of CompactL0OnClose is true.
func (Options) WithDir ¶
WithDir returns a new Options value with Dir set to the given value.
Dir is the path of the directory where key data will be stored in. If it doesn't exist, Badger will try to create it for you. This is set automatically to be the path given to `DefaultOptions`.
func (Options) WithEventLogging ¶
WithEventLogging returns a new Options value with EventLogging set to the given value.
EventLogging provides a way to enable or disable trace.EventLog logging.
The default value of EventLogging is true.
func (Options) WithLevelOneSize ¶
WithLevelOneSize returns a new Options value with LevelOneSize set to the given value.
LevelOneSize sets the maximum total size for Level 1.
The default value of LevelOneSize is 20MB.
func (Options) WithLevelSizeMultiplier ¶
WithLevelSizeMultiplier returns a new Options value with LevelSizeMultiplier set to the given value.
LevelSizeMultiplier sets the ratio between the maximum sizes of contiguous levels in the LSM. Once a level grows to be larger than this ratio allowed, the compaction process will be
triggered.
The default value of LevelSizeMultiplier is 10.
func (Options) WithLogRotatesToFlush ¶
WithLogRotatesToFlush returns a new Options value with LogRotatesToFlush set to the given value.
LogRotatesToFlush sets the number of value log file rotates after which the Memtables are flushed to disk. This is useful in write loads with fewer keys and larger values. This work load would fill up the value logs quickly, while not filling up the Memtables. Thus, on a crash and restart, the value log head could cause the replay of a good number of value log files which can slow things on start.
The default value of LogRotatesToFlush is 2.
func (Options) WithLogger ¶
WithLogger returns a new Options value with Logger set to the given value.
Logger provides a way to configure what logger each value of badger.DB uses.
The default value of Logger writes to stderr using the log package from the Go standard library.
func (Options) WithMaxLevels ¶
WithMaxLevels returns a new Options value with MaxLevels set to the given value.
Maximum number of levels of compaction allowed in the LSM.
The default value of MaxLevels is 7.
func (Options) WithMaxTableSize ¶
WithMaxTableSize returns a new Options value with MaxTableSize set to the given value.
MaxTableSize sets the maximum size in bytes for each LSM table or file.
The default value of MaxTableSize is 64MB.
func (Options) WithNumCompactors ¶
WithNumCompactors returns a new Options value with NumCompactors set to the given value.
NumCompactors sets the number of compaction workers to run concurrently. Setting this to zero stops compactions, which could eventually cause writes to block forever.
The default value of NumCompactors is 2.
func (Options) WithNumLevelZeroTables ¶
WithNumLevelZeroTables returns a new Options value with NumLevelZeroTables set to the given value.
NumLevelZeroTables sets the maximum number of Level 0 tables before compaction starts.
The default value of NumLevelZeroTables is 5.
func (Options) WithNumLevelZeroTablesStall ¶
WithNumLevelZeroTablesStall returns a new Options value with NumLevelZeroTablesStall set to the given value.
NumLevelZeroTablesStall sets the number of Level 0 tables that once reached causes the DB to stall until compaction succeeds.
The default value of NumLevelZeroTablesStall is 10.
func (Options) WithNumMemtables ¶
WithNumMemtables returns a new Options value with NumMemtables set to the given value.
NumMemtables sets the maximum number of tables to keep in memory before stalling.
The default value of NumMemtables is 5.
func (Options) WithNumVersionsToKeep ¶
WithNumVersionsToKeep returns a new Options value with NumVersionsToKeep set to the given value.
NumVersionsToKeep sets how many versions to keep per key at most.
The default value of NumVersionsToKeep is 1.
func (Options) WithReadOnly ¶
WithReadOnly returns a new Options value with ReadOnly set to the given value.
When ReadOnly is true the DB will be opened on read-only mode. Multiple processes can open the same Badger DB. Note: if the DB being opened had crashed before and has vlog data to be replayed, ReadOnly will cause Open to fail with an appropriate message.
The default value of ReadOnly is false.
func (Options) WithSyncWrites ¶
WithSyncWrites returns a new Options value with SyncWrites set to the given value.
When SyncWrites is true all writes are synced to disk. Setting this to false would achieve better performance, but may cause data loss in case of crash.
The default value of SyncWrites is true.
func (Options) WithTableLoadingMode ¶
func (opt Options) WithTableLoadingMode(val options.FileLoadingMode) Options
WithTableLoadingMode returns a new Options value with TableLoadingMode set to the given value.
TableLoadingMode indicates which file loading mode should be used for the LSM tree data files.
The default value of TableLoadingMode is options.MemoryMap.
func (Options) WithTruncate ¶
WithTruncate returns a new Options value with Truncate set to the given value.
Truncate indicates whether value log files should be truncated to delete corrupt data, if any. This option is ignored when ReadOnly is true.
The default value of Truncate is false.
func (Options) WithValueDir ¶
WithValueDir returns a new Options value with ValueDir set to the given value.
ValueDir is the path of the directory where value data will be stored in. If it doesn't exist, Badger will try to create it for you. This is set automatically to be the path given to `DefaultOptions`.
func (Options) WithValueLogFileSize ¶
WithValueLogFileSize returns a new Options value with ValueLogFileSize set to the given value.
ValueLogFileSize sets the maximum size of a single value log file.
The default value of ValueLogFileSize is 1GB.
func (Options) WithValueLogLoadingMode ¶
func (opt Options) WithValueLogLoadingMode(val options.FileLoadingMode) Options
WithValueLogLoadingMode returns a new Options value with ValueLogLoadingMode set to the given value.
ValueLogLoadingMode indicates which file loading mode should be used for the value log data files.
The default value of ValueLogLoadingMode is options.MemoryMap.
func (Options) WithValueLogMaxEntries ¶
WithValueLogMaxEntries returns a new Options value with ValueLogMaxEntries set to the given value.
ValueLogMaxEntries sets the maximum number of entries a value log file can hold approximately. A actual size limit of a value log file is the minimum of ValueLogFileSize and ValueLogMaxEntries.
The default value of ValueLogMaxEntries is one million (1000000).
func (Options) WithValueThreshold ¶
WithValueThreshold returns a new Options value with ValueThreshold set to the given value.
ValueThreshold sets the threshold used to decide whether a value is stored directly in the LSM tree or separatedly in the log value files.
The default value of ValueThreshold is 32, but LSMOnlyOptions sets it to 65500.
func (Options) WithVerifyValueChecksum ¶
WithVerifyValueChecksum returns a new Options value with VerifyValueChecksum set to the given value.
When VerifyValueChecksum is set to true, checksum will be verified for every entry read from the value log. If the value is stored in SST (value size less than value threshold) then the checksum validation will not be done.
The default value of VerifyValueChecksum is False.
type Sequence ¶
Sequence represents a Badger sequence.
type Stream ¶
type Stream struct { // Prefix to only iterate over certain range of keys. If set to nil (default), Stream would // iterate over the entire DB. Prefix []byte // Number of goroutines to use for iterating over key ranges. Defaults to 16. NumGo int // Badger would produce log entries in Infof to indicate the progress of Stream. LogPrefix can // be used to help differentiate them from other activities. Default is "Badger.Stream". LogPrefix string // ChooseKey is invoked each time a new key is encountered. Note that this is not called // on every version of the value, only the first encountered version (i.e. the highest version // of the value a key has). ChooseKey can be left nil to select all keys. // // Note: Calls to ChooseKey are concurrent. ChooseKey func(item *Item) bool // KeyToList, similar to ChooseKey, is only invoked on the highest version of the value. It // is upto the caller to iterate over the versions and generate zero, one or more KVs. It // is expected that the user would advance the iterator to go through the versions of the // values. However, the user MUST immediately return from this function on the first encounter // with a mismatching key. See example usage in ToList function. Can be left nil to use ToList // function by default. // // Note: Calls to KeyToList are concurrent. KeyToList func(key []byte, itr *Iterator) (*pb.KVList, error) // This is the method where Stream sends the final output. All calls to Send are done by a // single goroutine, i.e. logic within Send method can expect single threaded execution. Send func(*pb.KVList) error // contains filtered or unexported fields }
Stream provides a framework to concurrently iterate over a snapshot of Badger, pick up key-values, batch them up and call Send. Stream does concurrent iteration over many smaller key ranges. It does NOT send keys in lexicographical sorted order. To get keys in sorted order, use Iterator.
func (*Stream) Backup ¶
Backup dumps a protobuf-encoded list of all entries in the database into the given writer, that are newer than the specified version. It returns a timestamp indicating when the entries were dumped which can be passed into a later invocation to generate an incremental dump, of entries that have been added/modified since the last invocation of Stream.Backup().
This can be used to backup the data in a database at a given point in time.
func (*Stream) Orchestrate ¶
Orchestrate runs Stream. It picks up ranges from the SSTables, then runs NumGo number of goroutines to iterate over these ranges and batch up KVs in lists. It concurrently runs a single goroutine to pick these lists, batch them up further and send to Output.Send. Orchestrate also spits logs out to Infof, using provided LogPrefix. Note that all calls to Output.Send are serial. In case any of these steps encounter an error, Orchestrate would stop execution and return that error. Orchestrate can be called multiple times, but in serial order.
type StreamWriter ¶
type StreamWriter struct {
// contains filtered or unexported fields
}
StreamWriter is used to write data coming from multiple streams. The streams must not have any overlapping key ranges. Within each stream, the keys must be sorted. Badger Stream framework is capable of generating such an output. So, this StreamWriter can be used at the other end to build BadgerDB at a much faster pace by writing SSTables (and value logs) directly to LSM tree levels without causing any compactions at all. This is way faster than using batched writer or using transactions, but only applicable in situations where the keys are pre-sorted and the DB is being bootstrapped. Existing data would get deleted when using this writer. So, this is only useful when restoring from backup or replicating DB across servers.
StreamWriter should not be called on in-use DB instances. It is designed only to bootstrap new DBs.
func (*StreamWriter) Flush ¶
func (sw *StreamWriter) Flush() error
Flush is called once we are done writing all the entries. It syncs DB directories. It also updates Oracle with maxVersion found in all entries (if DB is not managed).
func (*StreamWriter) Prepare ¶
func (sw *StreamWriter) Prepare() error
Prepare should be called before writing any entry to StreamWriter. It deletes all data present in existing DB, stops compactions and any writes being done by other means. Be very careful when calling Prepare, because it could result in permanent data loss. Not calling Prepare would result in a corrupt Badger instance.
type TableInfo ¶
type TableInfo struct { ID uint64 Level int Left []byte Right []byte KeyCount uint64 // Number of keys in the table }
TableInfo represents the information about a table.
type TableManifest ¶
TableManifest contains information about a specific level in the LSM tree.
type Txn ¶
type Txn struct {
// contains filtered or unexported fields
}
Txn represents a Badger transaction.
func (*Txn) Commit ¶
Commit commits the transaction, following these steps:
1. If there are no writes, return immediately.
2. Check if read rows were updated since txn started. If so, return ErrConflict.
3. If no conflict, generate a commit timestamp and update written rows' commit ts.
4. Batch up all writes, write them to value log and LSM tree.
5. If callback is provided, Badger will return immediately after checking for conflicts. Writes to the database will happen in the background. If there is a conflict, an error will be returned and the callback will not run. If there are no conflicts, the callback will be called in the background upon successful completion of writes or any error during write.
If error is nil, the transaction is successfully committed. In case of a non-nil error, the LSM tree won't be updated, so there's no need for any rollback.
func (*Txn) CommitAt ¶
CommitAt commits the transaction, following the same logic as Commit(), but at the given commit timestamp. This will panic if not used with managed transactions.
This is only useful for databases built on top of Badger (like Dgraph), and can be ignored by most users.
func (*Txn) CommitWith ¶
CommitWith acts like Commit, but takes a callback, which gets run via a goroutine to avoid blocking this function. The callback is guaranteed to run, so it is safe to increment sync.WaitGroup before calling CommitWith, and decrementing it in the callback; to block until all callbacks are run.
func (*Txn) Delete ¶
Delete deletes a key.
This is done by adding a delete marker for the key at commit timestamp. Any reads happening before this timestamp would be unaffected. Any reads after this commit would see the deletion.
The current transaction keeps a reference to the key byte slice argument. Users must not modify the key until the end of the transaction.
func (*Txn) Discard ¶
func (txn *Txn) Discard()
Discard discards a created transaction. This method is very important and must be called. Commit method calls this internally, however, calling this multiple times doesn't cause any issues. So, this can safely be called via a defer right when transaction is created.
NOTE: If any operations are run on a discarded transaction, ErrDiscardedTxn is returned.
func (*Txn) Get ¶
Get looks for key and returns corresponding Item. If key is not found, ErrKeyNotFound is returned.
func (*Txn) NewIterator ¶
func (txn *Txn) NewIterator(opt IteratorOptions) *Iterator
NewIterator returns a new iterator. Depending upon the options, either only keys, or both key-value pairs would be fetched. The keys are returned in lexicographically sorted order. Using prefetch is recommended if you're doing a long running iteration, for performance.
Multiple Iterators: For a read-only txn, multiple iterators can be running simultaneously. However, for a read-write txn, only one can be running at one time to avoid race conditions, because Txn is thread-unsafe.
Example ¶
dir, err := ioutil.TempDir("", "badger-test") if err != nil { panic(err) } defer removeDir(dir) db, err := Open(DefaultOptions(dir)) if err != nil { panic(err) } defer db.Close() bkey := func(i int) []byte { return []byte(fmt.Sprintf("%09d", i)) } bval := func(i int) []byte { return []byte(fmt.Sprintf("%025d", i)) } txn := db.NewTransaction(true) // Fill in 1000 items n := 1000 for i := 0; i < n; i++ { err := txn.SetEntry(NewEntry(bkey(i), bval(i))) if err != nil { panic(err) } } err = txn.Commit() if err != nil { panic(err) } opt := DefaultIteratorOptions opt.PrefetchSize = 10 // Iterate over 1000 items var count int err = db.View(func(txn *Txn) error { it := txn.NewIterator(opt) defer it.Close() for it.Rewind(); it.Valid(); it.Next() { count++ } return nil }) if err != nil { panic(err) } fmt.Printf("Counted %d elements", count)
Output: Counted 1000 elements
func (*Txn) NewKeyIterator ¶
func (txn *Txn) NewKeyIterator(key []byte, opt IteratorOptions) *Iterator
NewKeyIterator is just like NewIterator, but allows the user to iterate over all versions of a single key. Internally, it sets the Prefix option in provided opt, and uses that prefix to additionally run bloom filter lookups before picking tables from the LSM tree.
func (*Txn) Set ¶
Set adds a key-value pair to the database. It will return ErrReadOnlyTxn if update flag was set to false when creating the transaction.
The current transaction keeps a reference to the key and val byte slice arguments. Users must not modify key and val until the end of the transaction.
type WriteBatch ¶
WriteBatch holds the necessary info to perform batched writes.
func (*WriteBatch) Cancel ¶
func (wb *WriteBatch) Cancel()
Cancel function must be called if there's a chance that Flush might not get called. If neither Flush or Cancel is called, the transaction oracle would never get a chance to clear out the row commit timestamp map, thus causing an unbounded memory consumption. Typically, you can call Cancel as a defer statement right after NewWriteBatch is called.
Note that any committed writes would still go through despite calling Cancel.
func (*WriteBatch) Delete ¶
func (wb *WriteBatch) Delete(k []byte) error
Delete is equivalent of Txn.Delete.
func (*WriteBatch) Error ¶
func (wb *WriteBatch) Error() error
Error returns any errors encountered so far. No commits would be run once an error is detected.
func (*WriteBatch) Flush ¶
func (wb *WriteBatch) Flush() error
Flush must be called at the end to ensure that any pending writes get committed to Badger. Flush returns any error stored by WriteBatch.
func (*WriteBatch) Set ¶
func (wb *WriteBatch) Set(k, v []byte) error
Set is equivalent of Txn.Set().
func (*WriteBatch) SetEntry ¶
func (wb *WriteBatch) SetEntry(e *Entry) error
SetEntry is the equivalent of Txn.SetEntry.
func (*WriteBatch) SetMaxPendingTxns ¶
func (wb *WriteBatch) SetMaxPendingTxns(max int)
SetMaxPendingTxns sets a limit on maximum number of pending transactions while writing batches. This function should be called before using WriteBatch. Default value of MaxPendingTxns is 16 to minimise memory usage.