Documentation
¶
Overview ¶
Package mergeplan provides a segment merge planning approach that's inspired by Lucene's TieredMergePolicy.java and descriptions like http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
Index ¶
- Variables
- func CalcBudget(totalSize int64, firstTierSize int64, o *MergePlanOptions) (budgetNumSegments int)
- func ScoreSegments(segments []Segment, o *MergePlanOptions) float64
- func ToBarChart(prefix string, barMax int, segments []Segment, plan *MergePlan) string
- type MergePlan
- type MergePlanOptions
- type MergeTask
- type Segment
Constants ¶
This section is empty.
Variables ¶
var DefaultMergePlanOptions = MergePlanOptions{
MaxSegmentsPerTier: 10,
MaxSegmentSize: 5000000,
TierGrowth: 10.0,
SegmentsPerMergeTask: 10,
FloorSegmentSize: 2000,
ReclaimDeletesWeight: 2.0,
}
Suggested default options.
Functions ¶
func CalcBudget ¶
func CalcBudget(totalSize int64, firstTierSize int64, o *MergePlanOptions) ( budgetNumSegments int)
Compute the number of segments that would be needed to cover the totalSize, by climbing up a logarithmically growing staircase of segment tiers.
func ScoreSegments ¶
func ScoreSegments(segments []Segment, o *MergePlanOptions) float64
Smaller result score is better.
Types ¶
type MergePlan ¶
type MergePlan struct {
Tasks []*MergeTask
}
A MergePlan is the result of the Plan() API.
The planner doesn’t know how or whether these tasks are executed -- that’s up to a separate merge execution system, which might execute these tasks concurrently or not, and which might execute all the tasks or not.
type MergePlanOptions ¶
type MergePlanOptions struct { // Max # segments per logarithmic tier, or max width of any // logarithmic “step”. Smaller values mean more merging but fewer // segments. Should be >= SegmentsPerMergeTask, else you'll have // too much merging. MaxSegmentsPerTier int // Max size of any segment produced after merging. Actual // merging, however, may produce segment sizes different than the // planner’s predicted sizes. MaxSegmentSize int64 // The growth factor for each tier in a staircase of idealized // segments computed by CalcBudget(). TierGrowth float64 // The number of segments in any resulting MergeTask. e.g., // len(result.Tasks[ * ].Segments) == SegmentsPerMergeTask. SegmentsPerMergeTask int // Small segments are rounded up to this size, i.e., treated as // equal (floor) size for consideration. This is to prevent lots // of tiny segments from resulting in a long tail in the index. FloorSegmentSize int64 // Controls how aggressively merges that reclaim more deletions // are favored. Higher values will more aggressively target // merges that reclaim deletions, but be careful not to go so high // that way too much merging takes place; a value of 3.0 is // probably nearly too high. A value of 0.0 means deletions don't // impact merge selection. ReclaimDeletesWeight float64 // Optional, defaults to mergeplan.CalcBudget(). CalcBudget func(totalSize int64, firstTierSize int64, o *MergePlanOptions) (budgetNumSegments int) // Optional, defaults to mergeplan.ScoreSegments(). ScoreSegments func(segments []Segment, o *MergePlanOptions) float64 // Optional. Logger func(string) }
The MergePlanOptions is designed to be reusable between planning calls.
func (*MergePlanOptions) RaiseToFloorSegmentSize ¶
func (o *MergePlanOptions) RaiseToFloorSegmentSize(s int64) int64
Returns the higher of the input or FloorSegmentSize.
type MergeTask ¶
type MergeTask struct {
Segments []Segment
}
A MergeTask represents several segments that should be merged together into a single segment.
type Segment ¶
type Segment interface { // Unique id of the segment -- used for sorting. Id() uint64 // Full segment size (the size before any logical deletions). FullSize() int64 // Size of the live data of the segment; i.e., FullSize() minus // any logical deletions. LiveSize() int64 }
A Segment represents the information that the planner needs to calculate segment merging.