-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add support for percent of totals in MetricsViewAggregation #5348
Conversation
proto/rill/runtime/v1/queries.proto
Outdated
@@ -389,6 +389,7 @@ message MetricsViewAggregationMeasure { | |||
MetricsViewAggregationMeasureComputeComparisonValue comparison_value = 7; | |||
MetricsViewAggregationMeasureComputeComparisonDelta comparison_delta = 8; | |||
MetricsViewAggregationMeasureComputeComparisonRatio comparison_ratio = 9; | |||
MetricsViewAggregationMeasurePercentOfTotal percent_of_total = 10; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: put "compute" in the name like for the other computed measure types: MetricsViewAggregationMeasureComputePercentOfTotal
On reflection, I think this might be simpler to solve with an additional query. I think there might actually be a simple way to achieve that. The new For inspiration, you can see this rewriter which turns relative time ranges into fixed time ranges, sometimes making a separate query to get the min/max timestamps of the metrics view: rill/runtime/metricsview/executor_rewrite_time.go Lines 14 to 16 in b67474d
|
d72a32b
to
87d22a5
Compare
runtime/metricsview/ast.go
Outdated
Expression: fmt.Sprintf("%s*100/%f", a.dialect.EscapeIdentifier(m.Name), qm.Compute.PercentOfTotal.Total), | ||
Type: runtimev1.MetricsViewSpec_MEASURE_TYPE_DERIVED, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- I wonder if we could make it simple and inject the expression directly? I.e. use something like
fmt.Sprintf("(%s)/%f", m.Expression, qm.Compute.PercentOfTotal.Total)
? I think it would work even with subquery measures, at leastselect (select 100)/10
works in both DuckDB and Druid. - Not sure, but maybe the
%f
should be%#f
, which always adds a decimal point? Otherwise, if both operands are whole numbers, could it end up doing integer division? (Not sure about the SQL semantics here.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- We are calculating the measure value itself in a subquery since it might be used for comparison columns as well. So adding the expression here will recompute it wont it?
- Update to be safer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, if you think it will mostly be queried in requests that also query the plain measure value, then I agree the derived approach is better
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ya. I would assume all instances of totals in dashboard right now can be replaced with this.
totalsQry := &Query{ | ||
MetricsView: qry.MetricsView, | ||
Measures: measures, | ||
TimeRange: qry.TimeRange, | ||
Where: qry.Where, | ||
TimeZone: qry.TimeZone, | ||
} | ||
|
||
e, err := NewExecutor(ctx, e.rt, e.instanceID, mv, sec, e.priority) | ||
if err != nil { | ||
return err | ||
} | ||
defer e.Close() | ||
|
||
res, err := e.Query(ctx, totalsQry, nil) | ||
if err != nil { | ||
return err | ||
} | ||
defer res.Close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The exactify rewriter does a couple things differently that I think we should also do here:
- Explicitly set all members of the
Query{}
struct to reduce chance of forgetting to handle one if it's changed in the future - Explicitly call
NewAST
ande.olap.Execute
to a) prevent any chance infinite recursion, b) only apply the rewrites that make sense in this context (I think that would be onlyrewriteApproxComparisons
andrewriteDruidGroups
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this not add some complexity? Any new rewrite need to be added in multiple places and the author needs to be aware of it. Not sure if it can be solved as is right now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah it's a little annoying, but I think it's best to explicitly add the rewrites each time at this point. To avoid recursion dangers and because the rewrites are a very mixed bag (some require a specific order of other rewrites, some are expensive to run, some don't make sense in the context of certain calls).
If we end up adding more rewrites, it may make sense to refactor how they're configured to avoid the repetition.
70f31ac
to
5dd3ae8
Compare
To add support for percent of totals based filter criteria in alerts, we need to add percent of totals as a variant of a measure.
This PR adds that support to MetricsViewAggregation by calculating totals and rewriting percent of totals with
measure/<calculated_total>