Query cache interface implementation for timeseries & metrics views #1446

egor-ryashin · 2022-12-14T17:06:53Z

No description provided.

…-impl

…-impl-timeseries # Conflicts: # runtime/queries/column_api_benchmark_test.go # runtime/server/queries_timeseries.go

egor-ryashin · 2022-12-14T17:19:16Z

cc @AdityaHegde

egor-ryashin · 2022-12-15T14:04:15Z

I've added caching for metrics view queries. It's complete. cc @AdityaHegde

nishantmonu51

Verified and tested the changes on a high level.
Performance wise most of the cache lookups resolve in ~10-15ms. that gives a very nice boost to query profiling.

nishantmonu51

Glanced through on a high level and tested the changes by running from branch, overall things seem to be working fine.
Left comments for few methods that seem generic and might be pulled into utility classes, feel free to ignore those comments.
Din't looked into the detailed API logic itself, as i believe that is majorly a refactor of the current logic.

nishantmonu51 · 2022-12-16T13:39:47Z

runtime/queries/column_timeseries.go

+ * Importantly, this function runs very fast. For more information about the original M4 method,
+ * see http://www.vldb.org/pvldb/vol7/p797-jugel.pdf
+ */
+func (q *ColumnTimeseries) createTimestampRollupReduction( // metadata: DatabaseMetadata,


remove commented metadata ?

nishantmonu51 · 2022-12-16T13:41:26Z

runtime/queries/column_timeseries.go

+	return m
+}
+
+func convertToDateTruncSpecifier(specifier runtimev1.TimeGrain) string {


I wonder if we need similar method in other APIs as well, If yes, we can move to timeutil

Looks like it's only used for this particular query.

But for shared SQL-building util functions (i.e. that are used by multiple queries), they should be in a central place. Maybe in queries/sqlutil.go?

nishantmonu51 · 2022-12-16T13:42:27Z

runtime/queries/metricsview.go

+
+// valToPB converts any value to a google.protobuf.Value. It's similar to
+// structpb.NewValue, but adds support for a few extra primitive types.
+func valToPB(v any) (*structpb.Value, error) {


looks like it belongs to protobuf util ? and might be reused?

Yes, this is now replicated here and in runtime/server/query.go. Maybe put it (and related functions) in runtime/server/pbutil as ToValue(v any) (*structpb.Value, error)?

nishantmonu51 · 2022-12-16T13:43:04Z

runtime/queries/metricsview.go

+	return res
+}
+
+func protobufValueToAny(val *structpb.Value) (any, error) {


move to util ?

Related to my comment about valToPB, this could become pbutil.FromValue(v *structpb.Value) (any, error)

begelundmuller

Only the questions around "ts" and tsAlias are important, the rest of the suggestions are nits. Once we address "ts", we can merge and fix other comments in a follow-up PRs.

In general, I found the logic in column_timeseries.go pretty intricate. It would be good to improve the structure and commenting in there to make it easier to understand. But I also know parts of it can be merged with the metrics views logic, so let's do that first.

begelundmuller · 2022-12-16T08:59:15Z

runtime/connectors/connectors.go

@@ -99,7 +99,7 @@ type SamplePolicy struct {
 func (s *Source) Validate() error {
 	connector, ok := Connectors[s.Connector]
 	if !ok {
-		return fmt.Errorf("connector: not found")
+		return fmt.Errorf("connector: not found " + s.Connector)


Consider doing return fmt.Errorf("connector: not found %q", s.Connector) (will insert value in quotes, so empty strings are clearer in output)

begelundmuller · 2022-12-16T09:57:53Z

runtime/queries/column_timeseries.go

+func (q *ColumnTimeseries) Key() string {
+	r, err := json.Marshal(q)
+	if err != nil {
+		panic(fmt.Errorf("ColumnTimeseries: failed to marshal: %w", err))


no need to change, just fyi: panics have stack traces and should mainly be dev errors (any error that's expected in production should be handled as an error object, e.g. returned to the user), so it's usually fine to just do panic(err)

begelundmuller · 2022-12-18T14:48:37Z

runtime/queries/column_timeseries.go

+		return fmt.Errorf("not available for dialect '%s'", olap.Dialect())
+	}
+
+	timeRange, err := q.normaliseTimeRange(ctx, rt, instanceID, priority)


normaliseTimeRange sounds like a util function, but it actually runs queries. Consider resolveNormalizedTimeRange or something like that.

It's subjective.

begelundmuller · 2022-12-18T14:49:08Z

runtime/queries/column_timeseries.go

+	}
+	var measures = normaliseMeasures(q.Measures)


nit: newline

begelundmuller · 2022-12-18T14:50:06Z

runtime/queries/column_timeseries.go

+	var timestampColumn = q.TimestampColumnName
+	var tableName = q.TableName


Remove these (e.g. just use q.TableName directly)

begelundmuller · 2022-12-18T16:19:48Z

runtime/queries/column_timeseries.go

+			switch x := v.(type) {
+			case int32:
+				value.Records[k] = float64(x)
+			case int64:
+				value.Records[k] = float64(x)
+			case float32:
+				value.Records[k] = float64(x)
+			case float64:
+				value.Records[k] = x
+			default:
+				return nil, fmt.Errorf("unknown type %T ", v)
+			}


This cast doesn't feel safe, since any measure expression can be passed. For now, we can add *big.Int case and we'll probably be safe. But when merging this query with the metrics view code, this should be addressed more generally.

I guess that requires changes on UI side.

We would need to cast *big.Int to a float for UI (and result would not be exact if exceeding), but though it's bad practice, that's what frontend currently wants (we do it other places as well).

There are also int8, int16, uint8, uint16, and maybe other measure result types I'm not thinking of.

begelundmuller · 2022-12-18T16:28:59Z

runtime/queries/metricsview.go

+
+// valToPB converts any value to a google.protobuf.Value. It's similar to
+// structpb.NewValue, but adds support for a few extra primitive types.
+func valToPB(v any) (*structpb.Value, error) {


Yes, this is now replicated here and in runtime/server/query.go. Maybe put it (and related functions) in runtime/server/pbutil as ToValue(v any) (*structpb.Value, error)?

begelundmuller · 2022-12-18T16:30:17Z

runtime/queries/metricsview.go

+	return res
+}
+
+func protobufValueToAny(val *structpb.Value) (any, error) {


Related to my comment about valToPB, this could become pbutil.FromValue(v *structpb.Value) (any, error)

begelundmuller · 2022-12-18T16:41:20Z

runtime/queries/column_timeseries.go

+        -- does not have that value.
+        SELECT 
+          ` + getCoalesceStatementsMeasures(measures) + `,
+          template.` + tsAlias + ` as ts from template


Is it safe to select this using as ts given that a measure might be named ts? Shouldn't it also use tsAlias?

Yes, it will be safer.

begelundmuller · 2022-12-18T16:41:48Z

runtime/queries/column_timeseries.go

+		value.Ts = row["ts"].(time.Time).Format(IsoFormat)
+		delete(row, "ts")


Shouldn't this pass in and use tsAlias?

No, because in the final output we replace tsAlias name to ts, please see

-- does not have that value. SELECT ` + getCoalesceStatementsMeasures(measures) + `, template.` + tsAlias + ` as ts from template LEFT OUTER JOIN series ON template.` + tsAlias + ` = series.` + tsAlias + ` ORDER BY template.` + tsAlias + `

What if getCoalesceStatementsMeasures contains a measure named ts?

* addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * merge fix * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments Co-authored-by: egor-ryashin <[email protected]>

…1446) * caching: metricsview timeseries Co-authored-by: egor-ryashin <[email protected]>

* addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * merge fix * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments * addressing review comments Co-authored-by: egor-ryashin <[email protected]>

egor-ryashin added 24 commits December 6, 2022 18:31

queries-caching-impl

7f1700b

queries-caching-impl

c133a22

caching for time grain query

3126c35

caching: column numeric histogram

eda43c1

caching: column numeric histogram

c680489

Merge remote-tracking branch 'origin/main' into query-cache-interface…

235356f

…-impl

caching: table cardinality fix

7d095d3

caching: rug histogram

4d6a455

caching: time range

3895bf1

caching: time range

c9b94ac

caching: column cardinality

5d93116

caching: rollup interval

b4d6095

caching: column with all nulls

0670dec

caching: code style

d34946b

caching: rug

7822a6a

caching: code style

5f45640

caching: ts fix

8923aab

Merge remote-tracking branch 'origin/main' into query-cache-interface…

0d8f659

…-impl

caching: ts fix

5c94aa9

caching: ts fix

fd3d586

caching: code style

3750567

Merge remote-tracking branch 'origin/main' into query-cache-interface…

c69f7fd

…-impl

caching: timeseries

1f0f02f

Merge remote-tracking branch 'origin/main' into query-cache-interface…

954d9ad

…-impl-timeseries # Conflicts: # runtime/queries/column_api_benchmark_test.go # runtime/server/queries_timeseries.go

egor-ryashin requested a review from begelundmuller December 14, 2022 17:19

egor-ryashin marked this pull request as ready for review December 14, 2022 17:19

egor-ryashin added 2 commits December 15, 2022 13:01

caching: table columns

7c33d50

caching: metricsview totals

3387238

egor-ryashin added 3 commits December 15, 2022 17:22

caching: metricsview toplist

b8c3bad

caching: metricsview toplist

0e89a64

caching: metricsview timeseries

40c7c27

egor-ryashin force-pushed the query-cache-interface-impl-timeseries branch from 3110831 to 40c7c27 Compare December 15, 2022 14:22

egor-ryashin changed the title ~~Query cache interface implementation for timeseries~~ Query cache interface implementation for timeseries & metrics views Dec 15, 2022

This was referenced Dec 16, 2022

Feature: dashboards without time series #1459

Merged

Runtime: support metrics without a time dimension #1366

Closed

nishantmonu51 reviewed Dec 16, 2022

View reviewed changes

begelundmuller approved these changes Dec 18, 2022

View reviewed changes

caching: metricsview timeseries

8330ae1

begelundmuller approved these changes Dec 19, 2022

View reviewed changes

egor-ryashin merged commit 3584a6f into main Dec 19, 2022

egor-ryashin deleted the query-cache-interface-impl-timeseries branch December 19, 2022 11:54

begelundmuller mentioned this pull request Dec 19, 2022

Platform meta #1173

Closed

egor-ryashin mentioned this pull request Dec 19, 2022

Following up refactoring relating to #1446 #1474

Merged

djbarnwal pushed a commit that referenced this pull request Aug 3, 2023

Query cache interface implementation for timeseries & metrics views (#…

b389319

…1446) * caching: metricsview timeseries Co-authored-by: egor-ryashin <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Query cache interface implementation for timeseries & metrics views #1446

Query cache interface implementation for timeseries & metrics views #1446

egor-ryashin commented Dec 14, 2022

egor-ryashin commented Dec 14, 2022

egor-ryashin commented Dec 15, 2022

nishantmonu51 left a comment

nishantmonu51 left a comment

nishantmonu51 Dec 16, 2022

nishantmonu51 Dec 16, 2022

begelundmuller Dec 18, 2022

nishantmonu51 Dec 16, 2022

begelundmuller Dec 18, 2022

nishantmonu51 Dec 16, 2022

begelundmuller Dec 18, 2022

begelundmuller left a comment

begelundmuller Dec 16, 2022

begelundmuller Dec 16, 2022

begelundmuller Dec 18, 2022

egor-ryashin Dec 19, 2022

begelundmuller Dec 18, 2022

begelundmuller Dec 18, 2022

begelundmuller Dec 18, 2022

egor-ryashin Dec 19, 2022

begelundmuller Dec 20, 2022

begelundmuller Dec 18, 2022

begelundmuller Dec 18, 2022

begelundmuller Dec 18, 2022

egor-ryashin Dec 19, 2022

begelundmuller Dec 18, 2022

egor-ryashin Dec 19, 2022

begelundmuller Dec 19, 2022

		var timestampColumn = q.TimestampColumnName
		var tableName = q.TableName

		value.Ts = row["ts"].(time.Time).Format(IsoFormat)
		delete(row, "ts")

Query cache interface implementation for timeseries & metrics views #1446

Query cache interface implementation for timeseries & metrics views #1446

Conversation

egor-ryashin commented Dec 14, 2022

egor-ryashin commented Dec 14, 2022

egor-ryashin commented Dec 15, 2022

nishantmonu51 left a comment

Choose a reason for hiding this comment

nishantmonu51 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

begelundmuller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment