stats/opentelemetry: separate out interceptors for tracing and metrics #8063

janardhanvissa · 2025-02-03T11:09:00Z

RELEASE NOTES: None

codecov · 2025-02-03T11:13:43Z

Codecov Report

Attention: Patch coverage is 74.48980% with 50 lines in your changes missing coverage. Please review.

Project coverage is 82.01%. Comparing base (ce35fd4) to head (cc12a84).

Files with missing lines	Patch %	Lines
stats/opentelemetry/client_tracing.go	56.71%	23 Missing and 6 partials ⚠️
stats/opentelemetry/server_tracing.go	57.69%	7 Missing and 4 partials ⚠️
stats/opentelemetry/client_metrics.go	85.45%	5 Missing and 3 partials ⚠️
stats/opentelemetry/server_metrics.go	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8063      +/-   ##
==========================================
- Coverage   82.16%   82.01%   -0.15%     
==========================================
  Files         410      410              
  Lines       40248    40342      +94     
==========================================
+ Hits        33068    33087      +19     
- Misses       5830     5881      +51     
- Partials     1350     1374      +24

Files with missing lines	Coverage Δ
stats/opentelemetry/opentelemetry.go	`77.90% <100.00%> (+2.90%)`	⬆️
stats/opentelemetry/server_metrics.go	`86.14% <90.90%> (-3.68%)`	⬇️
stats/opentelemetry/client_metrics.go	`82.35% <85.45%> (-7.05%)`	⬇️
stats/opentelemetry/server_tracing.go	`69.44% <57.69%> (-30.56%)`	⬇️
stats/opentelemetry/client_tracing.go	`66.27% <56.71%> (-17.73%)`	⬇️

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

purnesh42H · 2025-02-05T19:15:18Z

@janardhanvissa its not clear what is the intention of this refactor. The follow up from opentelemetry tracing API PR was to create separate interceptors for metrics and traces. Right now, single interceptor is handling both trace and metrics options. Once we have separate unary and stream interceptor each for tracing and metrics, we don't have to check for options disabled/enabled everytime.

purnesh42H

Please see this discussion https://github.com/grpc/grpc-go/pull/7852/files#r1909469701 and modify accordingly.

stats/opentelemetry/client_metrics.go

stats/opentelemetry/server_metrics.go

stats/opentelemetry/client_metrics.go

stats/opentelemetry/client_tracing.go

stats/opentelemetry/server_metrics.go

stats/opentelemetry/server_tracing.go

purnesh42H

LGTM

purnesh42H · 2025-03-28T19:19:28Z

@dfawley for second review

dfawley

Thanks for the cleanup! This definitely looks better, but I think it can be improved even more.

dfawley · 2025-03-28T19:51:36Z

stats/opentelemetry/opentelemetry.go

+	if !o.isTracingEnabled() {
+		return do
+	}
+	tracingHandler := &clientTracingHandler{options: o}


Can you call this cth, or make the above metricsHandler instead?

I looked at the current code and with this refactor, its fine to call the clientStatsHandler as metricsHandler. @janardhanvissa

dfawley · 2025-03-28T19:51:58Z

stats/opentelemetry/opentelemetry.go

@@ -117,10 +113,19 @@ type MetricsOptions struct {
 // MeterProvider. If the passed in Meter Provider does not have the view
 // configured for an individual metric turned on, the API call in this component
 // will create a default view for that metric.
+//
+// For the traces supported by this instrumentation code, provide an
+// implementation of a TextMapPropagator and OpenTelemetry TracerProvider.
 func DialOption(o Options) grpc.DialOption {
 	csh := &clientStatsHandler{options: o}


Isn't it possible to have metrics disabled and tracing enabled?

So, right now if metrics are disabled, the clientStatsHandler behaves as no-op for any type of metric work but its still always added. @janardhanvissa could you try not this interceptor if metrics is disabled? If there is no issue, then probably the right thing to do.

dfawley · 2025-03-28T19:53:36Z

stats/opentelemetry/client_metrics.go

 	estats "google.golang.org/grpc/experimental/stats"
 	istats "google.golang.org/grpc/internal/stats"
 	"google.golang.org/grpc/metadata"
 	"google.golang.org/grpc/stats"
 	"google.golang.org/grpc/status"
-
-	otelattribute "go.opentelemetry.io/otel/attribute"
-	otelmetric "go.opentelemetry.io/otel/metric"
 )

 type clientStatsHandler struct {


Should we call this clientMetricsHandler instead? "stats handler" is a specific thing that makes both metrics and tracing work. Since this only handles metrics, it probably shouldn't use "stats".

dfawley · 2025-03-28T19:55:45Z

stats/opentelemetry/client_metrics.go

+	ci := getCallInfo(ctx)
+	if ci == nil {
+		if logger.V(2) {
+			logger.Info("Creating new CallInfo since its not present in context in clientStatsHandler unaryInterceptor")
+		}
+		ci = &callInfo{
+			target: cc.CanonicalTarget(),
+			method: determineMethod(method, opts...),
+		}
+		ctx = setCallInfo(ctx, ci)
 	}


Why are we doing this? I would expect every unary interceptor start should be for a new call attempt, so there should never be anything in the context already? Am I missing something?

so, with this refactor, we have 2 stats handler now 1) metrics and 2) traces. The stats handlers are executed in the order in which they are added. So, if metrics handler executes first, we need to make sure that tracing handler doesn't create a new call info rather use the existing one and add the tracing stuff there. Vice-versa is also true if in future we change the order of interceptors. Basically, each interceptor needs to check if call info already exist or not and then add its info to existing one if present otherwise create a new one.

dfawley · 2025-03-28T19:56:23Z

stats/opentelemetry/client_metrics.go

+	ci := getCallInfo(ctx)
+	if ci == nil {
+		if logger.V(2) {
+			logger.Info("Creating new CallInfo since its not present in context in clientStatsHandler streamInterceptor")
+		}
+		ci = &callInfo{
+			target: cc.CanonicalTarget(),
+			method: determineMethod(method, opts...),
+		}
+		ctx = setCallInfo(ctx, ci)
 	}


Same reason.

dfawley · 2025-03-28T19:59:35Z

stats/opentelemetry/client_tracing.go

+
+func (h *clientTracingHandler) unaryInterceptor(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
+	ci := getCallInfo(ctx)
+	if ci == nil {


dfawley · 2025-03-28T19:59:43Z

stats/opentelemetry/client_tracing.go

+
+func (h *clientTracingHandler) streamInterceptor(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
+	ci := getCallInfo(ctx)
+	if ci == nil {


dfawley · 2025-03-28T20:01:54Z

stats/opentelemetry/client_tracing.go

+
+// perCallTraces sets the span status based on the RPC result and ends the span.
+// It is used to finalize tracing for both unary and streaming calls.
+func (h *clientTracingHandler) perCallTraces(err error, ts trace.Span) {


This won't have all of the per-call trace events, right? When we add the resolver delay in #8074 anyway.

So probably finishTrace() or endCall or something indicating it handles the end of the RPC?

renamed as finishTrace()

dfawley · 2025-03-28T20:03:36Z

stats/opentelemetry/client_tracing.go

+func (h *clientTracingHandler) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context {
+	ri := getRPCInfo(ctx)
+	var ai *attemptInfo
+	if ri == nil {


dfawley · 2025-03-28T20:06:35Z

stats/opentelemetry/server_tracing.go

+func (h *serverTracingHandler) unaryInterceptor(ctx context.Context, req any, _ *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
+	return handler(ctx, req)
+}
+
+func (h *serverTracingHandler) streamInterceptor(srv any, ss grpc.ServerStream, _ *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
+	return handler(srv, ss)
+}


These just shouldn't exist, and don't register an interceptor.

…tion of rpcInfo and attemptInfo

purnesh42H · 2025-04-03T16:18:03Z

stats/opentelemetry/client_tracing.go

@@ -130,7 +130,7 @@ func (h *clientTracingHandler) HandleConn(context.Context, stats.ConnStats) {}
 func (h *clientTracingHandler) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context {
 	ri := getRPCInfo(ctx)
 	var ai *attemptInfo
-	if ri == nil {
+	if ri.ai == nil {


I think here also we can't assume ri is not nil if the order of stats handlers changes.

purnesh42H · 2025-04-03T16:20:11Z

stats/opentelemetry/opentelemetry.go

+		metricsHandler := &clientMetricsHandler{options: o}
+		metricsHandler.initializeMetrics()
+		unaryInterceptors = append(unaryInterceptors, metricsHandler.unaryInterceptor)
+		streamInterceptors = append(streamInterceptors, metricsHandler.streamInterceptor)


this is changing the current order. Let's avoid that. Keep only 2 variables metricsInterceptors and tracesInterceptors and add metricsInterceptors before traces

purnesh42H · 2025-04-03T16:21:38Z

stats/opentelemetry/opentelemetry.go

+		streamInterceptors = append(streamInterceptors, tracingHandler.streamInterceptor)
+		do = append(do, grpc.WithStatsHandler(tracingHandler))
+	}
+	if len(unaryInterceptors) > 0 {


these ifs will change to metrics and traces

purnesh42H · 2025-04-03T16:24:47Z

stats/opentelemetry/server_tracing.go

 // TagRPC implements per RPC attempt context management for traces.
 func (h *serverTracingHandler) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context {
 	ri := getRPCInfo(ctx)
 	var ai *attemptInfo
-	if ri == nil {
+	if ri.ai == nil {


same here. We can't assume ri to be not nil here.

janardhankrishna-sai and others added 8 commits December 23, 2024 07:09

replace dial with newclient

7ddbe46

Merge branch 'master' of https://github.com/janardhanvissa/grpc-go

0486355

Merge branch 'grpc:master' into master

53fa9cd

Merge branch 'grpc:master' into master

4435b8a

Merge branch 'grpc:master' into master

a413555

Merge branch 'grpc:master' into master

4e203c3

Merge branch 'grpc:master' into master

e9ad552

refactor tracing and metrics interceptors separately

71804b4

janardhankrishna-sai added 2 commits February 3, 2025 11:21

adding opentelemetry tracing for client-server

4b3bd26

fixing vet issues

69df069

janardhanvissa force-pushed the refactor-tracing-metrics branch from 69df069 to 71804b4 Compare February 3, 2025 11:48

janardhankrishna-sai added 6 commits February 3, 2025 12:23

reverting newclient changes and fixing vet issues

770f430

reverting otel trace for client/server changes

b97a3ca

fixing vet issue

c89f3c9

renaming receiver name

3f07e48

unused param fix

5e8a4a5

fixing vet issue

76e422a

purnesh42H assigned janardhanvissa Feb 5, 2025

purnesh42H self-requested a review February 5, 2025 19:15

purnesh42H requested changes Feb 5, 2025

View reviewed changes

janardhankrishna-sai added 6 commits February 12, 2025 08:08

refactor client interceptor separately for traces and metrics

8fa0b03

moving tracing code to client_tracing.go file

1f41a49

revert previous commit

d74c61d

adding separate interceptors for traces and metrics of server

170eef6

separating HandleRPC interceptors of traces and metrics

7f5f539

updating client and server metrics

50999a0

purnesh42H assigned purnesh42H and unassigned janardhanvissa Feb 18, 2025

janardhankrishna-sai added 2 commits March 20, 2025 08:55

fixing vet issue

0b89cf6

fixing linter issue

42c08a0

janardhanvissa requested a review from purnesh42H March 20, 2025 09:11

janardhanvissa removed their assignment Mar 20, 2025

purnesh42H reviewed Mar 24, 2025

View reviewed changes

stats/opentelemetry/client_metrics.go Show resolved Hide resolved

stats/opentelemetry/server_metrics.go Show resolved Hide resolved

purnesh42H assigned janardhanvissa Mar 24, 2025

addressing review comments

5bb48b3

janardhanvissa requested a review from purnesh42H March 25, 2025 14:03

purnesh42H reviewed Mar 25, 2025

View reviewed changes

addressing review comments

fb8025d

janardhanvissa requested a review from purnesh42H March 26, 2025 17:54

purnesh42H approved these changes Mar 28, 2025

View reviewed changes

purnesh42H assigned dfawley Mar 28, 2025

purnesh42H requested a review from dfawley March 28, 2025 19:19

purnesh42H unassigned janardhanvissa Mar 28, 2025

dfawley reviewed Mar 28, 2025

View reviewed changes

dfawley assigned janardhanvissa and unassigned dfawley Mar 28, 2025

janardhankrishna-sai added 2 commits April 1, 2025 14:56

addressing review comments

cc39d1a

Initializes ri to &rpcInfo{} when it's nil, separating the initializa…

eb8321f

…tion of rpcInfo and attemptInfo

eshitachandwani modified the milestones: 1.72 Release, 1.73 Release Apr 3, 2025

purnesh42H reviewed Apr 3, 2025

View reviewed changes

janardhanvissa and others added 3 commits April 4, 2025 11:27

Merge branch 'master' into refactor-tracing-metrics

c858ed2

fixing vet issue

5c57489

addressing review comments

cc12a84

janardhanvissa removed their assignment Apr 4, 2025

janardhanvissa requested a review from dfawley April 4, 2025 23:51

arjan-bal assigned dfawley Apr 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stats/opentelemetry: separate out interceptors for tracing and metrics #8063

stats/opentelemetry: separate out interceptors for tracing and metrics #8063

janardhanvissa commented Feb 3, 2025 •

edited by purnesh42H

Loading

codecov bot commented Feb 3, 2025 •

edited

Loading

purnesh42H commented Feb 5, 2025

purnesh42H left a comment

purnesh42H left a comment

purnesh42H commented Mar 28, 2025

dfawley left a comment

dfawley Mar 28, 2025

purnesh42H Apr 1, 2025 •

edited

Loading

janardhanvissa Apr 1, 2025

dfawley Mar 28, 2025

purnesh42H Apr 1, 2025

janardhanvissa Apr 1, 2025

dfawley Mar 28, 2025

janardhanvissa Apr 1, 2025

dfawley Mar 28, 2025

purnesh42H Apr 1, 2025

dfawley Mar 28, 2025

purnesh42H Apr 1, 2025

dfawley Mar 28, 2025

dfawley Mar 28, 2025

dfawley Mar 28, 2025

janardhanvissa Apr 1, 2025

dfawley Mar 28, 2025

dfawley Mar 28, 2025

janardhanvissa Apr 1, 2025

purnesh42H Apr 3, 2025

janardhanvissa Apr 4, 2025

purnesh42H Apr 3, 2025

janardhanvissa Apr 4, 2025

purnesh42H Apr 3, 2025

janardhanvissa Apr 4, 2025

purnesh42H Apr 3, 2025

janardhanvissa Apr 4, 2025

stats/opentelemetry: separate out interceptors for tracing and metrics #8063

Are you sure you want to change the base?

stats/opentelemetry: separate out interceptors for tracing and metrics #8063

Conversation

janardhanvissa commented Feb 3, 2025 • edited by purnesh42H Loading

codecov bot commented Feb 3, 2025 • edited Loading

Codecov Report

purnesh42H commented Feb 5, 2025

purnesh42H left a comment

Choose a reason for hiding this comment

purnesh42H left a comment

Choose a reason for hiding this comment

purnesh42H commented Mar 28, 2025

dfawley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purnesh42H Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janardhanvissa commented Feb 3, 2025 •

edited by purnesh42H

Loading

codecov bot commented Feb 3, 2025 •

edited

Loading

purnesh42H Apr 1, 2025 •

edited

Loading