Error when using AWS_LAMBDA_EXEC_WRAPPER '/opt/otel-proxy-handler'

[msailes]

I have a Lambda function with the following signature. These APIGateway classes represent the HTTP API version of API Gateway.

public class ApiGatewayGetAllProductRequestHandler implements RequestHandler<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse> {

This causes an error with the OTEL Layer implementation using 'AWS_LAMBDA_EXEC_WRAPPER' = '/opt/otel-proxy-handler'

My Lambda function throws and error and doesn't complete and I get the following error message:

START RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba Version: $LATEST
EXTENSION	Name: collector	State: Ready	Events: [INVOKE,SHUTDOWN]
2022/08/04 12:53:29 [collector] Received event: {
	"eventType": "INVOKE",
	"deadlineMs": 1659617612128,
	"requestId": "0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba",
	"invokedFunctionArn": "arn:aws:lambda:eu-west-1:123456789012:function:JVMPerfTestStack-GetAllProductFunction9216DFAB-EAi3pX6W38xp",
	"tracing": {
		"type": "X-Amzn-Trace-Id",
		"value": "Root=1-62ebc144-782d1b5979d1d6b165c4873f;Parent=1a8f548170e20f69;Sampled=1"
	}
}
2022/08/04 12:53:29 [collector] Waiting for event...
2022-08-04T12:53:29.309Z	INFO	loggingexporter/logging_exporter.go:56	MetricsExporter	{"#metrics": 1}
Could not map API Gateway event body to requested parameter type: class com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent: java.lang.IllegalStateException
java.lang.IllegalStateException: Could not map API Gateway event body to requested parameter type: class com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.map(TracingRequestApiGatewayWrapper.java:41)
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.LambdaParameters.toArray(LambdaParameters.java:24)
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestWrapperBase.doHandleRequest(TracingRequestWrapperBase.java:63)
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.doHandleRequest(TracingRequestApiGatewayWrapper.java:49)
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.doHandleRequest(TracingRequestApiGatewayWrapper.java:21)
	at io.opentelemetry.instrumentation.awslambdacore.v1_0.TracingRequestHandler.handleRequest(TracingRequestHandler.java:79)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'eyJ0ZXN0IjoiYm9keSJ9': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"eyJ0ZXN0IjoiYm9keSJ9"; line: 1, column: 21]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:720)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2903)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1949)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:781)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4684)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4586)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3516)
	at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.map(TracingRequestApiGatewayWrapper.java:39)
	... 5 more

END RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba
REPORT RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba	Duration: 257.58 ms	Billed Duration: 258 ms	Memory Size: 2048 MB	Max Memory Used: 211 MB	Init Duration: 4067.70 ms	
XRAY TraceId: 1-62ebc144-782d1b5979d1d6b165c4873f	SegmentId: 1a8f548170e20f69	Sampled: true	

The error comes from this line

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/f236b2d4c9a0813b2dddf3d35928c79b21beb6d5/instrumentation/aws-lambda/aws-lambda-events-2.2/library/src/main/java/io/opentelemetry/instrumentation/awslambdaevents/v2_2/TracingRequestApiGatewayWrapper.java#L39

Where the code only expects 'APIGatewayProxyRequestEvent'. Other event objects will be used by customers and should be supported by the otel-proxy-handler.

[willarmiros]

Short term mitigation: Update https://aws-otel.github.io/docs/getting-started/lambda/lambda-java doc page to indicate only the 1 kind of APIGW event object is supported

Long term mitigation: actually fix the Lambda Java instrumentation code to support both types of objects.

[github-actions]

This issue was marked stale. It will be closed in 30 days without additional activity.

[arun-annamalai]

Any updates on a fix here?

[tylerbenson]

@arun-annamalai not yet. Any interest in submitting a PR fix?

[serkan-ozal]

@tylerbenson @msailes I think this issue should be fixed by this: open-telemetry/opentelemetry-java-instrumentation#11868
With this PR, existing built-in serialization/deserialization mechanism is being used for the events. But to get the benefit of this PR, /opt/otel-handler must be used instead of /opt/otel-proxy-handler

[serkan-ozal]

@msailes I think, this issue should be fixed in the latest Java wrapper layer version and by using /opt/otel-handler instead of /opt/otel-proxy-handler. Can you verify that?

[msailes]

/opt/otel-handler handles both request types. Does that have the same functionality as /opt/otel-proxy-handler?

[serkan-ozal]

@msailes as far as I see, they are similar in terms of functionality. The only minor difference I can see is that when you return non APIGatewayProxyResponseEvent typed response, TracingRequestApiGatewayWrapper (used by /opt/otel-proxy-handler script) wraps it inside APIGatewayProxyResponseEvent, but TracingRequestWrapper (used by /opt/otel-handler script) doesn't.

[msailes]

@serkan-ozal the documentation at https://aws-otel.github.io/docs/getting-started/lambda/lambda-java#enable-auto-instrumentation-for-your-lambda-function seems to suggest that there is more functionality.

[serkan-ozal]

@msailes if you're talking about "HTTP context propagation" mentioned in the doc you shared, it is already supported by TracingRequestWrapper impl too.
You can check the implementations here:

• https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-lambda/aws-lambda-events-2.2/library/src/main/java/io/opentelemetry/instrumentation/awslambdaevents/v2_2/TracingRequestWrapper.java
• https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-lambda/aws-lambda-events-2.2/library/src/main/java/io/opentelemetry/instrumentation/awslambdaevents/v2_2/TracingRequestApiGatewayWrapper.java

[msailes]

Fantastic, so it sounds like there is an opportunity to simplify the docs as well.