Skip to content

feat(v2): new logging module #1435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 77 commits into from
Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
0e2443d
logging module independent from implementation, based on slf4j
jeromevdl Nov 22, 2022
6bda23c
log4j2 implementation of the logging module
jeromevdl Nov 22, 2022
3d6ef5d
logback implementation of the logging module
jeromevdl Nov 22, 2022
6e7227d
update parent pom
jeromevdl Nov 22, 2022
c094ee4
services
jeromevdl Nov 22, 2022
6520996
rebase + fix build
jeromevdl Sep 17, 2023
9fa5aac
fix tests
jeromevdl Sep 17, 2023
1b98b65
fix tests
jeromevdl Sep 17, 2023
381082d
add ignored file
jeromevdl Sep 17, 2023
c4f4b62
fix example
jeromevdl Sep 17, 2023
cd692e1
replace log4j with slf4j everywhere
jeromevdl Sep 17, 2023
5c0469d
use the implementation
jeromevdl Sep 18, 2023
eff1dc1
not for cdk (previous version)
jeromevdl Sep 18, 2023
afc6f64
fix build
jeromevdl Sep 18, 2023
e6121df
fix e2e
jeromevdl Sep 18, 2023
fb4f7a0
fix e2e logging
jeromevdl Sep 19, 2023
9ba07fa
fix e2e logging + field names
jeromevdl Sep 19, 2023
8db9b7c
new tests
jeromevdl Sep 19, 2023
135e2d5
move implementations to logging folder
jeromevdl Sep 19, 2023
eb88735
proper project naming
jeromevdl Sep 19, 2023
7b04add
add slf4j-log4j
jeromevdl Sep 19, 2023
7e1f48e
fix spotbug
jeromevdl Sep 19, 2023
4990e18
use JsonConfig from serialization module
jeromevdl Sep 19, 2023
198b5dc
complete tests
jeromevdl Sep 19, 2023
923ed76
add aspectj for compilation in the IDE (scope provided)
jeromevdl Sep 19, 2023
36cc83d
complete tests for log4j implementation
jeromevdl Sep 20, 2023
4749c5c
add services files
jeromevdl Sep 21, 2023
ab9ce9c
complete tests for log4j
jeromevdl Sep 21, 2023
bdee1f3
complete tests for logback
jeromevdl Sep 21, 2023
788dc97
fixe timezone
jeromevdl Sep 21, 2023
895843e
fix checkstyle
jeromevdl Sep 22, 2023
e5e3f18
improve correlationId:
jeromevdl Sep 22, 2023
e13876b
add support for environment variable POWERTOOLS_LOGGER_LOG_EVENT
jeromevdl Sep 22, 2023
988d46e
load env var at init
jeromevdl Sep 22, 2023
4a5d790
remove junit-pioneer and mock env var
jeromevdl Sep 22, 2023
0853b88
sampling rate should be final, maybe
jeromevdl Sep 22, 2023
f6dcad8
sonar lint
jeromevdl Sep 22, 2023
7ef955b
remove slf4j binding (does not improve perfs)
jeromevdl Sep 27, 2023
cdc07a5
fix unit tests (env var)
jeromevdl Oct 12, 2023
87bd0c2
remove mockito
jeromevdl Oct 26, 2023
5ffa5bd
fix logging examples
jeromevdl Oct 26, 2023
c1b8e66
fix logging examples #2
jeromevdl Oct 26, 2023
3462c8f
fix logging examples #3
jeromevdl Oct 26, 2023
f35c211
fix logging examples #4
jeromevdl Oct 27, 2023
bb6f87e
fix logging examples #5
jeromevdl Oct 27, 2023
3505679
merge v2 into logging
jeromevdl Nov 23, 2023
d94a0c5
add more tests
jeromevdl Nov 24, 2023
8a4f05e
refactor and cleanup
jeromevdl Nov 24, 2023
7d9336b
custom message resolver for json strings
jeromevdl Nov 27, 2023
3206439
fix
jeromevdl Nov 27, 2023
2dca173
logback message json
jeromevdl Nov 27, 2023
48dfa9a
logback message json
jeromevdl Nov 27, 2023
232d9ef
fix maven warnings
jeromevdl Nov 27, 2023
c35fbc7
fix tests
jeromevdl Nov 27, 2023
5934215
fix gradle build
jeromevdl Nov 27, 2023
6a93f9f
log messages as JSON (optional)
jeromevdl Nov 30, 2023
87d6730
log responses and log exceptions
jeromevdl Dec 1, 2023
ecb352e
merge v2 into logging
jeromevdl Dec 1, 2023
5ba6d1a
fix modules in e2e-parent
jeromevdl Dec 1, 2023
75f866f
Apply suggestions from code review
jeromevdl Dec 6, 2023
313f6da
post review changes
jeromevdl Dec 6, 2023
cff2ee6
fix gradle build
jeromevdl Dec 7, 2023
4572bf4
fix e2e tests
jeromevdl Dec 7, 2023
12b2844
update poms (transitive dependency for log4j)
jeromevdl Dec 7, 2023
74b377a
documentation
jeromevdl Dec 7, 2023
8cc5c66
spotbugs
jeromevdl Dec 7, 2023
13e18dc
Apply suggestions from code review on documentation
jeromevdl Dec 8, 2023
b1d94a2
cleanup
jeromevdl Dec 7, 2023
4d78308
doc and test completion
jeromevdl Dec 7, 2023
11efab3
update elastic doc
jeromevdl Dec 8, 2023
83233c7
improve doc
jeromevdl Dec 8, 2023
3f6412c
improve doc
jeromevdl Dec 8, 2023
64e79dd
code review
jeromevdl Dec 11, 2023
e7edb57
fix tests
jeromevdl Dec 11, 2023
fd334db
code review
jeromevdl Dec 11, 2023
634d13e
code review last episode
jeromevdl Dec 11, 2023
440d0c1
merge v2 into logging
jeromevdl Dec 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
complete tests for log4j implementation
  • Loading branch information
jeromevdl committed Oct 26, 2023
commit 36cc83d082d490c60e67354569aa4eae480adae5
35 changes: 35 additions & 0 deletions powertools-logging/powertools-logging-log4j/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,45 @@
<artifactId>jsonassert</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit-pioneer</groupId>
<artifactId>junit-pioneer</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>dev.aspectj</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.13.1</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<complianceLevel>${maven.compiler.target}</complianceLevel>
<aspectLibraries>
<aspectLibrary>
<groupId>software.amazon.lambda</groupId>
<artifactId>powertools-logging</artifactId>
</aspectLibrary>
</aspectLibraries>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjtools</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,19 @@
*
*/

package software.amazon.lambda.powertools.logging.internal;
package org.apache.logging.log4j.layout.template.json.resolver;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolver;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverConfig;
import org.apache.logging.log4j.layout.template.json.util.JsonWriter;
import org.apache.logging.log4j.util.ReadOnlyStringMap;
import software.amazon.lambda.powertools.common.internal.LambdaConstants;
import software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields;

/**
* Custom {@link org.apache.logging.log4j.layout.template.json.resolver.TemplateResolver}
* used by {@link org.apache.logging.log4j.layout.template.json.JsonTemplateLayout}
* to be able to recognize powertools fields in the LambdaJsonLayout.json file.
*/
final class PowertoolsResolver implements EventResolver {

private static final EventResolver COLD_START_RESOLVER = new EventResolver() {
Expand Down Expand Up @@ -87,7 +92,11 @@ public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
public boolean isResolvable(LogEvent logEvent) {
final String samplingRate =
logEvent.getContextData().getValue(PowertoolsLoggedFields.SAMPLING_RATE.getName());
return null != samplingRate;
try {
return (null != samplingRate && Float.parseFloat(samplingRate) > 0.f);
} catch (NumberFormatException nfe) {
return false;
}
}

@Override
Expand All @@ -98,12 +107,21 @@ public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
}
};

private static final EventResolver XRAY_TRACE_RESOLVER =
(final LogEvent logEvent, final JsonWriter jsonWriter) -> {
final String traceId =
logEvent.getContextData().getValue(PowertoolsLoggedFields.FUNCTION_TRACE_ID.getName());
jsonWriter.writeString(traceId);
};
private static final EventResolver XRAY_TRACE_RESOLVER = new EventResolver() {
@Override
public boolean isResolvable(LogEvent logEvent) {
final String traceId =
logEvent.getContextData().getValue(PowertoolsLoggedFields.FUNCTION_TRACE_ID.getName());
return null != traceId;
}

@Override
public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
final String traceId =
logEvent.getContextData().getValue(PowertoolsLoggedFields.FUNCTION_TRACE_ID.getName());
jsonWriter.writeString(traceId);
}
};

private static final EventResolver SERVICE_RESOLVER =
(final LogEvent logEvent, final JsonWriter jsonWriter) -> {
Expand All @@ -113,7 +131,7 @@ public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {

private static final EventResolver REGION_RESOLVER =
(final LogEvent logEvent, final JsonWriter jsonWriter) ->
jsonWriter.writeString(System.getenv("AWS_REGION"));
jsonWriter.writeString(System.getenv(LambdaConstants.AWS_REGION_ENV));

private static final EventResolver ACCOUNT_ID_RESOLVER = new EventResolver() {
@Override
Expand All @@ -132,7 +150,7 @@ public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
private static final EventResolver NON_POWERTOOLS_FIELD_RESOLVER =
(LogEvent logEvent, JsonWriter jsonWriter) -> {
StringBuilder stringBuilder = jsonWriter.getStringBuilder();
// remove dummy field to kick inn powertools resolver
// remove dummy field to kick in powertools resolver
stringBuilder.setLength(stringBuilder.length() - 4);

// Inject all the context information.
Expand Down Expand Up @@ -195,10 +213,6 @@ public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
}
}

static String getName() {
return "powertools";
}

@Override
public void resolve(LogEvent value, JsonWriter jsonWriter) {
internalResolver.resolve(value, jsonWriter);
Expand All @@ -207,6 +221,6 @@ public void resolve(LogEvent value, JsonWriter jsonWriter) {
@Override
public boolean isResolvable(LogEvent value) {
ReadOnlyStringMap contextData = value.getContextData();
return null != contextData && !contextData.isEmpty() && internalResolver.isResolvable();
return null != contextData && !contextData.isEmpty() && internalResolver.isResolvable(value);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,17 @@
*
*/

package software.amazon.lambda.powertools.logging.internal;
package org.apache.logging.log4j.layout.template.json.resolver;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverFactory;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolver;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverConfig;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverFactory;

@Plugin(name = "PowertoolsResolverFactory", category = TemplateResolverFactory.CATEGORY)
public final class PowertoolsResolverFactory implements EventResolverFactory {

private static final PowertoolsResolverFactory INSTANCE = new PowertoolsResolverFactory();
private static final String RESOLVER_NAME = "powertools";

private PowertoolsResolverFactory() {
}
Expand All @@ -38,7 +34,7 @@ public static PowertoolsResolverFactory getInstance() {

@Override
public String getName() {
return PowertoolsResolver.getName();
return RESOLVER_NAME;
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,6 @@
"$resolver": "powertools",
"field": "service_version"
},
"event.dataset": {
"$resolver": "powertools",
"field": "service_name"
},
"process.thread.name": {
"$resolver": "thread",
"field": "name"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
/*
* Copyright 2023 Amazon.com, Inc. or its affiliates.
* Licensed under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.logging.log4j.layout.template.json.resolver;

import static org.apache.commons.lang3.reflect.FieldUtils.writeStaticField;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.contentOf;
import static org.mockito.Mockito.when;
import static org.mockito.MockitoAnnotations.openMocks;

import com.amazonaws.services.lambda.runtime.Context;
import java.io.File;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.file.NoSuchFileException;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junitpioneer.jupiter.SetEnvironmentVariable;
import org.mockito.Mock;
import org.slf4j.MDC;
import software.amazon.lambda.powertools.common.internal.LambdaHandlerProcessor;
import software.amazon.lambda.powertools.logging.internal.handler.PowertoolsLogEnabled;

class PowerToolsResolverFactoryTest {

@Mock
private Context context;

@BeforeEach
void setUp() throws IllegalAccessException, IOException {
openMocks(this);
MDC.clear();
writeStaticField(LambdaHandlerProcessor.class, "IS_COLD_START", null, true);
setupContext();
// Make sure file is cleaned up before running tests
try {
FileChannel.open(Paths.get("target/logfile.json"), StandardOpenOption.WRITE).truncate(0).close();
FileChannel.open(Paths.get("target/ecslogfile.json"), StandardOpenOption.WRITE).truncate(0).close();
} catch (NoSuchFileException e) {
// file may not exist on the first launch
}
}

@Test
@SetEnvironmentVariable(key = "POWERTOOLS_SERVICE_NAME", value = "testLog4j")
@SetEnvironmentVariable(key = "POWERTOOLS_LOGGER_SAMPLE_RATE", value = "0.000000001")
@SetEnvironmentVariable(key = "_X_AMZN_TRACE_ID", value = "Root=1-63441c4a-abcdef012345678912345678")
void shouldLogInJsonFormat() {
PowertoolsLogEnabled handler = new PowertoolsLogEnabled();
handler.handleRequest("Input", context);

File logFile = new File("target/logfile.json");
assertThat(contentOf(logFile)).startsWith(
"{\"cold_start\":true,\"function_arn\":\"arn:aws:lambda:eu-west-1:012345678910:function:testFunction:1\",\"function_memory_size\":1024,\"function_name\":\"testFunction\",\"function_request_id\":\"RequestId\",\"function_version\":\"1\",\"level\":\"INFO\",\"message\":\"Test event\",\"sampling_rate\":1.0E-9,\"service\":\"testLog4j\",\"timestamp\":")
.endsWith("\"xray_trace_id\":\"1-63441c4a-abcdef012345678912345678\",\"myKey\":\"myValue\"}\n");
}

@Test
@SetEnvironmentVariable(key = "AWS_REGION", value = "eu-central-1")
@SetEnvironmentVariable(key = "POWERTOOLS_SERVICE_NAME", value = "testLog4jEcs")
@SetEnvironmentVariable(key = "_X_AMZN_TRACE_ID", value = "Root=1-63441c4a-abcdef012345678912345678")
void shouldLogInEcsFormat() {
PowertoolsLogEnabled handler = new PowertoolsLogEnabled();
handler.handleRequest("Input", context);

File logFile = new File("target/ecslogfile.json");
assertThat(contentOf(logFile)).endsWith(
"\"ecs.version\":\"1.2.0\",\"log.level\":\"INFO\",\"message\":\"Test event\",\"service.name\":\"testLog4j\",\"service.version\":\"1\",\"process.thread.name\":\"main\",\"log.logger\":\"software.amazon.lambda.powertools.logging.internal.handler.PowertoolsLogEnabled\",\"cloud.provider\":\"aws\",\"cloud.service.name\":\"lambda\",\"cloud.region\":\"eu-central-1\",\"cloud.account.id\":\"012345678910\",\"faas.coldstart\":true,\"faas.id\":\"arn:aws:lambda:eu-west-1:012345678910:function:testFunction:1\",\"faas.memory\":1024,\"faas.name\":\"testFunction\",\"faas.execution\":\"RequestId\",\"faas.version\":\"1\",\"myKey\":\"myValue\",\"trace.id\":\"1-63441c4a-abcdef012345678912345678\"}\n")
.startsWith("{\"@timestamp\":\"");
}

private void setupContext() {
when(context.getFunctionName()).thenReturn("testFunction");
when(context.getInvokedFunctionArn()).thenReturn(
"arn:aws:lambda:eu-west-1:012345678910:function:testFunction:1");
when(context.getFunctionVersion()).thenReturn("1");
when(context.getMemoryLimitInMB()).thenReturn(1024);
when(context.getAwsRequestId()).thenReturn("RequestId");
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/*
* Copyright 2023 Amazon.com, Inc. or its affiliates.
* Licensed under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.logging.log4j.layout.template.json.resolver;

import static org.assertj.core.api.Assertions.assertThat;
import static software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields.FUNCTION_ARN;
import static software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields.FUNCTION_COLD_START;
import static software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields.FUNCTION_MEMORY_SIZE;
import static software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields.SAMPLING_RATE;

import java.util.HashMap;
import java.util.Map;
import org.apache.logging.log4j.core.impl.Log4jLogEvent;
import org.apache.logging.log4j.layout.template.json.util.JsonWriter;
import org.apache.logging.log4j.util.SortedArrayStringMap;
import org.apache.logging.log4j.util.StringMap;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import org.junitpioneer.jupiter.SetEnvironmentVariable;
import software.amazon.lambda.powertools.logging.internal.PowertoolsLoggedFields;

class PowertoolsResolverTest {

@ParameterizedTest
@EnumSource(value = PowertoolsLoggedFields.class,
mode = EnumSource.Mode.EXCLUDE,
names = {"FUNCTION_MEMORY_SIZE", "SAMPLING_RATE", "FUNCTION_COLD_START"})
void shouldResolveFunctionStringInfo(PowertoolsLoggedFields field) {
String result = resolveField(field.getName(), "value");
assertThat(result).isEqualTo("\"value\"");
}

@Test
void shouldResolveMemorySize() {
String result = resolveField(FUNCTION_MEMORY_SIZE.getName(), "42");
assertThat(result).isEqualTo("42");
}

@Test
void shouldResolveSamplingRate() {
String result = resolveField(SAMPLING_RATE.getName(), "0.4");
assertThat(result).isEqualTo("0.4");
}

@Test
void shouldResolveColdStart() {
String result = resolveField(FUNCTION_COLD_START.getName(), "true");
assertThat(result).isEqualTo("true");
}

@Test
void shouldResolveAccountId() {
String result = resolveField(FUNCTION_ARN.getName(), "account_id", "arn:aws:lambda:us-east-2:123456789012:function:my-function");
assertThat(result).isEqualTo("\"123456789012\"");
}

@Test
@SetEnvironmentVariable(key = "AWS_REGION", value = "eu-central-2")
void shouldResolveRegion() {
String result = resolveField("region", "dummy, will use the env var");
assertThat(result).isEqualTo("\"eu-central-2\"");
}

private static String resolveField(String field, String value) {
return resolveField(field, field, value);
}

private static String resolveField(String data, String field, String value) {
Map<String, Object> configMap = new HashMap<>();
configMap.put("field", field);

TemplateResolverConfig config = new TemplateResolverConfig(configMap);
PowertoolsResolver resolver = new PowertoolsResolver(config);
JsonWriter writer = JsonWriter
.newBuilder()
.setMaxStringLength(1000)
.setTruncatedStringSuffix("")
.build();

StringMap contextMap = new SortedArrayStringMap();
contextMap.putValue(data, value);

Log4jLogEvent logEvent = Log4jLogEvent.newBuilder().setContextData(contextMap).build();
if (resolver.isResolvable(logEvent)) {
resolver.resolve(logEvent, writer);
}

return writer.getStringBuilder().toString();
}
}
Loading