[AWS] Add CloudWatch logs latency config option #5777

zmoog · 2023-04-03T21:57:26Z

Motivation

If the logs collection starts at 10:00, then the input will gather logs in this time window:

start: 09:59
end: 10:00

Unfortunately, logs sometimes become available in CloudWatch with a delay; in these circumstances, the input of the CloudWatch logs could miss them.

For example, if a log event is:

created: 09:59:30
available 10:00:30

The input will miss that event with the default latency setting ¹.

However, the CloudWatch Logs input has a setting called latency that allows users to adjust the logs collection time window to cope with the AWS services' latency.

Change description

Make the CloudWatch Logs input setting option latency available in the integration settings.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes AWS Cloudwatch integration missing latency option #5712

Screenshots

Settings UI

Documentation

The input will pick that log event up at the next restart if "start position" is set to "beginning". ↩

elasticmachine · 2023-04-03T22:48:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-04T13:48:40.658+0000
Duration: 52 min 33 sec

Test stats 🧪

Test	Results
Failed	0
Passed	188
Skipped	4
Total	192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-04-03T22:48:55Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`15/15`)	💚
Files	93.75% (`15/16`)	👎 -6.25
Classes	93.75% (`15/16`)	👎 -6.25
Methods	86.131% (`236/274`)	👎 -9.107
Lines	85.925% (`7387/8597`)	👎 -8.88
Conditionals	100.0% (`0/0`)	💚

endorama · 2023-04-04T12:43:30Z

This PR closes #5712

endorama · 2023-04-04T12:46:05Z

packages/aws/data_stream/cloudwatch_logs/manifest.yml

@@ -149,6 +149,12 @@ streams:
        show_user: false
        default: 200ms
        description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account.
+      - name: latency


I see there is a description in the screenshot and I expected to see it here. Is this missing the description field?

Thanks (again) for noticing this!

I missed the line with the description field because it was attached to another change for a different PR 🤦🤦

Added with 17afdcf

endorama · 2023-04-04T12:46:55Z

packages/aws/_dev/build/docs/cloudwatch.md

+
+Log events on the busies log groups may require a longer time before they are available to CloudWatch Logs.
+
+The CloudWatch integration offers the `latency` setting to cope with this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago.


I would document the format expected by this setting, as it may not be familiar. (es a link to the documentation for the underlying string parser that converts strings to time durations)

Good point! Adding it.

What do you think about this?

Details at 2091d4e

zmoog · 2023-04-04T13:19:43Z

This PR closes #5712

Oh boy, thank you for the heads up; I forgot to add a reference to this issue! 🤦 (added in the PR description)

elasticmachine · 2023-04-04T16:45:32Z

Package aws - 1.33.0 containing this change is available at https://epr.elastic.co/search?package=aws

zmoog added enhancement New feature or request Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Apr 3, 2023

zmoog self-assigned this Apr 3, 2023

zmoog added 3 commits April 4, 2023 13:44

Add latency config option

e59b7b4

Update changelog

7097ee7

Update docs

5e0d1b7

zmoog force-pushed the zmoog/add-cloudwatch-logs-latency-option branch from a57e176 to 5e0d1b7 Compare April 4, 2023 11:45

zmoog marked this pull request as ready for review April 4, 2023 11:45

zmoog requested a review from a team as a code owner April 4, 2023 11:45

dmathieu approved these changes Apr 4, 2023

View reviewed changes

endorama reviewed Apr 4, 2023

View reviewed changes

zmoog added 2 commits April 4, 2023 15:25

Adding missing latency field description

17afdcf

Latency field: add samples and link to reference

2091d4e

zmoog requested a review from endorama April 4, 2023 14:06

zmoog merged commit 04e6661 into main Apr 4, 2023

zmoog deleted the zmoog/add-cloudwatch-logs-latency-option branch April 4, 2023 16:23

andrewkroh added the Integration:aws AWS label Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[AWS] Add CloudWatch logs latency config option #5777

[AWS] Add CloudWatch logs latency config option #5777

Uh oh!

zmoog commented Apr 3, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Apr 3, 2023 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Apr 3, 2023 •

edited

Loading

Uh oh!

endorama commented Apr 4, 2023

Uh oh!

endorama Apr 4, 2023 •

edited

Loading

Uh oh!

zmoog Apr 4, 2023

Uh oh!

zmoog Apr 4, 2023

Uh oh!

endorama Apr 4, 2023

Uh oh!

zmoog Apr 4, 2023

Uh oh!

zmoog Apr 4, 2023

Uh oh!

zmoog commented Apr 4, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Apr 4, 2023

Uh oh!

Uh oh!


		Log events on the busies log groups may require a longer time before they are available to CloudWatch Logs.

		The CloudWatch integration offers the `latency` setting to cope with this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago.

[AWS] Add CloudWatch logs latency config option #5777

[AWS] Add CloudWatch logs latency config option #5777

Uh oh!

Conversation

zmoog commented Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Change description

Checklist

Related issues

Screenshots

Settings UI

Documentation

Footnotes

Uh oh!

elasticmachine commented Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

endorama commented Apr 4, 2023

Uh oh!

endorama Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmoog Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

endorama Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Apr 4, 2023

Uh oh!

Uh oh!

zmoog commented Apr 3, 2023 •

edited

Loading

elasticmachine commented Apr 3, 2023 •

edited

Loading

elasticmachine commented Apr 3, 2023 •

edited

Loading

endorama Apr 4, 2023 •

edited

Loading

zmoog commented Apr 4, 2023 •

edited

Loading