Skip to content

Add cloudwatch input into AWS package for log collection #2323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jan 11, 2022
Merged

Add cloudwatch input into AWS package for log collection #2323

merged 12 commits into from
Jan 11, 2022

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Dec 8, 2021

What does this PR do?

We have aws-cloudwatch input in Filebeat as a separate input but never been used in any fileset under AWS module. This PR is to add aws-cloudwatch as an option for users to ingest logs using AWS package.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Related issues

Screenshots

For Cloudtrail logs:

Screen Shot 2021-12-08 at 2 38 10 PM

@kaiyan-sheng kaiyan-sheng self-assigned this Dec 8, 2021
@elasticmachine
Copy link

elasticmachine commented Dec 8, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-01-05T22:54:25.079+0000

  • Duration: 28 min 56 sec

  • Commit: 79e1227

Test stats 🧪

Test Results
Failed 0
Passed 323
Skipped 0
Total 323

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review December 9, 2021 21:17
@kaiyan-sheng kaiyan-sheng requested a review from aspacca December 14, 2021 04:38
@P1llus
Copy link
Member

P1llus commented Dec 14, 2021

@kaiyan-sheng If this is a new raw input package, can you add in a choice in the UI/menu for custom ingest pipeline as well? Similar to the TCP and UDP one.
Also are they not allowed to choose their datastream name similar to other input packages?

@kaiyan-sheng
Copy link
Contributor Author

Hey @P1llus 👋 This PR is not adding aws-cloudwatch as a new raw input package. It's simply adopting it in logs data streams in the existing aws package. Good to know what I need to pay attention to when adding a new raw input package though! Thank you!!

Copy link
Contributor

@aspacca aspacca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would ask @endorama to review that all the titles/descriptions are following the new guidelines

but for that just minor discrepancies on the equivalent files for every data stream

@@ -193,3 +194,107 @@ streams:
type: bool
multi: false
default: false
- input: aws-cloudwatch
template_path: aws-cloudwatch.yml.hbs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the equivalent files for elb_logs, firewall_logs and waf have enabled: false

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep choose which input to be enabled by default is discussed in #2318.

@kaiyan-sheng kaiyan-sheng merged commit b38094b into elastic:master Jan 11, 2022
@kaiyan-sheng kaiyan-sheng deleted the add_cloudwatch_input branch January 11, 2022 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support AWS logs with cloudwatch input
4 participants