AWS Integration enable v9 Kibana support #12637

gizas · 2025-02-06T10:59:34Z

Enhancement

Proposed commit message

WHAT: Enabling support for AWS and AWS Custom Logs integrations for 9.0 version
WHY: Is needed in order to enable above integrations in version 9.0.0

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone Pr
elastic-package build with v0.109.1
elastic-package stack up -d -v --version=9.0.0-SNAPSHOT to install a local ES
Install an agent + Fleet with prementioned integrations enabled

Related issues

Relates Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

Screenshots

No errors in Fleet:

Cloudwatch Metrics:

Cloudwatch logs

AWS S3 Logs:

AWS Custom Logs Integration

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-06T11:02:23Z

packages/aws/data_stream/guardduty/elasticsearch/ingest_pipeline/default.yml

@@ -15,6 +15,11 @@ processors:
      target_field: event.original
      ignore_missing: true
      if: 'ctx.event?.original == null'
+  - remove:


To remove errors for JSE0001:

Error: building package failed: invalid content found in built zip package: found 2 validation errors: 1. file "/Users/andreasgkizas/elastic/integrations3/integrations/build/packages/aws-2.39.0.zip/data_stream/securityhub_insights/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.158.remove: if is required 2. file "/Users/andreasgkizas/elastic/integrations3/integrations/build/packages/aws-2.39.0.zip/data_stream/securityhub_insights/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.158.remove.field: rename "message" to "event.original" processor requires remove "message" processor (JSE00001)

gizas · 2025-02-06T11:02:47Z

packages/aws/validation.yml

@@ -1,4 +1,3 @@
 errors:
  exclude_checks:
    - SVR00004 # References in dashboards.
-    - SVR00005 # Kibana version for saved tags.


Not needed anymore, it can built without this

elasticmachine · 2025-02-06T11:03:28Z

💔 Build Failed

Buildkite Build
Commit: c574f06

Failed CI Steps

✅ Check go sources

History

gizas · 2025-02-06T11:46:13Z

/test stack 9.0.0-SNAPSHOT

Signed-off-by: Andreas Gkizas <[email protected]>

…ment9

gizas · 2025-02-07T11:34:35Z

/test stack 9.0.0-SNAPSHOT

elasticmachine · 2025-02-07T11:46:23Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: 6d2f19d

Failed CI Steps

Check integrations aws

elasticmachine · 2025-02-07T11:50:37Z

💔 Build Failed

Buildkite Build
Commit: 6d2f19d

Failed CI Steps

Check integrations aws

gizas · 2025-02-07T11:56:43Z

/test

gizas · 2025-02-07T11:57:04Z

/test stack 9.0.0-SNAPSHOT

elasticmachine · 2025-02-07T12:11:29Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: 6d2f19d

Failed CI Steps

Check integrations aws

gizas · 2025-02-07T12:54:38Z

/test

gizas · 2025-02-07T14:12:12Z

/test

gizas · 2025-02-10T13:40:44Z

9.0.0 done as part of this #12503

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-10T15:20:54Z

/test

…ment9

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-11T15:32:24Z

packages/aws/data_stream/waf/_dev/test/system/test-default-config.yml

@@ -7,7 +7,11 @@ vars:
 skip_ignored_fields:
  - aws.waf.terminating_rule_match_details.location
  - aws.waf.non_terminating_matching_rules.ruleMatchDetails.location
+  - aws.waf.non_terminating_matching_rules.ruleMatchDetails.action


@elastic/security-service-integrations team can you please help understand here why I have those failures?

See https://buildkite.com/elastic/integrations/builds/22118

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-11T15:45:13Z

packages/aws/data_stream/waf/fields/fields.yml

@@ -13,10 +13,59 @@
      type: nested
      description: |
        The list of non-terminating rules in the rule group that match the request. These are always COUNT rules (non-terminating rules that match)
+      fields:


Needed to solve the pipeline errors of https://buildkite.com/elastic/integrations/builds/22047

Big thanks to @zmoog for details

Is this because these fields existed in the beats metricset and not incorporated for Integrations ?

Because of this, for spec +3.0.1, subobject fields need to be explicitly defined.

Specifically this: elastic/elastic-package#1489

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-12T10:15:27Z

/test stack 9.0.0-SNAPSHOT

elastic-vault-github-plugin-prod · 2025-02-12T11:09:00Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-02-12T11:09:46Z

💚 Build Succeeded

Buildkite Build
Commit: 61a0699

History

💔 Build #22182 failed beae77a
💔 Build #22177 failed f54effa
💔 Build #22175 failed 35b2605
💔 Build #22125 failed 172cde4
💔 Build #22124 failed d5a7e02

gizas · 2025-02-12T12:23:52Z

@elastic/security-service-integrations can I have a review please?

kcreddy · 2025-02-14T05:31:22Z

packages/aws/manifest.yml

@@ -1,7 +1,7 @@
-format_version: 3.0.0
+format_version: 3.3.1


As per suggestion from: https://github.com/elastic/ingest-dev/issues/4484#issuecomment-2551112710

packages/aws/data_stream/waf/fields/fields.yml

kcreddy · 2025-02-14T06:05:52Z

packages/aws/data_stream/waf/fields/fields.yml

@@ -13,10 +13,59 @@
      type: nested
      description: |
        The list of non-terminating rules in the rule group that match the request. These are always COUNT rules (non-terminating rules that match)
+      fields:


Specifically this: elastic/elastic-package#1489

packages/aws_logs/manifest.yml

Signed-off-by: Andreas Gkizas <[email protected]>

gizas · 2025-02-14T12:00:52Z

/test stack 9.0.0-SNAPSHOT

elastic-sonarqube · 2025-02-14T13:29:15Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-02-14T13:29:17Z

💚 Build Succeeded

Buildkite Build
Commit: af7b741

elasticmachine · 2025-02-14T13:31:44Z

💚 Build Succeeded

Buildkite Build
Commit: af7b741

gizas · 2025-02-17T06:55:16Z

@kcreddy a final review please ?

elastic-vault-github-plugin-prod · 2025-02-17T07:37:54Z

Package aws - 2.40.0 containing this change is available at https://epr.elastic.co/package/aws/2.40.0/

elastic-vault-github-plugin-prod · 2025-02-17T07:37:55Z

Package aws_logs - 1.7.0 containing this change is available at https://epr.elastic.co/package/aws_logs/1.7.0/

* aws commit to enable v9 support Signed-off-by: Andreas Gkizas <[email protected]> --------- Signed-off-by: Andreas Gkizas <[email protected]>

aws commit to enable v9 support

c574f06

Signed-off-by: Andreas Gkizas <[email protected]>

gizas requested review from a team as code owners February 6, 2025 10:59

gizas added Integration:aws AWS Integration:aws_logs Custom AWS Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Feb 6, 2025

gizas commented Feb 6, 2025

View reviewed changes

fixing prs

7e0215d

Signed-off-by: Andreas Gkizas <[email protected]>

gizas mentioned this pull request Feb 6, 2025

[Meta] Obs-ds-hosted-services Integration enablement for 9.0.0 #12529

Closed

14 tasks

gizas added 2 commits February 6, 2025 14:58

fixing prs and adding manifest value

c28034c

Signed-off-by: Andreas Gkizas <[email protected]>

Merge branch 'main' of github.com:elastic/integrations into awsenable…

1998a1d

…ment9

gizas closed this Feb 7, 2025

gizas reopened this Feb 7, 2025

Merge branch 'main' of github.com:elastic/integrations into awsenable…

6d2f19d

…ment9

gizas closed this Feb 10, 2025

merging with main

6f95f85

Signed-off-by: Andreas Gkizas <[email protected]>

gizas reopened this Feb 10, 2025

gizas added 2 commits February 11, 2025 13:26

Merge branch 'main' of github.com:elastic/integrations into awsenable…

3ba561f

…ment9

updating waf tests

5382883

Signed-off-by: Andreas Gkizas <[email protected]>

gizas commented Feb 11, 2025

View reviewed changes

updating waf tests

d5a7e02

Signed-off-by: Andreas Gkizas <[email protected]>

gizas commented Feb 11, 2025

View reviewed changes

gizas added 5 commits February 11, 2025 17:57

updating waf tests

172cde4

Signed-off-by: Andreas Gkizas <[email protected]>

updating waf tests

35b2605

Signed-off-by: Andreas Gkizas <[email protected]>

updating waf tests

f54effa

Signed-off-by: Andreas Gkizas <[email protected]>

updating waf tests

beae77a

Signed-off-by: Andreas Gkizas <[email protected]>

updating waf tests1

61a0699

Signed-off-by: Andreas Gkizas <[email protected]>

kcreddy reviewed Feb 14, 2025

View reviewed changes

address comments

af7b741

Signed-off-by: Andreas Gkizas <[email protected]>

ishleenk17 approved these changes Feb 14, 2025

View reviewed changes

MichaelKatsoulis approved these changes Feb 14, 2025

View reviewed changes

kcreddy approved these changes Feb 17, 2025

View reviewed changes

gizas merged commit f1b34ed into main Feb 17, 2025
7 checks passed

gizas deleted the awsenablement9 branch February 17, 2025 07:21

flexitrev pushed a commit that referenced this pull request Mar 20, 2025

AWS Integration enable v9 Kibana support (#12637)

127cd6d

* aws commit to enable v9 support Signed-off-by: Andreas Gkizas <[email protected]> --------- Signed-off-by: Andreas Gkizas <[email protected]>

AWS Integration enable v9 Kibana support #12637

AWS Integration enable v9 Kibana support #12637

Uh oh!

Conversation

gizas commented Feb 6, 2025

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

No errors in Fleet:

Cloudwatch Metrics:

Cloudwatch logs

AWS S3 Logs:

AWS Custom Logs Integration

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

gizas commented Feb 6, 2025

Uh oh!

gizas commented Feb 7, 2025

Uh oh!

elasticmachine commented Feb 7, 2025

⏳ Build in-progress, with failures

Failed CI Steps

Uh oh!

elasticmachine commented Feb 7, 2025

💔 Build Failed

Failed CI Steps

Uh oh!

gizas commented Feb 7, 2025

Uh oh!

gizas commented Feb 7, 2025

Uh oh!

elasticmachine commented Feb 7, 2025

⏳ Build in-progress, with failures

Failed CI Steps

Uh oh!

gizas commented Feb 7, 2025

Uh oh!

gizas commented Feb 7, 2025

Uh oh!

gizas commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gizas commented Feb 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gizas commented Feb 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented Feb 12, 2025

💚 Build Succeeded

History

Uh oh!

gizas commented Feb 12, 2025

Uh oh!

kcreddy Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

elasticmachine commented Feb 6, 2025 •

edited

Loading

gizas commented Feb 10, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 12, 2025 •

edited

Loading

kcreddy Feb 14, 2025 •

edited

Loading