[AWS] Handle duplicate fields in Network Firewall Logs data stream

[moxarth-rathod]

Proposed commit message

aws: network firewall logs ingest pipeline duplicate field error

For Network Firewall Logs, fixed the handling of duplicate fields error with the help of 
`allow_duplicate_keys` option in the JSON processor.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/aws directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes [AWS] Network Firewall logs ingest pipeline duplicate field error #5071

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[ShourieG]

@moxarth-rathod, is there any scenario where this duplicate key could be present in the mapping logic ? If so, we should have test sample data that has such duplicates and see the result.

[moxarth-rathod]

@moxarth-rathod, is there any scenario where this duplicate key could be present in the mapping logic ? If so, we should have test sample data that has such duplicates and see the result.

@ShourieG Yes, i've already added a test case to fulfil this scenario, in the new test case these fields - subject and issuerdn are duplicated.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5128a97

History

• 💚 Build #25557 succeeded c315fc9
• 💚 Build #25124 succeeded 1010cba

cc @moxarth-rathod

[ShourieG]

LGTM

[elastic-vault-github-plugin-prod]

Package aws - 3.3.1 containing this change is available at https://epr.elastic.co/package/aws/3.3.1/