aws/data_stream/route53_public_logs: Handle non-%HOSTNAME entries

[cafuego]

Proposed commit message

Change the HOSTNAME pattern to DATA in the grok snippet to match route53 public logs, so that DNS records that contain characters not included in [0-9A-Za-z][0-9A-Za-z-] can still be processed.

This notably includes entries such as _dmarc, _spf and various types of domain ownership verification records.

Those are a non-exhaustive list that are being dropped from my logs, but note that the DNS spec doesn't prevent other characters being present. The spec does not inform what a client might query for, which is what we're dealing with, not a zone file.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Use the provided patterns in packages/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[cafuego]

Yup, still relevant and does not contain a dependency on xz 🤪

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ishleenk17]

/test

[ishleenk17]

@cafuego : cAn we please resolve the merge conflicts

[cafuego]

/test

[cafuego]

@shmsr Yes, but buildkite seems to be unhappy about something; it's not completing the test or pinging back with the result (which has been fine previously)

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

The alternative would be to provided an augmented hostname pattern that includes all the codepoints that occur in DNS names. This only requires that we add _ to the possible codepoints.

Suggested change

	- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
	pattern_definitions:
	- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DNS_QUESTION:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
	pattern_definitions:
	DNS_QUESTION: \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

[cafuego]

At the very least you also need to add an escaped backslash, or you'll still miss any encoded UTF-8 data.

DNS entries can also contain any binary rubbish you'd care to put in (although Route53 may not allow you to add that) so from a log analysis perspective it's important to grab those logs and allow them to be analysed.

[efd6]

DNS entries can also contain any binary rubbish you'd care to put in

That's horrifying, but point well made.

[efd6]

/test

[cafuego]

Aww, sad trombone. I'll update the test.

[efd6]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f93d5e4

History

• 💔 Build #13811 failed fe9303c
• 💔 Build #10881 failed e34c87a

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
28.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[efd6]

Thanks

[elasticmachine]

Package aws - 2.21.0 containing this change is available at https://epr.elastic.co/search?package=aws

[cafuego]

@efd6 thanks mate!