[Sophos] UTM data stream: replace Javascript processing with ingest pipelines #6816

chemamartinez · 2023-07-05T07:10:26Z

What does this PR do?

This pull request updates the Sophos integration to:

Support DNS, DHCP, HTTP, and Packet Filter logs from UTM. Ingest pipelines have been added to process those events in favor of the current Javascript scripts.
Created dashboards for the event types mentioned below.
Fixed a small issue in the xg data stream about an incompatible event type and category.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes Sophos UTM #6184

Screenshots

Integration page and configuration

Dashboards

elasticmachine · 2023-07-05T07:17:00Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-07-05T07:36:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-24T10:08:14.705+0000
Duration: 19 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	38
Skipped	0
Total	38

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-05T07:36:14Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`17/17`)	💚
Classes	100.0% (`17/17`)	💚
Methods	100.0% (`125/125`)	💚 4.673
Lines	90.899% (`2487/2736`)	👍 9.911
Conditionals	100.0% (`0/0`)	💚

P1llus

Not reviewed packetfilter yet, nor the pipelines, just the test logs and expected.json for them. Will do the rest a bit later.

packages/sophos/data_stream/utm/_dev/test/pipeline/test-sophos-utm-dhcp.log-expected.json

P1llus · 2023-07-05T08:58:46Z

packages/sophos/data_stream/utm/_dev/test/pipeline/test-sophos-utm-dhcp.log-expected.json

+                ]
+            },
+            "host": {
+                "hostname": "sophos-test-vm2"


Are these the host or the observer? I would say its the observer maybe? Unsure if it was SIEM that required host.name or host.hostname for it to appear, but we could check that out if necessary.

I would say it is the hostname where the original log was created since Sophos reads directly from the original log files, based on the list here. In addition, the timestamp read from logs is considered the original timestamp of the event.

But, I am not 100% sure here.

packages/sophos/data_stream/utm/_dev/test/pipeline/test-sophos-utm-dhcp.log-expected.json

P1llus · 2023-07-05T09:12:51Z

packages/sophos/data_stream/utm/_dev/test/pipeline/test-sophos-utm-http.log-expected.json

+                        "country": "United States",
+                        "dnstime": 5,
+                        "filteraction": "REF_HTTP_ACTION",
+                        "fullreqtime": 32181,


Would this match with event.duration? Unsure as I have not checked their docs

It seems so, but I am not 100% sure either, most of HTTP events have several time-related fields, such as:

authtime="0" dnstime="249" aptptime="0" cattime="206" avscantime="0" fullreqtime="1499"

It is complicated to say for sure as I couldn't find any docs related to these fields.

packages/sophos/data_stream/utm/_dev/test/pipeline/test-sophos-utm-http.log-expected.json

efd6 · 2023-07-19T23:06:08Z

packages/sophos/data_stream/xg/elasticsearch/ingest_pipeline/event.yml

+    value: start
    if: "['17701', '17704', '17707', '17710', '17713'].contains(ctx.event?.code)"


For future spelunkers, https://docs.sophos.com/nsg/sophos-firewall/17.5/Help/en-us/webhelp/onlinehelp/nsg/sfos/concepts/LogMessages.html.

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/default.yml

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/packetfilter.yml

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/http.yml

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/packetfilter.yml

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/default.yml

elasticmachine · 2023-07-24T10:54:18Z

Package sophos - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=sophos

elasticmachine · 2023-07-24T10:54:19Z

Package sophos - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=sophos

elasticmachine · 2023-07-24T10:54:20Z

Package sophos - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=sophos

elasticmachine · 2023-07-24T10:54:21Z

Package sophos - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=sophos

elasticmachine · 2023-07-24T10:54:22Z

Package sophos - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=sophos

chemamartinez added enhancement New feature or request Team:Security-External Integrations Integration:sophos Sophos labels Jul 5, 2023

chemamartinez self-assigned this Jul 5, 2023

chemamartinez added 8 commits July 5, 2023 09:12

Fix invalid event type for XG data stream

9723bd7

Update Sophos UTM data stream

fa7243c

Add event outcome and log source to UTM events

b4b3c32

Generate sample event and add it to package docs

92e53b1

Convert some fields from keyword to long

c7b3e2a

Create dashboards for Sophos UTM

81369fa

Add screenshots of Sophos UTM dashboards

4408e57

Update changelog

33e94b3

chemamartinez force-pushed the 6184-sophos-utm branch from 6c0f620 to 33e94b3 Compare July 5, 2023 07:16

chemamartinez marked this pull request as ready for review July 5, 2023 07:16

chemamartinez requested a review from a team as a code owner July 5, 2023 07:16

P1llus reviewed Jul 5, 2023

View reviewed changes

chemamartinez added 3 commits July 14, 2023 18:46

Add requested changes

f92babb

Adapt dashboards to requested changes

af30b24

Add service to event.action for some cases

39a94ad

chemamartinez requested a review from P1llus July 18, 2023 06:59

efd6 reviewed Jul 19, 2023

View reviewed changes

Add requested changes in ingest pipelines

bcddffb

efd6 reviewed Jul 24, 2023

View reviewed changes

packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Remove extra conditions

7127517

chemamartinez requested a review from efd6 July 24, 2023 07:20

efd6 approved these changes Jul 24, 2023

View reviewed changes

Lowercase observer type

577586a

chemamartinez merged commit 42f49ac into elastic:main Jul 24, 2023

chemamartinez deleted the 6184-sophos-utm branch February 6, 2025 10:30

		value: start
		if: "['17701', '17704', '17707', '17710', '17713'].contains(ctx.event?.code)"

[Sophos] UTM data stream: replace Javascript processing with ingest pipelines #6816

[Sophos] UTM data stream: replace Javascript processing with ingest pipelines #6816

Uh oh!

Conversation

chemamartinez commented Jul 5, 2023

What does this PR do?

Checklist

Related issues

Screenshots

Uh oh!

elasticmachine commented Jul 5, 2023

Uh oh!

elasticmachine commented Jul 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Jul 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

P1llus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

P1llus Jul 5, 2023

Choose a reason for hiding this comment

Uh oh!

chemamartinez Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

P1llus Jul 5, 2023

Choose a reason for hiding this comment

Uh oh!

chemamartinez Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Jul 19, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

Uh oh!

elasticmachine commented Jul 5, 2023 •

edited

Loading

elasticmachine commented Jul 5, 2023 •

edited

Loading