[ti_threatq] Update confidence scoring logic, IOC ingestion batch size, ECS field mapping issues and dashboard #12968

janvi-elastic · 2025-03-05T12:55:37Z

Type of change

Enhancement

Proposed commit message

Added pagination logic in data collection.
Update batch size to 1000.
Update confidence scoring logic.
Update ECS field mapping and dashboard.
Add agentless deployment

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ti_threatq directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: ti_threatq - START ---
╭────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_threatq │             │ asset     │ dashboard ti_threatq-a05fd810-78f1-11ec-a97c-7db1518ab848 is loaded │ PASS   │      1.186µs │
│ ti_threatq │             │ asset     │ dashboard ti_threatq-ab289de0-78f1-11ec-a97c-7db1518ab848 is loaded │ PASS   │        278ns │
│ ti_threatq │             │ asset     │ dashboard ti_threatq-b45b0c40-78f1-11ec-a97c-7db1518ab848 is loaded │ PASS   │        242ns │
│ ti_threatq │ threat      │ asset     │ index_template logs-ti_threatq.threat is loaded                     │ PASS   │        210ns │
│ ti_threatq │ threat      │ asset     │ ingest_pipeline logs-ti_threatq.threat-1.32.0 is loaded             │ PASS   │        301ns │
╰────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_threatq - END   ---
Done
Run pipeline tests for the package
--- Test results for package: ti_threatq - START ---
╭────────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                      │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_threatq │ threat      │ pipeline  │ (ingest pipeline warnings test-threatq-no-preserve-ndjson.log) │ PASS   │ 331.579152ms │
│ ti_threatq │ threat      │ pipeline  │ (ingest pipeline warnings test-threatq-sample-ndjson.log)      │ PASS   │ 300.842746ms │
│ ti_threatq │ threat      │ pipeline  │ test-threatq-no-preserve-ndjson.log                            │ PASS   │ 168.900169ms │
│ ti_threatq │ threat      │ pipeline  │ test-threatq-sample-ndjson.log                                 │ PASS   │ 253.134895ms │
╰────────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_threatq - END   ---
Done
Run static tests for the package
--- Test results for package: ti_threatq - START ---
╭────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ ti_threatq │ threat      │ static    │ Verify sample_event.json │ PASS   │ 119.912582ms │
╰────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_threatq - END   ---
Done
--- Test results for package: ti_threatq - START ---
╭────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ ti_threatq │ threat      │ system    │ default   │ PASS   │ 38.029587969s │
╰────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: ti_threatq - END   ---
Done

Related issues

Closes [ti_threatq] Confidence scoring & performance fixes #10783

Screenshot

…mapping issues and dashboard

elasticmachine · 2025-03-05T13:06:19Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-03-05T13:13:42Z

🚀 Benchmarks report

Package `ti_threatq` 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`threat`	2557.54	1375.52	-1182.02 (-46.22%)	💔

To see the full report comment with /test benchmark fullreport

packages/ti_threatq/_dev/build/docs/README.md

packages/ti_threatq/changelog.yml

packages/ti_threatq/data_stream/threat/agent/stream/httpjson.yml.hbs

packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

kcreddy · 2025-03-05T13:38:45Z

...eatq/data_stream/threat/_dev/test/pipeline/test-threatq-no-preserve-ndjson.log-expected.json

+            ],
+            "threat": {
+                "indicator": {
+                    "description": "This is a sample description.",


Can you give an real-world (anonymized) example of this value?
I would like to see if it fits the ECS description: https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-indicator-description

packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

packages/ti_threatq/img/threatq-overview.png

packages/ti_threatq/manifest.yml

efd6 · 2025-03-09T20:57:47Z

packages/ti_threatq/kibana/dashboard/ti_threatq-a05fd810-78f1-11ec-a97c-7db1518ab848.json

@@ -154,7 +154,7 @@
                        "description": "",
                        "params": {
                            "fontSize": 12,
-                            "markdown": "**Navigation**\n\n**[ThreatQ Overview (This Page)](/app/dashboards#/view/ti_threatq-a05fd810-78f1-11ec-a97c-7db1518ab848)**  \n[ThreatQ Files](/app/dashboards#/view/ti_threatq-ab289de0-78f1-11ec-a97c-7db1518ab848)  \n[ThreatQ URLs](/app/dashboards#/view/ti_threatq-b45b0c40-78f1-11ec-a97c-7db1518ab848)  \n\n[Integrations Page](/app/integrations/detail/ti_threatq/overview)\n\n\n**Overview**\n\nThis dashboard is a health overview related to the ThreatQ integration.\n\nThe dashboard is made to provide general statistics and show the health of the ingestion of indicators from ThreatQ.  \n\nIt shows the ingestion rates (by default it fetches new updates every 10 minutes) and provides a few filters for drilling down to specific indicator types retrieved from ThreatQ.",
+                            "markdown": "**Navigation**\n\n**This Page**  \n[ThreatQ Files](/app/dashboards#/view/ti_threatq-ab289de0-78f1-11ec-a97c-7db1518ab848)  \n[ThreatQ URLs](/app/dashboards#/view/ti_threatq-b45b0c40-78f1-11ec-a97c-7db1518ab848)  \n\n[Integrations Page](/app/integrations/detail/ti_threatq/overview)\n\n\n**Overview**\n\nThis dashboard is a health overview related to the ThreatQ integration.\n\nThe dashboard is made to provide general statistics and show the health of the ingestion of indicators from ThreatQ.  \n\nIt shows the ingestion rates (by default it fetches new updates every 10 minutes) and provides a few filters for drilling down to specific indicator types retrieved from ThreatQ.",


This is not right. The "This Page" in the gist is a place-holder, this should be "ThreatQ Overview". The 'this page'-ness is indicated by the bolding and non-link status of the text.

Also, all the lines have two spaces before the new line; this is not necessary.

Same below.

efd6 · 2025-03-09T21:01:31Z

packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

@@ -627,7 +627,7 @@ processors:
          tag: date_sources_created_at
          target_field: _ingest._value.created_at
          formats:
-            - ISO8601
+            - 'yyyy-MM-dd HH:mm:ss'


Do we know that all the timestamps are in this format?

Yes, In live logs we have date in this format. And also in the existing pipeline, we have used this format.

efd6 · 2025-03-10T06:48:04Z

/test

kcreddy

From previous review:
#12968 (comment)
and
#12968 (comment)

kcreddy · 2025-03-10T07:02:53Z

...eatq/data_stream/threat/_dev/test/pipeline/test-threatq-no-preserve-ndjson.log-expected.json

+            ],
+            "threat": {
+                "indicator": {
+                    "description": "IP x.x.x.x was observed delivering the Angler EK.",


Looks like this value is taken from ECS doc. What I meant is to add a real-world sample value (sanitized) from Threatq.

kcreddy · 2025-03-10T07:12:59Z

packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

-      field: threatq.ioc_expired_at
+      field: threatq.ioc_expires_at


No, here the intention is to set threatq.ioc_expired_at to either threatq.expires_at or threatq.expired_at. Based on that, we do indicator expiration. There is no field threatq.ioc_expires_at that we are going to need.

….expires_at

elasticmachine · 2025-03-10T08:51:57Z

💚 Build Succeeded

Buildkite Build
Commit: 1192c90

History

💚 Build #23165 succeeded 7cbf1e6
💚 Build #23084 succeeded 242012f
💚 Build #23037 succeeded 14d1534

elastic-sonarqube · 2025-03-10T08:52:01Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

kcreddy · 2025-03-10T10:03:34Z

...eatq/data_stream/threat/_dev/test/pipeline/test-threatq-no-preserve-ndjson.log-expected.json

@@ -452,7 +452,7 @@
            ],
            "threat": {
                "indicator": {
-                    "description": "IP x.x.x.x was observed delivering the Angler EK.",
+                    "description": "No longer poses a serious threat",


This value doesn't bode well with the ECS definition of threat.indicator.description, https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-indicator-description which represents an action. Lets remove this field.

Actually we are not getting description in live logs. We have taken this response from the sample log which is already present in existing integration. And also threat.indicator.description ecs mapping is also present in existing integration.

kcreddy

LGTM. Can be merged after @efd6 approval.

packages/ti_threatq/kibana/dashboard/ti_threatq-a05fd810-78f1-11ec-a97c-7db1518ab848.json

packages/ti_threatq/kibana/dashboard/ti_threatq-ab289de0-78f1-11ec-a97c-7db1518ab848.json

packages/ti_threatq/kibana/dashboard/ti_threatq-b45b0c40-78f1-11ec-a97c-7db1518ab848.json

elastic-vault-github-plugin-prod · 2025-03-11T09:51:09Z

Package ti_threatq - 1.32.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.32.0/

…e, ECS field mapping issues and dashboard (#12968) * Update confidence scoring logic, IOC ingestion batch size, ECS field mapping issues and dashboard * Update changelog * Resolve review comments * Update id and indicator_id field type to long and dashboard * Update description in log and set threatq.ioc_expired_at from threatq.expires_at

Update confidence scoring logic, IOC ingestion batch size, ECS field …

6533b80

…mapping issues and dashboard

janvi-elastic requested a review from a team as a code owner March 5, 2025 12:55

Update changelog

14d1534

kcreddy added enhancement New feature or request Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:ti_threatq ThreatQuotient (Partner supported) labels Mar 5, 2025

kcreddy reviewed Mar 5, 2025

View reviewed changes

Resolve review comments

242012f

janvi-elastic requested review from kcreddy and efd6 March 6, 2025 17:09

efd6 reviewed Mar 9, 2025

View reviewed changes

Update id and indicator_id field type to long and dashboard

7cbf1e6

janvi-elastic requested a review from efd6 March 10, 2025 06:50

kcreddy reviewed Mar 10, 2025

View reviewed changes

Update description in log and set threatq.ioc_expired_at from threatq…

1192c90

….expires_at

janvi-elastic requested a review from kcreddy March 10, 2025 08:55

kcreddy reviewed Mar 10, 2025

View reviewed changes

kcreddy approved these changes Mar 10, 2025

View reviewed changes

efd6 reviewed Mar 10, 2025

View reviewed changes

efd6 approved these changes Mar 11, 2025

View reviewed changes

kcreddy merged commit 7686e48 into elastic:main Mar 11, 2025
7 checks passed

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Mar 13, 2025

[ti_threatq] Update confidence scoring logic, IOC ingestion batch size, ECS field mapping issues and dashboard #12968

[ti_threatq] Update confidence scoring logic, IOC ingestion batch size, ECS field mapping issues and dashboard #12968

Conversation

janvi-elastic commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshot

Uh oh!

elasticmachine commented Mar 5, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package ti_threatq 👍(0) 💚(0) 💔(1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Mar 9, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janvi-elastic Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 commented Mar 10, 2025

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Mar 10, 2025

Quality Gate passed

Uh oh!

kcreddy Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

janvi-elastic Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 11, 2025

Uh oh!

Uh oh!

janvi-elastic commented Mar 5, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Mar 5, 2025 •

edited

Loading

Package `ti_threatq` 👍(0) 💚(0) 💔(1)

efd6 Mar 9, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading