[Connector API] Support soft-deletes of connectors #118669

jedrazb · 2024-12-13T14:23:43Z

Soft deletes for connectors

Add support for soft-deletes of connectors. Why?

In order to improve UX on connectors on agentless we need to track what connectors were deleted vs. what connector records are not yet created

Changes

introduce deleted flag in the connector index mappings, indicates whether connector has been soft deleted
adapt delete, get and list operations logic to support this feature
- if we pass include_deleted=true flag we return also deleted connectors in the response
update docs
add yaml tests and unit tests

Related work

In a follow-up PR I will address adding index mapping migrations for connector index. I'm coordinating here with @navarone-feekery as he might be adding SystemIndexDecriptor for connector indices soon, this will help updating mappings a lot as there is SystemIndexMappingUpdateService
follow-up: support force deletes, e.g pass force=true to delete endpoint to completely remove the connector

github-actions · 2024-12-13T14:23:55Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-12-13T14:40:31Z

Hi @jedrazb, I've created a changelog YAML for you.

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

jedrazb · 2024-12-17T15:59:26Z

@elasticmachine merge upstream

navarone-feekery

LGTM!

…n use

jedrazb · 2024-12-19T16:03:10Z

can a new Connector have an index_name value identical to a soft-deleted Connector

That was the single edge case I didn't cover because we have custom check (with query) for index names already in use ... fixed in 7525c99

timgrein

Left some comments, but I think nothing preventing this PR from being merged

docs/reference/connector/apis/get-connector-api.asciidoc

rest-api-spec/src/main/resources/rest-api-spec/api/connector.list.json

...earch/src/main/java/org/elasticsearch/xpack/application/connector/ConnectorIndexService.java

...plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/Connector.java

Co-authored-by: Tim Grein <[email protected]>

…ist.json Co-authored-by: Tim Grein <[email protected]>

jedrazb · 2024-12-30T13:21:08Z

@elasticmachine merge upstream

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

docs/reference/connector/apis/delete-connector-api.asciidoc

… flag

jedrazb · 2025-01-08T14:56:08Z

New changes:

remove serialisation by wire for Connector class and get/list actions (since it's not needed) same as in Make the Get API keys API local only #105497
rename REST param deleted to include_deleted in get/list actions as IMO it conveys its purpose better

jedrazb · 2025-01-09T07:23:46Z

@elasticmachine merge upstream

navarone-feekery

Looks good! Just a few questions, nothing blocking.

...plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/Connector.java

...earch/src/main/java/org/elasticsearch/xpack/application/connector/ConnectorIndexService.java

...h/src/main/java/org/elasticsearch/xpack/application/connector/action/GetConnectorAction.java

.../main/java/org/elasticsearch/xpack/application/connector/action/RestListConnectorAction.java

jedrazb · 2025-01-09T11:17:53Z

@elasticmachine merge upstream

elasticmachine · 2025-01-09T11:17:56Z

merge conflict between base and head

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

jedrazb and others added 12 commits December 9, 2024 17:02

[Connector API] Add interface for soft-deletes

93ac07c

Define connector deleted system index

12567b2

Got soft-delete logic working

4fdc2ca

Add unit tests

d9be216

Add yaml e2e test and attempt to update permissions

183ed7a

Fix permissions

397032b

Update docs

d8523e0

Merge branch 'main' into support-soft-deletes-to-connectors

2640383

Fix docs

6e1197c

Update docs/changelog/118282.yaml

3d77bb2

Change logic

bf98ea8

Fix tests

b4eb411

jedrazb added :SearchOrg/Extract&Transform Label for the Search E&T team Team:Search - Extract & Transform v9.0.0 labels Dec 13, 2024

jedrazb added 2 commits December 13, 2024 15:24

Remove unnecessary privilege from yaml rest test

1b0845f

Update changelog

b51848e

jedrazb added the >feature label Dec 13, 2024

Update docs/changelog/118669.yaml

b206844

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

3c0c5eb

jedrazb mentioned this pull request Dec 13, 2024

[Connector API] Support soft deletes of connectors #118282

Closed

jedrazb added 3 commits December 16, 2024 10:26

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

5e02200

Adapt yaml tests

6ff5a8d

Merge branch 'support-soft-deletes-connectors-change-mapping' of gith…

8557414

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

elasticmachine and others added 2 commits December 17, 2024 15:59

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

471643f

Undo changes to muted-tests.yml

e697134

jedrazb marked this pull request as ready for review December 17, 2024 16:02

navarone-feekery approved these changes Dec 19, 2024

View reviewed changes

Exclude soft-deleted connector from checks about index_name already i…

7525c99

…n use

timgrein reviewed Dec 19, 2024

View reviewed changes

jedrazb and others added 3 commits December 30, 2024 13:19

Update docs/reference/connector/apis/get-connector-api.asciidoc

af7ad48

Co-authored-by: Tim Grein <[email protected]>

Update rest-api-spec/src/main/resources/rest-api-spec/api/connector.l…

f09aad4

…ist.json Co-authored-by: Tim Grein <[email protected]>

Adapt comments, add connector wire serializing test

7a9a90d

elasticmachine and others added 3 commits December 30, 2024 13:21

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

7d1e901

Introduce new transport versions for passing the delete flag

8d4ec53

Merge branch 'support-soft-deletes-connectors-change-mapping' of gith…

bda4f8d

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

seanstory reviewed Jan 2, 2025

View reviewed changes

docs/reference/connector/apis/delete-connector-api.asciidoc Show resolved Hide resolved

jedrazb mentioned this pull request Jan 7, 2025

[Agent] Don't auto-create connector doc if connector was deleted elastic/connectors#3077

Merged

8 tasks

jedrazb added 2 commits January 8, 2025 15:35

Get rid of wire serialisation, use include_deleted instead of deleted…

ea9394e

… flag

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

27bab3c

jedrazb added 2 commits January 8, 2025 16:19

Remove unused import

90862f2

Final tweaks

4d848cb

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

dec45e6

navarone-feekery approved these changes Jan 9, 2025

View reviewed changes

jedrazb added 2 commits January 9, 2025 12:15

Merge branch 'main' into support-soft-deletes-connectors-change-mapping

8c7c24d

Adapt variable name in rest layer

d2700e8

Merge branch 'support-soft-deletes-connectors-change-mapping' of gith…

ca7bbde

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping

jedrazb enabled auto-merge (squash) January 9, 2025 11:53

jedrazb merged commit 750a0ab into elastic:main Jan 9, 2025
16 checks passed

seanstory mentioned this pull request Jan 9, 2025

register 8.x deprecations elastic/kibana#205500

Merged

4 tasks

pquentin mentioned this pull request Jan 20, 2025

Adding new URL params to Connector API endpoints + new type to Connector types elastic/elasticsearch-specification#3549

Merged

[Connector API] Support soft-deletes of connectors #118669

[Connector API] Support soft-deletes of connectors #118669

Uh oh!

Conversation

jedrazb commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Soft deletes for connectors

Add support for soft-deletes of connectors. Why?

Changes

Related work

Uh oh!

github-actions bot commented Dec 13, 2024

Uh oh!

elasticsearchmachine commented Dec 13, 2024

Uh oh!

jedrazb commented Dec 17, 2024

Uh oh!

navarone-feekery left a comment

Choose a reason for hiding this comment

Uh oh!

jedrazb commented Dec 19, 2024

Uh oh!

timgrein left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jedrazb commented Dec 30, 2024

Uh oh!

Uh oh!

jedrazb commented Jan 8, 2025

Uh oh!

jedrazb commented Jan 9, 2025

Uh oh!

navarone-feekery left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jedrazb commented Jan 9, 2025

Uh oh!

elasticmachine commented Jan 9, 2025

Uh oh!

Uh oh!

Uh oh!

jedrazb commented Dec 13, 2024 •

edited

Loading