Skip to content

Adding chunking settings to IbmWatsonxService #114914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 21, 2024

Conversation

dan-rubinstein
Copy link
Member

@dan-rubinstein dan-rubinstein commented Oct 16, 2024

Issue - #117188

Description

This change introduces:

  • Chunking settings as a new setting to provide when creating embedding inference endpoints for the IbmWatsonxService

Note: Chunking currently only runs when large documents are ingested to an index with an inference field.

Testing

  • Ran unit tests and ensured that all succeed
  • Tested the following cases with each service with the feature flag enabled:
    • Creating an embedding endpoint with no chunking_settings provided generates a model with default chunking settings (keeps existing sentence based chunking strategy with default configurations).
    • Creating an embedding endpoint with chunking_settings for a word based chunking strategy provided generates a model that uses the word based chunking strategy.
    • Creating an embedding endpoint with chunking_settings for a sentence based chunking strategy provided generates a model that uses the sentence based chunking strategy.

@elasticsearchmachine elasticsearchmachine added v9.0.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Oct 16, 2024
@dan-rubinstein dan-rubinstein removed the external-contributor Pull request authored by a developer outside the Elasticsearch team label Oct 16, 2024
@dan-rubinstein
Copy link
Member Author

@elasticmachine merge upstream

@dan-rubinstein dan-rubinstein added >enhancement :ml Machine learning Team:ML Meta label for the ML team v8.17.0 labels Nov 7, 2024
@elasticsearchmachine
Copy link
Collaborator

Hi @dan-rubinstein, I've created a changelog YAML for you.

@dan-rubinstein dan-rubinstein marked this pull request as ready for review November 7, 2024 17:55
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

Copy link
Member

@davidkyle davidkyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dan-rubinstein dan-rubinstein added auto-backport Automatically create backport pull requests when merged v8.18.0 and removed v8.17.0 labels Nov 18, 2024
@dan-rubinstein
Copy link
Member Author

@elasticmachine merge upstream

@dan-rubinstein
Copy link
Member Author

@elasticmachine merge upstream

@dan-rubinstein dan-rubinstein merged commit 573b8a9 into elastic:main Nov 21, 2024
16 checks passed
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.x

dan-rubinstein added a commit to dan-rubinstein/elasticsearch that referenced this pull request Nov 21, 2024
* Adding chunking settings to IbmWatsonxService

* Removing feature flag

* Update docs/changelog/114914.yaml

---------

Co-authored-by: Elastic Machine <[email protected]>
elasticsearchmachine pushed a commit that referenced this pull request Nov 21, 2024
* Adding chunking settings to IbmWatsonxService

* Removing feature flag

* Update docs/changelog/114914.yaml

---------

Co-authored-by: Elastic Machine <[email protected]>
smalyshev pushed a commit to smalyshev/elasticsearch that referenced this pull request Nov 22, 2024
* Adding chunking settings to IbmWatsonxService

* Removing feature flag

* Update docs/changelog/114914.yaml

---------

Co-authored-by: Elastic Machine <[email protected]>
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this pull request Nov 28, 2024
* Adding chunking settings to IbmWatsonxService

* Removing feature flag

* Update docs/changelog/114914.yaml

---------

Co-authored-by: Elastic Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged >enhancement :ml Machine learning Team:ML Meta label for the ML team v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants