[Transform] Auto-migrate max_page_search_size #119348
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`max_page_search_size` had been deprecated in Pivot and migrated to Settings in ES 7.x. We are removing the key in Pivot in 9.x. It is still possible to create Pivots with `max_page_search_size`, though a deprecation is logged. To prep for its removal, this change automatically migrates `max_page_search_size` from Pivot to Settings as if the user called `_update`. - `PUT _transform` with `max_page_search_size` in Pivot will migrate the value to Settings as part of the Transform's creation. The user will receive an additional deprecation warning informing of this migration. - Existing transforms with `max_page_search_size` in Pivot will migrate the value to Settings as part of node assignment during start. Transform Messages and Elasticsearch logs will record this event so users will know what happened.
|
Hi @prwhelan, I've created a changelog YAML for you. |
|
Pinging @elastic/ml-core (Team:ML) |
| assertThat(createTransformResponse.get("acknowledged"), equalTo(Boolean.TRUE)); | ||
|
|
||
| Map<String, Object> transform = getTransformConfig(transformId, BASIC_AUTH_VALUE_TRANSFORM_ADMIN_WITH_SOME_DATA_ACCESS); | ||
| assertThat(XContentMapValues.extractValue("pivot.max_page_search_size", transform), equalTo(null)); |
There was a problem hiding this comment.
Minor: Can just use assertNull here.
| .withThrowable(validationException) | ||
| .log( | ||
| "Failed to validate auto-migrated Config. Please use the _update API to correct and update " | ||
| + "the Transform configuration. Continuing with old config." |
There was a problem hiding this comment.
Minor: Extra whitespace in between sentences.
| private TransformConfig autoMigrateConfig(Request request) { | ||
| var config = transformConfigAutoMigration.migrate(request.getConfig()); | ||
| if (config != request.getConfig()) { | ||
| deprecationLogger.warn( |
There was a problem hiding this comment.
Could this logging logic be contained within the transformConfigAutoMigration.migrate(...) function logic? Seems like something we'd want to happen whenever we call migrate and adding the logic there instead would also add a deprecation log when we call migrateAndSave.
| import static org.mockito.Mockito.verify; | ||
| import static org.mockito.Mockito.verifyNoInteractions; | ||
|
|
||
| public class TransformConfigAutoMigrationTests extends ESTestCase { |
There was a problem hiding this comment.
Can we add a test for when the transform has a max_page_search_size in both pivot config and settings and we just set the value to null? I know it's not a case we expect to actually hit but the code does specifically has this as an implementation decision we made so it's nice to have a test for it so we don't accidentally change this behavior.
|
@elasticmachine update branch |
|
@elasticmachine update branch |
|
@elasticmachine update branch |
|
Verified with latest changes, settings |
prwhelan
added a commit
to prwhelan/elasticsearch
that referenced
this pull request
Jan 7, 2025
`max_page_search_size` had been deprecated in Pivot and migrated to Settings in ES 7.x. We are removing the key in Pivot in 9.x. It is still possible to create Pivots with `max_page_search_size`, though a deprecation is logged. To prep for its removal, this change automatically migrates `max_page_search_size` from Pivot to Settings as if the user called `_update`. - `PUT _transform` with `max_page_search_size` in Pivot will migrate the value to Settings as part of the Transform's creation. The user will receive an additional deprecation warning informing of this migration. - Existing transforms with `max_page_search_size` in Pivot will migrate the value to Settings as part of node assignment during start. Transform Messages and Elasticsearch logs will record this event so users will know what happened.
prwhelan
added a commit
that referenced
this pull request
Jan 7, 2025
`max_page_search_size` had been deprecated in Pivot and migrated to Settings in ES 7.x. We are removing the key in Pivot in 9.x. It is still possible to create Pivots with `max_page_search_size`, though a deprecation is logged. To prep for its removal, this change automatically migrates `max_page_search_size` from Pivot to Settings as if the user called `_update`. - `PUT _transform` with `max_page_search_size` in Pivot will migrate the value to Settings as part of the Transform's creation. The user will receive an additional deprecation warning informing of this migration. - Existing transforms with `max_page_search_size` in Pivot will migrate the value to Settings as part of node assignment during start. Transform Messages and Elasticsearch logs will record this event so users will know what happened.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
max_page_search_sizehad been deprecated in Pivot and migrated to Settings in ES 7.x. We are removing the key in Pivot in 9.x. It is still possible to create Pivots withmax_page_search_size, though a deprecation is logged. To prep for its removal, this change automatically migratesmax_page_search_sizefrom Pivot to Settings as if the user called_update.PUT _transformwithmax_page_search_sizein Pivot will migrate the value to Settings as part of the Transform's creation. The user will receive an additional deprecation warning informing of this migration.max_page_search_sizein Pivot will migrate the value to Settings as part of node assignment during start. Transform Messages and Elasticsearch logs will record this event so users will know what happened.