Do not capture ClusterChangedEvent in IndicesStore call to #onClusterStateShardsClosed #120193
Conversation
…StateShardsClosed on IndicesClusterStateService#onClusterStateShardsClosed to be notified when the shard is closed and go through the process of deleting the local contents. The listener was using a lambda that captured the entire ClusterChangedEvent and in some cases if the shard was too slow to be closed we could keep capturing new cluster state instances and finally OOM the node. This commit just avoids capturing the entire ClusterChangedEvent.
|
Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing) |
|
Hi @fcofdez, I've created a changelog YAML for you. |
|
We noticed about this after a production node went OOM and upon the heap dump inspection we could observe how the internal There's the question of whether or not we should change how |
There was a problem hiding this comment.
LGTM
There's the question of whether or not we should change how
IndicesStoreschedules the calls todeleteShardIfExistElseWhere
Yes, similar to #74149 this whole area seems rather overcomplicated and inefficient.
|
Thanks for the review David! |
#108145 introduced async close of
IndexShard. Before that changeIndicesStorecalleddeleteShardIfExistElseWheredirectly when needed, but now it rely onIndicesClusterStateService#onClusterStateShardsClosedto be notified when the shard is closed and go through the process of deleting the local contents. The listener was using a lambda that captured the entireClusterChangedEventand in some cases if the shard was too slow to be closed we could keep capturing new cluster state instances and finally OOM the node. This commit just avoids capturing the entireClusterChangedEvent.