Snapshot Repository with azure Managed Identity

[muecahit94]

Elasticsearch Version

8.18.1, 9.0.1

Installed Plugins

No response

Java Version

24

OS Version

Red Hat Enterprise Linux release 9.5 (Plow)

Problem Description

Hi,
We have ECK-Stack installed on our AKS clusters and use snapshots to back up our Elasticsearch instances to a Storage account on Azure.
At the moment we use the Access Key option to access the Storage Account, but we want to change it and use Managed Identities.

We have set it up as described here, but it works just sometimes and very unstable.
When I want to create a Snapshot Repository, it takes very long or runs sometimes in a timeout, or when I want to verify repository(test), it takes long and works or fails with in a timeout.
The same things with the Access Key works very smooth and fast.

We get alot of these warnings when I want to create/edit a Snapshot Repository or test the connection, and every click on Kibana takes very long, also when it runs in timeout when I do things over the Console. Logs are added in the Logs section

The Setup for this testing enviroment has a Master and a Data Node.

Versions:
ECK-Operator Helm: 3.0.0
ECK-Stack Helm: 0.15.0
Elasticsearch: 9.0.1
Kibana: 9.0.1

Steps to Reproduce

Install ECK-Operator, then start an ECK-Stack with a master and a data node.
Prepare Snapshot Repository as described here and create a snapshot.

Logs (if relevant)

{"@timestamp":"2025-05-13T12:53:07.695Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][azure_event_loop][T#1]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.reactor.netty.core","trace.id":"3ece9635beb5887764779e7e95c58c77","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)\n\tat reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)\n\tat reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)\n\tat com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T12:53:10.775Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][azure_event_loop][T#1]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.reactor.netty.core","trace.id":"3ece9635beb5887764779e7e95c58c77","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)\n\tat reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)\n\tat reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)\n\tat com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T12:53:10.843Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][azure_event_loop][T#1]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.reactor.netty.core","trace.id":"3ece9635beb5887764779e7e95c58c77","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)\n\tat reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)\n\tat reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)\n\tat com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T12:53:11.651Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][azure_event_loop][T#1]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.reactor.netty.core","trace.id":"3ece9635beb5887764779e7e95c58c77","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)\n\tat reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)\n\tat reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)\n\tat com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

Stack trace is:

org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]
	at org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)
	at org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)
	at org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)
	at org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)
	at org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)
	at java.base/java.lang.Thread.setDaemon(Thread.java)
	at reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)
	at io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)
	at reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)
	at reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)
	at reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)
	at reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)
	at reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)
	at reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)
	at reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)
	at reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)
	at reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)
	at reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)
	at reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)
	at reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)
	at reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)
	at reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)
	at com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)
	at com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
	at com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
	at com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)
	at com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
	at com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
	at com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
	at com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
	at com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
	at com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
	at com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
	at com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:251)
	at org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)
	at org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)
	at org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)

Strange that this doesn't come up in the tests that exercise this functionality.

[DaveCTurner]

Strange that this doesn't come up in the tests that exercise this functionality.

Ah ok it seems it's this:

elasticsearch/modules/repository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureClientProvider.java

Lines 186 to 188 in 8ad2723

	if (DISABLE_INSTANCE_DISCOVERY) {
	credentialBuilder.disableInstanceDiscovery();
	}

We can't validate the test metadata endpoint against https://login.microsoft.com/ because it's not a real metadata endpoint.

[DaveCTurner]

I believe it should work to override the policy with a workaround on each node as follows:

mkdir -p ${ES_PATH_CONF}/jvm.options.d && echo '-Des.entitlements.policy.repository-azure=dmVyc2lvbnM6CiAtIDguMTguMAogLSA4LjE4LjEKIC0gOS4wLjAKIC0gOS4wLjEKcG9saWN5OgogaW8ubmV0dHkuY29tbW9uOgogIC0gb3V0Ym91bmRfbmV0d29yawogIC0gbWFuYWdlX3RocmVhZHMKICAtIGZpbGVzOgogICAgLSBwYXRoOiAiL2V0Yy9vcy1yZWxlYXNlIgogICAgICBtb2RlOiAicmVhZCIKICAgIC0gcGF0aDogIi91c3IvbGliL29zLXJlbGVhc2UiCiAgICAgIG1vZGU6ICJyZWFkIgogICAgLSBwYXRoOiAiL3Byb2Mvc3lzL25ldC9jb3JlL3NvbWF4Y29ubiIKICAgICAgbW9kZTogcmVhZAogY29tLmF6dXJlLmlkZW50aXR5OgogIC0gb3V0Ym91bmRfbmV0d29yawogIC0gZmlsZXM6CiAgICAtIHJlbGF0aXZlX3BhdGg6ICJzdG9yYWdlLWF6dXJlLyIKICAgICAgcmVsYXRpdmVfdG86IGNvbmZpZwogICAgICBtb2RlOiByZWFkCiByZWFjdG9yLmNvcmU6CiAgLSBtYW5hZ2VfdGhyZWFkcwogcmVhY3Rvci5uZXR0eS5jb3JlOgogIC0gbWFuYWdlX3RocmVhZHMK' > ${ES_PATH_CONF}/jvm.options.d/workaround-128046.options

#128047 will fix this properly.

[muecahit94]

Thanks for the fast response.
I have created an extra mount on ${ES_PATH_CONF}/jvm.options.d, added the file and restarted the pods.
Unfortunately it didnt solve the issue, I get still alot of these warnings.
I would anyway wait for the fix to be released, but this workaround doesnt seem to work unfortunately.

{"@timestamp":"2025-05-13T15:31:22.627Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][azure_event_loop][T#1]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.reactor.netty.core","trace.id":"f19a34964c8f8f681d9acc5ec52fc2a0","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [reactor.netty.core], class [class reactor.netty.resources.DefaultLoopResources$EventLoopFactory], entitlement [manage_threads]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFlagEntitlement(PolicyManager.java:632)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkEntitlementPresent(PolicyManager.java:722)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkManageThreadsEntitlement(PolicyManager.java:714)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_lang_Thread$setDaemon(ElasticsearchEntitlementChecker.java:2665)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources$EventLoopFactory.newThread(DefaultLoopResources.java:300)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:955)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:838)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)\n\tat java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:124)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:117)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.ColocatedEventLoopGroup.<init>(ColocatedEventLoopGroup.java:59)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.LoopResources.colocate(LoopResources.java:87)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.cacheNioClientLoops(DefaultLoopResources.java:179)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:147)\n\tat reactor.netty.core@1.0.45/reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)\n\tat reactor.netty.core@1.0.45/reactor.netty.resources.PooledConnectionProvider.lambda$acquire$2(PooledConnectionProvider.java:157)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:273)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58)\n\tat reactor.core@3.4.38/reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)\n\tat reactor.core@3.4.38/reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)\n\tat reactor.core@3.4.38/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)\n\tat reactor.netty.http@1.0.45/reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:276)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.subscribe(Mono.java:4491)\n\tat reactor.core@3.4.38/reactor.core.publisher.Mono.block(Mono.java:1741)\n\tat com.azure.http.netty@1.15.3/com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:195)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpLoggingPolicy.processSync(HttpLoggingPolicy.java:175)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.implementation.http.policy.InstrumentationPolicy.processSync(InstrumentationPolicy.java:101)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)\n\tat com.azure.core@1.51.0/com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)\n\tat com.azure.core@1.51.0/com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:67)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:137)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:52)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:299)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:249)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:360)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:107)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:39)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:23)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:46)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j@1.16.2/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\n\tat io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:28)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T15:31:22.763Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot][T#2]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"c5a04aae756580b1c1eb12bb8a5c0937","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:690)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:518)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:517)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:431)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:126)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3812)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:61)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:38)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:219)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:131)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.executeDirect(TransportAction.java:55)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.HandledTransportAction.lambda$new$0(HandledTransportAction.java:51)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:579)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:632)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:621)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:114)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:201)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.authenticate(ServerTransportFilter.java:127)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:643)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:90)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1098)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}

[muecahit94]

These are the warnings, if I want to list the snapshots from a snapshot-repository, ends in a timeout:

{"@timestamp":"2025-05-13T15:48:27.330Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot_meta][T#19]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"28ff2ad8435cb6c9c12685e67140c53d","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getInputStream(AzureBlobStore.java:345)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.openInputStream(AzureBlobContainer.java:71)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.readBlob(AzureBlobContainer.java:88)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:124)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$getOneSnapshotInfo$27(BlobStoreRepository.java:2017)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T15:48:27.331Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot_meta][T#20]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"28ff2ad8435cb6c9c12685e67140c53d","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getInputStream(AzureBlobStore.java:345)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.openInputStream(AzureBlobContainer.java:71)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.readBlob(AzureBlobContainer.java:88)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:124)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$getOneSnapshotInfo$27(BlobStoreRepository.java:2017)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T15:48:27.333Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot_meta][T#21]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"28ff2ad8435cb6c9c12685e67140c53d","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getInputStream(AzureBlobStore.java:345)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.openInputStream(AzureBlobContainer.java:71)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.readBlob(AzureBlobContainer.java:88)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:124)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$getOneSnapshotInfo$27(BlobStoreRepository.java:2017)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T15:48:27.335Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot_meta][T#22]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"28ff2ad8435cb6c9c12685e67140c53d","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getInputStream(AzureBlobStore.java:345)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.openInputStream(AzureBlobContainer.java:71)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.readBlob(AzureBlobContainer.java:88)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:124)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$getOneSnapshotInfo$27(BlobStoreRepository.java:2017)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}

[DaveCTurner]

The IntelliJCacheAccessor ones are odd, are you running Elasticsearch from within IntelliJ or something? Anyway I think they will be benign, it'll just make this auth method fail but that's not the one you're trying to use anyway.

I'm asking my colleagues for help with the override. I fear that if the workaround doesn't work then the change in #128047 won't fix it either.

[DaveCTurner]

Apologies I think I made a mistake in my YAML formatting, now fixed. Please could you try the workaround in my earlier comment again?

[muecahit94]

Yes, the problem is now partially fixed, everything works faster and more reliable, but the snapshots are still not as fast as with the access-key.

Only the following warnings are left, and no I dont start it in Intelij, its running in docker on a kubernetes cluster, everything set up by ECK-Operator/Stack

{"@timestamp":"2025-05-13T16:12:44.868Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot][T#6]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"c4d89986098f05b96ae2c27789302d56","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:690)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:518)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:517)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:431)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:126)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:121)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2216)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:549)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T16:12:44.944Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot][T#5]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"c4d89986098f05b96ae2c27789302d56","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:690)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:518)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:251)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:46)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:517)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:431)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:126)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3812)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:61)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:38)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:219)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:131)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.executeDirect(TransportAction.java:55)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.HandledTransportAction.lambda$new$0(HandledTransportAction.java:51)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:579)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:632)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:621)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:114)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:201)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.authenticate(ServerTransportFilter.java:127)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:643)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:90)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1098)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}
{"@timestamp":"2025-05-13T16:12:45.019Z", "log.level": "WARN", "message":"Not entitled: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-masters-0][snapshot][T#5]","log.logger":"org.elasticsearch.entitlement.runtime.policy.PolicyManager.repository-azure.com.azure.identity","trace.id":"c4d89986098f05b96ae2c27789302d56","elasticsearch.cluster.uuid":"ID","elasticsearch.node.id":"ID","elasticsearch.node.name":"elasticsearch-es-masters-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.entitlement.runtime.api.NotEntitledException","error.message":"component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]","error.stack_trace":"org.elasticsearch.entitlement.runtime.api.NotEntitledException: component [repository-azure], module [com.azure.identity], class [class com.azure.identity.implementation.IntelliJCacheAccessor], entitlement [file], operation [read], path [/usr/share/elasticsearch/AzureToolsForIntelliJ/AuthMethodDetails.json]\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:690)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:511)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:475)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:454)\n\tat org.elasticsearch.entitlement@9.0.1/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_File$exists(ElasticsearchEntitlementChecker.java:1538)\n\tat java.base/java.io.File.exists(File.java)\n\tat com.azure.identity@1.13.2/com.azure.identity.implementation.IntelliJCacheAccessor.getAuthDetailsIfAvailable(IntelliJCacheAccessor.java:303)\n\tat com.azure.identity@1.13.2/com.azure.identity.IntelliJCredential.<init>(IntelliJCredential.java:92)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:281)\n\tat com.azure.identity@1.13.2/com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:264)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:194)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:100)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:694)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobStore.getInputStream(AzureBlobStore.java:345)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.openInputStream(AzureBlobContainer.java:71)\n\tat org.elasticsearch.repository.azure@9.0.1/org.elasticsearch.repositories.azure.AzureBlobContainer.readBlob(AzureBlobContainer.java:88)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3825)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:61)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.repositories.VerifyNodeRepositoryAction$TransportAction.doExecute(VerifyNodeRepositoryAction.java:38)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:219)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:131)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.TransportAction.executeDirect(TransportAction.java:55)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.support.HandledTransportAction.lambda$new$0(HandledTransportAction.java:51)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:579)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:632)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:621)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:685)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:317)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:114)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:201)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.authenticate(ServerTransportFilter.java:127)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105)\n\tat org.elasticsearch.security@9.0.1/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:643)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:90)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1098)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.server@9.0.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\n"}