Hive: Clean up expired metastore clients #7310

frankliee · 2023-04-09T11:34:04Z

Fixed a bug that could leak hive client connections.

Caffeine cache will not always call removelistener by default, so an extra scheduler is required to invoke removelistener.

See: google/guava#3295.

ConeyLiu · 2023-04-10T09:04:29Z

core/src/main/java/org/apache/iceberg/ClientPoolImpl.java

    return poolSize;
  }
+
+  public boolean isClosed() {


package-private is enough? For UT, we could mark it as VisibleForTesting

ClientPoolImpl and UT are in different packages, so package-private is not enough.
But I will add VisibleForTesting.

cc @chenjunjiedada

@VisibleForTesting + public cannot pass Iceberg's style check, so I use public.

Since boolean isClosed() is read-only function, public will not introduce risks.

Besides, int poolSize() is this class is also public.

chenjunjiedada · 2023-04-10T04:03:35Z

hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java

          Caffeine.newBuilder()
              .expireAfterAccess(evictionInterval, TimeUnit.MILLISECONDS)
              .removalListener((ignored, value, cause) -> ((HiveClientPool) value).close())
+              .scheduler(Scheduler.forScheduledExecutorService(Executors.newScheduledThreadPool(2)))


Nit: Iceberg has a thread pool util ThreadPools.newScheduledPool that you can use.

chenjunjiedada · 2023-04-10T04:50:43Z

core/src/main/java/org/apache/iceberg/ClientPoolImpl.java

    return poolSize;
  }
+
+  public boolean isClosed() {


Do we need this to be public?

See the above comment.

chenjunjiedada

LGTM

nastra · 2023-04-13T07:21:42Z

Caffeine cache will not always call removelistener by default, so an extra scheduler is required to invoke removelistener.

From how I understand how Caffeine does this is that it performs the cleanup whenever the cache is being accessed or modified. Below is an excerpt from the Javadoc:

If expireAfter, expireAfterWrite, or expireAfterAccess is requested then entries may be evicted on each cache modification, on occasional cache accesses, or on calls to Cache.cleanUp. A scheduler(Scheduler) may be specified to provide prompt removal of expired entries rather than waiting until activity triggers the periodic maintenance. Expired entries may be counted by Cache.estimatedSize(), but will never be visible to read or write operations.

That means the removal listener will be called. And this can be seen by modifying the test to

    Awaitility.await()
        .atMost(10, TimeUnit.SECONDS)
        .untilAsserted(
            () -> {
              Assert.assertTrue(clientPool1.isClosed());
              Assert.assertTrue(clientPool2.isClosed());
            });

The question for this PR should rather be how critical it is that the removal is performed immediately vs waiting until activity triggers the periodic maintenance.

frankliee · 2023-04-13T12:15:58Z

Caffeine cache will not always call removelistener by default, so an extra scheduler is required to invoke removelistener.

From how I understand how Caffeine does this is that it performs the cleanup whenever the cache is being accessed or modified. Below is an excerpt from the Javadoc:

If expireAfter, expireAfterWrite, or expireAfterAccess is requested then entries may be evicted on each cache modification, on occasional cache accesses, or on calls to Cache.cleanUp. A scheduler(Scheduler) may be specified to provide prompt removal of expired entries rather than waiting until activity triggers the periodic maintenance. Expired entries may be counted by Cache.estimatedSize(), but will never be visible to read or write operations.

That means the removal listener will be called. And this can be seen by modifying the test to
    Awaitility.await()
        .atMost(10, TimeUnit.SECONDS)
        .untilAsserted(
            () -> {
              Assert.assertTrue(clientPool1.isClosed());
              Assert.assertTrue(clientPool2.isClosed());
            });
The question for this PR should rather be how critical it is that the removal is performed immediately vs waiting until activity triggers the periodic maintenance.

I understand your concerns, and this PR chooses to call removal immediately for the following two reasons.

Caffeine's default stratey will postpone removal as much as possible to reduce overhead. In this Iceberg, removal will be call for each 5 min by default, so the overhead is acceptable.
Iceberg has a conf called "client.pool.cache.eviction-interval-ms" to ensure the time of eviction, but the uncertainty of Caffeine default stratey will make it strange to predict. This PR makes it meaningful.

nastra · 2023-04-13T16:02:55Z

I think it would be good here to get @pvary's opinion as I'm not very familiar with Hive and how critical it is in this particular case that we perform removal immediately

pvary · 2023-04-17T15:11:39Z

I think it would be good here to get @pvary's opinion as I'm not very familiar with Hive and how critical it is in this particular case that we perform removal immediately

I have seen cases with Flink when the open HMS client numbers are kept increasing. I was suspicious about the Caffeine cleanup, but was not able to pin down the the issue. That said, I am not sure the issue was the same.

@frankliee: did you have a concrete issue which needs fixing? What was it?

Thanks, Peter

frankliee · 2023-04-18T02:36:24Z

I think it would be good here to get @pvary's opinion as I'm not very familiar with Hive and how critical it is in this particular case that we perform removal immediately

I have seen cases with Flink when the open HMS client numbers are kept increasing. I was suspicious about the Caffeine cleanup, but was not able to pin down the the issue. That said, I am not sure the issue was the same.

@frankliee: did you have a concrete issue which needs fixing? What was it?

Thanks, Peter

In another project, I found guava cache can leak connections, when the cache will not be "set" for a long time.
So Iceberg has the similar risk.

pvary · 2023-04-18T04:15:04Z

hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java

  private synchronized void init() {
    if (clientPoolCache == null) {
+      // Since Caffeine does not ensure that removalListener will be involved after expiration
+      // We use a scheduler with 2 threads to clean up expired clients.


Why this is size 2? Why not 1?

I agree that this should be rather 1 instead of 2

Could you please update the comment?

Thanks for your correction. I have updated the comment. @pvary

pvary · 2023-04-18T04:15:52Z

hive-metastore/src/test/java/org/apache/iceberg/hive/TestCachedClientPool.java

            .getIfPresent(CachedClientPool.extractKey(null, hiveConf)));
+
+    // The client has been really closed.
+    Assert.assertTrue(clientPool1.isClosed());


Just for the record: Did we test that this test fails without the fix?

I've checked this when reviewing and the test fails on master with these 2 checks added. However, the test passes with some wait time

Awaitility.await() .atMost(10, TimeUnit.SECONDS) .untilAsserted( () -> { Assert.assertTrue(clientPool1.isClosed()); Assert.assertTrue(clientPool2.isClosed()); });

because cache removal will eventually be executed (just not immediately).

Caffeine's doc describes its complicated cleanup behavior, and introduces the scheduler to prompt cleanup.

By default, Caffeine does not perform cleanup and evict values "automatically" or instantly after a value expires. Instead, it performs small amounts of maintenance work after write operations or occasionally after read operations if writes are rare.
( see: https://github.com/ben-manes/caffeine/wiki/Cleanup )

The unpredictability of removal could increase the pressure on HMS, when there are many applications (including Flink and Spark).

chenjunjiedada · 2023-04-25T04:00:42Z

@nastra @pvary Any more options on this? Is it ready to go?

pvary · 2023-04-26T15:20:18Z

Thanks for the review @chenjunjiedada, @nastra, @ConeyLiu for the review, and @frankliee for the PR

github-actions bot added core hive labels Apr 9, 2023

frankliee force-pushed the hive-cache branch from 6c8838e to 94fca9a Compare April 9, 2023 11:37

Hive: Clean up expired metastore clients to avoid zombie connections

785fd77

frankliee force-pushed the hive-cache branch from 94fca9a to 785fd77 Compare April 10, 2023 04:02

ConeyLiu reviewed Apr 10, 2023

View reviewed changes

chenjunjiedada reviewed Apr 10, 2023

View reviewed changes

fix comment

58a29cc

frankliee force-pushed the hive-cache branch from 2967832 to 58a29cc Compare April 11, 2023 02:16

chenjunjiedada approved these changes Apr 11, 2023

View reviewed changes

chenjunjiedada requested review from nastra and pvary April 13, 2023 03:40

pvary reviewed Apr 18, 2023

View reviewed changes

fix comment

d4c456e

fix comment

8b49176

pvary merged commit bbacaf4 into apache:master Apr 26, 2023

manisin pushed a commit to Snowflake-Labs/iceberg that referenced this pull request May 9, 2023

Hive: Clean up expired metastore clients (apache#7310)

5ba743b

Hive: Clean up expired metastore clients #7310

Hive: Clean up expired metastore clients #7310

Uh oh!

Conversation

frankliee commented Apr 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankliee Apr 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjunjiedada left a comment

Choose a reason for hiding this comment

Uh oh!

nastra commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

frankliee commented Apr 13, 2023

Uh oh!

nastra commented Apr 13, 2023

Uh oh!

pvary commented Apr 17, 2023

Uh oh!

frankliee commented Apr 18, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankliee Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjunjiedada commented Apr 25, 2023

Uh oh!

pvary commented Apr 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

frankliee Apr 10, 2023 •

edited

Loading

nastra commented Apr 13, 2023 •

edited

Loading

frankliee Apr 26, 2023 •

edited

Loading

nastra Apr 18, 2023 •

edited

Loading