[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor

[zhangwl9]

Why are the changes needed?

Close #4037.

Brief change log

• The new feature involves two methods in "TableRuntimeRefreshExecutor".
  • #getNextExecutingTime()
    • When dynamic refresh interval is enabled，retrieving the latest refresh interval from "DefaultTableRuntime" to determine the next execution time for "TableRuntimeRefreshExecutor#execute"
  • #execute()
    • Depending on the evaluated results, determine whether the table requires optimization.
    • When the evaluation table requires no optimization, extend the interval for its next refresh (incrementally increasing the interval by a fixed step).
    • When evaluating table requiring optimization, shorten its next refresh interval (halving the interval each time).
• dynamic refresh interval in [ refresh-tables.interval(1 min)，max_interval(1 hour)]

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[zhangwl9]

@zhoujinsong @turboFei @xxubai Could you help me review it when you are free? Thanks!

[Copilot]

Pull request overview

This PR implements a dynamic refresh interval feature for table metadata refreshing in TableRuntimeRefreshExecutor. The feature uses an AIMD (Additive Increase Multiplicative Decrease) algorithm to adjust refresh intervals based on whether tables need optimization, helping to reduce resource consumption for healthy tables while maintaining responsiveness for tables requiring optimization.

Changes:

• Added configuration properties for adaptive refresh intervals with configurable maximum interval and increase step
• Implemented adaptive interval calculation logic that increases intervals for healthy tables and decreases them for tables needing optimization
• Added comprehensive test coverage for the adaptive interval functionality

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/user-guides/configurations.md	Added documentation for three new configuration properties related to adaptive refresh intervals
amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java	Defined constants for adaptive refresh table properties and their default values
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java	Added fields, getters, setters, and updated equals/hashCode/toString methods for adaptive refresh configuration
amoro-ams/src/main/java/org/apache/amoro/server/table/TableConfigurations.java	Integrated parsing of adaptive refresh properties from table properties into OptimizingConfig
amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableRuntime.java	Added fields and methods to track latest refresh interval and optimization evaluation status
amoro-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java	Implemented core adaptive interval logic with AIMD algorithm and updated execute method to apply adaptive intervals
amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java	Added comprehensive test cases covering various adaptive interval scenarios including boundary conditions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.